2024-04-16T02:14:26.094 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T02:14:26.094 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi --verbose --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416 -- /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416/orig.config.yaml 2024-04-16T02:14:26.150 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T02:14:26.250 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416 branch: main description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7657416' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.1 ' name: teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 3f7db60e5ba4c4d053b594b7ec58703d2833c55b owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8042 sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 sleep_before_teardown: 0 subset: 25/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 3f7db60e5ba4c4d053b594b7ec58703d2833c55b targets: smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDcdvOwGEwiTGWVVq3/lH/SMtW1QGOfmchUl+HmD/ASByfF6XNvi0iddJC9FioCIYXYIjr7IqkS4umjtvk+qDJQ= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOifDiy5cEQMY0nTfO0+g7qJXRNVwrokpMt23C0FmhTNikwXD+9wRdztbEptjIYY2T0+SX93q7RVH8dcfYwC918= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.1 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.1 roleless: true - print: '**** done end installing v18.2.1 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-15_20:08:15 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-16T02:14:26.250 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa; will attempt to use it 2024-04-16T02:14:26.251 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks 2024-04-16T02:14:26.251 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T02:14:26.253 INFO:teuthology.task.internal:Checking packages... 2024-04-16T02:14:26.280 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '87dbfb632e974cb2c28d17423071d2d833b53b29' 2024-04-16T02:14:26.281 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T02:14:26.281 INFO:teuthology.packaging:ref: None 2024-04-16T02:14:26.281 INFO:teuthology.packaging:tag: None 2024-04-16T02:14:26.281 INFO:teuthology.packaging:branch: main 2024-04-16T02:14:26.281 INFO:teuthology.packaging:sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:14:26.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-16T02:14:26.623 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3100.g3f7db60e 2024-04-16T02:14:26.625 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T02:14:26.634 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T02:14:26.634 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T02:14:26.656 INFO:teuthology.task.internal:Saving configuration 2024-04-16T02:14:26.677 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T02:14:26.688 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T02:14:26.715 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 02:09:17.255584', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDcdvOwGEwiTGWVVq3/lH/SMtW1QGOfmchUl+HmD/ASByfF6XNvi0iddJC9FioCIYXYIjr7IqkS4umjtvk+qDJQ='} 2024-04-16T02:14:26.736 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 02:09:17.257462', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOifDiy5cEQMY0nTfO0+g7qJXRNVwrokpMt23C0FmhTNikwXD+9wRdztbEptjIYY2T0+SX93q7RVH8dcfYwC918='} 2024-04-16T02:14:26.736 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T02:14:26.752 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-16T02:14:26.752 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-16T02:14:26.752 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T02:14:26.832 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f40cf4320d0>, signals=[15]) 2024-04-16T02:14:26.832 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T02:14:26.838 INFO:teuthology.task.internal:Opening connections... 2024-04-16T02:14:26.838 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:14:26.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:14:26.971 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-04-16T02:14:26.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:14:27.061 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T02:14:27.070 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-04-16T02:14:27.088 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-04-16T02:14:27.089 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-04-16T02:14:27.146 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T02:14:27.147 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-04-16T02:14:27.147 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T02:14:27.147 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T02:14:27.147 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T02:14:27.147 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-04-16T02:14:27.180 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-04-16T02:14:27.198 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-04-16T02:14:27.198 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-04-16T02:14:27.256 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-04-16T02:14:27.256 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-04-16T02:14:27.256 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-04-16T02:14:27.257 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-04-16T02:14:27.257 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-04-16T02:14:27.257 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-04-16T02:14:27.257 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T02:14:27.257 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-04-16T02:14:27.257 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-04-16T02:14:27.258 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T02:14:27.258 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-04-16T02:14:27.258 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T02:14:27.258 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T02:14:27.258 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T02:14:27.259 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-04-16T02:14:27.280 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T02:14:27.296 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T02:14:27.304 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T02:14:27.304 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-04-16T02:14:27.307 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-04-16T02:14:27.323 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T02:14:27.331 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T02:14:27.331 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-04-16T02:14:27.364 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-04-16T02:14:27.469 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T02:14:27.526 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T02:14:27.542 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-16T02:14:27.542 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-16T02:14:27.543 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T02:14:27.543 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T02:14:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T02:14:27.559 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-04-16T02:14:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T02:14:27.560 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-04-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-361.el9.x86_64 2024-04-16T02:14:27.616 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-361.el9.x86_64 2024-04-16T02:14:27.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-04-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-361.el9.x86_64 2024-04-16T02:14:27.617 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-361.el9.x86_64 2024-04-16T02:14:27.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-04-16T02:14:29.966 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-16T02:14:29.976 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-16T02:14:31.541 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-04-16T02:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-04-16T02:14:33.807 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-04-16T02:14:35.742 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-16T02:14:39.784 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-16T02:14:41.243 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-04-16T02:14:42.172 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 804 kB/s | 8.0 MB 00:10 2024-04-16T02:14:43.272 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-16T02:14:44.106 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-16T02:14:48.469 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-16T02:14:48.624 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-04-16T02:14:50.174 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-04-16T02:14:50.827 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T02:14:50.827 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T02:14:50.933 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:14:50.948 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:14:50.949 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:14:50.950 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:14:50.950 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-04-16T02:14:50.950 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:14:50.950 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 77 M 2024-04-16T02:14:50.950 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 123 M 2024-04-16T02:14:50.951 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.4 MB/s | 277 kB 00:00 2024-04-16T02:14:51.988 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 15 MB/s | 15 MB 00:01 2024-04-16T02:14:52.109 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-16T02:14:52.588 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-16T02:14:52.938 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-04-16T02:14:52.941 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:14:52.941 INFO:teuthology.orchestra.run.smithi151.stdout:Total 39 MB/s | 77 MB 00:01 2024-04-16T02:14:52.942 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:14:53.305 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:14:53.306 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:14:54.188 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:14:54.188 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:14:56.546 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:14:57.175 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T02:14:57.389 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T02:14:59.532 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T02:14:59.532 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T02:14:59.621 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:14:59.632 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:14:59.632 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-16T02:14:59.632 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:14:59.632 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:14:59.632 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T02:14:59.632 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-16T02:14:59.632 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T02:14:59.633 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T02:14:59.633 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T02:14:59.633 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:14:59.633 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:14:59.633 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:14:59.633 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-04-16T02:14:59.633 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:14:59.634 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 77 M 2024-04-16T02:14:59.634 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 123 M 2024-04-16T02:14:59.634 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:14:59.801 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.6 MB/s | 277 kB 00:00 2024-04-16T02:15:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T02:15:00.244 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 25 MB/s | 15 MB 00:00 2024-04-16T02:15:00.274 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T02:15:00.877 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 26 MB/s | 28 MB 00:01 2024-04-16T02:15:01.194 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-04-16T02:15:01.197 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:15:01.197 INFO:teuthology.orchestra.run.smithi153.stdout:Total 49 MB/s | 77 MB 00:01 2024-04-16T02:15:01.197 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:15:01.595 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:15:01.596 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:15:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:15:05.025 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T02:15:05.140 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:05.587 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T02:15:05.677 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T02:15:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:13.357 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T02:15:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:18.191 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:39.315 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:39.315 INFO:teuthology.orchestra.run.smithi151.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T02:15:39.315 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:15:39.358 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:39.737 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:39.737 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T02:15:39.737 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T02:15:39.737 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T02:15:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:15:40.167 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:15:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T02:15:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T02:15:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T02:15:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T02:15:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:15:40.167 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:15:40.483 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T02:15:40.578 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T02:15:40.579 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T02:15:40.579 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T02:15:40.579 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T02:15:40.579 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T02:15:40.579 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-16T02:15:40.579 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T02:15:40.579 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T02:15:40.660 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T02:15:40.661 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T02:15:40.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2024-04-16T02:15:40.694 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2024-04-16T02:15:40.694 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2024-04-16T02:15:40.694 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 68.el9 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8264254 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2024-04-16T02:15:40.695 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T02:15:40.696 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2024-04-16T02:15:40.696 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2024-04-16T02:15:40.696 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:15:40.696 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T02:15:40.696 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T02:15:40.696 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T02:15:40.696 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2024-04-16T02:15:40.696 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:15:40.696 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T02:15:40.697 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T02:15:40.697 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-438.el9.x86_64 2024-04-16T02:15:40.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T02:15:41.228 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-04-16T02:15:42.647 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-04-16T02:15:42.703 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-04-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.VPgLZpjIzP 2024-04-16T02:15:42.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VPgLZpjIzP 2024-04-16T02:15:42.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.VPgLZpjIzP 2024-04-16T02:15:42.929 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.VPgLZpjIzP is 6KB 2024-04-16T02:15:42.977 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.VPgLZpjIzP 2024-04-16T02:15:42.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T02:15:43.056 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T02:15:43.056 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T02:15:43.056 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T02:15:43.056 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T02:15:43.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T02:15:43.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-04-16T02:15:47.336 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:47.336 INFO:teuthology.orchestra.run.smithi153.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T02:15:47.336 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:15:47.379 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:47.781 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:47.781 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T02:15:47.781 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T02:15:47.781 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:15:48.471 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T02:15:48.569 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T02:15:48.569 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T02:15:48.569 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T02:15:48.569 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T02:15:48.570 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T02:15:48.570 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-16T02:15:48.570 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T02:15:48.570 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T02:15:48.650 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T02:15:48.651 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T02:15:48.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-04-16T02:15:48.684 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-04-16T02:15:48.684 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-04-16T02:15:48.684 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-04-16T02:15:48.684 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-04-16T02:15:48.685 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:15:48.686 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T02:15:48.686 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T02:15:48.686 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T02:15:48.686 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-04-16T02:15:48.686 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:15:48.686 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T02:15:48.687 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T02:15:48.687 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-438.el9.x86_64 2024-04-16T02:15:48.687 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T02:15:49.301 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-04-16T02:15:50.721 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T02:15:50.756 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-04-16T02:15:50.758 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-04-16T02:15:50.772 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.zFILLhIA41 2024-04-16T02:15:50.773 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zFILLhIA41 2024-04-16T02:15:50.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.zFILLhIA41 2024-04-16T02:15:50.984 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.zFILLhIA41 is 6KB 2024-04-16T02:15:51.033 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.zFILLhIA41 2024-04-16T02:15:51.048 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T02:15:51.112 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T02:15:51.112 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T02:15:51.112 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T02:15:51.113 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T02:15:51.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T02:15:51.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-04-16T02:16:13.230 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-16T02:16:13.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T02:16:13.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:16:21.302 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-16T02:16:21.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-16T02:16:21.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:16:31.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-04-16T02:16:40.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T02:16:40.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:16:43.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-04-16T02:16:55.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-04-16T02:16:55.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T02:16:55.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:17:04.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-16T02:17:04.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:17:23.234 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-04-16T02:17:35.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-16T02:17:35.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:17:35.710 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-16T02:17:36.108 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-16T02:17:36.108 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-438.el9.x86_64"... 2024-04-16T02:17:36.109 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-04-16T02:17:36.123 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-438.el9.x86_64 2024-04-16T02:17:36.123 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T02:17:36.123 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T02:17:36.124 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T02:17:37.124 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-16T02:17:37.125 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T02:17:37.189 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-04-16T02:17:37.226 DEBUG:teuthology.parallel:result is None 2024-04-16T02:17:55.842 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T02:18:10.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T02:18:10.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:18:11.212 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-04-16T02:18:11.603 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T02:18:11.603 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T02:18:11.603 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-04-16T02:18:11.619 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-438.el9.x86_64 2024-04-16T02:18:11.619 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T02:18:11.619 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T02:18:11.619 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T02:18:12.620 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-16T02:18:12.621 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T02:18:12.687 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-04-16T02:18:12.726 DEBUG:teuthology.parallel:result is None 2024-04-16T02:18:12.727 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T02:18:12.736 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T02:18:12.736 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T02:18:12.738 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T02:18:12.758 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T02:18:12.769 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T02:18:12.777 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T02:18:12.777 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T02:18:12.797 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T02:18:12.827 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T02:18:12.833 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T02:18:12.833 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T02:18:12.863 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T02:18:12.897 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T02:18:12.907 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T02:18:12.911 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T02:18:12.923 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T02:18:12.925 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T02:18:12.931 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T02:18:12.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T02:18:12.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T02:18:12.986 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T02:18:12.995 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T02:18:12.995 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T02:18:13.026 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T02:18:13.043 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T02:18:13.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T02:18:13.175 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T02:18:13.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T02:18:13.302 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:18:13.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T02:18:13.370 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T02:18:13.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T02:18:13.471 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T02:18:13.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T02:18:13.599 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:18:13.599 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T02:18:13.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-04-16T02:18:13.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-04-16T02:18:13.728 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T02:18:13.763 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T02:18:14.047 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T02:18:14.055 INFO:teuthology.task.internal:Starting timer... 2024-04-16T02:18:14.055 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T02:18:14.065 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T02:18:14.073 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-16T02:18:14.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-04-16T02:18:14.116 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-04-16T02:18:14.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-04-16T02:18:14.184 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-04-16T02:18:14.186 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T02:18:14.186 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-04-16T02:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-04-16T02:18:14.211 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-04-16T02:18:14.240 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-04-16T02:18:14.241 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi151.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-04-16T02:18:14.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T02:18:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T02:18:14.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T02:18:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T02:18:14.310 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T02:18:14.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-04-16T02:18:14.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-04-16T02:18:14.393 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T02:18:14.402 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-16T02:18:14.407 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-16T02:18:14.527 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T02:18:14.543 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T02:18:14.544 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi151.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T02:22:36.890 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi151.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-04-16T02:22:36.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T02:22:36.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:22:36.982 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-04-16T02:22:37.062 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T02:22:37.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-16T02:22:37.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T02:22:37.152 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-16T02:22:37.238 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-16T02:22:37.238 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T02:22:37.249 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T02:22:37.249 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T02:22:37.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T02:22:37.252 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T02:22:37.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T02:22:37.282 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T02:22:37.300 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T02:22:37.313 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T02:22:37.331 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T02:22:37.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-04-16T02:22:37.346 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-04-16T02:22:37.363 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T02:22:37.363 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-04-16T02:22:37.378 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-04-16T02:22:37.380 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T02:22:37.396 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T02:22:37.413 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T02:22:37.439 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-04-16T02:22:37.442 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T02:22:37.442 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-04-16T02:22:37.442 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T02:22:37.443 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T02:22:37.443 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T02:22:37.443 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T02:22:37.466 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-16T02:22:37.469 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T02:22:37.469 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-16T02:22:37.469 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T02:22:37.469 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T02:22:37.470 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T02:22:37.470 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T02:22:37.471 INFO:teuthology.run_tasks:Running task install... 2024-04-16T02:22:37.479 DEBUG:teuthology.task.install:project ceph 2024-04-16T02:22:37.480 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '87dbfb632e974cb2c28d17423071d2d833b53b29'}} 2024-04-16T02:22:37.480 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.1', 'flavor': 'default', 'sha1': '87dbfb632e974cb2c28d17423071d2d833b53b29'} 2024-04-16T02:22:37.480 INFO:teuthology.task.install:Using flavor: default 2024-04-16T02:22:37.488 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T02:22:37.488 INFO:teuthology.task.install:extra packages: [] 2024-04-16T02:22:37.488 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '87dbfb632e974cb2c28d17423071d2d833b53b29', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-16T02:22:37.488 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-16T02:22:37.488 INFO:teuthology.packaging:ref: None 2024-04-16T02:22:37.488 INFO:teuthology.packaging:tag: v18.2.1 2024-04-16T02:22:37.488 INFO:teuthology.packaging:branch: None 2024-04-16T02:22:37.489 INFO:teuthology.packaging:sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:22:39.833 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.1^{} -> 7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-16T02:22:39.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-16T02:22:39.836 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '87dbfb632e974cb2c28d17423071d2d833b53b29', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-16T02:22:39.837 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-16T02:22:39.837 INFO:teuthology.packaging:ref: None 2024-04-16T02:22:39.837 INFO:teuthology.packaging:tag: v18.2.1 2024-04-16T02:22:39.837 INFO:teuthology.packaging:branch: None 2024-04-16T02:22:39.837 INFO:teuthology.packaging:sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:22:39.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-16T02:22:40.060 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-04-16T02:22:40.060 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-16T02:22:40.063 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-04-16T02:22:40.064 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-16T02:22:40.103 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T02:22:40.103 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:22:40.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T02:22:40.115 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T02:22:40.115 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:22:40.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T02:22:40.138 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T02:22:40.138 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-16T02:22:40.138 INFO:teuthology.packaging:ref: None 2024-04-16T02:22:40.138 INFO:teuthology.packaging:tag: v18.2.1 2024-04-16T02:22:40.138 INFO:teuthology.packaging:branch: None 2024-04-16T02:22:40.138 INFO:teuthology.packaging:sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:22:40.138 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T02:22:40.149 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T02:22:40.150 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-16T02:22:40.150 INFO:teuthology.packaging:ref: None 2024-04-16T02:22:40.150 INFO:teuthology.packaging:tag: v18.2.1 2024-04-16T02:22:40.150 INFO:teuthology.packaging:branch: None 2024-04-16T02:22:40.150 INFO:teuthology.packaging:sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:22:40.150 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T02:22:40.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T02:22:40.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T02:22:40.298 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T02:22:40.307 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T02:22:40.326 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2024-04-16T02:22:40.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2024-04-16T02:22:40.335 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-04-16T02:22:40.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-04-16T02:22:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:74 files removed 2024-04-16T02:22:40.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw 2024-04-16T02:22:40.641 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-04-16T02:22:40.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-04-16T02:22:41.108 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 417 kB/s | 76 kB 00:00 2024-04-16T02:22:41.118 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 522 kB/s | 76 kB 00:00 2024-04-16T02:22:41.319 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 62 kB/s | 9.4 kB 00:00 2024-04-16T02:22:41.341 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 51 kB/s | 9.4 kB 00:00 2024-04-16T02:22:41.509 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 17 kB/s | 2.2 kB 00:00 2024-04-16T02:22:41.511 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 14 kB/s | 2.2 kB 00:00 2024-04-16T02:22:41.978 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-04-16T02:22:41.980 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-04-16T02:22:43.447 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-04-16T02:22:43.470 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-04-16T02:22:44.236 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-04-16T02:22:47.953 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-04-16T02:22:52.223 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.2 MB 00:00 2024-04-16T02:22:53.103 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 883 kB/s | 8.0 MB 00:09 2024-04-16T02:22:54.139 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-16T02:22:55.856 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-16T02:23:00.873 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-16T02:23:00.997 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 19 MB 00:06 2024-04-16T02:23:03.600 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:23:03.627 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:03.627 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-16T02:23:03.628 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T02:23:03.629 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:03.630 INFO:teuthology.orchestra.run.smithi151.stdout:Install 28 Packages 2024-04-16T02:23:03.631 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2024-04-16T02:23:03.631 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:03.631 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 53 M 2024-04-16T02:23:03.631 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:23:03.890 INFO:teuthology.orchestra.run.smithi151.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 32 MB/s | 5.2 MB 00:00 2024-04-16T02:23:03.923 INFO:teuthology.orchestra.run.smithi151.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 723 kB/s | 24 kB 00:00 2024-04-16T02:23:03.957 INFO:teuthology.orchestra.run.smithi151.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-04-16T02:23:03.991 INFO:teuthology.orchestra.run.smithi151.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-16T02:23:04.091 INFO:teuthology.orchestra.run.smithi151.stdout:(5/30): librgw2-18.2.1-0.el9.x86_64.rpm 45 MB/s | 4.5 MB 00:00 2024-04-16T02:23:04.177 INFO:teuthology.orchestra.run.smithi151.stdout:(6/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 17 MB/s | 7.7 MB 00:00 2024-04-16T02:23:04.192 INFO:teuthology.orchestra.run.smithi151.stdout:(7/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 448 kB/s | 45 kB 00:00 2024-04-16T02:23:04.217 INFO:teuthology.orchestra.run.smithi151.stdout:(8/30): python3-ceph-common-18.2.1-0.el9.x86_64 2.9 MB/s | 124 kB 00:00 2024-04-16T02:23:04.242 INFO:teuthology.orchestra.run.smithi151.stdout:(9/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 3.1 MB/s | 161 kB 00:00 2024-04-16T02:23:04.268 INFO:teuthology.orchestra.run.smithi151.stdout:(10/30): python3-rados-18.2.1-0.el9.x86_64.rpm 6.2 MB/s | 321 kB 00:00 2024-04-16T02:23:04.293 INFO:teuthology.orchestra.run.smithi151.stdout:(11/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 5.8 MB/s | 297 kB 00:00 2024-04-16T02:23:04.318 INFO:teuthology.orchestra.run.smithi151.stdout:(12/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-04-16T02:23:04.577 INFO:teuthology.orchestra.run.smithi151.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-04-16T02:23:04.602 INFO:teuthology.orchestra.run.smithi151.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 343 kB/s | 106 kB 00:00 2024-04-16T02:23:04.627 INFO:teuthology.orchestra.run.smithi151.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 558 kB/s | 159 kB 00:00 2024-04-16T02:23:04.652 INFO:teuthology.orchestra.run.smithi151.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 589 kB/s | 44 kB 00:00 2024-04-16T02:23:04.702 INFO:teuthology.orchestra.run.smithi151.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-04-16T02:23:04.728 INFO:teuthology.orchestra.run.smithi151.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-04-16T02:23:04.761 INFO:teuthology.orchestra.run.smithi151.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 380 kB/s | 41 kB 00:00 2024-04-16T02:23:04.928 INFO:teuthology.orchestra.run.smithi151.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 492 kB/s | 111 kB 00:00 2024-04-16T02:23:04.979 INFO:teuthology.orchestra.run.smithi151.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2024-04-16T02:23:05.004 INFO:teuthology.orchestra.run.smithi151.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-04-16T02:23:05.029 INFO:teuthology.orchestra.run.smithi151.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-04-16T02:23:05.063 INFO:teuthology.orchestra.run.smithi151.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-16T02:23:05.146 INFO:teuthology.orchestra.run.smithi151.stdout:(25/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-16T02:23:05.172 INFO:teuthology.orchestra.run.smithi151.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-16T02:23:05.247 INFO:teuthology.orchestra.run.smithi151.stdout:(27/30): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-04-16T02:23:05.297 INFO:teuthology.orchestra.run.smithi151.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-16T02:23:05.364 INFO:teuthology.orchestra.run.smithi151.stdout:(29/30): librados2-18.2.1-0.el9.x86_64.rpm 17 MB/s | 3.3 MB 00:00 2024-04-16T02:23:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:(30/30): librbd1-18.2.1-0.el9.x86_64.rpm 17 MB/s | 3.0 MB 00:00 2024-04-16T02:23:05.427 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:05.428 INFO:teuthology.orchestra.run.smithi151.stdout:Total 29 MB/s | 53 MB 00:01 2024-04-16T02:23:05.428 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:23:05.466 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:23:05.466 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:23:05.921 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:23:05.921 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:23:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.2 MB 00:01 2024-04-16T02:23:07.328 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:23:07.422 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-16T02:23:07.452 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-16T02:23:07.503 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-16T02:23:07.506 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-04-16T02:23:07.870 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-16T02:23:07.909 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-16T02:23:08.002 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-16T02:23:08.081 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-04-16T02:23:08.128 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-16T02:23:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-16T02:23:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-16T02:23:08.262 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-04-16T02:23:08.328 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-16T02:23:08.354 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-16T02:23:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-16T02:23:08.526 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-16T02:23:08.554 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-16T02:23:08.594 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-16T02:23:08.626 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-16T02:23:08.645 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-16T02:23:08.681 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-16T02:23:08.713 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-16T02:23:09.012 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-16T02:23:09.043 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-16T02:23:09.133 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-16T02:23:09.165 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-04-16T02:23:09.200 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-16T02:23:09.601 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-16T02:23:09.629 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-16T02:23:09.651 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-16T02:23:09.700 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-16T02:23:09.776 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-04-16T02:23:09.806 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-04-16T02:23:11.836 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-16T02:23:11.869 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-16T02:23:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-16T02:23:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-16T02:23:12.573 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-16T02:23:12.573 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T02:23:12.573 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T02:23:12.573 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:12.598 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-16T02:23:14.699 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-16T02:23:17.434 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:23:17.446 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:23:17.446 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-16T02:23:17.446 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T02:23:17.447 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-04-16T02:23:17.448 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout:Install 28 Packages 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-04-16T02:23:17.449 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:17.450 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 53 M 2024-04-16T02:23:17.450 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:23:17.700 INFO:teuthology.orchestra.run.smithi153.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 32 MB/s | 5.2 MB 00:00 2024-04-16T02:23:17.717 INFO:teuthology.orchestra.run.smithi153.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 1.4 MB/s | 24 kB 00:00 2024-04-16T02:23:17.750 INFO:teuthology.orchestra.run.smithi153.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-04-16T02:23:17.784 INFO:teuthology.orchestra.run.smithi153.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-16T02:23:17.901 INFO:teuthology.orchestra.run.smithi153.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 21 MB/s | 7.7 MB 00:00 2024-04-16T02:23:17.926 INFO:teuthology.orchestra.run.smithi153.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-04-16T02:23:17.968 INFO:teuthology.orchestra.run.smithi153.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 2.9 MB/s | 124 kB 00:00 2024-04-16T02:23:17.994 INFO:teuthology.orchestra.run.smithi153.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 6.3 MB/s | 161 kB 00:00 2024-04-16T02:23:18.019 INFO:teuthology.orchestra.run.smithi153.stdout:(9/30): python3-rados-18.2.1-0.el9.x86_64.rpm 13 MB/s | 321 kB 00:00 2024-04-16T02:23:18.094 INFO:teuthology.orchestra.run.smithi153.stdout:(10/30): librgw2-18.2.1-0.el9.x86_64.rpm 14 MB/s | 4.5 MB 00:00 2024-04-16T02:23:18.120 INFO:teuthology.orchestra.run.smithi153.stdout:(11/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 2.9 MB/s | 297 kB 00:00 2024-04-16T02:23:18.145 INFO:teuthology.orchestra.run.smithi153.stdout:(12/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-04-16T02:23:18.395 INFO:teuthology.orchestra.run.smithi153.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 21 MB/s | 18 MB 00:00 2024-04-16T02:23:18.429 INFO:teuthology.orchestra.run.smithi153.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 343 kB/s | 106 kB 00:00 2024-04-16T02:23:18.454 INFO:teuthology.orchestra.run.smithi153.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 764 kB/s | 44 kB 00:00 2024-04-16T02:23:18.479 INFO:teuthology.orchestra.run.smithi153.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 476 kB/s | 159 kB 00:00 2024-04-16T02:23:18.638 INFO:teuthology.orchestra.run.smithi153.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 661 kB 00:00 2024-04-16T02:23:18.714 INFO:teuthology.orchestra.run.smithi153.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 282 kB 00:00 2024-04-16T02:23:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 159 kB/s | 41 kB 00:00 2024-04-16T02:23:18.914 INFO:teuthology.orchestra.run.smithi153.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 402 kB/s | 111 kB 00:00 2024-04-16T02:23:19.030 INFO:teuthology.orchestra.run.smithi153.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 981 kB/s | 309 kB 00:00 2024-04-16T02:23:19.106 INFO:teuthology.orchestra.run.smithi153.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 135 kB/s | 26 kB 00:00 2024-04-16T02:23:19.131 INFO:teuthology.orchestra.run.smithi153.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 491 kB/s | 49 kB 00:00 2024-04-16T02:23:19.256 INFO:teuthology.orchestra.run.smithi153.stdout:(24/30): libarrow-9.0.0-10.el9.x86_64.rpm 8.5 MB/s | 4.4 MB 00:00 2024-04-16T02:23:19.281 INFO:teuthology.orchestra.run.smithi153.stdout:(25/30): libunwind-1.6.2-1.el9.x86_64.rpm 457 kB/s | 67 kB 00:00 2024-04-16T02:23:19.307 INFO:teuthology.orchestra.run.smithi153.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-04-16T02:23:19.382 INFO:teuthology.orchestra.run.smithi153.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 44 MB/s | 3.3 MB 00:00 2024-04-16T02:23:19.449 INFO:teuthology.orchestra.run.smithi153.stdout:(28/30): librbd1-18.2.1-0.el9.x86_64.rpm 45 MB/s | 3.0 MB 00:00 2024-04-16T02:23:19.474 INFO:teuthology.orchestra.run.smithi153.stdout:(29/30): re2-20211101-3.el9.x86_64.rpm 889 kB/s | 192 kB 00:00 2024-04-16T02:23:19.583 INFO:teuthology.orchestra.run.smithi153.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 5.7 MB/s | 1.6 MB 00:00 2024-04-16T02:23:19.588 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:19.588 INFO:teuthology.orchestra.run.smithi153.stdout:Total 25 MB/s | 53 MB 00:02 2024-04-16T02:23:19.589 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:23:19.625 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:23:19.626 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:23:20.067 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:23:20.068 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:23:21.456 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:23:21.537 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-16T02:23:21.579 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-16T02:23:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-16T02:23:22.001 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-16T02:23:22.037 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-16T02:23:22.145 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-16T02:23:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-04-16T02:23:22.263 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-16T02:23:22.326 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-16T02:23:22.354 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-16T02:23:22.407 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-04-16T02:23:22.440 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-16T02:23:22.440 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2024-04-16T02:23:22.440 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2024-04-16T02:23:22.440 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2024-04-16T02:23:22.440 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2024-04-16T02:23:22.441 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2024-04-16T02:23:22.441 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2024-04-16T02:23:22.441 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2024-04-16T02:23:22.441 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:22.473 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-16T02:23:22.499 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-16T02:23:22.557 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-16T02:23:22.663 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-16T02:23:22.698 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-16T02:23:22.739 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-16T02:23:22.788 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-16T02:23:22.815 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-16T02:23:22.851 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-16T02:23:22.884 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-16T02:23:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-16T02:23:23.203 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-16T02:23:23.213 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-16T02:23:23.229 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-16T02:23:23.230 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:23.230 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T02:23:23.230 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T02:23:23.230 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T02:23:23.230 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:23.232 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-16T02:23:23.304 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-16T02:23:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-16T02:23:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-16T02:23:23.334 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-04-16T02:23:23.371 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-16T02:23:23.774 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-16T02:23:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-16T02:23:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-16T02:23:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-04-16T02:23:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-04-16T02:23:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-16T02:23:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-16T02:23:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-16T02:23:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-16T02:23:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-16T02:23:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-16T02:23:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-04-16T02:23:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-16T02:23:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-04-16T02:23:25.506 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T02:23:25.507 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.508 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.509 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T02:23:25.509 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.509 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.509 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:25.509 INFO:teuthology.orchestra.run.smithi151.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T02:23:25.509 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T02:23:25.509 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:25.509 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:23:25.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test 2024-04-16T02:23:25.966 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-16T02:23:25.998 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-16T02:23:26.432 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:26 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:23:26.568 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-16T02:23:26.592 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:23:26.593 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:26.593 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T02:23:26.593 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-04-16T02:23:26.594 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:26.595 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 41 M 2024-04-16T02:23:26.595 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 167 M 2024-04-16T02:23:26.595 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:23:26.609 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-16T02:23:26.688 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-16T02:23:26.688 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T02:23:26.688 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T02:23:26.688 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:26.727 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-16T02:23:26.862 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 165 kB 00:00 2024-04-16T02:23:26.920 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-04-16T02:23:26.946 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-04-16T02:23:26.971 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-16T02:23:27.463 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): ceph-test-18.2.1-0.el9.x86_64.rpm 54 MB/s | 40 MB 00:00 2024-04-16T02:23:27.555 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): jq-1.6-16.el9.x86_64.rpm 307 kB/s | 187 kB 00:00 2024-04-16T02:23:27.558 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:27.558 INFO:teuthology.orchestra.run.smithi151.stdout:Total 42 MB/s | 41 MB 00:00 2024-04-16T02:23:27.574 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:23:27.596 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:23:27.596 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:23:27.933 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:23:27.933 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:23:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:23:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T02:23:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T02:23:28.639 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T02:23:28.690 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T02:23:28.718 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-04-16T02:23:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-04-16T02:23:32.568 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-04-16T02:23:34.789 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-04-16T02:23:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-04-16T02:23:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-04-16T02:23:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T02:23:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T02:23:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T02:23:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T02:23:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:35.128 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:23:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-04-16T02:23:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T02:23:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T02:23:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:35.128 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:23:35.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph 2024-04-16T02:23:36.021 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:36 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:23:36.176 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:23:36.178 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T02:23:36.178 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:36.178 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:23:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-04-16T02:23:36.178 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T02:23:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-04-16T02:23:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T02:23:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:36.180 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:23:36.181 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:36.181 INFO:teuthology.orchestra.run.smithi151.stdout:Install 33 Packages 2024-04-16T02:23:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:36.181 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 30 M 2024-04-16T02:23:36.181 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 110 M 2024-04-16T02:23:36.181 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:23:36.430 INFO:teuthology.orchestra.run.smithi151.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 62 kB/s | 6.4 kB 00:00 2024-04-16T02:23:36.471 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-16T02:23:36.472 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-04-16T02:23:36.472 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-04-16T02:23:36.472 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-04-16T02:23:36.472 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-04-16T02:23:36.472 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-04-16T02:23:36.472 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-04-16T02:23:36.472 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-04-16T02:23:36.472 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:36.472 INFO:teuthology.orchestra.run.smithi151.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 9.9 MB/s | 1.4 MB 00:00 2024-04-16T02:23:36.523 INFO:teuthology.orchestra.run.smithi151.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-04-16T02:23:36.548 INFO:teuthology.orchestra.run.smithi151.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.4 MB/s | 242 kB 00:00 2024-04-16T02:23:36.624 INFO:teuthology.orchestra.run.smithi151.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-04-16T02:23:36.749 INFO:teuthology.orchestra.run.smithi151.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 310 kB/s | 32 kB 00:00 2024-04-16T02:23:36.774 INFO:teuthology.orchestra.run.smithi151.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 638 kB/s | 144 kB 00:00 2024-04-16T02:23:36.800 INFO:teuthology.orchestra.run.smithi151.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-04-16T02:23:36.825 INFO:teuthology.orchestra.run.smithi151.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2024-04-16T02:23:37.017 INFO:teuthology.orchestra.run.smithi151.stdout:(10/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 32 MB/s | 18 MB 00:00 2024-04-16T02:23:37.043 INFO:teuthology.orchestra.run.smithi151.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 992 kB/s | 241 kB 00:00 2024-04-16T02:23:37.193 INFO:teuthology.orchestra.run.smithi151.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 3.1 MB/s | 1.1 MB 00:00 2024-04-16T02:23:37.207 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-16T02:23:37.233 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-16T02:23:37.233 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:37.233 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T02:23:37.233 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T02:23:37.233 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T02:23:37.233 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:37.235 INFO:teuthology.orchestra.run.smithi151.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 475 kB/s | 103 kB 00:00 2024-04-16T02:23:37.235 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-16T02:23:37.260 INFO:teuthology.orchestra.run.smithi151.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 571 kB/s | 124 kB 00:00 2024-04-16T02:23:37.286 INFO:teuthology.orchestra.run.smithi151.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.0 MB/s | 187 kB 00:00 2024-04-16T02:23:37.318 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-16T02:23:37.318 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-16T02:23:37.344 INFO:teuthology.orchestra.run.smithi151.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-04-16T02:23:37.536 INFO:teuthology.orchestra.run.smithi151.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 158 kB/s | 43 kB 00:00 2024-04-16T02:23:37.595 INFO:teuthology.orchestra.run.smithi151.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-04-16T02:23:37.654 INFO:teuthology.orchestra.run.smithi151.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 467 kB/s | 172 kB 00:00 2024-04-16T02:23:37.679 INFO:teuthology.orchestra.run.smithi151.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-04-16T02:23:37.712 INFO:teuthology.orchestra.run.smithi151.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-04-16T02:23:37.738 INFO:teuthology.orchestra.run.smithi151.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-04-16T02:23:37.755 INFO:teuthology.orchestra.run.smithi151.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-04-16T02:23:37.772 INFO:teuthology.orchestra.run.smithi151.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 332 kB/s | 20 kB 00:00 2024-04-16T02:23:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 911 kB/s | 46 kB 00:00 2024-04-16T02:23:37.831 INFO:teuthology.orchestra.run.smithi151.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-04-16T02:23:37.856 INFO:teuthology.orchestra.run.smithi151.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 243 kB/s | 16 kB 00:00 2024-04-16T02:23:37.890 INFO:teuthology.orchestra.run.smithi151.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-16T02:23:37.915 INFO:teuthology.orchestra.run.smithi151.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-04-16T02:23:37.941 INFO:teuthology.orchestra.run.smithi151.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 425 kB/s | 36 kB 00:00 2024-04-16T02:23:38.000 INFO:teuthology.orchestra.run.smithi151.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-04-16T02:23:38.025 INFO:teuthology.orchestra.run.smithi151.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-04-16T02:23:38.050 INFO:teuthology.orchestra.run.smithi151.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-04-16T02:23:38.056 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:38.056 INFO:teuthology.orchestra.run.smithi151.stdout:Total 16 MB/s | 30 MB 00:01 2024-04-16T02:23:38.069 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:23:38.108 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:23:38.108 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:23:38.433 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-16T02:23:38.433 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-04-16T02:23:38.433 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-04-16T02:23:38.433 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-04-16T02:23:38.433 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-04-16T02:23:38.433 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-04-16T02:23:38.433 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-04-16T02:23:38.433 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-04-16T02:23:38.433 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-16T02:23:38.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-16T02:23:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-04-16T02:23:38.460 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:23:38.460 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:23:39.576 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-16T02:23:39.576 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:39.576 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-04-16T02:23:39.576 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.576 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:23:39.576 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.577 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T02:23:39.578 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:39.579 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:23:39.759 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:23:39.819 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-16T02:23:39.843 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-16T02:23:39.876 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-16T02:23:39.906 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-04-16T02:23:39.969 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-16T02:23:40.047 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-16T02:23:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-16T02:23:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-16T02:23:40.231 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-16T02:23:40.270 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-16T02:23:40.319 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-16T02:23:40.378 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-16T02:23:40.492 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:26 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:23:40.576 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-16T02:23:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-16T02:23:40.648 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:23:40.649 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:23:40.649 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-16T02:23:40.649 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:23:40.649 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:23:40.649 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-04-16T02:23:40.650 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:40.651 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 41 M 2024-04-16T02:23:40.651 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 167 M 2024-04-16T02:23:40.651 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:23:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-16T02:23:40.715 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-16T02:23:40.745 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-16T02:23:40.821 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-16T02:23:40.856 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T02:23:40.882 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 165 kB 00:00 2024-04-16T02:23:40.889 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-16T02:23:40.955 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-16T02:23:40.990 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-16T02:23:41.025 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-16T02:23:41.114 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-16T02:23:41.156 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-16T02:23:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-16T02:23:41.368 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-16T02:23:41.391 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): ceph-test-18.2.1-0.el9.x86_64.rpm 62 MB/s | 40 MB 00:00 2024-04-16T02:23:41.416 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 324 kB/s | 217 kB 00:00 2024-04-16T02:23:41.441 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): socat-1.7.4.1-5.el9.x86_64.rpm 538 kB/s | 300 kB 00:00 2024-04-16T02:23:41.458 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 949 kB/s | 63 kB 00:00 2024-04-16T02:23:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T02:23:41.654 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-04-16T02:23:41.679 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-16T02:23:41.705 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-16T02:23:41.705 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:41.705 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T02:23:41.706 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T02:23:41.706 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T02:23:41.706 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:42.034 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): jq-1.6-16.el9.x86_64.rpm 316 kB/s | 187 kB 00:00 2024-04-16T02:23:42.037 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:42.037 INFO:teuthology.orchestra.run.smithi153.stdout:Total 29 MB/s | 41 MB 00:01 2024-04-16T02:23:42.052 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:23:42.073 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:23:42.073 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:23:42.412 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:23:42.412 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:23:42.929 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:23:43.019 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T02:23:43.059 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T02:23:43.091 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T02:23:43.134 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T02:23:43.162 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-04-16T02:23:43.265 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-16T02:23:43.294 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-16T02:23:43.295 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:43.295 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T02:23:43.295 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T02:23:43.295 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T02:23:43.295 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:43.750 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-16T02:23:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-16T02:23:43.775 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:43.775 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T02:23:43.775 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T02:23:43.775 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T02:23:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:43.975 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-16T02:23:43.999 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-16T02:23:43.999 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:44.000 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T02:23:44.000 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T02:23:44.000 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T02:23:44.000 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:44.065 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-16T02:23:45.854 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-16T02:23:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-16T02:23:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-16T02:23:45.857 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-16T02:23:45.857 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T02:23:45.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-16T02:23:45.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-16T02:23:45.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-16T02:23:45.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-16T02:23:45.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-16T02:23:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-16T02:23:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:46.680 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:23:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T02:23:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T02:23:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T02:23:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T02:23:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T02:23:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:46.683 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:23:46.920 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-04-16T02:23:47.004 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-04-16T02:23:47.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base 2024-04-16T02:23:47.615 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:47 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:23:47.715 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:23:47.766 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:23:47.767 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T02:23:47.767 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:23:47.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install cephadm 2024-04-16T02:23:48.419 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:48 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:23:48.572 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:48.573 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 221 k 2024-04-16T02:23:48.574 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 213 k 2024-04-16T02:23:48.574 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:23:48.646 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm-18.2.1-0.el9.noarch.rpm 3.0 MB/s | 221 kB 00:00 2024-04-16T02:23:48.646 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:48.646 INFO:teuthology.orchestra.run.smithi151.stdout:Total 3.0 MB/s | 221 kB 00:00 2024-04-16T02:23:48.647 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:23:48.651 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:23:48.652 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:23:48.654 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:23:48.654 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:23:48.727 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:23:49.043 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-04-16T02:23:49.044 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-04-16T02:23:49.044 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-04-16T02:23:49.044 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T02:23:49.044 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T02:23:49.044 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T02:23:49.134 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-16T02:23:49.227 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-16T02:23:49.457 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T02:23:49.457 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:49.457 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:23:49.457 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-04-16T02:23:49.457 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T02:23:49.457 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T02:23:49.457 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:49.457 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:23:49.574 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-16T02:23:49.740 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-04-16T02:23:49.847 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-16T02:23:49.847 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:49.847 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:23:49.847 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-04-16T02:23:49.847 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:49.847 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:23:50.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T02:23:50.326 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:36 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:23:50.480 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:23:50.482 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:23:50.482 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-16T02:23:50.482 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:23:50.482 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:23:50.482 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T02:23:50.483 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T02:23:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T02:23:50.485 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T02:23:50.485 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T02:23:50.485 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T02:23:50.485 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T02:23:50.485 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:50.485 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:23:50.485 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:23:50.485 INFO:teuthology.orchestra.run.smithi153.stdout:Install 33 Packages 2024-04-16T02:23:50.485 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:50.486 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 30 M 2024-04-16T02:23:50.486 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 110 M 2024-04-16T02:23:50.486 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:23:50.667 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:50 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:23:50.795 INFO:teuthology.orchestra.run.smithi153.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 67 kB/s | 6.4 kB 00:00 2024-04-16T02:23:50.825 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:23:50.825 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:50.825 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-04-16T02:23:50.825 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:50.826 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:23:50.826 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-04-16T02:23:50.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:50.826 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:23:50.826 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:50.826 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T02:23:50.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:50.827 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 142 k 2024-04-16T02:23:50.827 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 395 k 2024-04-16T02:23:50.827 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:23:50.846 INFO:teuthology.orchestra.run.smithi153.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 9.9 MB/s | 1.4 MB 00:00 2024-04-16T02:23:50.896 INFO:teuthology.orchestra.run.smithi153.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-04-16T02:23:50.899 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.9 MB/s | 142 kB 00:00 2024-04-16T02:23:50.899 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:50.900 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.9 MB/s | 142 kB 00:00 2024-04-16T02:23:50.900 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:23:50.907 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:23:50.907 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:23:50.921 INFO:teuthology.orchestra.run.smithi153.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.5 MB/s | 242 kB 00:00 2024-04-16T02:23:50.961 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:23:50.961 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:23:50.988 INFO:teuthology.orchestra.run.smithi153.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-04-16T02:23:51.105 INFO:teuthology.orchestra.run.smithi153.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 340 kB/s | 32 kB 00:00 2024-04-16T02:23:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:23:51.130 INFO:teuthology.orchestra.run.smithi153.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 691 kB/s | 144 kB 00:00 2024-04-16T02:23:51.156 INFO:teuthology.orchestra.run.smithi153.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 939 kB/s | 47 kB 00:00 2024-04-16T02:23:51.181 INFO:teuthology.orchestra.run.smithi153.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 764 kB/s | 39 kB 00:00 2024-04-16T02:23:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:23:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:23:51.209 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:51.210 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T02:23:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:51.215 INFO:teuthology.orchestra.run.smithi153.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-04-16T02:23:51.415 INFO:teuthology.orchestra.run.smithi153.stdout:(11/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 31 MB/s | 18 MB 00:00 2024-04-16T02:23:51.449 INFO:teuthology.orchestra.run.smithi153.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 4.3 MB/s | 1.1 MB 00:00 2024-04-16T02:23:51.474 INFO:teuthology.orchestra.run.smithi153.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 398 kB/s | 103 kB 00:00 2024-04-16T02:23:51.499 INFO:teuthology.orchestra.run.smithi153.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-04-16T02:23:51.524 INFO:teuthology.orchestra.run.smithi153.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-16T02:23:51.624 INFO:teuthology.orchestra.run.smithi153.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 833 kB/s | 125 kB 00:00 2024-04-16T02:23:51.658 INFO:teuthology.orchestra.run.smithi153.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2024-04-16T02:23:51.708 INFO:teuthology.orchestra.run.smithi153.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-04-16T02:23:51.734 INFO:teuthology.orchestra.run.smithi153.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 822 kB/s | 172 kB 00:00 2024-04-16T02:23:51.759 INFO:teuthology.orchestra.run.smithi153.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-04-16T02:23:51.784 INFO:teuthology.orchestra.run.smithi153.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-04-16T02:23:51.809 INFO:teuthology.orchestra.run.smithi153.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-04-16T02:23:51.835 INFO:teuthology.orchestra.run.smithi153.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-04-16T02:23:51.877 INFO:teuthology.orchestra.run.smithi153.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-04-16T02:23:51.893 INFO:teuthology.orchestra.run.smithi153.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-04-16T02:23:51.919 INFO:teuthology.orchestra.run.smithi153.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 942 kB/s | 79 kB 00:00 2024-04-16T02:23:51.944 INFO:teuthology.orchestra.run.smithi153.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-16T02:23:51.961 INFO:teuthology.orchestra.run.smithi153.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 244 kB/s | 16 kB 00:00 2024-04-16T02:23:51.986 INFO:teuthology.orchestra.run.smithi153.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-04-16T02:23:52.003 INFO:teuthology.orchestra.run.smithi153.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 608 kB/s | 36 kB 00:00 2024-04-16T02:23:52.054 INFO:teuthology.orchestra.run.smithi153.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 401 kB/s | 20 kB 00:00 2024-04-16T02:23:52.104 INFO:teuthology.orchestra.run.smithi153.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-04-16T02:23:52.129 INFO:teuthology.orchestra.run.smithi153.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-04-16T02:23:52.133 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:52.133 INFO:teuthology.orchestra.run.smithi153.stdout:Total 18 MB/s | 30 MB 00:01 2024-04-16T02:23:52.147 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:23:52.169 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:23:52.169 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:52.169 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:23:52.169 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-04-16T02:23:52.169 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:52.170 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:23:52.187 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:23:52.187 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:23:52.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr 2024-04-16T02:23:52.552 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:23:52.552 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:23:53.023 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:53 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:23:53.122 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:23:53.174 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:23:53.175 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T02:23:53.175 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:23:53.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T02:23:53.777 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:23:53.831 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-16T02:23:53.832 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:53 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:23:53.863 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-16T02:23:53.896 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-16T02:23:53.985 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:23:53.986 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:53.986 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T02:23:53.986 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:53.986 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:23:53.986 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-04-16T02:23:53.986 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout:Install 7 Packages 2024-04-16T02:23:53.987 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:53.988 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.1 M 2024-04-16T02:23:53.988 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 8.5 M 2024-04-16T02:23:53.988 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:23:53.989 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-16T02:23:54.058 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-16T02:23:54.099 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-16T02:23:54.180 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-16T02:23:54.233 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-16T02:23:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-16T02:23:54.290 INFO:teuthology.orchestra.run.smithi151.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 216 kB/s | 23 kB 00:00 2024-04-16T02:23:54.307 INFO:teuthology.orchestra.run.smithi151.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 118 kB/s | 15 kB 00:00 2024-04-16T02:23:54.320 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-16T02:23:54.349 INFO:teuthology.orchestra.run.smithi151.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 11 MB/s | 1.8 MB 00:00 2024-04-16T02:23:54.379 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-16T02:23:54.516 INFO:teuthology.orchestra.run.smithi151.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 40 kB/s | 9.0 kB 00:00 2024-04-16T02:23:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-16T02:23:54.583 INFO:teuthology.orchestra.run.smithi151.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 148 kB/s | 41 kB 00:00 2024-04-16T02:23:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 115 kB/s | 31 kB 00:00 2024-04-16T02:23:54.617 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-16T02:23:54.661 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-16T02:23:54.699 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-16T02:23:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-16T02:23:54.784 INFO:teuthology.orchestra.run.smithi151.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 705 kB/s | 188 kB 00:00 2024-04-16T02:23:54.788 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:54.788 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.6 MB/s | 2.1 MB 00:00 2024-04-16T02:23:54.788 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:23:54.795 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:23:54.796 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:23:54.797 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-16T02:23:54.831 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T02:23:54.855 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:23:54.855 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:23:54.864 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-16T02:23:54.904 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-16T02:23:54.940 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-16T02:23:54.974 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-16T02:23:55.063 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-16T02:23:55.105 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-16T02:23:55.146 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:23:55.178 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-16T02:23:55.211 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-16T02:23:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-16T02:23:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-16T02:23:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-16T02:23:55.341 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-16T02:23:55.352 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-04-16T02:23:55.464 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T02:23:55.587 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-04-16T02:23:55.618 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-04-16T02:23:55.621 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-16T02:23:55.643 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-16T02:23:55.669 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-16T02:23:55.669 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:55.669 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T02:23:55.669 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T02:23:55.669 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T02:23:55.670 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:56.192 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-16T02:23:56.192 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-04-16T02:23:56.192 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-04-16T02:23:56.192 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-04-16T02:23:56.192 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-16T02:23:56.192 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-16T02:23:56.193 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-16T02:23:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-16T02:23:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:56.486 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:23:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-04-16T02:23:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-04-16T02:23:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-04-16T02:23:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-16T02:23:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-16T02:23:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T02:23:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T02:23:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:56.487 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:23:56.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T02:23:57.169 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-16T02:23:57.196 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-16T02:23:57.196 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:57.196 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T02:23:57.196 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T02:23:57.196 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T02:23:57.196 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:57.323 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:57 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:23:57.476 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:23:57.477 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:57.477 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T02:23:57.478 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T02:23:57.479 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T02:23:57.479 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T02:23:57.479 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:57.479 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:23:57.479 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:23:57.479 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-04-16T02:23:57.479 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:23:57.479 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 37 M 2024-04-16T02:23:57.480 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 209 M 2024-04-16T02:23:57.480 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:23:57.653 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-16T02:23:57.678 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-16T02:23:57.678 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:57.678 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T02:23:57.679 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T02:23:57.679 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T02:23:57.679 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:57.887 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-16T02:23:57.919 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-16T02:23:57.919 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:23:57.919 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T02:23:57.919 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T02:23:57.919 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T02:23:57.919 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:23:57.978 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-16T02:23:58.151 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 244 kB/s | 32 kB 00:00 2024-04-16T02:23:58.268 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): ceph-mgr-diskprediction-local-18.2.1-0. 30 MB/s | 7.4 MB 00:00 2024-04-16T02:23:58.293 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 121 kB/s | 17 kB 00:00 2024-04-16T02:23:58.318 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 682 kB/s | 34 kB 00:00 2024-04-16T02:23:58.519 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-04-16T02:23:58.594 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-04-16T02:23:58.620 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-04-16T02:23:58.895 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.4 MB/s | 3.0 MB 00:00 2024-04-16T02:23:59.070 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-04-16T02:23:59.313 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-04-16T02:23:59.404 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-04-16T02:23:59.438 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): libquadmath-11.4.1-3.el9.x86_64.rpm 361 kB/s | 195 kB 00:00 2024-04-16T02:23:59.440 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:23:59.441 INFO:teuthology.orchestra.run.smithi151.stdout:Total 19 MB/s | 37 MB 00:01 2024-04-16T02:23:59.482 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:23:59.507 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:23:59.507 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:23:59.851 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-16T02:23:59.851 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-04-16T02:23:59.851 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-04-16T02:23:59.851 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-04-16T02:23:59.851 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-04-16T02:23:59.851 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-04-16T02:23:59.851 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-16T02:23:59.852 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-16T02:23:59.853 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-16T02:23:59.945 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:23:59.945 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:24:00.687 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:24:00.760 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T02:24:00.772 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T02:24:00.773 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T02:24:00.774 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:00.775 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:00.792 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T02:24:00.889 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T02:24:00.946 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T02:24:01.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-04-16T02:24:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T02:24:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T02:24:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T02:24:01.694 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:47 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:01.793 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:01.845 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:01.846 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-16T02:24:01.846 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:01.903 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T02:24:01.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-04-16T02:24:02.019 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T02:24:02.496 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:48 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:02.648 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:02.648 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 221 k 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 213 k 2024-04-16T02:24:02.649 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:02.722 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-18.2.1-0.el9.noarch.rpm 3.0 MB/s | 221 kB 00:00 2024-04-16T02:24:02.722 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:02.722 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.9 MB/s | 221 kB 00:00 2024-04-16T02:24:02.722 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:02.727 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:02.727 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:02.730 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:02.730 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:02.802 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:03.176 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-16T02:24:03.260 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-16T02:24:03.264 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T02:24:03.623 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-16T02:24:03.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-16T02:24:03.894 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:03.894 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:03.894 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-04-16T02:24:03.894 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:03.894 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:04.099 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T02:24:04.305 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T02:24:04.336 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-16T02:24:04.685 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:50 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:04.842 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:04.843 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:04.843 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-04-16T02:24:04.843 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:04.843 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:04.843 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-04-16T02:24:04.843 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:04.844 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:04.844 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:04.844 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-16T02:24:04.844 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:04.844 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 142 k 2024-04-16T02:24:04.844 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 395 k 2024-04-16T02:24:04.844 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:04.910 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 2.1 MB/s | 142 kB 00:00 2024-04-16T02:24:04.911 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:04.911 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.1 MB/s | 142 kB 00:00 2024-04-16T02:24:04.911 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:04.918 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:04.918 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:04.970 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:04.970 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:05.110 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:05.169 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:05.194 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:05.195 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:24:05.195 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T02:24:05.195 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:06.158 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:06.158 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:06.421 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-04-16T02:24:07.019 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:53 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:07.119 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T02:24:07.151 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T02:24:07.152 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T02:24:07.170 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:07.171 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-16T02:24:07.171 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:07.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T02:24:07.584 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T02:24:07.585 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T02:24:07.585 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T02:24:07.585 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T02:24:07.585 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T02:24:07.585 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T02:24:07.585 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:07.585 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:07.808 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:53 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:07.961 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:07.961 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-16T02:24:07.962 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:07.963 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:07.963 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:07.963 INFO:teuthology.orchestra.run.smithi153.stdout:Install 7 Packages 2024-04-16T02:24:07.963 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:07.963 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.1 M 2024-04-16T02:24:07.963 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 8.5 M 2024-04-16T02:24:07.964 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:07.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-rook 2024-04-16T02:24:08.280 INFO:teuthology.orchestra.run.smithi153.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 198 kB/s | 23 kB 00:00 2024-04-16T02:24:08.305 INFO:teuthology.orchestra.run.smithi153.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 103 kB/s | 15 kB 00:00 2024-04-16T02:24:08.355 INFO:teuthology.orchestra.run.smithi153.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 9.1 MB/s | 1.8 MB 00:00 2024-04-16T02:24:08.431 INFO:teuthology.orchestra.run.smithi153.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 60 kB/s | 9.0 kB 00:00 2024-04-16T02:24:08.498 INFO:teuthology.orchestra.run.smithi153.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 213 kB/s | 41 kB 00:00 2024-04-16T02:24:08.564 INFO:teuthology.orchestra.run.smithi153.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 147 kB/s | 31 kB 00:00 2024-04-16T02:24:08.574 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:08 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:08.648 INFO:teuthology.orchestra.run.smithi153.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 868 kB/s | 188 kB 00:00 2024-04-16T02:24:08.650 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:08.650 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.0 MB/s | 2.1 MB 00:00 2024-04-16T02:24:08.650 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:08.658 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:08.658 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:08.722 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:08.722 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:08.729 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:08.730 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:08.730 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T02:24:08.730 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T02:24:08.731 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T02:24:08.732 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T02:24:08.732 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:08.732 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:24:08.732 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:08.732 INFO:teuthology.orchestra.run.smithi151.stdout:Install 13 Packages 2024-04-16T02:24:08.732 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:08.732 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.0 M 2024-04-16T02:24:08.733 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2024-04-16T02:24:08.733 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:24:09.028 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:09.107 INFO:teuthology.orchestra.run.smithi151.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 459 kB/s | 50 kB 00:00 2024-04-16T02:24:09.108 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-16T02:24:09.140 INFO:teuthology.orchestra.run.smithi151.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-04-16T02:24:09.142 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-16T02:24:09.158 INFO:teuthology.orchestra.run.smithi151.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 153 kB/s | 24 kB 00:00 2024-04-16T02:24:09.183 INFO:teuthology.orchestra.run.smithi151.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-16T02:24:09.188 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-16T02:24:09.208 INFO:teuthology.orchestra.run.smithi151.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-04-16T02:24:09.221 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-16T02:24:09.234 INFO:teuthology.orchestra.run.smithi151.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-04-16T02:24:09.249 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-04-16T02:24:09.250 INFO:teuthology.orchestra.run.smithi151.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 638 kB/s | 43 kB 00:00 2024-04-16T02:24:09.486 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-04-16T02:24:09.526 INFO:teuthology.orchestra.run.smithi151.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-04-16T02:24:09.528 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-16T02:24:09.543 INFO:teuthology.orchestra.run.smithi151.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-04-16T02:24:09.643 INFO:teuthology.orchestra.run.smithi151.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 517 kB/s | 203 kB 00:00 2024-04-16T02:24:09.694 INFO:teuthology.orchestra.run.smithi151.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 392 kB/s | 59 kB 00:00 2024-04-16T02:24:09.719 INFO:teuthology.orchestra.run.smithi151.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-04-16T02:24:09.886 INFO:teuthology.orchestra.run.smithi151.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-04-16T02:24:09.890 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:09.890 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-04-16T02:24:09.891 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:24:09.903 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:24:09.904 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:24:09.994 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:24:09.994 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:24:10.045 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-16T02:24:10.045 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-04-16T02:24:10.045 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-04-16T02:24:10.045 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-04-16T02:24:10.045 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-16T02:24:10.045 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-16T02:24:10.045 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-16T02:24:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-16T02:24:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:10.349 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-04-16T02:24:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-04-16T02:24:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-04-16T02:24:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-16T02:24:10.350 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-16T02:24:10.350 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T02:24:10.350 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T02:24:10.350 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:10.350 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:10.514 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:24:10.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T02:24:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T02:24:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T02:24:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T02:24:10.831 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T02:24:10.872 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T02:24:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T02:24:11.075 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T02:24:11.125 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T02:24:11.149 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:57 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:11.303 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:11.304 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:11.304 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-16T02:24:11.304 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:11.304 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:11.304 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-04-16T02:24:11.304 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-16T02:24:11.304 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:11.305 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-04-16T02:24:11.306 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:11.306 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 37 M 2024-04-16T02:24:11.306 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 209 M 2024-04-16T02:24:11.306 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:11.588 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 32 kB 00:00 2024-04-16T02:24:11.621 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 523 kB/s | 17 kB 00:00 2024-04-16T02:24:11.714 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.1-0. 28 MB/s | 7.4 MB 00:00 2024-04-16T02:24:11.739 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 292 kB/s | 34 kB 00:00 2024-04-16T02:24:11.852 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T02:24:11.897 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T02:24:11.923 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-04-16T02:24:11.931 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T02:24:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T02:24:11.998 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-04-16T02:24:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-16T02:24:12.048 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-04-16T02:24:12.073 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.4 MB/s | 217 kB 00:00 2024-04-16T02:24:12.223 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-16T02:24:12.374 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-04-16T02:24:12.399 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-04-16T02:24:12.433 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-04-16T02:24:12.434 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:12.435 INFO:teuthology.orchestra.run.smithi153.stdout:Total 33 MB/s | 37 MB 00:01 2024-04-16T02:24:12.475 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:12.498 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:12.498 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-16T02:24:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-04-16T02:24:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T02:24:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T02:24:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T02:24:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T02:24:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T02:24:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T02:24:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T02:24:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T02:24:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T02:24:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T02:24:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T02:24:12.932 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:12.932 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T02:24:13.245 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T02:24:13.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T02:24:13.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T02:24:13.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T02:24:13.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T02:24:13.246 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:13.246 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:13.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T02:24:13.626 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:13.714 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T02:24:13.745 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T02:24:13.838 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T02:24:13.882 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T02:24:14.060 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T02:24:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T02:24:14.092 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:14 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T02:24:14.244 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:14.245 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:14.245 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T02:24:14.245 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:14.245 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:24:14.245 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-04-16T02:24:14.245 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-04-16T02:24:14.246 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:14.247 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 6.8 M 2024-04-16T02:24:14.247 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 33 M 2024-04-16T02:24:14.247 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:24:14.628 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-04-16T02:24:14.711 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-16T02:24:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T02:24:14.929 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-04-16T02:24:14.943 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T02:24:15.095 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 160 kB/s | 58 kB 00:00 2024-04-16T02:24:15.187 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 982 kB/s | 548 kB 00:00 2024-04-16T02:24:15.204 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 186 kB/s | 51 kB 00:00 2024-04-16T02:24:15.207 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:15.207 INFO:teuthology.orchestra.run.smithi151.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2024-04-16T02:24:15.220 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:24:15.231 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:24:15.231 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:24:15.319 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:24:15.320 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:24:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:24:15.778 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T02:24:15.831 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T02:24:16.185 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T02:24:16.350 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T02:24:16.427 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T02:24:16.496 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T02:24:16.520 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-16T02:24:17.204 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T02:24:17.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-16T02:24:17.344 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-16T02:24:17.344 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-04-16T02:24:17.344 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T02:24:17.344 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T02:24:17.344 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T02:24:17.344 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T02:24:17.658 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T02:24:17.658 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:17.658 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:24:17.658 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-04-16T02:24:17.658 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T02:24:17.658 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T02:24:17.659 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T02:24:17.659 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T02:24:17.659 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T02:24:17.659 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:17.659 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:17.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse 2024-04-16T02:24:18.494 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:18 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:18.647 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:18.647 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-04-16T02:24:18.648 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:18.649 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 918 k 2024-04-16T02:24:18.649 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.7 M 2024-04-16T02:24:18.649 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:24:18.750 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-04-16T02:24:18.809 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 492 kB/s | 79 kB 00:00 2024-04-16T02:24:18.810 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:18.810 INFO:teuthology.orchestra.run.smithi151.stdout:Total 5.5 MB/s | 918 kB 00:00 2024-04-16T02:24:18.810 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:24:18.831 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:24:18.831 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:24:18.903 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:24:18.903 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:24:19.089 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:24:19.215 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T02:24:19.283 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-16T02:24:19.696 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T02:24:19.697 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T02:24:20.011 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-16T02:24:20.012 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-04-16T02:24:20.128 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T02:24:20.128 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:20.128 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:20.128 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:20.129 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T02:24:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:20.359 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:24:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T02:24:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:20.359 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:20.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-04-16T02:24:20.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel 2024-04-16T02:24:21.096 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:07 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:21.192 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:21 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:21.251 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:21.252 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:21.252 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-16T02:24:21.252 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:21.252 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:21.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-04-16T02:24:21.252 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T02:24:21.253 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T02:24:21.254 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:21.254 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:21.254 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:21.254 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-04-16T02:24:21.254 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:21.254 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-04-16T02:24:21.255 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-04-16T02:24:21.255 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:21.342 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:21.346 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:21.346 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:21.346 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:24:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-04-16T02:24:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:21.347 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:24:21.347 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:21.347 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T02:24:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:21.347 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 127 k 2024-04-16T02:24:21.347 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 456 k 2024-04-16T02:24:21.347 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:24:21.417 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.7 MB/s | 127 kB 00:00 2024-04-16T02:24:21.417 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:21.417 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.7 MB/s | 127 kB 00:00 2024-04-16T02:24:21.418 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:24:21.423 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:24:21.423 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:24:21.478 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:24:21.479 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:24:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:24:21.721 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 473 kB/s | 50 kB 00:00 2024-04-16T02:24:21.731 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:21.746 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 186 kB/s | 24 kB 00:00 2024-04-16T02:24:21.771 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-04-16T02:24:21.796 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-16T02:24:21.822 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-16T02:24:21.846 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-04-16T02:24:21.872 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-04-16T02:24:21.947 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 150 kB/s | 15 kB 00:00 2024-04-16T02:24:21.972 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-04-16T02:24:22.031 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-04-16T02:24:22.056 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 708 kB/s | 59 kB 00:00 2024-04-16T02:24:22.082 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-04-16T02:24:22.173 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:22.349 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.5 MB/s | 1.0 MB 00:00 2024-04-16T02:24:22.350 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:22.351 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-04-16T02:24:22.351 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:22.364 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:22.364 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:22.455 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:22.455 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:22.474 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:24:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:22.474 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:22.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2 2024-04-16T02:24:22.994 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:23.162 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T02:24:23.216 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T02:24:23.272 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T02:24:23.317 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T02:24:23.334 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:23 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:23.359 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T02:24:23.436 INFO:teuthology.orchestra.run.smithi151.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:23.452 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T02:24:23.490 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:23.491 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T02:24:23.491 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:23.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs-devel 2024-04-16T02:24:23.569 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T02:24:23.618 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T02:24:24.140 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:24 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:24.291 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:24.292 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:24.292 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:24.292 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:24.292 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:24:24.292 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-04-16T02:24:24.292 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:24.292 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:24:24.292 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:24.293 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T02:24:24.293 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:24.293 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 31 k 2024-04-16T02:24:24.293 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 139 k 2024-04-16T02:24:24.293 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:24:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T02:24:24.357 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 480 kB/s | 31 kB 00:00 2024-04-16T02:24:24.358 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:24.358 INFO:teuthology.orchestra.run.smithi151.stdout:Total 470 kB/s | 31 kB 00:00 2024-04-16T02:24:24.358 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:24:24.361 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:24:24.361 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:24:24.390 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:24:24.390 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:24:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T02:24:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T02:24:24.472 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:24:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T02:24:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-16T02:24:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:24.939 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:25.189 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:25.189 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:25.189 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:24:25.189 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:25.189 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:25.190 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:25.217 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T02:24:25.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T02:24:25.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2 2024-04-16T02:24:25.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T02:24:25.598 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:25.598 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:25.598 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-04-16T02:24:25.598 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T02:24:25.598 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:25.599 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:25.854 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T02:24:25.982 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:25 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:26.082 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:26.135 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:26.136 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T02:24:26.136 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:26.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1 2024-04-16T02:24:26.433 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:12 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:26.584 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:26.585 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:26.585 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-16T02:24:26.585 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:26.585 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:26.585 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-04-16T02:24:26.585 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-04-16T02:24:26.586 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:26.587 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.8 M 2024-04-16T02:24:26.587 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-04-16T02:24:26.587 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:26.794 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:26 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:26.884 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.8 MB/s | 132 kB 00:00 2024-04-16T02:24:26.895 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:26.948 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:26.949 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T02:24:26.949 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:27.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados 2024-04-16T02:24:27.092 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 888 kB/s | 248 kB 00:00 2024-04-16T02:24:27.326 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-04-16T02:24:27.376 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-04-16T02:24:27.426 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-04-16T02:24:27.610 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2024-04-16T02:24:27.611 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:27 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:27.614 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:27.614 INFO:teuthology.orchestra.run.smithi153.stdout:Total 6.6 MB/s | 6.8 MB 00:01 2024-04-16T02:24:27.627 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:27.637 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:27.638 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:27.710 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:27.727 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:27.727 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:27.763 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:27.764 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T02:24:27.764 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:27.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw 2024-04-16T02:24:28.033 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:28.213 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T02:24:28.270 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T02:24:28.427 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:28 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:28.541 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:28.594 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:28.595 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T02:24:28.595 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:28.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs 2024-04-16T02:24:28.794 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T02:24:28.873 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T02:24:28.932 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T02:24:28.965 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-16T02:24:29.245 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:29 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:29.344 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:29.396 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:29.397 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T02:24:29.397 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:29.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd 2024-04-16T02:24:29.845 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-16T02:24:29.845 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-04-16T02:24:29.845 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T02:24:29.845 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T02:24:29.845 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T02:24:29.846 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T02:24:30.068 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:30 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:30.148 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:30.169 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:30.222 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:30.223 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T02:24:30.223 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:30.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse 2024-04-16T02:24:30.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-04-16T02:24:30.878 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:30 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:30.992 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:16 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:31.032 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:31.032 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:31.033 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:31.033 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:31.033 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:24:31.033 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-04-16T02:24:31.033 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:31.033 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:24:31.033 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:31.033 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T02:24:31.033 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:31.034 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 86 k 2024-04-16T02:24:31.034 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 226 k 2024-04-16T02:24:31.034 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:24:31.120 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 998 kB/s | 86 kB 00:00 2024-04-16T02:24:31.121 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:31.121 INFO:teuthology.orchestra.run.smithi151.stdout:Total 984 kB/s | 86 kB 00:00 2024-04-16T02:24:31.121 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:24:31.128 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:24:31.128 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:24:31.147 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:31.147 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:31.147 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:31.147 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:31.148 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:31.148 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-04-16T02:24:31.148 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-16T02:24:31.148 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T02:24:31.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:31.148 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:31.148 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:31.148 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-04-16T02:24:31.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:31.149 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 918 k 2024-04-16T02:24:31.149 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.7 M 2024-04-16T02:24:31.149 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:31.182 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:24:31.182 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:24:31.246 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 8.5 MB/s | 839 kB 00:00 2024-04-16T02:24:31.304 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 509 kB/s | 79 kB 00:00 2024-04-16T02:24:31.305 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:31.305 INFO:teuthology.orchestra.run.smithi153.stdout:Total 5.7 MB/s | 918 kB 00:00 2024-04-16T02:24:31.306 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:31.325 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:31.325 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:24:31.397 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:31.398 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:31.434 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:31.579 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:31.718 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T02:24:31.786 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-16T02:24:31.823 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:32.124 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:32.124 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:32.124 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:24:32.124 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:32.124 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:32.124 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:32.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror 2024-04-16T02:24:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-16T02:24:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-04-16T02:24:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T02:24:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:32.857 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T02:24:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:32.857 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:32.986 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:32 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:33.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-04-16T02:24:33.143 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:33.144 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:33.144 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:33.144 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:24:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-16T02:24:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:33.144 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:24:33.144 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:33.144 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T02:24:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:33.145 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.0 M 2024-04-16T02:24:33.145 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 12 M 2024-04-16T02:24:33.145 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:24:33.277 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 23 MB/s | 3.0 MB 00:00 2024-04-16T02:24:33.277 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:33.278 INFO:teuthology.orchestra.run.smithi151.stdout:Total 22 MB/s | 3.0 MB 00:00 2024-04-16T02:24:33.278 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:24:33.287 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:24:33.287 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:24:33.359 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:24:33.359 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:24:33.692 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:19 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:33.765 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:24:33.828 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:33.843 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-16T02:24:33.844 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 127 k 2024-04-16T02:24:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 456 k 2024-04-16T02:24:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:33.853 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:24:33.853 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T02:24:33.854 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T02:24:33.854 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T02:24:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:33.910 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.9 MB/s | 127 kB 00:00 2024-04-16T02:24:33.911 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:33.911 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.8 MB/s | 127 kB 00:00 2024-04-16T02:24:33.911 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:33.917 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:33.917 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:33.971 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:33.971 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:34.112 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:34.200 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:34.666 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:34.907 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:34.908 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:34.908 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:34.908 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:34.908 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:34.908 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:34.986 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:34.986 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:34.986 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:24:34.987 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:34.987 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:34.987 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:35.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-04-16T02:24:35.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd 2024-04-16T02:24:35.790 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:21 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:35.848 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:35 ago on Tue 16 Apr 2024 02:23:00 AM UTC. 2024-04-16T02:24:35.893 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:35.948 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:35.949 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-16T02:24:35.949 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T02:24:36.002 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:36.002 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:36.002 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:36.002 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T02:24:36.002 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-04-16T02:24:36.002 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:36.002 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T02:24:36.002 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T02:24:36.003 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T02:24:36.003 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:36.003 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 171 k 2024-04-16T02:24:36.003 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 494 k 2024-04-16T02:24:36.003 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T02:24:36.020 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-04-16T02:24:36.067 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.6 MB/s | 171 kB 00:00 2024-04-16T02:24:36.068 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:36.068 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.5 MB/s | 171 kB 00:00 2024-04-16T02:24:36.068 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T02:24:36.076 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T02:24:36.077 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T02:24:36.132 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T02:24:36.133 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T02:24:36.275 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T02:24:36.382 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:36.595 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:22 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:36.747 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:36.747 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:36.748 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:36.748 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:36.748 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:36.748 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-04-16T02:24:36.748 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:36.748 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:36.748 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:36.748 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-16T02:24:36.748 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:36.749 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 31 k 2024-04-16T02:24:36.749 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 139 k 2024-04-16T02:24:36.749 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:36.822 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 425 kB/s | 31 kB 00:00 2024-04-16T02:24:36.822 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:36.822 INFO:teuthology.orchestra.run.smithi153.stdout:Total 418 kB/s | 31 kB 00:00 2024-04-16T02:24:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:36.826 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:36.826 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:36.848 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:36.859 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:36.860 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:36.945 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:37.045 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:37.118 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T02:24:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:24:37.118 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T02:24:37.360 DEBUG:teuthology.parallel:result is None 2024-04-16T02:24:37.440 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:37.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:37.703 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:37.704 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:37.704 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:37.704 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:37.704 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:37.896 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-04-16T02:24:38.482 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:24 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:38.583 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:38.636 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:38.637 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-16T02:24:38.638 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:38.712 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-04-16T02:24:39.300 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:25 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:39.399 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:39.453 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:39.454 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-16T02:24:39.454 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:39.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-04-16T02:24:40.137 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:26 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:40.236 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:40.289 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:40.290 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-16T02:24:40.290 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:40.358 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-04-16T02:24:40.956 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:26 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:41.056 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:41.110 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:41.111 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-16T02:24:41.111 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:41.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-04-16T02:24:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:27 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:41.866 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:41.918 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:41.920 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-16T02:24:41.920 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:41.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-04-16T02:24:42.584 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:28 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:42.683 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-16T02:24:42.738 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:42.739 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-16T02:24:42.739 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:42.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-04-16T02:24:43.396 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:29 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:43.551 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:43.551 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:43.551 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:43.551 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 86 k 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 226 k 2024-04-16T02:24:43.552 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:43.625 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2024-04-16T02:24:43.626 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:43.626 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.1 MB/s | 86 kB 00:00 2024-04-16T02:24:43.626 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:43.633 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:43.633 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:43.687 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:43.688 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:43.821 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:43.931 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:44.348 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:44.654 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:44.655 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:44.655 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:44.655 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:44.655 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:44.655 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:44.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-04-16T02:24:45.507 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:31 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:45.664 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.0 M 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 12 M 2024-04-16T02:24:45.665 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:45.761 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 31 MB/s | 3.0 MB 00:00 2024-04-16T02:24:45.762 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:45.762 INFO:teuthology.orchestra.run.smithi153.stdout:Total 31 MB/s | 3.0 MB 00:00 2024-04-16T02:24:45.763 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:45.771 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:45.771 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:45.843 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:45.844 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:46.304 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:46.330 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:46.330 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T02:24:46.330 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T02:24:46.330 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T02:24:46.330 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T02:24:46.330 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:47.402 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:47.402 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:47.402 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:47.402 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:47.402 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:47.402 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:47.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-04-16T02:24:48.232 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:34 ago on Tue 16 Apr 2024 02:23:14 AM UTC. 2024-04-16T02:24:48.386 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:48.387 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 171 k 2024-04-16T02:24:48.388 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 494 k 2024-04-16T02:24:48.388 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-16T02:24:48.456 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.4 MB/s | 171 kB 00:00 2024-04-16T02:24:48.457 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:24:48.457 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.4 MB/s | 171 kB 00:00 2024-04-16T02:24:48.457 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-16T02:24:48.465 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-16T02:24:48.465 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-16T02:24:48.522 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-16T02:24:48.522 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-16T02:24:48.664 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-16T02:24:48.755 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:49.218 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:49.502 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-16T02:24:49.502 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:49.502 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-16T02:24:49.502 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-04-16T02:24:49.502 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:24:49.502 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-16T02:24:49.778 DEBUG:teuthology.parallel:result is None 2024-04-16T02:24:49.778 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-16T02:24:49.778 INFO:teuthology.packaging:ref: None 2024-04-16T02:24:49.778 INFO:teuthology.packaging:tag: v18.2.1 2024-04-16T02:24:49.778 INFO:teuthology.packaging:branch: None 2024-04-16T02:24:49.779 INFO:teuthology.packaging:sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:24:49.779 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-16T02:24:49.923 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T02:24:49.950 INFO:teuthology.orchestra.run.smithi151.stdout:18.2.1-0.el9 2024-04-16T02:24:49.951 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-04-16T02:24:49.951 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-16T02:24:49.953 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-16T02:24:49.953 INFO:teuthology.packaging:ref: None 2024-04-16T02:24:49.953 INFO:teuthology.packaging:tag: v18.2.1 2024-04-16T02:24:49.953 INFO:teuthology.packaging:branch: None 2024-04-16T02:24:49.953 INFO:teuthology.packaging:sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:24:49.954 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-16T02:24:50.185 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T02:24:50.207 INFO:teuthology.orchestra.run.smithi153.stdout:18.2.1-0.el9 2024-04-16T02:24:50.208 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-04-16T02:24:50.208 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-16T02:24:50.210 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T02:24:50.210 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:24:50.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T02:24:50.244 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:24:50.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T02:24:50.277 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T02:24:50.277 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:24:50.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T02:24:50.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T02:24:50.387 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:24:50.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T02:24:50.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T02:24:50.484 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T02:24:50.484 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:24:50.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T02:24:50.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T02:24:50.584 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:24:50.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T02:24:50.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T02:24:50.681 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T02:24:50.681 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:24:50.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T02:24:50.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T02:24:50.779 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:24:50.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T02:24:50.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T02:24:50.877 INFO:teuthology.run_tasks:Running task print... 2024-04-16T02:24:50.888 INFO:teuthology.task.print:**** done install task... 2024-04-16T02:24:50.888 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-16T02:24:50.989 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.1', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '87dbfb632e974cb2c28d17423071d2d833b53b29'} 2024-04-16T02:24:50.990 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.1 2024-04-16T02:24:50.990 INFO:tasks.cephadm:Cluster fsid is 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:24:50.990 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-16T02:24:50.990 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-16T02:24:50.991 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi151': '172.21.15.151', 'mon.smithi153': '172.21.15.153'} 2024-04-16T02:24:50.991 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-16T02:24:50.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo hostname $(hostname -s) 2024-04-16T02:24:51.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-04-16T02:24:51.050 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-04-16T02:24:51.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:24:51.269 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '87dbfb632e974cb2c28d17423071d2d833b53b29', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54897/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3084-g87dbfb63', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3084.g87dbfb63'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/87dbfb632e974cb2c28d17423071d2d833b53b29/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-14 16:30:23.821701', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/87dbfb632e974cb2c28d17423071d2d833b53b29/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-16T02:24:51.517 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref reef, sha1 f1f10a1c441d6255eaa3105af60f09fbca74c76c from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-04-16T02:24:51.519 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/reef/f1f10a1c441d6255eaa3105af60f09fbca74c76c/centos/9/x86_64/flavors/default/cephadm 2024-04-16T02:24:51.519 DEBUG:teuthology.orchestra.run.smithi151:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/f1f10a1c441d6255eaa3105af60f09fbca74c76c/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-16T02:24:52.088 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 Apr 16 02:24 /home/ubuntu/cephtest/cephadm 2024-04-16T02:24:52.089 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/f1f10a1c441d6255eaa3105af60f09fbca74c76c/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-16T02:24:52.497 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 Apr 16 02:24 /home/ubuntu/cephtest/cephadm 2024-04-16T02:24:52.497 DEBUG:teuthology.orchestra.run.smithi151:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-16T02:24:52.515 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-16T02:24:52.548 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.1 on all hosts... 2024-04-16T02:24:52.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-16T02:24:52.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-16T02:24:53.356 INFO:teuthology.orchestra.run.smithi151.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-16T02:24:53.475 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-16T02:25:10.941 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:25:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-16T02:25:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-16T02:25:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "repo_digests": [ 2024-04-16T02:25:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-16T02:25:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-16T02:25:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-04-16T02:25:10.942 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:25:13.998 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-16T02:25:13.998 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-16T02:25:13.999 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-16T02:25:13.999 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-04-16T02:25:13.999 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-16T02:25:13.999 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-16T02:25:13.999 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-16T02:25:13.999 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-16T02:25:14.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph 2024-04-16T02:25:14.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-04-16T02:25:14.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /etc/ceph 2024-04-16T02:25:14.127 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-04-16T02:25:14.158 INFO:tasks.cephadm:Writing seed config... 2024-04-16T02:25:14.159 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-16T02:25:14.159 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-16T02:25:14.159 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-16T02:25:14.159 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-16T02:25:14.159 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-16T02:25:14.159 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-16T02:25:14.159 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-16T02:25:14.159 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-16T02:25:14.159 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-16T02:25:14.160 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-16T02:25:14.161 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-16T02:25:14.162 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-16T02:25:14.162 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-16T02:25:14.162 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-16T02:25:14.162 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:25:14.162 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-16T02:25:14.185 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 80d1e958-fb98-11ee-bc8f-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-16T02:25:14.186 DEBUG:teuthology.orchestra.run.smithi151:mon.smithi151> sudo journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151.service 2024-04-16T02:25:14.228 INFO:tasks.cephadm:Bootstrapping... 2024-04-16T02:25:14.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 -v bootstrap --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.151 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-16T02:25:14.388 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T02:25:14.388 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.1', '-v', 'bootstrap', '--fsid', '80d1e958-fb98-11ee-bc8f-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.151', '--skip-admin-label'] 2024-04-16T02:25:14.414 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T02:25:14.414 INFO:teuthology.orchestra.run.smithi151.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-16T02:25:14.414 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying podman|docker is present... 2024-04-16T02:25:14.438 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T02:25:14.438 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying lvm2 is present... 2024-04-16T02:25:14.438 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying time synchronization is in place... 2024-04-16T02:25:14.445 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-16T02:25:14.446 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-16T02:25:14.451 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-16T02:25:14.452 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-04-16T02:25:14.458 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-04-16T02:25:14.464 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-04-16T02:25:14.464 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-04-16T02:25:14.464 INFO:teuthology.orchestra.run.smithi151.stdout:Repeating the final host check... 2024-04-16T02:25:14.488 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T02:25:14.488 INFO:teuthology.orchestra.run.smithi151.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-16T02:25:14.488 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl is present 2024-04-16T02:25:14.488 INFO:teuthology.orchestra.run.smithi151.stdout:lvcreate is present 2024-04-16T02:25:14.495 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-16T02:25:14.495 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-16T02:25:14.501 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-16T02:25:14.501 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-04-16T02:25:14.507 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-04-16T02:25:14.514 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-04-16T02:25:14.514 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-04-16T02:25:14.514 INFO:teuthology.orchestra.run.smithi151.stdout:Host looks OK 2024-04-16T02:25:14.514 INFO:teuthology.orchestra.run.smithi151.stdout:Cluster fsid: 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:25:14.514 INFO:teuthology.orchestra.run.smithi151.stdout:Acquiring lock 139763342913888 on /run/cephadm/80d1e958-fb98-11ee-bc8f-c7b262605968.lock 2024-04-16T02:25:14.514 INFO:teuthology.orchestra.run.smithi151.stdout:Lock 139763342913888 acquired on /run/cephadm/80d1e958-fb98-11ee-bc8f-c7b262605968.lock 2024-04-16T02:25:14.515 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 3300 ... 2024-04-16T02:25:14.515 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 6789 ... 2024-04-16T02:25:14.515 INFO:teuthology.orchestra.run.smithi151.stdout:Base mon IP(s) is [172.21.15.151:3300, 172.21.15.151:6789], mon addrv is [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-04-16T02:25:14.518 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.151 metric 100 2024-04-16T02:25:14.518 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.151 metric 100 2024-04-16T02:25:14.521 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-16T02:25:14.521 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-16T02:25:14.521 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1764sec hoplimit 64 pref medium 2024-04-16T02:25:14.523 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-16T02:25:14.523 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-16T02:25:14.523 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-16T02:25:14.523 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-16T02:25:14.523 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6f63/64 scope link 2024-04-16T02:25:14.523 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-16T02:25:14.524 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-04-16T02:25:14.524 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-04-16T02:25:14.524 INFO:teuthology.orchestra.run.smithi151.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-16T02:25:14.524 INFO:teuthology.orchestra.run.smithi151.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-16T02:25:14.525 INFO:teuthology.orchestra.run.smithi151.stdout:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-16T02:25:19.102 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-16T02:25:19.102 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.1... 2024-04-16T02:25:19.102 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Getting image source signatures 2024-04-16T02:25:19.103 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:a733d3c618b71f19c168ebecd1953429dce2c1631835ca182e9551c36dce5989 2024-04-16T02:25:19.103 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-04-16T02:25:19.103 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying config sha256:5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-16T02:25:19.103 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-16T02:25:27.720 INFO:teuthology.orchestra.run.smithi151.stdout:ceph: stdout ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-16T02:25:27.721 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-16T02:25:27.721 INFO:teuthology.orchestra.run.smithi151.stdout:Extracting ceph user uid/gid from container image... 2024-04-16T02:25:28.598 INFO:teuthology.orchestra.run.smithi151.stdout:stat: stdout 167 167 2024-04-16T02:25:28.599 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial keys... 2024-04-16T02:25:29.454 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQCY4R1mpBLBNhAAweUIgLA//E5KPydCXmLCNw== 2024-04-16T02:25:30.301 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQCZ4R1m8BcvLBAA3InDOtigQCWxShzDqGwjWA== 2024-04-16T02:25:31.174 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQCa4R1mnmtzIhAAy5s1O56UBEVJrIk33bYnuA== 2024-04-16T02:25:31.175 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial monmap... 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool for smithi151 [v2:172.21.15.151:3300,v1:172.21.15.151:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout:setting min_mon_release = pacific 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: set fsid to 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:25:32.067 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mon... 2024-04-16T02:25:33.136 INFO:teuthology.orchestra.run.smithi151.stdout:create mon.smithi151 on 2024-04-16T02:25:33.346 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-16T02:25:33.520 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-16T02:25:33.700 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968.target. 2024-04-16T02:25:33.700 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968.target. 2024-04-16T02:25:33.951 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151 2024-04-16T02:25:33.951 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151.service: Unit ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151.service not loaded. 2024-04-16T02:25:34.118 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968.target.wants/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151.service → /etc/systemd/system/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@.service. 2024-04-16T02:25:34.481 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:34 smithi151 podman[28612]: 2024-04-16 02:25:34.377244483 +0000 UTC m=+0.123026860 container create 69ba8bf2a050df122ecb7367fdf50f381760e4c25ac8971561edd65718505290 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=HEAD, org.label-schema.schema-version=1.0) 2024-04-16T02:25:34.659 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T02:25:34.659 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-04-16T02:25:34.666 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T02:25:34.666 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-04-16T02:25:34.666 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-04-16T02:25:34.666 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-04-16T02:25:34.667 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon to start... 2024-04-16T02:25:34.667 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon... 2024-04-16T02:25:34.741 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:34 smithi151 podman[28612]: 2024-04-16 02:25:34.563076882 +0000 UTC m=+0.308859272 container init 69ba8bf2a050df122ecb7367fdf50f381760e4c25ac8971561edd65718505290 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-04-16T02:25:34.741 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:34 smithi151 podman[28612]: 2024-04-16 02:25:34.566571886 +0000 UTC m=+0.312354263 container start 69ba8bf2a050df122ecb7367fdf50f381760e4c25ac8971561edd65718505290 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.29.1) 2024-04-16T02:25:34.741 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:34 smithi151 ceph-mon[28630]: mkfs 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:25:34.741 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:34 smithi151 ceph-mon[28630]: mon.smithi151 is new leader, mons smithi151 in quorum (ranks 0) 2024-04-16T02:25:34.741 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:34 smithi151 bash[28612]: 69ba8bf2a050df122ecb7367fdf50f381760e4c25ac8971561edd65718505290 2024-04-16T02:25:34.742 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:34 smithi151 systemd[1]: Started Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:25:35.694 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout cluster: 2024-04-16T02:25:35.694 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout id: 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout services: 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi151 (age 0.603312s) 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout data: 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pgs: 2024-04-16T02:25:35.695 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.131+0000 7fa179949700 1 Processor -- start 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.131+0000 7fa179949700 1 -- start start 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.132+0000 7fa179949700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa174105850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.132+0000 7fa179949700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa174105e20 con 0x7fa174105430 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.132+0000 7fa172ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa174105850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.132+0000 7fa172ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa174105850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56660/0 (socket says 172.21.15.151:56660) 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.132+0000 7fa172ffd700 1 -- 172.21.15.151:0/1574793459 learned_addr learned my addr 172.21.15.151:0/1574793459 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.132+0000 7fa172ffd700 1 -- 172.21.15.151:0/1574793459 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa174106630 con 0x7fa174105430 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa172ffd700 1 --2- 172.21.15.151:0/1574793459 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa174105850 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa1680098a0 tx=0x7fa168009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee887440c71a642 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa171ffb700 1 -- 172.21.15.151:0/1574793459 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa168004180 con 0x7fa174105430 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa171ffb700 1 -- 172.21.15.151:0/1574793459 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1680042e0 con 0x7fa174105430 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa171ffb700 1 -- 172.21.15.151:0/1574793459 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1680036a0 con 0x7fa174105430 2024-04-16T02:25:35.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa179949700 1 -- 172.21.15.151:0/1574793459 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 msgr2=0x7fa174105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa179949700 1 --2- 172.21.15.151:0/1574793459 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa174105850 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa1680098a0 tx=0x7fa168009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa179949700 1 -- 172.21.15.151:0/1574793459 shutdown_connections 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa179949700 1 --2- 172.21.15.151:0/1574793459 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa174105850 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa179949700 1 -- 172.21.15.151:0/1574793459 >> 172.21.15.151:0/1574793459 conn(0x7fa1741009b0 msgr2=0x7fa174102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa179949700 1 -- 172.21.15.151:0/1574793459 shutdown_connections 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.133+0000 7fa179949700 1 -- 172.21.15.151:0/1574793459 wait complete. 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.134+0000 7fa179949700 1 Processor -- start 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.134+0000 7fa179949700 1 -- start start 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.134+0000 7fa179949700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa1741999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.134+0000 7fa179949700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa174199ee0 con 0x7fa174105430 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.134+0000 7fa172ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa1741999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.134+0000 7fa172ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa1741999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56670/0 (socket says 172.21.15.151:56670) 2024-04-16T02:25:35.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.134+0000 7fa172ffd700 1 -- 172.21.15.151:0/858674216 learned_addr learned my addr 172.21.15.151:0/858674216 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.135+0000 7fa172ffd700 1 -- 172.21.15.151:0/858674216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa168009550 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.135+0000 7fa172ffd700 1 --2- 172.21.15.151:0/858674216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa1741999a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa168004000 tx=0x7fa168004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.135+0000 7fa178947700 1 -- 172.21.15.151:0/858674216 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa168003ce0 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.135+0000 7fa178947700 1 -- 172.21.15.151:0/858674216 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168020070 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.135+0000 7fa179949700 1 -- 172.21.15.151:0/858674216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17419a0e0 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.135+0000 7fa178947700 1 -- 172.21.15.151:0/858674216 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa168011630 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.135+0000 7fa179949700 1 -- 172.21.15.151:0/858674216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17419a580 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.135+0000 7fa178947700 1 -- 172.21.15.151:0/858674216 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa168022030 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.135+0000 7fa178947700 1 -- 172.21.15.151:0/858674216 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa168025d10 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.136+0000 7fa179949700 1 -- 172.21.15.151:0/858674216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa174193250 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.138+0000 7fa178947700 1 -- 172.21.15.151:0/858674216 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa168025480 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.207+0000 7fa179949700 1 -- 172.21.15.151:0/858674216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fa17402cc70 con 0x7fa174105430 2024-04-16T02:25:35.698 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.208+0000 7fa178947700 1 -- 172.21.15.151:0/858674216 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fa168011c20 con 0x7fa174105430 2024-04-16T02:25:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.208+0000 7fa179949700 1 -- 172.21.15.151:0/858674216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 msgr2=0x7fa1741999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.208+0000 7fa179949700 1 --2- 172.21.15.151:0/858674216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa1741999a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa168004000 tx=0x7fa168004750 comp rx=0 tx=0).stop 2024-04-16T02:25:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.208+0000 7fa179949700 1 -- 172.21.15.151:0/858674216 shutdown_connections 2024-04-16T02:25:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.208+0000 7fa179949700 1 --2- 172.21.15.151:0/858674216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa174105430 0x7fa1741999a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.208+0000 7fa179949700 1 -- 172.21.15.151:0/858674216 >> 172.21.15.151:0/858674216 conn(0x7fa1741009b0 msgr2=0x7fa174079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.208+0000 7fa179949700 1 -- 172.21.15.151:0/858674216 shutdown_connections 2024-04-16T02:25:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:35.208+0000 7fa179949700 1 -- 172.21.15.151:0/858674216 wait complete. 2024-04-16T02:25:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:mon is available 2024-04-16T02:25:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:Assimilating anything we can from ceph.conf... 2024-04-16T02:25:35.894 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:35 smithi151 ceph-mon[28630]: mon.smithi151 is new leader, mons smithi151 in quorum (ranks 0) 2024-04-16T02:25:35.894 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:35 smithi151 ceph-mon[28630]: monmap e1: 1 mons at {smithi151=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T02:25:35.894 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:35 smithi151 ceph-mon[28630]: fsmap 2024-04-16T02:25:35.894 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:35 smithi151 ceph-mon[28630]: osdmap e1: 0 total, 0 up, 0 in 2024-04-16T02:25:35.894 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:35 smithi151 ceph-mon[28630]: mgrmap e1: no daemons active 2024-04-16T02:25:35.894 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:35 smithi151 ceph-mon[28630]: from='client.? 172.21.15.151:0/858674216' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-16T02:25:36.725 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:36.725 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:36.726 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-04-16T02:25:36.727 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-16T02:25:36.727 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.094+0000 7f92f899c700 1 Processor -- start 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.094+0000 7f92f899c700 1 -- start start 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.095+0000 7f92f899c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f4106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.095+0000 7f92f899c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4106f50 con 0x7f92f4106560 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.095+0000 7f92f259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f4106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.095+0000 7f92f259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f4106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56674/0 (socket says 172.21.15.151:56674) 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.095+0000 7f92f259c700 1 -- 172.21.15.151:0/760040207 learned_addr learned my addr 172.21.15.151:0/760040207 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.095+0000 7f92f259c700 1 -- 172.21.15.151:0/760040207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f4107760 con 0x7f92f4106560 2024-04-16T02:25:36.728 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.095+0000 7f92f259c700 1 --2- 172.21.15.151:0/760040207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f4106980 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0098a0 tx=0x7f92ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=825a3842d8e09bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f159a700 1 -- 172.21.15.151:0/760040207 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92ec004180 con 0x7f92f4106560 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f159a700 1 -- 172.21.15.151:0/760040207 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec0042e0 con 0x7f92f4106560 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f159a700 1 -- 172.21.15.151:0/760040207 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92ec0036a0 con 0x7f92f4106560 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f899c700 1 -- 172.21.15.151:0/760040207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 msgr2=0x7f92f4106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f899c700 1 --2- 172.21.15.151:0/760040207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f4106980 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0098a0 tx=0x7f92ec009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f899c700 1 -- 172.21.15.151:0/760040207 shutdown_connections 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f899c700 1 --2- 172.21.15.151:0/760040207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f4106980 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f899c700 1 -- 172.21.15.151:0/760040207 >> 172.21.15.151:0/760040207 conn(0x7f92f4101b40 msgr2=0x7f92f4103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f899c700 1 -- 172.21.15.151:0/760040207 shutdown_connections 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.096+0000 7f92f899c700 1 -- 172.21.15.151:0/760040207 wait complete. 2024-04-16T02:25:36.729 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.097+0000 7f92f899c700 1 Processor -- start 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.097+0000 7f92f899c700 1 -- start start 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.097+0000 7f92f899c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f41999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.097+0000 7f92f899c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4199f00 con 0x7f92f4106560 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.097+0000 7f92f259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f41999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.097+0000 7f92f259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f41999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56684/0 (socket says 172.21.15.151:56684) 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.097+0000 7f92f259c700 1 -- 172.21.15.151:0/1874315155 learned_addr learned my addr 172.21.15.151:0/1874315155 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.098+0000 7f92f259c700 1 -- 172.21.15.151:0/1874315155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec009550 con 0x7f92f4106560 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.098+0000 7f92f259c700 1 --2- 172.21.15.151:0/1874315155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f41999c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f92ec004000 tx=0x7f92ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.098+0000 7f92eb7fe700 1 -- 172.21.15.151:0/1874315155 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92ec003ce0 con 0x7f92f4106560 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.098+0000 7f92eb7fe700 1 -- 172.21.15.151:0/1874315155 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec020070 con 0x7f92f4106560 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.098+0000 7f92eb7fe700 1 -- 172.21.15.151:0/1874315155 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92ec011630 con 0x7f92f4106560 2024-04-16T02:25:36.730 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.098+0000 7f92f899c700 1 -- 172.21.15.151:0/1874315155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f419a100 con 0x7f92f4106560 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.098+0000 7f92f899c700 1 -- 172.21.15.151:0/1874315155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f419a5a0 con 0x7f92f4106560 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.098+0000 7f92eb7fe700 1 -- 172.21.15.151:0/1874315155 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f92ec022030 con 0x7f92f4106560 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.098+0000 7f92eb7fe700 1 -- 172.21.15.151:0/1874315155 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f92ec025d40 con 0x7f92f4106560 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.099+0000 7f92f899c700 1 -- 172.21.15.151:0/1874315155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f4193250 con 0x7f92f4106560 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.101+0000 7f92eb7fe700 1 -- 172.21.15.151:0/1874315155 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f92ec025480 con 0x7f92f4106560 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.169+0000 7f92f899c700 1 -- 172.21.15.151:0/1874315155 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f92f402d0f0 con 0x7f92f4106560 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.172+0000 7f92eb7fe700 1 -- 172.21.15.151:0/1874315155 <== mon.0 v2:172.21.15.151:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f92ec011c20 con 0x7f92f4106560 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.172+0000 7f92eb7fe700 1 -- 172.21.15.151:0/1874315155 <== mon.0 v2:172.21.15.151:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f92f402d0f0 con 0x7f92f4106560 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.173+0000 7f92f899c700 1 -- 172.21.15.151:0/1874315155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 msgr2=0x7f92f41999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.173+0000 7f92f899c700 1 --2- 172.21.15.151:0/1874315155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f41999c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f92ec004000 tx=0x7f92ec004750 comp rx=0 tx=0).stop 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.173+0000 7f92f899c700 1 -- 172.21.15.151:0/1874315155 shutdown_connections 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.173+0000 7f92f899c700 1 --2- 172.21.15.151:0/1874315155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92f4106560 0x7f92f41999c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.173+0000 7f92f899c700 1 -- 172.21.15.151:0/1874315155 >> 172.21.15.151:0/1874315155 conn(0x7f92f4101b40 msgr2=0x7f92f418df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:36.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.173+0000 7f92f899c700 1 -- 172.21.15.151:0/1874315155 shutdown_connections 2024-04-16T02:25:36.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:36.173+0000 7f92f899c700 1 -- 172.21.15.151:0/1874315155 wait complete. 2024-04-16T02:25:36.732 INFO:teuthology.orchestra.run.smithi151.stdout:Generating new minimal ceph.conf... 2024-04-16T02:25:36.931 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:36 smithi151 ceph-mon[28630]: from='client.? 172.21.15.151:0/1874315155' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-16T02:25:36.931 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:36 smithi151 ceph-mon[28630]: from='client.? 172.21.15.151:0/1874315155' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-16T02:25:37.788 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.158+0000 7fbd4ef84700 1 Processor -- start 2024-04-16T02:25:37.788 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.158+0000 7fbd4ef84700 1 -- start start 2024-04-16T02:25:37.788 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.158+0000 7fbd4ef84700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4807a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:37.788 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.158+0000 7fbd4ef84700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4807a890 con 0x7fbd4807be60 2024-04-16T02:25:37.788 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.158+0000 7fbd4cd20700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4807a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:37.788 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.158+0000 7fbd4cd20700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4807a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44032/0 (socket says 172.21.15.151:44032) 2024-04-16T02:25:37.788 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.158+0000 7fbd4cd20700 1 -- 172.21.15.151:0/623602635 learned_addr learned my addr 172.21.15.151:0/623602635 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:37.788 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.159+0000 7fbd4cd20700 1 -- 172.21.15.151:0/623602635 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4807a9d0 con 0x7fbd4807be60 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.159+0000 7fbd4cd20700 1 --2- 172.21.15.151:0/623602635 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4807a2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c0098a0 tx=0x7fbd3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b4d5e4fbbabb88 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.159+0000 7fbd477fe700 1 -- 172.21.15.151:0/623602635 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd3c004180 con 0x7fbd4807be60 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.159+0000 7fbd477fe700 1 -- 172.21.15.151:0/623602635 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbd3c0042e0 con 0x7fbd4807be60 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.159+0000 7fbd477fe700 1 -- 172.21.15.151:0/623602635 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd3c011a50 con 0x7fbd4807be60 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.159+0000 7fbd4ef84700 1 -- 172.21.15.151:0/623602635 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 msgr2=0x7fbd4807a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.159+0000 7fbd4ef84700 1 --2- 172.21.15.151:0/623602635 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4807a2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c0098a0 tx=0x7fbd3c009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4ef84700 1 -- 172.21.15.151:0/623602635 shutdown_connections 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4ef84700 1 --2- 172.21.15.151:0/623602635 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4807a2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4ef84700 1 -- 172.21.15.151:0/623602635 >> 172.21.15.151:0/623602635 conn(0x7fbd48101b40 msgr2=0x7fbd48103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4ef84700 1 -- 172.21.15.151:0/623602635 shutdown_connections 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4ef84700 1 -- 172.21.15.151:0/623602635 wait complete. 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4ef84700 1 Processor -- start 2024-04-16T02:25:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4ef84700 1 -- start start 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4ef84700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4819c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4ef84700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4819c910 con 0x7fbd4807be60 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.160+0000 7fbd4cd20700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4819c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd4cd20700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4819c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44036/0 (socket says 172.21.15.151:44036) 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd4cd20700 1 -- 172.21.15.151:0/2902645369 learned_addr learned my addr 172.21.15.151:0/2902645369 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd4cd20700 1 -- 172.21.15.151:0/2902645369 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3c009550 con 0x7fbd4807be60 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd4cd20700 1 --2- 172.21.15.151:0/2902645369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4819c3d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c003de0 tx=0x7fbd3c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd45ffb700 1 -- 172.21.15.151:0/2902645369 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd3c003930 con 0x7fbd4807be60 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd45ffb700 1 -- 172.21.15.151:0/2902645369 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbd3c003a90 con 0x7fbd4807be60 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd4ef84700 1 -- 172.21.15.151:0/2902645369 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4819cb10 con 0x7fbd4807be60 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd45ffb700 1 -- 172.21.15.151:0/2902645369 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd3c019730 con 0x7fbd4807be60 2024-04-16T02:25:37.790 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd4ef84700 1 -- 172.21.15.151:0/2902645369 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4819cfb0 con 0x7fbd4807be60 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd45ffb700 1 -- 172.21.15.151:0/2902645369 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbd3c019c80 con 0x7fbd4807be60 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.161+0000 7fbd45ffb700 1 -- 172.21.15.151:0/2902645369 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbd3c02c3a0 con 0x7fbd4807be60 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.162+0000 7fbd4ef84700 1 -- 172.21.15.151:0/2902645369 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd480623c0 con 0x7fbd4807be60 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.164+0000 7fbd45ffb700 1 -- 172.21.15.151:0/2902645369 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fbd3c01abc0 con 0x7fbd4807be60 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.232+0000 7fbd4ef84700 1 -- 172.21.15.151:0/2902645369 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fbd4819f970 con 0x7fbd4807be60 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.232+0000 7fbd45ffb700 1 -- 172.21.15.151:0/2902645369 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fbd3c01a430 con 0x7fbd4807be60 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.233+0000 7fbd4ef84700 1 -- 172.21.15.151:0/2902645369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 msgr2=0x7fbd4819c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.233+0000 7fbd4ef84700 1 --2- 172.21.15.151:0/2902645369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4819c3d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c003de0 tx=0x7fbd3c011ab0 comp rx=0 tx=0).stop 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.233+0000 7fbd4ef84700 1 -- 172.21.15.151:0/2902645369 shutdown_connections 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.233+0000 7fbd4ef84700 1 --2- 172.21.15.151:0/2902645369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd4807be60 0x7fbd4819c3d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:37.791 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.233+0000 7fbd4ef84700 1 -- 172.21.15.151:0/2902645369 >> 172.21.15.151:0/2902645369 conn(0x7fbd48101b40 msgr2=0x7fbd481028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:37.792 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.233+0000 7fbd4ef84700 1 -- 172.21.15.151:0/2902645369 shutdown_connections 2024-04-16T02:25:37.792 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:37.233+0000 7fbd4ef84700 1 -- 172.21.15.151:0/2902645369 wait complete. 2024-04-16T02:25:37.792 INFO:teuthology.orchestra.run.smithi151.stdout:Restarting the monitor... 2024-04-16T02:25:37.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:37 smithi151 ceph-mon[28630]: from='client.? 172.21.15.151:0/2902645369' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:25:37.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:37 smithi151 systemd[1]: Stopping Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:25:38.493 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:38 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[28626]: 2024-04-16T02:25:38.034+0000 7fcbd3c78700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi151 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T02:25:38.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:38 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[28626]: 2024-04-16T02:25:38.034+0000 7fcbd3c78700 -1 mon.smithi151@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T02:25:38.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:38 smithi151 podman[28876]: 2024-04-16 02:25:38.065752757 +0000 UTC m=+0.118793302 container died 69ba8bf2a050df122ecb7367fdf50f381760e4c25ac8971561edd65718505290 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-04-16T02:25:38.993 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:38 smithi151 podman[28876]: 2024-04-16 02:25:38.709412213 +0000 UTC m=+0.762452755 container cleanup 69ba8bf2a050df122ecb7367fdf50f381760e4c25ac8971561edd65718505290 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_CLEAN=True) 2024-04-16T02:25:38.993 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:38 smithi151 bash[28876]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151 2024-04-16T02:25:39.391 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 podman[28889]: 2024-04-16 02:25:39.010212905 +0000 UTC m=+0.945120807 container remove 69ba8bf2a050df122ecb7367fdf50f381760e4c25ac8971561edd65718505290 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-16T02:25:39.391 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151.service: Deactivated successfully. 2024-04-16T02:25:39.391 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 systemd[1]: Stopped Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:25:39.391 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 systemd[1]: Starting Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:25:39.646 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 podman[28985]: 2024-04-16 02:25:39.294342059 +0000 UTC m=+0.017916180 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph:v18.2.1 2024-04-16T02:25:39.646 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 podman[28985]: 2024-04-16 02:25:39.394302635 +0000 UTC m=+0.117876753 container create 051460b1db9f73ba31fe8575ade143d8b81ea32bf1082dff6ce456e8dc968247 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-04-16T02:25:39.646 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 podman[28985]: 2024-04-16 02:25:39.613495595 +0000 UTC m=+0.337069710 container init 051460b1db9f73ba31fe8575ade143d8b81ea32bf1082dff6ce456e8dc968247 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-04-16T02:25:39.646 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 podman[28985]: 2024-04-16 02:25:39.616595957 +0000 UTC m=+0.340170072 container start 051460b1db9f73ba31fe8575ade143d8b81ea32bf1082dff6ce456e8dc968247 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T02:25:39.646 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T02:25:39.646 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-04-16T02:25:39.646 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: pidfile_write: ignore empty --pid-file 2024-04-16T02:25:39.646 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: load: jerasure load: lrc 2024-04-16T02:25:39.692 INFO:teuthology.orchestra.run.smithi151.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: RocksDB version: 7.9.2 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Git sha 0 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Compile date 2023-12-11 22:07:34 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: DB SUMMARY 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: DB Session ID: 1T7SHW8BVC804WLIG7U6 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: CURRENT file: CURRENT 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi151/store.db dir, Total Num: 1, files: 000008.sst 2024-04-16T02:25:39.905 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi151/store.db: 000009.log size: 98387 ; 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.error_if_exists: 0 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.create_if_missing: 0 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.paranoid_checks: 1 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.env: 0x55ebc30f0720 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.info_log: 0x55ebc517b2e0 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.statistics: (nil) 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.use_fsync: 0 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_log_file_size: 0 2024-04-16T02:25:39.906 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.allow_fallocate: 1 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.use_direct_reads: 0 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.db_log_dir: 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.wal_dir: 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T02:25:39.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.write_buffer_manager: 0x55ebc43ec140 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.unordered_write: 0 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T02:25:39.908 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.row_cache: None 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.wal_filter: None 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.two_write_queues: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.wal_compression: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.atomic_flush: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.log_readahead_size: 0 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T02:25:39.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T02:25:39.910 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T02:25:39.910 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T02:25:39.910 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T02:25:39.910 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T02:25:39.910 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T02:25:39.910 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_background_jobs: 2 2024-04-16T02:25:39.910 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_background_compactions: -1 2024-04-16T02:25:39.910 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_subcompactions: 1 2024-04-16T02:25:39.911 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_open_files: -1 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T02:25:39.912 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_background_flushes: -1 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Compression algorithms supported: 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: kZSTD supported: 0 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: kXpressCompression supported: 0 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: kZlibCompression supported: 1 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: kSnappyCompression supported: 1 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: kLZ4Compression supported: 1 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: kBZip2Compression supported: 0 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi151/store.db/MANIFEST-000010 2024-04-16T02:25:39.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.merge_operator: 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_filter: None 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ebc517b3e0) 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-04-16T02:25:39.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: index_type: 0 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: data_block_index_type: 0 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: index_shortening: 1 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: checksum: 4 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: no_block_cache: 0 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_cache: 0x55ebc448d1f0 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_cache_options: 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: capacity : 536870912 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: num_shard_bits : 4 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: strict_capacity_limit : 0 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-04-16T02:25:39.915 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_cache_compressed: (nil) 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: persistent_cache: (nil) 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_size: 4096 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_size_deviation: 10 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_restart_interval: 16 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: index_block_restart_interval: 1 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: metadata_block_size: 4096 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: partition_filters: 0 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: use_delta_encoding: 1 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: filter_policy: bloomfilter 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: whole_key_filtering: 1 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: verify_compression: 0 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-04-16T02:25:39.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: format_version: 5 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: enable_index_compression: 1 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_align: 0 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: max_auto_readahead_size: 262144 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: prepopulate_block_cache: 0 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: initial_auto_readahead_size: 8192 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression: NoCompression 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.num_levels: 7 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T02:25:39.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T02:25:39.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T02:25:39.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T02:25:39.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T02:25:39.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T02:25:39.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T02:25:39.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T02:25:39.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T02:25:39.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T02:25:39.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T02:25:39.919 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T02:25:39.920 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.table_properties_collectors: 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.inplace_update_support: 0 2024-04-16T02:25:39.921 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.bloom_locality: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.max_successive_merges: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.ttl: 2592000 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T02:25:39.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.enable_blob_files: false 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.min_blob_size: 0 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi151/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 494c98a9-92d5-4767-8003-d4a132cd652d 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713234339646635, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-16T02:25:39.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713234339647642, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93631, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91653, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84262, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713234339, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "494c98a9-92d5-4767-8003-d4a132cd652d", "db_session_id": "1T7SHW8BVC804WLIG7U6", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713234339647723, "job": 1, "event": "recovery_finished"} 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi151/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ebc452a000 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: DB pointer 0x55ebc4516000 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ** DB Stats ** 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T02:25:39.924 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ** Compaction Stats [default] ** 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: L0 2/0 93.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Sum 2/0 93.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ** Compaction Stats [default] ** 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:25:39.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Cumulative compaction: 0.00 GB write, 29.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Interval compaction: 0.00 GB write, 29.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Block cache BinnedLRUCache@0x55ebc448d1f0#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T02:25:39.926 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: starting mon.smithi151 rank 0 at public addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] at bind addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi151 fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mon.smithi151@-1(???) e1 preinit fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mon.smithi151@-1(???).mds e1 new map 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mon.smithi151@-1(???).mds e1 print_map 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: e1 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: legacy client fscid: -1 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: No filesystems configured 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mon.smithi151@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mon.smithi151@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mon.smithi151@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mon.smithi151@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mon.smithi151@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta expand map: {default=false} 2024-04-16T02:25:39.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta from 'false' to 'false' 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta expanded map: {default=false} 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta expand map: {default=info} 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta from 'info' to 'info' 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta expanded map: {default=info} 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta expand map: {default=daemon} 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta expanded map: {default=daemon} 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta expand map: {default=debug} 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta from 'debug' to 'debug' 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: expand_channel_meta expanded map: {default=debug} 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 bash[28985]: 051460b1db9f73ba31fe8575ade143d8b81ea32bf1082dff6ce456e8dc968247 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 systemd[1]: Started Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mon.smithi151 is new leader, mons smithi151 in quorum (ranks 0) 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: monmap e1: 1 mons at {smithi151=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: fsmap 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: osdmap e1: 0 total, 0 up, 0 in 2024-04-16T02:25:39.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:39 smithi151 ceph-mon[29003]: mgrmap e1: no daemons active 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.110+0000 7f2f2bfab700 1 Processor -- start 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.110+0000 7f2f2bfab700 1 -- start start 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.111+0000 7f2f2bfab700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f24104c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.111+0000 7f2f2bfab700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f24105230 con 0x7f2f24104840 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.111+0000 7f2f29d47700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f24104c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.111+0000 7f2f29d47700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f24104c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44048/0 (socket says 172.21.15.151:44048) 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.111+0000 7f2f29d47700 1 -- 172.21.15.151:0/348867911 learned_addr learned my addr 172.21.15.151:0/348867911 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.111+0000 7f2f29d47700 1 -- 172.21.15.151:0/348867911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f241077d0 con 0x7f2f24104840 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f29d47700 1 --2- 172.21.15.151:0/348867911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f24104c60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2f200098a0 tx=0x7f2f20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbe9c1762fcfbb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f28d45700 1 -- 172.21.15.151:0/348867911 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f20004180 con 0x7f2f24104840 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f28d45700 1 -- 172.21.15.151:0/348867911 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2f200042e0 con 0x7f2f24104840 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f28d45700 1 -- 172.21.15.151:0/348867911 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f20011a50 con 0x7f2f24104840 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f2bfab700 1 -- 172.21.15.151:0/348867911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 msgr2=0x7f2f24104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f2bfab700 1 --2- 172.21.15.151:0/348867911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f24104c60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2f200098a0 tx=0x7f2f20009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f2bfab700 1 -- 172.21.15.151:0/348867911 shutdown_connections 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f2bfab700 1 --2- 172.21.15.151:0/348867911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f24104c60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f2bfab700 1 -- 172.21.15.151:0/348867911 >> 172.21.15.151:0/348867911 conn(0x7f2f24100170 msgr2=0x7f2f241025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f2bfab700 1 -- 172.21.15.151:0/348867911 shutdown_connections 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.112+0000 7f2f2bfab700 1 -- 172.21.15.151:0/348867911 wait complete. 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.113+0000 7f2f2bfab700 1 Processor -- start 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.113+0000 7f2f2bfab700 1 -- start start 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.113+0000 7f2f2bfab700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f2419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.113+0000 7f2f2bfab700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2419a740 con 0x7f2f24104840 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.113+0000 7f2f29d47700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f2419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:40.685 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.113+0000 7f2f29d47700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f2419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44060/0 (socket says 172.21.15.151:44060) 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.113+0000 7f2f29d47700 1 -- 172.21.15.151:0/1013169877 learned_addr learned my addr 172.21.15.151:0/1013169877 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.114+0000 7f2f29d47700 1 -- 172.21.15.151:0/1013169877 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20009550 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.114+0000 7f2f29d47700 1 --2- 172.21.15.151:0/1013169877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f2419a200 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009520 tx=0x7f2f20011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.114+0000 7f2f16ffd700 1 -- 172.21.15.151:0/1013169877 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f20003a80 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.114+0000 7f2f16ffd700 1 -- 172.21.15.151:0/1013169877 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2f20003be0 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.114+0000 7f2f2bfab700 1 -- 172.21.15.151:0/1013169877 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2419a940 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.114+0000 7f2f16ffd700 1 -- 172.21.15.151:0/1013169877 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f200197b0 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.114+0000 7f2f2bfab700 1 -- 172.21.15.151:0/1013169877 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2419ade0 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.114+0000 7f2f16ffd700 1 -- 172.21.15.151:0/1013169877 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2f2002d440 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.114+0000 7f2f16ffd700 1 -- 172.21.15.151:0/1013169877 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2f20024800 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.115+0000 7f2f2bfab700 1 -- 172.21.15.151:0/1013169877 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f240623c0 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.117+0000 7f2f16ffd700 1 -- 172.21.15.151:0/1013169877 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f2f2001ac30 con 0x7f2f24104840 2024-04-16T02:25:40.686 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.183+0000 7f2f2bfab700 1 -- 172.21.15.151:0/1013169877 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f2f2410aa80 con 0x7f2f24104840 2024-04-16T02:25:40.687 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.186+0000 7f2f16ffd700 1 -- 172.21.15.151:0/1013169877 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2f20005a20 con 0x7f2f24104840 2024-04-16T02:25:40.687 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.186+0000 7f2f2bfab700 1 -- 172.21.15.151:0/1013169877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 msgr2=0x7f2f2419a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:40.687 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.186+0000 7f2f2bfab700 1 --2- 172.21.15.151:0/1013169877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f2419a200 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009520 tx=0x7f2f20011ab0 comp rx=0 tx=0).stop 2024-04-16T02:25:40.687 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.186+0000 7f2f2bfab700 1 -- 172.21.15.151:0/1013169877 shutdown_connections 2024-04-16T02:25:40.687 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.186+0000 7f2f2bfab700 1 --2- 172.21.15.151:0/1013169877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24104840 0x7f2f2419a200 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:40.687 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.186+0000 7f2f2bfab700 1 -- 172.21.15.151:0/1013169877 >> 172.21.15.151:0/1013169877 conn(0x7f2f24100170 msgr2=0x7f2f24101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:40.687 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.187+0000 7f2f2bfab700 1 -- 172.21.15.151:0/1013169877 shutdown_connections 2024-04-16T02:25:40.687 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:40.187+0000 7f2f2bfab700 1 -- 172.21.15.151:0/1013169877 wait complete. 2024-04-16T02:25:40.687 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-16T02:25:40.702 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-16T02:25:40.702 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mgr... 2024-04-16T02:25:40.702 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-16T02:25:40.703 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-16T02:25:40.703 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-16T02:25:40.939 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mgr.smithi151.uqqame 2024-04-16T02:25:40.940 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mgr.smithi151.uqqame.service: Unit ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mgr.smithi151.uqqame.service not loaded. 2024-04-16T02:25:41.083 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968.target.wants/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mgr.smithi151.uqqame.service → /etc/systemd/system/ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@.service. 2024-04-16T02:25:41.243 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:41 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1013169877' entity='client.admin' 2024-04-16T02:25:41.644 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T02:25:41.644 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-04-16T02:25:41.651 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T02:25:41.651 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-04-16T02:25:41.651 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-04-16T02:25:41.651 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-04-16T02:25:41.660 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T02:25:41.660 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-04-16T02:25:41.667 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T02:25:41.667 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-04-16T02:25:41.667 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-04-16T02:25:41.668 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-16T02:25:41.668 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr to start... 2024-04-16T02:25:41.668 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr... 2024-04-16T02:25:42.493 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:42 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2727009846' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T02:25:42.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:42.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-04-16T02:25:42.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "80d1e958-fb98-11ee-bc8f-c7b262605968", 2024-04-16T02:25:42.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T02:25:42.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T02:25:42.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T02:25:42.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T02:25:42.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:42.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T02:25:42.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T02:25:42.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-04-16T02:25:42.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "smithi151" 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T02:25:42.879 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T02:25:42.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T02:25:42.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T02:25:42.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T02:25:34.606666+0000", 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.138+0000 7f366ad5c700 1 Processor -- start 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.138+0000 7f366ad5c700 1 -- start start 2024-04-16T02:25:42.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.138+0000 7f366ad5c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664106e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.138+0000 7f366ad5c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3664107440 con 0x7f3664106a50 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.138+0000 7f3668af8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.138+0000 7f3668af8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44088/0 (socket says 172.21.15.151:44088) 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.138+0000 7f3668af8700 1 -- 172.21.15.151:0/2320429120 learned_addr learned my addr 172.21.15.151:0/2320429120 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.139+0000 7f3668af8700 1 -- 172.21.15.151:0/2320429120 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3664105660 con 0x7f3664106a50 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.139+0000 7f3668af8700 1 --2- 172.21.15.151:0/2320429120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664106e70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f365c0098a0 tx=0x7f365c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d590ad6ec64a54 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.139+0000 7f36637fe700 1 -- 172.21.15.151:0/2320429120 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f365c004180 con 0x7f3664106a50 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.139+0000 7f36637fe700 1 -- 172.21.15.151:0/2320429120 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f365c0042e0 con 0x7f3664106a50 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.139+0000 7f36637fe700 1 -- 172.21.15.151:0/2320429120 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f365c011a50 con 0x7f3664106a50 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.140+0000 7f366ad5c700 1 -- 172.21.15.151:0/2320429120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 msgr2=0x7f3664106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.140+0000 7f366ad5c700 1 --2- 172.21.15.151:0/2320429120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664106e70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f365c0098a0 tx=0x7f365c009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.140+0000 7f366ad5c700 1 -- 172.21.15.151:0/2320429120 shutdown_connections 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.140+0000 7f366ad5c700 1 --2- 172.21.15.151:0/2320429120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664106e70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.140+0000 7f366ad5c700 1 -- 172.21.15.151:0/2320429120 >> 172.21.15.151:0/2320429120 conn(0x7f3664100190 msgr2=0x7f36641025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.140+0000 7f366ad5c700 1 -- 172.21.15.151:0/2320429120 shutdown_connections 2024-04-16T02:25:42.891 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.140+0000 7f366ad5c700 1 -- 172.21.15.151:0/2320429120 wait complete. 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.140+0000 7f366ad5c700 1 Processor -- start 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.140+0000 7f366ad5c700 1 -- start start 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f366ad5c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664190640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f366ad5c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3664190b80 con 0x7f3664106a50 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f3668af8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664190640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f3668af8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664190640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44096/0 (socket says 172.21.15.151:44096) 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f3668af8700 1 -- 172.21.15.151:0/2727009846 learned_addr learned my addr 172.21.15.151:0/2727009846 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f3668af8700 1 -- 172.21.15.151:0/2727009846 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c009550 con 0x7f3664106a50 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f3668af8700 1 --2- 172.21.15.151:0/2727009846 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664190640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f365c006af0 tx=0x7f365c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f3661ffb700 1 -- 172.21.15.151:0/2727009846 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f365c003b90 con 0x7f3664106a50 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f3661ffb700 1 -- 172.21.15.151:0/2727009846 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f365c01a8a0 con 0x7f3664106a50 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f366ad5c700 1 -- 172.21.15.151:0/2727009846 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366418ec90 con 0x7f3664106a50 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.141+0000 7f3661ffb700 1 -- 172.21.15.151:0/2727009846 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f365c019880 con 0x7f3664106a50 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.142+0000 7f366ad5c700 1 -- 172.21.15.151:0/2727009846 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366418f130 con 0x7f3664106a50 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.142+0000 7f3661ffb700 1 -- 172.21.15.151:0/2727009846 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f365c02d440 con 0x7f3664106a50 2024-04-16T02:25:42.892 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.142+0000 7f3661ffb700 1 -- 172.21.15.151:0/2727009846 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f365c024800 con 0x7f3664106a50 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.143+0000 7f366ad5c700 1 -- 172.21.15.151:0/2727009846 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36640623c0 con 0x7f3664106a50 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.145+0000 7f3661ffb700 1 -- 172.21.15.151:0/2727009846 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f365c01aa10 con 0x7f3664106a50 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.266+0000 7f366ad5c700 1 -- 172.21.15.151:0/2727009846 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f36641098a0 con 0x7f3664106a50 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.266+0000 7f3661ffb700 1 -- 172.21.15.151:0/2727009846 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f365c01abb0 con 0x7f3664106a50 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.267+0000 7f366ad5c700 1 -- 172.21.15.151:0/2727009846 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 msgr2=0x7f3664190640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.267+0000 7f366ad5c700 1 --2- 172.21.15.151:0/2727009846 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664190640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f365c006af0 tx=0x7f365c011a90 comp rx=0 tx=0).stop 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.267+0000 7f366ad5c700 1 -- 172.21.15.151:0/2727009846 shutdown_connections 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.267+0000 7f366ad5c700 1 --2- 172.21.15.151:0/2727009846 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3664106a50 0x7f3664190640 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.267+0000 7f366ad5c700 1 -- 172.21.15.151:0/2727009846 >> 172.21.15.151:0/2727009846 conn(0x7f3664100190 msgr2=0x7f3664104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.267+0000 7f366ad5c700 1 -- 172.21.15.151:0/2727009846 shutdown_connections 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:42.267+0000 7f366ad5c700 1 -- 172.21.15.151:0/2727009846 wait complete. 2024-04-16T02:25:42.893 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (1/15)... 2024-04-16T02:25:45.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:45 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/4114312791' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T02:25:45.918 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "80d1e958-fb98-11ee-bc8f-c7b262605968", 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T02:25:45.919 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "smithi151" 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T02:25:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T02:25:45.928 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T02:25:45.928 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T02:25:45.928 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T02:25:45.928 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:45.928 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T02:25:45.929 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T02:25:34.606666+0000", 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T02:25:45.930 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.319+0000 7f6b27606700 1 Processor -- start 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.319+0000 7f6b27606700 1 -- start start 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.319+0000 7f6b27606700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20106f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.319+0000 7f6b27606700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b201074e0 con 0x7f6b20106af0 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.319+0000 7f6b253a2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20106f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.319+0000 7f6b253a2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20106f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44110/0 (socket says 172.21.15.151:44110) 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.319+0000 7f6b253a2700 1 -- 172.21.15.151:0/3704059291 learned_addr learned my addr 172.21.15.151:0/3704059291 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.319+0000 7f6b253a2700 1 -- 172.21.15.151:0/3704059291 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b201056c0 con 0x7f6b20106af0 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.320+0000 7f6b253a2700 1 --2- 172.21.15.151:0/3704059291 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20106f10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0098a0 tx=0x7f6b1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c8c1da513d3edf server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.320+0000 7f6b17fff700 1 -- 172.21.15.151:0/3704059291 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b1c004180 con 0x7f6b20106af0 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.320+0000 7f6b17fff700 1 -- 172.21.15.151:0/3704059291 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6b1c0042e0 con 0x7f6b20106af0 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.320+0000 7f6b17fff700 1 -- 172.21.15.151:0/3704059291 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b1c011a50 con 0x7f6b20106af0 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.320+0000 7f6b27606700 1 -- 172.21.15.151:0/3704059291 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 msgr2=0x7f6b20106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:45.931 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.320+0000 7f6b27606700 1 --2- 172.21.15.151:0/3704059291 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20106f10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0098a0 tx=0x7f6b1c009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.320+0000 7f6b27606700 1 -- 172.21.15.151:0/3704059291 shutdown_connections 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.320+0000 7f6b27606700 1 --2- 172.21.15.151:0/3704059291 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20106f10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.320+0000 7f6b27606700 1 -- 172.21.15.151:0/3704059291 >> 172.21.15.151:0/3704059291 conn(0x7f6b20100190 msgr2=0x7f6b201025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.321+0000 7f6b27606700 1 -- 172.21.15.151:0/3704059291 shutdown_connections 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.321+0000 7f6b27606700 1 -- 172.21.15.151:0/3704059291 wait complete. 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.321+0000 7f6b27606700 1 Processor -- start 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.321+0000 7f6b27606700 1 -- start start 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.321+0000 7f6b27606700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20105110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.321+0000 7f6b27606700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2019cf50 con 0x7f6b20106af0 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.321+0000 7f6b253a2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20105110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.321+0000 7f6b253a2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20105110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44116/0 (socket says 172.21.15.151:44116) 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.321+0000 7f6b253a2700 1 -- 172.21.15.151:0/4114312791 learned_addr learned my addr 172.21.15.151:0/4114312791 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.322+0000 7f6b253a2700 1 -- 172.21.15.151:0/4114312791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c009550 con 0x7f6b20106af0 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.322+0000 7f6b253a2700 1 --2- 172.21.15.151:0/4114312791 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20105110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c011a00 tx=0x7f6b1c011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.322+0000 7f6b167fc700 1 -- 172.21.15.151:0/4114312791 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b1c003ac0 con 0x7f6b20106af0 2024-04-16T02:25:45.932 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.322+0000 7f6b167fc700 1 -- 172.21.15.151:0/4114312791 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6b1c0248a0 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.322+0000 7f6b27606700 1 -- 172.21.15.151:0/4114312791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2019d090 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.322+0000 7f6b167fc700 1 -- 172.21.15.151:0/4114312791 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b1c01a8b0 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.322+0000 7f6b27606700 1 -- 172.21.15.151:0/4114312791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2019d530 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.322+0000 7f6b167fc700 1 -- 172.21.15.151:0/4114312791 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6b1c02d3f0 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.322+0000 7f6b167fc700 1 -- 172.21.15.151:0/4114312791 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6b1c019790 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.323+0000 7f6b27606700 1 -- 172.21.15.151:0/4114312791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b200623c0 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.327+0000 7f6b167fc700 1 -- 172.21.15.151:0/4114312791 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f6b1c024430 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.396+0000 7f6b27606700 1 -- 172.21.15.151:0/4114312791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6b201098c0 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.397+0000 7f6b167fc700 1 -- 172.21.15.151:0/4114312791 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f6b1c035090 con 0x7f6b20106af0 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.398+0000 7f6b27606700 1 -- 172.21.15.151:0/4114312791 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 msgr2=0x7f6b20105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.398+0000 7f6b27606700 1 --2- 172.21.15.151:0/4114312791 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20105110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c011a00 tx=0x7f6b1c011ae0 comp rx=0 tx=0).stop 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.398+0000 7f6b27606700 1 -- 172.21.15.151:0/4114312791 shutdown_connections 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.398+0000 7f6b27606700 1 --2- 172.21.15.151:0/4114312791 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b20106af0 0x7f6b20105110 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.398+0000 7f6b27606700 1 -- 172.21.15.151:0/4114312791 >> 172.21.15.151:0/4114312791 conn(0x7f6b20100190 msgr2=0x7f6b20100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:45.934 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.398+0000 7f6b27606700 1 -- 172.21.15.151:0/4114312791 shutdown_connections 2024-04-16T02:25:45.934 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:45.398+0000 7f6b27606700 1 -- 172.21.15.151:0/4114312791 wait complete. 2024-04-16T02:25:45.934 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (2/15)... 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: Activating manager daemon smithi151.uqqame 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: mgrmap e2: smithi151.uqqame(active, starting, since 0.00338992s) 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: Manager daemon smithi151.uqqame is now available 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/trash_purge_schedule"}]: dispatch 2024-04-16T02:25:48.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' 2024-04-16T02:25:48.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:48 smithi151 ceph-mon[29003]: from='mgr.14100 172.21.15.151:0/4107101790' entity='mgr.smithi151.uqqame' 2024-04-16T02:25:49.342 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:49.345 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-04-16T02:25:49.345 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "80d1e958-fb98-11ee-bc8f-c7b262605968", 2024-04-16T02:25:49.345 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T02:25:49.345 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T02:25:49.345 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T02:25:49.345 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T02:25:49.345 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:49.345 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "smithi151" 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T02:25:49.346 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:49.347 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T02:25:49.347 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T02:25:49.347 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T02:25:49.354 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T02:25:49.355 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:49.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T02:25:34.606666+0000", 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.391+0000 7fc6a14da700 1 Processor -- start 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.391+0000 7fc6a14da700 1 -- start start 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.391+0000 7fc6a14da700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c07a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.391+0000 7fc6a14da700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c07a8d0 con 0x7fc69c07bea0 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.392+0000 7fc69affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c07a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:49.357 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.392+0000 7fc69affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c07a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42124/0 (socket says 172.21.15.151:42124) 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.392+0000 7fc69affd700 1 -- 172.21.15.151:0/3455017113 learned_addr learned my addr 172.21.15.151:0/3455017113 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.392+0000 7fc69affd700 1 -- 172.21.15.151:0/3455017113 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c07aa10 con 0x7fc69c07bea0 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.392+0000 7fc69affd700 1 --2- 172.21.15.151:0/3455017113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c07a300 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0098a0 tx=0x7fc68c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e949b62527875720 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.392+0000 7fc699ffb700 1 -- 172.21.15.151:0/3455017113 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c004180 con 0x7fc69c07bea0 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.392+0000 7fc699ffb700 1 -- 172.21.15.151:0/3455017113 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc68c0042e0 con 0x7fc69c07bea0 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.392+0000 7fc699ffb700 1 -- 172.21.15.151:0/3455017113 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c011a50 con 0x7fc69c07bea0 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.393+0000 7fc6a14da700 1 -- 172.21.15.151:0/3455017113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 msgr2=0x7fc69c07a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.393+0000 7fc6a14da700 1 --2- 172.21.15.151:0/3455017113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c07a300 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0098a0 tx=0x7fc68c009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.393+0000 7fc6a14da700 1 -- 172.21.15.151:0/3455017113 shutdown_connections 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.393+0000 7fc6a14da700 1 --2- 172.21.15.151:0/3455017113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c07a300 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.393+0000 7fc6a14da700 1 -- 172.21.15.151:0/3455017113 >> 172.21.15.151:0/3455017113 conn(0x7fc69c101b40 msgr2=0x7fc69c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.393+0000 7fc6a14da700 1 -- 172.21.15.151:0/3455017113 shutdown_connections 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.393+0000 7fc6a14da700 1 -- 172.21.15.151:0/3455017113 wait complete. 2024-04-16T02:25:49.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.393+0000 7fc6a14da700 1 Processor -- start 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc6a14da700 1 -- start start 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc6a14da700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c19c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc6a14da700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c19c8d0 con 0x7fc69c07bea0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc69affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c19c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc69affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c19c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42126/0 (socket says 172.21.15.151:42126) 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc69affd700 1 -- 172.21.15.151:0/2795304963 learned_addr learned my addr 172.21.15.151:0/2795304963 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc69affd700 1 -- 172.21.15.151:0/2795304963 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c009550 con 0x7fc69c07bea0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc69affd700 1 --2- 172.21.15.151:0/2795304963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c19c390 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc68c000c00 tx=0x7fc68c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc683fff700 1 -- 172.21.15.151:0/2795304963 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c003b90 con 0x7fc69c07bea0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc683fff700 1 -- 172.21.15.151:0/2795304963 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc68c0248b0 con 0x7fc69c07bea0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc6a14da700 1 -- 172.21.15.151:0/2795304963 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69c19cad0 con 0x7fc69c07bea0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.394+0000 7fc683fff700 1 -- 172.21.15.151:0/2795304963 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c01a8e0 con 0x7fc69c07bea0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.395+0000 7fc6a14da700 1 -- 172.21.15.151:0/2795304963 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c19cf70 con 0x7fc69c07bea0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.395+0000 7fc683fff700 1 -- 172.21.15.151:0/2795304963 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 2) v1 ==== 45039+0+0 (secure 0 0 0) 0x7fc68c01aba0 con 0x7fc69c07bea0 2024-04-16T02:25:49.359 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.396+0000 7fc6a14da700 1 -- 172.21.15.151:0/2795304963 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c10c8a0 con 0x7fc69c07bea0 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.396+0000 7fc683fff700 1 -- 172.21.15.151:0/2795304963 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc68c04be70 con 0x7fc69c07bea0 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.398+0000 7fc683fff700 1 -- 172.21.15.151:0/2795304963 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fc68c023790 con 0x7fc69c07bea0 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.466+0000 7fc6a14da700 1 -- 172.21.15.151:0/2795304963 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc69c19fdb0 con 0x7fc69c07bea0 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.467+0000 7fc683fff700 1 -- 172.21.15.151:0/2795304963 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fc68c0233a0 con 0x7fc69c07bea0 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.467+0000 7fc6a14da700 1 -- 172.21.15.151:0/2795304963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 msgr2=0x7fc69c19c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.467+0000 7fc6a14da700 1 --2- 172.21.15.151:0/2795304963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c19c390 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc68c000c00 tx=0x7fc68c011ab0 comp rx=0 tx=0).stop 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.468+0000 7fc6a14da700 1 -- 172.21.15.151:0/2795304963 shutdown_connections 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.468+0000 7fc6a14da700 1 --2- 172.21.15.151:0/2795304963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69c07bea0 0x7fc69c19c390 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.468+0000 7fc6a14da700 1 -- 172.21.15.151:0/2795304963 >> 172.21.15.151:0/2795304963 conn(0x7fc69c101b40 msgr2=0x7fc69c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.468+0000 7fc6a14da700 1 -- 172.21.15.151:0/2795304963 shutdown_connections 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:48.468+0000 7fc6a14da700 1 -- 172.21.15.151:0/2795304963 wait complete. 2024-04-16T02:25:49.360 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (3/15)... 2024-04-16T02:25:49.743 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:49 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2795304963' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T02:25:49.743 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:49 smithi151 ceph-mon[29003]: mgrmap e3: smithi151.uqqame(active, since 1.00736s) 2024-04-16T02:25:51.739 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:51 smithi151 ceph-mon[29003]: mgrmap e4: smithi151.uqqame(active, since 2s) 2024-04-16T02:25:52.564 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:52.564 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "80d1e958-fb98-11ee-bc8f-c7b262605968", 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "smithi151" 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:52.565 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-16T02:25:52.566 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T02:25:52.566 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:52.566 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-16T02:25:52.566 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T02:25:52.566 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:52.566 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T02:25:52.566 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:52.567 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T02:25:52.567 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T02:25:52.567 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T02:25:52.576 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T02:25:52.576 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T02:25:52.576 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T02:25:52.576 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T02:25:52.577 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:52.578 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T02:25:34.606666+0000", 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.756+0000 7f9ff144e700 1 Processor -- start 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.756+0000 7f9ff144e700 1 -- start start 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.756+0000 7f9ff144e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.756+0000 7f9ff144e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec079e50 con 0x7f9fec107020 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.757+0000 7f9feaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.757+0000 7f9feaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42132/0 (socket says 172.21.15.151:42132) 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.757+0000 7f9feaffd700 1 -- 172.21.15.151:0/2483131781 learned_addr learned my addr 172.21.15.151:0/2483131781 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.757+0000 7f9feaffd700 1 -- 172.21.15.151:0/2483131781 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec109950 con 0x7f9fec107020 2024-04-16T02:25:52.579 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.757+0000 7f9feaffd700 1 --2- 172.21.15.151:0/2483131781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec109410 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc009e80 tx=0x7f9fdc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eecf3bb3138ceab server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.757+0000 7f9fe9ffb700 1 -- 172.21.15.151:0/2483131781 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fdc00e070 con 0x7f9fec107020 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.757+0000 7f9fe9ffb700 1 -- 172.21.15.151:0/2483131781 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9fdc003e90 con 0x7f9fec107020 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.757+0000 7f9fe9ffb700 1 -- 172.21.15.151:0/2483131781 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fdc004380 con 0x7f9fec107020 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.758+0000 7f9ff144e700 1 -- 172.21.15.151:0/2483131781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 msgr2=0x7f9fec109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.758+0000 7f9ff144e700 1 --2- 172.21.15.151:0/2483131781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec109410 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc009e80 tx=0x7f9fdc00b330 comp rx=0 tx=0).stop 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.758+0000 7f9ff144e700 1 -- 172.21.15.151:0/2483131781 shutdown_connections 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.758+0000 7f9ff144e700 1 --2- 172.21.15.151:0/2483131781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec109410 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.758+0000 7f9ff144e700 1 -- 172.21.15.151:0/2483131781 >> 172.21.15.151:0/2483131781 conn(0x7f9fec1009f0 msgr2=0x7f9fec102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.758+0000 7f9ff144e700 1 -- 172.21.15.151:0/2483131781 shutdown_connections 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.758+0000 7f9ff144e700 1 -- 172.21.15.151:0/2483131781 wait complete. 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.759+0000 7f9ff144e700 1 Processor -- start 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.759+0000 7f9ff144e700 1 -- start start 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.759+0000 7f9ff144e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.759+0000 7f9ff144e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec19a650 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.759+0000 7f9feaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.759+0000 7f9feaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42144/0 (socket says 172.21.15.151:42144) 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.759+0000 7f9feaffd700 1 -- 172.21.15.151:0/2461004308 learned_addr learned my addr 172.21.15.151:0/2461004308 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.759+0000 7f9feaffd700 1 -- 172.21.15.151:0/2461004308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc009b30 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.760+0000 7f9feaffd700 1 --2- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec19a110 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc000f80 tx=0x7f9fdc004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.760+0000 7f9fe3fff700 1 -- 172.21.15.151:0/2461004308 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fdc00e040 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.760+0000 7f9fe3fff700 1 -- 172.21.15.151:0/2461004308 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9fdc003cf0 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.760+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec19a850 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.760+0000 7f9fe3fff700 1 -- 172.21.15.151:0/2461004308 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fdc01b930 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.760+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec19ad50 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.761+0000 7f9fe3fff700 1 -- 172.21.15.151:0/2461004308 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f9fdc019040 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.761+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec10a6a0 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.761+0000 7f9fe3fff700 1 --2- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f9fd80382d0 0x7f9fd803a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.761+0000 7f9fe3fff700 1 -- 172.21.15.151:0/2461004308 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9fdc04bc00 con 0x7f9fec107020 2024-04-16T02:25:52.581 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.761+0000 7f9fea7fc700 1 --2- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f9fd80382d0 0x7f9fd803a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.762+0000 7f9fea7fc700 1 --2- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f9fd80382d0 0x7f9fd803a790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4006fd0 tx=0x7f9fd4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:51.766+0000 7f9fe3fff700 1 -- 172.21.15.151:0/2461004308 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9fdc011980 con 0x7f9fec107020 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.042+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9fec10a890 con 0x7f9fec107020 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.043+0000 7f9fe3fff700 1 -- 172.21.15.151:0/2461004308 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f9fdc01f090 con 0x7f9fec107020 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f9fd80382d0 msgr2=0x7f9fd803a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 --2- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f9fd80382d0 0x7f9fd803a790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4006fd0 tx=0x7f9fd4006e40 comp rx=0 tx=0).stop 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 msgr2=0x7f9fec19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 --2- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec19a110 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc000f80 tx=0x7f9fdc004920 comp rx=0 tx=0).stop 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 shutdown_connections 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 --2- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f9fd80382d0 0x7f9fd803a790 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 --2- 172.21.15.151:0/2461004308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fec107020 0x7f9fec19a110 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 >> 172.21.15.151:0/2461004308 conn(0x7f9fec1009f0 msgr2=0x7f9fec101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 shutdown_connections 2024-04-16T02:25:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.045+0000 7f9ff144e700 1 -- 172.21.15.151:0/2461004308 wait complete. 2024-04-16T02:25:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:mgr is available 2024-04-16T02:25:52.944 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:52 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2461004308' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-04-16T02:25:53.717 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.943+0000 7f40d9359700 1 Processor -- start 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.943+0000 7f40d9359700 1 -- start start 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.943+0000 7f40d9359700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d4105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.943+0000 7f40d9359700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d4105e60 con 0x7f40d4105470 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.944+0000 7f40d2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d4105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.944+0000 7f40d2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d4105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42148/0 (socket says 172.21.15.151:42148) 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.944+0000 7f40d2ffd700 1 -- 172.21.15.151:0/4294585367 learned_addr learned my addr 172.21.15.151:0/4294585367 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.944+0000 7f40d2ffd700 1 -- 172.21.15.151:0/4294585367 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4106670 con 0x7f40d4105470 2024-04-16T02:25:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.944+0000 7f40d2ffd700 1 --2- 172.21.15.151:0/4294585367 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d4105890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40c40098a0 tx=0x7f40c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117ebf296a75d218 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d27fc700 1 -- 172.21.15.151:0/4294585367 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c4004180 con 0x7f40d4105470 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d27fc700 1 -- 172.21.15.151:0/4294585367 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f40c40042e0 con 0x7f40d4105470 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d27fc700 1 -- 172.21.15.151:0/4294585367 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c4011a50 con 0x7f40d4105470 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d9359700 1 -- 172.21.15.151:0/4294585367 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 msgr2=0x7f40d4105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d9359700 1 --2- 172.21.15.151:0/4294585367 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d4105890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40c40098a0 tx=0x7f40c4009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d9359700 1 -- 172.21.15.151:0/4294585367 shutdown_connections 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d9359700 1 --2- 172.21.15.151:0/4294585367 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d4105890 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d9359700 1 -- 172.21.15.151:0/4294585367 >> 172.21.15.151:0/4294585367 conn(0x7f40d41009f0 msgr2=0x7f40d4102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d9359700 1 -- 172.21.15.151:0/4294585367 shutdown_connections 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.945+0000 7f40d9359700 1 -- 172.21.15.151:0/4294585367 wait complete. 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.946+0000 7f40d9359700 1 Processor -- start 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.946+0000 7f40d9359700 1 -- start start 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.946+0000 7f40d9359700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.946+0000 7f40d9359700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d419a6e0 con 0x7f40d4105470 2024-04-16T02:25:53.719 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.946+0000 7f40d2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.946+0000 7f40d2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42156/0 (socket says 172.21.15.151:42156) 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.946+0000 7f40d2ffd700 1 -- 172.21.15.151:0/1415823196 learned_addr learned my addr 172.21.15.151:0/1415823196 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.947+0000 7f40d2ffd700 1 -- 172.21.15.151:0/1415823196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c4009550 con 0x7f40d4105470 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.947+0000 7f40d2ffd700 1 --2- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d419a1a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f40c4006af0 tx=0x7f40c4011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.947+0000 7f40d0ff9700 1 -- 172.21.15.151:0/1415823196 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c4003b90 con 0x7f40d4105470 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.947+0000 7f40d0ff9700 1 -- 172.21.15.151:0/1415823196 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f40c401a8a0 con 0x7f40d4105470 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.947+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d419a8e0 con 0x7f40d4105470 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.947+0000 7f40d0ff9700 1 -- 172.21.15.151:0/1415823196 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c4019880 con 0x7f40d4105470 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.947+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d419ad80 con 0x7f40d4105470 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.948+0000 7f40d0ff9700 1 -- 172.21.15.151:0/1415823196 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f40c4019b40 con 0x7f40d4105470 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.948+0000 7f40d0ff9700 1 --2- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f40c0038290 0x7f40c003a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.948+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d40623c0 con 0x7f40d4105470 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.948+0000 7f40d0ff9700 1 -- 172.21.15.151:0/1415823196 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f40c404c960 con 0x7f40d4105470 2024-04-16T02:25:53.720 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.948+0000 7f40cbfff700 1 --2- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f40c0038290 0x7f40c003a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.949+0000 7f40cbfff700 1 --2- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f40c0038290 0x7f40c003a750 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f40bc006fd0 tx=0x7f40bc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:52.953+0000 7f40d0ff9700 1 -- 172.21.15.151:0/1415823196 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f40c402c3a0 con 0x7f40d4105470 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.138+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f40d419dc70 con 0x7f40d4105470 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.138+0000 7f40d0ff9700 1 -- 172.21.15.151:0/1415823196 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f40c40172b0 con 0x7f40d4105470 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f40c0038290 msgr2=0x7f40c003a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 --2- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f40c0038290 0x7f40c003a750 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f40bc006fd0 tx=0x7f40bc006e40 comp rx=0 tx=0).stop 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 msgr2=0x7f40d419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 --2- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d419a1a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f40c4006af0 tx=0x7f40c4011a90 comp rx=0 tx=0).stop 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 shutdown_connections 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 --2- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f40c0038290 0x7f40c003a750 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 --2- 172.21.15.151:0/1415823196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d4105470 0x7f40d419a1a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 >> 172.21.15.151:0/1415823196 conn(0x7f40d41009f0 msgr2=0x7f40d4079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 shutdown_connections 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:53.141+0000 7f40d9359700 1 -- 172.21.15.151:0/1415823196 wait complete. 2024-04-16T02:25:53.721 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling cephadm module... 2024-04-16T02:25:53.945 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:53 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1415823196' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-16T02:25:54.913 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:54 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/343524904' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.142+0000 7f5eb74b6700 1 Processor -- start 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.142+0000 7f5eb74b6700 1 -- start start 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5eb74b6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb01093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5eb74b6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb0074680 con 0x7f5eb0106fe0 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5eb5252700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb01093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5eb5252700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb01093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42166/0 (socket says 172.21.15.151:42166) 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5eb5252700 1 -- 172.21.15.151:0/3891775836 learned_addr learned my addr 172.21.15.151:0/3891775836 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5eb5252700 1 -- 172.21.15.151:0/3891775836 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb0109910 con 0x7f5eb0106fe0 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5eb5252700 1 --2- 172.21.15.151:0/3891775836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb01093d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098a0 tx=0x7f5eac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fad35be4a9266b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:55.164 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5ea7fff700 1 -- 172.21.15.151:0/3891775836 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eac004180 con 0x7f5eb0106fe0 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5ea7fff700 1 -- 172.21.15.151:0/3891775836 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5eac0042e0 con 0x7f5eb0106fe0 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.143+0000 7f5ea7fff700 1 -- 172.21.15.151:0/3891775836 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eac011a50 con 0x7f5eb0106fe0 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.144+0000 7f5eb74b6700 1 -- 172.21.15.151:0/3891775836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 msgr2=0x7f5eb01093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.144+0000 7f5eb74b6700 1 --2- 172.21.15.151:0/3891775836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb01093d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098a0 tx=0x7f5eac009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.144+0000 7f5eb74b6700 1 -- 172.21.15.151:0/3891775836 shutdown_connections 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.144+0000 7f5eb74b6700 1 --2- 172.21.15.151:0/3891775836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb01093d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.144+0000 7f5eb74b6700 1 -- 172.21.15.151:0/3891775836 >> 172.21.15.151:0/3891775836 conn(0x7f5eb01009f0 msgr2=0x7f5eb0102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.144+0000 7f5eb74b6700 1 -- 172.21.15.151:0/3891775836 shutdown_connections 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.144+0000 7f5eb74b6700 1 -- 172.21.15.151:0/3891775836 wait complete. 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5eb74b6700 1 Processor -- start 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5eb74b6700 1 -- start start 2024-04-16T02:25:55.165 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5eb74b6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5eb74b6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb019a670 con 0x7f5eb0106fe0 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5eb5252700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5eb5252700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42178/0 (socket says 172.21.15.151:42178) 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5eb5252700 1 -- 172.21.15.151:0/343524904 learned_addr learned my addr 172.21.15.151:0/343524904 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5eb5252700 1 -- 172.21.15.151:0/343524904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac009550 con 0x7f5eb0106fe0 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5eb5252700 1 --2- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb019a130 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5eac000c00 tx=0x7f5eac011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5ea67fc700 1 -- 172.21.15.151:0/343524904 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eac003b90 con 0x7f5eb0106fe0 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.145+0000 7f5ea67fc700 1 -- 172.21.15.151:0/343524904 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5eac0248b0 con 0x7f5eb0106fe0 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.146+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb019a870 con 0x7f5eb0106fe0 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.146+0000 7f5ea67fc700 1 -- 172.21.15.151:0/343524904 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eac01a8e0 con 0x7f5eb0106fe0 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.146+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb019ad10 con 0x7f5eb0106fe0 2024-04-16T02:25:55.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.146+0000 7f5ea67fc700 1 -- 172.21.15.151:0/343524904 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f5eac01aba0 con 0x7f5eb0106fe0 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.146+0000 7f5ea67fc700 1 --2- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f5e9c0382d0 0x7f5e9c03a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.147+0000 7f5ea67fc700 1 -- 172.21.15.151:0/343524904 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5eac04c160 con 0x7f5eb0106fe0 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.147+0000 7f5eb4a51700 1 --2- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f5e9c0382d0 0x7f5e9c03a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.147+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb00623c0 con 0x7f5eb0106fe0 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.147+0000 7f5eb4a51700 1 --2- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f5e9c0382d0 0x7f5e9c03a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0006fd0 tx=0x7f5ea0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.152+0000 7f5ea67fc700 1 -- 172.21.15.151:0/343524904 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5eac0233a0 con 0x7f5eb0106fe0 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.392+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f5eb019d6d0 con 0x7f5eb0106fe0 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.660+0000 7f5ea67fc700 1 -- 172.21.15.151:0/343524904 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f5eac018350 con 0x7f5eb0106fe0 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.661+0000 7f5ea67fc700 1 -- 172.21.15.151:0/343524904 <== mon.0 v2:172.21.15.151:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f5eac02a2a0 con 0x7f5eb0106fe0 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f5e9c0382d0 msgr2=0x7f5e9c03a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 --2- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f5e9c0382d0 0x7f5e9c03a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0006fd0 tx=0x7f5ea0006e40 comp rx=0 tx=0).stop 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 msgr2=0x7f5eb019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:55.167 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 --2- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb019a130 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5eac000c00 tx=0x7f5eac011ab0 comp rx=0 tx=0).stop 2024-04-16T02:25:55.168 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 shutdown_connections 2024-04-16T02:25:55.168 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 --2- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f5e9c0382d0 0x7f5e9c03a790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:55.168 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 --2- 172.21.15.151:0/343524904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5eb0106fe0 0x7f5eb019a130 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:55.168 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 >> 172.21.15.151:0/343524904 conn(0x7f5eb01009f0 msgr2=0x7f5eb0101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:55.168 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 shutdown_connections 2024-04-16T02:25:55.168 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:54.664+0000 7f5eb74b6700 1 -- 172.21.15.151:0/343524904 wait complete. 2024-04-16T02:25:55.839 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:55 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/343524904' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-16T02:25:55.839 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:55 smithi151 ceph-mon[29003]: mgrmap e5: smithi151.uqqame(active, since 6s) 2024-04-16T02:25:56.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-04-16T02:25:56.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-16T02:25:56.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T02:25:56.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "smithi151.uqqame", 2024-04-16T02:25:56.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-16T02:25:56.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.633+0000 7f1e357bd700 1 Processor -- start 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.633+0000 7f1e357bd700 1 -- start start 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.633+0000 7f1e357bd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e301093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.633+0000 7f1e357bd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30074680 con 0x7f1e30106fe0 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.633+0000 7f1e2effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e301093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.633+0000 7f1e2effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e301093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42200/0 (socket says 172.21.15.151:42200) 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.633+0000 7f1e2effd700 1 -- 172.21.15.151:0/825735211 learned_addr learned my addr 172.21.15.151:0/825735211 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.634+0000 7f1e2effd700 1 -- 172.21.15.151:0/825735211 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30109910 con 0x7f1e30106fe0 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.634+0000 7f1e2effd700 1 --2- 172.21.15.151:0/825735211 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e301093d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1e240098a0 tx=0x7f1e24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f33ac32d60f3795 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.634+0000 7f1e2dffb700 1 -- 172.21.15.151:0/825735211 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e24004180 con 0x7f1e30106fe0 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.634+0000 7f1e2dffb700 1 -- 172.21.15.151:0/825735211 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1e240042e0 con 0x7f1e30106fe0 2024-04-16T02:25:56.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.634+0000 7f1e2dffb700 1 -- 172.21.15.151:0/825735211 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e24011a50 con 0x7f1e30106fe0 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.635+0000 7f1e357bd700 1 -- 172.21.15.151:0/825735211 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 msgr2=0x7f1e301093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.635+0000 7f1e357bd700 1 --2- 172.21.15.151:0/825735211 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e301093d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1e240098a0 tx=0x7f1e24009bb0 comp rx=0 tx=0).stop 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.635+0000 7f1e357bd700 1 -- 172.21.15.151:0/825735211 shutdown_connections 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.635+0000 7f1e357bd700 1 --2- 172.21.15.151:0/825735211 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e301093d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.635+0000 7f1e357bd700 1 -- 172.21.15.151:0/825735211 >> 172.21.15.151:0/825735211 conn(0x7f1e301009f0 msgr2=0x7f1e30102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.635+0000 7f1e357bd700 1 -- 172.21.15.151:0/825735211 shutdown_connections 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.635+0000 7f1e357bd700 1 -- 172.21.15.151:0/825735211 wait complete. 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.635+0000 7f1e357bd700 1 Processor -- start 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.635+0000 7f1e357bd700 1 -- start start 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e357bd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e3019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e357bd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3019a670 con 0x7f1e30106fe0 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e2effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e3019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:25:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e2effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e3019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42202/0 (socket says 172.21.15.151:42202) 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e2effd700 1 -- 172.21.15.151:0/2957970047 learned_addr learned my addr 172.21.15.151:0/2957970047 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e2effd700 1 -- 172.21.15.151:0/2957970047 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e24009550 con 0x7f1e30106fe0 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e2effd700 1 --2- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e3019a130 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1e24000c00 tx=0x7f1e24011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e17fff700 1 -- 172.21.15.151:0/2957970047 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e24003b90 con 0x7f1e30106fe0 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e17fff700 1 -- 172.21.15.151:0/2957970047 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1e240248b0 con 0x7f1e30106fe0 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.636+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3019a870 con 0x7f1e30106fe0 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.637+0000 7f1e17fff700 1 -- 172.21.15.151:0/2957970047 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e2401a8e0 con 0x7f1e30106fe0 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.637+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3019ad10 con 0x7f1e30106fe0 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.637+0000 7f1e17fff700 1 -- 172.21.15.151:0/2957970047 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f1e2401aba0 con 0x7f1e30106fe0 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.638+0000 7f1e17fff700 1 --2- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f1e18038320 0x7f1e1803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.638+0000 7f1e17fff700 1 -- 172.21.15.151:0/2957970047 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1e2404c0c0 con 0x7f1e30106fe0 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.638+0000 7f1e2e7fc700 1 -- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f1e18038320 msgr2=0x7f1e1803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/39987226 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.638+0000 7f1e2e7fc700 1 --2- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f1e18038320 0x7f1e1803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:25:56.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.638+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e300623c0 con 0x7f1e30106fe0 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.643+0000 7f1e17fff700 1 -- 172.21.15.151:0/2957970047 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1e240237b0 con 0x7f1e30106fe0 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.839+0000 7f1e2e7fc700 1 -- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f1e18038320 msgr2=0x7f1e1803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/39987226 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.839+0000 7f1e2e7fc700 1 --2- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f1e18038320 0x7f1e1803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.917+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1e3019d820 con 0x7f1e30106fe0 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.918+0000 7f1e17fff700 1 -- 172.21.15.151:0/2957970047 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f1e240172b0 con 0x7f1e30106fe0 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f1e18038320 msgr2=0x7f1e1803a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 --2- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f1e18038320 0x7f1e1803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 msgr2=0x7f1e3019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 --2- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e3019a130 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1e24000c00 tx=0x7f1e24011ab0 comp rx=0 tx=0).stop 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 shutdown_connections 2024-04-16T02:25:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 --2- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7f1e18038320 0x7f1e1803a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:56.426 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 --2- 172.21.15.151:0/2957970047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e30106fe0 0x7f1e3019a130 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:25:56.426 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 >> 172.21.15.151:0/2957970047 conn(0x7f1e301009f0 msgr2=0x7f1e30101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:25:56.426 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 shutdown_connections 2024-04-16T02:25:56.427 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:55.920+0000 7f1e357bd700 1 -- 172.21.15.151:0/2957970047 wait complete. 2024-04-16T02:25:56.427 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-04-16T02:25:56.427 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 5... 2024-04-16T02:25:56.674 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:25:56 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2957970047' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-16T02:26:01.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: Active manager daemon smithi151.uqqame restarted 2024-04-16T02:26:01.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: Activating manager daemon smithi151.uqqame 2024-04-16T02:26:01.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: osdmap e2: 0 total, 0 up, 0 in 2024-04-16T02:26:01.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: mgrmap e6: smithi151.uqqame(active, starting, since 0.00320624s) 2024-04-16T02:26:01.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:26:01.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:26:01.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:26:01.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:26:01.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:26:01.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: Manager daemon smithi151.uqqame is now available 2024-04-16T02:26:01.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:01.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:01.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:26:01.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:26:01.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:26:01.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:26:01.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:01 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/trash_purge_schedule"}]: dispatch 2024-04-16T02:26:02.725 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:02 smithi151 ceph-mon[29003]: Found migration_current of "None". Setting to last migration. 2024-04-16T02:26:02.725 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:02 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:02.725 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:02 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:02.725 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:02 smithi151 ceph-mon[29003]: mgrmap e7: smithi151.uqqame(active, since 1.00639s) 2024-04-16T02:26:02.977 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-04-16T02:26:02.977 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-16T02:26:02.977 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-16T02:26:02.977 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.814+0000 7fb2006ff700 1 Processor -- start 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.814+0000 7fb2006ff700 1 -- start start 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.814+0000 7fb2006ff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.814+0000 7fb2006ff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8105e60 con 0x7fb1f8105470 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.815+0000 7fb1fe49b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.815+0000 7fb1fe49b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40998/0 (socket says 172.21.15.151:40998) 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.815+0000 7fb1fe49b700 1 -- 172.21.15.151:0/1250846627 learned_addr learned my addr 172.21.15.151:0/1250846627 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.815+0000 7fb1fe49b700 1 -- 172.21.15.151:0/1250846627 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8106670 con 0x7fb1f8105470 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.815+0000 7fb1fe49b700 1 --2- 172.21.15.151:0/1250846627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8105890 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80098a0 tx=0x7fb1e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97aea310b5ef3d0c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.815+0000 7fb1fd499700 1 -- 172.21.15.151:0/1250846627 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e8004180 con 0x7fb1f8105470 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.815+0000 7fb1fd499700 1 -- 172.21.15.151:0/1250846627 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb1e80042e0 con 0x7fb1f8105470 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.815+0000 7fb1fd499700 1 -- 172.21.15.151:0/1250846627 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e8011a50 con 0x7fb1f8105470 2024-04-16T02:26:02.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.816+0000 7fb2006ff700 1 -- 172.21.15.151:0/1250846627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 msgr2=0x7fb1f8105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.816+0000 7fb2006ff700 1 --2- 172.21.15.151:0/1250846627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8105890 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80098a0 tx=0x7fb1e8009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.816+0000 7fb2006ff700 1 -- 172.21.15.151:0/1250846627 shutdown_connections 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.816+0000 7fb2006ff700 1 --2- 172.21.15.151:0/1250846627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8105890 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.816+0000 7fb2006ff700 1 -- 172.21.15.151:0/1250846627 >> 172.21.15.151:0/1250846627 conn(0x7fb1f81009f0 msgr2=0x7fb1f8102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.816+0000 7fb2006ff700 1 -- 172.21.15.151:0/1250846627 shutdown_connections 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.816+0000 7fb2006ff700 1 -- 172.21.15.151:0/1250846627 wait complete. 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.817+0000 7fb2006ff700 1 Processor -- start 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.817+0000 7fb2006ff700 1 -- start start 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.817+0000 7fb2006ff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8190640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.817+0000 7fb2006ff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8190b80 con 0x7fb1f8105470 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.817+0000 7fb1fe49b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8190640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.817+0000 7fb1fe49b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8190640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41002/0 (socket says 172.21.15.151:41002) 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.817+0000 7fb1fe49b700 1 -- 172.21.15.151:0/2361576829 learned_addr learned my addr 172.21.15.151:0/2361576829 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.817+0000 7fb1fe49b700 1 -- 172.21.15.151:0/2361576829 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8009550 con 0x7fb1f8105470 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.817+0000 7fb1fe49b700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8190640 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8006af0 tx=0x7fb1e8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.818+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e8003b90 con 0x7fb1f8105470 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.818+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb1e801a8a0 con 0x7fb1f8105470 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.818+0000 7fb2006ff700 1 -- 172.21.15.151:0/2361576829 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f818ec90 con 0x7fb1f8105470 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.818+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e8019880 con 0x7fb1f8105470 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.818+0000 7fb2006ff700 1 -- 172.21.15.151:0/2361576829 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f818f130 con 0x7fb1f8105470 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.818+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fb1e8019af0 con 0x7fb1f8105470 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.819+0000 7fb1ef7fe700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 0x7fb1e403a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.819+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 --> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1e403af00 con 0x7fb1e4038330 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.819+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb1e804d510 con 0x7fb1f8105470 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.819+0000 7fb1fdc9a700 1 -- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 msgr2=0x7fb1e403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/39987226 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:56.819+0000 7fb1fdc9a700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 0x7fb1e403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:57.020+0000 7fb1fdc9a700 1 -- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 msgr2=0x7fb1e403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/39987226 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:57.020+0000 7fb1fdc9a700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 0x7fb1e403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T02:26:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:57.421+0000 7fb1fdc9a700 1 -- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 msgr2=0x7fb1e403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/39987226 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:57.421+0000 7fb1fdc9a700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 0x7fb1e403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:58.222+0000 7fb1fdc9a700 1 -- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 msgr2=0x7fb1e403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/39987226 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:58.222+0000 7fb1fdc9a700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 0x7fb1e403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:59.823+0000 7fb1fdc9a700 1 -- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 msgr2=0x7fb1e403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/39987226 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:25:59.823+0000 7fb1fdc9a700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 0x7fb1e403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:01.418+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mgrmap(e 6) v1 ==== 45050+0+0 (secure 0 0 0) 0x7fb1e8016d60 con 0x7fb1f8105470 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:01.419+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 msgr2=0x7fb1e403a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:01.419+0000 7fb1ef7fe700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 0x7fb1e403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.421+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7fb1e802acf0 con 0x7fb1f8105470 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.421+0000 7fb1ef7fe700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fb1e403bc60 0x7fb1e403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.421+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1e403af00 con 0x7fb1e403bc60 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.421+0000 7fb1fdc9a700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fb1e403bc60 0x7fb1e403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.422+0000 7fb1fdc9a700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fb1e403bc60 0x7fb1e403e050 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4003a10 tx=0x7fb1f40092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.422+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fb1e403af00 con 0x7fb1e403bc60 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 -- 172.21.15.151:0/2361576829 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb1f81901e0 con 0x7fb1e403bc60 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb1ef7fe700 1 -- 172.21.15.151:0/2361576829 <== mgr.14120 v2:172.21.15.151:6800/2400549256 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb1f81901e0 con 0x7fb1e403bc60 2024-04-16T02:26:02.981 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 -- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fb1e403bc60 msgr2=0x7fb1e403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fb1e403bc60 0x7fb1e403e050 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4003a10 tx=0x7fb1f40092b0 comp rx=0 tx=0).stop 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 -- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 msgr2=0x7fb1f8190640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8190640 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8006af0 tx=0x7fb1e8011a90 comp rx=0 tx=0).stop 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 -- 172.21.15.151:0/2361576829 shutdown_connections 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fb1e403bc60 0x7fb1e403e050 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f8105470 0x7fb1f8190640 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 --2- 172.21.15.151:0/2361576829 >> [v2:172.21.15.151:6800/39987226,v1:172.21.15.151:6801/39987226] conn(0x7fb1e4038330 0x7fb1e403a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 -- 172.21.15.151:0/2361576829 >> 172.21.15.151:0/2361576829 conn(0x7fb1f81009f0 msgr2=0x7fb1f8074130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 -- 172.21.15.151:0/2361576829 shutdown_connections 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:02.429+0000 7fb2006ff700 1 -- 172.21.15.151:0/2361576829 wait complete. 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 5 is available 2024-04-16T02:26:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:Setting orchestrator backend to cephadm... 2024-04-16T02:26:03.627 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:03 smithi151 ceph-mon[29003]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-16T02:26:03.628 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:03 smithi151 ceph-mon[29003]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-16T02:26:03.628 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:03 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:26:04.222 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.412+0000 7f4a3b27e700 1 Processor -- start 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.412+0000 7f4a3b27e700 1 -- start start 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.412+0000 7f4a3b27e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3407a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.412+0000 7f4a3b27e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3407a8f0 con 0x7f4a3407bec0 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.413+0000 7f4a3901a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3407a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.413+0000 7f4a3901a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3407a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41056/0 (socket says 172.21.15.151:41056) 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.413+0000 7f4a3901a700 1 -- 172.21.15.151:0/3444700718 learned_addr learned my addr 172.21.15.151:0/3444700718 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.413+0000 7f4a3901a700 1 -- 172.21.15.151:0/3444700718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3407aa30 con 0x7f4a3407bec0 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.413+0000 7f4a3901a700 1 --2- 172.21.15.151:0/3444700718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3407a320 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a300098a0 tx=0x7f4a30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25f80f3ae9b079 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.413+0000 7f4a2bfff700 1 -- 172.21.15.151:0/3444700718 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a30004180 con 0x7f4a3407bec0 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.413+0000 7f4a2bfff700 1 -- 172.21.15.151:0/3444700718 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4a300042e0 con 0x7f4a3407bec0 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.413+0000 7f4a2bfff700 1 -- 172.21.15.151:0/3444700718 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a30011a50 con 0x7f4a3407bec0 2024-04-16T02:26:04.223 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.414+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3444700718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 msgr2=0x7f4a3407a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.414+0000 7f4a3b27e700 1 --2- 172.21.15.151:0/3444700718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3407a320 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a300098a0 tx=0x7f4a30009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.414+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3444700718 shutdown_connections 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.414+0000 7f4a3b27e700 1 --2- 172.21.15.151:0/3444700718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3407a320 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.414+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3444700718 >> 172.21.15.151:0/3444700718 conn(0x7f4a34101b40 msgr2=0x7f4a34103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.414+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3444700718 shutdown_connections 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.414+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3444700718 wait complete. 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.414+0000 7f4a3b27e700 1 Processor -- start 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a3b27e700 1 -- start start 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a3b27e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3419c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a3b27e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3419c9b0 con 0x7f4a3407bec0 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a3901a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3419c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a3901a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3419c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41062/0 (socket says 172.21.15.151:41062) 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a3901a700 1 -- 172.21.15.151:0/3962128209 learned_addr learned my addr 172.21.15.151:0/3962128209 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a3901a700 1 -- 172.21.15.151:0/3962128209 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30009550 con 0x7f4a3407bec0 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a3901a700 1 --2- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3419c470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009520 tx=0x7f4a30011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:04.224 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a2a7fc700 1 -- 172.21.15.151:0/3962128209 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a30003a80 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a2a7fc700 1 -- 172.21.15.151:0/3962128209 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4a30003be0 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3419cbb0 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.415+0000 7f4a2a7fc700 1 -- 172.21.15.151:0/3962128209 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a300197b0 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.416+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3419d050 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.416+0000 7f4a2a7fc700 1 -- 172.21.15.151:0/3962128209 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7f4a30019aa0 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.417+0000 7f4a2a7fc700 1 --2- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4a200381f0 0x7f4a2003a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.417+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3410c860 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.417+0000 7f4a2a7fc700 1 -- 172.21.15.151:0/3962128209 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4a3004c320 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.417+0000 7f4a38819700 1 --2- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4a200381f0 0x7f4a2003a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.417+0000 7f4a38819700 1 --2- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4a200381f0 0x7f4a2003a6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4a24006fd0 tx=0x7f4a24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.422+0000 7f4a2a7fc700 1 -- 172.21.15.151:0/3962128209 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4a3001ac30 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.471+0000 7f4a2a7fc700 1 -- 172.21.15.151:0/3962128209 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f4a3001a430 con 0x7f4a3407bec0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.627+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a34103e90 con 0x7f4a200381f0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.635+0000 7f4a2a7fc700 1 -- 172.21.15.151:0/3962128209 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4a34103e90 con 0x7f4a200381f0 2024-04-16T02:26:04.225 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4a200381f0 msgr2=0x7f4a2003a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:04.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 --2- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4a200381f0 0x7f4a2003a6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4a24006fd0 tx=0x7f4a24006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:04.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 msgr2=0x7f4a3419c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:04.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 --2- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3419c470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009520 tx=0x7f4a30011ab0 comp rx=0 tx=0).stop 2024-04-16T02:26:04.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 shutdown_connections 2024-04-16T02:26:04.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 --2- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4a200381f0 0x7f4a2003a6b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:04.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 --2- 172.21.15.151:0/3962128209 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a3407bec0 0x7f4a3419c470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:04.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 >> 172.21.15.151:0/3962128209 conn(0x7f4a34101b40 msgr2=0x7f4a34103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:04.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 shutdown_connections 2024-04-16T02:26:04.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:03.637+0000 7f4a3b27e700 1 -- 172.21.15.151:0/3962128209 wait complete. 2024-04-16T02:26:04.743 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:04 smithi151 ceph-mon[29003]: [16/Apr/2024:02:26:03] ENGINE Bus STARTING 2024-04-16T02:26:04.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:04 smithi151 ceph-mon[29003]: [16/Apr/2024:02:26:03] ENGINE Serving on http://172.21.15.151:8765 2024-04-16T02:26:04.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:04 smithi151 ceph-mon[29003]: [16/Apr/2024:02:26:03] ENGINE Serving on https://172.21.15.151:7150 2024-04-16T02:26:04.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:04 smithi151 ceph-mon[29003]: [16/Apr/2024:02:26:03] ENGINE Bus STARTED 2024-04-16T02:26:04.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:04 smithi151 ceph-mon[29003]: mgrmap e8: smithi151.uqqame(active, since 2s) 2024-04-16T02:26:04.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:04 smithi151 ceph-mon[29003]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:04.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:04 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:04.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:04 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:26:05.420 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-16T02:26:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.669+0000 7febd37af700 1 Processor -- start 2024-04-16T02:26:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.669+0000 7febd37af700 1 -- start start 2024-04-16T02:26:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.669+0000 7febd37af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc106e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.669+0000 7febd37af700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febcc1073f0 con 0x7febcc106a00 2024-04-16T02:26:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.669+0000 7febd154b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.669+0000 7febd154b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc106e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41064/0 (socket says 172.21.15.151:41064) 2024-04-16T02:26:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.669+0000 7febd154b700 1 -- 172.21.15.151:0/773650985 learned_addr learned my addr 172.21.15.151:0/773650985 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.670+0000 7febd154b700 1 -- 172.21.15.151:0/773650985 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febcc105660 con 0x7febcc106a00 2024-04-16T02:26:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.670+0000 7febd154b700 1 --2- 172.21.15.151:0/773650985 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc106e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7febc80098a0 tx=0x7febc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988828b9c4355bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.670+0000 7febc3fff700 1 -- 172.21.15.151:0/773650985 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc8004180 con 0x7febcc106a00 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.670+0000 7febc3fff700 1 -- 172.21.15.151:0/773650985 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7febc80042e0 con 0x7febcc106a00 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.670+0000 7febc3fff700 1 -- 172.21.15.151:0/773650985 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc8011a50 con 0x7febcc106a00 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.670+0000 7febd37af700 1 -- 172.21.15.151:0/773650985 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 msgr2=0x7febcc106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.670+0000 7febd37af700 1 --2- 172.21.15.151:0/773650985 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc106e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7febc80098a0 tx=0x7febc8009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.671+0000 7febd37af700 1 -- 172.21.15.151:0/773650985 shutdown_connections 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.671+0000 7febd37af700 1 --2- 172.21.15.151:0/773650985 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc106e20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.671+0000 7febd37af700 1 -- 172.21.15.151:0/773650985 >> 172.21.15.151:0/773650985 conn(0x7febcc100190 msgr2=0x7febcc1025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.671+0000 7febd37af700 1 -- 172.21.15.151:0/773650985 shutdown_connections 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.671+0000 7febd37af700 1 -- 172.21.15.151:0/773650985 wait complete. 2024-04-16T02:26:05.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.671+0000 7febd37af700 1 Processor -- start 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.671+0000 7febd37af700 1 -- start start 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.671+0000 7febd37af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc1050d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.671+0000 7febd37af700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febcc19cf00 con 0x7febcc106a00 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febd154b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc1050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febd154b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc1050d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41066/0 (socket says 172.21.15.151:41066) 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febd154b700 1 -- 172.21.15.151:0/448728108 learned_addr learned my addr 172.21.15.151:0/448728108 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febd154b700 1 -- 172.21.15.151:0/448728108 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8009550 con 0x7febcc106a00 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febd154b700 1 --2- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc1050d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7febc8003f80 tx=0x7febc8011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febc27fc700 1 -- 172.21.15.151:0/448728108 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc8003a80 con 0x7febcc106a00 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febc27fc700 1 -- 172.21.15.151:0/448728108 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7febc8003be0 con 0x7febcc106a00 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febcc19d040 con 0x7febcc106a00 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febc27fc700 1 -- 172.21.15.151:0/448728108 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc8019820 con 0x7febcc106a00 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.672+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febcc19d4e0 con 0x7febcc106a00 2024-04-16T02:26:05.423 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.673+0000 7febc27fc700 1 -- 172.21.15.151:0/448728108 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7febc8019a50 con 0x7febcc106a00 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.673+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febcc0623c0 con 0x7febcc106a00 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.673+0000 7febc27fc700 1 --2- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7febb8038320 0x7febb803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.673+0000 7febc27fc700 1 -- 172.21.15.151:0/448728108 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7febc804c7d0 con 0x7febcc106a00 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.673+0000 7febd0d4a700 1 --2- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7febb8038320 0x7febb803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.674+0000 7febd0d4a700 1 --2- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7febb8038320 0x7febb803a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febbc006fd0 tx=0x7febbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.678+0000 7febc27fc700 1 -- 172.21.15.151:0/448728108 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7febc802b750 con 0x7febcc106a00 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.869+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7febcc109870 con 0x7febb8038320 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.869+0000 7febc27fc700 1 -- 172.21.15.151:0/448728108 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7febcc109870 con 0x7febb8038320 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.871+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7febb8038320 msgr2=0x7febb803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.871+0000 7febd37af700 1 --2- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7febb8038320 0x7febb803a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febbc006fd0 tx=0x7febbc006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.871+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 msgr2=0x7febcc1050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.871+0000 7febd37af700 1 --2- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc1050d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7febc8003f80 tx=0x7febc8011ab0 comp rx=0 tx=0).stop 2024-04-16T02:26:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.872+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 shutdown_connections 2024-04-16T02:26:05.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.872+0000 7febd37af700 1 --2- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7febb8038320 0x7febb803a7e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:05.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.872+0000 7febd37af700 1 --2- 172.21.15.151:0/448728108 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febcc106a00 0x7febcc1050d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:05.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.872+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 >> 172.21.15.151:0/448728108 conn(0x7febcc100190 msgr2=0x7febcc101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:05.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.872+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 shutdown_connections 2024-04-16T02:26:05.425 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:04.872+0000 7febd37af700 1 -- 172.21.15.151:0/448728108 wait complete. 2024-04-16T02:26:05.425 INFO:teuthology.orchestra.run.smithi151.stdout:Generating ssh key... 2024-04-16T02:26:05.993 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:05 smithi151 ceph-mon[29003]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:06.884 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.826+0000 7f8f91a7d700 1 Processor -- start 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.826+0000 7f8f91a7d700 1 -- start start 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.826+0000 7f8f91a7d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.826+0000 7f8f91a7d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c07a8b0 con 0x7f8f8c07be80 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.826+0000 7f8f8b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.826+0000 7f8f8b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41068/0 (socket says 172.21.15.151:41068) 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.826+0000 7f8f8b7fe700 1 -- 172.21.15.151:0/3786720135 learned_addr learned my addr 172.21.15.151:0/3786720135 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f8b7fe700 1 -- 172.21.15.151:0/3786720135 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c07a9f0 con 0x7f8f8c07be80 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f8b7fe700 1 --2- 172.21.15.151:0/3786720135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c07a2e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0098a0 tx=0x7f8f7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc34980e7106c1a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f8a7fc700 1 -- 172.21.15.151:0/3786720135 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f7c004180 con 0x7f8f8c07be80 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f8a7fc700 1 -- 172.21.15.151:0/3786720135 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8f7c0042e0 con 0x7f8f8c07be80 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f8a7fc700 1 -- 172.21.15.151:0/3786720135 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f7c011a50 con 0x7f8f8c07be80 2024-04-16T02:26:06.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f91a7d700 1 -- 172.21.15.151:0/3786720135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 msgr2=0x7f8f8c07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f91a7d700 1 --2- 172.21.15.151:0/3786720135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c07a2e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0098a0 tx=0x7f8f7c009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f91a7d700 1 -- 172.21.15.151:0/3786720135 shutdown_connections 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f91a7d700 1 --2- 172.21.15.151:0/3786720135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c07a2e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.827+0000 7f8f91a7d700 1 -- 172.21.15.151:0/3786720135 >> 172.21.15.151:0/3786720135 conn(0x7f8f8c101b00 msgr2=0x7f8f8c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.828+0000 7f8f91a7d700 1 -- 172.21.15.151:0/3786720135 shutdown_connections 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.828+0000 7f8f91a7d700 1 -- 172.21.15.151:0/3786720135 wait complete. 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.828+0000 7f8f91a7d700 1 Processor -- start 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.828+0000 7f8f91a7d700 1 -- start start 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.828+0000 7f8f91a7d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.828+0000 7f8f91a7d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c19c990 con 0x7f8f8c07be80 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f8b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f8b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c19c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41080/0 (socket says 172.21.15.151:41080) 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f8b7fe700 1 -- 172.21.15.151:0/815737696 learned_addr learned my addr 172.21.15.151:0/815737696 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:06.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f8b7fe700 1 -- 172.21.15.151:0/815737696 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c009550 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f8b7fe700 1 --2- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c19c450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009520 tx=0x7f8f7c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f88ff9700 1 -- 172.21.15.151:0/815737696 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f7c003a80 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8c19cb90 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f88ff9700 1 -- 172.21.15.151:0/815737696 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8f7c003be0 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f88ff9700 1 -- 172.21.15.151:0/815737696 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f7c0197b0 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.829+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8c19d030 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.830+0000 7f8f88ff9700 1 -- 172.21.15.151:0/815737696 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f8f7c01a430 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.830+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f8c10c8e0 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.830+0000 7f8f88ff9700 1 --2- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8f740382d0 0x7f8f7403a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.830+0000 7f8f88ff9700 1 -- 172.21.15.151:0/815737696 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8f7c04c5d0 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.830+0000 7f8f8affd700 1 --2- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8f740382d0 0x7f8f7403a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.831+0000 7f8f8affd700 1 --2- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8f740382d0 0x7f8f7403a790 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f80006fd0 tx=0x7f8f80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:05.835+0000 7f8f88ff9700 1 -- 172.21.15.151:0/815737696 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8f7c018530 con 0x7f8f8c07be80 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.024+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f8c19fe70 con 0x7f8f740382d0 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.294+0000 7f8f88ff9700 1 -- 172.21.15.151:0/815737696 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8f8c19fe70 con 0x7f8f740382d0 2024-04-16T02:26:06.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8f740382d0 msgr2=0x7f8f7403a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:06.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 --2- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8f740382d0 0x7f8f7403a790 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f80006fd0 tx=0x7f8f80006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:06.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 msgr2=0x7f8f8c19c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:06.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 --2- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c19c450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009520 tx=0x7f8f7c011ab0 comp rx=0 tx=0).stop 2024-04-16T02:26:06.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 shutdown_connections 2024-04-16T02:26:06.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 --2- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8f740382d0 0x7f8f7403a790 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:06.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 --2- 172.21.15.151:0/815737696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f8c07be80 0x7f8f8c19c450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:06.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 >> 172.21.15.151:0/815737696 conn(0x7f8f8c101b00 msgr2=0x7f8f8c1031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:06.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 shutdown_connections 2024-04-16T02:26:06.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:06.297+0000 7f8f91a7d700 1 -- 172.21.15.151:0/815737696 wait complete. 2024-04-16T02:26:07.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:07 smithi151 ceph-mon[29003]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:07.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:07 smithi151 ceph-mon[29003]: Generating ssh key... 2024-04-16T02:26:07.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:07 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:07.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:07 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:07.991 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1AKgxXyJ9Bybzp1KmEqfNrRdo3SuuX7B9fTWy4MpZCxjNwRSZ9T8AgylOikbKw5FQ0xp3yFerxgdNeRlcfWmQrtAOS7OU1CoOcCCxGvMVb8Ztm6zTV8YDsNDNzs104pLcFj9nXcwR+u2jR9BxD4i7G8XRudEWwYjLQ+/i+vWaDHg+JeVhC/abHWhqwNI8jXaZFrzl38qTdOlArzmmNbHE0G7nhQGJ1u2odJbrWNLXWNxyPyNsiPYS1XFz6/gvqIDhoIGZQaWiA3WfgW2NP5ybHvFRxoAsy9twQ7bOMKFi0DbhPpFD2zog+LIU8YklMQlkZcPz/3ZWCAOV61ZK44/sGe4y9TpwN0GgYynLld4Qgpm4oA3GexT4H0+FzbI6kKemmIcKgDYsvAuDdPT7JMcKOqWCe2esIRLyTZSAEtRnRchRr5suV/vk+YpkJat8p8HjrZBTb5bCiRHUuI/9YKgLdP7WKfcQ2wf2MNVpUqWLWt1EBm5gjxhLlnsz/8baSSU= ceph-80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:26:07.991 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.237+0000 7f82f7079700 1 Processor -- start 2024-04-16T02:26:07.991 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.237+0000 7f82f7079700 1 -- start start 2024-04-16T02:26:07.991 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.237+0000 7f82f7079700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f0104c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:07.991 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.237+0000 7f82f7079700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0105230 con 0x7f82f0104840 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.237+0000 7f82f4e15700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f0104c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.237+0000 7f82f4e15700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f0104c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51752/0 (socket says 172.21.15.151:51752) 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.237+0000 7f82f4e15700 1 -- 172.21.15.151:0/130887451 learned_addr learned my addr 172.21.15.151:0/130887451 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.238+0000 7f82f4e15700 1 -- 172.21.15.151:0/130887451 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f01077d0 con 0x7f82f0104840 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.238+0000 7f82f4e15700 1 --2- 172.21.15.151:0/130887451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f0104c60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f82e40098a0 tx=0x7f82e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c935d568b5c78896 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.238+0000 7f82ef7fe700 1 -- 172.21.15.151:0/130887451 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e4004180 con 0x7f82f0104840 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.238+0000 7f82ef7fe700 1 -- 172.21.15.151:0/130887451 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f82e40042e0 con 0x7f82f0104840 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.238+0000 7f82ef7fe700 1 -- 172.21.15.151:0/130887451 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e4011a50 con 0x7f82f0104840 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 -- 172.21.15.151:0/130887451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 msgr2=0x7f82f0104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 --2- 172.21.15.151:0/130887451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f0104c60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f82e40098a0 tx=0x7f82e4009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 -- 172.21.15.151:0/130887451 shutdown_connections 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 --2- 172.21.15.151:0/130887451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f0104c60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 -- 172.21.15.151:0/130887451 >> 172.21.15.151:0/130887451 conn(0x7f82f0100190 msgr2=0x7f82f01025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 -- 172.21.15.151:0/130887451 shutdown_connections 2024-04-16T02:26:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 -- 172.21.15.151:0/130887451 wait complete. 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 Processor -- start 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 -- start start 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.239+0000 7f82f7079700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f019a740 con 0x7f82f0104840 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82f4e15700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82f4e15700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51766/0 (socket says 172.21.15.151:51766) 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82f4e15700 1 -- 172.21.15.151:0/2184752121 learned_addr learned my addr 172.21.15.151:0/2184752121 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82f4e15700 1 -- 172.21.15.151:0/2184752121 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e4009550 con 0x7f82f0104840 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82f4e15700 1 --2- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f019a200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f82e4003de0 tx=0x7f82e4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82edffb700 1 -- 172.21.15.151:0/2184752121 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e4003930 con 0x7f82f0104840 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82edffb700 1 -- 172.21.15.151:0/2184752121 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f82e4003a90 con 0x7f82f0104840 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82edffb700 1 -- 172.21.15.151:0/2184752121 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e40196c0 con 0x7f82f0104840 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f019a940 con 0x7f82f0104840 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.240+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f019ade0 con 0x7f82f0104840 2024-04-16T02:26:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.241+0000 7f82edffb700 1 -- 172.21.15.151:0/2184752121 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f82e401abc0 con 0x7f82f0104840 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.241+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f00623c0 con 0x7f82f0104840 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.241+0000 7f82edffb700 1 --2- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f82e0038370 0x7f82e003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.241+0000 7f82edffb700 1 -- 172.21.15.151:0/2184752121 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f82e404c3e0 con 0x7f82f0104840 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.242+0000 7f82effff700 1 --2- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f82e0038370 0x7f82e003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.242+0000 7f82effff700 1 --2- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f82e0038370 0x7f82e003a830 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f82dc006fd0 tx=0x7f82dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.246+0000 7f82edffb700 1 -- 172.21.15.151:0/2184752121 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f82e4019930 con 0x7f82f0104840 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.439+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f82f010a9d0 con 0x7f82e0038370 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.439+0000 7f82edffb700 1 -- 172.21.15.151:0/2184752121 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f82f010a9d0 con 0x7f82e0038370 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.441+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f82e0038370 msgr2=0x7f82e003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.441+0000 7f82f7079700 1 --2- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f82e0038370 0x7f82e003a830 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f82dc006fd0 tx=0x7f82dc006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.441+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 msgr2=0x7f82f019a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.441+0000 7f82f7079700 1 --2- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f019a200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f82e4003de0 tx=0x7f82e4011ab0 comp rx=0 tx=0).stop 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.441+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 shutdown_connections 2024-04-16T02:26:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.441+0000 7f82f7079700 1 --2- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f82e0038370 0x7f82e003a830 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:07.995 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.441+0000 7f82f7079700 1 --2- 172.21.15.151:0/2184752121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82f0104840 0x7f82f019a200 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:07.995 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.441+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 >> 172.21.15.151:0/2184752121 conn(0x7f82f0100190 msgr2=0x7f82f0101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:07.995 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.442+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 shutdown_connections 2024-04-16T02:26:07.995 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:07.442+0000 7f82f7079700 1 -- 172.21.15.151:0/2184752121 wait complete. 2024-04-16T02:26:07.995 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-16T02:26:07.995 INFO:teuthology.orchestra.run.smithi151.stdout:Adding key to root@localhost authorized_keys... 2024-04-16T02:26:07.995 INFO:teuthology.orchestra.run.smithi151.stdout:Adding host smithi151... 2024-04-16T02:26:08.624 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:08 smithi151 ceph-mon[29003]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:09.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:09 smithi151 ceph-mon[29003]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "addr": "172.21.15.151", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:10.993 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:10 smithi151 ceph-mon[29003]: Deploying cephadm binary to smithi151 2024-04-16T02:26:11.818 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Added host 'smithi151' with addr '172.21.15.151' 2024-04-16T02:26:11.818 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.423+0000 7f6d9e161700 1 Processor -- start 2024-04-16T02:26:11.818 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.423+0000 7f6d9e161700 1 -- start start 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.423+0000 7f6d9e161700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9807a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.423+0000 7f6d9e161700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9807a8b0 con 0x7f6d9807be80 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.424+0000 7f6d977fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9807a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.424+0000 7f6d977fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9807a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51774/0 (socket says 172.21.15.151:51774) 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.424+0000 7f6d977fe700 1 -- 172.21.15.151:0/2087867807 learned_addr learned my addr 172.21.15.151:0/2087867807 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.424+0000 7f6d977fe700 1 -- 172.21.15.151:0/2087867807 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9807a9f0 con 0x7f6d9807be80 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.424+0000 7f6d977fe700 1 --2- 172.21.15.151:0/2087867807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9807a2e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009e80 tx=0x7f6d8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c1eebf533e197f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.424+0000 7f6d967fc700 1 -- 172.21.15.151:0/2087867807 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d8800e070 con 0x7f6d9807be80 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.424+0000 7f6d967fc700 1 -- 172.21.15.151:0/2087867807 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6d88003e90 con 0x7f6d9807be80 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.424+0000 7f6d967fc700 1 -- 172.21.15.151:0/2087867807 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d88004380 con 0x7f6d9807be80 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.425+0000 7f6d9e161700 1 -- 172.21.15.151:0/2087867807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 msgr2=0x7f6d9807a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.425+0000 7f6d9e161700 1 --2- 172.21.15.151:0/2087867807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9807a2e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009e80 tx=0x7f6d8800b330 comp rx=0 tx=0).stop 2024-04-16T02:26:11.819 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.425+0000 7f6d9e161700 1 -- 172.21.15.151:0/2087867807 shutdown_connections 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.425+0000 7f6d9e161700 1 --2- 172.21.15.151:0/2087867807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9807a2e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.425+0000 7f6d9e161700 1 -- 172.21.15.151:0/2087867807 >> 172.21.15.151:0/2087867807 conn(0x7f6d98101b40 msgr2=0x7f6d98103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.425+0000 7f6d9e161700 1 -- 172.21.15.151:0/2087867807 shutdown_connections 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.425+0000 7f6d9e161700 1 -- 172.21.15.151:0/2087867807 wait complete. 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.425+0000 7f6d9e161700 1 Processor -- start 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.426+0000 7f6d9e161700 1 -- start start 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.426+0000 7f6d9e161700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.426+0000 7f6d9e161700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9819c920 con 0x7f6d9807be80 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.426+0000 7f6d977fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.426+0000 7f6d977fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9819c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51780/0 (socket says 172.21.15.151:51780) 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.426+0000 7f6d977fe700 1 -- 172.21.15.151:0/621743186 learned_addr learned my addr 172.21.15.151:0/621743186 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.426+0000 7f6d977fe700 1 -- 172.21.15.151:0/621743186 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d88009b30 con 0x7f6d9807be80 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.426+0000 7f6d977fe700 1 --2- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9819c3e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009e50 tx=0x7f6d88004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:11.820 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.426+0000 7f6d94ff9700 1 -- 172.21.15.151:0/621743186 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d8800e040 con 0x7f6d9807be80 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.427+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9819cb20 con 0x7f6d9807be80 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.427+0000 7f6d94ff9700 1 -- 172.21.15.151:0/621743186 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6d88003cf0 con 0x7f6d9807be80 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.427+0000 7f6d94ff9700 1 -- 172.21.15.151:0/621743186 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d8801b770 con 0x7f6d9807be80 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.427+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9819d020 con 0x7f6d9807be80 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.427+0000 7f6d94ff9700 1 -- 172.21.15.151:0/621743186 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f6d88019040 con 0x7f6d9807be80 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.427+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d98062430 con 0x7f6d9807be80 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.428+0000 7f6d94ff9700 1 --2- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6d80038320 0x7f6d8003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.428+0000 7f6d94ff9700 1 -- 172.21.15.151:0/621743186 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6d8804bd60 con 0x7f6d9807be80 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.428+0000 7f6d96ffd700 1 --2- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6d80038320 0x7f6d8003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.428+0000 7f6d96ffd700 1 --2- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6d80038320 0x7f6d8003a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c006fd0 tx=0x7f6d8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.432+0000 7f6d94ff9700 1 -- 172.21.15.151:0/621743186 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6d98062430 con 0x7f6d9807be80 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:08.623+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi151", "addr": "172.21.15.151", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d98062430 con 0x7f6d80038320 2024-04-16T02:26:11.821 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.278+0000 7f6d94ff9700 1 -- 172.21.15.151:0/621743186 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6d98062430 con 0x7f6d80038320 2024-04-16T02:26:11.822 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6d80038320 msgr2=0x7f6d8003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:11.822 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 --2- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6d80038320 0x7f6d8003a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c006fd0 tx=0x7f6d8c006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:11.822 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 msgr2=0x7f6d9819c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:11.822 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 --2- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9819c3e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009e50 tx=0x7f6d88004920 comp rx=0 tx=0).stop 2024-04-16T02:26:11.822 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 shutdown_connections 2024-04-16T02:26:11.822 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 --2- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6d80038320 0x7f6d8003a7e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:11.823 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 --2- 172.21.15.151:0/621743186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9807be80 0x7f6d9819c3e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:11.823 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 >> 172.21.15.151:0/621743186 conn(0x7f6d98101b40 msgr2=0x7f6d981028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:11.823 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 shutdown_connections 2024-04-16T02:26:11.823 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:11.281+0000 7f6d9e161700 1 -- 172.21.15.151:0/621743186 wait complete. 2024-04-16T02:26:11.823 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying mon service with default placement... 2024-04-16T02:26:12.465 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:12 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:12.465 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:12 smithi151 ceph-mon[29003]: Added host smithi151 2024-04-16T02:26:12.465 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:12 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:26:13.724 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:13 smithi151 ceph-mon[29003]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:13.724 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:13 smithi151 ceph-mon[29003]: Saving service mon spec with placement count:5 2024-04-16T02:26:13.724 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:13 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:13.724 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:13 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:13.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-16T02:26:13.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.240+0000 7f8abc327700 1 Processor -- start 2024-04-16T02:26:13.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.240+0000 7f8abc327700 1 -- start start 2024-04-16T02:26:13.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.240+0000 7f8abc327700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab4105880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:13.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.240+0000 7f8abc327700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4105e50 con 0x7f8ab4105460 2024-04-16T02:26:13.731 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.241+0000 7f8aba0c3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab4105880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.241+0000 7f8aba0c3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab4105880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51796/0 (socket says 172.21.15.151:51796) 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.241+0000 7f8aba0c3700 1 -- 172.21.15.151:0/1228566681 learned_addr learned my addr 172.21.15.151:0/1228566681 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.241+0000 7f8aba0c3700 1 -- 172.21.15.151:0/1228566681 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab4106660 con 0x7f8ab4105460 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.241+0000 7f8aba0c3700 1 --2- 172.21.15.151:0/1228566681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab4105880 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00098a0 tx=0x7f8ab0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf380f619616bbba server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.241+0000 7f8ab90c1700 1 -- 172.21.15.151:0/1228566681 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab0004180 con 0x7f8ab4105460 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.241+0000 7f8ab90c1700 1 -- 172.21.15.151:0/1228566681 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8ab00042e0 con 0x7f8ab4105460 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.241+0000 7f8ab90c1700 1 -- 172.21.15.151:0/1228566681 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab0011a50 con 0x7f8ab4105460 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.242+0000 7f8abc327700 1 -- 172.21.15.151:0/1228566681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 msgr2=0x7f8ab4105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.242+0000 7f8abc327700 1 --2- 172.21.15.151:0/1228566681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab4105880 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00098a0 tx=0x7f8ab0009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.242+0000 7f8abc327700 1 -- 172.21.15.151:0/1228566681 shutdown_connections 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.242+0000 7f8abc327700 1 --2- 172.21.15.151:0/1228566681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab4105880 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:13.732 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.242+0000 7f8abc327700 1 -- 172.21.15.151:0/1228566681 >> 172.21.15.151:0/1228566681 conn(0x7f8ab4100a20 msgr2=0x7f8ab4102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.242+0000 7f8abc327700 1 -- 172.21.15.151:0/1228566681 shutdown_connections 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.242+0000 7f8abc327700 1 -- 172.21.15.151:0/1228566681 wait complete. 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.243+0000 7f8abc327700 1 Processor -- start 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.243+0000 7f8abc327700 1 -- start start 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.243+0000 7f8abc327700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.243+0000 7f8abc327700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab419a750 con 0x7f8ab4105460 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.243+0000 7f8aba0c3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.243+0000 7f8aba0c3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51802/0 (socket says 172.21.15.151:51802) 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.243+0000 7f8aba0c3700 1 -- 172.21.15.151:0/599221792 learned_addr learned my addr 172.21.15.151:0/599221792 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.243+0000 7f8aba0c3700 1 -- 172.21.15.151:0/599221792 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab0009550 con 0x7f8ab4105460 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.243+0000 7f8aba0c3700 1 --2- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab419a210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009520 tx=0x7f8ab0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.244+0000 7f8aab7fe700 1 -- 172.21.15.151:0/599221792 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab0003a80 con 0x7f8ab4105460 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.244+0000 7f8aab7fe700 1 -- 172.21.15.151:0/599221792 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8ab0003be0 con 0x7f8ab4105460 2024-04-16T02:26:13.733 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.244+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab419a950 con 0x7f8ab4105460 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.244+0000 7f8aab7fe700 1 -- 172.21.15.151:0/599221792 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab00197b0 con 0x7f8ab4105460 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.244+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab419adf0 con 0x7f8ab4105460 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.244+0000 7f8aab7fe700 1 -- 172.21.15.151:0/599221792 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f8ab001a430 con 0x7f8ab4105460 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.245+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab40623c0 con 0x7f8ab4105460 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.245+0000 7f8aab7fe700 1 --2- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8aa0038320 0x7f8aa003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.245+0000 7f8aab7fe700 1 -- 172.21.15.151:0/599221792 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8ab004c530 con 0x7f8ab4105460 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.245+0000 7f8ab98c2700 1 --2- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8aa0038320 0x7f8aa003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.245+0000 7f8ab98c2700 1 --2- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8aa0038320 0x7f8aa003a7e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4006fd0 tx=0x7f8aa4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.250+0000 7f8aab7fe700 1 -- 172.21.15.151:0/599221792 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8ab0018530 con 0x7f8ab4105460 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.460+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ab419dc30 con 0x7f8aa0038320 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.465+0000 7f8aab7fe700 1 -- 172.21.15.151:0/599221792 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8ab419dc30 con 0x7f8aa0038320 2024-04-16T02:26:13.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8aa0038320 msgr2=0x7f8aa003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 --2- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8aa0038320 0x7f8aa003a7e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4006fd0 tx=0x7f8aa4006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 msgr2=0x7f8ab419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 --2- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab419a210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009520 tx=0x7f8ab0011ab0 comp rx=0 tx=0).stop 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 shutdown_connections 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 --2- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f8aa0038320 0x7f8aa003a7e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 --2- 172.21.15.151:0/599221792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ab4105460 0x7f8ab419a210 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 >> 172.21.15.151:0/599221792 conn(0x7f8ab4100a20 msgr2=0x7f8ab4079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 shutdown_connections 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:12.467+0000 7f8abc327700 1 -- 172.21.15.151:0/599221792 wait complete. 2024-04-16T02:26:13.735 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying mgr service with default placement... 2024-04-16T02:26:15.200 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:14 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:15.201 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:14 smithi151 ceph-mon[29003]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:15.201 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:14 smithi151 ceph-mon[29003]: Saving service mgr spec with placement count:2 2024-04-16T02:26:15.201 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:14 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:15.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-16T02:26:15.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.170+0000 7f0829c3a700 1 Processor -- start 2024-04-16T02:26:15.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.170+0000 7f0829c3a700 1 -- start start 2024-04-16T02:26:15.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.170+0000 7f0829c3a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f0824105230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.170+0000 7f0829c3a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0824079e30 con 0x7f0824104e10 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.171+0000 7f08237fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f0824105230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.171+0000 7f08237fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f0824105230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51806/0 (socket says 172.21.15.151:51806) 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.171+0000 7f08237fe700 1 -- 172.21.15.151:0/2968782994 learned_addr learned my addr 172.21.15.151:0/2968782994 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.171+0000 7f08237fe700 1 -- 172.21.15.151:0/2968782994 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0824105770 con 0x7f0824104e10 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.171+0000 7f08237fe700 1 --2- 172.21.15.151:0/2968782994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f0824105230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0814009e80 tx=0x7f081400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9835a5657697c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.171+0000 7f08227fc700 1 -- 172.21.15.151:0/2968782994 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f081400e070 con 0x7f0824104e10 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.171+0000 7f08227fc700 1 -- 172.21.15.151:0/2968782994 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f081400b8c0 con 0x7f0824104e10 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.171+0000 7f08227fc700 1 -- 172.21.15.151:0/2968782994 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08140044d0 con 0x7f0824104e10 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.172+0000 7f0829c3a700 1 -- 172.21.15.151:0/2968782994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 msgr2=0x7f0824105230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.172+0000 7f0829c3a700 1 --2- 172.21.15.151:0/2968782994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f0824105230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0814009e80 tx=0x7f081400b330 comp rx=0 tx=0).stop 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.172+0000 7f0829c3a700 1 -- 172.21.15.151:0/2968782994 shutdown_connections 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.172+0000 7f0829c3a700 1 --2- 172.21.15.151:0/2968782994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f0824105230 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:15.477 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.172+0000 7f0829c3a700 1 -- 172.21.15.151:0/2968782994 >> 172.21.15.151:0/2968782994 conn(0x7f08241009f0 msgr2=0x7f0824102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.172+0000 7f0829c3a700 1 -- 172.21.15.151:0/2968782994 shutdown_connections 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.172+0000 7f0829c3a700 1 -- 172.21.15.151:0/2968782994 wait complete. 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.172+0000 7f0829c3a700 1 Processor -- start 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f0829c3a700 1 -- start start 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f0829c3a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f082419c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f0829c3a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082419c920 con 0x7f0824104e10 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f08237fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f082419c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f08237fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f082419c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51818/0 (socket says 172.21.15.151:51818) 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f08237fe700 1 -- 172.21.15.151:0/2426722861 learned_addr learned my addr 172.21.15.151:0/2426722861 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f08237fe700 1 -- 172.21.15.151:0/2426722861 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0814009b30 con 0x7f0824104e10 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f08237fe700 1 --2- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f082419c3e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0814009e50 tx=0x7f08140049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f0820ff9700 1 -- 172.21.15.151:0/2426722861 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f081400e040 con 0x7f0824104e10 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082419cb20 con 0x7f0824104e10 2024-04-16T02:26:15.478 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f0820ff9700 1 -- 172.21.15.151:0/2426722861 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f081400bcc0 con 0x7f0824104e10 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f0820ff9700 1 -- 172.21.15.151:0/2426722861 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0814012670 con 0x7f0824104e10 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.173+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082419d020 con 0x7f0824104e10 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.174+0000 7f0820ff9700 1 -- 172.21.15.151:0/2426722861 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f0814019040 con 0x7f0824104e10 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.175+0000 7f0820ff9700 1 --2- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f080c038330 0x7f080c03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.175+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0824195c90 con 0x7f0824104e10 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.175+0000 7f0820ff9700 1 -- 172.21.15.151:0/2426722861 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f081404bec0 con 0x7f0824104e10 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.175+0000 7f0822ffd700 1 --2- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f080c038330 0x7f080c03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.175+0000 7f0822ffd700 1 --2- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f080c038330 0x7f080c03a7f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0818006fd0 tx=0x7f0818006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.180+0000 7f0820ff9700 1 -- 172.21.15.151:0/2426722861 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f081401f020 con 0x7f0824104e10 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.407+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f08240611d0 con 0x7f080c038330 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.411+0000 7f0820ff9700 1 -- 172.21.15.151:0/2426722861 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f08240611d0 con 0x7f080c038330 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.413+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f080c038330 msgr2=0x7f080c03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.413+0000 7f0829c3a700 1 --2- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f080c038330 0x7f080c03a7f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0818006fd0 tx=0x7f0818006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:15.480 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.413+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 msgr2=0x7f082419c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:15.480 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.413+0000 7f0829c3a700 1 --2- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f082419c3e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0814009e50 tx=0x7f08140049f0 comp rx=0 tx=0).stop 2024-04-16T02:26:15.480 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.413+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 shutdown_connections 2024-04-16T02:26:15.480 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.413+0000 7f0829c3a700 1 --2- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f080c038330 0x7f080c03a7f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:15.480 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.413+0000 7f0829c3a700 1 --2- 172.21.15.151:0/2426722861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0824104e10 0x7f082419c3e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:15.480 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.413+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 >> 172.21.15.151:0/2426722861 conn(0x7f08241009f0 msgr2=0x7f08241909d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:15.480 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.414+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 shutdown_connections 2024-04-16T02:26:15.480 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:14.414+0000 7f0829c3a700 1 -- 172.21.15.151:0/2426722861 wait complete. 2024-04-16T02:26:15.480 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying crash service with default placement... 2024-04-16T02:26:16.170 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:15 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:16.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-16T02:26:16.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.944+0000 7f1bae6fc700 1 Processor -- start 2024-04-16T02:26:16.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.944+0000 7f1bae6fc700 1 -- start start 2024-04-16T02:26:16.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.944+0000 7f1bae6fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba8109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:16.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.944+0000 7f1bae6fc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8079e50 con 0x7f1ba8107020 2024-04-16T02:26:16.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.944+0000 7f1ba7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba8109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:16.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.944+0000 7f1ba7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba8109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51828/0 (socket says 172.21.15.151:51828) 2024-04-16T02:26:16.734 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.944+0000 7f1ba7fff700 1 -- 172.21.15.151:0/282782664 learned_addr learned my addr 172.21.15.151:0/282782664 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.945+0000 7f1ba7fff700 1 -- 172.21.15.151:0/282782664 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba8109950 con 0x7f1ba8107020 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.945+0000 7f1ba7fff700 1 --2- 172.21.15.151:0/282782664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba8109410 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1b980098a0 tx=0x7f1b98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15aaace299ce38d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.945+0000 7f1ba6ffd700 1 -- 172.21.15.151:0/282782664 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b98004180 con 0x7f1ba8107020 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.945+0000 7f1ba6ffd700 1 -- 172.21.15.151:0/282782664 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b980042e0 con 0x7f1ba8107020 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.945+0000 7f1ba6ffd700 1 -- 172.21.15.151:0/282782664 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b98011af0 con 0x7f1ba8107020 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.946+0000 7f1bae6fc700 1 -- 172.21.15.151:0/282782664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 msgr2=0x7f1ba8109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.946+0000 7f1bae6fc700 1 --2- 172.21.15.151:0/282782664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba8109410 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1b980098a0 tx=0x7f1b98009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.946+0000 7f1bae6fc700 1 -- 172.21.15.151:0/282782664 shutdown_connections 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.946+0000 7f1bae6fc700 1 --2- 172.21.15.151:0/282782664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba8109410 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.946+0000 7f1bae6fc700 1 -- 172.21.15.151:0/282782664 >> 172.21.15.151:0/282782664 conn(0x7f1ba81009f0 msgr2=0x7f1ba8102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.946+0000 7f1bae6fc700 1 -- 172.21.15.151:0/282782664 shutdown_connections 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.946+0000 7f1bae6fc700 1 -- 172.21.15.151:0/282782664 wait complete. 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.946+0000 7f1bae6fc700 1 Processor -- start 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.946+0000 7f1bae6fc700 1 -- start start 2024-04-16T02:26:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1bae6fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1bae6fc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba819a6f0 con 0x7f1ba8107020 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1ba7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1ba7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51830/0 (socket says 172.21.15.151:51830) 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1ba7fff700 1 -- 172.21.15.151:0/354041055 learned_addr learned my addr 172.21.15.151:0/354041055 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1ba7fff700 1 -- 172.21.15.151:0/354041055 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b98009550 con 0x7f1ba8107020 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1ba7fff700 1 --2- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba819a1b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b98006ba0 tx=0x7f1b98011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1ba57fa700 1 -- 172.21.15.151:0/354041055 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b98003b00 con 0x7f1ba8107020 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1ba57fa700 1 -- 172.21.15.151:0/354041055 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b9801a8e0 con 0x7f1ba8107020 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1ba57fa700 1 -- 172.21.15.151:0/354041055 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b980199c0 con 0x7f1ba8107020 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.947+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba819a8f0 con 0x7f1ba8107020 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.948+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba819ad90 con 0x7f1ba8107020 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.948+0000 7f1ba57fa700 1 -- 172.21.15.151:0/354041055 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f1b98019c30 con 0x7f1ba8107020 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.948+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba810a6e0 con 0x7f1ba8107020 2024-04-16T02:26:16.736 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.949+0000 7f1ba57fa700 1 --2- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f1b90038320 0x7f1b9003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.949+0000 7f1ba57fa700 1 -- 172.21.15.151:0/354041055 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1b9804c5f0 con 0x7f1ba8107020 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.949+0000 7f1ba77fe700 1 --2- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f1b90038320 0x7f1b9003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.949+0000 7f1ba77fe700 1 --2- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f1b90038320 0x7f1b9003a7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c006fd0 tx=0x7f1b9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:15.953+0000 7f1ba57fa700 1 -- 172.21.15.151:0/354041055 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1b9802b700 con 0x7f1ba8107020 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.170+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ba8101e30 con 0x7f1b90038320 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.177+0000 7f1ba57fa700 1 -- 172.21.15.151:0/354041055 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f1ba8101e30 con 0x7f1b90038320 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f1b90038320 msgr2=0x7f1b9003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 --2- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f1b90038320 0x7f1b9003a7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c006fd0 tx=0x7f1b9c006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 msgr2=0x7f1ba819a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 --2- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba819a1b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b98006ba0 tx=0x7f1b98011b50 comp rx=0 tx=0).stop 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 shutdown_connections 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 --2- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f1b90038320 0x7f1b9003a7e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 --2- 172.21.15.151:0/354041055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ba8107020 0x7f1ba819a1b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 >> 172.21.15.151:0/354041055 conn(0x7f1ba81009f0 msgr2=0x7f1ba8101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:16.738 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 shutdown_connections 2024-04-16T02:26:16.738 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:16.180+0000 7f1bae6fc700 1 -- 172.21.15.151:0/354041055 wait complete. 2024-04-16T02:26:16.738 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying ceph-exporter service with default placement... 2024-04-16T02:26:17.243 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:17 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:17.243 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:17 smithi151 ceph-mon[29003]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:17.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:17 smithi151 ceph-mon[29003]: Saving service crash spec with placement * 2024-04-16T02:26:17.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:17 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:18.333 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-16T02:26:18.333 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.149+0000 7ff9222df700 1 Processor -- start 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.149+0000 7ff9222df700 1 -- start start 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.149+0000 7ff9222df700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.149+0000 7ff9222df700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c105e60 con 0x7ff91c105470 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.149+0000 7ff91bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.149+0000 7ff91bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50846/0 (socket says 172.21.15.151:50846) 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.149+0000 7ff91bfff700 1 -- 172.21.15.151:0/2622669338 learned_addr learned my addr 172.21.15.151:0/2622669338 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.150+0000 7ff91bfff700 1 -- 172.21.15.151:0/2622669338 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c106670 con 0x7ff91c105470 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.150+0000 7ff91bfff700 1 --2- 172.21.15.151:0/2622669338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c105890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00bb40 tx=0x7ff90c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab336d10cde8e4b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.150+0000 7ff91affd700 1 -- 172.21.15.151:0/2622669338 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff90c00dc00 con 0x7ff91c105470 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.150+0000 7ff91affd700 1 -- 172.21.15.151:0/2622669338 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff90c0044e0 con 0x7ff91c105470 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.150+0000 7ff91affd700 1 -- 172.21.15.151:0/2622669338 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff90c010470 con 0x7ff91c105470 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.151+0000 7ff9222df700 1 -- 172.21.15.151:0/2622669338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 msgr2=0x7ff91c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:18.334 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.151+0000 7ff9222df700 1 --2- 172.21.15.151:0/2622669338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c105890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00bb40 tx=0x7ff90c00be50 comp rx=0 tx=0).stop 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.151+0000 7ff9222df700 1 -- 172.21.15.151:0/2622669338 shutdown_connections 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.151+0000 7ff9222df700 1 --2- 172.21.15.151:0/2622669338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c105890 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.151+0000 7ff9222df700 1 -- 172.21.15.151:0/2622669338 >> 172.21.15.151:0/2622669338 conn(0x7ff91c1009f0 msgr2=0x7ff91c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.151+0000 7ff9222df700 1 -- 172.21.15.151:0/2622669338 shutdown_connections 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.151+0000 7ff9222df700 1 -- 172.21.15.151:0/2622669338 wait complete. 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.151+0000 7ff9222df700 1 Processor -- start 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.151+0000 7ff9222df700 1 -- start start 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.152+0000 7ff9222df700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.152+0000 7ff9222df700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c1984e0 con 0x7ff91c105470 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.152+0000 7ff91bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.152+0000 7ff91bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50852/0 (socket says 172.21.15.151:50852) 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.152+0000 7ff91bfff700 1 -- 172.21.15.151:0/226950204 learned_addr learned my addr 172.21.15.151:0/226950204 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.152+0000 7ff91bfff700 1 -- 172.21.15.151:0/226950204 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c00b7f0 con 0x7ff91c105470 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.152+0000 7ff91bfff700 1 --2- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c197fa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff90c003fc0 tx=0x7ff90c0040a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.152+0000 7ff9197fa700 1 -- 172.21.15.151:0/226950204 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff90c01e070 con 0x7ff91c105470 2024-04-16T02:26:18.335 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.153+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c1986e0 con 0x7ff91c105470 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.153+0000 7ff9197fa700 1 -- 172.21.15.151:0/226950204 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff90c00de20 con 0x7ff91c105470 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.153+0000 7ff9197fa700 1 -- 172.21.15.151:0/226950204 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff90c022ac0 con 0x7ff91c105470 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.153+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c198b80 con 0x7ff91c105470 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.153+0000 7ff9197fa700 1 -- 172.21.15.151:0/226950204 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff90c02c440 con 0x7ff91c105470 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.154+0000 7ff9197fa700 1 --2- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff904038320 0x7ff90403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.154+0000 7ff9197fa700 1 -- 172.21.15.151:0/226950204 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff90c04fa20 con 0x7ff91c105470 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.154+0000 7ff91b7fe700 1 --2- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff904038320 0x7ff90403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.154+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91c191d00 con 0x7ff91c105470 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.154+0000 7ff91b7fe700 1 --2- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff904038320 0x7ff90403a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff910006fd0 tx=0x7ff910006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.159+0000 7ff9197fa700 1 -- 172.21.15.151:0/226950204 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff90c027030 con 0x7ff91c105470 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.366+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7ff91c0611d0 con 0x7ff904038320 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.371+0000 7ff9197fa700 1 -- 172.21.15.151:0/226950204 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7ff91c0611d0 con 0x7ff904038320 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.373+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff904038320 msgr2=0x7ff90403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.373+0000 7ff9222df700 1 --2- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff904038320 0x7ff90403a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff910006fd0 tx=0x7ff910006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:18.336 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.373+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 msgr2=0x7ff91c197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.373+0000 7ff9222df700 1 --2- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c197fa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff90c003fc0 tx=0x7ff90c0040a0 comp rx=0 tx=0).stop 2024-04-16T02:26:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.373+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 shutdown_connections 2024-04-16T02:26:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.373+0000 7ff9222df700 1 --2- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff904038320 0x7ff90403a7e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.373+0000 7ff9222df700 1 --2- 172.21.15.151:0/226950204 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff91c105470 0x7ff91c197fa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.373+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 >> 172.21.15.151:0/226950204 conn(0x7ff91c1009f0 msgr2=0x7ff91c18ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.374+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 shutdown_connections 2024-04-16T02:26:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:17.374+0000 7ff9222df700 1 -- 172.21.15.151:0/226950204 wait complete. 2024-04-16T02:26:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying prometheus service with default placement... 2024-04-16T02:26:18.634 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:18 smithi151 ceph-mon[29003]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:18.634 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:18 smithi151 ceph-mon[29003]: Saving service ceph-exporter spec with placement * 2024-04-16T02:26:18.634 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:18 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:19.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-16T02:26:19.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.813+0000 7fa90a9a0700 1 Processor -- start 2024-04-16T02:26:19.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.813+0000 7fa90a9a0700 1 -- start start 2024-04-16T02:26:19.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa90a9a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa904105870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:19.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa90a9a0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa904105e40 con 0x7fa904105450 2024-04-16T02:26:19.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa903fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa904105870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:19.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa903fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa904105870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50860/0 (socket says 172.21.15.151:50860) 2024-04-16T02:26:19.886 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa903fff700 1 -- 172.21.15.151:0/3379324975 learned_addr learned my addr 172.21.15.151:0/3379324975 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa903fff700 1 -- 172.21.15.151:0/3379324975 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa904106650 con 0x7fa904105450 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa903fff700 1 --2- 172.21.15.151:0/3379324975 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa904105870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa8f40098a0 tx=0x7fa8f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8aaece418a494a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa902ffd700 1 -- 172.21.15.151:0/3379324975 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8f4004180 con 0x7fa904105450 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa902ffd700 1 -- 172.21.15.151:0/3379324975 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa8f40042e0 con 0x7fa904105450 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.814+0000 7fa902ffd700 1 -- 172.21.15.151:0/3379324975 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8f4011af0 con 0x7fa904105450 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.815+0000 7fa90a9a0700 1 -- 172.21.15.151:0/3379324975 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 msgr2=0x7fa904105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.815+0000 7fa90a9a0700 1 --2- 172.21.15.151:0/3379324975 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa904105870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa8f40098a0 tx=0x7fa8f4009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.815+0000 7fa90a9a0700 1 -- 172.21.15.151:0/3379324975 shutdown_connections 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.815+0000 7fa90a9a0700 1 --2- 172.21.15.151:0/3379324975 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa904105870 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.815+0000 7fa90a9a0700 1 -- 172.21.15.151:0/3379324975 >> 172.21.15.151:0/3379324975 conn(0x7fa9041009b0 msgr2=0x7fa904102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.815+0000 7fa90a9a0700 1 -- 172.21.15.151:0/3379324975 shutdown_connections 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.815+0000 7fa90a9a0700 1 -- 172.21.15.151:0/3379324975 wait complete. 2024-04-16T02:26:19.887 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.816+0000 7fa90a9a0700 1 Processor -- start 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.816+0000 7fa90a9a0700 1 -- start start 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.816+0000 7fa90a9a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa90419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.816+0000 7fa90a9a0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90419a7b0 con 0x7fa904105450 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.816+0000 7fa903fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa90419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.816+0000 7fa903fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa90419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50870/0 (socket says 172.21.15.151:50870) 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.816+0000 7fa903fff700 1 -- 172.21.15.151:0/579351664 learned_addr learned my addr 172.21.15.151:0/579351664 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.816+0000 7fa903fff700 1 -- 172.21.15.151:0/579351664 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f4009550 con 0x7fa904105450 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.817+0000 7fa903fff700 1 --2- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa90419a270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4011670 tx=0x7fa8f4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.817+0000 7fa9017fa700 1 -- 172.21.15.151:0/579351664 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8f40039c0 con 0x7fa904105450 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.817+0000 7fa9017fa700 1 -- 172.21.15.151:0/579351664 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa8f4003b20 con 0x7fa904105450 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.817+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90419a9b0 con 0x7fa904105450 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.817+0000 7fa9017fa700 1 -- 172.21.15.151:0/579351664 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8f40247f0 con 0x7fa904105450 2024-04-16T02:26:19.888 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.817+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90419ae50 con 0x7fa904105450 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.818+0000 7fa9017fa700 1 -- 172.21.15.151:0/579351664 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa8f4024ab0 con 0x7fa904105450 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.818+0000 7fa9017fa700 1 --2- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fa8ec0382e0 0x7fa8ec03a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.818+0000 7fa9017fa700 1 -- 172.21.15.151:0/579351664 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa8f404c2f0 con 0x7fa904105450 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.818+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9040623c0 con 0x7fa904105450 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.818+0000 7fa9037fe700 1 --2- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fa8ec0382e0 0x7fa8ec03a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.818+0000 7fa9037fe700 1 --2- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fa8ec0382e0 0x7fa8ec03a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8006fd0 tx=0x7fa8f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:18.823+0000 7fa9017fa700 1 -- 172.21.15.151:0/579351664 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa8f4022310 con 0x7fa904105450 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.026+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa904102cd0 con 0x7fa8ec0382e0 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.030+0000 7fa9017fa700 1 -- 172.21.15.151:0/579351664 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa904102cd0 con 0x7fa8ec0382e0 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.032+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fa8ec0382e0 msgr2=0x7fa8ec03a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.032+0000 7fa90a9a0700 1 --2- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fa8ec0382e0 0x7fa8ec03a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8006fd0 tx=0x7fa8f8006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.033+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 msgr2=0x7fa90419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:19.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.033+0000 7fa90a9a0700 1 --2- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa90419a270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4011670 tx=0x7fa8f4011aa0 comp rx=0 tx=0).stop 2024-04-16T02:26:19.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.033+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 shutdown_connections 2024-04-16T02:26:19.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.033+0000 7fa90a9a0700 1 --2- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fa8ec0382e0 0x7fa8ec03a7a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:19.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.033+0000 7fa90a9a0700 1 --2- 172.21.15.151:0/579351664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa904105450 0x7fa90419a270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:19.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.033+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 >> 172.21.15.151:0/579351664 conn(0x7fa9041009b0 msgr2=0x7fa9041020b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:19.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.033+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 shutdown_connections 2024-04-16T02:26:19.890 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:19.033+0000 7fa90a9a0700 1 -- 172.21.15.151:0/579351664 wait complete. 2024-04-16T02:26:19.890 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying grafana service with default placement... 2024-04-16T02:26:20.138 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:20 smithi151 ceph-mon[29003]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:20.138 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:20 smithi151 ceph-mon[29003]: Saving service prometheus spec with placement count:1 2024-04-16T02:26:20.138 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:20 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:21.099 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-16T02:26:21.099 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.346+0000 7f95c184d700 1 Processor -- start 2024-04-16T02:26:21.099 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.346+0000 7f95c184d700 1 -- start start 2024-04-16T02:26:21.099 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.346+0000 7f95c184d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc1093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:21.099 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.346+0000 7f95c184d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc074680 con 0x7f95bc106fe0 2024-04-16T02:26:21.099 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.346+0000 7f95baffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc1093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:21.099 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.347+0000 7f95baffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc1093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50886/0 (socket says 172.21.15.151:50886) 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.347+0000 7f95baffd700 1 -- 172.21.15.151:0/639785831 learned_addr learned my addr 172.21.15.151:0/639785831 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.347+0000 7f95baffd700 1 -- 172.21.15.151:0/639785831 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc109910 con 0x7f95bc106fe0 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.347+0000 7f95baffd700 1 --2- 172.21.15.151:0/639785831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc1093d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0098a0 tx=0x7f95ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102e123840a48015 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.347+0000 7f95b9ffb700 1 -- 172.21.15.151:0/639785831 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95ac004180 con 0x7f95bc106fe0 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.347+0000 7f95b9ffb700 1 -- 172.21.15.151:0/639785831 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f95ac0042e0 con 0x7f95bc106fe0 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.347+0000 7f95b9ffb700 1 -- 172.21.15.151:0/639785831 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95ac011af0 con 0x7f95bc106fe0 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.348+0000 7f95c184d700 1 -- 172.21.15.151:0/639785831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 msgr2=0x7f95bc1093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.348+0000 7f95c184d700 1 --2- 172.21.15.151:0/639785831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc1093d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0098a0 tx=0x7f95ac009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.348+0000 7f95c184d700 1 -- 172.21.15.151:0/639785831 shutdown_connections 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.348+0000 7f95c184d700 1 --2- 172.21.15.151:0/639785831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc1093d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.348+0000 7f95c184d700 1 -- 172.21.15.151:0/639785831 >> 172.21.15.151:0/639785831 conn(0x7f95bc1009f0 msgr2=0x7f95bc102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.348+0000 7f95c184d700 1 -- 172.21.15.151:0/639785831 shutdown_connections 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.348+0000 7f95c184d700 1 -- 172.21.15.151:0/639785831 wait complete. 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.348+0000 7f95c184d700 1 Processor -- start 2024-04-16T02:26:21.100 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95c184d700 1 -- start start 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95c184d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95c184d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc19a6b0 con 0x7f95bc106fe0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95baffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95baffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50888/0 (socket says 172.21.15.151:50888) 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95baffd700 1 -- 172.21.15.151:0/409193275 learned_addr learned my addr 172.21.15.151:0/409193275 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95baffd700 1 -- 172.21.15.151:0/409193275 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac009550 con 0x7f95bc106fe0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95baffd700 1 --2- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc19a170 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95ac006ba0 tx=0x7f95ac011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95c084b700 1 -- 172.21.15.151:0/409193275 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95ac003b00 con 0x7f95bc106fe0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95c084b700 1 -- 172.21.15.151:0/409193275 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f95ac01a8e0 con 0x7f95bc106fe0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc19a8b0 con 0x7f95bc106fe0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.349+0000 7f95c084b700 1 -- 172.21.15.151:0/409193275 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95ac0199c0 con 0x7f95bc106fe0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.350+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc19ad50 con 0x7f95bc106fe0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.350+0000 7f95c084b700 1 -- 172.21.15.151:0/409193275 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f95ac019c80 con 0x7f95bc106fe0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.350+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95bc10a700 con 0x7f95bc106fe0 2024-04-16T02:26:21.101 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.351+0000 7f95c084b700 1 --2- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f95a80382d0 0x7f95a803a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.351+0000 7f95c084b700 1 -- 172.21.15.151:0/409193275 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f95ac04c820 con 0x7f95bc106fe0 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.351+0000 7f95ba7fc700 1 --2- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f95a80382d0 0x7f95a803a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.351+0000 7f95ba7fc700 1 --2- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f95a80382d0 0x7f95a803a790 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95a400ad30 tx=0x7f95a40093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.355+0000 7f95c084b700 1 -- 172.21.15.151:0/409193275 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f95ac02b700 con 0x7f95bc106fe0 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.557+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f95bc101e50 con 0x7f95a80382d0 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.561+0000 7f95c084b700 1 -- 172.21.15.151:0/409193275 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f95bc101e50 con 0x7f95a80382d0 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f95a80382d0 msgr2=0x7f95a803a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 --2- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f95a80382d0 0x7f95a803a790 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95a400ad30 tx=0x7f95a40093f0 comp rx=0 tx=0).stop 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 msgr2=0x7f95bc19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 --2- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc19a170 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95ac006ba0 tx=0x7f95ac011b50 comp rx=0 tx=0).stop 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 shutdown_connections 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 --2- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f95a80382d0 0x7f95a803a790 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 --2- 172.21.15.151:0/409193275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95bc106fe0 0x7f95bc19a170 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:21.102 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 >> 172.21.15.151:0/409193275 conn(0x7f95bc1009f0 msgr2=0x7f95bc101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:21.103 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 shutdown_connections 2024-04-16T02:26:21.103 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:20.563+0000 7f95c184d700 1 -- 172.21.15.151:0/409193275 wait complete. 2024-04-16T02:26:21.103 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying node-exporter service with default placement... 2024-04-16T02:26:21.993 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:21 smithi151 ceph-mon[29003]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:21.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:21 smithi151 ceph-mon[29003]: Saving service grafana spec with placement count:1 2024-04-16T02:26:21.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:21 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:22.272 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-16T02:26:22.272 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.503+0000 7f6509ec0700 1 Processor -- start 2024-04-16T02:26:22.272 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.503+0000 7f6509ec0700 1 -- start start 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.503+0000 7f6509ec0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f6504106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.503+0000 7f6509ec0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6504106f50 con 0x7f6504106560 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.503+0000 7f65037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f6504106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.503+0000 7f65037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f6504106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50896/0 (socket says 172.21.15.151:50896) 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.503+0000 7f65037fe700 1 -- 172.21.15.151:0/1033180374 learned_addr learned my addr 172.21.15.151:0/1033180374 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.503+0000 7f65037fe700 1 -- 172.21.15.151:0/1033180374 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6504107760 con 0x7f6504106560 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f65037fe700 1 --2- 172.21.15.151:0/1033180374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f6504106980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f64f40098a0 tx=0x7f64f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d69613f6d70583 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f65027fc700 1 -- 172.21.15.151:0/1033180374 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64f4004180 con 0x7f6504106560 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f65027fc700 1 -- 172.21.15.151:0/1033180374 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f64f40042e0 con 0x7f6504106560 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f65027fc700 1 -- 172.21.15.151:0/1033180374 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64f4011af0 con 0x7f6504106560 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f6509ec0700 1 -- 172.21.15.151:0/1033180374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 msgr2=0x7f6504106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f6509ec0700 1 --2- 172.21.15.151:0/1033180374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f6504106980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f64f40098a0 tx=0x7f64f4009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:22.273 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f6509ec0700 1 -- 172.21.15.151:0/1033180374 shutdown_connections 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f6509ec0700 1 --2- 172.21.15.151:0/1033180374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f6504106980 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f6509ec0700 1 -- 172.21.15.151:0/1033180374 >> 172.21.15.151:0/1033180374 conn(0x7f6504101b40 msgr2=0x7f6504103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f6509ec0700 1 -- 172.21.15.151:0/1033180374 shutdown_connections 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.504+0000 7f6509ec0700 1 -- 172.21.15.151:0/1033180374 wait complete. 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.505+0000 7f6509ec0700 1 Processor -- start 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.505+0000 7f6509ec0700 1 -- start start 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.505+0000 7f6509ec0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f650419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.505+0000 7f6509ec0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650419a780 con 0x7f6504106560 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.505+0000 7f65037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f650419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.505+0000 7f65037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f650419a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50906/0 (socket says 172.21.15.151:50906) 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.505+0000 7f65037fe700 1 -- 172.21.15.151:0/2298229253 learned_addr learned my addr 172.21.15.151:0/2298229253 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.506+0000 7f65037fe700 1 -- 172.21.15.151:0/2298229253 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f4009550 con 0x7f6504106560 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.506+0000 7f65037fe700 1 --2- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f650419a240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64f4011670 tx=0x7f64f4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:22.274 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.506+0000 7f6500ff9700 1 -- 172.21.15.151:0/2298229253 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64f40039c0 con 0x7f6504106560 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.506+0000 7f6500ff9700 1 -- 172.21.15.151:0/2298229253 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f64f4003b20 con 0x7f6504106560 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.506+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650419a980 con 0x7f6504106560 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.506+0000 7f6500ff9700 1 -- 172.21.15.151:0/2298229253 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64f40247f0 con 0x7f6504106560 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.506+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650419ae20 con 0x7f6504106560 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.507+0000 7f6500ff9700 1 -- 172.21.15.151:0/2298229253 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f64f401ac70 con 0x7f6504106560 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.507+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65040623c0 con 0x7f6504106560 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.507+0000 7f6500ff9700 1 --2- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f64ec038330 0x7f64ec03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.507+0000 7f6500ff9700 1 -- 172.21.15.151:0/2298229253 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f64f404c330 con 0x7f6504106560 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.507+0000 7f6502ffd700 1 --2- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f64ec038330 0x7f64ec03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.508+0000 7f6502ffd700 1 --2- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f64ec038330 0x7f64ec03a7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f64f8006fd0 tx=0x7f64f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.512+0000 7f6500ff9700 1 -- 172.21.15.151:0/2298229253 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f64f4024a60 con 0x7f6504106560 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.716+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f650410a960 con 0x7f64ec038330 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.721+0000 7f6500ff9700 1 -- 172.21.15.151:0/2298229253 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f650410a960 con 0x7f64ec038330 2024-04-16T02:26:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f64ec038330 msgr2=0x7f64ec03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 --2- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f64ec038330 0x7f64ec03a7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f64f8006fd0 tx=0x7f64f8006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 msgr2=0x7f650419a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 --2- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f650419a240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64f4011670 tx=0x7f64f4011aa0 comp rx=0 tx=0).stop 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 shutdown_connections 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 --2- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f64ec038330 0x7f64ec03a7f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 --2- 172.21.15.151:0/2298229253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6504106560 0x7f650419a240 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 >> 172.21.15.151:0/2298229253 conn(0x7f6504101b40 msgr2=0x7f6504102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 shutdown_connections 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:21.723+0000 7f6509ec0700 1 -- 172.21.15.151:0/2298229253 wait complete. 2024-04-16T02:26:22.276 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying alertmanager service with default placement... 2024-04-16T02:26:22.644 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:22 smithi151 ceph-mon[29003]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:26:22.644 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:22 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:23.521 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.644+0000 7f6fcc4a4700 1 Processor -- start 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.644+0000 7f6fcc4a4700 1 -- start start 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.644+0000 7f6fcc4a4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc4105d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.644+0000 7f6fcc4a4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4106360 con 0x7f6fc4105970 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.644+0000 7f6fca240700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc4105d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.644+0000 7f6fca240700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc4105d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50920/0 (socket says 172.21.15.151:50920) 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.644+0000 7f6fca240700 1 -- 172.21.15.151:0/1126557585 learned_addr learned my addr 172.21.15.151:0/1126557585 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.645+0000 7f6fca240700 1 -- 172.21.15.151:0/1126557585 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4106690 con 0x7f6fc4105970 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.645+0000 7f6fca240700 1 --2- 172.21.15.151:0/1126557585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc4105d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00098a0 tx=0x7f6fc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f95713eda26bdbb server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.645+0000 7f6fc923e700 1 -- 172.21.15.151:0/1126557585 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fc0004180 con 0x7f6fc4105970 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.645+0000 7f6fc923e700 1 -- 172.21.15.151:0/1126557585 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6fc00042e0 con 0x7f6fc4105970 2024-04-16T02:26:23.522 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.645+0000 7f6fc923e700 1 -- 172.21.15.151:0/1126557585 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fc0011af0 con 0x7f6fc4105970 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.646+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/1126557585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 msgr2=0x7f6fc4105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.646+0000 7f6fcc4a4700 1 --2- 172.21.15.151:0/1126557585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc4105d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00098a0 tx=0x7f6fc0009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.646+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/1126557585 shutdown_connections 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.646+0000 7f6fcc4a4700 1 --2- 172.21.15.151:0/1126557585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc4105d90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.646+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/1126557585 >> 172.21.15.151:0/1126557585 conn(0x7f6fc4100190 msgr2=0x7f6fc41025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.646+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/1126557585 shutdown_connections 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.646+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/1126557585 wait complete. 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.647+0000 7f6fcc4a4700 1 Processor -- start 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.647+0000 7f6fcc4a4700 1 -- start start 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.647+0000 7f6fcc4a4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc41990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.647+0000 7f6fcc4a4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4199610 con 0x7f6fc4105970 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.647+0000 7f6fca240700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc41990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.647+0000 7f6fca240700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc41990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50930/0 (socket says 172.21.15.151:50930) 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.647+0000 7f6fca240700 1 -- 172.21.15.151:0/2640499111 learned_addr learned my addr 172.21.15.151:0/2640499111 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.647+0000 7f6fca240700 1 -- 172.21.15.151:0/2640499111 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc0009550 con 0x7f6fc4105970 2024-04-16T02:26:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.647+0000 7f6fca240700 1 --2- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc41990d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006ba0 tx=0x7f6fc0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.648+0000 7f6fbb7fe700 1 -- 172.21.15.151:0/2640499111 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fc0003b00 con 0x7f6fc4105970 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.648+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc4199810 con 0x7f6fc4105970 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.648+0000 7f6fbb7fe700 1 -- 172.21.15.151:0/2640499111 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6fc001a8e0 con 0x7f6fc4105970 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.648+0000 7f6fbb7fe700 1 -- 172.21.15.151:0/2640499111 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fc00199c0 con 0x7f6fc4105970 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.648+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc4199c30 con 0x7f6fc4105970 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.649+0000 7f6fbb7fe700 1 -- 172.21.15.151:0/2640499111 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f6fc0019c30 con 0x7f6fc4105970 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.649+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc40623c0 con 0x7f6fc4105970 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.649+0000 7f6fbb7fe700 1 --2- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6fb003a530 0x7f6fb003c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.649+0000 7f6fbb7fe700 1 -- 172.21.15.151:0/2640499111 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6fc004c710 con 0x7f6fc4105970 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.649+0000 7f6fc9a3f700 1 --2- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6fb003a530 0x7f6fb003c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.649+0000 7f6fc9a3f700 1 --2- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6fb003a530 0x7f6fb003c9f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4006fd0 tx=0x7f6fb4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.657+0000 7f6fbb7fe700 1 -- 172.21.15.151:0/2640499111 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6fc002d440 con 0x7f6fc4105970 2024-04-16T02:26:23.524 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.865+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fc4104ea0 con 0x7f6fb003a530 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.871+0000 7f6fbb7fe700 1 -- 172.21.15.151:0/2640499111 <== mgr.14120 v2:172.21.15.151:6800/2400549256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f6fc4104ea0 con 0x7f6fb003a530 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6fb003a530 msgr2=0x7f6fb003c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 --2- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6fb003a530 0x7f6fb003c9f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4006fd0 tx=0x7f6fb4006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 msgr2=0x7f6fc41990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 --2- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc41990d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006ba0 tx=0x7f6fc0011b50 comp rx=0 tx=0).stop 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 shutdown_connections 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 --2- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f6fb003a530 0x7f6fb003c9f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 --2- 172.21.15.151:0/2640499111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc4105970 0x7f6fc41990d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 >> 172.21.15.151:0/2640499111 conn(0x7f6fc4100190 msgr2=0x7f6fc4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 shutdown_connections 2024-04-16T02:26:23.525 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:22.873+0000 7f6fcc4a4700 1 -- 172.21.15.151:0/2640499111 wait complete. 2024-04-16T02:26:23.766 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:23 smithi151 ceph-mon[29003]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:23.766 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:23 smithi151 ceph-mon[29003]: Saving service node-exporter spec with placement * 2024-04-16T02:26:23.766 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:23 smithi151 ceph-mon[29003]: from='mgr.14120 172.21.15.151:0/1655875404' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:24.743 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:24 smithi151 ceph-mon[29003]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:24.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:24 smithi151 ceph-mon[29003]: Saving service alertmanager spec with placement count:1 2024-04-16T02:26:24.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:24 smithi151 ceph-mon[29003]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:26:24.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:24 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/382298551' entity='client.admin' 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.945+0000 7f4ee9942700 1 Processor -- start 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.945+0000 7f4ee9942700 1 -- start start 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee9942700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee4106960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee9942700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee4106f30 con 0x7f4ee4106540 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee4106960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee4106960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50938/0 (socket says 172.21.15.151:50938) 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee2ffd700 1 -- 172.21.15.151:0/3046863476 learned_addr learned my addr 172.21.15.151:0/3046863476 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee2ffd700 1 -- 172.21.15.151:0/3046863476 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee4107740 con 0x7f4ee4106540 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee2ffd700 1 --2- 172.21.15.151:0/3046863476 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee4106960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc0098a0 tx=0x7f4ecc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c03abd7ba3c38b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee1ffb700 1 -- 172.21.15.151:0/3046863476 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ecc004180 con 0x7f4ee4106540 2024-04-16T02:26:24.769 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee1ffb700 1 -- 172.21.15.151:0/3046863476 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4ecc0042e0 con 0x7f4ee4106540 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.946+0000 7f4ee1ffb700 1 -- 172.21.15.151:0/3046863476 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ecc011af0 con 0x7f4ee4106540 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.947+0000 7f4ee9942700 1 -- 172.21.15.151:0/3046863476 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 msgr2=0x7f4ee4106960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.947+0000 7f4ee9942700 1 --2- 172.21.15.151:0/3046863476 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee4106960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc0098a0 tx=0x7f4ecc009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.947+0000 7f4ee9942700 1 -- 172.21.15.151:0/3046863476 shutdown_connections 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.947+0000 7f4ee9942700 1 --2- 172.21.15.151:0/3046863476 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee4106960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.947+0000 7f4ee9942700 1 -- 172.21.15.151:0/3046863476 >> 172.21.15.151:0/3046863476 conn(0x7f4ee4101b00 msgr2=0x7f4ee4103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.947+0000 7f4ee9942700 1 -- 172.21.15.151:0/3046863476 shutdown_connections 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.947+0000 7f4ee9942700 1 -- 172.21.15.151:0/3046863476 wait complete. 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.948+0000 7f4ee9942700 1 Processor -- start 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.948+0000 7f4ee9942700 1 -- start start 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.948+0000 7f4ee9942700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.948+0000 7f4ee9942700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee419a7e0 con 0x7f4ee4106540 2024-04-16T02:26:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.948+0000 7f4ee2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.948+0000 7f4ee2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50950/0 (socket says 172.21.15.151:50950) 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.948+0000 7f4ee2ffd700 1 -- 172.21.15.151:0/382298551 learned_addr learned my addr 172.21.15.151:0/382298551 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.948+0000 7f4ee2ffd700 1 -- 172.21.15.151:0/382298551 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc009550 con 0x7f4ee4106540 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.948+0000 7f4ee2ffd700 1 --2- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee419a2a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc011670 tx=0x7f4ecc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.949+0000 7f4ee8940700 1 -- 172.21.15.151:0/382298551 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ecc0039f0 con 0x7f4ee4106540 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.949+0000 7f4ee8940700 1 -- 172.21.15.151:0/382298551 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4ecc003b50 con 0x7f4ee4106540 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.949+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee419a9e0 con 0x7f4ee4106540 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.949+0000 7f4ee8940700 1 -- 172.21.15.151:0/382298551 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ecc0247c0 con 0x7f4ee4106540 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.949+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee419ae80 con 0x7f4ee4106540 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.950+0000 7f4ee8940700 1 -- 172.21.15.151:0/382298551 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f4ecc01ac00 con 0x7f4ee4106540 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.950+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee4194380 con 0x7f4ee4106540 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.950+0000 7f4ee8940700 1 --2- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4ed0038370 0x7f4ed003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.950+0000 7f4ee8940700 1 -- 172.21.15.151:0/382298551 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4ecc04c370 con 0x7f4ee4106540 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.950+0000 7f4ee27fc700 1 --2- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4ed0038370 0x7f4ed003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:24.771 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.950+0000 7f4ee27fc700 1 --2- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4ed0038370 0x7f4ed003a830 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4006fd0 tx=0x7f4ed4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:23.954+0000 7f4ee8940700 1 -- 172.21.15.151:0/382298551 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4ecc010380 con 0x7f4ee4106540 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.143+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4ee402d090 con 0x7f4ee4106540 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.147+0000 7f4ee8940700 1 -- 172.21.15.151:0/382298551 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4ecc018b40 con 0x7f4ee4106540 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4ed0038370 msgr2=0x7f4ed003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 --2- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4ed0038370 0x7f4ed003a830 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4006fd0 tx=0x7f4ed4006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 msgr2=0x7f4ee419a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 --2- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee419a2a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc011670 tx=0x7f4ecc011b50 comp rx=0 tx=0).stop 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 shutdown_connections 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 --2- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f4ed0038370 0x7f4ed003a830 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 --2- 172.21.15.151:0/382298551 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ee4106540 0x7f4ee419a2a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 >> 172.21.15.151:0/382298551 conn(0x7f4ee4101b00 msgr2=0x7f4ee4103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 shutdown_connections 2024-04-16T02:26:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:24.150+0000 7f4ee9942700 1 -- 172.21.15.151:0/382298551 wait complete. 2024-04-16T02:26:25.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.183+0000 7feaf5812700 1 Processor -- start 2024-04-16T02:26:25.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.183+0000 7feaf5812700 1 -- start start 2024-04-16T02:26:25.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.183+0000 7feaf5812700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf01093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:25.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.183+0000 7feaf5812700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0079e90 con 0x7feaf0106fe0 2024-04-16T02:26:25.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.184+0000 7feaeeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf01093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:25.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.184+0000 7feaeeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf01093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50958/0 (socket says 172.21.15.151:50958) 2024-04-16T02:26:25.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.184+0000 7feaeeffd700 1 -- 172.21.15.151:0/2867306599 learned_addr learned my addr 172.21.15.151:0/2867306599 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:25.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.184+0000 7feaeeffd700 1 -- 172.21.15.151:0/2867306599 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf0109910 con 0x7feaf0106fe0 2024-04-16T02:26:25.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.184+0000 7feaeeffd700 1 --2- 172.21.15.151:0/2867306599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf01093d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feae00098a0 tx=0x7feae0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7091657cf45b0f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:25.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.184+0000 7feaee7fc700 1 -- 172.21.15.151:0/2867306599 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae0004180 con 0x7feaf0106fe0 2024-04-16T02:26:25.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.184+0000 7feaee7fc700 1 -- 172.21.15.151:0/2867306599 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feae00042e0 con 0x7feaf0106fe0 2024-04-16T02:26:25.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.185+0000 7feaee7fc700 1 -- 172.21.15.151:0/2867306599 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae0011af0 con 0x7feaf0106fe0 2024-04-16T02:26:25.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.185+0000 7feaf5812700 1 -- 172.21.15.151:0/2867306599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 msgr2=0x7feaf01093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:25.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.185+0000 7feaf5812700 1 --2- 172.21.15.151:0/2867306599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf01093d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feae00098a0 tx=0x7feae0009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:25.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.185+0000 7feaf5812700 1 -- 172.21.15.151:0/2867306599 shutdown_connections 2024-04-16T02:26:25.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.185+0000 7feaf5812700 1 --2- 172.21.15.151:0/2867306599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf01093d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.185+0000 7feaf5812700 1 -- 172.21.15.151:0/2867306599 >> 172.21.15.151:0/2867306599 conn(0x7feaf01009f0 msgr2=0x7feaf0102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.185+0000 7feaf5812700 1 -- 172.21.15.151:0/2867306599 shutdown_connections 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.185+0000 7feaf5812700 1 -- 172.21.15.151:0/2867306599 wait complete. 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.186+0000 7feaf5812700 1 Processor -- start 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.186+0000 7feaf5812700 1 -- start start 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.186+0000 7feaf5812700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.186+0000 7feaf5812700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf019a6c0 con 0x7feaf0106fe0 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.186+0000 7feaeeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.186+0000 7feaeeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50974/0 (socket says 172.21.15.151:50974) 2024-04-16T02:26:25.949 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.186+0000 7feaeeffd700 1 -- 172.21.15.151:0/3656032668 learned_addr learned my addr 172.21.15.151:0/3656032668 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.186+0000 7feaeeffd700 1 -- 172.21.15.151:0/3656032668 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae0009550 con 0x7feaf0106fe0 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.186+0000 7feaeeffd700 1 --2- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf019a180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feae0006ba0 tx=0x7feae0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.187+0000 7feaf4810700 1 -- 172.21.15.151:0/3656032668 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae0003b00 con 0x7feaf0106fe0 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.187+0000 7feaf4810700 1 -- 172.21.15.151:0/3656032668 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feae001a8e0 con 0x7feaf0106fe0 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.187+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf019a8c0 con 0x7feaf0106fe0 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.187+0000 7feaf4810700 1 -- 172.21.15.151:0/3656032668 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae00199c0 con 0x7feaf0106fe0 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.187+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf019ad60 con 0x7feaf0106fe0 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.187+0000 7feaf4810700 1 -- 172.21.15.151:0/3656032668 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7feae0019c80 con 0x7feaf0106fe0 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.187+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf010a700 con 0x7feaf0106fe0 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.188+0000 7feaf4810700 1 --2- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7feadc0382d0 0x7feadc03a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:25.950 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.188+0000 7feaf4810700 1 -- 172.21.15.151:0/3656032668 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feae004c610 con 0x7feaf0106fe0 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.188+0000 7feae7fff700 1 --2- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7feadc0382d0 0x7feadc03a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.188+0000 7feae7fff700 1 --2- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7feadc0382d0 0x7feadc03a790 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fead8006fd0 tx=0x7fead8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.192+0000 7feaf4810700 1 -- 172.21.15.151:0/3656032668 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7feae0018530 con 0x7feaf0106fe0 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.382+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7feaf019dba0 con 0x7feaf0106fe0 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.385+0000 7feaf4810700 1 -- 172.21.15.151:0/3656032668 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7feae0018350 con 0x7feaf0106fe0 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7feadc0382d0 msgr2=0x7feadc03a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 --2- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7feadc0382d0 0x7feadc03a790 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fead8006fd0 tx=0x7fead8006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 msgr2=0x7feaf019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 --2- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf019a180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feae0006ba0 tx=0x7feae0011b50 comp rx=0 tx=0).stop 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 shutdown_connections 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 --2- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7feadc0382d0 0x7feadc03a790 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 --2- 172.21.15.151:0/3656032668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feaf0106fe0 0x7feaf019a180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 >> 172.21.15.151:0/3656032668 conn(0x7feaf01009f0 msgr2=0x7feaf0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 shutdown_connections 2024-04-16T02:26:25.952 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:25.388+0000 7feaf5812700 1 -- 172.21.15.151:0/3656032668 wait complete. 2024-04-16T02:26:25.952 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling the dashboard module... 2024-04-16T02:26:26.629 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:26 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3656032668' entity='client.admin' 2024-04-16T02:26:26.629 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:26 smithi151 ceph-mon[29003]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:26:27.716 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:27 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1445765393' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-16T02:26:28.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.377+0000 7fdbf0737700 1 Processor -- start 2024-04-16T02:26:28.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.377+0000 7fdbf0737700 1 -- start start 2024-04-16T02:26:28.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.377+0000 7fdbf0737700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe8104c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:28.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.377+0000 7fdbf0737700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe8105250 con 0x7fdbe8104860 2024-04-16T02:26:28.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.378+0000 7fdbee4d3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe8104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:28.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.378+0000 7fdbee4d3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe8104c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50988/0 (socket says 172.21.15.151:50988) 2024-04-16T02:26:28.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.378+0000 7fdbee4d3700 1 -- 172.21.15.151:0/4221538064 learned_addr learned my addr 172.21.15.151:0/4221538064 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.378+0000 7fdbee4d3700 1 -- 172.21.15.151:0/4221538064 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe8105390 con 0x7fdbe8104860 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.378+0000 7fdbee4d3700 1 --2- 172.21.15.151:0/4221538064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe8104c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009e80 tx=0x7fdbe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348aa46b2237011f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.378+0000 7fdbed4d1700 1 -- 172.21.15.151:0/4221538064 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbe400e070 con 0x7fdbe8104860 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.378+0000 7fdbed4d1700 1 -- 172.21.15.151:0/4221538064 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdbe400b8c0 con 0x7fdbe8104860 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.378+0000 7fdbed4d1700 1 -- 172.21.15.151:0/4221538064 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbe40044d0 con 0x7fdbe8104860 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.379+0000 7fdbf0737700 1 -- 172.21.15.151:0/4221538064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 msgr2=0x7fdbe8104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.379+0000 7fdbf0737700 1 --2- 172.21.15.151:0/4221538064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe8104c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009e80 tx=0x7fdbe400b330 comp rx=0 tx=0).stop 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.379+0000 7fdbf0737700 1 -- 172.21.15.151:0/4221538064 shutdown_connections 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.379+0000 7fdbf0737700 1 --2- 172.21.15.151:0/4221538064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe8104c80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.379+0000 7fdbf0737700 1 -- 172.21.15.151:0/4221538064 >> 172.21.15.151:0/4221538064 conn(0x7fdbe8100190 msgr2=0x7fdbe81025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.379+0000 7fdbf0737700 1 -- 172.21.15.151:0/4221538064 shutdown_connections 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.379+0000 7fdbf0737700 1 -- 172.21.15.151:0/4221538064 wait complete. 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbf0737700 1 Processor -- start 2024-04-16T02:26:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbf0737700 1 -- start start 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbf0737700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbf0737700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe819a730 con 0x7fdbe8104860 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbee4d3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbee4d3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50998/0 (socket says 172.21.15.151:50998) 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbee4d3700 1 -- 172.21.15.151:0/1445765393 learned_addr learned my addr 172.21.15.151:0/1445765393 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbee4d3700 1 -- 172.21.15.151:0/1445765393 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe4009b30 con 0x7fdbe8104860 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbee4d3700 1 --2- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe819a1f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40072a0 tx=0x7fdbe40049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.380+0000 7fdbdf7fe700 1 -- 172.21.15.151:0/1445765393 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbe400e040 con 0x7fdbe8104860 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.381+0000 7fdbdf7fe700 1 -- 172.21.15.151:0/1445765393 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdbe400bd80 con 0x7fdbe8104860 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.381+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe819a930 con 0x7fdbe8104860 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.381+0000 7fdbdf7fe700 1 -- 172.21.15.151:0/1445765393 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbe40126c0 con 0x7fdbe8104860 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.381+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe819ae30 con 0x7fdbe8104860 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.381+0000 7fdbdf7fe700 1 -- 172.21.15.151:0/1445765393 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fdbe4019040 con 0x7fdbe8104860 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.381+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe80623c0 con 0x7fdbe8104860 2024-04-16T02:26:28.069 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.382+0000 7fdbdf7fe700 1 --2- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fdbd4038330 0x7fdbd403a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.382+0000 7fdbdf7fe700 1 -- 172.21.15.151:0/1445765393 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdbe404be40 con 0x7fdbe8104860 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.382+0000 7fdbedcd2700 1 --2- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fdbd4038330 0x7fdbd403a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.382+0000 7fdbedcd2700 1 --2- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fdbd4038330 0x7fdbd403a7f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8006fd0 tx=0x7fdbd8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.386+0000 7fdbdf7fe700 1 -- 172.21.15.151:0/1445765393 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdbe402cb40 con 0x7fdbe8104860 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:26.629+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fdbe8108500 con 0x7fdbe8104860 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.391+0000 7fdbdf7fe700 1 -- 172.21.15.151:0/1445765393 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fdbe40244a0 con 0x7fdbe8104860 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.393+0000 7fdbdf7fe700 1 -- 172.21.15.151:0/1445765393 <== mon.0 v2:172.21.15.151:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fdbe4050740 con 0x7fdbe8104860 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fdbd4038330 msgr2=0x7fdbd403a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 --2- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fdbd4038330 0x7fdbd403a7f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8006fd0 tx=0x7fdbd8006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 msgr2=0x7fdbe819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 --2- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe819a1f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40072a0 tx=0x7fdbe40049f0 comp rx=0 tx=0).stop 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 shutdown_connections 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 --2- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7fdbd4038330 0x7fdbd403a7f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:28.070 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 --2- 172.21.15.151:0/1445765393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe8104860 0x7fdbe819a1f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:28.071 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 >> 172.21.15.151:0/1445765393 conn(0x7fdbe8100190 msgr2=0x7fdbe8100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:28.071 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 shutdown_connections 2024-04-16T02:26:28.071 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:27.395+0000 7fdbf0737700 1 -- 172.21.15.151:0/1445765393 wait complete. 2024-04-16T02:26:28.693 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:28 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1445765393' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-16T02:26:28.693 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:28 smithi151 ceph-mon[29003]: mgrmap e9: smithi151.uqqame(active, since 25s) 2024-04-16T02:26:29.314 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "smithi151.uqqame", 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.487+0000 7ff4ff702700 1 Processor -- start 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.487+0000 7ff4ff702700 1 -- start start 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.487+0000 7ff4ff702700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f8109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.487+0000 7ff4ff702700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f8079e50 con 0x7ff4f8107020 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.488+0000 7ff4fd49e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f8109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.488+0000 7ff4fd49e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f8109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57860/0 (socket says 172.21.15.151:57860) 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.488+0000 7ff4fd49e700 1 -- 172.21.15.151:0/1813485308 learned_addr learned my addr 172.21.15.151:0/1813485308 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.488+0000 7ff4fd49e700 1 -- 172.21.15.151:0/1813485308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8109950 con 0x7ff4f8107020 2024-04-16T02:26:29.315 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.488+0000 7ff4fd49e700 1 --2- 172.21.15.151:0/1813485308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f8109410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40098a0 tx=0x7ff4f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6108ed521e073b7b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.488+0000 7ff4effff700 1 -- 172.21.15.151:0/1813485308 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f4004180 con 0x7ff4f8107020 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.488+0000 7ff4effff700 1 -- 172.21.15.151:0/1813485308 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff4f40042e0 con 0x7ff4f8107020 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.488+0000 7ff4effff700 1 -- 172.21.15.151:0/1813485308 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f4011af0 con 0x7ff4f8107020 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.489+0000 7ff4ff702700 1 -- 172.21.15.151:0/1813485308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 msgr2=0x7ff4f8109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.489+0000 7ff4ff702700 1 --2- 172.21.15.151:0/1813485308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f8109410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40098a0 tx=0x7ff4f4009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.489+0000 7ff4ff702700 1 -- 172.21.15.151:0/1813485308 shutdown_connections 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.489+0000 7ff4ff702700 1 --2- 172.21.15.151:0/1813485308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f8109410 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.489+0000 7ff4ff702700 1 -- 172.21.15.151:0/1813485308 >> 172.21.15.151:0/1813485308 conn(0x7ff4f81009f0 msgr2=0x7ff4f8102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.489+0000 7ff4ff702700 1 -- 172.21.15.151:0/1813485308 shutdown_connections 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.489+0000 7ff4ff702700 1 -- 172.21.15.151:0/1813485308 wait complete. 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.490+0000 7ff4ff702700 1 Processor -- start 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.490+0000 7ff4ff702700 1 -- start start 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.490+0000 7ff4ff702700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:29.316 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.490+0000 7ff4ff702700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f819a6f0 con 0x7ff4f8107020 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.490+0000 7ff4fd49e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.490+0000 7ff4fd49e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57870/0 (socket says 172.21.15.151:57870) 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.490+0000 7ff4fd49e700 1 -- 172.21.15.151:0/243136470 learned_addr learned my addr 172.21.15.151:0/243136470 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.490+0000 7ff4fd49e700 1 -- 172.21.15.151:0/243136470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f4009550 con 0x7ff4f8107020 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.490+0000 7ff4fd49e700 1 --2- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f819a1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4011ad0 tx=0x7ff4f4011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.491+0000 7ff4ee7fc700 1 -- 172.21.15.151:0/243136470 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f4003ab0 con 0x7ff4f8107020 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.491+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f819a8f0 con 0x7ff4f8107020 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.491+0000 7ff4ee7fc700 1 -- 172.21.15.151:0/243136470 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff4f4003c10 con 0x7ff4f8107020 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.491+0000 7ff4ee7fc700 1 -- 172.21.15.151:0/243136470 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f40198e0 con 0x7ff4f8107020 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.491+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f819ad90 con 0x7ff4f8107020 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.492+0000 7ff4ee7fc700 1 -- 172.21.15.151:0/243136470 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7ff4f4019ba0 con 0x7ff4f8107020 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.492+0000 7ff4ee7fc700 1 --2- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff4e4038370 0x7ff4e403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.492+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f810a6e0 con 0x7ff4f8107020 2024-04-16T02:26:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.492+0000 7ff4fcc9d700 1 -- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff4e4038370 msgr2=0x7ff4e403a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/2400549256 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.492+0000 7ff4ee7fc700 1 -- 172.21.15.151:0/243136470 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff4f404c190 con 0x7ff4f8107020 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.492+0000 7ff4fcc9d700 1 --2- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff4e4038370 0x7ff4e403a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.497+0000 7ff4ee7fc700 1 -- 172.21.15.151:0/243136470 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff4f4010d50 con 0x7ff4f8107020 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.693+0000 7ff4fcc9d700 1 -- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff4e4038370 msgr2=0x7ff4e403a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/2400549256 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.693+0000 7ff4fcc9d700 1 --2- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff4e4038370 0x7ff4e403a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.743+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff4f819db70 con 0x7ff4f8107020 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.743+0000 7ff4ee7fc700 1 -- 172.21.15.151:0/243136470 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7ff4f4010b70 con 0x7ff4f8107020 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff4e4038370 msgr2=0x7ff4e403a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 --2- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff4e4038370 0x7ff4e403a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 msgr2=0x7ff4f819a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 --2- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f819a1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4011ad0 tx=0x7ff4f4011bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 shutdown_connections 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 --2- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7ff4e4038370 0x7ff4e403a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:29.318 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 --2- 172.21.15.151:0/243136470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f8107020 0x7ff4f819a1b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:29.319 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 >> 172.21.15.151:0/243136470 conn(0x7ff4f81009f0 msgr2=0x7ff4f8101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:29.319 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 shutdown_connections 2024-04-16T02:26:29.319 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:28.746+0000 7ff4ff702700 1 -- 172.21.15.151:0/243136470 wait complete. 2024-04-16T02:26:29.319 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-04-16T02:26:29.319 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 9... 2024-04-16T02:26:29.560 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:29 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/243136470' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-16T02:26:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: Active manager daemon smithi151.uqqame restarted 2024-04-16T02:26:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: Activating manager daemon smithi151.uqqame 2024-04-16T02:26:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: osdmap e3: 0 total, 0 up, 0 in 2024-04-16T02:26:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: mgrmap e10: smithi151.uqqame(active, starting, since 0.00390925s) 2024-04-16T02:26:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:26:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:26:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:26:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:26:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:26:34.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: Manager daemon smithi151.uqqame is now available 2024-04-16T02:26:34.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:26:34.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:26:34.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:34 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:26:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:35 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/trash_purge_schedule"}]: dispatch 2024-04-16T02:26:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:35 smithi151 ceph-mon[29003]: mgrmap e11: smithi151.uqqame(active, since 1.00573s) 2024-04-16T02:26:35.942 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.786+0000 7f59822b1700 1 Processor -- start 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.786+0000 7f59822b1700 1 -- start start 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f59822b1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f59822b1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c105e60 con 0x7f597c105470 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f597bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f597bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57886/0 (socket says 172.21.15.151:57886) 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f597bfff700 1 -- 172.21.15.151:0/3267268455 learned_addr learned my addr 172.21.15.151:0/3267268455 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:35.943 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f597bfff700 1 -- 172.21.15.151:0/3267268455 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c106670 con 0x7f597c105470 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f597bfff700 1 --2- 172.21.15.151:0/3267268455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c105890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f596c0098a0 tx=0x7f596c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0cb9c5bcc1dcfd server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f597affd700 1 -- 172.21.15.151:0/3267268455 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f596c004180 con 0x7f597c105470 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f597affd700 1 -- 172.21.15.151:0/3267268455 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f596c0042e0 con 0x7f597c105470 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.787+0000 7f597affd700 1 -- 172.21.15.151:0/3267268455 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f596c011af0 con 0x7f597c105470 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.788+0000 7f59822b1700 1 -- 172.21.15.151:0/3267268455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 msgr2=0x7f597c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.788+0000 7f59822b1700 1 --2- 172.21.15.151:0/3267268455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c105890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f596c0098a0 tx=0x7f596c009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.788+0000 7f59822b1700 1 -- 172.21.15.151:0/3267268455 shutdown_connections 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.788+0000 7f59822b1700 1 --2- 172.21.15.151:0/3267268455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c105890 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.788+0000 7f59822b1700 1 -- 172.21.15.151:0/3267268455 >> 172.21.15.151:0/3267268455 conn(0x7f597c1009f0 msgr2=0x7f597c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.788+0000 7f59822b1700 1 -- 172.21.15.151:0/3267268455 shutdown_connections 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.788+0000 7f59822b1700 1 -- 172.21.15.151:0/3267268455 wait complete. 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.789+0000 7f59822b1700 1 Processor -- start 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.789+0000 7f59822b1700 1 -- start start 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f59822b1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c079b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:35.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f59822b1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c19cf00 con 0x7f597c105470 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f597bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c079b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f597bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c079b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57900/0 (socket says 172.21.15.151:57900) 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f597bfff700 1 -- 172.21.15.151:0/652542089 learned_addr learned my addr 172.21.15.151:0/652542089 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f597bfff700 1 -- 172.21.15.151:0/652542089 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c009550 con 0x7f597c105470 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f597bfff700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c079b30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f596c000c00 tx=0x7f596c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f596c003a70 con 0x7f597c105470 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f596c003bd0 con 0x7f597c105470 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f59822b1700 1 -- 172.21.15.151:0/652542089 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597c19d040 con 0x7f597c105470 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f596c0238c0 con 0x7f597c105470 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.790+0000 7f59822b1700 1 -- 172.21.15.151:0/652542089 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597c19d4e0 con 0x7f597c105470 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.791+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f596c023b80 con 0x7f597c105470 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.792+0000 7f59797fa700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 0x7f596803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.792+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 --> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f596803af40 con 0x7f5968038370 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.792+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f596c04c210 con 0x7f597c105470 2024-04-16T02:26:35.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.792+0000 7f597b7fe700 1 -- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 msgr2=0x7f596803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/2400549256 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.792+0000 7f597b7fe700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 0x7f596803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.993+0000 7f597b7fe700 1 -- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 msgr2=0x7f596803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/2400549256 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:29.993+0000 7f597b7fe700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 0x7f596803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:30.394+0000 7f597b7fe700 1 -- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 msgr2=0x7f596803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/2400549256 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:30.394+0000 7f597b7fe700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 0x7f596803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:31.194+0000 7f597b7fe700 1 -- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 msgr2=0x7f596803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/2400549256 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:31.194+0000 7f597b7fe700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 0x7f596803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:32.795+0000 7f597b7fe700 1 -- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 msgr2=0x7f596803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/2400549256 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:32.795+0000 7f597b7fe700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 0x7f596803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:34.118+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mgrmap(e 10) v1 ==== 45063+0+0 (secure 0 0 0) 0x7f596c0193c0 con 0x7f597c105470 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:34.118+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 msgr2=0x7f596803a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:34.118+0000 7f59797fa700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 0x7f596803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.119+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f596c019740 con 0x7f597c105470 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.119+0000 7f59797fa700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f596803bde0 0x7f596803e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.119+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 --> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f596803af40 con 0x7f596803bde0 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.119+0000 7f597b7fe700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f596803bde0 0x7f596803e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.120+0000 7f597b7fe700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f596803bde0 0x7f596803e1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5964003a10 tx=0x7f59640092b0 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.120+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 <== mgr.14164 v2:172.21.15.151:6800/3544869641 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f596803af40 con 0x7f596803bde0 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.126+0000 7f59822b1700 1 -- 172.21.15.151:0/652542089 --> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f597c0796d0 con 0x7f596803bde0 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59797fa700 1 -- 172.21.15.151:0/652542089 <== mgr.14164 v2:172.21.15.151:6800/3544869641 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f597c0796d0 con 0x7f596803bde0 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 -- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f596803bde0 msgr2=0x7f596803e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f596803bde0 0x7f596803e1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5964003a10 tx=0x7f59640092b0 comp rx=0 tx=0).stop 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 -- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 msgr2=0x7f597c079b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c079b30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f596c000c00 tx=0x7f596c011b50 comp rx=0 tx=0).stop 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 -- 172.21.15.151:0/652542089 shutdown_connections 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f596803bde0 0x7f596803e1d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f597c105470 0x7f597c079b30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:35.947 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 --2- 172.21.15.151:0/652542089 >> [v2:172.21.15.151:6800/2400549256,v1:172.21.15.151:6801/2400549256] conn(0x7f5968038370 0x7f596803a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:35.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 -- 172.21.15.151:0/652542089 >> 172.21.15.151:0/652542089 conn(0x7f597c1009f0 msgr2=0x7f597c101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:35.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 -- 172.21.15.151:0/652542089 shutdown_connections 2024-04-16T02:26:35.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:35.127+0000 7f59822b1700 1 -- 172.21.15.151:0/652542089 wait complete. 2024-04-16T02:26:35.948 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 9 is available 2024-04-16T02:26:35.948 INFO:teuthology.orchestra.run.smithi151.stdout:Generating a dashboard self-signed certificate... 2024-04-16T02:26:36.320 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:36 smithi151 ceph-mon[29003]: [16/Apr/2024:02:26:34] ENGINE Bus STARTING 2024-04-16T02:26:36.320 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:36 smithi151 ceph-mon[29003]: [16/Apr/2024:02:26:34] ENGINE Serving on http://172.21.15.151:8765 2024-04-16T02:26:36.320 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:36 smithi151 ceph-mon[29003]: [16/Apr/2024:02:26:35] ENGINE Serving on https://172.21.15.151:7150 2024-04-16T02:26:36.320 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:36 smithi151 ceph-mon[29003]: [16/Apr/2024:02:26:35] ENGINE Bus STARTED 2024-04-16T02:26:36.320 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:36 smithi151 ceph-mon[29003]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-16T02:26:36.321 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:36 smithi151 ceph-mon[29003]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-16T02:26:36.321 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:36 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:37.255 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.422+0000 7f5aeebd0700 1 Processor -- start 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.422+0000 7f5aeebd0700 1 -- start start 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.422+0000 7f5aeebd0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae807a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.422+0000 7f5aeebd0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae807a8b0 con 0x7f5ae807be80 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.422+0000 7f5aec96c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae807a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.422+0000 7f5aec96c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae807a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57976/0 (socket says 172.21.15.151:57976) 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.422+0000 7f5aec96c700 1 -- 172.21.15.151:0/1806933988 learned_addr learned my addr 172.21.15.151:0/1806933988 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.422+0000 7f5aec96c700 1 -- 172.21.15.151:0/1806933988 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae807a9f0 con 0x7f5ae807be80 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.422+0000 7f5aec96c700 1 --2- 172.21.15.151:0/1806933988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae807a2e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5adc0098a0 tx=0x7f5adc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1136fc2a1e308b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5ae77fe700 1 -- 172.21.15.151:0/1806933988 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc004180 con 0x7f5ae807be80 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5ae77fe700 1 -- 172.21.15.151:0/1806933988 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5adc0042e0 con 0x7f5ae807be80 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5ae77fe700 1 -- 172.21.15.151:0/1806933988 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc011af0 con 0x7f5ae807be80 2024-04-16T02:26:37.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5aeebd0700 1 -- 172.21.15.151:0/1806933988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 msgr2=0x7f5ae807a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5aeebd0700 1 --2- 172.21.15.151:0/1806933988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae807a2e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5adc0098a0 tx=0x7f5adc009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5aeebd0700 1 -- 172.21.15.151:0/1806933988 shutdown_connections 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5aeebd0700 1 --2- 172.21.15.151:0/1806933988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae807a2e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5aeebd0700 1 -- 172.21.15.151:0/1806933988 >> 172.21.15.151:0/1806933988 conn(0x7f5ae8101b40 msgr2=0x7f5ae8103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5aeebd0700 1 -- 172.21.15.151:0/1806933988 shutdown_connections 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.423+0000 7f5aeebd0700 1 -- 172.21.15.151:0/1806933988 wait complete. 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.424+0000 7f5aeebd0700 1 Processor -- start 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.424+0000 7f5aeebd0700 1 -- start start 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.424+0000 7f5aeebd0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.424+0000 7f5aeebd0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae819c9a0 con 0x7f5ae807be80 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.424+0000 7f5aec96c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:37.257 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.424+0000 7f5aec96c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae819c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57988/0 (socket says 172.21.15.151:57988) 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.424+0000 7f5aec96c700 1 -- 172.21.15.151:0/588031065 learned_addr learned my addr 172.21.15.151:0/588031065 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.424+0000 7f5aec96c700 1 -- 172.21.15.151:0/588031065 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc009550 con 0x7f5ae807be80 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.424+0000 7f5aec96c700 1 --2- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae819c460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5adc011670 tx=0x7f5adc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.425+0000 7f5ae5ffb700 1 -- 172.21.15.151:0/588031065 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc0039c0 con 0x7f5ae807be80 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.425+0000 7f5ae5ffb700 1 -- 172.21.15.151:0/588031065 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5adc003b20 con 0x7f5ae807be80 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.425+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae819cba0 con 0x7f5ae807be80 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.425+0000 7f5ae5ffb700 1 -- 172.21.15.151:0/588031065 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc0247f0 con 0x7f5ae807be80 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.425+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae819d040 con 0x7f5ae807be80 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.426+0000 7f5ae5ffb700 1 -- 172.21.15.151:0/588031065 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f5adc01a430 con 0x7f5ae807be80 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.426+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae810c920 con 0x7f5ae807be80 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.426+0000 7f5ae5ffb700 1 --2- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f5ad8038260 0x7f5ad803a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.426+0000 7f5ae5ffb700 1 -- 172.21.15.151:0/588031065 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5adc04c3f0 con 0x7f5ae807be80 2024-04-16T02:26:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.426+0000 7f5ae7fff700 1 --2- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f5ad8038260 0x7f5ad803a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.426+0000 7f5ae7fff700 1 --2- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f5ad8038260 0x7f5ad803a720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4006fd0 tx=0x7f5ad4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.431+0000 7f5ae5ffb700 1 -- 172.21.15.151:0/588031065 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5adc018530 con 0x7f5ae807be80 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.572+0000 7f5ae5ffb700 1 -- 172.21.15.151:0/588031065 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f5adc02d3f0 con 0x7f5ae807be80 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.638+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 --> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ae8103e00 con 0x7f5ad8038260 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.681+0000 7f5ae5ffb700 1 -- 172.21.15.151:0/588031065 <== mgr.14164 v2:172.21.15.151:6800/3544869641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f5ae8103e00 con 0x7f5ad8038260 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f5ad8038260 msgr2=0x7f5ad803a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 --2- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f5ad8038260 0x7f5ad803a720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4006fd0 tx=0x7f5ad4006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 msgr2=0x7f5ae819c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 --2- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae819c460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5adc011670 tx=0x7f5adc011aa0 comp rx=0 tx=0).stop 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 shutdown_connections 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 --2- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f5ad8038260 0x7f5ad803a720 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 --2- 172.21.15.151:0/588031065 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ae807be80 0x7f5ae819c460 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 >> 172.21.15.151:0/588031065 conn(0x7f5ae8101b40 msgr2=0x7f5ae81031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 shutdown_connections 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:36.684+0000 7f5aeebd0700 1 -- 172.21.15.151:0/588031065 wait complete. 2024-04-16T02:26:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial admin user... 2024-04-16T02:26:37.676 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:37 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:37.676 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:37 smithi151 ceph-mon[29003]: mgrmap e12: smithi151.uqqame(active, since 2s) 2024-04-16T02:26:37.676 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:37 smithi151 ceph-mon[29003]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:37.676 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:37 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:37.676 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:37 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:38.833 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$.m.B27LTfVdMwUN8gqoF9eiI8tvziokuFKDAZOKgW1A02txVpE74K", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713234398, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-16T02:26:38.833 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.704+0000 7fcdc95f8700 1 Processor -- start 2024-04-16T02:26:38.833 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.704+0000 7fcdc95f8700 1 -- start start 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc95f8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc4106960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc95f8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4106f30 con 0x7fcdc4106540 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc4106960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc4106960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41920/0 (socket says 172.21.15.151:41920) 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc2ffd700 1 -- 172.21.15.151:0/1078644100 learned_addr learned my addr 172.21.15.151:0/1078644100 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc2ffd700 1 -- 172.21.15.151:0/1078644100 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc4107740 con 0x7fcdc4106540 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc2ffd700 1 --2- 172.21.15.151:0/1078644100 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc4106960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40098a0 tx=0x7fcdb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d347352a02cd4559 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc27fc700 1 -- 172.21.15.151:0/1078644100 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdb4004180 con 0x7fcdc4106540 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc27fc700 1 -- 172.21.15.151:0/1078644100 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcdb40042e0 con 0x7fcdc4106540 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.705+0000 7fcdc27fc700 1 -- 172.21.15.151:0/1078644100 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdb4011af0 con 0x7fcdc4106540 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.706+0000 7fcdc95f8700 1 -- 172.21.15.151:0/1078644100 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 msgr2=0x7fcdc4106960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:38.834 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.706+0000 7fcdc95f8700 1 --2- 172.21.15.151:0/1078644100 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc4106960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40098a0 tx=0x7fcdb4009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.706+0000 7fcdc95f8700 1 -- 172.21.15.151:0/1078644100 shutdown_connections 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.706+0000 7fcdc95f8700 1 --2- 172.21.15.151:0/1078644100 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc4106960 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.706+0000 7fcdc95f8700 1 -- 172.21.15.151:0/1078644100 >> 172.21.15.151:0/1078644100 conn(0x7fcdc4101b00 msgr2=0x7fcdc4103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.706+0000 7fcdc95f8700 1 -- 172.21.15.151:0/1078644100 shutdown_connections 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.706+0000 7fcdc95f8700 1 -- 172.21.15.151:0/1078644100 wait complete. 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.707+0000 7fcdc95f8700 1 Processor -- start 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.707+0000 7fcdc95f8700 1 -- start start 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.707+0000 7fcdc95f8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.707+0000 7fcdc95f8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc419a7e0 con 0x7fcdc4106540 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.707+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.707+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41922/0 (socket says 172.21.15.151:41922) 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.707+0000 7fcdc2ffd700 1 -- 172.21.15.151:0/239084031 learned_addr learned my addr 172.21.15.151:0/239084031 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.707+0000 7fcdc2ffd700 1 -- 172.21.15.151:0/239084031 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb4009550 con 0x7fcdc4106540 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.707+0000 7fcdc2ffd700 1 --2- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc419a2a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4011670 tx=0x7fcdb4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.708+0000 7fcdc0ff9700 1 -- 172.21.15.151:0/239084031 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdb40039c0 con 0x7fcdc4106540 2024-04-16T02:26:38.835 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.708+0000 7fcdc0ff9700 1 -- 172.21.15.151:0/239084031 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcdb4003b20 con 0x7fcdc4106540 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.708+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc419a9e0 con 0x7fcdc4106540 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.708+0000 7fcdc0ff9700 1 -- 172.21.15.151:0/239084031 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdb40247f0 con 0x7fcdc4106540 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.708+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc419ae80 con 0x7fcdc4106540 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.709+0000 7fcdc0ff9700 1 -- 172.21.15.151:0/239084031 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fcdb4024ab0 con 0x7fcdc4106540 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.709+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc4194380 con 0x7fcdc4106540 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.709+0000 7fcdc0ff9700 1 --2- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcdb0038330 0x7fcdb003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.709+0000 7fcdc0ff9700 1 -- 172.21.15.151:0/239084031 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcdb404c6c0 con 0x7fcdc4106540 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.709+0000 7fcdba7fc700 1 --2- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcdb0038330 0x7fcdb003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.709+0000 7fcdba7fc700 1 --2- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcdb0038330 0x7fcdb003a7f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcdac006fd0 tx=0x7fcdac006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.713+0000 7fcdc0ff9700 1 -- 172.21.15.151:0/239084031 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcdb4022310 con 0x7fcdc4106540 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:37.944+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 --> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fcdc402d090 con 0x7fcdb0038330 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.169+0000 7fcdc0ff9700 1 -- 172.21.15.151:0/239084031 <== mgr.14164 v2:172.21.15.151:6800/3544869641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fcdc402d090 con 0x7fcdb0038330 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcdb0038330 msgr2=0x7fcdb003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:38.836 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 --2- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcdb0038330 0x7fcdb003a7f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcdac006fd0 tx=0x7fcdac006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 msgr2=0x7fcdc419a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 --2- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc419a2a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4011670 tx=0x7fcdb4011aa0 comp rx=0 tx=0).stop 2024-04-16T02:26:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 shutdown_connections 2024-04-16T02:26:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 --2- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcdb0038330 0x7fcdb003a7f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 --2- 172.21.15.151:0/239084031 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc4106540 0x7fcdc419a2a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 >> 172.21.15.151:0/239084031 conn(0x7fcdc4101b00 msgr2=0x7fcdc4103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 shutdown_connections 2024-04-16T02:26:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:38.172+0000 7fcdc95f8700 1 -- 172.21.15.151:0/239084031 wait complete. 2024-04-16T02:26:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:Fetching dashboard port number... 2024-04-16T02:26:39.398 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:39 smithi151 ceph-mon[29003]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:39.398 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:39 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:40.218 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:40 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/4113726881' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-16T02:26:40.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 8443 2024-04-16T02:26:40.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.282+0000 7f55f5052700 1 Processor -- start 2024-04-16T02:26:40.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.282+0000 7f55f5052700 1 -- start start 2024-04-16T02:26:40.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.282+0000 7f55f5052700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:40.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.282+0000 7f55f5052700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0105e60 con 0x7f55f0105470 2024-04-16T02:26:40.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.282+0000 7f55eed9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:40.226 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.282+0000 7f55eed9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41930/0 (socket says 172.21.15.151:41930) 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.282+0000 7f55eed9d700 1 -- 172.21.15.151:0/237885915 learned_addr learned my addr 172.21.15.151:0/237885915 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.283+0000 7f55eed9d700 1 -- 172.21.15.151:0/237885915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f0106670 con 0x7f55f0105470 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.283+0000 7f55eed9d700 1 --2- 172.21.15.151:0/237885915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0105890 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f55e00098a0 tx=0x7f55e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307d41fb1cf93897 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.283+0000 7f55edd9b700 1 -- 172.21.15.151:0/237885915 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e0004180 con 0x7f55f0105470 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.283+0000 7f55edd9b700 1 -- 172.21.15.151:0/237885915 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f55e00042e0 con 0x7f55f0105470 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.283+0000 7f55edd9b700 1 -- 172.21.15.151:0/237885915 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e0011af0 con 0x7f55f0105470 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.283+0000 7f55f5052700 1 -- 172.21.15.151:0/237885915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 msgr2=0x7f55f0105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.283+0000 7f55f5052700 1 --2- 172.21.15.151:0/237885915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0105890 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f55e00098a0 tx=0x7f55e0009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.284+0000 7f55f5052700 1 -- 172.21.15.151:0/237885915 shutdown_connections 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.284+0000 7f55f5052700 1 --2- 172.21.15.151:0/237885915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0105890 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.284+0000 7f55f5052700 1 -- 172.21.15.151:0/237885915 >> 172.21.15.151:0/237885915 conn(0x7f55f01009f0 msgr2=0x7f55f0102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.284+0000 7f55f5052700 1 -- 172.21.15.151:0/237885915 shutdown_connections 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.284+0000 7f55f5052700 1 -- 172.21.15.151:0/237885915 wait complete. 2024-04-16T02:26:40.227 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.284+0000 7f55f5052700 1 Processor -- start 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.284+0000 7f55f5052700 1 -- start start 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55f5052700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0074320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55f5052700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f019cee0 con 0x7f55f0105470 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55eed9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0074320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55eed9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0074320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41940/0 (socket says 172.21.15.151:41940) 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55eed9d700 1 -- 172.21.15.151:0/4113726881 learned_addr learned my addr 172.21.15.151:0/4113726881 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55eed9d700 1 -- 172.21.15.151:0/4113726881 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0009550 con 0x7f55f0105470 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55eed9d700 1 --2- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0074320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f55e0006ba0 tx=0x7f55e0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55d7fff700 1 -- 172.21.15.151:0/4113726881 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e0003b00 con 0x7f55f0105470 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55d7fff700 1 -- 172.21.15.151:0/4113726881 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f55e001a8e0 con 0x7f55f0105470 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55d7fff700 1 -- 172.21.15.151:0/4113726881 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e0019990 con 0x7f55f0105470 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.285+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f019d020 con 0x7f55f0105470 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.286+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f019d4c0 con 0x7f55f0105470 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.286+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f00623c0 con 0x7f55f0105470 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.286+0000 7f55d7fff700 1 -- 172.21.15.151:0/4113726881 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f55e0019bc0 con 0x7f55f0105470 2024-04-16T02:26:40.228 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.287+0000 7f55d7fff700 1 --2- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f55d8038320 0x7f55d803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.287+0000 7f55d7fff700 1 -- 172.21.15.151:0/4113726881 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f55e004c720 con 0x7f55f0105470 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.287+0000 7f55ee59c700 1 --2- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f55d8038320 0x7f55d803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.287+0000 7f55ee59c700 1 --2- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f55d8038320 0x7f55d803a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f55e4006fd0 tx=0x7f55e4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.291+0000 7f55d7fff700 1 -- 172.21.15.151:0/4113726881 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f55e002d440 con 0x7f55f0105470 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.484+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f55f01098a0 con 0x7f55f0105470 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.484+0000 7f55d7fff700 1 -- 172.21.15.151:0/4113726881 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f55e00172b0 con 0x7f55f0105470 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f55d8038320 msgr2=0x7f55d803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 --2- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f55d8038320 0x7f55d803a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f55e4006fd0 tx=0x7f55e4006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 msgr2=0x7f55f0074320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 --2- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0074320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f55e0006ba0 tx=0x7f55e0011b50 comp rx=0 tx=0).stop 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 shutdown_connections 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 --2- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f55d8038320 0x7f55d803a7e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 --2- 172.21.15.151:0/4113726881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55f0105470 0x7f55f0074320 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 >> 172.21.15.151:0/4113726881 conn(0x7f55f01009f0 msgr2=0x7f55f0101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:40.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 shutdown_connections 2024-04-16T02:26:40.230 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:39.486+0000 7f55f5052700 1 -- 172.21.15.151:0/4113726881 wait complete. 2024-04-16T02:26:40.234 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T02:26:40.234 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-04-16T02:26:40.242 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T02:26:40.242 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-04-16T02:26:40.242 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-04-16T02:26:40.242 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-16T02:26:40.244 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph Dashboard is now available at: 2024-04-16T02:26:40.244 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:40.244 INFO:teuthology.orchestra.run.smithi151.stdout: URL: https://smithi151.front.sepia.ceph.com:8443/ 2024-04-16T02:26:40.244 INFO:teuthology.orchestra.run.smithi151.stdout: User: admin 2024-04-16T02:26:40.244 INFO:teuthology.orchestra.run.smithi151.stdout: Password: r31qur8uvn 2024-04-16T02:26:40.244 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:40.244 INFO:teuthology.orchestra.run.smithi151.stdout:Saving cluster configuration to /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config directory 2024-04-16T02:26:40.245 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling autotune for osd_memory_target 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.707+0000 7f6952470700 1 Processor -- start 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.707+0000 7f6952470700 1 -- start start 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.707+0000 7f6952470700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c07a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.707+0000 7f6952470700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c07a8d0 con 0x7f694c07bea0 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.707+0000 7f694bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c07a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.707+0000 7f694bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c07a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41956/0 (socket says 172.21.15.151:41956) 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.707+0000 7f694bfff700 1 -- 172.21.15.151:0/3213531388 learned_addr learned my addr 172.21.15.151:0/3213531388 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.708+0000 7f694bfff700 1 -- 172.21.15.151:0/3213531388 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c07aa10 con 0x7f694c07bea0 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.708+0000 7f694bfff700 1 --2- 172.21.15.151:0/3213531388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c07a300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f693c0098a0 tx=0x7f693c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca27432486c81a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.708+0000 7f694b7fe700 1 -- 172.21.15.151:0/3213531388 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f693c004180 con 0x7f694c07bea0 2024-04-16T02:26:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.708+0000 7f694b7fe700 1 -- 172.21.15.151:0/3213531388 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f693c0042e0 con 0x7f694c07bea0 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.708+0000 7f694b7fe700 1 -- 172.21.15.151:0/3213531388 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f693c011af0 con 0x7f694c07bea0 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.709+0000 7f6952470700 1 -- 172.21.15.151:0/3213531388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 msgr2=0x7f694c07a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.709+0000 7f6952470700 1 --2- 172.21.15.151:0/3213531388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c07a300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f693c0098a0 tx=0x7f693c009bb0 comp rx=0 tx=0).stop 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.709+0000 7f6952470700 1 -- 172.21.15.151:0/3213531388 shutdown_connections 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.709+0000 7f6952470700 1 --2- 172.21.15.151:0/3213531388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c07a300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.709+0000 7f6952470700 1 -- 172.21.15.151:0/3213531388 >> 172.21.15.151:0/3213531388 conn(0x7f694c101b40 msgr2=0x7f694c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.709+0000 7f6952470700 1 -- 172.21.15.151:0/3213531388 shutdown_connections 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.709+0000 7f6952470700 1 -- 172.21.15.151:0/3213531388 wait complete. 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.710+0000 7f6952470700 1 Processor -- start 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.710+0000 7f6952470700 1 -- start start 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.710+0000 7f6952470700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.710+0000 7f6952470700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c19c9a0 con 0x7f694c07bea0 2024-04-16T02:26:41.484 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.710+0000 7f694bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.710+0000 7f694bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c19c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41972/0 (socket says 172.21.15.151:41972) 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.710+0000 7f694bfff700 1 -- 172.21.15.151:0/3225378262 learned_addr learned my addr 172.21.15.151:0/3225378262 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.710+0000 7f694bfff700 1 -- 172.21.15.151:0/3225378262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c009550 con 0x7f694c07bea0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.710+0000 7f694bfff700 1 --2- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c19c460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f693c011670 tx=0x7f693c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.711+0000 7f6949ffb700 1 -- 172.21.15.151:0/3225378262 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f693c0039f0 con 0x7f694c07bea0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.711+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694c19cba0 con 0x7f694c07bea0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.711+0000 7f6949ffb700 1 -- 172.21.15.151:0/3225378262 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f693c003b50 con 0x7f694c07bea0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.711+0000 7f6949ffb700 1 -- 172.21.15.151:0/3225378262 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f693c0247c0 con 0x7f694c07bea0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.711+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694c19d040 con 0x7f694c07bea0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.711+0000 7f6949ffb700 1 -- 172.21.15.151:0/3225378262 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f693c01a430 con 0x7f694c07bea0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.712+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694c10c900 con 0x7f694c07bea0 2024-04-16T02:26:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.712+0000 7f6949ffb700 1 --2- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f6938038410 0x7f693803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.712+0000 7f6949ffb700 1 -- 172.21.15.151:0/3225378262 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f693c04c5a0 con 0x7f694c07bea0 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.712+0000 7f6943fff700 1 --2- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f6938038410 0x7f693803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.712+0000 7f6943fff700 1 --2- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f6938038410 0x7f693803a8d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6934006fd0 tx=0x7f6934006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.717+0000 7f6949ffb700 1 -- 172.21.15.151:0/3225378262 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f693c018b40 con 0x7f694c07bea0 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.903+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f694c19fe20 con 0x7f694c07bea0 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.904+0000 7f6949ffb700 1 -- 172.21.15.151:0/3225378262 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f693c018b40 con 0x7f694c07bea0 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f6938038410 msgr2=0x7f693803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 --2- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f6938038410 0x7f693803a8d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6934006fd0 tx=0x7f6934006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 msgr2=0x7f694c19c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 --2- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c19c460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f693c011670 tx=0x7f693c011b50 comp rx=0 tx=0).stop 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 shutdown_connections 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 --2- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f6938038410 0x7f693803a8d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:41.486 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 --2- 172.21.15.151:0/3225378262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f694c07bea0 0x7f694c19c460 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:41.487 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 >> 172.21.15.151:0/3225378262 conn(0x7f694c101b40 msgr2=0x7f694c103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:41.487 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 shutdown_connections 2024-04-16T02:26:41.487 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:40.906+0000 7f6952470700 1 -- 172.21.15.151:0/3225378262 wait complete. 2024-04-16T02:26:41.993 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:41 smithi151 ceph-mon[29003]: mgrmap e13: smithi151.uqqame(active, since 6s) 2024-04-16T02:26:42.739 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.903+0000 7f9837be1700 1 Processor -- start 2024-04-16T02:26:42.739 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.903+0000 7f9837be1700 1 -- start start 2024-04-16T02:26:42.739 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f9837be1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f98301093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:42.739 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f9837be1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9830074680 con 0x7f9830106fe0 2024-04-16T02:26:42.739 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f983597d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f98301093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:42.739 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f983597d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f98301093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41986/0 (socket says 172.21.15.151:41986) 2024-04-16T02:26:42.739 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f983597d700 1 -- 172.21.15.151:0/4282485582 learned_addr learned my addr 172.21.15.151:0/4282485582 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f983597d700 1 -- 172.21.15.151:0/4282485582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830109910 con 0x7f9830106fe0 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f983597d700 1 --2- 172.21.15.151:0/4282485582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f98301093d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f982c009e80 tx=0x7f982c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6a9e4dd1d626e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f983497b700 1 -- 172.21.15.151:0/4282485582 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f982c00e070 con 0x7f9830106fe0 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f983497b700 1 -- 172.21.15.151:0/4282485582 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f982c00b8c0 con 0x7f9830106fe0 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.904+0000 7f983497b700 1 -- 172.21.15.151:0/4282485582 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f982c0044d0 con 0x7f9830106fe0 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.905+0000 7f9837be1700 1 -- 172.21.15.151:0/4282485582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 msgr2=0x7f98301093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.905+0000 7f9837be1700 1 --2- 172.21.15.151:0/4282485582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f98301093d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f982c009e80 tx=0x7f982c00b330 comp rx=0 tx=0).stop 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.905+0000 7f9837be1700 1 -- 172.21.15.151:0/4282485582 shutdown_connections 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.905+0000 7f9837be1700 1 --2- 172.21.15.151:0/4282485582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f98301093d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.905+0000 7f9837be1700 1 -- 172.21.15.151:0/4282485582 >> 172.21.15.151:0/4282485582 conn(0x7f98301009f0 msgr2=0x7f9830102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.905+0000 7f9837be1700 1 -- 172.21.15.151:0/4282485582 shutdown_connections 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.905+0000 7f9837be1700 1 -- 172.21.15.151:0/4282485582 wait complete. 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f9837be1700 1 Processor -- start 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f9837be1700 1 -- start start 2024-04-16T02:26:42.740 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f9837be1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f983019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f9837be1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983019a660 con 0x7f9830106fe0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f983597d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f983019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f983597d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f983019a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41992/0 (socket says 172.21.15.151:41992) 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f983597d700 1 -- 172.21.15.151:0/4209654427 learned_addr learned my addr 172.21.15.151:0/4209654427 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f983597d700 1 -- 172.21.15.151:0/4209654427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c009b30 con 0x7f9830106fe0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f983597d700 1 --2- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f983019a120 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f982c0072a0 tx=0x7f982c0049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f9826ffd700 1 -- 172.21.15.151:0/4209654427 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f982c00e040 con 0x7f9830106fe0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f9826ffd700 1 -- 172.21.15.151:0/4209654427 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f982c00bd80 con 0x7f9830106fe0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983019a860 con 0x7f9830106fe0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f9826ffd700 1 -- 172.21.15.151:0/4209654427 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f982c0126c0 con 0x7f9830106fe0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.906+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983019ad60 con 0x7f9830106fe0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.907+0000 7f9826ffd700 1 -- 172.21.15.151:0/4209654427 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f982c019040 con 0x7f9830106fe0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.907+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98300623c0 con 0x7f9830106fe0 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.908+0000 7f9826ffd700 1 --2- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f981c038420 0x7f981c03a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:42.741 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.908+0000 7f9826ffd700 1 -- 172.21.15.151:0/4209654427 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f982c04bfc0 con 0x7f9830106fe0 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.908+0000 7f983517c700 1 --2- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f981c038420 0x7f981c03a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.908+0000 7f983517c700 1 --2- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f981c038420 0x7f981c03a8e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9820006fd0 tx=0x7f9820006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:41.912+0000 7f9826ffd700 1 -- 172.21.15.151:0/4209654427 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f982c01a9e0 con 0x7f9830106fe0 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.212+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f983019d780 con 0x7f9830106fe0 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.215+0000 7f9826ffd700 1 -- 172.21.15.151:0/4209654427 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f982c01f020 con 0x7f9830106fe0 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f981c038420 msgr2=0x7f981c03a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 --2- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f981c038420 0x7f981c03a8e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9820006fd0 tx=0x7f9820006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 msgr2=0x7f983019a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 --2- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f983019a120 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f982c0072a0 tx=0x7f982c0049f0 comp rx=0 tx=0).stop 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 shutdown_connections 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 --2- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f981c038420 0x7f981c03a8e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 --2- 172.21.15.151:0/4209654427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9830106fe0 0x7f983019a120 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:42.742 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 >> 172.21.15.151:0/4209654427 conn(0x7f98301009f0 msgr2=0x7f98301016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 shutdown_connections 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-04-16T02:26:42.217+0000 7f9837be1700 1 -- 172.21.15.151:0/4209654427 wait complete. 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout:Or, if you are only running a single cluster on this host: 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-16T02:26:42.744 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:42.745 INFO:teuthology.orchestra.run.smithi151.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-16T02:26:42.745 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:42.745 INFO:teuthology.orchestra.run.smithi151.stdout: ceph telemetry on 2024-04-16T02:26:42.745 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:42.745 INFO:teuthology.orchestra.run.smithi151.stdout:For more information see: 2024-04-16T02:26:42.745 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:42.745 INFO:teuthology.orchestra.run.smithi151.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-16T02:26:42.745 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:26:42.745 INFO:teuthology.orchestra.run.smithi151.stdout:Bootstrap complete. 2024-04-16T02:26:42.773 INFO:tasks.cephadm:Fetching config... 2024-04-16T02:26:42.773 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:26:42.773 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-16T02:26:42.789 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-16T02:26:42.790 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:26:42.790 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-16T02:26:42.846 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-16T02:26:42.846 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:26:42.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/keyring of=/dev/stdout 2024-04-16T02:26:42.915 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-16T02:26:42.915 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:26:42.915 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-16T02:26:42.972 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-16T02:26:42.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1AKgxXyJ9Bybzp1KmEqfNrRdo3SuuX7B9fTWy4MpZCxjNwRSZ9T8AgylOikbKw5FQ0xp3yFerxgdNeRlcfWmQrtAOS7OU1CoOcCCxGvMVb8Ztm6zTV8YDsNDNzs104pLcFj9nXcwR+u2jR9BxD4i7G8XRudEWwYjLQ+/i+vWaDHg+JeVhC/abHWhqwNI8jXaZFrzl38qTdOlArzmmNbHE0G7nhQGJ1u2odJbrWNLXWNxyPyNsiPYS1XFz6/gvqIDhoIGZQaWiA3WfgW2NP5ybHvFRxoAsy9twQ7bOMKFi0DbhPpFD2zog+LIU8YklMQlkZcPz/3ZWCAOV61ZK44/sGe4y9TpwN0GgYynLld4Qgpm4oA3GexT4H0+FzbI6kKemmIcKgDYsvAuDdPT7JMcKOqWCe2esIRLyTZSAEtRnRchRr5suV/vk+YpkJat8p8HjrZBTb5bCiRHUuI/9YKgLdP7WKfcQ2wf2MNVpUqWLWt1EBm5gjxhLlnsz/8baSSU= ceph-80d1e958-fb98-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-16T02:26:43.054 INFO:teuthology.orchestra.run.smithi151.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1AKgxXyJ9Bybzp1KmEqfNrRdo3SuuX7B9fTWy4MpZCxjNwRSZ9T8AgylOikbKw5FQ0xp3yFerxgdNeRlcfWmQrtAOS7OU1CoOcCCxGvMVb8Ztm6zTV8YDsNDNzs104pLcFj9nXcwR+u2jR9BxD4i7G8XRudEWwYjLQ+/i+vWaDHg+JeVhC/abHWhqwNI8jXaZFrzl38qTdOlArzmmNbHE0G7nhQGJ1u2odJbrWNLXWNxyPyNsiPYS1XFz6/gvqIDhoIGZQaWiA3WfgW2NP5ybHvFRxoAsy9twQ7bOMKFi0DbhPpFD2zog+LIU8YklMQlkZcPz/3ZWCAOV61ZK44/sGe4y9TpwN0GgYynLld4Qgpm4oA3GexT4H0+FzbI6kKemmIcKgDYsvAuDdPT7JMcKOqWCe2esIRLyTZSAEtRnRchRr5suV/vk+YpkJat8p8HjrZBTb5bCiRHUuI/9YKgLdP7WKfcQ2wf2MNVpUqWLWt1EBm5gjxhLlnsz/8baSSU= ceph-80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:26:43.069 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1AKgxXyJ9Bybzp1KmEqfNrRdo3SuuX7B9fTWy4MpZCxjNwRSZ9T8AgylOikbKw5FQ0xp3yFerxgdNeRlcfWmQrtAOS7OU1CoOcCCxGvMVb8Ztm6zTV8YDsNDNzs104pLcFj9nXcwR+u2jR9BxD4i7G8XRudEWwYjLQ+/i+vWaDHg+JeVhC/abHWhqwNI8jXaZFrzl38qTdOlArzmmNbHE0G7nhQGJ1u2odJbrWNLXWNxyPyNsiPYS1XFz6/gvqIDhoIGZQaWiA3WfgW2NP5ybHvFRxoAsy9twQ7bOMKFi0DbhPpFD2zog+LIU8YklMQlkZcPz/3ZWCAOV61ZK44/sGe4y9TpwN0GgYynLld4Qgpm4oA3GexT4H0+FzbI6kKemmIcKgDYsvAuDdPT7JMcKOqWCe2esIRLyTZSAEtRnRchRr5suV/vk+YpkJat8p8HjrZBTb5bCiRHUuI/9YKgLdP7WKfcQ2wf2MNVpUqWLWt1EBm5gjxhLlnsz/8baSSU= ceph-80d1e958-fb98-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-16T02:26:43.117 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1AKgxXyJ9Bybzp1KmEqfNrRdo3SuuX7B9fTWy4MpZCxjNwRSZ9T8AgylOikbKw5FQ0xp3yFerxgdNeRlcfWmQrtAOS7OU1CoOcCCxGvMVb8Ztm6zTV8YDsNDNzs104pLcFj9nXcwR+u2jR9BxD4i7G8XRudEWwYjLQ+/i+vWaDHg+JeVhC/abHWhqwNI8jXaZFrzl38qTdOlArzmmNbHE0G7nhQGJ1u2odJbrWNLXWNxyPyNsiPYS1XFz6/gvqIDhoIGZQaWiA3WfgW2NP5ybHvFRxoAsy9twQ7bOMKFi0DbhPpFD2zog+LIU8YklMQlkZcPz/3ZWCAOV61ZK44/sGe4y9TpwN0GgYynLld4Qgpm4oA3GexT4H0+FzbI6kKemmIcKgDYsvAuDdPT7JMcKOqWCe2esIRLyTZSAEtRnRchRr5suV/vk+YpkJat8p8HjrZBTb5bCiRHUuI/9YKgLdP7WKfcQ2wf2MNVpUqWLWt1EBm5gjxhLlnsz/8baSSU= ceph-80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:26:43.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-16T02:26:43.339 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:26:43.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:43 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/4209654427' entity='client.admin' 2024-04-16T02:26:46.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.004+0000 7f890b55e700 1 -- 172.21.15.151:0/2862042864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 msgr2=0x7f89040731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:46.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.004+0000 7f890b55e700 1 --2- 172.21.15.151:0/2862042864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 0x7f89040731c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8900009910 tx=0x7f8900009c20 comp rx=0 tx=0).stop 2024-04-16T02:26:46.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.004+0000 7f890b55e700 1 -- 172.21.15.151:0/2862042864 shutdown_connections 2024-04-16T02:26:46.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.004+0000 7f890b55e700 1 --2- 172.21.15.151:0/2862042864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 0x7f89040731c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:46.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.004+0000 7f890b55e700 1 -- 172.21.15.151:0/2862042864 >> 172.21.15.151:0/2862042864 conn(0x7f89040fbe30 msgr2=0x7f89040fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:46.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.004+0000 7f890b55e700 1 -- 172.21.15.151:0/2862042864 shutdown_connections 2024-04-16T02:26:46.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.004+0000 7f890b55e700 1 -- 172.21.15.151:0/2862042864 wait complete. 2024-04-16T02:26:46.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.005+0000 7f890b55e700 1 Processor -- start 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.005+0000 7f890b55e700 1 -- start start 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.005+0000 7f890b55e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 0x7f8904197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.005+0000 7f890b55e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89041980f0 con 0x7f8904074d60 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.005+0000 7f89092fa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 0x7f8904197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.005+0000 7f89092fa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 0x7f8904197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42004/0 (socket says 172.21.15.151:42004) 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.005+0000 7f89092fa700 1 -- 172.21.15.151:0/213628262 learned_addr learned my addr 172.21.15.151:0/213628262 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.005+0000 7f89092fa700 1 -- 172.21.15.151:0/213628262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89000095f0 con 0x7f8904074d60 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.005+0000 7f89092fa700 1 --2- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 0x7f8904197bb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8900005b40 tx=0x7f8900004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.006+0000 7f88fa7fc700 1 -- 172.21.15.151:0/213628262 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f890001d070 con 0x7f8904074d60 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.006+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89041982f0 con 0x7f8904074d60 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.006+0000 7f88fa7fc700 1 -- 172.21.15.151:0/213628262 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f890000be10 con 0x7f8904074d60 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.006+0000 7f88fa7fc700 1 -- 172.21.15.151:0/213628262 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f890000f460 con 0x7f8904074d60 2024-04-16T02:26:46.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.006+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8904198790 con 0x7f8904074d60 2024-04-16T02:26:46.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.007+0000 7f88fa7fc700 1 -- 172.21.15.151:0/213628262 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f890000f6d0 con 0x7f8904074d60 2024-04-16T02:26:46.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.007+0000 7f88fa7fc700 1 --2- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f88f00383c0 0x7f88f003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:46.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.007+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89041916d0 con 0x7f8904074d60 2024-04-16T02:26:46.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.007+0000 7f88fa7fc700 1 -- 172.21.15.151:0/213628262 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f890004c570 con 0x7f8904074d60 2024-04-16T02:26:46.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.007+0000 7f8908af9700 1 --2- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f88f00383c0 0x7f88f003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:46.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.007+0000 7f8908af9700 1 --2- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f88f00383c0 0x7f88f003a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f88f4006fd0 tx=0x7f88f4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:46.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.012+0000 7f88fa7fc700 1 -- 172.21.15.151:0/213628262 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8900027070 con 0x7f8904074d60 2024-04-16T02:26:46.196 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:45 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:46.196 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:45 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:46.196 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:45 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:26:46.196 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:45 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:46.196 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:45 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:26:46.196 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:45 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-16T02:26:46.196 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:45 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:26:46.196 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:45 smithi151 ceph-mon[29003]: Deploying daemon ceph-exporter.smithi151 on smithi151 2024-04-16T02:26:46.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.226+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f890402cc70 con 0x7f8904074d60 2024-04-16T02:26:46.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.230+0000 7f88fa7fc700 1 -- 172.21.15.151:0/213628262 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f890002c440 con 0x7f8904074d60 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.232+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f88f00383c0 msgr2=0x7f88f003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.232+0000 7f890b55e700 1 --2- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f88f00383c0 0x7f88f003a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f88f4006fd0 tx=0x7f88f4006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.232+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 msgr2=0x7f8904197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.232+0000 7f890b55e700 1 --2- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 0x7f8904197bb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8900005b40 tx=0x7f8900004740 comp rx=0 tx=0).stop 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.232+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 shutdown_connections 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.233+0000 7f890b55e700 1 --2- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f88f00383c0 0x7f88f003a880 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.233+0000 7f890b55e700 1 --2- 172.21.15.151:0/213628262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8904074d60 0x7f8904197bb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.233+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 >> 172.21.15.151:0/213628262 conn(0x7f89040fbe30 msgr2=0x7f89040fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.233+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 shutdown_connections 2024-04-16T02:26:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:46.233+0000 7f890b55e700 1 -- 172.21.15.151:0/213628262 wait complete. 2024-04-16T02:26:47.306 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:47 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/213628262' entity='client.admin' 2024-04-16T02:26:47.307 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-16T02:26:47.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-16T02:26:48.360 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:26:50.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.067+0000 7f024a503700 1 -- 172.21.15.151:0/3225342024 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 msgr2=0x7f02440ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:50.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.067+0000 7f024a503700 1 --2- 172.21.15.151:0/3225342024 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 0x7f02440ffbb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0234009910 tx=0x7f0234009c20 comp rx=0 tx=0).stop 2024-04-16T02:26:50.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.067+0000 7f024a503700 1 -- 172.21.15.151:0/3225342024 shutdown_connections 2024-04-16T02:26:50.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.067+0000 7f024a503700 1 --2- 172.21.15.151:0/3225342024 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 0x7f02440ffbb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:50.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.067+0000 7f024a503700 1 -- 172.21.15.151:0/3225342024 >> 172.21.15.151:0/3225342024 conn(0x7f02440fad10 msgr2=0x7f02440fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:50.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.068+0000 7f024a503700 1 -- 172.21.15.151:0/3225342024 shutdown_connections 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.068+0000 7f024a503700 1 -- 172.21.15.151:0/3225342024 wait complete. 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.068+0000 7f024a503700 1 Processor -- start 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.068+0000 7f024a503700 1 -- start start 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.068+0000 7f024a503700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 0x7f02440fccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.068+0000 7f024a503700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02440fd210 con 0x7f02440ff790 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f0243fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 0x7f02440fccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f0243fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 0x7f02440fccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33504/0 (socket says 172.21.15.151:33504) 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f0243fff700 1 -- 172.21.15.151:0/405569499 learned_addr learned my addr 172.21.15.151:0/405569499 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f0243fff700 1 -- 172.21.15.151:0/405569499 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02340095f0 con 0x7f02440ff790 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f0243fff700 1 --2- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 0x7f02440fccd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0234005950 tx=0x7f0234009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f02417fa700 1 -- 172.21.15.151:0/405569499 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f023401d070 con 0x7f02440ff790 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f02417fa700 1 -- 172.21.15.151:0/405569499 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f023400be10 con 0x7f02440ff790 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02440fb320 con 0x7f02440ff790 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f02417fa700 1 -- 172.21.15.151:0/405569499 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f023400f460 con 0x7f02440ff790 2024-04-16T02:26:50.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.069+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02440fb7c0 con 0x7f02440ff790 2024-04-16T02:26:50.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.070+0000 7f02417fa700 1 -- 172.21.15.151:0/405569499 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f023400f690 con 0x7f02440ff790 2024-04-16T02:26:50.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.070+0000 7f02417fa700 1 --2- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f022c0383c0 0x7f022c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.070+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02440623c0 con 0x7f02440ff790 2024-04-16T02:26:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.070+0000 7f02417fa700 1 -- 172.21.15.151:0/405569499 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f023404c6c0 con 0x7f02440ff790 2024-04-16T02:26:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.070+0000 7f02437fe700 1 --2- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f022c0383c0 0x7f022c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.071+0000 7f02437fe700 1 --2- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f022c0383c0 0x7f022c03a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0238006fd0 tx=0x7f0238006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:50.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.075+0000 7f02417fa700 1 -- 172.21.15.151:0/405569499 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0234027070 con 0x7f02440ff790 2024-04-16T02:26:50.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.272+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 --> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f02440fba90 con 0x7f022c0383c0 2024-04-16T02:26:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.276+0000 7f02417fa700 1 -- 172.21.15.151:0/405569499 <== mgr.14164 v2:172.21.15.151:6800/3544869641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f02440fba90 con 0x7f022c0383c0 2024-04-16T02:26:50.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f022c0383c0 msgr2=0x7f022c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:50.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 --2- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f022c0383c0 0x7f022c03a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0238006fd0 tx=0x7f0238006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:50.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 msgr2=0x7f02440fccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:50.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 --2- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 0x7f02440fccd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0234005950 tx=0x7f0234009e60 comp rx=0 tx=0).stop 2024-04-16T02:26:50.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 shutdown_connections 2024-04-16T02:26:50.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 --2- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f022c0383c0 0x7f022c03a880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:50.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 --2- 172.21.15.151:0/405569499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02440ff790 0x7f02440fccd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:50.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 >> 172.21.15.151:0/405569499 conn(0x7f02440fad10 msgr2=0x7f0244069140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:50.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 shutdown_connections 2024-04-16T02:26:50.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:50.278+0000 7f024a503700 1 -- 172.21.15.151:0/405569499 wait complete. 2024-04-16T02:26:50.962 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-04-16T02:26:50.962 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:26:50.962 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-04-16T02:26:50.983 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:26:50.983 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:26:51.041 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-04-16T02:26:51.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch host add smithi153 2024-04-16T02:26:51.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:51.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:51.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:51.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:51.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:51.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:51.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi151", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:26:51.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi151", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T02:26:51.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:26:51.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:51 smithi151 ceph-mon[29003]: Deploying daemon crash.smithi151 on smithi151 2024-04-16T02:26:51.745 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:26:54.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.877+0000 7fccacfe6700 1 -- 172.21.15.151:0/3539977698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 msgr2=0x7fcca80ff330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:54.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.877+0000 7fccacfe6700 1 --2- 172.21.15.151:0/3539977698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 0x7fcca80ff330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800aa20 tx=0x7fcc9800ad30 comp rx=0 tx=0).stop 2024-04-16T02:26:54.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.877+0000 7fccacfe6700 1 -- 172.21.15.151:0/3539977698 shutdown_connections 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.877+0000 7fccacfe6700 1 --2- 172.21.15.151:0/3539977698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 0x7fcca80ff330 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.877+0000 7fccacfe6700 1 -- 172.21.15.151:0/3539977698 >> 172.21.15.151:0/3539977698 conn(0x7fcca80fa4b0 msgr2=0x7fcca80fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.877+0000 7fccacfe6700 1 -- 172.21.15.151:0/3539977698 shutdown_connections 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.877+0000 7fccacfe6700 1 -- 172.21.15.151:0/3539977698 wait complete. 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.877+0000 7fccacfe6700 1 Processor -- start 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.877+0000 7fccacfe6700 1 -- start start 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fccacfe6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 0x7fcca81959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fccacfe6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca8195f30 con 0x7fcca80fef10 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fcca659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 0x7fcca81959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fcca659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 0x7fcca81959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33530/0 (socket says 172.21.15.151:33530) 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fcca659c700 1 -- 172.21.15.151:0/1704470804 learned_addr learned my addr 172.21.15.151:0/1704470804 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fcca659c700 1 -- 172.21.15.151:0/1704470804 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9800a700 con 0x7fcca80fef10 2024-04-16T02:26:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fcca659c700 1 --2- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 0x7fcca81959f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcc98005e00 tx=0x7fcc98005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fcc977fe700 1 -- 172.21.15.151:0/1704470804 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9801d070 con 0x7fcca80fef10 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca8196130 con 0x7fcca80fef10 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fcc977fe700 1 -- 172.21.15.151:0/1704470804 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcc98005500 con 0x7fcca80fef10 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fcc977fe700 1 -- 172.21.15.151:0/1704470804 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9800e740 con 0x7fcca80fef10 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.878+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca81965d0 con 0x7fcca80fef10 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.879+0000 7fcc977fe700 1 -- 172.21.15.151:0/1704470804 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fcc9800ea00 con 0x7fcca80fef10 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.879+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca818f1c0 con 0x7fcca80fef10 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.879+0000 7fcc977fe700 1 --2- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcc900383c0 0x7fcc9003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.879+0000 7fcc977fe700 1 -- 172.21.15.151:0/1704470804 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcc98012070 con 0x7fcca80fef10 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.879+0000 7fcca5d9b700 1 --2- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcc900383c0 0x7fcc9003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.880+0000 7fcca5d9b700 1 --2- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcc900383c0 0x7fcc9003a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c006fd0 tx=0x7fcc9c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:26:54.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:53.884+0000 7fcc977fe700 1 -- 172.21.15.151:0/1704470804 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcc98005c00 con 0x7fcca80fef10 2024-04-16T02:26:54.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:54.092+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 --> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) v1 -- 0x7fcca80611d0 con 0x7fcc900383c0 2024-04-16T02:26:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:55 smithi151 ceph-mon[29003]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:26:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:55 smithi151 ceph-mon[29003]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:26:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:55 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:55 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:55 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:55 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:55 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:55.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:55 smithi151 ceph-mon[29003]: Deploying daemon node-exporter.smithi151 on smithi151 2024-04-16T02:26:56.493 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:56 smithi151 ceph-mon[29003]: Deploying cephadm binary to smithi153 2024-04-16T02:26:56.709 INFO:teuthology.orchestra.run.smithi151.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-04-16T02:26:56.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.707+0000 7fcc977fe700 1 -- 172.21.15.151:0/1704470804 <== mgr.14164 v2:172.21.15.151:6800/3544869641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcca80611d0 con 0x7fcc900383c0 2024-04-16T02:26:56.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcc900383c0 msgr2=0x7fcc9003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 --2- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcc900383c0 0x7fcc9003a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c006fd0 tx=0x7fcc9c006e40 comp rx=0 tx=0).stop 2024-04-16T02:26:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 msgr2=0x7fcca81959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:26:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 --2- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 0x7fcca81959f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcc98005e00 tx=0x7fcc98005000 comp rx=0 tx=0).stop 2024-04-16T02:26:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 shutdown_connections 2024-04-16T02:26:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 --2- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fcc900383c0 0x7fcc9003a880 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 --2- 172.21.15.151:0/1704470804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcca80fef10 0x7fcca81959f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:26:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 >> 172.21.15.151:0/1704470804 conn(0x7fcca80fa4b0 msgr2=0x7fcca818c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:26:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 shutdown_connections 2024-04-16T02:26:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:26:56.710+0000 7fccacfe6700 1 -- 172.21.15.151:0/1704470804 wait complete. 2024-04-16T02:26:57.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:57 smithi151 ceph-mon[29003]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:26:57.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:57 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:58.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-16T02:26:58.479 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:26:58.493 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:58 smithi151 ceph-mon[29003]: Added host smithi153 2024-04-16T02:26:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:59 smithi151 ceph-mon[29003]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:26:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:59 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:59 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:59 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:59 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:26:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:26:59 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:00.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.024+0000 7f20f2d2e700 1 -- 172.21.15.151:0/2590025583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 msgr2=0x7f20ec0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:00.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.024+0000 7f20f2d2e700 1 --2- 172.21.15.151:0/2590025583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 0x7f20ec0ffbc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009910 tx=0x7f20e0009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:00.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.024+0000 7f20f2d2e700 1 -- 172.21.15.151:0/2590025583 shutdown_connections 2024-04-16T02:27:00.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.024+0000 7f20f2d2e700 1 --2- 172.21.15.151:0/2590025583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 0x7f20ec0ffbc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:00.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.024+0000 7f20f2d2e700 1 -- 172.21.15.151:0/2590025583 >> 172.21.15.151:0/2590025583 conn(0x7f20ec0fad60 msgr2=0x7f20ec0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:00.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.024+0000 7f20f2d2e700 1 -- 172.21.15.151:0/2590025583 shutdown_connections 2024-04-16T02:27:00.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.024+0000 7f20f2d2e700 1 -- 172.21.15.151:0/2590025583 wait complete. 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.025+0000 7f20f2d2e700 1 Processor -- start 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.025+0000 7f20f2d2e700 1 -- start start 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.025+0000 7f20f2d2e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 0x7f20ec1948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.025+0000 7f20f2d2e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec194de0 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.025+0000 7f20f0aca700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 0x7f20ec1948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.025+0000 7f20f0aca700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 0x7f20ec1948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34342/0 (socket says 172.21.15.151:34342) 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.025+0000 7f20f0aca700 1 -- 172.21.15.151:0/1197478262 learned_addr learned my addr 172.21.15.151:0/1197478262 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.026+0000 7f20f0aca700 1 -- 172.21.15.151:0/1197478262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e00095f0 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.026+0000 7f20f0aca700 1 --2- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 0x7f20ec1948a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f20e0005b40 tx=0x7f20e0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.026+0000 7f20e9ffb700 1 -- 172.21.15.151:0/1197478262 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20e001d070 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.026+0000 7f20e9ffb700 1 -- 172.21.15.151:0/1197478262 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f20e000be10 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.026+0000 7f20e9ffb700 1 -- 172.21.15.151:0/1197478262 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20e000f460 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.026+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ec194fe0 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.026+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec195480 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.027+0000 7f20e9ffb700 1 -- 172.21.15.151:0/1197478262 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f20e00227e0 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.027+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ec18e3c0 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.027+0000 7f20e9ffb700 1 --2- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f20dc038410 0x7f20dc03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:00.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.027+0000 7f20e9ffb700 1 -- 172.21.15.151:0/1197478262 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f20e004c470 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.027+0000 7f20ebfff700 1 --2- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f20dc038410 0x7f20dc03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:00.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.028+0000 7f20ebfff700 1 --2- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f20dc038410 0x7f20dc03a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f20d8006fd0 tx=0x7f20d8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:00.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.032+0000 7f20e9ffb700 1 -- 172.21.15.151:0/1197478262 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f20e002bba0 con 0x7f20ec0ff7a0 2024-04-16T02:27:00.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.245+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 --> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20ec0611d0 con 0x7f20dc038410 2024-04-16T02:27:00.246 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.246+0000 7f20e9ffb700 1 -- 172.21.15.151:0/1197478262 <== mgr.14164 v2:172.21.15.151:6800/3544869641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f20ec0611d0 con 0x7f20dc038410 2024-04-16T02:27:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:27:00.247 INFO:teuthology.orchestra.run.smithi151.stdout:[{"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-04-16T02:27:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.248+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f20dc038410 msgr2=0x7f20dc03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.248+0000 7f20f2d2e700 1 --2- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f20dc038410 0x7f20dc03a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f20d8006fd0 tx=0x7f20d8006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.248+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 msgr2=0x7f20ec1948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.248+0000 7f20f2d2e700 1 --2- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 0x7f20ec1948a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f20e0005b40 tx=0x7f20e0004740 comp rx=0 tx=0).stop 2024-04-16T02:27:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.248+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 shutdown_connections 2024-04-16T02:27:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.248+0000 7f20f2d2e700 1 --2- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f20dc038410 0x7f20dc03a8d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.248+0000 7f20f2d2e700 1 --2- 172.21.15.151:0/1197478262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20ec0ff7a0 0x7f20ec1948a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.248+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 >> 172.21.15.151:0/1197478262 conn(0x7f20ec0fad60 msgr2=0x7f20ec0fb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.249+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 shutdown_connections 2024-04-16T02:27:00.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:00.249+0000 7f20f2d2e700 1 -- 172.21.15.151:0/1197478262 wait complete. 2024-04-16T02:27:00.493 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:00 smithi151 ceph-mon[29003]: Deploying daemon alertmanager.smithi151 on smithi151 2024-04-16T02:27:01.152 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-16T02:27:01.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd crush tunables default 2024-04-16T02:27:01.387 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:27:01.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:01 smithi151 ceph-mon[29003]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:01.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:01 smithi151 ceph-mon[29003]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T02:27:03.311 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:03 smithi151 ceph-mon[29003]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:03.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.545+0000 7f4e0aae5700 1 -- 172.21.15.151:0/4200258601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 msgr2=0x7f4e041037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:03.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.545+0000 7f4e0aae5700 1 --2- 172.21.15.151:0/4200258601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 0x7f4e041037d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009910 tx=0x7f4df8009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:03.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.545+0000 7f4e0aae5700 1 -- 172.21.15.151:0/4200258601 shutdown_connections 2024-04-16T02:27:03.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.545+0000 7f4e0aae5700 1 --2- 172.21.15.151:0/4200258601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 0x7f4e041037d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:03.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.545+0000 7f4e0aae5700 1 -- 172.21.15.151:0/4200258601 >> 172.21.15.151:0/4200258601 conn(0x7f4e040fad10 msgr2=0x7f4e040fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:03.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.545+0000 7f4e0aae5700 1 -- 172.21.15.151:0/4200258601 shutdown_connections 2024-04-16T02:27:03.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.545+0000 7f4e0aae5700 1 -- 172.21.15.151:0/4200258601 wait complete. 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.546+0000 7f4e0aae5700 1 Processor -- start 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.546+0000 7f4e0aae5700 1 -- start start 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.546+0000 7f4e0aae5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 0x7f4e04195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.546+0000 7f4e0aae5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04195e90 con 0x7f4e041013e0 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e08881700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 0x7f4e04195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e08881700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 0x7f4e04195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34352/0 (socket says 172.21.15.151:34352) 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e08881700 1 -- 172.21.15.151:0/2672003410 learned_addr learned my addr 172.21.15.151:0/2672003410 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e08881700 1 -- 172.21.15.151:0/2672003410 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df80095f0 con 0x7f4e041013e0 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e08881700 1 --2- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 0x7f4e04195950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4df8005b40 tx=0x7f4df8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e01ffb700 1 -- 172.21.15.151:0/2672003410 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df801d070 con 0x7f4e041013e0 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e01ffb700 1 -- 172.21.15.151:0/2672003410 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4df800be10 con 0x7f4e041013e0 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e04196090 con 0x7f4e041013e0 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e01ffb700 1 -- 172.21.15.151:0/2672003410 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df800f460 con 0x7f4e041013e0 2024-04-16T02:27:03.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.547+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e04196530 con 0x7f4e041013e0 2024-04-16T02:27:03.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.548+0000 7f4e01ffb700 1 -- 172.21.15.151:0/2672003410 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f4df800f5c0 con 0x7f4e041013e0 2024-04-16T02:27:03.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.548+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0418f4c0 con 0x7f4e041013e0 2024-04-16T02:27:03.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.548+0000 7f4e01ffb700 1 --2- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4df4038410 0x7f4df403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:03.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.549+0000 7f4e01ffb700 1 -- 172.21.15.151:0/2672003410 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4df804c3c0 con 0x7f4e041013e0 2024-04-16T02:27:03.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.549+0000 7f4e03fff700 1 --2- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4df4038410 0x7f4df403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:03.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.549+0000 7f4e03fff700 1 --2- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4df4038410 0x7f4df403a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4df0006fd0 tx=0x7f4df0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:03.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.553+0000 7f4e01ffb700 1 -- 172.21.15.151:0/2672003410 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4df802a360 con 0x7f4e041013e0 2024-04-16T02:27:03.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:03.756+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4e0402cc70 con 0x7f4e041013e0 2024-04-16T02:27:04.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.182+0000 7f4e01ffb700 1 -- 172.21.15.151:0/2672003410 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4df8027020 con 0x7f4e041013e0 2024-04-16T02:27:04.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.184+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4df4038410 msgr2=0x7f4df403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:04.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.184+0000 7f4e0aae5700 1 --2- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4df4038410 0x7f4df403a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4df0006fd0 tx=0x7f4df0006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:04.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.184+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 msgr2=0x7f4e04195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:04.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.184+0000 7f4e0aae5700 1 --2- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 0x7f4e04195950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4df8005b40 tx=0x7f4df8004740 comp rx=0 tx=0).stop 2024-04-16T02:27:04.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.184+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 shutdown_connections 2024-04-16T02:27:04.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.184+0000 7f4e0aae5700 1 --2- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4df4038410 0x7f4df403a8d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:04.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.184+0000 7f4e0aae5700 1 --2- 172.21.15.151:0/2672003410 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e041013e0 0x7f4e04195950 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:04.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.185+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 >> 172.21.15.151:0/2672003410 conn(0x7f4e040fad10 msgr2=0x7f4e040fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:04.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.185+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 shutdown_connections 2024-04-16T02:27:04.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:27:04.185+0000 7f4e0aae5700 1 -- 172.21.15.151:0/2672003410 wait complete. 2024-04-16T02:27:04.187 INFO:teuthology.orchestra.run.smithi151.stderr:adjusted tunables profile to default 2024-04-16T02:27:04.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:04 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2672003410' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-16T02:27:05.340 INFO:tasks.cephadm:Adding mon.smithi151 on smithi151 2024-04-16T02:27:05.341 INFO:tasks.cephadm:Adding mon.smithi153 on smithi153 2024-04-16T02:27:05.341 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch apply mon '2;smithi151:172.21.15.151=smithi151;smithi153:172.21.15.153=smithi153' 2024-04-16T02:27:05.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:05 smithi151 ceph-mon[29003]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:05.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:05 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2672003410' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-16T02:27:05.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:05 smithi151 ceph-mon[29003]: osdmap e4: 0 total, 0 up, 0 in 2024-04-16T02:27:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:05.597 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:07.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.063+0000 7fe5ff408700 1 -- 172.21.15.153:0/3461127416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 msgr2=0x7fe5f80ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:07.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.063+0000 7fe5ff408700 1 --2- 172.21.15.153:0/3461127416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 0x7fe5f80ffbb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009910 tx=0x7fe5e8009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:07.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.063+0000 7fe5ff408700 1 -- 172.21.15.153:0/3461127416 shutdown_connections 2024-04-16T02:27:07.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.063+0000 7fe5ff408700 1 --2- 172.21.15.153:0/3461127416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 0x7fe5f80ffbb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:07.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.063+0000 7fe5ff408700 1 -- 172.21.15.153:0/3461127416 >> 172.21.15.153:0/3461127416 conn(0x7fe5f80fad10 msgr2=0x7fe5f80fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:07.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.063+0000 7fe5ff408700 1 -- 172.21.15.153:0/3461127416 shutdown_connections 2024-04-16T02:27:07.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.063+0000 7fe5ff408700 1 -- 172.21.15.153:0/3461127416 wait complete. 2024-04-16T02:27:07.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.063+0000 7fe5ff408700 1 Processor -- start 2024-04-16T02:27:07.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.064+0000 7fe5ff408700 1 -- start start 2024-04-16T02:27:07.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.064+0000 7fe5ff408700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 0x7fe5f81948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:07.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.064+0000 7fe5ff408700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f8194de0 con 0x7fe5f80ff790 2024-04-16T02:27:07.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.064+0000 7fe5fd1a4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 0x7fe5f81948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:07.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.064+0000 7fe5fd1a4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 0x7fe5f81948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:57674/0 (socket says 172.21.15.153:57674) 2024-04-16T02:27:07.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.064+0000 7fe5fd1a4700 1 -- 172.21.15.153:0/1650748312 learned_addr learned my addr 172.21.15.153:0/1650748312 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:07.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.064+0000 7fe5fd1a4700 1 -- 172.21.15.153:0/1650748312 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80095f0 con 0x7fe5f80ff790 2024-04-16T02:27:07.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.064+0000 7fe5fd1a4700 1 --2- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 0x7fe5f81948a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8005b40 tx=0x7fe5e8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:07.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.065+0000 7fe5ee7fc700 1 -- 172.21.15.153:0/1650748312 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5e801d070 con 0x7fe5f80ff790 2024-04-16T02:27:07.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.065+0000 7fe5ee7fc700 1 -- 172.21.15.153:0/1650748312 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe5e800be10 con 0x7fe5f80ff790 2024-04-16T02:27:07.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.065+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f8194fe0 con 0x7fe5f80ff790 2024-04-16T02:27:07.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.065+0000 7fe5ee7fc700 1 -- 172.21.15.153:0/1650748312 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5e800f460 con 0x7fe5f80ff790 2024-04-16T02:27:07.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.065+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f8195480 con 0x7fe5f80ff790 2024-04-16T02:27:07.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.065+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f818e3b0 con 0x7fe5f80ff790 2024-04-16T02:27:07.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.066+0000 7fe5ee7fc700 1 -- 172.21.15.153:0/1650748312 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fe5e80227e0 con 0x7fe5f80ff790 2024-04-16T02:27:07.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.066+0000 7fe5ee7fc700 1 --2- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fe5e40383c0 0x7fe5e403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:07.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.066+0000 7fe5ee7fc700 1 -- 172.21.15.153:0/1650748312 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe5e804c430 con 0x7fe5f80ff790 2024-04-16T02:27:07.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.066+0000 7fe5fc9a3700 1 --2- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fe5e40383c0 0x7fe5e403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:07.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.066+0000 7fe5fc9a3700 1 --2- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fe5e40383c0 0x7fe5e403a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4006fd0 tx=0x7fe5f4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:07.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.071+0000 7fe5ee7fc700 1 -- 172.21.15.153:0/1650748312 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe5e802bba0 con 0x7fe5f80ff790 2024-04-16T02:27:07.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.268+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 --> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi151:172.21.15.151=smithi151;smithi153:172.21.15.153=smithi153", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5f8061340 con 0x7fe5e40383c0 2024-04-16T02:27:07.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.274+0000 7fe5ee7fc700 1 -- 172.21.15.153:0/1650748312 <== mgr.14164 v2:172.21.15.151:6800/3544869641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe5f8061340 con 0x7fe5e40383c0 2024-04-16T02:27:07.275 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mon update... 2024-04-16T02:27:07.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fe5e40383c0 msgr2=0x7fe5e403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:07.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 --2- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fe5e40383c0 0x7fe5e403a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4006fd0 tx=0x7fe5f4006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:07.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 msgr2=0x7fe5f81948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:07.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 --2- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 0x7fe5f81948a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8005b40 tx=0x7fe5e8004740 comp rx=0 tx=0).stop 2024-04-16T02:27:07.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 shutdown_connections 2024-04-16T02:27:07.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 --2- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fe5e40383c0 0x7fe5e403a880 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:07.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 --2- 172.21.15.153:0/1650748312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5f80ff790 0x7fe5f81948a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:07.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 >> 172.21.15.153:0/1650748312 conn(0x7fe5f80fad10 msgr2=0x7fe5f80fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:07.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 shutdown_connections 2024-04-16T02:27:07.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:07.276+0000 7fe5ff408700 1 -- 172.21.15.153:0/1650748312 wait complete. 2024-04-16T02:27:07.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:07 smithi151 ceph-mon[29003]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:07.907 DEBUG:teuthology.orchestra.run.smithi153:mon.smithi153> sudo journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi153.service 2024-04-16T02:27:07.909 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:07.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:08.346 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:08.645 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:08 smithi151 ceph-mon[29003]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi151:172.21.15.151=smithi151;smithi153:172.21.15.153=smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:27:08.645 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:08 smithi151 ceph-mon[29003]: Saving service mon spec with placement smithi151:172.21.15.151=smithi151;smithi153:172.21.15.153=smithi153;count:2 2024-04-16T02:27:08.645 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:08 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:09.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:09 smithi151 ceph-mon[29003]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:09.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:09 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:09.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:09 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:09.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:09 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:09.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:09 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:09.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:09 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:09.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:09 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:09.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.801+0000 7f917f29e700 1 -- 172.21.15.153:0/4180886332 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 msgr2=0x7f91780ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:09.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.801+0000 7f917f29e700 1 --2- 172.21.15.153:0/4180886332 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 0x7f91780ffbc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9168009910 tx=0x7f9168009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:09.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.801+0000 7f917f29e700 1 -- 172.21.15.153:0/4180886332 shutdown_connections 2024-04-16T02:27:09.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.801+0000 7f917f29e700 1 --2- 172.21.15.153:0/4180886332 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 0x7f91780ffbc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:09.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.801+0000 7f917f29e700 1 -- 172.21.15.153:0/4180886332 >> 172.21.15.153:0/4180886332 conn(0x7f91780fad20 msgr2=0x7f91780fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:09.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.801+0000 7f917f29e700 1 -- 172.21.15.153:0/4180886332 shutdown_connections 2024-04-16T02:27:09.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.801+0000 7f917f29e700 1 -- 172.21.15.153:0/4180886332 wait complete. 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.802+0000 7f917f29e700 1 Processor -- start 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.802+0000 7f917f29e700 1 -- start start 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.802+0000 7f917f29e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 0x7f91780fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.802+0000 7f917f29e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91780fd220 con 0x7f91780ff7a0 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.802+0000 7f917d03a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 0x7f91780fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.802+0000 7f917d03a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 0x7f91780fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:47368/0 (socket says 172.21.15.153:47368) 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.802+0000 7f917d03a700 1 -- 172.21.15.153:0/1899249136 learned_addr learned my addr 172.21.15.153:0/1899249136 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.802+0000 7f917d03a700 1 -- 172.21.15.153:0/1899249136 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91680095f0 con 0x7f91780ff7a0 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.803+0000 7f917d03a700 1 --2- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 0x7f91780fcce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9168005b40 tx=0x7f9168004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.803+0000 7f916e7fc700 1 -- 172.21.15.153:0/1899249136 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f916801d070 con 0x7f91780ff7a0 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.803+0000 7f916e7fc700 1 -- 172.21.15.153:0/1899249136 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f916800be10 con 0x7f91780ff7a0 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.803+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91780fb330 con 0x7f91780ff7a0 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.803+0000 7f916e7fc700 1 -- 172.21.15.153:0/1899249136 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f916800f460 con 0x7f91780ff7a0 2024-04-16T02:27:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.803+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91780fb7d0 con 0x7f91780ff7a0 2024-04-16T02:27:09.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.804+0000 7f916e7fc700 1 -- 172.21.15.153:0/1899249136 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f916800b960 con 0x7f91780ff7a0 2024-04-16T02:27:09.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.804+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91780623c0 con 0x7f91780ff7a0 2024-04-16T02:27:09.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.804+0000 7f916e7fc700 1 --2- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f91640383c0 0x7f916403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:09.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.804+0000 7f916e7fc700 1 -- 172.21.15.153:0/1899249136 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f916804c480 con 0x7f91780ff7a0 2024-04-16T02:27:09.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.804+0000 7f917c839700 1 --2- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f91640383c0 0x7f916403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:09.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.805+0000 7f917c839700 1 --2- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f91640383c0 0x7f916403a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9174006fd0 tx=0x7f9174006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:09.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:09.809+0000 7f916e7fc700 1 -- 172.21.15.153:0/1899249136 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f916802a360 con 0x7f91780ff7a0 2024-04-16T02:27:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.072+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f917818edb0 con 0x7f91780ff7a0 2024-04-16T02:27:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.073+0000 7f916e7fc700 1 -- 172.21.15.153:0/1899249136 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9168027070 con 0x7f91780ff7a0 2024-04-16T02:27:10.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:10.074 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f91640383c0 msgr2=0x7f916403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 --2- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f91640383c0 0x7f916403a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9174006fd0 tx=0x7f9174006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 msgr2=0x7f91780fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 --2- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 0x7f91780fcce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9168005b40 tx=0x7f9168004740 comp rx=0 tx=0).stop 2024-04-16T02:27:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 shutdown_connections 2024-04-16T02:27:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 --2- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f91640383c0 0x7f916403a880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 --2- 172.21.15.153:0/1899249136 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91780ff7a0 0x7f91780fcce0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 >> 172.21.15.153:0/1899249136 conn(0x7f91780fad20 msgr2=0x7f9178068810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 shutdown_connections 2024-04-16T02:27:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:10.075+0000 7f917f29e700 1 -- 172.21.15.153:0/1899249136 wait complete. 2024-04-16T02:27:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:10.493 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:10 smithi151 ceph-mon[29003]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-16T02:27:10.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:10 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:10.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:10 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:10.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:10 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-16T02:27:10.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:10 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-16T02:27:10.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:10 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:10.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:10 smithi151 ceph-mon[29003]: Deploying daemon grafana.smithi151 on smithi151 2024-04-16T02:27:10.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:10 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/1899249136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:11.641 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:11.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:11.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:11 smithi151 ceph-mon[29003]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:11.849 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:11.898 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:13.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.287+0000 7ff86e4a7700 1 -- 172.21.15.153:0/313726422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 msgr2=0x7ff8680ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:13.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.287+0000 7ff86e4a7700 1 --2- 172.21.15.153:0/313726422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 0x7ff8680ffbc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff850009910 tx=0x7ff850009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:13.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.287+0000 7ff86e4a7700 1 -- 172.21.15.153:0/313726422 shutdown_connections 2024-04-16T02:27:13.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.287+0000 7ff86e4a7700 1 --2- 172.21.15.153:0/313726422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 0x7ff8680ffbc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.287+0000 7ff86e4a7700 1 -- 172.21.15.153:0/313726422 >> 172.21.15.153:0/313726422 conn(0x7ff8680fad20 msgr2=0x7ff8680fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.287+0000 7ff86e4a7700 1 -- 172.21.15.153:0/313726422 shutdown_connections 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.287+0000 7ff86e4a7700 1 -- 172.21.15.153:0/313726422 wait complete. 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.288+0000 7ff86e4a7700 1 Processor -- start 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.288+0000 7ff86e4a7700 1 -- start start 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.288+0000 7ff86e4a7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 0x7ff8680693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.288+0000 7ff86e4a7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8681964f0 con 0x7ff8680ff7a0 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.288+0000 7ff867fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 0x7ff8680693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.288+0000 7ff867fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 0x7ff8680693d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:47382/0 (socket says 172.21.15.153:47382) 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.288+0000 7ff867fff700 1 -- 172.21.15.153:0/1332195216 learned_addr learned my addr 172.21.15.153:0/1332195216 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.289+0000 7ff867fff700 1 -- 172.21.15.153:0/1332195216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8500095f0 con 0x7ff8680ff7a0 2024-04-16T02:27:13.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.289+0000 7ff867fff700 1 --2- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 0x7ff8680693d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff850005950 tx=0x7ff850009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:13.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.289+0000 7ff8657fa700 1 -- 172.21.15.153:0/1332195216 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff85001d070 con 0x7ff8680ff7a0 2024-04-16T02:27:13.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.289+0000 7ff8657fa700 1 -- 172.21.15.153:0/1332195216 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff85000be10 con 0x7ff8680ff7a0 2024-04-16T02:27:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.289+0000 7ff8657fa700 1 -- 172.21.15.153:0/1332195216 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff85000f460 con 0x7ff8680ff7a0 2024-04-16T02:27:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.289+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86819ab00 con 0x7ff8680ff7a0 2024-04-16T02:27:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.289+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86819ae60 con 0x7ff8680ff7a0 2024-04-16T02:27:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.290+0000 7ff8657fa700 1 -- 172.21.15.153:0/1332195216 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7ff85000b960 con 0x7ff8680ff7a0 2024-04-16T02:27:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.290+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86818e3c0 con 0x7ff8680ff7a0 2024-04-16T02:27:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.290+0000 7ff8657fa700 1 --2- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff854038410 0x7ff85403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.290+0000 7ff8657fa700 1 -- 172.21.15.153:0/1332195216 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff85004c420 con 0x7ff8680ff7a0 2024-04-16T02:27:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.290+0000 7ff8677fe700 1 --2- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff854038410 0x7ff85403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.291+0000 7ff8677fe700 1 --2- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff854038410 0x7ff85403a8d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff858006fd0 tx=0x7ff858006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:13.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.295+0000 7ff8657fa700 1 -- 172.21.15.153:0/1332195216 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff85002bba0 con 0x7ff8680ff7a0 2024-04-16T02:27:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.557+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff86802cc70 con 0x7ff8680ff7a0 2024-04-16T02:27:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.557+0000 7ff8657fa700 1 -- 172.21.15.153:0/1332195216 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff850027030 con 0x7ff8680ff7a0 2024-04-16T02:27:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:13.559 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:13.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.559+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff854038410 msgr2=0x7ff85403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:13.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.559+0000 7ff86e4a7700 1 --2- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff854038410 0x7ff85403a8d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff858006fd0 tx=0x7ff858006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:13.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.559+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 msgr2=0x7ff8680693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.559+0000 7ff86e4a7700 1 --2- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 0x7ff8680693d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff850005950 tx=0x7ff850009e60 comp rx=0 tx=0).stop 2024-04-16T02:27:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.560+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 shutdown_connections 2024-04-16T02:27:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.560+0000 7ff86e4a7700 1 --2- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff854038410 0x7ff85403a8d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.560+0000 7ff86e4a7700 1 --2- 172.21.15.153:0/1332195216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8680ff7a0 0x7ff8680693d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.560+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 >> 172.21.15.153:0/1332195216 conn(0x7ff8680fad20 msgr2=0x7ff8680fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.560+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 shutdown_connections 2024-04-16T02:27:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:13.560+0000 7ff86e4a7700 1 -- 172.21.15.153:0/1332195216 wait complete. 2024-04-16T02:27:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:13.743 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:13 smithi151 ceph-mon[29003]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:14.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:14 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/1332195216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:14.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:14 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:15.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:15.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:15.244 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:15.292 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:15.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:15 smithi151 ceph-mon[29003]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:16.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.715+0000 7f2e3606f700 1 -- 172.21.15.153:0/1308810535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 msgr2=0x7f2e30103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:16.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.715+0000 7f2e3606f700 1 --2- 172.21.15.153:0/1308810535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 0x7f2e30103720 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e20009910 tx=0x7f2e20009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:16.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.715+0000 7f2e3606f700 1 -- 172.21.15.153:0/1308810535 shutdown_connections 2024-04-16T02:27:16.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.715+0000 7f2e3606f700 1 --2- 172.21.15.153:0/1308810535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 0x7f2e30103720 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:16.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.715+0000 7f2e3606f700 1 -- 172.21.15.153:0/1308810535 >> 172.21.15.153:0/1308810535 conn(0x7f2e300fad20 msgr2=0x7f2e300fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:16.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.715+0000 7f2e3606f700 1 -- 172.21.15.153:0/1308810535 shutdown_connections 2024-04-16T02:27:16.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.715+0000 7f2e3606f700 1 -- 172.21.15.153:0/1308810535 wait complete. 2024-04-16T02:27:16.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.715+0000 7f2e3606f700 1 Processor -- start 2024-04-16T02:27:16.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.715+0000 7f2e3606f700 1 -- start start 2024-04-16T02:27:16.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e3606f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 0x7f2e30195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:16.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e3606f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30195e70 con 0x7f2e30101330 2024-04-16T02:27:16.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e2f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 0x7f2e30195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:16.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e2f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 0x7f2e30195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:47406/0 (socket says 172.21.15.153:47406) 2024-04-16T02:27:16.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e2f7fe700 1 -- 172.21.15.153:0/1030186931 learned_addr learned my addr 172.21.15.153:0/1030186931 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e2f7fe700 1 -- 172.21.15.153:0/1030186931 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e200095f0 con 0x7f2e30101330 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e2f7fe700 1 --2- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 0x7f2e30195930 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2e20005b40 tx=0x7f2e20004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e2cff9700 1 -- 172.21.15.153:0/1030186931 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e2001d070 con 0x7f2e30101330 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e2cff9700 1 -- 172.21.15.153:0/1030186931 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2e2000be10 con 0x7f2e30101330 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e2cff9700 1 -- 172.21.15.153:0/1030186931 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e2000f460 con 0x7f2e30101330 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e30196070 con 0x7f2e30101330 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.716+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e30196510 con 0x7f2e30101330 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.717+0000 7f2e2cff9700 1 -- 172.21.15.153:0/1030186931 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f2e2000b960 con 0x7f2e30101330 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.717+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3018f4c0 con 0x7f2e30101330 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.717+0000 7f2e2cff9700 1 --2- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f2e18038410 0x7f2e1803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.718+0000 7f2e2cff9700 1 -- 172.21.15.153:0/1030186931 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2e2004c560 con 0x7f2e30101330 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.718+0000 7f2e2effd700 1 --2- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f2e18038410 0x7f2e1803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:16.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.718+0000 7f2e2effd700 1 --2- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f2e18038410 0x7f2e1803a8d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e24006fd0 tx=0x7f2e24006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:16.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.724+0000 7f2e2cff9700 1 -- 172.21.15.153:0/1030186931 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2e2002bb60 con 0x7f2e30101330 2024-04-16T02:27:16.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.983+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e3002cc70 con 0x7f2e30101330 2024-04-16T02:27:16.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.984+0000 7f2e2cff9700 1 -- 172.21.15.153:0/1030186931 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2e20027030 con 0x7f2e30101330 2024-04-16T02:27:16.985 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:16.986 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:16.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.986+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f2e18038410 msgr2=0x7f2e1803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.986+0000 7f2e3606f700 1 --2- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f2e18038410 0x7f2e1803a8d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e24006fd0 tx=0x7f2e24006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.986+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 msgr2=0x7f2e30195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.986+0000 7f2e3606f700 1 --2- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 0x7f2e30195930 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2e20005b40 tx=0x7f2e20004740 comp rx=0 tx=0).stop 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.986+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 shutdown_connections 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.986+0000 7f2e3606f700 1 --2- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f2e18038410 0x7f2e1803a8d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.986+0000 7f2e3606f700 1 --2- 172.21.15.153:0/1030186931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e30101330 0x7f2e30195930 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.986+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 >> 172.21.15.153:0/1030186931 conn(0x7f2e300fad20 msgr2=0x7f2e300fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.987+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 shutdown_connections 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:16.987+0000 7f2e3606f700 1 -- 172.21.15.153:0/1030186931 wait complete. 2024-04-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:17.743 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:17 smithi151 ceph-mon[29003]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:17.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:17 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/1030186931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:18.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:18.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:18.785 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:18.834 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:19.743 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:19 smithi151 ceph-mon[29003]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.300+0000 7fec43948700 1 -- 172.21.15.153:0/1284682784 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 msgr2=0x7fec3c100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.300+0000 7fec43948700 1 --2- 172.21.15.153:0/1284682784 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 0x7fec3c100cd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009910 tx=0x7fec2c009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.300+0000 7fec43948700 1 -- 172.21.15.153:0/1284682784 shutdown_connections 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.300+0000 7fec43948700 1 --2- 172.21.15.153:0/1284682784 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 0x7fec3c100cd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.300+0000 7fec43948700 1 -- 172.21.15.153:0/1284682784 >> 172.21.15.153:0/1284682784 conn(0x7fec3c0fbe30 msgr2=0x7fec3c0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.300+0000 7fec43948700 1 -- 172.21.15.153:0/1284682784 shutdown_connections 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.300+0000 7fec43948700 1 -- 172.21.15.153:0/1284682784 wait complete. 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.301+0000 7fec43948700 1 Processor -- start 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.301+0000 7fec43948700 1 -- start start 2024-04-16T02:27:20.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.301+0000 7fec43948700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 0x7fec3c1959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:20.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.301+0000 7fec43948700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c195f00 con 0x7fec3c1008b0 2024-04-16T02:27:20.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.301+0000 7fec416e4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 0x7fec3c1959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:20.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.301+0000 7fec416e4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 0x7fec3c1959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:33048/0 (socket says 172.21.15.153:33048) 2024-04-16T02:27:20.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.301+0000 7fec416e4700 1 -- 172.21.15.153:0/3507903862 learned_addr learned my addr 172.21.15.153:0/3507903862 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.302+0000 7fec416e4700 1 -- 172.21.15.153:0/3507903862 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c0095f0 con 0x7fec3c1008b0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.302+0000 7fec416e4700 1 --2- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 0x7fec3c1959c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fec2c005b40 tx=0x7fec2c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.302+0000 7fec327fc700 1 -- 172.21.15.153:0/3507903862 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec2c01d070 con 0x7fec3c1008b0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.302+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3c196100 con 0x7fec3c1008b0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.302+0000 7fec327fc700 1 -- 172.21.15.153:0/3507903862 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fec2c00be10 con 0x7fec3c1008b0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.302+0000 7fec327fc700 1 -- 172.21.15.153:0/3507903862 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec2c00f460 con 0x7fec3c1008b0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.302+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3c1965a0 con 0x7fec3c1008b0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.303+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3c18f4d0 con 0x7fec3c1008b0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.303+0000 7fec327fc700 1 -- 172.21.15.153:0/3507903862 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fec2c0227e0 con 0x7fec3c1008b0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.303+0000 7fec327fc700 1 --2- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fec280383c0 0x7fec2803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.303+0000 7fec327fc700 1 -- 172.21.15.153:0/3507903862 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fec2c04c560 con 0x7fec3c1008b0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.303+0000 7fec40ee3700 1 --2- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fec280383c0 0x7fec2803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:20.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.304+0000 7fec40ee3700 1 --2- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fec280383c0 0x7fec2803a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fec38006fd0 tx=0x7fec38006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:20.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.308+0000 7fec327fc700 1 -- 172.21.15.153:0/3507903862 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fec2c02bba0 con 0x7fec3c1008b0 2024-04-16T02:27:20.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.568+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec3c02cc70 con 0x7fec3c1008b0 2024-04-16T02:27:20.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.569+0000 7fec327fc700 1 -- 172.21.15.153:0/3507903862 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fec2c027020 con 0x7fec3c1008b0 2024-04-16T02:27:20.569 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:20.569 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:20.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fec280383c0 msgr2=0x7fec2803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:20.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 --2- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fec280383c0 0x7fec2803a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fec38006fd0 tx=0x7fec38006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:20.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 msgr2=0x7fec3c1959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:20.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 --2- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 0x7fec3c1959c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fec2c005b40 tx=0x7fec2c004740 comp rx=0 tx=0).stop 2024-04-16T02:27:20.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 shutdown_connections 2024-04-16T02:27:20.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 --2- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fec280383c0 0x7fec2803a880 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:20.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 --2- 172.21.15.153:0/3507903862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec3c1008b0 0x7fec3c1959c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:20.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 >> 172.21.15.153:0/3507903862 conn(0x7fec3c0fbe30 msgr2=0x7fec3c0fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:20.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 shutdown_connections 2024-04-16T02:27:20.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:20.571+0000 7fec43948700 1 -- 172.21.15.153:0/3507903862 wait complete. 2024-04-16T02:27:20.573 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:20.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:20 smithi151 ceph-mon[29003]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:20.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:20 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/3507903862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:22.169 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:22.169 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:22.376 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:22.423 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:23.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:23 smithi151 ceph-mon[29003]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.905+0000 7f78bc282700 1 -- 172.21.15.153:0/1856315029 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 msgr2=0x7f78b40ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.905+0000 7f78bc282700 1 --2- 172.21.15.153:0/1856315029 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 0x7f78b40ffbc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00b0a0 tx=0x7f78ac00b3b0 comp rx=0 tx=0).stop 2024-04-16T02:27:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.905+0000 7f78bc282700 1 -- 172.21.15.153:0/1856315029 shutdown_connections 2024-04-16T02:27:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.905+0000 7f78bc282700 1 --2- 172.21.15.153:0/1856315029 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 0x7f78b40ffbc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.905+0000 7f78bc282700 1 -- 172.21.15.153:0/1856315029 >> 172.21.15.153:0/1856315029 conn(0x7f78b40fad20 msgr2=0x7f78b40fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.905+0000 7f78bc282700 1 -- 172.21.15.153:0/1856315029 shutdown_connections 2024-04-16T02:27:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.905+0000 7f78bc282700 1 -- 172.21.15.153:0/1856315029 wait complete. 2024-04-16T02:27:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.906+0000 7f78bc282700 1 Processor -- start 2024-04-16T02:27:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.906+0000 7f78bc282700 1 -- start start 2024-04-16T02:27:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.906+0000 7f78bc282700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 0x7f78b4068b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.906+0000 7f78bc282700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b41986d0 con 0x7f78b40ff7a0 2024-04-16T02:27:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.906+0000 7f78ba01e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 0x7f78b4068b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.906+0000 7f78ba01e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 0x7f78b4068b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:33068/0 (socket says 172.21.15.153:33068) 2024-04-16T02:27:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.906+0000 7f78ba01e700 1 -- 172.21.15.153:0/2253559716 learned_addr learned my addr 172.21.15.153:0/2253559716 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.906+0000 7f78ba01e700 1 -- 172.21.15.153:0/2253559716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac009d00 con 0x7f78b40ff7a0 2024-04-16T02:27:23.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.907+0000 7f78ba01e700 1 --2- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 0x7f78b4068b20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00b830 tx=0x7f78ac00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.907+0000 7f78b37fe700 1 -- 172.21.15.153:0/2253559716 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78ac00e040 con 0x7f78b40ff7a0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.907+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b4198810 con 0x7f78b40ff7a0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.907+0000 7f78b37fe700 1 -- 172.21.15.153:0/2253559716 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f78ac012af0 con 0x7f78b40ff7a0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.907+0000 7f78b37fe700 1 -- 172.21.15.153:0/2253559716 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78ac01d9f0 con 0x7f78b40ff7a0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.907+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b4198c30 con 0x7f78b40ff7a0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.908+0000 7f78b37fe700 1 -- 172.21.15.153:0/2253559716 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f78ac01dcb0 con 0x7f78b40ff7a0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.908+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b418e3c0 con 0x7f78b40ff7a0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.908+0000 7f78b37fe700 1 --2- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f789c0383c0 0x7f789c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.908+0000 7f78b37fe700 1 -- 172.21.15.153:0/2253559716 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f78ac04c980 con 0x7f78b40ff7a0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.908+0000 7f78b981d700 1 --2- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f789c0383c0 0x7f789c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:23.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.909+0000 7f78b981d700 1 --2- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f789c0383c0 0x7f789c03a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f78a8006fd0 tx=0x7f78a8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:23.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:23.913+0000 7f78b37fe700 1 -- 172.21.15.153:0/2253559716 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f78ac01abf0 con 0x7f78b40ff7a0 2024-04-16T02:27:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.177+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78b402cc70 con 0x7f78b40ff7a0 2024-04-16T02:27:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.178+0000 7f78b37fe700 1 -- 172.21.15.153:0/2253559716 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f78ac01abf0 con 0x7f78b40ff7a0 2024-04-16T02:27:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:24.178 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f789c0383c0 msgr2=0x7f789c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 --2- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f789c0383c0 0x7f789c03a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f78a8006fd0 tx=0x7f78a8006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 msgr2=0x7f78b4068b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 --2- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 0x7f78b4068b20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00b830 tx=0x7f78ac00bd40 comp rx=0 tx=0).stop 2024-04-16T02:27:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 shutdown_connections 2024-04-16T02:27:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 --2- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f789c0383c0 0x7f789c03a880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 --2- 172.21.15.153:0/2253559716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78b40ff7a0 0x7f78b4068b20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 >> 172.21.15.153:0/2253559716 conn(0x7f78b40fad20 msgr2=0x7f78b40fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 shutdown_connections 2024-04-16T02:27:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:24.180+0000 7f78bc282700 1 -- 172.21.15.153:0/2253559716 wait complete. 2024-04-16T02:27:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:24.487 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:24 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/2253559716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:25.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:25 smithi151 ceph-mon[29003]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:25.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:25 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:25 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:25 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:25 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:25 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:25 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:25 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:25 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:25.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:25.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:26.015 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:26.444 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:26 smithi151 ceph-mon[29003]: Deploying daemon prometheus.smithi151 on smithi151 2024-04-16T02:27:27.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.360+0000 7fb28cf74700 1 -- 172.21.15.153:0/2615663189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 msgr2=0x7fb288073220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:27.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.360+0000 7fb28cf74700 1 --2- 172.21.15.153:0/2615663189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 0x7fb288073220 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb278009910 tx=0x7fb278009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:27.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.360+0000 7fb28cf74700 1 -- 172.21.15.153:0/2615663189 shutdown_connections 2024-04-16T02:27:27.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.360+0000 7fb28cf74700 1 --2- 172.21.15.153:0/2615663189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 0x7fb288073220 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:27.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.360+0000 7fb28cf74700 1 -- 172.21.15.153:0/2615663189 >> 172.21.15.153:0/2615663189 conn(0x7fb2880fbe30 msgr2=0x7fb2880fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:27.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.360+0000 7fb28cf74700 1 -- 172.21.15.153:0/2615663189 shutdown_connections 2024-04-16T02:27:27.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.360+0000 7fb28cf74700 1 -- 172.21.15.153:0/2615663189 wait complete. 2024-04-16T02:27:27.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.361+0000 7fb28cf74700 1 Processor -- start 2024-04-16T02:27:27.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.361+0000 7fb28cf74700 1 -- start start 2024-04-16T02:27:27.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.361+0000 7fb28cf74700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 0x7fb288197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.361+0000 7fb28cf74700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2881980b0 con 0x7fb288074dc0 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.361+0000 7fb28659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 0x7fb288197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.361+0000 7fb28659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 0x7fb288197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:33092/0 (socket says 172.21.15.153:33092) 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.361+0000 7fb28659c700 1 -- 172.21.15.153:0/786783349 learned_addr learned my addr 172.21.15.153:0/786783349 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.361+0000 7fb28659c700 1 -- 172.21.15.153:0/786783349 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2780095f0 con 0x7fb288074dc0 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.362+0000 7fb28659c700 1 --2- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 0x7fb288197b70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb278005b40 tx=0x7fb278004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.362+0000 7fb27f7fe700 1 -- 172.21.15.153:0/786783349 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb27801d070 con 0x7fb288074dc0 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.362+0000 7fb27f7fe700 1 -- 172.21.15.153:0/786783349 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb27800be10 con 0x7fb288074dc0 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.362+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2881982b0 con 0x7fb288074dc0 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.362+0000 7fb27f7fe700 1 -- 172.21.15.153:0/786783349 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb27800f460 con 0x7fb288074dc0 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.362+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb288198750 con 0x7fb288074dc0 2024-04-16T02:27:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.363+0000 7fb27f7fe700 1 -- 172.21.15.153:0/786783349 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fb27800f6d0 con 0x7fb288074dc0 2024-04-16T02:27:27.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.363+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2881916e0 con 0x7fb288074dc0 2024-04-16T02:27:27.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.363+0000 7fb27f7fe700 1 --2- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fb2740383c0 0x7fb27403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:27.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.363+0000 7fb27f7fe700 1 -- 172.21.15.153:0/786783349 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb27804c690 con 0x7fb288074dc0 2024-04-16T02:27:27.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.363+0000 7fb285d9b700 1 --2- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fb2740383c0 0x7fb27403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:27.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.364+0000 7fb285d9b700 1 --2- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fb2740383c0 0x7fb27403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb270006fd0 tx=0x7fb270006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.368+0000 7fb27f7fe700 1 -- 172.21.15.153:0/786783349 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb27802a360 con 0x7fb288074dc0 2024-04-16T02:27:27.493 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:27 smithi151 ceph-mon[29003]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.633+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb28802cc70 con 0x7fb288074dc0 2024-04-16T02:27:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.633+0000 7fb27f7fe700 1 -- 172.21.15.153:0/786783349 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb278027030 con 0x7fb288074dc0 2024-04-16T02:27:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:27.634 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.635+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fb2740383c0 msgr2=0x7fb27403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.635+0000 7fb28cf74700 1 --2- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fb2740383c0 0x7fb27403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb270006fd0 tx=0x7fb270006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.635+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 msgr2=0x7fb288197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.635+0000 7fb28cf74700 1 --2- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 0x7fb288197b70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb278005b40 tx=0x7fb278004740 comp rx=0 tx=0).stop 2024-04-16T02:27:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.635+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 shutdown_connections 2024-04-16T02:27:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.635+0000 7fb28cf74700 1 --2- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fb2740383c0 0x7fb27403a880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.635+0000 7fb28cf74700 1 --2- 172.21.15.153:0/786783349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb288074dc0 0x7fb288197b70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.635+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 >> 172.21.15.153:0/786783349 conn(0x7fb2880fbe30 msgr2=0x7fb2880fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.636+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 shutdown_connections 2024-04-16T02:27:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:27.636+0000 7fb28cf74700 1 -- 172.21.15.153:0/786783349 wait complete. 2024-04-16T02:27:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:28.493 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:28 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/786783349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:29.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:29.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:29.342 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:29.390 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:29.493 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:29 smithi151 ceph-mon[29003]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:29.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:29 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.838+0000 7ff178e4c700 1 -- 172.21.15.153:0/1887624778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 msgr2=0x7ff1741022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.838+0000 7ff178e4c700 1 --2- 172.21.15.153:0/1887624778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 0x7ff1741022b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff168009910 tx=0x7ff168009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.838+0000 7ff178e4c700 1 -- 172.21.15.153:0/1887624778 shutdown_connections 2024-04-16T02:27:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.838+0000 7ff178e4c700 1 --2- 172.21.15.153:0/1887624778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 0x7ff1741022b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.838+0000 7ff178e4c700 1 -- 172.21.15.153:0/1887624778 >> 172.21.15.153:0/1887624778 conn(0x7ff1740fa4d0 msgr2=0x7ff1740fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.838+0000 7ff178e4c700 1 -- 172.21.15.153:0/1887624778 shutdown_connections 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.838+0000 7ff178e4c700 1 -- 172.21.15.153:0/1887624778 wait complete. 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.839+0000 7ff178e4c700 1 Processor -- start 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.839+0000 7ff178e4c700 1 -- start start 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.839+0000 7ff178e4c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 0x7ff1741959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.839+0000 7ff178e4c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff174195ef0 con 0x7ff174101e90 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.839+0000 7ff17259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 0x7ff1741959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.839+0000 7ff17259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 0x7ff1741959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:36652/0 (socket says 172.21.15.153:36652) 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.839+0000 7ff17259c700 1 -- 172.21.15.153:0/2212737385 learned_addr learned my addr 172.21.15.153:0/2212737385 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.839+0000 7ff17259c700 1 -- 172.21.15.153:0/2212737385 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1680095f0 con 0x7ff174101e90 2024-04-16T02:27:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.840+0000 7ff17259c700 1 --2- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 0x7ff1741959b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff168005950 tx=0x7ff168009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.840+0000 7ff1637fe700 1 -- 172.21.15.153:0/2212737385 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff16801d070 con 0x7ff174101e90 2024-04-16T02:27:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.840+0000 7ff1637fe700 1 -- 172.21.15.153:0/2212737385 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff16800be10 con 0x7ff174101e90 2024-04-16T02:27:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.840+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1741960f0 con 0x7ff174101e90 2024-04-16T02:27:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.840+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff174196590 con 0x7ff174101e90 2024-04-16T02:27:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.840+0000 7ff1637fe700 1 -- 172.21.15.153:0/2212737385 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff16800f460 con 0x7ff174101e90 2024-04-16T02:27:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.841+0000 7ff1637fe700 1 -- 172.21.15.153:0/2212737385 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7ff16800f690 con 0x7ff174101e90 2024-04-16T02:27:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.841+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff154005320 con 0x7ff174101e90 2024-04-16T02:27:30.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.841+0000 7ff1637fe700 1 --2- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff15c0383c0 0x7ff15c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:30.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.841+0000 7ff1637fe700 1 -- 172.21.15.153:0/2212737385 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff16804c5d0 con 0x7ff174101e90 2024-04-16T02:27:30.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.841+0000 7ff171d9b700 1 --2- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff15c0383c0 0x7ff15c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:30.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.842+0000 7ff171d9b700 1 --2- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff15c0383c0 0x7ff15c03a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff164006fd0 tx=0x7ff164006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:30.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:30.846+0000 7ff1637fe700 1 -- 172.21.15.153:0/2212737385 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff16802a360 con 0x7ff174101e90 2024-04-16T02:27:30.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:30 smithi151 ceph-mon[29003]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:31.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.107+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1540059f0 con 0x7ff174101e90 2024-04-16T02:27:31.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.108+0000 7ff1637fe700 1 -- 172.21.15.153:0/2212737385 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff168027070 con 0x7ff174101e90 2024-04-16T02:27:31.108 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:31.108 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:31.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff15c0383c0 msgr2=0x7ff15c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:31.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 --2- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff15c0383c0 0x7ff15c03a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff164006fd0 tx=0x7ff164006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:31.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 msgr2=0x7ff1741959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:31.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 --2- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 0x7ff1741959b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff168005950 tx=0x7ff168009e60 comp rx=0 tx=0).stop 2024-04-16T02:27:31.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 shutdown_connections 2024-04-16T02:27:31.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 --2- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7ff15c0383c0 0x7ff15c03a880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:31.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 --2- 172.21.15.153:0/2212737385 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff174101e90 0x7ff1741959b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:31.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 >> 172.21.15.153:0/2212737385 conn(0x7ff1740fa4d0 msgr2=0x7ff1740fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:31.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 shutdown_connections 2024-04-16T02:27:31.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:31.110+0000 7ff178e4c700 1 -- 172.21.15.153:0/2212737385 wait complete. 2024-04-16T02:27:31.111 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:31.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:31 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/2212737385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:32.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:32.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:32.931 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:32.979 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:32.993 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:32 smithi151 ceph-mon[29003]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:34.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.260+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/999079982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 msgr2=0x7f4d240ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:34.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.260+0000 7f4d2bb1c700 1 --2- 172.21.15.153:0/999079982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 0x7f4d240ffbc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009910 tx=0x7f4d20009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:34.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.260+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/999079982 shutdown_connections 2024-04-16T02:27:34.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.260+0000 7f4d2bb1c700 1 --2- 172.21.15.153:0/999079982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 0x7f4d240ffbc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:34.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.260+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/999079982 >> 172.21.15.153:0/999079982 conn(0x7f4d240fad20 msgr2=0x7f4d240fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:34.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.260+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/999079982 shutdown_connections 2024-04-16T02:27:34.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.260+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/999079982 wait complete. 2024-04-16T02:27:34.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d2bb1c700 1 Processor -- start 2024-04-16T02:27:34.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d2bb1c700 1 -- start start 2024-04-16T02:27:34.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d2bb1c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 0x7f4d240fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:34.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d2bb1c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d240fd220 con 0x7f4d240ff7a0 2024-04-16T02:27:34.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d298b8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 0x7f4d240fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:34.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d298b8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 0x7f4d240fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:36672/0 (socket says 172.21.15.153:36672) 2024-04-16T02:27:34.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d298b8700 1 -- 172.21.15.153:0/877205727 learned_addr learned my addr 172.21.15.153:0/877205727 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:34.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d298b8700 1 -- 172.21.15.153:0/877205727 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d200095f0 con 0x7f4d240ff7a0 2024-04-16T02:27:34.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d298b8700 1 --2- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 0x7f4d240fcce0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4d20005b40 tx=0x7f4d20004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:34.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.261+0000 7f4d1affd700 1 -- 172.21.15.153:0/877205727 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d2001d070 con 0x7f4d240ff7a0 2024-04-16T02:27:34.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.262+0000 7f4d1affd700 1 -- 172.21.15.153:0/877205727 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4d2000be10 con 0x7f4d240ff7a0 2024-04-16T02:27:34.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.262+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d240fb330 con 0x7f4d240ff7a0 2024-04-16T02:27:34.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.262+0000 7f4d1affd700 1 -- 172.21.15.153:0/877205727 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d2000f460 con 0x7f4d240ff7a0 2024-04-16T02:27:34.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.262+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d240fb7d0 con 0x7f4d240ff7a0 2024-04-16T02:27:34.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.262+0000 7f4d1affd700 1 -- 172.21.15.153:0/877205727 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f4d2000b960 con 0x7f4d240ff7a0 2024-04-16T02:27:34.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.262+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d240623c0 con 0x7f4d240ff7a0 2024-04-16T02:27:34.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.263+0000 7f4d1affd700 1 --2- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4d100383c0 0x7f4d1003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:34.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.263+0000 7f4d1affd700 1 -- 172.21.15.153:0/877205727 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4d2004c480 con 0x7f4d240ff7a0 2024-04-16T02:27:34.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.263+0000 7f4d290b7700 1 --2- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4d100383c0 0x7f4d1003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:34.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.263+0000 7f4d290b7700 1 --2- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4d100383c0 0x7f4d1003a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d14006fd0 tx=0x7f4d14006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:34.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.267+0000 7f4d1affd700 1 -- 172.21.15.153:0/877205727 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4d2002a360 con 0x7f4d240ff7a0 2024-04-16T02:27:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.534+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d2418edb0 con 0x7f4d240ff7a0 2024-04-16T02:27:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.535+0000 7f4d1affd700 1 -- 172.21.15.153:0/877205727 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4d20027030 con 0x7f4d240ff7a0 2024-04-16T02:27:34.535 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:34.535 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4d100383c0 msgr2=0x7f4d1003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 --2- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4d100383c0 0x7f4d1003a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d14006fd0 tx=0x7f4d14006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 msgr2=0x7f4d240fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 --2- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 0x7f4d240fcce0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4d20005b40 tx=0x7f4d20004740 comp rx=0 tx=0).stop 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 shutdown_connections 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 --2- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f4d100383c0 0x7f4d1003a880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 --2- 172.21.15.153:0/877205727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d240ff7a0 0x7f4d240fcce0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 >> 172.21.15.153:0/877205727 conn(0x7f4d240fad20 msgr2=0x7f4d24069100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 shutdown_connections 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:34.537+0000 7f4d2bb1c700 1 -- 172.21.15.153:0/877205727 wait complete. 2024-04-16T02:27:34.538 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:35 smithi151 ceph-mon[29003]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:27:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:35 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/877205727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:35 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:35 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:35 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:35.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:35 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-16T02:27:36.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:36.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:36.316 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:36.365 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:37.351 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:37 smithi151 ceph-mon[29003]: from='mgr.14164 172.21.15.151:0/4218705472' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-16T02:27:37.351 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:37 smithi151 ceph-mon[29003]: mgrmap e14: smithi151.uqqame(active, since 61s) 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.778+0000 7fd30f9a4700 1 -- 172.21.15.153:0/2916399247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 msgr2=0x7fd308103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.778+0000 7fd30f9a4700 1 --2- 172.21.15.153:0/2916399247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 0x7fd308103720 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd304009910 tx=0x7fd304009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.778+0000 7fd30f9a4700 1 -- 172.21.15.153:0/2916399247 shutdown_connections 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.778+0000 7fd30f9a4700 1 --2- 172.21.15.153:0/2916399247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 0x7fd308103720 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.778+0000 7fd30f9a4700 1 -- 172.21.15.153:0/2916399247 >> 172.21.15.153:0/2916399247 conn(0x7fd3080fad20 msgr2=0x7fd3080fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.778+0000 7fd30f9a4700 1 -- 172.21.15.153:0/2916399247 shutdown_connections 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.778+0000 7fd30f9a4700 1 -- 172.21.15.153:0/2916399247 wait complete. 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.778+0000 7fd30f9a4700 1 Processor -- start 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.778+0000 7fd30f9a4700 1 -- start start 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd30f9a4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 0x7fd308195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:37.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd30f9a4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd308195e70 con 0x7fd308101330 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd30d740700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 0x7fd308195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd30d740700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 0x7fd308195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:36698/0 (socket says 172.21.15.153:36698) 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd30d740700 1 -- 172.21.15.153:0/3460720213 learned_addr learned my addr 172.21.15.153:0/3460720213 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd30d740700 1 -- 172.21.15.153:0/3460720213 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3040095f0 con 0x7fd308101330 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd30d740700 1 --2- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 0x7fd308195930 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd304005b40 tx=0x7fd304004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd2fe7fc700 1 -- 172.21.15.153:0/3460720213 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd30401d070 con 0x7fd308101330 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd2fe7fc700 1 -- 172.21.15.153:0/3460720213 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd30400be10 con 0x7fd308101330 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd308196070 con 0x7fd308101330 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd2fe7fc700 1 -- 172.21.15.153:0/3460720213 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd30400f460 con 0x7fd308101330 2024-04-16T02:27:37.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.779+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd308196510 con 0x7fd308101330 2024-04-16T02:27:37.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.780+0000 7fd2fe7fc700 1 -- 172.21.15.153:0/3460720213 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 14) v1 ==== 45354+0+0 (secure 0 0 0) 0x7fd30400f6d0 con 0x7fd308101330 2024-04-16T02:27:37.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.780+0000 7fd2fe7fc700 1 --2- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fd2f4038410 0x7fd2f403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:37.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.780+0000 7fd2fe7fc700 1 -- 172.21.15.153:0/3460720213 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd30404c6a0 con 0x7fd308101330 2024-04-16T02:27:37.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.780+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30818f4c0 con 0x7fd308101330 2024-04-16T02:27:37.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.780+0000 7fd30cf3f700 1 -- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fd2f4038410 msgr2=0x7fd2f403a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/3544869641 2024-04-16T02:27:37.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.780+0000 7fd30cf3f700 1 --2- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fd2f4038410 0x7fd2f403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:27:37.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.786+0000 7fd2fe7fc700 1 -- 172.21.15.153:0/3460720213 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd304027070 con 0x7fd308101330 2024-04-16T02:27:37.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.981+0000 7fd30cf3f700 1 -- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fd2f4038410 msgr2=0x7fd2f403a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/3544869641 2024-04-16T02:27:37.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:37.981+0000 7fd30cf3f700 1 --2- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fd2f4038410 0x7fd2f403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T02:27:38.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.048+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd30802cc70 con 0x7fd308101330 2024-04-16T02:27:38.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.049+0000 7fd2fe7fc700 1 -- 172.21.15.153:0/3460720213 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fd30402c440 con 0x7fd308101330 2024-04-16T02:27:38.049 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:38.049 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fd2f4038410 msgr2=0x7fd2f403a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 --2- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fd2f4038410 0x7fd2f403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 msgr2=0x7fd308195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 --2- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 0x7fd308195930 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd304005b40 tx=0x7fd304004740 comp rx=0 tx=0).stop 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 shutdown_connections 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 --2- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7fd2f4038410 0x7fd2f403a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 --2- 172.21.15.153:0/3460720213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd308101330 0x7fd308195930 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 >> 172.21.15.153:0/3460720213 conn(0x7fd3080fad20 msgr2=0x7fd3080fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 shutdown_connections 2024-04-16T02:27:38.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:38.051+0000 7fd30f9a4700 1 -- 172.21.15.153:0/3460720213 wait complete. 2024-04-16T02:27:38.052 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:38.299 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:38 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/3460720213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:39.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:39.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:41.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.400+0000 7f08453c1700 1 -- 172.21.15.153:0/1155462830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 msgr2=0x7f08400ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:41.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.400+0000 7f08453c1700 1 --2- 172.21.15.153:0/1155462830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 0x7f08400ffbc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0830009910 tx=0x7f0830009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:41.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.400+0000 7f08453c1700 1 -- 172.21.15.153:0/1155462830 shutdown_connections 2024-04-16T02:27:41.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.400+0000 7f08453c1700 1 --2- 172.21.15.153:0/1155462830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 0x7f08400ffbc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:41.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.400+0000 7f08453c1700 1 -- 172.21.15.153:0/1155462830 >> 172.21.15.153:0/1155462830 conn(0x7f08400fad20 msgr2=0x7f08400fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:41.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.401+0000 7f08453c1700 1 -- 172.21.15.153:0/1155462830 shutdown_connections 2024-04-16T02:27:41.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.401+0000 7f08453c1700 1 -- 172.21.15.153:0/1155462830 wait complete. 2024-04-16T02:27:41.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.401+0000 7f08453c1700 1 Processor -- start 2024-04-16T02:27:41.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.401+0000 7f08453c1700 1 -- start start 2024-04-16T02:27:41.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f08453c1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 0x7f08401948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:41.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f08453c1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0840194de0 con 0x7f08400ff7a0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f083effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 0x7f08401948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f083effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 0x7f08401948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:58172/0 (socket says 172.21.15.153:58172) 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f083effd700 1 -- 172.21.15.153:0/2557501809 learned_addr learned my addr 172.21.15.153:0/2557501809 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f083effd700 1 -- 172.21.15.153:0/2557501809 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08300095f0 con 0x7f08400ff7a0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f083effd700 1 --2- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 0x7f08401948a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0830005b40 tx=0x7f0830004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f0827fff700 1 -- 172.21.15.153:0/2557501809 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083001d070 con 0x7f08400ff7a0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f0827fff700 1 -- 172.21.15.153:0/2557501809 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f083000be10 con 0x7f08400ff7a0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0840194fe0 con 0x7f08400ff7a0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f0827fff700 1 -- 172.21.15.153:0/2557501809 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083000f460 con 0x7f08400ff7a0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.402+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0840195480 con 0x7f08400ff7a0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.403+0000 7f0827fff700 1 -- 172.21.15.153:0/2557501809 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 14) v1 ==== 45354+0+0 (secure 0 0 0) 0x7f083000f6d0 con 0x7f08400ff7a0 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.403+0000 7f0827fff700 1 --2- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f0828038410 0x7f082803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.403+0000 7f0827fff700 1 -- 172.21.15.153:0/2557501809 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f083004c500 con 0x7f08400ff7a0 2024-04-16T02:27:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.403+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084018e3c0 con 0x7f08400ff7a0 2024-04-16T02:27:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.403+0000 7f083e7fc700 1 -- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f0828038410 msgr2=0x7f082803a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/3544869641 2024-04-16T02:27:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.404+0000 7f083e7fc700 1 --2- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f0828038410 0x7f082803a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:27:41.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.409+0000 7f0827fff700 1 -- 172.21.15.153:0/2557501809 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0830027070 con 0x7f08400ff7a0 2024-04-16T02:27:41.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.605+0000 7f083e7fc700 1 -- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f0828038410 msgr2=0x7f082803a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.151:6800/3544869641 2024-04-16T02:27:41.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.605+0000 7f083e7fc700 1 --2- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f0828038410 0x7f082803a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T02:27:41.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.676+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08400619a0 con 0x7f08400ff7a0 2024-04-16T02:27:41.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.677+0000 7f0827fff700 1 -- 172.21.15.153:0/2557501809 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f083002c440 con 0x7f08400ff7a0 2024-04-16T02:27:41.677 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:41.677 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:41.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f0828038410 msgr2=0x7f082803a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 --2- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f0828038410 0x7f082803a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 msgr2=0x7f08401948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 --2- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 0x7f08401948a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0830005b40 tx=0x7f0830004740 comp rx=0 tx=0).stop 2024-04-16T02:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 shutdown_connections 2024-04-16T02:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 --2- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:6800/3544869641,v1:172.21.15.151:6801/3544869641] conn(0x7f0828038410 0x7f082803a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 --2- 172.21.15.153:0/2557501809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08400ff7a0 0x7f08401948a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 >> 172.21.15.153:0/2557501809 conn(0x7f08400fad20 msgr2=0x7f08400fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 shutdown_connections 2024-04-16T02:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:41.679+0000 7f08453c1700 1 -- 172.21.15.153:0/2557501809 wait complete. 2024-04-16T02:27:41.680 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:41.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:41 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/2557501809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:43.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:43.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: Active manager daemon smithi151.uqqame restarted 2024-04-16T02:27:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: Activating manager daemon smithi151.uqqame 2024-04-16T02:27:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T02:27:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: mgrmap e15: smithi151.uqqame(active, starting, since 0.00392475s) 2024-04-16T02:27:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:27:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:27:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:27:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:27:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:27:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: Manager daemon smithi151.uqqame is now available 2024-04-16T02:27:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:27:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:27:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:27:43.455 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:43.511 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:44.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:27:44.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/trash_purge_schedule"}]: dispatch 2024-04-16T02:27:44.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:44.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:43 smithi151 ceph-mon[29003]: mgrmap e16: smithi151.uqqame(active, since 1.00597s) 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.966+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/3065599845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 msgr2=0x7f3b9c100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.966+0000 7f3ba0cc9700 1 --2- 172.21.15.153:0/3065599845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 0x7f3b9c100cd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009910 tx=0x7f3b8c009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.966+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/3065599845 shutdown_connections 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.966+0000 7f3ba0cc9700 1 --2- 172.21.15.153:0/3065599845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 0x7f3b9c100cd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.966+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/3065599845 >> 172.21.15.153:0/3065599845 conn(0x7f3b9c0fbe30 msgr2=0x7f3b9c0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.966+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/3065599845 shutdown_connections 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.966+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/3065599845 wait complete. 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.967+0000 7f3ba0cc9700 1 Processor -- start 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.967+0000 7f3ba0cc9700 1 -- start start 2024-04-16T02:27:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.967+0000 7f3ba0cc9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 0x7f3b9c1959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.967+0000 7f3ba0cc9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c195f10 con 0x7f3b9c1008b0 2024-04-16T02:27:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.967+0000 7f3b9a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 0x7f3b9c1959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.967+0000 7f3b9a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 0x7f3b9c1959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:58190/0 (socket says 172.21.15.153:58190) 2024-04-16T02:27:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.967+0000 7f3b9a59c700 1 -- 172.21.15.153:0/541873030 learned_addr learned my addr 172.21.15.153:0/541873030 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:44.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.967+0000 7f3b9a59c700 1 -- 172.21.15.153:0/541873030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c0095f0 con 0x7f3b9c1008b0 2024-04-16T02:27:44.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.967+0000 7f3b9a59c700 1 --2- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 0x7f3b9c1959d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c005b40 tx=0x7f3b8c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:44.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.968+0000 7f3b8b7fe700 1 -- 172.21.15.153:0/541873030 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b8c01d070 con 0x7f3b9c1008b0 2024-04-16T02:27:44.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.968+0000 7f3b8b7fe700 1 -- 172.21.15.153:0/541873030 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3b8c00be10 con 0x7f3b9c1008b0 2024-04-16T02:27:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.968+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c196110 con 0x7f3b9c1008b0 2024-04-16T02:27:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.968+0000 7f3b8b7fe700 1 -- 172.21.15.153:0/541873030 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b8c00f460 con 0x7f3b9c1008b0 2024-04-16T02:27:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.968+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c1965b0 con 0x7f3b9c1008b0 2024-04-16T02:27:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.969+0000 7f3b8b7fe700 1 -- 172.21.15.153:0/541873030 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 45204+0+0 (secure 0 0 0) 0x7f3b8c00f6d0 con 0x7f3b9c1008b0 2024-04-16T02:27:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.969+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c18f4d0 con 0x7f3b9c1008b0 2024-04-16T02:27:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.969+0000 7f3b8b7fe700 1 --2- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3b84038270 0x7f3b8403a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.969+0000 7f3b8b7fe700 1 -- 172.21.15.153:0/541873030 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f3b8c04c300 con 0x7f3b9c1008b0 2024-04-16T02:27:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.969+0000 7f3b99d9b700 1 --2- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3b84038270 0x7f3b8403a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:44.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.970+0000 7f3b99d9b700 1 --2- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3b84038270 0x7f3b8403a730 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3b90006fd0 tx=0x7f3b90006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:44.975+0000 7f3b8b7fe700 1 -- 172.21.15.153:0/541873030 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3b8c02a360 con 0x7f3b9c1008b0 2024-04-16T02:27:45.239 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:44 smithi151 ceph-mon[29003]: [16/Apr/2024:02:27:43] ENGINE Bus STARTING 2024-04-16T02:27:45.239 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:44 smithi151 ceph-mon[29003]: [16/Apr/2024:02:27:43] ENGINE Serving on http://172.21.15.151:8765 2024-04-16T02:27:45.239 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:44 smithi151 ceph-mon[29003]: [16/Apr/2024:02:27:43] ENGINE Serving on https://172.21.15.151:7150 2024-04-16T02:27:45.239 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:44 smithi151 ceph-mon[29003]: [16/Apr/2024:02:27:43] ENGINE Bus STARTED 2024-04-16T02:27:45.239 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:44 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.251+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b9c02cc70 con 0x7f3b9c1008b0 2024-04-16T02:27:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.252+0000 7f3b8b7fe700 1 -- 172.21.15.153:0/541873030 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3b8c027020 con 0x7f3b9c1008b0 2024-04-16T02:27:45.253 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:45.254 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3b84038270 msgr2=0x7f3b8403a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 --2- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3b84038270 0x7f3b8403a730 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3b90006fd0 tx=0x7f3b90006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 msgr2=0x7f3b9c1959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 --2- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 0x7f3b9c1959d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c005b40 tx=0x7f3b8c004740 comp rx=0 tx=0).stop 2024-04-16T02:27:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 shutdown_connections 2024-04-16T02:27:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 --2- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3b84038270 0x7f3b8403a730 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 --2- 172.21.15.153:0/541873030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b9c1008b0 0x7f3b9c1959d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 >> 172.21.15.153:0/541873030 conn(0x7f3b9c0fbe30 msgr2=0x7f3b9c0fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 shutdown_connections 2024-04-16T02:27:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:45.254+0000 7f3ba0cc9700 1 -- 172.21.15.153:0/541873030 wait complete. 2024-04-16T02:27:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:46.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:45 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/541873030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:46.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:45 smithi151 ceph-mon[29003]: mgrmap e17: smithi151.uqqame(active, since 2s) 2024-04-16T02:27:46.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:46.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:47.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:46 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:47.739 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:47.801 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T02:27:48.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:48 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:48.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:48 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:48.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:48 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:48.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:48 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:48.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:48 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.063+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2620503995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 msgr2=0x7f8c70105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.063+0000 7f8c75ab6700 1 --2- 172.21.15.153:0/2620503995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 0x7f8c70105b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8c60009910 tx=0x7f8c60009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.063+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2620503995 shutdown_connections 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.063+0000 7f8c75ab6700 1 --2- 172.21.15.153:0/2620503995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 0x7f8c70105b60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.063+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2620503995 >> 172.21.15.153:0/2620503995 conn(0x7f8c700752b0 msgr2=0x7f8c700756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.064+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2620503995 shutdown_connections 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.064+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2620503995 wait complete. 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.064+0000 7f8c75ab6700 1 Processor -- start 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.064+0000 7f8c75ab6700 1 -- start start 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.064+0000 7f8c75ab6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 0x7f8c70196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:50.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.064+0000 7f8c75ab6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c70197120 con 0x7f8c70105780 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c6f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 0x7f8c70196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c6f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 0x7f8c70196be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:39900/0 (socket says 172.21.15.153:39900) 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c6f7fe700 1 -- 172.21.15.153:0/2407902359 learned_addr learned my addr 172.21.15.153:0/2407902359 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c6f7fe700 1 -- 172.21.15.153:0/2407902359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c600095f0 con 0x7f8c70105780 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c6f7fe700 1 --2- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 0x7f8c70196be0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8c60005b40 tx=0x7f8c60004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c6d7fa700 1 -- 172.21.15.153:0/2407902359 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c6001d070 con 0x7f8c70105780 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c6d7fa700 1 -- 172.21.15.153:0/2407902359 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8c600044d0 con 0x7f8c70105780 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c701972c0 con 0x7f8c70105780 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c6d7fa700 1 -- 172.21.15.153:0/2407902359 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c60003930 con 0x7f8c70105780 2024-04-16T02:27:50.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.065+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c70197680 con 0x7f8c70105780 2024-04-16T02:27:50.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.066+0000 7f8c6d7fa700 1 -- 172.21.15.153:0/2407902359 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f8c60003bf0 con 0x7f8c70105780 2024-04-16T02:27:50.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.066+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c701039c0 con 0x7f8c70105780 2024-04-16T02:27:50.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.066+0000 7f8c6d7fa700 1 --2- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8c5c0384b0 0x7f8c5c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:50.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.067+0000 7f8c6d7fa700 1 -- 172.21.15.153:0/2407902359 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f8c6004c460 con 0x7f8c70105780 2024-04-16T02:27:50.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.067+0000 7f8c67fff700 1 --2- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8c5c0384b0 0x7f8c5c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:50.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.067+0000 7f8c67fff700 1 --2- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8c5c0384b0 0x7f8c5c03a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8c58006fd0 tx=0x7f8c58006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:50.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.072+0000 7f8c6d7fa700 1 -- 172.21.15.153:0/2407902359 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8c6002a540 con 0x7f8c70105780 2024-04-16T02:27:50.289 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:50.289 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:50.289 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:50.289 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:50.290 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:27:50.290 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:27:50.290 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:27:50.290 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: Updating smithi151:/etc/ceph/ceph.conf 2024-04-16T02:27:50.290 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-16T02:27:50.290 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:50.290 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.338+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c700623c0 con 0x7f8c70105780 2024-04-16T02:27:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.338+0000 7f8c6d7fa700 1 -- 172.21.15.153:0/2407902359 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8c6002a360 con 0x7f8c70105780 2024-04-16T02:27:50.340 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:50.340 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.340+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8c5c0384b0 msgr2=0x7f8c5c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.340+0000 7f8c75ab6700 1 --2- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8c5c0384b0 0x7f8c5c03a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8c58006fd0 tx=0x7f8c58006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.340+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 msgr2=0x7f8c70196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.340+0000 7f8c75ab6700 1 --2- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 0x7f8c70196be0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8c60005b40 tx=0x7f8c60004c80 comp rx=0 tx=0).stop 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.340+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 shutdown_connections 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.340+0000 7f8c75ab6700 1 --2- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8c5c0384b0 0x7f8c5c03a970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.340+0000 7f8c75ab6700 1 --2- 172.21.15.153:0/2407902359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c70105780 0x7f8c70196be0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.340+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 >> 172.21.15.153:0/2407902359 conn(0x7f8c700752b0 msgr2=0x7f8c700fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.341+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 shutdown_connections 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:50.341+0000 7f8c75ab6700 1 -- 172.21.15.153:0/2407902359 wait complete. 2024-04-16T02:27:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:51.328 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:27:51.328 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:27:51.328 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:27:51.328 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:27:51.328 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:27:51.329 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/2407902359' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:51.329 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:51.329 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:51.329 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:51.329 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:27:51.329 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-16T02:27:51.329 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:27:52.011 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:52.012 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:52.220 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:27:52.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:52 smithi151 ceph-mon[29003]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:27:52.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:52 smithi151 ceph-mon[29003]: Deploying daemon ceph-exporter.smithi153 on smithi153 2024-04-16T02:27:54.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.070+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/3357607246 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 msgr2=0x7f2a08106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:54.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.070+0000 7f2a0fbc9700 1 --2- 172.21.15.153:0/3357607246 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 0x7f2a08106c10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2a04009910 tx=0x7f2a04009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:54.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.070+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/3357607246 shutdown_connections 2024-04-16T02:27:54.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.070+0000 7f2a0fbc9700 1 --2- 172.21.15.153:0/3357607246 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 0x7f2a08106c10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:54.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.070+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/3357607246 >> 172.21.15.153:0/3357607246 conn(0x7f2a080fc380 msgr2=0x7f2a080fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.070+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/3357607246 shutdown_connections 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.070+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/3357607246 wait complete. 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.071+0000 7f2a0fbc9700 1 Processor -- start 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.071+0000 7f2a0fbc9700 1 -- start start 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.071+0000 7f2a0fbc9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 0x7f2a0819bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.071+0000 7f2a0fbc9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0819bff0 con 0x7f2a08106830 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.071+0000 7f2a0d965700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 0x7f2a0819bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.071+0000 7f2a0d965700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 0x7f2a0819bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:39922/0 (socket says 172.21.15.153:39922) 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.071+0000 7f2a0d965700 1 -- 172.21.15.153:0/2227221876 learned_addr learned my addr 172.21.15.153:0/2227221876 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.071+0000 7f2a0d965700 1 -- 172.21.15.153:0/2227221876 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a040095f0 con 0x7f2a08106830 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.071+0000 7f2a0d965700 1 --2- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 0x7f2a0819bab0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2a04005b40 tx=0x7f2a04004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.072+0000 7f29feffd700 1 -- 172.21.15.153:0/2227221876 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a0401d070 con 0x7f2a08106830 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.072+0000 7f29feffd700 1 -- 172.21.15.153:0/2227221876 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2a040044d0 con 0x7f2a08106830 2024-04-16T02:27:54.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.072+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0819c1f0 con 0x7f2a08106830 2024-04-16T02:27:54.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.072+0000 7f29feffd700 1 -- 172.21.15.153:0/2227221876 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a04003910 con 0x7f2a08106830 2024-04-16T02:27:54.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.072+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a08195b30 con 0x7f2a08106830 2024-04-16T02:27:54.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.073+0000 7f29feffd700 1 -- 172.21.15.153:0/2227221876 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f2a04003b90 con 0x7f2a08106830 2024-04-16T02:27:54.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.073+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a080623c0 con 0x7f2a08106830 2024-04-16T02:27:54.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.073+0000 7f29feffd700 1 --2- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f29f40384b0 0x7f29f403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:54.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.073+0000 7f29feffd700 1 -- 172.21.15.153:0/2227221876 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f2a0404c540 con 0x7f2a08106830 2024-04-16T02:27:54.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.073+0000 7f2a0d164700 1 --2- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f29f40384b0 0x7f29f403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:54.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.074+0000 7f2a0d164700 1 --2- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f29f40384b0 0x7f29f403a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29f8006fd0 tx=0x7f29f8006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:54.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.078+0000 7f29feffd700 1 -- 172.21.15.153:0/2227221876 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2a0402a540 con 0x7f2a08106830 2024-04-16T02:27:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.361+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a081964f0 con 0x7f2a08106830 2024-04-16T02:27:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.362+0000 7f29feffd700 1 -- 172.21.15.153:0/2227221876 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2a0402a360 con 0x7f2a08106830 2024-04-16T02:27:54.363 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:54.363 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f29f40384b0 msgr2=0x7f29f403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 --2- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f29f40384b0 0x7f29f403a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29f8006fd0 tx=0x7f29f8006e40 comp rx=0 tx=0).stop 2024-04-16T02:27:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 msgr2=0x7f2a0819bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 --2- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 0x7f2a0819bab0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2a04005b40 tx=0x7f2a04004c80 comp rx=0 tx=0).stop 2024-04-16T02:27:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 shutdown_connections 2024-04-16T02:27:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 --2- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f29f40384b0 0x7f29f403a970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 --2- 172.21.15.153:0/2227221876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a08106830 0x7f2a0819bab0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 >> 172.21.15.153:0/2227221876 conn(0x7f2a080fc380 msgr2=0x7f2a08108c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 shutdown_connections 2024-04-16T02:27:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:54.364+0000 7f2a0fbc9700 1 -- 172.21.15.153:0/2227221876 wait complete. 2024-04-16T02:27:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:54.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:54 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/2227221876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:56.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:27:56.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:27:56.716 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:27:56.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:56.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:56.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:56.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:56.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:27:56.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T02:27:56.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:27:56.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:56 smithi151 ceph-mon[29003]: Deploying daemon crash.smithi153 on smithi153 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.213+0000 7ffb1490c700 1 -- 172.21.15.153:0/258370346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 msgr2=0x7ffb10105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.213+0000 7ffb1490c700 1 --2- 172.21.15.153:0/258370346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 0x7ffb10105b60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009910 tx=0x7ffb08009c20 comp rx=0 tx=0).stop 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.213+0000 7ffb1490c700 1 -- 172.21.15.153:0/258370346 shutdown_connections 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.213+0000 7ffb1490c700 1 --2- 172.21.15.153:0/258370346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 0x7ffb10105b60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.213+0000 7ffb1490c700 1 -- 172.21.15.153:0/258370346 >> 172.21.15.153:0/258370346 conn(0x7ffb100752b0 msgr2=0x7ffb100756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.214+0000 7ffb1490c700 1 -- 172.21.15.153:0/258370346 shutdown_connections 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.214+0000 7ffb1490c700 1 -- 172.21.15.153:0/258370346 wait complete. 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.214+0000 7ffb1490c700 1 Processor -- start 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.214+0000 7ffb1490c700 1 -- start start 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.214+0000 7ffb1490c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 0x7ffb10195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.214+0000 7ffb1490c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb101962e0 con 0x7ffb10105780 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.215+0000 7ffb0e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 0x7ffb10195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:58.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.215+0000 7ffb0e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 0x7ffb10195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:39944/0 (socket says 172.21.15.153:39944) 2024-04-16T02:27:58.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.215+0000 7ffb0e59c700 1 -- 172.21.15.153:0/2102212542 learned_addr learned my addr 172.21.15.153:0/2102212542 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:27:58.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.215+0000 7ffb0e59c700 1 -- 172.21.15.153:0/2102212542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb080095f0 con 0x7ffb10105780 2024-04-16T02:27:58.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.215+0000 7ffb0e59c700 1 --2- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 0x7ffb10195da0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb08005b40 tx=0x7ffb08004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:58.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.216+0000 7ffb077fe700 1 -- 172.21.15.153:0/2102212542 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb0801d070 con 0x7ffb10105780 2024-04-16T02:27:58.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.216+0000 7ffb077fe700 1 -- 172.21.15.153:0/2102212542 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffb080044d0 con 0x7ffb10105780 2024-04-16T02:27:58.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.216+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb101964e0 con 0x7ffb10105780 2024-04-16T02:27:58.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.216+0000 7ffb077fe700 1 -- 172.21.15.153:0/2102212542 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb08003930 con 0x7ffb10105780 2024-04-16T02:27:58.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.216+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb101968a0 con 0x7ffb10105780 2024-04-16T02:27:58.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.217+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb100623c0 con 0x7ffb10105780 2024-04-16T02:27:58.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.217+0000 7ffb077fe700 1 -- 172.21.15.153:0/2102212542 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7ffb08003bf0 con 0x7ffb10105780 2024-04-16T02:27:58.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.217+0000 7ffb077fe700 1 --2- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ffafc0384b0 0x7ffafc03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:27:58.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.217+0000 7ffb077fe700 1 -- 172.21.15.153:0/2102212542 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ffb0804c460 con 0x7ffb10105780 2024-04-16T02:27:58.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.217+0000 7ffb0dd9b700 1 --2- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ffafc0384b0 0x7ffafc03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:27:58.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.217+0000 7ffb0dd9b700 1 --2- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ffafc0384b0 0x7ffafc03a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800ad30 tx=0x7ffaf80093f0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:27:58.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.222+0000 7ffb077fe700 1 -- 172.21.15.153:0/2102212542 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffb0802a540 con 0x7ffb10105780 2024-04-16T02:27:58.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.497+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb1019a890 con 0x7ffb10105780 2024-04-16T02:27:58.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.497+0000 7ffb077fe700 1 -- 172.21.15.153:0/2102212542 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ffb0802a360 con 0x7ffb10105780 2024-04-16T02:27:58.502 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:27:58.502 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:27:58.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.499+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ffafc0384b0 msgr2=0x7ffafc03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:58.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.499+0000 7ffb1490c700 1 --2- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ffafc0384b0 0x7ffafc03a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800ad30 tx=0x7ffaf80093f0 comp rx=0 tx=0).stop 2024-04-16T02:27:58.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.499+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 msgr2=0x7ffb10195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:27:58.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.499+0000 7ffb1490c700 1 --2- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 0x7ffb10195da0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb08005b40 tx=0x7ffb08004c80 comp rx=0 tx=0).stop 2024-04-16T02:27:58.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.500+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 shutdown_connections 2024-04-16T02:27:58.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.500+0000 7ffb1490c700 1 --2- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ffafc0384b0 0x7ffafc03a970 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:58.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.500+0000 7ffb1490c700 1 --2- 172.21.15.153:0/2102212542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb10105780 0x7ffb10195da0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:27:58.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.500+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 >> 172.21.15.153:0/2102212542 conn(0x7ffb100752b0 msgr2=0x7ffb10067fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:27:58.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.500+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 shutdown_connections 2024-04-16T02:27:58.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:27:58.500+0000 7ffb1490c700 1 -- 172.21.15.153:0/2102212542 wait complete. 2024-04-16T02:27:58.503 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:27:59.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:59.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:27:59.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:58 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/2102212542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:27:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:27:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:00.169 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:28:00.169 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:28:00.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:27:59 smithi151 ceph-mon[29003]: Deploying daemon node-exporter.smithi153 on smithi153 2024-04-16T02:28:00.377 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:28:02.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.872+0000 7fc47c639700 1 -- 172.21.15.153:0/3370915858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 msgr2=0x7fc474106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:02.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.872+0000 7fc47c639700 1 --2- 172.21.15.153:0/3370915858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 0x7fc474106bf0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc470009910 tx=0x7fc470009c20 comp rx=0 tx=0).stop 2024-04-16T02:28:02.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.872+0000 7fc47c639700 1 -- 172.21.15.153:0/3370915858 shutdown_connections 2024-04-16T02:28:02.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.872+0000 7fc47c639700 1 --2- 172.21.15.153:0/3370915858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 0x7fc474106bf0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:02.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.872+0000 7fc47c639700 1 -- 172.21.15.153:0/3370915858 >> 172.21.15.153:0/3370915858 conn(0x7fc4740fc320 msgr2=0x7fc4740fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:28:02.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.872+0000 7fc47c639700 1 -- 172.21.15.153:0/3370915858 shutdown_connections 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.872+0000 7fc47c639700 1 -- 172.21.15.153:0/3370915858 wait complete. 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.873+0000 7fc47c639700 1 Processor -- start 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.873+0000 7fc47c639700 1 -- start start 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.873+0000 7fc47c639700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 0x7fc474195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.873+0000 7fc47c639700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4741962a0 con 0x7fc474106810 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.873+0000 7fc47a3d5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 0x7fc474195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.873+0000 7fc47a3d5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 0x7fc474195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:46048/0 (socket says 172.21.15.153:46048) 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.873+0000 7fc47a3d5700 1 -- 172.21.15.153:0/1044023761 learned_addr learned my addr 172.21.15.153:0/1044023761 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.873+0000 7fc47a3d5700 1 -- 172.21.15.153:0/1044023761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4700095f0 con 0x7fc474106810 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.874+0000 7fc47a3d5700 1 --2- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 0x7fc474195d60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc470005b40 tx=0x7fc470004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.874+0000 7fc46b7fe700 1 -- 172.21.15.153:0/1044023761 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc47001d070 con 0x7fc474106810 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.874+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4741964a0 con 0x7fc474106810 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.874+0000 7fc46b7fe700 1 -- 172.21.15.153:0/1044023761 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc4700044d0 con 0x7fc474106810 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.874+0000 7fc46b7fe700 1 -- 172.21.15.153:0/1044023761 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc470003910 con 0x7fc474106810 2024-04-16T02:28:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.874+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc474196920 con 0x7fc474106810 2024-04-16T02:28:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.875+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4740623c0 con 0x7fc474106810 2024-04-16T02:28:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.875+0000 7fc46b7fe700 1 -- 172.21.15.153:0/1044023761 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7fc47000b960 con 0x7fc474106810 2024-04-16T02:28:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.875+0000 7fc46b7fe700 1 --2- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc4600384b0 0x7fc46003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.875+0000 7fc46b7fe700 1 -- 172.21.15.153:0/1044023761 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc47004c5b0 con 0x7fc474106810 2024-04-16T02:28:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.875+0000 7fc479bd4700 1 --2- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc4600384b0 0x7fc46003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.876+0000 7fc479bd4700 1 --2- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc4600384b0 0x7fc46003a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc464006fd0 tx=0x7fc464006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:02.880+0000 7fc46b7fe700 1 -- 172.21.15.153:0/1044023761 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc47002a540 con 0x7fc474106810 2024-04-16T02:28:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.149+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4741045a0 con 0x7fc474106810 2024-04-16T02:28:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.150+0000 7fc46b7fe700 1 -- 172.21.15.153:0/1044023761 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fc47002a360 con 0x7fc474106810 2024-04-16T02:28:03.151 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:28:03.152 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:28:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc4600384b0 msgr2=0x7fc46003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 --2- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc4600384b0 0x7fc46003a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc464006fd0 tx=0x7fc464006e40 comp rx=0 tx=0).stop 2024-04-16T02:28:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 msgr2=0x7fc474195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 --2- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 0x7fc474195d60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc470005b40 tx=0x7fc470004c80 comp rx=0 tx=0).stop 2024-04-16T02:28:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 shutdown_connections 2024-04-16T02:28:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 --2- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc4600384b0 0x7fc46003a970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 --2- 172.21.15.153:0/1044023761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc474106810 0x7fc474195d60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 >> 172.21.15.153:0/1044023761 conn(0x7fc4740fc320 msgr2=0x7fc474108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:28:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 shutdown_connections 2024-04-16T02:28:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:03.152+0000 7fc47c639700 1 -- 172.21.15.153:0/1044023761 wait complete. 2024-04-16T02:28:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:28:04.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:03 smithi151 ceph-mon[29003]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:04.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:03 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:04.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:03 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/1044023761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:28:04.742 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:28:04.743 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:28:05.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:05.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:05.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:05.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:05.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:28:05.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-16T02:28:05.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:28:05.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:05.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:04 smithi151 ceph-mon[29003]: Deploying daemon mgr.smithi153.esbtzf on smithi153 2024-04-16T02:28:05.516 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:28:06.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:05 smithi151 ceph-mon[29003]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:07.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.477+0000 7f95bbea0700 1 -- 172.21.15.153:0/1315622327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 msgr2=0x7f95b4106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:07.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.477+0000 7f95bbea0700 1 --2- 172.21.15.153:0/1315622327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 0x7f95b4106bf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009910 tx=0x7f95b0009c20 comp rx=0 tx=0).stop 2024-04-16T02:28:07.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.477+0000 7f95bbea0700 1 -- 172.21.15.153:0/1315622327 shutdown_connections 2024-04-16T02:28:07.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.477+0000 7f95bbea0700 1 --2- 172.21.15.153:0/1315622327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 0x7f95b4106bf0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:07.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.477+0000 7f95bbea0700 1 -- 172.21.15.153:0/1315622327 >> 172.21.15.153:0/1315622327 conn(0x7f95b40fc320 msgr2=0x7f95b40fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:28:07.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.477+0000 7f95bbea0700 1 -- 172.21.15.153:0/1315622327 shutdown_connections 2024-04-16T02:28:07.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.477+0000 7f95bbea0700 1 -- 172.21.15.153:0/1315622327 wait complete. 2024-04-16T02:28:07.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.477+0000 7f95bbea0700 1 Processor -- start 2024-04-16T02:28:07.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.478+0000 7f95bbea0700 1 -- start start 2024-04-16T02:28:07.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.478+0000 7f95bbea0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 0x7f95b4195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:07.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.478+0000 7f95bbea0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b41962a0 con 0x7f95b4106810 2024-04-16T02:28:07.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.478+0000 7f95b9c3c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 0x7f95b4195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:07.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.478+0000 7f95b9c3c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 0x7f95b4195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.153:46092/0 (socket says 172.21.15.153:46092) 2024-04-16T02:28:07.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.478+0000 7f95b9c3c700 1 -- 172.21.15.153:0/678154557 learned_addr learned my addr 172.21.15.153:0/678154557 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:28:07.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.478+0000 7f95b9c3c700 1 -- 172.21.15.153:0/678154557 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b00095f0 con 0x7f95b4106810 2024-04-16T02:28:07.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.479+0000 7f95b9c3c700 1 --2- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 0x7f95b4195d60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95b0005b40 tx=0x7f95b0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.479+0000 7f95aaffd700 1 -- 172.21.15.153:0/678154557 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b001d070 con 0x7f95b4106810 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.479+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b41964a0 con 0x7f95b4106810 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.479+0000 7f95aaffd700 1 -- 172.21.15.153:0/678154557 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f95b00044d0 con 0x7f95b4106810 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.479+0000 7f95aaffd700 1 -- 172.21.15.153:0/678154557 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b0003910 con 0x7f95b4106810 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.479+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b4196920 con 0x7f95b4106810 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.480+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b40623c0 con 0x7f95b4106810 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.480+0000 7f95aaffd700 1 -- 172.21.15.153:0/678154557 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f95b0003b90 con 0x7f95b4106810 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.480+0000 7f95aaffd700 1 --2- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f95a00384b0 0x7f95a003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.480+0000 7f95aaffd700 1 -- 172.21.15.153:0/678154557 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f95b004c420 con 0x7f95b4106810 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.480+0000 7f95b943b700 1 --2- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f95a00384b0 0x7f95a003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:07.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.481+0000 7f95b943b700 1 --2- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f95a00384b0 0x7f95a003a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f95a4006fd0 tx=0x7f95a4006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:07.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.485+0000 7f95aaffd700 1 -- 172.21.15.153:0/678154557 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f95b002a540 con 0x7f95b4106810 2024-04-16T02:28:07.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.780+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95b41045a0 con 0x7f95b4106810 2024-04-16T02:28:07.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.781+0000 7f95aaffd700 1 -- 172.21.15.153:0/678154557 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f95b002a360 con 0x7f95b4106810 2024-04-16T02:28:07.782 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:28:07.782 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:25:31.524258Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T02:28:07.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f95a00384b0 msgr2=0x7f95a003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:07.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 --2- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f95a00384b0 0x7f95a003a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f95a4006fd0 tx=0x7f95a4006e40 comp rx=0 tx=0).stop 2024-04-16T02:28:07.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 msgr2=0x7f95b4195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:07.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 --2- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 0x7f95b4195d60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95b0005b40 tx=0x7f95b0004c80 comp rx=0 tx=0).stop 2024-04-16T02:28:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 shutdown_connections 2024-04-16T02:28:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 --2- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f95a00384b0 0x7f95a003a970 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 --2- 172.21.15.153:0/678154557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95b4106810 0x7f95b4195d60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 >> 172.21.15.153:0/678154557 conn(0x7f95b40fc320 msgr2=0x7f95b4108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:28:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 shutdown_connections 2024-04-16T02:28:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:07.783+0000 7f95bbea0700 1 -- 172.21.15.153:0/678154557 wait complete. 2024-04-16T02:28:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-16T02:28:08.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:08.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:08.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:08.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:08.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:08.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:28:08.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:08.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: Deploying daemon mon.smithi153 on smithi153 2024-04-16T02:28:08.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/678154557' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:28:08.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:09.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T02:28:09.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T02:28:10.357 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T02:28:10.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:10 smithi151 ceph-mon[29003]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:11.795 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 podman[36710]: 2024-04-16 02:28:11.720738719 +0000 UTC m=+0.148430474 container create 5f3186f6d9f908f44962f139f9d40629028d9ed81e4938887f9b9e8cbdcf907a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-16T02:28:12.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 podman[36710]: 2024-04-16 02:28:11.964936582 +0000 UTC m=+0.392628341 container init 5f3186f6d9f908f44962f139f9d40629028d9ed81e4938887f9b9e8cbdcf907a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-16T02:28:12.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 podman[36710]: 2024-04-16 02:28:11.968361572 +0000 UTC m=+0.396053343 container start 5f3186f6d9f908f44962f139f9d40629028d9ed81e4938887f9b9e8cbdcf907a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.schema-version=1.0) 2024-04-16T02:28:12.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T02:28:12.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-04-16T02:28:12.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: pidfile_write: ignore empty --pid-file 2024-04-16T02:28:12.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: load: jerasure load: lrc 2024-04-16T02:28:12.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: RocksDB version: 7.9.2 2024-04-16T02:28:12.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Git sha 0 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Compile date 2023-12-11 22:07:34 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: DB SUMMARY 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: DB Session ID: IVAZLMYGB30RD3UVJUZA 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: CURRENT file: CURRENT 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi153/store.db dir, Total Num: 0, files: 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi153/store.db: 000004.log size: 511 ; 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.error_if_exists: 0 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.create_if_missing: 0 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.paranoid_checks: 1 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T02:28:12.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T02:28:12.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.env: 0x558064ea8720 2024-04-16T02:28:12.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T02:28:12.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.info_log: 0x5580670d12e0 2024-04-16T02:28:12.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T02:28:12.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.statistics: (nil) 2024-04-16T02:28:12.152 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.use_fsync: 0 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_log_file_size: 0 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.allow_fallocate: 1 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.use_direct_reads: 0 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.db_log_dir: 2024-04-16T02:28:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.wal_dir: 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.write_buffer_manager: 0x558066342140 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T02:28:12.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.unordered_write: 0 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.row_cache: None 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.wal_filter: None 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.two_write_queues: 0 2024-04-16T02:28:12.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.wal_compression: 0 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.atomic_flush: 0 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.log_readahead_size: 0 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_background_jobs: 2 2024-04-16T02:28:12.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_background_compactions: -1 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_subcompactions: 1 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_open_files: -1 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T02:28:12.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_background_flushes: -1 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Compression algorithms supported: 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: kZSTD supported: 0 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: kXpressCompression supported: 0 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: kZlibCompression supported: 1 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: kSnappyCompression supported: 1 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: kLZ4Compression supported: 1 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: kBZip2Compression supported: 0 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000005 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T02:28:12.158 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.merge_operator: 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_filter: None 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5580670d13e0) 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_type: 0 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_index_type: 0 2024-04-16T02:28:12.159 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_shortening: 1 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: checksum: 4 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: no_block_cache: 0 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache: 0x5580663e31f0 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_options: 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: capacity : 536870912 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_shard_bits : 4 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: strict_capacity_limit : 0 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_compressed: (nil) 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: persistent_cache: (nil) 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size: 4096 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size_deviation: 10 2024-04-16T02:28:12.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_restart_interval: 16 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_block_restart_interval: 1 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: metadata_block_size: 4096 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: partition_filters: 0 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: use_delta_encoding: 1 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: filter_policy: bloomfilter 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: whole_key_filtering: 1 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: verify_compression: 0 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: format_version: 5 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_index_compression: 1 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_align: 0 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_auto_readahead_size: 262144 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: prepopulate_block_cache: 0 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: initial_auto_readahead_size: 8192 2024-04-16T02:28:12.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression: NoCompression 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.num_levels: 7 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T02:28:12.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T02:28:12.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T02:28:12.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T02:28:12.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.table_properties_collectors: 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.inplace_update_support: 0 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.bloom_locality: 0 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.max_successive_merges: 0 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.ttl: 2592000 2024-04-16T02:28:12.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.enable_blob_files: false 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.min_blob_size: 0 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-16T02:28:12.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fb9d9b85-0cc5-4a26-8461-7ecbdf8de661 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713234491998266, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713234491998700, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713234491, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fb9d9b85-0cc5-4a26-8461-7ecbdf8de661", "db_session_id": "IVAZLMYGB30RD3UVJUZA", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713234491998781, "job": 1, "event": "recovery_finished"} 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:11 smithi153 ceph-mon[36747]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi153/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558066480000 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: rocksdb: DB pointer 0x55806646c000 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** DB Stats ** 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T02:28:12.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T02:28:12.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T02:28:12.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Block cache BinnedLRUCache@0x5580663e31f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153 does not exist in monmap, will attempt to join an existing cluster 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: using public_addr v2:172.21.15.153:0/0 -> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: starting mon.smithi153 rank -1 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi153 fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(???) e0 preinit fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).mds e1 new map 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).mds e1 print_map 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: e1 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: legacy client fscid: -1 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: No filesystems configured 2024-04-16T02:28:12.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:12.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: Deploying daemon mon.smithi153 on smithi153 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: from='client.? 172.21.15.153:0/678154557' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: mon.smithi153@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 bash[36710]: 5f3186f6d9f908f44962f139f9d40629028d9ed81e4938887f9b9e8cbdcf907a 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta expand map: {default=false} 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta from 'false' to 'false' 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta expanded map: {default=false} 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta expand map: {default=info} 2024-04-16T02:28:12.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta from 'info' to 'info' 2024-04-16T02:28:12.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta expanded map: {default=info} 2024-04-16T02:28:12.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta expand map: {default=daemon} 2024-04-16T02:28:12.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-16T02:28:12.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta expanded map: {default=daemon} 2024-04-16T02:28:12.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta expand map: {default=debug} 2024-04-16T02:28:12.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta from 'debug' to 'debug' 2024-04-16T02:28:12.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 ceph-mon[36747]: expand_channel_meta expanded map: {default=debug} 2024-04-16T02:28:12.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:12 smithi153 systemd[1]: Started Ceph mon.smithi153 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: mon.smithi151 calling monitor election 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/3493296898' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: mon.smithi153 calling monitor election 2024-04-16T02:28:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: mon.smithi151 is new leader, mons smithi151,smithi153 in quorum (ranks 0,1) 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: monmap e2: 2 mons at {smithi151=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: fsmap 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: mgrmap e17: smithi151.uqqame(active, since 34s) 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/3493296898' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: overall HEALTH_OK 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/3493296898' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/3493296898' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:28:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: mon.smithi151 calling monitor election 2024-04-16T02:28:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:28:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3493296898' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:28:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: mon.smithi153 calling monitor election 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: mon.smithi151 is new leader, mons smithi151,smithi153 in quorum (ranks 0,1) 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: monmap e2: 2 mons at {smithi151=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: fsmap 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: mgrmap e17: smithi151.uqqame(active, since 34s) 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3493296898' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: overall HEALTH_OK 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3493296898' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3493296898' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:28:17.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:17.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:18.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:18 smithi151 ceph-mon[29003]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:18.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:18.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:18 smithi151 ceph-mon[29003]: mgrmap e18: smithi151.uqqame(active, since 34s), standbys: smithi153.esbtzf 2024-04-16T02:28:18.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:18.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi153.esbtzf", "id": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:28:18.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:18.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:28:18.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:18.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:18 smithi153 ceph-mon[36747]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:18 smithi153 ceph-mon[36747]: mgrmap e18: smithi151.uqqame(active, since 34s), standbys: smithi153.esbtzf 2024-04-16T02:28:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi153.esbtzf", "id": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:28:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:28:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:28:18.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.539+0000 7fe9c64d0700 1 -- 172.21.15.153:0/2489918460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a80062a0 msgr2=0x7fe9a8008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:18.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.539+0000 7fe9c64d0700 1 --2- 172.21.15.153:0/2489918460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a80062a0 0x7fe9a8008710 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0100920 tx=0x7fe9b000fe90 comp rx=0 tx=0).stop 2024-04-16T02:28:18.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.539+0000 7fe9c64d0700 1 -- 172.21.15.153:0/2489918460 shutdown_connections 2024-04-16T02:28:18.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.539+0000 7fe9c64d0700 1 --2- 172.21.15.153:0/2489918460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a80062a0 0x7fe9a8008710 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:18.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.539+0000 7fe9c64d0700 1 -- 172.21.15.153:0/2489918460 >> 172.21.15.153:0/2489918460 conn(0x7fe9c00fb390 msgr2=0x7fe9c00fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:28:18.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.539+0000 7fe9c64d0700 1 -- 172.21.15.153:0/2489918460 shutdown_connections 2024-04-16T02:28:18.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.539+0000 7fe9c64d0700 1 -- 172.21.15.153:0/2489918460 wait complete. 2024-04-16T02:28:18.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9c64d0700 1 Processor -- start 2024-04-16T02:28:18.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9c64d0700 1 -- start start 2024-04-16T02:28:18.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9c64d0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a80062a0 0x7fe9c0194ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9c64d0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 0x7fe9c01998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9c64d0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0195a50 con 0x7fe9a80062a0 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9c64d0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0195bc0 con 0x7fe9c0195430 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9bf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 0x7fe9c01998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9bf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 0x7fe9c01998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48874/0 (socket says 172.21.15.153:48874) 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9bf7fe700 1 -- 172.21.15.153:0/3544196462 learned_addr learned my addr 172.21.15.153:0/3544196462 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9bf7fe700 1 -- 172.21.15.153:0/3544196462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 msgr2=0x7fe9c01998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9bf7fe700 1 -- 172.21.15.153:0/3544196462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 msgr2=0x7fe9c01998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9bf7fe700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 0x7fe9c01998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9bffff700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a80062a0 0x7fe9c0194ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.540+0000 7fe9bf7fe700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 0x7fe9c01998a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.541+0000 7fe9bffff700 1 -- 172.21.15.153:0/3544196462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 msgr2=0x7fe9c01998a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.541+0000 7fe9bffff700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 0x7fe9c01998a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:18.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.541+0000 7fe9bffff700 1 -- 172.21.15.153:0/3544196462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b000f8e0 con 0x7fe9a80062a0 2024-04-16T02:28:18.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.541+0000 7fe9bffff700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a80062a0 0x7fe9c0194ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0010010 tx=0x7fe9b001d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:18.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.541+0000 7fe9bd7fa700 1 -- 172.21.15.153:0/3544196462 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9b0005220 con 0x7fe9a80062a0 2024-04-16T02:28:18.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.541+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c0199ea0 con 0x7fe9a80062a0 2024-04-16T02:28:18.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.541+0000 7fe9bd7fa700 1 -- 172.21.15.153:0/3544196462 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe9b0003680 con 0x7fe9a80062a0 2024-04-16T02:28:18.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.541+0000 7fe9bd7fa700 1 -- 172.21.15.153:0/3544196462 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9b0011740 con 0x7fe9a80062a0 2024-04-16T02:28:18.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.541+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c019a3f0 con 0x7fe9a80062a0 2024-04-16T02:28:18.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.543+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c019a720 con 0x7fe9a80062a0 2024-04-16T02:28:18.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.543+0000 7fe9bd7fa700 1 -- 172.21.15.153:0/3544196462 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7fe9b0011a00 con 0x7fe9a80062a0 2024-04-16T02:28:18.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.544+0000 7fe9bd7fa700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe9ac06c560 0x7fe9ac06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:18.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.544+0000 7fe9bd7fa700 1 -- 172.21.15.153:0/3544196462 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fe9b008d680 con 0x7fe9a80062a0 2024-04-16T02:28:18.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.544+0000 7fe9bf7fe700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe9ac06c560 0x7fe9ac06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:18.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.544+0000 7fe9bf7fe700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe9ac06c560 0x7fe9ac06ea20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009180 tx=0x7fe9b4008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:18.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.549+0000 7fe9bd7fa700 1 -- 172.21.15.153:0/3544196462 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe9b00590a0 con 0x7fe9a80062a0 2024-04-16T02:28:18.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.823+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9c019fb40 con 0x7fe9a80062a0 2024-04-16T02:28:18.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.824+0000 7fe9bd7fa700 1 -- 172.21.15.153:0/3544196462 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1026 (secure 0 0 0) 0x7fe9b008ce40 con 0x7fe9a80062a0 2024-04-16T02:28:18.824 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-16T02:28:18.825 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","modified":"2024-04-16T02:28:12.026363Z","created":"2024-04-16T02:25:31.524258Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi151","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-16T02:28:18.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe9ac06c560 msgr2=0x7fe9ac06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:18.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe9ac06c560 0x7fe9ac06ea20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009180 tx=0x7fe9b4008040 comp rx=0 tx=0).stop 2024-04-16T02:28:18.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a80062a0 msgr2=0x7fe9c0194ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a80062a0 0x7fe9c0194ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0010010 tx=0x7fe9b001d9b0 comp rx=0 tx=0).stop 2024-04-16T02:28:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 shutdown_connections 2024-04-16T02:28:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe9ac06c560 0x7fe9ac06ea20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a80062a0 0x7fe9c0194ef0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 --2- 172.21.15.153:0/3544196462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9c0195430 0x7fe9c01998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 >> 172.21.15.153:0/3544196462 conn(0x7fe9c00fb390 msgr2=0x7fe9c0101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:28:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 shutdown_connections 2024-04-16T02:28:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:28:18.827+0000 7fe9c64d0700 1 -- 172.21.15.153:0/3544196462 wait complete. 2024-04-16T02:28:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-04-16T02:28:19.451 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:19 smithi153 ceph-mon[36747]: from='client.? 172.21.15.153:0/3544196462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:28:19.472 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-16T02:28:19.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph config generate-minimal-conf 2024-04-16T02:28:19.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:19 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/3544196462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T02:28:19.693 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:28:20.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:20 smithi151 ceph-mon[29003]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:20.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:20 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:20.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:20 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:20.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:20 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:20.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:20 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:28:20.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:20 smithi153 ceph-mon[36747]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:20.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:20 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:20.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:20 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:20.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:20 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:20.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:20 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:28:21.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.370+0000 7fc99d03d700 1 -- 172.21.15.151:0/1927351412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 msgr2=0x7fc9980ff500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:21.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.370+0000 7fc99d03d700 1 --2- 172.21.15.151:0/1927351412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 0x7fc9980ff500 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc980009910 tx=0x7fc980009c20 comp rx=0 tx=0).stop 2024-04-16T02:28:21.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.370+0000 7fc99d03d700 1 -- 172.21.15.151:0/1927351412 shutdown_connections 2024-04-16T02:28:21.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.370+0000 7fc99d03d700 1 --2- 172.21.15.151:0/1927351412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 0x7fc9980ff500 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:21.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.370+0000 7fc99d03d700 1 -- 172.21.15.151:0/1927351412 >> 172.21.15.151:0/1927351412 conn(0x7fc998075df0 msgr2=0x7fc998076200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:28:21.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.370+0000 7fc99d03d700 1 -- 172.21.15.151:0/1927351412 shutdown_connections 2024-04-16T02:28:21.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.370+0000 7fc99d03d700 1 -- 172.21.15.151:0/1927351412 wait complete. 2024-04-16T02:28:21.372 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: Updating smithi151:/etc/ceph/ceph.conf 2024-04-16T02:28:21.372 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-16T02:28:21.372 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.372 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.373 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.373 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.373 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.373 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:28:21.373 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:28:21.373 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:21 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:21.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.371+0000 7fc99d03d700 1 Processor -- start 2024-04-16T02:28:21.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.371+0000 7fc99d03d700 1 -- start start 2024-04-16T02:28:21.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc99d03d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 0x7fc9981000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc99d03d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9981005e0 0x7fc998100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc99d03d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9981a20a0 con 0x7fc9980ff120 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc99d03d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9981a21e0 con 0x7fc9981005e0 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc996d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 0x7fc9981000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc996d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 0x7fc9981000a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38202/0 (socket says 172.21.15.151:38202) 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc996d9d700 1 -- 172.21.15.151:0/2439759583 learned_addr learned my addr 172.21.15.151:0/2439759583 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc996d9d700 1 -- 172.21.15.151:0/2439759583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9981005e0 msgr2=0x7fc998100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc996d9d700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9981005e0 0x7fc998100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc996d9d700 1 -- 172.21.15.151:0/2439759583 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9800095f0 con 0x7fc9980ff120 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.372+0000 7fc996d9d700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 0x7fc9981000a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc9800092e0 tx=0x7fc98000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.373+0000 7fc98ffff700 1 -- 172.21.15.151:0/2439759583 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc98001d070 con 0x7fc9980ff120 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.373+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9981a2320 con 0x7fc9980ff120 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.373+0000 7fc98ffff700 1 -- 172.21.15.151:0/2439759583 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc980005140 con 0x7fc9980ff120 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.373+0000 7fc98ffff700 1 -- 172.21.15.151:0/2439759583 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc980017440 con 0x7fc9980ff120 2024-04-16T02:28:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.373+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9981a27c0 con 0x7fc9980ff120 2024-04-16T02:28:21.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.374+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9980623c0 con 0x7fc9980ff120 2024-04-16T02:28:21.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.374+0000 7fc98ffff700 1 -- 172.21.15.151:0/2439759583 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7fc98000f460 con 0x7fc9980ff120 2024-04-16T02:28:21.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.375+0000 7fc98ffff700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc98406c5c0 0x7fc98406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:21.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.375+0000 7fc98ffff700 1 -- 172.21.15.151:0/2439759583 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc98008bb80 con 0x7fc9980ff120 2024-04-16T02:28:21.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.375+0000 7fc99659c700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc98406c5c0 0x7fc98406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:21.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.375+0000 7fc99659c700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc98406c5c0 0x7fc98406ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9880098e0 tx=0x7fc988006dd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:21.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.380+0000 7fc98ffff700 1 -- 172.21.15.151:0/2439759583 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc980057440 con 0x7fc9980ff120 2024-04-16T02:28:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: Updating smithi151:/etc/ceph/ceph.conf 2024-04-16T02:28:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-16T02:28:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:21.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:28:21.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:28:21.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:21 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:21.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.572+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc9981a2bf0 con 0x7fc9980ff120 2024-04-16T02:28:21.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.573+0000 7fc98ffff700 1 -- 172.21.15.151:0/2439759583 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7fc98008b340 con 0x7fc9980ff120 2024-04-16T02:28:21.573 INFO:teuthology.orchestra.run.smithi151.stdout:# minimal ceph.conf for 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:28:21.574 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-04-16T02:28:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:28:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: mon_host = [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-04-16T02:28:21.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc98406c5c0 msgr2=0x7fc98406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:21.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc98406c5c0 0x7fc98406ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9880098e0 tx=0x7fc988006dd0 comp rx=0 tx=0).stop 2024-04-16T02:28:21.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 msgr2=0x7fc9981000a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:21.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 0x7fc9981000a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc9800092e0 tx=0x7fc98000bda0 comp rx=0 tx=0).stop 2024-04-16T02:28:21.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 shutdown_connections 2024-04-16T02:28:21.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc98406c5c0 0x7fc98406ea80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:21.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9980ff120 0x7fc9981000a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:21.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 --2- 172.21.15.151:0/2439759583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9981005e0 0x7fc998100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:21.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 >> 172.21.15.151:0/2439759583 conn(0x7fc998075df0 msgr2=0x7fc998106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:28:21.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 shutdown_connections 2024-04-16T02:28:21.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:21.575+0000 7fc99d03d700 1 -- 172.21.15.151:0/2439759583 wait complete. 2024-04-16T02:28:22.087 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-16T02:28:22.088 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:28:22.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.conf 2024-04-16T02:28:22.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:22 smithi151 ceph-mon[29003]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:28:22.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:22 smithi151 ceph-mon[29003]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:28:22.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:22 smithi151 ceph-mon[29003]: Reconfiguring mon.smithi151 (unknown last config time)... 2024-04-16T02:28:22.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:22 smithi151 ceph-mon[29003]: Reconfiguring daemon mon.smithi151 on smithi151 2024-04-16T02:28:22.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:22 smithi151 ceph-mon[29003]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:22.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:22 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2439759583' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:22.159 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:28:22.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:28:22.229 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:28:22.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-04-16T02:28:22.266 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:28:22.266 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:28:22.340 INFO:tasks.cephadm:Deploying OSDs... 2024-04-16T02:28:22.340 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:28:22.340 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T02:28:22.356 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T02:28:22.356 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-04-16T02:28:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T02:28:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T02:28:22.411 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-16T02:28:22.411 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:28:22.411 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T02:28:22.411 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-04-16 02:27:34.458160027 +0000 2024-04-16T02:28:22.411 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-04-16 02:22:08.674035745 +0000 2024-04-16T02:28:22.411 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-04-16 02:22:08.674035745 +0000 2024-04-16T02:28:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-04-16 02:22:08.674035745 +0000 2024-04-16T02:28:22.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T02:28:22.481 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-04-16T02:28:22.482 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-04-16T02:28:22.482 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.00018955 s, 2.7 MB/s 2024-04-16T02:28:22.483 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T02:28:22.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:22 smithi153 ceph-mon[36747]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:28:22.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:22 smithi153 ceph-mon[36747]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:28:22.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:22 smithi153 ceph-mon[36747]: Reconfiguring mon.smithi151 (unknown last config time)... 2024-04-16T02:28:22.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:22 smithi153 ceph-mon[36747]: Reconfiguring daemon mon.smithi151 on smithi151 2024-04-16T02:28:22.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:22 smithi153 ceph-mon[36747]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:22.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:22 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2439759583' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:22.541 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-04-16T02:28:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T02:28:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T02:28:22.597 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-16T02:28:22.597 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:28:22.597 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T02:28:22.597 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-04-16 02:27:34.458160027 +0000 2024-04-16T02:28:22.597 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-04-16 02:22:09.022031164 +0000 2024-04-16T02:28:22.597 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-04-16 02:22:09.022031164 +0000 2024-04-16T02:28:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-04-16 02:22:09.022031164 +0000 2024-04-16T02:28:22.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T02:28:22.664 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-04-16T02:28:22.664 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-04-16T02:28:22.664 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000182743 s, 2.8 MB/s 2024-04-16T02:28:22.665 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T02:28:22.721 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-04-16T02:28:22.777 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T02:28:22.777 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T02:28:22.777 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-16T02:28:22.777 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:28:22.777 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T02:28:22.777 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-04-16 02:27:34.459160014 +0000 2024-04-16T02:28:22.777 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-04-16 02:22:09.361026702 +0000 2024-04-16T02:28:22.777 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-04-16 02:22:09.361026702 +0000 2024-04-16T02:28:22.777 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-04-16 02:22:09.361026702 +0000 2024-04-16T02:28:22.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T02:28:22.845 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-04-16T02:28:22.845 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-04-16T02:28:22.845 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000195499 s, 2.6 MB/s 2024-04-16T02:28:22.847 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T02:28:22.903 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-04-16T02:28:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T02:28:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T02:28:22.959 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-16T02:28:22.959 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:28:22.959 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T02:28:22.959 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-04-16 02:27:34.459160014 +0000 2024-04-16T02:28:22.959 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-04-16 02:22:09.694022318 +0000 2024-04-16T02:28:22.959 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-04-16 02:22:09.694022318 +0000 2024-04-16T02:28:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-04-16 02:22:09.694022318 +0000 2024-04-16T02:28:22.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T02:28:23.025 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-04-16T02:28:23.025 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-04-16T02:28:23.025 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000232435 s, 2.2 MB/s 2024-04-16T02:28:23.026 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T02:28:23.083 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:28:23.083 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T02:28:23.103 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T02:28:23.103 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-04-16T02:28:23.161 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T02:28:23.161 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T02:28:23.162 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-16T02:28:23.162 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:28:23.162 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T02:28:23.162 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-16 02:28:10.659489838 +0000 2024-04-16T02:28:23.162 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-16 02:22:15.487188258 +0000 2024-04-16T02:28:23.162 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-16 02:22:15.487188258 +0000 2024-04-16T02:28:23.162 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-16 02:22:15.487188258 +0000 2024-04-16T02:28:23.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T02:28:23.232 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-16T02:28:23.232 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-16T02:28:23.232 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000286468 s, 1.8 MB/s 2024-04-16T02:28:23.234 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T02:28:23.293 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-04-16T02:28:23.350 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T02:28:23.350 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T02:28:23.350 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-16T02:28:23.350 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:28:23.350 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T02:28:23.350 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-16 02:28:10.659489838 +0000 2024-04-16T02:28:23.350 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-16 02:22:15.838181645 +0000 2024-04-16T02:28:23.350 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-16 02:22:15.838181645 +0000 2024-04-16T02:28:23.351 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-16 02:22:15.838181645 +0000 2024-04-16T02:28:23.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T02:28:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-16T02:28:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-16T02:28:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000243689 s, 2.1 MB/s 2024-04-16T02:28:23.420 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T02:28:23.478 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-04-16T02:28:23.534 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T02:28:23.534 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T02:28:23.534 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-16T02:28:23.535 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:28:23.535 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T02:28:23.535 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-16 02:28:10.660489820 +0000 2024-04-16T02:28:23.535 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-16 02:22:16.200174824 +0000 2024-04-16T02:28:23.535 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-16 02:22:16.200174824 +0000 2024-04-16T02:28:23.535 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-16 02:22:16.200174824 +0000 2024-04-16T02:28:23.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T02:28:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-16T02:28:23.604 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-16T02:28:23.604 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000194277 s, 2.6 MB/s 2024-04-16T02:28:23.605 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T02:28:23.662 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-04-16T02:28:23.718 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T02:28:23.718 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T02:28:23.718 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-16T02:28:23.718 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:28:23.718 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T02:28:23.718 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-16 02:28:10.660489820 +0000 2024-04-16T02:28:23.719 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-16 02:22:16.545168324 +0000 2024-04-16T02:28:23.719 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-16 02:22:16.545168324 +0000 2024-04-16T02:28:23.719 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-16 02:22:16.545168324 +0000 2024-04-16T02:28:23.719 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T02:28:23.787 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-16T02:28:23.787 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-16T02:28:23.787 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000187587 s, 2.7 MB/s 2024-04-16T02:28:23.788 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T02:28:23.845 INFO:tasks.cephadm:Deploying osd.0 on smithi151 with /dev/vg_nvme/lv_4... 2024-04-16T02:28:23.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-16T02:28:24.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:24 smithi151 ceph-mon[29003]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:24.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:24 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:24.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:24 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:24.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:24 smithi151 ceph-mon[29003]: Reconfiguring mgr.smithi151.uqqame (unknown last config time)... 2024-04-16T02:28:24.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:24 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi151.uqqame", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:28:24.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:24 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:28:24.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:24 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:24.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:24 smithi151 ceph-mon[29003]: Reconfiguring daemon mgr.smithi151.uqqame on smithi151 2024-04-16T02:28:24.466 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:28:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:24 smithi153 ceph-mon[36747]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:24 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:24 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:24 smithi153 ceph-mon[36747]: Reconfiguring mgr.smithi151.uqqame (unknown last config time)... 2024-04-16T02:28:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:24 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi151.uqqame", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:28:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:24 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:28:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:24 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:24.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:24 smithi153 ceph-mon[36747]: Reconfiguring daemon mgr.smithi151.uqqame on smithi151 2024-04-16T02:28:25.558 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:25 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:25.559 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:25 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:25.559 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:25 smithi151 ceph-mon[29003]: Reconfiguring ceph-exporter.smithi151 (monmap changed)... 2024-04-16T02:28:25.559 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:25 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:28:25.559 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:25 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:25.559 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:25 smithi151 ceph-mon[29003]: Reconfiguring daemon ceph-exporter.smithi151 on smithi151 2024-04-16T02:28:25.559 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:25 smithi151 ceph-mon[29003]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:26.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:25 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:26.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:25 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:26.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:25 smithi153 ceph-mon[36747]: Reconfiguring ceph-exporter.smithi151 (monmap changed)... 2024-04-16T02:28:26.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:25 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:28:26.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:25 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:26.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:25 smithi153 ceph-mon[36747]: Reconfiguring daemon ceph-exporter.smithi151 on smithi151 2024-04-16T02:28:26.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:25 smithi153 ceph-mon[36747]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:27.431 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-16T02:28:27.431 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-16T02:28:27.431 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-04-16T02:28:27.431 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-04-16T02:28:27.431 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114802 s, 913 MB/s 2024-04-16T02:28:27.431 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-04-16T02:28:28.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:27 smithi151 ceph-mon[29003]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:28.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:28.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:28.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:27 smithi151 ceph-mon[29003]: Reconfiguring crash.smithi151 (monmap changed)... 2024-04-16T02:28:28.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi151", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:28:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:27 smithi151 ceph-mon[29003]: Reconfiguring daemon crash.smithi151 on smithi151 2024-04-16T02:28:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:28:28.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:27 smithi153 ceph-mon[36747]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:28.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:28.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:28.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:27 smithi153 ceph-mon[36747]: Reconfiguring crash.smithi151 (monmap changed)... 2024-04-16T02:28:28.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi151", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:28:28.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:28.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:27 smithi153 ceph-mon[36747]: Reconfiguring daemon crash.smithi151 on smithi151 2024-04-16T02:28:28.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:28:28.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_4 2024-04-16T02:28:29.030 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:28:30.127 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:29 smithi151 ceph-mon[29003]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:30.128 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:30.128 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:30.128 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:29 smithi151 ceph-mon[29003]: Reconfiguring alertmanager.smithi151 (dependencies changed)... 2024-04-16T02:28:30.128 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:29 smithi151 ceph-mon[29003]: Reconfiguring daemon alertmanager.smithi151 on smithi151 2024-04-16T02:28:30.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:29 smithi153 ceph-mon[36747]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:30.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:30.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:30.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:29 smithi153 ceph-mon[36747]: Reconfiguring alertmanager.smithi151 (dependencies changed)... 2024-04-16T02:28:30.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:29 smithi153 ceph-mon[36747]: Reconfiguring daemon alertmanager.smithi151 on smithi151 2024-04-16T02:28:32.211 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:31 smithi151 ceph-mon[29003]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:32.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:31 smithi153 ceph-mon[36747]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:32.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.321+0000 7f7741ea9700 1 -- 172.21.15.151:0/802670408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 msgr2=0x7f773c100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:32.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.321+0000 7f7741ea9700 1 --2- 172.21.15.151:0/802670408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 0x7f773c100ff0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f772c009910 tx=0x7f772c009c20 comp rx=0 tx=0).stop 2024-04-16T02:28:32.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.321+0000 7f7741ea9700 1 -- 172.21.15.151:0/802670408 shutdown_connections 2024-04-16T02:28:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.321+0000 7f7741ea9700 1 --2- 172.21.15.151:0/802670408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f773c101dd0 0x7f773c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.321+0000 7f7741ea9700 1 --2- 172.21.15.151:0/802670408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 0x7f773c100ff0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.321+0000 7f7741ea9700 1 -- 172.21.15.151:0/802670408 >> 172.21.15.151:0/802670408 conn(0x7f773c0fc170 msgr2=0x7f773c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:28:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.321+0000 7f7741ea9700 1 -- 172.21.15.151:0/802670408 shutdown_connections 2024-04-16T02:28:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.321+0000 7f7741ea9700 1 -- 172.21.15.151:0/802670408 wait complete. 2024-04-16T02:28:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f7741ea9700 1 Processor -- start 2024-04-16T02:28:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f7741ea9700 1 -- start start 2024-04-16T02:28:32.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f7741ea9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 0x7f773c196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f7741ea9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f773c101dd0 0x7f773c196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f7741ea9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c196f60 con 0x7f773c100bd0 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f7741ea9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c1970a0 con 0x7f773c101dd0 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f773b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 0x7f773c196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f773b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 0x7f773c196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58070/0 (socket says 172.21.15.151:58070) 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f773b7fe700 1 -- 172.21.15.151:0/1068702668 learned_addr learned my addr 172.21.15.151:0/1068702668 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.322+0000 7f773affd700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f773c101dd0 0x7f773c196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.323+0000 7f773b7fe700 1 -- 172.21.15.151:0/1068702668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f773c101dd0 msgr2=0x7f773c196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.323+0000 7f773b7fe700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f773c101dd0 0x7f773c196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.323+0000 7f773b7fe700 1 -- 172.21.15.151:0/1068702668 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c0095f0 con 0x7f773c100bd0 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.323+0000 7f773b7fe700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 0x7f773c196400 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f772c00bda0 tx=0x7f772c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.323+0000 7f7738ff9700 1 -- 172.21.15.151:0/1068702668 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f772c01d070 con 0x7f773c100bd0 2024-04-16T02:28:32.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.323+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c19baf0 con 0x7f773c100bd0 2024-04-16T02:28:32.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.323+0000 7f7738ff9700 1 -- 172.21.15.151:0/1068702668 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f772c004c50 con 0x7f773c100bd0 2024-04-16T02:28:32.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.323+0000 7f7738ff9700 1 -- 172.21.15.151:0/1068702668 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f772c00f690 con 0x7f773c100bd0 2024-04-16T02:28:32.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.323+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c19c060 con 0x7f773c100bd0 2024-04-16T02:28:32.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.324+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c1097b0 con 0x7f773c100bd0 2024-04-16T02:28:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.324+0000 7f7738ff9700 1 -- 172.21.15.151:0/1068702668 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f772c0229b0 con 0x7f773c100bd0 2024-04-16T02:28:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.325+0000 7f7738ff9700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f772406c460 0x7f772406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:28:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.325+0000 7f7738ff9700 1 -- 172.21.15.151:0/1068702668 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f772c08be40 con 0x7f773c100bd0 2024-04-16T02:28:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.325+0000 7f773affd700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f772406c460 0x7f772406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:28:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.325+0000 7f773affd700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f772406c460 0x7f772406e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7730006fd0 tx=0x7f7730008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:28:32.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.329+0000 7f7738ff9700 1 -- 172.21.15.151:0/1068702668 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f772c05b360 con 0x7f773c100bd0 2024-04-16T02:28:32.532 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:28:32.532+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f773c19c7b0 con 0x7f772406c460 2024-04-16T02:28:33.232 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:32 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:28:33.233 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:32 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:28:33.233 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:32 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:33.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:32 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:28:33.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:32 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:28:33.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:32 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:34.138 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:33 smithi151 ceph-mon[29003]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:28:34.138 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:33 smithi151 ceph-mon[29003]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:34.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:33 smithi153 ceph-mon[36747]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:28:34.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:33 smithi153 ceph-mon[36747]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:35.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:35 smithi151 ceph-mon[29003]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:36.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:35 smithi153 ceph-mon[36747]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:37.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:37 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:37.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:37 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:37.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:37 smithi151 ceph-mon[29003]: Reconfiguring grafana.smithi151 (dependencies changed)... 2024-04-16T02:28:37.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:37 smithi151 ceph-mon[29003]: Reconfiguring daemon grafana.smithi151 on smithi151 2024-04-16T02:28:37.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:37 smithi151 ceph-mon[29003]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:37.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:37 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:37.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:37 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:37.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:37 smithi153 ceph-mon[36747]: Reconfiguring grafana.smithi151 (dependencies changed)... 2024-04-16T02:28:37.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:37 smithi153 ceph-mon[36747]: Reconfiguring daemon grafana.smithi151 on smithi151 2024-04-16T02:28:37.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:37 smithi153 ceph-mon[36747]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:38.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:38 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3479494299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8da4d50b-626b-4cf3-9524-ea84e7733113"}]: dispatch 2024-04-16T02:28:38.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:38 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3479494299' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8da4d50b-626b-4cf3-9524-ea84e7733113"}]': finished 2024-04-16T02:28:38.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:38 smithi151 ceph-mon[29003]: osdmap e6: 1 total, 0 up, 1 in 2024-04-16T02:28:38.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:38 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:28:38.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:38 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/583308498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:28:38.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:38 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3479494299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8da4d50b-626b-4cf3-9524-ea84e7733113"}]: dispatch 2024-04-16T02:28:38.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:38 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3479494299' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8da4d50b-626b-4cf3-9524-ea84e7733113"}]': finished 2024-04-16T02:28:38.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:38 smithi153 ceph-mon[36747]: osdmap e6: 1 total, 0 up, 1 in 2024-04-16T02:28:38.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:38 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:28:38.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:38 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/583308498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:28:39.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:39 smithi151 ceph-mon[29003]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:39.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:39 smithi153 ceph-mon[36747]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:42.056 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:41 smithi151 ceph-mon[29003]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:42.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:41 smithi153 ceph-mon[36747]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:43.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:42 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:28:43.237 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:28:44.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:43 smithi151 ceph-mon[29003]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:44.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:43 smithi153 ceph-mon[36747]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:45.951 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:45 smithi151 ceph-mon[29003]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:46.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:45 smithi153 ceph-mon[36747]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:47.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:47 smithi151 ceph-mon[29003]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:47.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:47 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:47.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:47 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:47.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:47 smithi151 ceph-mon[29003]: Reconfiguring prometheus.smithi151 (dependencies changed)... 2024-04-16T02:28:48.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:47 smithi153 ceph-mon[36747]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:48.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:47 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:48.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:47 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:48.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:47 smithi153 ceph-mon[36747]: Reconfiguring prometheus.smithi151 (dependencies changed)... 2024-04-16T02:28:49.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:48 smithi151 ceph-mon[29003]: Reconfiguring daemon prometheus.smithi151 on smithi151 2024-04-16T02:28:49.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:48 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T02:28:49.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:48 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:49.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:48 smithi153 ceph-mon[36747]: Reconfiguring daemon prometheus.smithi151 on smithi151 2024-04-16T02:28:49.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:48 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T02:28:49.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:48 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:49.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:49 smithi151 ceph-mon[29003]: Deploying daemon osd.0 on smithi151 2024-04-16T02:28:49.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:49 smithi151 ceph-mon[29003]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:50.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:49 smithi153 ceph-mon[36747]: Deploying daemon osd.0 on smithi151 2024-04-16T02:28:50.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:49 smithi153 ceph-mon[36747]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:52.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:51 smithi151 ceph-mon[29003]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:52.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:51 smithi153 ceph-mon[36747]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:53.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:53 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:53.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:53 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:53.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:53 smithi153 ceph-mon[36747]: Reconfiguring ceph-exporter.smithi153 (monmap changed)... 2024-04-16T02:28:53.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:53 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:28:53.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:53 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:53.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:53 smithi153 ceph-mon[36747]: Reconfiguring daemon ceph-exporter.smithi153 on smithi153 2024-04-16T02:28:53.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:53 smithi153 ceph-mon[36747]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:53.808 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:53 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:53.808 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:53 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:53.808 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:53 smithi151 ceph-mon[29003]: Reconfiguring ceph-exporter.smithi153 (monmap changed)... 2024-04-16T02:28:53.808 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:53 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:28:53.808 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:53 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:53.809 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:53 smithi151 ceph-mon[29003]: Reconfiguring daemon ceph-exporter.smithi153 on smithi153 2024-04-16T02:28:53.809 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:53 smithi151 ceph-mon[29003]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:55.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:55.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:55.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:28:55.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:55.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:55.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:55.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:28:55.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:56.219 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:56 smithi151 ceph-mon[29003]: Reconfiguring crash.smithi153 (monmap changed)... 2024-04-16T02:28:56.219 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:56 smithi151 ceph-mon[29003]: Reconfiguring daemon crash.smithi153 on smithi153 2024-04-16T02:28:56.220 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:56 smithi151 ceph-mon[29003]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:56.220 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:56.220 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:56.220 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:28:56.220 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:28:56.220 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:56.468 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:56 smithi153 ceph-mon[36747]: Reconfiguring crash.smithi153 (monmap changed)... 2024-04-16T02:28:56.468 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:56 smithi153 ceph-mon[36747]: Reconfiguring daemon crash.smithi153 on smithi153 2024-04-16T02:28:56.468 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:56 smithi153 ceph-mon[36747]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:56.468 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:56.468 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:56.469 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:28:56.469 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:28:56.469 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:57.323 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:57 smithi151 ceph-mon[29003]: Reconfiguring mgr.smithi153.esbtzf (monmap changed)... 2024-04-16T02:28:57.323 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:57 smithi151 ceph-mon[29003]: Reconfiguring daemon mgr.smithi153.esbtzf on smithi153 2024-04-16T02:28:57.323 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:57.323 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:57.323 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:28:57.323 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:28:57.323 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:57.353 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:57 smithi153 ceph-mon[36747]: Reconfiguring mgr.smithi153.esbtzf (monmap changed)... 2024-04-16T02:28:57.353 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:57 smithi153 ceph-mon[36747]: Reconfiguring daemon mgr.smithi153.esbtzf on smithi153 2024-04-16T02:28:57.353 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:57.354 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:57.354 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:28:57.354 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:28:57.354 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:28:58.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: Reconfiguring mon.smithi153 (monmap changed)... 2024-04-16T02:28:58.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: Reconfiguring daemon mon.smithi153 on smithi153 2024-04-16T02:28:58.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:58.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:28:58.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T02:28:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T02:28:58.359 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: Reconfiguring mon.smithi153 (monmap changed)... 2024-04-16T02:28:58.360 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: Reconfiguring daemon mon.smithi153 on smithi153 2024-04-16T02:28:58.360 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:28:58.360 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:28:58.360 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:58.360 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:58.360 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:58.360 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:58.360 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T02:28:58.360 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T02:28:59.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:59.384 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:28:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:28:59.437 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T02:28:59.437 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T02:28:59.437 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:59.437 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T02:28:59.437 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T02:28:59.437 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T02:28:59.437 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi151.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T02:28:59.437 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:59.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:28:59.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:28:59.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T02:28:59.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi151.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T02:28:59.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:28:59.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:28:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:29:00.388 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:00 smithi151 ceph-mon[29003]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:29:00.388 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:00 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:00.388 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:00 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:00.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:00 smithi153 ceph-mon[36747]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:29:00.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:00 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:00.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:00 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:01.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.270+0000 7f7738ff9700 1 -- 172.21.15.151:0/1068702668 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f773c19c7b0 con 0x7f772406c460 2024-04-16T02:29:01.274 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 0 on host 'smithi151' 2024-04-16T02:29:01.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.272+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f772406c460 msgr2=0x7f772406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:01.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.272+0000 7f7741ea9700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f772406c460 0x7f772406e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7730006fd0 tx=0x7f7730008040 comp rx=0 tx=0).stop 2024-04-16T02:29:01.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.272+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 msgr2=0x7f773c196400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:01.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.272+0000 7f7741ea9700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 0x7f773c196400 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f772c00bda0 tx=0x7f772c00be80 comp rx=0 tx=0).stop 2024-04-16T02:29:01.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.272+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 shutdown_connections 2024-04-16T02:29:01.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.272+0000 7f7741ea9700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f772406c460 0x7f772406e920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:01.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.272+0000 7f7741ea9700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f773c100bd0 0x7f773c196400 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:01.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.272+0000 7f7741ea9700 1 --2- 172.21.15.151:0/1068702668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f773c101dd0 0x7f773c196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:01.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.272+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 >> 172.21.15.151:0/1068702668 conn(0x7f773c0fc170 msgr2=0x7f773c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:29:01.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.273+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 shutdown_connections 2024-04-16T02:29:01.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:01.273+0000 7f7741ea9700 1 -- 172.21.15.151:0/1068702668 wait complete. 2024-04-16T02:29:01.908 DEBUG:teuthology.orchestra.run.smithi151:osd.0> sudo journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.0.service 2024-04-16T02:29:01.910 INFO:tasks.cephadm:Deploying osd.1 on smithi151 with /dev/vg_nvme/lv_3... 2024-04-16T02:29:01.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-16T02:29:02.432 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:02 smithi151 ceph-mon[29003]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:29:02.432 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:02 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:02.432 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:02 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:02.432 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:02 smithi151 ceph-mon[29003]: from='osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T02:29:02.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:02 smithi153 ceph-mon[36747]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:29:02.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:02 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:02.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:02 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:02.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:02 smithi153 ceph-mon[36747]: from='osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T02:29:03.353 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:29:03.431 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:03 smithi151 ceph-mon[29003]: from='osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T02:29:03.432 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:03 smithi151 ceph-mon[29003]: osdmap e7: 1 total, 0 up, 1 in 2024-04-16T02:29:03.432 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:03 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:29:03.432 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:03 smithi151 ceph-mon[29003]: from='osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:29:03.432 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:03 smithi151 ceph-mon[29003]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:29:03.432 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:29:03 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[47348]: 2024-04-16T02:29:03.228+0000 7f26c2b31700 -1 osd.0 0 waiting for initial osdmap 2024-04-16T02:29:03.432 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:29:03 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[47348]: 2024-04-16T02:29:03.231+0000 7f26be926700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:29:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:03 smithi153 ceph-mon[36747]: from='osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T02:29:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:03 smithi153 ceph-mon[36747]: osdmap e7: 1 total, 0 up, 1 in 2024-04-16T02:29:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:03 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:29:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:03 smithi153 ceph-mon[36747]: from='osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:29:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:03 smithi153 ceph-mon[36747]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:29:04.485 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:04 smithi151 ceph-mon[29003]: from='osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-04-16T02:29:04.485 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:04 smithi151 ceph-mon[29003]: osdmap e8: 1 total, 0 up, 1 in 2024-04-16T02:29:04.485 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:29:04.485 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:29:04.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:04 smithi153 ceph-mon[36747]: from='osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-04-16T02:29:04.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:04 smithi153 ceph-mon[36747]: osdmap e8: 1 total, 0 up, 1 in 2024-04-16T02:29:04.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:04 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:29:04.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:04 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:29:05.357 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:05 smithi151 ceph-mon[29003]: purged_snaps scrub starts 2024-04-16T02:29:05.357 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:05 smithi151 ceph-mon[29003]: purged_snaps scrub ok 2024-04-16T02:29:05.357 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:05 smithi151 ceph-mon[29003]: osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] boot 2024-04-16T02:29:05.357 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:05 smithi151 ceph-mon[29003]: osdmap e9: 1 total, 1 up, 1 in 2024-04-16T02:29:05.357 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:29:05.357 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:05 smithi151 ceph-mon[29003]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:29:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:05 smithi153 ceph-mon[36747]: purged_snaps scrub starts 2024-04-16T02:29:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:05 smithi153 ceph-mon[36747]: purged_snaps scrub ok 2024-04-16T02:29:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:05 smithi153 ceph-mon[36747]: osd.0 [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] boot 2024-04-16T02:29:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:05 smithi153 ceph-mon[36747]: osdmap e9: 1 total, 1 up, 1 in 2024-04-16T02:29:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:29:05.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:05 smithi153 ceph-mon[36747]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T02:29:06.221 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-16T02:29:06.222 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-16T02:29:06.222 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-04-16T02:29:06.222 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-04-16T02:29:06.222 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134642 s, 779 MB/s 2024-04-16T02:29:06.222 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-04-16T02:29:06.831 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:06 smithi151 ceph-mon[29003]: osdmap e10: 1 total, 1 up, 1 in 2024-04-16T02:29:06.831 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:06.831 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:06.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_3 2024-04-16T02:29:07.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:06 smithi153 ceph-mon[36747]: osdmap e10: 1 total, 1 up, 1 in 2024-04-16T02:29:07.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:07.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:07.066 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:29:07.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:07 smithi151 ceph-mon[29003]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:08.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:07 smithi153 ceph-mon[36747]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:09.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.270+0000 7f4571c7b700 1 -- 172.21.15.151:0/1220567247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 msgr2=0x7f456c102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:09.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.270+0000 7f4571c7b700 1 --2- 172.21.15.151:0/1220567247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 0x7f456c102260 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4554009910 tx=0x7f4554009c20 comp rx=0 tx=0).stop 2024-04-16T02:29:09.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.270+0000 7f4571c7b700 1 -- 172.21.15.151:0/1220567247 shutdown_connections 2024-04-16T02:29:09.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.270+0000 7f4571c7b700 1 --2- 172.21.15.151:0/1220567247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 0x7f456c102260 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:09.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.270+0000 7f4571c7b700 1 --2- 172.21.15.151:0/1220567247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f456c100c00 0x7f456c101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:09.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.270+0000 7f4571c7b700 1 -- 172.21.15.151:0/1220567247 >> 172.21.15.151:0/1220567247 conn(0x7f456c0fc180 msgr2=0x7f456c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:29:09.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.270+0000 7f4571c7b700 1 -- 172.21.15.151:0/1220567247 shutdown_connections 2024-04-16T02:29:09.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.270+0000 7f4571c7b700 1 -- 172.21.15.151:0/1220567247 wait complete. 2024-04-16T02:29:09.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4571c7b700 1 Processor -- start 2024-04-16T02:29:09.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4571c7b700 1 -- start start 2024-04-16T02:29:09.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4571c7b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f456c100c00 0x7f456c196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:29:09.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4571c7b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 0x7f456c1968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:29:09.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4571c7b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c196ee0 con 0x7f456c101e00 2024-04-16T02:29:09.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4571c7b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c197020 con 0x7f456c100c00 2024-04-16T02:29:09.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4562ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 0x7f456c1968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4562ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 0x7f456c1968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53068/0 (socket says 172.21.15.151:53068) 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4562ffd700 1 -- 172.21.15.151:0/2522293382 learned_addr learned my addr 172.21.15.151:0/2522293382 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f456b7fe700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f456c100c00 0x7f456c196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4562ffd700 1 -- 172.21.15.151:0/2522293382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f456c100c00 msgr2=0x7f456c196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.271+0000 7f4562ffd700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f456c100c00 0x7f456c196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.272+0000 7f4562ffd700 1 -- 172.21.15.151:0/2522293382 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45540095f0 con 0x7f456c101e00 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.272+0000 7f456b7fe700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f456c100c00 0x7f456c196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.272+0000 7f4562ffd700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 0x7f456c1968c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f455400bd70 tx=0x7f455400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.272+0000 7f45697fa700 1 -- 172.21.15.151:0/2522293382 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f455401d070 con 0x7f456c101e00 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.272+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456c19ba70 con 0x7f456c101e00 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.272+0000 7f45697fa700 1 -- 172.21.15.151:0/2522293382 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4554022470 con 0x7f456c101e00 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.272+0000 7f45697fa700 1 -- 172.21.15.151:0/2522293382 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f455400f670 con 0x7f456c101e00 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.272+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456c19bf60 con 0x7f456c101e00 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.273+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456c109760 con 0x7f456c101e00 2024-04-16T02:29:09.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.273+0000 7f45697fa700 1 -- 172.21.15.151:0/2522293382 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f4554022a50 con 0x7f456c101e00 2024-04-16T02:29:09.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.274+0000 7f45697fa700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f455806c460 0x7f455806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:29:09.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.274+0000 7f45697fa700 1 -- 172.21.15.151:0/2522293382 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f455408c030 con 0x7f456c101e00 2024-04-16T02:29:09.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.274+0000 7f456b7fe700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f455806c460 0x7f455806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:29:09.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.274+0000 7f456b7fe700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f455806c460 0x7f455806e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f455c005fd0 tx=0x7f455c005dc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:29:09.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.279+0000 7f45697fa700 1 -- 172.21.15.151:0/2522293382 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f455405b1d0 con 0x7f456c101e00 2024-04-16T02:29:09.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:09.479+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f456c19c660 con 0x7f455806c460 2024-04-16T02:29:09.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:09 smithi151 ceph-mon[29003]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:09.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:09 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:29:09.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:09 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:29:09.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:09 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:09 smithi153 ceph-mon[36747]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:09 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:29:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:09 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:29:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:09 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:10.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:10 smithi151 ceph-mon[29003]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:29:11.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:10 smithi153 ceph-mon[36747]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:29:11.974 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:11 smithi151 ceph-mon[29003]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:11.975 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:11.975 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:11.975 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:29:11.975 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:11.975 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:11.975 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:29:11.975 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:11.975 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:29:12.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:11 smithi153 ceph-mon[36747]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:29:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:29:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:29:13.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:12 smithi151 ceph-mon[29003]: Detected new or changed devices on smithi151 2024-04-16T02:29:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:12 smithi151 ceph-mon[29003]: Adjusting osd_memory_target on smithi151 to 11840M 2024-04-16T02:29:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:12 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:29:13.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:12 smithi153 ceph-mon[36747]: Detected new or changed devices on smithi151 2024-04-16T02:29:13.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:12 smithi153 ceph-mon[36747]: Adjusting osd_memory_target on smithi151 to 11840M 2024-04-16T02:29:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:29:14.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:13 smithi151 ceph-mon[29003]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:14.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:13 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1618487165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d2980b5-1d30-445c-91ae-f08074b6d8f4"}]: dispatch 2024-04-16T02:29:14.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:13 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1618487165' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d2980b5-1d30-445c-91ae-f08074b6d8f4"}]': finished 2024-04-16T02:29:14.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:13 smithi151 ceph-mon[29003]: osdmap e11: 2 total, 1 up, 2 in 2024-04-16T02:29:14.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:13 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:14.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:13 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:14.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:13 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:14.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:13 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2875854669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:29:14.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:13 smithi153 ceph-mon[36747]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:13 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1618487165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d2980b5-1d30-445c-91ae-f08074b6d8f4"}]: dispatch 2024-04-16T02:29:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:13 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1618487165' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d2980b5-1d30-445c-91ae-f08074b6d8f4"}]': finished 2024-04-16T02:29:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:13 smithi153 ceph-mon[36747]: osdmap e11: 2 total, 1 up, 2 in 2024-04-16T02:29:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:13 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:13 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:13 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:13 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2875854669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:29:15.968 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:15 smithi151 ceph-mon[29003]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:16.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:15 smithi153 ceph-mon[36747]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:17.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:17.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:17.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:17.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:29:17.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:17.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:17 smithi151 ceph-mon[29003]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:17.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:17.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:17.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:17.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:29:17.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:17.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:17 smithi153 ceph-mon[36747]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:20.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:19 smithi151 ceph-mon[29003]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:20.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:19 smithi153 ceph-mon[36747]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:21.162 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:20 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T02:29:21.162 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:20 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:21.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:20 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T02:29:21.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:20 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:22.179 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:21 smithi151 ceph-mon[29003]: Deploying daemon osd.1 on smithi151 2024-04-16T02:29:22.179 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:21 smithi151 ceph-mon[29003]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:22.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:21 smithi153 ceph-mon[36747]: Deploying daemon osd.1 on smithi151 2024-04-16T02:29:22.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:21 smithi153 ceph-mon[36747]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:24.179 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:23 smithi151 ceph-mon[29003]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:24.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:23 smithi153 ceph-mon[36747]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:25.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:25 smithi151 ceph-mon[29003]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:26.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:25 smithi153 ceph-mon[36747]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:27.680 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:27.680 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:27.681 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:29:27.681 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:27 smithi153 ceph-mon[36747]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:27.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:27.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:27.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:29:27.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:27 smithi151 ceph-mon[29003]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:28.393 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:28 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:29:28.393 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:28 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:28.393 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:28 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:28.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:28 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:29:28.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:28 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:28.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:28 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:29.692 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:29 smithi151 ceph-mon[29003]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:29.741 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 1 on host 'smithi151' 2024-04-16T02:29:29.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.740+0000 7f45697fa700 1 -- 172.21.15.151:0/2522293382 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f456c19c660 con 0x7f455806c460 2024-04-16T02:29:29.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f455806c460 msgr2=0x7f455806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:29.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f455806c460 0x7f455806e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f455c005fd0 tx=0x7f455c005dc0 comp rx=0 tx=0).stop 2024-04-16T02:29:29.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 msgr2=0x7f456c1968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:29.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 0x7f456c1968c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f455400bd70 tx=0x7f455400be50 comp rx=0 tx=0).stop 2024-04-16T02:29:29.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 shutdown_connections 2024-04-16T02:29:29.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f455806c460 0x7f455806e920 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:29.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f456c100c00 0x7f456c196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:29.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 --2- 172.21.15.151:0/2522293382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f456c101e00 0x7f456c1968c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:29.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 >> 172.21.15.151:0/2522293382 conn(0x7f456c0fc180 msgr2=0x7f456c105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:29:29.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 shutdown_connections 2024-04-16T02:29:29.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:29.742+0000 7f4571c7b700 1 -- 172.21.15.151:0/2522293382 wait complete. 2024-04-16T02:29:29.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:29 smithi153 ceph-mon[36747]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:30.515 DEBUG:teuthology.orchestra.run.smithi151:osd.1> sudo journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.1.service 2024-04-16T02:29:30.517 INFO:tasks.cephadm:Deploying osd.2 on smithi151 with /dev/vg_nvme/lv_2... 2024-04-16T02:29:30.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-16T02:29:30.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:30 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:30.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:30 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:30.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:30 smithi151 ceph-mon[29003]: from='osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T02:29:31.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:30 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:31.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:30 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:31.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:30 smithi153 ceph-mon[36747]: from='osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T02:29:31.293 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:29:31.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:31 smithi151 ceph-mon[29003]: from='osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T02:29:31.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:31 smithi151 ceph-mon[29003]: osdmap e12: 2 total, 1 up, 2 in 2024-04-16T02:29:31.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:31 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:31.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:31 smithi151 ceph-mon[29003]: from='osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:29:31.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:31 smithi151 ceph-mon[29003]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:31.995 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:29:31 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[55137]: 2024-04-16T02:29:31.733+0000 7fc67c30b700 -1 osd.1 0 waiting for initial osdmap 2024-04-16T02:29:31.995 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:29:31 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[55137]: 2024-04-16T02:29:31.737+0000 7fc6750fa700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:29:32.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:31 smithi153 ceph-mon[36747]: from='osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T02:29:32.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:31 smithi153 ceph-mon[36747]: osdmap e12: 2 total, 1 up, 2 in 2024-04-16T02:29:32.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:31 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:32.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:31 smithi153 ceph-mon[36747]: from='osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:29:32.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:31 smithi153 ceph-mon[36747]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-16T02:29:32.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:32 smithi151 ceph-mon[29003]: from='osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-04-16T02:29:32.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:32 smithi151 ceph-mon[29003]: osdmap e13: 2 total, 1 up, 2 in 2024-04-16T02:29:32.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:32 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:32 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:33.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:32 smithi153 ceph-mon[36747]: from='osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-04-16T02:29:33.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:32 smithi153 ceph-mon[36747]: osdmap e13: 2 total, 1 up, 2 in 2024-04-16T02:29:33.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:32 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:33.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:32 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:33.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:33 smithi151 ceph-mon[29003]: purged_snaps scrub starts 2024-04-16T02:29:33.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:33 smithi151 ceph-mon[29003]: purged_snaps scrub ok 2024-04-16T02:29:33.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:33 smithi151 ceph-mon[29003]: osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] boot 2024-04-16T02:29:33.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:33 smithi151 ceph-mon[29003]: osdmap e14: 2 total, 2 up, 2 in 2024-04-16T02:29:33.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:33 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:33.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:33 smithi151 ceph-mon[29003]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:34.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:33 smithi153 ceph-mon[36747]: purged_snaps scrub starts 2024-04-16T02:29:34.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:33 smithi153 ceph-mon[36747]: purged_snaps scrub ok 2024-04-16T02:29:34.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:33 smithi153 ceph-mon[36747]: osd.1 [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] boot 2024-04-16T02:29:34.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:33 smithi153 ceph-mon[36747]: osdmap e14: 2 total, 2 up, 2 in 2024-04-16T02:29:34.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:33 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:29:34.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:33 smithi153 ceph-mon[36747]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:34.464 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-16T02:29:34.464 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-16T02:29:34.464 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-04-16T02:29:34.464 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-04-16T02:29:34.464 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127124 s, 825 MB/s 2024-04-16T02:29:34.464 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-04-16T02:29:35.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:34 smithi153 ceph-mon[36747]: osdmap e15: 2 total, 2 up, 2 in 2024-04-16T02:29:35.176 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:34 smithi151 ceph-mon[29003]: osdmap e15: 2 total, 2 up, 2 in 2024-04-16T02:29:35.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_2 2024-04-16T02:29:35.660 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:29:36.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:35 smithi153 ceph-mon[36747]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:36.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:36.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:36.080 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:35 smithi151 ceph-mon[29003]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:36.080 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:36.080 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:37.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:37 smithi151 ceph-mon[29003]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:38.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:37 smithi153 ceph-mon[36747]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 -- 172.21.15.151:0/3874058650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 msgr2=0x7f47580ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 --2- 172.21.15.151:0/3874058650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 0x7f47580ffed0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4754009910 tx=0x7f4754009c20 comp rx=0 tx=0).stop 2024-04-16T02:29:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 -- 172.21.15.151:0/3874058650 shutdown_connections 2024-04-16T02:29:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 --2- 172.21.15.151:0/3874058650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4758100cb0 0x7f4758101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 --2- 172.21.15.151:0/3874058650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 0x7f47580ffed0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 -- 172.21.15.151:0/3874058650 >> 172.21.15.151:0/3874058650 conn(0x7f47580fb030 msgr2=0x7f47580fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:29:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 -- 172.21.15.151:0/3874058650 shutdown_connections 2024-04-16T02:29:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 -- 172.21.15.151:0/3874058650 wait complete. 2024-04-16T02:29:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 Processor -- start 2024-04-16T02:29:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.406+0000 7f47606dd700 1 -- start start 2024-04-16T02:29:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f47606dd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 0x7f47581952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:29:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f47606dd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4758100cb0 0x7f4758195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:29:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f47606dd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4758195e40 con 0x7f47580ffab0 2024-04-16T02:29:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f47606dd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4758195f80 con 0x7f4758100cb0 2024-04-16T02:29:38.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f475e479700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 0x7f47581952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f475e479700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 0x7f47581952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58946/0 (socket says 172.21.15.151:58946) 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f475e479700 1 -- 172.21.15.151:0/2365723032 learned_addr learned my addr 172.21.15.151:0/2365723032 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f475e479700 1 -- 172.21.15.151:0/2365723032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4758100cb0 msgr2=0x7f4758195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f475e479700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4758100cb0 0x7f4758195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f475e479700 1 -- 172.21.15.151:0/2365723032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47540095f0 con 0x7f47580ffab0 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f475e479700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 0x7f47581952e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f475400bda0 tx=0x7f475400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f474f7fe700 1 -- 172.21.15.151:0/2365723032 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f475401d070 con 0x7f47580ffab0 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f474f7fe700 1 -- 172.21.15.151:0/2365723032 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4754004c50 con 0x7f47580ffab0 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475819a9d0 con 0x7f47580ffab0 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f474f7fe700 1 -- 172.21.15.151:0/2365723032 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f475400f690 con 0x7f47580ffab0 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.407+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475819af40 con 0x7f47580ffab0 2024-04-16T02:29:38.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.408+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4758108690 con 0x7f47580ffab0 2024-04-16T02:29:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.409+0000 7f474f7fe700 1 -- 172.21.15.151:0/2365723032 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f47540229b0 con 0x7f47580ffab0 2024-04-16T02:29:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.409+0000 7f474f7fe700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f474406c460 0x7f474406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:29:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.409+0000 7f475dc78700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f474406c460 0x7f474406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:29:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.409+0000 7f474f7fe700 1 -- 172.21.15.151:0/2365723032 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f475408c280 con 0x7f47580ffab0 2024-04-16T02:29:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.410+0000 7f475dc78700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f474406c460 0x7f474406e920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47480097b0 tx=0x7f4748006d20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:29:38.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.414+0000 7f474f7fe700 1 -- 172.21.15.151:0/2365723032 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f475405b240 con 0x7f47580ffab0 2024-04-16T02:29:38.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:38.618+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4758105600 con 0x7f474406c460 2024-04-16T02:29:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:38 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:29:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:38 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:29:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:38 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:38 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:29:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:38 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:29:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:38 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:40.229 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:39 smithi151 ceph-mon[29003]: from='client.14288 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:29:40.229 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:39 smithi151 ceph-mon[29003]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:40.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:39 smithi153 ceph-mon[36747]: from='client.14288 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:29:40.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:39 smithi153 ceph-mon[36747]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: Detected new or changed devices on smithi151 2024-04-16T02:29:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:29:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:29:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: Adjusting osd_memory_target on smithi151 to 5920M 2024-04-16T02:29:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:29:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:29:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:41 smithi153 ceph-mon[36747]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:41.656 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: Detected new or changed devices on smithi151 2024-04-16T02:29:41.656 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: Adjusting osd_memory_target on smithi151 to 5920M 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:29:41.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:41 smithi151 ceph-mon[29003]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:42.611 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:42 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3277562488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64e4e7ce-5c3c-48c8-be3d-c9936b6866b6"}]: dispatch 2024-04-16T02:29:42.612 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:42 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3277562488' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64e4e7ce-5c3c-48c8-be3d-c9936b6866b6"}]': finished 2024-04-16T02:29:42.612 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:42 smithi151 ceph-mon[29003]: osdmap e16: 3 total, 2 up, 3 in 2024-04-16T02:29:42.612 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:29:42.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:42 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3277562488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64e4e7ce-5c3c-48c8-be3d-c9936b6866b6"}]: dispatch 2024-04-16T02:29:42.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:42 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3277562488' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64e4e7ce-5c3c-48c8-be3d-c9936b6866b6"}]': finished 2024-04-16T02:29:42.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:42 smithi153 ceph-mon[36747]: osdmap e16: 3 total, 2 up, 3 in 2024-04-16T02:29:42.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:42 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:29:43.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:43.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:43.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:43 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/917107021' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:29:43.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:43 smithi151 ceph-mon[29003]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:43.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:29:43.671 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:43.671 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:43.671 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:43 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/917107021' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:29:43.671 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:43 smithi153 ceph-mon[36747]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:43.671 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:29:45.832 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:45 smithi151 ceph-mon[29003]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:45.832 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:45.832 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:45.832 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:45.832 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:29:45.832 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:46.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:45 smithi153 ceph-mon[36747]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:29:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:48.213 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:47 smithi151 ceph-mon[29003]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:48.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:47 smithi153 ceph-mon[36747]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:50.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:49 smithi151 ceph-mon[29003]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:50.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:49 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T02:29:50.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:49 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:50.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:49 smithi153 ceph-mon[36747]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:50.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:49 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T02:29:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:49 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:29:50.976 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:50 smithi151 ceph-mon[29003]: Deploying daemon osd.2 on smithi151 2024-04-16T02:29:51.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:50 smithi153 ceph-mon[36747]: Deploying daemon osd.2 on smithi151 2024-04-16T02:29:52.161 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:51 smithi151 ceph-mon[29003]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:52.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:51 smithi153 ceph-mon[36747]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:53 smithi151 ceph-mon[29003]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:54.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:53 smithi153 ceph-mon[36747]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:56.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:56 smithi151 ceph-mon[29003]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:56.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:56.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:56.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:29:56.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:56 smithi153 ceph-mon[36747]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:56.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:56.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:56.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:29:58.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:58 smithi151 ceph-mon[29003]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:58.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:29:58.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:58 smithi153 ceph-mon[36747]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:58.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:29:58.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.724+0000 7f474f7fe700 1 -- 172.21.15.151:0/2365723032 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4758105600 con 0x7f474406c460 2024-04-16T02:29:58.726 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 2 on host 'smithi151' 2024-04-16T02:29:58.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f474406c460 msgr2=0x7f474406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:58.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f474406c460 0x7f474406e920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47480097b0 tx=0x7f4748006d20 comp rx=0 tx=0).stop 2024-04-16T02:29:58.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 msgr2=0x7f47581952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:29:58.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 0x7f47581952e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f475400bda0 tx=0x7f475400be80 comp rx=0 tx=0).stop 2024-04-16T02:29:58.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 shutdown_connections 2024-04-16T02:29:58.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f474406c460 0x7f474406e920 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:58.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47580ffab0 0x7f47581952e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:58.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 --2- 172.21.15.151:0/2365723032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4758100cb0 0x7f4758195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:29:58.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 >> 172.21.15.151:0/2365723032 conn(0x7f47580fb030 msgr2=0x7f4758103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:29:58.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 shutdown_connections 2024-04-16T02:29:58.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:29:58.727+0000 7f47606dd700 1 -- 172.21.15.151:0/2365723032 wait complete. 2024-04-16T02:29:59.728 DEBUG:teuthology.orchestra.run.smithi151:osd.2> sudo journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.2.service 2024-04-16T02:29:59.730 INFO:tasks.cephadm:Deploying osd.3 on smithi153 with /dev/vg_nvme/lv_4... 2024-04-16T02:29:59.731 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-16T02:29:59.952 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T02:29:59.969 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:59.970 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:29:59.970 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:59 smithi151 ceph-mon[29003]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:29:59.970 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:29:59 smithi151 ceph-mon[29003]: from='osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T02:30:00.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:00.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:00.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:59 smithi153 ceph-mon[36747]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:30:00.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:29:59 smithi153 ceph-mon[36747]: from='osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T02:30:00.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:00 smithi151 ceph-mon[29003]: from='osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T02:30:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:00 smithi151 ceph-mon[29003]: osdmap e17: 3 total, 2 up, 3 in 2024-04-16T02:30:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:00 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:00 smithi151 ceph-mon[29003]: from='osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:30:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:00 smithi151 ceph-mon[29003]: overall HEALTH_OK 2024-04-16T02:30:00.995 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[63000]: 2024-04-16T02:30:00.691+0000 7f6a9143a700 -1 osd.2 0 waiting for initial osdmap 2024-04-16T02:30:00.995 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[63000]: 2024-04-16T02:30:00.711+0000 7f6a8da30700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:30:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:00 smithi153 ceph-mon[36747]: from='osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T02:30:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:00 smithi153 ceph-mon[36747]: osdmap e17: 3 total, 2 up, 3 in 2024-04-16T02:30:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:00 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:00 smithi153 ceph-mon[36747]: from='osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:30:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:00 smithi153 ceph-mon[36747]: overall HEALTH_OK 2024-04-16T02:30:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-16T02:30:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-16T02:30:01.803 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-16T02:30:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-16T02:30:01.803 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130957 s, 801 MB/s 2024-04-16T02:30:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-16T02:30:01.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:01 smithi151 ceph-mon[29003]: from='osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-04-16T02:30:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:01 smithi151 ceph-mon[29003]: osdmap e18: 3 total, 2 up, 3 in 2024-04-16T02:30:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:01 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:01 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:01 smithi151 ceph-mon[29003]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:30:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:01 smithi153 ceph-mon[36747]: from='osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-04-16T02:30:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:01 smithi153 ceph-mon[36747]: osdmap e18: 3 total, 2 up, 3 in 2024-04-16T02:30:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:01 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:01 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:01 smithi153 ceph-mon[36747]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-16T02:30:02.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-04-16T02:30:02.657 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T02:30:02.700 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:02 smithi151 ceph-mon[29003]: purged_snaps scrub starts 2024-04-16T02:30:02.700 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:02 smithi151 ceph-mon[29003]: purged_snaps scrub ok 2024-04-16T02:30:02.700 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:02 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:02.700 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:02 smithi151 ceph-mon[29003]: osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] boot 2024-04-16T02:30:02.700 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:02 smithi151 ceph-mon[29003]: osdmap e19: 3 total, 3 up, 3 in 2024-04-16T02:30:02.700 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:02 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:02.984 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:02 smithi153 ceph-mon[36747]: purged_snaps scrub starts 2024-04-16T02:30:02.984 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:02 smithi153 ceph-mon[36747]: purged_snaps scrub ok 2024-04-16T02:30:02.984 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:02 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:02.984 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:02 smithi153 ceph-mon[36747]: osd.2 [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] boot 2024-04-16T02:30:02.985 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:02 smithi153 ceph-mon[36747]: osdmap e19: 3 total, 3 up, 3 in 2024-04-16T02:30:02.985 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:02 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:30:02.985 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:02 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:02.985 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:02 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:02.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:02 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:02.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:02 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:03.973 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:03 smithi153 ceph-mon[36747]: osdmap e20: 3 total, 3 up, 3 in 2024-04-16T02:30:03.973 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:03 smithi153 ceph-mon[36747]: pgmap v78: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:03.973 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:03 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-16T02:30:03.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:03 smithi151 ceph-mon[29003]: osdmap e20: 3 total, 3 up, 3 in 2024-04-16T02:30:03.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:03 smithi151 ceph-mon[29003]: pgmap v78: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:03.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:03 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-16T02:30:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.219+0000 7f17aca31700 1 -- 172.21.15.153:0/183531375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a8100cb0 msgr2=0x7f17a8101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.219+0000 7f17aca31700 1 --2- 172.21.15.153:0/183531375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a8100cb0 0x7f17a8101110 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1790009910 tx=0x7f1790009c20 comp rx=0 tx=0).stop 2024-04-16T02:30:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.219+0000 7f17aca31700 1 -- 172.21.15.153:0/183531375 shutdown_connections 2024-04-16T02:30:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.219+0000 7f17aca31700 1 --2- 172.21.15.153:0/183531375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a8100cb0 0x7f17a8101110 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.219+0000 7f17aca31700 1 --2- 172.21.15.153:0/183531375 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17a80ffab0 0x7f17a80ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.219+0000 7f17aca31700 1 -- 172.21.15.153:0/183531375 >> 172.21.15.153:0/183531375 conn(0x7f17a80fb050 msgr2=0x7f17a80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:30:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.219+0000 7f17aca31700 1 -- 172.21.15.153:0/183531375 shutdown_connections 2024-04-16T02:30:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.219+0000 7f17aca31700 1 -- 172.21.15.153:0/183531375 wait complete. 2024-04-16T02:30:04.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17aca31700 1 Processor -- start 2024-04-16T02:30:04.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17aca31700 1 -- start start 2024-04-16T02:30:04.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17aca31700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a80ffab0 0x7f17a8195230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:30:04.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17aca31700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17a8100cb0 0x7f17a8195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:30:04.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17aca31700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8195d90 con 0x7f17a8100cb0 2024-04-16T02:30:04.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17aca31700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8195ed0 con 0x7f17a80ffab0 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17a659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a80ffab0 0x7f17a8195230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17a659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a80ffab0 0x7f17a8195230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54506/0 (socket says 172.21.15.153:54506) 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17a659c700 1 -- 172.21.15.153:0/2590167078 learned_addr learned my addr 172.21.15.153:0/2590167078 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.220+0000 7f17a5d9b700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17a8100cb0 0x7f17a8195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.221+0000 7f17a659c700 1 -- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17a8100cb0 msgr2=0x7f17a8195770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.221+0000 7f17a659c700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17a8100cb0 0x7f17a8195770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.221+0000 7f17a659c700 1 -- 172.21.15.153:0/2590167078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17900095f0 con 0x7f17a80ffab0 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.221+0000 7f17a659c700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a80ffab0 0x7f17a8195230 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f179800d900 tx=0x7f179800dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.221+0000 7f179f7fe700 1 -- 172.21.15.153:0/2590167078 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179800b920 con 0x7f17a80ffab0 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.221+0000 7f179f7fe700 1 -- 172.21.15.153:0/2590167078 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1798010460 con 0x7f17a80ffab0 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.221+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a819a980 con 0x7f17a80ffab0 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.221+0000 7f179f7fe700 1 -- 172.21.15.153:0/2590167078 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179800f590 con 0x7f17a80ffab0 2024-04-16T02:30:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.221+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a819aed0 con 0x7f17a80ffab0 2024-04-16T02:30:04.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.222+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a8108610 con 0x7f17a80ffab0 2024-04-16T02:30:04.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.222+0000 7f179f7fe700 1 -- 172.21.15.153:0/2590167078 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f179800ba80 con 0x7f17a80ffab0 2024-04-16T02:30:04.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.223+0000 7f179f7fe700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f179406c460 0x7f179406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:30:04.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.223+0000 7f179f7fe700 1 -- 172.21.15.153:0/2590167078 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3221+0+0 (secure 0 0 0) 0x7f179808ab20 con 0x7f17a80ffab0 2024-04-16T02:30:04.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.223+0000 7f17a5d9b700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f179406c460 0x7f179406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:30:04.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.224+0000 7f17a5d9b700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f179406c460 0x7f179406e920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f179000b5c0 tx=0x7f1790005ee0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:30:04.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.228+0000 7f179f7fe700 1 -- 172.21.15.153:0/2590167078 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1798059730 con 0x7f17a80ffab0 2024-04-16T02:30:04.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:04.422+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f17a8105600 con 0x7f179406c460 2024-04-16T02:30:04.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:04 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-16T02:30:04.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:04 smithi153 ceph-mon[36747]: osdmap e21: 3 total, 3 up, 3 in 2024-04-16T02:30:04.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:04 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-16T02:30:04.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:04 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:30:04.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:04 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:30:04.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:04 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:04.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-16T02:30:04.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:04 smithi151 ceph-mon[29003]: osdmap e21: 3 total, 3 up, 3 in 2024-04-16T02:30:04.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-16T02:30:04.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:30:04.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:30:04.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:04 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:05.494 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67361]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T02:30:05.495 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67361]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:05.495 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67361]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:05.495 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67361]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:05.826 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67430]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T02:30:05.826 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67430]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:05.826 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67430]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:05.826 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67430]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:05.826 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:30:05.826 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-16T02:30:05.826 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: osdmap e22: 3 total, 3 up, 3 in 2024-04-16T02:30:05.826 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: Detected new or changed devices on smithi151 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:05.827 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: osdmap e22: 3 total, 3 up, 3 in 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: Detected new or changed devices on smithi151 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:06.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:06.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:06.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:06.088 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67520]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T02:30:06.089 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67520]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:06.089 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67520]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:06.089 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:30:05 smithi151 sudo[67520]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:06.342 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67529]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T02:30:06.342 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67529]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:06.342 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67529]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:06.342 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67529]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:06.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67546]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T02:30:06.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67546]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:06.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67546]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:06.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67546]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:06.873 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67607]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T02:30:06.873 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67607]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:06.873 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67607]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:06.873 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67607]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:07.133 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67673]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T02:30:07.133 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67673]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:07.133 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67673]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:07.133 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:06 smithi151 sudo[67673]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:07.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 ceph-mon[36747]: Adjusting osd_memory_target on smithi151 to 3946M 2024-04-16T02:30:07.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 ceph-mon[36747]: osdmap e23: 3 total, 3 up, 3 in 2024-04-16T02:30:07.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 ceph-mon[36747]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T02:30:07.400 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 sudo[40607]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T02:30:07.400 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 sudo[40607]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:07.400 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 sudo[40607]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:07.400 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 sudo[40607]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:07.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:07 smithi151 sudo[67678]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T02:30:07.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:07 smithi151 sudo[67678]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:07.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:07 smithi151 sudo[67678]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:07.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:07 smithi151 sudo[67678]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:07.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:07 smithi151 ceph-mon[29003]: Adjusting osd_memory_target on smithi151 to 3946M 2024-04-16T02:30:07.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:07 smithi151 ceph-mon[29003]: osdmap e23: 3 total, 3 up, 3 in 2024-04-16T02:30:07.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:07 smithi151 ceph-mon[29003]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T02:30:07.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 sudo[40610]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T02:30:07.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 sudo[40610]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T02:30:07.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 sudo[40610]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T02:30:07.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:07 smithi153 sudo[40610]: pam_unix(sudo:session): session closed for user root 2024-04-16T02:30:08.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:08.145+0000 7f179f7fe700 1 -- 172.21.15.153:0/2590167078 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f179800bd30 con 0x7f17a80ffab0 2024-04-16T02:30:08.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='client.? 172.21.15.153:0/1371953939' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1194f132-bf2d-4e0c-ab93-f3344ff64d76"}]: dispatch 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1194f132-bf2d-4e0c-ab93-f3344ff64d76"}]: dispatch 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1194f132-bf2d-4e0c-ab93-f3344ff64d76"}]': finished 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: osdmap e24: 4 total, 3 up, 4 in 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T02:30:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/1371953939' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1194f132-bf2d-4e0c-ab93-f3344ff64d76"}]: dispatch 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1194f132-bf2d-4e0c-ab93-f3344ff64d76"}]: dispatch 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1194f132-bf2d-4e0c-ab93-f3344ff64d76"}]': finished 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: osdmap e24: 4 total, 3 up, 4 in 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:08.452 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:30:08.453 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:30:08.453 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T02:30:08.453 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:09.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:09 smithi151 ceph-mon[29003]: mgrmap e19: smithi151.uqqame(active, since 2m), standbys: smithi153.esbtzf 2024-04-16T02:30:09.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:09 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/519086592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:30:09.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:09 smithi153 ceph-mon[36747]: mgrmap e19: smithi151.uqqame(active, since 2m), standbys: smithi153.esbtzf 2024-04-16T02:30:09.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:09 smithi153 ceph-mon[36747]: from='client.? 172.21.15.153:0/519086592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:30:10.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:10 smithi153 ceph-mon[36747]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:10.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:10 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:10.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:10 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:10.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:10 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:10.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:10 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:10.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:10 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:10.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:10 smithi151 ceph-mon[29003]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:10.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:10 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:10.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:10 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:10.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:10 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:10.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:10 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:10.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:10 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:11.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:11 smithi151 ceph-mon[29003]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:11.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:11 smithi153 ceph-mon[36747]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:13.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:30:13.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:12 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:30:14.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:13 smithi151 ceph-mon[29003]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:14.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:13 smithi153 ceph-mon[36747]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:15 smithi151 ceph-mon[29003]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:15 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T02:30:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:15 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:15 smithi151 ceph-mon[29003]: Deploying daemon osd.3 on smithi153 2024-04-16T02:30:16.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:15 smithi153 ceph-mon[36747]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:16.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:15 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T02:30:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:15 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:15 smithi153 ceph-mon[36747]: Deploying daemon osd.3 on smithi153 2024-04-16T02:30:18.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:17 smithi153 ceph-mon[36747]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:18.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:17 smithi151 ceph-mon[29003]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:20.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:19 smithi151 ceph-mon[29003]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:20.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:19 smithi153 ceph-mon[36747]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:21.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:20 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:20 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:20 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:21.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:20 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:21.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:20 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:21.159 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:20 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:22.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:22 smithi153 ceph-mon[36747]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:22.368 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:22 smithi151 ceph-mon[29003]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:23.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.833+0000 7f179f7fe700 1 -- 172.21.15.153:0/2590167078 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f17a8105600 con 0x7f179406c460 2024-04-16T02:30:23.834 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 3 on host 'smithi153' 2024-04-16T02:30:23.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f179406c460 msgr2=0x7f179406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:23.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f179406c460 0x7f179406e920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f179000b5c0 tx=0x7f1790005ee0 comp rx=0 tx=0).stop 2024-04-16T02:30:23.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a80ffab0 msgr2=0x7f17a8195230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:23.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a80ffab0 0x7f17a8195230 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f179800d900 tx=0x7f179800dc10 comp rx=0 tx=0).stop 2024-04-16T02:30:23.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 shutdown_connections 2024-04-16T02:30:23.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f179406c460 0x7f179406e920 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:23.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a80ffab0 0x7f17a8195230 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:23.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 --2- 172.21.15.153:0/2590167078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17a8100cb0 0x7f17a8195770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:23.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 >> 172.21.15.153:0/2590167078 conn(0x7f17a80fb050 msgr2=0x7f17a8103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:30:23.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 shutdown_connections 2024-04-16T02:30:23.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:23.835+0000 7f17aca31700 1 -- 172.21.15.153:0/2590167078 wait complete. 2024-04-16T02:30:24.379 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:24 smithi153 ceph-mon[36747]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:24.380 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:24 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:24.380 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:24 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:24.380 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:24 smithi153 ceph-mon[36747]: from='osd.3 [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T02:30:24.380 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:24 smithi153 ceph-mon[36747]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T02:30:24.405 DEBUG:teuthology.orchestra.run.smithi153:osd.3> sudo journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.3.service 2024-04-16T02:30:24.407 INFO:tasks.cephadm:Deploying osd.4 on smithi153 with /dev/vg_nvme/lv_3... 2024-04-16T02:30:24.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-16T02:30:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:24 smithi151 ceph-mon[29003]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:24 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:24.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:24 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:24.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:24 smithi151 ceph-mon[29003]: from='osd.3 [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T02:30:24.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:24 smithi151 ceph-mon[29003]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T02:30:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T02:30:25.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:25.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:25.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-mon[36747]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T02:30:25.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-mon[36747]: osdmap e25: 4 total, 3 up, 4 in 2024-04-16T02:30:25.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:25.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-mon[36747]: from='osd.3 [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:30:25.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-mon[36747]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:30:25.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-mon[36747]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:25.957 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[42707]: 2024-04-16T02:30:25.816+0000 7f80cf7bc700 -1 osd.3 0 waiting for initial osdmap 2024-04-16T02:30:25.957 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:30:25 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[42707]: 2024-04-16T02:30:25.822+0000 7f80cbdb2700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:30:25.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:25 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:25.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:25 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:25 smithi151 ceph-mon[29003]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T02:30:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:25 smithi151 ceph-mon[29003]: osdmap e25: 4 total, 3 up, 4 in 2024-04-16T02:30:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:25 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:25 smithi151 ceph-mon[29003]: from='osd.3 [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:30:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:25 smithi151 ceph-mon[29003]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:30:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:25 smithi151 ceph-mon[29003]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-16T02:30:26.480 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-16T02:30:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-16T02:30:26.481 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-16T02:30:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-16T02:30:26.481 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149245 s, 703 MB/s 2024-04-16T02:30:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-16T02:30:27.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:26 smithi153 ceph-mon[36747]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-16T02:30:27.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:26 smithi153 ceph-mon[36747]: osdmap e26: 4 total, 3 up, 4 in 2024-04-16T02:30:27.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:26 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:27.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:26 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:27.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:26 smithi151 ceph-mon[29003]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-16T02:30:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:26 smithi151 ceph-mon[29003]: osdmap e26: 4 total, 3 up, 4 in 2024-04-16T02:30:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:26 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:26 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:27 smithi153 ceph-mon[36747]: purged_snaps scrub starts 2024-04-16T02:30:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:27 smithi153 ceph-mon[36747]: purged_snaps scrub ok 2024-04-16T02:30:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:27 smithi153 ceph-mon[36747]: osd.3 [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] boot 2024-04-16T02:30:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:27 smithi153 ceph-mon[36747]: osdmap e27: 4 total, 4 up, 4 in 2024-04-16T02:30:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:27 smithi153 ceph-mon[36747]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:28.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-04-16T02:30:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:27 smithi151 ceph-mon[29003]: purged_snaps scrub starts 2024-04-16T02:30:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:27 smithi151 ceph-mon[29003]: purged_snaps scrub ok 2024-04-16T02:30:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:27 smithi151 ceph-mon[29003]: osd.3 [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] boot 2024-04-16T02:30:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:27 smithi151 ceph-mon[29003]: osdmap e27: 4 total, 4 up, 4 in 2024-04-16T02:30:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:30:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:27 smithi151 ceph-mon[29003]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:28.428 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T02:30:28.980 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:28 smithi153 ceph-mon[36747]: osdmap e28: 4 total, 4 up, 4 in 2024-04-16T02:30:28.980 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:28 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:30:28.980 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:28 smithi153 ceph-mon[36747]: osdmap e29: 4 total, 4 up, 4 in 2024-04-16T02:30:29.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:28 smithi151 ceph-mon[29003]: osdmap e28: 4 total, 4 up, 4 in 2024-04-16T02:30:29.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:28 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:30:29.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:28 smithi151 ceph-mon[29003]: osdmap e29: 4 total, 4 up, 4 in 2024-04-16T02:30:30.111 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:30.111 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.111 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.111 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:30.111 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.111 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:30.111 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:30.112 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.112 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: osdmap e30: 4 total, 4 up, 4 in 2024-04-16T02:30:30.112 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:30.112 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:30.112 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:30.112 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:29 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: osdmap e30: 4 total, 4 up, 4 in 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:29 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:30.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.599+0000 7f9766382700 1 -- 172.21.15.153:0/315227805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760101670 msgr2=0x7f9760103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:30.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.599+0000 7f9766382700 1 --2- 172.21.15.153:0/315227805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760101670 0x7f9760103a60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9750009910 tx=0x7f9750009c20 comp rx=0 tx=0).stop 2024-04-16T02:30:30.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.599+0000 7f9766382700 1 -- 172.21.15.153:0/315227805 shutdown_connections 2024-04-16T02:30:30.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.599+0000 7f9766382700 1 --2- 172.21.15.153:0/315227805 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9760103fa0 0x7f9760106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:30.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.599+0000 7f9766382700 1 --2- 172.21.15.153:0/315227805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760101670 0x7f9760103a60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:30.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.599+0000 7f9766382700 1 -- 172.21.15.153:0/315227805 >> 172.21.15.153:0/315227805 conn(0x7f97600fb080 msgr2=0x7f97600fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:30:30.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.599+0000 7f9766382700 1 -- 172.21.15.153:0/315227805 shutdown_connections 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.599+0000 7f9766382700 1 -- 172.21.15.153:0/315227805 wait complete. 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.599+0000 7f9766382700 1 Processor -- start 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f9766382700 1 -- start start 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f9766382700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9760101670 0x7f9760196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f9766382700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760103fa0 0x7f9760196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f9766382700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9760196e00 con 0x7f9760101670 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f9766382700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9760196f40 con 0x7f9760103fa0 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f975f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760103fa0 0x7f9760196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f975f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760103fa0 0x7f9760196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41270/0 (socket says 172.21.15.153:41270) 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f975f7fe700 1 -- 172.21.15.153:0/1349105435 learned_addr learned my addr 172.21.15.153:0/1349105435 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f975ffff700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9760101670 0x7f9760196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:30:30.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f975f7fe700 1 -- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9760101670 msgr2=0x7f9760196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:30.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f975f7fe700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9760101670 0x7f9760196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:30.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.600+0000 7f975f7fe700 1 -- 172.21.15.153:0/1349105435 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500095f0 con 0x7f9760103fa0 2024-04-16T02:30:30.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.601+0000 7f975f7fe700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760103fa0 0x7f9760196870 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f975400c930 tx=0x7f975400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:30:30.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.601+0000 7f975d7fa700 1 -- 172.21.15.153:0/1349105435 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9754007ad0 con 0x7f9760103fa0 2024-04-16T02:30:30.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.601+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976019ba00 con 0x7f9760103fa0 2024-04-16T02:30:30.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.601+0000 7f975d7fa700 1 -- 172.21.15.153:0/1349105435 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f975400ce80 con 0x7f9760103fa0 2024-04-16T02:30:30.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.601+0000 7f975d7fa700 1 -- 172.21.15.153:0/1349105435 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9754018630 con 0x7f9760103fa0 2024-04-16T02:30:30.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.601+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976019bf20 con 0x7f9760103fa0 2024-04-16T02:30:30.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.602+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9760190550 con 0x7f9760103fa0 2024-04-16T02:30:30.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.602+0000 7f975d7fa700 1 -- 172.21.15.153:0/1349105435 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f9754007c30 con 0x7f9760103fa0 2024-04-16T02:30:30.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.603+0000 7f975d7fa700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f974806c4e0 0x7f974806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:30:30.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.603+0000 7f975d7fa700 1 -- 172.21.15.153:0/1349105435 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f975408b110 con 0x7f9760103fa0 2024-04-16T02:30:30.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.603+0000 7f975ffff700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f974806c4e0 0x7f974806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:30:30.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.603+0000 7f975ffff700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f974806c4e0 0x7f974806e9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f975000b5c0 tx=0x7f97500052a0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:30:30.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.607+0000 7f975d7fa700 1 -- 172.21.15.153:0/1349105435 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9754059a20 con 0x7f9760103fa0 2024-04-16T02:30:30.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:30.804+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f97600611d0 con 0x7f974806c4e0 2024-04-16T02:30:31.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:30 smithi151 ceph-mon[29003]: Detected new or changed devices on smithi153 2024-04-16T02:30:31.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:30 smithi151 ceph-mon[29003]: Adjusting osd_memory_target on smithi153 to 14912M 2024-04-16T02:30:31.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:30 smithi151 ceph-mon[29003]: osdmap e31: 4 total, 4 up, 4 in 2024-04-16T02:30:31.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:30 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:30:31.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:30 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:30:31.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:30 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:31.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:30 smithi153 ceph-mon[36747]: Detected new or changed devices on smithi153 2024-04-16T02:30:31.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:30 smithi153 ceph-mon[36747]: Adjusting osd_memory_target on smithi153 to 14912M 2024-04-16T02:30:31.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:30 smithi153 ceph-mon[36747]: osdmap e31: 4 total, 4 up, 4 in 2024-04-16T02:30:31.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:30 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:30:31.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:30 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:30:31.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:30 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:32.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:31 smithi153 ceph-mon[36747]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:30:32.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:31 smithi153 ceph-mon[36747]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:31 smithi151 ceph-mon[29003]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:30:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:31 smithi151 ceph-mon[29003]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:34.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:33 smithi151 ceph-mon[29003]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-16T02:30:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:33 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/1961027563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d803d8b-eb35-4a0f-9d36-de11b743a123"}]: dispatch 2024-04-16T02:30:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:33 smithi151 ceph-mon[29003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d803d8b-eb35-4a0f-9d36-de11b743a123"}]: dispatch 2024-04-16T02:30:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:33 smithi151 ceph-mon[29003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d803d8b-eb35-4a0f-9d36-de11b743a123"}]': finished 2024-04-16T02:30:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:33 smithi151 ceph-mon[29003]: osdmap e32: 5 total, 4 up, 5 in 2024-04-16T02:30:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:33 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:33 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/20695753' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:30:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:33 smithi153 ceph-mon[36747]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-16T02:30:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:33 smithi153 ceph-mon[36747]: from='client.? 172.21.15.153:0/1961027563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d803d8b-eb35-4a0f-9d36-de11b743a123"}]: dispatch 2024-04-16T02:30:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:33 smithi153 ceph-mon[36747]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d803d8b-eb35-4a0f-9d36-de11b743a123"}]: dispatch 2024-04-16T02:30:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:33 smithi153 ceph-mon[36747]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d803d8b-eb35-4a0f-9d36-de11b743a123"}]': finished 2024-04-16T02:30:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:33 smithi153 ceph-mon[36747]: osdmap e32: 5 total, 4 up, 5 in 2024-04-16T02:30:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:33 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:33 smithi153 ceph-mon[36747]: from='client.? 172.21.15.153:0/20695753' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:30:35.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:35 smithi153 ceph-mon[36747]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-16T02:30:35.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:35 smithi151 ceph-mon[29003]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-16T02:30:38.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:37 smithi151 ceph-mon[29003]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-16T02:30:38.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:37 smithi153 ceph-mon[36747]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-16T02:30:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:39 smithi153 ceph-mon[36747]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-16T02:30:40.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:39 smithi151 ceph-mon[29003]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-16T02:30:41.186 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:40 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T02:30:41.186 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:40 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:41.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:40 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T02:30:41.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:40 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:42.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:41 smithi153 ceph-mon[36747]: Deploying daemon osd.4 on smithi153 2024-04-16T02:30:42.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:41 smithi153 ceph-mon[36747]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-16T02:30:42.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:41 smithi151 ceph-mon[29003]: Deploying daemon osd.4 on smithi153 2024-04-16T02:30:42.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:41 smithi151 ceph-mon[29003]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-16T02:30:43.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:42 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:30:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:30:43.984 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:43 smithi153 ceph-mon[36747]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:44.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:43 smithi151 ceph-mon[29003]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:45.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:45 smithi153 ceph-mon[36747]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:45.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:45.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:45.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:45.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:45 smithi151 ceph-mon[29003]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:45.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:45.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:45.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:48.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:47 smithi153 ceph-mon[36747]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:48.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:47 smithi151 ceph-mon[29003]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.767+0000 7f975d7fa700 1 -- 172.21.15.153:0/1349105435 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f97600611d0 con 0x7f974806c4e0 2024-04-16T02:30:48.769 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 4 on host 'smithi153' 2024-04-16T02:30:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.769+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f974806c4e0 msgr2=0x7f974806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.769+0000 7f9766382700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f974806c4e0 0x7f974806e9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f975000b5c0 tx=0x7f97500052a0 comp rx=0 tx=0).stop 2024-04-16T02:30:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.769+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760103fa0 msgr2=0x7f9760196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.769+0000 7f9766382700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760103fa0 0x7f9760196870 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f975400c930 tx=0x7f975400cc40 comp rx=0 tx=0).stop 2024-04-16T02:30:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.770+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 shutdown_connections 2024-04-16T02:30:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.770+0000 7f9766382700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f974806c4e0 0x7f974806e9a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.770+0000 7f9766382700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9760101670 0x7f9760196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.770+0000 7f9766382700 1 --2- 172.21.15.153:0/1349105435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9760103fa0 0x7f9760196870 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.770+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 >> 172.21.15.153:0/1349105435 conn(0x7f97600fb080 msgr2=0x7f97600fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:30:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.770+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 shutdown_connections 2024-04-16T02:30:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:48.770+0000 7f9766382700 1 -- 172.21.15.153:0/1349105435 wait complete. 2024-04-16T02:30:49.363 DEBUG:teuthology.orchestra.run.smithi153:osd.4> sudo journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.4.service 2024-04-16T02:30:49.366 INFO:tasks.cephadm:Deploying osd.5 on smithi153 with /dev/vg_nvme/lv_2... 2024-04-16T02:30:49.366 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-16T02:30:49.638 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:30:49 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[49579]: 2024-04-16T02:30:49.529+0000 7f077ea3f640 -1 osd.4 0 log_to_monitors true 2024-04-16T02:30:50.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:49 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:49 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:49 smithi153 ceph-mon[36747]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:49 smithi153 ceph-mon[36747]: from='osd.4 [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T02:30:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:49 smithi153 ceph-mon[36747]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T02:30:50.143 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T02:30:50.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:49 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:49 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:49 smithi151 ceph-mon[29003]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:49 smithi151 ceph-mon[29003]: from='osd.4 [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T02:30:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:49 smithi151 ceph-mon[29003]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T02:30:50.960 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:30:50 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[49579]: 2024-04-16T02:30:50.760+0000 7f07750a5700 -1 osd.4 0 waiting for initial osdmap 2024-04-16T02:30:50.960 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:30:50 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[49579]: 2024-04-16T02:30:50.769+0000 7f076be90700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:30:50.961 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:50 smithi153 ceph-mon[36747]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T02:30:50.961 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:50 smithi153 ceph-mon[36747]: osdmap e33: 5 total, 4 up, 5 in 2024-04-16T02:30:50.961 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:50 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:50.961 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:50 smithi153 ceph-mon[36747]: from='osd.4 [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:30:50.961 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:50 smithi153 ceph-mon[36747]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:30:51.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:50 smithi151 ceph-mon[29003]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T02:30:51.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:50 smithi151 ceph-mon[29003]: osdmap e33: 5 total, 4 up, 5 in 2024-04-16T02:30:51.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:51.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:50 smithi151 ceph-mon[29003]: from='osd.4 [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:30:51.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:50 smithi151 ceph-mon[29003]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:30:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:51 smithi153 ceph-mon[36747]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-16T02:30:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:51 smithi153 ceph-mon[36747]: osdmap e34: 5 total, 4 up, 5 in 2024-04-16T02:30:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:51 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:51 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:51 smithi153 ceph-mon[36747]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:51 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:51 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:52.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:51 smithi151 ceph-mon[29003]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-16T02:30:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:51 smithi151 ceph-mon[29003]: osdmap e34: 5 total, 4 up, 5 in 2024-04-16T02:30:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:51 smithi151 ceph-mon[29003]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-16T02:30:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:51 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:52.286 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-16T02:30:52.286 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-16T02:30:52.287 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-16T02:30:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-16T02:30:52.287 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127869 s, 820 MB/s 2024-04-16T02:30:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-16T02:30:52.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-04-16T02:30:53.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:52 smithi153 ceph-mon[36747]: purged_snaps scrub starts 2024-04-16T02:30:53.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:52 smithi153 ceph-mon[36747]: purged_snaps scrub ok 2024-04-16T02:30:53.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:52 smithi153 ceph-mon[36747]: osd.4 [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] boot 2024-04-16T02:30:53.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:52 smithi153 ceph-mon[36747]: osdmap e35: 5 total, 5 up, 5 in 2024-04-16T02:30:53.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:52 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:53.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:52 smithi151 ceph-mon[29003]: purged_snaps scrub starts 2024-04-16T02:30:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:52 smithi151 ceph-mon[29003]: purged_snaps scrub ok 2024-04-16T02:30:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:52 smithi151 ceph-mon[29003]: osd.4 [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] boot 2024-04-16T02:30:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:52 smithi151 ceph-mon[29003]: osdmap e35: 5 total, 5 up, 5 in 2024-04-16T02:30:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:52 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:30:53.782 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T02:30:53.904 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:53 smithi153 ceph-mon[36747]: osdmap e36: 5 total, 5 up, 5 in 2024-04-16T02:30:53.904 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:53 smithi153 ceph-mon[36747]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:30:54.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:53 smithi151 ceph-mon[29003]: osdmap e36: 5 total, 5 up, 5 in 2024-04-16T02:30:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:53 smithi151 ceph-mon[29003]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:30:55.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.935+0000 7fedda2f0700 1 -- 172.21.15.153:0/1577117069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 msgr2=0x7fedd4100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:55.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.935+0000 7fedda2f0700 1 --2- 172.21.15.153:0/1577117069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 0x7fedd4100ff0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009910 tx=0x7fedc4009c20 comp rx=0 tx=0).stop 2024-04-16T02:30:55.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.935+0000 7fedda2f0700 1 -- 172.21.15.153:0/1577117069 shutdown_connections 2024-04-16T02:30:55.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.935+0000 7fedda2f0700 1 --2- 172.21.15.153:0/1577117069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fedd4101dd0 0x7fedd4102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:55.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.935+0000 7fedda2f0700 1 --2- 172.21.15.153:0/1577117069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 0x7fedd4100ff0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:55.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.935+0000 7fedda2f0700 1 -- 172.21.15.153:0/1577117069 >> 172.21.15.153:0/1577117069 conn(0x7fedd40fc190 msgr2=0x7fedd40fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:30:55.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.935+0000 7fedda2f0700 1 -- 172.21.15.153:0/1577117069 shutdown_connections 2024-04-16T02:30:55.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.935+0000 7fedda2f0700 1 -- 172.21.15.153:0/1577117069 wait complete. 2024-04-16T02:30:55.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.936+0000 7fedda2f0700 1 Processor -- start 2024-04-16T02:30:55.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.936+0000 7fedda2f0700 1 -- start start 2024-04-16T02:30:55.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.936+0000 7fedda2f0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 0x7fedd4196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:30:55.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.936+0000 7fedda2f0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fedd4101dd0 0x7fedd4196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:30:55.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.936+0000 7fedda2f0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd4196eb0 con 0x7fedd4101dd0 2024-04-16T02:30:55.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.936+0000 7fedda2f0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd4196ff0 con 0x7fedd4100bd0 2024-04-16T02:30:55.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.936+0000 7fedd3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 0x7fedd4196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:30:55.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.936+0000 7fedd3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 0x7fedd4196350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49984/0 (socket says 172.21.15.153:49984) 2024-04-16T02:30:55.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.936+0000 7fedd3fff700 1 -- 172.21.15.153:0/3853048416 learned_addr learned my addr 172.21.15.153:0/3853048416 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:30:55.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedd37fe700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fedd4101dd0 0x7fedd4196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedd3fff700 1 -- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fedd4101dd0 msgr2=0x7fedd4196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedd3fff700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fedd4101dd0 0x7fedd4196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedd3fff700 1 -- 172.21.15.153:0/3853048416 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc40095f0 con 0x7fedd4100bd0 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedd3fff700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 0x7fedd4196350 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fedc400bda0 tx=0x7fedc400be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedd17fa700 1 -- 172.21.15.153:0/3853048416 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedc401d070 con 0x7fedd4100bd0 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedd17fa700 1 -- 172.21.15.153:0/3853048416 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fedc400f460 con 0x7fedd4100bd0 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd419ba40 con 0x7fedd4100bd0 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedd17fa700 1 -- 172.21.15.153:0/3853048416 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedc4021640 con 0x7fedd4100bd0 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.937+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd419bf30 con 0x7fedd4100bd0 2024-04-16T02:30:55.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.938+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd4109730 con 0x7fedd4100bd0 2024-04-16T02:30:55.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.938+0000 7fedd17fa700 1 -- 172.21.15.153:0/3853048416 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fedc400fa00 con 0x7fedd4100bd0 2024-04-16T02:30:55.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.939+0000 7fedd17fa700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fedbc06c4e0 0x7fedbc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:30:55.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.939+0000 7fedd17fa700 1 -- 172.21.15.153:0/3853048416 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4206+0+0 (secure 0 0 0) 0x7fedc408c630 con 0x7fedd4100bd0 2024-04-16T02:30:55.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.939+0000 7fedd37fe700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fedbc06c4e0 0x7fedbc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:30:55.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.940+0000 7fedd37fe700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fedbc06c4e0 0x7fedbc06e9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005950 tx=0x7fedc80058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:30:55.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:55.943+0000 7fedd17fa700 1 -- 172.21.15.153:0/3853048416 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fedc405ad90 con 0x7fedd4100bd0 2024-04-16T02:30:55.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:55 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:56.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:30:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:55 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:30:56.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:30:56.142+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fedd4106720 con 0x7fedbc06c4e0 2024-04-16T02:30:56.902 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:56 smithi153 ceph-mon[36747]: Detected new or changed devices on smithi153 2024-04-16T02:30:56.902 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:56 smithi153 ceph-mon[36747]: Adjusting osd_memory_target on smithi153 to 7456M 2024-04-16T02:30:56.902 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:56 smithi153 ceph-mon[36747]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:30:56.902 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:30:56.902 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:30:56.902 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:56 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:56.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:56 smithi151 ceph-mon[29003]: Detected new or changed devices on smithi153 2024-04-16T02:30:56.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:56 smithi151 ceph-mon[29003]: Adjusting osd_memory_target on smithi153 to 7456M 2024-04-16T02:30:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:56 smithi151 ceph-mon[29003]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:30:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T02:30:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T02:30:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:56 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:30:57.940 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:57 smithi153 ceph-mon[36747]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:30:57.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:57 smithi151 ceph-mon[29003]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:30:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:30:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:58 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/2627835035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d0d2460-ef76-481b-ad51-afb80d05bc72"}]: dispatch 2024-04-16T02:30:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:58 smithi151 ceph-mon[29003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d0d2460-ef76-481b-ad51-afb80d05bc72"}]: dispatch 2024-04-16T02:30:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:58 smithi151 ceph-mon[29003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d0d2460-ef76-481b-ad51-afb80d05bc72"}]': finished 2024-04-16T02:30:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:58 smithi151 ceph-mon[29003]: osdmap e37: 6 total, 5 up, 6 in 2024-04-16T02:30:58.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:58 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:30:59.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:30:59.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:58 smithi153 ceph-mon[36747]: from='client.? 172.21.15.153:0/2627835035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d0d2460-ef76-481b-ad51-afb80d05bc72"}]: dispatch 2024-04-16T02:30:59.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:58 smithi153 ceph-mon[36747]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d0d2460-ef76-481b-ad51-afb80d05bc72"}]: dispatch 2024-04-16T02:30:59.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:58 smithi153 ceph-mon[36747]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d0d2460-ef76-481b-ad51-afb80d05bc72"}]': finished 2024-04-16T02:30:59.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:58 smithi153 ceph-mon[36747]: osdmap e37: 6 total, 5 up, 6 in 2024-04-16T02:30:59.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:58 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:30:59.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:59 smithi151 ceph-mon[29003]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:30:59.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:30:59 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/2671439018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:31:00.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:59 smithi153 ceph-mon[36747]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:00.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:30:59 smithi153 ceph-mon[36747]: from='client.? 172.21.15.153:0/2671439018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T02:31:02.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:01 smithi151 ceph-mon[29003]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:02.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:01 smithi153 ceph-mon[36747]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:04.101 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:03 smithi153 ceph-mon[36747]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:04.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:03 smithi151 ceph-mon[29003]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:05.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:05 smithi151 ceph-mon[29003]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:06.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:05 smithi153 ceph-mon[36747]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:06.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T02:31:06.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:31:06.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:06 smithi151 ceph-mon[29003]: Deploying daemon osd.5 on smithi153 2024-04-16T02:31:07.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T02:31:07.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:31:07.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:06 smithi153 ceph-mon[36747]: Deploying daemon osd.5 on smithi153 2024-04-16T02:31:07.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:07 smithi153 ceph-mon[36747]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:07.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:07 smithi151 ceph-mon[29003]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:10.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:09 smithi153 ceph-mon[36747]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:10.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:09 smithi151 ceph-mon[29003]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:12.354 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:12 smithi153 ceph-mon[36747]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:12.355 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:12.355 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:12.355 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:31:12.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:12 smithi151 ceph-mon[29003]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:12.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:12 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:12.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:12 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:12.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:12 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:31:13.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:13 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:31:13.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:13 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:31:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.851+0000 7fedd17fa700 1 -- 172.21.15.153:0/3853048416 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fedd4106720 con 0x7fedbc06c4e0 2024-04-16T02:31:13.853 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 5 on host 'smithi153' 2024-04-16T02:31:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fedbc06c4e0 msgr2=0x7fedbc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fedbc06c4e0 0x7fedbc06e9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005950 tx=0x7fedc80058e0 comp rx=0 tx=0).stop 2024-04-16T02:31:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 msgr2=0x7fedd4196350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 0x7fedd4196350 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fedc400bda0 tx=0x7fedc400be80 comp rx=0 tx=0).stop 2024-04-16T02:31:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 shutdown_connections 2024-04-16T02:31:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fedbc06c4e0 0x7fedbc06e9a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:13.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedd4100bd0 0x7fedd4196350 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:13.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 --2- 172.21.15.153:0/3853048416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fedd4101dd0 0x7fedd4196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:13.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 >> 172.21.15.153:0/3853048416 conn(0x7fedd40fc190 msgr2=0x7fedd4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:13.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 shutdown_connections 2024-04-16T02:31:13.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:13.854+0000 7fedda2f0700 1 -- 172.21.15.153:0/3853048416 wait complete. 2024-04-16T02:31:14.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:14 smithi153 ceph-mon[36747]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:14.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:14 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:14.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:14 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:14.427 DEBUG:teuthology.orchestra.run.smithi153:osd.5> sudo journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.5.service 2024-04-16T02:31:14.429 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-16T02:31:14.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-16T02:31:14.462 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:14 smithi151 ceph-mon[29003]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:14.462 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:14 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:14.462 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:14 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:14.654 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:15.270 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:31:14 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[56540]: 2024-04-16T02:31:14.958+0000 7f8449363640 -1 osd.5 0 log_to_monitors true 2024-04-16T02:31:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:15 smithi151 ceph-mon[29003]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:15 smithi151 ceph-mon[29003]: from='osd.5 [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T02:31:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:15 smithi151 ceph-mon[29003]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T02:31:16.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:15 smithi153 ceph-mon[36747]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:16.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:15 smithi153 ceph-mon[36747]: from='osd.5 [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T02:31:16.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:15 smithi153 ceph-mon[36747]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T02:31:16.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.351+0000 7fa67a39f700 1 -- 172.21.15.151:0/1524782842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 msgr2=0x7fa674106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:16.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.351+0000 7fa67a39f700 1 --2- 172.21.15.151:0/1524782842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 0x7fa674106190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa668009910 tx=0x7fa668009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:16.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.351+0000 7fa67a39f700 1 -- 172.21.15.151:0/1524782842 shutdown_connections 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.351+0000 7fa67a39f700 1 --2- 172.21.15.151:0/1524782842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 0x7fa674106190 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.351+0000 7fa67a39f700 1 --2- 172.21.15.151:0/1524782842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa674101470 0x7fa674103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.351+0000 7fa67a39f700 1 -- 172.21.15.151:0/1524782842 >> 172.21.15.151:0/1524782842 conn(0x7fa6740fae40 msgr2=0x7fa6740fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.351+0000 7fa67a39f700 1 -- 172.21.15.151:0/1524782842 shutdown_connections 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.351+0000 7fa67a39f700 1 -- 172.21.15.151:0/1524782842 wait complete. 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa67a39f700 1 Processor -- start 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa67a39f700 1 -- start start 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa67a39f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa674101470 0x7fa674196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa67a39f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 0x7fa6741966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa67a39f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa674196ce0 con 0x7fa674103da0 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa67a39f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa674196e20 con 0x7fa674101470 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa678b9c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 0x7fa6741966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:16.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa678b9c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 0x7fa6741966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44258/0 (socket says 172.21.15.151:44258) 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa678b9c700 1 -- 172.21.15.151:0/2733292741 learned_addr learned my addr 172.21.15.151:0/2733292741 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.352+0000 7fa67939d700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa674101470 0x7fa674196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.353+0000 7fa678b9c700 1 -- 172.21.15.151:0/2733292741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa674101470 msgr2=0x7fa674196180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.353+0000 7fa678b9c700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa674101470 0x7fa674196180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.353+0000 7fa678b9c700 1 -- 172.21.15.151:0/2733292741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6680095f0 con 0x7fa674103da0 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.353+0000 7fa678b9c700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 0x7fa6741966c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa668000c00 tx=0x7fa66800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.353+0000 7fa6727fc700 1 -- 172.21.15.151:0/2733292741 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa66801d070 con 0x7fa674103da0 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.353+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67419b870 con 0x7fa674103da0 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.353+0000 7fa6727fc700 1 -- 172.21.15.151:0/2733292741 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa668005dc0 con 0x7fa674103da0 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.353+0000 7fa6727fc700 1 -- 172.21.15.151:0/2733292741 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6680226a0 con 0x7fa674103da0 2024-04-16T02:31:16.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.353+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa67419bd60 con 0x7fa674103da0 2024-04-16T02:31:16.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.354+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa674190350 con 0x7fa674103da0 2024-04-16T02:31:16.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.354+0000 7fa6727fc700 1 -- 172.21.15.151:0/2733292741 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa6680038f0 con 0x7fa674103da0 2024-04-16T02:31:16.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.355+0000 7fa6727fc700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa65c06c4e0 0x7fa65c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:16.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.355+0000 7fa67939d700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa65c06c4e0 0x7fa65c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:16.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.355+0000 7fa6727fc700 1 -- 172.21.15.151:0/2733292741 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fa66808c900 con 0x7fa674103da0 2024-04-16T02:31:16.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.355+0000 7fa67939d700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa65c06c4e0 0x7fa65c06e9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa66400f540 tx=0x7fa66400b040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:16.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.360+0000 7fa6727fc700 1 -- 172.21.15.151:0/2733292741 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa66805afd0 con 0x7fa674103da0 2024-04-16T02:31:16.554 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.553+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa67402cc70 con 0x7fa674103da0 2024-04-16T02:31:16.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.553+0000 7fa6727fc700 1 -- 172.21.15.151:0/2733292741 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7fa668003680 con 0x7fa674103da0 2024-04-16T02:31:16.555 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.555+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa65c06c4e0 msgr2=0x7fa65c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.555+0000 7fa67a39f700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa65c06c4e0 0x7fa65c06e9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa66400f540 tx=0x7fa66400b040 comp rx=0 tx=0).stop 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.555+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 msgr2=0x7fa6741966c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.555+0000 7fa67a39f700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 0x7fa6741966c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa668000c00 tx=0x7fa66800f690 comp rx=0 tx=0).stop 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.556+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 shutdown_connections 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.556+0000 7fa67a39f700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa65c06c4e0 0x7fa65c06e9a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.556+0000 7fa67a39f700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa674101470 0x7fa674196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.556+0000 7fa67a39f700 1 --2- 172.21.15.151:0/2733292741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa674103da0 0x7fa6741966c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.556+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 >> 172.21.15.151:0/2733292741 conn(0x7fa6740fae40 msgr2=0x7fa6740fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.556+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 shutdown_connections 2024-04-16T02:31:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:16.556+0000 7fa67a39f700 1 -- 172.21.15.151:0/2733292741 wait complete. 2024-04-16T02:31:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:16 smithi151 ceph-mon[29003]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T02:31:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:16 smithi151 ceph-mon[29003]: osdmap e38: 6 total, 5 up, 6 in 2024-04-16T02:31:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:16 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:16 smithi151 ceph-mon[29003]: from='osd.5 [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:31:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:16 smithi151 ceph-mon[29003]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:31:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:16 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:16 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:16 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2733292741' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T02:31:17.007 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[56540]: 2024-04-16T02:31:16.735+0000 7f843e1c6700 -1 osd.5 0 waiting for initial osdmap 2024-04-16T02:31:17.007 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[56540]: 2024-04-16T02:31:16.743+0000 7f84387b8700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:31:17.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-mon[36747]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T02:31:17.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-mon[36747]: osdmap e38: 6 total, 5 up, 6 in 2024-04-16T02:31:17.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:17.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-mon[36747]: from='osd.5 [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:31:17.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-mon[36747]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:31:17.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:17.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:17.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:16 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2733292741' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T02:31:17.279 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1713234651,"num_in_osds":6,"osd_in_since":1713234658,"num_remapped_pgs":0} 2024-04-16T02:31:17.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:17 smithi153 ceph-mon[36747]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-16T02:31:17.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:17 smithi153 ceph-mon[36747]: osdmap e39: 6 total, 5 up, 6 in 2024-04-16T02:31:17.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:17.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:17.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:17 smithi153 ceph-mon[36747]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:17.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:17 smithi151 ceph-mon[29003]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-16T02:31:17.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:17 smithi151 ceph-mon[29003]: osdmap e39: 6 total, 5 up, 6 in 2024-04-16T02:31:17.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:17.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:17.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:17 smithi151 ceph-mon[29003]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-16T02:31:18.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-16T02:31:18.493 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:18.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:18 smithi151 ceph-mon[29003]: purged_snaps scrub starts 2024-04-16T02:31:18.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:18 smithi151 ceph-mon[29003]: purged_snaps scrub ok 2024-04-16T02:31:18.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:18.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:18 smithi151 ceph-mon[29003]: osd.5 [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] boot 2024-04-16T02:31:18.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:18 smithi151 ceph-mon[29003]: osdmap e40: 6 total, 6 up, 6 in 2024-04-16T02:31:18.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:19.012 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:18 smithi153 ceph-mon[36747]: purged_snaps scrub starts 2024-04-16T02:31:19.012 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:18 smithi153 ceph-mon[36747]: purged_snaps scrub ok 2024-04-16T02:31:19.012 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:19.012 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:18 smithi153 ceph-mon[36747]: osd.5 [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] boot 2024-04-16T02:31:19.012 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:18 smithi153 ceph-mon[36747]: osdmap e40: 6 total, 6 up, 6 in 2024-04-16T02:31:19.012 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: osdmap e41: 6 total, 6 up, 6 in 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:31:19.947 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:31:19.948 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:19.948 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:31:19.948 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:31:19.948 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:31:19.948 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:19 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: osdmap e41: 6 total, 6 up, 6 in 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:31:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:31:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:31:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:31:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:19 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:31:20.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.173+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3490408517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c101db0 msgr2=0x7f4f9c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.173+0000 7f4fa20a8700 1 --2- 172.21.15.151:0/3490408517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c101db0 0x7f4f9c102210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4f84009910 tx=0x7f4f84009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.173+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3490408517 shutdown_connections 2024-04-16T02:31:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.173+0000 7f4fa20a8700 1 --2- 172.21.15.151:0/3490408517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c101db0 0x7f4f9c102210 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.173+0000 7f4fa20a8700 1 --2- 172.21.15.151:0/3490408517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f9c100bb0 0x7f4f9c100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.173+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3490408517 >> 172.21.15.151:0/3490408517 conn(0x7f4f9c0fc110 msgr2=0x7f4f9c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.173+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3490408517 shutdown_connections 2024-04-16T02:31:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.173+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3490408517 wait complete. 2024-04-16T02:31:20.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.173+0000 7f4fa20a8700 1 Processor -- start 2024-04-16T02:31:20.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4fa20a8700 1 -- start start 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4fa20a8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c100bb0 0x7f4f9c196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4fa20a8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f9c101db0 0x7f4f9c196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4fa20a8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c196f60 con 0x7f4f9c100bb0 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4fa20a8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c1970a0 con 0x7f4f9c101db0 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4f9b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c100bb0 0x7f4f9c196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4f9b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c100bb0 0x7f4f9c196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58016/0 (socket says 172.21.15.151:58016) 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4f9b7fe700 1 -- 172.21.15.151:0/3845790725 learned_addr learned my addr 172.21.15.151:0/3845790725 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4f9affd700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f9c101db0 0x7f4f9c196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4f9b7fe700 1 -- 172.21.15.151:0/3845790725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f9c101db0 msgr2=0x7f4f9c196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4f9b7fe700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f9c101db0 0x7f4f9c196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4f9b7fe700 1 -- 172.21.15.151:0/3845790725 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f840095f0 con 0x7f4f9c100bb0 2024-04-16T02:31:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4f9b7fe700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c100bb0 0x7f4f9c196400 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00b9c0 tx=0x7f4f8c00bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.174+0000 7f4f98ff9700 1 -- 172.21.15.151:0/3845790725 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c00d8b0 con 0x7f4f9c100bb0 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.175+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c19bb50 con 0x7f4f9c100bb0 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.175+0000 7f4f98ff9700 1 -- 172.21.15.151:0/3845790725 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4f8c015420 con 0x7f4f9c100bb0 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.175+0000 7f4f98ff9700 1 -- 172.21.15.151:0/3845790725 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c00c610 con 0x7f4f9c100bb0 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.175+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c19c0a0 con 0x7f4f9c100bb0 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.176+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c1097b0 con 0x7f4f9c100bb0 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.176+0000 7f4f98ff9700 1 -- 172.21.15.151:0/3845790725 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4f8c015590 con 0x7f4f9c100bb0 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.176+0000 7f4f98ff9700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4f8806c4e0 0x7f4f8806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.177+0000 7f4f9affd700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4f8806c4e0 0x7f4f8806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.177+0000 7f4f98ff9700 1 -- 172.21.15.151:0/3845790725 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4f8c011070 con 0x7f4f9c100bb0 2024-04-16T02:31:20.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.177+0000 7f4f9affd700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4f8806c4e0 0x7f4f8806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400b5c0 tx=0x7f4f840052a0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:20.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.181+0000 7f4f98ff9700 1 -- 172.21.15.151:0/3845790725 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4f8c04f0e0 con 0x7f4f9c100bb0 2024-04-16T02:31:20.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.370+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f4f9c1099a0 con 0x7f4f9c100bb0 2024-04-16T02:31:20.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.370+0000 7f4f98ff9700 1 -- 172.21.15.151:0/3845790725 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f4f8c0594f0 con 0x7f4f9c100bb0 2024-04-16T02:31:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:31:20.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4f8806c4e0 msgr2=0x7f4f8806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4f8806c4e0 0x7f4f8806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400b5c0 tx=0x7f4f840052a0 comp rx=0 tx=0).stop 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c100bb0 msgr2=0x7f4f9c196400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c100bb0 0x7f4f9c196400 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00b9c0 tx=0x7f4f8c00bcd0 comp rx=0 tx=0).stop 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 shutdown_connections 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4f8806c4e0 0x7f4f8806e9a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f9c100bb0 0x7f4f9c196400 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 --2- 172.21.15.151:0/3845790725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f9c101db0 0x7f4f9c196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 >> 172.21.15.151:0/3845790725 conn(0x7f4f9c0fc110 msgr2=0x7f4f9c104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 shutdown_connections 2024-04-16T02:31:20.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:20.372+0000 7f4fa20a8700 1 -- 172.21.15.151:0/3845790725 wait complete. 2024-04-16T02:31:20.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:20 smithi151 ceph-mon[29003]: Detected new or changed devices on smithi153 2024-04-16T02:31:20.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:20 smithi151 ceph-mon[29003]: Adjusting osd_memory_target on smithi153 to 4970M 2024-04-16T02:31:20.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:20 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3845790725' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T02:31:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:20 smithi153 ceph-mon[36747]: Detected new or changed devices on smithi153 2024-04-16T02:31:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:20 smithi153 ceph-mon[36747]: Adjusting osd_memory_target on smithi153 to 4970M 2024-04-16T02:31:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:20 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3845790725' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T02:31:21.152 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1713234677,"num_in_osds":6,"osd_in_since":1713234658,"num_remapped_pgs":0} 2024-04-16T02:31:21.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T02:31:21.367 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:22.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:21 smithi153 ceph-mon[36747]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:22.152 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:21 smithi151 ceph-mon[29003]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:23.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 -- 172.21.15.151:0/2787573681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350101d60 msgr2=0x7f6350102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 --2- 172.21.15.151:0/2787573681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350101d60 0x7f6350102140 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6338009940 tx=0x7f6338009c50 comp rx=0 tx=0).stop 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 -- 172.21.15.151:0/2787573681 shutdown_connections 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 --2- 172.21.15.151:0/2787573681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6350102680 0x7f635010ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 --2- 172.21.15.151:0/2787573681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350101d60 0x7f6350102140 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 -- 172.21.15.151:0/2787573681 >> 172.21.15.151:0/2787573681 conn(0x7f63500fb550 msgr2=0x7f63500fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 -- 172.21.15.151:0/2787573681 shutdown_connections 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 -- 172.21.15.151:0/2787573681 wait complete. 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 Processor -- start 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.107+0000 7f63566cd700 1 -- start start 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f63566cd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6350101d60 0x7f635019c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f63566cd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350102680 0x7f635019ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f63566cd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635019d0f0 con 0x7f6350102680 2024-04-16T02:31:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f63566cd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63501965a0 con 0x7f6350101d60 2024-04-16T02:31:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f634f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350102680 0x7f635019ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f634f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350102680 0x7f635019ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58034/0 (socket says 172.21.15.151:58034) 2024-04-16T02:31:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f634f7fe700 1 -- 172.21.15.151:0/231034478 learned_addr learned my addr 172.21.15.151:0/231034478 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f634ffff700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6350101d60 0x7f635019c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f634f7fe700 1 -- 172.21.15.151:0/231034478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6350101d60 msgr2=0x7f635019c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f634f7fe700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6350101d60 0x7f635019c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.108+0000 7f634f7fe700 1 -- 172.21.15.151:0/231034478 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63380095f0 con 0x7f6350102680 2024-04-16T02:31:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.109+0000 7f634f7fe700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350102680 0x7f635019ca60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f634000c8f0 tx=0x7f634000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.109+0000 7f634d7fa700 1 -- 172.21.15.151:0/231034478 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6340007ad0 con 0x7f6350102680 2024-04-16T02:31:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.109+0000 7f634d7fa700 1 -- 172.21.15.151:0/231034478 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f634000ce90 con 0x7f6350102680 2024-04-16T02:31:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.109+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6350196880 con 0x7f6350102680 2024-04-16T02:31:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.109+0000 7f634d7fa700 1 -- 172.21.15.151:0/231034478 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6340018630 con 0x7f6350102680 2024-04-16T02:31:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.109+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6350196dd0 con 0x7f6350102680 2024-04-16T02:31:23.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.111+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6350108270 con 0x7f6350102680 2024-04-16T02:31:23.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.111+0000 7f634d7fa700 1 -- 172.21.15.151:0/231034478 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f634000f450 con 0x7f6350102680 2024-04-16T02:31:23.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.111+0000 7f634d7fa700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f633c06c4e0 0x7f633c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:23.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.111+0000 7f634d7fa700 1 -- 172.21.15.151:0/231034478 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f634008b480 con 0x7f6350102680 2024-04-16T02:31:23.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.111+0000 7f634ffff700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f633c06c4e0 0x7f633c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:23.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.112+0000 7f634ffff700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f633c06c4e0 0x7f633c06e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63380059d0 tx=0x7f6338005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:23.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.116+0000 7f634d7fa700 1 -- 172.21.15.151:0/231034478 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6340059a30 con 0x7f6350102680 2024-04-16T02:31:23.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.307+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f635004ea90 con 0x7f6350102680 2024-04-16T02:31:23.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.308+0000 7f634d7fa700 1 -- 172.21.15.151:0/231034478 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11396 (secure 0 0 0) 0x7f63400595c0 con 0x7f6350102680 2024-04-16T02:31:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:31:23.309 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":41,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","created":"2024-04-16T02:25:34.606206+0000","modified":"2024-04-16T02:31:18.735658+0000","last_up_change":"2024-04-16T02:31:17.730938+0000","last_in_change":"2024-04-16T02:30:58.536477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T02:30:03.137221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8da4d50b-626b-4cf3-9524-ea84e7733113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6803","nonce":1205378476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6805","nonce":1205378476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6809","nonce":1205378476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6807","nonce":1205378476}]},"public_addr":"172.21.15.151:6803/1205378476","cluster_addr":"172.21.15.151:6805/1205378476","heartbeat_back_addr":"172.21.15.151:6809/1205378476","heartbeat_front_addr":"172.21.15.151:6807/1205378476","state":["exists","up"]},{"osd":1,"uuid":"8d2980b5-1d30-445c-91ae-f08074b6d8f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6811","nonce":1648693895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6813","nonce":1648693895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6817","nonce":1648693895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6815","nonce":1648693895}]},"public_addr":"172.21.15.151:6811/1648693895","cluster_addr":"172.21.15.151:6813/1648693895","heartbeat_back_addr":"172.21.15.151:6817/1648693895","heartbeat_front_addr":"172.21.15.151:6815/1648693895","state":["exists","up"]},{"osd":2,"uuid":"64e4e7ce-5c3c-48c8-be3d-c9936b6866b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6819","nonce":2687133389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6821","nonce":2687133389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6825","nonce":2687133389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6823","nonce":2687133389}]},"public_addr":"172.21.15.151:6819/2687133389","cluster_addr":"172.21.15.151:6821/2687133389","heartbeat_back_addr":"172.21.15.151:6825/2687133389","heartbeat_front_addr":"172.21.15.151:6823/2687133389","state":["exists","up"]},{"osd":3,"uuid":"1194f132-bf2d-4e0c-ab93-f3344ff64d76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6801","nonce":2298379944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6803","nonce":2298379944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6807","nonce":2298379944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6805","nonce":2298379944}]},"public_addr":"172.21.15.153:6801/2298379944","cluster_addr":"172.21.15.153:6803/2298379944","heartbeat_back_addr":"172.21.15.153:6807/2298379944","heartbeat_front_addr":"172.21.15.153:6805/2298379944","state":["exists","up"]},{"osd":4,"uuid":"9d803d8b-eb35-4a0f-9d36-de11b743a123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6809","nonce":2873952441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6811","nonce":2873952441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6815","nonce":2873952441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6813","nonce":2873952441}]},"public_addr":"172.21.15.153:6809/2873952441","cluster_addr":"172.21.15.153:6811/2873952441","heartbeat_back_addr":"172.21.15.153:6815/2873952441","heartbeat_front_addr":"172.21.15.153:6813/2873952441","state":["exists","up"]},{"osd":5,"uuid":"7d0d2460-ef76-481b-ad51-afb80d05bc72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6817","nonce":1792791923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6819","nonce":1792791923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6823","nonce":1792791923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6821","nonce":1792791923}]},"public_addr":"172.21.15.153:6817/1792791923","cluster_addr":"172.21.15.153:6819/1792791923","heartbeat_back_addr":"172.21.15.153:6823/1792791923","heartbeat_front_addr":"172.21.15.153:6821/1792791923","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:02.729782+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:31.268279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:59.891666+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:24.930390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:50.555079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:31:15.969056+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6800/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6801/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6800/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3454798699":"2024-04-17T02:26:34.117371+0000","172.21.15.151:0/681541897":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6800/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/2293925528":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/4272586248":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/3129110405":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3798542210":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/969025109":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/1923729027":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/2684608614":"2024-04-17T02:26:34.117371+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T02:31:23.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f633c06c4e0 msgr2=0x7f633c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:23.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f633c06c4e0 0x7f633c06e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63380059d0 tx=0x7f6338005960 comp rx=0 tx=0).stop 2024-04-16T02:31:23.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350102680 msgr2=0x7f635019ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:23.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350102680 0x7f635019ca60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f634000c8f0 tx=0x7f634000ccb0 comp rx=0 tx=0).stop 2024-04-16T02:31:23.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 shutdown_connections 2024-04-16T02:31:23.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f633c06c4e0 0x7f633c06e9a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:23.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6350101d60 0x7f635019c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:23.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 --2- 172.21.15.151:0/231034478 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350102680 0x7f635019ca60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:23.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 >> 172.21.15.151:0/231034478 conn(0x7f63500fb550 msgr2=0x7f63501053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:23.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 shutdown_connections 2024-04-16T02:31:23.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:23.310+0000 7f63566cd700 1 -- 172.21.15.151:0/231034478 wait complete. 2024-04-16T02:31:23.935 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T02:30:03.137221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T02:31:23.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-16T02:31:24.147 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:24.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:23 smithi151 ceph-mon[29003]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:24.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:23 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/231034478' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T02:31:24.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:23 smithi153 ceph-mon[36747]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:23 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/231034478' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T02:31:25.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.737+0000 7f0a06c27700 1 -- 172.21.15.151:0/758135593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00100dd0 msgr2=0x7f0a001011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.737+0000 7f0a06c27700 1 --2- 172.21.15.151:0/758135593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00100dd0 0x7f0a001011b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009910 tx=0x7f09f8009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.737+0000 7f0a06c27700 1 -- 172.21.15.151:0/758135593 shutdown_connections 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.737+0000 7f0a06c27700 1 --2- 172.21.15.151:0/758135593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00101780 0x7f0a001057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.737+0000 7f0a06c27700 1 --2- 172.21.15.151:0/758135593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00100dd0 0x7f0a001011b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.737+0000 7f0a06c27700 1 -- 172.21.15.151:0/758135593 >> 172.21.15.151:0/758135593 conn(0x7f0a000fc640 msgr2=0x7f0a000fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.737+0000 7f0a06c27700 1 -- 172.21.15.151:0/758135593 shutdown_connections 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.737+0000 7f0a06c27700 1 -- 172.21.15.151:0/758135593 wait complete. 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.738+0000 7f0a06c27700 1 Processor -- start 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.738+0000 7f0a06c27700 1 -- start start 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.738+0000 7f0a06c27700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00100dd0 0x7f0a0019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.738+0000 7f0a06c27700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00101780 0x7f0a0019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:25.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.738+0000 7f0a06c27700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0019d0e0 con 0x7f0a00101780 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.738+0000 7f0a06c27700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a00196590 con 0x7f0a00100dd0 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.738+0000 7f09fffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00101780 0x7f0a0019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.738+0000 7f09fffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00101780 0x7f0a0019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58052/0 (socket says 172.21.15.151:58052) 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.738+0000 7f09fffff700 1 -- 172.21.15.151:0/3183978757 learned_addr learned my addr 172.21.15.151:0/3183978757 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f0a049c3700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00100dd0 0x7f0a0019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f09fffff700 1 -- 172.21.15.151:0/3183978757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00100dd0 msgr2=0x7f0a0019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f09fffff700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00100dd0 0x7f0a0019c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f09fffff700 1 -- 172.21.15.151:0/3183978757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f80095f0 con 0x7f0a00101780 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f09fffff700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00101780 0x7f0a0019ca50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f09f000ada0 tx=0x7f09f000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f09fdffb700 1 -- 172.21.15.151:0/3183978757 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09f000f070 con 0x7f0a00101780 2024-04-16T02:31:25.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f09fdffb700 1 -- 172.21.15.151:0/3183978757 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f09f0007bc0 con 0x7f0a00101780 2024-04-16T02:31:25.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a00196870 con 0x7f0a00101780 2024-04-16T02:31:25.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f09fdffb700 1 -- 172.21.15.151:0/3183978757 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09f0005630 con 0x7f0a00101780 2024-04-16T02:31:25.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.739+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a00196dc0 con 0x7f0a00101780 2024-04-16T02:31:25.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.740+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a00109170 con 0x7f0a00101780 2024-04-16T02:31:25.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.741+0000 7f09fdffb700 1 -- 172.21.15.151:0/3183978757 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f09f00058a0 con 0x7f0a00101780 2024-04-16T02:31:25.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.741+0000 7f09fdffb700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f09ec06c4e0 0x7f09ec06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:25.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.741+0000 7f09fdffb700 1 -- 172.21.15.151:0/3183978757 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f09f008ae30 con 0x7f0a00101780 2024-04-16T02:31:25.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.741+0000 7f0a049c3700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f09ec06c4e0 0x7f09ec06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:25.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.741+0000 7f0a049c3700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f09ec06c4e0 0x7f09ec06e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09f800b5c0 tx=0x7f09f8005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:25.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.746+0000 7f09fdffb700 1 -- 172.21.15.151:0/3183978757 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f09f00593e0 con 0x7f0a00101780 2024-04-16T02:31:25.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.933+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a00197060 con 0x7f0a00101780 2024-04-16T02:31:25.934 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:25 smithi151 ceph-mon[29003]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:25.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.933+0000 7f09fdffb700 1 -- 172.21.15.151:0/3183978757 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09f0058f70 con 0x7f0a00101780 2024-04-16T02:31:25.935 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 1 2024-04-16T02:31:25.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.935+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f09ec06c4e0 msgr2=0x7f09ec06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:25.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.935+0000 7f0a06c27700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f09ec06c4e0 0x7f09ec06e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09f800b5c0 tx=0x7f09f8005960 comp rx=0 tx=0).stop 2024-04-16T02:31:25.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.935+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00101780 msgr2=0x7f0a0019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:25.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.935+0000 7f0a06c27700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00101780 0x7f0a0019ca50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f09f000ada0 tx=0x7f09f000c330 comp rx=0 tx=0).stop 2024-04-16T02:31:25.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.936+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 shutdown_connections 2024-04-16T02:31:25.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.936+0000 7f0a06c27700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f09ec06c4e0 0x7f09ec06e9a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:25.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.936+0000 7f0a06c27700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a00100dd0 0x7f0a0019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:25.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.936+0000 7f0a06c27700 1 --2- 172.21.15.151:0/3183978757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a00101780 0x7f0a0019ca50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:25.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.936+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 >> 172.21.15.151:0/3183978757 conn(0x7f0a000fc640 msgr2=0x7f0a000fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:25.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.936+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 shutdown_connections 2024-04-16T02:31:25.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:25.936+0000 7f0a06c27700 1 -- 172.21.15.151:0/3183978757 wait complete. 2024-04-16T02:31:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:25 smithi153 ceph-mon[36747]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:26.605 INFO:tasks.cephadm:Setting up client nodes... 2024-04-16T02:31:26.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-16T02:31:26.819 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:26.941 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:26 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3183978757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T02:31:27.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:26 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3183978757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T02:31:27.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:27 smithi151 ceph-mon[29003]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:28.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:27 smithi153 ceph-mon[36747]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:28.531 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.530+0000 7fef3a1f7700 1 -- 172.21.15.151:0/3749263057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 msgr2=0x7fef340ff2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:28.532 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.530+0000 7fef3a1f7700 1 --2- 172.21.15.151:0/3749263057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 0x7fef340ff2f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fef2400b0a0 tx=0x7fef2400b3b0 comp rx=0 tx=0).stop 2024-04-16T02:31:28.532 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.530+0000 7fef3a1f7700 1 -- 172.21.15.151:0/3749263057 shutdown_connections 2024-04-16T02:31:28.532 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.530+0000 7fef3a1f7700 1 --2- 172.21.15.151:0/3749263057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101eb0 0x7fef34102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:28.532 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.530+0000 7fef3a1f7700 1 --2- 172.21.15.151:0/3749263057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 0x7fef340ff2f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:28.532 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.530+0000 7fef3a1f7700 1 -- 172.21.15.151:0/3749263057 >> 172.21.15.151:0/3749263057 conn(0x7fef340fa810 msgr2=0x7fef340fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:28.532 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.530+0000 7fef3a1f7700 1 -- 172.21.15.151:0/3749263057 shutdown_connections 2024-04-16T02:31:28.532 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.530+0000 7fef3a1f7700 1 -- 172.21.15.151:0/3749263057 wait complete. 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef3a1f7700 1 Processor -- start 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef3a1f7700 1 -- start start 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef3a1f7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 0x7fef34196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef3a1f7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101eb0 0x7fef34196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef3a1f7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef34196fa0 con 0x7fef340feed0 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef3a1f7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef341970e0 con 0x7fef34101eb0 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef337fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 0x7fef34196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef337fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 0x7fef34196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59322/0 (socket says 172.21.15.151:59322) 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef337fe700 1 -- 172.21.15.151:0/4105138893 learned_addr learned my addr 172.21.15.151:0/4105138893 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.531+0000 7fef32ffd700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101eb0 0x7fef34196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:28.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.532+0000 7fef337fe700 1 -- 172.21.15.151:0/4105138893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101eb0 msgr2=0x7fef34196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:28.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.532+0000 7fef337fe700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101eb0 0x7fef34196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:28.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.532+0000 7fef337fe700 1 -- 172.21.15.151:0/4105138893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24009d00 con 0x7fef340feed0 2024-04-16T02:31:28.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.532+0000 7fef337fe700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 0x7fef34196440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fef24006130 tx=0x7fef2400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:28.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.532+0000 7fef30ff9700 1 -- 172.21.15.151:0/4105138893 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2400e040 con 0x7fef340feed0 2024-04-16T02:31:28.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.532+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3419bb30 con 0x7fef340feed0 2024-04-16T02:31:28.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.532+0000 7fef30ff9700 1 -- 172.21.15.151:0/4105138893 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fef240044f0 con 0x7fef340feed0 2024-04-16T02:31:28.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.532+0000 7fef30ff9700 1 -- 172.21.15.151:0/4105138893 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2401d820 con 0x7fef340feed0 2024-04-16T02:31:28.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.532+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3419c020 con 0x7fef340feed0 2024-04-16T02:31:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.533+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef34109770 con 0x7fef340feed0 2024-04-16T02:31:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.533+0000 7fef30ff9700 1 -- 172.21.15.151:0/4105138893 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fef24004040 con 0x7fef340feed0 2024-04-16T02:31:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.534+0000 7fef30ff9700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fef1c06c4e0 0x7fef1c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.534+0000 7fef30ff9700 1 -- 172.21.15.151:0/4105138893 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fef2408ce60 con 0x7fef340feed0 2024-04-16T02:31:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.534+0000 7fef32ffd700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fef1c06c4e0 0x7fef1c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.534+0000 7fef32ffd700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fef1c06c4e0 0x7fef1c06e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef28005950 tx=0x7fef2800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.538+0000 7fef30ff9700 1 -- 172.21.15.151:0/4105138893 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fef2405b410 con 0x7fef340feed0 2024-04-16T02:31:28.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.806+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fef3419c660 con 0x7fef340feed0 2024-04-16T02:31:28.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.809+0000 7fef30ff9700 1 -- 172.21.15.151:0/4105138893 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7fef2405afa0 con 0x7fef340feed0 2024-04-16T02:31:28.811 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-04-16T02:31:28.811 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQAA4x1mITweMBAAuivI40T+Hpv/RX9tiMnpWw== 2024-04-16T02:31:28.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.811+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fef1c06c4e0 msgr2=0x7fef1c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:28.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.811+0000 7fef3a1f7700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fef1c06c4e0 0x7fef1c06e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef28005950 tx=0x7fef2800b410 comp rx=0 tx=0).stop 2024-04-16T02:31:28.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.811+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 msgr2=0x7fef34196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:28.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.811+0000 7fef3a1f7700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 0x7fef34196440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fef24006130 tx=0x7fef2400cd20 comp rx=0 tx=0).stop 2024-04-16T02:31:28.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.811+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 shutdown_connections 2024-04-16T02:31:28.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.811+0000 7fef3a1f7700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fef1c06c4e0 0x7fef1c06e9a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:28.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.812+0000 7fef3a1f7700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340feed0 0x7fef34196440 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:28.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.812+0000 7fef3a1f7700 1 --2- 172.21.15.151:0/4105138893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101eb0 0x7fef34196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:28.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.812+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 >> 172.21.15.151:0/4105138893 conn(0x7fef340fa810 msgr2=0x7fef341050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:28.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.812+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 shutdown_connections 2024-04-16T02:31:28.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:28.812+0000 7fef3a1f7700 1 -- 172.21.15.151:0/4105138893 wait complete. 2024-04-16T02:31:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:28 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:31:29.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:28 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:31:29.470 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:31:29.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-16T02:31:29.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T02:31:29.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-16T02:31:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T02:31:29.861 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:29 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/4105138893' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T02:31:29.861 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:29 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/4105138893' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T02:31:29.861 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:29 smithi153 ceph-mon[36747]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:29.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:29 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/4105138893' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T02:31:29.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:29 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/4105138893' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T02:31:29.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:29 smithi151 ceph-mon[29003]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:31.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.310+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/2833486037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 msgr2=0x7f6d78102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:31.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.310+0000 7f6d7c8c4700 1 --2- 172.21.15.153:0/2833486037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 0x7f6d78102220 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009910 tx=0x7f6d6c009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:31.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.310+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/2833486037 shutdown_connections 2024-04-16T02:31:31.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.310+0000 7f6d7c8c4700 1 --2- 172.21.15.153:0/2833486037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 0x7f6d78102220 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:31.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.310+0000 7f6d7c8c4700 1 --2- 172.21.15.153:0/2833486037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d78100bc0 0x7f6d78100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:31.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.310+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/2833486037 >> 172.21.15.153:0/2833486037 conn(0x7f6d780fc140 msgr2=0x7f6d780fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:31.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.310+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/2833486037 shutdown_connections 2024-04-16T02:31:31.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.310+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/2833486037 wait complete. 2024-04-16T02:31:31.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d7c8c4700 1 Processor -- start 2024-04-16T02:31:31.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d7c8c4700 1 -- start start 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d7c8c4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d78100bc0 0x7f6d78196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d7c8c4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 0x7f6d78196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d7c8c4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78196ea0 con 0x7f6d78100bc0 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d7c8c4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78196fe0 con 0x7f6d78101dc0 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d75d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 0x7f6d78196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d75d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 0x7f6d78196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55278/0 (socket says 172.21.15.153:55278) 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d75d9b700 1 -- 172.21.15.153:0/3459283269 learned_addr learned my addr 172.21.15.153:0/3459283269 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d7659c700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d78100bc0 0x7f6d78196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d75d9b700 1 -- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d78100bc0 msgr2=0x7f6d78196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d75d9b700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d78100bc0 0x7f6d78196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.311+0000 7f6d75d9b700 1 -- 172.21.15.153:0/3459283269 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c0095f0 con 0x7f6d78101dc0 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.312+0000 7f6d75d9b700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 0x7f6d78196880 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c006010 tx=0x7f6d6c00bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:31.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.312+0000 7f6d677fe700 1 -- 172.21.15.153:0/3459283269 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d6c01d070 con 0x7f6d78101dc0 2024-04-16T02:31:31.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.312+0000 7f6d677fe700 1 -- 172.21.15.153:0/3459283269 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6d6c00f460 con 0x7f6d78101dc0 2024-04-16T02:31:31.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.312+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7819ba30 con 0x7f6d78101dc0 2024-04-16T02:31:31.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.312+0000 7f6d677fe700 1 -- 172.21.15.153:0/3459283269 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d6c021640 con 0x7f6d78101dc0 2024-04-16T02:31:31.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.312+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7819bf20 con 0x7f6d78101dc0 2024-04-16T02:31:31.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.313+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d78109720 con 0x7f6d78101dc0 2024-04-16T02:31:31.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.313+0000 7f6d677fe700 1 -- 172.21.15.153:0/3459283269 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6d6c003a60 con 0x7f6d78101dc0 2024-04-16T02:31:31.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.314+0000 7f6d677fe700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f6d6006c4e0 0x7f6d6006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:31.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.314+0000 7f6d677fe700 1 -- 172.21.15.153:0/3459283269 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6d6c08c710 con 0x7f6d78101dc0 2024-04-16T02:31:31.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.314+0000 7f6d7659c700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f6d6006c4e0 0x7f6d6006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:31.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.315+0000 7f6d7659c700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f6d6006c4e0 0x7f6d6006e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d78101c20 tx=0x7f6d6800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:31.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.318+0000 7f6d677fe700 1 -- 172.21.15.153:0/3459283269 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6d6c05ad40 con 0x7f6d78101dc0 2024-04-16T02:31:31.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.579+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f6d7819c670 con 0x7f6d78101dc0 2024-04-16T02:31:31.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.584+0000 7f6d677fe700 1 -- 172.21.15.153:0/3459283269 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f6d6c05a8d0 con 0x7f6d78101dc0 2024-04-16T02:31:31.586 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-04-16T02:31:31.586 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAD4x1mOn2tIhAAKApPpcRK2Eds6ixuNY4riQ== 2024-04-16T02:31:31.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f6d6006c4e0 msgr2=0x7f6d6006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:31.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f6d6006c4e0 0x7f6d6006e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d78101c20 tx=0x7f6d6800b410 comp rx=0 tx=0).stop 2024-04-16T02:31:31.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 msgr2=0x7f6d78196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:31.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 0x7f6d78196880 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c006010 tx=0x7f6d6c00bd70 comp rx=0 tx=0).stop 2024-04-16T02:31:31.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 shutdown_connections 2024-04-16T02:31:31.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f6d6006c4e0 0x7f6d6006e9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:31.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d78100bc0 0x7f6d78196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:31.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 --2- 172.21.15.153:0/3459283269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78101dc0 0x7f6d78196880 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:31.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 >> 172.21.15.153:0/3459283269 conn(0x7f6d780fc140 msgr2=0x7f6d78104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:31.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 shutdown_connections 2024-04-16T02:31:31.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:31:31.586+0000 7f6d7c8c4700 1 -- 172.21.15.153:0/3459283269 wait complete. 2024-04-16T02:31:32.180 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:31 smithi153 ceph-mon[36747]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:32.180 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:31 smithi153 ceph-mon[36747]: from='client.? 172.21.15.153:0/3459283269' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T02:31:32.180 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:31 smithi153 ceph-mon[36747]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T02:31:32.180 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:31 smithi153 ceph-mon[36747]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T02:31:32.202 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:31:32.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-16T02:31:32.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-16T02:31:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:31 smithi151 ceph-mon[29003]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:31 smithi151 ceph-mon[29003]: from='client.? 172.21.15.153:0/3459283269' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T02:31:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:31 smithi151 ceph-mon[29003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T02:31:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:31 smithi151 ceph-mon[29003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T02:31:32.248 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T02:31:32.248 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-16T02:31:32.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph mgr dump --format=json 2024-04-16T02:31:32.474 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:34.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.059+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2425782614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb80ffa40 msgr2=0x7f4cb80ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:34.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.059+0000 7f4cbc80c700 1 --2- 172.21.15.151:0/2425782614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb80ffa40 0x7f4cb80ffec0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009910 tx=0x7f4ca0009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.059+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2425782614 shutdown_connections 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.059+0000 7f4cbc80c700 1 --2- 172.21.15.151:0/2425782614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb80ffa40 0x7f4cb80ffec0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.059+0000 7f4cbc80c700 1 --2- 172.21.15.151:0/2425782614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cb8105a40 0x7f4cb8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.060+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2425782614 >> 172.21.15.151:0/2425782614 conn(0x7f4cb80fb550 msgr2=0x7f4cb80fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.060+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2425782614 shutdown_connections 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.060+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2425782614 wait complete. 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.060+0000 7f4cbc80c700 1 Processor -- start 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.060+0000 7f4cbc80c700 1 -- start start 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cbc80c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cb80ffa40 0x7f4cb8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cbc80c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb8105a40 0x7f4cb8198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cbc80c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb8199520 con 0x7f4cb8105a40 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cbc80c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb819d330 con 0x7f4cb80ffa40 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cb5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb8105a40 0x7f4cb8198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:34.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cb5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb8105a40 0x7f4cb8198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59332/0 (socket says 172.21.15.151:59332) 2024-04-16T02:31:34.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cb5d9b700 1 -- 172.21.15.151:0/2395237289 learned_addr learned my addr 172.21.15.151:0/2395237289 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:34.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cb5d9b700 1 -- 172.21.15.151:0/2395237289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cb80ffa40 msgr2=0x7f4cb8198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:34.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cb5d9b700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cb80ffa40 0x7f4cb8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:34.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cb5d9b700 1 -- 172.21.15.151:0/2395237289 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca00095f0 con 0x7f4cb8105a40 2024-04-16T02:31:34.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cb5d9b700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb8105a40 0x7f4cb8198e90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0006010 tx=0x7f4ca000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:34.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4caf7fe700 1 -- 172.21.15.151:0/2395237289 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca001d070 con 0x7f4cb8105a40 2024-04-16T02:31:34.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb819d5b0 con 0x7f4cb8105a40 2024-04-16T02:31:34.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.061+0000 7f4caf7fe700 1 -- 172.21.15.151:0/2395237289 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4ca000f460 con 0x7f4cb8105a40 2024-04-16T02:31:34.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.062+0000 7f4caf7fe700 1 -- 172.21.15.151:0/2395237289 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca0021640 con 0x7f4cb8105a40 2024-04-16T02:31:34.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.062+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb819daa0 con 0x7f4cb8105a40 2024-04-16T02:31:34.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.062+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb8107fd0 con 0x7f4cb8105a40 2024-04-16T02:31:34.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.063+0000 7f4caf7fe700 1 -- 172.21.15.151:0/2395237289 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4ca000faf0 con 0x7f4cb8105a40 2024-04-16T02:31:34.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.064+0000 7f4caf7fe700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4ca406c4e0 0x7f4ca406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:34.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.064+0000 7f4caf7fe700 1 -- 172.21.15.151:0/2395237289 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4ca008cb10 con 0x7f4cb8105a40 2024-04-16T02:31:34.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.064+0000 7f4cb659c700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4ca406c4e0 0x7f4ca406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:34.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.064+0000 7f4cb659c700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4ca406c4e0 0x7f4ca406e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800b420 tx=0x7f4ca8008540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:34.069 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.068+0000 7f4caf7fe700 1 -- 172.21.15.151:0/2395237289 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4ca005b1e0 con 0x7f4cb8105a40 2024-04-16T02:31:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:33 smithi151 ceph-mon[29003]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:34.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:33 smithi153 ceph-mon[36747]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:34.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.311+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4cb804ea90 con 0x7f4cb8105a40 2024-04-16T02:31:34.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.315+0000 7f4caf7fe700 1 -- 172.21.15.151:0/2395237289 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173046 (secure 0 0 0) 0x7f4ca005ad70 con 0x7f4cb8105a40 2024-04-16T02:31:34.318 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4ca406c4e0 msgr2=0x7f4ca406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4ca406c4e0 0x7f4ca406e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800b420 tx=0x7f4ca8008540 comp rx=0 tx=0).stop 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb8105a40 msgr2=0x7f4cb8198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb8105a40 0x7f4cb8198e90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0006010 tx=0x7f4ca000be20 comp rx=0 tx=0).stop 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 shutdown_connections 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4ca406c4e0 0x7f4ca406e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cb80ffa40 0x7f4cb8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 --2- 172.21.15.151:0/2395237289 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cb8105a40 0x7f4cb8198e90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 >> 172.21.15.151:0/2395237289 conn(0x7f4cb80fb550 msgr2=0x7f4cb80fbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 shutdown_connections 2024-04-16T02:31:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:34.317+0000 7f4cbc80c700 1 -- 172.21.15.151:0/2395237289 wait complete. 2024-04-16T02:31:34.934 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":19,"active_gid":14215,"active_name":"smithi151.uqqame","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2753390843},{"type":"v1","addr":"172.21.15.151:6801","nonce":2753390843}]},"active_addr":"172.21.15.151:6801/2753390843","active_change":"2024-04-16T02:27:42.822052+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14238,"name":"smithi153.esbtzf","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.151:8443/","prometheus":"http://172.21.15.151:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":1299604123}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":671603278}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":3091708247}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":661820069}]}]} 2024-04-16T02:31:34.938 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-16T02:31:34.938 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-16T02:31:34.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T02:31:35.150 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:34 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2395237289' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-16T02:31:35.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:34 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2395237289' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-16T02:31:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:35 smithi151 ceph-mon[29003]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:36.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:35 smithi153 ceph-mon[36747]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:36.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.879+0000 7fa0ee069700 1 -- 172.21.15.151:0/3364334962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8101770 msgr2=0x7fa0e81057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.879+0000 7fa0ee069700 1 --2- 172.21.15.151:0/3364334962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8101770 0x7fa0e81057c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009910 tx=0x7fa0dc009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.879+0000 7fa0ee069700 1 -- 172.21.15.151:0/3364334962 shutdown_connections 2024-04-16T02:31:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.879+0000 7fa0ee069700 1 --2- 172.21.15.151:0/3364334962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8101770 0x7fa0e81057c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.879+0000 7fa0ee069700 1 --2- 172.21.15.151:0/3364334962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8100dc0 0x7fa0e81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.879+0000 7fa0ee069700 1 -- 172.21.15.151:0/3364334962 >> 172.21.15.151:0/3364334962 conn(0x7fa0e80fc650 msgr2=0x7fa0e80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.879+0000 7fa0ee069700 1 -- 172.21.15.151:0/3364334962 shutdown_connections 2024-04-16T02:31:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.879+0000 7fa0ee069700 1 -- 172.21.15.151:0/3364334962 wait complete. 2024-04-16T02:31:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.879+0000 7fa0ee069700 1 Processor -- start 2024-04-16T02:31:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0ee069700 1 -- start start 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0ee069700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8100dc0 0x7fa0e819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0ee069700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8101770 0x7fa0e819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0ee069700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e819d0a0 con 0x7fa0e8100dc0 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0ee069700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e81965e0 con 0x7fa0e8101770 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0e77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8100dc0 0x7fa0e819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0e77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8100dc0 0x7fa0e819c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38924/0 (socket says 172.21.15.151:38924) 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0e77fe700 1 -- 172.21.15.151:0/1861355085 learned_addr learned my addr 172.21.15.151:0/1861355085 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0e6ffd700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8101770 0x7fa0e819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0e77fe700 1 -- 172.21.15.151:0/1861355085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8101770 msgr2=0x7fa0e819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0e77fe700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8101770 0x7fa0e819caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0e77fe700 1 -- 172.21.15.151:0/1861355085 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc0095f0 con 0x7fa0e8100dc0 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.880+0000 7fa0e77fe700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8100dc0 0x7fa0e819c560 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800c930 tx=0x7fa0d800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:36.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.881+0000 7fa0e4ff9700 1 -- 172.21.15.151:0/1861355085 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d8007ad0 con 0x7fa0e8100dc0 2024-04-16T02:31:36.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.881+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e81968c0 con 0x7fa0e8100dc0 2024-04-16T02:31:36.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.881+0000 7fa0e4ff9700 1 -- 172.21.15.151:0/1861355085 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa0d8007c30 con 0x7fa0e8100dc0 2024-04-16T02:31:36.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.881+0000 7fa0e4ff9700 1 -- 172.21.15.151:0/1861355085 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d8018630 con 0x7fa0e8100dc0 2024-04-16T02:31:36.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.881+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e8196e10 con 0x7fa0e8100dc0 2024-04-16T02:31:36.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.882+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e81091c0 con 0x7fa0e8100dc0 2024-04-16T02:31:36.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.882+0000 7fa0e4ff9700 1 -- 172.21.15.151:0/1861355085 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa0d800f450 con 0x7fa0e8100dc0 2024-04-16T02:31:36.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.882+0000 7fa0e4ff9700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa0d006c4e0 0x7fa0d006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:36.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.883+0000 7fa0e4ff9700 1 -- 172.21.15.151:0/1861355085 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa0d808b6d0 con 0x7fa0e8100dc0 2024-04-16T02:31:36.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.883+0000 7fa0e6ffd700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa0d006c4e0 0x7fa0d006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:36.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.883+0000 7fa0e6ffd700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa0d006c4e0 0x7fa0d006e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc006010 tx=0x7fa0dc005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:36.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:36.887+0000 7fa0e4ff9700 1 -- 172.21.15.151:0/1861355085 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa0d8059c80 con 0x7fa0e8100dc0 2024-04-16T02:31:37.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.081+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0e8197090 con 0x7fa0e8100dc0 2024-04-16T02:31:37.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.081+0000 7fa0e4ff9700 1 -- 172.21.15.151:0/1861355085 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11396 (secure 0 0 0) 0x7fa0d8059810 con 0x7fa0e8100dc0 2024-04-16T02:31:37.084 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:31:37.084 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":41,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","created":"2024-04-16T02:25:34.606206+0000","modified":"2024-04-16T02:31:18.735658+0000","last_up_change":"2024-04-16T02:31:17.730938+0000","last_in_change":"2024-04-16T02:30:58.536477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T02:30:03.137221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8da4d50b-626b-4cf3-9524-ea84e7733113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6803","nonce":1205378476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6805","nonce":1205378476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6809","nonce":1205378476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6807","nonce":1205378476}]},"public_addr":"172.21.15.151:6803/1205378476","cluster_addr":"172.21.15.151:6805/1205378476","heartbeat_back_addr":"172.21.15.151:6809/1205378476","heartbeat_front_addr":"172.21.15.151:6807/1205378476","state":["exists","up"]},{"osd":1,"uuid":"8d2980b5-1d30-445c-91ae-f08074b6d8f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6811","nonce":1648693895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6813","nonce":1648693895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6817","nonce":1648693895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6815","nonce":1648693895}]},"public_addr":"172.21.15.151:6811/1648693895","cluster_addr":"172.21.15.151:6813/1648693895","heartbeat_back_addr":"172.21.15.151:6817/1648693895","heartbeat_front_addr":"172.21.15.151:6815/1648693895","state":["exists","up"]},{"osd":2,"uuid":"64e4e7ce-5c3c-48c8-be3d-c9936b6866b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6819","nonce":2687133389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6821","nonce":2687133389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6825","nonce":2687133389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6823","nonce":2687133389}]},"public_addr":"172.21.15.151:6819/2687133389","cluster_addr":"172.21.15.151:6821/2687133389","heartbeat_back_addr":"172.21.15.151:6825/2687133389","heartbeat_front_addr":"172.21.15.151:6823/2687133389","state":["exists","up"]},{"osd":3,"uuid":"1194f132-bf2d-4e0c-ab93-f3344ff64d76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6801","nonce":2298379944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6803","nonce":2298379944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6807","nonce":2298379944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6805","nonce":2298379944}]},"public_addr":"172.21.15.153:6801/2298379944","cluster_addr":"172.21.15.153:6803/2298379944","heartbeat_back_addr":"172.21.15.153:6807/2298379944","heartbeat_front_addr":"172.21.15.153:6805/2298379944","state":["exists","up"]},{"osd":4,"uuid":"9d803d8b-eb35-4a0f-9d36-de11b743a123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6809","nonce":2873952441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6811","nonce":2873952441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6815","nonce":2873952441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6813","nonce":2873952441}]},"public_addr":"172.21.15.153:6809/2873952441","cluster_addr":"172.21.15.153:6811/2873952441","heartbeat_back_addr":"172.21.15.153:6815/2873952441","heartbeat_front_addr":"172.21.15.153:6813/2873952441","state":["exists","up"]},{"osd":5,"uuid":"7d0d2460-ef76-481b-ad51-afb80d05bc72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6817","nonce":1792791923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6819","nonce":1792791923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6823","nonce":1792791923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6821","nonce":1792791923}]},"public_addr":"172.21.15.153:6817/1792791923","cluster_addr":"172.21.15.153:6819/1792791923","heartbeat_back_addr":"172.21.15.153:6823/1792791923","heartbeat_front_addr":"172.21.15.153:6821/1792791923","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:02.729782+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:31.268279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:59.891666+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:24.930390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:50.555079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:31:15.969056+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6800/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6801/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6800/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3454798699":"2024-04-17T02:26:34.117371+0000","172.21.15.151:0/681541897":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6800/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/2293925528":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/4272586248":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/3129110405":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3798542210":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/969025109":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/1923729027":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/2684608614":"2024-04-17T02:26:34.117371+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T02:31:37.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.083+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa0d006c4e0 msgr2=0x7fa0d006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:37.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.083+0000 7fa0ee069700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa0d006c4e0 0x7fa0d006e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc006010 tx=0x7fa0dc005960 comp rx=0 tx=0).stop 2024-04-16T02:31:37.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.083+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8100dc0 msgr2=0x7fa0e819c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:37.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.084+0000 7fa0ee069700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8100dc0 0x7fa0e819c560 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800c930 tx=0x7fa0d800ccf0 comp rx=0 tx=0).stop 2024-04-16T02:31:37.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.084+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 shutdown_connections 2024-04-16T02:31:37.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.084+0000 7fa0ee069700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa0d006c4e0 0x7fa0d006e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:37.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.084+0000 7fa0ee069700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e8100dc0 0x7fa0e819c560 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:37.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.084+0000 7fa0ee069700 1 --2- 172.21.15.151:0/1861355085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8101770 0x7fa0e819caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:37.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.084+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 >> 172.21.15.151:0/1861355085 conn(0x7fa0e80fc650 msgr2=0x7fa0e80fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:37.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.084+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 shutdown_connections 2024-04-16T02:31:37.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:37.084+0000 7fa0ee069700 1 -- 172.21.15.151:0/1861355085 wait complete. 2024-04-16T02:31:37.705 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-16T02:31:37.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T02:31:37.916 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:37.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:37 smithi151 ceph-mon[29003]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:37.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:37 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1861355085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T02:31:38.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:37 smithi153 ceph-mon[36747]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:38.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:37 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1861355085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T02:31:39.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 -- 172.21.15.151:0/3266723140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 msgr2=0x7f4c800ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 --2- 172.21.15.151:0/3266723140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 0x7f4c800ffec0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4c70009910 tx=0x7f4c70009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 -- 172.21.15.151:0/3266723140 shutdown_connections 2024-04-16T02:31:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 --2- 172.21.15.151:0/3266723140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 0x7f4c800ffec0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 --2- 172.21.15.151:0/3266723140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c80105a40 0x7f4c80105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 -- 172.21.15.151:0/3266723140 >> 172.21.15.151:0/3266723140 conn(0x7f4c800fb550 msgr2=0x7f4c800fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 -- 172.21.15.151:0/3266723140 shutdown_connections 2024-04-16T02:31:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 -- 172.21.15.151:0/3266723140 wait complete. 2024-04-16T02:31:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 Processor -- start 2024-04-16T02:31:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.634+0000 7f4c85f38700 1 -- start start 2024-04-16T02:31:39.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c85f38700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 0x7f4c801975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c85f38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c80105a40 0x7f4c80197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c85f38700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c80198150 con 0x7f4c800ffa40 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c85f38700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c80198290 con 0x7f4c80105a40 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 0x7f4c801975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 0x7f4c801975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38930/0 (socket says 172.21.15.151:38930) 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7f7fe700 1 -- 172.21.15.151:0/2503144350 learned_addr learned my addr 172.21.15.151:0/2503144350 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7effd700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c80105a40 0x7f4c80197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7f7fe700 1 -- 172.21.15.151:0/2503144350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c80105a40 msgr2=0x7f4c80197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7f7fe700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c80105a40 0x7f4c80197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7f7fe700 1 -- 172.21.15.151:0/2503144350 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c700095f0 con 0x7f4c800ffa40 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7f7fe700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 0x7f4c801975f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400d900 tx=0x7f4c7400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7cff9700 1 -- 172.21.15.151:0/2503144350 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c74004a00 con 0x7f4c800ffa40 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c801a0660 con 0x7f4c800ffa40 2024-04-16T02:31:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.635+0000 7f4c7cff9700 1 -- 172.21.15.151:0/2503144350 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4c740054d0 con 0x7f4c800ffa40 2024-04-16T02:31:39.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.636+0000 7f4c7cff9700 1 -- 172.21.15.151:0/2503144350 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c74009d00 con 0x7f4c800ffa40 2024-04-16T02:31:39.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.636+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c801a0b20 con 0x7f4c800ffa40 2024-04-16T02:31:39.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.636+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c80107fd0 con 0x7f4c800ffa40 2024-04-16T02:31:39.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.637+0000 7f4c7cff9700 1 -- 172.21.15.151:0/2503144350 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4c74005020 con 0x7f4c800ffa40 2024-04-16T02:31:39.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.637+0000 7f4c7cff9700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4c6806c4e0 0x7f4c6806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:39.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.638+0000 7f4c7effd700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4c6806c4e0 0x7f4c6806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:39.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.638+0000 7f4c7cff9700 1 -- 172.21.15.151:0/2503144350 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4c7408b440 con 0x7f4c800ffa40 2024-04-16T02:31:39.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.638+0000 7f4c7effd700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4c6806c4e0 0x7f4c6806e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4c80069660 tx=0x7f4c7001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:39.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.642+0000 7f4c7cff9700 1 -- 172.21.15.151:0/2503144350 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4c74055f60 con 0x7f4c800ffa40 2024-04-16T02:31:39.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.833+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c8002cc70 con 0x7f4c800ffa40 2024-04-16T02:31:39.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.834+0000 7f4c7cff9700 1 -- 172.21.15.151:0/2503144350 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11396 (secure 0 0 0) 0x7f4c74009e60 con 0x7f4c800ffa40 2024-04-16T02:31:39.836 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:31:39.836 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":41,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","created":"2024-04-16T02:25:34.606206+0000","modified":"2024-04-16T02:31:18.735658+0000","last_up_change":"2024-04-16T02:31:17.730938+0000","last_in_change":"2024-04-16T02:30:58.536477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T02:30:03.137221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8da4d50b-626b-4cf3-9524-ea84e7733113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6803","nonce":1205378476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6805","nonce":1205378476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6809","nonce":1205378476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6807","nonce":1205378476}]},"public_addr":"172.21.15.151:6803/1205378476","cluster_addr":"172.21.15.151:6805/1205378476","heartbeat_back_addr":"172.21.15.151:6809/1205378476","heartbeat_front_addr":"172.21.15.151:6807/1205378476","state":["exists","up"]},{"osd":1,"uuid":"8d2980b5-1d30-445c-91ae-f08074b6d8f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6811","nonce":1648693895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6813","nonce":1648693895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6817","nonce":1648693895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6815","nonce":1648693895}]},"public_addr":"172.21.15.151:6811/1648693895","cluster_addr":"172.21.15.151:6813/1648693895","heartbeat_back_addr":"172.21.15.151:6817/1648693895","heartbeat_front_addr":"172.21.15.151:6815/1648693895","state":["exists","up"]},{"osd":2,"uuid":"64e4e7ce-5c3c-48c8-be3d-c9936b6866b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6819","nonce":2687133389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6821","nonce":2687133389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6825","nonce":2687133389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6823","nonce":2687133389}]},"public_addr":"172.21.15.151:6819/2687133389","cluster_addr":"172.21.15.151:6821/2687133389","heartbeat_back_addr":"172.21.15.151:6825/2687133389","heartbeat_front_addr":"172.21.15.151:6823/2687133389","state":["exists","up"]},{"osd":3,"uuid":"1194f132-bf2d-4e0c-ab93-f3344ff64d76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6801","nonce":2298379944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6803","nonce":2298379944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6807","nonce":2298379944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6805","nonce":2298379944}]},"public_addr":"172.21.15.153:6801/2298379944","cluster_addr":"172.21.15.153:6803/2298379944","heartbeat_back_addr":"172.21.15.153:6807/2298379944","heartbeat_front_addr":"172.21.15.153:6805/2298379944","state":["exists","up"]},{"osd":4,"uuid":"9d803d8b-eb35-4a0f-9d36-de11b743a123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6809","nonce":2873952441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6811","nonce":2873952441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6815","nonce":2873952441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6813","nonce":2873952441}]},"public_addr":"172.21.15.153:6809/2873952441","cluster_addr":"172.21.15.153:6811/2873952441","heartbeat_back_addr":"172.21.15.153:6815/2873952441","heartbeat_front_addr":"172.21.15.153:6813/2873952441","state":["exists","up"]},{"osd":5,"uuid":"7d0d2460-ef76-481b-ad51-afb80d05bc72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6817","nonce":1792791923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6819","nonce":1792791923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6823","nonce":1792791923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6821","nonce":1792791923}]},"public_addr":"172.21.15.153:6817/1792791923","cluster_addr":"172.21.15.153:6819/1792791923","heartbeat_back_addr":"172.21.15.153:6823/1792791923","heartbeat_front_addr":"172.21.15.153:6821/1792791923","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:02.729782+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:31.268279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:59.891666+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:24.930390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:50.555079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:31:15.969056+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6800/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6801/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6800/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3454798699":"2024-04-17T02:26:34.117371+0000","172.21.15.151:0/681541897":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6800/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/2293925528":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/4272586248":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/3129110405":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3798542210":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/969025109":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/1923729027":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/2684608614":"2024-04-17T02:26:34.117371+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T02:31:39.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4c6806c4e0 msgr2=0x7f4c6806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4c6806c4e0 0x7f4c6806e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4c80069660 tx=0x7f4c7001a040 comp rx=0 tx=0).stop 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 msgr2=0x7f4c801975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 0x7f4c801975f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400d900 tx=0x7f4c7400dc10 comp rx=0 tx=0).stop 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 shutdown_connections 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4c6806c4e0 0x7f4c6806e9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c800ffa40 0x7f4c801975f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 --2- 172.21.15.151:0/2503144350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c80105a40 0x7f4c80197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 >> 172.21.15.151:0/2503144350 conn(0x7f4c800fb550 msgr2=0x7f4c800fc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 shutdown_connections 2024-04-16T02:31:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:39.836+0000 7f4c85f38700 1 -- 172.21.15.151:0/2503144350 wait complete. 2024-04-16T02:31:40.189 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:39 smithi151 ceph-mon[29003]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:40.189 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:39 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2503144350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T02:31:40.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:39 smithi153 ceph-mon[36747]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:39 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2503144350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T02:31:40.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-16T02:31:40.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-16T02:31:40.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-16T02:31:40.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-16T02:31:40.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-16T02:31:40.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-16T02:31:40.828 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:40.839 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:41.585 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:41.585 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:41.586 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:41.586 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:41.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:41 smithi151 ceph-mon[29003]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:42.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:41 smithi153 ceph-mon[36747]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:43.076 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:31:43.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:42 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:31:44.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:43 smithi151 ceph-mon[29003]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:44.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:43 smithi153 ceph-mon[36747]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:45.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.111+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/3067117371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac0ffbd0 msgr2=0x7fc0ac100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:45.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.111+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/3067117371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac0ffbd0 0x7fc0ac100050 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc098009910 tx=0x7fc098009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:45.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.111+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/3067117371 shutdown_connections 2024-04-16T02:31:45.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.111+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/3067117371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac0ffbd0 0x7fc0ac100050 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:45.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.111+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/3067117371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac105a40 0x7fc0ac105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:45.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.111+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/3067117371 >> 172.21.15.151:0/3067117371 conn(0x7fc0ac0755a0 msgr2=0x7fc0ac0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:45.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.111+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/3067117371 shutdown_connections 2024-04-16T02:31:45.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.111+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/3067117371 wait complete. 2024-04-16T02:31:45.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b2dd2700 1 Processor -- start 2024-04-16T02:31:45.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b2dd2700 1 -- start start 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b2dd2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac0ffbd0 0x7fc0ac197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b2dd2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac105a40 0x7fc0ac197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b2dd2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac1981b0 con 0x7fc0ac105a40 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b2dd2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac198320 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b0b6e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac0ffbd0 0x7fc0ac197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0abfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac105a40 0x7fc0ac197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b0b6e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac0ffbd0 0x7fc0ac197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:43536/0 (socket says 172.21.15.151:43536) 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0abfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac105a40 0x7fc0ac197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38952/0 (socket says 172.21.15.151:38952) 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b0b6e700 1 -- 172.21.15.151:0/75593593 learned_addr learned my addr 172.21.15.151:0/75593593 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b0b6e700 1 -- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac105a40 msgr2=0x7fc0ac197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b0b6e700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac105a40 0x7fc0ac197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0b0b6e700 1 -- 172.21.15.151:0/75593593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980095f0 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.112+0000 7fc0abfff700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac105a40 0x7fc0ac197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:31:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.113+0000 7fc0b0b6e700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac0ffbd0 0x7fc0ac197590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000e920 tx=0x7fc0a000ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.113+0000 7fc0a9ffb700 1 -- 172.21.15.151:0/75593593 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0a000c9d0 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.113+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ac1a0600 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.113+0000 7fc0a9ffb700 1 -- 172.21.15.151:0/75593593 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc0a0010430 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.113+0000 7fc0a9ffb700 1 -- 172.21.15.151:0/75593593 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0a0018650 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.113+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ac1a0b20 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.114+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0ac10cec0 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.115+0000 7fc0a9ffb700 1 -- 172.21.15.151:0/75593593 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc0a00105a0 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.115+0000 7fc0a9ffb700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc09c06c4e0 0x7fc09c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.115+0000 7fc0a9ffb700 1 -- 172.21.15.151:0/75593593 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc0a0014070 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.115+0000 7fc0abfff700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc09c06c4e0 0x7fc09c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.115+0000 7fc0a9ffb700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] conn(0x7fc09c0721c0 0x7fc09c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.115+0000 7fc0a9ffb700 1 -- 172.21.15.151:0/75593593 --> [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc09c074c90 con 0x7fc09c0721c0 2024-04-16T02:31:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.115+0000 7fc0a9ffb700 1 -- 172.21.15.151:0/75593593 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0a008c500 con 0x7fc0ac0ffbd0 2024-04-16T02:31:45.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.115+0000 7fc0b136f700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] conn(0x7fc09c0721c0 0x7fc09c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:45.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.115+0000 7fc0abfff700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc09c06c4e0 0x7fc09c06e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac069040 tx=0x7fc098005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:45.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.116+0000 7fc0b136f700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] conn(0x7fc09c0721c0 0x7fc09c0745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:45.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.117+0000 7fc0a9ffb700 1 -- 172.21.15.151:0/75593593 <== osd.3 v2:172.21.15.153:6800/2298379944 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc09c074c90 con 0x7fc09c0721c0 2024-04-16T02:31:45.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.132+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 --> [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc0ac04ea90 con 0x7fc09c0721c0 2024-04-16T02:31:45.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.132+0000 7fc0a9ffb700 1 -- 172.21.15.151:0/75593593 <== osd.3 v2:172.21.15.153:6800/2298379944 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc0ac04ea90 con 0x7fc09c0721c0 2024-04-16T02:31:45.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] conn(0x7fc09c0721c0 msgr2=0x7fc09c0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] conn(0x7fc09c0721c0 0x7fc09c0745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc09c06c4e0 msgr2=0x7fc09c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc09c06c4e0 0x7fc09c06e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac069040 tx=0x7fc098005940 comp rx=0 tx=0).stop 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac0ffbd0 msgr2=0x7fc0ac197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac0ffbd0 0x7fc0ac197590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000e920 tx=0x7fc0a000ec30 comp rx=0 tx=0).stop 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 shutdown_connections 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fc09c06c4e0 0x7fc09c06e9a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:6800/2298379944,v1:172.21.15.153:6801/2298379944] conn(0x7fc09c0721c0 0x7fc09c0745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac0ffbd0 0x7fc0ac197590 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 --2- 172.21.15.151:0/75593593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0ac105a40 0x7fc0ac197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 >> 172.21.15.151:0/75593593 conn(0x7fc0ac0755a0 msgr2=0x7fc0ac0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 shutdown_connections 2024-04-16T02:31:45.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:45.133+0000 7fc0b2dd2700 1 -- 172.21.15.151:0/75593593 wait complete. 2024-04-16T02:31:45.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:45 smithi151 ceph-mon[29003]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:45 smithi153 ceph-mon[36747]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:46.612 INFO:teuthology.orchestra.run.smithi151.stdout:115964117009 2024-04-16T02:31:46.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-16T02:31:47.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.732+0000 7f8a548d8700 1 -- 172.21.15.151:0/2292050022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 msgr2=0x7f8a50068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:47.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.732+0000 7f8a548d8700 1 --2- 172.21.15.151:0/2292050022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 0x7f8a50068a50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8a40009910 tx=0x7f8a40009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.733+0000 7f8a548d8700 1 -- 172.21.15.151:0/2292050022 shutdown_connections 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.733+0000 7f8a548d8700 1 --2- 172.21.15.151:0/2292050022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a50069020 0x7f8a50105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.733+0000 7f8a548d8700 1 --2- 172.21.15.151:0/2292050022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 0x7f8a50068a50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.733+0000 7f8a548d8700 1 -- 172.21.15.151:0/2292050022 >> 172.21.15.151:0/2292050022 conn(0x7f8a500755a0 msgr2=0x7f8a500759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.733+0000 7f8a548d8700 1 -- 172.21.15.151:0/2292050022 shutdown_connections 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.733+0000 7f8a548d8700 1 -- 172.21.15.151:0/2292050022 wait complete. 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.733+0000 7f8a548d8700 1 Processor -- start 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.733+0000 7f8a548d8700 1 -- start start 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a548d8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 0x7f8a5019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a548d8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a50069020 0x7f8a5019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a548d8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5019d0e0 con 0x7f8a50068670 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a548d8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a50196590 con 0x7f8a50069020 2024-04-16T02:31:47.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a4e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 0x7f8a5019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a4e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 0x7f8a5019c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50140/0 (socket says 172.21.15.151:50140) 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a4e59c700 1 -- 172.21.15.151:0/1727413894 learned_addr learned my addr 172.21.15.151:0/1727413894 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a4dd9b700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a50069020 0x7f8a5019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a4e59c700 1 -- 172.21.15.151:0/1727413894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a50069020 msgr2=0x7f8a5019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a4e59c700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a50069020 0x7f8a5019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a4e59c700 1 -- 172.21.15.151:0/1727413894 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a400095f0 con 0x7f8a50068670 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a4e59c700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 0x7f8a5019c510 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8a40006010 tx=0x7f8a4000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a477fe700 1 -- 172.21.15.151:0/1727413894 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a4001d070 con 0x7f8a50068670 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a477fe700 1 -- 172.21.15.151:0/1727413894 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8a40004f50 con 0x7f8a50068670 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a477fe700 1 -- 172.21.15.151:0/1727413894 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a40021640 con 0x7f8a50068670 2024-04-16T02:31:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.734+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a50196810 con 0x7f8a50068670 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.735+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a50196d00 con 0x7f8a50068670 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.735+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a5010e080 con 0x7f8a50068670 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.736+0000 7f8a477fe700 1 -- 172.21.15.151:0/1727413894 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8a4000f460 con 0x7f8a50068670 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.737+0000 7f8a477fe700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8a3c06c4e0 0x7f8a3c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.737+0000 7f8a477fe700 1 -- 172.21.15.151:0/1727413894 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8a4008c740 con 0x7f8a50068670 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.737+0000 7f8a4dd9b700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8a3c06c4e0 0x7f8a3c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.737+0000 7f8a477fe700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] conn(0x7f8a3c0721c0 0x7f8a3c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.737+0000 7f8a477fe700 1 -- 172.21.15.151:0/1727413894 --> [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a3c074c90 con 0x7f8a3c0721c0 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.737+0000 7f8a477fe700 1 -- 172.21.15.151:0/1727413894 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a4008ee10 con 0x7f8a50068670 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.737+0000 7f8a4ed9d700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] conn(0x7f8a3c0721c0 0x7f8a3c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:47.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.737+0000 7f8a4dd9b700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8a3c06c4e0 0x7f8a3c06e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8a50107750 tx=0x7f8a380094f0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:47.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.738+0000 7f8a4ed9d700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] conn(0x7f8a3c0721c0 0x7f8a3c0745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:47.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.738+0000 7f8a477fe700 1 -- 172.21.15.151:0/1727413894 <== osd.0 v2:172.21.15.151:6802/1205378476 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8a3c074c90 con 0x7f8a3c0721c0 2024-04-16T02:31:47.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.754+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 --> [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a5004ea90 con 0x7f8a3c0721c0 2024-04-16T02:31:47.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.754+0000 7f8a477fe700 1 -- 172.21.15.151:0/1727413894 <== osd.0 v2:172.21.15.151:6802/1205378476 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8a5004ea90 con 0x7f8a3c0721c0 2024-04-16T02:31:47.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] conn(0x7f8a3c0721c0 msgr2=0x7f8a3c0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] conn(0x7f8a3c0721c0 0x7f8a3c0745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8a3c06c4e0 msgr2=0x7f8a3c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8a3c06c4e0 0x7f8a3c06e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8a50107750 tx=0x7f8a380094f0 comp rx=0 tx=0).stop 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 msgr2=0x7f8a5019c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 0x7f8a5019c510 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8a40006010 tx=0x7f8a4000be20 comp rx=0 tx=0).stop 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 shutdown_connections 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6802/1205378476,v1:172.21.15.151:6803/1205378476] conn(0x7f8a3c0721c0 0x7f8a3c0745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8a3c06c4e0 0x7f8a3c06e9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a50068670 0x7f8a5019c510 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 --2- 172.21.15.151:0/1727413894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a50069020 0x7f8a5019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:47.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.755+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 >> 172.21.15.151:0/1727413894 conn(0x7f8a500755a0 msgr2=0x7f8a500fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:47.758 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.756+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 shutdown_connections 2024-04-16T02:31:47.758 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:47.756+0000 7f8a548d8700 1 -- 172.21.15.151:0/1727413894 wait complete. 2024-04-16T02:31:48.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:47 smithi151 ceph-mon[29003]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:48.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:47 smithi153 ceph-mon[36747]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.078+0000 7f85a9b2f700 1 -- 172.21.15.151:0/671380592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a40ffbd0 msgr2=0x7f85a4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.078+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/671380592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a40ffbd0 0x7f85a4100050 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8598009910 tx=0x7f8598009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.078+0000 7f85a9b2f700 1 -- 172.21.15.151:0/671380592 shutdown_connections 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.078+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/671380592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a40ffbd0 0x7f85a4100050 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.078+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/671380592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85a4105a40 0x7f85a4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.078+0000 7f85a9b2f700 1 -- 172.21.15.151:0/671380592 >> 172.21.15.151:0/671380592 conn(0x7f85a40755a0 msgr2=0x7f85a40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.079+0000 7f85a9b2f700 1 -- 172.21.15.151:0/671380592 shutdown_connections 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.079+0000 7f85a9b2f700 1 -- 172.21.15.151:0/671380592 wait complete. 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.079+0000 7f85a9b2f700 1 Processor -- start 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.079+0000 7f85a9b2f700 1 -- start start 2024-04-16T02:31:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a9b2f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85a40ffbd0 0x7f85a419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a9b2f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a4105a40 0x7f85a419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a9b2f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a419d0d0 con 0x7f85a4105a40 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a9b2f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a4196580 con 0x7f85a40ffbd0 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a4105a40 0x7f85a419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a4105a40 0x7f85a419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50150/0 (socket says 172.21.15.151:50150) 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a2ffd700 1 -- 172.21.15.151:0/3818326488 learned_addr learned my addr 172.21.15.151:0/3818326488 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a37fe700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85a40ffbd0 0x7f85a419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a2ffd700 1 -- 172.21.15.151:0/3818326488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85a40ffbd0 msgr2=0x7f85a419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a2ffd700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85a40ffbd0 0x7f85a419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a2ffd700 1 -- 172.21.15.151:0/3818326488 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980095f0 con 0x7f85a4105a40 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a37fe700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85a40ffbd0 0x7f85a419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a2ffd700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a4105a40 0x7f85a419ca40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f859800fa00 tx=0x7f859800fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a0ff9700 1 -- 172.21.15.151:0/3818326488 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f859801d070 con 0x7f85a4105a40 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a4196800 con 0x7f85a4105a40 2024-04-16T02:31:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.080+0000 7f85a0ff9700 1 -- 172.21.15.151:0/3818326488 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8598005640 con 0x7f85a4105a40 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.081+0000 7f85a0ff9700 1 -- 172.21.15.151:0/3818326488 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85980226a0 con 0x7f85a4105a40 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.081+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a4196cf0 con 0x7f85a4105a40 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.081+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f85a410cf40 con 0x7f85a4105a40 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.082+0000 7f85a0ff9700 1 -- 172.21.15.151:0/3818326488 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8598003680 con 0x7f85a4105a40 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.082+0000 7f85a0ff9700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f858c06c4e0 0x7f858c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.083+0000 7f85a0ff9700 1 -- 172.21.15.151:0/3818326488 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f859808c930 con 0x7f85a4105a40 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.083+0000 7f85a37fe700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f858c06c4e0 0x7f858c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.083+0000 7f85a0ff9700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] conn(0x7f858c0721c0 0x7f858c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.083+0000 7f85a0ff9700 1 -- 172.21.15.151:0/3818326488 --> [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f858c074c90 con 0x7f858c0721c0 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.083+0000 7f85a0ff9700 1 -- 172.21.15.151:0/3818326488 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85980912a0 con 0x7f85a4105a40 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.083+0000 7f85a3fff700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] conn(0x7f858c0721c0 0x7f858c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.083+0000 7f85a37fe700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f858c06c4e0 0x7f858c06e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8594009be0 tx=0x7f8594009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.083+0000 7f85a3fff700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] conn(0x7f858c0721c0 0x7f858c0745e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.084+0000 7f85a0ff9700 1 -- 172.21.15.151:0/3818326488 <== osd.1 v2:172.21.15.151:6810/1648693895 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f858c074c90 con 0x7f858c0721c0 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.100+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 --> [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f85a404ea90 con 0x7f858c0721c0 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.100+0000 7f85a0ff9700 1 -- 172.21.15.151:0/3818326488 <== osd.1 v2:172.21.15.151:6810/1648693895 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f85a404ea90 con 0x7f858c0721c0 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.100+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] conn(0x7f858c0721c0 msgr2=0x7f858c0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.100+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] conn(0x7f858c0721c0 0x7f858c0745e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f858c06c4e0 msgr2=0x7f858c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f858c06c4e0 0x7f858c06e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8594009be0 tx=0x7f8594009260 comp rx=0 tx=0).stop 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a4105a40 msgr2=0x7f85a419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a4105a40 0x7f85a419ca40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f859800fa00 tx=0x7f859800fae0 comp rx=0 tx=0).stop 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 shutdown_connections 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f858c06c4e0 0x7f858c06e9a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85a40ffbd0 0x7f85a419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:6810/1648693895,v1:172.21.15.151:6811/1648693895] conn(0x7f858c0721c0 0x7f858c0745e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 --2- 172.21.15.151:0/3818326488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85a4105a40 0x7f85a419ca40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 >> 172.21.15.151:0/3818326488 conn(0x7f85a40755a0 msgr2=0x7f85a4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 shutdown_connections 2024-04-16T02:31:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:48.101+0000 7f85a9b2f700 1 -- 172.21.15.151:0/3818326488 wait complete. 2024-04-16T02:31:49.648 INFO:teuthology.orchestra.run.smithi151.stdout:60129542174 2024-04-16T02:31:49.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-16T02:31:50.179 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:49 smithi151 ceph-mon[29003]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:50.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:49 smithi153 ceph-mon[36747]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:50.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.282+0000 7f2a1cc60700 1 -- 172.21.15.151:0/2412472227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18100dc0 msgr2=0x7f2a181011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.282+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/2412472227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18100dc0 0x7f2a181011a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2a08009910 tx=0x7f2a08009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:50.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.282+0000 7f2a1cc60700 1 -- 172.21.15.151:0/2412472227 shutdown_connections 2024-04-16T02:31:50.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.282+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/2412472227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a18101770 0x7f2a181057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.282+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/2412472227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18100dc0 0x7f2a181011a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.282+0000 7f2a1cc60700 1 -- 172.21.15.151:0/2412472227 >> 172.21.15.151:0/2412472227 conn(0x7f2a180fc650 msgr2=0x7f2a180fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:50.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.282+0000 7f2a1cc60700 1 -- 172.21.15.151:0/2412472227 shutdown_connections 2024-04-16T02:31:50.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.282+0000 7f2a1cc60700 1 -- 172.21.15.151:0/2412472227 wait complete. 2024-04-16T02:31:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a1cc60700 1 Processor -- start 2024-04-16T02:31:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a1cc60700 1 -- start start 2024-04-16T02:31:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a1cc60700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a18100dc0 0x7f2a1819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a1cc60700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18101770 0x7f2a1819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a1cc60700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1819d110 con 0x7f2a18101770 2024-04-16T02:31:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a1cc60700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a181965c0 con 0x7f2a18100dc0 2024-04-16T02:31:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a15d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18101770 0x7f2a1819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a15d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18101770 0x7f2a1819ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50178/0 (socket says 172.21.15.151:50178) 2024-04-16T02:31:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a15d9b700 1 -- 172.21.15.151:0/1841883191 learned_addr learned my addr 172.21.15.151:0/1841883191 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a1659c700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a18100dc0 0x7f2a1819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a15d9b700 1 -- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a18100dc0 msgr2=0x7f2a1819c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a15d9b700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a18100dc0 0x7f2a1819c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a15d9b700 1 -- 172.21.15.151:0/1841883191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a080095f0 con 0x7f2a18101770 2024-04-16T02:31:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.283+0000 7f2a15d9b700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18101770 0x7f2a1819ca80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00d8d0 tx=0x7f2a0c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.284+0000 7f2a077fe700 1 -- 172.21.15.151:0/1841883191 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a0c0041f0 con 0x7f2a18101770 2024-04-16T02:31:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.284+0000 7f2a077fe700 1 -- 172.21.15.151:0/1841883191 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2a0c007620 con 0x7f2a18101770 2024-04-16T02:31:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.284+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a181968a0 con 0x7f2a18101770 2024-04-16T02:31:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.284+0000 7f2a077fe700 1 -- 172.21.15.151:0/1841883191 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a0c0056b0 con 0x7f2a18101770 2024-04-16T02:31:50.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.284+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a18196df0 con 0x7f2a18101770 2024-04-16T02:31:50.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.284+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a1810e080 con 0x7f2a18101770 2024-04-16T02:31:50.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.285+0000 7f2a077fe700 1 -- 172.21.15.151:0/1841883191 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2a0c004750 con 0x7f2a18101770 2024-04-16T02:31:50.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.285+0000 7f2a077fe700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2a0006c4e0 0x7f2a0006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:50.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.286+0000 7f2a077fe700 1 -- 172.21.15.151:0/1841883191 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2a0c08b9f0 con 0x7f2a18101770 2024-04-16T02:31:50.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.286+0000 7f2a1659c700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2a0006c4e0 0x7f2a0006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:50.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.286+0000 7f2a077fe700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] conn(0x7f2a000721c0 0x7f2a000745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:50.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.286+0000 7f2a077fe700 1 -- 172.21.15.151:0/1841883191 --> [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a00074c90 con 0x7f2a000721c0 2024-04-16T02:31:50.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.286+0000 7f2a077fe700 1 -- 172.21.15.151:0/1841883191 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a0c08be50 con 0x7f2a18101770 2024-04-16T02:31:50.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.286+0000 7f2a16d9d700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] conn(0x7f2a000721c0 0x7f2a000745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:50.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.286+0000 7f2a1659c700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2a0006c4e0 0x7f2a0006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2a08006010 tx=0x7f2a0801a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:50.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.286+0000 7f2a16d9d700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] conn(0x7f2a000721c0 0x7f2a000745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:50.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.287+0000 7f2a077fe700 1 -- 172.21.15.151:0/1841883191 <== osd.5 v2:172.21.15.153:6816/1792791923 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2a00074c90 con 0x7f2a000721c0 2024-04-16T02:31:50.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.303+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 --> [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2a1804ea90 con 0x7f2a000721c0 2024-04-16T02:31:50.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.303+0000 7f2a077fe700 1 -- 172.21.15.151:0/1841883191 <== osd.5 v2:172.21.15.153:6816/1792791923 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2a1804ea90 con 0x7f2a000721c0 2024-04-16T02:31:50.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] conn(0x7f2a000721c0 msgr2=0x7f2a000745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] conn(0x7f2a000721c0 0x7f2a000745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2a0006c4e0 msgr2=0x7f2a0006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2a0006c4e0 0x7f2a0006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2a08006010 tx=0x7f2a0801a040 comp rx=0 tx=0).stop 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18101770 msgr2=0x7f2a1819ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18101770 0x7f2a1819ca80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00d8d0 tx=0x7f2a0c00dbe0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 shutdown_connections 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:6816/1792791923,v1:172.21.15.153:6817/1792791923] conn(0x7f2a000721c0 0x7f2a000745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2a0006c4e0 0x7f2a0006e9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a18100dc0 0x7f2a1819c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 --2- 172.21.15.151:0/1841883191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a18101770 0x7f2a1819ca80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 >> 172.21.15.151:0/1841883191 conn(0x7f2a180fc650 msgr2=0x7f2a180fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 shutdown_connections 2024-04-16T02:31:50.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.304+0000 7f2a1cc60700 1 -- 172.21.15.151:0/1841883191 wait complete. 2024-04-16T02:31:50.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.523+0000 7f3133c97700 1 -- 172.21.15.151:0/2403203023 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 msgr2=0x7f312c0ff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.523+0000 7f3133c97700 1 --2- 172.21.15.151:0/2403203023 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 0x7f312c0ff8e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3128009910 tx=0x7f3128009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.523+0000 7f3133c97700 1 -- 172.21.15.151:0/2403203023 shutdown_connections 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.523+0000 7f3133c97700 1 --2- 172.21.15.151:0/2403203023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f312c0ffeb0 0x7f312c105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.523+0000 7f3133c97700 1 --2- 172.21.15.151:0/2403203023 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 0x7f312c0ff8e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.523+0000 7f3133c97700 1 -- 172.21.15.151:0/2403203023 >> 172.21.15.151:0/2403203023 conn(0x7f312c074d80 msgr2=0x7f312c075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.524+0000 7f3133c97700 1 -- 172.21.15.151:0/2403203023 shutdown_connections 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.524+0000 7f3133c97700 1 -- 172.21.15.151:0/2403203023 wait complete. 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.524+0000 7f3133c97700 1 Processor -- start 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.524+0000 7f3133c97700 1 -- start start 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.524+0000 7f3133c97700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 0x7f312c19c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.524+0000 7f3133c97700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f312c0ffeb0 0x7f312c19cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.524+0000 7f3133c97700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c19d1b0 con 0x7f312c0ff500 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.524+0000 7f3133c97700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c196660 con 0x7f312c0ffeb0 2024-04-16T02:31:50.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3131a33700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 0x7f312c19c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3131a33700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 0x7f312c19c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50184/0 (socket says 172.21.15.151:50184) 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3131a33700 1 -- 172.21.15.151:0/826860869 learned_addr learned my addr 172.21.15.151:0/826860869 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3131232700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f312c0ffeb0 0x7f312c19cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3131a33700 1 -- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f312c0ffeb0 msgr2=0x7f312c19cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3131a33700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f312c0ffeb0 0x7f312c19cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3131a33700 1 -- 172.21.15.151:0/826860869 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31280095f0 con 0x7f312c0ff500 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3131a33700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 0x7f312c19c5e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3128006010 tx=0x7f312800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3122ffd700 1 -- 172.21.15.151:0/826860869 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f312801d070 con 0x7f312c0ff500 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c1968e0 con 0x7f312c0ff500 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3122ffd700 1 -- 172.21.15.151:0/826860869 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3128004fa0 con 0x7f312c0ff500 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3122ffd700 1 -- 172.21.15.151:0/826860869 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f312800f690 con 0x7f312c0ff500 2024-04-16T02:31:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.525+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c196dd0 con 0x7f312c0ff500 2024-04-16T02:31:50.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.526+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f312c10e160 con 0x7f312c0ff500 2024-04-16T02:31:50.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.527+0000 7f3122ffd700 1 -- 172.21.15.151:0/826860869 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3128022470 con 0x7f312c0ff500 2024-04-16T02:31:50.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.527+0000 7f3122ffd700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f311806c4e0 0x7f311806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:50.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.527+0000 7f3122ffd700 1 -- 172.21.15.151:0/826860869 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f312808d6e0 con 0x7f312c0ff500 2024-04-16T02:31:50.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.527+0000 7f3131232700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f311806c4e0 0x7f311806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:50.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.527+0000 7f3122ffd700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] conn(0x7f31180721c0 0x7f31180745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:50.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.527+0000 7f3122ffd700 1 -- 172.21.15.151:0/826860869 --> [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3118074c90 con 0x7f31180721c0 2024-04-16T02:31:50.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.527+0000 7f3122ffd700 1 -- 172.21.15.151:0/826860869 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f312808db40 con 0x7f312c0ff500 2024-04-16T02:31:50.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.528+0000 7f3131232700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f311806c4e0 0x7f311806e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f312c1077b0 tx=0x7f311c009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:50.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.528+0000 7f3132234700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] conn(0x7f31180721c0 0x7f31180745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:50.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.528+0000 7f3132234700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] conn(0x7f31180721c0 0x7f31180745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:50.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.529+0000 7f3122ffd700 1 -- 172.21.15.151:0/826860869 <== osd.4 v2:172.21.15.153:6808/2873952441 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3118074c90 con 0x7f31180721c0 2024-04-16T02:31:50.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.545+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 --> [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f312c04ea90 con 0x7f31180721c0 2024-04-16T02:31:50.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.546+0000 7f3122ffd700 1 -- 172.21.15.151:0/826860869 <== osd.4 v2:172.21.15.153:6808/2873952441 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f312c04ea90 con 0x7f31180721c0 2024-04-16T02:31:50.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.546+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] conn(0x7f31180721c0 msgr2=0x7f31180745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.546+0000 7f3133c97700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] conn(0x7f31180721c0 0x7f31180745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.546+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f311806c4e0 msgr2=0x7f311806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.546+0000 7f3133c97700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f311806c4e0 0x7f311806e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f312c1077b0 tx=0x7f311c009190 comp rx=0 tx=0).stop 2024-04-16T02:31:50.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.546+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 msgr2=0x7f312c19c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:50.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.546+0000 7f3133c97700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 0x7f312c19c5e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3128006010 tx=0x7f312800bd70 comp rx=0 tx=0).stop 2024-04-16T02:31:50.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.546+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 shutdown_connections 2024-04-16T02:31:50.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.546+0000 7f3133c97700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:6808/2873952441,v1:172.21.15.153:6809/2873952441] conn(0x7f31180721c0 0x7f31180745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.547+0000 7f3133c97700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f311806c4e0 0x7f311806e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.547+0000 7f3133c97700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f312c0ff500 0x7f312c19c5e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.547+0000 7f3133c97700 1 --2- 172.21.15.151:0/826860869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f312c0ffeb0 0x7f312c19cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:50.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.547+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 >> 172.21.15.151:0/826860869 conn(0x7f312c074d80 msgr2=0x7f312c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:50.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.547+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 shutdown_connections 2024-04-16T02:31:50.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:50.547+0000 7f3133c97700 1 -- 172.21.15.151:0/826860869 wait complete. 2024-04-16T02:31:51.130 INFO:teuthology.orchestra.run.smithi151.stdout:171798691850 2024-04-16T02:31:51.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-16T02:31:52.065 INFO:teuthology.orchestra.run.smithi151.stdout:38654705700 2024-04-16T02:31:52.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-16T02:31:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:51 smithi151 ceph-mon[29003]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:52.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:51 smithi153 ceph-mon[36747]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:52.304 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:52.774 INFO:teuthology.orchestra.run.smithi151.stdout:150323855374 2024-04-16T02:31:52.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-16T02:31:52.781 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:53.808 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:53.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:53 smithi151 ceph-mon[29003]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.253+0000 7f2908540700 1 -- 172.21.15.151:0/2400787153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 msgr2=0x7f2900105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.253+0000 7f2908540700 1 --2- 172.21.15.151:0/2400787153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 0x7f2900105e20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b0a0 tx=0x7f28f000b3b0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.253+0000 7f2908540700 1 -- 172.21.15.151:0/2400787153 shutdown_connections 2024-04-16T02:31:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.253+0000 7f2908540700 1 --2- 172.21.15.151:0/2400787153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29000ffbd0 0x7f2900100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.253+0000 7f2908540700 1 --2- 172.21.15.151:0/2400787153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 0x7f2900105e20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.253+0000 7f2908540700 1 -- 172.21.15.151:0/2400787153 >> 172.21.15.151:0/2400787153 conn(0x7f29000755a0 msgr2=0x7f29000759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.253+0000 7f2908540700 1 -- 172.21.15.151:0/2400787153 shutdown_connections 2024-04-16T02:31:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.253+0000 7f2908540700 1 -- 172.21.15.151:0/2400787153 wait complete. 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.254+0000 7f2908540700 1 Processor -- start 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.254+0000 7f2908540700 1 -- start start 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.254+0000 7f2908540700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29000ffbd0 0x7f2900198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.254+0000 7f2908540700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 0x7f2900198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.254+0000 7f2908540700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2900199530 con 0x7f2900105a40 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.254+0000 7f2908540700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290019d340 con 0x7f29000ffbd0 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.254+0000 7f2905adb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 0x7f2900198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f2905adb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 0x7f2900198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50198/0 (socket says 172.21.15.151:50198) 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f2905adb700 1 -- 172.21.15.151:0/2045699350 learned_addr learned my addr 172.21.15.151:0/2045699350 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f29062dc700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29000ffbd0 0x7f2900198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f2905adb700 1 -- 172.21.15.151:0/2045699350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29000ffbd0 msgr2=0x7f2900198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f2905adb700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29000ffbd0 0x7f2900198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f2905adb700 1 -- 172.21.15.151:0/2045699350 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f0009d00 con 0x7f2900105a40 2024-04-16T02:31:54.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f2905adb700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 0x7f2900198ea0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f28f400c8a0 tx=0x7f28f400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:54.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f28ff7fe700 1 -- 172.21.15.151:0/2045699350 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28f400cea0 con 0x7f2900105a40 2024-04-16T02:31:54.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f28ff7fe700 1 -- 172.21.15.151:0/2045699350 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f28f4004830 con 0x7f2900105a40 2024-04-16T02:31:54.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290019d620 con 0x7f2900105a40 2024-04-16T02:31:54.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.255+0000 7f28ff7fe700 1 -- 172.21.15.151:0/2045699350 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28f40055b0 con 0x7f2900105a40 2024-04-16T02:31:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.256+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290019db70 con 0x7f2900105a40 2024-04-16T02:31:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.257+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f290010cec0 con 0x7f2900105a40 2024-04-16T02:31:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.257+0000 7f28ff7fe700 1 -- 172.21.15.151:0/2045699350 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f28f400f470 con 0x7f2900105a40 2024-04-16T02:31:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.257+0000 7f28ff7fe700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f28e806c4e0 0x7f28e806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.258+0000 7f28ff7fe700 1 -- 172.21.15.151:0/2045699350 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f28f408b220 con 0x7f2900105a40 2024-04-16T02:31:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.258+0000 7f29062dc700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f28e806c4e0 0x7f28e806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.258+0000 7f28ff7fe700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] conn(0x7f28e80721c0 0x7f28e80745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.258+0000 7f28ff7fe700 1 -- 172.21.15.151:0/2045699350 --> [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28e8074c90 con 0x7f28e80721c0 2024-04-16T02:31:54.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.258+0000 7f28ff7fe700 1 -- 172.21.15.151:0/2045699350 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28f408d8f0 con 0x7f2900105a40 2024-04-16T02:31:54.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.258+0000 7f2906add700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] conn(0x7f28e80721c0 0x7f28e80745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:54.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.258+0000 7f29062dc700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f28e806c4e0 0x7f28e806e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b830 tx=0x7f28f00097c0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:54.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.258+0000 7f2906add700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] conn(0x7f28e80721c0 0x7f28e80745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:54.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.259+0000 7f28ff7fe700 1 -- 172.21.15.151:0/2045699350 <== osd.2 v2:172.21.15.151:6818/2687133389 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f28e8074c90 con 0x7f28e80721c0 2024-04-16T02:31:54.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:53 smithi153 ceph-mon[36747]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:54.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.276+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 --> [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f290004ea90 con 0x7f28e80721c0 2024-04-16T02:31:54.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.276+0000 7f28ff7fe700 1 -- 172.21.15.151:0/2045699350 <== osd.2 v2:172.21.15.151:6818/2687133389 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f290004ea90 con 0x7f28e80721c0 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] conn(0x7f28e80721c0 msgr2=0x7f28e80745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] conn(0x7f28e80721c0 0x7f28e80745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f28e806c4e0 msgr2=0x7f28e806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f28e806c4e0 0x7f28e806e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b830 tx=0x7f28f00097c0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 msgr2=0x7f2900198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 0x7f2900198ea0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f28f400c8a0 tx=0x7f28f400cc60 comp rx=0 tx=0).stop 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 shutdown_connections 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f28e806c4e0 0x7f28e806e9a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:6818/2687133389,v1:172.21.15.151:6819/2687133389] conn(0x7f28e80721c0 0x7f28e80745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29000ffbd0 0x7f2900198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 --2- 172.21.15.151:0/2045699350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2900105a40 0x7f2900198ea0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:54.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 >> 172.21.15.151:0/2045699350 conn(0x7f29000755a0 msgr2=0x7f29000fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:54.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 shutdown_connections 2024-04-16T02:31:54.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:54.277+0000 7f2908540700 1 -- 172.21.15.151:0/2045699350 wait complete. 2024-04-16T02:31:54.959 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:54.959 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:55.757 INFO:teuthology.orchestra.run.smithi151.stdout:81604378650 2024-04-16T02:31:55.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-16T02:31:55.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:55 smithi153 ceph-mon[36747]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:55 smithi151 ceph-mon[29003]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:58.057 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:57 smithi151 ceph-mon[29003]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:58.057 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:31:58.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:57 smithi153 ceph-mon[36747]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:31:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:31:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.226+0000 7fa6a92c8700 1 -- 172.21.15.151:0/1394803248 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a4101d80 msgr2=0x7fa6a4102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.226+0000 7fa6a92c8700 1 --2- 172.21.15.151:0/1394803248 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a4101d80 0x7fa6a4102160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa69400b0a0 tx=0x7fa69400b3b0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.226+0000 7fa6a92c8700 1 -- 172.21.15.151:0/1394803248 shutdown_connections 2024-04-16T02:31:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.226+0000 7fa6a92c8700 1 --2- 172.21.15.151:0/1394803248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a41026a0 0x7fa6a410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.226+0000 7fa6a92c8700 1 --2- 172.21.15.151:0/1394803248 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a4101d80 0x7fa6a4102160 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.226+0000 7fa6a92c8700 1 -- 172.21.15.151:0/1394803248 >> 172.21.15.151:0/1394803248 conn(0x7fa6a40fb550 msgr2=0x7fa6a40fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.226+0000 7fa6a92c8700 1 -- 172.21.15.151:0/1394803248 shutdown_connections 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.226+0000 7fa6a92c8700 1 -- 172.21.15.151:0/1394803248 wait complete. 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a92c8700 1 Processor -- start 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a92c8700 1 -- start start 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a92c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4101d80 0x7fa6a419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a92c8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a41026a0 0x7fa6a419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a92c8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a419d100 con 0x7fa6a41026a0 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a92c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a41965b0 con 0x7fa6a4101d80 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a41026a0 0x7fa6a419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a41026a0 0x7fa6a419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48026/0 (socket says 172.21.15.151:48026) 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a27fc700 1 -- 172.21.15.151:0/2346143479 learned_addr learned my addr 172.21.15.151:0/2346143479 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a2ffd700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4101d80 0x7fa6a419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a27fc700 1 -- 172.21.15.151:0/2346143479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4101d80 msgr2=0x7fa6a419c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a27fc700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4101d80 0x7fa6a419c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.227+0000 7fa6a27fc700 1 -- 172.21.15.151:0/2346143479 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa694009d00 con 0x7fa6a41026a0 2024-04-16T02:31:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.228+0000 7fa6a27fc700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a41026a0 0x7fa6a419ca70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00d8d0 tx=0x7fa68c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.228+0000 7fa69bfff700 1 -- 172.21.15.151:0/2346143479 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa68c004a00 con 0x7fa6a41026a0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.228+0000 7fa69bfff700 1 -- 172.21.15.151:0/2346143479 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa68c0054d0 con 0x7fa6a41026a0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.228+0000 7fa69bfff700 1 -- 172.21.15.151:0/2346143479 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa68c00fc20 con 0x7fa6a41026a0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.228+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a4196890 con 0x7fa6a41026a0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.228+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a4196de0 con 0x7fa6a41026a0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.229+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a4108290 con 0x7fa6a41026a0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.229+0000 7fa69bfff700 1 -- 172.21.15.151:0/2346143479 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa68c00fd80 con 0x7fa6a41026a0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.230+0000 7fa69bfff700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa69006c4e0 0x7fa69006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.230+0000 7fa69bfff700 1 -- 172.21.15.151:0/2346143479 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa68c08acf0 con 0x7fa6a41026a0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.230+0000 7fa6a2ffd700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa69006c4e0 0x7fa69006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.230+0000 7fa6a2ffd700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa69006c4e0 0x7fa69006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa694000f80 tx=0x7fa6940060c0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.235+0000 7fa69bfff700 1 -- 172.21.15.151:0/2346143479 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa68c059320 con 0x7fa6a41026a0 2024-04-16T02:31:59.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.432+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa6a41970d0 con 0x7fa6a41026a0 2024-04-16T02:31:59.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.432+0000 7fa69bfff700 1 -- 172.21.15.151:0/2346143479 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa68c017090 con 0x7fa6a41026a0 2024-04-16T02:31:59.434 INFO:teuthology.orchestra.run.smithi151.stdout:150323855376 2024-04-16T02:31:59.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.434+0000 7f1b84d60700 1 -- 172.21.15.151:0/1498963827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 msgr2=0x7f1b80105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:59.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.434+0000 7f1b84d60700 1 --2- 172.21.15.151:0/1498963827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 0x7f1b80105e20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009910 tx=0x7f1b70009c20 comp rx=0 tx=0).stop 2024-04-16T02:31:59.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.434+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa69006c4e0 msgr2=0x7fa69006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:59.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.434+0000 7fa6a92c8700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa69006c4e0 0x7fa69006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa694000f80 tx=0x7fa6940060c0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.434+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a41026a0 msgr2=0x7fa6a419ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:59.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.434+0000 7fa6a92c8700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a41026a0 0x7fa6a419ca70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00d8d0 tx=0x7fa68c00dbe0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7f1b84d60700 1 -- 172.21.15.151:0/1498963827 shutdown_connections 2024-04-16T02:31:59.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7f1b84d60700 1 --2- 172.21.15.151:0/1498963827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b800ffa40 0x7f1b800ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7f1b84d60700 1 --2- 172.21.15.151:0/1498963827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 0x7f1b80105e20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7f1b84d60700 1 -- 172.21.15.151:0/1498963827 >> 172.21.15.151:0/1498963827 conn(0x7f1b800fb570 msgr2=0x7f1b800fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:59.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 shutdown_connections 2024-04-16T02:31:59.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7fa6a92c8700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa69006c4e0 0x7fa69006e9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7fa6a92c8700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4101d80 0x7fa6a419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7fa6a92c8700 1 --2- 172.21.15.151:0/2346143479 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6a41026a0 0x7fa6a419ca70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 >> 172.21.15.151:0/2346143479 conn(0x7fa6a40fb550 msgr2=0x7fa6a41053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:59.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7f1b84d60700 1 -- 172.21.15.151:0/1498963827 shutdown_connections 2024-04-16T02:31:59.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7f1b84d60700 1 -- 172.21.15.151:0/1498963827 wait complete. 2024-04-16T02:31:59.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 shutdown_connections 2024-04-16T02:31:59.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7fa6a92c8700 1 -- 172.21.15.151:0/2346143479 wait complete. 2024-04-16T02:31:59.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.435+0000 7f1b84d60700 1 Processor -- start 2024-04-16T02:31:59.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b84d60700 1 -- start start 2024-04-16T02:31:59.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b84d60700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b800ffa40 0x7f1b80195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:59.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b84d60700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 0x7f1b80195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:59.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b84d60700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80196270 con 0x7f1b80105a40 2024-04-16T02:31:59.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b84d60700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8019a000 con 0x7f1b800ffa40 2024-04-16T02:31:59.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b7dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 0x7f1b80195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b7dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 0x7f1b80195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48046/0 (socket says 172.21.15.151:48046) 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b7dd9b700 1 -- 172.21.15.151:0/4038473004 learned_addr learned my addr 172.21.15.151:0/4038473004 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b7e59c700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b800ffa40 0x7f1b80195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b7dd9b700 1 -- 172.21.15.151:0/4038473004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b800ffa40 msgr2=0x7f1b80195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b7dd9b700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b800ffa40 0x7f1b80195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b7dd9b700 1 -- 172.21.15.151:0/4038473004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b700095f0 con 0x7f1b80105a40 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b7e59c700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b800ffa40 0x7f1b80195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.436+0000 7f1b7dd9b700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 0x7f1b80195b90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009970 tx=0x7f1b74009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.437+0000 7f1b6f7fe700 1 -- 172.21.15.151:0/4038473004 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b74012070 con 0x7f1b80105a40 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.437+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8019a280 con 0x7f1b80105a40 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.437+0000 7f1b6f7fe700 1 -- 172.21.15.151:0/4038473004 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b74007ae0 con 0x7f1b80105a40 2024-04-16T02:31:59.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.437+0000 7f1b6f7fe700 1 -- 172.21.15.151:0/4038473004 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b74017c90 con 0x7f1b80105a40 2024-04-16T02:31:59.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.437+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8019a7d0 con 0x7f1b80105a40 2024-04-16T02:31:59.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.438+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8004ea90 con 0x7f1b80105a40 2024-04-16T02:31:59.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.439+0000 7f1b6f7fe700 1 -- 172.21.15.151:0/4038473004 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1b74007c50 con 0x7f1b80105a40 2024-04-16T02:31:59.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.439+0000 7f1b6f7fe700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1b6806c4e0 0x7f1b6806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:31:59.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.440+0000 7f1b6f7fe700 1 -- 172.21.15.151:0/4038473004 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1b7408c990 con 0x7f1b80105a40 2024-04-16T02:31:59.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.440+0000 7f1b7e59c700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1b6806c4e0 0x7f1b6806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:31:59.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.441+0000 7f1b7e59c700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1b6806c4e0 0x7f1b6806e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000b5c0 tx=0x7f1b7001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:31:59.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.445+0000 7f1b6f7fe700 1 -- 172.21.15.151:0/4038473004 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1b740575e0 con 0x7f1b80105a40 2024-04-16T02:31:59.447 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:31:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.642+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1b8019a410 con 0x7f1b80105a40 2024-04-16T02:31:59.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.642+0000 7f1b6f7fe700 1 -- 172.21.15.151:0/4038473004 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b8019a410 con 0x7f1b80105a40 2024-04-16T02:31:59.644 INFO:teuthology.orchestra.run.smithi151.stdout:60129542176 2024-04-16T02:31:59.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1b6806c4e0 msgr2=0x7f1b6806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:59.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1b6806c4e0 0x7f1b6806e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000b5c0 tx=0x7f1b7001a040 comp rx=0 tx=0).stop 2024-04-16T02:31:59.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 msgr2=0x7f1b80195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:31:59.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 0x7f1b80195b90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009970 tx=0x7f1b74009d30 comp rx=0 tx=0).stop 2024-04-16T02:31:59.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 shutdown_connections 2024-04-16T02:31:59.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1b6806c4e0 0x7f1b6806e9a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b800ffa40 0x7f1b80195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 --2- 172.21.15.151:0/4038473004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b80105a40 0x7f1b80195b90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:31:59.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 >> 172.21.15.151:0/4038473004 conn(0x7f1b800fb570 msgr2=0x7f1b800fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:31:59.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 shutdown_connections 2024-04-16T02:31:59.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:31:59.644+0000 7f1b84d60700 1 -- 172.21.15.151:0/4038473004 wait complete. 2024-04-16T02:32:00.151 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:59 smithi151 ceph-mon[29003]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:00.151 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:59 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2346143479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-16T02:32:00.151 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:31:59 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/4038473004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-16T02:32:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:59 smithi153 ceph-mon[36747]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:59 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2346143479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-16T02:32:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:31:59 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/4038473004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-16T02:32:01.100 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855374 got 150323855376 for osd.4 2024-04-16T02:32:01.100 DEBUG:teuthology.parallel:result is None 2024-04-16T02:32:01.808 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-04-16T02:32:01.808 DEBUG:teuthology.parallel:result is None 2024-04-16T02:32:02.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:01 smithi151 ceph-mon[29003]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:02.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:01 smithi153 ceph-mon[36747]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:03.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.964+0000 7f4893e09700 1 -- 172.21.15.151:0/3154969612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c0ffbd0 msgr2=0x7f488c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.964+0000 7f4893e09700 1 --2- 172.21.15.151:0/3154969612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c0ffbd0 0x7f488c100050 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f487c009910 tx=0x7f487c009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.964+0000 7f4893e09700 1 -- 172.21.15.151:0/3154969612 shutdown_connections 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.964+0000 7f4893e09700 1 --2- 172.21.15.151:0/3154969612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c0ffbd0 0x7f488c100050 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.964+0000 7f4893e09700 1 --2- 172.21.15.151:0/3154969612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c105a40 0x7f488c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.964+0000 7f4893e09700 1 -- 172.21.15.151:0/3154969612 >> 172.21.15.151:0/3154969612 conn(0x7f488c0755a0 msgr2=0x7f488c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.964+0000 7f4893e09700 1 -- 172.21.15.151:0/3154969612 shutdown_connections 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.964+0000 7f4893e09700 1 -- 172.21.15.151:0/3154969612 wait complete. 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f4893e09700 1 Processor -- start 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f4893e09700 1 -- start start 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f4893e09700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c0ffbd0 0x7f488c101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f4893e09700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c105a40 0x7f488c101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f4893e09700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c102090 con 0x7f488c105a40 2024-04-16T02:32:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f4893e09700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c1021d0 con 0x7f488c0ffbd0 2024-04-16T02:32:03.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f48913a4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c105a40 0x7f488c101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:03.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f48913a4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c105a40 0x7f488c101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48086/0 (socket says 172.21.15.151:48086) 2024-04-16T02:32:03.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f48913a4700 1 -- 172.21.15.151:0/3510589762 learned_addr learned my addr 172.21.15.151:0/3510589762 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:03.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.965+0000 7f4891ba5700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c0ffbd0 0x7f488c101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f48913a4700 1 -- 172.21.15.151:0/3510589762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c0ffbd0 msgr2=0x7f488c101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f48913a4700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c0ffbd0 0x7f488c101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f48913a4700 1 -- 172.21.15.151:0/3510589762 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c0095f0 con 0x7f488c105a40 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f4891ba5700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c0ffbd0 0x7f488c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f48913a4700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c105a40 0x7f488c101b50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f487c006010 tx=0x7f487c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f4882ffd700 1 -- 172.21.15.151:0/3510589762 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487c01d070 con 0x7f488c105a40 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f4882ffd700 1 -- 172.21.15.151:0/3510589762 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f487c00f460 con 0x7f488c105a40 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c102450 con 0x7f488c105a40 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f4882ffd700 1 -- 172.21.15.151:0/3510589762 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487c021640 con 0x7f488c105a40 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.966+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c1a2c20 con 0x7f488c105a40 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.967+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c108fe0 con 0x7f488c105a40 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.968+0000 7f4882ffd700 1 -- 172.21.15.151:0/3510589762 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f487c00fab0 con 0x7f488c105a40 2024-04-16T02:32:03.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.968+0000 7f4882ffd700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f487806c4e0 0x7f487806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:03.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.968+0000 7f4882ffd700 1 -- 172.21.15.151:0/3510589762 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f487c08c730 con 0x7f488c105a40 2024-04-16T02:32:03.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.968+0000 7f4891ba5700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f487806c4e0 0x7f487806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:03.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.968+0000 7f4891ba5700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f487806c4e0 0x7f487806e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4888005950 tx=0x7f4888005f50 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:03.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.972+0000 7f4882ffd700 1 -- 172.21.15.151:0/3510589762 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f487c0572d0 con 0x7f488c105a40 2024-04-16T02:32:03.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:03 smithi151 ceph-mon[29003]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:04.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 -- 172.21.15.151:0/1957753578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0100da0 msgr2=0x7fe8a0101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:04.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 --2- 172.21.15.151:0/1957753578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0100da0 0x7fe8a0101180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe890009910 tx=0x7fe890009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:04.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 -- 172.21.15.151:0/1957753578 shutdown_connections 2024-04-16T02:32:04.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 --2- 172.21.15.151:0/1957753578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a0101750 0x7fe8a01057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 --2- 172.21.15.151:0/1957753578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0100da0 0x7fe8a0101180 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 -- 172.21.15.151:0/1957753578 >> 172.21.15.151:0/1957753578 conn(0x7fe8a00fc630 msgr2=0x7fe8a00fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 -- 172.21.15.151:0/1957753578 shutdown_connections 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 -- 172.21.15.151:0/1957753578 wait complete. 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 Processor -- start 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.844+0000 7fe8a55af700 1 -- start start 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe8a55af700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a0100da0 0x7fe8a019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe8a55af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0101750 0x7fe8a019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe8a55af700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a019d210 con 0x7fe8a0101750 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe8a55af700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a01966c0 con 0x7fe8a0100da0 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe89e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0101750 0x7fe8a019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe89e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0101750 0x7fe8a019cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48076/0 (socket says 172.21.15.151:48076) 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe89e7fc700 1 -- 172.21.15.151:0/1180122125 learned_addr learned my addr 172.21.15.151:0/1180122125 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe89effd700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a0100da0 0x7fe8a019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:04.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe89e7fc700 1 -- 172.21.15.151:0/1180122125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a0100da0 msgr2=0x7fe8a019c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe89e7fc700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a0100da0 0x7fe8a019c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe89e7fc700 1 -- 172.21.15.151:0/1180122125 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900095f0 con 0x7fe8a0101750 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe89e7fc700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0101750 0x7fe8a019cb80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe89400b480 tx=0x7fe89400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe887fff700 1 -- 172.21.15.151:0/1180122125 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8940118c0 con 0x7fe8a0101750 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe887fff700 1 -- 172.21.15.151:0/1180122125 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe894010380 con 0x7fe8a0101750 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a01969a0 con 0x7fe8a0101750 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe887fff700 1 -- 172.21.15.151:0/1180122125 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89400f4f0 con 0x7fe8a0101750 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.845+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a0196ef0 con 0x7fe8a0101750 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.846+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a0109140 con 0x7fe8a0101750 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.847+0000 7fe887fff700 1 -- 172.21.15.151:0/1180122125 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe8940104f0 con 0x7fe8a0101750 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.847+0000 7fe887fff700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe88806c4e0 0x7fe88806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.847+0000 7fe89effd700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe88806c4e0 0x7fe88806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.847+0000 7fe887fff700 1 -- 172.21.15.151:0/1180122125 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe89408b060 con 0x7fe8a0101750 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.848+0000 7fe89effd700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe88806c4e0 0x7fe88806e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe890006010 tx=0x7fe89000b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:03.852+0000 7fe887fff700 1 -- 172.21.15.151:0/1180122125 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe894059690 con 0x7fe8a0101750 2024-04-16T02:32:04.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.087+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe8a004ea90 con 0x7fe8a0101750 2024-04-16T02:32:04.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.087+0000 7fe887fff700 1 -- 172.21.15.151:0/1180122125 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe894059220 con 0x7fe8a0101750 2024-04-16T02:32:04.122 INFO:teuthology.orchestra.run.smithi151.stdout:171798691852 2024-04-16T02:32:04.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe88806c4e0 msgr2=0x7fe88806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:04.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe88806c4e0 0x7fe88806e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe890006010 tx=0x7fe89000b540 comp rx=0 tx=0).stop 2024-04-16T02:32:04.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0101750 msgr2=0x7fe8a019cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:04.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0101750 0x7fe8a019cb80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe89400b480 tx=0x7fe89400b840 comp rx=0 tx=0).stop 2024-04-16T02:32:04.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 shutdown_connections 2024-04-16T02:32:04.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe88806c4e0 0x7fe88806e9a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:04.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a0100da0 0x7fe8a019c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:04.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 --2- 172.21.15.151:0/1180122125 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a0101750 0x7fe8a019cb80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:04.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 >> 172.21.15.151:0/1180122125 conn(0x7fe8a00fc630 msgr2=0x7fe8a00fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:04.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 shutdown_connections 2024-04-16T02:32:04.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.089+0000 7fe8a55af700 1 -- 172.21.15.151:0/1180122125 wait complete. 2024-04-16T02:32:04.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.210+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f488c1089c0 con 0x7f488c105a40 2024-04-16T02:32:04.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.210+0000 7f4882ffd700 1 -- 172.21.15.151:0/3510589762 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f487c05a8f0 con 0x7f488c105a40 2024-04-16T02:32:04.212 INFO:teuthology.orchestra.run.smithi151.stdout:115964117013 2024-04-16T02:32:04.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.212+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f487806c4e0 msgr2=0x7f487806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:04.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.212+0000 7f4893e09700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f487806c4e0 0x7f487806e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4888005950 tx=0x7f4888005f50 comp rx=0 tx=0).stop 2024-04-16T02:32:04.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.212+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c105a40 msgr2=0x7f488c101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:04.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.212+0000 7f4893e09700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c105a40 0x7f488c101b50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f487c006010 tx=0x7f487c00be20 comp rx=0 tx=0).stop 2024-04-16T02:32:04.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.213+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 shutdown_connections 2024-04-16T02:32:04.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.213+0000 7f4893e09700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f487806c4e0 0x7f487806e9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:04.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.213+0000 7f4893e09700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c0ffbd0 0x7f488c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:04.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.213+0000 7f4893e09700 1 --2- 172.21.15.151:0/3510589762 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f488c105a40 0x7f488c101b50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:04.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.213+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 >> 172.21.15.151:0/3510589762 conn(0x7f488c0755a0 msgr2=0x7f488c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:04.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.213+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 shutdown_connections 2024-04-16T02:32:04.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:04.213+0000 7f4893e09700 1 -- 172.21.15.151:0/3510589762 wait complete. 2024-04-16T02:32:04.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:03 smithi153 ceph-mon[36747]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:04.928 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117009 got 115964117013 for osd.3 2024-04-16T02:32:04.928 DEBUG:teuthology.parallel:result is None 2024-04-16T02:32:05.159 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:04 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1180122125' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-16T02:32:05.160 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:04 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3510589762' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-16T02:32:05.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:04 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1180122125' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-16T02:32:05.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:04 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3510589762' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-16T02:32:05.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/2772116420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 msgr2=0x7fe0ac1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:05.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 --2- 172.21.15.151:0/2772116420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 0x7fe0ac1011c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0009910 tx=0x7fe0a0009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:05.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/2772116420 shutdown_connections 2024-04-16T02:32:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 --2- 172.21.15.151:0/2772116420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0ac101790 0x7fe0ac1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 --2- 172.21.15.151:0/2772116420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 0x7fe0ac1011c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/2772116420 >> 172.21.15.151:0/2772116420 conn(0x7fe0ac0fc670 msgr2=0x7fe0ac0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/2772116420 shutdown_connections 2024-04-16T02:32:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/2772116420 wait complete. 2024-04-16T02:32:05.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 Processor -- start 2024-04-16T02:32:05.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.362+0000 7fe0b2d2c700 1 -- start start 2024-04-16T02:32:05.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b2d2c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 0x7fe0ac19c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:05.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b2d2c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0ac101790 0x7fe0ac19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:05.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b2d2c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac19d070 con 0x7fe0ac100de0 2024-04-16T02:32:05.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b2d2c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac1965b0 con 0x7fe0ac101790 2024-04-16T02:32:05.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b0ac8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 0x7fe0ac19c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b0ac8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 0x7fe0ac19c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48106/0 (socket says 172.21.15.151:48106) 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b0ac8700 1 -- 172.21.15.151:0/252903873 learned_addr learned my addr 172.21.15.151:0/252903873 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0abfff700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0ac101790 0x7fe0ac19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b0ac8700 1 -- 172.21.15.151:0/252903873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0ac101790 msgr2=0x7fe0ac19ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b0ac8700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0ac101790 0x7fe0ac19ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b0ac8700 1 -- 172.21.15.151:0/252903873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a00095f0 con 0x7fe0ac100de0 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0abfff700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0ac101790 0x7fe0ac19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b0ac8700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 0x7fe0ac19c530 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0006010 tx=0x7fe0a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0a9ffb700 1 -- 172.21.15.151:0/252903873 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0a001d070 con 0x7fe0ac100de0 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0a9ffb700 1 -- 172.21.15.151:0/252903873 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe0a000f460 con 0x7fe0ac100de0 2024-04-16T02:32:05.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0a9ffb700 1 -- 172.21.15.151:0/252903873 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0a0021640 con 0x7fe0ac100de0 2024-04-16T02:32:05.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.363+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac196830 con 0x7fe0ac100de0 2024-04-16T02:32:05.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.364+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac196d20 con 0x7fe0ac100de0 2024-04-16T02:32:05.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.364+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac109180 con 0x7fe0ac100de0 2024-04-16T02:32:05.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.365+0000 7fe0a9ffb700 1 -- 172.21.15.151:0/252903873 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe0a000fab0 con 0x7fe0ac100de0 2024-04-16T02:32:05.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.365+0000 7fe0a9ffb700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe09c06c4e0 0x7fe09c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:05.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.366+0000 7fe0a9ffb700 1 -- 172.21.15.151:0/252903873 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe0a008c7d0 con 0x7fe0ac100de0 2024-04-16T02:32:05.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.366+0000 7fe0abfff700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe09c06c4e0 0x7fe09c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:05.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.366+0000 7fe0abfff700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe09c06c4e0 0x7fe09c06e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe098005fd0 tx=0x7fe098005e30 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:05.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.370+0000 7fe0a9ffb700 1 -- 172.21.15.151:0/252903873 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe0a005ad80 con 0x7fe0ac100de0 2024-04-16T02:32:05.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.571+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe0ac108b60 con 0x7fe0ac100de0 2024-04-16T02:32:05.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.571+0000 7fd958f89700 1 -- 172.21.15.151:0/1774586590 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 msgr2=0x7fd954104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:05.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.571+0000 7fd958f89700 1 --2- 172.21.15.151:0/1774586590 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 0x7fd954104e30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd944009910 tx=0x7fd944009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:05.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.572+0000 7fe0a9ffb700 1 -- 172.21.15.151:0/252903873 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe0a005a910 con 0x7fe0ac100de0 2024-04-16T02:32:05.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.572+0000 7fd958f89700 1 -- 172.21.15.151:0/1774586590 shutdown_connections 2024-04-16T02:32:05.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.572+0000 7fd958f89700 1 --2- 172.21.15.151:0/1774586590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd954100180 0x7fd954100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.572+0000 7fd958f89700 1 --2- 172.21.15.151:0/1774586590 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 0x7fd954104e30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.572+0000 7fd958f89700 1 -- 172.21.15.151:0/1774586590 >> 172.21.15.151:0/1774586590 conn(0x7fd954075e30 msgr2=0x7fd9540fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:05.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.572+0000 7fd958f89700 1 -- 172.21.15.151:0/1774586590 shutdown_connections 2024-04-16T02:32:05.575 INFO:teuthology.orchestra.run.smithi151.stdout:38654705703 2024-04-16T02:32:05.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.572+0000 7fd958f89700 1 -- 172.21.15.151:0/1774586590 wait complete. 2024-04-16T02:32:05.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd958f89700 1 Processor -- start 2024-04-16T02:32:05.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd958f89700 1 -- start start 2024-04-16T02:32:05.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd958f89700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd954100180 0x7fd954196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:05.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd958f89700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 0x7fd954196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd958f89700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd954197280 con 0x7fd954104a50 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd958f89700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95419b010 con 0x7fd954100180 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd951d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 0x7fd954196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd951d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 0x7fd954196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48126/0 (socket says 172.21.15.151:48126) 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd951d9b700 1 -- 172.21.15.151:0/1740960867 learned_addr learned my addr 172.21.15.151:0/1740960867 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd95259c700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd954100180 0x7fd954196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd951d9b700 1 -- 172.21.15.151:0/1740960867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd954100180 msgr2=0x7fd954196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd951d9b700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd954100180 0x7fd954196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd951d9b700 1 -- 172.21.15.151:0/1740960867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9440095f0 con 0x7fd954104a50 2024-04-16T02:32:05.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.573+0000 7fd95259c700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd954100180 0x7fd954196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:32:05.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fd951d9b700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 0x7fd954196ba0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd94800e920 tx=0x7fd94800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:05.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fd9437fe700 1 -- 172.21.15.151:0/1740960867 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd948009790 con 0x7fd954104a50 2024-04-16T02:32:05.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fd9437fe700 1 -- 172.21.15.151:0/1740960867 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd94800cba0 con 0x7fd954104a50 2024-04-16T02:32:05.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fd9437fe700 1 -- 172.21.15.151:0/1740960867 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd948018630 con 0x7fd954104a50 2024-04-16T02:32:05.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95419b2f0 con 0x7fd954104a50 2024-04-16T02:32:05.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe09c06c4e0 msgr2=0x7fe09c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:05.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fe0b2d2c700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe09c06c4e0 0x7fe09c06e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe098005fd0 tx=0x7fe098005e30 comp rx=0 tx=0).stop 2024-04-16T02:32:05.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 msgr2=0x7fe0ac19c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:05.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fe0b2d2c700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 0x7fe0ac19c530 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0006010 tx=0x7fe0a000be20 comp rx=0 tx=0).stop 2024-04-16T02:32:05.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95419b7f0 con 0x7fd954104a50 2024-04-16T02:32:05.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 shutdown_connections 2024-04-16T02:32:05.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fe0b2d2c700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe09c06c4e0 0x7fe09c06e9a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fe0b2d2c700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0ac100de0 0x7fe0ac19c530 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.574+0000 7fe0b2d2c700 1 --2- 172.21.15.151:0/252903873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0ac101790 0x7fe0ac19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.575+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 >> 172.21.15.151:0/252903873 conn(0x7fe0ac0fc670 msgr2=0x7fe0ac0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:05.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.575+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 shutdown_connections 2024-04-16T02:32:05.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.575+0000 7fe0b2d2c700 1 -- 172.21.15.151:0/252903873 wait complete. 2024-04-16T02:32:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.575+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95404ea90 con 0x7fd954104a50 2024-04-16T02:32:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.576+0000 7fd9437fe700 1 -- 172.21.15.151:0/1740960867 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd9480188f0 con 0x7fd954104a50 2024-04-16T02:32:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.576+0000 7fd9437fe700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd93c06c4e0 0x7fd93c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.576+0000 7fd9437fe700 1 -- 172.21.15.151:0/1740960867 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd948014070 con 0x7fd954104a50 2024-04-16T02:32:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.576+0000 7fd95259c700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd93c06c4e0 0x7fd93c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.576+0000 7fd95259c700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd93c06c4e0 0x7fd93c06e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd944000c00 tx=0x7fd944009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.580+0000 7fd9437fe700 1 -- 172.21.15.151:0/1740960867 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd9480520e0 con 0x7fd954104a50 2024-04-16T02:32:05.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.777+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd9541979c0 con 0x7fd954104a50 2024-04-16T02:32:05.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.778+0000 7fd9437fe700 1 -- 172.21.15.151:0/1740960867 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd948059df0 con 0x7fd954104a50 2024-04-16T02:32:05.780 INFO:teuthology.orchestra.run.smithi151.stdout:81604378652 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd93c06c4e0 msgr2=0x7fd93c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd93c06c4e0 0x7fd93c06e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd944000c00 tx=0x7fd944009f70 comp rx=0 tx=0).stop 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 msgr2=0x7fd954196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 0x7fd954196ba0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd94800e920 tx=0x7fd94800ece0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 shutdown_connections 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd93c06c4e0 0x7fd93c06e9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd954100180 0x7fd954196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 --2- 172.21.15.151:0/1740960867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd954104a50 0x7fd954196ba0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 >> 172.21.15.151:0/1740960867 conn(0x7fd954075e30 msgr2=0x7fd9540fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:05.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 shutdown_connections 2024-04-16T02:32:05.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:05.780+0000 7fd958f89700 1 -- 172.21.15.151:0/1740960867 wait complete. 2024-04-16T02:32:06.011 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691850 got 171798691852 for osd.5 2024-04-16T02:32:06.011 DEBUG:teuthology.parallel:result is None 2024-04-16T02:32:06.222 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:05 smithi151 ceph-mon[29003]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:06.222 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:05 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/252903873' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-16T02:32:06.222 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:05 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1740960867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-16T02:32:06.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:05 smithi153 ceph-mon[36747]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:05 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/252903873' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-16T02:32:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:05 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1740960867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-16T02:32:06.801 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378650 got 81604378652 for osd.2 2024-04-16T02:32:06.802 DEBUG:teuthology.parallel:result is None 2024-04-16T02:32:07.323 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705703 for osd.0 2024-04-16T02:32:07.323 DEBUG:teuthology.parallel:result is None 2024-04-16T02:32:07.323 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-16T02:32:07.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-16T02:32:07.542 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:08.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:07 smithi151 ceph-mon[29003]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:07 smithi153 ceph-mon[36747]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.172+0000 7f7a7dded700 1 -- 172.21.15.151:0/2855188616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a780ffbd0 msgr2=0x7f7a78100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.172+0000 7f7a7dded700 1 --2- 172.21.15.151:0/2855188616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a780ffbd0 0x7f7a78100050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009910 tx=0x7f7a68009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.172+0000 7f7a7dded700 1 -- 172.21.15.151:0/2855188616 shutdown_connections 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.172+0000 7f7a7dded700 1 --2- 172.21.15.151:0/2855188616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a780ffbd0 0x7f7a78100050 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.172+0000 7f7a7dded700 1 --2- 172.21.15.151:0/2855188616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a78105a40 0x7f7a78105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.172+0000 7f7a7dded700 1 -- 172.21.15.151:0/2855188616 >> 172.21.15.151:0/2855188616 conn(0x7f7a780755a0 msgr2=0x7f7a780759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.172+0000 7f7a7dded700 1 -- 172.21.15.151:0/2855188616 shutdown_connections 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.172+0000 7f7a7dded700 1 -- 172.21.15.151:0/2855188616 wait complete. 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a7dded700 1 Processor -- start 2024-04-16T02:32:09.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a7dded700 1 -- start start 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a7dded700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a780ffbd0 0x7f7a7819c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a7dded700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a78105a40 0x7f7a7819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a7dded700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7819d140 con 0x7f7a78105a40 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a7dded700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a781965f0 con 0x7f7a780ffbd0 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a76ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a78105a40 0x7f7a7819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a76ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a78105a40 0x7f7a7819cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36580/0 (socket says 172.21.15.151:36580) 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a76ffd700 1 -- 172.21.15.151:0/1832780738 learned_addr learned my addr 172.21.15.151:0/1832780738 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.173+0000 7f7a777fe700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a780ffbd0 0x7f7a7819c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a76ffd700 1 -- 172.21.15.151:0/1832780738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a780ffbd0 msgr2=0x7f7a7819c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a76ffd700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a780ffbd0 0x7f7a7819c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a76ffd700 1 -- 172.21.15.151:0/1832780738 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680095f0 con 0x7f7a78105a40 2024-04-16T02:32:09.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a777fe700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a780ffbd0 0x7f7a7819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:32:09.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a76ffd700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a78105a40 0x7f7a7819cab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7a68006010 tx=0x7f7a6800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:09.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a74ff9700 1 -- 172.21.15.151:0/1832780738 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a6801d070 con 0x7f7a78105a40 2024-04-16T02:32:09.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a74ff9700 1 -- 172.21.15.151:0/1832780738 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7a6800f460 con 0x7f7a78105a40 2024-04-16T02:32:09.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a78196870 con 0x7f7a78105a40 2024-04-16T02:32:09.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a74ff9700 1 -- 172.21.15.151:0/1832780738 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a68021640 con 0x7f7a78105a40 2024-04-16T02:32:09.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.174+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a78196d60 con 0x7f7a78105a40 2024-04-16T02:32:09.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.175+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a78109070 con 0x7f7a78105a40 2024-04-16T02:32:09.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.175+0000 7f7a74ff9700 1 -- 172.21.15.151:0/1832780738 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7a6800faf0 con 0x7f7a78105a40 2024-04-16T02:32:09.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.176+0000 7f7a74ff9700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f7a6006c4e0 0x7f7a6006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:09.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.176+0000 7f7a74ff9700 1 -- 172.21.15.151:0/1832780738 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7a6808c870 con 0x7f7a78105a40 2024-04-16T02:32:09.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.176+0000 7f7a777fe700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f7a6006c4e0 0x7f7a6006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:09.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.176+0000 7f7a777fe700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f7a6006c4e0 0x7f7a6006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c0099d0 tx=0x7f7a6c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:09.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.180+0000 7f7a74ff9700 1 -- 172.21.15.151:0/1832780738 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7a6805ae20 con 0x7f7a78105a40 2024-04-16T02:32:09.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.364+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7a7802ccf0 con 0x7f7a6006c4e0 2024-04-16T02:32:09.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.366+0000 7f7a74ff9700 1 -- 172.21.15.151:0/1832780738 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28024 (secure 0 0 0) 0x7f7a7802ccf0 con 0x7f7a6006c4e0 2024-04-16T02:32:09.368 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f7a6006c4e0 msgr2=0x7f7a6006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f7a6006c4e0 0x7f7a6006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c0099d0 tx=0x7f7a6c008040 comp rx=0 tx=0).stop 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a78105a40 msgr2=0x7f7a7819cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a78105a40 0x7f7a7819cab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7a68006010 tx=0x7f7a6800be20 comp rx=0 tx=0).stop 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 shutdown_connections 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f7a6006c4e0 0x7f7a6006e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a780ffbd0 0x7f7a7819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 --2- 172.21.15.151:0/1832780738 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a78105a40 0x7f7a7819cab0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 >> 172.21.15.151:0/1832780738 conn(0x7f7a780755a0 msgr2=0x7f7a78068610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 shutdown_connections 2024-04-16T02:32:09.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:09.368+0000 7f7a7dded700 1 -- 172.21.15.151:0/1832780738 wait complete. 2024-04-16T02:32:09.371 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-04-16T02:32:09.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:09 smithi151 ceph-mon[29003]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:10.063 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-04-16T02:32:08.872144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002056"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":185,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-16T02:31:17.733795+0000","last_change":"2024-04-16T02:30:51.760405+0000","last_active":"2024-04-16T02:31:17.733795+0000","last_peered":"2024-04-16T02:31:17.733795+0000","last_clean":"2024-04-16T02:31:17.733795+0000","last_became_active":"2024-04-16T02:30:51.759984+0000","last_became_peered":"2024-04-16T02:30:51.759984+0000","last_unstale":"2024-04-16T02:31:17.733795+0000","last_undegraded":"2024-04-16T02:31:17.733795+0000","last_fullsized":"2024-04-16T02:31:17.733795+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T02:30:03.707564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T02:30:03.707564+0000","last_clean_scrub_stamp":"2024-04-16T02:30:03.707564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T05:23:59.150162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.622}]},{"osd":1,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.44500000000000001}]},{"osd":2,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.56899999999999995}]},{"osd":3,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.311},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.32600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 02:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Tue Apr 16 02:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.443},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.501}]},{"osd":2,"last update":"Tue Apr 16 02:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.54500000000000004}]},{"osd":4,"last update":"Tue Apr 16 02:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 02:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34000000000000002},"min":{"1min":0.26400000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.34399999999999997,"15min":0.34000000000000002},"min":{"1min":0.23300000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.27400000000000002}]},{"osd":1,"last update":"Tue Apr 16 02:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.35299999999999998,"15min":0.35099999999999998},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.30499999999999999}]},{"osd":3,"last update":"Tue Apr 16 02:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.42599999999999999}]},{"osd":4,"last update":"Tue Apr 16 02:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 16 02:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34999999999999998,"15min":0.35199999999999998},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.375,"15min":0.377},"min":{"1min":0.26400000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.47799999999999998,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.34699999999999998}]},{"osd":2,"last update":"Tue Apr 16 02:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.30499999999999999}]},{"osd":3,"last update":"Tue Apr 16 02:31:27 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.53400000000000003}]},{"osd":4,"last update":"Tue Apr 16 02:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 02:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.29899999999999999,"15min":0.28899999999999998},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.29399999999999998},"min":{"1min":0.23999999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.438}]},{"osd":2,"last update":"Tue Apr 16 02:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.502,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35999999999999999,"15min":0.36199999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.46999999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.41599999999999998}]},{"osd":3,"last update":"Tue Apr 16 02:31:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.497},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T02:32:10.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-16T02:32:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:09 smithi153 ceph-mon[36747]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:10.273 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:10.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:10 smithi151 ceph-mon[29003]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T02:32:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:10 smithi153 ceph-mon[36747]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T02:32:11.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.749+0000 7fb9e47b7700 1 -- 172.21.15.151:0/222654495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc105a40 msgr2=0x7fb9dc105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.749+0000 7fb9e47b7700 1 --2- 172.21.15.151:0/222654495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc105a40 0x7fb9dc105e20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009910 tx=0x7fb9cc009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.749+0000 7fb9e47b7700 1 -- 172.21.15.151:0/222654495 shutdown_connections 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.749+0000 7fb9e47b7700 1 --2- 172.21.15.151:0/222654495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc0ffbd0 0x7fb9dc100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.749+0000 7fb9e47b7700 1 --2- 172.21.15.151:0/222654495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc105a40 0x7fb9dc105e20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.749+0000 7fb9e47b7700 1 -- 172.21.15.151:0/222654495 >> 172.21.15.151:0/222654495 conn(0x7fb9dc0755a0 msgr2=0x7fb9dc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.749+0000 7fb9e47b7700 1 -- 172.21.15.151:0/222654495 shutdown_connections 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.749+0000 7fb9e47b7700 1 -- 172.21.15.151:0/222654495 wait complete. 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e47b7700 1 Processor -- start 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e47b7700 1 -- start start 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e47b7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc0ffbd0 0x7fb9dc197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e47b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc105a40 0x7fb9dc197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e47b7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc198190 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e47b7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc1982d0 con 0x7fb9dc105a40 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e2553700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc0ffbd0 0x7fb9dc197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e2553700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc0ffbd0 0x7fb9dc197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36598/0 (socket says 172.21.15.151:36598) 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e2553700 1 -- 172.21.15.151:0/1453936825 learned_addr learned my addr 172.21.15.151:0/1453936825 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.750+0000 7fb9e1d52700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc105a40 0x7fb9dc197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.751+0000 7fb9e2553700 1 -- 172.21.15.151:0/1453936825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc105a40 msgr2=0x7fb9dc197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.751+0000 7fb9e2553700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc105a40 0x7fb9dc197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.751+0000 7fb9e2553700 1 -- 172.21.15.151:0/1453936825 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc0095f0 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.751+0000 7fb9e2553700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc0ffbd0 0x7fb9dc197630 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc006010 tx=0x7fb9cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.751+0000 7fb9d37fe700 1 -- 172.21.15.151:0/1453936825 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9cc01d070 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.751+0000 7fb9d37fe700 1 -- 172.21.15.151:0/1453936825 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb9cc00f460 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.751+0000 7fb9d37fe700 1 -- 172.21.15.151:0/1453936825 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9cc021640 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.751+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc1a06a0 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.751+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc1a0ab0 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.752+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc108040 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.752+0000 7fb9d37fe700 1 -- 172.21.15.151:0/1453936825 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb9cc00faf0 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.753+0000 7fb9d37fe700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb9c806c4e0 0x7fb9c806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:11.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.753+0000 7fb9d37fe700 1 -- 172.21.15.151:0/1453936825 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb9cc08c780 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.753+0000 7fb9e1d52700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb9c806c4e0 0x7fb9c806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:11.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.753+0000 7fb9e1d52700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb9c806c4e0 0x7fb9c806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8006fd0 tx=0x7fb9d8008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:11.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.757+0000 7fb9d37fe700 1 -- 172.21.15.151:0/1453936825 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb9cc057320 con 0x7fb9dc0ffbd0 2024-04-16T02:32:11.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.944+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9dc02cdd0 con 0x7fb9c806c4e0 2024-04-16T02:32:11.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.946+0000 7fb9d37fe700 1 -- 172.21.15.151:0/1453936825 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27912 (secure 0 0 0) 0x7fb9dc02cdd0 con 0x7fb9c806c4e0 2024-04-16T02:32:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:32:11.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb9c806c4e0 msgr2=0x7fb9c806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:11.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb9c806c4e0 0x7fb9c806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8006fd0 tx=0x7fb9d8008040 comp rx=0 tx=0).stop 2024-04-16T02:32:11.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc0ffbd0 msgr2=0x7fb9dc197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:11.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc0ffbd0 0x7fb9dc197630 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc006010 tx=0x7fb9cc00be20 comp rx=0 tx=0).stop 2024-04-16T02:32:11.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 shutdown_connections 2024-04-16T02:32:11.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb9c806c4e0 0x7fb9c806e9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:11.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9dc0ffbd0 0x7fb9dc197630 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:11.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 --2- 172.21.15.151:0/1453936825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc105a40 0x7fb9dc197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:11.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 >> 172.21.15.151:0/1453936825 conn(0x7fb9dc0755a0 msgr2=0x7fb9dc0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:11.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 shutdown_connections 2024-04-16T02:32:11.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:11.949+0000 7fb9e47b7700 1 -- 172.21.15.151:0/1453936825 wait complete. 2024-04-16T02:32:11.951 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-04-16T02:32:12.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:11 smithi151 ceph-mon[29003]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:12.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:11 smithi153 ceph-mon[36747]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:12.659 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-04-16T02:32:10.872464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002081"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":185,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-16T02:31:17.733795+0000","last_change":"2024-04-16T02:30:51.760405+0000","last_active":"2024-04-16T02:31:17.733795+0000","last_peered":"2024-04-16T02:31:17.733795+0000","last_clean":"2024-04-16T02:31:17.733795+0000","last_became_active":"2024-04-16T02:30:51.759984+0000","last_became_peered":"2024-04-16T02:30:51.759984+0000","last_unstale":"2024-04-16T02:31:17.733795+0000","last_undegraded":"2024-04-16T02:31:17.733795+0000","last_fullsized":"2024-04-16T02:31:17.733795+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T02:30:03.707564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T02:30:03.707564+0000","last_clean_scrub_stamp":"2024-04-16T02:30:03.707564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T05:23:59.150162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.622}]},{"osd":1,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.44500000000000001}]},{"osd":2,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.56899999999999995}]},{"osd":3,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.311},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.32600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 02:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Tue Apr 16 02:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.443},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.501}]},{"osd":2,"last update":"Tue Apr 16 02:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.54500000000000004}]},{"osd":4,"last update":"Tue Apr 16 02:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 02:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34000000000000002},"min":{"1min":0.26400000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.34399999999999997,"15min":0.34000000000000002},"min":{"1min":0.23300000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.27400000000000002}]},{"osd":1,"last update":"Tue Apr 16 02:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.35299999999999998,"15min":0.35099999999999998},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.30499999999999999}]},{"osd":3,"last update":"Tue Apr 16 02:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.42599999999999999}]},{"osd":4,"last update":"Tue Apr 16 02:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 16 02:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34999999999999998,"15min":0.35199999999999998},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.375,"15min":0.377},"min":{"1min":0.26400000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.47799999999999998,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.316}]},{"osd":2,"last update":"Tue Apr 16 02:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.505,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.372,"15min":0.38100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44900000000000001,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.23699999999999999}]},{"osd":3,"last update":"Tue Apr 16 02:31:27 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.61699999999999999}]},{"osd":4,"last update":"Tue Apr 16 02:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 02:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.29899999999999999,"15min":0.28899999999999998},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.29399999999999998},"min":{"1min":0.23999999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.438}]},{"osd":2,"last update":"Tue Apr 16 02:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.502,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35999999999999999,"15min":0.36199999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.46999999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.41599999999999998}]},{"osd":3,"last update":"Tue Apr 16 02:31:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Apr 16 02:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.497},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T02:32:12.660 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-16T02:32:12.660 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T02:32:12.661 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-16T02:32:12.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph health --format=json 2024-04-16T02:32:12.869 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:12 smithi151 ceph-mon[29003]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T02:32:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:12 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:32:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:12 smithi153 ceph-mon[36747]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T02:32:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:32:14.072 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:13 smithi151 ceph-mon[29003]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:13 smithi153 ceph-mon[36747]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:14.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.582+0000 7f22510e6700 1 -- 172.21.15.151:0/1130991293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 msgr2=0x7f224c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:14.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.582+0000 7f22510e6700 1 --2- 172.21.15.151:0/1130991293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 0x7f224c1011a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f223c00b0a0 tx=0x7f223c00b3b0 comp rx=0 tx=0).stop 2024-04-16T02:32:14.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.582+0000 7f22510e6700 1 -- 172.21.15.151:0/1130991293 shutdown_connections 2024-04-16T02:32:14.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.582+0000 7f22510e6700 1 --2- 172.21.15.151:0/1130991293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f224c101770 0x7f224c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:14.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.582+0000 7f22510e6700 1 --2- 172.21.15.151:0/1130991293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 0x7f224c1011a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:14.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.582+0000 7f22510e6700 1 -- 172.21.15.151:0/1130991293 >> 172.21.15.151:0/1130991293 conn(0x7f224c0fc650 msgr2=0x7f224c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.582+0000 7f22510e6700 1 -- 172.21.15.151:0/1130991293 shutdown_connections 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.582+0000 7f22510e6700 1 -- 172.21.15.151:0/1130991293 wait complete. 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.583+0000 7f22510e6700 1 Processor -- start 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.583+0000 7f22510e6700 1 -- start start 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f22510e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 0x7f224c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f22510e6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f224c101770 0x7f224c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f22510e6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c19d110 con 0x7f224c100dc0 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f22510e6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c1965c0 con 0x7f224c101770 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f224ad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 0x7f224c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f224ad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 0x7f224c19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36622/0 (socket says 172.21.15.151:36622) 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f224ad9d700 1 -- 172.21.15.151:0/1698635966 learned_addr learned my addr 172.21.15.151:0/1698635966 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f224ad9d700 1 -- 172.21.15.151:0/1698635966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f224c101770 msgr2=0x7f224c19ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f224a59c700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f224c101770 0x7f224c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:14.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f224ad9d700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f224c101770 0x7f224c19ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:14.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f224ad9d700 1 -- 172.21.15.151:0/1698635966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c009d00 con 0x7f224c100dc0 2024-04-16T02:32:14.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f224ad9d700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 0x7f224c19c540 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f223c000f80 tx=0x7f223c00cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:14.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f2243fff700 1 -- 172.21.15.151:0/1698635966 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f223c00e040 con 0x7f224c100dc0 2024-04-16T02:32:14.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f2243fff700 1 -- 172.21.15.151:0/1698635966 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f223c003de0 con 0x7f224c100dc0 2024-04-16T02:32:14.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f2243fff700 1 -- 172.21.15.151:0/1698635966 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f223c01d9a0 con 0x7f224c100dc0 2024-04-16T02:32:14.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.584+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c196840 con 0x7f224c100dc0 2024-04-16T02:32:14.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.585+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c196d30 con 0x7f224c100dc0 2024-04-16T02:32:14.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.585+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c109160 con 0x7f224c100dc0 2024-04-16T02:32:14.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.586+0000 7f2243fff700 1 -- 172.21.15.151:0/1698635966 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f223c004040 con 0x7f224c100dc0 2024-04-16T02:32:14.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.586+0000 7f2243fff700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f223806c4e0 0x7f223806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:14.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.587+0000 7f2243fff700 1 -- 172.21.15.151:0/1698635966 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f223c08ce70 con 0x7f224c100dc0 2024-04-16T02:32:14.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.587+0000 7f224a59c700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f223806c4e0 0x7f223806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:14.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.587+0000 7f224a59c700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f223806c4e0 0x7f223806e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f224c1059b0 tx=0x7f22340094f0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:14.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.591+0000 7f2243fff700 1 -- 172.21.15.151:0/1698635966 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f223c05b420 con 0x7f224c100dc0 2024-04-16T02:32:14.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.826+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f224c04ea90 con 0x7f224c100dc0 2024-04-16T02:32:14.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.826+0000 7f2243fff700 1 -- 172.21.15.151:0/1698635966 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f223c05afb0 con 0x7f224c100dc0 2024-04-16T02:32:14.828 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:32:14.828 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T02:32:14.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.828+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f223806c4e0 msgr2=0x7f223806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:14.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.828+0000 7f22510e6700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f223806c4e0 0x7f223806e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f224c1059b0 tx=0x7f22340094f0 comp rx=0 tx=0).stop 2024-04-16T02:32:14.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.828+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 msgr2=0x7f224c19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:14.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.828+0000 7f22510e6700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 0x7f224c19c540 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f223c000f80 tx=0x7f223c00cdd0 comp rx=0 tx=0).stop 2024-04-16T02:32:14.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.829+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 shutdown_connections 2024-04-16T02:32:14.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.829+0000 7f22510e6700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f223806c4e0 0x7f223806e9a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:14.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.829+0000 7f22510e6700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f224c100dc0 0x7f224c19c540 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.829+0000 7f22510e6700 1 --2- 172.21.15.151:0/1698635966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f224c101770 0x7f224c19ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.829+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 >> 172.21.15.151:0/1698635966 conn(0x7f224c0fc650 msgr2=0x7f224c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.829+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 shutdown_connections 2024-04-16T02:32:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:14.829+0000 7f22510e6700 1 -- 172.21.15.151:0/1698635966 wait complete. 2024-04-16T02:32:15.468 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-16T02:32:15.468 INFO:tasks.cephadm:Setup complete, yielding 2024-04-16T02:32:15.469 INFO:teuthology.run_tasks:Running task print... 2024-04-16T02:32:15.478 INFO:teuthology.task.print:**** done end installing v18.2.1 cephadm ... 2024-04-16T02:32:15.478 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T02:32:15.495 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:32:15.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-16T02:32:15.706 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:15 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1698635966' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-16T02:32:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:15 smithi151 ceph-mon[29003]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:16.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:15 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1698635966' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-16T02:32:16.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:15 smithi153 ceph-mon[36747]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:17.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.277+0000 7f3040eb4700 1 -- 172.21.15.151:0/1061699081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 msgr2=0x7f303c100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:17.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.277+0000 7f3040eb4700 1 --2- 172.21.15.151:0/1061699081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 0x7f303c100fe0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f302c009910 tx=0x7f302c009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:17.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.278+0000 7f3040eb4700 1 -- 172.21.15.151:0/1061699081 shutdown_connections 2024-04-16T02:32:17.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.278+0000 7f3040eb4700 1 --2- 172.21.15.151:0/1061699081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c101dc0 0x7f303c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:17.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.278+0000 7f3040eb4700 1 --2- 172.21.15.151:0/1061699081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 0x7f303c100fe0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:17.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.278+0000 7f3040eb4700 1 -- 172.21.15.151:0/1061699081 >> 172.21.15.151:0/1061699081 conn(0x7f303c0fc140 msgr2=0x7f303c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:17.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.278+0000 7f3040eb4700 1 -- 172.21.15.151:0/1061699081 shutdown_connections 2024-04-16T02:32:17.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.278+0000 7f3040eb4700 1 -- 172.21.15.151:0/1061699081 wait complete. 2024-04-16T02:32:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f3040eb4700 1 Processor -- start 2024-04-16T02:32:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f3040eb4700 1 -- start start 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f3040eb4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 0x7f303c196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f3040eb4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c101dc0 0x7f303c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f3040eb4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c196e60 con 0x7f303c100bc0 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f3040eb4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c196fa0 con 0x7f303c101dc0 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f303a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 0x7f303c196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f303a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 0x7f303c196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35150/0 (socket says 172.21.15.151:35150) 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f303a59c700 1 -- 172.21.15.151:0/2857165710 learned_addr learned my addr 172.21.15.151:0/2857165710 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f3039d9b700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c101dc0 0x7f303c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f303a59c700 1 -- 172.21.15.151:0/2857165710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c101dc0 msgr2=0x7f303c1968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f303a59c700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c101dc0 0x7f303c1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f303a59c700 1 -- 172.21.15.151:0/2857165710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c0095f0 con 0x7f303c100bc0 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f303a59c700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 0x7f303c196390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f302c00bda0 tx=0x7f302c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.279+0000 7f302b7fe700 1 -- 172.21.15.151:0/2857165710 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f302c01d070 con 0x7f303c100bc0 2024-04-16T02:32:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.280+0000 7f302b7fe700 1 -- 172.21.15.151:0/2857165710 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f302c004c50 con 0x7f303c100bc0 2024-04-16T02:32:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.280+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c19ba00 con 0x7f303c100bc0 2024-04-16T02:32:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.280+0000 7f302b7fe700 1 -- 172.21.15.151:0/2857165710 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f302c00f690 con 0x7f303c100bc0 2024-04-16T02:32:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.280+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c19bec0 con 0x7f303c100bc0 2024-04-16T02:32:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.281+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c109720 con 0x7f303c100bc0 2024-04-16T02:32:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.281+0000 7f302b7fe700 1 -- 172.21.15.151:0/2857165710 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f302c022a20 con 0x7f303c100bc0 2024-04-16T02:32:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.281+0000 7f302b7fe700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f302406c4e0 0x7f302406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.282+0000 7f302b7fe700 1 -- 172.21.15.151:0/2857165710 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f302c08cc00 con 0x7f303c100bc0 2024-04-16T02:32:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.282+0000 7f3039d9b700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f302406c4e0 0x7f302406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.282+0000 7f3039d9b700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f302406c4e0 0x7f302406e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f303000ba60 tx=0x7f3030005d50 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:17.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.286+0000 7f302b7fe700 1 -- 172.21.15.151:0/2857165710 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f302c092020 con 0x7f303c100bc0 2024-04-16T02:32:17.477 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.475+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f303c1098e0 con 0x7f303c100bc0 2024-04-16T02:32:17.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.480+0000 7f302b7fe700 1 -- 172.21.15.151:0/2857165710 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f302c05b260 con 0x7f303c100bc0 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f302406c4e0 msgr2=0x7f302406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f302406c4e0 0x7f302406e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f303000ba60 tx=0x7f3030005d50 comp rx=0 tx=0).stop 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 msgr2=0x7f303c196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 0x7f303c196390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f302c00bda0 tx=0x7f302c00be80 comp rx=0 tx=0).stop 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 shutdown_connections 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f302406c4e0 0x7f302406e9a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f303c100bc0 0x7f303c196390 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 --2- 172.21.15.151:0/2857165710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c101dc0 0x7f303c1968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 >> 172.21.15.151:0/2857165710 conn(0x7f303c0fc140 msgr2=0x7f303c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 shutdown_connections 2024-04-16T02:32:17.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:17.482+0000 7f3040eb4700 1 -- 172.21.15.151:0/2857165710 wait complete. 2024-04-16T02:32:18.101 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:17 smithi151 ceph-mon[29003]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:18.101 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:17 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2857165710' entity='client.admin' 2024-04-16T02:32:18.101 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:17 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:32:18.122 INFO:teuthology.run_tasks:Running task print... 2024-04-16T02:32:18.130 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-16T02:32:18.130 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T02:32:18.138 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:32:18.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch status' 2024-04-16T02:32:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:17 smithi153 ceph-mon[36747]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:17 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2857165710' entity='client.admin' 2024-04-16T02:32:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:17 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:32:18.360 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:18.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:18.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:32:18.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:18 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:19.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:19.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:32:19.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:18 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:19.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.963+0000 7f06af59d700 1 -- 172.21.15.151:0/2961505713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 msgr2=0x7f06a81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:19.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.963+0000 7f06af59d700 1 --2- 172.21.15.151:0/2961505713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 0x7f06a81008b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0698009910 tx=0x7f0698009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:19.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.963+0000 7f06af59d700 1 -- 172.21.15.151:0/2961505713 shutdown_connections 2024-04-16T02:32:19.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.963+0000 7f06af59d700 1 --2- 172.21.15.151:0/2961505713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 0x7f06a81008b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.963+0000 7f06af59d700 1 --2- 172.21.15.151:0/2961505713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a80ff250 0x7f06a80ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.963+0000 7f06af59d700 1 -- 172.21.15.151:0/2961505713 >> 172.21.15.151:0/2961505713 conn(0x7f06a80fa7f0 msgr2=0x7f06a80fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.963+0000 7f06af59d700 1 -- 172.21.15.151:0/2961505713 shutdown_connections 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.963+0000 7f06af59d700 1 -- 172.21.15.151:0/2961505713 wait complete. 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.963+0000 7f06af59d700 1 Processor -- start 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06af59d700 1 -- start start 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06af59d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a80ff250 0x7f06a81021d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06af59d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 0x7f06a819b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06af59d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a819b610 con 0x7f06a8100450 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06af59d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a819b750 con 0x7f06a80ff250 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06acb38700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 0x7f06a819b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06acb38700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 0x7f06a819b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35172/0 (socket says 172.21.15.151:35172) 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06acb38700 1 -- 172.21.15.151:0/1215466774 learned_addr learned my addr 172.21.15.151:0/1215466774 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06ad339700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a80ff250 0x7f06a81021d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:19.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06acb38700 1 -- 172.21.15.151:0/1215466774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a80ff250 msgr2=0x7f06a81021d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:19.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06acb38700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a80ff250 0x7f06a81021d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:19.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06acb38700 1 -- 172.21.15.151:0/1215466774 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980095f0 con 0x7f06a8100450 2024-04-16T02:32:19.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f06acb38700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 0x7f06a819b0d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f069800bd70 tx=0x7f069800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:19.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.964+0000 7f069e7fc700 1 -- 172.21.15.151:0/1215466774 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f069801d070 con 0x7f06a8100450 2024-04-16T02:32:19.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.965+0000 7f069e7fc700 1 -- 172.21.15.151:0/1215466774 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0698005940 con 0x7f06a8100450 2024-04-16T02:32:19.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.965+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a819b9d0 con 0x7f06a8100450 2024-04-16T02:32:19.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.965+0000 7f069e7fc700 1 -- 172.21.15.151:0/1215466774 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0698021430 con 0x7f06a8100450 2024-04-16T02:32:19.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.965+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a819bec0 con 0x7f06a8100450 2024-04-16T02:32:19.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.966+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a8107db0 con 0x7f06a8100450 2024-04-16T02:32:19.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.966+0000 7f069e7fc700 1 -- 172.21.15.151:0/1215466774 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f069800f860 con 0x7f06a8100450 2024-04-16T02:32:19.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.966+0000 7f069e7fc700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f069406c4e0 0x7f069406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:19.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.966+0000 7f069e7fc700 1 -- 172.21.15.151:0/1215466774 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f069808c8e0 con 0x7f06a8100450 2024-04-16T02:32:19.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.966+0000 7f06ad339700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f069406c4e0 0x7f069406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:19.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.967+0000 7f06ad339700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f069406c4e0 0x7f069406e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f06a81002b0 tx=0x7f06a400a3b0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:19.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:19.971+0000 7f069e7fc700 1 -- 172.21.15.151:0/1215466774 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f069805ae90 con 0x7f06a8100450 2024-04-16T02:32:20.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.181+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f06a8104790 con 0x7f069406c4e0 2024-04-16T02:32:20.183 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:19 smithi151 ceph-mon[29003]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:20.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.182+0000 7f069e7fc700 1 -- 172.21.15.151:0/1215466774 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f06a8104790 con 0x7f069406c4e0 2024-04-16T02:32:20.183 INFO:teuthology.orchestra.run.smithi151.stdout:Backend: cephadm 2024-04-16T02:32:20.184 INFO:teuthology.orchestra.run.smithi151.stdout:Available: Yes 2024-04-16T02:32:20.184 INFO:teuthology.orchestra.run.smithi151.stdout:Paused: No 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f069406c4e0 msgr2=0x7f069406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f069406c4e0 0x7f069406e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f06a81002b0 tx=0x7f06a400a3b0 comp rx=0 tx=0).stop 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 msgr2=0x7f06a819b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 0x7f06a819b0d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f069800bd70 tx=0x7f069800bda0 comp rx=0 tx=0).stop 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 shutdown_connections 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f069406c4e0 0x7f069406e9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a80ff250 0x7f06a81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 --2- 172.21.15.151:0/1215466774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06a8100450 0x7f06a819b0d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 >> 172.21.15.151:0/1215466774 conn(0x7f06a80fa7f0 msgr2=0x7f06a80fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 shutdown_connections 2024-04-16T02:32:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:20.184+0000 7f06af59d700 1 -- 172.21.15.151:0/1215466774 wait complete. 2024-04-16T02:32:20.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:19 smithi153 ceph-mon[36747]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:20.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-16T02:32:21.038 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:22.072 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:21 smithi151 ceph-mon[29003]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:22.072 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:21 smithi151 ceph-mon[29003]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:21 smithi153 ceph-mon[36747]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:21 smithi153 ceph-mon[36747]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.593+0000 7f04a4140700 1 -- 172.21.15.151:0/1708223058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c104040 msgr2=0x7f049c106430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.593+0000 7f04a4140700 1 --2- 172.21.15.151:0/1708223058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c104040 0x7f049c106430 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0490009910 tx=0x7f0490009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.593+0000 7f04a4140700 1 -- 172.21.15.151:0/1708223058 shutdown_connections 2024-04-16T02:32:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.593+0000 7f04a4140700 1 --2- 172.21.15.151:0/1708223058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c104040 0x7f049c106430 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.593+0000 7f04a4140700 1 --2- 172.21.15.151:0/1708223058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f049c101710 0x7f049c103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.593+0000 7f04a4140700 1 -- 172.21.15.151:0/1708223058 >> 172.21.15.151:0/1708223058 conn(0x7f049c0fb080 msgr2=0x7f049c0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.593+0000 7f04a4140700 1 -- 172.21.15.151:0/1708223058 shutdown_connections 2024-04-16T02:32:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.593+0000 7f04a4140700 1 -- 172.21.15.151:0/1708223058 wait complete. 2024-04-16T02:32:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.594+0000 7f04a4140700 1 Processor -- start 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.594+0000 7f04a4140700 1 -- start start 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.594+0000 7f04a4140700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c101710 0x7f049c1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.594+0000 7f04a4140700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f049c104040 0x7f049c196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.594+0000 7f04a4140700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c196f50 con 0x7f049c101710 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.594+0000 7f04a4140700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c197090 con 0x7f049c104040 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.594+0000 7f04a1edc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c101710 0x7f049c1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.594+0000 7f04a1edc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c101710 0x7f049c1963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35176/0 (socket says 172.21.15.151:35176) 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.594+0000 7f04a1edc700 1 -- 172.21.15.151:0/3457157235 learned_addr learned my addr 172.21.15.151:0/3457157235 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f04a16db700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f049c104040 0x7f049c196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f04a1edc700 1 -- 172.21.15.151:0/3457157235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f049c104040 msgr2=0x7f049c196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f04a1edc700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f049c104040 0x7f049c196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f04a1edc700 1 -- 172.21.15.151:0/3457157235 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04900095f0 con 0x7f049c101710 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f04a16db700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f049c104040 0x7f049c196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:32:22.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f04a1edc700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c101710 0x7f049c1963f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f049800b4b0 tx=0x7f049800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f048effd700 1 -- 172.21.15.151:0/3457157235 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0498011800 con 0x7f049c101710 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c19bb40 con 0x7f049c101710 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f048effd700 1 -- 172.21.15.151:0/3457157235 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0498011e10 con 0x7f049c101710 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f048effd700 1 -- 172.21.15.151:0/3457157235 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f049800f2e0 con 0x7f049c101710 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.595+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c19c090 con 0x7f049c101710 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.596+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c1905e0 con 0x7f049c101710 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.596+0000 7f048effd700 1 -- 172.21.15.151:0/3457157235 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0498010380 con 0x7f049c101710 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.597+0000 7f048effd700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f048806c4e0 0x7f048806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.597+0000 7f048effd700 1 -- 172.21.15.151:0/3457157235 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f049808b130 con 0x7f049c101710 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.597+0000 7f04a16db700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f048806c4e0 0x7f048806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.597+0000 7f04a16db700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f048806c4e0 0x7f048806e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f049000b5c0 tx=0x7f0490005e20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:22.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.601+0000 7f048effd700 1 -- 172.21.15.151:0/3457157235 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f04980596e0 con 0x7f049c101710 2024-04-16T02:32:22.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.811+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f049c0611d0 con 0x7f048806c4e0 2024-04-16T02:32:22.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.820+0000 7f048effd700 1 -- 172.21.15.151:0/3457157235 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f049c0611d0 con 0x7f048806c4e0 2024-04-16T02:32:22.821 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (3m) 2m ago 5m 26.2M - 0.25.0 c8568f914cd2 9f7a3dfb7aa4 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (5m) 2m ago 5m 7079k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (4m) 66s ago 4m 6975k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (5m) 2m ago 5m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (4m) 66s ago 4m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (3m) 2m ago 4m 77.7M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:9283,8765,8443 running (6m) 2m ago 6m 482M - 18.2.1 5be31c24972a 9e98c5bc45da 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (4m) 66s ago 4m 442M - 18.2.1 5be31c24972a 6795a633edf9 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (6m) 2m ago 6m 46.5M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (4m) 66s ago 4m 40.6M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (5m) 2m ago 5m 16.0M - 1.5.0 0da6a335fe13 7748f5130613 2024-04-16T02:32:22.822 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (4m) 66s ago 4m 16.2M - 1.5.0 0da6a335fe13 6a289489fe28 2024-04-16T02:32:22.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (3m) 2m ago 3m 40.1M 3946M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:32:22.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (2m) 2m ago 2m 40.8M 3946M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:32:22.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (2m) 2m ago 2m 40.6M 3946M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:32:22.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (2m) 66s ago 2m 41.7M 4970M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:32:22.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (97s) 66s ago 97s 41.8M 4970M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:32:22.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (72s) 66s ago 71s 11.5M 4970M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:32:22.823 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (3m) 2m ago 4m 39.0M - 2.43.0 a07b618ecd1d 9a1c8cd0261d 2024-04-16T02:32:22.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f048806c4e0 msgr2=0x7f048806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:22.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f048806c4e0 0x7f048806e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f049000b5c0 tx=0x7f0490005e20 comp rx=0 tx=0).stop 2024-04-16T02:32:22.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c101710 msgr2=0x7f049c1963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:22.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c101710 0x7f049c1963f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f049800b4b0 tx=0x7f049800b870 comp rx=0 tx=0).stop 2024-04-16T02:32:22.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 shutdown_connections 2024-04-16T02:32:22.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f048806c4e0 0x7f048806e9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:22.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f049c101710 0x7f049c1963f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:22.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 --2- 172.21.15.151:0/3457157235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f049c104040 0x7f049c196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:22.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 >> 172.21.15.151:0/3457157235 conn(0x7f049c0fb080 msgr2=0x7f049c0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:22.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.822+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 shutdown_connections 2024-04-16T02:32:22.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:22.823+0000 7f04a4140700 1 -- 172.21.15.151:0/3457157235 wait complete. 2024-04-16T02:32:23.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-16T02:32:23.735 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:24.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:23 smithi151 ceph-mon[29003]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:24.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:23 smithi151 ceph-mon[29003]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:24.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:23 smithi153 ceph-mon[36747]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:23 smithi153 ceph-mon[36747]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:25.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.340+0000 7fd5caf61700 1 -- 172.21.15.151:0/3490823068 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c40ffcd0 msgr2=0x7fd5c4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:25.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.340+0000 7fd5caf61700 1 --2- 172.21.15.151:0/3490823068 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c40ffcd0 0x7fd5c4100150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009910 tx=0x7fd5b0009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:25.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.340+0000 7fd5caf61700 1 -- 172.21.15.151:0/3490823068 shutdown_connections 2024-04-16T02:32:25.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.340+0000 7fd5caf61700 1 --2- 172.21.15.151:0/3490823068 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c40ffcd0 0x7fd5c4100150 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.340+0000 7fd5caf61700 1 --2- 172.21.15.151:0/3490823068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5c4101070 0x7fd5c4101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.340+0000 7fd5caf61700 1 -- 172.21.15.151:0/3490823068 >> 172.21.15.151:0/3490823068 conn(0x7fd5c40fa7c0 msgr2=0x7fd5c40fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.341+0000 7fd5caf61700 1 -- 172.21.15.151:0/3490823068 shutdown_connections 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.341+0000 7fd5caf61700 1 -- 172.21.15.151:0/3490823068 wait complete. 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.341+0000 7fd5caf61700 1 Processor -- start 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.341+0000 7fd5caf61700 1 -- start start 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.341+0000 7fd5caf61700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5c40ffcd0 0x7fd5c40ff720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.341+0000 7fd5caf61700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c4101070 0x7fd5c419b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.341+0000 7fd5caf61700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c419b6e0 con 0x7fd5c4101070 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.341+0000 7fd5caf61700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c419b820 con 0x7fd5c40ffcd0 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.341+0000 7fd5c3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c4101070 0x7fd5c419b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:25.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5c3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c4101070 0x7fd5c419b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35194/0 (socket says 172.21.15.151:35194) 2024-04-16T02:32:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5c3fff700 1 -- 172.21.15.151:0/3769216962 learned_addr learned my addr 172.21.15.151:0/3769216962 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5c3fff700 1 -- 172.21.15.151:0/3769216962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5c40ffcd0 msgr2=0x7fd5c40ff720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5c3fff700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5c40ffcd0 0x7fd5c40ff720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5c3fff700 1 -- 172.21.15.151:0/3769216962 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b00095f0 con 0x7fd5c4101070 2024-04-16T02:32:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5c3fff700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c4101070 0x7fd5c419b110 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0006010 tx=0x7fd5b000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5c1ffb700 1 -- 172.21.15.151:0/3769216962 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5b001d070 con 0x7fd5c4101070 2024-04-16T02:32:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5c1ffb700 1 -- 172.21.15.151:0/3769216962 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd5b000f460 con 0x7fd5c4101070 2024-04-16T02:32:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c419baa0 con 0x7fd5c4101070 2024-04-16T02:32:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5c1ffb700 1 -- 172.21.15.151:0/3769216962 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5b0021640 con 0x7fd5c4101070 2024-04-16T02:32:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.342+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c419c010 con 0x7fd5c4101070 2024-04-16T02:32:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.343+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c4108690 con 0x7fd5c4101070 2024-04-16T02:32:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.344+0000 7fd5c1ffb700 1 -- 172.21.15.151:0/3769216962 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd5b0004b20 con 0x7fd5c4101070 2024-04-16T02:32:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.344+0000 7fd5c1ffb700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd5b406c4e0 0x7fd5b406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.344+0000 7fd5c1ffb700 1 -- 172.21.15.151:0/3769216962 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd5b008c970 con 0x7fd5c4101070 2024-04-16T02:32:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.344+0000 7fd5c8cfd700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd5b406c4e0 0x7fd5b406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.345+0000 7fd5c8cfd700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd5b406c4e0 0x7fd5b406e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80097b0 tx=0x7fd5b8006d20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:25.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.349+0000 7fd5c1ffb700 1 -- 172.21.15.151:0/3769216962 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd5b005af20 con 0x7fd5c4101070 2024-04-16T02:32:25.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.551+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5c4105630 con 0x7fd5b406c4e0 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.555+0000 7fd5c1ffb700 1 -- 172.21.15.151:0/3769216962 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1310 (secure 0 0 0) 0x7fd5c4105630 con 0x7fd5b406c4e0 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter 2/2 2m ago 6m * 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:crash 2/2 2m ago 6m * 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:mgr 2/2 2m ago 6m count:2 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:mon 2/2 2m ago 5m smithi151:172.21.15.151=smithi151;smithi153:172.21.15.153=smithi153;count:2 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd 6 2m ago - 2024-04-16T02:32:25.557 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-16T02:32:25.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.557+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd5b406c4e0 msgr2=0x7fd5b406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:25.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.557+0000 7fd5caf61700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd5b406c4e0 0x7fd5b406e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80097b0 tx=0x7fd5b8006d20 comp rx=0 tx=0).stop 2024-04-16T02:32:25.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.557+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c4101070 msgr2=0x7fd5c419b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:25.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.557+0000 7fd5caf61700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c4101070 0x7fd5c419b110 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0006010 tx=0x7fd5b000be20 comp rx=0 tx=0).stop 2024-04-16T02:32:25.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.557+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 shutdown_connections 2024-04-16T02:32:25.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.558+0000 7fd5caf61700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd5b406c4e0 0x7fd5b406e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:25.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.558+0000 7fd5caf61700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5c40ffcd0 0x7fd5c40ff720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:25.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.558+0000 7fd5caf61700 1 --2- 172.21.15.151:0/3769216962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5c4101070 0x7fd5c419b110 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.558+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 >> 172.21.15.151:0/3769216962 conn(0x7fd5c40fa7c0 msgr2=0x7fd5c4103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.558+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 shutdown_connections 2024-04-16T02:32:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:25.558+0000 7fd5caf61700 1 -- 172.21.15.151:0/3769216962 wait complete. 2024-04-16T02:32:25.859 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:25 smithi151 ceph-mon[29003]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:25 smithi153 ceph-mon[36747]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:26.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-16T02:32:26.411 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:26.822 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:26 smithi151 ceph-mon[29003]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:27.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:26 smithi153 ceph-mon[36747]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:27.959 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.956+0000 7f48c9b68700 1 -- 172.21.15.151:0/958822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4101dc0 msgr2=0x7f48c4102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:27.959 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.956+0000 7f48c9b68700 1 --2- 172.21.15.151:0/958822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4101dc0 0x7f48c4102220 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f48ac009910 tx=0x7f48ac009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.956+0000 7f48c9b68700 1 -- 172.21.15.151:0/958822 shutdown_connections 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.956+0000 7f48c9b68700 1 --2- 172.21.15.151:0/958822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4101dc0 0x7f48c4102220 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.956+0000 7f48c9b68700 1 --2- 172.21.15.151:0/958822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4100bc0 0x7f48c4100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.956+0000 7f48c9b68700 1 -- 172.21.15.151:0/958822 >> 172.21.15.151:0/958822 conn(0x7f48c40fc140 msgr2=0x7f48c40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.956+0000 7f48c9b68700 1 -- 172.21.15.151:0/958822 shutdown_connections 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.956+0000 7f48c9b68700 1 -- 172.21.15.151:0/958822 wait complete. 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c9b68700 1 Processor -- start 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c9b68700 1 -- start start 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c9b68700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4100bc0 0x7f48c4074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c9b68700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4101dc0 0x7f48c4073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c9b68700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c40736c0 con 0x7f48c4100bc0 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c9b68700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c4073800 con 0x7f48c4101dc0 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4100bc0 0x7f48c4074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4100bc0 0x7f48c4074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40178/0 (socket says 172.21.15.151:40178) 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c37fe700 1 -- 172.21.15.151:0/918614434 learned_addr learned my addr 172.21.15.151:0/918614434 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.957+0000 7f48c2ffd700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4101dc0 0x7f48c4073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.958+0000 7f48c37fe700 1 -- 172.21.15.151:0/918614434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4101dc0 msgr2=0x7f48c4073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.958+0000 7f48c37fe700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4101dc0 0x7f48c4073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.958+0000 7f48c37fe700 1 -- 172.21.15.151:0/918614434 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac0095f0 con 0x7f48c4100bc0 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.958+0000 7f48c37fe700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4100bc0 0x7f48c4074b30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f48b400d900 tx=0x7f48b400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.958+0000 7f48c0ff9700 1 -- 172.21.15.151:0/918614434 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48b4009900 con 0x7f48c4100bc0 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.958+0000 7f48c0ff9700 1 -- 172.21.15.151:0/918614434 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f48b4010460 con 0x7f48c4100bc0 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.958+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c4073ae0 con 0x7f48c4100bc0 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.958+0000 7f48c0ff9700 1 -- 172.21.15.151:0/918614434 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48b400f5b0 con 0x7f48c4100bc0 2024-04-16T02:32:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.958+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c419e3f0 con 0x7f48c4100bc0 2024-04-16T02:32:27.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.959+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c4109740 con 0x7f48c4100bc0 2024-04-16T02:32:27.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.959+0000 7f48c0ff9700 1 -- 172.21.15.151:0/918614434 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f48b40105d0 con 0x7f48c4100bc0 2024-04-16T02:32:27.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.960+0000 7f48c0ff9700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f48b006c4e0 0x7f48b006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:27.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.960+0000 7f48c0ff9700 1 -- 172.21.15.151:0/918614434 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f48b408af90 con 0x7f48c4100bc0 2024-04-16T02:32:27.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.960+0000 7f48c2ffd700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f48b006c4e0 0x7f48b006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:27.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.960+0000 7f48c2ffd700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f48b006c4e0 0x7f48b006e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f48ac006010 tx=0x7f48ac00b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:27.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:27.964+0000 7f48c0ff9700 1 -- 172.21.15.151:0/918614434 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f48b40595c0 con 0x7f48c4100bc0 2024-04-16T02:32:27.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:27 smithi151 ceph-mon[29003]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:28.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:27 smithi153 ceph-mon[36747]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:28.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.158+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f48c4106710 con 0x7f48b006c4e0 2024-04-16T02:32:28.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.160+0000 7f48c0ff9700 1 -- 172.21.15.151:0/918614434 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f48c4106710 con 0x7f48b006c4e0 2024-04-16T02:32:28.161 INFO:teuthology.orchestra.run.smithi151.stdout:HOST ADDR LABELS STATUS 2024-04-16T02:32:28.161 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151 172.21.15.151 2024-04-16T02:32:28.161 INFO:teuthology.orchestra.run.smithi151.stdout:smithi153 172.21.15.153 2024-04-16T02:32:28.161 INFO:teuthology.orchestra.run.smithi151.stdout:2 hosts in cluster 2024-04-16T02:32:28.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f48b006c4e0 msgr2=0x7f48b006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:28.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f48b006c4e0 0x7f48b006e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f48ac006010 tx=0x7f48ac00b540 comp rx=0 tx=0).stop 2024-04-16T02:32:28.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4100bc0 msgr2=0x7f48c4074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:28.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4100bc0 0x7f48c4074b30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f48b400d900 tx=0x7f48b400dc10 comp rx=0 tx=0).stop 2024-04-16T02:32:28.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 shutdown_connections 2024-04-16T02:32:28.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f48b006c4e0 0x7f48b006e9a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:28.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48c4100bc0 0x7f48c4074b30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:28.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 --2- 172.21.15.151:0/918614434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4101dc0 0x7f48c4073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:28.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 >> 172.21.15.151:0/918614434 conn(0x7f48c40fc140 msgr2=0x7f48c4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:28.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 shutdown_connections 2024-04-16T02:32:28.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:28.162+0000 7f48c9b68700 1 -- 172.21.15.151:0/918614434 wait complete. 2024-04-16T02:32:28.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-16T02:32:28.976 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:28 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:32:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:28 smithi151 ceph-mon[29003]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:29.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:28 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:32:29.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:28 smithi153 ceph-mon[36747]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:29 smithi153 ceph-mon[36747]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:30.030 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:29 smithi151 ceph-mon[29003]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:30.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.492+0000 7fed029e2700 1 -- 172.21.15.151:0/3882110869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 msgr2=0x7fecfc0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:30.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.492+0000 7fed029e2700 1 --2- 172.21.15.151:0/3882110869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 0x7fecfc0ffed0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fecec009910 tx=0x7fecec009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:30.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.492+0000 7fed029e2700 1 -- 172.21.15.151:0/3882110869 shutdown_connections 2024-04-16T02:32:30.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.492+0000 7fed029e2700 1 --2- 172.21.15.151:0/3882110869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecfc100cb0 0x7fecfc101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:30.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.492+0000 7fed029e2700 1 --2- 172.21.15.151:0/3882110869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 0x7fecfc0ffed0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:30.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.492+0000 7fed029e2700 1 -- 172.21.15.151:0/3882110869 >> 172.21.15.151:0/3882110869 conn(0x7fecfc0fb030 msgr2=0x7fecfc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:30.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.492+0000 7fed029e2700 1 -- 172.21.15.151:0/3882110869 shutdown_connections 2024-04-16T02:32:30.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.493+0000 7fed029e2700 1 -- 172.21.15.151:0/3882110869 wait complete. 2024-04-16T02:32:30.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.493+0000 7fed029e2700 1 Processor -- start 2024-04-16T02:32:30.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.493+0000 7fed029e2700 1 -- start start 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.493+0000 7fed029e2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 0x7fecfc1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.493+0000 7fed029e2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecfc100cb0 0x7fecfc196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.493+0000 7fed029e2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc196f50 con 0x7fecfc0ffab0 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.493+0000 7fed029e2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc197090 con 0x7fecfc100cb0 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecfbfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 0x7fecfc1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecfbfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 0x7fecfc1963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40190/0 (socket says 172.21.15.151:40190) 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecfbfff700 1 -- 172.21.15.151:0/3652374615 learned_addr learned my addr 172.21.15.151:0/3652374615 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecfbfff700 1 -- 172.21.15.151:0/3652374615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecfc100cb0 msgr2=0x7fecfc196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecfb7fe700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecfc100cb0 0x7fecfc196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecfbfff700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecfc100cb0 0x7fecfc196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecfbfff700 1 -- 172.21.15.151:0/3652374615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecec0095f0 con 0x7fecfc0ffab0 2024-04-16T02:32:30.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecfb7fe700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecfc100cb0 0x7fecfc196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:32:30.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecfbfff700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 0x7fecfc1963f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fecec00bda0 tx=0x7fecec00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:30.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecf97fa700 1 -- 172.21.15.151:0/3652374615 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecec01d070 con 0x7fecfc0ffab0 2024-04-16T02:32:30.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecf97fa700 1 -- 172.21.15.151:0/3652374615 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fecec004c50 con 0x7fecfc0ffab0 2024-04-16T02:32:30.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc19bae0 con 0x7fecfc0ffab0 2024-04-16T02:32:30.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fecf97fa700 1 -- 172.21.15.151:0/3652374615 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecec00f690 con 0x7fecfc0ffab0 2024-04-16T02:32:30.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.494+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc19c050 con 0x7fecfc0ffab0 2024-04-16T02:32:30.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.495+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecfc108690 con 0x7fecfc0ffab0 2024-04-16T02:32:30.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.496+0000 7fecf97fa700 1 -- 172.21.15.151:0/3652374615 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fecec022a20 con 0x7fecfc0ffab0 2024-04-16T02:32:30.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.496+0000 7fecf97fa700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fece406c4e0 0x7fece406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:30.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.496+0000 7fecfb7fe700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fece406c4e0 0x7fece406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:30.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.496+0000 7fecf97fa700 1 -- 172.21.15.151:0/3652374615 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fecec08cc00 con 0x7fecfc0ffab0 2024-04-16T02:32:30.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.497+0000 7fecfb7fe700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fece406c4e0 0x7fece406e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fecf0007900 tx=0x7fecf0008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:30.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.501+0000 7fecf97fa700 1 -- 172.21.15.151:0/3652374615 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fecec092020 con 0x7fecfc0ffab0 2024-04-16T02:32:30.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.698+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fecfc068b20 con 0x7fece406c4e0 2024-04-16T02:32:30.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.700+0000 7fecf97fa700 1 -- 172.21.15.151:0/3652374615 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fecfc068b20 con 0x7fece406c4e0 2024-04-16T02:32:30.701 INFO:teuthology.orchestra.run.smithi151.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-16T02:32:30.701 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-16T02:32:30.701 INFO:teuthology.orchestra.run.smithi151.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 71s ago Has a FileSystem, LVM detected 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fece406c4e0 msgr2=0x7fece406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fece406c4e0 0x7fece406e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fecf0007900 tx=0x7fecf0008040 comp rx=0 tx=0).stop 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 msgr2=0x7fecfc1963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 0x7fecfc1963f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fecec00bda0 tx=0x7fecec00be80 comp rx=0 tx=0).stop 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 shutdown_connections 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fece406c4e0 0x7fece406e9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fecfc0ffab0 0x7fecfc1963f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 --2- 172.21.15.151:0/3652374615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecfc100cb0 0x7fecfc196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 >> 172.21.15.151:0/3652374615 conn(0x7fecfc0fb030 msgr2=0x7fecfc068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 shutdown_connections 2024-04-16T02:32:30.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:30.702+0000 7fed029e2700 1 -- 172.21.15.151:0/3652374615 wait complete. 2024-04-16T02:32:31.308 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T02:32:31.317 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:32:31.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-16T02:32:31.525 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:31 smithi151 ceph-mon[29003]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:31 smithi151 ceph-mon[29003]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:32.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:31 smithi153 ceph-mon[36747]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:32.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:31 smithi153 ceph-mon[36747]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.219+0000 7fe36fc60700 1 -- 172.21.15.151:0/3062909120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 msgr2=0x7fe368100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.219+0000 7fe36fc60700 1 --2- 172.21.15.151:0/3062909120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 0x7fe368100fd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe36400b0a0 tx=0x7fe36400b3b0 comp rx=0 tx=0).stop 2024-04-16T02:32:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.219+0000 7fe36fc60700 1 -- 172.21.15.151:0/3062909120 shutdown_connections 2024-04-16T02:32:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.219+0000 7fe36fc60700 1 --2- 172.21.15.151:0/3062909120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe368101db0 0x7fe368102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.219+0000 7fe36fc60700 1 --2- 172.21.15.151:0/3062909120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 0x7fe368100fd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.219+0000 7fe36fc60700 1 -- 172.21.15.151:0/3062909120 >> 172.21.15.151:0/3062909120 conn(0x7fe3680fc110 msgr2=0x7fe3680fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.219+0000 7fe36fc60700 1 -- 172.21.15.151:0/3062909120 shutdown_connections 2024-04-16T02:32:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.219+0000 7fe36fc60700 1 -- 172.21.15.151:0/3062909120 wait complete. 2024-04-16T02:32:33.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36fc60700 1 Processor -- start 2024-04-16T02:32:33.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36fc60700 1 -- start start 2024-04-16T02:32:33.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36fc60700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 0x7fe368196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:33.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36fc60700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe368101db0 0x7fe3681969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:33.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36fc60700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe368196fd0 con 0x7fe368100bb0 2024-04-16T02:32:33.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36fc60700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe368197110 con 0x7fe368101db0 2024-04-16T02:32:33.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36d9fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 0x7fe368196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:33.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36d9fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 0x7fe368196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40206/0 (socket says 172.21.15.151:40206) 2024-04-16T02:32:33.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36d9fc700 1 -- 172.21.15.151:0/343698638 learned_addr learned my addr 172.21.15.151:0/343698638 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36d1fb700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe368101db0 0x7fe3681969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36d9fc700 1 -- 172.21.15.151:0/343698638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe368101db0 msgr2=0x7fe3681969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36d9fc700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe368101db0 0x7fe3681969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.220+0000 7fe36d9fc700 1 -- 172.21.15.151:0/343698638 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe364009d00 con 0x7fe368100bb0 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.221+0000 7fe36d9fc700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 0x7fe368196470 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe3640077b0 tx=0x7fe364003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.221+0000 7fe35effd700 1 -- 172.21.15.151:0/343698638 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe36400e040 con 0x7fe368100bb0 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.221+0000 7fe35effd700 1 -- 172.21.15.151:0/343698638 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe36400ce60 con 0x7fe368100bb0 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.221+0000 7fe35effd700 1 -- 172.21.15.151:0/343698638 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe36401db10 con 0x7fe368100bb0 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.221+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36819bb60 con 0x7fe368100bb0 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.221+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36819c050 con 0x7fe368100bb0 2024-04-16T02:32:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.222+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe368109790 con 0x7fe368100bb0 2024-04-16T02:32:33.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.222+0000 7fe35effd700 1 -- 172.21.15.151:0/343698638 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe36401dc90 con 0x7fe368100bb0 2024-04-16T02:32:33.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.223+0000 7fe35effd700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe35406c4e0 0x7fe35406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:33.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.223+0000 7fe35effd700 1 -- 172.21.15.151:0/343698638 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe36408cf10 con 0x7fe368100bb0 2024-04-16T02:32:33.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.223+0000 7fe36d1fb700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe35406c4e0 0x7fe35406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:33.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.223+0000 7fe36d1fb700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe35406c4e0 0x7fe35406e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe358006fd0 tx=0x7fe358008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:33.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.227+0000 7fe35effd700 1 -- 172.21.15.151:0/343698638 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe36405b540 con 0x7fe368100bb0 2024-04-16T02:32:33.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:33.441+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe368106700 con 0x7fe35406c4e0 2024-04-16T02:32:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:33 smithi151 ceph-mon[29003]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:33 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-16T02:32:34.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:33 smithi153 ceph-mon[36747]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:33 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-16T02:32:34.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.968+0000 7fe35effd700 1 -- 172.21.15.151:0/343698638 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe368106700 con 0x7fe35406c4e0 2024-04-16T02:32:34.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe35406c4e0 msgr2=0x7fe35406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe35406c4e0 0x7fe35406e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe358006fd0 tx=0x7fe358008040 comp rx=0 tx=0).stop 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 msgr2=0x7fe368196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 0x7fe368196470 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe3640077b0 tx=0x7fe364003ce0 comp rx=0 tx=0).stop 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 shutdown_connections 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fe35406c4e0 0x7fe35406e9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe368100bb0 0x7fe368196470 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 --2- 172.21.15.151:0/343698638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe368101db0 0x7fe3681969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 >> 172.21.15.151:0/343698638 conn(0x7fe3680fc110 msgr2=0x7fe368104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 shutdown_connections 2024-04-16T02:32:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:34.971+0000 7fe36fc60700 1 -- 172.21.15.151:0/343698638 wait complete. 2024-04-16T02:32:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:34 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[28999]: 2024-04-16T02:32:34.940+0000 7ffbca5da700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T02:32:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:34 smithi151 ceph-mon[29003]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:34 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-16T02:32:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:34 smithi151 ceph-mon[29003]: osdmap e42: 6 total, 6 up, 6 in 2024-04-16T02:32:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:34 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-16T02:32:35.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:34 smithi153 ceph-mon[36747]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:32:35.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:34 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-16T02:32:35.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:34 smithi153 ceph-mon[36747]: osdmap e42: 6 total, 6 up, 6 in 2024-04-16T02:32:35.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:34 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-16T02:32:35.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T02:32:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: pgmap v176: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-16T02:32:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: osdmap e43: 6 total, 6 up, 6 in 2024-04-16T02:32:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-16T02:32:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T02:32:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: osdmap e44: 6 total, 6 up, 6 in 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: fsmap cephfs:0 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: Saving service mds.cephfs spec with placement count:4 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.bdpnxq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.bdpnxq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: Deploying daemon mds.cephfs.smithi151.bdpnxq on smithi151 2024-04-16T02:32:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:35 smithi151 ceph-mon[29003]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: pgmap v176: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: osdmap e43: 6 total, 6 up, 6 in 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: osdmap e44: 6 total, 6 up, 6 in 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: fsmap cephfs:0 2024-04-16T02:32:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: Saving service mds.cephfs spec with placement count:4 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.bdpnxq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.bdpnxq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: Deploying daemon mds.cephfs.smithi151.bdpnxq on smithi151 2024-04-16T02:32:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:35 smithi153 ceph-mon[36747]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-16T02:32:36.889 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:37.133 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:36 smithi151 ceph-mon[29003]: osdmap e45: 6 total, 6 up, 6 in 2024-04-16T02:32:37.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:36 smithi153 ceph-mon[36747]: osdmap e45: 6 total, 6 up, 6 in 2024-04-16T02:32:38.175 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:37 smithi151 ceph-mon[29003]: pgmap v180: 65 pgs: 6 creating+peering, 25 active+clean, 34 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:38.175 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:37 smithi151 ceph-mon[29003]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-16T02:32:38.175 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:37 smithi151 ceph-mon[29003]: osdmap e46: 6 total, 6 up, 6 in 2024-04-16T02:32:38.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:37 smithi153 ceph-mon[36747]: pgmap v180: 65 pgs: 6 creating+peering, 25 active+clean, 34 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:38.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:37 smithi153 ceph-mon[36747]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-16T02:32:38.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:37 smithi153 ceph-mon[36747]: osdmap e46: 6 total, 6 up, 6 in 2024-04-16T02:32:38.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.663+0000 7fdb4f04c700 1 -- 172.21.15.151:0/2544878624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 msgr2=0x7fdb48100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:38.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.663+0000 7fdb4f04c700 1 --2- 172.21.15.151:0/2544878624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 0x7fdb48100fe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009910 tx=0x7fdb3c009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:38.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.663+0000 7fdb4f04c700 1 -- 172.21.15.151:0/2544878624 shutdown_connections 2024-04-16T02:32:38.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.663+0000 7fdb4f04c700 1 --2- 172.21.15.151:0/2544878624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb48101dc0 0x7fdb48102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:38.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.663+0000 7fdb4f04c700 1 --2- 172.21.15.151:0/2544878624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 0x7fdb48100fe0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:38.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.663+0000 7fdb4f04c700 1 -- 172.21.15.151:0/2544878624 >> 172.21.15.151:0/2544878624 conn(0x7fdb480fc140 msgr2=0x7fdb480fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:38.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.663+0000 7fdb4f04c700 1 -- 172.21.15.151:0/2544878624 shutdown_connections 2024-04-16T02:32:38.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.663+0000 7fdb4f04c700 1 -- 172.21.15.151:0/2544878624 wait complete. 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb4f04c700 1 Processor -- start 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb4f04c700 1 -- start start 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb4f04c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 0x7fdb48074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb4f04c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb48101dc0 0x7fdb48073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb4f04c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480736c0 con 0x7fdb48100bc0 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb4f04c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48073800 con 0x7fdb48101dc0 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb4cde8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 0x7fdb48074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb4cde8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 0x7fdb48074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45050/0 (socket says 172.21.15.151:45050) 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb4cde8700 1 -- 172.21.15.151:0/790181333 learned_addr learned my addr 172.21.15.151:0/790181333 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.664+0000 7fdb47fff700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb48101dc0 0x7fdb48073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.665+0000 7fdb4cde8700 1 -- 172.21.15.151:0/790181333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb48101dc0 msgr2=0x7fdb48073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.665+0000 7fdb4cde8700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb48101dc0 0x7fdb48073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.665+0000 7fdb4cde8700 1 -- 172.21.15.151:0/790181333 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0095f0 con 0x7fdb48100bc0 2024-04-16T02:32:38.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.665+0000 7fdb4cde8700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 0x7fdb48074b30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00bda0 tx=0x7fdb3c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:38.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.665+0000 7fdb45ffb700 1 -- 172.21.15.151:0/790181333 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3c01d070 con 0x7fdb48100bc0 2024-04-16T02:32:38.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.665+0000 7fdb45ffb700 1 -- 172.21.15.151:0/790181333 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdb3c004c50 con 0x7fdb48100bc0 2024-04-16T02:32:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.665+0000 7fdb45ffb700 1 -- 172.21.15.151:0/790181333 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3c00f690 con 0x7fdb48100bc0 2024-04-16T02:32:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.665+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb48073a80 con 0x7fdb48100bc0 2024-04-16T02:32:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.665+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb48073f70 con 0x7fdb48100bc0 2024-04-16T02:32:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.666+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb48109720 con 0x7fdb48100bc0 2024-04-16T02:32:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.667+0000 7fdb45ffb700 1 -- 172.21.15.151:0/790181333 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fdb3c022a20 con 0x7fdb48100bc0 2024-04-16T02:32:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.667+0000 7fdb45ffb700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fdb3806c4e0 0x7fdb3806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.667+0000 7fdb45ffb700 1 -- 172.21.15.151:0/790181333 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fdb3c08ce10 con 0x7fdb48100bc0 2024-04-16T02:32:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.667+0000 7fdb47fff700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fdb3806c4e0 0x7fdb3806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.667+0000 7fdb47fff700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fdb3806c4e0 0x7fdb3806e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdb34005ea0 tx=0x7fdb34005e30 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:38.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.672+0000 7fdb45ffb700 1 -- 172.21.15.151:0/790181333 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdb3c05b2d0 con 0x7fdb48100bc0 2024-04-16T02:32:38.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.915+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdb4819e870 con 0x7fdb48100bc0 2024-04-16T02:32:38.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.915+0000 7fdb45ffb700 1 -- 172.21.15.151:0/790181333 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7fdb3c027460 con 0x7fdb48100bc0 2024-04-16T02:32:38.917 INFO:teuthology.orchestra.run.smithi151.stdout:e2 2024-04-16T02:32:38.917 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:32:38.917 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 2 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:32:34.941449+0000 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:32:38.918 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 0 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 1 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:in 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 0 2024-04-16T02:32:38.919 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:32:38.920 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:32:38.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.917+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fdb3806c4e0 msgr2=0x7fdb3806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:38.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fdb3806c4e0 0x7fdb3806e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdb34005ea0 tx=0x7fdb34005e30 comp rx=0 tx=0).stop 2024-04-16T02:32:38.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 msgr2=0x7fdb48074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:38.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 0x7fdb48074b30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00bda0 tx=0x7fdb3c00be80 comp rx=0 tx=0).stop 2024-04-16T02:32:38.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 shutdown_connections 2024-04-16T02:32:38.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fdb3806c4e0 0x7fdb3806e9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:38.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb48100bc0 0x7fdb48074b30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:38.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 --2- 172.21.15.151:0/790181333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb48101dc0 0x7fdb48073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:38.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 >> 172.21.15.151:0/790181333 conn(0x7fdb480fc140 msgr2=0x7fdb48104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:38.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 shutdown_connections 2024-04-16T02:32:38.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:38.918+0000 7fdb4f04c700 1 -- 172.21.15.151:0/790181333 wait complete. 2024-04-16T02:32:38.921 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 2 2024-04-16T02:32:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:38 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/790181333' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:32:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:38 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/790181333' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:32:39.800 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T02:32:39.810 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:32:39.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-16T02:32:40.024 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:40.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: pgmap v182: 65 pgs: 6 creating+peering, 59 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:40.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:40.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:40.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ogdcjk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ogdcjk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: daemon mds.cephfs.smithi151.bdpnxq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: Cluster is now healthy 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] up:boot 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: fsmap cephfs:1 {0=cephfs.smithi151.bdpnxq=up:creating} 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:32:40.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:39 smithi151 ceph-mon[29003]: daemon mds.cephfs.smithi151.bdpnxq is now active in filesystem cephfs as rank 0 2024-04-16T02:32:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: pgmap v182: 65 pgs: 6 creating+peering, 59 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:32:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ogdcjk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ogdcjk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: daemon mds.cephfs.smithi151.bdpnxq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: Cluster is now healthy 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] up:boot 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: fsmap cephfs:1 {0=cephfs.smithi151.bdpnxq=up:creating} 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:32:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:39 smithi153 ceph-mon[36747]: daemon mds.cephfs.smithi151.bdpnxq is now active in filesystem cephfs as rank 0 2024-04-16T02:32:41.127 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:40 smithi151 ceph-mon[29003]: Deploying daemon mds.cephfs.smithi153.ogdcjk on smithi153 2024-04-16T02:32:41.127 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:40 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] up:active 2024-04-16T02:32:41.127 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:40 smithi151 ceph-mon[29003]: fsmap cephfs:1 {0=cephfs.smithi151.bdpnxq=up:active} 2024-04-16T02:32:41.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:40 smithi153 ceph-mon[36747]: Deploying daemon mds.cephfs.smithi153.ogdcjk on smithi153 2024-04-16T02:32:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:40 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] up:active 2024-04-16T02:32:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:40 smithi153 ceph-mon[36747]: fsmap cephfs:1 {0=cephfs.smithi151.bdpnxq=up:active} 2024-04-16T02:32:41.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.659+0000 7f3e6e103700 1 -- 172.21.15.151:0/4152630996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e680ffad0 msgr2=0x7f3e680ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:41.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.659+0000 7f3e6e103700 1 --2- 172.21.15.151:0/4152630996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e680ffad0 0x7f3e680ffef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009910 tx=0x7f3e50009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:41.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.659+0000 7f3e6e103700 1 -- 172.21.15.151:0/4152630996 shutdown_connections 2024-04-16T02:32:41.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.659+0000 7f3e6e103700 1 --2- 172.21.15.151:0/4152630996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e68100cd0 0x7f3e68101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:41.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.659+0000 7f3e6e103700 1 --2- 172.21.15.151:0/4152630996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e680ffad0 0x7f3e680ffef0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:41.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.659+0000 7f3e6e103700 1 -- 172.21.15.151:0/4152630996 >> 172.21.15.151:0/4152630996 conn(0x7f3e680fb050 msgr2=0x7f3e680fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.659+0000 7f3e6e103700 1 -- 172.21.15.151:0/4152630996 shutdown_connections 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.659+0000 7f3e6e103700 1 -- 172.21.15.151:0/4152630996 wait complete. 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.660+0000 7f3e6e103700 1 Processor -- start 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.660+0000 7f3e6e103700 1 -- start start 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.660+0000 7f3e6e103700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e680ffad0 0x7f3e68069910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.660+0000 7f3e6e103700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e68100cd0 0x7f3e6819b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.660+0000 7f3e6e103700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6819b730 con 0x7f3e68100cd0 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.660+0000 7f3e6e103700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6819b8a0 con 0x7f3e680ffad0 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e66ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e68100cd0 0x7f3e6819b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e66ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e68100cd0 0x7f3e6819b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45072/0 (socket says 172.21.15.151:45072) 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e66ffd700 1 -- 172.21.15.151:0/1432887956 learned_addr learned my addr 172.21.15.151:0/1432887956 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e677fe700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e680ffad0 0x7f3e68069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e66ffd700 1 -- 172.21.15.151:0/1432887956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e680ffad0 msgr2=0x7f3e68069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e66ffd700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e680ffad0 0x7f3e68069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:41.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e66ffd700 1 -- 172.21.15.151:0/1432887956 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e500095f0 con 0x7f3e68100cd0 2024-04-16T02:32:41.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e66ffd700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e68100cd0 0x7f3e6819b160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800b4b0 tx=0x7f3e5800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:41.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e64ff9700 1 -- 172.21.15.151:0/1432887956 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e58011800 con 0x7f3e68100cd0 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e64ff9700 1 -- 172.21.15.151:0/1432887956 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3e58011e10 con 0x7f3e68100cd0 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6819bb80 con 0x7f3e68100cd0 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e64ff9700 1 -- 172.21.15.151:0/1432887956 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e5800f2e0 con 0x7f3e68100cd0 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.661+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6819c150 con 0x7f3e68100cd0 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.662+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e681086b0 con 0x7f3e68100cd0 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.663+0000 7f3e64ff9700 1 -- 172.21.15.151:0/1432887956 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3e58010380 con 0x7f3e68100cd0 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.663+0000 7f3e64ff9700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3e5406c4e0 0x7f3e5406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.663+0000 7f3e64ff9700 1 -- 172.21.15.151:0/1432887956 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f3e5808b3c0 con 0x7f3e68100cd0 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.663+0000 7f3e677fe700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3e5406c4e0 0x7f3e5406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.664+0000 7f3e677fe700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3e5406c4e0 0x7f3e5406e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e50006010 tx=0x7f3e5000b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:41.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.668+0000 7f3e64ff9700 1 -- 172.21.15.151:0/1432887956 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3e58059630 con 0x7f3e68100cd0 2024-04-16T02:32:41.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:41.903+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f3e6819c7c0 con 0x7f3e68100cd0 2024-04-16T02:32:42.018 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:41 smithi153 ceph-mon[36747]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-16T02:32:42.018 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:41 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1432887956' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-16T02:32:42.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:41 smithi151 ceph-mon[29003]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-16T02:32:42.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:41 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1432887956' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-16T02:32:42.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.746+0000 7f3e64ff9700 1 -- 172.21.15.151:0/1432887956 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v5) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3e580591c0 con 0x7f3e68100cd0 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3e5406c4e0 msgr2=0x7f3e5406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3e5406c4e0 0x7f3e5406e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e50006010 tx=0x7f3e5000b540 comp rx=0 tx=0).stop 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e68100cd0 msgr2=0x7f3e6819b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e68100cd0 0x7f3e6819b160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800b4b0 tx=0x7f3e5800b7c0 comp rx=0 tx=0).stop 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 shutdown_connections 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f3e5406c4e0 0x7f3e5406e9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e680ffad0 0x7f3e68069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 --2- 172.21.15.151:0/1432887956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e68100cd0 0x7f3e6819b160 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 >> 172.21.15.151:0/1432887956 conn(0x7f3e680fb050 msgr2=0x7f3e68103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 shutdown_connections 2024-04-16T02:32:42.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:42.749+0000 7f3e6e103700 1 -- 172.21.15.151:0/1432887956 wait complete. 2024-04-16T02:32:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.slrqog", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.slrqog", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: Deploying daemon mds.cephfs.smithi151.slrqog on smithi151 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1432887956' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.153:6824/1319024054,v1:172.21.15.153:6825/1319024054] up:boot 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: daemon mds.cephfs.smithi153.ogdcjk assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: fsmap cephfs:1 {0=cephfs.smithi151.bdpnxq=up:active} 1 up:standby 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:32:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:creating} 2024-04-16T02:32:43.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: daemon mds.cephfs.smithi153.ogdcjk is now active in filesystem cephfs as rank 1 2024-04-16T02:32:43.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:43.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:32:43.381 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.slrqog", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.slrqog", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: Deploying daemon mds.cephfs.smithi151.slrqog on smithi151 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1432887956' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.153:6824/1319024054,v1:172.21.15.153:6825/1319024054] up:boot 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: daemon mds.cephfs.smithi153.ogdcjk assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: fsmap cephfs:1 {0=cephfs.smithi151.bdpnxq=up:active} 1 up:standby 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:32:43.382 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:creating} 2024-04-16T02:32:43.383 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: daemon mds.cephfs.smithi153.ogdcjk is now active in filesystem cephfs as rank 1 2024-04-16T02:32:43.383 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:43.383 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:32:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:44 smithi151 ceph-mon[29003]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s wr, 6 op/s 2024-04-16T02:32:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:44 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.153:6824/1319024054,v1:172.21.15.153:6825/1319024054] up:active 2024-04-16T02:32:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:44 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 2024-04-16T02:32:44.305 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T02:32:44.315 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:32:44.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-16T02:32:44.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:44 smithi153 ceph-mon[36747]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s wr, 6 op/s 2024-04-16T02:32:44.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:44 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.153:6824/1319024054,v1:172.21.15.153:6825/1319024054] up:active 2024-04-16T02:32:44.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:44 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 2024-04-16T02:32:44.909 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s wr, 6 op/s 2024-04-16T02:32:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.mqbgav", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.mqbgav", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: Deploying daemon mds.cephfs.smithi153.mqbgav on smithi153 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: Cluster is now healthy 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:boot 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 1 up:standby 2024-04-16T02:32:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:45 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:32:46.162 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s wr, 6 op/s 2024-04-16T02:32:46.162 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.mqbgav", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.mqbgav", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: Deploying daemon mds.cephfs.smithi153.mqbgav on smithi153 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: Cluster is now healthy 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:boot 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 1 up:standby 2024-04-16T02:32:46.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:45 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:32:46.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.669+0000 7f193ac19700 1 -- 172.21.15.151:0/803795432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 msgr2=0x7f19341063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:46.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.669+0000 7f193ac19700 1 --2- 172.21.15.151:0/803795432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 0x7f19341063e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1924009910 tx=0x7f1924009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:46.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.669+0000 7f193ac19700 1 -- 172.21.15.151:0/803795432 shutdown_connections 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.669+0000 7f193ac19700 1 --2- 172.21.15.151:0/803795432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 0x7f19341063e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.669+0000 7f193ac19700 1 --2- 172.21.15.151:0/803795432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19341016c0 0x7f1934103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.669+0000 7f193ac19700 1 -- 172.21.15.151:0/803795432 >> 172.21.15.151:0/803795432 conn(0x7f19340fb030 msgr2=0x7f19340fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.669+0000 7f193ac19700 1 -- 172.21.15.151:0/803795432 shutdown_connections 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.669+0000 7f193ac19700 1 -- 172.21.15.151:0/803795432 wait complete. 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f193ac19700 1 Processor -- start 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f193ac19700 1 -- start start 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f193ac19700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19341016c0 0x7f1934101020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f193ac19700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 0x7f19340ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f193ac19700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340ffbb0 con 0x7f1934103ff0 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f193ac19700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340ffcf0 con 0x7f19341016c0 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f192bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 0x7f19340ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f19389b5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19341016c0 0x7f1934101020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f192bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 0x7f19340ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45202/0 (socket says 172.21.15.151:45202) 2024-04-16T02:32:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f192bfff700 1 -- 172.21.15.151:0/3954857402 learned_addr learned my addr 172.21.15.151:0/3954857402 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:46.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f192bfff700 1 -- 172.21.15.151:0/3954857402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19341016c0 msgr2=0x7f1934101020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:46.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f192bfff700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19341016c0 0x7f1934101020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:46.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f192bfff700 1 -- 172.21.15.151:0/3954857402 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19240095f0 con 0x7f1934103ff0 2024-04-16T02:32:46.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f192bfff700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 0x7f19340ff670 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1924000c00 tx=0x7f192400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:46.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.670+0000 7f1929ffb700 1 -- 172.21.15.151:0/3954857402 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f192401d070 con 0x7f1934103ff0 2024-04-16T02:32:46.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.671+0000 7f1929ffb700 1 -- 172.21.15.151:0/3954857402 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1924005dc0 con 0x7f1934103ff0 2024-04-16T02:32:46.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.671+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19340fff70 con 0x7f1934103ff0 2024-04-16T02:32:46.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.671+0000 7f1929ffb700 1 -- 172.21.15.151:0/3954857402 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19240226a0 con 0x7f1934103ff0 2024-04-16T02:32:46.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.671+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1934100460 con 0x7f1934103ff0 2024-04-16T02:32:46.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.672+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1934190520 con 0x7f1934103ff0 2024-04-16T02:32:46.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.672+0000 7f1929ffb700 1 -- 172.21.15.151:0/3954857402 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f19240038f0 con 0x7f1934103ff0 2024-04-16T02:32:46.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.673+0000 7f1929ffb700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f191c06c4e0 0x7f191c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:46.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.673+0000 7f19389b5700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f191c06c4e0 0x7f191c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:46.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.673+0000 7f1929ffb700 1 -- 172.21.15.151:0/3954857402 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f192408cd30 con 0x7f1934103ff0 2024-04-16T02:32:46.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.673+0000 7f19389b5700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f191c06c4e0 0x7f191c06e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1930009c60 tx=0x7f1930009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:46.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.677+0000 7f1929ffb700 1 -- 172.21.15.151:0/3954857402 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f192405afa0 con 0x7f1934103ff0 2024-04-16T02:32:46.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:46.909+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f193402cc70 con 0x7f1934103ff0 2024-04-16T02:32:47.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.915+0000 7f1929ffb700 1 -- 172.21.15.151:0/3954857402 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7f1924003680 con 0x7f1934103ff0 2024-04-16T02:32:47.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f191c06c4e0 msgr2=0x7f191c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f191c06c4e0 0x7f191c06e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1930009c60 tx=0x7f1930009260 comp rx=0 tx=0).stop 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 msgr2=0x7f19340ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 0x7f19340ff670 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1924000c00 tx=0x7f192400f690 comp rx=0 tx=0).stop 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 shutdown_connections 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f191c06c4e0 0x7f191c06e9a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19341016c0 0x7f1934101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 --2- 172.21.15.151:0/3954857402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1934103ff0 0x7f19340ff670 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 >> 172.21.15.151:0/3954857402 conn(0x7f19340fb030 msgr2=0x7f19340fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 shutdown_connections 2024-04-16T02:32:47.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:47.919+0000 7f193ac19700 1 -- 172.21.15.151:0/3954857402 wait complete. 2024-04-16T02:32:48.227 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:47 smithi151 ceph-mon[29003]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 7 op/s 2024-04-16T02:32:48.228 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:47 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3954857402' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-16T02:32:48.228 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:47 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.228 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:47 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.228 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:47 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.228 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:47 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.228 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:47 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.228 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:47 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:32:48.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:47 smithi153 ceph-mon[36747]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 7 op/s 2024-04-16T02:32:48.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:47 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3954857402' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-16T02:32:48.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:47 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:47 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:47 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:47 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:47 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:48.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:47 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:32:48.490 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T02:32:48.502 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:32:48.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data false' 2024-04-16T02:32:49.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:48 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3954857402' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-16T02:32:49.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:48 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] up:boot 2024-04-16T02:32:49.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:48 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] up:active 2024-04-16T02:32:49.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:48 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 2 up:standby 2024-04-16T02:32:49.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:48 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:32:49.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:48 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:49.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:48 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 2 up:standby-replay 2024-04-16T02:32:49.222 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:48 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3954857402' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-16T02:32:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:48 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] up:boot 2024-04-16T02:32:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:48 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] up:active 2024-04-16T02:32:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:48 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 2 up:standby 2024-04-16T02:32:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:48 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:32:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:48 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:48 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 2 up:standby-replay 2024-04-16T02:32:49.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 257 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-16T02:32:49.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:standby-replay 2024-04-16T02:32:49.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: Dropping low affinity active daemon mds.cephfs.smithi153.ogdcjk in favor of higher affinity standby. 2024-04-16T02:32:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: Replacing daemon mds.cephfs.smithi153.ogdcjk as rank 1 with standby daemon mds.cephfs.smithi151.slrqog 2024-04-16T02:32:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T02:32:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 2 up:standby-replay 2024-04-16T02:32:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T02:32:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:replay} 1 up:standby-replay 2024-04-16T02:32:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:49 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 257 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-16T02:32:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:standby-replay 2024-04-16T02:32:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: Dropping low affinity active daemon mds.cephfs.smithi153.ogdcjk in favor of higher affinity standby. 2024-04-16T02:32:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: Replacing daemon mds.cephfs.smithi153.ogdcjk as rank 1 with standby daemon mds.cephfs.smithi151.slrqog 2024-04-16T02:32:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T02:32:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi153.ogdcjk=up:active} 2 up:standby-replay 2024-04-16T02:32:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T02:32:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:replay} 1 up:standby-replay 2024-04-16T02:32:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:49 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 -- 172.21.15.151:0/1863446228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc0feec0 msgr2=0x7f2ddc0ff2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 --2- 172.21.15.151:0/1863446228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc0feec0 0x7f2ddc0ff2e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00b0a0 tx=0x7f2dcc00b3b0 comp rx=0 tx=0).stop 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 -- 172.21.15.151:0/1863446228 shutdown_connections 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 --2- 172.21.15.151:0/1863446228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ddc101ea0 0x7f2ddc102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 --2- 172.21.15.151:0/1863446228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc0feec0 0x7f2ddc0ff2e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 -- 172.21.15.151:0/1863446228 >> 172.21.15.151:0/1863446228 conn(0x7f2ddc0fa800 msgr2=0x7f2ddc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 -- 172.21.15.151:0/1863446228 shutdown_connections 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 -- 172.21.15.151:0/1863446228 wait complete. 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 Processor -- start 2024-04-16T02:32:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.828+0000 7f2de1bb7700 1 -- start start 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2de1bb7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ddc0feec0 0x7f2ddc101800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2de1bb7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc101ea0 0x7f2ddc0ffe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2de1bb7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc100390 con 0x7f2ddc101ea0 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2de1bb7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc1004d0 con 0x7f2ddc0feec0 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2ddaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc101ea0 0x7f2ddc0ffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2ddaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc101ea0 0x7f2ddc0ffe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45220/0 (socket says 172.21.15.151:45220) 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2ddaffd700 1 -- 172.21.15.151:0/2125683917 learned_addr learned my addr 172.21.15.151:0/2125683917 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2ddb7fe700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ddc0feec0 0x7f2ddc101800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2ddaffd700 1 -- 172.21.15.151:0/2125683917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ddc0feec0 msgr2=0x7f2ddc101800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2ddaffd700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ddc0feec0 0x7f2ddc101800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2ddaffd700 1 -- 172.21.15.151:0/2125683917 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc009d00 con 0x7f2ddc101ea0 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2ddaffd700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc101ea0 0x7f2ddc0ffe50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000d8d0 tx=0x7f2dd000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:50.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2dd8ff9700 1 -- 172.21.15.151:0/2125683917 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dd0009940 con 0x7f2ddc101ea0 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc1007b0 con 0x7f2ddc101ea0 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2dd8ff9700 1 -- 172.21.15.151:0/2125683917 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2dd0010460 con 0x7f2ddc101ea0 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2dd8ff9700 1 -- 172.21.15.151:0/2125683917 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dd000f5f0 con 0x7f2ddc101ea0 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.829+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc19e3e0 con 0x7f2ddc101ea0 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.830+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc109760 con 0x7f2ddc101ea0 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.831+0000 7f2dd8ff9700 1 -- 172.21.15.151:0/2125683917 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2dd00105d0 con 0x7f2ddc101ea0 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.831+0000 7f2dd8ff9700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2dc406c4e0 0x7f2dc406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.831+0000 7f2dd8ff9700 1 -- 172.21.15.151:0/2125683917 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2dd008b890 con 0x7f2ddc101ea0 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.831+0000 7f2ddb7fe700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2dc406c4e0 0x7f2dc406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:50.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.832+0000 7f2ddb7fe700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2dc406c4e0 0x7f2dc406e9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009480 tx=0x7f2dcc000f40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:50.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:50.836+0000 7f2dd8ff9700 1 -- 172.21.15.151:0/2125683917 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2dd0059aa0 con 0x7f2ddc101ea0 2024-04-16T02:32:51.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.089+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f2ddc19e990 con 0x7f2ddc101ea0 2024-04-16T02:32:51.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:50 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] up:boot 2024-04-16T02:32:51.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:50 smithi151 ceph-mon[29003]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:replay} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:51.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:32:51.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:50 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] up:boot 2024-04-16T02:32:51.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:50 smithi153 ceph-mon[36747]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:replay} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:51.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:50 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:32:51.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.941+0000 7f2dd8ff9700 1 -- 172.21.15.151:0/2125683917 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v15) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2dd0059630 con 0x7f2ddc101ea0 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2dc406c4e0 msgr2=0x7f2dc406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2dc406c4e0 0x7f2dc406e9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009480 tx=0x7f2dcc000f40 comp rx=0 tx=0).stop 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc101ea0 msgr2=0x7f2ddc0ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc101ea0 0x7f2ddc0ffe50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000d8d0 tx=0x7f2dd000dc90 comp rx=0 tx=0).stop 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 shutdown_connections 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f2dc406c4e0 0x7f2dc406e9a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ddc0feec0 0x7f2ddc101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 --2- 172.21.15.151:0/2125683917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ddc101ea0 0x7f2ddc0ffe50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 >> 172.21.15.151:0/2125683917 conn(0x7f2ddc0fa800 msgr2=0x7f2ddc1050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 shutdown_connections 2024-04-16T02:32:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:51.943+0000 7f2de1bb7700 1 -- 172.21.15.151:0/2125683917 wait complete. 2024-04-16T02:32:51.946 INFO:teuthology.orchestra.run.smithi151.stderr:inline data disabled 2024-04-16T02:32:52.205 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:51 smithi151 ceph-mon[29003]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-16T02:32:52.205 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:51 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:resolve 2024-04-16T02:32:52.205 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:51 smithi151 ceph-mon[29003]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:resolve} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:52.206 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:51 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2125683917' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-16T02:32:52.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:51 smithi153 ceph-mon[36747]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-16T02:32:52.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:51 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:resolve 2024-04-16T02:32:52.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:51 smithi153 ceph-mon[36747]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:resolve} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:52.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:51 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2125683917' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-16T02:32:52.689 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T02:32:52.699 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:32:52.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T02:32:53.116 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:53.199 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:52 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:reconnect 2024-04-16T02:32:53.199 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:52 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2125683917' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-16T02:32:53.199 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:52 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] up:standby-replay 2024-04-16T02:32:53.199 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:52 smithi151 ceph-mon[29003]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:reconnect} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:52 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:reconnect 2024-04-16T02:32:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:52 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2125683917' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-16T02:32:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:52 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] up:standby-replay 2024-04-16T02:32:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:52 smithi153 ceph-mon[36747]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:reconnect} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:53 smithi151 ceph-mon[29003]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 819 B/s wr, 6 op/s 2024-04-16T02:32:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:53 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:rejoin 2024-04-16T02:32:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:53 smithi151 ceph-mon[29003]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:rejoin} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:53 smithi151 ceph-mon[29003]: daemon mds.cephfs.smithi151.slrqog is now active in filesystem cephfs as rank 1 2024-04-16T02:32:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:53 smithi153 ceph-mon[36747]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 819 B/s wr, 6 op/s 2024-04-16T02:32:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:53 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:rejoin 2024-04-16T02:32:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:53 smithi153 ceph-mon[36747]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:rejoin} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:53 smithi153 ceph-mon[36747]: daemon mds.cephfs.smithi151.slrqog is now active in filesystem cephfs as rank 1 2024-04-16T02:32:55.179 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:54 smithi151 ceph-mon[29003]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T02:32:55.179 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:54 smithi151 ceph-mon[29003]: Cluster is now healthy 2024-04-16T02:32:55.179 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:54 smithi151 ceph-mon[29003]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:active 2024-04-16T02:32:55.179 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:54 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:55.179 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:54 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:32:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:54 smithi153 ceph-mon[36747]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T02:32:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:54 smithi153 ceph-mon[36747]: Cluster is now healthy 2024-04-16T02:32:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:54 smithi153 ceph-mon[36747]: mds.? [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] up:active 2024-04-16T02:32:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:54 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 1 up:standby-replay 1 up:standby 2024-04-16T02:32:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:54 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:32:55.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.569+0000 7f8f6d857700 1 -- 172.21.15.151:0/682716742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 msgr2=0x7f8f68101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:55.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.569+0000 7f8f6d857700 1 --2- 172.21.15.151:0/682716742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 0x7f8f68101000 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8f5800b0a0 tx=0x7f8f5800b3b0 comp rx=0 tx=0).stop 2024-04-16T02:32:55.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.569+0000 7f8f6d857700 1 -- 172.21.15.151:0/682716742 shutdown_connections 2024-04-16T02:32:55.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.569+0000 7f8f6d857700 1 --2- 172.21.15.151:0/682716742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68101de0 0x7f8f68102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:55.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.569+0000 7f8f6d857700 1 --2- 172.21.15.151:0/682716742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 0x7f8f68101000 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:55.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.569+0000 7f8f6d857700 1 -- 172.21.15.151:0/682716742 >> 172.21.15.151:0/682716742 conn(0x7f8f680fc140 msgr2=0x7f8f680fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:55.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.569+0000 7f8f6d857700 1 -- 172.21.15.151:0/682716742 shutdown_connections 2024-04-16T02:32:55.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.569+0000 7f8f6d857700 1 -- 172.21.15.151:0/682716742 wait complete. 2024-04-16T02:32:55.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f6d857700 1 Processor -- start 2024-04-16T02:32:55.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f6d857700 1 -- start start 2024-04-16T02:32:55.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f6d857700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 0x7f8f68196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:55.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f6d857700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68101de0 0x7f8f681969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:55.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f6d857700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68196ff0 con 0x7f8f68100be0 2024-04-16T02:32:55.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f6d857700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68197130 con 0x7f8f68101de0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f66ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 0x7f8f68196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f66ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 0x7f8f68196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45238/0 (socket says 172.21.15.151:45238) 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f66ffd700 1 -- 172.21.15.151:0/35444461 learned_addr learned my addr 172.21.15.151:0/35444461 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f66ffd700 1 -- 172.21.15.151:0/35444461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68101de0 msgr2=0x7f8f681969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f66ffd700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68101de0 0x7f8f681969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f66ffd700 1 -- 172.21.15.151:0/35444461 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f58009d00 con 0x7f8f68100be0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f66ffd700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 0x7f8f68196490 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8f580077b0 tx=0x7f8f58003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f6c855700 1 -- 172.21.15.151:0/35444461 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f5800e040 con 0x7f8f68100be0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f6c855700 1 -- 172.21.15.151:0/35444461 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8f5800ce60 con 0x7f8f68100be0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.570+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6819bb80 con 0x7f8f68100be0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.571+0000 7f8f6c855700 1 -- 172.21.15.151:0/35444461 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f5801db10 con 0x7f8f68100be0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.571+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6819c0f0 con 0x7f8f68100be0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.571+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f681097c0 con 0x7f8f68100be0 2024-04-16T02:32:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.572+0000 7f8f6c855700 1 -- 172.21.15.151:0/35444461 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8f5801dc90 con 0x7f8f68100be0 2024-04-16T02:32:55.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.572+0000 7f8f6c855700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8f5006c4e0 0x7f8f5006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:55.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.572+0000 7f8f6c855700 1 -- 172.21.15.151:0/35444461 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8f5808d260 con 0x7f8f68100be0 2024-04-16T02:32:55.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.573+0000 7f8f57fff700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8f5006c4e0 0x7f8f5006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:55.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.573+0000 7f8f57fff700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8f5006c4e0 0x7f8f5006e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0097b0 tx=0x7f8f5c006d20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:55.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.577+0000 7f8f6c855700 1 -- 172.21.15.151:0/35444461 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8f5805b4f0 con 0x7f8f68100be0 2024-04-16T02:32:55.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.812+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8f6819c730 con 0x7f8f68100be0 2024-04-16T02:32:55.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.813+0000 7f8f6c855700 1 -- 172.21.15.151:0/35444461 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1851 (secure 0 0 0) 0x7f8f5805b080 con 0x7f8f68100be0 2024-04-16T02:32:55.815 INFO:teuthology.orchestra.run.smithi151.stdout:e18 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 18 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:32:53.952055+0000 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:32:55.816 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:32:55.817 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:32:55.818 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:32:55.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8f5006c4e0 msgr2=0x7f8f5006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:55.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8f5006c4e0 0x7f8f5006e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0097b0 tx=0x7f8f5c006d20 comp rx=0 tx=0).stop 2024-04-16T02:32:55.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 msgr2=0x7f8f68196490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:55.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 0x7f8f68196490 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8f580077b0 tx=0x7f8f58003ce0 comp rx=0 tx=0).stop 2024-04-16T02:32:55.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 shutdown_connections 2024-04-16T02:32:55.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f8f5006c4e0 0x7f8f5006e9a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:55.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f68100be0 0x7f8f68196490 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:55.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 --2- 172.21.15.151:0/35444461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68101de0 0x7f8f681969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:55.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 >> 172.21.15.151:0/35444461 conn(0x7f8f680fc140 msgr2=0x7f8f68105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:55.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 shutdown_connections 2024-04-16T02:32:55.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:55.815+0000 7f8f6d857700 1 -- 172.21.15.151:0/35444461 wait complete. 2024-04-16T02:32:55.820 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 18 2024-04-16T02:32:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:55 smithi151 ceph-mon[29003]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 102 B/s wr, 5 op/s 2024-04-16T02:32:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:55 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/35444461' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:32:56.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:55 smithi153 ceph-mon[36747]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 102 B/s wr, 5 op/s 2024-04-16T02:32:56.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:55 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/35444461' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:32:56.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-16T02:32:56.733 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:32:58.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:57 smithi151 ceph-mon[29003]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-16T02:32:58.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:58.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:58.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:58.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:32:58.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:58.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:32:58.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:58.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:32:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:57 smithi153 ceph-mon[36747]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-16T02:32:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:32:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:32:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:32:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:32:58.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.385+0000 7f68298a5700 1 -- 172.21.15.151:0/219960473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824101dc0 msgr2=0x7f6824102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:58.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.385+0000 7f68298a5700 1 --2- 172.21.15.151:0/219960473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824101dc0 0x7f6824102220 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6814009910 tx=0x7f6814009c20 comp rx=0 tx=0).stop 2024-04-16T02:32:58.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.385+0000 7f68298a5700 1 -- 172.21.15.151:0/219960473 shutdown_connections 2024-04-16T02:32:58.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.385+0000 7f68298a5700 1 --2- 172.21.15.151:0/219960473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824101dc0 0x7f6824102220 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:58.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.385+0000 7f68298a5700 1 --2- 172.21.15.151:0/219960473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6824100bc0 0x7f6824100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:58.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.385+0000 7f68298a5700 1 -- 172.21.15.151:0/219960473 >> 172.21.15.151:0/219960473 conn(0x7f68240fc140 msgr2=0x7f68240fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.385+0000 7f68298a5700 1 -- 172.21.15.151:0/219960473 shutdown_connections 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.385+0000 7f68298a5700 1 -- 172.21.15.151:0/219960473 wait complete. 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.386+0000 7f68298a5700 1 Processor -- start 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.386+0000 7f68298a5700 1 -- start start 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.386+0000 7f68298a5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824100bc0 0x7f6824196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.386+0000 7f68298a5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6824101dc0 0x7f68241968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.386+0000 7f68298a5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6824196ed0 con 0x7f6824100bc0 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.386+0000 7f68298a5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6824197010 con 0x7f6824101dc0 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f6822ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824100bc0 0x7f6824196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f6822ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824100bc0 0x7f6824196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53694/0 (socket says 172.21.15.151:53694) 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f6822ffd700 1 -- 172.21.15.151:0/1106563293 learned_addr learned my addr 172.21.15.151:0/1106563293 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f68227fc700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6824101dc0 0x7f68241968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f6822ffd700 1 -- 172.21.15.151:0/1106563293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6824101dc0 msgr2=0x7f68241968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f6822ffd700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6824101dc0 0x7f68241968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:58.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f6822ffd700 1 -- 172.21.15.151:0/1106563293 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68140095f0 con 0x7f6824100bc0 2024-04-16T02:32:58.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f6822ffd700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824100bc0 0x7f6824196370 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f681800e8f0 tx=0x7f681800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:58.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f68288a3700 1 -- 172.21.15.151:0/1106563293 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6818009790 con 0x7f6824100bc0 2024-04-16T02:32:58.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f68288a3700 1 -- 172.21.15.151:0/1106563293 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f681800c810 con 0x7f6824100bc0 2024-04-16T02:32:58.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682419bac0 con 0x7f6824100bc0 2024-04-16T02:32:58.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f68288a3700 1 -- 172.21.15.151:0/1106563293 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6818003d20 con 0x7f6824100bc0 2024-04-16T02:32:58.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.387+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682419c010 con 0x7f6824100bc0 2024-04-16T02:32:58.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.388+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6824109720 con 0x7f6824100bc0 2024-04-16T02:32:58.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.389+0000 7f68288a3700 1 -- 172.21.15.151:0/1106563293 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f68180041f0 con 0x7f6824100bc0 2024-04-16T02:32:58.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.389+0000 7f68288a3700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f680c06c4e0 0x7f680c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:32:58.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.389+0000 7f68288a3700 1 -- 172.21.15.151:0/1106563293 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6818014070 con 0x7f6824100bc0 2024-04-16T02:32:58.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.389+0000 7f68227fc700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f680c06c4e0 0x7f680c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:32:58.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.390+0000 7f68227fc700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f680c06c4e0 0x7f680c06e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6814009fd0 tx=0x7f6814005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:32:58.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.394+0000 7f68288a3700 1 -- 172.21.15.151:0/1106563293 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6818059d90 con 0x7f6824100bc0 2024-04-16T02:32:58.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.640+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6824109910 con 0x7f6824100bc0 2024-04-16T02:32:58.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.641+0000 7f68288a3700 1 -- 172.21.15.151:0/1106563293 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4776 (secure 0 0 0) 0x7f6818059920 con 0x7f6824100bc0 2024-04-16T02:32:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f680c06c4e0 msgr2=0x7f680c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f680c06c4e0 0x7f680c06e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6814009fd0 tx=0x7f6814005940 comp rx=0 tx=0).stop 2024-04-16T02:32:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824100bc0 msgr2=0x7f6824196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:32:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824100bc0 0x7f6824196370 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f681800e8f0 tx=0x7f681800ec00 comp rx=0 tx=0).stop 2024-04-16T02:32:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 shutdown_connections 2024-04-16T02:32:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f680c06c4e0 0x7f680c06e9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6824100bc0 0x7f6824196370 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 --2- 172.21.15.151:0/1106563293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6824101dc0 0x7f68241968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:32:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 >> 172.21.15.151:0/1106563293 conn(0x7f68240fc140 msgr2=0x7f6824104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:32:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 shutdown_connections 2024-04-16T02:32:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:32:58.643+0000 7f68298a5700 1 -- 172.21.15.151:0/1106563293 wait complete. 2024-04-16T02:32:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 18 2024-04-16T02:32:58.656 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:32:59.217 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:58 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1106563293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T02:32:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:58 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1106563293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T02:32:59.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-16T02:32:59.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:59 smithi151 ceph-mon[29003]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-16T02:32:59.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:32:59.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:32:59 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:33:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:59 smithi153 ceph-mon[36747]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-16T02:33:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:33:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:32:59 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:33:00.645 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:33:02.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:01 smithi151 ceph-mon[29003]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T02:33:02.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:01 smithi153 ceph-mon[36747]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T02:33:02.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 -- 172.21.15.151:0/390060173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c101660 msgr2=0x7f268c103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 --2- 172.21.15.151:0/390060173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c101660 0x7f268c103a50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2688009910 tx=0x7f2688009c20 comp rx=0 tx=0).stop 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 -- 172.21.15.151:0/390060173 shutdown_connections 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 --2- 172.21.15.151:0/390060173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268c103f90 0x7f268c106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 --2- 172.21.15.151:0/390060173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c101660 0x7f268c103a50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 -- 172.21.15.151:0/390060173 >> 172.21.15.151:0/390060173 conn(0x7f268c0fb050 msgr2=0x7f268c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 -- 172.21.15.151:0/390060173 shutdown_connections 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 -- 172.21.15.151:0/390060173 wait complete. 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 Processor -- start 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 -- start start 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268c101660 0x7f268c1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c103f90 0x7f268c196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c196f50 con 0x7f268c103f90 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.740+0000 7f2694497700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c197090 con 0x7f268c101660 2024-04-16T02:33:02.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2691a32700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c103f90 0x7f268c196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2691a32700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c103f90 0x7f268c196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53712/0 (socket says 172.21.15.151:53712) 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2691a32700 1 -- 172.21.15.151:0/2397968056 learned_addr learned my addr 172.21.15.151:0/2397968056 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2691a32700 1 -- 172.21.15.151:0/2397968056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268c101660 msgr2=0x7f268c1963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2692233700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268c101660 0x7f268c1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2691a32700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268c101660 0x7f268c1963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2691a32700 1 -- 172.21.15.151:0/2397968056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26880095f0 con 0x7f268c103f90 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2691a32700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c103f90 0x7f268c196930 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f267c00b4b0 tx=0x7f267c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f26837fe700 1 -- 172.21.15.151:0/2397968056 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f267c011800 con 0x7f268c103f90 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268c19bb40 con 0x7f268c103f90 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f26837fe700 1 -- 172.21.15.151:0/2397968056 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f267c011e10 con 0x7f268c103f90 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268c19c110 con 0x7f268c103f90 2024-04-16T02:33:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.741+0000 7f26837fe700 1 -- 172.21.15.151:0/2397968056 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f267c01a400 con 0x7f268c103f90 2024-04-16T02:33:02.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.742+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2670005320 con 0x7f268c103f90 2024-04-16T02:33:02.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.743+0000 7f26837fe700 1 -- 172.21.15.151:0/2397968056 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f267c011960 con 0x7f268c103f90 2024-04-16T02:33:02.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.743+0000 7f26837fe700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f267806c4e0 0x7f267806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:02.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.743+0000 7f26837fe700 1 -- 172.21.15.151:0/2397968056 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f267c08b3f0 con 0x7f268c103f90 2024-04-16T02:33:02.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.743+0000 7f2692233700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f267806c4e0 0x7f267806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:02.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.744+0000 7f2692233700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f267806c4e0 0x7f267806e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f268800b5c0 tx=0x7f2688005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:02.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:02.748+0000 7f26837fe700 1 -- 172.21.15.151:0/2397968056 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f267c059600 con 0x7f268c103f90 2024-04-16T02:33:03.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.027+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f2670005190 con 0x7f268c103f90 2024-04-16T02:33:03.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.027+0000 7f26837fe700 1 -- 172.21.15.151:0/2397968056 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+83 (secure 0 0 0) 0x7f267c059190 con 0x7f268c103f90 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f267806c4e0 msgr2=0x7f267806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f267806c4e0 0x7f267806e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f268800b5c0 tx=0x7f2688005960 comp rx=0 tx=0).stop 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c103f90 msgr2=0x7f268c196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c103f90 0x7f268c196930 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f267c00b4b0 tx=0x7f267c00b870 comp rx=0 tx=0).stop 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 shutdown_connections 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f267806c4e0 0x7f267806e9a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268c101660 0x7f268c1963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 --2- 172.21.15.151:0/2397968056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f268c103f90 0x7f268c196930 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 >> 172.21.15.151:0/2397968056 conn(0x7f268c0fb050 msgr2=0x7f268c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 shutdown_connections 2024-04-16T02:33:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:03.030+0000 7f2694497700 1 -- 172.21.15.151:0/2397968056 wait complete. 2024-04-16T02:33:03.045 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:33:04.125 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:03 smithi151 ceph-mon[29003]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T02:33:04.125 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:03 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2397968056' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-16T02:33:04.148 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-16T02:33:04.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T02:33:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:03 smithi153 ceph-mon[36747]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T02:33:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:03 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2397968056' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.283+0000 7f1e47ebd640 1 -- 172.21.15.151:0/2315239372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 msgr2=0x7f1e40100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.283+0000 7f1e47ebd640 1 --2- 172.21.15.151:0/2315239372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 0x7f1e40100d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009880 tx=0x7f1e3c02bce0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.283+0000 7f1e47ebd640 1 -- 172.21.15.151:0/2315239372 shutdown_connections 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.283+0000 7f1e47ebd640 1 --2- 172.21.15.151:0/2315239372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e40101290 0x7f1e40103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.283+0000 7f1e47ebd640 1 --2- 172.21.15.151:0/2315239372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 0x7f1e40100d50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.283+0000 7f1e47ebd640 1 -- 172.21.15.151:0/2315239372 >> 172.21.15.151:0/2315239372 conn(0x7f1e400fa4e0 msgr2=0x7f1e400fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.283+0000 7f1e47ebd640 1 -- 172.21.15.151:0/2315239372 shutdown_connections 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.283+0000 7f1e47ebd640 1 -- 172.21.15.151:0/2315239372 wait complete. 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.284+0000 7f1e47ebd640 1 Processor -- start 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.284+0000 7f1e47ebd640 1 -- start start 2024-04-16T02:33:04.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.284+0000 7f1e47ebd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 0x7f1e4019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.284+0000 7f1e47ebd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e40101290 0x7f1e4019c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.284+0000 7f1e47ebd640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4019c9c0 con 0x7f1e400fe920 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.284+0000 7f1e47ebd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4019cb30 con 0x7f1e40101290 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.284+0000 7f1e45c59640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 0x7f1e4019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.284+0000 7f1e45c59640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 0x7f1e4019beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53724/0 (socket says 172.21.15.151:53724) 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.284+0000 7f1e45c59640 1 -- 172.21.15.151:0/654137615 learned_addr learned my addr 172.21.15.151:0/654137615 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e45458640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e40101290 0x7f1e4019c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e45c59640 1 -- 172.21.15.151:0/654137615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e40101290 msgr2=0x7f1e4019c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e45c59640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e40101290 0x7f1e4019c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e45c59640 1 -- 172.21.15.151:0/654137615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c009560 con 0x7f1e400fe920 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e45c59640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 0x7f1e4019beb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009430 tx=0x7f1e3c02cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e36ffd640 1 -- 172.21.15.151:0/654137615 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e3c004520 con 0x7f1e400fe920 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e36ffd640 1 -- 172.21.15.151:0/654137615 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1e3c0054b0 con 0x7f1e400fe920 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e401a1560 con 0x7f1e400fe920 2024-04-16T02:33:04.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e36ffd640 1 -- 172.21.15.151:0/654137615 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e3c034440 con 0x7f1e400fe920 2024-04-16T02:33:04.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.285+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e401a1a60 con 0x7f1e400fe920 2024-04-16T02:33:04.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.286+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e40196010 con 0x7f1e400fe920 2024-04-16T02:33:04.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.287+0000 7f1e36ffd640 1 -- 172.21.15.151:0/654137615 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1e3c005000 con 0x7f1e400fe920 2024-04-16T02:33:04.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.287+0000 7f1e36ffd640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1e2c06c510 0x7f1e2c06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:04.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.287+0000 7f1e36ffd640 1 -- 172.21.15.151:0/654137615 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1e3c0adef0 con 0x7f1e400fe920 2024-04-16T02:33:04.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.287+0000 7f1e45458640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1e2c06c510 0x7f1e2c06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:04.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.287+0000 7f1e45458640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1e2c06c510 0x7f1e2c06e9d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1e30005fd0 tx=0x7f1e30005e20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:04.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.292+0000 7f1e36ffd640 1 -- 172.21.15.151:0/654137615 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1e3c0786f0 con 0x7f1e400fe920 2024-04-16T02:33:04.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.473+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e4010f370 con 0x7f1e400fe920 2024-04-16T02:33:04.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.474+0000 7f1e36ffd640 1 -- 172.21.15.151:0/654137615 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+15201 (secure 0 0 0) 0x7f1e3c07bd10 con 0x7f1e400fe920 2024-04-16T02:33:04.476 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:33:04.476 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":47,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","created":"2024-04-16T02:25:34.606206+0000","modified":"2024-04-16T02:32:48.923598+0000","last_up_change":"2024-04-16T02:31:17.730938+0000","last_in_change":"2024-04-16T02:30:58.536477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T02:30:03.137221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T02:32:33.681623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T02:32:34.178442+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8da4d50b-626b-4cf3-9524-ea84e7733113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6803","nonce":1205378476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6805","nonce":1205378476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6809","nonce":1205378476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1205378476},{"type":"v1","addr":"172.21.15.151:6807","nonce":1205378476}]},"public_addr":"172.21.15.151:6803/1205378476","cluster_addr":"172.21.15.151:6805/1205378476","heartbeat_back_addr":"172.21.15.151:6809/1205378476","heartbeat_front_addr":"172.21.15.151:6807/1205378476","state":["exists","up"]},{"osd":1,"uuid":"8d2980b5-1d30-445c-91ae-f08074b6d8f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6811","nonce":1648693895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6813","nonce":1648693895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6817","nonce":1648693895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1648693895},{"type":"v1","addr":"172.21.15.151:6815","nonce":1648693895}]},"public_addr":"172.21.15.151:6811/1648693895","cluster_addr":"172.21.15.151:6813/1648693895","heartbeat_back_addr":"172.21.15.151:6817/1648693895","heartbeat_front_addr":"172.21.15.151:6815/1648693895","state":["exists","up"]},{"osd":2,"uuid":"64e4e7ce-5c3c-48c8-be3d-c9936b6866b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6819","nonce":2687133389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6821","nonce":2687133389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6825","nonce":2687133389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2687133389},{"type":"v1","addr":"172.21.15.151:6823","nonce":2687133389}]},"public_addr":"172.21.15.151:6819/2687133389","cluster_addr":"172.21.15.151:6821/2687133389","heartbeat_back_addr":"172.21.15.151:6825/2687133389","heartbeat_front_addr":"172.21.15.151:6823/2687133389","state":["exists","up"]},{"osd":3,"uuid":"1194f132-bf2d-4e0c-ab93-f3344ff64d76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6801","nonce":2298379944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6803","nonce":2298379944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6807","nonce":2298379944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2298379944},{"type":"v1","addr":"172.21.15.153:6805","nonce":2298379944}]},"public_addr":"172.21.15.153:6801/2298379944","cluster_addr":"172.21.15.153:6803/2298379944","heartbeat_back_addr":"172.21.15.153:6807/2298379944","heartbeat_front_addr":"172.21.15.153:6805/2298379944","state":["exists","up"]},{"osd":4,"uuid":"9d803d8b-eb35-4a0f-9d36-de11b743a123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6809","nonce":2873952441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6811","nonce":2873952441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6815","nonce":2873952441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2873952441},{"type":"v1","addr":"172.21.15.153:6813","nonce":2873952441}]},"public_addr":"172.21.15.153:6809/2873952441","cluster_addr":"172.21.15.153:6811/2873952441","heartbeat_back_addr":"172.21.15.153:6815/2873952441","heartbeat_front_addr":"172.21.15.153:6813/2873952441","state":["exists","up"]},{"osd":5,"uuid":"7d0d2460-ef76-481b-ad51-afb80d05bc72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6817","nonce":1792791923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6819","nonce":1792791923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6823","nonce":1792791923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1792791923},{"type":"v1","addr":"172.21.15.153:6821","nonce":1792791923}]},"public_addr":"172.21.15.153:6817/1792791923","cluster_addr":"172.21.15.153:6819/1792791923","heartbeat_back_addr":"172.21.15.153:6823/1792791923","heartbeat_front_addr":"172.21.15.153:6821/1792791923","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:02.729782+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:31.268279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:59.891666+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:24.930390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:50.555079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:31:15.969056+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6825/1319024054":"2024-04-17T02:32:48.923574+0000","172.21.15.151:0/1923729027":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/969025109":"2024-04-17T02:26:01.417600+0000","172.21.15.151:6801/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/2293925528":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/3129110405":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6800/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/2684608614":"2024-04-17T02:26:34.117371+0000","172.21.15.151:0/4272586248":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/681541897":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3454798699":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6800/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.151:0/3798542210":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6800/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.153:6824/1319024054":"2024-04-17T02:32:48.923574+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T02:33:04.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1e2c06c510 msgr2=0x7f1e2c06e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1e2c06c510 0x7f1e2c06e9d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1e30005fd0 tx=0x7f1e30005e20 comp rx=0 tx=0).stop 2024-04-16T02:33:04.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 msgr2=0x7f1e4019beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 0x7f1e4019beb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009430 tx=0x7f1e3c02cec0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 shutdown_connections 2024-04-16T02:33:04.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e40101290 0x7f1e4019c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1e2c06c510 0x7f1e2c06e9d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 --2- 172.21.15.151:0/654137615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e400fe920 0x7f1e4019beb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 >> 172.21.15.151:0/654137615 conn(0x7f1e400fa4e0 msgr2=0x7f1e400fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:04.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 shutdown_connections 2024-04-16T02:33:04.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.476+0000 7f1e47ebd640 1 -- 172.21.15.151:0/654137615 wait complete. 2024-04-16T02:33:04.490 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T02:30:03.137221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T02:32:33.681623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T02:32:34.178442+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T02:33:04.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T02:33:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.597+0000 7f7161841640 1 -- 172.21.15.151:0/4277537892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 msgr2=0x7f715c0730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.597+0000 7f7161841640 1 --2- 172.21.15.151:0/4277537892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 0x7f715c0730e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7144009880 tx=0x7f714402bd70 comp rx=0 tx=0).stop 2024-04-16T02:33:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.597+0000 7f7161841640 1 -- 172.21.15.151:0/4277537892 shutdown_connections 2024-04-16T02:33:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.597+0000 7f7161841640 1 --2- 172.21.15.151:0/4277537892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 0x7f715c0730e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.597+0000 7f7161841640 1 --2- 172.21.15.151:0/4277537892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f715c074260 0x7f715c0726b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.597+0000 7f7161841640 1 -- 172.21.15.151:0/4277537892 >> 172.21.15.151:0/4277537892 conn(0x7f715c0fb680 msgr2=0x7f715c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.597+0000 7f7161841640 1 -- 172.21.15.151:0/4277537892 shutdown_connections 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f7161841640 1 -- 172.21.15.151:0/4277537892 wait complete. 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f7161841640 1 Processor -- start 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f7161841640 1 -- start start 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f7161841640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 0x7f715c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f7161841640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f715c074260 0x7f715c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f7161841640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c198640 con 0x7f715c072c80 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f7161841640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c1987b0 con 0x7f715c074260 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f715affd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 0x7f715c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f715affd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 0x7f715c197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53750/0 (socket says 172.21.15.151:53750) 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.598+0000 7f715affd640 1 -- 172.21.15.151:0/3906932642 learned_addr learned my addr 172.21.15.151:0/3906932642 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.599+0000 7f715affd640 1 -- 172.21.15.151:0/3906932642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f715c074260 msgr2=0x7f715c198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.599+0000 7f715affd640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f715c074260 0x7f715c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.599+0000 7f715affd640 1 -- 172.21.15.151:0/3906932642 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144009560 con 0x7f715c072c80 2024-04-16T02:33:04.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.599+0000 7f715affd640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 0x7f715c197b30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f714c00cc60 tx=0x7f714c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:04.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.599+0000 7f716083f640 1 -- 172.21.15.151:0/3906932642 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f714c004b20 con 0x7f715c072c80 2024-04-16T02:33:04.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.599+0000 7f716083f640 1 -- 172.21.15.151:0/3906932642 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f714c004c80 con 0x7f715c072c80 2024-04-16T02:33:04.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.599+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715c19d240 con 0x7f715c072c80 2024-04-16T02:33:04.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.599+0000 7f716083f640 1 -- 172.21.15.151:0/3906932642 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f714c00f660 con 0x7f715c072c80 2024-04-16T02:33:04.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.599+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c19d7a0 con 0x7f715c072c80 2024-04-16T02:33:04.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.600+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c10b000 con 0x7f715c072c80 2024-04-16T02:33:04.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.601+0000 7f716083f640 1 -- 172.21.15.151:0/3906932642 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f714c00f920 con 0x7f715c072c80 2024-04-16T02:33:04.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.601+0000 7f716083f640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f714806c450 0x7f714806e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:04.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.601+0000 7f716083f640 1 -- 172.21.15.151:0/3906932642 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f714c08ba20 con 0x7f715c072c80 2024-04-16T02:33:04.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.601+0000 7f715a7fc640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f714806c450 0x7f714806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:04.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.602+0000 7f715a7fc640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f714806c450 0x7f714806e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f714402c850 tx=0x7f7144000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:04.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.605+0000 7f716083f640 1 -- 172.21.15.151:0/3906932642 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f714c0561a0 con 0x7f715c072c80 2024-04-16T02:33:04.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.781+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f715c19da20 con 0x7f715c072c80 2024-04-16T02:33:04.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.781+0000 7f716083f640 1 -- 172.21.15.151:0/3906932642 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f714c0597c0 con 0x7f715c072c80 2024-04-16T02:33:04.783 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 1 2024-04-16T02:33:04.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.783+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f714806c450 msgr2=0x7f714806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f714806c450 0x7f714806e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f714402c850 tx=0x7f7144000bc0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 msgr2=0x7f715c197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 0x7f715c197b30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f714c00cc60 tx=0x7f714c00a620 comp rx=0 tx=0).stop 2024-04-16T02:33:04.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 shutdown_connections 2024-04-16T02:33:04.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f715c074260 0x7f715c198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f714806c450 0x7f714806e910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 --2- 172.21.15.151:0/3906932642 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f715c072c80 0x7f715c197b30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 >> 172.21.15.151:0/3906932642 conn(0x7f715c0fb680 msgr2=0x7f715c105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:04.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 shutdown_connections 2024-04-16T02:33:04.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.784+0000 7f7161841640 1 -- 172.21.15.151:0/3906932642 wait complete. 2024-04-16T02:33:04.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T02:33:04.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 -- 172.21.15.151:0/109563495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 msgr2=0x7f75cc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 --2- 172.21.15.151:0/109563495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 0x7f75cc102f00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f75c80098b0 tx=0x7f75c802bcd0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 -- 172.21.15.151:0/109563495 shutdown_connections 2024-04-16T02:33:04.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 --2- 172.21.15.151:0/109563495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75cc103440 0x7f75cc105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 --2- 172.21.15.151:0/109563495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 0x7f75cc102f00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 -- 172.21.15.151:0/109563495 >> 172.21.15.151:0/109563495 conn(0x7f75cc0fa4e0 msgr2=0x7f75cc0fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 -- 172.21.15.151:0/109563495 shutdown_connections 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 -- 172.21.15.151:0/109563495 wait complete. 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 Processor -- start 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.910+0000 7f75d30a4640 1 -- start start 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d30a4640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 0x7f75cc1959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d30a4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75cc103440 0x7f75cc195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d30a4640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc1964f0 con 0x7f75cc100b10 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d30a4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc196660 con 0x7f75cc103440 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d0e40640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 0x7f75cc1959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d0e40640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 0x7f75cc1959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53762/0 (socket says 172.21.15.151:53762) 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d0e40640 1 -- 172.21.15.151:0/2447789945 learned_addr learned my addr 172.21.15.151:0/2447789945 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d0e40640 1 -- 172.21.15.151:0/2447789945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75cc103440 msgr2=0x7f75cc195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75c3fff640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75cc103440 0x7f75cc195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d0e40640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75cc103440 0x7f75cc195f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:04.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d0e40640 1 -- 172.21.15.151:0/2447789945 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c8009560 con 0x7f75cc100b10 2024-04-16T02:33:04.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d0e40640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 0x7f75cc1959e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009880 tx=0x7f75c8004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:04.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75c1ffb640 1 -- 172.21.15.151:0/2447789945 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75c8005000 con 0x7f75cc100b10 2024-04-16T02:33:04.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75c1ffb640 1 -- 172.21.15.151:0/2447789945 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f75c8005610 con 0x7f75cc100b10 2024-04-16T02:33:04.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75c1ffb640 1 -- 172.21.15.151:0/2447789945 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75c8034440 con 0x7f75cc100b10 2024-04-16T02:33:04.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.911+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc19b090 con 0x7f75cc100b10 2024-04-16T02:33:04.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.912+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc19b540 con 0x7f75cc100b10 2024-04-16T02:33:04.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.913+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc108e00 con 0x7f75cc100b10 2024-04-16T02:33:04.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.914+0000 7f75c1ffb640 1 -- 172.21.15.151:0/2447789945 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f75c8003880 con 0x7f75cc100b10 2024-04-16T02:33:04.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.914+0000 7f75c1ffb640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f75b406c450 0x7f75b406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:04.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.914+0000 7f75c1ffb640 1 -- 172.21.15.151:0/2447789945 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f75c80ade20 con 0x7f75cc100b10 2024-04-16T02:33:04.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.914+0000 7f75c3fff640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f75b406c450 0x7f75b406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:04.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.915+0000 7f75c3fff640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f75b406c450 0x7f75b406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005fd0 tx=0x7f75bc005850 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:04.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:04.920+0000 7f75c1ffb640 1 -- 172.21.15.151:0/2447789945 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f75c80785a0 con 0x7f75cc100b10 2024-04-16T02:33:05.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.112+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f75cc19b870 con 0x7f75cc100b10 2024-04-16T02:33:05.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.113+0000 7f75c1ffb640 1 -- 172.21.15.151:0/2447789945 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f75c807bbc0 con 0x7f75cc100b10 2024-04-16T02:33:05.115 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 32 2024-04-16T02:33:05.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f75b406c450 msgr2=0x7f75b406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f75b406c450 0x7f75b406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005fd0 tx=0x7f75bc005850 comp rx=0 tx=0).stop 2024-04-16T02:33:05.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 msgr2=0x7f75cc1959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 0x7f75cc1959e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009880 tx=0x7f75c8004e30 comp rx=0 tx=0).stop 2024-04-16T02:33:05.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 shutdown_connections 2024-04-16T02:33:05.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75cc103440 0x7f75cc195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f75b406c450 0x7f75b406e910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 --2- 172.21.15.151:0/2447789945 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75cc100b10 0x7f75cc1959e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 >> 172.21.15.151:0/2447789945 conn(0x7f75cc0fa4e0 msgr2=0x7f75cc0fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:05.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 shutdown_connections 2024-04-16T02:33:05.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.115+0000 7f75d30a4640 1 -- 172.21.15.151:0/2447789945 wait complete. 2024-04-16T02:33:05.127 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:04 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/654137615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T02:33:05.127 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:04 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3906932642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T02:33:05.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T02:33:05.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 -- 172.21.15.151:0/1771821543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7254100180 msgr2=0x7f7254100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 --2- 172.21.15.151:0/1771821543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7254100180 0x7f7254100600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7248009c20 tx=0x7f724802c1b0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 -- 172.21.15.151:0/1771821543 shutdown_connections 2024-04-16T02:33:05.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 --2- 172.21.15.151:0/1771821543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7254100180 0x7f7254100600 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 --2- 172.21.15.151:0/1771821543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72540fef90 0x7f72540ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 -- 172.21.15.151:0/1771821543 >> 172.21.15.151:0/1771821543 conn(0x7f72540fa4e0 msgr2=0x7f72540fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:05.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 -- 172.21.15.151:0/1771821543 shutdown_connections 2024-04-16T02:33:05.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 -- 172.21.15.151:0/1771821543 wait complete. 2024-04-16T02:33:05.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 Processor -- start 2024-04-16T02:33:05.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 -- start start 2024-04-16T02:33:05.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72540fef90 0x7f7254195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:05.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7254100180 0x7f7254195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:05.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7254196430 con 0x7f72540fef90 2024-04-16T02:33:05.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725c426640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72541965a0 con 0x7f7254100180 2024-04-16T02:33:05.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.238+0000 7f725a1c2640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72540fef90 0x7f7254195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:05.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f725a1c2640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72540fef90 0x7f7254195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53778/0 (socket says 172.21.15.151:53778) 2024-04-16T02:33:05.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f725a1c2640 1 -- 172.21.15.151:0/2842848791 learned_addr learned my addr 172.21.15.151:0/2842848791 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:05.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f725a1c2640 1 -- 172.21.15.151:0/2842848791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7254100180 msgr2=0x7f7254195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f725a1c2640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7254100180 0x7f7254195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f725a1c2640 1 -- 172.21.15.151:0/2842848791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7248009900 con 0x7f72540fef90 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f725a1c2640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72540fef90 0x7f7254195920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f724c00d6f0 tx=0x7f724c00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f72537fe640 1 -- 172.21.15.151:0/2842848791 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f724c009a30 con 0x7f72540fef90 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f72537fe640 1 -- 172.21.15.151:0/2842848791 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f724c0044d0 con 0x7f72540fef90 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725419b030 con 0x7f72540fef90 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f72537fe640 1 -- 172.21.15.151:0/2842848791 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f724c003df0 con 0x7f72540fef90 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.239+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725419b5e0 con 0x7f72540fef90 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.240+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7254107b70 con 0x7f72540fef90 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.240+0000 7f72537fe640 1 -- 172.21.15.151:0/2842848791 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f724c0035f0 con 0x7f72540fef90 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.241+0000 7f72537fe640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f723c06c400 0x7f723c06e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.241+0000 7f72599c1640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f723c06c400 0x7f723c06e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.241+0000 7f72537fe640 1 -- 172.21.15.151:0/2842848791 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f724c08b2f0 con 0x7f72540fef90 2024-04-16T02:33:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.241+0000 7f72599c1640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f723c06c400 0x7f723c06e8c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7248006010 tx=0x7f7248009800 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:05.247 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.246+0000 7f72537fe640 1 -- 172.21.15.151:0/2842848791 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f724c055a70 con 0x7f72540fef90 2024-04-16T02:33:05.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:04 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/654137615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T02:33:05.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:04 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3906932642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T02:33:05.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.413+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f725419bb80 con 0x7f72540fef90 2024-04-16T02:33:05.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.413+0000 7f72537fe640 1 -- 172.21.15.151:0/2842848791 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f724c059090 con 0x7f72540fef90 2024-04-16T02:33:05.415 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 32 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f723c06c400 msgr2=0x7f723c06e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f723c06c400 0x7f723c06e8c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7248006010 tx=0x7f7248009800 comp rx=0 tx=0).stop 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72540fef90 msgr2=0x7f7254195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72540fef90 0x7f7254195920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f724c00d6f0 tx=0x7f724c00dbb0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 shutdown_connections 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7254100180 0x7f7254195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f723c06c400 0x7f723c06e8c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 --2- 172.21.15.151:0/2842848791 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72540fef90 0x7f7254195920 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 >> 172.21.15.151:0/2842848791 conn(0x7f72540fa4e0 msgr2=0x7f7254067350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 shutdown_connections 2024-04-16T02:33:05.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.415+0000 7f725c426640 1 -- 172.21.15.151:0/2842848791 wait complete. 2024-04-16T02:33:05.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T02:33:05.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.530+0000 7f04d2041640 1 -- 172.21.15.151:0/452405453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc1001b0 msgr2=0x7f04cc100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.530+0000 7f04d2041640 1 --2- 172.21.15.151:0/452405453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc1001b0 0x7f04cc100630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009880 tx=0x7f04b402bce0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.530+0000 7f04d2041640 1 -- 172.21.15.151:0/452405453 shutdown_connections 2024-04-16T02:33:05.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.530+0000 7f04d2041640 1 --2- 172.21.15.151:0/452405453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc1001b0 0x7f04cc100630 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.530+0000 7f04d2041640 1 --2- 172.21.15.151:0/452405453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04cc0fefc0 0x7f04cc0ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.530+0000 7f04d2041640 1 -- 172.21.15.151:0/452405453 >> 172.21.15.151:0/452405453 conn(0x7f04cc0fa530 msgr2=0x7f04cc0fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.530+0000 7f04d2041640 1 -- 172.21.15.151:0/452405453 shutdown_connections 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.530+0000 7f04d2041640 1 -- 172.21.15.151:0/452405453 wait complete. 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04d2041640 1 Processor -- start 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04d2041640 1 -- start start 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04d2041640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc0fefc0 0x7f04cc195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04d2041640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04cc1001b0 0x7f04cc195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04d2041640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc196410 con 0x7f04cc0fefc0 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04d2041640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc196580 con 0x7f04cc1001b0 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04cb7fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc0fefc0 0x7f04cc195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04cb7fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc0fefc0 0x7f04cc195900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53802/0 (socket says 172.21.15.151:53802) 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04cb7fe640 1 -- 172.21.15.151:0/1548711055 learned_addr learned my addr 172.21.15.151:0/1548711055 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04c3fff640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04cc1001b0 0x7f04cc195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04cb7fe640 1 -- 172.21.15.151:0/1548711055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04cc1001b0 msgr2=0x7f04cc195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04cb7fe640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04cc1001b0 0x7f04cc195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04cb7fe640 1 -- 172.21.15.151:0/1548711055 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009560 con 0x7f04cc0fefc0 2024-04-16T02:33:05.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.531+0000 7f04c3fff640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04cc1001b0 0x7f04cc195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:33:05.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.532+0000 7f04cb7fe640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc0fefc0 0x7f04cc195900 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00d6f0 tx=0x7f04bc00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:05.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.532+0000 7f04c97fa640 1 -- 172.21.15.151:0/1548711055 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04bc009a30 con 0x7f04cc0fefc0 2024-04-16T02:33:05.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.532+0000 7f04c97fa640 1 -- 172.21.15.151:0/1548711055 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f04bc0044d0 con 0x7f04cc0fefc0 2024-04-16T02:33:05.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.532+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04cc19afb0 con 0x7f04cc0fefc0 2024-04-16T02:33:05.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.532+0000 7f04c97fa640 1 -- 172.21.15.151:0/1548711055 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04bc003df0 con 0x7f04cc0fefc0 2024-04-16T02:33:05.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.532+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04cc19b510 con 0x7f04cc0fefc0 2024-04-16T02:33:05.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.533+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04cc107d30 con 0x7f04cc0fefc0 2024-04-16T02:33:05.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.533+0000 7f04c97fa640 1 -- 172.21.15.151:0/1548711055 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f04bc0035f0 con 0x7f04cc0fefc0 2024-04-16T02:33:05.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.534+0000 7f04c97fa640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f04b806c4b0 0x7f04b806e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:05.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.534+0000 7f04c97fa640 1 -- 172.21.15.151:0/1548711055 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f04bc08b2f0 con 0x7f04cc0fefc0 2024-04-16T02:33:05.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.534+0000 7f04c3fff640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f04b806c4b0 0x7f04b806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:05.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.534+0000 7f04c3fff640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f04b806c4b0 0x7f04b806e970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f04cc196e10 tx=0x7f04b402c7c0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:05.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.538+0000 7f04c97fa640 1 -- 172.21.15.151:0/1548711055 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f04bc055a70 con 0x7f04cc0fefc0 2024-04-16T02:33:05.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.741+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04cc19b780 con 0x7f04cc0fefc0 2024-04-16T02:33:05.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.742+0000 7f04c97fa640 1 -- 172.21.15.151:0/1548711055 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4776 (secure 0 0 0) 0x7f04bc059090 con 0x7f04cc0fefc0 2024-04-16T02:33:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:33:05.744 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:32:53.952055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14478,"mds_1":14490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14478":{"gid":14478,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":3,"state":"up:active","state_seq":4,"addr":"172.21.15.151:6827/1662886502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1662886502},{"type":"v1","addr":"172.21.15.151:6827","nonce":1662886502}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.151:6829/37453862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":37453862},{"type":"v1","addr":"172.21.15.151:6829","nonce":37453862}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14502":{"gid":14502,"name":"cephfs.smithi153.ogdcjk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6825/3333609621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3333609621},{"type":"v1","addr":"172.21.15.153:6825","nonce":3333609621}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24275":{"gid":24275,"name":"cephfs.smithi153.mqbgav","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.153:6827/2749127439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2749127439},{"type":"v1","addr":"172.21.15.153:6827","nonce":2749127439}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-16T02:33:05.744 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 18 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f04b806c4b0 msgr2=0x7f04b806e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f04b806c4b0 0x7f04b806e970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f04cc196e10 tx=0x7f04b402c7c0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc0fefc0 msgr2=0x7f04cc195900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc0fefc0 0x7f04cc195900 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00d6f0 tx=0x7f04bc00dbb0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 shutdown_connections 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04cc1001b0 0x7f04cc195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f04b806c4b0 0x7f04b806e970 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 --2- 172.21.15.151:0/1548711055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04cc0fefc0 0x7f04cc195900 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 >> 172.21.15.151:0/1548711055 conn(0x7f04cc0fa530 msgr2=0x7f04cc0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 shutdown_connections 2024-04-16T02:33:05.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:05.744+0000 7f04d2041640 1 -- 172.21.15.151:0/1548711055 wait complete. 2024-04-16T02:33:05.757 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-04-16T02:33:05.757 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-16T02:33:05.781 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-16T02:33:05.781 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-16T02:33:05.781 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-16T02:33:05.782 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T02:33:05.782 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:05.782 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-16T02:33:05.782 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T02:33:05.782 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:05.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:05.782 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-04-16T02:33:05.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:05.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link delete ceph-brx 2024-04-16T02:33:05.881 INFO:teuthology.orchestra.run.smithi153.stderr:Cannot find device "ceph-brx" 2024-04-16T02:33:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T02:33:05.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:05.884 DEBUG:teuthology.orchestra.run.smithi151:> ip netns list 2024-04-16T02:33:05.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:05.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link delete ceph-brx 2024-04-16T02:33:05.970 INFO:teuthology.orchestra.run.smithi151.stderr:Cannot find device "ceph-brx" 2024-04-16T02:33:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T02:33:05.971 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-16T02:33:05.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-16T02:33:06.221 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:33:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:06 smithi153 ceph-mon[36747]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T02:33:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:33:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:33:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:33:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:33:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:33:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:06 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2447789945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T02:33:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:06 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2842848791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T02:33:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:06 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1548711055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T02:33:06.343 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:06 smithi151 ceph-mon[29003]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T02:33:06.343 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:33:06.343 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:33:06.343 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:33:06.343 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:33:06.343 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:33:06.344 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:06 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2447789945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T02:33:06.344 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:06 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2842848791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T02:33:06.344 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:06 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1548711055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T02:33:07.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.881+0000 7f752be46700 1 -- 172.21.15.151:0/1832779701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 msgr2=0x7f752410cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:07.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.881+0000 7f752be46700 1 --2- 172.21.15.151:0/1832779701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 0x7f752410cee0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7518009910 tx=0x7f7518009c20 comp rx=0 tx=0).stop 2024-04-16T02:33:07.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.881+0000 7f752be46700 1 -- 172.21.15.151:0/1832779701 shutdown_connections 2024-04-16T02:33:07.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.881+0000 7f752be46700 1 --2- 172.21.15.151:0/1832779701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 0x7f752410cee0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.881+0000 7f752be46700 1 --2- 172.21.15.151:0/1832779701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7524073160 0x7f7524073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.881+0000 7f752be46700 1 -- 172.21.15.151:0/1832779701 >> 172.21.15.151:0/1832779701 conn(0x7f75240fc670 msgr2=0x7f75240fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.881+0000 7f752be46700 1 -- 172.21.15.151:0/1832779701 shutdown_connections 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.881+0000 7f752be46700 1 -- 172.21.15.151:0/1832779701 wait complete. 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.882+0000 7f752be46700 1 Processor -- start 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.882+0000 7f752be46700 1 -- start start 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.882+0000 7f752be46700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7524073160 0x7f752419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.882+0000 7f752be46700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 0x7f752419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.882+0000 7f752be46700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752419f2d0 con 0x7f7524073a80 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.882+0000 7f752be46700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7524198780 con 0x7f7524073160 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.882+0000 7f75293e1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 0x7f752419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.882+0000 7f75293e1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 0x7f752419ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53044/0 (socket says 172.21.15.151:53044) 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.882+0000 7f75293e1700 1 -- 172.21.15.151:0/1998380513 learned_addr learned my addr 172.21.15.151:0/1998380513 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:07.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f7529be2700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7524073160 0x7f752419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f75293e1700 1 -- 172.21.15.151:0/1998380513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7524073160 msgr2=0x7f752419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f75293e1700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7524073160 0x7f752419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f75293e1700 1 -- 172.21.15.151:0/1998380513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180095f0 con 0x7f7524073a80 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f7529be2700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7524073160 0x7f752419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f75293e1700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 0x7f752419ec40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f751800bd70 tx=0x7f751800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f7522ffd700 1 -- 172.21.15.151:0/1998380513 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f751801d070 con 0x7f7524073a80 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f7522ffd700 1 -- 172.21.15.151:0/1998380513 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7518022470 con 0x7f7524073a80 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7524198a00 con 0x7f7524073a80 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f7522ffd700 1 -- 172.21.15.151:0/1998380513 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f751800f690 con 0x7f7524073a80 2024-04-16T02:33:07.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.883+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7524198ef0 con 0x7f7524073a80 2024-04-16T02:33:07.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.884+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752410a5e0 con 0x7f7524073a80 2024-04-16T02:33:07.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.885+0000 7f7522ffd700 1 -- 172.21.15.151:0/1998380513 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f75180225e0 con 0x7f7524073a80 2024-04-16T02:33:07.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.885+0000 7f7522ffd700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f750c06c4e0 0x7f750c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:07.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.885+0000 7f7529be2700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f750c06c4e0 0x7f750c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:07.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.885+0000 7f7522ffd700 1 -- 172.21.15.151:0/1998380513 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f751808cdd0 con 0x7f7524073a80 2024-04-16T02:33:07.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.886+0000 7f7529be2700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f750c06c4e0 0x7f750c06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f751c0060b0 tx=0x7f751c0076a0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:07.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:07.890+0000 7f7522ffd700 1 -- 172.21.15.151:0/1998380513 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f751805b110 con 0x7f7524073a80 2024-04-16T02:33:08.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.128+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f752404ea90 con 0x7f7524073a80 2024-04-16T02:33:08.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.129+0000 7f7522ffd700 1 -- 172.21.15.151:0/1998380513 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f751805aca0 con 0x7f7524073a80 2024-04-16T02:33:08.131 INFO:teuthology.orchestra.run.smithi151.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-16T02:33:08.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.131+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f750c06c4e0 msgr2=0x7f750c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:08.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.131+0000 7f752be46700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f750c06c4e0 0x7f750c06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f751c0060b0 tx=0x7f751c0076a0 comp rx=0 tx=0).stop 2024-04-16T02:33:08.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.131+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 msgr2=0x7f752419ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:08.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.131+0000 7f752be46700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 0x7f752419ec40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f751800bd70 tx=0x7f751800be50 comp rx=0 tx=0).stop 2024-04-16T02:33:08.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.131+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 shutdown_connections 2024-04-16T02:33:08.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.131+0000 7f752be46700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f750c06c4e0 0x7f750c06e9a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:08.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.131+0000 7f752be46700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7524073160 0x7f752419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:08.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.131+0000 7f752be46700 1 --2- 172.21.15.151:0/1998380513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7524073a80 0x7f752419ec40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:08.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.131+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 >> 172.21.15.151:0/1998380513 conn(0x7f75240fc670 msgr2=0x7f7524107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:08.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.132+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 shutdown_connections 2024-04-16T02:33:08.134 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:08 smithi151 ceph-mon[29003]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T02:33:08.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08.132+0000 7f752be46700 1 -- 172.21.15.151:0/1998380513 wait complete. 2024-04-16T02:33:08.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:08 smithi153 ceph-mon[36747]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T02:33:08.732 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T02:33:08.732 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T02:33:08.732 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi151.front.sepia.ceph.com 2024-04-16T02:33:08.732 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T02:33:08.732 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:08.732 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T02:33:08.732 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T02:33:08.732 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T02:33:08.732 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T02:33:08.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:08.733 DEBUG:teuthology.orchestra.run.smithi151:> ip addr 2024-04-16T02:33:08.750 INFO:teuthology.orchestra.run.smithi151.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T02:33:08.750 INFO:teuthology.orchestra.run.smithi151.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T02:33:08.750 INFO:teuthology.orchestra.run.smithi151.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T02:33:08.750 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-04-16T02:33:08.750 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 ::1/128 scope host 2024-04-16T02:33:08.750 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-04-16T02:33:08.750 INFO:teuthology.orchestra.run.smithi151.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T02:33:08.750 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:da:6e:0c brd ff:ff:ff:ff:ff:ff 2024-04-16T02:33:08.750 INFO:teuthology.orchestra.run.smithi151.stdout: altname enp7s0f0 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:da:6e:0d brd ff:ff:ff:ff:ff:ff 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout: altname enp7s0f1 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:88:6f:62 brd ff:ff:ff:ff:ff:ff 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:88:6f:63 brd ff:ff:ff:ff:ff:ff 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout: inet 172.21.15.151/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft 42265sec preferred_lft 42265sec 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 fe80::ec4:7aff:fe88:6f63/64 scope link 2024-04-16T02:33:08.751 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-04-16T02:33:08.752 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T02:33:08.752 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:08.752 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-04-16T02:33:08.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link add name ceph-brx type bridge 2024-04-16T02:33:08.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip addr flush dev ceph-brx 2024-04-16T02:33:08.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link set ceph-brx up 2024-04-16T02:33:08.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T02:33:08.752 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-04-16T02:33:08.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:08.955 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:08.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:08.962 DEBUG:teuthology.orchestra.run.smithi151:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T02:33:08.990 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-04-16T02:33:08.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:08.992 DEBUG:teuthology.orchestra.run.smithi151:> route 2024-04-16T02:33:09.062 INFO:teuthology.orchestra.run.smithi151.stdout:Kernel IP routing table 2024-04-16T02:33:09.062 INFO:teuthology.orchestra.run.smithi151.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T02:33:09.062 INFO:teuthology.orchestra.run.smithi151.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T02:33:09.062 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T02:33:09.062 INFO:teuthology.orchestra.run.smithi151.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T02:33:09.062 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:09.062 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-04-16T02:33:09.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T02:33:09.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T02:33:09.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T02:33:09.063 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-04-16T02:33:09.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:09.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:09 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1998380513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T02:33:09.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:09.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:09.394 DEBUG:teuthology.orchestra.run.smithi151:> ip netns list 2024-04-16T02:33:09.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:09.409 DEBUG:teuthology.orchestra.run.smithi151:> ip netns list-id 2024-04-16T02:33:09.465 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:09.465 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-04-16T02:33:09.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T02:33:09.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-16T02:33:09.466 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-04-16T02:33:09.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:09 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1998380513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T02:33:09.554 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:09.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:09.590 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-16T02:33:09.591 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:09.591 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-04-16T02:33:09.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-16T02:33:09.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T02:33:09.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-16T02:33:09.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-16T02:33:09.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-16T02:33:09.591 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-04-16T02:33:09.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:09.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:09.737 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:09.737 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-04-16T02:33:09.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link set brx.0 up 2024-04-16T02:33:09.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T02:33:09.737 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-04-16T02:33:09.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:09.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:09.820 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-16T02:33:09.820 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T02:33:09.820 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:09.836 INFO:teuthology.orchestra.run.smithi151.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T02:33:09.836 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T02:33:09.836 DEBUG:teuthology.orchestra.run.smithi151:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:09.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo modprobe fuse 2024-04-16T02:33:09.965 DEBUG:teuthology.orchestra.run.smithi151:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T02:33:10.022 INFO:teuthology.orchestra.run.smithi151.stdout:/proc 2024-04-16T02:33:10.022 INFO:teuthology.orchestra.run.smithi151.stdout:/sys 2024-04-16T02:33:10.022 INFO:teuthology.orchestra.run.smithi151.stdout:/dev 2024-04-16T02:33:10.022 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/security 2024-04-16T02:33:10.022 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/shm 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/pts 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/run 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/cgroup 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/pstore 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/bpf 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/ 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/selinux 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/hugepages 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/mqueue 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/debug 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/tracing 2024-04-16T02:33:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/fuse/connections 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/config 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/run/user/1000 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/062417d8fb3a3289505b8ca9f4f6811004bef9b0d2bf75a1217fe249e0982a78/merged 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/5f22b7de27a4d653004e3be721fd845cd923ddee77980ff9670c1b91c17c4e85/merged 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/run/user/0 2024-04-16T02:33:10.024 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/6357ea1c94e6e12c2f4688b54f65049d3322f67ddaa1e5c569b2c68536399f53/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/971059cd0b7f140eef77f40bd807a23b9535e57930489a2d73b0f9a6889d286e/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/37f9e885bcf2fa8ab9190e1164045ffe20a5129c1989c3da76745e4da37c487d/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/650c2684f63d2bca6f91dfc928a21e657e4b9524a4ed70d7a2e21dae8e253e21/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/a8e470f3fc5a832440be9872b058c9a1e83f5b67080b006f9f3bc6ac6bec795f/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/9d8c5cbcc9837e5cd91854ab1f1f0a8819ed251c35b7c13696bb339d0fdf9d49/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/87f99492310c8c383491c4cf426a7465e7a7ea7a73e6327779ef3fdca67c8d5f/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/082287a45bf43d9f8de66c302b24f3e1d602a6304eac979a86ed34f7ce0749ef/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/2fc8105469cfc0f2fb8f6754926db2feaa64ae52c2b26219db86565495913613/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/1bfeb767cb6171ddb23a7b1d1aaeadc3274fe525e5e7a5c6264f818b7bb1617f/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/9fd722d80e4f484d8179911b4c737234a575c7a2e615e7fcd529e47aef6f812e/merged 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T02:33:10.025 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T02:33:10.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:10.026 DEBUG:teuthology.orchestra.run.smithi151:> ls /sys/fs/fuse/connections 2024-04-16T02:33:10.078 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-16T02:33:10.078 DEBUG:teuthology.orchestra.run.smithi151:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-16T02:33:10.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo modprobe fuse 2024-04-16T02:33:10.149 DEBUG:teuthology.orchestra.run.smithi151:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T02:33:10.205 INFO:teuthology.orchestra.run.smithi151.stdout:/proc 2024-04-16T02:33:10.205 INFO:teuthology.orchestra.run.smithi151.stdout:/sys 2024-04-16T02:33:10.205 INFO:teuthology.orchestra.run.smithi151.stdout:/dev 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/security 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/shm 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/pts 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/run 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/cgroup 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/pstore 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/bpf 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/ 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/selinux 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/hugepages 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/mqueue 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/debug 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/tracing 2024-04-16T02:33:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/fuse/connections 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/config 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/run/user/1000 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/062417d8fb3a3289505b8ca9f4f6811004bef9b0d2bf75a1217fe249e0982a78/merged 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/5f22b7de27a4d653004e3be721fd845cd923ddee77980ff9670c1b91c17c4e85/merged 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/run/user/0 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/6357ea1c94e6e12c2f4688b54f65049d3322f67ddaa1e5c569b2c68536399f53/merged 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/971059cd0b7f140eef77f40bd807a23b9535e57930489a2d73b0f9a6889d286e/merged 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/37f9e885bcf2fa8ab9190e1164045ffe20a5129c1989c3da76745e4da37c487d/merged 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/650c2684f63d2bca6f91dfc928a21e657e4b9524a4ed70d7a2e21dae8e253e21/merged 2024-04-16T02:33:10.207 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/a8e470f3fc5a832440be9872b058c9a1e83f5b67080b006f9f3bc6ac6bec795f/merged 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/9d8c5cbcc9837e5cd91854ab1f1f0a8819ed251c35b7c13696bb339d0fdf9d49/merged 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/87f99492310c8c383491c4cf426a7465e7a7ea7a73e6327779ef3fdca67c8d5f/merged 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/082287a45bf43d9f8de66c302b24f3e1d602a6304eac979a86ed34f7ce0749ef/merged 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/2fc8105469cfc0f2fb8f6754926db2feaa64ae52c2b26219db86565495913613/merged 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/1bfeb767cb6171ddb23a7b1d1aaeadc3274fe525e5e7a5c6264f818b7bb1617f/merged 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/containers/storage/overlay/9fd722d80e4f484d8179911b4c737234a575c7a2e615e7fcd529e47aef6f812e/merged 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T02:33:10.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:10.208 DEBUG:teuthology.orchestra.run.smithi151:> ls /sys/fs/fuse/connections 2024-04-16T02:33:10.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi151.stderr:2024-04-16T02:33:10.227+0000 7f4dc4e35480 -1 init, newargv = 0x558d88ea2430 newargc=15 2024-04-16T02:33:10.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi151.stderr:ceph-fuse[77179]: starting ceph client 2024-04-16T02:33:10.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi151.stderr:ceph-fuse[77179]: starting fuse 2024-04-16T02:33:10.245 INFO:teuthology.orchestra.run.smithi151.stdout:159 2024-04-16T02:33:10.245 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [159] 2024-04-16T02:33:10.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo stdin-killer -- python3 -c ' 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> import glob 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> import re 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> import os 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> import subprocess 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> def _find_admin_socket(client_name): 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> files = glob.glob(asok_path) 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> # Given a non-glob path, it better be there 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> if "*" not in asok_path: 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> assert(len(files) == 1) 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> return files[0] 2024-04-16T02:33:10.246 DEBUG:teuthology.orchestra.run.smithi151:> 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> for f in files: 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> contents = proc_f.read() 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> if mountpoint in contents: 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> return f 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> print(_find_admin_socket("client.0")) 2024-04-16T02:33:10.247 DEBUG:teuthology.orchestra.run.smithi151:> ' 2024-04-16T02:33:10.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:10 smithi151 ceph-mon[29003]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-16T02:33:10.368 INFO:teuthology.orchestra.run.smithi151.stdout:/var/run/ceph/ceph-client.0.77179.asok 2024-04-16T02:33:10.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:10.379 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.77179.asok 2024-04-16T02:33:10.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:10.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.77179.asok status 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "metadata": { 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "entity_id": "0", 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "hostname": "smithi151", 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "pid": "77179", 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "root": "/" 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "dentry_count": 0, 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "dentry_pinned_count": 0, 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "id": 14546, 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "inst": { 2024-04-16T02:33:10.518 INFO:teuthology.orchestra.run.smithi151.stdout: "name": { 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "client", 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "num": 14546 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": { 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "192.168.0.1:0", 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1232858561 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": { 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "192.168.0.1:0", 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1232858561 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "inst_str": "client.14546 192.168.0.1:0/1232858561", 2024-04-16T02:33:10.519 INFO:teuthology.orchestra.run.smithi151.stdout: "addr_str": "192.168.0.1:0/1232858561", 2024-04-16T02:33:10.520 INFO:teuthology.orchestra.run.smithi151.stdout: "inode_count": 1, 2024-04-16T02:33:10.520 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_epoch": 18, 2024-04-16T02:33:10.520 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_epoch": 47, 2024-04-16T02:33:10.520 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_epoch_barrier": 0, 2024-04-16T02:33:10.520 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklisted": false, 2024-04-16T02:33:10.520 INFO:teuthology.orchestra.run.smithi151.stdout: "fs_name": "cephfs" 2024-04-16T02:33:10.520 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:33:10.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:10 smithi153 ceph-mon[36747]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-16T02:33:10.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-16T02:33:10.736 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:33:12.131 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:12 smithi151 ceph-mon[29003]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:33:12.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.348+0000 7fa510219700 1 -- 172.21.15.151:0/3235066713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 msgr2=0x7fa508101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:12.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.348+0000 7fa510219700 1 --2- 172.21.15.151:0/3235066713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 0x7fa508101180 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009910 tx=0x7fa4f8009c20 comp rx=0 tx=0).stop 2024-04-16T02:33:12.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.348+0000 7fa510219700 1 -- 172.21.15.151:0/3235066713 shutdown_connections 2024-04-16T02:33:12.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.348+0000 7fa510219700 1 --2- 172.21.15.151:0/3235066713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa508101750 0x7fa5081057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:12.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.348+0000 7fa510219700 1 --2- 172.21.15.151:0/3235066713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 0x7fa508101180 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.348+0000 7fa510219700 1 -- 172.21.15.151:0/3235066713 >> 172.21.15.151:0/3235066713 conn(0x7fa5080fc630 msgr2=0x7fa5080fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.348+0000 7fa510219700 1 -- 172.21.15.151:0/3235066713 shutdown_connections 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.348+0000 7fa510219700 1 -- 172.21.15.151:0/3235066713 wait complete. 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa510219700 1 Processor -- start 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa510219700 1 -- start start 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa510219700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 0x7fa50819c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa510219700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa508101750 0x7fa50819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa510219700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50819d170 con 0x7fa508100da0 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa510219700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa508196620 con 0x7fa508101750 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa50dfb5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 0x7fa50819c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa50dfb5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 0x7fa50819c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53064/0 (socket says 172.21.15.151:53064) 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa50dfb5700 1 -- 172.21.15.151:0/3623091698 learned_addr learned my addr 172.21.15.151:0/3623091698 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa50d7b4700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa508101750 0x7fa50819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:12.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa50dfb5700 1 -- 172.21.15.151:0/3623091698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa508101750 msgr2=0x7fa50819cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:12.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa50dfb5700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa508101750 0x7fa50819cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:12.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.349+0000 7fa50dfb5700 1 -- 172.21.15.151:0/3623091698 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f80095f0 con 0x7fa508100da0 2024-04-16T02:33:12.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.350+0000 7fa50dfb5700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 0x7fa50819c5a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8006010 tx=0x7fa4f800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:12.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.350+0000 7fa4feffd700 1 -- 172.21.15.151:0/3623091698 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f801d070 con 0x7fa508100da0 2024-04-16T02:33:12.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.350+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5081968a0 con 0x7fa508100da0 2024-04-16T02:33:12.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.350+0000 7fa4feffd700 1 -- 172.21.15.151:0/3623091698 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa4f800f460 con 0x7fa508100da0 2024-04-16T02:33:12.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.350+0000 7fa4feffd700 1 -- 172.21.15.151:0/3623091698 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f8021640 con 0x7fa508100da0 2024-04-16T02:33:12.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.350+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa508196d90 con 0x7fa508100da0 2024-04-16T02:33:12.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.351+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa508109140 con 0x7fa508100da0 2024-04-16T02:33:12.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.351+0000 7fa4feffd700 1 -- 172.21.15.151:0/3623091698 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa4f800faf0 con 0x7fa508100da0 2024-04-16T02:33:12.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.352+0000 7fa4feffd700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa4f406c4e0 0x7fa4f406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:12.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.352+0000 7fa50d7b4700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa4f406c4e0 0x7fa4f406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:12.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.352+0000 7fa4feffd700 1 -- 172.21.15.151:0/3623091698 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa4f808cb70 con 0x7fa508100da0 2024-04-16T02:33:12.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.352+0000 7fa50d7b4700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa4f406c4e0 0x7fa4f406e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa504005fd0 tx=0x7fa50400a560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:12.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.357+0000 7fa4feffd700 1 -- 172.21.15.151:0/3623091698 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa4f805ad80 con 0x7fa508100da0 2024-04-16T02:33:12.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:12 smithi153 ceph-mon[36747]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:33:12.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.590+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa50802ccf0 con 0x7fa508100da0 2024-04-16T02:33:12.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.590+0000 7fa4feffd700 1 -- 172.21.15.151:0/3623091698 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7fa4f805a910 con 0x7fa508100da0 2024-04-16T02:33:12.592 INFO:teuthology.orchestra.run.smithi151.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-16T02:33:12.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa4f406c4e0 msgr2=0x7fa4f406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:12.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa4f406c4e0 0x7fa4f406e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa504005fd0 tx=0x7fa50400a560 comp rx=0 tx=0).stop 2024-04-16T02:33:12.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 msgr2=0x7fa50819c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:12.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 0x7fa50819c5a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8006010 tx=0x7fa4f800be20 comp rx=0 tx=0).stop 2024-04-16T02:33:12.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 shutdown_connections 2024-04-16T02:33:12.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fa4f406c4e0 0x7fa4f406e9a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:12.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa508100da0 0x7fa50819c5a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:12.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 --2- 172.21.15.151:0/3623091698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa508101750 0x7fa50819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:12.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 >> 172.21.15.151:0/3623091698 conn(0x7fa5080fc630 msgr2=0x7fa5080fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:12.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 shutdown_connections 2024-04-16T02:33:12.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:12.592+0000 7fa510219700 1 -- 172.21.15.151:0/3623091698 wait complete. 2024-04-16T02:33:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:13 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3623091698' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T02:33:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:13 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:33:13.331 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T02:33:13.331 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T02:33:13.331 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi153.front.sepia.ceph.com 2024-04-16T02:33:13.331 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T02:33:13.331 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:13.331 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T02:33:13.331 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T02:33:13.331 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T02:33:13.332 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T02:33:13.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:13.332 DEBUG:teuthology.orchestra.run.smithi153:> ip addr 2024-04-16T02:33:13.351 INFO:teuthology.orchestra.run.smithi153.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 ::1/128 scope host 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b2 brd ff:ff:ff:ff:ff:ff 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f0 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b3 brd ff:ff:ff:ff:ff:ff 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f1 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:ea brd ff:ff:ff:ff:ff:ff 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T02:33:13.352 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:eb brd ff:ff:ff:ff:ff:ff 2024-04-16T02:33:13.353 INFO:teuthology.orchestra.run.smithi153.stdout: inet 172.21.15.153/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-16T02:33:13.353 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft 42255sec preferred_lft 42255sec 2024-04-16T02:33:13.353 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::ec4:7aff:fe88:6deb/64 scope link 2024-04-16T02:33:13.353 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-04-16T02:33:13.353 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T02:33:13.353 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:13.353 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-16T02:33:13.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add name ceph-brx type bridge 2024-04-16T02:33:13.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr flush dev ceph-brx 2024-04-16T02:33:13.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set ceph-brx up 2024-04-16T02:33:13.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T02:33:13.353 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-16T02:33:13.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:13 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3623091698' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T02:33:13.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:13 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:33:13.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:13.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:13.570 DEBUG:teuthology.orchestra.run.smithi153:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T02:33:13.640 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-04-16T02:33:13.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:13.642 DEBUG:teuthology.orchestra.run.smithi153:> route 2024-04-16T02:33:13.708 INFO:teuthology.orchestra.run.smithi153.stdout:Kernel IP routing table 2024-04-16T02:33:13.708 INFO:teuthology.orchestra.run.smithi153.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T02:33:13.708 INFO:teuthology.orchestra.run.smithi153.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T02:33:13.708 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T02:33:13.708 INFO:teuthology.orchestra.run.smithi153.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T02:33:13.709 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:13.709 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-16T02:33:13.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T02:33:13.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T02:33:13.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T02:33:13.709 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-16T02:33:13.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:14.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:14.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:14.035 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-04-16T02:33:14.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:14.051 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list-id 2024-04-16T02:33:14.107 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:14.107 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-16T02:33:14.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T02:33:14.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-16T02:33:14.107 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-16T02:33:14.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:14.234 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-16T02:33:14.235 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:14.235 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-16T02:33:14.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-16T02:33:14.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T02:33:14.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-16T02:33:14.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-16T02:33:14.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-16T02:33:14.235 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-16T02:33:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:14.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:14 smithi153 ceph-mon[36747]: pgmap v200: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s 2024-04-16T02:33:14.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:14.375 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T02:33:14.375 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-16T02:33:14.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set brx.0 up 2024-04-16T02:33:14.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T02:33:14.375 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-16T02:33:14.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T02:33:14.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:14.459 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-16T02:33:14.459 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T02:33:14.459 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:14.474 INFO:teuthology.orchestra.run.smithi153.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T02:33:14.474 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T02:33:14.474 DEBUG:teuthology.orchestra.run.smithi153:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:14.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:14 smithi151 ceph-mon[29003]: pgmap v200: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s 2024-04-16T02:33:14.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo modprobe fuse 2024-04-16T02:33:14.598 DEBUG:teuthology.orchestra.run.smithi153:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T02:33:14.655 INFO:teuthology.orchestra.run.smithi153.stdout:/proc 2024-04-16T02:33:14.655 INFO:teuthology.orchestra.run.smithi153.stdout:/sys 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/dev 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/security 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/shm 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/pts 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/run 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/cgroup 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/pstore 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/bpf 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/ 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/selinux 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/mqueue 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/hugepages 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/debug 2024-04-16T02:33:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/tracing 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/fuse/connections 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/config 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/run/user/1000 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/ceph 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/run/user/0 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/b4f9f39c42bb31438a9333a674014c1c70317405700523f2ccf2f48898f76df8/merged 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/2b423ca122929c9a68399c3924c1374025fdc45dea8d267c04f032284f45b23a/merged 2024-04-16T02:33:14.657 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/8d3ee33026c2637731dcf759d5c4dfc01ee7c52bce893a2a9054e414eed6af4f/merged 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/075c1007bd17b5efdbd580cd2f10359b8c41a88df82014240b494eb4a501faac/merged 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/836d6ff367268d4d083e3cf4d785e14fd467e51232ee9d44a18007154bc35e16/merged 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/5490c298b331e0da0b36de5df8358bc3eaf615072949928ac1ce1a33291fbb13/merged 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/6cae53082392af693a0c96861aee72559c00302dc60c663cd3a3e76e9155e764/merged 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/699b186c74875546dabea44fd80ce81d0b207bd65276d189faa1cdfb2686257d/merged 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/dccf57db3f5028a86f9db8eb4200e4f2fee6baad706be6f71b6143aa1ba94a93/merged 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/71d02d3d1a017fd7e39c67d5d301c8dd3004a2cc43279f478192b31b39dcb71c/merged 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T02:33:14.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:14.659 DEBUG:teuthology.orchestra.run.smithi153:> ls /sys/fs/fuse/connections 2024-04-16T02:33:14.711 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-16T02:33:14.712 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-16T02:33:14.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo modprobe fuse 2024-04-16T02:33:14.781 DEBUG:teuthology.orchestra.run.smithi153:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T02:33:14.838 INFO:teuthology.orchestra.run.smithi153.stdout:/proc 2024-04-16T02:33:14.838 INFO:teuthology.orchestra.run.smithi153.stdout:/sys 2024-04-16T02:33:14.838 INFO:teuthology.orchestra.run.smithi153.stdout:/dev 2024-04-16T02:33:14.838 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/security 2024-04-16T02:33:14.838 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/shm 2024-04-16T02:33:14.838 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/pts 2024-04-16T02:33:14.838 INFO:teuthology.orchestra.run.smithi153.stdout:/run 2024-04-16T02:33:14.838 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/cgroup 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/pstore 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/bpf 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/ 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/selinux 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/mqueue 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/hugepages 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/debug 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/tracing 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/fuse/connections 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/config 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T02:33:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/run/user/1000 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/ceph 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/run/user/0 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/b4f9f39c42bb31438a9333a674014c1c70317405700523f2ccf2f48898f76df8/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/2b423ca122929c9a68399c3924c1374025fdc45dea8d267c04f032284f45b23a/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/8d3ee33026c2637731dcf759d5c4dfc01ee7c52bce893a2a9054e414eed6af4f/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/075c1007bd17b5efdbd580cd2f10359b8c41a88df82014240b494eb4a501faac/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/836d6ff367268d4d083e3cf4d785e14fd467e51232ee9d44a18007154bc35e16/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/5490c298b331e0da0b36de5df8358bc3eaf615072949928ac1ce1a33291fbb13/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/6cae53082392af693a0c96861aee72559c00302dc60c663cd3a3e76e9155e764/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/699b186c74875546dabea44fd80ce81d0b207bd65276d189faa1cdfb2686257d/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/dccf57db3f5028a86f9db8eb4200e4f2fee6baad706be6f71b6143aa1ba94a93/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/71d02d3d1a017fd7e39c67d5d301c8dd3004a2cc43279f478192b31b39dcb71c/merged 2024-04-16T02:33:14.840 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns 2024-04-16T02:33:14.841 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T02:33:14.841 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T02:33:14.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:14.841 DEBUG:teuthology.orchestra.run.smithi153:> ls /sys/fs/fuse/connections 2024-04-16T02:33:14.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi153.stderr:2024-04-16T02:33:14.869+0000 7fa8cbc4b480 -1 init, newargv = 0x55d137630c90 newargc=15 2024-04-16T02:33:14.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi153.stderr:ceph-fuse[62876]: starting ceph client 2024-04-16T02:33:14.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi153.stderr:ceph-fuse[62876]: starting fuse 2024-04-16T02:33:14.887 INFO:teuthology.orchestra.run.smithi153.stdout:126 2024-04-16T02:33:14.887 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [126] 2024-04-16T02:33:14.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo stdin-killer -- python3 -c ' 2024-04-16T02:33:14.887 DEBUG:teuthology.orchestra.run.smithi153:> import glob 2024-04-16T02:33:14.887 DEBUG:teuthology.orchestra.run.smithi153:> import re 2024-04-16T02:33:14.887 DEBUG:teuthology.orchestra.run.smithi153:> import os 2024-04-16T02:33:14.887 DEBUG:teuthology.orchestra.run.smithi153:> import subprocess 2024-04-16T02:33:14.887 DEBUG:teuthology.orchestra.run.smithi153:> 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> def _find_admin_socket(client_name): 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> files = glob.glob(asok_path) 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> # Given a non-glob path, it better be there 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> if "*" not in asok_path: 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> assert(len(files) == 1) 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> return files[0] 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> for f in files: 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-16T02:33:14.888 DEBUG:teuthology.orchestra.run.smithi153:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-16T02:33:14.889 DEBUG:teuthology.orchestra.run.smithi153:> contents = proc_f.read() 2024-04-16T02:33:14.889 DEBUG:teuthology.orchestra.run.smithi153:> if mountpoint in contents: 2024-04-16T02:33:14.889 DEBUG:teuthology.orchestra.run.smithi153:> return f 2024-04-16T02:33:14.889 DEBUG:teuthology.orchestra.run.smithi153:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-16T02:33:14.889 DEBUG:teuthology.orchestra.run.smithi153:> 2024-04-16T02:33:14.889 DEBUG:teuthology.orchestra.run.smithi153:> print(_find_admin_socket("client.1")) 2024-04-16T02:33:14.889 DEBUG:teuthology.orchestra.run.smithi153:> ' 2024-04-16T02:33:15.011 INFO:teuthology.orchestra.run.smithi153.stdout:/var/run/ceph/ceph-client.1.62876.asok 2024-04-16T02:33:15.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-16T02:33:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T02:33:15.022 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.62876.asok 2024-04-16T02:33:15.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:15.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.62876.asok status 2024-04-16T02:33:15.159 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "metadata": { 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "entity_id": "1", 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "hostname": "smithi153", 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "pid": "62876", 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "root": "/" 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "dentry_count": 0, 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "dentry_pinned_count": 0, 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 24303, 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "inst": { 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": { 2024-04-16T02:33:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client", 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 24303 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": { 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "192.168.0.1:0", 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1797148833 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": { 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "192.168.0.1:0", 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1797148833 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "inst_str": "client.24303 192.168.0.1:0/1797148833", 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "addr_str": "192.168.0.1:0/1797148833", 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "inode_count": 1, 2024-04-16T02:33:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_epoch": 18, 2024-04-16T02:33:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_epoch": 47, 2024-04-16T02:33:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_epoch_barrier": 0, 2024-04-16T02:33:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: "blocklisted": false, 2024-04-16T02:33:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: "fs_name": "cephfs" 2024-04-16T02:33:15.162 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-16T02:33:15.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:15.168 DEBUG:teuthology.orchestra.run.smithi151:> stat --file-system '--printf=%T 2024-04-16T02:33:15.168 DEBUG:teuthology.orchestra.run.smithi151:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:15.186 INFO:teuthology.orchestra.run.smithi151.stdout:fuseblk 2024-04-16T02:33:15.186 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:15.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:15.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:15.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:15.261 DEBUG:teuthology.orchestra.run.smithi153:> stat --file-system '--printf=%T 2024-04-16T02:33:15.261 DEBUG:teuthology.orchestra.run.smithi153:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:15.277 INFO:teuthology.orchestra.run.smithi153.stdout:fuseblk 2024-04-16T02:33:15.277 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:15.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T02:33:15.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:15 smithi153 ceph-mon[36747]: pgmap v201: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-16T02:33:16.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:15 smithi151 ceph-mon[29003]: pgmap v201: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-16T02:33:18.216 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:17 smithi151 ceph-mon[29003]: pgmap v202: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-16T02:33:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:17 smithi153 ceph-mon[36747]: pgmap v202: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-16T02:33:19.311 INFO:teuthology.run_tasks:Running task print... 2024-04-16T02:33:19.322 INFO:teuthology.task.print:**** done client 2024-04-16T02:33:19.322 INFO:teuthology.run_tasks:Running task parallel... 2024-04-16T02:33:19.336 INFO:teuthology.task.parallel:starting parallel... 2024-04-16T02:33:19.336 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-16T02:33:19.336 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T02:33:19.337 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:33:19.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-16T02:33:19.337 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-16T02:33:19.337 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-16T02:33:19.341 INFO:tasks.workunit:Pulling workunits from ref 3f7db60e5ba4c4d053b594b7ec58703d2833c55b 2024-04-16T02:33:19.341 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-16T02:33:19.341 INFO:tasks.workunit:timeout=3h 2024-04-16T02:33:19.341 INFO:tasks.workunit:cleanup=True 2024-04-16T02:33:19.341 DEBUG:teuthology.orchestra.run.smithi151:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:19.362 INFO:teuthology.orchestra.run.smithi151.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:19.362 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-16T02:33:19.363 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 9fh/159d Inode: 1 Links: 2 2024-04-16T02:33:19.363 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:33:19.363 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-16T02:33:19.363 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-16T02:33:19.363 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-04-16 02:32:39.737720441 +0000 2024-04-16T02:33:19.363 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-04-16 02:33:15.257158734 +0000 2024-04-16T02:33:19.363 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2024-04-16T02:33:19.363 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-16T02:33:19.363 DEBUG:teuthology.orchestra.run.smithi151:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-16T02:33:19.440 DEBUG:teuthology.orchestra.run.smithi153:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:19.462 INFO:teuthology.orchestra.run.smithi153.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:19.462 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-16T02:33:19.462 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-04-16T02:33:19.462 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T02:33:19.462 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-16T02:33:19.462 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-16T02:33:19.462 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-16 02:33:19.432580349 +0000 2024-04-16T02:33:19.462 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-16 02:33:19.432580349 +0000 2024-04-16T02:33:19.462 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-04-16T02:33:19.462 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-16T02:33:19.462 DEBUG:teuthology.orchestra.run.smithi153:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-16T02:33:19.542 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3f7db60e5ba4c4d053b594b7ec58703d2833c55b 2024-04-16T02:33:19.542 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 3f7db60e5ba4c4d053b594b7ec58703d2833c55b 2024-04-16T02:33:19.559 INFO:tasks.workunit.client.0.smithi151.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-16T02:33:19.571 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:33:19.600 INFO:tasks.workunit.client.1.smithi153.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-16T02:33:19.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:19 smithi151 ceph-mon[29003]: pgmap v203: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:33:20.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:19 smithi153 ceph-mon[36747]: pgmap v203: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:33:21.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.229+0000 7fb468435700 1 -- 172.21.15.151:0/929505465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 msgr2=0x7fb4600ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:21.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.229+0000 7fb468435700 1 --2- 172.21.15.151:0/929505465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 0x7fb4600ffed0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009910 tx=0x7fb45c009c20 comp rx=0 tx=0).stop 2024-04-16T02:33:21.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.229+0000 7fb468435700 1 -- 172.21.15.151:0/929505465 shutdown_connections 2024-04-16T02:33:21.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.229+0000 7fb468435700 1 --2- 172.21.15.151:0/929505465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb460100cb0 0x7fb460101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:21.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.229+0000 7fb468435700 1 --2- 172.21.15.151:0/929505465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 0x7fb4600ffed0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:21.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.229+0000 7fb468435700 1 -- 172.21.15.151:0/929505465 >> 172.21.15.151:0/929505465 conn(0x7fb4600fb030 msgr2=0x7fb4600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:21.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.229+0000 7fb468435700 1 -- 172.21.15.151:0/929505465 shutdown_connections 2024-04-16T02:33:21.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.229+0000 7fb468435700 1 -- 172.21.15.151:0/929505465 wait complete. 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb468435700 1 Processor -- start 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb468435700 1 -- start start 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb468435700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 0x7fb4601952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb468435700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb460100cb0 0x7fb460195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb468435700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb460195e40 con 0x7fb4600ffab0 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb468435700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb460195f80 con 0x7fb460100cb0 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb4661d1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 0x7fb4601952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb4661d1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 0x7fb4601952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49584/0 (socket says 172.21.15.151:49584) 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb4661d1700 1 -- 172.21.15.151:0/3833032123 learned_addr learned my addr 172.21.15.151:0/3833032123 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.230+0000 7fb4659d0700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb460100cb0 0x7fb460195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.231+0000 7fb4661d1700 1 -- 172.21.15.151:0/3833032123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb460100cb0 msgr2=0x7fb460195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.231+0000 7fb4661d1700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb460100cb0 0x7fb460195820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.231+0000 7fb4661d1700 1 -- 172.21.15.151:0/3833032123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c0095f0 con 0x7fb4600ffab0 2024-04-16T02:33:21.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.231+0000 7fb4661d1700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 0x7fb4601952e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00bda0 tx=0x7fb45c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:21.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.231+0000 7fb4577fe700 1 -- 172.21.15.151:0/3833032123 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb45c01d070 con 0x7fb4600ffab0 2024-04-16T02:33:21.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.231+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46019a9d0 con 0x7fb4600ffab0 2024-04-16T02:33:21.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.231+0000 7fb4577fe700 1 -- 172.21.15.151:0/3833032123 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb45c004c50 con 0x7fb4600ffab0 2024-04-16T02:33:21.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.231+0000 7fb4577fe700 1 -- 172.21.15.151:0/3833032123 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb45c00f690 con 0x7fb4600ffab0 2024-04-16T02:33:21.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.231+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46019af40 con 0x7fb4600ffab0 2024-04-16T02:33:21.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.232+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb460108690 con 0x7fb4600ffab0 2024-04-16T02:33:21.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.233+0000 7fb4577fe700 1 -- 172.21.15.151:0/3833032123 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb45c0229b0 con 0x7fb4600ffab0 2024-04-16T02:33:21.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.233+0000 7fb4577fe700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb44c06c4e0 0x7fb44c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:21.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.233+0000 7fb4659d0700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb44c06c4e0 0x7fb44c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:21.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.233+0000 7fb4577fe700 1 -- 172.21.15.151:0/3833032123 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb45c08cfe0 con 0x7fb4600ffab0 2024-04-16T02:33:21.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.234+0000 7fb4659d0700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb44c06c4e0 0x7fb44c06e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb450006fd0 tx=0x7fb450008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:21.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.238+0000 7fb4577fe700 1 -- 172.21.15.151:0/3833032123 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb45c05b270 con 0x7fb4600ffab0 2024-04-16T02:33:21.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.431+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fb460108850 con 0x7fb4600ffab0 2024-04-16T02:33:21.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.431+0000 7fb4577fe700 1 -- 172.21.15.151:0/3833032123 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fb45c05ae00 con 0x7fb4600ffab0 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb44c06c4e0 msgr2=0x7fb44c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb44c06c4e0 0x7fb44c06e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb450006fd0 tx=0x7fb450008040 comp rx=0 tx=0).stop 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 msgr2=0x7fb4601952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 0x7fb4601952e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00bda0 tx=0x7fb45c00be80 comp rx=0 tx=0).stop 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 shutdown_connections 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb44c06c4e0 0x7fb44c06e9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4600ffab0 0x7fb4601952e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 --2- 172.21.15.151:0/3833032123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb460100cb0 0x7fb460195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 >> 172.21.15.151:0/3833032123 conn(0x7fb4600fb030 msgr2=0x7fb460103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 shutdown_connections 2024-04-16T02:33:21.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:21.433+0000 7fb468435700 1 -- 172.21.15.151:0/3833032123 wait complete. 2024-04-16T02:33:21.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:21 smithi151 ceph-mon[29003]: pgmap v204: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:33:22.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:21 smithi153 ceph-mon[36747]: pgmap v204: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:33:22.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-16T02:33:22.477 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:33:24.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:23 smithi151 ceph-mon[29003]: pgmap v205: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-16T02:33:24.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.259+0000 7fd5702f9700 1 -- 172.21.15.151:0/19877637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 msgr2=0x7fd5680ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.259+0000 7fd5702f9700 1 --2- 172.21.15.151:0/19877637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 0x7fd5680ffed0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd56400b0a0 tx=0x7fd56400b3b0 comp rx=0 tx=0).stop 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.259+0000 7fd5702f9700 1 -- 172.21.15.151:0/19877637 shutdown_connections 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.259+0000 7fd5702f9700 1 --2- 172.21.15.151:0/19877637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd568100cb0 0x7fd568101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.259+0000 7fd5702f9700 1 --2- 172.21.15.151:0/19877637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 0x7fd5680ffed0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.259+0000 7fd5702f9700 1 -- 172.21.15.151:0/19877637 >> 172.21.15.151:0/19877637 conn(0x7fd5680fb050 msgr2=0x7fd5680fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.259+0000 7fd5702f9700 1 -- 172.21.15.151:0/19877637 shutdown_connections 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.259+0000 7fd5702f9700 1 -- 172.21.15.151:0/19877637 wait complete. 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd5702f9700 1 Processor -- start 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd5702f9700 1 -- start start 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd5702f9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 0x7fd568196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd5702f9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd568100cb0 0x7fd5681968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:24.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd5702f9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568196ed0 con 0x7fd5680ffab0 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd5702f9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568197010 con 0x7fd568100cb0 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd56e095700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 0x7fd568196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd56e095700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 0x7fd568196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49604/0 (socket says 172.21.15.151:49604) 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd56e095700 1 -- 172.21.15.151:0/3193290420 learned_addr learned my addr 172.21.15.151:0/3193290420 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd56d894700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd568100cb0 0x7fd5681968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd56e095700 1 -- 172.21.15.151:0/3193290420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd568100cb0 msgr2=0x7fd5681968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd56e095700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd568100cb0 0x7fd5681968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd56e095700 1 -- 172.21.15.151:0/3193290420 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd564009d00 con 0x7fd5680ffab0 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd56d894700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd568100cb0 0x7fd5681968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.260+0000 7fd56e095700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 0x7fd568196370 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd5640077b0 tx=0x7fd564009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.261+0000 7fd55f7fe700 1 -- 172.21.15.151:0/3193290420 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56400e040 con 0x7fd5680ffab0 2024-04-16T02:33:24.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.261+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56819ba60 con 0x7fd5680ffab0 2024-04-16T02:33:24.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.261+0000 7fd55f7fe700 1 -- 172.21.15.151:0/3193290420 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd5640044f0 con 0x7fd5680ffab0 2024-04-16T02:33:24.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.261+0000 7fd55f7fe700 1 -- 172.21.15.151:0/3193290420 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56401d9e0 con 0x7fd5680ffab0 2024-04-16T02:33:24.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.261+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56819bfd0 con 0x7fd5680ffab0 2024-04-16T02:33:24.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.262+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd568108610 con 0x7fd5680ffab0 2024-04-16T02:33:24.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.262+0000 7fd55f7fe700 1 -- 172.21.15.151:0/3193290420 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd564004040 con 0x7fd5680ffab0 2024-04-16T02:33:24.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.263+0000 7fd55f7fe700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd55406c4e0 0x7fd55406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:24.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.263+0000 7fd55f7fe700 1 -- 172.21.15.151:0/3193290420 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd56408d2a0 con 0x7fd5680ffab0 2024-04-16T02:33:24.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.263+0000 7fd56d894700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd55406c4e0 0x7fd55406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:24.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.263+0000 7fd56d894700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd55406c4e0 0x7fd55406e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd558005950 tx=0x7fd5580058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:24.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.267+0000 7fd55f7fe700 1 -- 172.21.15.151:0/3193290420 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd56405b4b0 con 0x7fd5680ffab0 2024-04-16T02:33:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:23 smithi153 ceph-mon[36747]: pgmap v205: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-16T02:33:24.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.456+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fd56819c6d0 con 0x7fd5680ffab0 2024-04-16T02:33:24.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.457+0000 7fd55f7fe700 1 -- 172.21.15.151:0/3193290420 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7fd56405b040 con 0x7fd5680ffab0 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd55406c4e0 msgr2=0x7fd55406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd55406c4e0 0x7fd55406e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd558005950 tx=0x7fd5580058e0 comp rx=0 tx=0).stop 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 msgr2=0x7fd568196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 0x7fd568196370 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd5640077b0 tx=0x7fd564009810 comp rx=0 tx=0).stop 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 shutdown_connections 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fd55406c4e0 0x7fd55406e9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5680ffab0 0x7fd568196370 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 --2- 172.21.15.151:0/3193290420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd568100cb0 0x7fd5681968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 >> 172.21.15.151:0/3193290420 conn(0x7fd5680fb050 msgr2=0x7fd568068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 shutdown_connections 2024-04-16T02:33:24.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:24.459+0000 7fd5702f9700 1 -- 172.21.15.151:0/3193290420 wait complete. 2024-04-16T02:33:25.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-16T02:33:25.453 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:33:25.880 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:25 smithi151 ceph-mon[29003]: pgmap v206: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-16T02:33:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:25 smithi153 ceph-mon[36747]: pgmap v206: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-16T02:33:27.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.119+0000 7f698f74a700 1 -- 172.21.15.151:0/4225627926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 msgr2=0x7f6988101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.119+0000 7f698f74a700 1 --2- 172.21.15.151:0/4225627926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 0x7f6988101000 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6984009910 tx=0x7f6984009c20 comp rx=0 tx=0).stop 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.119+0000 7f698f74a700 1 -- 172.21.15.151:0/4225627926 shutdown_connections 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.119+0000 7f698f74a700 1 --2- 172.21.15.151:0/4225627926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6988101de0 0x7f6988102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.119+0000 7f698f74a700 1 --2- 172.21.15.151:0/4225627926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 0x7f6988101000 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.119+0000 7f698f74a700 1 -- 172.21.15.151:0/4225627926 >> 172.21.15.151:0/4225627926 conn(0x7f69880fc140 msgr2=0x7f69880fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698f74a700 1 -- 172.21.15.151:0/4225627926 shutdown_connections 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698f74a700 1 -- 172.21.15.151:0/4225627926 wait complete. 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698f74a700 1 Processor -- start 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698f74a700 1 -- start start 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698f74a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 0x7f6988196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698f74a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6988101de0 0x7f69881968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698f74a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6988196ef0 con 0x7f6988100be0 2024-04-16T02:33:27.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698f74a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6988197030 con 0x7f6988101de0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698d4e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 0x7f6988196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698d4e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 0x7f6988196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33844/0 (socket says 172.21.15.151:33844) 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.120+0000 7f698d4e6700 1 -- 172.21.15.151:0/4086126950 learned_addr learned my addr 172.21.15.151:0/4086126950 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f698cce5700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6988101de0 0x7f69881968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f698d4e6700 1 -- 172.21.15.151:0/4086126950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6988101de0 msgr2=0x7f69881968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f698d4e6700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6988101de0 0x7f69881968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f698d4e6700 1 -- 172.21.15.151:0/4086126950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69840095f0 con 0x7f6988100be0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f698d4e6700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 0x7f6988196390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f698400bda0 tx=0x7f698400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f697e7fc700 1 -- 172.21.15.151:0/4086126950 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f698401d070 con 0x7f6988100be0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698819ba80 con 0x7f6988100be0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f697e7fc700 1 -- 172.21.15.151:0/4086126950 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f698400f460 con 0x7f6988100be0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f697e7fc700 1 -- 172.21.15.151:0/4086126950 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6984021620 con 0x7f6988100be0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.121+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698819bf70 con 0x7f6988100be0 2024-04-16T02:33:27.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.122+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6988109740 con 0x7f6988100be0 2024-04-16T02:33:27.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.123+0000 7f697e7fc700 1 -- 172.21.15.151:0/4086126950 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f698400f5d0 con 0x7f6988100be0 2024-04-16T02:33:27.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.123+0000 7f697e7fc700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f697406c4e0 0x7f697406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:27.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.123+0000 7f697e7fc700 1 -- 172.21.15.151:0/4086126950 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f698408cb60 con 0x7f6988100be0 2024-04-16T02:33:27.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.123+0000 7f698cce5700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f697406c4e0 0x7f697406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:27.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.123+0000 7f698cce5700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f697406c4e0 0x7f697406e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6978005950 tx=0x7f697800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:27.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.127+0000 7f697e7fc700 1 -- 172.21.15.151:0/4086126950 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f698405ad70 con 0x7f6988100be0 2024-04-16T02:33:27.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.317+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f6988109900 con 0x7f6988100be0 2024-04-16T02:33:27.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.317+0000 7f697e7fc700 1 -- 172.21.15.151:0/4086126950 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f698405a900 con 0x7f6988100be0 2024-04-16T02:33:27.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f697406c4e0 msgr2=0x7f697406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:27.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f697406c4e0 0x7f697406e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6978005950 tx=0x7f697800b410 comp rx=0 tx=0).stop 2024-04-16T02:33:27.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 msgr2=0x7f6988196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:27.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 0x7f6988196390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f698400bda0 tx=0x7f698400bdd0 comp rx=0 tx=0).stop 2024-04-16T02:33:27.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 shutdown_connections 2024-04-16T02:33:27.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f697406c4e0 0x7f697406e9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:27.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6988100be0 0x7f6988196390 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:27.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 --2- 172.21.15.151:0/4086126950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6988101de0 0x7f69881968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:27.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 >> 172.21.15.151:0/4086126950 conn(0x7f69880fc140 msgr2=0x7f6988105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:27.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 shutdown_connections 2024-04-16T02:33:27.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:27.319+0000 7f698f74a700 1 -- 172.21.15.151:0/4086126950 wait complete. 2024-04-16T02:33:28.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-16T02:33:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:27 smithi151 ceph-mon[29003]: pgmap v207: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-16T02:33:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:33:28.266 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:33:28.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:27 smithi153 ceph-mon[36747]: pgmap v207: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-16T02:33:28.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:33:29.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.882+0000 7ff2a1988700 1 -- 172.21.15.151:0/2086109145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c101dc0 msgr2=0x7ff29c102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:29.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.882+0000 7ff2a1988700 1 --2- 172.21.15.151:0/2086109145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c101dc0 0x7ff29c102220 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff290009910 tx=0x7ff290009c20 comp rx=0 tx=0).stop 2024-04-16T02:33:29.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.883+0000 7ff2a1988700 1 -- 172.21.15.151:0/2086109145 shutdown_connections 2024-04-16T02:33:29.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.883+0000 7ff2a1988700 1 --2- 172.21.15.151:0/2086109145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c101dc0 0x7ff29c102220 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:29.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.883+0000 7ff2a1988700 1 --2- 172.21.15.151:0/2086109145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff29c100bc0 0x7ff29c100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:29.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.883+0000 7ff2a1988700 1 -- 172.21.15.151:0/2086109145 >> 172.21.15.151:0/2086109145 conn(0x7ff29c0fc140 msgr2=0x7ff29c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:33:29.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.883+0000 7ff2a1988700 1 -- 172.21.15.151:0/2086109145 shutdown_connections 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.883+0000 7ff2a1988700 1 -- 172.21.15.151:0/2086109145 wait complete. 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.883+0000 7ff2a1988700 1 Processor -- start 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.883+0000 7ff2a1988700 1 -- start start 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff2a1988700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c100bc0 0x7ff29c1963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff2a1988700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff29c101dc0 0x7ff29c196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff2a1988700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c196f40 con 0x7ff29c100bc0 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff2a1988700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c197080 con 0x7ff29c101dc0 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff29affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c100bc0 0x7ff29c1963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff29affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c100bc0 0x7ff29c1963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33852/0 (socket says 172.21.15.151:33852) 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff29affd700 1 -- 172.21.15.151:0/2808157984 learned_addr learned my addr 172.21.15.151:0/2808157984 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff29a7fc700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff29c101dc0 0x7ff29c196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:29.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff29affd700 1 -- 172.21.15.151:0/2808157984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff29c101dc0 msgr2=0x7ff29c196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:33:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff29affd700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff29c101dc0 0x7ff29c196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:33:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff29affd700 1 -- 172.21.15.151:0/2808157984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2900095f0 con 0x7ff29c100bc0 2024-04-16T02:33:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff29affd700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c100bc0 0x7ff29c1963e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00b480 tx=0x7ff28c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff2a0986700 1 -- 172.21.15.151:0/2808157984 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28c0118c0 con 0x7ff29c100bc0 2024-04-16T02:33:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29c19bb30 con 0x7ff29c100bc0 2024-04-16T02:33:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff2a0986700 1 -- 172.21.15.151:0/2808157984 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff28c010380 con 0x7ff29c100bc0 2024-04-16T02:33:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff2a0986700 1 -- 172.21.15.151:0/2808157984 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28c00f4f0 con 0x7ff29c100bc0 2024-04-16T02:33:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.884+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29c19c080 con 0x7ff29c100bc0 2024-04-16T02:33:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.885+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29c1097c0 con 0x7ff29c100bc0 2024-04-16T02:33:29.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.886+0000 7ff2a0986700 1 -- 172.21.15.151:0/2808157984 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff28c0104f0 con 0x7ff29c100bc0 2024-04-16T02:33:29.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.886+0000 7ff2a0986700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ff28406c4e0 0x7ff28406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:33:29.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.886+0000 7ff2a0986700 1 -- 172.21.15.151:0/2808157984 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff28c08b500 con 0x7ff29c100bc0 2024-04-16T02:33:29.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.886+0000 7ff29a7fc700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ff28406c4e0 0x7ff28406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:33:29.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.887+0000 7ff29a7fc700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ff28406c4e0 0x7ff28406e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff290006010 tx=0x7ff29000b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:33:29.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:29.891+0000 7ff2a0986700 1 -- 172.21.15.151:0/2808157984 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff28c059710 con 0x7ff29c100bc0 2024-04-16T02:33:30.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:33:30.101+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7ff29c106710 con 0x7ff28406c4e0 2024-04-16T02:33:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:29 smithi151 ceph-mon[29003]: pgmap v208: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-16T02:33:30.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:29 smithi153 ceph-mon[36747]: pgmap v208: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-16T02:33:31.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:30 smithi151 ceph-mon[29003]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:33:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:30 smithi153 ceph-mon[36747]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:33:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:31 smithi151 ceph-mon[29003]: pgmap v209: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-16T02:33:32.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:31 smithi153 ceph-mon[36747]: pgmap v209: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-16T02:33:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:33 smithi151 ceph-mon[29003]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-16T02:33:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:33 smithi153 ceph-mon[36747]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-16T02:33:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:35 smithi153 ceph-mon[36747]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-16T02:33:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:35 smithi151 ceph-mon[29003]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-16T02:33:38.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:37 smithi151 ceph-mon[29003]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-16T02:33:38.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:37 smithi153 ceph-mon[36747]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-16T02:33:40.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:39 smithi151 ceph-mon[29003]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-16T02:33:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:39 smithi153 ceph-mon[36747]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-16T02:33:42.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:41 smithi151 ceph-mon[29003]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-16T02:33:42.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:41 smithi153 ceph-mon[36747]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-16T02:33:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:33:43.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:42 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:33:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:43 smithi151 ceph-mon[29003]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:33:44.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:43 smithi153 ceph-mon[36747]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:33:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:45 smithi153 ceph-mon[36747]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:33:46.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:45 smithi151 ceph-mon[29003]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:33:48.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:47 smithi151 ceph-mon[29003]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:33:48.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:47 smithi153 ceph-mon[36747]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:33:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:49 smithi151 ceph-mon[29003]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:33:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:49 smithi153 ceph-mon[36747]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:33:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:51 smithi151 ceph-mon[29003]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:33:52.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:51 smithi153 ceph-mon[36747]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:33:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:53 smithi151 ceph-mon[29003]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:33:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:53 smithi153 ceph-mon[36747]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:33:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:55 smithi153 ceph-mon[36747]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:33:56.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:55 smithi151 ceph-mon[29003]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:33:58.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:57 smithi151 ceph-mon[29003]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:33:58.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:33:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:57 smithi153 ceph-mon[36747]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:33:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:34:00.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:33:59 smithi151 ceph-mon[29003]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:33:59 smithi153 ceph-mon[36747]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:02.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:01 smithi151 ceph-mon[29003]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:34:02.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:01 smithi153 ceph-mon[36747]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:34:04.206 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:03 smithi151 ceph-mon[29003]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:03 smithi153 ceph-mon[36747]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:05.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:05 smithi151 ceph-mon[29003]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:34:05.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:34:06.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:05 smithi153 ceph-mon[36747]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:34:06.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:34:07.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:34:07.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:34:07.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:07.114 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:34:07.114 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:34:07.114 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:07.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.182+0000 7ff2a0986700 1 -- 172.21.15.151:0/2808157984 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7ff29c106710 con 0x7ff28406c4e0 2024-04-16T02:34:07.185 INFO:teuthology.orchestra.run.smithi151.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ff28406c4e0 msgr2=0x7ff28406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ff28406c4e0 0x7ff28406e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff290006010 tx=0x7ff29000b540 comp rx=0 tx=0).stop 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c100bc0 msgr2=0x7ff29c1963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c100bc0 0x7ff29c1963e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00b480 tx=0x7ff28c00b840 comp rx=0 tx=0).stop 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 shutdown_connections 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7ff28406c4e0 0x7ff28406e9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c100bc0 0x7ff29c1963e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 --2- 172.21.15.151:0/2808157984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff29c101dc0 0x7ff29c196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 >> 172.21.15.151:0/2808157984 conn(0x7ff29c0fc140 msgr2=0x7ff29c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 shutdown_connections 2024-04-16T02:34:07.188 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:07.185+0000 7ff2a1988700 1 -- 172.21.15.151:0/2808157984 wait complete. 2024-04-16T02:34:07.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-16T02:34:08.428 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:08 smithi151 ceph-mon[29003]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:08.428 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:08 smithi151 ceph-mon[29003]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:34:08.428 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:08.428 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:34:08.428 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:34:08.429 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:34:08.429 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:08 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:08.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:08 smithi153 ceph-mon[36747]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:08.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:08 smithi153 ceph-mon[36747]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:34:08.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:08.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:34:08.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:34:08.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:34:08.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:08 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:09.254 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:34:09.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:09 smithi151 ceph-mon[29003]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:34:09.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:09 smithi151 ceph-mon[29003]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:09.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:09 smithi153 ceph-mon[36747]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:34:09.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:09 smithi153 ceph-mon[36747]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:10.472 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:10 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:10.472 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:10 smithi151 ceph-mon[29003]: Upgrade: Target is version 19.0.0-3084-g87dbfb63 (unknown) 2024-04-16T02:34:10.472 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:10 smithi151 ceph-mon[29003]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f'] 2024-04-16T02:34:10.472 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:10 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:34:10.473 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:10 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:10.473 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:10 smithi151 ceph-mon[29003]: Upgrade: Need to upgrade myself (mgr.smithi151.uqqame) 2024-04-16T02:34:10.473 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:10 smithi151 ceph-mon[29003]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f on smithi153 2024-04-16T02:34:10.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:10 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:10.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:10 smithi153 ceph-mon[36747]: Upgrade: Target is version 19.0.0-3084-g87dbfb63 (unknown) 2024-04-16T02:34:10.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:10 smithi153 ceph-mon[36747]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f'] 2024-04-16T02:34:10.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:10 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:34:10.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:10 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:10.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:10 smithi153 ceph-mon[36747]: Upgrade: Need to upgrade myself (mgr.smithi151.uqqame) 2024-04-16T02:34:10.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:10 smithi153 ceph-mon[36747]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f on smithi153 2024-04-16T02:34:11.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:11 smithi151 ceph-mon[29003]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:34:11.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:11 smithi153 ceph-mon[36747]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:34:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:12 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:34:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:12 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:34:14.211 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:13 smithi151 ceph-mon[29003]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:13 smithi153 ceph-mon[36747]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:14.635 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.769+0000 7f4da95ec700 1 -- 172.21.15.151:0/2305677876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 msgr2=0x7f4da40ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.769+0000 7f4da95ec700 1 --2- 172.21.15.151:0/2305677876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 0x7f4da40ffed0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009910 tx=0x7f4d94009c20 comp rx=0 tx=0).stop 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.770+0000 7f4da95ec700 1 -- 172.21.15.151:0/2305677876 shutdown_connections 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.770+0000 7f4da95ec700 1 --2- 172.21.15.151:0/2305677876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da4100cb0 0x7f4da4101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.770+0000 7f4da95ec700 1 --2- 172.21.15.151:0/2305677876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 0x7f4da40ffed0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.770+0000 7f4da95ec700 1 -- 172.21.15.151:0/2305677876 >> 172.21.15.151:0/2305677876 conn(0x7f4da40fb050 msgr2=0x7f4da40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.770+0000 7f4da95ec700 1 -- 172.21.15.151:0/2305677876 shutdown_connections 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.770+0000 7f4da95ec700 1 -- 172.21.15.151:0/2305677876 wait complete. 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.770+0000 7f4da95ec700 1 Processor -- start 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.770+0000 7f4da95ec700 1 -- start start 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da95ec700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 0x7f4da4069800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:14.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da95ec700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da4100cb0 0x7f4da419d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da95ec700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da4196f50 con 0x7f4da40ffab0 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da95ec700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da419da20 con 0x7f4da4100cb0 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 0x7f4da4069800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 0x7f4da4069800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50714/0 (socket says 172.21.15.151:50714) 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da2ffd700 1 -- 172.21.15.151:0/2375383007 learned_addr learned my addr 172.21.15.151:0/2375383007 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da2ffd700 1 -- 172.21.15.151:0/2375383007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da4100cb0 msgr2=0x7f4da419d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da2ffd700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da4100cb0 0x7f4da419d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da2ffd700 1 -- 172.21.15.151:0/2375383007 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d940095f0 con 0x7f4da40ffab0 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.771+0000 7f4da2ffd700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 0x7f4da4069800 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400bdd0 tx=0x7f4d9400beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.772+0000 7f4d8bfff700 1 -- 172.21.15.151:0/2375383007 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d9401d070 con 0x7f4da40ffab0 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.772+0000 7f4d8bfff700 1 -- 172.21.15.151:0/2375383007 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4d94004bb0 con 0x7f4da40ffab0 2024-04-16T02:34:14.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.772+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da419dc70 con 0x7f4da40ffab0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.772+0000 7f4d8bfff700 1 -- 172.21.15.151:0/2375383007 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d94021640 con 0x7f4da40ffab0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.772+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da419e110 con 0x7f4da40ffab0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.774+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da4108610 con 0x7f4da40ffab0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.774+0000 7f4d8bfff700 1 -- 172.21.15.151:0/2375383007 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4d94004d20 con 0x7f4da40ffab0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.774+0000 7f4d8bfff700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4d8c06c4e0 0x7f4d8c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.774+0000 7f4d8bfff700 1 -- 172.21.15.151:0/2375383007 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4d9408cc90 con 0x7f4da40ffab0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.783+0000 7f4da27fc700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4d8c06c4e0 0x7f4d8c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.783+0000 7f4d8bfff700 1 -- 172.21.15.151:0/2375383007 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4d9405aea0 con 0x7f4da40ffab0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:13.784+0000 7f4da27fc700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4d8c06c4e0 0x7f4d8c06e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4da4069d80 tx=0x7f4d9800b3f0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.208+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4da4105600 con 0x7f4d8c06c4e0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.210+0000 7f4d8bfff700 1 -- 172.21.15.151:0/2375383007 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f4da4105600 con 0x7f4d8c06c4e0 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4d8c06c4e0 msgr2=0x7f4d8c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:14.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4d8c06c4e0 0x7f4d8c06e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4da4069d80 tx=0x7f4d9800b3f0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 msgr2=0x7f4da4069800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 0x7f4da4069800 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400bdd0 tx=0x7f4d9400beb0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 shutdown_connections 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f4d8c06c4e0 0x7f4d8c06e9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4da40ffab0 0x7f4da4069800 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 --2- 172.21.15.151:0/2375383007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da4100cb0 0x7f4da419d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 >> 172.21.15.151:0/2375383007 conn(0x7f4da40fb050 msgr2=0x7f4da4103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.212+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 shutdown_connections 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.213+0000 7f4da95ec700 1 -- 172.21.15.151:0/2375383007 wait complete. 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.332+0000 7f45b4370700 1 -- 172.21.15.151:0/3945859285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45ac101140 msgr2=0x7f45ac101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.332+0000 7f45b4370700 1 --2- 172.21.15.151:0/3945859285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45ac101140 0x7f45ac101560 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009910 tx=0x7f45a8009c20 comp rx=0 tx=0).stop 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.332+0000 7f45b4370700 1 -- 172.21.15.151:0/3945859285 shutdown_connections 2024-04-16T02:34:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.332+0000 7f45b4370700 1 --2- 172.21.15.151:0/3945859285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac0fec40 0x7f45ac0ff0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.332+0000 7f45b4370700 1 --2- 172.21.15.151:0/3945859285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45ac101140 0x7f45ac101560 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.332+0000 7f45b4370700 1 -- 172.21.15.151:0/3945859285 >> 172.21.15.151:0/3945859285 conn(0x7f45ac0fa7c0 msgr2=0x7f45ac0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.333+0000 7f45b4370700 1 -- 172.21.15.151:0/3945859285 shutdown_connections 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.333+0000 7f45b4370700 1 -- 172.21.15.151:0/3945859285 wait complete. 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.334+0000 7f45b4370700 1 Processor -- start 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.334+0000 7f45b4370700 1 -- start start 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.334+0000 7f45b4370700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac0fec40 0x7f45ac1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.334+0000 7f45b4370700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45ac101140 0x7f45ac198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.334+0000 7f45b190b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45ac101140 0x7f45ac198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.334+0000 7f45b210c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac0fec40 0x7f45ac1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.334+0000 7f45b210c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac0fec40 0x7f45ac1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:44356/0 (socket says 172.21.15.151:44356) 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.334+0000 7f45b210c700 1 -- 172.21.15.151:0/2864412189 learned_addr learned my addr 172.21.15.151:0/2864412189 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.334+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac199100 con 0x7f45ac101140 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.335+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac199240 con 0x7f45ac0fec40 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.335+0000 7f45b210c700 1 -- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45ac101140 msgr2=0x7f45ac198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.335+0000 7f45b210c700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45ac101140 0x7f45ac198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.335+0000 7f45b210c700 1 -- 172.21.15.151:0/2864412189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a80095f0 con 0x7f45ac0fec40 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.335+0000 7f45b190b700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45ac101140 0x7f45ac198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.335+0000 7f45b210c700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac0fec40 0x7f45ac1985a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f45a8000c00 tx=0x7f45a800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.335+0000 7f45a37fe700 1 -- 172.21.15.151:0/2864412189 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45a801d070 con 0x7f45ac0fec40 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.336+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac19dc90 con 0x7f45ac0fec40 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.336+0000 7f45a37fe700 1 -- 172.21.15.151:0/2864412189 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f45a8005dc0 con 0x7f45ac0fec40 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.337+0000 7f45a37fe700 1 -- 172.21.15.151:0/2864412189 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45a80226a0 con 0x7f45ac0fec40 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.337+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac19e0c0 con 0x7f45ac0fec40 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.338+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac10b9f0 con 0x7f45ac0fec40 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.338+0000 7f45a37fe700 1 -- 172.21.15.151:0/2864412189 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f45a800faf0 con 0x7f45ac0fec40 2024-04-16T02:34:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.339+0000 7f45a37fe700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f459806c2d0 0x7f459806e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:14.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.339+0000 7f45a37fe700 1 -- 172.21.15.151:0/2864412189 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f45a808c380 con 0x7f45ac0fec40 2024-04-16T02:34:14.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.339+0000 7f45b190b700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f459806c2d0 0x7f459806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:14.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.339+0000 7f45b190b700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f459806c2d0 0x7f459806e790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f459c011490 tx=0x7f459c009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:14.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.348+0000 7f45a37fe700 1 -- 172.21.15.151:0/2864412189 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f45a805a500 con 0x7f45ac0fec40 2024-04-16T02:34:14.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.724+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45ac109a60 con 0x7f459806c2d0 2024-04-16T02:34:14.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.725+0000 7f45a37fe700 1 -- 172.21.15.151:0/2864412189 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f45ac109a60 con 0x7f459806c2d0 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f459806c2d0 msgr2=0x7f459806e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f459806c2d0 0x7f459806e790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f459c011490 tx=0x7f459c009260 comp rx=0 tx=0).stop 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac0fec40 msgr2=0x7f45ac1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac0fec40 0x7f45ac1985a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f45a8000c00 tx=0x7f45a800f690 comp rx=0 tx=0).stop 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 shutdown_connections 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f459806c2d0 0x7f459806e790 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac0fec40 0x7f45ac1985a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 --2- 172.21.15.151:0/2864412189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45ac101140 0x7f45ac198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 >> 172.21.15.151:0/2864412189 conn(0x7f45ac0fa7c0 msgr2=0x7f45ac108340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 shutdown_connections 2024-04-16T02:34:14.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.728+0000 7f45b4370700 1 -- 172.21.15.151:0/2864412189 wait complete. 2024-04-16T02:34:14.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.845+0000 7f40bcd72700 1 -- 172.21.15.151:0/2982114177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40b81016a0 msgr2=0x7f40b8103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:14.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.845+0000 7f40bcd72700 1 --2- 172.21.15.151:0/2982114177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40b81016a0 0x7f40b8103a90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009910 tx=0x7f40a8009c20 comp rx=0 tx=0).stop 2024-04-16T02:34:14.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.845+0000 7f40bcd72700 1 -- 172.21.15.151:0/2982114177 shutdown_connections 2024-04-16T02:34:14.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.845+0000 7f40bcd72700 1 --2- 172.21.15.151:0/2982114177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b8103fd0 0x7f40b81063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.845+0000 7f40bcd72700 1 --2- 172.21.15.151:0/2982114177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40b81016a0 0x7f40b8103a90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.845+0000 7f40bcd72700 1 -- 172.21.15.151:0/2982114177 >> 172.21.15.151:0/2982114177 conn(0x7f40b80fb030 msgr2=0x7f40b80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:14.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.845+0000 7f40bcd72700 1 -- 172.21.15.151:0/2982114177 shutdown_connections 2024-04-16T02:34:14.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.845+0000 7f40bcd72700 1 -- 172.21.15.151:0/2982114177 wait complete. 2024-04-16T02:34:14.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.846+0000 7f40bcd72700 1 Processor -- start 2024-04-16T02:34:14.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.846+0000 7f40bcd72700 1 -- start start 2024-04-16T02:34:14.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40bcd72700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b81016a0 0x7f40b8071d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:14.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40bcd72700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40b8103fd0 0x7f40b8072280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:14.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40bcd72700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b80728a0 con 0x7f40b8103fd0 2024-04-16T02:34:14.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40bcd72700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b81a3db0 con 0x7f40b81016a0 2024-04-16T02:34:14.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40b5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40b8103fd0 0x7f40b8072280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:14.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40b659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b81016a0 0x7f40b8071d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:14.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40b5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40b8103fd0 0x7f40b8072280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50752/0 (socket says 172.21.15.151:50752) 2024-04-16T02:34:14.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40b5d9b700 1 -- 172.21.15.151:0/2625949036 learned_addr learned my addr 172.21.15.151:0/2625949036 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:14.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40b659c700 1 -- 172.21.15.151:0/2625949036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40b8103fd0 msgr2=0x7f40b8072280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:14.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40b659c700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40b8103fd0 0x7f40b8072280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:14.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.847+0000 7f40b659c700 1 -- 172.21.15.151:0/2625949036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a80095f0 con 0x7f40b81016a0 2024-04-16T02:34:14.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.848+0000 7f40b659c700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b81016a0 0x7f40b8071d40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f40a8005b40 tx=0x7f40a800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:14.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.848+0000 7f40a77fe700 1 -- 172.21.15.151:0/2625949036 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40a801d070 con 0x7f40b81016a0 2024-04-16T02:34:14.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.848+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b80ff770 con 0x7f40b81016a0 2024-04-16T02:34:14.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.848+0000 7f40a77fe700 1 -- 172.21.15.151:0/2625949036 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f40a8005dc0 con 0x7f40b81016a0 2024-04-16T02:34:14.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.848+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b80ffba0 con 0x7f40b81016a0 2024-04-16T02:34:14.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.848+0000 7f40a77fe700 1 -- 172.21.15.151:0/2625949036 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40a8022470 con 0x7f40b81016a0 2024-04-16T02:34:14.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.849+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4098005320 con 0x7f40b81016a0 2024-04-16T02:34:14.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.850+0000 7f40a77fe700 1 -- 172.21.15.151:0/2625949036 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f40a80225d0 con 0x7f40b81016a0 2024-04-16T02:34:14.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.850+0000 7f40a77fe700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f40a006c530 0x7f40a006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:14.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.850+0000 7f40a77fe700 1 -- 172.21.15.151:0/2625949036 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f40a808cdf0 con 0x7f40b81016a0 2024-04-16T02:34:14.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.850+0000 7f40b5d9b700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f40a006c530 0x7f40a006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:14.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.851+0000 7f40b5d9b700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f40a006c530 0x7f40a006e9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f40b8072e30 tx=0x7f40ac009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:14.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:14.859+0000 7f40a77fe700 1 -- 172.21.15.151:0/2625949036 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f40a80574d0 con 0x7f40b81016a0 2024-04-16T02:34:15.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.074+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4098000bf0 con 0x7f40a006c530 2024-04-16T02:34:15.097 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (5m) 70s ago 7m 25.1M - 0.25.0 c8568f914cd2 9f7a3dfb7aa4 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (7m) 70s ago 7m 7570k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (6m) 75s ago 6m 7386k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (7m) 70s ago 7m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (6m) 75s ago 6m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (5m) 70s ago 6m 82.6M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (96s) 70s ago 95s 16.6M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (90s) 70s ago 90s 18.3M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (88s) 75s ago 87s 15.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (93s) 75s ago 93s 12.8M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:9283,8765,8443 running (8m) 70s ago 8m 490M - 18.2.1 5be31c24972a 9e98c5bc45da 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (6m) 75s ago 6m 443M - 18.2.1 5be31c24972a 6795a633edf9 2024-04-16T02:34:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (8m) 70s ago 8m 47.6M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (6m) 75s ago 6m 36.1M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (7m) 70s ago 7m 16.2M - 1.5.0 0da6a335fe13 7748f5130613 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (6m) 75s ago 6m 16.3M - 1.5.0 0da6a335fe13 6a289489fe28 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (5m) 70s ago 5m 47.5M 3946M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (4m) 70s ago 4m 47.0M 3946M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (4m) 70s ago 4m 46.8M 3946M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (3m) 75s ago 3m 46.7M 4970M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (3m) 75s ago 3m 46.2M 4970M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (3m) 75s ago 3m 43.5M 4970M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (5m) 70s ago 6m 58.9M - 2.43.0 a07b618ecd1d 9a1c8cd0261d 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.093+0000 7f40a77fe700 1 -- 172.21.15.151:0/2625949036 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f4098000bf0 con 0x7f40a006c530 2024-04-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.096+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f40a006c530 msgr2=0x7f40a006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.096+0000 7f40bcd72700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f40a006c530 0x7f40a006e9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f40b8072e30 tx=0x7f40ac009260 comp rx=0 tx=0).stop 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.096+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b81016a0 msgr2=0x7f40b8071d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.096+0000 7f40bcd72700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b81016a0 0x7f40b8071d40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f40a8005b40 tx=0x7f40a800fa00 comp rx=0 tx=0).stop 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.096+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 shutdown_connections 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.096+0000 7f40bcd72700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f40a006c530 0x7f40a006e9f0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f40b8072e30 tx=0x7f40ac009260 comp rx=0 tx=0).stop 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.096+0000 7f40bcd72700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b81016a0 0x7f40b8071d40 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f40a8005b40 tx=0x7f40a800fa00 comp rx=0 tx=0).stop 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.096+0000 7f40bcd72700 1 --2- 172.21.15.151:0/2625949036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40b8103fd0 0x7f40b8072280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.096+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 >> 172.21.15.151:0/2625949036 conn(0x7f40b80fb030 msgr2=0x7f40b80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.097+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 shutdown_connections 2024-04-16T02:34:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.097+0000 7f40bcd72700 1 -- 172.21.15.151:0/2625949036 wait complete. 2024-04-16T02:34:15.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.210+0000 7f86ac228700 1 -- 172.21.15.151:0/349756455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a4100440 msgr2=0x7f86a41008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.210+0000 7f86ac228700 1 --2- 172.21.15.151:0/349756455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a4100440 0x7f86a41008a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8694009910 tx=0x7f8694009c20 comp rx=0 tx=0).stop 2024-04-16T02:34:15.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.210+0000 7f86ac228700 1 -- 172.21.15.151:0/349756455 shutdown_connections 2024-04-16T02:34:15.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.210+0000 7f86ac228700 1 --2- 172.21.15.151:0/349756455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a4100440 0x7f86a41008a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.210+0000 7f86ac228700 1 --2- 172.21.15.151:0/349756455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a40ff240 0x7f86a40ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.210+0000 7f86ac228700 1 -- 172.21.15.151:0/349756455 >> 172.21.15.151:0/349756455 conn(0x7f86a40fa7c0 msgr2=0x7f86a40fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:15.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.210+0000 7f86ac228700 1 -- 172.21.15.151:0/349756455 shutdown_connections 2024-04-16T02:34:15.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.210+0000 7f86ac228700 1 -- 172.21.15.151:0/349756455 wait complete. 2024-04-16T02:34:15.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.211+0000 7f86ac228700 1 Processor -- start 2024-04-16T02:34:15.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.211+0000 7f86ac228700 1 -- start start 2024-04-16T02:34:15.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86ac228700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a40ff240 0x7f86a4193f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:15.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86ac228700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a4100440 0x7f86a41944b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86a9fc4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a40ff240 0x7f86a4193f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86ac228700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4194ad0 con 0x7f86a4100440 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86a97c3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a4100440 0x7f86a41944b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86a97c3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a4100440 0x7f86a41944b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50774/0 (socket says 172.21.15.151:50774) 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86a97c3700 1 -- 172.21.15.151:0/3442956759 learned_addr learned my addr 172.21.15.151:0/3442956759 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86a9fc4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a40ff240 0x7f86a4193f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:44398/0 (socket says 172.21.15.151:44398) 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4194c10 con 0x7f86a40ff240 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86a97c3700 1 -- 172.21.15.151:0/3442956759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a40ff240 msgr2=0x7f86a4193f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86a97c3700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a40ff240 0x7f86a4193f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.212+0000 7f86a97c3700 1 -- 172.21.15.151:0/3442956759 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0009520 con 0x7f86a4100440 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.213+0000 7f86a9fc4700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a40ff240 0x7f86a4193f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.213+0000 7f86a97c3700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a4100440 0x7f86a41944b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8694005d10 tx=0x7f869401cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.213+0000 7f869affd700 1 -- 172.21.15.151:0/3442956759 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8694021070 con 0x7f86a4100440 2024-04-16T02:34:15.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.213+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86940095f0 con 0x7f86a4100440 2024-04-16T02:34:15.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.213+0000 7f869affd700 1 -- 172.21.15.151:0/3442956759 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8694005dc0 con 0x7f86a4100440 2024-04-16T02:34:15.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.213+0000 7f869affd700 1 -- 172.21.15.151:0/3442956759 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f869400f940 con 0x7f86a4100440 2024-04-16T02:34:15.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.214+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a4199880 con 0x7f86a4100440 2024-04-16T02:34:15.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.215+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a4107510 con 0x7f86a4100440 2024-04-16T02:34:15.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.216+0000 7f869affd700 1 -- 172.21.15.151:0/3442956759 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8694025070 con 0x7f86a4100440 2024-04-16T02:34:15.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.216+0000 7f869affd700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f869006c530 0x7f869006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:15.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.217+0000 7f869affd700 1 -- 172.21.15.151:0/3442956759 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8694090fa0 con 0x7f86a4100440 2024-04-16T02:34:15.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.217+0000 7f86a9fc4700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f869006c530 0x7f869006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:15.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.217+0000 7f86a9fc4700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f869006c530 0x7f869006e9f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f86a0009fd0 tx=0x7f86a0009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:15.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.222+0000 7f869affd700 1 -- 172.21.15.151:0/3442956759 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f869405f190 con 0x7f86a4100440 2024-04-16T02:34:15.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.508+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f86a4107700 con 0x7f86a4100440 2024-04-16T02:34:15.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.508+0000 7f869affd700 1 -- 172.21.15.151:0/3442956759 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f869405ed20 con 0x7f86a4100440 2024-04-16T02:34:15.511 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:34:15.511 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:34:15.511 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:34:15.512 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:34:15.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.511+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f869006c530 msgr2=0x7f869006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.511+0000 7f86ac228700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f869006c530 0x7f869006e9f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f86a0009fd0 tx=0x7f86a0009260 comp rx=0 tx=0).stop 2024-04-16T02:34:15.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.511+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a4100440 msgr2=0x7f86a41944b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.511+0000 7f86ac228700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a4100440 0x7f86a41944b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8694005d10 tx=0x7f869401cde0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.511+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 shutdown_connections 2024-04-16T02:34:15.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.511+0000 7f86ac228700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f869006c530 0x7f869006e9f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.511+0000 7f86ac228700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86a40ff240 0x7f86a4193f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.511+0000 7f86ac228700 1 --2- 172.21.15.151:0/3442956759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a4100440 0x7f86a41944b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.511+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 >> 172.21.15.151:0/3442956759 conn(0x7f86a40fa7c0 msgr2=0x7f86a40fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:15.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.512+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 shutdown_connections 2024-04-16T02:34:15.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.512+0000 7f86ac228700 1 -- 172.21.15.151:0/3442956759 wait complete. 2024-04-16T02:34:15.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.660+0000 7f1d09b97700 1 -- 172.21.15.151:0/275065409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 msgr2=0x7f1d04103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.660+0000 7f1d09b97700 1 --2- 172.21.15.151:0/275065409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 0x7f1d04103a30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009910 tx=0x7f1cf4009c20 comp rx=0 tx=0).stop 2024-04-16T02:34:15.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.660+0000 7f1d09b97700 1 -- 172.21.15.151:0/275065409 shutdown_connections 2024-04-16T02:34:15.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.660+0000 7f1d09b97700 1 --2- 172.21.15.151:0/275065409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d04103f70 0x7f1d04106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.660+0000 7f1d09b97700 1 --2- 172.21.15.151:0/275065409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 0x7f1d04103a30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.660+0000 7f1d09b97700 1 -- 172.21.15.151:0/275065409 >> 172.21.15.151:0/275065409 conn(0x7f1d040fb050 msgr2=0x7f1d040fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:15.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.660+0000 7f1d09b97700 1 -- 172.21.15.151:0/275065409 shutdown_connections 2024-04-16T02:34:15.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.660+0000 7f1d09b97700 1 -- 172.21.15.151:0/275065409 wait complete. 2024-04-16T02:34:15.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.661+0000 7f1d09b97700 1 Processor -- start 2024-04-16T02:34:15.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.661+0000 7f1d09b97700 1 -- start start 2024-04-16T02:34:15.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.661+0000 7f1d09b97700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 0x7f1d04100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:15.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.661+0000 7f1d09b97700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d04103f70 0x7f1d040ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:15.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.661+0000 7f1d09b97700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d04199f60 con 0x7f1d04101640 2024-04-16T02:34:15.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.661+0000 7f1d09b97700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0419a0a0 con 0x7f1d04103f70 2024-04-16T02:34:15.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 0x7f1d04100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:15.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d02ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d04103f70 0x7f1d040ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:15.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 0x7f1d04100fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50796/0 (socket says 172.21.15.151:50796) 2024-04-16T02:34:15.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d037fe700 1 -- 172.21.15.151:0/1133465907 learned_addr learned my addr 172.21.15.151:0/1133465907 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:15.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d037fe700 1 -- 172.21.15.151:0/1133465907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d04103f70 msgr2=0x7f1d040ff5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d037fe700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d04103f70 0x7f1d040ff5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d037fe700 1 -- 172.21.15.151:0/1133465907 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf40095f0 con 0x7f1d04101640 2024-04-16T02:34:15.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d037fe700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 0x7f1d04100fa0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4000c00 tx=0x7f1cf400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:15.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d00ff9700 1 -- 172.21.15.151:0/1133465907 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cf401d070 con 0x7f1d04101640 2024-04-16T02:34:15.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d00ff9700 1 -- 172.21.15.151:0/1133465907 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1cf4005dc0 con 0x7f1d04101640 2024-04-16T02:34:15.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d00ff9700 1 -- 172.21.15.151:0/1133465907 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cf40226a0 con 0x7f1d04101640 2024-04-16T02:34:15.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.662+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d040ffb30 con 0x7f1d04101640 2024-04-16T02:34:15.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.663+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d040fffa0 con 0x7f1d04101640 2024-04-16T02:34:15.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.664+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d04190520 con 0x7f1d04101640 2024-04-16T02:34:15.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.665+0000 7f1d00ff9700 1 -- 172.21.15.151:0/1133465907 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1cf40038f0 con 0x7f1d04101640 2024-04-16T02:34:15.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.665+0000 7f1d00ff9700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1cec06c4e0 0x7f1cec06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:15.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.666+0000 7f1d00ff9700 1 -- 172.21.15.151:0/1133465907 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1cf408cd90 con 0x7f1d04101640 2024-04-16T02:34:15.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.666+0000 7f1d02ffd700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1cec06c4e0 0x7f1cec06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:15.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.666+0000 7f1d02ffd700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1cec06c4e0 0x7f1cec06e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009cb0 tx=0x7f1cf8009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:15.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.670+0000 7f1d00ff9700 1 -- 172.21.15.151:0/1133465907 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1cf405afa0 con 0x7f1d04101640 2024-04-16T02:34:15.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.943+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d040008d0 con 0x7f1cec06c4e0 2024-04-16T02:34:15.947 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:34:15.947 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:34:15.947 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:34:15.947 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-16T02:34:15.947 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [], 2024-04-16T02:34:15.948 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "0/2 daemons upgraded", 2024-04-16T02:34:15.948 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f image on host smithi153", 2024-04-16T02:34:15.948 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:34:15.948 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:34:15.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.944+0000 7f1d00ff9700 1 -- 172.21.15.151:0/1133465907 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f1d040008d0 con 0x7f1cec06c4e0 2024-04-16T02:34:15.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.946+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1cec06c4e0 msgr2=0x7f1cec06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.946+0000 7f1d09b97700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1cec06c4e0 0x7f1cec06e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009cb0 tx=0x7f1cf8009260 comp rx=0 tx=0).stop 2024-04-16T02:34:15.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.946+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 msgr2=0x7f1d04100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:15.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.946+0000 7f1d09b97700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 0x7f1d04100fa0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4000c00 tx=0x7f1cf400f690 comp rx=0 tx=0).stop 2024-04-16T02:34:15.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.946+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 shutdown_connections 2024-04-16T02:34:15.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.946+0000 7f1d09b97700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f1cec06c4e0 0x7f1cec06e9a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.946+0000 7f1d09b97700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d04101640 0x7f1d04100fa0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.946+0000 7f1d09b97700 1 --2- 172.21.15.151:0/1133465907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d04103f70 0x7f1d040ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:15.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.946+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 >> 172.21.15.151:0/1133465907 conn(0x7f1d040fb050 msgr2=0x7f1d040fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:15.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.947+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 shutdown_connections 2024-04-16T02:34:15.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:15.947+0000 7f1d09b97700 1 -- 172.21.15.151:0/1133465907 wait complete. 2024-04-16T02:34:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:15 smithi153 ceph-mon[36747]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:15 smithi153 ceph-mon[36747]: from='client.24321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:15 smithi153 ceph-mon[36747]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:34:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:15 smithi153 ceph-mon[36747]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:15 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3442956759' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:34:16.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:15 smithi151 ceph-mon[29003]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:16.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:15 smithi151 ceph-mon[29003]: from='client.24321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:16.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:15 smithi151 ceph-mon[29003]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:34:16.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:15 smithi151 ceph-mon[29003]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:16.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:15 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3442956759' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:34:17.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:16 smithi153 ceph-mon[36747]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:17.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:16 smithi151 ceph-mon[29003]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:18.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:17 smithi153 ceph-mon[36747]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:18.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:17 smithi151 ceph-mon[29003]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:20.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:19 smithi151 ceph-mon[29003]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:20.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:19 smithi153 ceph-mon[36747]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:22.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:21 smithi151 ceph-mon[29003]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:34:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:21 smithi153 ceph-mon[36747]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:34:24.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:23 smithi151 ceph-mon[29003]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:23 smithi153 ceph-mon[36747]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:25 smithi151 ceph-mon[29003]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:34:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:25 smithi153 ceph-mon[36747]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:34:27.955 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:27 smithi153 ceph-mon[36747]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:27.955 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:27 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:34:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:27 smithi151 ceph-mon[29003]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:28.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:27 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:34:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:29 smithi151 ceph-mon[29003]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:30.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:29 smithi153 ceph-mon[36747]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:31.173 INFO:tasks.workunit.client.0.smithi151.stderr:Note: switching to '3f7db60e5ba4c4d053b594b7ec58703d2833c55b'. 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr: 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr:state without impacting any branches by switching back to a branch. 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr: 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr: 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr: git switch -c 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr: 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr:Or undo this operation with: 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr: 2024-04-16T02:34:31.174 INFO:tasks.workunit.client.0.smithi151.stderr: git switch - 2024-04-16T02:34:31.175 INFO:tasks.workunit.client.0.smithi151.stderr: 2024-04-16T02:34:31.175 INFO:tasks.workunit.client.0.smithi151.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-16T02:34:31.175 INFO:tasks.workunit.client.0.smithi151.stderr: 2024-04-16T02:34:31.175 INFO:tasks.workunit.client.0.smithi151.stderr:HEAD is now at 3f7db60e5ba Merge pull request #56767 from adk3798/cephadm-log-only-match 2024-04-16T02:34:31.176 DEBUG:teuthology.orchestra.run.smithi151:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-16T02:34:31.407 INFO:tasks.workunit.client.0.smithi151.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-16T02:34:31.409 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-16T02:34:31.409 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-16T02:34:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:31 smithi151 ceph-mon[29003]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:34:32.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:31 smithi153 ceph-mon[36747]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:34:32.818 INFO:tasks.workunit.client.1.smithi153.stderr:Updating files: 91% (11415/12482) Updating files: 92% (11484/12482) Updating files: 93% (11609/12482) Updating files: 94% (11734/12482) Updating files: 95% (11858/12482) Updating files: 96% (11983/12482) Updating files: 97% (12108/12482) Updating files: 98% (12233/12482) Updating files: 99% (12358/12482) Updating files: 100% (12482/12482) Updating files: 100% (12482/12482), done. 2024-04-16T02:34:33.415 INFO:tasks.workunit.client.1.smithi153.stderr:Note: switching to '3f7db60e5ba4c4d053b594b7ec58703d2833c55b'. 2024-04-16T02:34:33.415 INFO:tasks.workunit.client.1.smithi153.stderr: 2024-04-16T02:34:33.415 INFO:tasks.workunit.client.1.smithi153.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-16T02:34:33.415 INFO:tasks.workunit.client.1.smithi153.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-16T02:34:33.415 INFO:tasks.workunit.client.1.smithi153.stderr:state without impacting any branches by switching back to a branch. 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr: 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr: 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr: git switch -c 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr: 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr:Or undo this operation with: 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr: 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr: git switch - 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr: 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr: 2024-04-16T02:34:33.416 INFO:tasks.workunit.client.1.smithi153.stderr:HEAD is now at 3f7db60e5ba Merge pull request #56767 from adk3798/cephadm-log-only-match 2024-04-16T02:34:33.417 DEBUG:teuthology.orchestra.run.smithi153:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-16T02:34:33.479 INFO:tasks.workunit.client.1.smithi153.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-16T02:34:33.482 INFO:tasks.workunit.client.1.smithi153.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-16T02:34:33.482 INFO:tasks.workunit.client.1.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-16T02:34:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:33 smithi151 ceph-mon[29003]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:34.254 INFO:tasks.workunit.client.1.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-16T02:34:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:33 smithi153 ceph-mon[36747]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T02:34:34.316 INFO:tasks.workunit.client.1.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-16T02:34:34.358 INFO:tasks.workunit.client.1.smithi153.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-16T02:34:34.359 INFO:tasks.workunit.client.1.smithi153.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-16T02:34:34.360 INFO:tasks.workunit.client.1.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-16T02:34:34.400 INFO:tasks.workunit.client.1.smithi153.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-16T02:34:34.404 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-16T02:34:34.404 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-16T02:34:34.461 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-16T02:34:34.464 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-16T02:34:34.465 DEBUG:teuthology.orchestra.run.smithi153:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7db60e5ba4c4d053b594b7ec58703d2833c55b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-16T02:34:34.534 INFO:tasks.workunit.client.1.smithi153.stderr:+ mkdir -p fsstress 2024-04-16T02:34:34.536 INFO:tasks.workunit.client.1.smithi153.stderr:+ pushd fsstress 2024-04-16T02:34:34.538 INFO:tasks.workunit.client.1.smithi153.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-16T02:34:34.538 INFO:tasks.workunit.client.1.smithi153.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-16T02:34:35.405 INFO:tasks.workunit.client.1.smithi153.stderr:+ tar xzf ltp-full.tgz 2024-04-16T02:34:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:35 smithi153 ceph-mon[36747]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:34:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:35 smithi151 ceph-mon[29003]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-16T02:34:37.149 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-16T02:34:37.230 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-16T02:34:37.274 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-16T02:34:37.276 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-16T02:34:37.276 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-16T02:34:37.317 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-16T02:34:37.321 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T02:34:37.321 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-16T02:34:37.337 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-16T02:34:37.339 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-16T02:34:37.339 DEBUG:teuthology.orchestra.run.smithi151:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7db60e5ba4c4d053b594b7ec58703d2833c55b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-16T02:34:37.405 INFO:tasks.workunit.client.0.smithi151.stderr:+ mkdir -p fsstress 2024-04-16T02:34:37.408 INFO:tasks.workunit.client.0.smithi151.stderr:+ pushd fsstress 2024-04-16T02:34:37.409 INFO:tasks.workunit.client.0.smithi151.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-16T02:34:37.410 INFO:tasks.workunit.client.0.smithi151.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-16T02:34:38.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:37 smithi153 ceph-mon[36747]: pgmap v242: 65 pgs: 65 active+clean; 2.2 MiB data, 172 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 143 KiB/s wr, 14 op/s 2024-04-16T02:34:38.255 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:37 smithi151 ceph-mon[29003]: pgmap v242: 65 pgs: 65 active+clean; 2.2 MiB data, 172 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 143 KiB/s wr, 14 op/s 2024-04-16T02:34:39.255 INFO:tasks.workunit.client.0.smithi151.stderr:+ tar xzf ltp-full.tgz 2024-04-16T02:34:40.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:39 smithi151 ceph-mon[29003]: pgmap v243: 65 pgs: 65 active+clean; 14 MiB data, 200 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-04-16T02:34:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:39 smithi153 ceph-mon[36747]: pgmap v243: 65 pgs: 65 active+clean; 14 MiB data, 200 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-04-16T02:34:42.202 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:41 smithi153 ceph-mon[36747]: pgmap v244: 65 pgs: 65 active+clean; 19 MiB data, 216 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-04-16T02:34:42.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:41 smithi151 ceph-mon[29003]: pgmap v244: 65 pgs: 65 active+clean; 19 MiB data, 216 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-04-16T02:34:43.098 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:42 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:34:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:42 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:34:44.102 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:43 smithi153 ceph-mon[36747]: pgmap v245: 65 pgs: 65 active+clean; 33 MiB data, 289 MiB used, 536 GiB / 536 GiB avail; 585 KiB/s rd, 2.8 MiB/s wr, 196 op/s 2024-04-16T02:34:44.103 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:44.103 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:34:44.103 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:34:44.103 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:43 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:34:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:43 smithi151 ceph-mon[29003]: pgmap v245: 65 pgs: 65 active+clean; 33 MiB data, 289 MiB used, 536 GiB / 536 GiB avail; 585 KiB/s rd, 2.8 MiB/s wr, 196 op/s 2024-04-16T02:34:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:34:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:34:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:43 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:34:45.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:44 smithi151 ceph-mon[29003]: Upgrade: Updating mgr.smithi153.esbtzf 2024-04-16T02:34:45.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:44 smithi151 ceph-mon[29003]: Deploying daemon mgr.smithi153.esbtzf on smithi153 2024-04-16T02:34:45.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:44 smithi153 ceph-mon[36747]: Upgrade: Updating mgr.smithi153.esbtzf 2024-04-16T02:34:45.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:44 smithi153 ceph-mon[36747]: Deploying daemon mgr.smithi153.esbtzf on smithi153 2024-04-16T02:34:46.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.093+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/3840193810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3bc0ffab0 msgr2=0x7fb3bc0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:46.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.093+0000 7fb3c2ac9700 1 --2- 172.21.15.151:0/3840193810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3bc0ffab0 0x7fb3bc0ffed0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009910 tx=0x7fb3b0009c20 comp rx=0 tx=0).stop 2024-04-16T02:34:46.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.093+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/3840193810 shutdown_connections 2024-04-16T02:34:46.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.093+0000 7fb3c2ac9700 1 --2- 172.21.15.151:0/3840193810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3bc100cb0 0x7fb3bc101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.093+0000 7fb3c2ac9700 1 --2- 172.21.15.151:0/3840193810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3bc0ffab0 0x7fb3bc0ffed0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.093+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/3840193810 >> 172.21.15.151:0/3840193810 conn(0x7fb3bc0fb050 msgr2=0x7fb3bc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:46.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.093+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/3840193810 shutdown_connections 2024-04-16T02:34:46.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.093+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/3840193810 wait complete. 2024-04-16T02:34:46.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.094+0000 7fb3c2ac9700 1 Processor -- start 2024-04-16T02:34:46.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.094+0000 7fb3c2ac9700 1 -- start start 2024-04-16T02:34:46.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.094+0000 7fb3c2ac9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3bc0ffab0 0x7fb3bc103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:46.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.094+0000 7fb3c2ac9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3bc100cb0 0x7fb3bc1020b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:46.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.094+0000 7fb3c2ac9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc199fa0 con 0x7fb3bc0ffab0 2024-04-16T02:34:46.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.094+0000 7fb3c2ac9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc19a0e0 con 0x7fb3bc100cb0 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.094+0000 7fb3bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3bc100cb0 0x7fb3bc1020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.094+0000 7fb3bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3bc100cb0 0x7fb3bc1020b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:37440/0 (socket says 172.21.15.151:37440) 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.094+0000 7fb3bbfff700 1 -- 172.21.15.151:0/1625080036 learned_addr learned my addr 172.21.15.151:0/1625080036 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3c0865700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3bc0ffab0 0x7fb3bc103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3bbfff700 1 -- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3bc0ffab0 msgr2=0x7fb3bc103a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3bbfff700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3bc0ffab0 0x7fb3bc103a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3bbfff700 1 -- 172.21.15.151:0/1625080036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b00095f0 con 0x7fb3bc100cb0 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3c0865700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3bc0ffab0 0x7fb3bc103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3bbfff700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3bc100cb0 0x7fb3bc1020b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3a800e920 tx=0x7fb3a800ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3b9ffb700 1 -- 172.21.15.151:0/1625080036 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a8009790 con 0x7fb3bc100cb0 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc19a360 con 0x7fb3bc100cb0 2024-04-16T02:34:46.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3b9ffb700 1 -- 172.21.15.151:0/1625080036 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb3a800cba0 con 0x7fb3bc100cb0 2024-04-16T02:34:46.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3b9ffb700 1 -- 172.21.15.151:0/1625080036 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a8018630 con 0x7fb3bc100cb0 2024-04-16T02:34:46.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.095+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc19a8b0 con 0x7fb3bc100cb0 2024-04-16T02:34:46.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.097+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc108610 con 0x7fb3bc100cb0 2024-04-16T02:34:46.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.097+0000 7fb3b9ffb700 1 -- 172.21.15.151:0/1625080036 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb3a8010430 con 0x7fb3bc100cb0 2024-04-16T02:34:46.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.098+0000 7fb3b9ffb700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb3ac06c4e0 0x7fb3ac06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:46.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.098+0000 7fb3b9ffb700 1 -- 172.21.15.151:0/1625080036 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb3a8014070 con 0x7fb3bc100cb0 2024-04-16T02:34:46.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.098+0000 7fb3c0865700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb3ac06c4e0 0x7fb3ac06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:46.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.098+0000 7fb3c0865700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb3ac06c4e0 0x7fb3ac06e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009fd0 tx=0x7fb3b0005fb0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:46.105 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.103+0000 7fb3b9ffb700 1 -- 172.21.15.151:0/1625080036 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb3a80520e0 con 0x7fb3bc100cb0 2024-04-16T02:34:46.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:45 smithi151 ceph-mon[29003]: pgmap v246: 65 pgs: 65 active+clean; 40 MiB data, 311 MiB used, 536 GiB / 536 GiB avail; 585 KiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-04-16T02:34:46.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:45 smithi153 ceph-mon[36747]: pgmap v246: 65 pgs: 65 active+clean; 40 MiB data, 311 MiB used, 536 GiB / 536 GiB avail; 585 KiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-04-16T02:34:46.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.403+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3bc069390 con 0x7fb3ac06c4e0 2024-04-16T02:34:46.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.404+0000 7fb3b9ffb700 1 -- 172.21.15.151:0/1625080036 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fb3bc069390 con 0x7fb3ac06c4e0 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb3ac06c4e0 msgr2=0x7fb3ac06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb3ac06c4e0 0x7fb3ac06e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009fd0 tx=0x7fb3b0005fb0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3bc100cb0 msgr2=0x7fb3bc1020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3bc100cb0 0x7fb3bc1020b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3a800e920 tx=0x7fb3a800ece0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 shutdown_connections 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb3ac06c4e0 0x7fb3ac06e9a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3bc0ffab0 0x7fb3bc103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 --2- 172.21.15.151:0/1625080036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3bc100cb0 0x7fb3bc1020b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 >> 172.21.15.151:0/1625080036 conn(0x7fb3bc0fb050 msgr2=0x7fb3bc068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.407+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 shutdown_connections 2024-04-16T02:34:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.408+0000 7fb3c2ac9700 1 -- 172.21.15.151:0/1625080036 wait complete. 2024-04-16T02:34:46.424 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:34:46.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.550+0000 7f67bce9d700 1 -- 172.21.15.151:0/1270090500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 msgr2=0x7f67b8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:46.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.550+0000 7f67bce9d700 1 --2- 172.21.15.151:0/1270090500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 0x7f67b8102310 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009910 tx=0x7f67ac009c20 comp rx=0 tx=0).stop 2024-04-16T02:34:46.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.550+0000 7f67bce9d700 1 -- 172.21.15.151:0/1270090500 shutdown_connections 2024-04-16T02:34:46.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.550+0000 7f67bce9d700 1 --2- 172.21.15.151:0/1270090500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 0x7f67b8102310 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.550+0000 7f67bce9d700 1 --2- 172.21.15.151:0/1270090500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b80feed0 0x7f67b80ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.550+0000 7f67bce9d700 1 -- 172.21.15.151:0/1270090500 >> 172.21.15.151:0/1270090500 conn(0x7f67b80fa810 msgr2=0x7f67b80fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.551+0000 7f67bce9d700 1 -- 172.21.15.151:0/1270090500 shutdown_connections 2024-04-16T02:34:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.551+0000 7f67bce9d700 1 -- 172.21.15.151:0/1270090500 wait complete. 2024-04-16T02:34:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.551+0000 7f67bce9d700 1 Processor -- start 2024-04-16T02:34:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.551+0000 7f67bce9d700 1 -- start start 2024-04-16T02:34:46.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67bce9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b80feed0 0x7f67b81963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:46.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67bce9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 0x7f67b8196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:46.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67bce9d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b8196f50 con 0x7f67b8101eb0 2024-04-16T02:34:46.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67bce9d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b8197090 con 0x7f67b80feed0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67b5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 0x7f67b8196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67b5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 0x7f67b8196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33004/0 (socket says 172.21.15.151:33004) 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67b5d9b700 1 -- 172.21.15.151:0/2534274654 learned_addr learned my addr 172.21.15.151:0/2534274654 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67b659c700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b80feed0 0x7f67b81963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67b5d9b700 1 -- 172.21.15.151:0/2534274654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b80feed0 msgr2=0x7f67b81963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67b5d9b700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b80feed0 0x7f67b81963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.552+0000 7f67b5d9b700 1 -- 172.21.15.151:0/2534274654 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac0095f0 con 0x7f67b8101eb0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.553+0000 7f67b5d9b700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 0x7f67b8196930 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00fa00 tx=0x7f67ac00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.553+0000 7f67a77fe700 1 -- 172.21.15.151:0/2534274654 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67ac01d070 con 0x7f67b8101eb0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.553+0000 7f67a77fe700 1 -- 172.21.15.151:0/2534274654 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f67ac00fe90 con 0x7f67b8101eb0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.553+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b819bae0 con 0x7f67b8101eb0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.553+0000 7f67a77fe700 1 -- 172.21.15.151:0/2534274654 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67ac0226a0 con 0x7f67b8101eb0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.553+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b819bfd0 con 0x7f67b8101eb0 2024-04-16T02:34:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.554+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6798005320 con 0x7f67b8101eb0 2024-04-16T02:34:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.555+0000 7f67a77fe700 1 -- 172.21.15.151:0/2534274654 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f67ac0050d0 con 0x7f67b8101eb0 2024-04-16T02:34:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.556+0000 7f67a77fe700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f67a006c4e0 0x7f67a006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.556+0000 7f67a77fe700 1 -- 172.21.15.151:0/2534274654 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f67ac08cb90 con 0x7f67b8101eb0 2024-04-16T02:34:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.556+0000 7f67b659c700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f67a006c4e0 0x7f67a006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:46.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.556+0000 7f67b659c700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f67a006c4e0 0x7f67a006e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f67b8068b70 tx=0x7f67a8009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:46.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.563+0000 7f67a77fe700 1 -- 172.21.15.151:0/2534274654 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f67ac05ae20 con 0x7f67b8101eb0 2024-04-16T02:34:46.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.883+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6798000bf0 con 0x7f67a006c4e0 2024-04-16T02:34:46.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.884+0000 7f67a77fe700 1 -- 172.21.15.151:0/2534274654 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f6798000bf0 con 0x7f67a006c4e0 2024-04-16T02:34:46.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f67a006c4e0 msgr2=0x7f67a006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:46.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f67a006c4e0 0x7f67a006e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f67b8068b70 tx=0x7f67a8009260 comp rx=0 tx=0).stop 2024-04-16T02:34:46.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 msgr2=0x7f67b8196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:46.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 0x7f67b8196930 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00fa00 tx=0x7f67ac00fa30 comp rx=0 tx=0).stop 2024-04-16T02:34:46.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 shutdown_connections 2024-04-16T02:34:46.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f67a006c4e0 0x7f67a006e9a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b80feed0 0x7f67b81963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 --2- 172.21.15.151:0/2534274654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67b8101eb0 0x7f67b8196930 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:46.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 >> 172.21.15.151:0/2534274654 conn(0x7f67b80fa810 msgr2=0x7f67b81050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:46.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 shutdown_connections 2024-04-16T02:34:46.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:46.888+0000 7f67bce9d700 1 -- 172.21.15.151:0/2534274654 wait complete. 2024-04-16T02:34:47.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.049+0000 7fb918f7f700 1 -- 172.21.15.151:0/511234191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9140ffad0 msgr2=0x7fb9140ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:47.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.049+0000 7fb918f7f700 1 --2- 172.21.15.151:0/511234191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9140ffad0 0x7fb9140ffef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb904009910 tx=0x7fb904009c20 comp rx=0 tx=0).stop 2024-04-16T02:34:47.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.049+0000 7fb918f7f700 1 -- 172.21.15.151:0/511234191 shutdown_connections 2024-04-16T02:34:47.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.049+0000 7fb918f7f700 1 --2- 172.21.15.151:0/511234191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb914100cd0 0x7fb914101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.049+0000 7fb918f7f700 1 --2- 172.21.15.151:0/511234191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9140ffad0 0x7fb9140ffef0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.049+0000 7fb918f7f700 1 -- 172.21.15.151:0/511234191 >> 172.21.15.151:0/511234191 conn(0x7fb9140fb050 msgr2=0x7fb9140fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.049+0000 7fb918f7f700 1 -- 172.21.15.151:0/511234191 shutdown_connections 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.049+0000 7fb918f7f700 1 -- 172.21.15.151:0/511234191 wait complete. 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.050+0000 7fb918f7f700 1 Processor -- start 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.050+0000 7fb918f7f700 1 -- start start 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb918f7f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9140ffad0 0x7fb9140698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb918f7f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb914100cd0 0x7fb91419b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb918f7f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91419b680 con 0x7fb9140ffad0 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb918f7f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91419b7c0 con 0x7fb914100cd0 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb91259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9140ffad0 0x7fb9140698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb911d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb914100cd0 0x7fb91419b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb91259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9140ffad0 0x7fb9140698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33028/0 (socket says 172.21.15.151:33028) 2024-04-16T02:34:47.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb91259c700 1 -- 172.21.15.151:0/3897275076 learned_addr learned my addr 172.21.15.151:0/3897275076 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb91259c700 1 -- 172.21.15.151:0/3897275076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb914100cd0 msgr2=0x7fb91419b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb91259c700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb914100cd0 0x7fb91419b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb91259c700 1 -- 172.21.15.151:0/3897275076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9040095f0 con 0x7fb9140ffad0 2024-04-16T02:34:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.051+0000 7fb91259c700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9140ffad0 0x7fb9140698f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb904005b40 tx=0x7fb90400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.052+0000 7fb9037fe700 1 -- 172.21.15.151:0/3897275076 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb90401d070 con 0x7fb9140ffad0 2024-04-16T02:34:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.052+0000 7fb9037fe700 1 -- 172.21.15.151:0/3897275076 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb904005dc0 con 0x7fb9140ffad0 2024-04-16T02:34:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.052+0000 7fb9037fe700 1 -- 172.21.15.151:0/3897275076 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb904022470 con 0x7fb9140ffad0 2024-04-16T02:34:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.052+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb91419ba40 con 0x7fb9140ffad0 2024-04-16T02:34:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.052+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91419bf30 con 0x7fb9140ffad0 2024-04-16T02:34:47.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.053+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb914108630 con 0x7fb9140ffad0 2024-04-16T02:34:47.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.054+0000 7fb9037fe700 1 -- 172.21.15.151:0/3897275076 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb904003680 con 0x7fb9140ffad0 2024-04-16T02:34:47.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.054+0000 7fb9037fe700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb8fc06c4e0 0x7fb8fc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:47.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.054+0000 7fb9037fe700 1 -- 172.21.15.151:0/3897275076 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb90408cd30 con 0x7fb9140ffad0 2024-04-16T02:34:47.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.055+0000 7fb911d9b700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb8fc06c4e0 0x7fb8fc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:47.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.055+0000 7fb911d9b700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb8fc06c4e0 0x7fb8fc06e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb908009c60 tx=0x7fb908009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:47.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.062+0000 7fb9037fe700 1 -- 172.21.15.151:0/3897275076 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb90405afc0 con 0x7fb9140ffad0 2024-04-16T02:34:47.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.361+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb914105620 con 0x7fb8fc06c4e0 2024-04-16T02:34:47.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.380+0000 7fb9037fe700 1 -- 172.21.15.151:0/3897275076 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7fb914105620 con 0x7fb8fc06c4e0 2024-04-16T02:34:47.382 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:34:47.382 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (6m) 102s ago 7m 25.1M - 0.25.0 c8568f914cd2 9f7a3dfb7aa4 2024-04-16T02:34:47.382 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (7m) 102s ago 7m 7570k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (6m) 107s ago 6m 7386k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (7m) 102s ago 7m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (6m) 107s ago 6m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (6m) 102s ago 7m 82.6M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (2m) 102s ago 2m 16.6M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (2m) 102s ago 2m 18.3M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (2m) 107s ago 119s 15.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (2m) 107s ago 2m 12.8M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:9283,8765,8443 running (9m) 102s ago 9m 490M - 18.2.1 5be31c24972a 9e98c5bc45da 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (6m) 107s ago 6m 443M - 18.2.1 5be31c24972a 6795a633edf9 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (9m) 102s ago 9m 47.6M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (6m) 107s ago 6m 36.1M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (7m) 102s ago 7m 16.2M - 1.5.0 0da6a335fe13 7748f5130613 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (6m) 107s ago 6m 16.3M - 1.5.0 0da6a335fe13 6a289489fe28 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (5m) 102s ago 5m 47.5M 3946M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:34:47.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (5m) 102s ago 5m 47.0M 3946M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:34:47.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (4m) 102s ago 4m 46.8M 3946M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:34:47.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (4m) 107s ago 4m 46.7M 4970M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:34:47.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (4m) 107s ago 4m 46.2M 4970M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:34:47.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (3m) 107s ago 3m 43.5M 4970M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:34:47.384 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (5m) 102s ago 7m 58.9M - 2.43.0 a07b618ecd1d 9a1c8cd0261d 2024-04-16T02:34:47.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb8fc06c4e0 msgr2=0x7fb8fc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:47.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb8fc06c4e0 0x7fb8fc06e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb908009c60 tx=0x7fb908009260 comp rx=0 tx=0).stop 2024-04-16T02:34:47.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9140ffad0 msgr2=0x7fb9140698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:47.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9140ffad0 0x7fb9140698f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb904005b40 tx=0x7fb90400fa00 comp rx=0 tx=0).stop 2024-04-16T02:34:47.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 shutdown_connections 2024-04-16T02:34:47.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7fb8fc06c4e0 0x7fb8fc06e9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9140ffad0 0x7fb9140698f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 --2- 172.21.15.151:0/3897275076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb914100cd0 0x7fb91419b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 >> 172.21.15.151:0/3897275076 conn(0x7fb9140fb050 msgr2=0x7fb914103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:47.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 shutdown_connections 2024-04-16T02:34:47.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.383+0000 7fb918f7f700 1 -- 172.21.15.151:0/3897275076 wait complete. 2024-04-16T02:34:47.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.498+0000 7f292941a700 1 -- 172.21.15.151:0/3243515432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2924100bf0 msgr2=0x7f2924101010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:47.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.498+0000 7f292941a700 1 --2- 172.21.15.151:0/3243515432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2924100bf0 0x7f2924101010 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2914009910 tx=0x7f2914009c20 comp rx=0 tx=0).stop 2024-04-16T02:34:47.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.498+0000 7f292941a700 1 -- 172.21.15.151:0/3243515432 shutdown_connections 2024-04-16T02:34:47.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.498+0000 7f292941a700 1 --2- 172.21.15.151:0/3243515432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2924101df0 0x7f2924102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.498+0000 7f292941a700 1 --2- 172.21.15.151:0/3243515432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2924100bf0 0x7f2924101010 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.498+0000 7f292941a700 1 -- 172.21.15.151:0/3243515432 >> 172.21.15.151:0/3243515432 conn(0x7f29240fc190 msgr2=0x7f29240fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:47.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.498+0000 7f292941a700 1 -- 172.21.15.151:0/3243515432 shutdown_connections 2024-04-16T02:34:47.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.499+0000 7f292941a700 1 -- 172.21.15.151:0/3243515432 wait complete. 2024-04-16T02:34:47.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.499+0000 7f292941a700 1 Processor -- start 2024-04-16T02:34:47.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.499+0000 7f292941a700 1 -- start start 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f292941a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2924100bf0 0x7f2924196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f292941a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2924101df0 0x7f29241968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f292941a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2924196ee0 con 0x7f2924101df0 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f292941a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2924197020 con 0x7f2924100bf0 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f2922ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2924100bf0 0x7f2924196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f2922ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2924100bf0 0x7f2924196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:39286/0 (socket says 172.21.15.151:39286) 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f2922ffd700 1 -- 172.21.15.151:0/3461908785 learned_addr learned my addr 172.21.15.151:0/3461908785 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f29227fc700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2924101df0 0x7f29241968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f2922ffd700 1 -- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2924101df0 msgr2=0x7f29241968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f2922ffd700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2924101df0 0x7f29241968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f2922ffd700 1 -- 172.21.15.151:0/3461908785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29140095f0 con 0x7f2924100bf0 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f2922ffd700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2924100bf0 0x7f2924196380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2914000c00 tx=0x7f291400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.500+0000 7f290bfff700 1 -- 172.21.15.151:0/3461908785 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f291401d070 con 0x7f2924100bf0 2024-04-16T02:34:47.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.501+0000 7f290bfff700 1 -- 172.21.15.151:0/3461908785 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2914005dc0 con 0x7f2924100bf0 2024-04-16T02:34:47.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.501+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f292419ba70 con 0x7f2924100bf0 2024-04-16T02:34:47.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.501+0000 7f290bfff700 1 -- 172.21.15.151:0/3461908785 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2914022470 con 0x7f2924100bf0 2024-04-16T02:34:47.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.501+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f292419bf60 con 0x7f2924100bf0 2024-04-16T02:34:47.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.502+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2924109750 con 0x7f2924100bf0 2024-04-16T02:34:47.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.502+0000 7f290bfff700 1 -- 172.21.15.151:0/3461908785 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2914003810 con 0x7f2924100bf0 2024-04-16T02:34:47.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.503+0000 7f290bfff700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f290c06c4e0 0x7f290c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:47.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.503+0000 7f290bfff700 1 -- 172.21.15.151:0/3461908785 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f291408ce60 con 0x7f2924100bf0 2024-04-16T02:34:47.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.503+0000 7f29227fc700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f290c06c4e0 0x7f290c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:47.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.504+0000 7f29227fc700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f290c06c4e0 0x7f290c06e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2918009c60 tx=0x7f2918009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:47.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.508+0000 7f290bfff700 1 -- 172.21.15.151:0/3461908785 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f291405b010 con 0x7f2924100bf0 2024-04-16T02:34:47.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.929+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2924109940 con 0x7f2924100bf0 2024-04-16T02:34:47.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.929+0000 7f290bfff700 1 -- 172.21.15.151:0/3461908785 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f291405aba0 con 0x7f2924100bf0 2024-04-16T02:34:47.931 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:34:47.931 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:34:47.931 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:34:47.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:34:47.932 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:34:47.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f290c06c4e0 msgr2=0x7f290c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:47.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f290c06c4e0 0x7f290c06e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2918009c60 tx=0x7f2918009260 comp rx=0 tx=0).stop 2024-04-16T02:34:47.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2924100bf0 msgr2=0x7f2924196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:47.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2924100bf0 0x7f2924196380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2914000c00 tx=0x7f291400f690 comp rx=0 tx=0).stop 2024-04-16T02:34:47.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 shutdown_connections 2024-04-16T02:34:47.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f290c06c4e0 0x7f290c06e9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2924100bf0 0x7f2924196380 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 --2- 172.21.15.151:0/3461908785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2924101df0 0x7f29241968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:47.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 >> 172.21.15.151:0/3461908785 conn(0x7f29240fc190 msgr2=0x7f2924105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:47.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 shutdown_connections 2024-04-16T02:34:47.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:47.932+0000 7f292941a700 1 -- 172.21.15.151:0/3461908785 wait complete. 2024-04-16T02:34:48.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.080+0000 7f57788dc700 1 -- 172.21.15.151:0/1858775619 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57740ff240 msgr2=0x7f57740ff660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:48.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.080+0000 7f57788dc700 1 --2- 172.21.15.151:0/1858775619 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57740ff240 0x7f57740ff660 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f576c00b0a0 tx=0x7f576c00b3b0 comp rx=0 tx=0).stop 2024-04-16T02:34:48.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.080+0000 7f57788dc700 1 -- 172.21.15.151:0/1858775619 shutdown_connections 2024-04-16T02:34:48.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.080+0000 7f57788dc700 1 --2- 172.21.15.151:0/1858775619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5774100440 0x7f57741008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:48.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.080+0000 7f57788dc700 1 --2- 172.21.15.151:0/1858775619 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57740ff240 0x7f57740ff660 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:48.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.080+0000 7f57788dc700 1 -- 172.21.15.151:0/1858775619 >> 172.21.15.151:0/1858775619 conn(0x7f57740fa7c0 msgr2=0x7f57740fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:48.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.080+0000 7f57788dc700 1 -- 172.21.15.151:0/1858775619 shutdown_connections 2024-04-16T02:34:48.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.080+0000 7f57788dc700 1 -- 172.21.15.151:0/1858775619 wait complete. 2024-04-16T02:34:48.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f57788dc700 1 Processor -- start 2024-04-16T02:34:48.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f57788dc700 1 -- start start 2024-04-16T02:34:48.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f57788dc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740ff240 0x7f57741963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:48.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f57788dc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5774100440 0x7f57741968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:48.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f57788dc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5774196f00 con 0x7f5774100440 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f57788dc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5774197040 con 0x7f57740ff240 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f5771d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5774100440 0x7f57741968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f5771d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5774100440 0x7f57741968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33050/0 (socket says 172.21.15.151:33050) 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f5771d9b700 1 -- 172.21.15.151:0/2514936959 learned_addr learned my addr 172.21.15.151:0/2514936959 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.081+0000 7f577259c700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740ff240 0x7f57741963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.082+0000 7f5771d9b700 1 -- 172.21.15.151:0/2514936959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740ff240 msgr2=0x7f57741963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.082+0000 7f5771d9b700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740ff240 0x7f57741963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.082+0000 7f5771d9b700 1 -- 172.21.15.151:0/2514936959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c009d00 con 0x7f5774100440 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.082+0000 7f5771d9b700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5774100440 0x7f57741968e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f576400ea30 tx=0x7f576400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.082+0000 7f576b7fe700 1 -- 172.21.15.151:0/2514936959 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f576400cca0 con 0x7f5774100440 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.082+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577419baf0 con 0x7f5774100440 2024-04-16T02:34:48.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.082+0000 7f576b7fe700 1 -- 172.21.15.151:0/2514936959 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f576400ce00 con 0x7f5774100440 2024-04-16T02:34:48.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.082+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577419c0c0 con 0x7f5774100440 2024-04-16T02:34:48.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.082+0000 7f576b7fe700 1 -- 172.21.15.151:0/2514936959 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5764015720 con 0x7f5774100440 2024-04-16T02:34:48.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.083+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5774107510 con 0x7f5774100440 2024-04-16T02:34:48.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.084+0000 7f576b7fe700 1 -- 172.21.15.151:0/2514936959 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f57640159e0 con 0x7f5774100440 2024-04-16T02:34:48.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.085+0000 7f576b7fe700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f576006c4e0 0x7f576006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:34:48.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.085+0000 7f576b7fe700 1 -- 172.21.15.151:0/2514936959 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f576408bcf0 con 0x7f5774100440 2024-04-16T02:34:48.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.085+0000 7f577259c700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f576006c4e0 0x7f576006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:34:48.088 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.086+0000 7f577259c700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f576006c4e0 0x7f576006e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f576c00b780 tx=0x7f576c0060c0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:34:48.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.095+0000 7f576b7fe700 1 -- 172.21.15.151:0/2514936959 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5764059f00 con 0x7f5774100440 2024-04-16T02:34:48.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:47 smithi153 ceph-mon[36747]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:48.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:47 smithi153 ceph-mon[36747]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:48.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:47 smithi153 ceph-mon[36747]: pgmap v247: 65 pgs: 65 active+clean; 48 MiB data, 373 MiB used, 536 GiB / 536 GiB avail; 927 KiB/s rd, 4.1 MiB/s wr, 343 op/s 2024-04-16T02:34:48.157 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:47 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3461908785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:34:48.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:47 smithi151 ceph-mon[29003]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:48.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:47 smithi151 ceph-mon[29003]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:48.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:47 smithi151 ceph-mon[29003]: pgmap v247: 65 pgs: 65 active+clean; 48 MiB data, 373 MiB used, 536 GiB / 536 GiB avail; 927 KiB/s rd, 4.1 MiB/s wr, 343 op/s 2024-04-16T02:34:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:47 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3461908785' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:34:48.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.412+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 --> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5774104d90 con 0x7f576006c4e0 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.413+0000 7f576b7fe700 1 -- 172.21.15.151:0/2514936959 <== mgr.14215 v2:172.21.15.151:6800/2753390843 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f5774104d90 con 0x7f576006c4e0 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [], 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "0/2 daemons upgraded", 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mgr daemons", 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:34:48.415 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:34:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.415+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f576006c4e0 msgr2=0x7f576006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.415+0000 7f57788dc700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f576006c4e0 0x7f576006e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f576c00b780 tx=0x7f576c0060c0 comp rx=0 tx=0).stop 2024-04-16T02:34:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.415+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5774100440 msgr2=0x7f57741968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:34:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.415+0000 7f57788dc700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5774100440 0x7f57741968e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f576400ea30 tx=0x7f576400ed40 comp rx=0 tx=0).stop 2024-04-16T02:34:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.416+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 shutdown_connections 2024-04-16T02:34:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.416+0000 7f57788dc700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:6800/2753390843,v1:172.21.15.151:6801/2753390843] conn(0x7f576006c4e0 0x7f576006e9a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.416+0000 7f57788dc700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740ff240 0x7f57741963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.416+0000 7f57788dc700 1 --2- 172.21.15.151:0/2514936959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5774100440 0x7f57741968e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:34:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.416+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 >> 172.21.15.151:0/2514936959 conn(0x7f57740fa7c0 msgr2=0x7f5774103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:34:48.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.416+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 shutdown_connections 2024-04-16T02:34:48.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:34:48.416+0000 7f57788dc700 1 -- 172.21.15.151:0/2514936959 wait complete. 2024-04-16T02:34:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:48 smithi151 ceph-mon[29003]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:49.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:48 smithi153 ceph-mon[36747]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:49 smithi151 ceph-mon[29003]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:49 smithi151 ceph-mon[29003]: pgmap v248: 65 pgs: 65 active+clean; 53 MiB data, 408 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 4.4 MiB/s wr, 426 op/s 2024-04-16T02:34:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:49 smithi153 ceph-mon[36747]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:34:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:49 smithi153 ceph-mon[36747]: pgmap v248: 65 pgs: 65 active+clean; 53 MiB data, 408 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 4.4 MiB/s wr, 426 op/s 2024-04-16T02:34:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:50 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:34:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:50 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:50 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:50 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:34:51.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:50 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:34:51.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:51.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:34:51.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:50 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:34:52.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:51 smithi153 ceph-mon[36747]: pgmap v249: 65 pgs: 65 active+clean; 60 MiB data, 427 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 416 op/s 2024-04-16T02:34:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:51 smithi151 ceph-mon[29003]: pgmap v249: 65 pgs: 65 active+clean; 60 MiB data, 427 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 416 op/s 2024-04-16T02:34:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:53 smithi151 ceph-mon[29003]: pgmap v250: 65 pgs: 65 active+clean; 77 MiB data, 481 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 479 op/s 2024-04-16T02:34:54.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:53 smithi153 ceph-mon[36747]: pgmap v250: 65 pgs: 65 active+clean; 77 MiB data, 481 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 479 op/s 2024-04-16T02:34:55.877 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:55 smithi153 ceph-mon[36747]: pgmap v251: 65 pgs: 65 active+clean; 80 MiB data, 505 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 403 op/s 2024-04-16T02:34:56.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:55 smithi151 ceph-mon[29003]: pgmap v251: 65 pgs: 65 active+clean; 80 MiB data, 505 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 403 op/s 2024-04-16T02:34:58.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:57 smithi151 ceph-mon[29003]: pgmap v252: 65 pgs: 65 active+clean; 92 MiB data, 523 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 441 op/s 2024-04-16T02:34:58.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:57 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:34:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:57 smithi153 ceph-mon[36747]: pgmap v252: 65 pgs: 65 active+clean; 92 MiB data, 523 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 441 op/s 2024-04-16T02:34:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:57 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:34:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:58 smithi151 ceph-mon[29003]: Standby manager daemon smithi153.esbtzf restarted 2024-04-16T02:34:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:58 smithi151 ceph-mon[29003]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:34:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:58 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:34:59.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:58 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:34:59.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:58 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:34:59.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:58 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:34:59.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:58 smithi153 ceph-mon[36747]: Standby manager daemon smithi153.esbtzf restarted 2024-04-16T02:34:59.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:58 smithi153 ceph-mon[36747]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:34:59.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:58 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:34:59.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:58 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:34:59.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:58 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:34:59.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:58 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:35:00.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:59 smithi151 ceph-mon[29003]: pgmap v253: 65 pgs: 65 active+clean; 101 MiB data, 541 MiB used, 536 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 465 op/s 2024-04-16T02:35:00.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:34:59 smithi151 ceph-mon[29003]: mgrmap e20: smithi151.uqqame(active, since 7m), standbys: smithi153.esbtzf 2024-04-16T02:35:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:59 smithi153 ceph-mon[36747]: pgmap v253: 65 pgs: 65 active+clean; 101 MiB data, 541 MiB used, 536 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 465 op/s 2024-04-16T02:35:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:34:59 smithi153 ceph-mon[36747]: mgrmap e20: smithi151.uqqame(active, since 7m), standbys: smithi153.esbtzf 2024-04-16T02:35:02.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:01 smithi151 ceph-mon[29003]: pgmap v254: 65 pgs: 65 active+clean; 103 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 409 op/s 2024-04-16T02:35:02.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:01 smithi153 ceph-mon[36747]: pgmap v254: 65 pgs: 65 active+clean; 103 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 409 op/s 2024-04-16T02:35:04.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:03 smithi151 ceph-mon[29003]: pgmap v255: 65 pgs: 65 active+clean; 115 MiB data, 609 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.9 MiB/s wr, 490 op/s 2024-04-16T02:35:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:03 smithi153 ceph-mon[36747]: pgmap v255: 65 pgs: 65 active+clean; 115 MiB data, 609 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.9 MiB/s wr, 490 op/s 2024-04-16T02:35:06.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:06.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:05 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:06.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:05 smithi151 ceph-mon[29003]: pgmap v256: 65 pgs: 65 active+clean; 116 MiB data, 621 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 443 op/s 2024-04-16T02:35:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:05 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:05 smithi153 ceph-mon[36747]: pgmap v256: 65 pgs: 65 active+clean; 116 MiB data, 621 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 443 op/s 2024-04-16T02:35:07.107 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:07.108 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:06 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:07.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:07.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:06 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:08.133 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:07 smithi153 ceph-mon[36747]: pgmap v257: 65 pgs: 65 active+clean; 123 MiB data, 664 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 485 op/s 2024-04-16T02:35:08.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:07 smithi151 ceph-mon[29003]: pgmap v257: 65 pgs: 65 active+clean; 123 MiB data, 664 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 485 op/s 2024-04-16T02:35:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:09 smithi151 ceph-mon[29003]: pgmap v258: 65 pgs: 65 active+clean; 134 MiB data, 704 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 484 op/s 2024-04-16T02:35:10.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:09 smithi153 ceph-mon[36747]: pgmap v258: 65 pgs: 65 active+clean; 134 MiB data, 704 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 484 op/s 2024-04-16T02:35:11.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:11.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:11.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:35:11.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:35:11.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr fail", "who": "smithi151.uqqame"}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "mgr fail", "who": "smithi151.uqqame"}]': finished 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: mgrmap e21: smithi153.esbtzf(active, starting, since 0.011935s) 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr metadata", "who": "smithi153.esbtzf", "id": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:35:11.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: Manager daemon smithi153.esbtzf is now available 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.esbtzf/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.esbtzf/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.esbtzf/trash_purge_schedule"}]: dispatch 2024-04-16T02:35:11.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:11 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.esbtzf/trash_purge_schedule"}]: dispatch 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr fail", "who": "smithi151.uqqame"}]: dispatch 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:35:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.14215 172.21.15.151:0/4050822284' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "mgr fail", "who": "smithi151.uqqame"}]': finished 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: mgrmap e21: smithi153.esbtzf(active, starting, since 0.011935s) 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr metadata", "who": "smithi153.esbtzf", "id": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: Manager daemon smithi153.esbtzf is now available 2024-04-16T02:35:11.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:35:11.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:35:11.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.esbtzf/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:35:11.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.esbtzf/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:35:11.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.esbtzf/trash_purge_schedule"}]: dispatch 2024-04-16T02:35:11.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:11 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.esbtzf/trash_purge_schedule"}]: dispatch 2024-04-16T02:35:12.698 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:12 smithi151 ceph-mon[29003]: [16/Apr/2024:02:35:10] ENGINE Bus STARTING 2024-04-16T02:35:12.699 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:12 smithi151 ceph-mon[29003]: [16/Apr/2024:02:35:11] ENGINE Serving on https://172.21.15.153:7150 2024-04-16T02:35:12.699 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:12 smithi151 ceph-mon[29003]: Deploying cephadm binary to smithi151 2024-04-16T02:35:12.699 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:12 smithi151 ceph-mon[29003]: Deploying cephadm binary to smithi153 2024-04-16T02:35:12.699 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:12 smithi151 ceph-mon[29003]: [16/Apr/2024:02:35:11] ENGINE Serving on http://172.21.15.153:8765 2024-04-16T02:35:12.699 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:12 smithi151 ceph-mon[29003]: [16/Apr/2024:02:35:11] ENGINE Bus STARTED 2024-04-16T02:35:12.699 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:12 smithi151 ceph-mon[29003]: mgrmap e22: smithi153.esbtzf(active, since 1.01087s) 2024-04-16T02:35:12.699 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:12 smithi151 ceph-mon[29003]: pgmap v3: 65 pgs: 65 active+clean; 157 MiB data, 761 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:35:12.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:12 smithi153 ceph-mon[36747]: [16/Apr/2024:02:35:10] ENGINE Bus STARTING 2024-04-16T02:35:12.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:12 smithi153 ceph-mon[36747]: [16/Apr/2024:02:35:11] ENGINE Serving on https://172.21.15.153:7150 2024-04-16T02:35:12.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:12 smithi153 ceph-mon[36747]: Deploying cephadm binary to smithi151 2024-04-16T02:35:12.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:12 smithi153 ceph-mon[36747]: Deploying cephadm binary to smithi153 2024-04-16T02:35:12.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:12 smithi153 ceph-mon[36747]: [16/Apr/2024:02:35:11] ENGINE Serving on http://172.21.15.153:8765 2024-04-16T02:35:12.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:12 smithi153 ceph-mon[36747]: [16/Apr/2024:02:35:11] ENGINE Bus STARTED 2024-04-16T02:35:12.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:12 smithi153 ceph-mon[36747]: mgrmap e22: smithi153.esbtzf(active, since 1.01087s) 2024-04-16T02:35:12.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:12 smithi153 ceph-mon[36747]: pgmap v3: 65 pgs: 65 active+clean; 157 MiB data, 761 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:35:13.634 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:13 smithi151 ceph-mon[29003]: pgmap v4: 65 pgs: 65 active+clean; 157 MiB data, 761 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:35:13.634 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:13 smithi151 ceph-mon[29003]: mgrmap e23: smithi153.esbtzf(active, since 2s) 2024-04-16T02:35:13.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:13 smithi153 ceph-mon[36747]: pgmap v4: 65 pgs: 65 active+clean; 157 MiB data, 761 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:35:13.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:13 smithi153 ceph-mon[36747]: mgrmap e23: smithi153.esbtzf(active, since 2s) 2024-04-16T02:35:15.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:15 smithi151 ceph-mon[29003]: pgmap v5: 65 pgs: 65 active+clean; 157 MiB data, 761 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:35:15.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:15 smithi151 ceph-mon[29003]: mgrmap e24: smithi153.esbtzf(active, since 4s) 2024-04-16T02:35:15.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:15 smithi153 ceph-mon[36747]: pgmap v5: 65 pgs: 65 active+clean; 157 MiB data, 761 MiB used, 536 GiB / 536 GiB avail 2024-04-16T02:35:15.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:15 smithi153 ceph-mon[36747]: mgrmap e24: smithi153.esbtzf(active, since 4s) 2024-04-16T02:35:17.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:17 smithi151 ceph-mon[29003]: pgmap v6: 65 pgs: 65 active+clean; 164 MiB data, 786 MiB used, 536 GiB / 536 GiB avail; 955 KiB/s rd, 1.6 MiB/s wr, 179 op/s 2024-04-16T02:35:17.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:17 smithi153 ceph-mon[36747]: pgmap v6: 65 pgs: 65 active+clean; 164 MiB data, 786 MiB used, 536 GiB / 536 GiB avail; 955 KiB/s rd, 1.6 MiB/s wr, 179 op/s 2024-04-16T02:35:18.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.570+0000 7f08977fe700 1 -- 172.21.15.151:0/908626673 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0884004a00 con 0x7f0898100cb0 2024-04-16T02:35:18.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.571+0000 7f089f0ea700 1 -- 172.21.15.151:0/908626673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0898100cb0 msgr2=0x7f0898101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:18.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.571+0000 7f089f0ea700 1 --2- 172.21.15.151:0/908626673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0898100cb0 0x7f0898101110 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0884009910 tx=0x7f0884009c20 comp rx=0 tx=0).stop 2024-04-16T02:35:18.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.571+0000 7f089f0ea700 1 -- 172.21.15.151:0/908626673 shutdown_connections 2024-04-16T02:35:18.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.571+0000 7f089f0ea700 1 --2- 172.21.15.151:0/908626673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0898100cb0 0x7f0898101110 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:18.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.571+0000 7f089f0ea700 1 --2- 172.21.15.151:0/908626673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08980ffab0 0x7f08980ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:18.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.571+0000 7f089f0ea700 1 -- 172.21.15.151:0/908626673 >> 172.21.15.151:0/908626673 conn(0x7f08980fb050 msgr2=0x7f08980fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:18.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.572+0000 7f089f0ea700 1 -- 172.21.15.151:0/908626673 shutdown_connections 2024-04-16T02:35:18.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.572+0000 7f089f0ea700 1 -- 172.21.15.151:0/908626673 wait complete. 2024-04-16T02:35:18.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.573+0000 7f089f0ea700 1 Processor -- start 2024-04-16T02:35:18.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.573+0000 7f089f0ea700 1 -- start start 2024-04-16T02:35:18.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.573+0000 7f089f0ea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08980ffab0 0x7f0898196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:18.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.574+0000 7f089f0ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0898100cb0 0x7f08981968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:18.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.574+0000 7f089f0ea700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898196ec0 con 0x7f08980ffab0 2024-04-16T02:35:18.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.574+0000 7f089ce86700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08980ffab0 0x7f0898196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:18.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.574+0000 7f089ce86700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08980ffab0 0x7f0898196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34806/0 (socket says 172.21.15.151:34806) 2024-04-16T02:35:18.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.574+0000 7f089ce86700 1 -- 172.21.15.151:0/3164390530 learned_addr learned my addr 172.21.15.151:0/3164390530 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:18.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.574+0000 7f0897fff700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0898100cb0 0x7f08981968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:18.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.574+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898197000 con 0x7f0898100cb0 2024-04-16T02:35:18.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.575+0000 7f089ce86700 1 -- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0898100cb0 msgr2=0x7f08981968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:18.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.575+0000 7f089ce86700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0898100cb0 0x7f08981968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:18.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.575+0000 7f089ce86700 1 -- 172.21.15.151:0/3164390530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08840095f0 con 0x7f08980ffab0 2024-04-16T02:35:18.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.575+0000 7f089ce86700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08980ffab0 0x7f0898196360 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f088c00e8f0 tx=0x7f088c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:18.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.575+0000 7f0895ffb700 1 -- 172.21.15.151:0/3164390530 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f088c009790 con 0x7f08980ffab0 2024-04-16T02:35:18.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.575+0000 7f0895ffb700 1 -- 172.21.15.151:0/3164390530 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f088c00c810 con 0x7f08980ffab0 2024-04-16T02:35:18.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.575+0000 7f0895ffb700 1 -- 172.21.15.151:0/3164390530 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f088c003d20 con 0x7f08980ffab0 2024-04-16T02:35:18.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.575+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089819bab0 con 0x7f08980ffab0 2024-04-16T02:35:18.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.576+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089819c000 con 0x7f08980ffab0 2024-04-16T02:35:18.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.577+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0898109710 con 0x7f08980ffab0 2024-04-16T02:35:18.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.578+0000 7f0895ffb700 1 -- 172.21.15.151:0/3164390530 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 24) v1 ==== 46699+0+0 (secure 0 0 0) 0x7f088c0041f0 con 0x7f08980ffab0 2024-04-16T02:35:18.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.579+0000 7f0895ffb700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f08880398c0 0x7f088803bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:18.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.579+0000 7f0895ffb700 1 -- 172.21.15.151:0/3164390530 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f088c014070 con 0x7f08980ffab0 2024-04-16T02:35:18.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.579+0000 7f0897fff700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f08880398c0 0x7f088803bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:18.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.580+0000 7f0897fff700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f08880398c0 0x7f088803bd80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f088400b600 tx=0x7f0884005fb0 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:18.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.587+0000 7f0895ffb700 1 -- 172.21.15.151:0/3164390530 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f088c0173d0 con 0x7f08980ffab0 2024-04-16T02:35:18.910 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.908+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0898105600 con 0x7f08880398c0 2024-04-16T02:35:18.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.910+0000 7f0895ffb700 1 -- 172.21.15.151:0/3164390530 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f0898105600 con 0x7f08880398c0 2024-04-16T02:35:18.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f08880398c0 msgr2=0x7f088803bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:18.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f08880398c0 0x7f088803bd80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f088400b600 tx=0x7f0884005fb0 comp rx=0 tx=0).stop 2024-04-16T02:35:18.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08980ffab0 msgr2=0x7f0898196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:18.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08980ffab0 0x7f0898196360 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f088c00e8f0 tx=0x7f088c00ec00 comp rx=0 tx=0).stop 2024-04-16T02:35:18.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 shutdown_connections 2024-04-16T02:35:18.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f08880398c0 0x7f088803bd80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:18.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08980ffab0 0x7f0898196360 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:18.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 --2- 172.21.15.151:0/3164390530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0898100cb0 0x7f08981968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:18.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 >> 172.21.15.151:0/3164390530 conn(0x7f08980fb050 msgr2=0x7f0898103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:18.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 shutdown_connections 2024-04-16T02:35:18.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:18.912+0000 7f089f0ea700 1 -- 172.21.15.151:0/3164390530 wait complete. 2024-04-16T02:35:18.924 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:35:19.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.038+0000 7fb0f110f700 1 -- 172.21.15.151:0/3492448974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 msgr2=0x7fb0ec103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:19.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.038+0000 7fb0f110f700 1 --2- 172.21.15.151:0/3492448974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 0x7fb0ec103a30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009870 tx=0x7fb0dc009b80 comp rx=0 tx=0).stop 2024-04-16T02:35:19.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.038+0000 7fb0f110f700 1 -- 172.21.15.151:0/3492448974 shutdown_connections 2024-04-16T02:35:19.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.038+0000 7fb0f110f700 1 --2- 172.21.15.151:0/3492448974 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0ec103f70 0x7fb0ec106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:19.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.038+0000 7fb0f110f700 1 --2- 172.21.15.151:0/3492448974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 0x7fb0ec103a30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:19.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.038+0000 7fb0f110f700 1 -- 172.21.15.151:0/3492448974 >> 172.21.15.151:0/3492448974 conn(0x7fb0ec0fb030 msgr2=0x7fb0ec0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:19.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.038+0000 7fb0f110f700 1 -- 172.21.15.151:0/3492448974 shutdown_connections 2024-04-16T02:35:19.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.038+0000 7fb0f110f700 1 -- 172.21.15.151:0/3492448974 wait complete. 2024-04-16T02:35:19.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.039+0000 7fb0f110f700 1 Processor -- start 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.039+0000 7fb0f110f700 1 -- start start 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0f110f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 0x7fb0ec1940f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0f110f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0ec103f70 0x7fb0ec194630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0f110f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec194b70 con 0x7fb0ec103f70 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0f110f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec194cb0 con 0x7fb0ec101640 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0e2d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 0x7fb0ec1940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0ead9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0ec103f70 0x7fb0ec194630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0e2d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 0x7fb0ec1940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:49064/0 (socket says 172.21.15.151:49064) 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0e2d9d700 1 -- 172.21.15.151:0/4112223718 learned_addr learned my addr 172.21.15.151:0/4112223718 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0ead9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0ec103f70 0x7fb0ec194630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34812/0 (socket says 172.21.15.151:34812) 2024-04-16T02:35:19.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0e2d9d700 1 -- 172.21.15.151:0/4112223718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0ec103f70 msgr2=0x7fb0ec194630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:19.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0e2d9d700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0ec103f70 0x7fb0ec194630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:19.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0e2d9d700 1 -- 172.21.15.151:0/4112223718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc009520 con 0x7fb0ec101640 2024-04-16T02:35:19.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.040+0000 7fb0e2d9d700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 0x7fb0ec1940f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009230 tx=0x7fb0dc00f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:19.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.041+0000 7fb0e8d99700 1 -- 172.21.15.151:0/4112223718 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0dc01c070 con 0x7fb0ec101640 2024-04-16T02:35:19.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.041+0000 7fb0e8d99700 1 -- 172.21.15.151:0/4112223718 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb0dc00fe40 con 0x7fb0ec101640 2024-04-16T02:35:19.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.041+0000 7fb0e8d99700 1 -- 172.21.15.151:0/4112223718 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0dc0173f0 con 0x7fb0ec101640 2024-04-16T02:35:19.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.041+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec1aa820 con 0x7fb0ec101640 2024-04-16T02:35:19.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.041+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec1aad40 con 0x7fb0ec101640 2024-04-16T02:35:19.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.042+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ec109430 con 0x7fb0ec101640 2024-04-16T02:35:19.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.043+0000 7fb0e8d99700 1 -- 172.21.15.151:0/4112223718 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 24) v1 ==== 46699+0+0 (secure 0 0 0) 0x7fb0dc020070 con 0x7fb0ec101640 2024-04-16T02:35:19.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.043+0000 7fb0e8d99700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fb0d80398c0 0x7fb0d803bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:19.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.043+0000 7fb0e8d99700 1 -- 172.21.15.151:0/4112223718 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fb0dc04ef60 con 0x7fb0ec101640 2024-04-16T02:35:19.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.043+0000 7fb0ead9d700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fb0d80398c0 0x7fb0d803bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:19.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.044+0000 7fb0ead9d700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fb0d80398c0 0x7fb0d803bd80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4005fd0 tx=0x7fb0d4009310 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:19.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.049+0000 7fb0e8d99700 1 -- 172.21.15.151:0/4112223718 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fb0dc02a950 con 0x7fb0ec101640 2024-04-16T02:35:19.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.421+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0ec0611d0 con 0x7fb0d80398c0 2024-04-16T02:35:19.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.422+0000 7fb0e8d99700 1 -- 172.21.15.151:0/4112223718 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fb0ec0611d0 con 0x7fb0d80398c0 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.424+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fb0d80398c0 msgr2=0x7fb0d803bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.424+0000 7fb0f110f700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fb0d80398c0 0x7fb0d803bd80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4005fd0 tx=0x7fb0d4009310 comp rx=0 tx=0).stop 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.424+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 msgr2=0x7fb0ec1940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.424+0000 7fb0f110f700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 0x7fb0ec1940f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009230 tx=0x7fb0dc00f740 comp rx=0 tx=0).stop 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.424+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 shutdown_connections 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.424+0000 7fb0f110f700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fb0d80398c0 0x7fb0d803bd80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.424+0000 7fb0f110f700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0ec101640 0x7fb0ec1940f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.424+0000 7fb0f110f700 1 --2- 172.21.15.151:0/4112223718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0ec103f70 0x7fb0ec194630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.424+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 >> 172.21.15.151:0/4112223718 conn(0x7fb0ec0fb030 msgr2=0x7fb0ec0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.425+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 shutdown_connections 2024-04-16T02:35:19.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.425+0000 7fb0f110f700 1 -- 172.21.15.151:0/4112223718 wait complete. 2024-04-16T02:35:19.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.588+0000 7fd74b6fc700 1 -- 172.21.15.151:0/3382112295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 msgr2=0x7fd744100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:19.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.588+0000 7fd74b6fc700 1 --2- 172.21.15.151:0/3382112295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 0x7fd744100150 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd734009910 tx=0x7fd734009c20 comp rx=0 tx=0).stop 2024-04-16T02:35:19.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.588+0000 7fd74b6fc700 1 -- 172.21.15.151:0/3382112295 shutdown_connections 2024-04-16T02:35:19.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.588+0000 7fd74b6fc700 1 --2- 172.21.15.151:0/3382112295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 0x7fd744100150 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:19.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.588+0000 7fd74b6fc700 1 --2- 172.21.15.151:0/3382112295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd744101070 0x7fd744101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:19.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.588+0000 7fd74b6fc700 1 -- 172.21.15.151:0/3382112295 >> 172.21.15.151:0/3382112295 conn(0x7fd7440fa7c0 msgr2=0x7fd7440fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:19.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.588+0000 7fd74b6fc700 1 -- 172.21.15.151:0/3382112295 shutdown_connections 2024-04-16T02:35:19.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.588+0000 7fd74b6fc700 1 -- 172.21.15.151:0/3382112295 wait complete. 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.589+0000 7fd74b6fc700 1 Processor -- start 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.589+0000 7fd74b6fc700 1 -- start start 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.589+0000 7fd74b6fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 0x7fd7440ff720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.589+0000 7fd74b6fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd744101070 0x7fd74419b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd74b6fc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74419b670 con 0x7fd7440ffcd0 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd749498700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 0x7fd7440ff720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd74b6fc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74419b7b0 con 0x7fd744101070 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd749498700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 0x7fd7440ff720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34840/0 (socket says 172.21.15.151:34840) 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd749498700 1 -- 172.21.15.151:0/2500619931 learned_addr learned my addr 172.21.15.151:0/2500619931 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:19.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd748c97700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd744101070 0x7fd74419b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:19.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd749498700 1 -- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd744101070 msgr2=0x7fd74419b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:19.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd749498700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd744101070 0x7fd74419b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:19.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd749498700 1 -- 172.21.15.151:0/2500619931 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7340095f0 con 0x7fd7440ffcd0 2024-04-16T02:35:19.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.590+0000 7fd749498700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 0x7fd7440ff720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e920 tx=0x7fd74000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:19.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.591+0000 7fd73a7fc700 1 -- 172.21.15.151:0/2500619931 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd74000c9d0 con 0x7fd7440ffcd0 2024-04-16T02:35:19.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.591+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74419ba90 con 0x7fd7440ffcd0 2024-04-16T02:35:19.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.591+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74419bfb0 con 0x7fd7440ffcd0 2024-04-16T02:35:19.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.591+0000 7fd73a7fc700 1 -- 172.21.15.151:0/2500619931 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd740010430 con 0x7fd7440ffcd0 2024-04-16T02:35:19.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.591+0000 7fd73a7fc700 1 -- 172.21.15.151:0/2500619931 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd740018650 con 0x7fd7440ffcd0 2024-04-16T02:35:19.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.592+0000 7fd73a7fc700 1 -- 172.21.15.151:0/2500619931 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 24) v1 ==== 46699+0+0 (secure 0 0 0) 0x7fd7400188c0 con 0x7fd7440ffcd0 2024-04-16T02:35:19.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.592+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd728005320 con 0x7fd7440ffcd0 2024-04-16T02:35:19.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.593+0000 7fd73a7fc700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd7300398c0 0x7fd73003bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:19.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.593+0000 7fd73a7fc700 1 -- 172.21.15.151:0/2500619931 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fd740014070 con 0x7fd7440ffcd0 2024-04-16T02:35:19.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.593+0000 7fd748c97700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd7300398c0 0x7fd73003bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:19.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.594+0000 7fd748c97700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd7300398c0 0x7fd73003bd80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd73400b6f0 tx=0x7fd734005a90 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:19.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.601+0000 7fd73a7fc700 1 -- 172.21.15.151:0/2500619931 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fd74001fdf0 con 0x7fd7440ffcd0 2024-04-16T02:35:19.745 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:19 smithi153 ceph-mon[36747]: pgmap v7: 65 pgs: 65 active+clean; 174 MiB data, 852 MiB used, 536 GiB / 536 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 344 op/s 2024-04-16T02:35:19.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:19 smithi151 ceph-mon[29003]: pgmap v7: 65 pgs: 65 active+clean; 174 MiB data, 852 MiB used, 536 GiB / 536 GiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 344 op/s 2024-04-16T02:35:19.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.973+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd728000bf0 con 0x7fd7300398c0 2024-04-16T02:35:19.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.996+0000 7fd73a7fc700 1 -- 172.21.15.151:0/2500619931 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd728000bf0 con 0x7fd7300398c0 2024-04-16T02:35:19.998 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:35:19.998 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (6m) 2m ago 8m 25.1M - 0.25.0 c8568f914cd2 9f7a3dfb7aa4 2024-04-16T02:35:19.998 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (8m) 2m ago 8m 7570k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (7m) 15s ago 7m 7608k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (8m) 2m ago 8m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (7m) 15s ago 7m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (6m) 2m ago 7m 82.6M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (2m) 2m ago 2m 16.6M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (2m) 2m ago 2m 18.3M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (2m) 15s ago 2m 52.3M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (2m) 15s ago 2m 23.8M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:9283,8765,8443 running (9m) 2m ago 9m 490M - 18.2.1 5be31c24972a 9e98c5bc45da 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (32s) 15s ago 7m 93.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 cd0d253df44f 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (9m) 2m ago 9m 47.6M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (7m) 15s ago 7m 36.6M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:35:19.999 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (8m) 2m ago 8m 16.2M - 1.5.0 0da6a335fe13 7748f5130613 2024-04-16T02:35:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 15s ago 7m 16.4M - 1.5.0 0da6a335fe13 6a289489fe28 2024-04-16T02:35:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (6m) 2m ago 6m 47.5M 3946M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:35:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (5m) 2m ago 5m 47.0M 3946M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:35:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (5m) 2m ago 5m 46.8M 3946M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:35:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (5m) 15s ago 4m 83.2M 4970M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:35:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (4m) 15s ago 4m 81.5M 4970M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:35:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (4m) 15s ago 4m 78.8M 4970M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:35:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (6m) 2m ago 7m 58.9M - 2.43.0 a07b618ecd1d 9a1c8cd0261d 2024-04-16T02:35:20.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd7300398c0 msgr2=0x7fd73003bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:20.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd7300398c0 0x7fd73003bd80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd73400b6f0 tx=0x7fd734005a90 comp rx=0 tx=0).stop 2024-04-16T02:35:20.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 msgr2=0x7fd7440ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:20.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 0x7fd7440ff720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e920 tx=0x7fd74000ec30 comp rx=0 tx=0).stop 2024-04-16T02:35:20.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 shutdown_connections 2024-04-16T02:35:20.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd7300398c0 0x7fd73003bd80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7440ffcd0 0x7fd7440ff720 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 --2- 172.21.15.151:0/2500619931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd744101070 0x7fd74419b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 >> 172.21.15.151:0/2500619931 conn(0x7fd7440fa7c0 msgr2=0x7fd744103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:20.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.998+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 shutdown_connections 2024-04-16T02:35:20.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:19.999+0000 7fd74b6fc700 1 -- 172.21.15.151:0/2500619931 wait complete. 2024-04-16T02:35:20.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.156+0000 7f5908344700 1 -- 172.21.15.151:0/715415266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 msgr2=0x7f590010a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:20.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.156+0000 7f5908344700 1 --2- 172.21.15.151:0/715415266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 0x7f590010a210 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f58f00074e0 tx=0x7f58f00077f0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.157+0000 7f5908344700 1 -- 172.21.15.151:0/715415266 shutdown_connections 2024-04-16T02:35:20.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.157+0000 7f5908344700 1 --2- 172.21.15.151:0/715415266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f590010a7e0 0x7f590010cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.157+0000 7f5908344700 1 --2- 172.21.15.151:0/715415266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 0x7f590010a210 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.157+0000 7f5908344700 1 -- 172.21.15.151:0/715415266 >> 172.21.15.151:0/715415266 conn(0x7f590006d9f0 msgr2=0x7f590006fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:20.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.157+0000 7f5908344700 1 -- 172.21.15.151:0/715415266 shutdown_connections 2024-04-16T02:35:20.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.157+0000 7f5908344700 1 -- 172.21.15.151:0/715415266 wait complete. 2024-04-16T02:35:20.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.157+0000 7f5908344700 1 Processor -- start 2024-04-16T02:35:20.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.158+0000 7f5908344700 1 -- start start 2024-04-16T02:35:20.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.158+0000 7f5908344700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 0x7f590019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.158+0000 7f5908344700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f590010a7e0 0x7f590019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.158+0000 7f5908344700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590019d2a0 con 0x7f5900107e20 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.158+0000 7f5908344700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590019d3e0 con 0x7f590010a7e0 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.158+0000 7f59060e0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 0x7f590019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.158+0000 7f59058df700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f590010a7e0 0x7f590019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.158+0000 7f59060e0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 0x7f590019c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34866/0 (socket says 172.21.15.151:34866) 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.158+0000 7f59060e0700 1 -- 172.21.15.151:0/3750164039 learned_addr learned my addr 172.21.15.151:0/3750164039 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.159+0000 7f59060e0700 1 -- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f590010a7e0 msgr2=0x7f590019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.159+0000 7f59060e0700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f590010a7e0 0x7f590019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.159+0000 7f59060e0700 1 -- 172.21.15.151:0/3750164039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f00071c0 con 0x7f5900107e20 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.159+0000 7f59058df700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f590010a7e0 0x7f590019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:35:20.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.159+0000 7f59060e0700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 0x7f590019c740 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009d70 tx=0x7f58f0005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:20.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.160+0000 7f58f77fe700 1 -- 172.21.15.151:0/3750164039 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58f0003850 con 0x7f5900107e20 2024-04-16T02:35:20.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.160+0000 7f58f77fe700 1 -- 172.21.15.151:0/3750164039 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f58f0003e60 con 0x7f5900107e20 2024-04-16T02:35:20.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.160+0000 7f58f77fe700 1 -- 172.21.15.151:0/3750164039 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58f0011690 con 0x7f5900107e20 2024-04-16T02:35:20.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.160+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59001a1e30 con 0x7f5900107e20 2024-04-16T02:35:20.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.160+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59001aaa40 con 0x7f5900107e20 2024-04-16T02:35:20.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.162+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590010fd00 con 0x7f5900107e20 2024-04-16T02:35:20.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.162+0000 7f58f77fe700 1 -- 172.21.15.151:0/3750164039 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 24) v1 ==== 46699+0+0 (secure 0 0 0) 0x7f58f0009460 con 0x7f5900107e20 2024-04-16T02:35:20.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.163+0000 7f58f77fe700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f58ec039910 0x7f58ec03bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:20.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.163+0000 7f58f77fe700 1 -- 172.21.15.151:0/3750164039 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f58f004e7f0 con 0x7f5900107e20 2024-04-16T02:35:20.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.163+0000 7f59058df700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f58ec039910 0x7f58ec03bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:20.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.164+0000 7f59058df700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f58ec039910 0x7f58ec03bdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00afd0 tx=0x7f58fc011040 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.171+0000 7f58f77fe700 1 -- 172.21.15.151:0/3750164039 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f58f0017a00 con 0x7f5900107e20 2024-04-16T02:35:20.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.558+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f59001ab160 con 0x7f5900107e20 2024-04-16T02:35:20.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.558+0000 7f58f77fe700 1 -- 172.21.15.151:0/3750164039 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f58f0015070 con 0x7f5900107e20 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 1 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:35:20.561 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-16T02:35:20.562 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 1 2024-04-16T02:35:20.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:35:20.562 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:35:20.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f58ec039910 msgr2=0x7f58ec03bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:20.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f58ec039910 0x7f58ec03bdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00afd0 tx=0x7f58fc011040 comp rx=0 tx=0).stop 2024-04-16T02:35:20.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 msgr2=0x7f590019c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:20.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 0x7f590019c740 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009d70 tx=0x7f58f0005270 comp rx=0 tx=0).stop 2024-04-16T02:35:20.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 shutdown_connections 2024-04-16T02:35:20.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f58ec039910 0x7f58ec03bdd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5900107e20 0x7f590019c740 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 --2- 172.21.15.151:0/3750164039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f590010a7e0 0x7f590019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 >> 172.21.15.151:0/3750164039 conn(0x7f590006d9f0 msgr2=0x7f590006fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:20.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 shutdown_connections 2024-04-16T02:35:20.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.562+0000 7f5908344700 1 -- 172.21.15.151:0/3750164039 wait complete. 2024-04-16T02:35:20.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.692+0000 7f6fa805b700 1 -- 172.21.15.151:0/1501550273 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0108400 msgr2=0x7f6fa0108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:20.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.692+0000 7f6fa805b700 1 --2- 172.21.15.151:0/1501550273 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0108400 0x7f6fa0108860 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009910 tx=0x7f6f90009c20 comp rx=0 tx=0).stop 2024-04-16T02:35:20.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.692+0000 7f6fa805b700 1 -- 172.21.15.151:0/1501550273 shutdown_connections 2024-04-16T02:35:20.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.692+0000 7f6fa805b700 1 --2- 172.21.15.151:0/1501550273 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0108400 0x7f6fa0108860 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.692+0000 7f6fa805b700 1 --2- 172.21.15.151:0/1501550273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fa0107200 0x7f6fa0107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.692+0000 7f6fa805b700 1 -- 172.21.15.151:0/1501550273 >> 172.21.15.151:0/1501550273 conn(0x7f6fa00761b0 msgr2=0x7f6fa0078610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:20.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.692+0000 7f6fa805b700 1 -- 172.21.15.151:0/1501550273 shutdown_connections 2024-04-16T02:35:20.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.692+0000 7f6fa805b700 1 -- 172.21.15.151:0/1501550273 wait complete. 2024-04-16T02:35:20.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.693+0000 7f6fa805b700 1 Processor -- start 2024-04-16T02:35:20.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.693+0000 7f6fa805b700 1 -- start start 2024-04-16T02:35:20.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.693+0000 7f6fa805b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0107200 0x7f6fa019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:20.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.693+0000 7f6fa805b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fa0108400 0x7f6fa019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:20.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.693+0000 7f6fa805b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa019d4c0 con 0x7f6fa0107200 2024-04-16T02:35:20.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.693+0000 7f6fa805b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa019d600 con 0x7f6fa0108400 2024-04-16T02:35:20.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.694+0000 7f6fa5df7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0107200 0x7f6fa019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:20.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.694+0000 7f6fa5df7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0107200 0x7f6fa019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34872/0 (socket says 172.21.15.151:34872) 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.694+0000 7f6fa5df7700 1 -- 172.21.15.151:0/3685434206 learned_addr learned my addr 172.21.15.151:0/3685434206 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.694+0000 7f6fa55f6700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fa0108400 0x7f6fa019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.694+0000 7f6fa5df7700 1 -- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fa0108400 msgr2=0x7f6fa019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.694+0000 7f6fa5df7700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fa0108400 0x7f6fa019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.694+0000 7f6fa5df7700 1 -- 172.21.15.151:0/3685434206 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f900095f0 con 0x7f6fa0107200 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.694+0000 7f6fa5df7700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0107200 0x7f6fa019c960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00e8f0 tx=0x7f6f9c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.695+0000 7f6f96ffd700 1 -- 172.21.15.151:0/3685434206 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c009790 con 0x7f6fa0107200 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.695+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa01a20b0 con 0x7f6fa0107200 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.695+0000 7f6f96ffd700 1 -- 172.21.15.151:0/3685434206 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6f9c00c810 con 0x7f6fa0107200 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.695+0000 7f6f96ffd700 1 -- 172.21.15.151:0/3685434206 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c003d20 con 0x7f6fa0107200 2024-04-16T02:35:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.695+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa01a2600 con 0x7f6fa0107200 2024-04-16T02:35:20.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.696+0000 7f6f96ffd700 1 -- 172.21.15.151:0/3685434206 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 24) v1 ==== 46699+0+0 (secure 0 0 0) 0x7f6f9c018450 con 0x7f6fa0107200 2024-04-16T02:35:20.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.696+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa010fd60 con 0x7f6fa0107200 2024-04-16T02:35:20.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.697+0000 7f6f96ffd700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f6f8c0398c0 0x7f6f8c03bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:20.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.697+0000 7f6f96ffd700 1 -- 172.21.15.151:0/3685434206 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f6f9c014070 con 0x7f6fa0107200 2024-04-16T02:35:20.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.697+0000 7f6fa55f6700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f6f8c0398c0 0x7f6f8c03bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:20.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.698+0000 7f6fa55f6700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f6f8c0398c0 0x7f6f8c03bd80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000b6f0 tx=0x7f6f90005a90 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:20.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:20.704+0000 7f6f96ffd700 1 -- 172.21.15.151:0/3685434206 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f6f9c01f340 con 0x7f6fa0107200 2024-04-16T02:35:20.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: Standby manager daemon smithi151.uqqame started 2024-04-16T02:35:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.151:0/3063485531' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi151.uqqame/crt"}]: dispatch 2024-04-16T02:35:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.151:0/3063485531' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:35:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.151:0/3063485531' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi151.uqqame/key"}]: dispatch 2024-04-16T02:35:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.151:0/3063485531' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:35:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:20 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3750164039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:35:21.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.018+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fa010cd50 con 0x7f6f8c0398c0 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.025+0000 7f6f96ffd700 1 -- 172.21.15.151:0/3685434206 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f6fa010cd50 con 0x7f6f8c0398c0 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [], 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "1/2 daemons upgraded", 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "", 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:35:21.027 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:35:21.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.028+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f6f8c0398c0 msgr2=0x7f6f8c03bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.028+0000 7f6fa805b700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f6f8c0398c0 0x7f6f8c03bd80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000b6f0 tx=0x7f6f90005a90 comp rx=0 tx=0).stop 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.028+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0107200 msgr2=0x7f6fa019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.028+0000 7f6fa805b700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0107200 0x7f6fa019c960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00e8f0 tx=0x7f6f9c00ec00 comp rx=0 tx=0).stop 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.028+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 shutdown_connections 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.029+0000 7f6fa805b700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f6f8c0398c0 0x7f6f8c03bd80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.029+0000 7f6fa805b700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fa0107200 0x7f6fa019c960 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.029+0000 7f6fa805b700 1 --2- 172.21.15.151:0/3685434206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fa0108400 0x7f6fa019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.029+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 >> 172.21.15.151:0/3685434206 conn(0x7f6fa00761b0 msgr2=0x7f6fa010b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.029+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 shutdown_connections 2024-04-16T02:35:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:21.029+0000 7f6fa805b700 1 -- 172.21.15.151:0/3685434206 wait complete. 2024-04-16T02:35:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: Standby manager daemon smithi151.uqqame started 2024-04-16T02:35:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.151:0/3063485531' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi151.uqqame/crt"}]: dispatch 2024-04-16T02:35:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.151:0/3063485531' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:35:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.151:0/3063485531' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi151.uqqame/key"}]: dispatch 2024-04-16T02:35:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.151:0/3063485531' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:35:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:20 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3750164039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:35:22.072 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:21 smithi151 ceph-mon[29003]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:22.072 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:21 smithi151 ceph-mon[29003]: pgmap v8: 65 pgs: 65 active+clean; 174 MiB data, 852 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 267 op/s 2024-04-16T02:35:22.072 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:21 smithi151 ceph-mon[29003]: mgrmap e25: smithi153.esbtzf(active, since 10s), standbys: smithi151.uqqame 2024-04-16T02:35:22.072 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:21 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:35:22.072 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:21 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:22.073 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:21 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:21 smithi153 ceph-mon[36747]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:21 smithi153 ceph-mon[36747]: pgmap v8: 65 pgs: 65 active+clean; 174 MiB data, 852 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 267 op/s 2024-04-16T02:35:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:21 smithi153 ceph-mon[36747]: mgrmap e25: smithi153.esbtzf(active, since 10s), standbys: smithi151.uqqame 2024-04-16T02:35:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:21 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:35:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:21 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:21 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:22.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:22 smithi151 ceph-mon[29003]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:23.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:22 smithi153 ceph-mon[36747]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:23.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:23 smithi151 ceph-mon[29003]: pgmap v9: 65 pgs: 65 active+clean; 195 MiB data, 903 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 368 op/s 2024-04-16T02:35:24.016 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:23 smithi153 ceph-mon[36747]: pgmap v9: 65 pgs: 65 active+clean; 195 MiB data, 903 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 368 op/s 2024-04-16T02:35:25.452 INFO:tasks.workunit.client.1.smithi153.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-16T02:35:25.460 INFO:tasks.workunit.client.1.smithi153.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-16T02:35:25.460 INFO:tasks.workunit.client.1.smithi153.stderr:+ make 2024-04-16T02:35:25.532 INFO:tasks.workunit.client.1.smithi153.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-16T02:35:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:25 smithi153 ceph-mon[36747]: pgmap v10: 65 pgs: 65 active+clean; 195 MiB data, 903 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-04-16T02:35:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:25 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:35:26.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:25 smithi151 ceph-mon[29003]: pgmap v10: 65 pgs: 65 active+clean; 195 MiB data, 903 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-04-16T02:35:26.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:25 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:35:26.286 INFO:tasks.workunit.client.1.smithi153.stderr:++ readlink -f fsstress 2024-04-16T02:35:26.288 INFO:tasks.workunit.client.1.smithi153.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-16T02:35:26.288 INFO:tasks.workunit.client.1.smithi153.stderr:+ popd 2024-04-16T02:35:26.290 INFO:tasks.workunit.client.1.smithi153.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-16T02:35:26.290 INFO:tasks.workunit.client.1.smithi153.stderr:+ popd 2024-04-16T02:35:26.290 INFO:tasks.workunit.client.1.smithi153.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-16T02:35:26.291 INFO:tasks.workunit.client.1.smithi153.stderr:++ mktemp -d -p . 2024-04-16T02:35:26.294 INFO:tasks.workunit.client.1.smithi153.stderr:+ T=./tmp.wcURSL0jp3 2024-04-16T02:35:26.294 INFO:tasks.workunit.client.1.smithi153.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.wcURSL0jp3 -l 1 -n 1000 -p 10 -v 2024-04-16T02:35:26.297 INFO:tasks.workunit.client.1.smithi153.stdout:seed = 1712940219 2024-04-16T02:35:26.303 INFO:tasks.workunit.client.1.smithi153.stdout:2/0: rmdir - no directory 2024-04-16T02:35:26.303 INFO:tasks.workunit.client.1.smithi153.stdout:2/1: chown . 662 1 2024-04-16T02:35:26.304 INFO:tasks.workunit.client.1.smithi153.stdout:2/2: stat - no entries 2024-04-16T02:35:26.304 INFO:tasks.workunit.client.1.smithi153.stdout:2/3: dwrite - no filename 2024-04-16T02:35:26.304 INFO:tasks.workunit.client.1.smithi153.stdout:2/4: rmdir - no directory 2024-04-16T02:35:26.304 INFO:tasks.workunit.client.1.smithi153.stdout:2/5: truncate - no filename 2024-04-16T02:35:26.304 INFO:tasks.workunit.client.1.smithi153.stdout:0/0: creat f0 x:0 0 0 2024-04-16T02:35:26.305 INFO:tasks.workunit.client.1.smithi153.stdout:3/0: write - no filename 2024-04-16T02:35:26.306 INFO:tasks.workunit.client.1.smithi153.stdout:1/0: creat f0 x:0 0 0 2024-04-16T02:35:26.307 INFO:tasks.workunit.client.1.smithi153.stdout:0/1: rename f0 to f1 0 2024-04-16T02:35:26.307 INFO:tasks.workunit.client.1.smithi153.stdout:0/2: dread - f1 zero size 2024-04-16T02:35:26.308 INFO:tasks.workunit.client.1.smithi153.stdout:2/6: creat f0 x:0 0 0 2024-04-16T02:35:26.308 INFO:tasks.workunit.client.1.smithi153.stdout:0/3: dread - f1 zero size 2024-04-16T02:35:26.308 INFO:tasks.workunit.client.1.smithi153.stdout:2/7: dread - f0 zero size 2024-04-16T02:35:26.309 INFO:tasks.workunit.client.1.smithi153.stdout:2/8: dread - f0 zero size 2024-04-16T02:35:26.309 INFO:tasks.workunit.client.1.smithi153.stdout:2/9: rmdir - no directory 2024-04-16T02:35:26.310 INFO:tasks.workunit.client.1.smithi153.stdout:4/0: rename - no filename 2024-04-16T02:35:26.310 INFO:tasks.workunit.client.1.smithi153.stdout:4/1: chown . 255529 1 2024-04-16T02:35:26.310 INFO:tasks.workunit.client.1.smithi153.stdout:4/2: rename - no filename 2024-04-16T02:35:26.310 INFO:tasks.workunit.client.1.smithi153.stdout:1/1: symlink l1 0 2024-04-16T02:35:26.310 INFO:tasks.workunit.client.1.smithi153.stdout:1/2: fdatasync f0 0 2024-04-16T02:35:26.311 INFO:tasks.workunit.client.1.smithi153.stdout:0/4: mknod c2 0 2024-04-16T02:35:26.311 INFO:tasks.workunit.client.1.smithi153.stdout:0/5: dread - f1 zero size 2024-04-16T02:35:26.312 INFO:tasks.workunit.client.1.smithi153.stdout:1/3: dread - f0 zero size 2024-04-16T02:35:26.312 INFO:tasks.workunit.client.1.smithi153.stdout:1/4: rmdir - no directory 2024-04-16T02:35:26.312 INFO:tasks.workunit.client.1.smithi153.stdout:3/1: creat f0 x:0 0 0 2024-04-16T02:35:26.312 INFO:tasks.workunit.client.1.smithi153.stdout:1/5: fdatasync f0 0 2024-04-16T02:35:26.313 INFO:tasks.workunit.client.1.smithi153.stdout:5/0: write - no filename 2024-04-16T02:35:26.313 INFO:tasks.workunit.client.1.smithi153.stdout:2/10: creat f1 x:0 0 0 2024-04-16T02:35:26.314 INFO:tasks.workunit.client.1.smithi153.stdout:3/2: creat f1 x:0 0 0 2024-04-16T02:35:26.315 INFO:tasks.workunit.client.1.smithi153.stdout:4/3: creat f0 x:0 0 0 2024-04-16T02:35:26.315 INFO:tasks.workunit.client.1.smithi153.stdout:0/6: link f1 f3 0 2024-04-16T02:35:26.315 INFO:tasks.workunit.client.1.smithi153.stdout:2/11: truncate f1 135075 0 2024-04-16T02:35:26.316 INFO:tasks.workunit.client.1.smithi153.stdout:2/12: fdatasync f1 0 2024-04-16T02:35:26.316 INFO:tasks.workunit.client.1.smithi153.stdout:1/6: mknod c2 0 2024-04-16T02:35:26.318 INFO:tasks.workunit.client.1.smithi153.stdout:3/3: mknod c2 0 2024-04-16T02:35:26.321 INFO:tasks.workunit.client.1.smithi153.stdout:3/4: chown f1 87640 1 2024-04-16T02:35:26.321 INFO:tasks.workunit.client.1.smithi153.stdout:0/7: write f1 [135906,33444] 0 2024-04-16T02:35:26.326 INFO:tasks.workunit.client.1.smithi153.stdout:7/0: dwrite - no filename 2024-04-16T02:35:26.326 INFO:tasks.workunit.client.1.smithi153.stdout:7/1: fdatasync - no filename 2024-04-16T02:35:26.327 INFO:tasks.workunit.client.1.smithi153.stdout:2/13: mknod c2 0 2024-04-16T02:35:26.327 INFO:tasks.workunit.client.1.smithi153.stdout:0/8: write f3 [988416,5187] 0 2024-04-16T02:35:26.327 INFO:tasks.workunit.client.1.smithi153.stdout:0/9: readlink - no filename 2024-04-16T02:35:26.327 INFO:tasks.workunit.client.1.smithi153.stdout:0/10: readlink - no filename 2024-04-16T02:35:26.327 INFO:tasks.workunit.client.1.smithi153.stdout:5/1: symlink l0 0 2024-04-16T02:35:26.328 INFO:tasks.workunit.client.1.smithi153.stdout:3/5: unlink f0 0 2024-04-16T02:35:26.328 INFO:tasks.workunit.client.1.smithi153.stdout:3/6: rmdir - no directory 2024-04-16T02:35:26.330 INFO:tasks.workunit.client.1.smithi153.stdout:3/7: write f1 [743937,45221] 0 2024-04-16T02:35:26.331 INFO:tasks.workunit.client.1.smithi153.stdout:4/4: dwrite f0 [0,4194304] 0 2024-04-16T02:35:26.331 INFO:tasks.workunit.client.1.smithi153.stdout:6/0: symlink l0 0 2024-04-16T02:35:26.331 INFO:tasks.workunit.client.1.smithi153.stdout:6/1: dread - no filename 2024-04-16T02:35:26.331 INFO:tasks.workunit.client.1.smithi153.stdout:8/0: dread - no filename 2024-04-16T02:35:26.333 INFO:tasks.workunit.client.1.smithi153.stdout:1/7: dwrite f0 [0,4194304] 0 2024-04-16T02:35:26.334 INFO:tasks.workunit.client.1.smithi153.stdout:2/14: mknod c3 0 2024-04-16T02:35:26.336 INFO:tasks.workunit.client.1.smithi153.stdout:5/2: unlink l0 0 2024-04-16T02:35:26.336 INFO:tasks.workunit.client.1.smithi153.stdout:5/3: fsync - no filename 2024-04-16T02:35:26.336 INFO:tasks.workunit.client.1.smithi153.stdout:5/4: dwrite - no filename 2024-04-16T02:35:26.336 INFO:tasks.workunit.client.1.smithi153.stdout:5/5: dread - no filename 2024-04-16T02:35:26.336 INFO:tasks.workunit.client.1.smithi153.stdout:5/6: dwrite - no filename 2024-04-16T02:35:26.339 INFO:tasks.workunit.client.1.smithi153.stdout:0/11: symlink l4 0 2024-04-16T02:35:26.341 INFO:tasks.workunit.client.1.smithi153.stdout:3/8: mknod c3 0 2024-04-16T02:35:26.343 INFO:tasks.workunit.client.1.smithi153.stdout:4/5: symlink l1 0 2024-04-16T02:35:26.347 INFO:tasks.workunit.client.1.smithi153.stdout:6/2: symlink l1 0 2024-04-16T02:35:26.347 INFO:tasks.workunit.client.1.smithi153.stdout:1/8: dread f0 [0,4194304] 0 2024-04-16T02:35:26.349 INFO:tasks.workunit.client.1.smithi153.stdout:7/2: mkdir d0 0 2024-04-16T02:35:26.349 INFO:tasks.workunit.client.1.smithi153.stdout:2/15: dwrite f0 [0,4194304] 0 2024-04-16T02:35:26.351 INFO:tasks.workunit.client.1.smithi153.stdout:5/7: mknod c1 0 2024-04-16T02:35:26.354 INFO:tasks.workunit.client.1.smithi153.stdout:0/12: creat f5 x:0 0 0 2024-04-16T02:35:26.354 INFO:tasks.workunit.client.1.smithi153.stdout:3/9: mknod c4 0 2024-04-16T02:35:26.361 INFO:tasks.workunit.client.1.smithi153.stdout:2/16: creat f4 x:0 0 0 2024-04-16T02:35:26.361 INFO:tasks.workunit.client.1.smithi153.stdout:8/1: creat f0 x:0 0 0 2024-04-16T02:35:26.362 INFO:tasks.workunit.client.1.smithi153.stdout:3/10: write f1 [978885,55920] 0 2024-04-16T02:35:26.362 INFO:tasks.workunit.client.1.smithi153.stdout:6/3: creat f2 x:0 0 0 2024-04-16T02:35:26.363 INFO:tasks.workunit.client.1.smithi153.stdout:5/8: symlink l2 0 2024-04-16T02:35:26.363 INFO:tasks.workunit.client.1.smithi153.stdout:5/9: dread - no filename 2024-04-16T02:35:26.364 INFO:tasks.workunit.client.1.smithi153.stdout:8/2: write f0 [260111,38122] 0 2024-04-16T02:35:26.366 INFO:tasks.workunit.client.1.smithi153.stdout:4/6: link f0 f2 0 2024-04-16T02:35:26.366 INFO:tasks.workunit.client.1.smithi153.stdout:4/7: chown l1 1542 1 2024-04-16T02:35:26.366 INFO:tasks.workunit.client.1.smithi153.stdout:5/10: chown c1 2037381 1 2024-04-16T02:35:26.366 INFO:tasks.workunit.client.1.smithi153.stdout:9/0: mknod c0 0 2024-04-16T02:35:26.366 INFO:tasks.workunit.client.1.smithi153.stdout:9/1: dwrite - no filename 2024-04-16T02:35:26.367 INFO:tasks.workunit.client.1.smithi153.stdout:5/11: chown c1 901346 1 2024-04-16T02:35:26.367 INFO:tasks.workunit.client.1.smithi153.stdout:5/12: rmdir - no directory 2024-04-16T02:35:26.367 INFO:tasks.workunit.client.1.smithi153.stdout:8/3: chown f0 412102 1 2024-04-16T02:35:26.367 INFO:tasks.workunit.client.1.smithi153.stdout:7/3: symlink d0/l1 0 2024-04-16T02:35:26.367 INFO:tasks.workunit.client.1.smithi153.stdout:7/4: fsync - no filename 2024-04-16T02:35:26.367 INFO:tasks.workunit.client.1.smithi153.stdout:7/5: write - no filename 2024-04-16T02:35:26.369 INFO:tasks.workunit.client.1.smithi153.stdout:2/17: mknod c5 0 2024-04-16T02:35:26.370 INFO:tasks.workunit.client.1.smithi153.stdout:2/18: write f1 [1073726,39460] 0 2024-04-16T02:35:26.371 INFO:tasks.workunit.client.1.smithi153.stdout:3/11: symlink l5 0 2024-04-16T02:35:26.372 INFO:tasks.workunit.client.1.smithi153.stdout:4/8: mkdir d3 0 2024-04-16T02:35:26.372 INFO:tasks.workunit.client.1.smithi153.stdout:3/12: readlink l5 0 2024-04-16T02:35:26.373 INFO:tasks.workunit.client.1.smithi153.stdout:9/2: rename c0 to c1 0 2024-04-16T02:35:26.375 INFO:tasks.workunit.client.1.smithi153.stdout:5/13: rename c1 to c3 0 2024-04-16T02:35:26.378 INFO:tasks.workunit.client.1.smithi153.stdout:7/6: creat d0/f2 x:0 0 0 2024-04-16T02:35:26.378 INFO:tasks.workunit.client.1.smithi153.stdout:2/19: creat f6 x:0 0 0 2024-04-16T02:35:26.379 INFO:tasks.workunit.client.1.smithi153.stdout:6/4: link f2 f3 0 2024-04-16T02:35:26.380 INFO:tasks.workunit.client.1.smithi153.stdout:3/13: mkdir d6 0 2024-04-16T02:35:26.381 INFO:tasks.workunit.client.1.smithi153.stdout:4/9: rename l1 to d3/l4 0 2024-04-16T02:35:26.382 INFO:tasks.workunit.client.1.smithi153.stdout:7/7: rename d0 to d0/d3 22 2024-04-16T02:35:26.383 INFO:tasks.workunit.client.1.smithi153.stdout:9/3: creat f2 x:0 0 0 2024-04-16T02:35:26.384 INFO:tasks.workunit.client.1.smithi153.stdout:9/4: dread - f2 zero size 2024-04-16T02:35:26.384 INFO:tasks.workunit.client.1.smithi153.stdout:7/8: readlink d0/l1 0 2024-04-16T02:35:26.384 INFO:tasks.workunit.client.1.smithi153.stdout:8/4: link f0 f1 0 2024-04-16T02:35:26.386 INFO:tasks.workunit.client.1.smithi153.stdout:2/20: mknod c7 0 2024-04-16T02:35:26.387 INFO:tasks.workunit.client.1.smithi153.stdout:9/5: write f2 [307732,72102] 0 2024-04-16T02:35:26.387 INFO:tasks.workunit.client.1.smithi153.stdout:6/5: rename f2 to f4 0 2024-04-16T02:35:26.387 INFO:tasks.workunit.client.1.smithi153.stdout:8/5: chown f1 155939 1 2024-04-16T02:35:26.390 INFO:tasks.workunit.client.1.smithi153.stdout:2/21: creat f8 x:0 0 0 2024-04-16T02:35:26.392 INFO:tasks.workunit.client.1.smithi153.stdout:8/6: unlink f0 0 2024-04-16T02:35:26.399 INFO:tasks.workunit.client.1.smithi153.stdout:3/14: dwrite f1 [0,4194304] 0 2024-04-16T02:35:26.401 INFO:tasks.workunit.client.1.smithi153.stdout:1/9: fsync f0 0 2024-04-16T02:35:26.402 INFO:tasks.workunit.client.1.smithi153.stdout:7/9: dwrite d0/f2 [0,4194304] 0 2024-04-16T02:35:26.405 INFO:tasks.workunit.client.1.smithi153.stdout:7/10: write d0/f2 [329712,48798] 0 2024-04-16T02:35:26.406 INFO:tasks.workunit.client.1.smithi153.stdout:0/13: fsync f3 0 2024-04-16T02:35:26.409 INFO:tasks.workunit.client.1.smithi153.stdout:1/10: mknod c3 0 2024-04-16T02:35:26.409 INFO:tasks.workunit.client.1.smithi153.stdout:2/22: dwrite f0 [0,4194304] 0 2024-04-16T02:35:26.413 INFO:tasks.workunit.client.1.smithi153.stdout:1/11: write f0 [2084577,95083] 0 2024-04-16T02:35:26.414 INFO:tasks.workunit.client.1.smithi153.stdout:3/15: mknod d6/c7 0 2024-04-16T02:35:26.415 INFO:tasks.workunit.client.1.smithi153.stdout:1/12: chown c2 425158446 1 2024-04-16T02:35:26.418 INFO:tasks.workunit.client.1.smithi153.stdout:2/23: mkdir d9 0 2024-04-16T02:35:26.421 INFO:tasks.workunit.client.1.smithi153.stdout:3/16: creat d6/f8 x:0 0 0 2024-04-16T02:35:26.422 INFO:tasks.workunit.client.1.smithi153.stdout:2/24: write f8 [215850,110370] 0 2024-04-16T02:35:26.422 INFO:tasks.workunit.client.1.smithi153.stdout:7/11: dread d0/f2 [0,4194304] 0 2024-04-16T02:35:26.431 INFO:tasks.workunit.client.1.smithi153.stdout:0/14: dwrite f1 [0,4194304] 0 2024-04-16T02:35:26.433 INFO:tasks.workunit.client.1.smithi153.stdout:2/25: dread f1 [0,4194304] 0 2024-04-16T02:35:26.434 INFO:tasks.workunit.client.1.smithi153.stdout:3/17: dread f1 [0,4194304] 0 2024-04-16T02:35:26.437 INFO:tasks.workunit.client.1.smithi153.stdout:9/6: chown c1 439992412 1 2024-04-16T02:35:26.439 INFO:tasks.workunit.client.1.smithi153.stdout:5/14: chown c3 26499 1 2024-04-16T02:35:26.440 INFO:tasks.workunit.client.1.smithi153.stdout:1/13: fdatasync f0 0 2024-04-16T02:35:26.444 INFO:tasks.workunit.client.1.smithi153.stdout:0/15: dwrite f1 [0,4194304] 0 2024-04-16T02:35:26.445 INFO:tasks.workunit.client.1.smithi153.stdout:3/18: mknod d6/c9 0 2024-04-16T02:35:26.446 INFO:tasks.workunit.client.1.smithi153.stdout:9/7: creat f3 x:0 0 0 2024-04-16T02:35:26.447 INFO:tasks.workunit.client.1.smithi153.stdout:4/10: readlink d3/l4 0 2024-04-16T02:35:26.449 INFO:tasks.workunit.client.1.smithi153.stdout:5/15: creat f4 x:0 0 0 2024-04-16T02:35:26.451 INFO:tasks.workunit.client.1.smithi153.stdout:2/26: dwrite f1 [0,4194304] 0 2024-04-16T02:35:26.451 INFO:tasks.workunit.client.1.smithi153.stdout:1/14: mkdir d4 0 2024-04-16T02:35:26.451 INFO:tasks.workunit.client.1.smithi153.stdout:0/16: mknod c6 0 2024-04-16T02:35:26.455 INFO:tasks.workunit.client.1.smithi153.stdout:2/27: read f0 [152408,129181] 0 2024-04-16T02:35:26.456 INFO:tasks.workunit.client.1.smithi153.stdout:4/11: unlink f0 0 2024-04-16T02:35:26.461 INFO:tasks.workunit.client.1.smithi153.stdout:2/28: write f4 [133173,106289] 0 2024-04-16T02:35:26.462 INFO:tasks.workunit.client.1.smithi153.stdout:3/19: dread f1 [0,4194304] 0 2024-04-16T02:35:26.465 INFO:tasks.workunit.client.1.smithi153.stdout:8/7: getdents . 0 2024-04-16T02:35:26.465 INFO:tasks.workunit.client.1.smithi153.stdout:2/29: write f8 [1117905,110976] 0 2024-04-16T02:35:26.467 INFO:tasks.workunit.client.1.smithi153.stdout:9/8: dwrite f3 [0,4194304] 0 2024-04-16T02:35:26.471 INFO:tasks.workunit.client.1.smithi153.stdout:0/17: mknod c7 0 2024-04-16T02:35:26.473 INFO:tasks.workunit.client.1.smithi153.stdout:7/12: truncate d0/f2 3532098 0 2024-04-16T02:35:26.474 INFO:tasks.workunit.client.1.smithi153.stdout:5/16: dwrite f4 [0,4194304] 0 2024-04-16T02:35:26.476 INFO:tasks.workunit.client.1.smithi153.stdout:0/18: write f3 [2282442,6659] 0 2024-04-16T02:35:26.476 INFO:tasks.workunit.client.1.smithi153.stdout:1/15: symlink d4/l5 0 2024-04-16T02:35:26.477 INFO:tasks.workunit.client.1.smithi153.stdout:0/19: rmdir - no directory 2024-04-16T02:35:26.480 INFO:tasks.workunit.client.1.smithi153.stdout:4/12: dread f2 [0,4194304] 0 2024-04-16T02:35:26.482 INFO:tasks.workunit.client.1.smithi153.stdout:9/9: unlink c1 0 2024-04-16T02:35:26.482 INFO:tasks.workunit.client.1.smithi153.stdout:9/10: rmdir - no directory 2024-04-16T02:35:26.485 INFO:tasks.workunit.client.1.smithi153.stdout:5/17: creat f5 x:0 0 0 2024-04-16T02:35:26.486 INFO:tasks.workunit.client.1.smithi153.stdout:2/30: dwrite f8 [0,4194304] 0 2024-04-16T02:35:26.489 INFO:tasks.workunit.client.1.smithi153.stdout:5/18: truncate f4 5049513 0 2024-04-16T02:35:26.492 INFO:tasks.workunit.client.1.smithi153.stdout:7/13: write d0/f2 [1308317,79302] 0 2024-04-16T02:35:26.493 INFO:tasks.workunit.client.1.smithi153.stdout:4/13: symlink d3/l5 0 2024-04-16T02:35:26.493 INFO:tasks.workunit.client.1.smithi153.stdout:0/20: dread f3 [0,4194304] 0 2024-04-16T02:35:26.493 INFO:tasks.workunit.client.1.smithi153.stdout:9/11: mkdir d4 0 2024-04-16T02:35:26.494 INFO:tasks.workunit.client.1.smithi153.stdout:7/14: chown d0 1315 1 2024-04-16T02:35:26.495 INFO:tasks.workunit.client.1.smithi153.stdout:0/21: write f5 [933908,40633] 0 2024-04-16T02:35:26.500 INFO:tasks.workunit.client.1.smithi153.stdout:3/20: dwrite f1 [0,4194304] 0 2024-04-16T02:35:26.502 INFO:tasks.workunit.client.1.smithi153.stdout:4/14: rename d3 to d3/d6 22 2024-04-16T02:35:26.503 INFO:tasks.workunit.client.1.smithi153.stdout:3/21: dread - d6/f8 zero size 2024-04-16T02:35:26.503 INFO:tasks.workunit.client.1.smithi153.stdout:9/12: dread f2 [0,4194304] 0 2024-04-16T02:35:26.505 INFO:tasks.workunit.client.1.smithi153.stdout:4/15: stat d3/l4 0 2024-04-16T02:35:26.505 INFO:tasks.workunit.client.1.smithi153.stdout:7/15: mkdir d0/d4 0 2024-04-16T02:35:26.507 INFO:tasks.workunit.client.1.smithi153.stdout:2/31: creat d9/fa x:0 0 0 2024-04-16T02:35:26.509 INFO:tasks.workunit.client.1.smithi153.stdout:1/16: link c3 d4/c6 0 2024-04-16T02:35:26.510 INFO:tasks.workunit.client.1.smithi153.stdout:7/16: write d0/f2 [3010574,38990] 0 2024-04-16T02:35:26.511 INFO:tasks.workunit.client.1.smithi153.stdout:5/19: dwrite f4 [4194304,4194304] 0 2024-04-16T02:35:26.513 INFO:tasks.workunit.client.1.smithi153.stdout:3/22: rename l5 to d6/la 0 2024-04-16T02:35:26.516 INFO:tasks.workunit.client.1.smithi153.stdout:0/22: dwrite f5 [0,4194304] 0 2024-04-16T02:35:26.516 INFO:tasks.workunit.client.1.smithi153.stdout:8/8: write f1 [1006220,12461] 0 2024-04-16T02:35:26.518 INFO:tasks.workunit.client.1.smithi153.stdout:5/20: symlink l6 0 2024-04-16T02:35:26.518 INFO:tasks.workunit.client.1.smithi153.stdout:0/23: chown f3 0 1 2024-04-16T02:35:26.520 INFO:tasks.workunit.client.1.smithi153.stdout:2/32: rename f4 to d9/fb 0 2024-04-16T02:35:26.520 INFO:tasks.workunit.client.1.smithi153.stdout:0/24: readlink l4 0 2024-04-16T02:35:26.521 INFO:tasks.workunit.client.1.smithi153.stdout:5/21: dread - f5 zero size 2024-04-16T02:35:26.523 INFO:tasks.workunit.client.1.smithi153.stdout:9/13: dread f3 [0,4194304] 0 2024-04-16T02:35:26.523 INFO:tasks.workunit.client.1.smithi153.stdout:7/17: dread d0/f2 [0,4194304] 0 2024-04-16T02:35:26.526 INFO:tasks.workunit.client.1.smithi153.stdout:0/25: chown l4 380219102 1 2024-04-16T02:35:26.529 INFO:tasks.workunit.client.1.smithi153.stdout:1/17: dread f0 [0,4194304] 0 2024-04-16T02:35:26.543 INFO:tasks.workunit.client.1.smithi153.stdout:3/23: rename c2 to d6/cb 0 2024-04-16T02:35:26.544 INFO:tasks.workunit.client.1.smithi153.stdout:8/9: mkdir d2 0 2024-04-16T02:35:26.545 INFO:tasks.workunit.client.1.smithi153.stdout:9/14: fdatasync f2 0 2024-04-16T02:35:26.546 INFO:tasks.workunit.client.1.smithi153.stdout:9/15: readlink - no filename 2024-04-16T02:35:26.548 INFO:tasks.workunit.client.1.smithi153.stdout:2/33: creat d9/fc x:0 0 0 2024-04-16T02:35:26.548 INFO:tasks.workunit.client.1.smithi153.stdout:3/24: write f1 [3089861,129921] 0 2024-04-16T02:35:26.548 INFO:tasks.workunit.client.1.smithi153.stdout:0/26: creat f8 x:0 0 0 2024-04-16T02:35:26.552 INFO:tasks.workunit.client.1.smithi153.stdout:2/34: dread - d9/fa zero size 2024-04-16T02:35:26.553 INFO:tasks.workunit.client.1.smithi153.stdout:5/22: dwrite f4 [4194304,4194304] 0 2024-04-16T02:35:26.554 INFO:tasks.workunit.client.1.smithi153.stdout:4/16: dwrite f2 [0,4194304] 0 2024-04-16T02:35:26.558 INFO:tasks.workunit.client.1.smithi153.stdout:7/18: dwrite d0/f2 [0,4194304] 0 2024-04-16T02:35:26.564 INFO:tasks.workunit.client.1.smithi153.stdout:4/17: dwrite f2 [0,4194304] 0 2024-04-16T02:35:26.565 INFO:tasks.workunit.client.1.smithi153.stdout:4/18: chown d3/l4 2386124 1 2024-04-16T02:35:26.567 INFO:tasks.workunit.client.1.smithi153.stdout:7/19: dread d0/f2 [0,4194304] 0 2024-04-16T02:35:26.782 INFO:tasks.workunit.client.1.smithi153.stdout:1/18: rename f0 to d4/f7 0 2024-04-16T02:35:26.782 INFO:tasks.workunit.client.1.smithi153.stdout:1/19: chown d4 1 1 2024-04-16T02:35:26.783 INFO:tasks.workunit.client.1.smithi153.stdout:0/27: creat f9 x:0 0 0 2024-04-16T02:35:26.783 INFO:tasks.workunit.client.1.smithi153.stdout:3/25: mknod d6/cc 0 2024-04-16T02:35:26.785 INFO:tasks.workunit.client.1.smithi153.stdout:2/35: mkdir d9/dd 0 2024-04-16T02:35:26.787 INFO:tasks.workunit.client.1.smithi153.stdout:4/19: mknod d3/c7 0 2024-04-16T02:35:26.789 INFO:tasks.workunit.client.1.smithi153.stdout:8/10: unlink f1 0 2024-04-16T02:35:26.789 INFO:tasks.workunit.client.1.smithi153.stdout:8/11: dread - no filename 2024-04-16T02:35:26.792 INFO:tasks.workunit.client.1.smithi153.stdout:9/16: rename f3 to d4/f5 0 2024-04-16T02:35:26.794 INFO:tasks.workunit.client.1.smithi153.stdout:3/26: mknod d6/cd 0 2024-04-16T02:35:26.796 INFO:tasks.workunit.client.1.smithi153.stdout:7/20: symlink d0/d4/l5 0 2024-04-16T02:35:26.797 INFO:tasks.workunit.client.1.smithi153.stdout:4/20: mknod d3/c8 0 2024-04-16T02:35:26.798 INFO:tasks.workunit.client.1.smithi153.stdout:4/21: write f2 [3075479,13729] 0 2024-04-16T02:35:26.801 INFO:tasks.workunit.client.1.smithi153.stdout:2/36: creat d9/dd/fe x:0 0 0 2024-04-16T02:35:26.802 INFO:tasks.workunit.client.1.smithi153.stdout:3/27: mkdir d6/de 0 2024-04-16T02:35:26.803 INFO:tasks.workunit.client.1.smithi153.stdout:7/21: read d0/f2 [612007,54181] 0 2024-04-16T02:35:26.804 INFO:tasks.workunit.client.1.smithi153.stdout:8/12: symlink d2/l3 0 2024-04-16T02:35:26.805 INFO:tasks.workunit.client.1.smithi153.stdout:9/17: dwrite d4/f5 [0,4194304] 0 2024-04-16T02:35:26.805 INFO:tasks.workunit.client.1.smithi153.stdout:8/13: rename d2 to d2/d4 22 2024-04-16T02:35:26.805 INFO:tasks.workunit.client.1.smithi153.stdout:8/14: dwrite - no filename 2024-04-16T02:35:26.805 INFO:tasks.workunit.client.1.smithi153.stdout:8/15: readlink d2/l3 0 2024-04-16T02:35:26.806 INFO:tasks.workunit.client.1.smithi153.stdout:4/22: rmdir d3 39 2024-04-16T02:35:26.808 INFO:tasks.workunit.client.1.smithi153.stdout:2/37: creat d9/dd/ff x:0 0 0 2024-04-16T02:35:26.810 INFO:tasks.workunit.client.1.smithi153.stdout:3/28: symlink d6/lf 0 2024-04-16T02:35:26.812 INFO:tasks.workunit.client.1.smithi153.stdout:7/22: write d0/f2 [2050483,32397] 0 2024-04-16T02:35:26.814 INFO:tasks.workunit.client.1.smithi153.stdout:9/18: mkdir d4/d6 0 2024-04-16T02:35:26.814 INFO:tasks.workunit.client.1.smithi153.stdout:4/23: dwrite f2 [4194304,4194304] 0 2024-04-16T02:35:26.815 INFO:tasks.workunit.client.1.smithi153.stdout:7/23: fsync d0/f2 0 2024-04-16T02:35:26.816 INFO:tasks.workunit.client.1.smithi153.stdout:8/16: creat d2/f5 x:0 0 0 2024-04-16T02:35:26.816 INFO:tasks.workunit.client.1.smithi153.stdout:7/24: write d0/f2 [21139,12167] 0 2024-04-16T02:35:26.817 INFO:tasks.workunit.client.1.smithi153.stdout:8/17: dread - d2/f5 zero size 2024-04-16T02:35:26.817 INFO:tasks.workunit.client.1.smithi153.stdout:7/25: chown d0/d4 40776 1 2024-04-16T02:35:26.818 INFO:tasks.workunit.client.1.smithi153.stdout:2/38: rename d9/fa to d9/f10 0 2024-04-16T02:35:26.818 INFO:tasks.workunit.client.1.smithi153.stdout:2/39: stat c3 0 2024-04-16T02:35:26.820 INFO:tasks.workunit.client.1.smithi153.stdout:3/29: readlink d6/la 0 2024-04-16T02:35:26.823 INFO:tasks.workunit.client.1.smithi153.stdout:9/19: dread f2 [0,4194304] 0 2024-04-16T02:35:26.823 INFO:tasks.workunit.client.1.smithi153.stdout:7/26: fdatasync d0/f2 0 2024-04-16T02:35:26.828 INFO:tasks.workunit.client.1.smithi153.stdout:9/20: dread d4/f5 [0,4194304] 0 2024-04-16T02:35:26.828 INFO:tasks.workunit.client.1.smithi153.stdout:2/40: symlink d9/dd/l11 0 2024-04-16T02:35:26.828 INFO:tasks.workunit.client.1.smithi153.stdout:9/21: stat d4/d6 0 2024-04-16T02:35:26.830 INFO:tasks.workunit.client.1.smithi153.stdout:4/24: mkdir d3/d9 0 2024-04-16T02:35:26.834 INFO:tasks.workunit.client.1.smithi153.stdout:2/41: unlink c3 0 2024-04-16T02:35:26.835 INFO:tasks.workunit.client.1.smithi153.stdout:3/30: getdents d6/de 0 2024-04-16T02:35:26.837 INFO:tasks.workunit.client.1.smithi153.stdout:4/25: rename f2 to d3/fa 0 2024-04-16T02:35:26.841 INFO:tasks.workunit.client.1.smithi153.stdout:2/42: mknod d9/c12 0 2024-04-16T02:35:26.842 INFO:tasks.workunit.client.1.smithi153.stdout:3/31: dwrite d6/f8 [0,4194304] 0 2024-04-16T02:35:26.842 INFO:tasks.workunit.client.1.smithi153.stdout:3/32: chown d6 472756 1 2024-04-16T02:35:26.844 INFO:tasks.workunit.client.1.smithi153.stdout:7/27: link d0/d4/l5 d0/d4/l6 0 2024-04-16T02:35:26.845 INFO:tasks.workunit.client.1.smithi153.stdout:2/43: symlink d9/dd/l13 0 2024-04-16T02:35:26.846 INFO:tasks.workunit.client.1.smithi153.stdout:3/33: mkdir d6/d10 0 2024-04-16T02:35:26.849 INFO:tasks.workunit.client.1.smithi153.stdout:3/34: chown d6/f8 0 1 2024-04-16T02:35:26.854 INFO:tasks.workunit.client.1.smithi153.stdout:4/26: dwrite d3/fa [0,4194304] 0 2024-04-16T02:35:26.855 INFO:tasks.workunit.client.1.smithi153.stdout:3/35: link d6/f8 d6/f11 0 2024-04-16T02:35:26.856 INFO:tasks.workunit.client.1.smithi153.stdout:7/28: dwrite d0/f2 [0,4194304] 0 2024-04-16T02:35:26.856 INFO:tasks.workunit.client.1.smithi153.stdout:3/36: chown d6 0 1 2024-04-16T02:35:26.869 INFO:tasks.workunit.client.1.smithi153.stdout:3/37: dwrite d6/f11 [0,4194304] 0 2024-04-16T02:35:26.870 INFO:tasks.workunit.client.1.smithi153.stdout:7/29: dwrite d0/f2 [0,4194304] 0 2024-04-16T02:35:26.877 INFO:tasks.workunit.client.1.smithi153.stdout:3/38: rename d6/f8 to d6/f12 0 2024-04-16T02:35:26.880 INFO:tasks.workunit.client.1.smithi153.stdout:3/39: write d6/f11 [426875,14052] 0 2024-04-16T02:35:26.885 INFO:tasks.workunit.client.1.smithi153.stdout:3/40: dread f1 [0,4194304] 0 2024-04-16T02:35:26.889 INFO:tasks.workunit.client.1.smithi153.stdout:3/41: rmdir d6/de 0 2024-04-16T02:35:26.892 INFO:tasks.workunit.client.1.smithi153.stdout:3/42: creat d6/d10/f13 x:0 0 0 2024-04-16T02:35:26.894 INFO:tasks.workunit.client.1.smithi153.stdout:3/43: symlink d6/l14 0 2024-04-16T02:35:26.917 INFO:tasks.workunit.client.1.smithi153.stdout:1/20: fsync d4/f7 0 2024-04-16T02:35:26.919 INFO:tasks.workunit.client.1.smithi153.stdout:0/28: getdents . 0 2024-04-16T02:35:26.921 INFO:tasks.workunit.client.1.smithi153.stdout:0/29: write f1 [2657251,102407] 0 2024-04-16T02:35:26.922 INFO:tasks.workunit.client.1.smithi153.stdout:1/21: unlink d4/c6 0 2024-04-16T02:35:26.924 INFO:tasks.workunit.client.1.smithi153.stdout:0/30: mknod ca 0 2024-04-16T02:35:26.924 INFO:tasks.workunit.client.1.smithi153.stdout:0/31: chown f8 600749 1 2024-04-16T02:35:26.925 INFO:tasks.workunit.client.1.smithi153.stdout:1/22: rmdir d4 39 2024-04-16T02:35:26.926 INFO:tasks.workunit.client.1.smithi153.stdout:9/22: truncate f2 426787 0 2024-04-16T02:35:26.929 INFO:tasks.workunit.client.1.smithi153.stdout:5/23: truncate f4 3138868 0 2024-04-16T02:35:26.931 INFO:tasks.workunit.client.1.smithi153.stdout:1/23: creat d4/f8 x:0 0 0 2024-04-16T02:35:26.931 INFO:tasks.workunit.client.1.smithi153.stdout:0/32: dwrite f5 [0,4194304] 0 2024-04-16T02:35:26.933 INFO:tasks.workunit.client.1.smithi153.stdout:9/23: dread d4/f5 [0,4194304] 0 2024-04-16T02:35:26.933 INFO:tasks.workunit.client.1.smithi153.stdout:9/24: chown d4/d6 273943612 1 2024-04-16T02:35:26.936 INFO:tasks.workunit.client.1.smithi153.stdout:1/24: symlink d4/l9 0 2024-04-16T02:35:26.937 INFO:tasks.workunit.client.1.smithi153.stdout:1/25: write d4/f7 [1683411,80895] 0 2024-04-16T02:35:26.937 INFO:tasks.workunit.client.1.smithi153.stdout:0/33: rename c6 to cb 0 2024-04-16T02:35:26.939 INFO:tasks.workunit.client.1.smithi153.stdout:9/25: write d4/f5 [999812,73994] 0 2024-04-16T02:35:26.942 INFO:tasks.workunit.client.1.smithi153.stdout:0/34: dread f1 [0,4194304] 0 2024-04-16T02:35:26.942 INFO:tasks.workunit.client.1.smithi153.stdout:9/26: mknod d4/c7 0 2024-04-16T02:35:26.943 INFO:tasks.workunit.client.1.smithi153.stdout:9/27: write d4/f5 [37323,55968] 0 2024-04-16T02:35:26.944 INFO:tasks.workunit.client.1.smithi153.stdout:8/18: rmdir d2 39 2024-04-16T02:35:26.945 INFO:tasks.workunit.client.1.smithi153.stdout:0/35: dread f3 [0,4194304] 0 2024-04-16T02:35:26.946 INFO:tasks.workunit.client.1.smithi153.stdout:2/44: rename d9/f10 to d9/dd/f14 0 2024-04-16T02:35:26.947 INFO:tasks.workunit.client.1.smithi153.stdout:9/28: rename d4/d6 to d4/d6/d8 22 2024-04-16T02:35:26.947 INFO:tasks.workunit.client.1.smithi153.stdout:2/45: chown f1 1808782365 1 2024-04-16T02:35:26.950 INFO:tasks.workunit.client.1.smithi153.stdout:8/19: creat d2/f6 x:0 0 0 2024-04-16T02:35:26.950 INFO:tasks.workunit.client.1.smithi153.stdout:0/36: dread f1 [0,4194304] 0 2024-04-16T02:35:26.951 INFO:tasks.workunit.client.1.smithi153.stdout:0/37: chown f1 104771 1 2024-04-16T02:35:26.951 INFO:tasks.workunit.client.1.smithi153.stdout:0/38: readlink l4 0 2024-04-16T02:35:26.951 INFO:tasks.workunit.client.1.smithi153.stdout:2/46: mknod d9/dd/c15 0 2024-04-16T02:35:26.952 INFO:tasks.workunit.client.1.smithi153.stdout:2/47: chown d9/dd/l11 43 1 2024-04-16T02:35:26.955 INFO:tasks.workunit.client.1.smithi153.stdout:8/20: rename d2/f5 to d2/f7 0 2024-04-16T02:35:26.955 INFO:tasks.workunit.client.1.smithi153.stdout:8/21: stat d2/f6 0 2024-04-16T02:35:26.955 INFO:tasks.workunit.client.1.smithi153.stdout:8/22: dread - d2/f6 zero size 2024-04-16T02:35:26.957 INFO:tasks.workunit.client.1.smithi153.stdout:7/30: rmdir d0 39 2024-04-16T02:35:26.959 INFO:tasks.workunit.client.1.smithi153.stdout:2/48: write d9/dd/f14 [920665,61912] 0 2024-04-16T02:35:26.960 INFO:tasks.workunit.client.1.smithi153.stdout:8/23: symlink d2/l8 0 2024-04-16T02:35:26.961 INFO:tasks.workunit.client.1.smithi153.stdout:8/24: read - d2/f7 zero size 2024-04-16T02:35:26.963 INFO:tasks.workunit.client.1.smithi153.stdout:0/39: dwrite f3 [4194304,4194304] 0 2024-04-16T02:35:26.963 INFO:tasks.workunit.client.1.smithi153.stdout:2/49: creat d9/dd/f16 x:0 0 0 2024-04-16T02:35:26.963 INFO:tasks.workunit.client.1.smithi153.stdout:0/40: rmdir - no directory 2024-04-16T02:35:26.963 INFO:tasks.workunit.client.1.smithi153.stdout:8/25: truncate d2/f7 5699 0 2024-04-16T02:35:26.964 INFO:tasks.workunit.client.1.smithi153.stdout:8/26: chown d2 5472 1 2024-04-16T02:35:26.967 INFO:tasks.workunit.client.1.smithi153.stdout:7/31: dread d0/f2 [0,4194304] 0 2024-04-16T02:35:26.970 INFO:tasks.workunit.client.1.smithi153.stdout:0/41: dread f3 [0,4194304] 0 2024-04-16T02:35:26.970 INFO:tasks.workunit.client.1.smithi153.stdout:2/50: creat d9/dd/f17 x:0 0 0 2024-04-16T02:35:26.970 INFO:tasks.workunit.client.1.smithi153.stdout:4/27: truncate d3/fa 4327686 0 2024-04-16T02:35:26.975 INFO:tasks.workunit.client.1.smithi153.stdout:7/32: unlink d0/d4/l6 0 2024-04-16T02:35:26.976 INFO:tasks.workunit.client.1.smithi153.stdout:3/44: truncate f1 3739041 0 2024-04-16T02:35:26.978 INFO:tasks.workunit.client.1.smithi153.stdout:3/45: fsync d6/d10/f13 0 2024-04-16T02:35:26.979 INFO:tasks.workunit.client.1.smithi153.stdout:0/42: dwrite f3 [0,4194304] 0 2024-04-16T02:35:26.981 INFO:tasks.workunit.client.1.smithi153.stdout:9/29: dread f2 [0,4194304] 0 2024-04-16T02:35:26.981 INFO:tasks.workunit.client.1.smithi153.stdout:3/46: symlink d6/l15 0 2024-04-16T02:35:26.982 INFO:tasks.workunit.client.1.smithi153.stdout:7/33: chown d0/d4/l5 86 1 2024-04-16T02:35:26.982 INFO:tasks.workunit.client.1.smithi153.stdout:9/30: write d4/f5 [2719671,11389] 0 2024-04-16T02:35:26.986 INFO:tasks.workunit.client.1.smithi153.stdout:0/43: mknod cc 0 2024-04-16T02:35:26.987 INFO:tasks.workunit.client.1.smithi153.stdout:5/24: dread f4 [0,4194304] 0 2024-04-16T02:35:26.987 INFO:tasks.workunit.client.1.smithi153.stdout:1/26: rmdir d4 39 2024-04-16T02:35:26.990 INFO:tasks.workunit.client.1.smithi153.stdout:7/34: write d0/f2 [369711,15308] 0 2024-04-16T02:35:26.993 INFO:tasks.workunit.client.1.smithi153.stdout:5/25: dwrite f5 [0,4194304] 0 2024-04-16T02:35:26.994 INFO:tasks.workunit.client.1.smithi153.stdout:5/26: read f4 [218525,130293] 0 2024-04-16T02:35:26.995 INFO:tasks.workunit.client.1.smithi153.stdout:0/44: mkdir dd 0 2024-04-16T02:35:26.996 INFO:tasks.workunit.client.1.smithi153.stdout:1/27: readlink d4/l5 0 2024-04-16T02:35:26.999 INFO:tasks.workunit.client.1.smithi153.stdout:7/35: creat d0/f7 x:0 0 0 2024-04-16T02:35:26.999 INFO:tasks.workunit.client.1.smithi153.stdout:0/45: dread f1 [0,4194304] 0 2024-04-16T02:35:27.051 INFO:tasks.workunit.client.1.smithi153.stdout:5/27: fsync f5 0 2024-04-16T02:35:27.054 INFO:tasks.workunit.client.1.smithi153.stdout:1/28: chown c3 230491 1 2024-04-16T02:35:27.055 INFO:tasks.workunit.client.1.smithi153.stdout:2/51: getdents d9/dd 0 2024-04-16T02:35:27.057 INFO:tasks.workunit.client.1.smithi153.stdout:3/47: rename d6/cb to d6/d10/c16 0 2024-04-16T02:35:27.058 INFO:tasks.workunit.client.1.smithi153.stdout:4/28: dread d3/fa [0,4194304] 0 2024-04-16T02:35:27.058 INFO:tasks.workunit.client.1.smithi153.stdout:7/36: symlink d0/l8 0 2024-04-16T02:35:27.061 INFO:tasks.workunit.client.1.smithi153.stdout:5/28: fsync f4 0 2024-04-16T02:35:27.063 INFO:tasks.workunit.client.1.smithi153.stdout:1/29: creat d4/fa x:0 0 0 2024-04-16T02:35:27.064 INFO:tasks.workunit.client.1.smithi153.stdout:1/30: stat d4/f7 0 2024-04-16T02:35:27.065 INFO:tasks.workunit.client.1.smithi153.stdout:1/31: fdatasync d4/fa 0 2024-04-16T02:35:27.065 INFO:tasks.workunit.client.1.smithi153.stdout:2/52: mkdir d9/dd/d18 0 2024-04-16T02:35:27.066 INFO:tasks.workunit.client.1.smithi153.stdout:3/48: mknod d6/c17 0 2024-04-16T02:35:27.067 INFO:tasks.workunit.client.1.smithi153.stdout:1/32: write d4/fa [864433,58137] 0 2024-04-16T02:35:27.067 INFO:tasks.workunit.client.1.smithi153.stdout:2/53: write d9/dd/f16 [761456,62722] 0 2024-04-16T02:35:27.069 INFO:tasks.workunit.client.1.smithi153.stdout:7/37: symlink d0/d4/l9 0 2024-04-16T02:35:27.071 INFO:tasks.workunit.client.1.smithi153.stdout:0/46: link f5 dd/fe 0 2024-04-16T02:35:27.071 INFO:tasks.workunit.client.1.smithi153.stdout:5/29: dwrite f4 [0,4194304] 0 2024-04-16T02:35:27.072 INFO:tasks.workunit.client.1.smithi153.stdout:4/29: mkdir d3/d9/db 0 2024-04-16T02:35:27.074 INFO:tasks.workunit.client.1.smithi153.stdout:5/30: chown f4 28391263 1 2024-04-16T02:35:27.074 INFO:tasks.workunit.client.1.smithi153.stdout:3/49: creat d6/d10/f18 x:0 0 0 2024-04-16T02:35:27.075 INFO:tasks.workunit.client.1.smithi153.stdout:7/38: truncate d0/f7 864363 0 2024-04-16T02:35:27.076 INFO:tasks.workunit.client.1.smithi153.stdout:1/33: unlink c3 0 2024-04-16T02:35:27.076 INFO:tasks.workunit.client.1.smithi153.stdout:5/31: chown f4 0 1 2024-04-16T02:35:27.076 INFO:tasks.workunit.client.1.smithi153.stdout:9/31: dwrite f2 [0,4194304] 0 2024-04-16T02:35:27.078 INFO:tasks.workunit.client.1.smithi153.stdout:1/34: chown l1 23418 1 2024-04-16T02:35:27.079 INFO:tasks.workunit.client.1.smithi153.stdout:5/32: write f5 [3831526,20740] 0 2024-04-16T02:35:27.081 INFO:tasks.workunit.client.1.smithi153.stdout:1/35: chown c2 1600 1 2024-04-16T02:35:27.084 INFO:tasks.workunit.client.1.smithi153.stdout:4/30: write d3/fa [4607481,46207] 0 2024-04-16T02:35:27.084 INFO:tasks.workunit.client.1.smithi153.stdout:3/50: creat d6/d10/f19 x:0 0 0 2024-04-16T02:35:27.087 INFO:tasks.workunit.client.1.smithi153.stdout:0/47: creat dd/ff x:0 0 0 2024-04-16T02:35:27.088 INFO:tasks.workunit.client.1.smithi153.stdout:2/54: dwrite d9/dd/f17 [0,4194304] 0 2024-04-16T02:35:27.089 INFO:tasks.workunit.client.1.smithi153.stdout:7/39: rename d0/f2 to d0/fa 0 2024-04-16T02:35:27.089 INFO:tasks.workunit.client.1.smithi153.stdout:7/40: readlink d0/l1 0 2024-04-16T02:35:27.091 INFO:tasks.workunit.client.1.smithi153.stdout:5/33: creat f7 x:0 0 0 2024-04-16T02:35:27.092 INFO:tasks.workunit.client.1.smithi153.stdout:5/34: dread - f7 zero size 2024-04-16T02:35:27.092 INFO:tasks.workunit.client.1.smithi153.stdout:5/35: readlink l2 0 2024-04-16T02:35:27.094 INFO:tasks.workunit.client.1.smithi153.stdout:1/36: rmdir d4 39 2024-04-16T02:35:27.096 INFO:tasks.workunit.client.1.smithi153.stdout:7/41: dread d0/f7 [0,4194304] 0 2024-04-16T02:35:27.106 INFO:tasks.workunit.client.1.smithi153.stdout:2/55: rename d9/dd/fe to d9/dd/f19 0 2024-04-16T02:35:27.106 INFO:tasks.workunit.client.1.smithi153.stdout:2/56: read - f6 zero size 2024-04-16T02:35:27.109 INFO:tasks.workunit.client.1.smithi153.stdout:2/57: truncate d9/dd/ff 1001688 0 2024-04-16T02:35:27.111 INFO:tasks.workunit.client.1.smithi153.stdout:1/37: write d4/fa [394856,85042] 0 2024-04-16T02:35:27.111 INFO:tasks.workunit.client.1.smithi153.stdout:7/42: write d0/fa [3665704,50457] 0 2024-04-16T02:35:27.111 INFO:tasks.workunit.client.1.smithi153.stdout:5/36: fsync f7 0 2024-04-16T02:35:27.113 INFO:tasks.workunit.client.1.smithi153.stdout:1/38: dread - d4/f8 zero size 2024-04-16T02:35:27.113 INFO:tasks.workunit.client.1.smithi153.stdout:7/43: readlink d0/d4/l9 0 2024-04-16T02:35:27.113 INFO:tasks.workunit.client.1.smithi153.stdout:3/51: stat f1 0 2024-04-16T02:35:27.115 INFO:tasks.workunit.client.1.smithi153.stdout:4/31: link d3/l4 d3/d9/lc 0 2024-04-16T02:35:27.115 INFO:tasks.workunit.client.1.smithi153.stdout:0/48: write f3 [3209608,12701] 0 2024-04-16T02:35:27.115 INFO:tasks.workunit.client.1.smithi153.stdout:1/39: readlink d4/l9 0 2024-04-16T02:35:27.116 INFO:tasks.workunit.client.1.smithi153.stdout:9/32: dwrite f2 [4194304,4194304] 0 2024-04-16T02:35:27.117 INFO:tasks.workunit.client.1.smithi153.stdout:2/58: unlink d9/dd/l13 0 2024-04-16T02:35:27.117 INFO:tasks.workunit.client.1.smithi153.stdout:7/44: write d0/fa [3836229,80363] 0 2024-04-16T02:35:27.119 INFO:tasks.workunit.client.1.smithi153.stdout:5/37: mkdir d8 0 2024-04-16T02:35:27.121 INFO:tasks.workunit.client.1.smithi153.stdout:3/52: creat d6/f1a x:0 0 0 2024-04-16T02:35:27.122 INFO:tasks.workunit.client.1.smithi153.stdout:4/32: mknod d3/cd 0 2024-04-16T02:35:27.125 INFO:tasks.workunit.client.1.smithi153.stdout:7/45: write d0/f7 [110314,124797] 0 2024-04-16T02:35:27.130 INFO:tasks.workunit.client.1.smithi153.stdout:2/59: link d9/fc d9/dd/f1a 0 2024-04-16T02:35:27.130 INFO:tasks.workunit.client.1.smithi153.stdout:0/49: dwrite f1 [0,4194304] 0 2024-04-16T02:35:27.130 INFO:tasks.workunit.client.1.smithi153.stdout:5/38: mknod d8/c9 0 2024-04-16T02:35:27.131 INFO:tasks.workunit.client.1.smithi153.stdout:1/40: link l1 d4/lb 0 2024-04-16T02:35:27.133 INFO:tasks.workunit.client.1.smithi153.stdout:7/46: mkdir d0/d4/db 0 2024-04-16T02:35:27.135 INFO:tasks.workunit.client.1.smithi153.stdout:3/53: dread d6/f12 [0,4194304] 0 2024-04-16T02:35:27.136 INFO:tasks.workunit.client.1.smithi153.stdout:5/39: unlink f7 0 2024-04-16T02:35:27.137 INFO:tasks.workunit.client.1.smithi153.stdout:4/33: dwrite d3/fa [0,4194304] 0 2024-04-16T02:35:27.137 INFO:tasks.workunit.client.1.smithi153.stdout:0/50: rmdir dd 39 2024-04-16T02:35:27.138 INFO:tasks.workunit.client.1.smithi153.stdout:3/54: dread - d6/d10/f18 zero size 2024-04-16T02:35:27.138 INFO:tasks.workunit.client.1.smithi153.stdout:0/51: fdatasync f9 0 2024-04-16T02:35:27.140 INFO:tasks.workunit.client.1.smithi153.stdout:3/55: write d6/d10/f19 [473169,100781] 0 2024-04-16T02:35:27.142 INFO:tasks.workunit.client.1.smithi153.stdout:1/41: mknod d4/cc 0 2024-04-16T02:35:27.142 INFO:tasks.workunit.client.1.smithi153.stdout:2/60: dread f8 [0,4194304] 0 2024-04-16T02:35:27.143 INFO:tasks.workunit.client.1.smithi153.stdout:1/42: write d4/f7 [1738135,114086] 0 2024-04-16T02:35:27.144 INFO:tasks.workunit.client.1.smithi153.stdout:1/43: chown c2 287524 1 2024-04-16T02:35:27.145 INFO:tasks.workunit.client.1.smithi153.stdout:4/34: dwrite d3/fa [4194304,4194304] 0 2024-04-16T02:35:27.152 INFO:tasks.workunit.client.1.smithi153.stdout:7/47: creat d0/d4/fc x:0 0 0 2024-04-16T02:35:27.152 INFO:tasks.workunit.client.1.smithi153.stdout:5/40: symlink d8/la 0 2024-04-16T02:35:27.153 INFO:tasks.workunit.client.1.smithi153.stdout:5/41: stat f5 0 2024-04-16T02:35:27.154 INFO:tasks.workunit.client.1.smithi153.stdout:3/56: rename d6/l14 to d6/d10/l1b 0 2024-04-16T02:35:27.155 INFO:tasks.workunit.client.1.smithi153.stdout:1/44: dwrite d4/fa [0,4194304] 0 2024-04-16T02:35:27.157 INFO:tasks.workunit.client.1.smithi153.stdout:2/61: mkdir d9/dd/d1b 0 2024-04-16T02:35:27.159 INFO:tasks.workunit.client.1.smithi153.stdout:9/33: truncate f2 5981810 0 2024-04-16T02:35:27.160 INFO:tasks.workunit.client.1.smithi153.stdout:9/34: chown d4 8116286 1 2024-04-16T02:35:27.160 INFO:tasks.workunit.client.1.smithi153.stdout:9/35: rename d4 to d4/d6/d9 22 2024-04-16T02:35:27.161 INFO:tasks.workunit.client.1.smithi153.stdout:0/52: creat dd/f10 x:0 0 0 2024-04-16T02:35:27.163 INFO:tasks.workunit.client.1.smithi153.stdout:3/57: creat d6/d10/f1c x:0 0 0 2024-04-16T02:35:27.165 INFO:tasks.workunit.client.1.smithi153.stdout:1/45: mkdir d4/dd 0 2024-04-16T02:35:27.166 INFO:tasks.workunit.client.1.smithi153.stdout:2/62: mkdir d9/dd/d1c 0 2024-04-16T02:35:27.168 INFO:tasks.workunit.client.1.smithi153.stdout:4/35: mkdir d3/d9/db/de 0 2024-04-16T02:35:27.169 INFO:tasks.workunit.client.1.smithi153.stdout:4/36: write d3/fa [800431,56530] 0 2024-04-16T02:35:27.169 INFO:tasks.workunit.client.1.smithi153.stdout:0/53: mknod dd/c11 0 2024-04-16T02:35:27.170 INFO:tasks.workunit.client.1.smithi153.stdout:4/37: write d3/fa [1863527,76950] 0 2024-04-16T02:35:27.170 INFO:tasks.workunit.client.1.smithi153.stdout:3/58: fsync d6/f11 0 2024-04-16T02:35:27.172 INFO:tasks.workunit.client.1.smithi153.stdout:1/46: creat d4/fe x:0 0 0 2024-04-16T02:35:27.175 INFO:tasks.workunit.client.1.smithi153.stdout:0/54: mknod dd/c12 0 2024-04-16T02:35:27.175 INFO:tasks.workunit.client.1.smithi153.stdout:4/38: fsync d3/fa 0 2024-04-16T02:35:27.176 INFO:tasks.workunit.client.1.smithi153.stdout:0/55: truncate f8 219078 0 2024-04-16T02:35:27.176 INFO:tasks.workunit.client.1.smithi153.stdout:0/56: chown f5 0 1 2024-04-16T02:35:27.178 INFO:tasks.workunit.client.1.smithi153.stdout:3/59: symlink d6/l1d 0 2024-04-16T02:35:27.178 INFO:tasks.workunit.client.1.smithi153.stdout:3/60: fsync d6/d10/f13 0 2024-04-16T02:35:27.180 INFO:tasks.workunit.client.1.smithi153.stdout:1/47: mkdir d4/df 0 2024-04-16T02:35:27.181 INFO:tasks.workunit.client.1.smithi153.stdout:1/48: write d4/fa [1912695,94306] 0 2024-04-16T02:35:27.182 INFO:tasks.workunit.client.1.smithi153.stdout:4/39: rename d3/fa to d3/ff 0 2024-04-16T02:35:27.185 INFO:tasks.workunit.client.1.smithi153.stdout:3/61: dwrite d6/f1a [0,4194304] 0 2024-04-16T02:35:27.185 INFO:tasks.workunit.client.1.smithi153.stdout:3/62: write d6/f1a [3073259,36996] 0 2024-04-16T02:35:27.187 INFO:tasks.workunit.client.1.smithi153.stdout:0/57: rename l4 to dd/l13 0 2024-04-16T02:35:27.190 INFO:tasks.workunit.client.1.smithi153.stdout:1/49: unlink d4/fa 0 2024-04-16T02:35:27.195 INFO:tasks.workunit.client.1.smithi153.stdout:3/63: creat d6/d10/f1e x:0 0 0 2024-04-16T02:35:27.197 INFO:tasks.workunit.client.1.smithi153.stdout:0/58: dwrite f9 [0,4194304] 0 2024-04-16T02:35:27.199 INFO:tasks.workunit.client.1.smithi153.stdout:7/48: truncate d0/fa 1820011 0 2024-04-16T02:35:27.200 INFO:tasks.workunit.client.1.smithi153.stdout:3/64: readlink d6/d10/l1b 0 2024-04-16T02:35:27.202 INFO:tasks.workunit.client.1.smithi153.stdout:9/36: truncate d4/f5 3882772 0 2024-04-16T02:35:27.202 INFO:tasks.workunit.client.1.smithi153.stdout:0/59: mkdir dd/d14 0 2024-04-16T02:35:27.203 INFO:tasks.workunit.client.1.smithi153.stdout:4/40: dwrite d3/ff [0,4194304] 0 2024-04-16T02:35:27.204 INFO:tasks.workunit.client.1.smithi153.stdout:3/65: write d6/f12 [686653,127383] 0 2024-04-16T02:35:27.205 INFO:tasks.workunit.client.1.smithi153.stdout:4/41: stat d3/l5 0 2024-04-16T02:35:27.206 INFO:tasks.workunit.client.1.smithi153.stdout:1/50: rmdir d4/df 0 2024-04-16T02:35:27.208 INFO:tasks.workunit.client.1.smithi153.stdout:7/49: mkdir d0/d4/db/dd 0 2024-04-16T02:35:27.210 INFO:tasks.workunit.client.1.smithi153.stdout:9/37: dread f2 [4194304,4194304] 0 2024-04-16T02:35:27.211 INFO:tasks.workunit.client.1.smithi153.stdout:7/50: write d0/f7 [1288661,92043] 0 2024-04-16T02:35:27.211 INFO:tasks.workunit.client.1.smithi153.stdout:3/66: symlink d6/d10/l1f 0 2024-04-16T02:35:27.212 INFO:tasks.workunit.client.1.smithi153.stdout:2/63: dwrite d9/dd/f1a [0,4194304] 0 2024-04-16T02:35:27.213 INFO:tasks.workunit.client.1.smithi153.stdout:9/38: rename d4/d6 to d4/d6/da 22 2024-04-16T02:35:27.213 INFO:tasks.workunit.client.1.smithi153.stdout:9/39: readlink - no filename 2024-04-16T02:35:27.213 INFO:tasks.workunit.client.1.smithi153.stdout:9/40: readlink - no filename 2024-04-16T02:35:27.213 INFO:tasks.workunit.client.1.smithi153.stdout:3/67: readlink d6/la 0 2024-04-16T02:35:27.213 INFO:tasks.workunit.client.1.smithi153.stdout:7/51: write d0/f7 [1738102,99287] 0 2024-04-16T02:35:27.214 INFO:tasks.workunit.client.1.smithi153.stdout:3/68: chown d6/l1d 12 1 2024-04-16T02:35:27.215 INFO:tasks.workunit.client.1.smithi153.stdout:0/60: stat dd/l13 0 2024-04-16T02:35:27.215 INFO:tasks.workunit.client.1.smithi153.stdout:2/64: write d9/dd/ff [1670736,44509] 0 2024-04-16T02:35:27.216 INFO:tasks.workunit.client.1.smithi153.stdout:3/69: write d6/d10/f18 [417354,30061] 0 2024-04-16T02:35:27.219 INFO:tasks.workunit.client.1.smithi153.stdout:3/70: chown d6/c9 4044 1 2024-04-16T02:35:27.219 INFO:tasks.workunit.client.1.smithi153.stdout:9/41: rmdir d4 39 2024-04-16T02:35:27.222 INFO:tasks.workunit.client.1.smithi153.stdout:7/52: symlink d0/d4/le 0 2024-04-16T02:35:27.223 INFO:tasks.workunit.client.1.smithi153.stdout:7/53: dread - d0/d4/fc zero size 2024-04-16T02:35:27.223 INFO:tasks.workunit.client.1.smithi153.stdout:0/61: symlink dd/l15 0 2024-04-16T02:35:27.225 INFO:tasks.workunit.client.1.smithi153.stdout:4/42: symlink d3/d9/db/de/l10 0 2024-04-16T02:35:27.228 INFO:tasks.workunit.client.1.smithi153.stdout:2/65: rename d9/dd/f1a to d9/dd/d1b/f1d 0 2024-04-16T02:35:27.231 INFO:tasks.workunit.client.1.smithi153.stdout:2/66: dread f1 [0,4194304] 0 2024-04-16T02:35:27.233 INFO:tasks.workunit.client.1.smithi153.stdout:0/62: creat dd/f16 x:0 0 0 2024-04-16T02:35:27.234 INFO:tasks.workunit.client.1.smithi153.stdout:1/51: link d4/l5 d4/dd/l10 0 2024-04-16T02:35:27.234 INFO:tasks.workunit.client.1.smithi153.stdout:0/63: write dd/f16 [368459,7119] 0 2024-04-16T02:35:27.235 INFO:tasks.workunit.client.1.smithi153.stdout:0/64: dread - dd/ff zero size 2024-04-16T02:35:27.235 INFO:tasks.workunit.client.1.smithi153.stdout:1/52: dread - d4/f8 zero size 2024-04-16T02:35:27.235 INFO:tasks.workunit.client.1.smithi153.stdout:4/43: mknod d3/c11 0 2024-04-16T02:35:27.236 INFO:tasks.workunit.client.1.smithi153.stdout:1/53: dread - d4/f8 zero size 2024-04-16T02:35:27.240 INFO:tasks.workunit.client.1.smithi153.stdout:2/67: symlink d9/dd/d1b/l1e 0 2024-04-16T02:35:27.242 INFO:tasks.workunit.client.1.smithi153.stdout:0/65: rename f3 to dd/d14/f17 0 2024-04-16T02:35:27.243 INFO:tasks.workunit.client.1.smithi153.stdout:7/54: rmdir d0 39 2024-04-16T02:35:27.244 INFO:tasks.workunit.client.1.smithi153.stdout:1/54: readlink d4/dd/l10 0 2024-04-16T02:35:27.249 INFO:tasks.workunit.client.1.smithi153.stdout:2/68: creat d9/dd/d1b/f1f x:0 0 0 2024-04-16T02:35:27.250 INFO:tasks.workunit.client.1.smithi153.stdout:1/55: dwrite d4/f7 [4194304,4194304] 0 2024-04-16T02:35:27.251 INFO:tasks.workunit.client.1.smithi153.stdout:1/56: chown d4/l9 1 1 2024-04-16T02:35:27.255 INFO:tasks.workunit.client.1.smithi153.stdout:7/55: read - d0/d4/fc zero size 2024-04-16T02:35:27.258 INFO:tasks.workunit.client.1.smithi153.stdout:1/57: readlink d4/l5 0 2024-04-16T02:35:27.259 INFO:tasks.workunit.client.1.smithi153.stdout:1/58: write d4/f8 [414253,98913] 0 2024-04-16T02:35:27.261 INFO:tasks.workunit.client.1.smithi153.stdout:9/42: fsync d4/f5 0 2024-04-16T02:35:27.265 INFO:tasks.workunit.client.1.smithi153.stdout:9/43: fdatasync f2 0 2024-04-16T02:35:27.266 INFO:tasks.workunit.client.1.smithi153.stdout:4/44: getdents d3/d9/db 0 2024-04-16T02:35:27.266 INFO:tasks.workunit.client.1.smithi153.stdout:9/44: fdatasync f2 0 2024-04-16T02:35:27.269 INFO:tasks.workunit.client.1.smithi153.stdout:4/45: chown d3/cd 173408276 1 2024-04-16T02:35:27.270 INFO:tasks.workunit.client.1.smithi153.stdout:9/45: creat d4/d6/fb x:0 0 0 2024-04-16T02:35:27.270 INFO:tasks.workunit.client.1.smithi153.stdout:4/46: fdatasync d3/ff 0 2024-04-16T02:35:27.271 INFO:tasks.workunit.client.1.smithi153.stdout:7/56: dread d0/fa [0,4194304] 0 2024-04-16T02:35:27.272 INFO:tasks.workunit.client.1.smithi153.stdout:0/66: dwrite dd/fe [0,4194304] 0 2024-04-16T02:35:27.276 INFO:tasks.workunit.client.1.smithi153.stdout:1/59: fdatasync d4/f7 0 2024-04-16T02:35:27.277 INFO:tasks.workunit.client.1.smithi153.stdout:4/47: creat d3/d9/db/de/f12 x:0 0 0 2024-04-16T02:35:27.277 INFO:tasks.workunit.client.1.smithi153.stdout:7/57: creat d0/d4/db/ff x:0 0 0 2024-04-16T02:35:27.280 INFO:tasks.workunit.client.1.smithi153.stdout:7/58: chown d0/d4/db/dd 50 1 2024-04-16T02:35:27.281 INFO:tasks.workunit.client.1.smithi153.stdout:4/48: write d3/ff [7787696,120080] 0 2024-04-16T02:35:27.281 INFO:tasks.workunit.client.1.smithi153.stdout:1/60: mknod d4/c11 0 2024-04-16T02:35:27.281 INFO:tasks.workunit.client.1.smithi153.stdout:7/59: chown d0/d4 3055 1 2024-04-16T02:35:27.282 INFO:tasks.workunit.client.1.smithi153.stdout:9/46: dwrite d4/d6/fb [0,4194304] 0 2024-04-16T02:35:27.283 INFO:tasks.workunit.client.1.smithi153.stdout:9/47: write d4/f5 [552547,20060] 0 2024-04-16T02:35:27.285 INFO:tasks.workunit.client.1.smithi153.stdout:1/61: mknod d4/c12 0 2024-04-16T02:35:27.285 INFO:tasks.workunit.client.1.smithi153.stdout:9/48: write d4/d6/fb [3808141,13079] 0 2024-04-16T02:35:27.285 INFO:tasks.workunit.client.1.smithi153.stdout:7/60: mknod d0/d4/c10 0 2024-04-16T02:35:27.293 INFO:tasks.workunit.client.1.smithi153.stdout:9/49: dwrite d4/f5 [0,4194304] 0 2024-04-16T02:35:27.294 INFO:tasks.workunit.client.1.smithi153.stdout:7/61: rename d0/l1 to d0/d4/db/dd/l11 0 2024-04-16T02:35:27.295 INFO:tasks.workunit.client.1.smithi153.stdout:1/62: mkdir d4/dd/d13 0 2024-04-16T02:35:27.297 INFO:tasks.workunit.client.1.smithi153.stdout:0/67: truncate f1 2894890 0 2024-04-16T02:35:27.299 INFO:tasks.workunit.client.1.smithi153.stdout:1/63: dread d4/f7 [4194304,4194304] 0 2024-04-16T02:35:27.299 INFO:tasks.workunit.client.1.smithi153.stdout:1/64: read - d4/fe zero size 2024-04-16T02:35:27.301 INFO:tasks.workunit.client.1.smithi153.stdout:7/62: creat d0/d4/db/f12 x:0 0 0 2024-04-16T02:35:27.301 INFO:tasks.workunit.client.1.smithi153.stdout:4/49: getdents d3/d9 0 2024-04-16T02:35:27.302 INFO:tasks.workunit.client.1.smithi153.stdout:4/50: readlink d3/d9/db/de/l10 0 2024-04-16T02:35:27.304 INFO:tasks.workunit.client.1.smithi153.stdout:0/68: creat dd/f18 x:0 0 0 2024-04-16T02:35:27.304 INFO:tasks.workunit.client.1.smithi153.stdout:0/69: dread - dd/f10 zero size 2024-04-16T02:35:27.305 INFO:tasks.workunit.client.1.smithi153.stdout:0/70: write dd/fe [3749582,17820] 0 2024-04-16T02:35:27.305 INFO:tasks.workunit.client.1.smithi153.stdout:7/63: mkdir d0/d13 0 2024-04-16T02:35:27.306 INFO:tasks.workunit.client.1.smithi153.stdout:4/51: rmdir d3/d9 39 2024-04-16T02:35:27.306 INFO:tasks.workunit.client.1.smithi153.stdout:0/71: read f8 [59263,73100] 0 2024-04-16T02:35:27.307 INFO:tasks.workunit.client.1.smithi153.stdout:0/72: fdatasync f9 0 2024-04-16T02:35:27.311 INFO:tasks.workunit.client.1.smithi153.stdout:7/64: mknod d0/d4/db/dd/c14 0 2024-04-16T02:35:27.313 INFO:tasks.workunit.client.1.smithi153.stdout:4/52: dread - d3/d9/db/de/f12 zero size 2024-04-16T02:35:27.313 INFO:tasks.workunit.client.1.smithi153.stdout:9/50: fsync d4/f5 0 2024-04-16T02:35:27.314 INFO:tasks.workunit.client.1.smithi153.stdout:9/51: write d4/d6/fb [3742185,2916] 0 2024-04-16T02:35:27.317 INFO:tasks.workunit.client.1.smithi153.stdout:7/65: rename d0/d4/le to d0/d4/db/dd/l15 0 2024-04-16T02:35:27.321 INFO:tasks.workunit.client.1.smithi153.stdout:4/53: creat d3/d9/db/de/f13 x:0 0 0 2024-04-16T02:35:27.321 INFO:tasks.workunit.client.1.smithi153.stdout:4/54: dread - d3/d9/db/de/f12 zero size 2024-04-16T02:35:27.322 INFO:tasks.workunit.client.1.smithi153.stdout:9/52: mkdir d4/d6/dc 0 2024-04-16T02:35:27.326 INFO:tasks.workunit.client.1.smithi153.stdout:9/53: creat d4/fd x:0 0 0 2024-04-16T02:35:27.329 INFO:tasks.workunit.client.1.smithi153.stdout:0/73: truncate f5 4139740 0 2024-04-16T02:35:27.329 INFO:tasks.workunit.client.1.smithi153.stdout:4/55: dwrite d3/d9/db/de/f12 [0,4194304] 0 2024-04-16T02:35:27.332 INFO:tasks.workunit.client.1.smithi153.stdout:7/66: write d0/fa [87722,84784] 0 2024-04-16T02:35:27.334 INFO:tasks.workunit.client.1.smithi153.stdout:4/56: creat d3/d9/f14 x:0 0 0 2024-04-16T02:35:27.336 INFO:tasks.workunit.client.1.smithi153.stdout:9/54: dwrite d4/f5 [0,4194304] 0 2024-04-16T02:35:27.337 INFO:tasks.workunit.client.1.smithi153.stdout:9/55: read d4/d6/fb [1149321,106790] 0 2024-04-16T02:35:27.338 INFO:tasks.workunit.client.1.smithi153.stdout:4/57: mknod d3/d9/c15 0 2024-04-16T02:35:27.339 INFO:tasks.workunit.client.1.smithi153.stdout:4/58: fsync d3/ff 0 2024-04-16T02:35:27.340 INFO:tasks.workunit.client.1.smithi153.stdout:4/59: read d3/ff [6013116,34477] 0 2024-04-16T02:35:27.341 INFO:tasks.workunit.client.1.smithi153.stdout:4/60: read d3/ff [3635857,95007] 0 2024-04-16T02:35:27.342 INFO:tasks.workunit.client.1.smithi153.stdout:0/74: write f1 [1837581,118754] 0 2024-04-16T02:35:27.343 INFO:tasks.workunit.client.1.smithi153.stdout:4/61: chown d3/ff 8933513 1 2024-04-16T02:35:27.346 INFO:tasks.workunit.client.1.smithi153.stdout:9/56: dwrite d4/d6/fb [0,4194304] 0 2024-04-16T02:35:27.346 INFO:tasks.workunit.client.1.smithi153.stdout:0/75: mkdir dd/d14/d19 0 2024-04-16T02:35:27.348 INFO:tasks.workunit.client.1.smithi153.stdout:9/57: write d4/fd [605098,73841] 0 2024-04-16T02:35:27.349 INFO:tasks.workunit.client.1.smithi153.stdout:4/62: getdents d3/d9/db 0 2024-04-16T02:35:27.351 INFO:tasks.workunit.client.1.smithi153.stdout:4/63: write d3/d9/db/de/f12 [2423287,97545] 0 2024-04-16T02:35:27.354 INFO:tasks.workunit.client.1.smithi153.stdout:4/64: dread - d3/d9/db/de/f13 zero size 2024-04-16T02:35:27.356 INFO:tasks.workunit.client.1.smithi153.stdout:9/58: dwrite f2 [0,4194304] 0 2024-04-16T02:35:27.358 INFO:tasks.workunit.client.1.smithi153.stdout:4/65: rmdir d3/d9/db 39 2024-04-16T02:35:27.360 INFO:tasks.workunit.client.1.smithi153.stdout:9/59: creat d4/d6/dc/fe x:0 0 0 2024-04-16T02:35:27.362 INFO:tasks.workunit.client.1.smithi153.stdout:9/60: read d4/f5 [2189453,79088] 0 2024-04-16T02:35:27.362 INFO:tasks.workunit.client.1.smithi153.stdout:4/66: symlink d3/l16 0 2024-04-16T02:35:27.369 INFO:tasks.workunit.client.1.smithi153.stdout:4/67: mkdir d3/d9/db/d17 0 2024-04-16T02:35:27.369 INFO:tasks.workunit.client.1.smithi153.stdout:0/76: dwrite dd/fe [0,4194304] 0 2024-04-16T02:35:27.370 INFO:tasks.workunit.client.1.smithi153.stdout:4/68: chown d3/d9/db 2 1 2024-04-16T02:35:27.371 INFO:tasks.workunit.client.1.smithi153.stdout:0/77: dread - dd/ff zero size 2024-04-16T02:35:27.372 INFO:tasks.workunit.client.1.smithi153.stdout:4/69: chown d3/d9/db/de/f12 212 1 2024-04-16T02:35:27.375 INFO:tasks.workunit.client.1.smithi153.stdout:4/70: chown d3/c7 123744072 1 2024-04-16T02:35:27.377 INFO:tasks.workunit.client.1.smithi153.stdout:0/78: getdents dd/d14/d19 0 2024-04-16T02:35:27.378 INFO:tasks.workunit.client.1.smithi153.stdout:9/61: dwrite d4/f5 [0,4194304] 0 2024-04-16T02:35:27.381 INFO:tasks.workunit.client.1.smithi153.stdout:0/79: rename f8 to dd/d14/d19/f1a 0 2024-04-16T02:35:27.384 INFO:tasks.workunit.client.1.smithi153.stdout:4/71: dwrite d3/d9/db/de/f12 [0,4194304] 0 2024-04-16T02:35:27.386 INFO:tasks.workunit.client.1.smithi153.stdout:9/62: rename f2 to d4/ff 0 2024-04-16T02:35:27.387 INFO:tasks.workunit.client.1.smithi153.stdout:0/80: dread dd/d14/f17 [0,4194304] 0 2024-04-16T02:35:27.388 INFO:tasks.workunit.client.1.smithi153.stdout:0/81: stat dd/l13 0 2024-04-16T02:35:27.389 INFO:tasks.workunit.client.1.smithi153.stdout:0/82: read f9 [4141592,57491] 0 2024-04-16T02:35:27.390 INFO:tasks.workunit.client.1.smithi153.stdout:9/63: dread d4/f5 [0,4194304] 0 2024-04-16T02:35:27.392 INFO:tasks.workunit.client.1.smithi153.stdout:4/72: getdents d3/d9/db/d17 0 2024-04-16T02:35:27.393 INFO:tasks.workunit.client.1.smithi153.stdout:4/73: fsync d3/d9/db/de/f13 0 2024-04-16T02:35:27.399 INFO:tasks.workunit.client.1.smithi153.stdout:0/83: link dd/ff dd/d14/d19/f1b 0 2024-04-16T02:35:27.401 INFO:tasks.workunit.client.1.smithi153.stdout:4/74: link d3/d9/db/de/l10 d3/d9/db/d17/l18 0 2024-04-16T02:35:27.403 INFO:tasks.workunit.client.1.smithi153.stdout:0/84: write dd/ff [594699,113777] 0 2024-04-16T02:35:27.403 INFO:tasks.workunit.client.1.smithi153.stdout:4/75: dread - d3/d9/f14 zero size 2024-04-16T02:35:27.405 INFO:tasks.workunit.client.1.smithi153.stdout:0/85: chown dd/f10 82 1 2024-04-16T02:35:27.405 INFO:tasks.workunit.client.1.smithi153.stdout:4/76: chown d3/c8 14 1 2024-04-16T02:35:27.405 INFO:tasks.workunit.client.1.smithi153.stdout:9/64: dwrite d4/ff [0,4194304] 0 2024-04-16T02:35:27.406 INFO:tasks.workunit.client.1.smithi153.stdout:0/86: dread - dd/f10 zero size 2024-04-16T02:35:27.407 INFO:tasks.workunit.client.1.smithi153.stdout:9/65: dread - d4/d6/dc/fe zero size 2024-04-16T02:35:27.410 INFO:tasks.workunit.client.1.smithi153.stdout:0/87: mkdir dd/d14/d19/d1c 0 2024-04-16T02:35:27.410 INFO:tasks.workunit.client.1.smithi153.stdout:4/77: chown d3/d9/db/de/l10 6884071 1 2024-04-16T02:35:27.413 INFO:tasks.workunit.client.1.smithi153.stdout:9/66: rename d4/f5 to d4/d6/dc/f10 0 2024-04-16T02:35:27.415 INFO:tasks.workunit.client.1.smithi153.stdout:0/88: mknod dd/d14/d19/c1d 0 2024-04-16T02:35:27.419 INFO:tasks.workunit.client.1.smithi153.stdout:4/78: dwrite d3/d9/db/de/f12 [0,4194304] 0 2024-04-16T02:35:27.422 INFO:tasks.workunit.client.1.smithi153.stdout:4/79: chown d3/d9/f14 269592 1 2024-04-16T02:35:27.423 INFO:tasks.workunit.client.1.smithi153.stdout:9/67: rmdir d4/d6 39 2024-04-16T02:35:27.426 INFO:tasks.workunit.client.1.smithi153.stdout:0/89: dwrite dd/f10 [0,4194304] 0 2024-04-16T02:35:27.430 INFO:tasks.workunit.client.1.smithi153.stdout:4/80: symlink d3/d9/l19 0 2024-04-16T02:35:27.432 INFO:tasks.workunit.client.1.smithi153.stdout:0/90: creat dd/d14/d19/f1e x:0 0 0 2024-04-16T02:35:27.436 INFO:tasks.workunit.client.1.smithi153.stdout:4/81: dread d3/d9/db/de/f12 [0,4194304] 0 2024-04-16T02:35:27.440 INFO:tasks.workunit.client.1.smithi153.stdout:4/82: creat d3/d9/f1a x:0 0 0 2024-04-16T02:35:27.440 INFO:tasks.workunit.client.1.smithi153.stdout:4/83: write d3/ff [2764432,18658] 0 2024-04-16T02:35:27.441 INFO:tasks.workunit.client.1.smithi153.stdout:4/84: dread - d3/d9/f1a zero size 2024-04-16T02:35:27.441 INFO:tasks.workunit.client.1.smithi153.stdout:4/85: chown d3/d9/c15 1834 1 2024-04-16T02:35:27.442 INFO:tasks.workunit.client.1.smithi153.stdout:4/86: write d3/d9/db/de/f13 [480101,58973] 0 2024-04-16T02:35:27.445 INFO:tasks.workunit.client.1.smithi153.stdout:4/87: mknod d3/d9/db/de/c1b 0 2024-04-16T02:35:27.448 INFO:tasks.workunit.client.1.smithi153.stdout:4/88: creat d3/f1c x:0 0 0 2024-04-16T02:35:27.449 INFO:tasks.workunit.client.1.smithi153.stdout:4/89: chown d3/d9/db/de/f12 105632291 1 2024-04-16T02:35:27.451 INFO:tasks.workunit.client.1.smithi153.stdout:4/90: symlink d3/l1d 0 2024-04-16T02:35:27.453 INFO:tasks.workunit.client.1.smithi153.stdout:4/91: symlink d3/d9/db/d17/l1e 0 2024-04-16T02:35:27.458 INFO:tasks.workunit.client.1.smithi153.stdout:4/92: link d3/l4 d3/d9/db/de/l1f 0 2024-04-16T02:35:27.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: pgmap v11: 65 pgs: 65 active+clean; 206 MiB data, 944 MiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 380 op/s 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: Adjusting osd_memory_target on smithi153 to 2240M 2024-04-16T02:35:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:27 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:27.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:27.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: pgmap v11: 65 pgs: 65 active+clean; 206 MiB data, 944 MiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 380 op/s 2024-04-16T02:35:27.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:27.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: Adjusting osd_memory_target on smithi153 to 2240M 2024-04-16T02:35:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:27 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:28.513 INFO:tasks.workunit.client.1.smithi153.stdout:8/27: sync 2024-04-16T02:35:28.513 INFO:tasks.workunit.client.1.smithi153.stdout:5/42: sync 2024-04-16T02:35:28.513 INFO:tasks.workunit.client.1.smithi153.stdout:3/71: sync 2024-04-16T02:35:28.513 INFO:tasks.workunit.client.1.smithi153.stdout:1/65: sync 2024-04-16T02:35:28.513 INFO:tasks.workunit.client.1.smithi153.stdout:2/69: sync 2024-04-16T02:35:28.513 INFO:tasks.workunit.client.1.smithi153.stdout:6/6: sync 2024-04-16T02:35:28.513 INFO:tasks.workunit.client.1.smithi153.stdout:7/67: sync 2024-04-16T02:35:28.513 INFO:tasks.workunit.client.1.smithi153.stdout:7/68: chown d0/f7 238155621 1 2024-04-16T02:35:28.514 INFO:tasks.workunit.client.1.smithi153.stdout:1/66: chown d4/f8 2537939 1 2024-04-16T02:35:28.514 INFO:tasks.workunit.client.1.smithi153.stdout:1/67: fdatasync d4/fe 0 2024-04-16T02:35:28.515 INFO:tasks.workunit.client.1.smithi153.stdout:6/7: creat f5 x:0 0 0 2024-04-16T02:35:28.515 INFO:tasks.workunit.client.1.smithi153.stdout:7/69: write d0/d4/db/f12 [590376,6044] 0 2024-04-16T02:35:28.515 INFO:tasks.workunit.client.1.smithi153.stdout:5/43: symlink d8/lb 0 2024-04-16T02:35:28.516 INFO:tasks.workunit.client.1.smithi153.stdout:6/8: dread - f4 zero size 2024-04-16T02:35:28.518 INFO:tasks.workunit.client.1.smithi153.stdout:5/44: chown d8/c9 1065 1 2024-04-16T02:35:28.519 INFO:tasks.workunit.client.1.smithi153.stdout:2/70: mkdir d9/dd/d1c/d20 0 2024-04-16T02:35:28.520 INFO:tasks.workunit.client.1.smithi153.stdout:7/70: read d0/d4/db/f12 [390018,127691] 0 2024-04-16T02:35:28.523 INFO:tasks.workunit.client.1.smithi153.stdout:8/28: dwrite d2/f6 [0,4194304] 0 2024-04-16T02:35:28.524 INFO:tasks.workunit.client.1.smithi153.stdout:1/68: dread d4/f7 [4194304,4194304] 0 2024-04-16T02:35:28.525 INFO:tasks.workunit.client.1.smithi153.stdout:1/69: write d4/f7 [5580029,47620] 0 2024-04-16T02:35:28.526 INFO:tasks.workunit.client.1.smithi153.stdout:5/45: dread f5 [0,4194304] 0 2024-04-16T02:35:28.527 INFO:tasks.workunit.client.1.smithi153.stdout:6/9: link f3 f6 0 2024-04-16T02:35:28.527 INFO:tasks.workunit.client.1.smithi153.stdout:6/10: read - f4 zero size 2024-04-16T02:35:28.528 INFO:tasks.workunit.client.1.smithi153.stdout:7/71: mkdir d0/d4/db/d16 0 2024-04-16T02:35:28.529 INFO:tasks.workunit.client.1.smithi153.stdout:6/11: chown f3 2327712 1 2024-04-16T02:35:28.531 INFO:tasks.workunit.client.1.smithi153.stdout:8/29: creat d2/f9 x:0 0 0 2024-04-16T02:35:28.532 INFO:tasks.workunit.client.1.smithi153.stdout:8/30: chown d2 578 1 2024-04-16T02:35:28.532 INFO:tasks.workunit.client.1.smithi153.stdout:2/71: dwrite f8 [0,4194304] 0 2024-04-16T02:35:28.533 INFO:tasks.workunit.client.1.smithi153.stdout:2/72: chown f8 137 1 2024-04-16T02:35:28.535 INFO:tasks.workunit.client.1.smithi153.stdout:5/46: unlink f5 0 2024-04-16T02:35:28.538 INFO:tasks.workunit.client.1.smithi153.stdout:6/12: mknod c7 0 2024-04-16T02:35:28.539 INFO:tasks.workunit.client.1.smithi153.stdout:7/72: mknod d0/d4/db/dd/c17 0 2024-04-16T02:35:28.541 INFO:tasks.workunit.client.1.smithi153.stdout:1/70: creat d4/dd/d13/f14 x:0 0 0 2024-04-16T02:35:28.543 INFO:tasks.workunit.client.1.smithi153.stdout:8/31: mknod d2/ca 0 2024-04-16T02:35:28.546 INFO:tasks.workunit.client.1.smithi153.stdout:8/32: write d2/f6 [1786550,45416] 0 2024-04-16T02:35:28.547 INFO:tasks.workunit.client.1.smithi153.stdout:6/13: rename l1 to l8 0 2024-04-16T02:35:28.548 INFO:tasks.workunit.client.1.smithi153.stdout:8/33: chown d2/l8 5944068 1 2024-04-16T02:35:28.548 INFO:tasks.workunit.client.1.smithi153.stdout:7/73: dwrite d0/f7 [0,4194304] 0 2024-04-16T02:35:28.554 INFO:tasks.workunit.client.1.smithi153.stdout:5/47: truncate f4 1819043 0 2024-04-16T02:35:28.554 INFO:tasks.workunit.client.1.smithi153.stdout:1/71: dwrite d4/dd/d13/f14 [0,4194304] 0 2024-04-16T02:35:28.554 INFO:tasks.workunit.client.1.smithi153.stdout:5/48: chown d8 98703969 1 2024-04-16T02:35:28.554 INFO:tasks.workunit.client.1.smithi153.stdout:5/49: chown c3 907 1 2024-04-16T02:35:28.555 INFO:tasks.workunit.client.1.smithi153.stdout:5/50: readlink l2 0 2024-04-16T02:35:28.564 INFO:tasks.workunit.client.1.smithi153.stdout:4/93: sync 2024-04-16T02:35:28.564 INFO:tasks.workunit.client.1.smithi153.stdout:9/68: sync 2024-04-16T02:35:28.564 INFO:tasks.workunit.client.1.smithi153.stdout:0/91: sync 2024-04-16T02:35:28.566 INFO:tasks.workunit.client.1.smithi153.stdout:8/34: creat d2/fb x:0 0 0 2024-04-16T02:35:28.566 INFO:tasks.workunit.client.1.smithi153.stdout:7/74: rename d0/l8 to d0/d4/db/l18 0 2024-04-16T02:35:28.570 INFO:tasks.workunit.client.1.smithi153.stdout:1/72: creat d4/dd/f15 x:0 0 0 2024-04-16T02:35:28.570 INFO:tasks.workunit.client.1.smithi153.stdout:5/51: creat d8/fc x:0 0 0 2024-04-16T02:35:28.570 INFO:tasks.workunit.client.1.smithi153.stdout:0/92: truncate dd/d14/d19/f1e 1039895 0 2024-04-16T02:35:28.570 INFO:tasks.workunit.client.1.smithi153.stdout:0/93: truncate dd/f16 1367837 0 2024-04-16T02:35:28.716 INFO:tasks.workunit.client.1.smithi153.stdout:3/72: sync 2024-04-16T02:35:28.717 INFO:tasks.workunit.client.1.smithi153.stdout:3/73: write d6/d10/f19 [810564,102495] 0 2024-04-16T02:35:28.745 INFO:tasks.workunit.client.1.smithi153.stdout:2/73: truncate d9/dd/d1b/f1d 1407062 0 2024-04-16T02:35:28.745 INFO:tasks.workunit.client.1.smithi153.stdout:2/74: truncate f6 410375 0 2024-04-16T02:35:28.750 INFO:tasks.workunit.client.1.smithi153.stdout:1/73: write d4/dd/d13/f14 [4452376,102144] 0 2024-04-16T02:35:28.764 INFO:tasks.workunit.client.1.smithi153.stdout:9/69: write d4/d6/fb [81103,117263] 0 2024-04-16T02:35:28.765 INFO:tasks.workunit.client.1.smithi153.stdout:9/70: stat d4/d6/dc/f10 0 2024-04-16T02:35:28.767 INFO:tasks.workunit.client.1.smithi153.stdout:8/35: creat d2/fc x:0 0 0 2024-04-16T02:35:28.770 INFO:tasks.workunit.client.1.smithi153.stdout:0/94: mkdir dd/d14/d19/d1f 0 2024-04-16T02:35:28.771 INFO:tasks.workunit.client.1.smithi153.stdout:0/95: stat dd/d14/d19 0 2024-04-16T02:35:28.772 INFO:tasks.workunit.client.1.smithi153.stdout:3/74: creat d6/f20 x:0 0 0 2024-04-16T02:35:28.774 INFO:tasks.workunit.client.1.smithi153.stdout:9/71: dwrite d4/d6/fb [4194304,4194304] 0 2024-04-16T02:35:28.774 INFO:tasks.workunit.client.1.smithi153.stdout:3/75: chown d6/d10/l1f 374 1 2024-04-16T02:35:28.775 INFO:tasks.workunit.client.1.smithi153.stdout:2/75: rmdir d9/dd/d1b 39 2024-04-16T02:35:28.775 INFO:tasks.workunit.client.1.smithi153.stdout:9/72: write d4/fd [18757,54302] 0 2024-04-16T02:35:28.776 INFO:tasks.workunit.client.1.smithi153.stdout:9/73: stat d4/fd 0 2024-04-16T02:35:28.776 INFO:tasks.workunit.client.1.smithi153.stdout:2/76: truncate d9/fb 580608 0 2024-04-16T02:35:28.780 INFO:tasks.workunit.client.1.smithi153.stdout:6/14: getdents . 0 2024-04-16T02:35:28.781 INFO:tasks.workunit.client.1.smithi153.stdout:4/94: mknod d3/d9/c20 0 2024-04-16T02:35:28.783 INFO:tasks.workunit.client.1.smithi153.stdout:8/36: symlink d2/ld 0 2024-04-16T02:35:28.784 INFO:tasks.workunit.client.1.smithi153.stdout:8/37: read d2/f7 [231,40486] 0 2024-04-16T02:35:28.785 INFO:tasks.workunit.client.1.smithi153.stdout:7/75: chown d0/d4/db/l18 75924 1 2024-04-16T02:35:28.788 INFO:tasks.workunit.client.1.smithi153.stdout:3/76: symlink d6/l21 0 2024-04-16T02:35:28.791 INFO:tasks.workunit.client.1.smithi153.stdout:2/77: symlink d9/dd/d1c/l21 0 2024-04-16T02:35:28.793 INFO:tasks.workunit.client.1.smithi153.stdout:9/74: rename d4/ff to d4/d6/dc/f11 0 2024-04-16T02:35:28.794 INFO:tasks.workunit.client.1.smithi153.stdout:9/75: read d4/d6/dc/f10 [3424024,50526] 0 2024-04-16T02:35:28.795 INFO:tasks.workunit.client.1.smithi153.stdout:6/15: creat f9 x:0 0 0 2024-04-16T02:35:28.795 INFO:tasks.workunit.client.1.smithi153.stdout:6/16: chown f3 105207 1 2024-04-16T02:35:28.796 INFO:tasks.workunit.client.1.smithi153.stdout:6/17: chown f6 16 1 2024-04-16T02:35:28.798 INFO:tasks.workunit.client.1.smithi153.stdout:8/38: mkdir d2/de 0 2024-04-16T02:35:28.800 INFO:tasks.workunit.client.1.smithi153.stdout:7/76: unlink d0/f7 0 2024-04-16T02:35:28.803 INFO:tasks.workunit.client.1.smithi153.stdout:2/78: mknod d9/c22 0 2024-04-16T02:35:28.804 INFO:tasks.workunit.client.1.smithi153.stdout:2/79: write f6 [661117,19646] 0 2024-04-16T02:35:28.806 INFO:tasks.workunit.client.1.smithi153.stdout:9/76: creat d4/d6/dc/f12 x:0 0 0 2024-04-16T02:35:28.807 INFO:tasks.workunit.client.1.smithi153.stdout:7/77: dwrite d0/fa [0,4194304] 0 2024-04-16T02:35:28.812 INFO:tasks.workunit.client.1.smithi153.stdout:8/39: creat d2/ff x:0 0 0 2024-04-16T02:35:28.813 INFO:tasks.workunit.client.1.smithi153.stdout:9/77: dread d4/d6/dc/f11 [0,4194304] 0 2024-04-16T02:35:28.818 INFO:tasks.workunit.client.1.smithi153.stdout:8/40: unlink d2/f6 0 2024-04-16T02:35:28.819 INFO:tasks.workunit.client.1.smithi153.stdout:9/78: rmdir d4/d6 39 2024-04-16T02:35:28.820 INFO:tasks.workunit.client.1.smithi153.stdout:8/41: write d2/fb [751597,94403] 0 2024-04-16T02:35:28.822 INFO:tasks.workunit.client.1.smithi153.stdout:8/42: truncate d2/f7 1044534 0 2024-04-16T02:35:28.823 INFO:tasks.workunit.client.1.smithi153.stdout:7/78: dwrite d0/d4/db/f12 [0,4194304] 0 2024-04-16T02:35:28.826 INFO:tasks.workunit.client.1.smithi153.stdout:9/79: fdatasync d4/d6/dc/f11 0 2024-04-16T02:35:28.827 INFO:tasks.workunit.client.1.smithi153.stdout:9/80: truncate d4/d6/dc/f10 4267612 0 2024-04-16T02:35:28.828 INFO:tasks.workunit.client.1.smithi153.stdout:8/43: rename d2/l8 to d2/l10 0 2024-04-16T02:35:28.828 INFO:tasks.workunit.client.1.smithi153.stdout:9/81: write d4/d6/dc/f11 [6344638,42508] 0 2024-04-16T02:35:28.828 INFO:tasks.workunit.client.1.smithi153.stdout:8/44: dread - d2/ff zero size 2024-04-16T02:35:28.829 INFO:tasks.workunit.client.1.smithi153.stdout:7/79: chown d0/d4/db/dd/l11 6868 1 2024-04-16T02:35:28.830 INFO:tasks.workunit.client.1.smithi153.stdout:8/45: fdatasync d2/fb 0 2024-04-16T02:35:28.832 INFO:tasks.workunit.client.1.smithi153.stdout:7/80: creat d0/d4/db/f19 x:0 0 0 2024-04-16T02:35:28.834 INFO:tasks.workunit.client.1.smithi153.stdout:8/46: creat d2/f11 x:0 0 0 2024-04-16T02:35:28.835 INFO:tasks.workunit.client.1.smithi153.stdout:7/81: dread d0/fa [0,4194304] 0 2024-04-16T02:35:28.837 INFO:tasks.workunit.client.1.smithi153.stdout:8/47: creat d2/f12 x:0 0 0 2024-04-16T02:35:28.843 INFO:tasks.workunit.client.1.smithi153.stdout:9/82: rename d4/d6/dc/f12 to d4/f13 0 2024-04-16T02:35:28.844 INFO:tasks.workunit.client.1.smithi153.stdout:9/83: chown d4/d6/dc/f10 14738865 1 2024-04-16T02:35:28.845 INFO:tasks.workunit.client.1.smithi153.stdout:8/48: mkdir d2/d13 0 2024-04-16T02:35:28.846 INFO:tasks.workunit.client.1.smithi153.stdout:9/84: symlink d4/d6/l14 0 2024-04-16T02:35:28.847 INFO:tasks.workunit.client.1.smithi153.stdout:9/85: dread - d4/d6/dc/fe zero size 2024-04-16T02:35:28.848 INFO:tasks.workunit.client.1.smithi153.stdout:9/86: readlink d4/d6/l14 0 2024-04-16T02:35:28.849 INFO:tasks.workunit.client.1.smithi153.stdout:8/49: rename d2/f11 to d2/f14 0 2024-04-16T02:35:28.850 INFO:tasks.workunit.client.1.smithi153.stdout:8/50: write d2/ff [30881,123085] 0 2024-04-16T02:35:28.853 INFO:tasks.workunit.client.1.smithi153.stdout:9/87: symlink d4/l15 0 2024-04-16T02:35:28.860 INFO:tasks.workunit.client.1.smithi153.stdout:8/51: dwrite d2/f9 [0,4194304] 0 2024-04-16T02:35:28.860 INFO:tasks.workunit.client.1.smithi153.stdout:8/52: read d2/f7 [171103,100118] 0 2024-04-16T02:35:28.861 INFO:tasks.workunit.client.1.smithi153.stdout:8/53: write d2/fc [756895,10726] 0 2024-04-16T02:35:28.869 INFO:tasks.workunit.client.1.smithi153.stdout:8/54: dwrite d2/fb [0,4194304] 0 2024-04-16T02:35:28.873 INFO:tasks.workunit.client.1.smithi153.stdout:8/55: creat d2/de/f15 x:0 0 0 2024-04-16T02:35:28.888 INFO:tasks.workunit.client.1.smithi153.stdout:3/77: sync 2024-04-16T02:35:28.891 INFO:tasks.workunit.client.1.smithi153.stdout:3/78: mkdir d6/d10/d22 0 2024-04-16T02:35:28.891 INFO:tasks.workunit.client.1.smithi153.stdout:3/79: readlink d6/d10/l1b 0 2024-04-16T02:35:28.892 INFO:tasks.workunit.client.1.smithi153.stdout:3/80: fsync d6/d10/f13 0 2024-04-16T02:35:28.894 INFO:tasks.workunit.client.1.smithi153.stdout:3/81: mknod d6/c23 0 2024-04-16T02:35:28.894 INFO:tasks.workunit.client.1.smithi153.stdout:3/82: stat c3 0 2024-04-16T02:35:28.910 INFO:tasks.workunit.client.0.smithi151.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-16T02:35:28.913 INFO:tasks.workunit.client.1.smithi153.stdout:5/52: write f4 [2142399,50526] 0 2024-04-16T02:35:28.913 INFO:tasks.workunit.client.1.smithi153.stdout:5/53: write f4 [2834026,118494] 0 2024-04-16T02:35:28.917 INFO:tasks.workunit.client.1.smithi153.stdout:5/54: dread f4 [0,4194304] 0 2024-04-16T02:35:28.918 INFO:tasks.workunit.client.1.smithi153.stdout:1/74: truncate d4/f7 1097188 0 2024-04-16T02:35:28.918 INFO:tasks.workunit.client.1.smithi153.stdout:5/55: dread - d8/fc zero size 2024-04-16T02:35:28.921 INFO:tasks.workunit.client.1.smithi153.stdout:5/56: dread f4 [0,4194304] 0 2024-04-16T02:35:28.921 INFO:tasks.workunit.client.1.smithi153.stdout:1/75: dread d4/dd/d13/f14 [0,4194304] 0 2024-04-16T02:35:28.922 INFO:tasks.workunit.client.1.smithi153.stdout:5/57: write d8/fc [712171,22348] 0 2024-04-16T02:35:28.923 INFO:tasks.workunit.client.1.smithi153.stdout:0/96: truncate dd/d14/d19/f1e 317172 0 2024-04-16T02:35:28.925 INFO:tasks.workunit.client.0.smithi151.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-16T02:35:28.925 INFO:tasks.workunit.client.0.smithi151.stderr:+ make 2024-04-16T02:35:28.926 INFO:tasks.workunit.client.1.smithi153.stdout:1/76: dread d4/f8 [0,4194304] 0 2024-04-16T02:35:28.926 INFO:tasks.workunit.client.1.smithi153.stdout:5/58: rename d8/fc to d8/fd 0 2024-04-16T02:35:28.926 INFO:tasks.workunit.client.1.smithi153.stdout:1/77: dread d4/f8 [0,4194304] 0 2024-04-16T02:35:28.927 INFO:tasks.workunit.client.1.smithi153.stdout:4/95: truncate d3/ff 788077 0 2024-04-16T02:35:28.928 INFO:tasks.workunit.client.1.smithi153.stdout:0/97: unlink cb 0 2024-04-16T02:35:28.930 INFO:tasks.workunit.client.1.smithi153.stdout:1/78: creat d4/dd/f16 x:0 0 0 2024-04-16T02:35:28.932 INFO:tasks.workunit.client.1.smithi153.stdout:4/96: creat d3/d9/db/d17/f21 x:0 0 0 2024-04-16T02:35:28.933 INFO:tasks.workunit.client.1.smithi153.stdout:6/18: dwrite f4 [0,4194304] 0 2024-04-16T02:35:28.934 INFO:tasks.workunit.client.1.smithi153.stdout:4/97: truncate d3/d9/f14 702242 0 2024-04-16T02:35:28.935 INFO:tasks.workunit.client.1.smithi153.stdout:4/98: chown d3/d9/db/de 19686 1 2024-04-16T02:35:28.936 INFO:tasks.workunit.client.1.smithi153.stdout:1/79: symlink d4/dd/d13/l17 0 2024-04-16T02:35:28.936 INFO:tasks.workunit.client.1.smithi153.stdout:1/80: read d4/dd/d13/f14 [3705280,86801] 0 2024-04-16T02:35:28.938 INFO:tasks.workunit.client.1.smithi153.stdout:6/19: rename l8 to la 0 2024-04-16T02:35:28.938 INFO:tasks.workunit.client.1.smithi153.stdout:6/20: dread - f5 zero size 2024-04-16T02:35:28.943 INFO:tasks.workunit.client.1.smithi153.stdout:1/81: fsync d4/dd/d13/f14 0 2024-04-16T02:35:28.943 INFO:tasks.workunit.client.1.smithi153.stdout:7/82: rmdir d0/d4/db 39 2024-04-16T02:35:28.944 INFO:tasks.workunit.client.1.smithi153.stdout:7/83: fdatasync d0/d4/fc 0 2024-04-16T02:35:28.945 INFO:tasks.workunit.client.1.smithi153.stdout:6/21: mknod cb 0 2024-04-16T02:35:28.945 INFO:tasks.workunit.client.1.smithi153.stdout:6/22: rmdir - no directory 2024-04-16T02:35:28.946 INFO:tasks.workunit.client.1.smithi153.stdout:6/23: read f6 [1760766,129672] 0 2024-04-16T02:35:28.947 INFO:tasks.workunit.client.1.smithi153.stdout:0/98: getdents dd/d14 0 2024-04-16T02:35:28.949 INFO:tasks.workunit.client.1.smithi153.stdout:1/82: mkdir d4/dd/d18 0 2024-04-16T02:35:28.950 INFO:tasks.workunit.client.1.smithi153.stdout:1/83: truncate d4/dd/f16 930180 0 2024-04-16T02:35:28.952 INFO:tasks.workunit.client.1.smithi153.stdout:7/84: write d0/d4/db/ff [83229,50273] 0 2024-04-16T02:35:28.954 INFO:tasks.workunit.client.1.smithi153.stdout:6/24: mkdir dc 0 2024-04-16T02:35:28.955 INFO:tasks.workunit.client.1.smithi153.stdout:6/25: chown f9 23441 1 2024-04-16T02:35:28.956 INFO:tasks.workunit.client.1.smithi153.stdout:1/84: dwrite d4/dd/f16 [0,4194304] 0 2024-04-16T02:35:28.957 INFO:tasks.workunit.client.1.smithi153.stdout:2/80: sync 2024-04-16T02:35:28.958 INFO:tasks.workunit.client.1.smithi153.stdout:1/85: write d4/dd/f15 [106082,73935] 0 2024-04-16T02:35:28.958 INFO:tasks.workunit.client.1.smithi153.stdout:4/99: getdents d3 0 2024-04-16T02:35:28.959 INFO:tasks.workunit.client.1.smithi153.stdout:7/85: chown d0/d4/l5 7217 1 2024-04-16T02:35:28.963 INFO:tasks.workunit.client.1.smithi153.stdout:1/86: write d4/dd/f15 [459971,102731] 0 2024-04-16T02:35:28.963 INFO:tasks.workunit.client.1.smithi153.stdout:7/86: truncate d0/d4/db/ff 962559 0 2024-04-16T02:35:28.963 INFO:tasks.workunit.client.1.smithi153.stdout:9/88: getdents d4 0 2024-04-16T02:35:28.963 INFO:tasks.workunit.client.1.smithi153.stdout:6/26: read f3 [3968435,77109] 0 2024-04-16T02:35:28.965 INFO:tasks.workunit.client.1.smithi153.stdout:9/89: readlink d4/d6/l14 0 2024-04-16T02:35:28.965 INFO:tasks.workunit.client.1.smithi153.stdout:6/27: readlink l0 0 2024-04-16T02:35:28.965 INFO:tasks.workunit.client.1.smithi153.stdout:6/28: chown f3 38 1 2024-04-16T02:35:28.966 INFO:tasks.workunit.client.1.smithi153.stdout:2/81: mknod d9/dd/c23 0 2024-04-16T02:35:28.966 INFO:tasks.workunit.client.1.smithi153.stdout:8/56: rmdir d2/de 39 2024-04-16T02:35:28.967 INFO:tasks.workunit.client.1.smithi153.stdout:2/82: truncate f6 1326798 0 2024-04-16T02:35:28.968 INFO:tasks.workunit.client.1.smithi153.stdout:1/87: symlink d4/dd/d13/l19 0 2024-04-16T02:35:28.969 INFO:tasks.workunit.client.1.smithi153.stdout:9/90: symlink d4/d6/dc/l16 0 2024-04-16T02:35:28.972 INFO:tasks.workunit.client.1.smithi153.stdout:4/100: dwrite d3/d9/db/de/f12 [0,4194304] 0 2024-04-16T02:35:28.976 INFO:tasks.workunit.client.1.smithi153.stdout:1/88: creat d4/dd/d13/f1a x:0 0 0 2024-04-16T02:35:28.976 INFO:tasks.workunit.client.1.smithi153.stdout:2/83: dwrite f8 [0,4194304] 0 2024-04-16T02:35:28.981 INFO:tasks.workunit.client.1.smithi153.stdout:8/57: creat d2/d13/f16 x:0 0 0 2024-04-16T02:35:28.981 INFO:tasks.workunit.client.1.smithi153.stdout:8/58: fsync d2/f12 0 2024-04-16T02:35:28.982 INFO:tasks.workunit.client.1.smithi153.stdout:9/91: rename d4/c7 to d4/d6/c17 0 2024-04-16T02:35:28.993 INFO:tasks.workunit.client.0.smithi151.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-16T02:35:28.996 INFO:tasks.workunit.client.1.smithi153.stdout:8/59: rename d2/fb to d2/f17 0 2024-04-16T02:35:28.997 INFO:tasks.workunit.client.1.smithi153.stdout:4/101: getdents d3/d9/db/d17 0 2024-04-16T02:35:28.998 INFO:tasks.workunit.client.1.smithi153.stdout:1/89: getdents d4/dd 0 2024-04-16T02:35:28.998 INFO:tasks.workunit.client.1.smithi153.stdout:1/90: chown d4/fe 64 1 2024-04-16T02:35:29.000 INFO:tasks.workunit.client.1.smithi153.stdout:8/60: write d2/f7 [1779043,21783] 0 2024-04-16T02:35:29.001 INFO:tasks.workunit.client.1.smithi153.stdout:8/61: write d2/f7 [2805715,102140] 0 2024-04-16T02:35:29.004 INFO:tasks.workunit.client.1.smithi153.stdout:4/102: mknod d3/d9/db/c22 0 2024-04-16T02:35:29.007 INFO:tasks.workunit.client.1.smithi153.stdout:6/29: link la dc/ld 0 2024-04-16T02:35:29.008 INFO:tasks.workunit.client.1.smithi153.stdout:4/103: unlink d3/c11 0 2024-04-16T02:35:29.015 INFO:tasks.workunit.client.1.smithi153.stdout:1/91: link c2 d4/dd/c1b 0 2024-04-16T02:35:29.017 INFO:tasks.workunit.client.1.smithi153.stdout:6/30: stat dc/ld 0 2024-04-16T02:35:29.018 INFO:tasks.workunit.client.1.smithi153.stdout:0/99: dwrite dd/d14/d19/f1e [0,4194304] 0 2024-04-16T02:35:29.018 INFO:tasks.workunit.client.1.smithi153.stdout:6/31: fdatasync f5 0 2024-04-16T02:35:29.018 INFO:tasks.workunit.client.1.smithi153.stdout:6/32: dread - f5 zero size 2024-04-16T02:35:29.021 INFO:tasks.workunit.client.1.smithi153.stdout:8/62: link d2/ca d2/c18 0 2024-04-16T02:35:29.024 INFO:tasks.workunit.client.1.smithi153.stdout:9/92: dread d4/d6/fb [0,4194304] 0 2024-04-16T02:35:29.031 INFO:tasks.workunit.client.1.smithi153.stdout:6/33: creat dc/fe x:0 0 0 2024-04-16T02:35:29.031 INFO:tasks.workunit.client.1.smithi153.stdout:6/34: chown f4 3288541 1 2024-04-16T02:35:29.032 INFO:tasks.workunit.client.1.smithi153.stdout:9/93: write d4/d6/dc/f10 [2419893,110441] 0 2024-04-16T02:35:29.033 INFO:tasks.workunit.client.1.smithi153.stdout:4/104: fsync d3/f1c 0 2024-04-16T02:35:29.035 INFO:tasks.workunit.client.1.smithi153.stdout:0/100: rmdir dd/d14/d19/d1f 0 2024-04-16T02:35:29.036 INFO:tasks.workunit.client.1.smithi153.stdout:0/101: chown dd/l15 92844 1 2024-04-16T02:35:29.039 INFO:tasks.workunit.client.1.smithi153.stdout:4/105: chown d3/d9/l19 40 1 2024-04-16T02:35:29.042 INFO:tasks.workunit.client.1.smithi153.stdout:9/94: dread d4/fd [0,4194304] 0 2024-04-16T02:35:29.042 INFO:tasks.workunit.client.1.smithi153.stdout:8/63: getdents d2 0 2024-04-16T02:35:29.044 INFO:tasks.workunit.client.1.smithi153.stdout:3/83: sync 2024-04-16T02:35:29.047 INFO:tasks.workunit.client.1.smithi153.stdout:3/84: dread - d6/f20 zero size 2024-04-16T02:35:29.048 INFO:tasks.workunit.client.1.smithi153.stdout:0/102: truncate dd/f10 1584965 0 2024-04-16T02:35:29.050 INFO:tasks.workunit.client.1.smithi153.stdout:4/106: creat d3/d9/f23 x:0 0 0 2024-04-16T02:35:29.051 INFO:tasks.workunit.client.1.smithi153.stdout:3/85: fsync d6/d10/f1c 0 2024-04-16T02:35:29.055 INFO:tasks.workunit.client.1.smithi153.stdout:9/95: dread d4/d6/dc/f11 [0,4194304] 0 2024-04-16T02:35:29.057 INFO:tasks.workunit.client.1.smithi153.stdout:1/92: dwrite d4/f7 [0,4194304] 0 2024-04-16T02:35:29.057 INFO:tasks.workunit.client.1.smithi153.stdout:8/64: dwrite d2/fc [0,4194304] 0 2024-04-16T02:35:29.059 INFO:tasks.workunit.client.1.smithi153.stdout:3/86: creat d6/f24 x:0 0 0 2024-04-16T02:35:29.060 INFO:tasks.workunit.client.1.smithi153.stdout:6/35: dwrite f4 [0,4194304] 0 2024-04-16T02:35:29.063 INFO:tasks.workunit.client.1.smithi153.stdout:4/107: mkdir d3/d9/d24 0 2024-04-16T02:35:29.065 INFO:tasks.workunit.client.1.smithi153.stdout:1/93: mknod d4/dd/d13/c1c 0 2024-04-16T02:35:29.066 INFO:tasks.workunit.client.1.smithi153.stdout:8/65: symlink d2/d13/l19 0 2024-04-16T02:35:29.068 INFO:tasks.workunit.client.1.smithi153.stdout:0/103: dread dd/d14/d19/f1e [0,4194304] 0 2024-04-16T02:35:29.069 INFO:tasks.workunit.client.1.smithi153.stdout:1/94: fsync d4/dd/d13/f1a 0 2024-04-16T02:35:29.070 INFO:tasks.workunit.client.1.smithi153.stdout:9/96: dread d4/d6/dc/f10 [0,4194304] 0 2024-04-16T02:35:29.071 INFO:tasks.workunit.client.1.smithi153.stdout:1/95: chown d4/dd 0 1 2024-04-16T02:35:29.071 INFO:tasks.workunit.client.1.smithi153.stdout:3/87: link c3 d6/c25 0 2024-04-16T02:35:29.072 INFO:tasks.workunit.client.1.smithi153.stdout:8/66: write d2/f14 [135484,123141] 0 2024-04-16T02:35:29.074 INFO:tasks.workunit.client.1.smithi153.stdout:4/108: mknod d3/d9/c25 0 2024-04-16T02:35:29.074 INFO:tasks.workunit.client.1.smithi153.stdout:3/88: truncate d6/f20 1035672 0 2024-04-16T02:35:29.076 INFO:tasks.workunit.client.1.smithi153.stdout:1/96: creat d4/dd/f1d x:0 0 0 2024-04-16T02:35:29.077 INFO:tasks.workunit.client.1.smithi153.stdout:8/67: symlink d2/d13/l1a 0 2024-04-16T02:35:29.077 INFO:tasks.workunit.client.1.smithi153.stdout:9/97: readlink d4/l15 0 2024-04-16T02:35:29.077 INFO:tasks.workunit.client.1.smithi153.stdout:6/36: dwrite dc/fe [0,4194304] 0 2024-04-16T02:35:29.079 INFO:tasks.workunit.client.1.smithi153.stdout:4/109: creat d3/d9/db/de/f26 x:0 0 0 2024-04-16T02:35:29.081 INFO:tasks.workunit.client.1.smithi153.stdout:8/68: fsync d2/f7 0 2024-04-16T02:35:29.083 INFO:tasks.workunit.client.1.smithi153.stdout:6/37: chown f6 253391 1 2024-04-16T02:35:29.085 INFO:tasks.workunit.client.1.smithi153.stdout:1/97: symlink d4/dd/d18/l1e 0 2024-04-16T02:35:29.085 INFO:tasks.workunit.client.1.smithi153.stdout:6/38: dread - f9 zero size 2024-04-16T02:35:29.086 INFO:tasks.workunit.client.1.smithi153.stdout:8/69: readlink d2/ld 0 2024-04-16T02:35:29.086 INFO:tasks.workunit.client.1.smithi153.stdout:9/98: write d4/d6/dc/f10 [1664063,124376] 0 2024-04-16T02:35:29.089 INFO:tasks.workunit.client.1.smithi153.stdout:4/110: getdents d3/d9/db 0 2024-04-16T02:35:29.090 INFO:tasks.workunit.client.1.smithi153.stdout:8/70: read d2/f14 [73819,55457] 0 2024-04-16T02:35:29.092 INFO:tasks.workunit.client.1.smithi153.stdout:1/98: rename d4/f8 to d4/f1f 0 2024-04-16T02:35:29.093 INFO:tasks.workunit.client.1.smithi153.stdout:0/104: dwrite dd/d14/d19/f1e [4194304,4194304] 0 2024-04-16T02:35:29.093 INFO:tasks.workunit.client.1.smithi153.stdout:5/59: sync 2024-04-16T02:35:29.093 INFO:tasks.workunit.client.1.smithi153.stdout:7/87: sync 2024-04-16T02:35:29.094 INFO:tasks.workunit.client.1.smithi153.stdout:3/89: dwrite d6/f11 [0,4194304] 0 2024-04-16T02:35:29.094 INFO:tasks.workunit.client.1.smithi153.stdout:9/99: write d4/d6/dc/f10 [4656064,124589] 0 2024-04-16T02:35:29.094 INFO:tasks.workunit.client.1.smithi153.stdout:4/111: fsync d3/d9/db/d17/f21 0 2024-04-16T02:35:29.102 INFO:tasks.workunit.client.1.smithi153.stdout:6/39: dwrite f3 [0,4194304] 0 2024-04-16T02:35:29.105 INFO:tasks.workunit.client.1.smithi153.stdout:4/112: write d3/d9/f23 [1025932,88351] 0 2024-04-16T02:35:29.113 INFO:tasks.workunit.client.1.smithi153.stdout:1/99: rmdir d4/dd 39 2024-04-16T02:35:29.115 INFO:tasks.workunit.client.1.smithi153.stdout:9/100: write d4/d6/fb [5203980,92701] 0 2024-04-16T02:35:29.116 INFO:tasks.workunit.client.1.smithi153.stdout:6/40: dread f6 [0,4194304] 0 2024-04-16T02:35:29.117 INFO:tasks.workunit.client.1.smithi153.stdout:5/60: dwrite f4 [0,4194304] 0 2024-04-16T02:35:29.118 INFO:tasks.workunit.client.1.smithi153.stdout:6/41: chown cb 529760 1 2024-04-16T02:35:29.118 INFO:tasks.workunit.client.1.smithi153.stdout:3/90: dwrite d6/f24 [0,4194304] 0 2024-04-16T02:35:29.122 INFO:tasks.workunit.client.1.smithi153.stdout:4/113: rename d3/d9/db/de/c1b to d3/c27 0 2024-04-16T02:35:29.123 INFO:tasks.workunit.client.1.smithi153.stdout:5/61: rename d8/lb to d8/le 0 2024-04-16T02:35:29.124 INFO:tasks.workunit.client.1.smithi153.stdout:0/105: write dd/f10 [1965773,121665] 0 2024-04-16T02:35:29.126 INFO:tasks.workunit.client.1.smithi153.stdout:8/71: truncate d2/f9 2053685 0 2024-04-16T02:35:29.126 INFO:tasks.workunit.client.1.smithi153.stdout:5/62: truncate f4 4706057 0 2024-04-16T02:35:29.128 INFO:tasks.workunit.client.1.smithi153.stdout:0/106: write dd/fe [3522122,53808] 0 2024-04-16T02:35:29.128 INFO:tasks.workunit.client.1.smithi153.stdout:9/101: dread d4/d6/fb [4194304,4194304] 0 2024-04-16T02:35:29.131 INFO:tasks.workunit.client.1.smithi153.stdout:4/114: write d3/d9/f14 [1728705,84529] 0 2024-04-16T02:35:29.132 INFO:tasks.workunit.client.1.smithi153.stdout:9/102: chown d4/l15 461556980 1 2024-04-16T02:35:29.133 INFO:tasks.workunit.client.1.smithi153.stdout:8/72: rename d2/fc to d2/d13/f1b 0 2024-04-16T02:35:29.134 INFO:tasks.workunit.client.1.smithi153.stdout:6/42: dwrite f9 [0,4194304] 0 2024-04-16T02:35:29.136 INFO:tasks.workunit.client.1.smithi153.stdout:9/103: write d4/d6/dc/f10 [3457336,129972] 0 2024-04-16T02:35:29.136 INFO:tasks.workunit.client.1.smithi153.stdout:3/91: getdents d6 0 2024-04-16T02:35:29.137 INFO:tasks.workunit.client.1.smithi153.stdout:3/92: write d6/d10/f19 [197784,81946] 0 2024-04-16T02:35:29.138 INFO:tasks.workunit.client.1.smithi153.stdout:3/93: chown d6/f1a 20903 1 2024-04-16T02:35:29.139 INFO:tasks.workunit.client.1.smithi153.stdout:0/107: mkdir dd/d14/d19/d20 0 2024-04-16T02:35:29.139 INFO:tasks.workunit.client.1.smithi153.stdout:0/108: fsync dd/f18 0 2024-04-16T02:35:29.143 INFO:tasks.workunit.client.1.smithi153.stdout:4/115: rmdir d3/d9 39 2024-04-16T02:35:29.143 INFO:tasks.workunit.client.1.smithi153.stdout:3/94: dread d6/f1a [0,4194304] 0 2024-04-16T02:35:29.144 INFO:tasks.workunit.client.1.smithi153.stdout:3/95: write d6/d10/f1e [903268,6051] 0 2024-04-16T02:35:29.153 INFO:tasks.workunit.client.1.smithi153.stdout:3/96: mknod d6/c26 0 2024-04-16T02:35:29.154 INFO:tasks.workunit.client.1.smithi153.stdout:6/43: dwrite f3 [0,4194304] 0 2024-04-16T02:35:29.154 INFO:tasks.workunit.client.1.smithi153.stdout:3/97: fsync d6/f12 0 2024-04-16T02:35:29.155 INFO:tasks.workunit.client.1.smithi153.stdout:9/104: rename d4/d6/c17 to d4/d6/c18 0 2024-04-16T02:35:29.205 INFO:tasks.workunit.client.1.smithi153.stdout:2/84: sync 2024-04-16T02:35:29.215 INFO:tasks.workunit.client.1.smithi153.stdout:4/116: mkdir d3/d9/db/d17/d28 0 2024-04-16T02:35:29.216 INFO:tasks.workunit.client.1.smithi153.stdout:3/98: creat d6/d10/f27 x:0 0 0 2024-04-16T02:35:29.218 INFO:tasks.workunit.client.1.smithi153.stdout:8/73: write d2/d13/f1b [1693777,12999] 0 2024-04-16T02:35:29.218 INFO:tasks.workunit.client.1.smithi153.stdout:0/109: write dd/d14/d19/f1a [1001566,95737] 0 2024-04-16T02:35:29.218 INFO:tasks.workunit.client.1.smithi153.stdout:9/105: rename d4/d6/dc/f10 to d4/d6/dc/f19 0 2024-04-16T02:35:29.221 INFO:tasks.workunit.client.1.smithi153.stdout:3/99: read d6/d10/f19 [306525,49710] 0 2024-04-16T02:35:29.222 INFO:tasks.workunit.client.1.smithi153.stdout:9/106: read d4/fd [442276,128340] 0 2024-04-16T02:35:29.223 INFO:tasks.workunit.client.1.smithi153.stdout:6/44: dwrite f4 [4194304,4194304] 0 2024-04-16T02:35:29.224 INFO:tasks.workunit.client.1.smithi153.stdout:8/74: mkdir d2/d1c 0 2024-04-16T02:35:29.225 INFO:tasks.workunit.client.1.smithi153.stdout:6/45: readlink l0 0 2024-04-16T02:35:29.273 INFO:tasks.workunit.client.1.smithi153.stdout:6/46: fdatasync f3 0 2024-04-16T02:35:29.278 INFO:tasks.workunit.client.1.smithi153.stdout:8/75: creat d2/d13/f1d x:0 0 0 2024-04-16T02:35:29.279 INFO:tasks.workunit.client.1.smithi153.stdout:9/107: link d4/d6/dc/f11 d4/d6/f1a 0 2024-04-16T02:35:29.279 INFO:tasks.workunit.client.1.smithi153.stdout:3/100: truncate d6/d10/f18 268154 0 2024-04-16T02:35:29.281 INFO:tasks.workunit.client.1.smithi153.stdout:4/117: dwrite d3/d9/db/de/f12 [0,4194304] 0 2024-04-16T02:35:29.281 INFO:tasks.workunit.client.1.smithi153.stdout:9/108: stat d4/d6/l14 0 2024-04-16T02:35:29.282 INFO:tasks.workunit.client.1.smithi153.stdout:8/76: symlink d2/de/l1e 0 2024-04-16T02:35:29.282 INFO:tasks.workunit.client.1.smithi153.stdout:3/101: chown d6/l15 3 1 2024-04-16T02:35:29.285 INFO:tasks.workunit.client.1.smithi153.stdout:2/85: dwrite d9/fc [0,4194304] 0 2024-04-16T02:35:29.287 INFO:tasks.workunit.client.1.smithi153.stdout:3/102: truncate d6/f12 4709336 0 2024-04-16T02:35:29.287 INFO:tasks.workunit.client.1.smithi153.stdout:6/47: dwrite f3 [0,4194304] 0 2024-04-16T02:35:29.287 INFO:tasks.workunit.client.1.smithi153.stdout:2/86: chown d9/dd/l11 813356 1 2024-04-16T02:35:29.287 INFO:tasks.workunit.client.1.smithi153.stdout:3/103: stat d6/cd 0 2024-04-16T02:35:29.290 INFO:tasks.workunit.client.1.smithi153.stdout:9/109: mkdir d4/d6/d1b 0 2024-04-16T02:35:29.290 INFO:tasks.workunit.client.1.smithi153.stdout:7/88: sync 2024-04-16T02:35:29.291 INFO:tasks.workunit.client.1.smithi153.stdout:9/110: dread - d4/d6/dc/fe zero size 2024-04-16T02:35:29.292 INFO:tasks.workunit.client.1.smithi153.stdout:4/118: creat d3/d9/f29 x:0 0 0 2024-04-16T02:35:29.292 INFO:tasks.workunit.client.1.smithi153.stdout:6/48: stat dc/ld 0 2024-04-16T02:35:29.293 INFO:tasks.workunit.client.1.smithi153.stdout:6/49: stat l0 0 2024-04-16T02:35:29.293 INFO:tasks.workunit.client.1.smithi153.stdout:8/77: mkdir d2/d1c/d1f 0 2024-04-16T02:35:29.294 INFO:tasks.workunit.client.1.smithi153.stdout:8/78: dread - d2/de/f15 zero size 2024-04-16T02:35:29.295 INFO:tasks.workunit.client.1.smithi153.stdout:7/89: read d0/fa [1304092,51346] 0 2024-04-16T02:35:29.295 INFO:tasks.workunit.client.1.smithi153.stdout:8/79: readlink d2/de/l1e 0 2024-04-16T02:35:29.295 INFO:tasks.workunit.client.1.smithi153.stdout:7/90: chown d0/d4 31906136 1 2024-04-16T02:35:29.296 INFO:tasks.workunit.client.1.smithi153.stdout:6/50: creat dc/ff x:0 0 0 2024-04-16T02:35:29.298 INFO:tasks.workunit.client.1.smithi153.stdout:3/104: creat d6/d10/d22/f28 x:0 0 0 2024-04-16T02:35:29.299 INFO:tasks.workunit.client.1.smithi153.stdout:4/119: mknod d3/c2a 0 2024-04-16T02:35:29.301 INFO:tasks.workunit.client.1.smithi153.stdout:8/80: unlink d2/d13/f16 0 2024-04-16T02:35:29.303 INFO:tasks.workunit.client.1.smithi153.stdout:6/51: mkdir dc/d10 0 2024-04-16T02:35:29.303 INFO:tasks.workunit.client.1.smithi153.stdout:9/111: getdents d4 0 2024-04-16T02:35:29.304 INFO:tasks.workunit.client.1.smithi153.stdout:3/105: symlink d6/d10/l29 0 2024-04-16T02:35:29.304 INFO:tasks.workunit.client.1.smithi153.stdout:9/112: chown d4/d6/dc/f19 16068 1 2024-04-16T02:35:29.304 INFO:tasks.workunit.client.1.smithi153.stdout:3/106: stat d6/f1a 0 2024-04-16T02:35:29.305 INFO:tasks.workunit.client.1.smithi153.stdout:4/120: chown d3/l16 13 1 2024-04-16T02:35:29.305 INFO:tasks.workunit.client.1.smithi153.stdout:7/91: mknod d0/d4/db/d16/c1a 0 2024-04-16T02:35:29.306 INFO:tasks.workunit.client.1.smithi153.stdout:2/87: getdents d9 0 2024-04-16T02:35:29.307 INFO:tasks.workunit.client.1.smithi153.stdout:6/52: creat dc/f11 x:0 0 0 2024-04-16T02:35:29.307 INFO:tasks.workunit.client.1.smithi153.stdout:9/113: rmdir d4 39 2024-04-16T02:35:29.308 INFO:tasks.workunit.client.1.smithi153.stdout:2/88: write d9/dd/f17 [1855487,50507] 0 2024-04-16T02:35:29.308 INFO:tasks.workunit.client.1.smithi153.stdout:3/107: creat d6/d10/d22/f2a x:0 0 0 2024-04-16T02:35:29.312 INFO:tasks.workunit.client.1.smithi153.stdout:3/108: chown d6/cc 137 1 2024-04-16T02:35:29.314 INFO:tasks.workunit.client.1.smithi153.stdout:7/92: dwrite d0/d4/db/f12 [0,4194304] 0 2024-04-16T02:35:29.318 INFO:tasks.workunit.client.1.smithi153.stdout:6/53: mkdir dc/d12 0 2024-04-16T02:35:29.330 INFO:tasks.workunit.client.1.smithi153.stdout:4/121: creat d3/d9/d24/f2b x:0 0 0 2024-04-16T02:35:29.332 INFO:tasks.workunit.client.1.smithi153.stdout:2/89: mkdir d9/d24 0 2024-04-16T02:35:29.332 INFO:tasks.workunit.client.1.smithi153.stdout:2/90: chown d9/dd 3 1 2024-04-16T02:35:29.335 INFO:tasks.workunit.client.1.smithi153.stdout:1/100: sync 2024-04-16T02:35:29.335 INFO:tasks.workunit.client.1.smithi153.stdout:5/63: sync 2024-04-16T02:35:29.336 INFO:tasks.workunit.client.1.smithi153.stdout:8/81: getdents d2/de 0 2024-04-16T02:35:29.337 INFO:tasks.workunit.client.1.smithi153.stdout:4/122: mkdir d3/d9/db/de/d2c 0 2024-04-16T02:35:29.341 INFO:tasks.workunit.client.1.smithi153.stdout:6/54: mknod dc/d10/c13 0 2024-04-16T02:35:29.341 INFO:tasks.workunit.client.1.smithi153.stdout:5/64: dread d8/fd [0,4194304] 0 2024-04-16T02:35:29.342 INFO:tasks.workunit.client.1.smithi153.stdout:6/55: write f3 [2610093,88506] 0 2024-04-16T02:35:29.347 INFO:tasks.workunit.client.1.smithi153.stdout:5/65: mkdir d8/df 0 2024-04-16T02:35:29.348 INFO:tasks.workunit.client.1.smithi153.stdout:4/123: dwrite d3/d9/f1a [0,4194304] 0 2024-04-16T02:35:29.348 INFO:tasks.workunit.client.1.smithi153.stdout:1/101: unlink d4/dd/d13/l19 0 2024-04-16T02:35:29.348 INFO:tasks.workunit.client.1.smithi153.stdout:5/66: read d8/fd [582816,5831] 0 2024-04-16T02:35:29.350 INFO:tasks.workunit.client.1.smithi153.stdout:8/82: write d2/f17 [5140688,127836] 0 2024-04-16T02:35:29.350 INFO:tasks.workunit.client.1.smithi153.stdout:1/102: write d4/dd/f15 [213991,28580] 0 2024-04-16T02:35:29.351 INFO:tasks.workunit.client.1.smithi153.stdout:3/109: getdents d6 0 2024-04-16T02:35:29.352 INFO:tasks.workunit.client.1.smithi153.stdout:4/124: write d3/d9/d24/f2b [537208,31760] 0 2024-04-16T02:35:29.353 INFO:tasks.workunit.client.1.smithi153.stdout:8/83: write d2/d13/f1b [4198089,22385] 0 2024-04-16T02:35:29.355 INFO:tasks.workunit.client.1.smithi153.stdout:7/93: getdents d0/d4/db 0 2024-04-16T02:35:29.355 INFO:tasks.workunit.client.1.smithi153.stdout:8/84: dread - d2/de/f15 zero size 2024-04-16T02:35:29.357 INFO:tasks.workunit.client.1.smithi153.stdout:7/94: rename d0/d4/db/d16 to d0/d4/db/d16/d1b 22 2024-04-16T02:35:29.358 INFO:tasks.workunit.client.1.smithi153.stdout:6/56: mknod dc/d12/c14 0 2024-04-16T02:35:29.360 INFO:tasks.workunit.client.1.smithi153.stdout:1/103: dread d4/dd/f15 [0,4194304] 0 2024-04-16T02:35:29.364 INFO:tasks.workunit.client.1.smithi153.stdout:8/85: mknod d2/d1c/c20 0 2024-04-16T02:35:29.365 INFO:tasks.workunit.client.1.smithi153.stdout:5/67: dwrite d8/fd [0,4194304] 0 2024-04-16T02:35:29.366 INFO:tasks.workunit.client.1.smithi153.stdout:5/68: readlink l2 0 2024-04-16T02:35:29.369 INFO:tasks.workunit.client.1.smithi153.stdout:1/104: mkdir d4/dd/d20 0 2024-04-16T02:35:29.373 INFO:tasks.workunit.client.1.smithi153.stdout:0/110: sync 2024-04-16T02:35:29.373 INFO:tasks.workunit.client.1.smithi153.stdout:7/95: creat d0/d13/f1c x:0 0 0 2024-04-16T02:35:29.374 INFO:tasks.workunit.client.1.smithi153.stdout:9/114: write d4/d6/f1a [4946243,4442] 0 2024-04-16T02:35:29.376 INFO:tasks.workunit.client.1.smithi153.stdout:1/105: symlink d4/dd/d13/l21 0 2024-04-16T02:35:29.377 INFO:tasks.workunit.client.1.smithi153.stdout:2/91: truncate d9/dd/f14 736387 0 2024-04-16T02:35:29.378 INFO:tasks.workunit.client.1.smithi153.stdout:1/106: fdatasync d4/dd/f16 0 2024-04-16T02:35:29.378 INFO:tasks.workunit.client.1.smithi153.stdout:8/86: symlink d2/d1c/d1f/l21 0 2024-04-16T02:35:29.380 INFO:tasks.workunit.client.1.smithi153.stdout:5/69: creat d8/df/f10 x:0 0 0 2024-04-16T02:35:29.380 INFO:tasks.workunit.client.1.smithi153.stdout:9/115: fdatasync d4/d6/dc/f19 0 2024-04-16T02:35:29.380 INFO:tasks.workunit.client.1.smithi153.stdout:1/107: write d4/dd/f16 [4884479,37553] 0 2024-04-16T02:35:29.398 INFO:tasks.workunit.client.1.smithi153.stdout:8/87: read d2/d13/f1b [3053161,122445] 0 2024-04-16T02:35:29.399 INFO:tasks.workunit.client.1.smithi153.stdout:0/111: dwrite dd/d14/d19/f1e [0,4194304] 0 2024-04-16T02:35:29.399 INFO:tasks.workunit.client.1.smithi153.stdout:9/116: stat d4/d6/d1b 0 2024-04-16T02:35:29.400 INFO:tasks.workunit.client.1.smithi153.stdout:4/125: truncate d3/d9/db/de/f12 2652644 0 2024-04-16T02:35:29.402 INFO:tasks.workunit.client.1.smithi153.stdout:6/57: truncate f4 4332066 0 2024-04-16T02:35:29.404 INFO:tasks.workunit.client.1.smithi153.stdout:6/58: write f5 [602004,10803] 0 2024-04-16T02:35:29.406 INFO:tasks.workunit.client.1.smithi153.stdout:2/92: mknod d9/d24/c25 0 2024-04-16T02:35:29.408 INFO:tasks.workunit.client.1.smithi153.stdout:0/112: unlink dd/d14/f17 0 2024-04-16T02:35:29.408 INFO:tasks.workunit.client.1.smithi153.stdout:4/126: mknod d3/d9/db/c2d 0 2024-04-16T02:35:29.409 INFO:tasks.workunit.client.1.smithi153.stdout:6/59: mkdir dc/d12/d15 0 2024-04-16T02:35:29.410 INFO:tasks.workunit.client.1.smithi153.stdout:3/110: dwrite d6/f1a [0,4194304] 0 2024-04-16T02:35:29.411 INFO:tasks.workunit.client.1.smithi153.stdout:5/70: link d8/df/f10 d8/df/f11 0 2024-04-16T02:35:29.412 INFO:tasks.workunit.client.1.smithi153.stdout:5/71: stat c3 0 2024-04-16T02:35:29.413 INFO:tasks.workunit.client.1.smithi153.stdout:1/108: creat d4/f22 x:0 0 0 2024-04-16T02:35:29.413 INFO:tasks.workunit.client.1.smithi153.stdout:9/117: dwrite d4/d6/f1a [0,4194304] 0 2024-04-16T02:35:29.413 INFO:tasks.workunit.client.1.smithi153.stdout:4/127: dread - d3/d9/db/d17/f21 zero size 2024-04-16T02:35:29.414 INFO:tasks.workunit.client.1.smithi153.stdout:6/60: symlink dc/l16 0 2024-04-16T02:35:29.416 INFO:tasks.workunit.client.1.smithi153.stdout:8/88: link d2/de/l1e d2/de/l22 0 2024-04-16T02:35:29.416 INFO:tasks.workunit.client.1.smithi153.stdout:2/93: mknod d9/dd/d1c/d20/c26 0 2024-04-16T02:35:29.419 INFO:tasks.workunit.client.1.smithi153.stdout:3/111: creat d6/f2b x:0 0 0 2024-04-16T02:35:29.419 INFO:tasks.workunit.client.1.smithi153.stdout:3/112: readlink d6/l1d 0 2024-04-16T02:35:29.421 INFO:tasks.workunit.client.1.smithi153.stdout:0/113: mknod dd/d14/d19/d1c/c21 0 2024-04-16T02:35:29.421 INFO:tasks.workunit.client.1.smithi153.stdout:9/118: write d4/d6/dc/f19 [4259994,35272] 0 2024-04-16T02:35:29.422 INFO:tasks.workunit.client.1.smithi153.stdout:0/114: read f9 [2823240,1982] 0 2024-04-16T02:35:29.424 INFO:tasks.workunit.client.1.smithi153.stdout:0/115: write dd/fe [1357264,86131] 0 2024-04-16T02:35:29.426 INFO:tasks.workunit.client.1.smithi153.stdout:9/119: dread d4/d6/f1a [4194304,4194304] 0 2024-04-16T02:35:29.427 INFO:tasks.workunit.client.1.smithi153.stdout:0/116: chown dd/d14/d19/d1c 72344 1 2024-04-16T02:35:29.428 INFO:tasks.workunit.client.1.smithi153.stdout:0/117: chown dd/f18 6580 1 2024-04-16T02:35:29.428 INFO:tasks.workunit.client.1.smithi153.stdout:2/94: dwrite d9/fc [0,4194304] 0 2024-04-16T02:35:29.429 INFO:tasks.workunit.client.1.smithi153.stdout:2/95: readlink d9/dd/l11 0 2024-04-16T02:35:29.429 INFO:tasks.workunit.client.1.smithi153.stdout:0/118: chown dd/d14/d19/f1e 769 1 2024-04-16T02:35:29.457 INFO:tasks.workunit.client.1.smithi153.stdout:1/109: mkdir d4/dd/d13/d23 0 2024-04-16T02:35:29.459 INFO:tasks.workunit.client.1.smithi153.stdout:4/128: mknod d3/d9/db/c2e 0 2024-04-16T02:35:29.461 INFO:tasks.workunit.client.1.smithi153.stdout:3/113: mkdir d6/d10/d22/d2c 0 2024-04-16T02:35:29.464 INFO:tasks.workunit.client.1.smithi153.stdout:9/120: symlink d4/d6/dc/l1c 0 2024-04-16T02:35:29.466 INFO:tasks.workunit.client.1.smithi153.stdout:2/96: truncate d9/dd/d1b/f1f 378379 0 2024-04-16T02:35:29.467 INFO:tasks.workunit.client.1.smithi153.stdout:2/97: stat d9/dd/d1b/l1e 0 2024-04-16T02:35:29.468 INFO:tasks.workunit.client.1.smithi153.stdout:5/72: link d8/la d8/df/l12 0 2024-04-16T02:35:29.469 INFO:tasks.workunit.client.1.smithi153.stdout:3/114: creat d6/f2d x:0 0 0 2024-04-16T02:35:29.470 INFO:tasks.workunit.client.1.smithi153.stdout:9/121: rmdir d4/d6 39 2024-04-16T02:35:29.471 INFO:tasks.workunit.client.1.smithi153.stdout:0/119: write f1 [1342608,48338] 0 2024-04-16T02:35:29.473 INFO:tasks.workunit.client.1.smithi153.stdout:5/73: dread d8/fd [0,4194304] 0 2024-04-16T02:35:29.473 INFO:tasks.workunit.client.1.smithi153.stdout:1/110: truncate d4/dd/d13/f14 378258 0 2024-04-16T02:35:29.474 INFO:tasks.workunit.client.1.smithi153.stdout:0/120: read dd/d14/d19/f1e [893135,8780] 0 2024-04-16T02:35:29.474 INFO:tasks.workunit.client.1.smithi153.stdout:6/61: truncate f6 4082316 0 2024-04-16T02:35:29.475 INFO:tasks.workunit.client.1.smithi153.stdout:0/121: write dd/ff [1648560,119073] 0 2024-04-16T02:35:29.476 INFO:tasks.workunit.client.1.smithi153.stdout:4/129: mknod d3/d9/c2f 0 2024-04-16T02:35:29.477 INFO:tasks.workunit.client.1.smithi153.stdout:0/122: write f1 [2817190,43795] 0 2024-04-16T02:35:29.478 INFO:tasks.workunit.client.1.smithi153.stdout:3/115: mkdir d6/d10/d22/d2e 0 2024-04-16T02:35:29.479 INFO:tasks.workunit.client.1.smithi153.stdout:7/96: sync 2024-04-16T02:35:29.480 INFO:tasks.workunit.client.1.smithi153.stdout:4/130: write d3/d9/db/d17/f21 [893315,112388] 0 2024-04-16T02:35:29.482 INFO:tasks.workunit.client.1.smithi153.stdout:3/116: chown d6/d10/f1e 49007007 1 2024-04-16T02:35:29.483 INFO:tasks.workunit.client.1.smithi153.stdout:5/74: dwrite f4 [0,4194304] 0 2024-04-16T02:35:29.484 INFO:tasks.workunit.client.1.smithi153.stdout:4/131: fdatasync d3/d9/db/de/f13 0 2024-04-16T02:35:29.485 INFO:tasks.workunit.client.1.smithi153.stdout:3/117: chown d6/d10/f13 2 1 2024-04-16T02:35:29.487 INFO:tasks.workunit.client.1.smithi153.stdout:0/123: dread dd/d14/d19/f1a [0,4194304] 0 2024-04-16T02:35:29.487 INFO:tasks.workunit.client.1.smithi153.stdout:3/118: rename d6/d10 to d6/d10/d2f 22 2024-04-16T02:35:29.489 INFO:tasks.workunit.client.1.smithi153.stdout:0/124: write dd/f16 [1006859,51981] 0 2024-04-16T02:35:29.489 INFO:tasks.workunit.client.1.smithi153.stdout:4/132: write d3/d9/db/d17/f21 [146570,69777] 0 2024-04-16T02:35:29.491 INFO:tasks.workunit.client.1.smithi153.stdout:0/125: chown dd/d14/d19/f1a 4 1 2024-04-16T02:35:29.494 INFO:tasks.workunit.client.1.smithi153.stdout:5/75: fdatasync d8/fd 0 2024-04-16T02:35:29.497 INFO:tasks.workunit.client.1.smithi153.stdout:9/122: symlink d4/l1d 0 2024-04-16T02:35:29.499 INFO:tasks.workunit.client.1.smithi153.stdout:3/119: rename d6/cc to d6/d10/c30 0 2024-04-16T02:35:29.500 INFO:tasks.workunit.client.1.smithi153.stdout:1/111: rename d4/cc to d4/c24 0 2024-04-16T02:35:29.500 INFO:tasks.workunit.client.1.smithi153.stdout:1/112: rename d4 to d4/dd/d25 22 2024-04-16T02:35:29.501 INFO:tasks.workunit.client.1.smithi153.stdout:1/113: read d4/f1f [68731,64876] 0 2024-04-16T02:35:29.502 INFO:tasks.workunit.client.1.smithi153.stdout:7/97: link d0/d4/db/f12 d0/d4/db/dd/f1d 0 2024-04-16T02:35:29.503 INFO:tasks.workunit.client.1.smithi153.stdout:5/76: stat d8/la 0 2024-04-16T02:35:29.505 INFO:tasks.workunit.client.1.smithi153.stdout:3/120: creat d6/d10/d22/f31 x:0 0 0 2024-04-16T02:35:29.506 INFO:tasks.workunit.client.1.smithi153.stdout:3/121: write d6/f2b [331232,68465] 0 2024-04-16T02:35:29.506 INFO:tasks.workunit.client.1.smithi153.stdout:0/126: link dd/f18 dd/f22 0 2024-04-16T02:35:29.507 INFO:tasks.workunit.client.1.smithi153.stdout:1/114: fdatasync d4/dd/f15 0 2024-04-16T02:35:29.508 INFO:tasks.workunit.client.1.smithi153.stdout:6/62: getdents dc 0 2024-04-16T02:35:29.509 INFO:tasks.workunit.client.1.smithi153.stdout:5/77: mknod d8/c13 0 2024-04-16T02:35:29.512 INFO:tasks.workunit.client.1.smithi153.stdout:3/122: rename d6/f24 to d6/d10/f32 0 2024-04-16T02:35:29.513 INFO:tasks.workunit.client.1.smithi153.stdout:0/127: dwrite f1 [0,4194304] 0 2024-04-16T02:35:29.514 INFO:tasks.workunit.client.1.smithi153.stdout:1/115: mkdir d4/dd/d13/d26 0 2024-04-16T02:35:29.515 INFO:tasks.workunit.client.1.smithi153.stdout:3/123: truncate d6/d10/f27 963667 0 2024-04-16T02:35:29.516 INFO:tasks.workunit.client.1.smithi153.stdout:5/78: dread f4 [0,4194304] 0 2024-04-16T02:35:29.517 INFO:tasks.workunit.client.1.smithi153.stdout:9/123: rename d4/d6 to d4/d6/d1b/d1e 22 2024-04-16T02:35:29.518 INFO:tasks.workunit.client.1.smithi153.stdout:3/124: write d6/d10/f27 [1236782,103565] 0 2024-04-16T02:35:29.519 INFO:tasks.workunit.client.1.smithi153.stdout:5/79: truncate d8/df/f10 1545 0 2024-04-16T02:35:29.520 INFO:tasks.workunit.client.1.smithi153.stdout:9/124: write d4/d6/dc/fe [448832,69494] 0 2024-04-16T02:35:29.523 INFO:tasks.workunit.client.1.smithi153.stdout:4/133: dwrite d3/ff [0,4194304] 0 2024-04-16T02:35:29.525 INFO:tasks.workunit.client.1.smithi153.stdout:6/63: fsync f6 0 2024-04-16T02:35:29.526 INFO:tasks.workunit.client.1.smithi153.stdout:5/80: mknod d8/c14 0 2024-04-16T02:35:29.528 INFO:tasks.workunit.client.1.smithi153.stdout:0/128: link f1 dd/d14/d19/d1c/f23 0 2024-04-16T02:35:29.530 INFO:tasks.workunit.client.1.smithi153.stdout:1/116: creat d4/dd/d13/d23/f27 x:0 0 0 2024-04-16T02:35:29.534 INFO:tasks.workunit.client.1.smithi153.stdout:6/64: creat dc/f17 x:0 0 0 2024-04-16T02:35:29.535 INFO:tasks.workunit.client.1.smithi153.stdout:5/81: mknod d8/c15 0 2024-04-16T02:35:29.535 INFO:tasks.workunit.client.1.smithi153.stdout:9/125: write d4/fd [1582034,57692] 0 2024-04-16T02:35:29.537 INFO:tasks.workunit.client.1.smithi153.stdout:0/129: symlink dd/d14/d19/l24 0 2024-04-16T02:35:29.538 INFO:tasks.workunit.client.1.smithi153.stdout:4/134: mknod d3/c30 0 2024-04-16T02:35:29.539 INFO:tasks.workunit.client.1.smithi153.stdout:6/65: fsync f3 0 2024-04-16T02:35:29.543 INFO:tasks.workunit.client.1.smithi153.stdout:6/66: dread - dc/ff zero size 2024-04-16T02:35:29.545 INFO:tasks.workunit.client.1.smithi153.stdout:0/130: dread dd/f16 [0,4194304] 0 2024-04-16T02:35:29.546 INFO:tasks.workunit.client.1.smithi153.stdout:5/82: write d8/df/f11 [359093,9264] 0 2024-04-16T02:35:29.548 INFO:tasks.workunit.client.1.smithi153.stdout:4/135: truncate d3/d9/db/de/f12 200408 0 2024-04-16T02:35:29.550 INFO:tasks.workunit.client.1.smithi153.stdout:1/117: dwrite d4/f1f [0,4194304] 0 2024-04-16T02:35:29.551 INFO:tasks.workunit.client.1.smithi153.stdout:4/136: stat d3/l16 0 2024-04-16T02:35:29.551 INFO:tasks.workunit.client.1.smithi153.stdout:9/126: link d4/d6/c18 d4/d6/d1b/c1f 0 2024-04-16T02:35:29.552 INFO:tasks.workunit.client.1.smithi153.stdout:9/127: readlink d4/d6/l14 0 2024-04-16T02:35:29.554 INFO:tasks.workunit.client.1.smithi153.stdout:0/131: rename dd/d14/d19/c1d to dd/d14/c25 0 2024-04-16T02:35:29.556 INFO:tasks.workunit.client.1.smithi153.stdout:9/128: rmdir d4/d6/d1b 39 2024-04-16T02:35:29.561 INFO:tasks.workunit.client.1.smithi153.stdout:6/67: dwrite dc/fe [0,4194304] 0 2024-04-16T02:35:29.563 INFO:tasks.workunit.client.1.smithi153.stdout:4/137: link d3/c27 d3/d9/db/d17/d28/c31 0 2024-04-16T02:35:29.585 INFO:tasks.workunit.client.1.smithi153.stdout:9/129: dread d4/fd [0,4194304] 0 2024-04-16T02:35:29.587 INFO:tasks.workunit.client.1.smithi153.stdout:5/83: dwrite f4 [4194304,4194304] 0 2024-04-16T02:35:29.590 INFO:tasks.workunit.client.1.smithi153.stdout:5/84: chown l6 0 1 2024-04-16T02:35:29.590 INFO:tasks.workunit.client.1.smithi153.stdout:1/118: dwrite d4/dd/f16 [4194304,4194304] 0 2024-04-16T02:35:29.593 INFO:tasks.workunit.client.1.smithi153.stdout:0/132: dwrite dd/d14/d19/f1b [0,4194304] 0 2024-04-16T02:35:29.593 INFO:tasks.workunit.client.1.smithi153.stdout:1/119: readlink d4/dd/d13/l21 0 2024-04-16T02:35:29.597 INFO:tasks.workunit.client.1.smithi153.stdout:1/120: dread - d4/f22 zero size 2024-04-16T02:35:29.599 INFO:tasks.workunit.client.1.smithi153.stdout:8/89: sync 2024-04-16T02:35:29.599 INFO:tasks.workunit.client.1.smithi153.stdout:6/68: dwrite dc/f11 [0,4194304] 0 2024-04-16T02:35:29.600 INFO:tasks.workunit.client.1.smithi153.stdout:1/121: dread - d4/dd/d13/f1a zero size 2024-04-16T02:35:29.606 INFO:tasks.workunit.client.1.smithi153.stdout:0/133: dwrite dd/fe [0,4194304] 0 2024-04-16T02:35:29.608 INFO:tasks.workunit.client.1.smithi153.stdout:0/134: write dd/fe [3603597,62483] 0 2024-04-16T02:35:29.608 INFO:tasks.workunit.client.1.smithi153.stdout:0/135: chown f9 9 1 2024-04-16T02:35:29.608 INFO:tasks.workunit.client.1.smithi153.stdout:4/138: dread d3/d9/d24/f2b [0,4194304] 0 2024-04-16T02:35:29.658 INFO:tasks.workunit.client.1.smithi153.stdout:2/98: sync 2024-04-16T02:35:29.702 INFO:tasks.workunit.client.1.smithi153.stdout:3/125: sync 2024-04-16T02:35:29.702 INFO:tasks.workunit.client.1.smithi153.stdout:7/98: sync 2024-04-16T02:35:29.703 INFO:tasks.workunit.client.1.smithi153.stdout:7/99: fdatasync d0/d4/db/ff 0 2024-04-16T02:35:29.729 INFO:tasks.workunit.client.1.smithi153.stdout:9/130: fdatasync d4/d6/dc/f11 0 2024-04-16T02:35:29.731 INFO:tasks.workunit.client.1.smithi153.stdout:5/85: unlink c3 0 2024-04-16T02:35:29.731 INFO:tasks.workunit.client.1.smithi153.stdout:5/86: readlink l6 0 2024-04-16T02:35:29.735 INFO:tasks.workunit.client.1.smithi153.stdout:6/69: symlink dc/l18 0 2024-04-16T02:35:29.736 INFO:tasks.workunit.client.1.smithi153.stdout:8/90: write d2/d13/f1b [874925,44052] 0 2024-04-16T02:35:29.736 INFO:tasks.workunit.client.1.smithi153.stdout:6/70: fsync dc/fe 0 2024-04-16T02:35:29.737 INFO:tasks.workunit.client.1.smithi153.stdout:0/136: rmdir dd/d14/d19/d1c 39 2024-04-16T02:35:29.737 INFO:tasks.workunit.client.1.smithi153.stdout:0/137: fsync dd/ff 0 2024-04-16T02:35:29.742 INFO:tasks.workunit.client.1.smithi153.stdout:7/100: creat d0/f1e x:0 0 0 2024-04-16T02:35:29.743 INFO:tasks.workunit.client.1.smithi153.stdout:9/131: dread - d4/f13 zero size 2024-04-16T02:35:29.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:29 smithi151 ceph-mon[29003]: pgmap v12: 65 pgs: 65 active+clean; 234 MiB data, 988 MiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 407 op/s 2024-04-16T02:35:29.746 INFO:tasks.workunit.client.1.smithi153.stdout:5/87: unlink d8/fd 0 2024-04-16T02:35:29.746 INFO:tasks.workunit.client.1.smithi153.stdout:1/122: mknod d4/dd/d13/d26/c28 0 2024-04-16T02:35:29.746 INFO:tasks.workunit.client.1.smithi153.stdout:5/88: write d8/df/f10 [868826,52195] 0 2024-04-16T02:35:29.748 INFO:tasks.workunit.client.1.smithi153.stdout:5/89: write f4 [2826155,129978] 0 2024-04-16T02:35:29.750 INFO:tasks.workunit.client.1.smithi153.stdout:4/139: mknod d3/c32 0 2024-04-16T02:35:29.751 INFO:tasks.workunit.client.1.smithi153.stdout:4/140: chown d3/d9/db/d17/d28 3511 1 2024-04-16T02:35:29.751 INFO:tasks.workunit.client.1.smithi153.stdout:0/138: mkdir dd/d26 0 2024-04-16T02:35:29.753 INFO:tasks.workunit.client.1.smithi153.stdout:3/126: creat d6/d10/d22/d2c/f33 x:0 0 0 2024-04-16T02:35:29.753 INFO:tasks.workunit.client.1.smithi153.stdout:3/127: truncate d6/d10/f13 665976 0 2024-04-16T02:35:29.753 INFO:tasks.workunit.client.1.smithi153.stdout:7/101: symlink d0/l1f 0 2024-04-16T02:35:29.755 INFO:tasks.workunit.client.1.smithi153.stdout:9/132: write d4/d6/fb [2317552,87039] 0 2024-04-16T02:35:29.756 INFO:tasks.workunit.client.1.smithi153.stdout:9/133: fsync d4/d6/dc/f19 0 2024-04-16T02:35:29.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:29 smithi153 ceph-mon[36747]: pgmap v12: 65 pgs: 65 active+clean; 234 MiB data, 988 MiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 407 op/s 2024-04-16T02:35:29.782 INFO:tasks.workunit.client.1.smithi153.stdout:1/123: mkdir d4/dd/d18/d29 0 2024-04-16T02:35:29.782 INFO:tasks.workunit.client.1.smithi153.stdout:6/71: getdents dc/d12/d15 0 2024-04-16T02:35:29.782 INFO:tasks.workunit.client.1.smithi153.stdout:4/141: symlink d3/d9/db/d17/l33 0 2024-04-16T02:35:29.783 INFO:tasks.workunit.client.1.smithi153.stdout:6/72: stat f4 0 2024-04-16T02:35:29.783 INFO:tasks.workunit.client.1.smithi153.stdout:3/128: symlink d6/l34 0 2024-04-16T02:35:29.785 INFO:tasks.workunit.client.1.smithi153.stdout:7/102: dread d0/d4/db/dd/f1d [0,4194304] 0 2024-04-16T02:35:29.785 INFO:tasks.workunit.client.1.smithi153.stdout:4/142: write d3/d9/f1a [1646484,92831] 0 2024-04-16T02:35:29.787 INFO:tasks.workunit.client.1.smithi153.stdout:9/134: dread d4/d6/f1a [4194304,4194304] 0 2024-04-16T02:35:29.789 INFO:tasks.workunit.client.1.smithi153.stdout:7/103: write d0/d4/db/f19 [475360,110020] 0 2024-04-16T02:35:29.792 INFO:tasks.workunit.client.1.smithi153.stdout:1/124: dread d4/dd/f16 [0,4194304] 0 2024-04-16T02:35:29.795 INFO:tasks.workunit.client.1.smithi153.stdout:3/129: dwrite d6/d10/f27 [0,4194304] 0 2024-04-16T02:35:29.796 INFO:tasks.workunit.client.1.smithi153.stdout:3/130: readlink d6/la 0 2024-04-16T02:35:29.797 INFO:tasks.workunit.client.1.smithi153.stdout:8/91: link d2/d13/f1b d2/f23 0 2024-04-16T02:35:29.797 INFO:tasks.workunit.client.1.smithi153.stdout:2/99: getdents d9/dd/d1b 0 2024-04-16T02:35:29.798 INFO:tasks.workunit.client.1.smithi153.stdout:6/73: unlink dc/d12/c14 0 2024-04-16T02:35:29.800 INFO:tasks.workunit.client.1.smithi153.stdout:6/74: dread - dc/f17 zero size 2024-04-16T02:35:29.802 INFO:tasks.workunit.client.1.smithi153.stdout:0/139: creat dd/d14/d19/f27 x:0 0 0 2024-04-16T02:35:29.802 INFO:tasks.workunit.client.1.smithi153.stdout:7/104: dwrite d0/d4/fc [0,4194304] 0 2024-04-16T02:35:29.804 INFO:tasks.workunit.client.1.smithi153.stdout:7/105: truncate d0/f1e 80866 0 2024-04-16T02:35:29.806 INFO:tasks.workunit.client.0.smithi151.stderr:++ readlink -f fsstress 2024-04-16T02:35:29.808 INFO:tasks.workunit.client.1.smithi153.stdout:7/106: dread d0/d4/fc [0,4194304] 0 2024-04-16T02:35:29.808 INFO:tasks.workunit.client.1.smithi153.stdout:1/125: getdents d4/dd/d20 0 2024-04-16T02:35:29.809 INFO:tasks.workunit.client.0.smithi151.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-16T02:35:29.809 INFO:tasks.workunit.client.0.smithi151.stderr:+ popd 2024-04-16T02:35:29.810 INFO:tasks.workunit.client.1.smithi153.stdout:0/140: fsync f1 0 2024-04-16T02:35:29.812 INFO:tasks.workunit.client.0.smithi151.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-16T02:35:29.813 INFO:tasks.workunit.client.0.smithi151.stderr:+ popd 2024-04-16T02:35:29.816 INFO:tasks.workunit.client.0.smithi151.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-16T02:35:29.817 INFO:tasks.workunit.client.0.smithi151.stderr:++ mktemp -d -p . 2024-04-16T02:35:29.819 INFO:tasks.workunit.client.1.smithi153.stdout:9/135: rename d4/d6/dc/fe to d4/d6/f20 0 2024-04-16T02:35:29.821 INFO:tasks.workunit.client.1.smithi153.stdout:4/143: link d3/d9/db/c22 d3/d9/db/d17/d28/c34 0 2024-04-16T02:35:29.821 INFO:tasks.workunit.client.1.smithi153.stdout:0/141: dwrite dd/ff [0,4194304] 0 2024-04-16T02:35:29.824 INFO:tasks.workunit.client.1.smithi153.stdout:0/142: chown dd/d14/d19 34541458 1 2024-04-16T02:35:29.826 INFO:tasks.workunit.client.1.smithi153.stdout:6/75: creat dc/d12/d15/f19 x:0 0 0 2024-04-16T02:35:29.833 INFO:tasks.workunit.client.1.smithi153.stdout:2/100: rename d9/dd/d1b to d9/d24/d27 0 2024-04-16T02:35:29.833 INFO:tasks.workunit.client.1.smithi153.stdout:4/144: dwrite d3/ff [4194304,4194304] 0 2024-04-16T02:35:29.835 INFO:tasks.workunit.client.1.smithi153.stdout:9/136: creat d4/d6/dc/f21 x:0 0 0 2024-04-16T02:35:29.837 INFO:tasks.workunit.client.1.smithi153.stdout:8/92: getdents d2/d1c 0 2024-04-16T02:35:29.837 INFO:tasks.workunit.client.1.smithi153.stdout:6/76: dwrite dc/f11 [0,4194304] 0 2024-04-16T02:35:29.841 INFO:tasks.workunit.client.1.smithi153.stdout:0/143: truncate dd/d14/d19/f1a 125790 0 2024-04-16T02:35:29.847 INFO:tasks.workunit.client.1.smithi153.stdout:6/77: dread f4 [0,4194304] 0 2024-04-16T02:35:29.847 INFO:tasks.workunit.client.1.smithi153.stdout:7/107: creat d0/d4/f20 x:0 0 0 2024-04-16T02:35:29.848 INFO:tasks.workunit.client.1.smithi153.stdout:7/108: chown d0/d13 0 1 2024-04-16T02:35:29.850 INFO:tasks.workunit.client.1.smithi153.stdout:0/144: mkdir dd/d28 0 2024-04-16T02:35:29.853 INFO:tasks.workunit.client.1.smithi153.stdout:6/78: mkdir dc/d1a 0 2024-04-16T02:35:29.855 INFO:tasks.workunit.client.1.smithi153.stdout:7/109: mknod d0/d13/c21 0 2024-04-16T02:35:29.857 INFO:tasks.workunit.client.1.smithi153.stdout:9/137: creat d4/d6/f22 x:0 0 0 2024-04-16T02:35:29.859 INFO:tasks.workunit.client.1.smithi153.stdout:2/101: getdents d9/dd/d1c 0 2024-04-16T02:35:29.859 INFO:tasks.workunit.client.1.smithi153.stdout:0/145: dwrite f9 [0,4194304] 0 2024-04-16T02:35:29.861 INFO:tasks.workunit.client.1.smithi153.stdout:9/138: fdatasync d4/d6/dc/f11 0 2024-04-16T02:35:29.862 INFO:tasks.workunit.client.1.smithi153.stdout:2/102: stat d9/dd/d1c 0 2024-04-16T02:35:29.863 INFO:tasks.workunit.client.1.smithi153.stdout:7/110: truncate d0/d4/db/ff 1365213 0 2024-04-16T02:35:29.864 INFO:tasks.workunit.client.1.smithi153.stdout:0/146: mknod dd/d14/c29 0 2024-04-16T02:35:29.864 INFO:tasks.workunit.client.1.smithi153.stdout:8/93: dwrite d2/f14 [0,4194304] 0 2024-04-16T02:35:29.865 INFO:tasks.workunit.client.1.smithi153.stdout:6/79: dread dc/f11 [0,4194304] 0 2024-04-16T02:35:29.866 INFO:tasks.workunit.client.1.smithi153.stdout:2/103: creat d9/dd/d1c/f28 x:0 0 0 2024-04-16T02:35:29.868 INFO:tasks.workunit.client.1.smithi153.stdout:2/104: write d9/fc [2118336,56462] 0 2024-04-16T02:35:29.870 INFO:tasks.workunit.client.1.smithi153.stdout:7/111: write d0/d4/fc [3144861,92979] 0 2024-04-16T02:35:29.871 INFO:tasks.workunit.client.1.smithi153.stdout:2/105: chown d9/dd/ff 0 1 2024-04-16T02:35:29.872 INFO:tasks.workunit.client.1.smithi153.stdout:2/106: stat f8 0 2024-04-16T02:35:29.873 INFO:tasks.workunit.client.1.smithi153.stdout:2/107: stat d9/dd/c15 0 2024-04-16T02:35:29.873 INFO:tasks.workunit.client.1.smithi153.stdout:6/80: dread dc/f11 [0,4194304] 0 2024-04-16T02:35:29.873 INFO:tasks.workunit.client.1.smithi153.stdout:5/90: sync 2024-04-16T02:35:29.874 INFO:tasks.workunit.client.1.smithi153.stdout:9/139: creat d4/f23 x:0 0 0 2024-04-16T02:35:29.875 INFO:tasks.workunit.client.1.smithi153.stdout:6/81: write f4 [2724605,77433] 0 2024-04-16T02:35:29.875 INFO:tasks.workunit.client.1.smithi153.stdout:5/91: write d8/df/f11 [1886302,105540] 0 2024-04-16T02:35:29.877 INFO:tasks.workunit.client.1.smithi153.stdout:0/147: mkdir dd/d14/d19/d20/d2a 0 2024-04-16T02:35:29.880 INFO:tasks.workunit.client.1.smithi153.stdout:8/94: truncate d2/f23 269738 0 2024-04-16T02:35:29.880 INFO:tasks.workunit.client.1.smithi153.stdout:8/95: read d2/f17 [2985526,23771] 0 2024-04-16T02:35:29.881 INFO:tasks.workunit.client.1.smithi153.stdout:7/112: creat d0/d4/db/dd/f22 x:0 0 0 2024-04-16T02:35:29.883 INFO:tasks.workunit.client.1.smithi153.stdout:7/113: dread d0/f1e [0,4194304] 0 2024-04-16T02:35:29.883 INFO:tasks.workunit.client.1.smithi153.stdout:7/114: stat d0/d13 0 2024-04-16T02:35:29.890 INFO:tasks.workunit.client.1.smithi153.stdout:2/108: creat d9/dd/d18/f29 x:0 0 0 2024-04-16T02:35:29.893 INFO:tasks.workunit.client.1.smithi153.stdout:7/115: symlink d0/d4/db/dd/l23 0 2024-04-16T02:35:29.894 INFO:tasks.workunit.client.1.smithi153.stdout:9/140: mkdir d4/d6/dc/d24 0 2024-04-16T02:35:29.895 INFO:tasks.workunit.client.1.smithi153.stdout:9/141: chown d4/d6/dc/f11 7 1 2024-04-16T02:35:29.896 INFO:tasks.workunit.client.1.smithi153.stdout:2/109: dread f6 [0,4194304] 0 2024-04-16T02:35:29.896 INFO:tasks.workunit.client.1.smithi153.stdout:6/82: link f5 dc/d10/f1b 0 2024-04-16T02:35:29.903 INFO:tasks.workunit.client.1.smithi153.stdout:8/96: link d2/f14 d2/de/f24 0 2024-04-16T02:35:29.906 INFO:tasks.workunit.client.1.smithi153.stdout:7/116: mknod d0/d4/db/dd/c24 0 2024-04-16T02:35:29.907 INFO:tasks.workunit.client.0.smithi151.stderr:+ T=./tmp.04Kqyy33Bl 2024-04-16T02:35:29.907 INFO:tasks.workunit.client.0.smithi151.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.04Kqyy33Bl -l 1 -n 1000 -p 10 -v 2024-04-16T02:35:29.907 INFO:tasks.workunit.client.1.smithi153.stdout:4/145: dwrite d3/d9/db/de/f12 [0,4194304] 0 2024-04-16T02:35:29.908 INFO:tasks.workunit.client.1.smithi153.stdout:7/117: read - d0/d4/db/dd/f22 zero size 2024-04-16T02:35:29.910 INFO:tasks.workunit.client.1.smithi153.stdout:4/146: write d3/d9/db/de/f12 [3688928,60252] 0 2024-04-16T02:35:29.912 INFO:tasks.workunit.client.0.smithi151.stdout:seed = 1712327497 2024-04-16T02:35:29.913 INFO:tasks.workunit.client.1.smithi153.stdout:5/92: truncate f4 1126748 0 2024-04-16T02:35:29.914 INFO:tasks.workunit.client.1.smithi153.stdout:5/93: chown d8/c15 84625108 1 2024-04-16T02:35:29.916 INFO:tasks.workunit.client.1.smithi153.stdout:9/142: dwrite d4/d6/f20 [0,4194304] 0 2024-04-16T02:35:29.918 INFO:tasks.workunit.client.0.smithi151.stdout:0/0: dwrite - no filename 2024-04-16T02:35:29.919 INFO:tasks.workunit.client.0.smithi151.stdout:0/1: rename - no filename 2024-04-16T02:35:29.922 INFO:tasks.workunit.client.0.smithi151.stdout:1/0: stat - no entries 2024-04-16T02:35:29.922 INFO:tasks.workunit.client.0.smithi151.stdout:1/1: read - no filename 2024-04-16T02:35:29.922 INFO:tasks.workunit.client.0.smithi151.stdout:1/2: truncate - no filename 2024-04-16T02:35:29.923 INFO:tasks.workunit.client.1.smithi153.stdout:8/97: dread d2/f9 [0,4194304] 0 2024-04-16T02:35:29.924 INFO:tasks.workunit.client.1.smithi153.stdout:6/83: creat dc/d1a/f1c x:0 0 0 2024-04-16T02:35:29.924 INFO:tasks.workunit.client.0.smithi151.stdout:2/0: chown . 129344331 1 2024-04-16T02:35:29.924 INFO:tasks.workunit.client.0.smithi151.stdout:0/2: mknod c0 0 2024-04-16T02:35:29.926 INFO:tasks.workunit.client.1.smithi153.stdout:0/148: rename dd/d14/d19/d1c/c21 to dd/d14/d19/c2b 0 2024-04-16T02:35:29.927 INFO:tasks.workunit.client.0.smithi151.stdout:3/0: fdatasync - no filename 2024-04-16T02:35:29.927 INFO:tasks.workunit.client.0.smithi151.stdout:3/1: chown . 112806 1 2024-04-16T02:35:29.928 INFO:tasks.workunit.client.0.smithi151.stdout:0/3: symlink l1 0 2024-04-16T02:35:29.928 INFO:tasks.workunit.client.0.smithi151.stdout:0/4: write - no filename 2024-04-16T02:35:29.929 INFO:tasks.workunit.client.1.smithi153.stdout:6/84: dread f5 [0,4194304] 0 2024-04-16T02:35:29.930 INFO:tasks.workunit.client.1.smithi153.stdout:6/85: write dc/d1a/f1c [976866,15833] 0 2024-04-16T02:35:29.930 INFO:tasks.workunit.client.0.smithi151.stdout:1/3: creat f0 x:0 0 0 2024-04-16T02:35:29.930 INFO:tasks.workunit.client.0.smithi151.stdout:2/1: creat f0 x:0 0 0 2024-04-16T02:35:29.931 INFO:tasks.workunit.client.1.smithi153.stdout:6/86: readlink dc/l18 0 2024-04-16T02:35:29.932 INFO:tasks.workunit.client.1.smithi153.stdout:5/94: rmdir d8/df 39 2024-04-16T02:35:29.933 INFO:tasks.workunit.client.0.smithi151.stdout:4/0: chown . 134 1 2024-04-16T02:35:29.933 INFO:tasks.workunit.client.0.smithi151.stdout:0/5: creat f2 x:0 0 0 2024-04-16T02:35:29.934 INFO:tasks.workunit.client.0.smithi151.stdout:0/6: chown f2 151133737 1 2024-04-16T02:35:29.934 INFO:tasks.workunit.client.0.smithi151.stdout:2/2: rename f0 to f1 0 2024-04-16T02:35:29.934 INFO:tasks.workunit.client.1.smithi153.stdout:2/110: creat d9/dd/f2a x:0 0 0 2024-04-16T02:35:29.935 INFO:tasks.workunit.client.0.smithi151.stdout:5/0: dread - no filename 2024-04-16T02:35:29.935 INFO:tasks.workunit.client.0.smithi151.stdout:5/1: write - no filename 2024-04-16T02:35:29.936 INFO:tasks.workunit.client.0.smithi151.stdout:2/3: creat f2 x:0 0 0 2024-04-16T02:35:29.937 INFO:tasks.workunit.client.0.smithi151.stdout:0/7: creat f3 x:0 0 0 2024-04-16T02:35:29.939 INFO:tasks.workunit.client.0.smithi151.stdout:7/0: write - no filename 2024-04-16T02:35:29.939 INFO:tasks.workunit.client.0.smithi151.stdout:7/1: readlink - no filename 2024-04-16T02:35:29.939 INFO:tasks.workunit.client.0.smithi151.stdout:7/2: write - no filename 2024-04-16T02:35:29.939 INFO:tasks.workunit.client.0.smithi151.stdout:7/3: truncate - no filename 2024-04-16T02:35:29.940 INFO:tasks.workunit.client.0.smithi151.stdout:4/1: mkdir d0 0 2024-04-16T02:35:29.940 INFO:tasks.workunit.client.0.smithi151.stdout:4/2: dread - no filename 2024-04-16T02:35:29.940 INFO:tasks.workunit.client.0.smithi151.stdout:4/3: truncate - no filename 2024-04-16T02:35:29.940 INFO:tasks.workunit.client.0.smithi151.stdout:4/4: dread - no filename 2024-04-16T02:35:29.940 INFO:tasks.workunit.client.0.smithi151.stdout:2/4: symlink l3 0 2024-04-16T02:35:29.942 INFO:tasks.workunit.client.1.smithi153.stdout:7/118: rename d0/d4/db/f19 to d0/d4/db/dd/f25 0 2024-04-16T02:35:29.942 INFO:tasks.workunit.client.0.smithi151.stdout:6/0: read - no filename 2024-04-16T02:35:29.942 INFO:tasks.workunit.client.0.smithi151.stdout:6/1: stat - no entries 2024-04-16T02:35:29.942 INFO:tasks.workunit.client.0.smithi151.stdout:5/2: symlink l0 0 2024-04-16T02:35:29.944 INFO:tasks.workunit.client.0.smithi151.stdout:4/5: symlink d0/l1 0 2024-04-16T02:35:29.945 INFO:tasks.workunit.client.0.smithi151.stdout:5/3: unlink l0 0 2024-04-16T02:35:29.947 INFO:tasks.workunit.client.1.smithi153.stdout:2/111: rename d9/dd to d9/d2b 0 2024-04-16T02:35:29.950 INFO:tasks.workunit.client.0.smithi151.stdout:0/8: dwrite f2 [0,4194304] 0 2024-04-16T02:35:29.976 INFO:tasks.workunit.client.1.smithi153.stdout:6/87: rename cb to dc/d12/c1d 0 2024-04-16T02:35:29.976 INFO:tasks.workunit.client.0.smithi151.stdout:2/5: dwrite f1 [0,4194304] 0 2024-04-16T02:35:29.977 INFO:tasks.workunit.client.1.smithi153.stdout:7/119: mkdir d0/d4/db/d26 0 2024-04-16T02:35:29.978 INFO:tasks.workunit.client.1.smithi153.stdout:6/88: write f6 [3903596,6099] 0 2024-04-16T02:35:29.978 INFO:tasks.workunit.client.1.smithi153.stdout:2/112: truncate f6 2315782 0 2024-04-16T02:35:29.980 INFO:tasks.workunit.client.1.smithi153.stdout:4/147: getdents d3/d9/db/d17 0 2024-04-16T02:35:29.981 INFO:tasks.workunit.client.0.smithi151.stdout:8/0: dwrite - no filename 2024-04-16T02:35:29.982 INFO:tasks.workunit.client.0.smithi151.stdout:8/1: stat - no entries 2024-04-16T02:35:29.982 INFO:tasks.workunit.client.0.smithi151.stdout:8/2: truncate - no filename 2024-04-16T02:35:29.982 INFO:tasks.workunit.client.0.smithi151.stdout:8/3: link - no file 2024-04-16T02:35:29.982 INFO:tasks.workunit.client.0.smithi151.stdout:4/6: creat d0/f2 x:0 0 0 2024-04-16T02:35:29.982 INFO:tasks.workunit.client.1.smithi153.stdout:8/98: getdents d2/d1c/d1f 0 2024-04-16T02:35:29.983 INFO:tasks.workunit.client.0.smithi151.stdout:5/4: creat f1 x:0 0 0 2024-04-16T02:35:29.984 INFO:tasks.workunit.client.0.smithi151.stdout:5/5: read - f1 zero size 2024-04-16T02:35:29.984 INFO:tasks.workunit.client.0.smithi151.stdout:5/6: readlink - no filename 2024-04-16T02:35:29.984 INFO:tasks.workunit.client.1.smithi153.stdout:6/89: creat dc/d12/d15/f1e x:0 0 0 2024-04-16T02:35:29.985 INFO:tasks.workunit.client.1.smithi153.stdout:5/95: getdents d8 0 2024-04-16T02:35:29.987 INFO:tasks.workunit.client.0.smithi151.stdout:0/9: rename f2 to f4 0 2024-04-16T02:35:29.988 INFO:tasks.workunit.client.1.smithi153.stdout:4/148: mkdir d3/d9/d24/d35 0 2024-04-16T02:35:29.989 INFO:tasks.workunit.client.0.smithi151.stdout:0/10: chown f3 31573737 1 2024-04-16T02:35:29.989 INFO:tasks.workunit.client.0.smithi151.stdout:0/11: dread - f3 zero size 2024-04-16T02:35:29.989 INFO:tasks.workunit.client.1.smithi153.stdout:6/90: dread f4 [0,4194304] 0 2024-04-16T02:35:29.989 INFO:tasks.workunit.client.1.smithi153.stdout:4/149: dread - d3/d9/db/de/f26 zero size 2024-04-16T02:35:29.989 INFO:tasks.workunit.client.1.smithi153.stdout:7/120: creat d0/d4/db/d26/f27 x:0 0 0 2024-04-16T02:35:29.990 INFO:tasks.workunit.client.1.smithi153.stdout:6/91: readlink dc/l16 0 2024-04-16T02:35:29.990 INFO:tasks.workunit.client.0.smithi151.stdout:2/6: mknod c4 0 2024-04-16T02:35:29.990 INFO:tasks.workunit.client.1.smithi153.stdout:6/92: chown dc/fe 33 1 2024-04-16T02:35:29.990 INFO:tasks.workunit.client.1.smithi153.stdout:4/150: write d3/d9/db/d17/f21 [1060533,122956] 0 2024-04-16T02:35:29.992 INFO:tasks.workunit.client.1.smithi153.stdout:4/151: write d3/ff [1078320,45068] 0 2024-04-16T02:35:29.993 INFO:tasks.workunit.client.0.smithi151.stdout:4/7: creat d0/f3 x:0 0 0 2024-04-16T02:35:29.994 INFO:tasks.workunit.client.0.smithi151.stdout:9/0: symlink l0 0 2024-04-16T02:35:29.998 INFO:tasks.workunit.client.1.smithi153.stdout:5/96: creat d8/df/f16 x:0 0 0 2024-04-16T02:35:29.998 INFO:tasks.workunit.client.1.smithi153.stdout:5/97: readlink l2 0 2024-04-16T02:35:29.998 INFO:tasks.workunit.client.1.smithi153.stdout:7/121: creat d0/d4/db/dd/f28 x:0 0 0 2024-04-16T02:35:29.998 INFO:tasks.workunit.client.1.smithi153.stdout:5/98: dread - d8/df/f16 zero size 2024-04-16T02:35:29.998 INFO:tasks.workunit.client.1.smithi153.stdout:6/93: dwrite f6 [0,4194304] 0 2024-04-16T02:35:29.999 INFO:tasks.workunit.client.0.smithi151.stdout:5/7: mkdir d2 0 2024-04-16T02:35:29.999 INFO:tasks.workunit.client.0.smithi151.stdout:5/8: chown f1 2371 1 2024-04-16T02:35:29.999 INFO:tasks.workunit.client.1.smithi153.stdout:2/113: getdents d9/d2b/d18 0 2024-04-16T02:35:29.999 INFO:tasks.workunit.client.0.smithi151.stdout:5/9: rename d2 to d2/d3 22 2024-04-16T02:35:30.000 INFO:tasks.workunit.client.1.smithi153.stdout:5/99: symlink d8/l17 0 2024-04-16T02:35:30.000 INFO:tasks.workunit.client.0.smithi151.stdout:2/7: symlink l5 0 2024-04-16T02:35:30.001 INFO:tasks.workunit.client.0.smithi151.stdout:5/10: write f1 [650213,50152] 0 2024-04-16T02:35:30.001 INFO:tasks.workunit.client.1.smithi153.stdout:1/126: sync 2024-04-16T02:35:30.001 INFO:tasks.workunit.client.1.smithi153.stdout:3/131: sync 2024-04-16T02:35:30.002 INFO:tasks.workunit.client.1.smithi153.stdout:7/122: fsync d0/d4/db/d26/f27 0 2024-04-16T02:35:30.002 INFO:tasks.workunit.client.1.smithi153.stdout:1/127: chown d4/l5 32917617 1 2024-04-16T02:35:30.004 INFO:tasks.workunit.client.1.smithi153.stdout:1/128: truncate d4/fe 131325 0 2024-04-16T02:35:30.004 INFO:tasks.workunit.client.0.smithi151.stdout:2/8: write f2 [852702,115948] 0 2024-04-16T02:35:30.004 INFO:tasks.workunit.client.0.smithi151.stdout:8/4: getdents . 0 2024-04-16T02:35:30.004 INFO:tasks.workunit.client.0.smithi151.stdout:8/5: dread - no filename 2024-04-16T02:35:30.004 INFO:tasks.workunit.client.0.smithi151.stdout:8/6: stat - no entries 2024-04-16T02:35:30.004 INFO:tasks.workunit.client.0.smithi151.stdout:8/7: dwrite - no filename 2024-04-16T02:35:30.004 INFO:tasks.workunit.client.0.smithi151.stdout:8/8: dread - no filename 2024-04-16T02:35:30.005 INFO:tasks.workunit.client.1.smithi153.stdout:7/123: fsync d0/d4/db/dd/f28 0 2024-04-16T02:35:30.005 INFO:tasks.workunit.client.1.smithi153.stdout:7/124: rename d0 to d0/d4/db/dd/d29 22 2024-04-16T02:35:30.005 INFO:tasks.workunit.client.0.smithi151.stdout:2/9: rmdir - no directory 2024-04-16T02:35:30.006 INFO:tasks.workunit.client.1.smithi153.stdout:6/94: mknod dc/c1f 0 2024-04-16T02:35:30.006 INFO:tasks.workunit.client.0.smithi151.stdout:9/1: unlink l0 0 2024-04-16T02:35:30.006 INFO:tasks.workunit.client.0.smithi151.stdout:9/2: rmdir - no directory 2024-04-16T02:35:30.006 INFO:tasks.workunit.client.0.smithi151.stdout:9/3: dwrite - no filename 2024-04-16T02:35:30.006 INFO:tasks.workunit.client.0.smithi151.stdout:9/4: dread - no filename 2024-04-16T02:35:30.007 INFO:tasks.workunit.client.1.smithi153.stdout:8/99: getdents d2 0 2024-04-16T02:35:30.009 INFO:tasks.workunit.client.1.smithi153.stdout:2/114: mkdir d9/d24/d27/d2c 0 2024-04-16T02:35:30.011 INFO:tasks.workunit.client.0.smithi151.stdout:4/8: dwrite d0/f3 [0,4194304] 0 2024-04-16T02:35:30.012 INFO:tasks.workunit.client.0.smithi151.stdout:2/10: unlink l3 0 2024-04-16T02:35:30.013 INFO:tasks.workunit.client.1.smithi153.stdout:1/129: dwrite d4/dd/d13/f1a [0,4194304] 0 2024-04-16T02:35:30.014 INFO:tasks.workunit.client.0.smithi151.stdout:9/5: mkdir d1 0 2024-04-16T02:35:30.014 INFO:tasks.workunit.client.0.smithi151.stdout:9/6: read - no filename 2024-04-16T02:35:30.015 INFO:tasks.workunit.client.1.smithi153.stdout:1/130: readlink d4/dd/l10 0 2024-04-16T02:35:30.015 INFO:tasks.workunit.client.1.smithi153.stdout:6/95: dread dc/f11 [0,4194304] 0 2024-04-16T02:35:30.016 INFO:tasks.workunit.client.1.smithi153.stdout:7/125: rmdir d0/d13 39 2024-04-16T02:35:30.018 INFO:tasks.workunit.client.0.smithi151.stdout:8/9: creat f0 x:0 0 0 2024-04-16T02:35:30.022 INFO:tasks.workunit.client.0.smithi151.stdout:8/10: write f0 [955340,72970] 0 2024-04-16T02:35:30.022 INFO:tasks.workunit.client.0.smithi151.stdout:8/11: readlink - no filename 2024-04-16T02:35:30.024 INFO:tasks.workunit.client.1.smithi153.stdout:2/115: rmdir d9/d2b/d1c/d20 39 2024-04-16T02:35:30.024 INFO:tasks.workunit.client.1.smithi153.stdout:8/100: symlink d2/d1c/d1f/l25 0 2024-04-16T02:35:30.024 INFO:tasks.workunit.client.0.smithi151.stdout:5/11: dwrite f1 [0,4194304] 0 2024-04-16T02:35:30.025 INFO:tasks.workunit.client.1.smithi153.stdout:1/131: creat d4/dd/d13/f2a x:0 0 0 2024-04-16T02:35:30.025 INFO:tasks.workunit.client.1.smithi153.stdout:7/126: rename d0/d4/db/dd/f1d to d0/d4/db/f2a 0 2024-04-16T02:35:30.025 INFO:tasks.workunit.client.1.smithi153.stdout:1/132: stat d4/dd 0 2024-04-16T02:35:30.025 INFO:tasks.workunit.client.1.smithi153.stdout:6/96: dread dc/f11 [0,4194304] 0 2024-04-16T02:35:30.027 INFO:tasks.workunit.client.0.smithi151.stdout:9/7: mkdir d1/d2 0 2024-04-16T02:35:30.027 INFO:tasks.workunit.client.1.smithi153.stdout:5/100: mknod d8/c18 0 2024-04-16T02:35:30.028 INFO:tasks.workunit.client.0.smithi151.stdout:5/12: write f1 [2488558,42635] 0 2024-04-16T02:35:30.028 INFO:tasks.workunit.client.0.smithi151.stdout:0/12: fsync f4 0 2024-04-16T02:35:30.030 INFO:tasks.workunit.client.0.smithi151.stdout:0/13: fdatasync f4 0 2024-04-16T02:35:30.030 INFO:tasks.workunit.client.1.smithi153.stdout:4/152: truncate d3/d9/db/de/f12 4006946 0 2024-04-16T02:35:30.031 INFO:tasks.workunit.client.0.smithi151.stdout:8/12: symlink l1 0 2024-04-16T02:35:30.031 INFO:tasks.workunit.client.1.smithi153.stdout:4/153: readlink d3/d9/db/d17/l33 0 2024-04-16T02:35:30.031 INFO:tasks.workunit.client.0.smithi151.stdout:0/14: fsync f4 0 2024-04-16T02:35:30.032 INFO:tasks.workunit.client.0.smithi151.stdout:8/13: stat f0 0 2024-04-16T02:35:30.032 INFO:tasks.workunit.client.0.smithi151.stdout:2/11: dwrite f1 [0,4194304] 0 2024-04-16T02:35:30.034 INFO:tasks.workunit.client.1.smithi153.stdout:3/132: getdents d6 0 2024-04-16T02:35:30.034 INFO:tasks.workunit.client.0.smithi151.stdout:4/9: dread d0/f3 [0,4194304] 0 2024-04-16T02:35:30.035 INFO:tasks.workunit.client.0.smithi151.stdout:4/10: chown d0/f2 0 1 2024-04-16T02:35:30.035 INFO:tasks.workunit.client.0.smithi151.stdout:0/15: truncate f3 664121 0 2024-04-16T02:35:30.036 INFO:tasks.workunit.client.1.smithi153.stdout:5/101: dread - d8/df/f16 zero size 2024-04-16T02:35:30.037 INFO:tasks.workunit.client.1.smithi153.stdout:6/97: creat dc/d12/d15/f20 x:0 0 0 2024-04-16T02:35:30.037 INFO:tasks.workunit.client.1.smithi153.stdout:5/102: fdatasync d8/df/f16 0 2024-04-16T02:35:30.037 INFO:tasks.workunit.client.0.smithi151.stdout:4/11: write d0/f2 [305995,69322] 0 2024-04-16T02:35:30.038 INFO:tasks.workunit.client.0.smithi151.stdout:8/14: creat f2 x:0 0 0 2024-04-16T02:35:30.039 INFO:tasks.workunit.client.1.smithi153.stdout:1/133: dread d4/dd/f16 [4194304,4194304] 0 2024-04-16T02:35:30.039 INFO:tasks.workunit.client.1.smithi153.stdout:5/103: write d8/df/f11 [2522779,101552] 0 2024-04-16T02:35:30.040 INFO:tasks.workunit.client.0.smithi151.stdout:9/8: rmdir d1/d2 0 2024-04-16T02:35:30.040 INFO:tasks.workunit.client.0.smithi151.stdout:9/9: dread - no filename 2024-04-16T02:35:30.041 INFO:tasks.workunit.client.1.smithi153.stdout:4/154: chown d3/d9/db/d17/l18 7903 1 2024-04-16T02:35:30.045 INFO:tasks.workunit.client.1.smithi153.stdout:6/98: fsync f5 0 2024-04-16T02:35:30.046 INFO:tasks.workunit.client.0.smithi151.stdout:8/15: rename f0 to f3 0 2024-04-16T02:35:30.046 INFO:tasks.workunit.client.0.smithi151.stdout:9/10: mkdir d1/d3 0 2024-04-16T02:35:30.046 INFO:tasks.workunit.client.0.smithi151.stdout:9/11: link - no file 2024-04-16T02:35:30.046 INFO:tasks.workunit.client.0.smithi151.stdout:9/12: write - no filename 2024-04-16T02:35:30.046 INFO:tasks.workunit.client.0.smithi151.stdout:9/13: readlink - no filename 2024-04-16T02:35:30.046 INFO:tasks.workunit.client.0.smithi151.stdout:9/14: fsync - no filename 2024-04-16T02:35:30.046 INFO:tasks.workunit.client.0.smithi151.stdout:9/15: unlink - no file 2024-04-16T02:35:30.048 INFO:tasks.workunit.client.1.smithi153.stdout:5/104: dwrite d8/df/f10 [0,4194304] 0 2024-04-16T02:35:30.048 INFO:tasks.workunit.client.0.smithi151.stdout:8/16: truncate f3 1591339 0 2024-04-16T02:35:30.049 INFO:tasks.workunit.client.0.smithi151.stdout:2/12: dwrite f1 [0,4194304] 0 2024-04-16T02:35:30.051 INFO:tasks.workunit.client.0.smithi151.stdout:8/17: dread - f2 zero size 2024-04-16T02:35:30.051 INFO:tasks.workunit.client.0.smithi151.stdout:9/16: symlink d1/l4 0 2024-04-16T02:35:30.051 INFO:tasks.workunit.client.1.smithi153.stdout:5/105: fdatasync d8/df/f16 0 2024-04-16T02:35:30.053 INFO:tasks.workunit.client.1.smithi153.stdout:5/106: chown l6 12686566 1 2024-04-16T02:35:30.055 INFO:tasks.workunit.client.1.smithi153.stdout:5/107: fsync d8/df/f16 0 2024-04-16T02:35:30.055 INFO:tasks.workunit.client.1.smithi153.stdout:7/127: dread d0/d4/db/f12 [0,4194304] 0 2024-04-16T02:35:30.055 INFO:tasks.workunit.client.0.smithi151.stdout:8/18: write f2 [990956,107054] 0 2024-04-16T02:35:30.055 INFO:tasks.workunit.client.0.smithi151.stdout:4/12: dread d0/f3 [0,4194304] 0 2024-04-16T02:35:30.057 INFO:tasks.workunit.client.0.smithi151.stdout:2/13: unlink f2 0 2024-04-16T02:35:30.057 INFO:tasks.workunit.client.1.smithi153.stdout:5/108: write d8/df/f10 [3519942,3684] 0 2024-04-16T02:35:30.057 INFO:tasks.workunit.client.0.smithi151.stdout:0/16: dwrite f4 [0,4194304] 0 2024-04-16T02:35:30.059 INFO:tasks.workunit.client.0.smithi151.stdout:9/17: creat d1/f5 x:0 0 0 2024-04-16T02:35:30.060 INFO:tasks.workunit.client.0.smithi151.stdout:0/17: write f3 [1498975,5445] 0 2024-04-16T02:35:30.063 INFO:tasks.workunit.client.1.smithi153.stdout:2/116: write f6 [1211926,14133] 0 2024-04-16T02:35:30.064 INFO:tasks.workunit.client.1.smithi153.stdout:8/101: write d2/d13/f1b [824737,102761] 0 2024-04-16T02:35:30.065 INFO:tasks.workunit.client.1.smithi153.stdout:2/117: write d9/d2b/d1c/f28 [434322,72975] 0 2024-04-16T02:35:30.065 INFO:tasks.workunit.client.0.smithi151.stdout:5/13: truncate f1 778308 0 2024-04-16T02:35:30.067 INFO:tasks.workunit.client.0.smithi151.stdout:0/18: unlink l1 0 2024-04-16T02:35:30.070 INFO:tasks.workunit.client.1.smithi153.stdout:6/99: creat dc/f21 x:0 0 0 2024-04-16T02:35:30.070 INFO:tasks.workunit.client.1.smithi153.stdout:2/118: dread d9/fb [0,4194304] 0 2024-04-16T02:35:30.070 INFO:tasks.workunit.client.1.smithi153.stdout:0/149: sync 2024-04-16T02:35:30.072 INFO:tasks.workunit.client.1.smithi153.stdout:9/143: sync 2024-04-16T02:35:30.072 INFO:tasks.workunit.client.0.smithi151.stdout:2/14: rename f1 to f6 0 2024-04-16T02:35:30.072 INFO:tasks.workunit.client.0.smithi151.stdout:9/18: dwrite d1/f5 [0,4194304] 0 2024-04-16T02:35:30.075 INFO:tasks.workunit.client.1.smithi153.stdout:7/128: rename d0/d4/db/dd to d0/d4/db/d2b 0 2024-04-16T02:35:30.075 INFO:tasks.workunit.client.1.smithi153.stdout:5/109: fdatasync d8/df/f10 0 2024-04-16T02:35:30.077 INFO:tasks.workunit.client.1.smithi153.stdout:8/102: creat d2/d1c/f26 x:0 0 0 2024-04-16T02:35:30.078 INFO:tasks.workunit.client.0.smithi151.stdout:0/19: rename f4 to f5 0 2024-04-16T02:35:30.080 INFO:tasks.workunit.client.1.smithi153.stdout:3/133: truncate d6/f20 829987 0 2024-04-16T02:35:30.083 INFO:tasks.workunit.client.1.smithi153.stdout:0/150: unlink f1 0 2024-04-16T02:35:30.084 INFO:tasks.workunit.client.1.smithi153.stdout:6/100: creat dc/d12/d15/f22 x:0 0 0 2024-04-16T02:35:30.085 INFO:tasks.workunit.client.0.smithi151.stdout:9/19: dwrite d1/f5 [0,4194304] 0 2024-04-16T02:35:30.087 INFO:tasks.workunit.client.1.smithi153.stdout:2/119: rename c2 to d9/d24/d27/d2c/c2d 0 2024-04-16T02:35:30.088 INFO:tasks.workunit.client.1.smithi153.stdout:4/155: write d3/d9/db/de/f12 [1749150,81588] 0 2024-04-16T02:35:30.088 INFO:tasks.workunit.client.1.smithi153.stdout:1/134: write d4/dd/d13/f14 [413546,25831] 0 2024-04-16T02:35:30.089 INFO:tasks.workunit.client.1.smithi153.stdout:2/120: chown d9/c22 53188 1 2024-04-16T02:35:30.090 INFO:tasks.workunit.client.1.smithi153.stdout:4/156: stat d3/d9/db/d17 0 2024-04-16T02:35:30.091 INFO:tasks.workunit.client.1.smithi153.stdout:1/135: truncate d4/fe 861745 0 2024-04-16T02:35:30.091 INFO:tasks.workunit.client.1.smithi153.stdout:5/110: stat d8/le 0 2024-04-16T02:35:30.092 INFO:tasks.workunit.client.1.smithi153.stdout:8/103: mkdir d2/d1c/d1f/d27 0 2024-04-16T02:35:30.093 INFO:tasks.workunit.client.1.smithi153.stdout:8/104: truncate d2/f12 863642 0 2024-04-16T02:35:30.095 INFO:tasks.workunit.client.1.smithi153.stdout:0/151: creat dd/d14/d19/d1c/f2c x:0 0 0 2024-04-16T02:35:30.095 INFO:tasks.workunit.client.0.smithi151.stdout:9/20: dread d1/f5 [0,4194304] 0 2024-04-16T02:35:30.095 INFO:tasks.workunit.client.1.smithi153.stdout:0/152: read dd/fe [3785121,35019] 0 2024-04-16T02:35:30.097 INFO:tasks.workunit.client.0.smithi151.stdout:0/20: fdatasync f5 0 2024-04-16T02:35:30.099 INFO:tasks.workunit.client.0.smithi151.stdout:5/14: link f1 d2/f4 0 2024-04-16T02:35:30.099 INFO:tasks.workunit.client.1.smithi153.stdout:9/144: rename d4/d6/dc/f19 to d4/d6/dc/d24/f25 0 2024-04-16T02:35:30.099 INFO:tasks.workunit.client.0.smithi151.stdout:8/19: truncate f3 714382 0 2024-04-16T02:35:30.100 INFO:tasks.workunit.client.1.smithi153.stdout:9/145: fdatasync d4/d6/dc/f11 0 2024-04-16T02:35:30.101 INFO:tasks.workunit.client.0.smithi151.stdout:9/21: dread d1/f5 [0,4194304] 0 2024-04-16T02:35:30.101 INFO:tasks.workunit.client.0.smithi151.stdout:0/21: symlink l6 0 2024-04-16T02:35:30.102 INFO:tasks.workunit.client.0.smithi151.stdout:5/15: rmdir d2 39 2024-04-16T02:35:30.102 INFO:tasks.workunit.client.1.smithi153.stdout:2/121: creat d9/d24/d27/f2e x:0 0 0 2024-04-16T02:35:30.103 INFO:tasks.workunit.client.0.smithi151.stdout:2/15: getdents . 0 2024-04-16T02:35:30.105 INFO:tasks.workunit.client.0.smithi151.stdout:8/20: symlink l4 0 2024-04-16T02:35:30.105 INFO:tasks.workunit.client.0.smithi151.stdout:4/13: truncate d0/f3 1407209 0 2024-04-16T02:35:30.106 INFO:tasks.workunit.client.1.smithi153.stdout:4/157: truncate d3/d9/d24/f2b 4037 0 2024-04-16T02:35:30.106 INFO:tasks.workunit.client.1.smithi153.stdout:4/158: chown d3/d9/db/d17/d28 13039673 1 2024-04-16T02:35:30.106 INFO:tasks.workunit.client.0.smithi151.stdout:9/22: dread d1/f5 [0,4194304] 0 2024-04-16T02:35:30.108 INFO:tasks.workunit.client.0.smithi151.stdout:0/22: unlink l6 0 2024-04-16T02:35:30.109 INFO:tasks.workunit.client.1.smithi153.stdout:0/153: write dd/f16 [1533148,55785] 0 2024-04-16T02:35:30.109 INFO:tasks.workunit.client.1.smithi153.stdout:0/154: readlink dd/l13 0 2024-04-16T02:35:30.110 INFO:tasks.workunit.client.1.smithi153.stdout:0/155: fsync dd/ff 0 2024-04-16T02:35:30.110 INFO:tasks.workunit.client.1.smithi153.stdout:4/159: dread d3/ff [0,4194304] 0 2024-04-16T02:35:30.111 INFO:tasks.workunit.client.1.smithi153.stdout:0/156: chown dd/d14 178398758 1 2024-04-16T02:35:30.111 INFO:tasks.workunit.client.0.smithi151.stdout:9/23: dread d1/f5 [0,4194304] 0 2024-04-16T02:35:30.112 INFO:tasks.workunit.client.0.smithi151.stdout:2/16: rename f6 to f7 0 2024-04-16T02:35:30.114 INFO:tasks.workunit.client.1.smithi153.stdout:4/160: dread d3/ff [4194304,4194304] 0 2024-04-16T02:35:30.115 INFO:tasks.workunit.client.1.smithi153.stdout:4/161: truncate d3/d9/f29 203283 0 2024-04-16T02:35:30.116 INFO:tasks.workunit.client.0.smithi151.stdout:4/14: symlink d0/l4 0 2024-04-16T02:35:30.116 INFO:tasks.workunit.client.0.smithi151.stdout:9/24: dread d1/f5 [0,4194304] 0 2024-04-16T02:35:30.116 INFO:tasks.workunit.client.1.smithi153.stdout:3/134: rename d6/f2b to d6/d10/d22/d2e/f35 0 2024-04-16T02:35:30.117 INFO:tasks.workunit.client.0.smithi151.stdout:0/23: mknod c7 0 2024-04-16T02:35:30.118 INFO:tasks.workunit.client.0.smithi151.stdout:5/16: creat d2/f5 x:0 0 0 2024-04-16T02:35:30.119 INFO:tasks.workunit.client.1.smithi153.stdout:7/129: write d0/d4/db/f12 [1341275,66043] 0 2024-04-16T02:35:30.120 INFO:tasks.workunit.client.0.smithi151.stdout:2/17: dread f7 [0,4194304] 0 2024-04-16T02:35:30.120 INFO:tasks.workunit.client.1.smithi153.stdout:2/122: fsync d9/d24/d27/f2e 0 2024-04-16T02:35:30.122 INFO:tasks.workunit.client.0.smithi151.stdout:5/17: write f1 [887596,74361] 0 2024-04-16T02:35:30.122 INFO:tasks.workunit.client.0.smithi151.stdout:9/25: mkdir d1/d3/d6 0 2024-04-16T02:35:30.126 INFO:tasks.workunit.client.0.smithi151.stdout:0/24: dread f3 [0,4194304] 0 2024-04-16T02:35:30.130 INFO:tasks.workunit.client.0.smithi151.stdout:9/26: creat d1/f7 x:0 0 0 2024-04-16T02:35:30.131 INFO:tasks.workunit.client.0.smithi151.stdout:0/25: dread f5 [0,4194304] 0 2024-04-16T02:35:30.136 INFO:tasks.workunit.client.0.smithi151.stdout:9/27: mknod d1/d3/c8 0 2024-04-16T02:35:30.136 INFO:tasks.workunit.client.1.smithi153.stdout:4/162: symlink d3/d9/db/d17/l36 0 2024-04-16T02:35:30.138 INFO:tasks.workunit.client.0.smithi151.stdout:0/26: mknod c8 0 2024-04-16T02:35:30.138 INFO:tasks.workunit.client.1.smithi153.stdout:6/101: rename f5 to dc/d12/d15/f23 0 2024-04-16T02:35:30.139 INFO:tasks.workunit.client.0.smithi151.stdout:9/28: write d1/f5 [405371,54586] 0 2024-04-16T02:35:30.140 INFO:tasks.workunit.client.0.smithi151.stdout:0/27: creat f9 x:0 0 0 2024-04-16T02:35:30.142 INFO:tasks.workunit.client.1.smithi153.stdout:9/146: mknod d4/c26 0 2024-04-16T02:35:30.142 INFO:tasks.workunit.client.0.smithi151.stdout:9/29: mknod d1/c9 0 2024-04-16T02:35:30.142 INFO:tasks.workunit.client.0.smithi151.stdout:4/15: truncate d0/f2 4980 0 2024-04-16T02:35:30.144 INFO:tasks.workunit.client.0.smithi151.stdout:0/28: creat fa x:0 0 0 2024-04-16T02:35:30.145 INFO:tasks.workunit.client.1.smithi153.stdout:7/130: mknod d0/d4/db/c2c 0 2024-04-16T02:35:30.145 INFO:tasks.workunit.client.0.smithi151.stdout:4/16: symlink d0/l5 0 2024-04-16T02:35:30.147 INFO:tasks.workunit.client.0.smithi151.stdout:2/18: truncate f7 2482099 0 2024-04-16T02:35:30.147 INFO:tasks.workunit.client.0.smithi151.stdout:8/21: dwrite f3 [0,4194304] 0 2024-04-16T02:35:30.148 INFO:tasks.workunit.client.0.smithi151.stdout:5/18: truncate f1 671292 0 2024-04-16T02:35:30.148 INFO:tasks.workunit.client.1.smithi153.stdout:0/157: symlink dd/d14/d19/l2d 0 2024-04-16T02:35:30.149 INFO:tasks.workunit.client.1.smithi153.stdout:3/135: dwrite d6/d10/d22/d2e/f35 [0,4194304] 0 2024-04-16T02:35:30.149 INFO:tasks.workunit.client.1.smithi153.stdout:0/158: stat dd/f16 0 2024-04-16T02:35:30.149 INFO:tasks.workunit.client.0.smithi151.stdout:8/22: write f3 [4023911,44785] 0 2024-04-16T02:35:30.150 INFO:tasks.workunit.client.0.smithi151.stdout:0/29: mknod cb 0 2024-04-16T02:35:30.150 INFO:tasks.workunit.client.0.smithi151.stdout:5/19: write d2/f5 [17410,47295] 0 2024-04-16T02:35:30.153 INFO:tasks.workunit.client.0.smithi151.stdout:4/17: mkdir d0/d6 0 2024-04-16T02:35:30.154 INFO:tasks.workunit.client.1.smithi153.stdout:1/136: getdents d4/dd/d18 0 2024-04-16T02:35:30.155 INFO:tasks.workunit.client.1.smithi153.stdout:2/123: symlink d9/d2b/d1c/l2f 0 2024-04-16T02:35:30.156 INFO:tasks.workunit.client.1.smithi153.stdout:7/131: stat d0/d13 0 2024-04-16T02:35:30.158 INFO:tasks.workunit.client.1.smithi153.stdout:8/105: link d2/c18 d2/c28 0 2024-04-16T02:35:30.158 INFO:tasks.workunit.client.0.smithi151.stdout:0/30: dread f5 [0,4194304] 0 2024-04-16T02:35:30.161 INFO:tasks.workunit.client.1.smithi153.stdout:4/163: mknod d3/d9/db/d17/d28/c37 0 2024-04-16T02:35:30.162 INFO:tasks.workunit.client.1.smithi153.stdout:3/136: mknod d6/d10/d22/d2e/c36 0 2024-04-16T02:35:30.162 INFO:tasks.workunit.client.1.smithi153.stdout:4/164: truncate d3/d9/f23 1181128 0 2024-04-16T02:35:30.165 INFO:tasks.workunit.client.1.smithi153.stdout:2/124: creat d9/d24/d27/f30 x:0 0 0 2024-04-16T02:35:30.165 INFO:tasks.workunit.client.1.smithi153.stdout:2/125: chown d9/d2b/f2a 489719 1 2024-04-16T02:35:30.166 INFO:tasks.workunit.client.0.smithi151.stdout:8/23: dwrite f3 [0,4194304] 0 2024-04-16T02:35:30.167 INFO:tasks.workunit.client.0.smithi151.stdout:0/31: mkdir dc 0 2024-04-16T02:35:30.167 INFO:tasks.workunit.client.0.smithi151.stdout:0/32: dread - fa zero size 2024-04-16T02:35:30.170 INFO:tasks.workunit.client.1.smithi153.stdout:7/132: symlink d0/d4/l2d 0 2024-04-16T02:35:30.170 INFO:tasks.workunit.client.0.smithi151.stdout:8/24: creat f5 x:0 0 0 2024-04-16T02:35:30.171 INFO:tasks.workunit.client.0.smithi151.stdout:2/19: getdents . 0 2024-04-16T02:35:30.172 INFO:tasks.workunit.client.1.smithi153.stdout:2/126: symlink d9/d2b/d18/l31 0 2024-04-16T02:35:30.172 INFO:tasks.workunit.client.0.smithi151.stdout:0/33: dread f5 [0,4194304] 0 2024-04-16T02:35:30.173 INFO:tasks.workunit.client.0.smithi151.stdout:8/25: mknod c6 0 2024-04-16T02:35:30.174 INFO:tasks.workunit.client.1.smithi153.stdout:9/147: link d4/fd d4/f27 0 2024-04-16T02:35:30.174 INFO:tasks.workunit.client.0.smithi151.stdout:8/26: write f2 [1125028,2836] 0 2024-04-16T02:35:30.176 INFO:tasks.workunit.client.1.smithi153.stdout:7/133: mknod d0/d4/db/d26/c2e 0 2024-04-16T02:35:30.177 INFO:tasks.workunit.client.0.smithi151.stdout:0/34: creat dc/fd x:0 0 0 2024-04-16T02:35:30.178 INFO:tasks.workunit.client.1.smithi153.stdout:9/148: read - d4/f13 zero size 2024-04-16T02:35:30.178 INFO:tasks.workunit.client.0.smithi151.stdout:8/27: mknod c7 0 2024-04-16T02:35:30.179 INFO:tasks.workunit.client.0.smithi151.stdout:8/28: truncate f5 389278 0 2024-04-16T02:35:30.179 INFO:tasks.workunit.client.0.smithi151.stdout:8/29: chown c6 1 1 2024-04-16T02:35:30.180 INFO:tasks.workunit.client.1.smithi153.stdout:9/149: dread - d4/d6/dc/f21 zero size 2024-04-16T02:35:30.180 INFO:tasks.workunit.client.0.smithi151.stdout:0/35: mkdir dc/de 0 2024-04-16T02:35:30.183 INFO:tasks.workunit.client.1.smithi153.stdout:2/127: dwrite d9/d2b/d1c/f28 [0,4194304] 0 2024-04-16T02:35:30.183 INFO:tasks.workunit.client.0.smithi151.stdout:8/30: symlink l8 0 2024-04-16T02:35:30.184 INFO:tasks.workunit.client.1.smithi153.stdout:9/150: write d4/fd [716263,11563] 0 2024-04-16T02:35:30.185 INFO:tasks.workunit.client.0.smithi151.stdout:0/36: creat dc/ff x:0 0 0 2024-04-16T02:35:30.186 INFO:tasks.workunit.client.1.smithi153.stdout:7/134: getdents d0 0 2024-04-16T02:35:30.190 INFO:tasks.workunit.client.1.smithi153.stdout:9/151: creat d4/d6/dc/d24/f28 x:0 0 0 2024-04-16T02:35:30.191 INFO:tasks.workunit.client.1.smithi153.stdout:0/159: write dd/f22 [801516,86307] 0 2024-04-16T02:35:30.193 INFO:tasks.workunit.client.1.smithi153.stdout:2/128: dwrite d9/fc [0,4194304] 0 2024-04-16T02:35:30.196 INFO:tasks.workunit.client.1.smithi153.stdout:8/106: write d2/f17 [5757176,128597] 0 2024-04-16T02:35:30.198 INFO:tasks.workunit.client.0.smithi151.stdout:4/18: write d0/f3 [11875,107261] 0 2024-04-16T02:35:30.198 INFO:tasks.workunit.client.1.smithi153.stdout:3/137: write f1 [2059963,73657] 0 2024-04-16T02:35:30.200 INFO:tasks.workunit.client.1.smithi153.stdout:0/160: fsync dd/f22 0 2024-04-16T02:35:30.203 INFO:tasks.workunit.client.1.smithi153.stdout:2/129: symlink d9/d2b/l32 0 2024-04-16T02:35:30.203 INFO:tasks.workunit.client.1.smithi153.stdout:8/107: truncate d2/f12 1770227 0 2024-04-16T02:35:30.206 INFO:tasks.workunit.client.1.smithi153.stdout:6/102: dwrite dc/d10/f1b [0,4194304] 0 2024-04-16T02:35:30.206 INFO:tasks.workunit.client.1.smithi153.stdout:8/108: stat d2/d13/f1d 0 2024-04-16T02:35:30.207 INFO:tasks.workunit.client.1.smithi153.stdout:5/111: sync 2024-04-16T02:35:30.207 INFO:tasks.workunit.client.1.smithi153.stdout:8/109: readlink d2/ld 0 2024-04-16T02:35:30.207 INFO:tasks.workunit.client.0.smithi151.stdout:8/31: dwrite f5 [0,4194304] 0 2024-04-16T02:35:30.208 INFO:tasks.workunit.client.1.smithi153.stdout:9/152: dwrite d4/f27 [0,4194304] 0 2024-04-16T02:35:30.208 INFO:tasks.workunit.client.1.smithi153.stdout:3/138: rename d6/l34 to d6/d10/d22/l37 0 2024-04-16T02:35:30.210 INFO:tasks.workunit.client.0.smithi151.stdout:0/37: dwrite fa [0,4194304] 0 2024-04-16T02:35:30.210 INFO:tasks.workunit.client.1.smithi153.stdout:3/139: fdatasync d6/d10/d22/f31 0 2024-04-16T02:35:30.210 INFO:tasks.workunit.client.1.smithi153.stdout:9/153: dread - d4/d6/dc/f21 zero size 2024-04-16T02:35:30.210 INFO:tasks.workunit.client.1.smithi153.stdout:6/103: creat dc/d12/f24 x:0 0 0 2024-04-16T02:35:30.213 INFO:tasks.workunit.client.1.smithi153.stdout:0/161: mknod dd/d14/d19/d20/d2a/c2e 0 2024-04-16T02:35:30.213 INFO:tasks.workunit.client.1.smithi153.stdout:6/104: write f3 [5111854,58565] 0 2024-04-16T02:35:30.214 INFO:tasks.workunit.client.1.smithi153.stdout:3/140: truncate d6/d10/d22/f28 39321 0 2024-04-16T02:35:30.214 INFO:tasks.workunit.client.1.smithi153.stdout:6/105: write f3 [3673550,88692] 0 2024-04-16T02:35:30.215 INFO:tasks.workunit.client.1.smithi153.stdout:3/141: stat d6/c7 0 2024-04-16T02:35:30.216 INFO:tasks.workunit.client.0.smithi151.stdout:5/20: dwrite d2/f4 [0,4194304] 0 2024-04-16T02:35:30.218 INFO:tasks.workunit.client.0.smithi151.stdout:5/21: dread d2/f5 [0,4194304] 0 2024-04-16T02:35:30.220 INFO:tasks.workunit.client.1.smithi153.stdout:5/112: rename d8/c13 to d8/df/c19 0 2024-04-16T02:35:30.221 INFO:tasks.workunit.client.1.smithi153.stdout:2/130: getdents d9 0 2024-04-16T02:35:30.226 INFO:tasks.workunit.client.1.smithi153.stdout:6/106: mknod dc/d12/c25 0 2024-04-16T02:35:30.230 INFO:tasks.workunit.client.1.smithi153.stdout:3/142: unlink d6/l1d 0 2024-04-16T02:35:30.231 INFO:tasks.workunit.client.1.smithi153.stdout:2/131: dwrite d9/d24/d27/f1d [0,4194304] 0 2024-04-16T02:35:30.232 INFO:tasks.workunit.client.1.smithi153.stdout:2/132: stat d9/d24/d27/l1e 0 2024-04-16T02:35:30.235 INFO:tasks.workunit.client.1.smithi153.stdout:9/154: rename d4/d6/fb to d4/d6/dc/d24/f29 0 2024-04-16T02:35:30.236 INFO:tasks.workunit.client.1.smithi153.stdout:9/155: chown d4/f27 114654 1 2024-04-16T02:35:30.237 INFO:tasks.workunit.client.1.smithi153.stdout:3/143: write d6/d10/f32 [3526784,38600] 0 2024-04-16T02:35:30.238 INFO:tasks.workunit.client.1.smithi153.stdout:8/110: link d2/d1c/d1f/l21 d2/d1c/d1f/d27/l29 0 2024-04-16T02:35:30.240 INFO:tasks.workunit.client.1.smithi153.stdout:9/156: mkdir d4/d6/d1b/d2a 0 2024-04-16T02:35:30.243 INFO:tasks.workunit.client.1.smithi153.stdout:3/144: dread d6/d10/f18 [0,4194304] 0 2024-04-16T02:35:30.243 INFO:tasks.workunit.client.1.smithi153.stdout:0/162: creat dd/d14/d19/d1c/f2f x:0 0 0 2024-04-16T02:35:30.244 INFO:tasks.workunit.client.1.smithi153.stdout:3/145: read d6/d10/f32 [3934167,37917] 0 2024-04-16T02:35:30.244 INFO:tasks.workunit.client.1.smithi153.stdout:0/163: chown dd/d14/d19/f1b 2118955 1 2024-04-16T02:35:30.246 INFO:tasks.workunit.client.1.smithi153.stdout:8/111: creat d2/d1c/d1f/d27/f2a x:0 0 0 2024-04-16T02:35:30.248 INFO:tasks.workunit.client.1.smithi153.stdout:9/157: symlink d4/d6/d1b/d2a/l2b 0 2024-04-16T02:35:30.249 INFO:tasks.workunit.client.1.smithi153.stdout:3/146: mknod d6/d10/d22/d2e/c38 0 2024-04-16T02:35:30.249 INFO:tasks.workunit.client.1.smithi153.stdout:9/158: truncate d4/d6/dc/d24/f28 440591 0 2024-04-16T02:35:30.250 INFO:tasks.workunit.client.1.smithi153.stdout:3/147: read - d6/d10/d22/d2c/f33 zero size 2024-04-16T02:35:30.251 INFO:tasks.workunit.client.1.smithi153.stdout:8/112: chown d2/d1c/d1f/d27/l29 0 1 2024-04-16T02:35:30.252 INFO:tasks.workunit.client.1.smithi153.stdout:9/159: creat d4/d6/d1b/d2a/f2c x:0 0 0 2024-04-16T02:35:30.252 INFO:tasks.workunit.client.1.smithi153.stdout:3/148: creat d6/f39 x:0 0 0 2024-04-16T02:35:30.253 INFO:tasks.workunit.client.1.smithi153.stdout:8/113: mknod d2/d1c/d1f/d27/c2b 0 2024-04-16T02:35:30.254 INFO:tasks.workunit.client.1.smithi153.stdout:9/160: unlink d4/d6/d1b/d2a/f2c 0 2024-04-16T02:35:30.257 INFO:tasks.workunit.client.1.smithi153.stdout:9/161: mkdir d4/d6/dc/d2d 0 2024-04-16T02:35:30.261 INFO:tasks.workunit.client.1.smithi153.stdout:3/149: dwrite f1 [0,4194304] 0 2024-04-16T02:35:30.263 INFO:tasks.workunit.client.1.smithi153.stdout:8/114: dwrite d2/f7 [0,4194304] 0 2024-04-16T02:35:30.263 INFO:tasks.workunit.client.1.smithi153.stdout:3/150: dread - d6/d10/d22/f2a zero size 2024-04-16T02:35:30.265 INFO:tasks.workunit.client.1.smithi153.stdout:3/151: truncate d6/f2d 942745 0 2024-04-16T02:35:30.266 INFO:tasks.workunit.client.1.smithi153.stdout:8/115: mkdir d2/d13/d2c 0 2024-04-16T02:35:30.266 INFO:tasks.workunit.client.1.smithi153.stdout:3/152: read - d6/d10/d22/d2c/f33 zero size 2024-04-16T02:35:30.267 INFO:tasks.workunit.client.1.smithi153.stdout:3/153: readlink d6/d10/l29 0 2024-04-16T02:35:30.271 INFO:tasks.workunit.client.1.smithi153.stdout:8/116: mkdir d2/d1c/d1f/d27/d2d 0 2024-04-16T02:35:30.273 INFO:tasks.workunit.client.1.smithi153.stdout:3/154: symlink d6/d10/l3a 0 2024-04-16T02:35:30.274 INFO:tasks.workunit.client.1.smithi153.stdout:3/155: write d6/d10/f13 [799940,96821] 0 2024-04-16T02:35:30.274 INFO:tasks.workunit.client.0.smithi151.stdout:5/22: mkdir d2/d6 0 2024-04-16T02:35:30.275 INFO:tasks.workunit.client.1.smithi153.stdout:3/156: fsync d6/d10/f1e 0 2024-04-16T02:35:30.276 INFO:tasks.workunit.client.0.smithi151.stdout:0/38: mknod dc/de/c10 0 2024-04-16T02:35:30.276 INFO:tasks.workunit.client.1.smithi153.stdout:8/117: creat d2/d1c/d1f/f2e x:0 0 0 2024-04-16T02:35:30.277 INFO:tasks.workunit.client.1.smithi153.stdout:2/133: truncate f0 2178977 0 2024-04-16T02:35:30.279 INFO:tasks.workunit.client.1.smithi153.stdout:3/157: mkdir d6/d10/d3b 0 2024-04-16T02:35:30.279 INFO:tasks.workunit.client.0.smithi151.stdout:4/19: rmdir d0/d6 0 2024-04-16T02:35:30.280 INFO:tasks.workunit.client.0.smithi151.stdout:5/23: getdents d2/d6 0 2024-04-16T02:35:30.282 INFO:tasks.workunit.client.1.smithi153.stdout:2/134: creat d9/f33 x:0 0 0 2024-04-16T02:35:30.283 INFO:tasks.workunit.client.1.smithi153.stdout:2/135: readlink d9/d24/d27/l1e 0 2024-04-16T02:35:30.284 INFO:tasks.workunit.client.1.smithi153.stdout:0/164: write dd/d14/d19/f1a [436156,91263] 0 2024-04-16T02:35:30.284 INFO:tasks.workunit.client.0.smithi151.stdout:4/20: dread d0/f2 [0,4194304] 0 2024-04-16T02:35:30.286 INFO:tasks.workunit.client.0.smithi151.stdout:0/39: dwrite dc/ff [0,4194304] 0 2024-04-16T02:35:30.286 INFO:tasks.workunit.client.1.smithi153.stdout:3/158: dwrite d6/f11 [0,4194304] 0 2024-04-16T02:35:30.289 INFO:tasks.workunit.client.1.smithi153.stdout:8/118: getdents d2/d1c 0 2024-04-16T02:35:30.290 INFO:tasks.workunit.client.0.smithi151.stdout:4/21: mknod d0/c7 0 2024-04-16T02:35:30.291 INFO:tasks.workunit.client.0.smithi151.stdout:4/22: write d0/f3 [468605,6581] 0 2024-04-16T02:35:30.293 INFO:tasks.workunit.client.1.smithi153.stdout:3/159: symlink d6/d10/d22/l3c 0 2024-04-16T02:35:30.293 INFO:tasks.workunit.client.0.smithi151.stdout:0/40: rename f5 to dc/f11 0 2024-04-16T02:35:30.294 INFO:tasks.workunit.client.1.smithi153.stdout:9/162: truncate d4/fd 4011473 0 2024-04-16T02:35:30.297 INFO:tasks.workunit.client.1.smithi153.stdout:8/119: mknod d2/c2f 0 2024-04-16T02:35:30.297 INFO:tasks.workunit.client.0.smithi151.stdout:4/23: creat d0/f8 x:0 0 0 2024-04-16T02:35:30.298 INFO:tasks.workunit.client.1.smithi153.stdout:3/160: creat d6/d10/d22/d2c/f3d x:0 0 0 2024-04-16T02:35:30.298 INFO:tasks.workunit.client.0.smithi151.stdout:0/41: symlink dc/de/l12 0 2024-04-16T02:35:30.299 INFO:tasks.workunit.client.1.smithi153.stdout:3/161: dread - d6/f39 zero size 2024-04-16T02:35:30.300 INFO:tasks.workunit.client.0.smithi151.stdout:4/24: mknod d0/c9 0 2024-04-16T02:35:30.301 INFO:tasks.workunit.client.1.smithi153.stdout:9/163: rename d4/l1d to d4/d6/d1b/d2a/l2e 0 2024-04-16T02:35:30.302 INFO:tasks.workunit.client.1.smithi153.stdout:9/164: write d4/d6/f20 [459510,81000] 0 2024-04-16T02:35:30.303 INFO:tasks.workunit.client.0.smithi151.stdout:4/25: truncate d0/f2 123196 0 2024-04-16T02:35:30.304 INFO:tasks.workunit.client.1.smithi153.stdout:8/120: symlink d2/d1c/d1f/d27/l30 0 2024-04-16T02:35:30.306 INFO:tasks.workunit.client.1.smithi153.stdout:3/162: symlink d6/d10/d22/d2e/l3e 0 2024-04-16T02:35:30.307 INFO:tasks.workunit.client.1.smithi153.stdout:3/163: chown d6/d10/d22/d2c/f33 3133 1 2024-04-16T02:35:30.308 INFO:tasks.workunit.client.1.smithi153.stdout:3/164: read - d6/d10/d22/d2c/f33 zero size 2024-04-16T02:35:30.308 INFO:tasks.workunit.client.1.smithi153.stdout:9/165: creat d4/d6/d1b/f2f x:0 0 0 2024-04-16T02:35:30.311 INFO:tasks.workunit.client.1.smithi153.stdout:9/166: creat d4/d6/dc/d24/f30 x:0 0 0 2024-04-16T02:35:30.311 INFO:tasks.workunit.client.1.smithi153.stdout:8/121: creat d2/d1c/d1f/d27/d2d/f31 x:0 0 0 2024-04-16T02:35:30.312 INFO:tasks.workunit.client.1.smithi153.stdout:8/122: chown d2/d13/d2c 6 1 2024-04-16T02:35:30.314 INFO:tasks.workunit.client.1.smithi153.stdout:8/123: dread d2/ff [0,4194304] 0 2024-04-16T02:35:30.318 INFO:tasks.workunit.client.1.smithi153.stdout:8/124: creat d2/d13/d2c/f32 x:0 0 0 2024-04-16T02:35:30.319 INFO:tasks.workunit.client.1.smithi153.stdout:8/125: read - d2/de/f15 zero size 2024-04-16T02:35:30.320 INFO:tasks.workunit.client.1.smithi153.stdout:0/165: dwrite dd/d14/d19/d1c/f23 [4194304,4194304] 0 2024-04-16T02:35:30.320 INFO:tasks.workunit.client.1.smithi153.stdout:8/126: readlink d2/de/l22 0 2024-04-16T02:35:30.323 INFO:tasks.workunit.client.1.smithi153.stdout:8/127: symlink d2/d1c/l33 0 2024-04-16T02:35:30.324 INFO:tasks.workunit.client.1.smithi153.stdout:8/128: chown d2/d1c/d1f/l25 198 1 2024-04-16T02:35:30.324 INFO:tasks.workunit.client.1.smithi153.stdout:0/166: creat dd/d14/d19/d20/f30 x:0 0 0 2024-04-16T02:35:30.325 INFO:tasks.workunit.client.1.smithi153.stdout:0/167: truncate dd/f18 1929645 0 2024-04-16T02:35:30.325 INFO:tasks.workunit.client.1.smithi153.stdout:0/168: stat dd/f16 0 2024-04-16T02:35:30.327 INFO:tasks.workunit.client.1.smithi153.stdout:8/129: mkdir d2/de/d34 0 2024-04-16T02:35:30.327 INFO:tasks.workunit.client.1.smithi153.stdout:0/169: creat dd/d14/d19/d20/f31 x:0 0 0 2024-04-16T02:35:30.328 INFO:tasks.workunit.client.0.smithi151.stdout:8/32: write f5 [4339448,1713] 0 2024-04-16T02:35:30.330 INFO:tasks.workunit.client.1.smithi153.stdout:8/130: rmdir d2/d1c/d1f/d27/d2d 39 2024-04-16T02:35:30.330 INFO:tasks.workunit.client.1.smithi153.stdout:8/131: read - d2/d13/f1d zero size 2024-04-16T02:35:30.331 INFO:tasks.workunit.client.0.smithi151.stdout:2/20: dread f7 [0,4194304] 0 2024-04-16T02:35:30.331 INFO:tasks.workunit.client.0.smithi151.stdout:8/33: mknod c9 0 2024-04-16T02:35:30.332 INFO:tasks.workunit.client.1.smithi153.stdout:0/170: mkdir dd/d32 0 2024-04-16T02:35:30.333 INFO:tasks.workunit.client.0.smithi151.stdout:2/21: creat f8 x:0 0 0 2024-04-16T02:35:30.334 INFO:tasks.workunit.client.1.smithi153.stdout:8/132: mkdir d2/de/d34/d35 0 2024-04-16T02:35:30.335 INFO:tasks.workunit.client.0.smithi151.stdout:2/22: write f8 [382393,89313] 0 2024-04-16T02:35:30.335 INFO:tasks.workunit.client.1.smithi153.stdout:0/171: mkdir dd/d32/d33 0 2024-04-16T02:35:30.336 INFO:tasks.workunit.client.0.smithi151.stdout:2/23: chown f7 5637 1 2024-04-16T02:35:30.337 INFO:tasks.workunit.client.0.smithi151.stdout:5/24: truncate f1 392572 0 2024-04-16T02:35:30.339 INFO:tasks.workunit.client.1.smithi153.stdout:8/133: dread d2/f17 [0,4194304] 0 2024-04-16T02:35:30.339 INFO:tasks.workunit.client.0.smithi151.stdout:8/34: dwrite f3 [0,4194304] 0 2024-04-16T02:35:30.339 INFO:tasks.workunit.client.0.smithi151.stdout:2/24: symlink l9 0 2024-04-16T02:35:30.341 INFO:tasks.workunit.client.0.smithi151.stdout:0/42: rmdir dc 39 2024-04-16T02:35:30.343 INFO:tasks.workunit.client.1.smithi153.stdout:0/172: dwrite dd/f18 [0,4194304] 0 2024-04-16T02:35:30.343 INFO:tasks.workunit.client.1.smithi153.stdout:8/134: write d2/d1c/d1f/d27/d2d/f31 [537426,68942] 0 2024-04-16T02:35:30.344 INFO:tasks.workunit.client.0.smithi151.stdout:8/35: symlink la 0 2024-04-16T02:35:30.344 INFO:tasks.workunit.client.0.smithi151.stdout:8/36: readlink l8 0 2024-04-16T02:35:30.344 INFO:tasks.workunit.client.0.smithi151.stdout:4/26: dread d0/f2 [0,4194304] 0 2024-04-16T02:35:30.346 INFO:tasks.workunit.client.0.smithi151.stdout:0/43: dread - dc/fd zero size 2024-04-16T02:35:30.346 INFO:tasks.workunit.client.0.smithi151.stdout:4/27: truncate d0/f8 747550 0 2024-04-16T02:35:30.347 INFO:tasks.workunit.client.0.smithi151.stdout:8/37: rename f5 to fb 0 2024-04-16T02:35:30.350 INFO:tasks.workunit.client.0.smithi151.stdout:4/28: creat d0/fa x:0 0 0 2024-04-16T02:35:30.350 INFO:tasks.workunit.client.0.smithi151.stdout:0/44: dread dc/ff [0,4194304] 0 2024-04-16T02:35:30.352 INFO:tasks.workunit.client.0.smithi151.stdout:8/38: rename c6 to cc 0 2024-04-16T02:35:30.354 INFO:tasks.workunit.client.0.smithi151.stdout:4/29: rename d0/fa to d0/fb 0 2024-04-16T02:35:30.355 INFO:tasks.workunit.client.1.smithi153.stdout:0/173: dread dd/d14/d19/f1e [0,4194304] 0 2024-04-16T02:35:30.356 INFO:tasks.workunit.client.1.smithi153.stdout:8/135: dwrite d2/d13/f1b [0,4194304] 0 2024-04-16T02:35:30.356 INFO:tasks.workunit.client.0.smithi151.stdout:0/45: mknod dc/c13 0 2024-04-16T02:35:30.359 INFO:tasks.workunit.client.0.smithi151.stdout:2/25: write f7 [2240249,108969] 0 2024-04-16T02:35:30.360 INFO:tasks.workunit.client.0.smithi151.stdout:4/30: write d0/fb [292529,40701] 0 2024-04-16T02:35:30.360 INFO:tasks.workunit.client.0.smithi151.stdout:0/46: write dc/f11 [4588570,46872] 0 2024-04-16T02:35:30.361 INFO:tasks.workunit.client.0.smithi151.stdout:0/47: chown dc/de 530 1 2024-04-16T02:35:30.363 INFO:tasks.workunit.client.0.smithi151.stdout:2/26: mknod ca 0 2024-04-16T02:35:30.364 INFO:tasks.workunit.client.1.smithi153.stdout:0/174: getdents dd/d14/d19 0 2024-04-16T02:35:30.364 INFO:tasks.workunit.client.1.smithi153.stdout:8/136: getdents d2/de 0 2024-04-16T02:35:30.365 INFO:tasks.workunit.client.1.smithi153.stdout:8/137: dread - d2/d13/d2c/f32 zero size 2024-04-16T02:35:30.367 INFO:tasks.workunit.client.0.smithi151.stdout:0/48: rename dc/de/l12 to dc/l14 0 2024-04-16T02:35:30.367 INFO:tasks.workunit.client.1.smithi153.stdout:0/175: truncate dd/d14/d19/f1e 2625836 0 2024-04-16T02:35:30.368 INFO:tasks.workunit.client.0.smithi151.stdout:2/27: mknod cb 0 2024-04-16T02:35:30.368 INFO:tasks.workunit.client.1.smithi153.stdout:8/138: creat d2/d1c/f36 x:0 0 0 2024-04-16T02:35:30.369 INFO:tasks.workunit.client.0.smithi151.stdout:4/31: dwrite d0/f3 [0,4194304] 0 2024-04-16T02:35:30.369 INFO:tasks.workunit.client.1.smithi153.stdout:8/139: truncate d2/d1c/f36 826591 0 2024-04-16T02:35:30.374 INFO:tasks.workunit.client.1.smithi153.stdout:8/140: rename d2/de/d34/d35 to d2/d13/d37 0 2024-04-16T02:35:30.375 INFO:tasks.workunit.client.1.smithi153.stdout:0/176: dwrite dd/d14/d19/d1c/f2f [0,4194304] 0 2024-04-16T02:35:30.377 INFO:tasks.workunit.client.0.smithi151.stdout:2/28: dwrite f8 [0,4194304] 0 2024-04-16T02:35:30.378 INFO:tasks.workunit.client.0.smithi151.stdout:8/39: getdents . 0 2024-04-16T02:35:30.378 INFO:tasks.workunit.client.0.smithi151.stdout:4/32: dread d0/f3 [0,4194304] 0 2024-04-16T02:35:30.379 INFO:tasks.workunit.client.1.smithi153.stdout:0/177: write dd/f22 [1445441,2905] 0 2024-04-16T02:35:30.380 INFO:tasks.workunit.client.0.smithi151.stdout:2/29: write f8 [2830430,361] 0 2024-04-16T02:35:30.380 INFO:tasks.workunit.client.0.smithi151.stdout:5/25: link d2/f4 d2/f7 0 2024-04-16T02:35:30.381 INFO:tasks.workunit.client.0.smithi151.stdout:2/30: chown l9 2306109 1 2024-04-16T02:35:30.382 INFO:tasks.workunit.client.0.smithi151.stdout:2/31: fdatasync f7 0 2024-04-16T02:35:30.382 INFO:tasks.workunit.client.0.smithi151.stdout:2/32: readlink l9 0 2024-04-16T02:35:30.383 INFO:tasks.workunit.client.0.smithi151.stdout:2/33: write f7 [729115,35853] 0 2024-04-16T02:35:30.384 INFO:tasks.workunit.client.1.smithi153.stdout:1/137: sync 2024-04-16T02:35:30.384 INFO:tasks.workunit.client.1.smithi153.stdout:4/165: sync 2024-04-16T02:35:30.385 INFO:tasks.workunit.client.0.smithi151.stdout:8/40: symlink ld 0 2024-04-16T02:35:30.386 INFO:tasks.workunit.client.1.smithi153.stdout:1/138: stat d4/dd/f16 0 2024-04-16T02:35:30.387 INFO:tasks.workunit.client.1.smithi153.stdout:7/135: sync 2024-04-16T02:35:30.387 INFO:tasks.workunit.client.1.smithi153.stdout:8/141: dwrite d2/d13/d2c/f32 [0,4194304] 0 2024-04-16T02:35:30.390 INFO:tasks.workunit.client.0.smithi151.stdout:0/49: readlink dc/l14 0 2024-04-16T02:35:30.391 INFO:tasks.workunit.client.1.smithi153.stdout:7/136: write d0/d4/db/f12 [4295714,35857] 0 2024-04-16T02:35:30.391 INFO:tasks.workunit.client.1.smithi153.stdout:8/142: symlink d2/de/l38 0 2024-04-16T02:35:30.395 INFO:tasks.workunit.client.0.smithi151.stdout:4/33: dwrite d0/f8 [0,4194304] 0 2024-04-16T02:35:30.395 INFO:tasks.workunit.client.1.smithi153.stdout:8/143: stat d2/c28 0 2024-04-16T02:35:30.396 INFO:tasks.workunit.client.0.smithi151.stdout:5/26: creat d2/d6/f8 x:0 0 0 2024-04-16T02:35:30.400 INFO:tasks.workunit.client.0.smithi151.stdout:8/41: dwrite f2 [0,4194304] 0 2024-04-16T02:35:30.401 INFO:tasks.workunit.client.0.smithi151.stdout:2/34: link l5 lc 0 2024-04-16T02:35:30.401 INFO:tasks.workunit.client.0.smithi151.stdout:0/50: write f3 [1198764,40777] 0 2024-04-16T02:35:30.401 INFO:tasks.workunit.client.1.smithi153.stdout:1/139: link d4/dd/f15 d4/dd/d13/d26/f2b 0 2024-04-16T02:35:30.402 INFO:tasks.workunit.client.1.smithi153.stdout:4/166: dread d3/ff [4194304,4194304] 0 2024-04-16T02:35:30.402 INFO:tasks.workunit.client.0.smithi151.stdout:5/27: creat d2/d6/f9 x:0 0 0 2024-04-16T02:35:30.404 INFO:tasks.workunit.client.0.smithi151.stdout:8/42: write f2 [2368004,13191] 0 2024-04-16T02:35:30.405 INFO:tasks.workunit.client.1.smithi153.stdout:7/137: dwrite d0/d4/fc [0,4194304] 0 2024-04-16T02:35:30.406 INFO:tasks.workunit.client.1.smithi153.stdout:1/140: chown d4/dd/d13/d23/f27 105 1 2024-04-16T02:35:30.408 INFO:tasks.workunit.client.0.smithi151.stdout:0/51: rmdir dc 39 2024-04-16T02:35:30.408 INFO:tasks.workunit.client.1.smithi153.stdout:1/141: read d4/dd/d13/d26/f2b [436302,12531] 0 2024-04-16T02:35:30.409 INFO:tasks.workunit.client.0.smithi151.stdout:0/52: dread - f9 zero size 2024-04-16T02:35:30.411 INFO:tasks.workunit.client.1.smithi153.stdout:4/167: rmdir d3/d9/d24 39 2024-04-16T02:35:30.412 INFO:tasks.workunit.client.0.smithi151.stdout:4/34: dwrite d0/f8 [0,4194304] 0 2024-04-16T02:35:30.413 INFO:tasks.workunit.client.1.smithi153.stdout:8/144: link d2/d1c/l33 d2/d13/l39 0 2024-04-16T02:35:30.415 INFO:tasks.workunit.client.1.smithi153.stdout:7/138: creat d0/d4/db/d2b/f2f x:0 0 0 2024-04-16T02:35:30.415 INFO:tasks.workunit.client.1.smithi153.stdout:8/145: truncate d2/f12 2818369 0 2024-04-16T02:35:30.418 INFO:tasks.workunit.client.1.smithi153.stdout:7/139: chown d0/d4/db/d2b/f22 112 1 2024-04-16T02:35:30.418 INFO:tasks.workunit.client.0.smithi151.stdout:0/53: fdatasync dc/f11 0 2024-04-16T02:35:30.421 INFO:tasks.workunit.client.1.smithi153.stdout:1/142: mknod d4/dd/d20/c2c 0 2024-04-16T02:35:30.422 INFO:tasks.workunit.client.1.smithi153.stdout:1/143: fsync d4/f7 0 2024-04-16T02:35:30.423 INFO:tasks.workunit.client.0.smithi151.stdout:8/43: dwrite fb [0,4194304] 0 2024-04-16T02:35:30.424 INFO:tasks.workunit.client.1.smithi153.stdout:7/140: mkdir d0/d4/db/d16/d30 0 2024-04-16T02:35:30.425 INFO:tasks.workunit.client.0.smithi151.stdout:4/35: rename d0/l5 to d0/lc 0 2024-04-16T02:35:30.426 INFO:tasks.workunit.client.1.smithi153.stdout:7/141: fdatasync d0/d13/f1c 0 2024-04-16T02:35:30.426 INFO:tasks.workunit.client.0.smithi151.stdout:8/44: creat fe x:0 0 0 2024-04-16T02:35:30.426 INFO:tasks.workunit.client.0.smithi151.stdout:8/45: chown l8 15894 1 2024-04-16T02:35:30.427 INFO:tasks.workunit.client.0.smithi151.stdout:4/36: chown d0/f2 0 1 2024-04-16T02:35:30.427 INFO:tasks.workunit.client.1.smithi153.stdout:8/146: dwrite d2/d13/f1b [0,4194304] 0 2024-04-16T02:35:30.430 INFO:tasks.workunit.client.0.smithi151.stdout:4/37: creat d0/fd x:0 0 0 2024-04-16T02:35:30.432 INFO:tasks.workunit.client.1.smithi153.stdout:7/142: creat d0/d4/db/d16/d30/f31 x:0 0 0 2024-04-16T02:35:30.433 INFO:tasks.workunit.client.1.smithi153.stdout:8/147: rename d2/d13/d37 to d2/d13/d3a 0 2024-04-16T02:35:30.434 INFO:tasks.workunit.client.1.smithi153.stdout:8/148: readlink d2/de/l1e 0 2024-04-16T02:35:30.436 INFO:tasks.workunit.client.1.smithi153.stdout:8/149: dread - d2/d1c/d1f/f2e zero size 2024-04-16T02:35:30.436 INFO:tasks.workunit.client.0.smithi151.stdout:4/38: dread d0/fb [0,4194304] 0 2024-04-16T02:35:30.436 INFO:tasks.workunit.client.0.smithi151.stdout:2/35: getdents . 0 2024-04-16T02:35:30.436 INFO:tasks.workunit.client.1.smithi153.stdout:1/144: link d4/dd/d20/c2c d4/c2d 0 2024-04-16T02:35:30.437 INFO:tasks.workunit.client.0.smithi151.stdout:2/36: chown f8 0 1 2024-04-16T02:35:30.437 INFO:tasks.workunit.client.0.smithi151.stdout:4/39: chown d0/l1 4787 1 2024-04-16T02:35:30.437 INFO:tasks.workunit.client.1.smithi153.stdout:1/145: dread - d4/dd/d13/f2a zero size 2024-04-16T02:35:30.438 INFO:tasks.workunit.client.1.smithi153.stdout:1/146: read d4/f7 [3661394,94226] 0 2024-04-16T02:35:30.439 INFO:tasks.workunit.client.1.smithi153.stdout:7/143: dread d0/d4/fc [0,4194304] 0 2024-04-16T02:35:30.439 INFO:tasks.workunit.client.0.smithi151.stdout:2/37: rename cb to cd 0 2024-04-16T02:35:30.440 INFO:tasks.workunit.client.1.smithi153.stdout:8/150: getdents d2/de/d34 0 2024-04-16T02:35:30.440 INFO:tasks.workunit.client.0.smithi151.stdout:4/40: creat d0/fe x:0 0 0 2024-04-16T02:35:30.441 INFO:tasks.workunit.client.0.smithi151.stdout:2/38: mkdir de 0 2024-04-16T02:35:30.442 INFO:tasks.workunit.client.0.smithi151.stdout:2/39: fdatasync f8 0 2024-04-16T02:35:30.442 INFO:tasks.workunit.client.1.smithi153.stdout:7/144: creat d0/d13/f32 x:0 0 0 2024-04-16T02:35:30.442 INFO:tasks.workunit.client.0.smithi151.stdout:4/41: symlink d0/lf 0 2024-04-16T02:35:30.444 INFO:tasks.workunit.client.1.smithi153.stdout:7/145: dread - d0/d4/f20 zero size 2024-04-16T02:35:30.444 INFO:tasks.workunit.client.0.smithi151.stdout:2/40: chown l5 5941552 1 2024-04-16T02:35:30.444 INFO:tasks.workunit.client.0.smithi151.stdout:4/42: fsync d0/f2 0 2024-04-16T02:35:30.447 INFO:tasks.workunit.client.0.smithi151.stdout:4/43: dread d0/fb [0,4194304] 0 2024-04-16T02:35:30.447 INFO:tasks.workunit.client.0.smithi151.stdout:5/28: write d2/f5 [70003,89003] 0 2024-04-16T02:35:30.447 INFO:tasks.workunit.client.1.smithi153.stdout:7/146: fsync d0/d4/db/d26/f27 0 2024-04-16T02:35:30.447 INFO:tasks.workunit.client.1.smithi153.stdout:1/147: dwrite d4/f7 [0,4194304] 0 2024-04-16T02:35:30.449 INFO:tasks.workunit.client.1.smithi153.stdout:5/113: sync 2024-04-16T02:35:30.449 INFO:tasks.workunit.client.1.smithi153.stdout:7/147: stat d0/d4/db/d16/d30/f31 0 2024-04-16T02:35:30.450 INFO:tasks.workunit.client.1.smithi153.stdout:8/151: dwrite d2/f7 [4194304,4194304] 0 2024-04-16T02:35:30.450 INFO:tasks.workunit.client.0.smithi151.stdout:4/44: write d0/fb [1069265,99158] 0 2024-04-16T02:35:30.451 INFO:tasks.workunit.client.0.smithi151.stdout:5/29: creat d2/d6/fa x:0 0 0 2024-04-16T02:35:30.453 INFO:tasks.workunit.client.0.smithi151.stdout:2/41: dwrite f7 [0,4194304] 0 2024-04-16T02:35:30.453 INFO:tasks.workunit.client.1.smithi153.stdout:1/148: mknod d4/dd/d18/c2e 0 2024-04-16T02:35:30.454 INFO:tasks.workunit.client.0.smithi151.stdout:4/45: dread d0/f2 [0,4194304] 0 2024-04-16T02:35:30.457 INFO:tasks.workunit.client.0.smithi151.stdout:5/30: fsync d2/f5 0 2024-04-16T02:35:30.460 INFO:tasks.workunit.client.1.smithi153.stdout:0/178: truncate dd/d14/d19/f1e 2691508 0 2024-04-16T02:35:30.460 INFO:tasks.workunit.client.0.smithi151.stdout:5/31: dread d2/f5 [0,4194304] 0 2024-04-16T02:35:30.464 INFO:tasks.workunit.client.1.smithi153.stdout:5/114: symlink d8/l1a 0 2024-04-16T02:35:30.464 INFO:tasks.workunit.client.0.smithi151.stdout:2/42: link f8 de/ff 0 2024-04-16T02:35:30.464 INFO:tasks.workunit.client.0.smithi151.stdout:2/43: stat l9 0 2024-04-16T02:35:30.465 INFO:tasks.workunit.client.0.smithi151.stdout:2/44: chown lc 437262 1 2024-04-16T02:35:30.465 INFO:tasks.workunit.client.1.smithi153.stdout:5/115: dread - d8/df/f16 zero size 2024-04-16T02:35:30.467 INFO:tasks.workunit.client.1.smithi153.stdout:1/149: mknod d4/c2f 0 2024-04-16T02:35:30.468 INFO:tasks.workunit.client.0.smithi151.stdout:5/32: rename d2/f4 to d2/fb 0 2024-04-16T02:35:30.469 INFO:tasks.workunit.client.1.smithi153.stdout:7/148: getdents d0/d4/db 0 2024-04-16T02:35:30.469 INFO:tasks.workunit.client.1.smithi153.stdout:4/168: dwrite d3/ff [0,4194304] 0 2024-04-16T02:35:30.469 INFO:tasks.workunit.client.1.smithi153.stdout:5/116: write d8/df/f10 [4658724,70153] 0 2024-04-16T02:35:30.471 INFO:tasks.workunit.client.1.smithi153.stdout:5/117: stat d8/df/f11 0 2024-04-16T02:35:30.471 INFO:tasks.workunit.client.1.smithi153.stdout:5/118: dread - d8/df/f16 zero size 2024-04-16T02:35:30.474 INFO:tasks.workunit.client.0.smithi151.stdout:0/54: truncate dc/ff 2352860 0 2024-04-16T02:35:30.478 INFO:tasks.workunit.client.0.smithi151.stdout:2/45: dwrite de/ff [0,4194304] 0 2024-04-16T02:35:30.478 INFO:tasks.workunit.client.0.smithi151.stdout:0/55: rename dc/de to dc/d15 0 2024-04-16T02:35:30.478 INFO:tasks.workunit.client.1.smithi153.stdout:5/119: mknod d8/df/c1b 0 2024-04-16T02:35:30.478 INFO:tasks.workunit.client.1.smithi153.stdout:7/149: dwrite d0/d4/db/d2b/f22 [0,4194304] 0 2024-04-16T02:35:30.480 INFO:tasks.workunit.client.1.smithi153.stdout:5/120: truncate d8/df/f11 4840275 0 2024-04-16T02:35:30.480 INFO:tasks.workunit.client.1.smithi153.stdout:5/121: dread - d8/df/f16 zero size 2024-04-16T02:35:30.481 INFO:tasks.workunit.client.0.smithi151.stdout:0/56: dread f3 [0,4194304] 0 2024-04-16T02:35:30.481 INFO:tasks.workunit.client.1.smithi153.stdout:4/169: truncate d3/d9/d24/f2b 315302 0 2024-04-16T02:35:30.483 INFO:tasks.workunit.client.0.smithi151.stdout:0/57: symlink dc/l16 0 2024-04-16T02:35:30.483 INFO:tasks.workunit.client.1.smithi153.stdout:7/150: symlink d0/d4/db/d26/l33 0 2024-04-16T02:35:30.484 INFO:tasks.workunit.client.1.smithi153.stdout:1/150: getdents d4/dd/d18 0 2024-04-16T02:35:30.485 INFO:tasks.workunit.client.1.smithi153.stdout:2/136: sync 2024-04-16T02:35:30.485 INFO:tasks.workunit.client.1.smithi153.stdout:3/165: sync 2024-04-16T02:35:30.486 INFO:tasks.workunit.client.1.smithi153.stdout:9/167: sync 2024-04-16T02:35:30.486 INFO:tasks.workunit.client.1.smithi153.stdout:6/107: sync 2024-04-16T02:35:30.487 INFO:tasks.workunit.client.1.smithi153.stdout:7/151: mkdir d0/d4/db/d34 0 2024-04-16T02:35:30.487 INFO:tasks.workunit.client.0.smithi151.stdout:0/58: creat dc/d15/f17 x:0 0 0 2024-04-16T02:35:30.490 INFO:tasks.workunit.client.1.smithi153.stdout:0/179: write dd/fe [516747,64403] 0 2024-04-16T02:35:30.495 INFO:tasks.workunit.client.1.smithi153.stdout:0/180: mknod dd/d14/c34 0 2024-04-16T02:35:30.495 INFO:tasks.workunit.client.1.smithi153.stdout:2/137: chown d9/d2b/l32 0 1 2024-04-16T02:35:30.496 INFO:tasks.workunit.client.0.smithi151.stdout:4/46: truncate d0/f3 2928120 0 2024-04-16T02:35:30.497 INFO:tasks.workunit.client.0.smithi151.stdout:4/47: dread - d0/fe zero size 2024-04-16T02:35:30.497 INFO:tasks.workunit.client.1.smithi153.stdout:3/166: rmdir d6/d10/d22/d2e 39 2024-04-16T02:35:30.497 INFO:tasks.workunit.client.1.smithi153.stdout:6/108: creat dc/d1a/f26 x:0 0 0 2024-04-16T02:35:30.499 INFO:tasks.workunit.client.1.smithi153.stdout:6/109: fdatasync f3 0 2024-04-16T02:35:30.500 INFO:tasks.workunit.client.0.smithi151.stdout:4/48: link d0/f2 d0/f10 0 2024-04-16T02:35:30.500 INFO:tasks.workunit.client.0.smithi151.stdout:4/49: readlink d0/l1 0 2024-04-16T02:35:30.502 INFO:tasks.workunit.client.1.smithi153.stdout:2/138: mkdir d9/d24/d34 0 2024-04-16T02:35:30.505 INFO:tasks.workunit.client.1.smithi153.stdout:3/167: mknod d6/d10/c3f 0 2024-04-16T02:35:30.508 INFO:tasks.workunit.client.0.smithi151.stdout:5/33: fdatasync d2/fb 0 2024-04-16T02:35:30.509 INFO:tasks.workunit.client.1.smithi153.stdout:0/181: dread - dd/d14/d19/d1c/f2c zero size 2024-04-16T02:35:30.510 INFO:tasks.workunit.client.1.smithi153.stdout:8/152: dwrite d2/de/f24 [4194304,4194304] 0 2024-04-16T02:35:30.512 INFO:tasks.workunit.client.1.smithi153.stdout:4/170: dwrite d3/d9/f1a [0,4194304] 0 2024-04-16T02:35:30.512 INFO:tasks.workunit.client.1.smithi153.stdout:9/168: dwrite d4/d6/d1b/f2f [0,4194304] 0 2024-04-16T02:35:30.513 INFO:tasks.workunit.client.1.smithi153.stdout:1/151: dread d4/dd/f16 [4194304,4194304] 0 2024-04-16T02:35:30.514 INFO:tasks.workunit.client.1.smithi153.stdout:2/139: unlink d9/fc 0 2024-04-16T02:35:30.515 INFO:tasks.workunit.client.0.smithi151.stdout:5/34: unlink f1 0 2024-04-16T02:35:30.515 INFO:tasks.workunit.client.1.smithi153.stdout:5/122: write f4 [2091107,67033] 0 2024-04-16T02:35:30.516 INFO:tasks.workunit.client.1.smithi153.stdout:7/152: link d0/d4/c10 d0/d13/c35 0 2024-04-16T02:35:30.516 INFO:tasks.workunit.client.1.smithi153.stdout:8/153: chown d2/de/l1e 145958 1 2024-04-16T02:35:30.517 INFO:tasks.workunit.client.0.smithi151.stdout:5/35: truncate d2/d6/f9 622293 0 2024-04-16T02:35:30.521 INFO:tasks.workunit.client.0.smithi151.stdout:5/36: mknod d2/d6/cc 0 2024-04-16T02:35:30.525 INFO:tasks.workunit.client.1.smithi153.stdout:6/110: dread f6 [0,4194304] 0 2024-04-16T02:35:30.525 INFO:tasks.workunit.client.0.smithi151.stdout:5/37: creat d2/fd x:0 0 0 2024-04-16T02:35:30.526 INFO:tasks.workunit.client.1.smithi153.stdout:7/153: write d0/d4/db/d26/f27 [592919,81421] 0 2024-04-16T02:35:30.530 INFO:tasks.workunit.client.0.smithi151.stdout:4/50: dwrite d0/f2 [0,4194304] 0 2024-04-16T02:35:30.531 INFO:tasks.workunit.client.0.smithi151.stdout:4/51: fdatasync d0/f8 0 2024-04-16T02:35:30.531 INFO:tasks.workunit.client.1.smithi153.stdout:2/140: mknod d9/d24/d27/c35 0 2024-04-16T02:35:30.531 INFO:tasks.workunit.client.1.smithi153.stdout:5/123: mknod d8/df/c1c 0 2024-04-16T02:35:30.532 INFO:tasks.workunit.client.1.smithi153.stdout:4/171: fdatasync d3/f1c 0 2024-04-16T02:35:30.532 INFO:tasks.workunit.client.1.smithi153.stdout:1/152: symlink d4/l30 0 2024-04-16T02:35:30.534 INFO:tasks.workunit.client.0.smithi151.stdout:0/59: dwrite dc/ff [0,4194304] 0 2024-04-16T02:35:30.534 INFO:tasks.workunit.client.1.smithi153.stdout:7/154: unlink d0/d4/db/c2c 0 2024-04-16T02:35:30.537 INFO:tasks.workunit.client.0.smithi151.stdout:0/60: creat dc/d15/f18 x:0 0 0 2024-04-16T02:35:30.537 INFO:tasks.workunit.client.1.smithi153.stdout:1/153: unlink d4/l30 0 2024-04-16T02:35:30.538 INFO:tasks.workunit.client.0.smithi151.stdout:4/52: getdents d0 0 2024-04-16T02:35:30.538 INFO:tasks.workunit.client.0.smithi151.stdout:0/61: chown dc/ff 802 1 2024-04-16T02:35:30.540 INFO:tasks.workunit.client.1.smithi153.stdout:2/141: rmdir d9/d2b/d1c 39 2024-04-16T02:35:30.542 INFO:tasks.workunit.client.0.smithi151.stdout:4/53: mkdir d0/d11 0 2024-04-16T02:35:30.542 INFO:tasks.workunit.client.1.smithi153.stdout:8/154: dwrite d2/f12 [0,4194304] 0 2024-04-16T02:35:30.543 INFO:tasks.workunit.client.1.smithi153.stdout:9/169: dwrite d4/f23 [0,4194304] 0 2024-04-16T02:35:30.543 INFO:tasks.workunit.client.1.smithi153.stdout:0/182: dread dd/ff [0,4194304] 0 2024-04-16T02:35:30.544 INFO:tasks.workunit.client.0.smithi151.stdout:0/62: dread f3 [0,4194304] 0 2024-04-16T02:35:30.544 INFO:tasks.workunit.client.1.smithi153.stdout:5/124: write d8/df/f10 [4310145,58217] 0 2024-04-16T02:35:30.546 INFO:tasks.workunit.client.0.smithi151.stdout:0/63: write dc/d15/f18 [193193,22478] 0 2024-04-16T02:35:30.546 INFO:tasks.workunit.client.1.smithi153.stdout:7/155: chown d0/d4/db/d2b/c24 985 1 2024-04-16T02:35:30.546 INFO:tasks.workunit.client.1.smithi153.stdout:0/183: truncate f9 5012385 0 2024-04-16T02:35:30.547 INFO:tasks.workunit.client.1.smithi153.stdout:8/155: chown d2/de/d34 27355 1 2024-04-16T02:35:30.549 INFO:tasks.workunit.client.1.smithi153.stdout:1/154: mkdir d4/dd/d13/d23/d31 0 2024-04-16T02:35:30.549 INFO:tasks.workunit.client.0.smithi151.stdout:4/54: symlink d0/l12 0 2024-04-16T02:35:30.551 INFO:tasks.workunit.client.1.smithi153.stdout:3/168: dwrite d6/d10/d22/d2e/f35 [4194304,4194304] 0 2024-04-16T02:35:30.551 INFO:tasks.workunit.client.1.smithi153.stdout:8/156: chown d2/d13/d3a 7516413 1 2024-04-16T02:35:30.551 INFO:tasks.workunit.client.0.smithi151.stdout:4/55: write d0/fd [491096,112221] 0 2024-04-16T02:35:30.554 INFO:tasks.workunit.client.1.smithi153.stdout:8/157: chown d2/d1c/d1f/d27/d2d 558759792 1 2024-04-16T02:35:30.554 INFO:tasks.workunit.client.1.smithi153.stdout:5/125: dread f4 [0,4194304] 0 2024-04-16T02:35:30.556 INFO:tasks.workunit.client.1.smithi153.stdout:5/126: chown l2 23774966 1 2024-04-16T02:35:30.557 INFO:tasks.workunit.client.1.smithi153.stdout:0/184: creat dd/d32/f35 x:0 0 0 2024-04-16T02:35:30.560 INFO:tasks.workunit.client.0.smithi151.stdout:0/64: dwrite dc/d15/f18 [0,4194304] 0 2024-04-16T02:35:30.562 INFO:tasks.workunit.client.1.smithi153.stdout:8/158: chown d2/l3 14609 1 2024-04-16T02:35:30.563 INFO:tasks.workunit.client.1.smithi153.stdout:1/155: dread d4/f1f [0,4194304] 0 2024-04-16T02:35:30.564 INFO:tasks.workunit.client.1.smithi153.stdout:3/169: dread d6/d10/f1e [0,4194304] 0 2024-04-16T02:35:30.564 INFO:tasks.workunit.client.1.smithi153.stdout:4/172: mkdir d3/d9/db/de/d2c/d38 0 2024-04-16T02:35:30.565 INFO:tasks.workunit.client.1.smithi153.stdout:2/142: mkdir d9/d24/d34/d36 0 2024-04-16T02:35:30.567 INFO:tasks.workunit.client.0.smithi151.stdout:5/38: write d2/fb [446584,78279] 0 2024-04-16T02:35:30.567 INFO:tasks.workunit.client.1.smithi153.stdout:7/156: rename d0/d4/db/d16/d30/f31 to d0/f36 0 2024-04-16T02:35:30.569 INFO:tasks.workunit.client.1.smithi153.stdout:3/170: fdatasync d6/d10/d22/d2c/f3d 0 2024-04-16T02:35:30.570 INFO:tasks.workunit.client.0.smithi151.stdout:4/56: dwrite d0/fb [0,4194304] 0 2024-04-16T02:35:30.571 INFO:tasks.workunit.client.1.smithi153.stdout:5/127: mknod d8/df/c1d 0 2024-04-16T02:35:30.572 INFO:tasks.workunit.client.1.smithi153.stdout:0/185: rmdir dd/d14/d19 39 2024-04-16T02:35:30.573 INFO:tasks.workunit.client.1.smithi153.stdout:9/170: dwrite d4/d6/dc/d24/f25 [0,4194304] 0 2024-04-16T02:35:30.575 INFO:tasks.workunit.client.0.smithi151.stdout:0/65: dwrite dc/d15/f17 [0,4194304] 0 2024-04-16T02:35:30.583 INFO:tasks.workunit.client.1.smithi153.stdout:8/159: dwrite d2/f23 [4194304,4194304] 0 2024-04-16T02:35:30.584 INFO:tasks.workunit.client.1.smithi153.stdout:2/143: mknod d9/d2b/d18/c37 0 2024-04-16T02:35:30.590 INFO:tasks.workunit.client.0.smithi151.stdout:0/66: dwrite dc/f11 [0,4194304] 0 2024-04-16T02:35:30.595 INFO:tasks.workunit.client.0.smithi151.stdout:0/67: creat dc/d15/f19 x:0 0 0 2024-04-16T02:35:30.596 INFO:tasks.workunit.client.0.smithi151.stdout:5/39: chown d2/d6 9089479 1 2024-04-16T02:35:30.600 INFO:tasks.workunit.client.0.smithi151.stdout:5/40: dread - d2/d6/fa zero size 2024-04-16T02:35:30.601 INFO:tasks.workunit.client.1.smithi153.stdout:8/160: write d2/d13/d2c/f32 [2347440,84386] 0 2024-04-16T02:35:30.603 INFO:tasks.workunit.client.0.smithi151.stdout:5/41: symlink d2/d6/le 0 2024-04-16T02:35:30.604 INFO:tasks.workunit.client.1.smithi153.stdout:1/156: truncate d4/dd/d13/d26/f2b 1178537 0 2024-04-16T02:35:30.606 INFO:tasks.workunit.client.1.smithi153.stdout:5/128: dwrite d8/df/f11 [0,4194304] 0 2024-04-16T02:35:30.607 INFO:tasks.workunit.client.0.smithi151.stdout:5/42: unlink d2/d6/cc 0 2024-04-16T02:35:30.607 INFO:tasks.workunit.client.0.smithi151.stdout:5/43: dread - d2/d6/fa zero size 2024-04-16T02:35:30.611 INFO:tasks.workunit.client.1.smithi153.stdout:5/129: dread d8/df/f10 [0,4194304] 0 2024-04-16T02:35:30.613 INFO:tasks.workunit.client.0.smithi151.stdout:5/44: dread d2/f7 [0,4194304] 0 2024-04-16T02:35:30.613 INFO:tasks.workunit.client.0.smithi151.stdout:5/45: readlink d2/d6/le 0 2024-04-16T02:35:30.615 INFO:tasks.workunit.client.0.smithi151.stdout:5/46: fdatasync d2/f5 0 2024-04-16T02:35:30.616 INFO:tasks.workunit.client.0.smithi151.stdout:0/68: rmdir dc/d15 39 2024-04-16T02:35:30.620 INFO:tasks.workunit.client.0.smithi151.stdout:0/69: mknod dc/d15/c1a 0 2024-04-16T02:35:30.628 INFO:tasks.workunit.client.0.smithi151.stdout:5/47: dwrite d2/fd [0,4194304] 0 2024-04-16T02:35:30.629 INFO:tasks.workunit.client.1.smithi153.stdout:3/171: write d6/d10/f19 [1731936,44042] 0 2024-04-16T02:35:30.630 INFO:tasks.workunit.client.0.smithi151.stdout:0/70: dwrite fa [0,4194304] 0 2024-04-16T02:35:30.631 INFO:tasks.workunit.client.0.smithi151.stdout:0/71: truncate f9 992960 0 2024-04-16T02:35:30.633 INFO:tasks.workunit.client.0.smithi151.stdout:5/48: creat d2/ff x:0 0 0 2024-04-16T02:35:30.637 INFO:tasks.workunit.client.0.smithi151.stdout:0/72: symlink dc/d15/l1b 0 2024-04-16T02:35:30.637 INFO:tasks.workunit.client.0.smithi151.stdout:0/73: readlink dc/d15/l1b 0 2024-04-16T02:35:30.637 INFO:tasks.workunit.client.0.smithi151.stdout:5/49: unlink d2/d6/le 0 2024-04-16T02:35:30.639 INFO:tasks.workunit.client.0.smithi151.stdout:0/74: dread f3 [0,4194304] 0 2024-04-16T02:35:30.655 INFO:tasks.workunit.client.1.smithi153.stdout:6/111: sync 2024-04-16T02:35:30.661 INFO:tasks.workunit.client.1.smithi153.stdout:6/112: dwrite dc/d12/d15/f19 [0,4194304] 0 2024-04-16T02:35:30.777 INFO:tasks.workunit.client.0.smithi151.stdout:0/75: link dc/d15/l1b dc/d15/l1c 0 2024-04-16T02:35:30.778 INFO:tasks.workunit.client.0.smithi151.stdout:0/76: symlink dc/d15/l1d 0 2024-04-16T02:35:30.784 INFO:tasks.workunit.client.0.smithi151.stdout:0/77: rename dc/d15/c10 to dc/c1e 0 2024-04-16T02:35:30.785 INFO:tasks.workunit.client.0.smithi151.stdout:0/78: chown dc/d15/l1d 1 1 2024-04-16T02:35:30.785 INFO:tasks.workunit.client.0.smithi151.stdout:0/79: readlink dc/l14 0 2024-04-16T02:35:30.790 INFO:tasks.workunit.client.0.smithi151.stdout:0/80: dread dc/d15/f17 [0,4194304] 0 2024-04-16T02:35:30.791 INFO:tasks.workunit.client.0.smithi151.stdout:0/81: write dc/ff [1820523,116678] 0 2024-04-16T02:35:30.797 INFO:tasks.workunit.client.0.smithi151.stdout:0/82: truncate f9 722483 0 2024-04-16T02:35:30.801 INFO:tasks.workunit.client.0.smithi151.stdout:0/83: dread fa [0,4194304] 0 2024-04-16T02:35:30.802 INFO:tasks.workunit.client.0.smithi151.stdout:0/84: chown dc/d15/l1c 9206317 1 2024-04-16T02:35:30.806 INFO:tasks.workunit.client.0.smithi151.stdout:0/85: creat dc/f1f x:0 0 0 2024-04-16T02:35:30.808 INFO:tasks.workunit.client.0.smithi151.stdout:0/86: mkdir dc/d20 0 2024-04-16T02:35:30.810 INFO:tasks.workunit.client.0.smithi151.stdout:0/87: mknod dc/c21 0 2024-04-16T02:35:30.817 INFO:tasks.workunit.client.0.smithi151.stdout:0/88: getdents dc 0 2024-04-16T02:35:30.820 INFO:tasks.workunit.client.0.smithi151.stdout:0/89: unlink f9 0 2024-04-16T02:35:30.823 INFO:tasks.workunit.client.0.smithi151.stdout:0/90: link dc/d15/l1d dc/d15/l22 0 2024-04-16T02:35:30.825 INFO:tasks.workunit.client.0.smithi151.stdout:0/91: creat dc/d15/f23 x:0 0 0 2024-04-16T02:35:30.825 INFO:tasks.workunit.client.0.smithi151.stdout:0/92: dread - dc/f1f zero size 2024-04-16T02:35:30.827 INFO:tasks.workunit.client.0.smithi151.stdout:0/93: mknod dc/c24 0 2024-04-16T02:35:30.829 INFO:tasks.workunit.client.0.smithi151.stdout:0/94: unlink dc/d15/f18 0 2024-04-16T02:35:30.831 INFO:tasks.workunit.client.0.smithi151.stdout:0/95: unlink dc/d15/l1c 0 2024-04-16T02:35:30.835 INFO:tasks.workunit.client.0.smithi151.stdout:0/96: dread dc/f11 [0,4194304] 0 2024-04-16T02:35:30.837 INFO:tasks.workunit.client.0.smithi151.stdout:0/97: creat dc/d15/f25 x:0 0 0 2024-04-16T02:35:30.839 INFO:tasks.workunit.client.0.smithi151.stdout:0/98: write dc/ff [3197561,11166] 0 2024-04-16T02:35:30.841 INFO:tasks.workunit.client.0.smithi151.stdout:0/99: write dc/f11 [1320900,123169] 0 2024-04-16T02:35:30.842 INFO:tasks.workunit.client.0.smithi151.stdout:0/100: chown dc/fd 214057 1 2024-04-16T02:35:30.845 INFO:tasks.workunit.client.0.smithi151.stdout:0/101: getdents dc/d20 0 2024-04-16T02:35:30.856 INFO:tasks.workunit.client.0.smithi151.stdout:0/102: rmdir dc/d15 39 2024-04-16T02:35:30.858 INFO:tasks.workunit.client.0.smithi151.stdout:0/103: fdatasync dc/d15/f19 0 2024-04-16T02:35:30.858 INFO:tasks.workunit.client.0.smithi151.stdout:0/104: write dc/ff [3090579,30931] 0 2024-04-16T02:35:30.860 INFO:tasks.workunit.client.0.smithi151.stdout:0/105: truncate dc/d15/f19 791072 0 2024-04-16T02:35:30.862 INFO:tasks.workunit.client.0.smithi151.stdout:0/106: chown dc/c13 179 1 2024-04-16T02:35:30.863 INFO:tasks.workunit.client.0.smithi151.stdout:0/107: chown dc/d15/l1b 1830 1 2024-04-16T02:35:30.864 INFO:tasks.workunit.client.0.smithi151.stdout:0/108: dread - dc/d15/f23 zero size 2024-04-16T02:35:30.864 INFO:tasks.workunit.client.1.smithi153.stdout:0/186: unlink dd/d14/d19/d20/f30 0 2024-04-16T02:35:30.865 INFO:tasks.workunit.client.1.smithi153.stdout:9/171: symlink d4/l31 0 2024-04-16T02:35:30.866 INFO:tasks.workunit.client.1.smithi153.stdout:8/161: symlink d2/de/d34/l3b 0 2024-04-16T02:35:30.866 INFO:tasks.workunit.client.0.smithi151.stdout:0/109: symlink dc/d20/l26 0 2024-04-16T02:35:30.867 INFO:tasks.workunit.client.0.smithi151.stdout:0/110: chown dc/c24 2473947 1 2024-04-16T02:35:30.867 INFO:tasks.workunit.client.1.smithi153.stdout:1/157: mknod d4/dd/d13/d23/c32 0 2024-04-16T02:35:30.867 INFO:tasks.workunit.client.1.smithi153.stdout:1/158: stat d4/dd/d18/l1e 0 2024-04-16T02:35:30.868 INFO:tasks.workunit.client.1.smithi153.stdout:5/130: creat d8/df/f1e x:0 0 0 2024-04-16T02:35:30.869 INFO:tasks.workunit.client.1.smithi153.stdout:5/131: write d8/df/f16 [745095,31372] 0 2024-04-16T02:35:30.871 INFO:tasks.workunit.client.1.smithi153.stdout:6/113: symlink dc/d10/l27 0 2024-04-16T02:35:30.874 INFO:tasks.workunit.client.0.smithi151.stdout:0/111: dwrite dc/d15/f17 [0,4194304] 0 2024-04-16T02:35:30.875 INFO:tasks.workunit.client.1.smithi153.stdout:8/162: creat d2/d1c/d1f/f3c x:0 0 0 2024-04-16T02:35:30.875 INFO:tasks.workunit.client.1.smithi153.stdout:6/114: truncate dc/d12/d15/f1e 548720 0 2024-04-16T02:35:30.876 INFO:tasks.workunit.client.0.smithi151.stdout:0/112: dread - dc/f1f zero size 2024-04-16T02:35:30.877 INFO:tasks.workunit.client.1.smithi153.stdout:8/163: rename d2 to d2/d1c/d3d 22 2024-04-16T02:35:30.878 INFO:tasks.workunit.client.1.smithi153.stdout:6/115: chown dc/d10/c13 977 1 2024-04-16T02:35:30.879 INFO:tasks.workunit.client.1.smithi153.stdout:7/157: dwrite d0/d4/db/d2b/f2f [0,4194304] 0 2024-04-16T02:35:30.881 INFO:tasks.workunit.client.1.smithi153.stdout:5/132: dwrite d8/df/f1e [0,4194304] 0 2024-04-16T02:35:30.882 INFO:tasks.workunit.client.1.smithi153.stdout:6/116: chown dc/d10/l27 173883425 1 2024-04-16T02:35:30.882 INFO:tasks.workunit.client.1.smithi153.stdout:4/173: link d3/d9/db/de/l1f d3/d9/db/d17/d28/l39 0 2024-04-16T02:35:30.884 INFO:tasks.workunit.client.1.smithi153.stdout:7/158: write d0/d4/db/f12 [2157758,31115] 0 2024-04-16T02:35:30.887 INFO:tasks.workunit.client.0.smithi151.stdout:0/113: dwrite f3 [0,4194304] 0 2024-04-16T02:35:30.887 INFO:tasks.workunit.client.1.smithi153.stdout:1/159: link d4/f1f d4/dd/d13/d26/f33 0 2024-04-16T02:35:30.889 INFO:tasks.workunit.client.1.smithi153.stdout:8/164: creat d2/de/f3e x:0 0 0 2024-04-16T02:35:30.889 INFO:tasks.workunit.client.1.smithi153.stdout:8/165: readlink d2/de/l22 0 2024-04-16T02:35:30.890 INFO:tasks.workunit.client.1.smithi153.stdout:8/166: chown d2/d13 333470 1 2024-04-16T02:35:30.893 INFO:tasks.workunit.client.1.smithi153.stdout:0/187: creat dd/d14/d19/d1c/f36 x:0 0 0 2024-04-16T02:35:30.896 INFO:tasks.workunit.client.0.smithi151.stdout:0/114: rename dc/fd to dc/f27 0 2024-04-16T02:35:30.897 INFO:tasks.workunit.client.0.smithi151.stdout:0/115: write dc/f1f [776355,89103] 0 2024-04-16T02:35:30.899 INFO:tasks.workunit.client.1.smithi153.stdout:8/167: dwrite d2/d13/f1d [0,4194304] 0 2024-04-16T02:35:30.899 INFO:tasks.workunit.client.0.smithi151.stdout:0/116: write dc/d15/f19 [508759,90277] 0 2024-04-16T02:35:30.899 INFO:tasks.workunit.client.1.smithi153.stdout:6/117: rename dc/l16 to dc/l28 0 2024-04-16T02:35:30.901 INFO:tasks.workunit.client.0.smithi151.stdout:0/117: chown dc/l16 1 1 2024-04-16T02:35:30.901 INFO:tasks.workunit.client.0.smithi151.stdout:0/118: dread - dc/d15/f23 zero size 2024-04-16T02:35:30.902 INFO:tasks.workunit.client.1.smithi153.stdout:4/174: creat d3/d9/d24/f3a x:0 0 0 2024-04-16T02:35:30.903 INFO:tasks.workunit.client.0.smithi151.stdout:0/119: readlink dc/d15/l1d 0 2024-04-16T02:35:30.903 INFO:tasks.workunit.client.1.smithi153.stdout:7/159: dread - d0/d13/f1c zero size 2024-04-16T02:35:30.905 INFO:tasks.workunit.client.1.smithi153.stdout:4/175: write d3/d9/d24/f3a [926707,42162] 0 2024-04-16T02:35:30.908 INFO:tasks.workunit.client.1.smithi153.stdout:5/133: mkdir d8/d1f 0 2024-04-16T02:35:30.908 INFO:tasks.workunit.client.1.smithi153.stdout:4/176: write d3/d9/db/d17/f21 [1309178,90345] 0 2024-04-16T02:35:30.909 INFO:tasks.workunit.client.1.smithi153.stdout:8/168: mkdir d2/de/d3f 0 2024-04-16T02:35:30.910 INFO:tasks.workunit.client.1.smithi153.stdout:7/160: dread d0/d4/db/f2a [0,4194304] 0 2024-04-16T02:35:30.911 INFO:tasks.workunit.client.1.smithi153.stdout:1/160: truncate d4/dd/f15 312558 0 2024-04-16T02:35:30.911 INFO:tasks.workunit.client.1.smithi153.stdout:7/161: write d0/d4/db/d2b/f2f [2833246,41729] 0 2024-04-16T02:35:30.914 INFO:tasks.workunit.client.0.smithi151.stdout:0/120: dwrite f3 [4194304,4194304] 0 2024-04-16T02:35:30.916 INFO:tasks.workunit.client.1.smithi153.stdout:5/134: fdatasync f4 0 2024-04-16T02:35:30.917 INFO:tasks.workunit.client.0.smithi151.stdout:0/121: fdatasync dc/f27 0 2024-04-16T02:35:30.917 INFO:tasks.workunit.client.1.smithi153.stdout:4/177: creat d3/d9/db/de/f3b x:0 0 0 2024-04-16T02:35:30.920 INFO:tasks.workunit.client.0.smithi151.stdout:0/122: creat dc/f28 x:0 0 0 2024-04-16T02:35:30.922 INFO:tasks.workunit.client.0.smithi151.stdout:0/123: creat dc/d15/f29 x:0 0 0 2024-04-16T02:35:30.924 INFO:tasks.workunit.client.1.smithi153.stdout:5/135: dwrite d8/df/f1e [0,4194304] 0 2024-04-16T02:35:30.928 INFO:tasks.workunit.client.1.smithi153.stdout:0/188: rename dd/d14/d19/d1c/f2c to dd/d14/f37 0 2024-04-16T02:35:30.929 INFO:tasks.workunit.client.1.smithi153.stdout:0/189: read - dd/d14/d19/f27 zero size 2024-04-16T02:35:30.934 INFO:tasks.workunit.client.1.smithi153.stdout:6/118: rename dc/d12/c25 to dc/d12/d15/c29 0 2024-04-16T02:35:30.934 INFO:tasks.workunit.client.1.smithi153.stdout:6/119: fsync f4 0 2024-04-16T02:35:30.935 INFO:tasks.workunit.client.0.smithi151.stdout:0/124: rmdir dc/d15 39 2024-04-16T02:35:30.935 INFO:tasks.workunit.client.1.smithi153.stdout:6/120: write dc/f17 [86709,119578] 0 2024-04-16T02:35:30.937 INFO:tasks.workunit.client.1.smithi153.stdout:5/136: mkdir d8/d1f/d20 0 2024-04-16T02:35:30.937 INFO:tasks.workunit.client.1.smithi153.stdout:6/121: write dc/d12/d15/f1e [223803,74] 0 2024-04-16T02:35:30.938 INFO:tasks.workunit.client.1.smithi153.stdout:6/122: truncate dc/f11 4264968 0 2024-04-16T02:35:30.939 INFO:tasks.workunit.client.0.smithi151.stdout:0/125: symlink dc/d15/l2a 0 2024-04-16T02:35:30.939 INFO:tasks.workunit.client.0.smithi151.stdout:0/126: fdatasync dc/f28 0 2024-04-16T02:35:30.939 INFO:tasks.workunit.client.1.smithi153.stdout:0/190: symlink dd/d26/l38 0 2024-04-16T02:35:30.940 INFO:tasks.workunit.client.1.smithi153.stdout:0/191: stat dd/d26 0 2024-04-16T02:35:30.940 INFO:tasks.workunit.client.1.smithi153.stdout:0/192: chown dd/d14/c34 2 1 2024-04-16T02:35:30.941 INFO:tasks.workunit.client.1.smithi153.stdout:7/162: rename d0/d4/db/d26/l33 to d0/d13/l37 0 2024-04-16T02:35:30.945 INFO:tasks.workunit.client.1.smithi153.stdout:4/178: link d3/c27 d3/d9/db/de/c3c 0 2024-04-16T02:35:30.946 INFO:tasks.workunit.client.0.smithi151.stdout:0/127: dwrite dc/d15/f17 [0,4194304] 0 2024-04-16T02:35:30.946 INFO:tasks.workunit.client.1.smithi153.stdout:6/123: mknod dc/d10/c2a 0 2024-04-16T02:35:30.948 INFO:tasks.workunit.client.0.smithi151.stdout:0/128: creat dc/d15/f2b x:0 0 0 2024-04-16T02:35:30.949 INFO:tasks.workunit.client.1.smithi153.stdout:5/137: dwrite d8/df/f11 [0,4194304] 0 2024-04-16T02:35:30.952 INFO:tasks.workunit.client.0.smithi151.stdout:0/129: dread fa [0,4194304] 0 2024-04-16T02:35:30.954 INFO:tasks.workunit.client.0.smithi151.stdout:0/130: mkdir dc/d20/d2c 0 2024-04-16T02:35:30.956 INFO:tasks.workunit.client.1.smithi153.stdout:7/163: rmdir d0/d4/db/d16/d30 0 2024-04-16T02:35:30.957 INFO:tasks.workunit.client.0.smithi151.stdout:0/131: creat dc/f2d x:0 0 0 2024-04-16T02:35:30.957 INFO:tasks.workunit.client.0.smithi151.stdout:0/132: stat dc/c13 0 2024-04-16T02:35:30.959 INFO:tasks.workunit.client.0.smithi151.stdout:0/133: write f3 [3068740,58367] 0 2024-04-16T02:35:30.963 INFO:tasks.workunit.client.0.smithi151.stdout:0/134: creat dc/d20/f2e x:0 0 0 2024-04-16T02:35:30.964 INFO:tasks.workunit.client.1.smithi153.stdout:7/164: dwrite d0/d4/f20 [0,4194304] 0 2024-04-16T02:35:30.967 INFO:tasks.workunit.client.1.smithi153.stdout:7/165: creat d0/d4/db/d2b/f38 x:0 0 0 2024-04-16T02:35:30.972 INFO:tasks.workunit.client.1.smithi153.stdout:7/166: dread d0/d4/db/d2b/f22 [0,4194304] 0 2024-04-16T02:35:30.973 INFO:tasks.workunit.client.1.smithi153.stdout:7/167: chown d0/d4/db/d16/c1a 248 1 2024-04-16T02:35:30.976 INFO:tasks.workunit.client.1.smithi153.stdout:7/168: creat d0/d4/db/f39 x:0 0 0 2024-04-16T02:35:30.976 INFO:tasks.workunit.client.0.smithi151.stdout:0/135: rmdir dc 39 2024-04-16T02:35:30.977 INFO:tasks.workunit.client.0.smithi151.stdout:0/136: fsync dc/f27 0 2024-04-16T02:35:30.978 INFO:tasks.workunit.client.1.smithi153.stdout:7/169: symlink d0/d4/db/d34/l3a 0 2024-04-16T02:35:30.979 INFO:tasks.workunit.client.1.smithi153.stdout:7/170: fdatasync d0/d4/db/d26/f27 0 2024-04-16T02:35:30.981 INFO:tasks.workunit.client.0.smithi151.stdout:0/137: rename dc/d20/f2e to dc/f2f 0 2024-04-16T02:35:30.984 INFO:tasks.workunit.client.0.smithi151.stdout:0/138: creat dc/d20/d2c/f30 x:0 0 0 2024-04-16T02:35:30.987 INFO:tasks.workunit.client.0.smithi151.stdout:0/139: rename dc/d15/l1d to dc/d15/l31 0 2024-04-16T02:35:30.991 INFO:tasks.workunit.client.1.smithi153.stdout:7/171: dread d0/fa [0,4194304] 0 2024-04-16T02:35:30.994 INFO:tasks.workunit.client.0.smithi151.stdout:0/140: dwrite dc/f27 [0,4194304] 0 2024-04-16T02:35:30.995 INFO:tasks.workunit.client.0.smithi151.stdout:0/141: fdatasync dc/d20/d2c/f30 0 2024-04-16T02:35:30.996 INFO:tasks.workunit.client.0.smithi151.stdout:0/142: write dc/d15/f23 [102227,55602] 0 2024-04-16T02:35:30.998 INFO:tasks.workunit.client.0.smithi151.stdout:0/143: write dc/f11 [4005240,50340] 0 2024-04-16T02:35:31.002 INFO:tasks.workunit.client.0.smithi151.stdout:0/144: dread dc/d15/f17 [0,4194304] 0 2024-04-16T02:35:31.005 INFO:tasks.workunit.client.0.smithi151.stdout:0/145: mkdir dc/d20/d2c/d32 0 2024-04-16T02:35:31.008 INFO:tasks.workunit.client.1.smithi153.stdout:2/144: truncate d9/d2b/f17 537603 0 2024-04-16T02:35:31.009 INFO:tasks.workunit.client.0.smithi151.stdout:0/146: rmdir dc/d20/d2c/d32 0 2024-04-16T02:35:31.010 INFO:tasks.workunit.client.0.smithi151.stdout:0/147: write dc/d15/f19 [576109,27070] 0 2024-04-16T02:35:31.011 INFO:tasks.workunit.client.1.smithi153.stdout:3/172: truncate f1 124931 0 2024-04-16T02:35:31.011 INFO:tasks.workunit.client.1.smithi153.stdout:3/173: truncate d6/f39 633163 0 2024-04-16T02:35:31.012 INFO:tasks.workunit.client.1.smithi153.stdout:9/172: truncate d4/fd 1662860 0 2024-04-16T02:35:31.013 INFO:tasks.workunit.client.1.smithi153.stdout:9/173: write d4/d6/f20 [2765033,42453] 0 2024-04-16T02:35:31.015 INFO:tasks.workunit.client.1.smithi153.stdout:9/174: truncate d4/d6/f22 202585 0 2024-04-16T02:35:31.017 INFO:tasks.workunit.client.1.smithi153.stdout:2/145: link d9/d2b/l11 d9/d24/d27/l38 0 2024-04-16T02:35:31.017 INFO:tasks.workunit.client.0.smithi151.stdout:0/148: dwrite dc/d15/f17 [0,4194304] 0 2024-04-16T02:35:31.018 INFO:tasks.workunit.client.1.smithi153.stdout:3/174: dwrite d6/d10/f27 [0,4194304] 0 2024-04-16T02:35:31.018 INFO:tasks.workunit.client.0.smithi151.stdout:0/149: fsync dc/d15/f29 0 2024-04-16T02:35:31.021 INFO:tasks.workunit.client.1.smithi153.stdout:2/146: unlink d9/d2b/l32 0 2024-04-16T02:35:31.030 INFO:tasks.workunit.client.1.smithi153.stdout:1/161: unlink d4/dd/f15 0 2024-04-16T02:35:31.031 INFO:tasks.workunit.client.1.smithi153.stdout:3/175: getdents d6/d10/d22 0 2024-04-16T02:35:31.032 INFO:tasks.workunit.client.1.smithi153.stdout:1/162: write d4/dd/d13/d23/f27 [44393,119454] 0 2024-04-16T02:35:31.034 INFO:tasks.workunit.client.1.smithi153.stdout:3/176: symlink d6/d10/d22/d2c/l40 0 2024-04-16T02:35:31.035 INFO:tasks.workunit.client.1.smithi153.stdout:8/169: truncate d2/f17 1827970 0 2024-04-16T02:35:31.038 INFO:tasks.workunit.client.1.smithi153.stdout:3/177: creat d6/d10/d3b/f41 x:0 0 0 2024-04-16T02:35:31.040 INFO:tasks.workunit.client.1.smithi153.stdout:3/178: write d6/d10/f32 [3065477,125642] 0 2024-04-16T02:35:31.040 INFO:tasks.workunit.client.1.smithi153.stdout:3/179: chown d6/d10/d3b 286729230 1 2024-04-16T02:35:31.041 INFO:tasks.workunit.client.1.smithi153.stdout:1/163: getdents d4/dd/d13/d23 0 2024-04-16T02:35:31.042 INFO:tasks.workunit.client.1.smithi153.stdout:0/193: truncate f5 3997375 0 2024-04-16T02:35:31.044 INFO:tasks.workunit.client.1.smithi153.stdout:1/164: creat d4/dd/d13/d26/f34 x:0 0 0 2024-04-16T02:35:31.045 INFO:tasks.workunit.client.1.smithi153.stdout:5/138: truncate d8/df/f10 230941 0 2024-04-16T02:35:31.047 INFO:tasks.workunit.client.1.smithi153.stdout:7/172: fsync d0/d4/db/d2b/f38 0 2024-04-16T02:35:31.049 INFO:tasks.workunit.client.1.smithi153.stdout:1/165: write d4/dd/d13/d23/f27 [401171,88116] 0 2024-04-16T02:35:31.051 INFO:tasks.workunit.client.1.smithi153.stdout:3/180: dwrite d6/d10/d22/f2a [0,4194304] 0 2024-04-16T02:35:31.053 INFO:tasks.workunit.client.1.smithi153.stdout:0/194: dread dd/f18 [0,4194304] 0 2024-04-16T02:35:31.056 INFO:tasks.workunit.client.1.smithi153.stdout:9/175: rename d4/fd to d4/d6/d1b/d2a/f32 0 2024-04-16T02:35:31.059 INFO:tasks.workunit.client.1.smithi153.stdout:4/179: dwrite d3/d9/d24/f2b [0,4194304] 0 2024-04-16T02:35:31.062 INFO:tasks.workunit.client.1.smithi153.stdout:0/195: dread dd/f22 [0,4194304] 0 2024-04-16T02:35:31.063 INFO:tasks.workunit.client.1.smithi153.stdout:9/176: mkdir d4/d33 0 2024-04-16T02:35:31.069 INFO:tasks.workunit.client.1.smithi153.stdout:2/147: mknod d9/d24/d27/c39 0 2024-04-16T02:35:31.071 INFO:tasks.workunit.client.1.smithi153.stdout:0/196: creat dd/d32/f39 x:0 0 0 2024-04-16T02:35:31.072 INFO:tasks.workunit.client.1.smithi153.stdout:9/177: write d4/d6/dc/d24/f29 [74736,66151] 0 2024-04-16T02:35:31.077 INFO:tasks.workunit.client.1.smithi153.stdout:2/148: rmdir d9/d24 39 2024-04-16T02:35:31.078 INFO:tasks.workunit.client.1.smithi153.stdout:5/139: dwrite d8/df/f10 [0,4194304] 0 2024-04-16T02:35:31.081 INFO:tasks.workunit.client.1.smithi153.stdout:4/180: rename d3/d9/db/d17/f21 to d3/d9/f3d 0 2024-04-16T02:35:31.084 INFO:tasks.workunit.client.1.smithi153.stdout:0/197: creat dd/d32/d33/f3a x:0 0 0 2024-04-16T02:35:31.088 INFO:tasks.workunit.client.1.smithi153.stdout:9/178: symlink d4/d6/l34 0 2024-04-16T02:35:31.091 INFO:tasks.workunit.client.1.smithi153.stdout:7/173: dwrite d0/fa [0,4194304] 0 2024-04-16T02:35:31.091 INFO:tasks.workunit.client.1.smithi153.stdout:5/140: dwrite d8/df/f10 [0,4194304] 0 2024-04-16T02:35:31.091 INFO:tasks.workunit.client.1.smithi153.stdout:4/181: write d3/d9/f3d [1255239,11028] 0 2024-04-16T02:35:31.092 INFO:tasks.workunit.client.1.smithi153.stdout:5/141: chown l6 29860 1 2024-04-16T02:35:31.094 INFO:tasks.workunit.client.1.smithi153.stdout:4/182: fdatasync d3/d9/d24/f3a 0 2024-04-16T02:35:31.098 INFO:tasks.workunit.client.1.smithi153.stdout:9/179: readlink d4/d6/d1b/d2a/l2e 0 2024-04-16T02:35:31.098 INFO:tasks.workunit.client.1.smithi153.stdout:7/174: mknod d0/d4/db/d16/c3b 0 2024-04-16T02:35:31.099 INFO:tasks.workunit.client.1.smithi153.stdout:5/142: creat d8/d1f/f21 x:0 0 0 2024-04-16T02:35:31.100 INFO:tasks.workunit.client.1.smithi153.stdout:9/180: chown d4/f13 0 1 2024-04-16T02:35:31.102 INFO:tasks.workunit.client.1.smithi153.stdout:7/175: write d0/d4/db/d2b/f2f [1464784,124399] 0 2024-04-16T02:35:31.102 INFO:tasks.workunit.client.1.smithi153.stdout:2/149: rename d9/d2b/d1c/l2f to d9/d2b/d1c/l3a 0 2024-04-16T02:35:31.105 INFO:tasks.workunit.client.1.smithi153.stdout:5/143: creat d8/df/f22 x:0 0 0 2024-04-16T02:35:31.107 INFO:tasks.workunit.client.1.smithi153.stdout:4/183: dwrite d3/d9/f14 [0,4194304] 0 2024-04-16T02:35:31.107 INFO:tasks.workunit.client.1.smithi153.stdout:5/144: chown d8/df/f1e 1919080989 1 2024-04-16T02:35:31.109 INFO:tasks.workunit.client.1.smithi153.stdout:4/184: write d3/d9/db/de/f26 [345325,41449] 0 2024-04-16T02:35:31.110 INFO:tasks.workunit.client.1.smithi153.stdout:9/181: rename d4/d6/dc/l16 to d4/d6/d1b/l35 0 2024-04-16T02:35:31.111 INFO:tasks.workunit.client.1.smithi153.stdout:2/150: truncate f0 1947239 0 2024-04-16T02:35:31.113 INFO:tasks.workunit.client.1.smithi153.stdout:7/176: mknod d0/d4/c3c 0 2024-04-16T02:35:31.113 INFO:tasks.workunit.client.1.smithi153.stdout:3/181: truncate f1 550692 0 2024-04-16T02:35:31.118 INFO:tasks.workunit.client.1.smithi153.stdout:3/182: creat d6/d10/d22/d2e/f42 x:0 0 0 2024-04-16T02:35:31.118 INFO:tasks.workunit.client.1.smithi153.stdout:4/185: dread d3/d9/d24/f3a [0,4194304] 0 2024-04-16T02:35:31.119 INFO:tasks.workunit.client.1.smithi153.stdout:2/151: creat d9/d24/d34/d36/f3b x:0 0 0 2024-04-16T02:35:31.119 INFO:tasks.workunit.client.1.smithi153.stdout:5/145: dwrite d8/df/f1e [0,4194304] 0 2024-04-16T02:35:31.121 INFO:tasks.workunit.client.1.smithi153.stdout:3/183: rmdir d6/d10/d3b 39 2024-04-16T02:35:31.122 INFO:tasks.workunit.client.1.smithi153.stdout:5/146: readlink d8/le 0 2024-04-16T02:35:31.126 INFO:tasks.workunit.client.1.smithi153.stdout:5/147: mkdir d8/df/d23 0 2024-04-16T02:35:31.127 INFO:tasks.workunit.client.1.smithi153.stdout:7/177: dread d0/d4/db/d2b/f2f [0,4194304] 0 2024-04-16T02:35:31.129 INFO:tasks.workunit.client.1.smithi153.stdout:0/198: link dd/fe dd/d14/f3b 0 2024-04-16T02:35:31.129 INFO:tasks.workunit.client.1.smithi153.stdout:9/182: dwrite d4/f13 [0,4194304] 0 2024-04-16T02:35:31.130 INFO:tasks.workunit.client.1.smithi153.stdout:5/148: write f4 [1113063,108404] 0 2024-04-16T02:35:31.131 INFO:tasks.workunit.client.1.smithi153.stdout:7/178: dread - d0/d13/f32 zero size 2024-04-16T02:35:31.136 INFO:tasks.workunit.client.1.smithi153.stdout:3/184: getdents d6 0 2024-04-16T02:35:31.136 INFO:tasks.workunit.client.1.smithi153.stdout:5/149: fdatasync d8/df/f11 0 2024-04-16T02:35:31.139 INFO:tasks.workunit.client.1.smithi153.stdout:2/152: dread d9/d2b/d1c/f28 [0,4194304] 0 2024-04-16T02:35:31.139 INFO:tasks.workunit.client.1.smithi153.stdout:4/186: dread d3/d9/db/de/f12 [0,4194304] 0 2024-04-16T02:35:31.139 INFO:tasks.workunit.client.1.smithi153.stdout:3/185: chown d6/d10/d22/d2e/f35 8520 1 2024-04-16T02:35:31.140 INFO:tasks.workunit.client.1.smithi153.stdout:5/150: write f4 [1994203,7635] 0 2024-04-16T02:35:31.142 INFO:tasks.workunit.client.1.smithi153.stdout:3/186: fdatasync d6/d10/d22/d2e/f42 0 2024-04-16T02:35:31.142 INFO:tasks.workunit.client.1.smithi153.stdout:2/153: unlink d9/f33 0 2024-04-16T02:35:31.143 INFO:tasks.workunit.client.1.smithi153.stdout:0/199: link dd/d14/d19/f1e dd/d28/f3c 0 2024-04-16T02:35:31.145 INFO:tasks.workunit.client.1.smithi153.stdout:9/183: dread d4/d6/dc/d24/f25 [0,4194304] 0 2024-04-16T02:35:31.145 INFO:tasks.workunit.client.1.smithi153.stdout:0/200: readlink dd/l15 0 2024-04-16T02:35:31.148 INFO:tasks.workunit.client.1.smithi153.stdout:7/179: getdents d0/d4/db/d26 0 2024-04-16T02:35:31.148 INFO:tasks.workunit.client.1.smithi153.stdout:3/187: fdatasync d6/d10/d3b/f41 0 2024-04-16T02:35:31.149 INFO:tasks.workunit.client.1.smithi153.stdout:7/180: stat d0/d4 0 2024-04-16T02:35:31.150 INFO:tasks.workunit.client.1.smithi153.stdout:9/184: write d4/d6/f22 [474947,11405] 0 2024-04-16T02:35:31.154 INFO:tasks.workunit.client.1.smithi153.stdout:3/188: dread d6/d10/d22/f28 [0,4194304] 0 2024-04-16T02:35:31.154 INFO:tasks.workunit.client.1.smithi153.stdout:2/154: fsync d9/d24/d27/f1d 0 2024-04-16T02:35:31.155 INFO:tasks.workunit.client.1.smithi153.stdout:4/187: dwrite d3/d9/f14 [0,4194304] 0 2024-04-16T02:35:31.155 INFO:tasks.workunit.client.1.smithi153.stdout:0/201: dread dd/d14/d19/f1a [0,4194304] 0 2024-04-16T02:35:31.157 INFO:tasks.workunit.client.1.smithi153.stdout:5/151: dwrite d8/df/f22 [0,4194304] 0 2024-04-16T02:35:31.162 INFO:tasks.workunit.client.1.smithi153.stdout:3/189: read d6/d10/f13 [751744,11697] 0 2024-04-16T02:35:31.162 INFO:tasks.workunit.client.1.smithi153.stdout:5/152: creat d8/df/f24 x:0 0 0 2024-04-16T02:35:31.162 INFO:tasks.workunit.client.1.smithi153.stdout:2/155: mknod d9/d24/d34/d36/c3c 0 2024-04-16T02:35:31.164 INFO:tasks.workunit.client.1.smithi153.stdout:4/188: write d3/d9/f3d [476898,37930] 0 2024-04-16T02:35:31.167 INFO:tasks.workunit.client.1.smithi153.stdout:5/153: write d8/df/f22 [2675218,93948] 0 2024-04-16T02:35:31.167 INFO:tasks.workunit.client.1.smithi153.stdout:0/202: write dd/d14/d19/d1c/f23 [1228277,129732] 0 2024-04-16T02:35:31.171 INFO:tasks.workunit.client.1.smithi153.stdout:9/185: dwrite d4/f13 [0,4194304] 0 2024-04-16T02:35:31.172 INFO:tasks.workunit.client.1.smithi153.stdout:7/181: dwrite d0/d13/f1c [0,4194304] 0 2024-04-16T02:35:31.172 INFO:tasks.workunit.client.1.smithi153.stdout:4/189: mkdir d3/d9/d24/d3e 0 2024-04-16T02:35:31.175 INFO:tasks.workunit.client.1.smithi153.stdout:2/156: rename d9/d2b/c15 to d9/d2b/d1c/c3d 0 2024-04-16T02:35:31.178 INFO:tasks.workunit.client.1.smithi153.stdout:5/154: rename d8/c18 to d8/c25 0 2024-04-16T02:35:31.179 INFO:tasks.workunit.client.1.smithi153.stdout:4/190: mkdir d3/d9/db/d17/d3f 0 2024-04-16T02:35:31.184 INFO:tasks.workunit.client.1.smithi153.stdout:9/186: dread d4/d6/f22 [0,4194304] 0 2024-04-16T02:35:31.185 INFO:tasks.workunit.client.1.smithi153.stdout:2/157: mknod d9/d24/d27/d2c/c3e 0 2024-04-16T02:35:31.188 INFO:tasks.workunit.client.1.smithi153.stdout:0/203: dwrite dd/d14/d19/d1c/f23 [4194304,4194304] 0 2024-04-16T02:35:31.189 INFO:tasks.workunit.client.1.smithi153.stdout:9/187: symlink d4/d6/d1b/l36 0 2024-04-16T02:35:31.190 INFO:tasks.workunit.client.1.smithi153.stdout:2/158: readlink d9/d2b/l11 0 2024-04-16T02:35:31.191 INFO:tasks.workunit.client.1.smithi153.stdout:8/170: sync 2024-04-16T02:35:31.191 INFO:tasks.workunit.client.1.smithi153.stdout:6/124: sync 2024-04-16T02:35:31.191 INFO:tasks.workunit.client.1.smithi153.stdout:1/166: sync 2024-04-16T02:35:31.192 INFO:tasks.workunit.client.1.smithi153.stdout:7/182: dread d0/d4/db/d2b/f2f [0,4194304] 0 2024-04-16T02:35:31.195 INFO:tasks.workunit.client.1.smithi153.stdout:0/204: symlink dd/d32/d33/l3d 0 2024-04-16T02:35:31.196 INFO:tasks.workunit.client.1.smithi153.stdout:9/188: creat d4/d6/d1b/f37 x:0 0 0 2024-04-16T02:35:31.197 INFO:tasks.workunit.client.1.smithi153.stdout:7/183: stat d0/d4 0 2024-04-16T02:35:31.198 INFO:tasks.workunit.client.1.smithi153.stdout:1/167: write d4/fe [595603,55577] 0 2024-04-16T02:35:31.201 INFO:tasks.workunit.client.1.smithi153.stdout:6/125: write dc/f17 [1113703,68393] 0 2024-04-16T02:35:31.201 INFO:tasks.workunit.client.1.smithi153.stdout:8/171: creat d2/de/d34/f40 x:0 0 0 2024-04-16T02:35:31.203 INFO:tasks.workunit.client.1.smithi153.stdout:2/159: mkdir d9/d2b/d18/d3f 0 2024-04-16T02:35:31.204 INFO:tasks.workunit.client.1.smithi153.stdout:5/155: dwrite d8/df/f1e [0,4194304] 0 2024-04-16T02:35:31.205 INFO:tasks.workunit.client.1.smithi153.stdout:4/191: dwrite d3/d9/f3d [0,4194304] 0 2024-04-16T02:35:31.205 INFO:tasks.workunit.client.1.smithi153.stdout:3/190: dwrite d6/d10/f18 [0,4194304] 0 2024-04-16T02:35:31.205 INFO:tasks.workunit.client.1.smithi153.stdout:0/205: write dd/d32/f39 [692496,116962] 0 2024-04-16T02:35:31.208 INFO:tasks.workunit.client.1.smithi153.stdout:1/168: write d4/dd/f1d [657564,11557] 0 2024-04-16T02:35:31.208 INFO:tasks.workunit.client.1.smithi153.stdout:7/184: write d0/d13/f1c [1377448,63503] 0 2024-04-16T02:35:31.211 INFO:tasks.workunit.client.1.smithi153.stdout:2/160: chown d9/d2b/d1c/l21 176 1 2024-04-16T02:35:31.211 INFO:tasks.workunit.client.1.smithi153.stdout:3/191: write d6/f1a [422063,58841] 0 2024-04-16T02:35:31.214 INFO:tasks.workunit.client.1.smithi153.stdout:7/185: stat d0/d4/db/d2b 0 2024-04-16T02:35:31.216 INFO:tasks.workunit.client.1.smithi153.stdout:3/192: write d6/d10/d22/d2c/f33 [24075,130635] 0 2024-04-16T02:35:31.219 INFO:tasks.workunit.client.1.smithi153.stdout:5/156: mknod d8/df/d23/c26 0 2024-04-16T02:35:31.219 INFO:tasks.workunit.client.1.smithi153.stdout:9/189: dwrite d4/d6/dc/d24/f29 [8388608,4194304] 0 2024-04-16T02:35:31.219 INFO:tasks.workunit.client.1.smithi153.stdout:5/157: chown d8/df 8 1 2024-04-16T02:35:31.220 INFO:tasks.workunit.client.1.smithi153.stdout:9/190: stat d4/d6/d1b/d2a/l2e 0 2024-04-16T02:35:31.220 INFO:tasks.workunit.client.1.smithi153.stdout:5/158: dread - d8/df/f24 zero size 2024-04-16T02:35:31.220 INFO:tasks.workunit.client.1.smithi153.stdout:8/172: link d2/l3 d2/de/l41 0 2024-04-16T02:35:31.221 INFO:tasks.workunit.client.1.smithi153.stdout:9/191: fdatasync d4/d6/dc/d24/f28 0 2024-04-16T02:35:31.222 INFO:tasks.workunit.client.1.smithi153.stdout:9/192: fdatasync d4/d6/dc/d24/f28 0 2024-04-16T02:35:31.223 INFO:tasks.workunit.client.1.smithi153.stdout:9/193: dread d4/d6/f22 [0,4194304] 0 2024-04-16T02:35:31.224 INFO:tasks.workunit.client.1.smithi153.stdout:9/194: chown d4/d33 3 1 2024-04-16T02:35:31.225 INFO:tasks.workunit.client.1.smithi153.stdout:3/193: write d6/d10/f13 [146361,88909] 0 2024-04-16T02:35:31.228 INFO:tasks.workunit.client.1.smithi153.stdout:9/195: dread d4/d6/dc/d24/f25 [0,4194304] 0 2024-04-16T02:35:31.229 INFO:tasks.workunit.client.1.smithi153.stdout:9/196: chown d4/l31 8 1 2024-04-16T02:35:31.229 INFO:tasks.workunit.client.1.smithi153.stdout:9/197: stat d4/d6 0 2024-04-16T02:35:31.230 INFO:tasks.workunit.client.1.smithi153.stdout:4/192: mknod d3/d9/db/de/d2c/d38/c40 0 2024-04-16T02:35:31.243 INFO:tasks.workunit.client.1.smithi153.stdout:9/198: chown d4/d6/dc/d24 28556 1 2024-04-16T02:35:31.243 INFO:tasks.workunit.client.1.smithi153.stdout:5/159: rename l6 to d8/d1f/l27 0 2024-04-16T02:35:31.244 INFO:tasks.workunit.client.1.smithi153.stdout:2/161: mkdir d9/d2b/d18/d3f/d40 0 2024-04-16T02:35:31.244 INFO:tasks.workunit.client.1.smithi153.stdout:5/160: write f4 [1420564,58151] 0 2024-04-16T02:35:31.244 INFO:tasks.workunit.client.1.smithi153.stdout:0/206: link f9 dd/d14/d19/d1c/f3e 0 2024-04-16T02:35:31.244 INFO:tasks.workunit.client.1.smithi153.stdout:8/173: dwrite d2/ff [0,4194304] 0 2024-04-16T02:35:31.244 INFO:tasks.workunit.client.1.smithi153.stdout:4/193: symlink d3/d9/db/d17/l41 0 2024-04-16T02:35:31.244 INFO:tasks.workunit.client.1.smithi153.stdout:4/194: fsync d3/d9/db/de/f3b 0 2024-04-16T02:35:31.244 INFO:tasks.workunit.client.1.smithi153.stdout:1/169: write d4/f1f [147734,418] 0 2024-04-16T02:35:31.246 INFO:tasks.workunit.client.1.smithi153.stdout:5/161: mkdir d8/df/d28 0 2024-04-16T02:35:31.249 INFO:tasks.workunit.client.1.smithi153.stdout:8/174: dread d2/f17 [0,4194304] 0 2024-04-16T02:35:31.250 INFO:tasks.workunit.client.1.smithi153.stdout:7/186: getdents d0/d13 0 2024-04-16T02:35:31.250 INFO:tasks.workunit.client.1.smithi153.stdout:8/175: chown d2/d1c/d1f/f3c 8981 1 2024-04-16T02:35:31.251 INFO:tasks.workunit.client.1.smithi153.stdout:1/170: chown d4/lb 1 1 2024-04-16T02:35:31.251 INFO:tasks.workunit.client.1.smithi153.stdout:8/176: chown d2/l10 22090340 1 2024-04-16T02:35:31.252 INFO:tasks.workunit.client.1.smithi153.stdout:5/162: mknod d8/d1f/c29 0 2024-04-16T02:35:31.256 INFO:tasks.workunit.client.1.smithi153.stdout:4/195: rename d3/c8 to d3/d9/d24/c42 0 2024-04-16T02:35:31.256 INFO:tasks.workunit.client.1.smithi153.stdout:5/163: rename d8/df to d8/df/d28/d2a 22 2024-04-16T02:35:31.256 INFO:tasks.workunit.client.1.smithi153.stdout:4/196: fdatasync d3/d9/f1a 0 2024-04-16T02:35:31.262 INFO:tasks.workunit.client.1.smithi153.stdout:8/177: symlink d2/d13/d2c/l42 0 2024-04-16T02:35:31.264 INFO:tasks.workunit.client.1.smithi153.stdout:5/164: dwrite d8/df/f10 [0,4194304] 0 2024-04-16T02:35:31.265 INFO:tasks.workunit.client.1.smithi153.stdout:5/165: read d8/df/f1e [3462140,122339] 0 2024-04-16T02:35:31.265 INFO:tasks.workunit.client.1.smithi153.stdout:7/187: dwrite d0/d4/db/d2b/f25 [0,4194304] 0 2024-04-16T02:35:31.265 INFO:tasks.workunit.client.1.smithi153.stdout:9/199: link d4/d6/d1b/d2a/f32 d4/d33/f38 0 2024-04-16T02:35:31.266 INFO:tasks.workunit.client.1.smithi153.stdout:4/197: mkdir d3/d9/db/d17/d28/d43 0 2024-04-16T02:35:31.266 INFO:tasks.workunit.client.1.smithi153.stdout:5/166: readlink d8/df/l12 0 2024-04-16T02:35:31.267 INFO:tasks.workunit.client.1.smithi153.stdout:7/188: fsync d0/d13/f1c 0 2024-04-16T02:35:31.269 INFO:tasks.workunit.client.1.smithi153.stdout:0/207: write dd/fe [33298,118796] 0 2024-04-16T02:35:31.270 INFO:tasks.workunit.client.1.smithi153.stdout:2/162: getdents d9/d24/d34/d36 0 2024-04-16T02:35:31.271 INFO:tasks.workunit.client.1.smithi153.stdout:0/208: chown dd/f18 5931117 1 2024-04-16T02:35:31.273 INFO:tasks.workunit.client.1.smithi153.stdout:8/178: unlink d2/ca 0 2024-04-16T02:35:31.277 INFO:tasks.workunit.client.1.smithi153.stdout:4/198: creat d3/d9/db/de/f44 x:0 0 0 2024-04-16T02:35:31.280 INFO:tasks.workunit.client.1.smithi153.stdout:2/163: mkdir d9/d24/d27/d2c/d41 0 2024-04-16T02:35:31.281 INFO:tasks.workunit.client.1.smithi153.stdout:8/179: readlink d2/l3 0 2024-04-16T02:35:31.285 INFO:tasks.workunit.client.1.smithi153.stdout:4/199: dread d3/d9/db/de/f13 [0,4194304] 0 2024-04-16T02:35:31.317 INFO:tasks.workunit.client.1.smithi153.stdout:9/200: mknod d4/d6/c39 0 2024-04-16T02:35:31.318 INFO:tasks.workunit.client.1.smithi153.stdout:9/201: dread - d4/d6/dc/d24/f30 zero size 2024-04-16T02:35:31.322 INFO:tasks.workunit.client.1.smithi153.stdout:4/200: mkdir d3/d9/db/d45 0 2024-04-16T02:35:31.322 INFO:tasks.workunit.client.1.smithi153.stdout:5/167: link d8/df/d23/c26 d8/df/d23/c2b 0 2024-04-16T02:35:31.323 INFO:tasks.workunit.client.1.smithi153.stdout:4/201: chown d3/c30 35 1 2024-04-16T02:35:31.324 INFO:tasks.workunit.client.1.smithi153.stdout:7/189: dwrite d0/d4/db/f2a [0,4194304] 0 2024-04-16T02:35:31.324 INFO:tasks.workunit.client.1.smithi153.stdout:1/171: truncate d4/dd/d13/d26/f33 3802048 0 2024-04-16T02:35:31.324 INFO:tasks.workunit.client.1.smithi153.stdout:7/190: chown d0/d13 1446131 1 2024-04-16T02:35:31.325 INFO:tasks.workunit.client.1.smithi153.stdout:1/172: fsync d4/dd/d13/f1a 0 2024-04-16T02:35:31.327 INFO:tasks.workunit.client.1.smithi153.stdout:7/191: write d0/d4/db/ff [833968,130187] 0 2024-04-16T02:35:31.327 INFO:tasks.workunit.client.1.smithi153.stdout:2/164: symlink d9/d2b/d1c/d20/l42 0 2024-04-16T02:35:31.330 INFO:tasks.workunit.client.1.smithi153.stdout:8/180: creat d2/de/d3f/f43 x:0 0 0 2024-04-16T02:35:31.330 INFO:tasks.workunit.client.1.smithi153.stdout:9/202: mkdir d4/d33/d3a 0 2024-04-16T02:35:31.331 INFO:tasks.workunit.client.1.smithi153.stdout:5/168: unlink d8/le 0 2024-04-16T02:35:31.335 INFO:tasks.workunit.client.1.smithi153.stdout:1/173: dwrite d4/f7 [4194304,4194304] 0 2024-04-16T02:35:31.343 INFO:tasks.workunit.client.1.smithi153.stdout:2/165: creat d9/d2b/d1c/d20/f43 x:0 0 0 2024-04-16T02:35:31.345 INFO:tasks.workunit.client.1.smithi153.stdout:9/203: mknod d4/d6/d1b/c3b 0 2024-04-16T02:35:31.347 INFO:tasks.workunit.client.1.smithi153.stdout:5/169: creat d8/df/f2c x:0 0 0 2024-04-16T02:35:31.347 INFO:tasks.workunit.client.1.smithi153.stdout:5/170: readlink d8/l17 0 2024-04-16T02:35:31.351 INFO:tasks.workunit.client.1.smithi153.stdout:4/202: creat d3/f46 x:0 0 0 2024-04-16T02:35:31.352 INFO:tasks.workunit.client.1.smithi153.stdout:9/204: symlink d4/d6/d1b/d2a/l3c 0 2024-04-16T02:35:31.353 INFO:tasks.workunit.client.1.smithi153.stdout:2/166: dread d9/d2b/f14 [0,4194304] 0 2024-04-16T02:35:31.353 INFO:tasks.workunit.client.1.smithi153.stdout:1/174: symlink d4/dd/d18/d29/l35 0 2024-04-16T02:35:31.354 INFO:tasks.workunit.client.1.smithi153.stdout:9/205: write d4/d6/dc/d24/f28 [1235176,16607] 0 2024-04-16T02:35:31.354 INFO:tasks.workunit.client.1.smithi153.stdout:5/171: mknod d8/df/d23/c2d 0 2024-04-16T02:35:31.355 INFO:tasks.workunit.client.1.smithi153.stdout:1/175: chown d4/f7 409952 1 2024-04-16T02:35:31.355 INFO:tasks.workunit.client.1.smithi153.stdout:9/206: chown d4/d33/d3a 7 1 2024-04-16T02:35:31.355 INFO:tasks.workunit.client.1.smithi153.stdout:1/176: chown d4/dd/d13/f2a 337 1 2024-04-16T02:35:31.357 INFO:tasks.workunit.client.1.smithi153.stdout:7/192: rename d0/d13/f1c to d0/f3d 0 2024-04-16T02:35:31.359 INFO:tasks.workunit.client.1.smithi153.stdout:4/203: creat d3/d9/db/d17/f47 x:0 0 0 2024-04-16T02:35:31.360 INFO:tasks.workunit.client.1.smithi153.stdout:4/204: truncate d3/d9/db/de/f13 729137 0 2024-04-16T02:35:31.362 INFO:tasks.workunit.client.1.smithi153.stdout:9/207: creat d4/d6/d1b/f3d x:0 0 0 2024-04-16T02:35:31.365 INFO:tasks.workunit.client.1.smithi153.stdout:8/181: getdents d2/d13/d3a 0 2024-04-16T02:35:31.366 INFO:tasks.workunit.client.1.smithi153.stdout:2/167: creat d9/d24/d27/d2c/d41/f44 x:0 0 0 2024-04-16T02:35:31.367 INFO:tasks.workunit.client.1.smithi153.stdout:4/205: symlink d3/d9/db/d17/d28/l48 0 2024-04-16T02:35:31.368 INFO:tasks.workunit.client.1.smithi153.stdout:4/206: fsync d3/ff 0 2024-04-16T02:35:31.368 INFO:tasks.workunit.client.1.smithi153.stdout:9/208: creat d4/d6/d1b/f3e x:0 0 0 2024-04-16T02:35:31.372 INFO:tasks.workunit.client.1.smithi153.stdout:5/172: rmdir d8 39 2024-04-16T02:35:31.374 INFO:tasks.workunit.client.1.smithi153.stdout:1/177: read d4/dd/d13/d26/f33 [1134041,23423] 0 2024-04-16T02:35:31.380 INFO:tasks.workunit.client.1.smithi153.stdout:7/193: dwrite d0/d4/db/d2b/f2f [0,4194304] 0 2024-04-16T02:35:31.380 INFO:tasks.workunit.client.1.smithi153.stdout:9/209: dwrite d4/d6/dc/d24/f28 [0,4194304] 0 2024-04-16T02:35:31.380 INFO:tasks.workunit.client.1.smithi153.stdout:1/178: rmdir d4/dd/d13/d23 39 2024-04-16T02:35:31.383 INFO:tasks.workunit.client.1.smithi153.stdout:8/182: dwrite d2/de/d34/f40 [0,4194304] 0 2024-04-16T02:35:31.383 INFO:tasks.workunit.client.1.smithi153.stdout:6/126: sync 2024-04-16T02:35:31.386 INFO:tasks.workunit.client.1.smithi153.stdout:2/168: getdents d9 0 2024-04-16T02:35:31.387 INFO:tasks.workunit.client.1.smithi153.stdout:5/173: rmdir d8/df/d23 39 2024-04-16T02:35:31.393 INFO:tasks.workunit.client.1.smithi153.stdout:8/183: write d2/d1c/f26 [993044,75688] 0 2024-04-16T02:35:31.395 INFO:tasks.workunit.client.1.smithi153.stdout:8/184: write d2/de/d34/f40 [2798974,42825] 0 2024-04-16T02:35:31.397 INFO:tasks.workunit.client.1.smithi153.stdout:6/127: rmdir dc/d12 39 2024-04-16T02:35:31.398 INFO:tasks.workunit.client.1.smithi153.stdout:4/207: link d3/d9/db/d17/d28/l48 d3/l49 0 2024-04-16T02:35:31.400 INFO:tasks.workunit.client.1.smithi153.stdout:4/208: write d3/d9/db/de/f13 [1094262,89522] 0 2024-04-16T02:35:31.402 INFO:tasks.workunit.client.1.smithi153.stdout:9/210: fsync d4/f27 0 2024-04-16T02:35:31.404 INFO:tasks.workunit.client.1.smithi153.stdout:7/194: write d0/f36 [997263,118637] 0 2024-04-16T02:35:31.405 INFO:tasks.workunit.client.1.smithi153.stdout:1/179: creat d4/f36 x:0 0 0 2024-04-16T02:35:31.407 INFO:tasks.workunit.client.1.smithi153.stdout:4/209: creat d3/d9/db/f4a x:0 0 0 2024-04-16T02:35:31.407 INFO:tasks.workunit.client.1.smithi153.stdout:5/174: chown d8/df/d23/c26 33262805 1 2024-04-16T02:35:31.409 INFO:tasks.workunit.client.1.smithi153.stdout:5/175: fdatasync f4 0 2024-04-16T02:35:31.412 INFO:tasks.workunit.client.1.smithi153.stdout:2/169: dwrite d9/d2b/f19 [0,4194304] 0 2024-04-16T02:35:31.412 INFO:tasks.workunit.client.1.smithi153.stdout:9/211: fsync d4/d6/d1b/d2a/f32 0 2024-04-16T02:35:31.413 INFO:tasks.workunit.client.1.smithi153.stdout:7/195: creat d0/d4/db/d2b/f3e x:0 0 0 2024-04-16T02:35:31.416 INFO:tasks.workunit.client.1.smithi153.stdout:1/180: dread d4/dd/f1d [0,4194304] 0 2024-04-16T02:35:31.416 INFO:tasks.workunit.client.1.smithi153.stdout:2/170: write d9/d2b/f19 [3005067,76516] 0 2024-04-16T02:35:31.417 INFO:tasks.workunit.client.1.smithi153.stdout:7/196: fdatasync d0/fa 0 2024-04-16T02:35:31.418 INFO:tasks.workunit.client.1.smithi153.stdout:8/185: dwrite d2/f12 [0,4194304] 0 2024-04-16T02:35:31.418 INFO:tasks.workunit.client.1.smithi153.stdout:9/212: creat d4/d33/f3f x:0 0 0 2024-04-16T02:35:31.420 INFO:tasks.workunit.client.1.smithi153.stdout:5/176: mkdir d8/df/d2e 0 2024-04-16T02:35:31.421 INFO:tasks.workunit.client.1.smithi153.stdout:5/177: chown d8/df/d23 6333744 1 2024-04-16T02:35:31.423 INFO:tasks.workunit.client.1.smithi153.stdout:5/178: write d8/df/f22 [2638066,129790] 0 2024-04-16T02:35:31.428 INFO:tasks.workunit.client.1.smithi153.stdout:8/186: rename d2/d1c/c20 to d2/de/d3f/c44 0 2024-04-16T02:35:31.428 INFO:tasks.workunit.client.1.smithi153.stdout:7/197: dwrite d0/d4/db/ff [0,4194304] 0 2024-04-16T02:35:31.432 INFO:tasks.workunit.client.1.smithi153.stdout:4/210: getdents d3/d9/db 0 2024-04-16T02:35:31.432 INFO:tasks.workunit.client.1.smithi153.stdout:1/181: mknod d4/c37 0 2024-04-16T02:35:31.434 INFO:tasks.workunit.client.1.smithi153.stdout:4/211: fdatasync d3/d9/db/de/f13 0 2024-04-16T02:35:31.435 INFO:tasks.workunit.client.1.smithi153.stdout:5/179: dwrite d8/df/f11 [0,4194304] 0 2024-04-16T02:35:31.438 INFO:tasks.workunit.client.1.smithi153.stdout:8/187: dread d2/f12 [0,4194304] 0 2024-04-16T02:35:31.440 INFO:tasks.workunit.client.1.smithi153.stdout:8/188: write d2/d13/f1d [687955,71553] 0 2024-04-16T02:35:31.440 INFO:tasks.workunit.client.1.smithi153.stdout:4/212: stat d3/cd 0 2024-04-16T02:35:31.470 INFO:tasks.workunit.client.1.smithi153.stdout:3/194: sync 2024-04-16T02:35:31.470 INFO:tasks.workunit.client.1.smithi153.stdout:0/209: sync 2024-04-16T02:35:31.484 INFO:tasks.workunit.client.1.smithi153.stdout:4/213: stat d3/d9/db/de/d2c/d38 0 2024-04-16T02:35:31.484 INFO:tasks.workunit.client.1.smithi153.stdout:5/180: chown d8/df/c19 14561 1 2024-04-16T02:35:31.484 INFO:tasks.workunit.client.1.smithi153.stdout:9/213: dread d4/d6/d1b/d2a/f32 [0,4194304] 0 2024-04-16T02:35:31.484 INFO:tasks.workunit.client.1.smithi153.stdout:2/171: truncate d9/d2b/f19 186359 0 2024-04-16T02:35:31.484 INFO:tasks.workunit.client.1.smithi153.stdout:3/195: stat d6/d10 0 2024-04-16T02:35:31.486 INFO:tasks.workunit.client.1.smithi153.stdout:4/214: dread - d3/d9/db/f4a zero size 2024-04-16T02:35:31.486 INFO:tasks.workunit.client.1.smithi153.stdout:0/210: mkdir dd/d32/d3f 0 2024-04-16T02:35:31.488 INFO:tasks.workunit.client.1.smithi153.stdout:3/196: chown d6/d10/l3a 78 1 2024-04-16T02:35:31.488 INFO:tasks.workunit.client.1.smithi153.stdout:5/181: mkdir d8/df/d2f 0 2024-04-16T02:35:31.489 INFO:tasks.workunit.client.1.smithi153.stdout:9/214: mkdir d4/d33/d3a/d40 0 2024-04-16T02:35:31.492 INFO:tasks.workunit.client.1.smithi153.stdout:3/197: mkdir d6/d10/d22/d43 0 2024-04-16T02:35:31.494 INFO:tasks.workunit.client.1.smithi153.stdout:8/189: dwrite d2/d13/f1b [0,4194304] 0 2024-04-16T02:35:31.495 INFO:tasks.workunit.client.1.smithi153.stdout:9/215: symlink d4/d6/dc/l41 0 2024-04-16T02:35:31.498 INFO:tasks.workunit.client.1.smithi153.stdout:7/198: dwrite d0/f1e [0,4194304] 0 2024-04-16T02:35:31.500 INFO:tasks.workunit.client.1.smithi153.stdout:4/215: rename d3/d9/db/de/f44 to d3/d9/db/d17/f4b 0 2024-04-16T02:35:31.500 INFO:tasks.workunit.client.1.smithi153.stdout:5/182: getdents d8 0 2024-04-16T02:35:31.501 INFO:tasks.workunit.client.1.smithi153.stdout:9/216: stat d4/f27 0 2024-04-16T02:35:31.502 INFO:tasks.workunit.client.1.smithi153.stdout:8/190: readlink d2/l10 0 2024-04-16T02:35:31.504 INFO:tasks.workunit.client.1.smithi153.stdout:4/216: chown d3/d9/db/de/f12 37 1 2024-04-16T02:35:31.507 INFO:tasks.workunit.client.1.smithi153.stdout:7/199: mkdir d0/d4/d3f 0 2024-04-16T02:35:31.508 INFO:tasks.workunit.client.1.smithi153.stdout:2/172: dwrite d9/d24/d27/d2c/d41/f44 [0,4194304] 0 2024-04-16T02:35:31.508 INFO:tasks.workunit.client.1.smithi153.stdout:9/217: creat d4/d6/dc/d24/f42 x:0 0 0 2024-04-16T02:35:31.509 INFO:tasks.workunit.client.1.smithi153.stdout:3/198: creat d6/d10/d22/d2c/f44 x:0 0 0 2024-04-16T02:35:31.511 INFO:tasks.workunit.client.1.smithi153.stdout:0/211: dwrite dd/d14/d19/d1c/f2f [0,4194304] 0 2024-04-16T02:35:31.513 INFO:tasks.workunit.client.1.smithi153.stdout:2/173: unlink f8 0 2024-04-16T02:35:31.513 INFO:tasks.workunit.client.1.smithi153.stdout:4/217: chown d3/d9/db/c22 3699492 1 2024-04-16T02:35:31.513 INFO:tasks.workunit.client.1.smithi153.stdout:9/218: write d4/d6/d1b/f3e [759893,102645] 0 2024-04-16T02:35:31.513 INFO:tasks.workunit.client.1.smithi153.stdout:3/199: write d6/f1a [4280998,104324] 0 2024-04-16T02:35:31.513 INFO:tasks.workunit.client.1.smithi153.stdout:7/200: creat d0/d4/db/d2b/f40 x:0 0 0 2024-04-16T02:35:31.515 INFO:tasks.workunit.client.1.smithi153.stdout:4/218: chown d3/c2a 107100236 1 2024-04-16T02:35:31.518 INFO:tasks.workunit.client.1.smithi153.stdout:3/200: write d6/f12 [4350124,118064] 0 2024-04-16T02:35:31.520 INFO:tasks.workunit.client.1.smithi153.stdout:0/212: creat dd/d28/f40 x:0 0 0 2024-04-16T02:35:31.523 INFO:tasks.workunit.client.1.smithi153.stdout:2/174: dread - d9/d24/d27/f2e zero size 2024-04-16T02:35:31.523 INFO:tasks.workunit.client.1.smithi153.stdout:7/201: truncate d0/d4/db/d26/f27 1707611 0 2024-04-16T02:35:31.523 INFO:tasks.workunit.client.1.smithi153.stdout:9/219: rename d4/d6/d1b/f3d to d4/d6/dc/d24/f43 0 2024-04-16T02:35:31.524 INFO:tasks.workunit.client.1.smithi153.stdout:4/219: rmdir d3/d9/db/de/d2c 39 2024-04-16T02:35:31.526 INFO:tasks.workunit.client.1.smithi153.stdout:7/202: rmdir d0 39 2024-04-16T02:35:31.526 INFO:tasks.workunit.client.1.smithi153.stdout:0/213: write dd/d14/d19/f27 [195746,10645] 0 2024-04-16T02:35:31.528 INFO:tasks.workunit.client.1.smithi153.stdout:9/220: symlink d4/d33/l44 0 2024-04-16T02:35:31.528 INFO:tasks.workunit.client.1.smithi153.stdout:0/214: readlink dd/l15 0 2024-04-16T02:35:31.528 INFO:tasks.workunit.client.1.smithi153.stdout:2/175: mkdir d9/d2b/d1c/d20/d45 0 2024-04-16T02:35:31.530 INFO:tasks.workunit.client.1.smithi153.stdout:4/220: mkdir d3/d9/db/d17/d28/d4c 0 2024-04-16T02:35:31.530 INFO:tasks.workunit.client.1.smithi153.stdout:8/191: dread d2/d1c/d1f/d27/d2d/f31 [0,4194304] 0 2024-04-16T02:35:31.532 INFO:tasks.workunit.client.1.smithi153.stdout:5/183: dwrite d8/df/f1e [0,4194304] 0 2024-04-16T02:35:31.532 INFO:tasks.workunit.client.1.smithi153.stdout:2/176: dread - d9/d2b/f2a zero size 2024-04-16T02:35:31.533 INFO:tasks.workunit.client.1.smithi153.stdout:3/201: dwrite d6/d10/f32 [0,4194304] 0 2024-04-16T02:35:31.533 INFO:tasks.workunit.client.1.smithi153.stdout:4/221: read d3/d9/f14 [2873034,41517] 0 2024-04-16T02:35:31.535 INFO:tasks.workunit.client.1.smithi153.stdout:7/203: getdents d0/d13 0 2024-04-16T02:35:31.536 INFO:tasks.workunit.client.1.smithi153.stdout:9/221: write d4/d6/dc/d24/f25 [1892191,120148] 0 2024-04-16T02:35:31.536 INFO:tasks.workunit.client.1.smithi153.stdout:8/192: write d2/de/d34/f40 [3559640,108990] 0 2024-04-16T02:35:31.538 INFO:tasks.workunit.client.1.smithi153.stdout:4/222: dread - d3/d9/db/de/f3b zero size 2024-04-16T02:35:31.539 INFO:tasks.workunit.client.1.smithi153.stdout:2/177: write d9/d24/d27/d2c/d41/f44 [3724811,41029] 0 2024-04-16T02:35:31.540 INFO:tasks.workunit.client.1.smithi153.stdout:5/184: creat d8/f30 x:0 0 0 2024-04-16T02:35:31.542 INFO:tasks.workunit.client.1.smithi153.stdout:7/204: mkdir d0/d4/db/d16/d41 0 2024-04-16T02:35:31.544 INFO:tasks.workunit.client.1.smithi153.stdout:0/215: dwrite dd/ff [0,4194304] 0 2024-04-16T02:35:31.544 INFO:tasks.workunit.client.1.smithi153.stdout:4/223: unlink d3/f46 0 2024-04-16T02:35:31.549 INFO:tasks.workunit.client.1.smithi153.stdout:9/222: dread d4/d6/d1b/f3e [0,4194304] 0 2024-04-16T02:35:31.552 INFO:tasks.workunit.client.1.smithi153.stdout:6/128: sync 2024-04-16T02:35:31.552 INFO:tasks.workunit.client.1.smithi153.stdout:9/223: readlink d4/d33/l44 0 2024-04-16T02:35:31.553 INFO:tasks.workunit.client.1.smithi153.stdout:2/178: creat d9/d2b/f46 x:0 0 0 2024-04-16T02:35:31.557 INFO:tasks.workunit.client.1.smithi153.stdout:9/224: readlink d4/d6/d1b/l36 0 2024-04-16T02:35:31.558 INFO:tasks.workunit.client.1.smithi153.stdout:6/129: mknod dc/d10/c2b 0 2024-04-16T02:35:31.559 INFO:tasks.workunit.client.1.smithi153.stdout:8/193: dwrite d2/de/f3e [0,4194304] 0 2024-04-16T02:35:31.561 INFO:tasks.workunit.client.1.smithi153.stdout:0/216: dread dd/d32/f39 [0,4194304] 0 2024-04-16T02:35:31.563 INFO:tasks.workunit.client.1.smithi153.stdout:8/194: rename d2/d13 to d2/d13/d45 22 2024-04-16T02:35:31.565 INFO:tasks.workunit.client.1.smithi153.stdout:7/205: dwrite d0/d4/db/d2b/f38 [0,4194304] 0 2024-04-16T02:35:31.566 INFO:tasks.workunit.client.1.smithi153.stdout:9/225: dread d4/f27 [0,4194304] 0 2024-04-16T02:35:31.567 INFO:tasks.workunit.client.1.smithi153.stdout:8/195: chown d2/d1c/d1f/d27 316 1 2024-04-16T02:35:31.568 INFO:tasks.workunit.client.1.smithi153.stdout:6/130: fdatasync dc/d12/d15/f1e 0 2024-04-16T02:35:31.568 INFO:tasks.workunit.client.1.smithi153.stdout:4/224: dwrite d3/d9/d24/f2b [0,4194304] 0 2024-04-16T02:35:31.570 INFO:tasks.workunit.client.1.smithi153.stdout:8/196: write d2/d1c/f26 [123000,23922] 0 2024-04-16T02:35:31.572 INFO:tasks.workunit.client.1.smithi153.stdout:0/217: creat dd/d28/f41 x:0 0 0 2024-04-16T02:35:31.572 INFO:tasks.workunit.client.1.smithi153.stdout:3/202: truncate d6/d10/f19 378622 0 2024-04-16T02:35:31.572 INFO:tasks.workunit.client.1.smithi153.stdout:4/225: dread - d3/d9/db/f4a zero size 2024-04-16T02:35:31.575 INFO:tasks.workunit.client.1.smithi153.stdout:6/131: write dc/d12/f24 [188759,113129] 0 2024-04-16T02:35:31.576 INFO:tasks.workunit.client.1.smithi153.stdout:3/203: fdatasync d6/d10/f1c 0 2024-04-16T02:35:31.577 INFO:tasks.workunit.client.1.smithi153.stdout:5/185: symlink d8/df/d23/l31 0 2024-04-16T02:35:31.578 INFO:tasks.workunit.client.1.smithi153.stdout:8/197: dread d2/d1c/f36 [0,4194304] 0 2024-04-16T02:35:31.578 INFO:tasks.workunit.client.1.smithi153.stdout:0/218: creat dd/d32/d33/f42 x:0 0 0 2024-04-16T02:35:31.579 INFO:tasks.workunit.client.1.smithi153.stdout:6/132: write dc/d12/d15/f22 [327836,27732] 0 2024-04-16T02:35:31.582 INFO:tasks.workunit.client.1.smithi153.stdout:9/226: getdents d4/d33 0 2024-04-16T02:35:31.582 INFO:tasks.workunit.client.1.smithi153.stdout:5/186: read - d8/df/f2c zero size 2024-04-16T02:35:31.583 INFO:tasks.workunit.client.1.smithi153.stdout:7/206: dread d0/f3d [0,4194304] 0 2024-04-16T02:35:31.586 INFO:tasks.workunit.client.1.smithi153.stdout:6/133: readlink dc/l28 0 2024-04-16T02:35:31.586 INFO:tasks.workunit.client.1.smithi153.stdout:9/227: stat d4/d6/dc/d24/f42 0 2024-04-16T02:35:31.587 INFO:tasks.workunit.client.1.smithi153.stdout:8/198: creat d2/d13/d3a/f46 x:0 0 0 2024-04-16T02:35:31.589 INFO:tasks.workunit.client.1.smithi153.stdout:1/182: sync 2024-04-16T02:35:31.590 INFO:tasks.workunit.client.1.smithi153.stdout:2/179: dwrite d9/d24/d27/f1d [4194304,4194304] 0 2024-04-16T02:35:31.590 INFO:tasks.workunit.client.1.smithi153.stdout:5/187: creat d8/df/d23/f32 x:0 0 0 2024-04-16T02:35:31.590 INFO:tasks.workunit.client.1.smithi153.stdout:6/134: rmdir dc/d10 39 2024-04-16T02:35:31.591 INFO:tasks.workunit.client.1.smithi153.stdout:4/226: dread d3/d9/f1a [0,4194304] 0 2024-04-16T02:35:31.591 INFO:tasks.workunit.client.1.smithi153.stdout:9/228: chown d4/d6/dc/l41 0 1 2024-04-16T02:35:31.592 INFO:tasks.workunit.client.1.smithi153.stdout:3/204: link d6/d10/f32 d6/d10/d22/d2e/f45 0 2024-04-16T02:35:31.592 INFO:tasks.workunit.client.1.smithi153.stdout:9/229: chown d4/d6 1510 1 2024-04-16T02:35:31.595 INFO:tasks.workunit.client.1.smithi153.stdout:1/183: mknod d4/dd/d18/d29/c38 0 2024-04-16T02:35:31.597 INFO:tasks.workunit.client.1.smithi153.stdout:6/135: rmdir dc/d12 39 2024-04-16T02:35:31.600 INFO:tasks.workunit.client.1.smithi153.stdout:8/199: dwrite d2/f7 [4194304,4194304] 0 2024-04-16T02:35:31.601 INFO:tasks.workunit.client.1.smithi153.stdout:4/227: creat d3/d9/d24/f4d x:0 0 0 2024-04-16T02:35:31.610 INFO:tasks.workunit.client.1.smithi153.stdout:9/230: rename d4/d6/l14 to d4/d33/l45 0 2024-04-16T02:35:31.610 INFO:tasks.workunit.client.1.smithi153.stdout:2/180: symlink d9/d2b/d1c/l47 0 2024-04-16T02:35:31.610 INFO:tasks.workunit.client.1.smithi153.stdout:5/188: creat d8/df/d2f/f33 x:0 0 0 2024-04-16T02:35:31.610 INFO:tasks.workunit.client.1.smithi153.stdout:1/184: creat d4/dd/d18/d29/f39 x:0 0 0 2024-04-16T02:35:31.611 INFO:tasks.workunit.client.1.smithi153.stdout:5/189: fsync f4 0 2024-04-16T02:35:31.614 INFO:tasks.workunit.client.1.smithi153.stdout:8/200: mkdir d2/d1c/d1f/d27/d2d/d47 0 2024-04-16T02:35:31.614 INFO:tasks.workunit.client.1.smithi153.stdout:6/136: write dc/d12/d15/f19 [1870699,114716] 0 2024-04-16T02:35:31.614 INFO:tasks.workunit.client.1.smithi153.stdout:7/207: link d0/d13/c21 d0/d4/db/c42 0 2024-04-16T02:35:31.615 INFO:tasks.workunit.client.1.smithi153.stdout:3/205: symlink d6/d10/d22/l46 0 2024-04-16T02:35:31.616 INFO:tasks.workunit.client.1.smithi153.stdout:0/219: truncate dd/d14/d19/f1b 1428946 0 2024-04-16T02:35:31.616 INFO:tasks.workunit.client.1.smithi153.stdout:1/185: write d4/dd/d18/d29/f39 [55749,20772] 0 2024-04-16T02:35:31.617 INFO:tasks.workunit.client.1.smithi153.stdout:6/137: dread - dc/d1a/f26 zero size 2024-04-16T02:35:31.618 INFO:tasks.workunit.client.1.smithi153.stdout:4/228: rename d3/d9/db/d17/d28/d43 to d3/d9/db/d17/d28/d4e 0 2024-04-16T02:35:31.620 INFO:tasks.workunit.client.1.smithi153.stdout:2/181: creat d9/d2b/d18/d3f/f48 x:0 0 0 2024-04-16T02:35:31.621 INFO:tasks.workunit.client.1.smithi153.stdout:0/220: read - dd/d32/d33/f3a zero size 2024-04-16T02:35:31.623 INFO:tasks.workunit.client.1.smithi153.stdout:1/186: mknod d4/dd/d18/c3a 0 2024-04-16T02:35:31.624 INFO:tasks.workunit.client.1.smithi153.stdout:6/138: mknod dc/c2c 0 2024-04-16T02:35:31.625 INFO:tasks.workunit.client.1.smithi153.stdout:7/208: write d0/d4/db/d2b/f40 [835125,90870] 0 2024-04-16T02:35:31.625 INFO:tasks.workunit.client.1.smithi153.stdout:9/231: dwrite d4/f27 [0,4194304] 0 2024-04-16T02:35:31.627 INFO:tasks.workunit.client.1.smithi153.stdout:4/229: creat d3/d9/db/d17/d3f/f4f x:0 0 0 2024-04-16T02:35:31.628 INFO:tasks.workunit.client.1.smithi153.stdout:2/182: creat d9/d2b/d18/d3f/f49 x:0 0 0 2024-04-16T02:35:31.632 INFO:tasks.workunit.client.1.smithi153.stdout:9/232: mkdir d4/d6/dc/d46 0 2024-04-16T02:35:31.633 INFO:tasks.workunit.client.1.smithi153.stdout:1/187: mknod d4/dd/d13/c3b 0 2024-04-16T02:35:31.634 INFO:tasks.workunit.client.1.smithi153.stdout:3/206: dwrite d6/d10/f1e [0,4194304] 0 2024-04-16T02:35:31.638 INFO:tasks.workunit.client.1.smithi153.stdout:1/188: fdatasync d4/f22 0 2024-04-16T02:35:31.641 INFO:tasks.workunit.client.1.smithi153.stdout:5/190: truncate d8/df/f11 739110 0 2024-04-16T02:35:31.645 INFO:tasks.workunit.client.1.smithi153.stdout:4/230: creat d3/d9/db/de/d2c/f50 x:0 0 0 2024-04-16T02:35:31.647 INFO:tasks.workunit.client.1.smithi153.stdout:6/139: dwrite dc/d12/f24 [0,4194304] 0 2024-04-16T02:35:31.648 INFO:tasks.workunit.client.1.smithi153.stdout:7/209: getdents d0/d4/db/d16 0 2024-04-16T02:35:31.650 INFO:tasks.workunit.client.1.smithi153.stdout:1/189: mknod d4/dd/d20/c3c 0 2024-04-16T02:35:31.652 INFO:tasks.workunit.client.1.smithi153.stdout:4/231: readlink d3/l4 0 2024-04-16T02:35:31.653 INFO:tasks.workunit.client.1.smithi153.stdout:2/183: dwrite d9/d2b/d18/f29 [0,4194304] 0 2024-04-16T02:35:31.654 INFO:tasks.workunit.client.1.smithi153.stdout:0/221: dread f9 [0,4194304] 0 2024-04-16T02:35:31.656 INFO:tasks.workunit.client.1.smithi153.stdout:3/207: rename d6/c25 to d6/d10/d22/d43/c47 0 2024-04-16T02:35:31.657 INFO:tasks.workunit.client.1.smithi153.stdout:9/233: dwrite d4/d6/dc/d24/f42 [0,4194304] 0 2024-04-16T02:35:31.661 INFO:tasks.workunit.client.1.smithi153.stdout:6/140: mkdir dc/d1a/d2d 0 2024-04-16T02:35:31.661 INFO:tasks.workunit.client.1.smithi153.stdout:7/210: mkdir d0/d4/db/d2b/d43 0 2024-04-16T02:35:31.664 INFO:tasks.workunit.client.1.smithi153.stdout:7/211: readlink d0/d4/l9 0 2024-04-16T02:35:31.668 INFO:tasks.workunit.client.1.smithi153.stdout:5/191: dread d8/df/f11 [0,4194304] 0 2024-04-16T02:35:31.669 INFO:tasks.workunit.client.1.smithi153.stdout:4/232: rmdir d3/d9/db/d17/d28/d4e 0 2024-04-16T02:35:31.670 INFO:tasks.workunit.client.1.smithi153.stdout:0/222: symlink dd/d32/l43 0 2024-04-16T02:35:31.670 INFO:tasks.workunit.client.1.smithi153.stdout:2/184: creat d9/d24/d27/f4a x:0 0 0 2024-04-16T02:35:31.671 INFO:tasks.workunit.client.1.smithi153.stdout:1/190: truncate d4/dd/d13/d26/f33 1707101 0 2024-04-16T02:35:31.671 INFO:tasks.workunit.client.1.smithi153.stdout:5/192: readlink d8/df/d23/l31 0 2024-04-16T02:35:31.672 INFO:tasks.workunit.client.1.smithi153.stdout:3/208: dwrite d6/f2d [0,4194304] 0 2024-04-16T02:35:31.672 INFO:tasks.workunit.client.1.smithi153.stdout:7/212: unlink d0/d4/db/d2b/l23 0 2024-04-16T02:35:31.673 INFO:tasks.workunit.client.1.smithi153.stdout:9/234: creat d4/f47 x:0 0 0 2024-04-16T02:35:31.674 INFO:tasks.workunit.client.1.smithi153.stdout:5/193: dread - d8/df/f2c zero size 2024-04-16T02:35:31.675 INFO:tasks.workunit.client.1.smithi153.stdout:2/185: symlink d9/d24/l4b 0 2024-04-16T02:35:31.675 INFO:tasks.workunit.client.1.smithi153.stdout:0/223: write dd/d28/f41 [376375,93475] 0 2024-04-16T02:35:31.676 INFO:tasks.workunit.client.1.smithi153.stdout:9/235: stat d4/d6/dc/d24/f25 0 2024-04-16T02:35:31.677 INFO:tasks.workunit.client.1.smithi153.stdout:2/186: write d9/d24/d27/f1d [5318593,67965] 0 2024-04-16T02:35:31.677 INFO:tasks.workunit.client.1.smithi153.stdout:7/213: symlink d0/d4/db/d16/l44 0 2024-04-16T02:35:31.678 INFO:tasks.workunit.client.1.smithi153.stdout:0/224: write dd/d14/d19/f27 [588464,41277] 0 2024-04-16T02:35:31.679 INFO:tasks.workunit.client.1.smithi153.stdout:3/209: symlink d6/d10/d3b/l48 0 2024-04-16T02:35:31.680 INFO:tasks.workunit.client.1.smithi153.stdout:2/187: dread d9/d2b/f19 [0,4194304] 0 2024-04-16T02:35:31.683 INFO:tasks.workunit.client.1.smithi153.stdout:1/191: creat d4/dd/d13/d26/f3d x:0 0 0 2024-04-16T02:35:31.684 INFO:tasks.workunit.client.1.smithi153.stdout:7/214: write d0/f3d [3359877,48857] 0 2024-04-16T02:35:31.686 INFO:tasks.workunit.client.1.smithi153.stdout:5/194: mknod d8/df/d2e/c34 0 2024-04-16T02:35:31.686 INFO:tasks.workunit.client.1.smithi153.stdout:2/188: creat d9/d24/d27/d2c/d41/f4c x:0 0 0 2024-04-16T02:35:31.688 INFO:tasks.workunit.client.1.smithi153.stdout:7/215: write d0/d4/db/d2b/f2f [4798386,112571] 0 2024-04-16T02:35:31.688 INFO:tasks.workunit.client.1.smithi153.stdout:4/233: link d3/d9/db/d17/d28/l48 d3/l51 0 2024-04-16T02:35:31.689 INFO:tasks.workunit.client.1.smithi153.stdout:5/195: chown d8/df/d23/c2d 12 1 2024-04-16T02:35:31.689 INFO:tasks.workunit.client.1.smithi153.stdout:3/210: mknod d6/d10/c49 0 2024-04-16T02:35:31.690 INFO:tasks.workunit.client.1.smithi153.stdout:4/234: write d3/d9/f29 [510531,66021] 0 2024-04-16T02:35:31.690 INFO:tasks.workunit.client.1.smithi153.stdout:9/236: rename d4/d6/dc/d24/f28 to d4/f48 0 2024-04-16T02:35:31.691 INFO:tasks.workunit.client.1.smithi153.stdout:9/237: chown d4 1798927175 1 2024-04-16T02:35:31.691 INFO:tasks.workunit.client.1.smithi153.stdout:1/192: symlink d4/dd/d13/d23/l3e 0 2024-04-16T02:35:31.693 INFO:tasks.workunit.client.1.smithi153.stdout:2/189: unlink d9/d24/d27/f2e 0 2024-04-16T02:35:31.694 INFO:tasks.workunit.client.1.smithi153.stdout:2/190: chown d9 6059 1 2024-04-16T02:35:31.697 INFO:tasks.workunit.client.1.smithi153.stdout:9/238: rmdir d4/d33 39 2024-04-16T02:35:31.712 INFO:tasks.workunit.client.1.smithi153.stdout:6/141: dread f4 [0,4194304] 0 2024-04-16T02:35:31.714 INFO:tasks.workunit.client.1.smithi153.stdout:5/196: getdents d8/d1f/d20 0 2024-04-16T02:35:31.715 INFO:tasks.workunit.client.1.smithi153.stdout:7/216: dwrite d0/d4/db/d2b/f40 [0,4194304] 0 2024-04-16T02:35:31.715 INFO:tasks.workunit.client.1.smithi153.stdout:2/191: read d9/d2b/f17 [518836,63291] 0 2024-04-16T02:35:31.717 INFO:tasks.workunit.client.1.smithi153.stdout:1/193: dread d4/f7 [4194304,4194304] 0 2024-04-16T02:35:31.719 INFO:tasks.workunit.client.1.smithi153.stdout:1/194: fsync d4/f36 0 2024-04-16T02:35:31.721 INFO:tasks.workunit.client.1.smithi153.stdout:2/192: symlink d9/d2b/d18/l4d 0 2024-04-16T02:35:31.723 INFO:tasks.workunit.client.1.smithi153.stdout:6/142: write dc/f11 [3334632,77709] 0 2024-04-16T02:35:31.724 INFO:tasks.workunit.client.1.smithi153.stdout:1/195: stat d4/dd/d18/d29/l35 0 2024-04-16T02:35:31.725 INFO:tasks.workunit.client.1.smithi153.stdout:3/211: getdents d6/d10/d22/d2e 0 2024-04-16T02:35:31.727 INFO:tasks.workunit.client.1.smithi153.stdout:6/143: symlink dc/d1a/l2e 0 2024-04-16T02:35:31.729 INFO:tasks.workunit.client.1.smithi153.stdout:0/225: dwrite f9 [4194304,4194304] 0 2024-04-16T02:35:31.732 INFO:tasks.workunit.client.1.smithi153.stdout:5/197: dwrite d8/df/f1e [0,4194304] 0 2024-04-16T02:35:31.733 INFO:tasks.workunit.client.1.smithi153.stdout:2/193: truncate d9/d2b/d18/d3f/f48 551020 0 2024-04-16T02:35:31.739 INFO:tasks.workunit.client.1.smithi153.stdout:3/212: link d6/d10/d22/d2e/f35 d6/d10/d22/d2e/f4a 0 2024-04-16T02:35:31.741 INFO:tasks.workunit.client.1.smithi153.stdout:2/194: chown d9/d24/d27/d2c/d41/f44 8214 1 2024-04-16T02:35:31.741 INFO:tasks.workunit.client.1.smithi153.stdout:5/198: creat d8/df/d28/f35 x:0 0 0 2024-04-16T02:35:31.744 INFO:tasks.workunit.client.1.smithi153.stdout:1/196: dwrite d4/dd/d13/d26/f34 [0,4194304] 0 2024-04-16T02:35:31.745 INFO:tasks.workunit.client.1.smithi153.stdout:6/144: dread f6 [0,4194304] 0 2024-04-16T02:35:31.745 INFO:tasks.workunit.client.1.smithi153.stdout:2/195: chown d9/c22 893461262 1 2024-04-16T02:35:31.747 INFO:tasks.workunit.client.1.smithi153.stdout:2/196: chown d9/d2b/f16 1730 1 2024-04-16T02:35:31.751 INFO:tasks.workunit.client.1.smithi153.stdout:3/213: read - d6/d10/d22/d2c/f44 zero size 2024-04-16T02:35:31.751 INFO:tasks.workunit.client.1.smithi153.stdout:6/145: chown dc/d1a/f26 21219612 1 2024-04-16T02:35:31.752 INFO:tasks.workunit.client.1.smithi153.stdout:0/226: dwrite dd/d14/d19/d1c/f2f [0,4194304] 0 2024-04-16T02:35:31.752 INFO:tasks.workunit.client.1.smithi153.stdout:1/197: unlink c2 0 2024-04-16T02:35:31.755 INFO:tasks.workunit.client.1.smithi153.stdout:6/146: creat dc/d12/d15/f2f x:0 0 0 2024-04-16T02:35:31.759 INFO:tasks.workunit.client.1.smithi153.stdout:5/199: rename d8/c25 to d8/df/c36 0 2024-04-16T02:35:31.759 INFO:tasks.workunit.client.1.smithi153.stdout:8/201: sync 2024-04-16T02:35:31.763 INFO:tasks.workunit.client.1.smithi153.stdout:6/147: dread dc/d12/d15/f1e [0,4194304] 0 2024-04-16T02:35:31.764 INFO:tasks.workunit.client.1.smithi153.stdout:0/227: rename dd/d14/d19/d20/d2a to dd/d26/d44 0 2024-04-16T02:35:31.764 INFO:tasks.workunit.client.1.smithi153.stdout:1/198: link d4/dd/d13/d23/l3e d4/dd/d18/l3f 0 2024-04-16T02:35:31.765 INFO:tasks.workunit.client.1.smithi153.stdout:2/197: dwrite d9/d24/d27/f30 [0,4194304] 0 2024-04-16T02:35:31.769 INFO:tasks.workunit.client.1.smithi153.stdout:1/199: creat d4/f40 x:0 0 0 2024-04-16T02:35:31.770 INFO:tasks.workunit.client.1.smithi153.stdout:2/198: creat d9/d24/d27/d2c/d41/f4e x:0 0 0 2024-04-16T02:35:31.771 INFO:tasks.workunit.client.1.smithi153.stdout:1/200: dread - d4/dd/d13/d26/f3d zero size 2024-04-16T02:35:31.772 INFO:tasks.workunit.client.1.smithi153.stdout:2/199: fsync d9/d2b/d1c/f28 0 2024-04-16T02:35:31.772 INFO:tasks.workunit.client.1.smithi153.stdout:8/202: dwrite d2/f12 [0,4194304] 0 2024-04-16T02:35:31.774 INFO:tasks.workunit.client.1.smithi153.stdout:1/201: chown d4/dd/d13/f14 871124403 1 2024-04-16T02:35:31.775 INFO:tasks.workunit.client.1.smithi153.stdout:1/202: chown d4/dd/d13 1776346 1 2024-04-16T02:35:31.776 INFO:tasks.workunit.client.1.smithi153.stdout:1/203: truncate d4/f22 577743 0 2024-04-16T02:35:31.788 INFO:tasks.workunit.client.1.smithi153.stdout:6/148: dread dc/d12/d15/f19 [0,4194304] 0 2024-04-16T02:35:31.875 INFO:tasks.workunit.client.1.smithi153.stdout:4/235: sync 2024-04-16T02:35:31.916 INFO:tasks.workunit.client.1.smithi153.stdout:9/239: sync 2024-04-16T02:35:31.916 INFO:tasks.workunit.client.1.smithi153.stdout:7/217: sync 2024-04-16T02:35:31.921 INFO:tasks.workunit.client.1.smithi153.stdout:3/214: dwrite d6/d10/d22/d2e/f45 [0,4194304] 0 2024-04-16T02:35:31.923 INFO:tasks.workunit.client.1.smithi153.stdout:4/236: mkdir d3/d9/db/de/d2c/d52 0 2024-04-16T02:35:31.924 INFO:tasks.workunit.client.1.smithi153.stdout:4/237: fdatasync d3/d9/d24/f4d 0 2024-04-16T02:35:31.928 INFO:tasks.workunit.client.1.smithi153.stdout:0/228: truncate dd/f22 1046923 0 2024-04-16T02:35:31.932 INFO:tasks.workunit.client.1.smithi153.stdout:2/200: write f1 [1916968,12808] 0 2024-04-16T02:35:31.936 INFO:tasks.workunit.client.1.smithi153.stdout:3/215: rename d6/d10/l3a to d6/d10/d22/d2c/l4b 0 2024-04-16T02:35:31.936 INFO:tasks.workunit.client.1.smithi153.stdout:6/149: mknod dc/d1a/d2d/c30 0 2024-04-16T02:35:31.939 INFO:tasks.workunit.client.1.smithi153.stdout:0/229: symlink dd/d28/l45 0 2024-04-16T02:35:31.940 INFO:tasks.workunit.client.1.smithi153.stdout:5/200: mkdir d8/df/d2e/d37 0 2024-04-16T02:35:31.941 INFO:tasks.workunit.client.1.smithi153.stdout:6/150: rmdir dc/d12 39 2024-04-16T02:35:31.941 INFO:tasks.workunit.client.1.smithi153.stdout:2/201: dread d9/d2b/f16 [0,4194304] 0 2024-04-16T02:35:31.941 INFO:tasks.workunit.client.1.smithi153.stdout:4/238: symlink d3/d9/db/de/l53 0 2024-04-16T02:35:31.943 INFO:tasks.workunit.client.1.smithi153.stdout:0/230: unlink dd/d32/d33/f3a 0 2024-04-16T02:35:31.945 INFO:tasks.workunit.client.1.smithi153.stdout:3/216: dwrite d6/f39 [0,4194304] 0 2024-04-16T02:35:31.946 INFO:tasks.workunit.client.1.smithi153.stdout:5/201: unlink d8/df/c1d 0 2024-04-16T02:35:31.948 INFO:tasks.workunit.client.1.smithi153.stdout:9/240: getdents d4/d6/dc 0 2024-04-16T02:35:31.950 INFO:tasks.workunit.client.1.smithi153.stdout:8/203: truncate d2/de/f24 623195 0 2024-04-16T02:35:31.955 INFO:tasks.workunit.client.1.smithi153.stdout:3/217: truncate d6/f20 809633 0 2024-04-16T02:35:31.955 INFO:tasks.workunit.client.1.smithi153.stdout:2/202: dwrite d9/d2b/ff [0,4194304] 0 2024-04-16T02:35:31.957 INFO:tasks.workunit.client.1.smithi153.stdout:4/239: dwrite d3/d9/f23 [0,4194304] 0 2024-04-16T02:35:31.959 INFO:tasks.workunit.client.1.smithi153.stdout:8/204: stat d2/c18 0 2024-04-16T02:35:31.962 INFO:tasks.workunit.client.1.smithi153.stdout:7/218: truncate d0/d4/f20 1529707 0 2024-04-16T02:35:31.964 INFO:tasks.workunit.client.1.smithi153.stdout:5/202: readlink d8/d1f/l27 0 2024-04-16T02:35:31.964 INFO:tasks.workunit.client.1.smithi153.stdout:8/205: dread - d2/de/f15 zero size 2024-04-16T02:35:31.969 INFO:tasks.workunit.client.1.smithi153.stdout:6/151: link dc/d12/d15/f19 dc/f31 0 2024-04-16T02:35:31.970 INFO:tasks.workunit.client.1.smithi153.stdout:9/241: dwrite d4/f47 [0,4194304] 0 2024-04-16T02:35:31.970 INFO:tasks.workunit.client.1.smithi153.stdout:8/206: chown d2/de/l22 22788 1 2024-04-16T02:35:31.975 INFO:tasks.workunit.client.1.smithi153.stdout:4/240: symlink d3/d9/db/d17/d28/l54 0 2024-04-16T02:35:31.975 INFO:tasks.workunit.client.1.smithi153.stdout:8/207: chown d2/de/d3f 8075 1 2024-04-16T02:35:31.977 INFO:tasks.workunit.client.1.smithi153.stdout:6/152: write dc/d12/d15/f1e [704963,109845] 0 2024-04-16T02:35:31.979 INFO:tasks.workunit.client.1.smithi153.stdout:8/208: dread - d2/d1c/d1f/f3c zero size 2024-04-16T02:35:31.979 INFO:tasks.workunit.client.1.smithi153.stdout:7/219: chown d0/d4/f20 30799 1 2024-04-16T02:35:31.980 INFO:tasks.workunit.client.1.smithi153.stdout:1/204: dwrite d4/f1f [0,4194304] 0 2024-04-16T02:35:31.982 INFO:tasks.workunit.client.1.smithi153.stdout:2/203: dwrite d9/d2b/d18/f29 [0,4194304] 0 2024-04-16T02:35:31.984 INFO:tasks.workunit.client.1.smithi153.stdout:8/209: dread - d2/de/d3f/f43 zero size 2024-04-16T02:35:31.986 INFO:tasks.workunit.client.1.smithi153.stdout:5/203: getdents d8/df 0 2024-04-16T02:35:31.986 INFO:tasks.workunit.client.1.smithi153.stdout:4/241: readlink d3/d9/db/d17/l36 0 2024-04-16T02:35:31.987 INFO:tasks.workunit.client.1.smithi153.stdout:6/153: symlink dc/d10/l32 0 2024-04-16T02:35:31.987 INFO:tasks.workunit.client.1.smithi153.stdout:9/242: dread d4/f27 [0,4194304] 0 2024-04-16T02:35:31.991 INFO:tasks.workunit.client.1.smithi153.stdout:8/210: fdatasync d2/d13/d3a/f46 0 2024-04-16T02:35:31.991 INFO:tasks.workunit.client.1.smithi153.stdout:5/204: fsync d8/df/f1e 0 2024-04-16T02:35:31.991 INFO:tasks.workunit.client.1.smithi153.stdout:5/205: chown d8/df/d23 384768 1 2024-04-16T02:35:31.992 INFO:tasks.workunit.client.1.smithi153.stdout:4/242: chown d3/d9/db/d17/d28 1026 1 2024-04-16T02:35:31.993 INFO:tasks.workunit.client.1.smithi153.stdout:6/154: truncate dc/d12/d15/f22 968411 0 2024-04-16T02:35:31.994 INFO:tasks.workunit.client.1.smithi153.stdout:6/155: stat dc/f31 0 2024-04-16T02:35:31.994 INFO:tasks.workunit.client.1.smithi153.stdout:7/220: rename d0/d4/db/d16/l44 to d0/d4/db/l45 0 2024-04-16T02:35:31.995 INFO:tasks.workunit.client.1.smithi153.stdout:0/231: truncate f9 2909418 0 2024-04-16T02:35:31.997 INFO:tasks.workunit.client.1.smithi153.stdout:1/205: write d4/dd/d13/f14 [191485,19272] 0 2024-04-16T02:35:31.999 INFO:tasks.workunit.client.1.smithi153.stdout:9/243: chown d4/d33/f3f 694 1 2024-04-16T02:35:31.999 INFO:tasks.workunit.client.1.smithi153.stdout:1/206: chown d4/dd/d13/f2a 59 1 2024-04-16T02:35:31.999 INFO:tasks.workunit.client.1.smithi153.stdout:2/204: unlink d9/d24/d27/c35 0 2024-04-16T02:35:32.001 INFO:tasks.workunit.client.1.smithi153.stdout:8/211: readlink d2/d1c/l33 0 2024-04-16T02:35:32.001 INFO:tasks.workunit.client.1.smithi153.stdout:3/218: dread d6/d10/d22/d2e/f4a [0,4194304] 0 2024-04-16T02:35:32.001 INFO:tasks.workunit.client.1.smithi153.stdout:6/156: fdatasync dc/f31 0 2024-04-16T02:35:32.002 INFO:tasks.workunit.client.1.smithi153.stdout:3/219: dread - d6/d10/d3b/f41 zero size 2024-04-16T02:35:32.005 INFO:tasks.workunit.client.1.smithi153.stdout:7/221: creat d0/d4/db/d26/f46 x:0 0 0 2024-04-16T02:35:32.005 INFO:tasks.workunit.client.1.smithi153.stdout:4/243: dread d3/d9/d24/f3a [0,4194304] 0 2024-04-16T02:35:32.006 INFO:tasks.workunit.client.1.smithi153.stdout:0/232: unlink dd/d14/f37 0 2024-04-16T02:35:32.007 INFO:tasks.workunit.client.1.smithi153.stdout:2/205: creat d9/d24/d34/f4f x:0 0 0 2024-04-16T02:35:32.008 INFO:tasks.workunit.client.1.smithi153.stdout:7/222: write d0/f1e [4599923,4049] 0 2024-04-16T02:35:32.011 INFO:tasks.workunit.client.1.smithi153.stdout:8/212: creat d2/d13/d2c/f48 x:0 0 0 2024-04-16T02:35:32.011 INFO:tasks.workunit.client.1.smithi153.stdout:6/157: dwrite dc/d12/d15/f2f [0,4194304] 0 2024-04-16T02:35:32.013 INFO:tasks.workunit.client.1.smithi153.stdout:5/206: creat d8/df/d2e/d37/f38 x:0 0 0 2024-04-16T02:35:32.015 INFO:tasks.workunit.client.1.smithi153.stdout:5/207: chown d8/df/d2e/d37 481 1 2024-04-16T02:35:32.016 INFO:tasks.workunit.client.1.smithi153.stdout:6/158: write dc/d12/d15/f20 [830570,32270] 0 2024-04-16T02:35:32.017 INFO:tasks.workunit.client.1.smithi153.stdout:4/244: rmdir d3 39 2024-04-16T02:35:32.018 INFO:tasks.workunit.client.1.smithi153.stdout:6/159: chown dc/f21 12552 1 2024-04-16T02:35:32.021 INFO:tasks.workunit.client.1.smithi153.stdout:7/223: dwrite d0/d4/db/d2b/f38 [0,4194304] 0 2024-04-16T02:35:32.022 INFO:tasks.workunit.client.1.smithi153.stdout:9/244: creat d4/d6/dc/d46/f49 x:0 0 0 2024-04-16T02:35:32.022 INFO:tasks.workunit.client.1.smithi153.stdout:0/233: dread dd/f16 [0,4194304] 0 2024-04-16T02:35:32.023 INFO:tasks.workunit.client.1.smithi153.stdout:2/206: symlink d9/d2b/d1c/d20/l50 0 2024-04-16T02:35:32.029 INFO:tasks.workunit.client.1.smithi153.stdout:5/208: rmdir d8/df/d2f 39 2024-04-16T02:35:32.032 INFO:tasks.workunit.client.1.smithi153.stdout:6/160: creat dc/d10/f33 x:0 0 0 2024-04-16T02:35:32.032 INFO:tasks.workunit.client.1.smithi153.stdout:0/234: dwrite dd/d14/d19/f27 [0,4194304] 0 2024-04-16T02:35:32.034 INFO:tasks.workunit.client.1.smithi153.stdout:7/224: unlink d0/d4/l5 0 2024-04-16T02:35:32.037 INFO:tasks.workunit.client.1.smithi153.stdout:7/225: chown d0/d4/db/d16/d41 63735487 1 2024-04-16T02:35:32.038 INFO:tasks.workunit.client.1.smithi153.stdout:3/220: creat d6/d10/d22/f4c x:0 0 0 2024-04-16T02:35:32.040 INFO:tasks.workunit.client.1.smithi153.stdout:4/245: creat d3/d9/db/d17/f55 x:0 0 0 2024-04-16T02:35:32.040 INFO:tasks.workunit.client.1.smithi153.stdout:1/207: link d4/c2f d4/dd/c41 0 2024-04-16T02:35:32.040 INFO:tasks.workunit.client.1.smithi153.stdout:1/208: stat d4/dd/d13 0 2024-04-16T02:35:32.043 INFO:tasks.workunit.client.1.smithi153.stdout:9/245: mkdir d4/d33/d3a/d40/d4a 0 2024-04-16T02:35:32.044 INFO:tasks.workunit.client.1.smithi153.stdout:7/226: mkdir d0/d4/db/d47 0 2024-04-16T02:35:32.045 INFO:tasks.workunit.client.1.smithi153.stdout:2/207: mknod d9/d24/d27/c51 0 2024-04-16T02:35:32.046 INFO:tasks.workunit.client.1.smithi153.stdout:5/209: fsync d8/df/d2e/d37/f38 0 2024-04-16T02:35:32.046 INFO:tasks.workunit.client.1.smithi153.stdout:4/246: creat d3/d9/db/de/d2c/f56 x:0 0 0 2024-04-16T02:35:32.047 INFO:tasks.workunit.client.1.smithi153.stdout:4/247: chown d3/d9/c2f 3573 1 2024-04-16T02:35:32.047 INFO:tasks.workunit.client.1.smithi153.stdout:0/235: truncate dd/f22 86029 0 2024-04-16T02:35:32.049 INFO:tasks.workunit.client.1.smithi153.stdout:6/161: link dc/d1a/f1c dc/d10/f34 0 2024-04-16T02:35:32.049 INFO:tasks.workunit.client.1.smithi153.stdout:0/236: dread dd/d32/f39 [0,4194304] 0 2024-04-16T02:35:32.049 INFO:tasks.workunit.client.1.smithi153.stdout:9/246: write d4/d6/d1b/f3e [263819,63595] 0 2024-04-16T02:35:32.050 INFO:tasks.workunit.client.1.smithi153.stdout:3/221: mkdir d6/d10/d22/d4d 0 2024-04-16T02:35:32.052 INFO:tasks.workunit.client.1.smithi153.stdout:1/209: creat d4/dd/d18/f42 x:0 0 0 2024-04-16T02:35:32.052 INFO:tasks.workunit.client.1.smithi153.stdout:7/227: mknod d0/d4/db/d47/c48 0 2024-04-16T02:35:32.052 INFO:tasks.workunit.client.1.smithi153.stdout:3/222: write d6/d10/d22/d2c/f3d [486132,53386] 0 2024-04-16T02:35:32.054 INFO:tasks.workunit.client.1.smithi153.stdout:7/228: chown d0/fa 300 1 2024-04-16T02:35:32.055 INFO:tasks.workunit.client.1.smithi153.stdout:2/208: mknod d9/d2b/c52 0 2024-04-16T02:35:32.056 INFO:tasks.workunit.client.1.smithi153.stdout:6/162: write dc/d10/f34 [575616,25296] 0 2024-04-16T02:35:32.056 INFO:tasks.workunit.client.1.smithi153.stdout:4/248: mknod d3/d9/c57 0 2024-04-16T02:35:32.058 INFO:tasks.workunit.client.1.smithi153.stdout:9/247: symlink d4/d6/dc/d2d/l4b 0 2024-04-16T02:35:32.058 INFO:tasks.workunit.client.1.smithi153.stdout:6/163: readlink dc/d1a/l2e 0 2024-04-16T02:35:32.059 INFO:tasks.workunit.client.1.smithi153.stdout:3/223: dread d6/d10/f13 [0,4194304] 0 2024-04-16T02:35:32.060 INFO:tasks.workunit.client.1.smithi153.stdout:2/209: fsync d9/d24/d27/d2c/d41/f44 0 2024-04-16T02:35:32.060 INFO:tasks.workunit.client.1.smithi153.stdout:9/248: chown d4/d6/dc/d2d 173633828 1 2024-04-16T02:35:32.060 INFO:tasks.workunit.client.1.smithi153.stdout:1/210: symlink d4/dd/d13/d23/d31/l43 0 2024-04-16T02:35:32.062 INFO:tasks.workunit.client.1.smithi153.stdout:1/211: fsync d4/fe 0 2024-04-16T02:35:32.062 INFO:tasks.workunit.client.1.smithi153.stdout:4/249: unlink d3/d9/d24/f3a 0 2024-04-16T02:35:32.063 INFO:tasks.workunit.client.1.smithi153.stdout:2/210: readlink d9/d24/d27/l38 0 2024-04-16T02:35:32.063 INFO:tasks.workunit.client.1.smithi153.stdout:5/210: dwrite d8/df/d2f/f33 [0,4194304] 0 2024-04-16T02:35:32.064 INFO:tasks.workunit.client.1.smithi153.stdout:6/164: mknod dc/d10/c35 0 2024-04-16T02:35:32.065 INFO:tasks.workunit.client.1.smithi153.stdout:4/250: dread - d3/d9/db/d17/f47 zero size 2024-04-16T02:35:32.065 INFO:tasks.workunit.client.1.smithi153.stdout:1/212: write d4/f1f [133531,53755] 0 2024-04-16T02:35:32.065 INFO:tasks.workunit.client.1.smithi153.stdout:6/165: chown dc/d10 169828 1 2024-04-16T02:35:32.067 INFO:tasks.workunit.client.1.smithi153.stdout:1/213: fdatasync d4/dd/d13/f1a 0 2024-04-16T02:35:32.069 INFO:tasks.workunit.client.1.smithi153.stdout:3/224: write d6/d10/d22/d2e/f4a [7921669,21283] 0 2024-04-16T02:35:32.071 INFO:tasks.workunit.client.1.smithi153.stdout:9/249: fdatasync d4/d6/dc/d24/f43 0 2024-04-16T02:35:32.072 INFO:tasks.workunit.client.1.smithi153.stdout:9/250: dread - d4/d6/dc/d24/f30 zero size 2024-04-16T02:35:32.075 INFO:tasks.workunit.client.1.smithi153.stdout:5/211: dread - d8/df/f24 zero size 2024-04-16T02:35:32.075 INFO:tasks.workunit.client.1.smithi153.stdout:2/211: creat d9/d24/d27/d2c/d41/f53 x:0 0 0 2024-04-16T02:35:32.080 INFO:tasks.workunit.client.1.smithi153.stdout:9/251: dwrite d4/f13 [0,4194304] 0 2024-04-16T02:35:32.081 INFO:tasks.workunit.client.1.smithi153.stdout:6/166: link dc/ff dc/d1a/f36 0 2024-04-16T02:35:32.082 INFO:tasks.workunit.client.1.smithi153.stdout:1/214: symlink d4/dd/d20/l44 0 2024-04-16T02:35:32.084 INFO:tasks.workunit.client.1.smithi153.stdout:7/229: link d0/d4/db/d2b/c24 d0/d4/c49 0 2024-04-16T02:35:32.086 INFO:tasks.workunit.client.1.smithi153.stdout:4/251: creat d3/d9/f58 x:0 0 0 2024-04-16T02:35:32.087 INFO:tasks.workunit.client.1.smithi153.stdout:6/167: creat dc/d10/f37 x:0 0 0 2024-04-16T02:35:32.088 INFO:tasks.workunit.client.1.smithi153.stdout:2/212: link d9/d2b/f19 d9/d2b/d1c/d20/d45/f54 0 2024-04-16T02:35:32.089 INFO:tasks.workunit.client.1.smithi153.stdout:3/225: dwrite d6/d10/f13 [0,4194304] 0 2024-04-16T02:35:32.089 INFO:tasks.workunit.client.1.smithi153.stdout:4/252: dread - d3/d9/db/de/d2c/f50 zero size 2024-04-16T02:35:32.092 INFO:tasks.workunit.client.1.smithi153.stdout:0/237: write f9 [1360278,3464] 0 2024-04-16T02:35:32.092 INFO:tasks.workunit.client.1.smithi153.stdout:2/213: dread - d9/d24/d27/d2c/d41/f4e zero size 2024-04-16T02:35:32.093 INFO:tasks.workunit.client.1.smithi153.stdout:5/212: dread f4 [0,4194304] 0 2024-04-16T02:35:32.096 INFO:tasks.workunit.client.1.smithi153.stdout:7/230: chown d0/d4/db/c42 0 1 2024-04-16T02:35:32.096 INFO:tasks.workunit.client.1.smithi153.stdout:1/215: symlink d4/dd/d18/d29/l45 0 2024-04-16T02:35:32.098 INFO:tasks.workunit.client.1.smithi153.stdout:6/168: mknod dc/d12/c38 0 2024-04-16T02:35:32.100 INFO:tasks.workunit.client.1.smithi153.stdout:0/238: write dd/d14/d19/d1c/f3e [2089368,76801] 0 2024-04-16T02:35:32.100 INFO:tasks.workunit.client.1.smithi153.stdout:5/213: write d8/f30 [663298,63562] 0 2024-04-16T02:35:32.100 INFO:tasks.workunit.client.1.smithi153.stdout:7/231: readlink d0/d4/db/d34/l3a 0 2024-04-16T02:35:32.101 INFO:tasks.workunit.client.1.smithi153.stdout:9/252: getdents d4/d6/d1b/d2a 0 2024-04-16T02:35:32.103 INFO:tasks.workunit.client.1.smithi153.stdout:6/169: mknod dc/d1a/c39 0 2024-04-16T02:35:32.105 INFO:tasks.workunit.client.1.smithi153.stdout:9/253: unlink d4/f23 0 2024-04-16T02:35:32.108 INFO:tasks.workunit.client.1.smithi153.stdout:4/253: dwrite d3/d9/db/f4a [0,4194304] 0 2024-04-16T02:35:32.108 INFO:tasks.workunit.client.1.smithi153.stdout:2/214: symlink d9/d2b/d18/d3f/d40/l55 0 2024-04-16T02:35:32.109 INFO:tasks.workunit.client.1.smithi153.stdout:6/170: dread - dc/d10/f37 zero size 2024-04-16T02:35:32.110 INFO:tasks.workunit.client.1.smithi153.stdout:5/214: dread - d8/df/d23/f32 zero size 2024-04-16T02:35:32.110 INFO:tasks.workunit.client.1.smithi153.stdout:7/232: truncate d0/d4/db/f39 274940 0 2024-04-16T02:35:32.111 INFO:tasks.workunit.client.1.smithi153.stdout:6/171: stat dc/f21 0 2024-04-16T02:35:32.113 INFO:tasks.workunit.client.1.smithi153.stdout:7/233: chown d0/d4/db/d2b/f2f 3 1 2024-04-16T02:35:32.113 INFO:tasks.workunit.client.1.smithi153.stdout:1/216: dread d4/dd/d13/d26/f2b [0,4194304] 0 2024-04-16T02:35:32.113 INFO:tasks.workunit.client.1.smithi153.stdout:0/239: dwrite dd/fe [0,4194304] 0 2024-04-16T02:35:32.117 INFO:tasks.workunit.client.1.smithi153.stdout:8/213: sync 2024-04-16T02:35:32.120 INFO:tasks.workunit.client.1.smithi153.stdout:6/172: creat dc/d1a/f3a x:0 0 0 2024-04-16T02:35:32.123 INFO:tasks.workunit.client.1.smithi153.stdout:7/234: creat d0/d13/f4a x:0 0 0 2024-04-16T02:35:32.123 INFO:tasks.workunit.client.1.smithi153.stdout:6/173: chown f4 13 1 2024-04-16T02:35:32.123 INFO:tasks.workunit.client.1.smithi153.stdout:5/215: write d8/df/d28/f35 [882502,50584] 0 2024-04-16T02:35:32.125 INFO:tasks.workunit.client.1.smithi153.stdout:0/240: rename dd/d14/d19/d1c/f23 to dd/d14/d19/f46 0 2024-04-16T02:35:32.130 INFO:tasks.workunit.client.1.smithi153.stdout:4/254: link d3/cd d3/d9/db/d17/d3f/c59 0 2024-04-16T02:35:32.130 INFO:tasks.workunit.client.1.smithi153.stdout:9/254: dwrite d4/f48 [0,4194304] 0 2024-04-16T02:35:32.131 INFO:tasks.workunit.client.1.smithi153.stdout:8/214: symlink d2/d1c/d1f/d27/d2d/l49 0 2024-04-16T02:35:32.134 INFO:tasks.workunit.client.1.smithi153.stdout:2/215: dwrite d9/d2b/f16 [0,4194304] 0 2024-04-16T02:35:32.134 INFO:tasks.workunit.client.1.smithi153.stdout:1/217: dwrite d4/f1f [0,4194304] 0 2024-04-16T02:35:32.134 INFO:tasks.workunit.client.1.smithi153.stdout:0/241: chown dd/d14/d19/f1b 3474 1 2024-04-16T02:35:32.136 INFO:tasks.workunit.client.1.smithi153.stdout:4/255: chown d3/l16 220673 1 2024-04-16T02:35:32.137 INFO:tasks.workunit.client.1.smithi153.stdout:0/242: readlink dd/d32/l43 0 2024-04-16T02:35:32.140 INFO:tasks.workunit.client.1.smithi153.stdout:9/255: write d4/d6/dc/d24/f29 [410895,71048] 0 2024-04-16T02:35:32.146 INFO:tasks.workunit.client.1.smithi153.stdout:2/216: fdatasync d9/d24/d27/d2c/d41/f4e 0 2024-04-16T02:35:32.148 INFO:tasks.workunit.client.1.smithi153.stdout:6/174: link dc/d12/c38 dc/d1a/d2d/c3b 0 2024-04-16T02:35:32.149 INFO:tasks.workunit.client.1.smithi153.stdout:3/226: dwrite d6/d10/d22/f28 [0,4194304] 0 2024-04-16T02:35:32.150 INFO:tasks.workunit.client.1.smithi153.stdout:5/216: dwrite f4 [0,4194304] 0 2024-04-16T02:35:32.153 INFO:tasks.workunit.client.1.smithi153.stdout:1/218: mknod d4/dd/d13/d23/d31/c46 0 2024-04-16T02:35:32.153 INFO:tasks.workunit.client.1.smithi153.stdout:6/175: readlink dc/d10/l32 0 2024-04-16T02:35:32.153 INFO:tasks.workunit.client.1.smithi153.stdout:4/256: creat d3/d9/d24/d3e/f5a x:0 0 0 2024-04-16T02:35:32.154 INFO:tasks.workunit.client.1.smithi153.stdout:8/215: dwrite d2/de/f3e [0,4194304] 0 2024-04-16T02:35:32.160 INFO:tasks.workunit.client.1.smithi153.stdout:2/217: creat d9/d24/d27/d2c/d41/f56 x:0 0 0 2024-04-16T02:35:32.163 INFO:tasks.workunit.client.1.smithi153.stdout:2/218: chown d9/d2b/d1c/d20/d45 16124 1 2024-04-16T02:35:32.164 INFO:tasks.workunit.client.1.smithi153.stdout:1/219: mknod d4/dd/d13/c47 0 2024-04-16T02:35:32.166 INFO:tasks.workunit.client.1.smithi153.stdout:1/220: write d4/f36 [162989,3479] 0 2024-04-16T02:35:32.168 INFO:tasks.workunit.client.1.smithi153.stdout:3/227: dwrite d6/d10/f1c [0,4194304] 0 2024-04-16T02:35:32.169 INFO:tasks.workunit.client.1.smithi153.stdout:4/257: dread d3/d9/f23 [0,4194304] 0 2024-04-16T02:35:32.170 INFO:tasks.workunit.client.1.smithi153.stdout:3/228: truncate d6/d10/d22/f28 5209984 0 2024-04-16T02:35:32.171 INFO:tasks.workunit.client.1.smithi153.stdout:4/258: chown d3/d9/db/d17/l1e 2 1 2024-04-16T02:35:32.172 INFO:tasks.workunit.client.1.smithi153.stdout:3/229: dread - d6/d10/d22/d2c/f44 zero size 2024-04-16T02:35:32.179 INFO:tasks.workunit.client.1.smithi153.stdout:3/230: dread d6/d10/d22/d2c/f33 [0,4194304] 0 2024-04-16T02:35:32.179 INFO:tasks.workunit.client.1.smithi153.stdout:3/231: chown d6/d10/f27 57164 1 2024-04-16T02:35:32.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:31 smithi151 ceph-mon[29003]: pgmap v13: 65 pgs: 65 active+clean; 234 MiB data, 988 MiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 281 op/s 2024-04-16T02:35:32.265 INFO:tasks.workunit.client.1.smithi153.stdout:0/243: sync 2024-04-16T02:35:32.266 INFO:tasks.workunit.client.1.smithi153.stdout:9/256: sync 2024-04-16T02:35:32.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:31 smithi153 ceph-mon[36747]: pgmap v13: 65 pgs: 65 active+clean; 234 MiB data, 988 MiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 281 op/s 2024-04-16T02:35:32.439 INFO:tasks.workunit.client.1.smithi153.stdout:8/216: link d2/de/l41 d2/d13/d2c/l4a 0 2024-04-16T02:35:32.446 INFO:tasks.workunit.client.1.smithi153.stdout:7/235: truncate d0/d4/db/d2b/f25 1804666 0 2024-04-16T02:35:32.447 INFO:tasks.workunit.client.1.smithi153.stdout:7/236: chown d0/d4/db/d26 37 1 2024-04-16T02:35:32.449 INFO:tasks.workunit.client.1.smithi153.stdout:6/176: rename dc/ld to dc/l3c 0 2024-04-16T02:35:32.450 INFO:tasks.workunit.client.1.smithi153.stdout:6/177: fsync dc/d12/d15/f22 0 2024-04-16T02:35:32.454 INFO:tasks.workunit.client.1.smithi153.stdout:2/219: dread f6 [0,4194304] 0 2024-04-16T02:35:32.481 INFO:tasks.workunit.client.1.smithi153.stdout:3/232: creat d6/d10/d22/d2e/f4e x:0 0 0 2024-04-16T02:35:32.483 INFO:tasks.workunit.client.1.smithi153.stdout:5/217: truncate d8/df/d2f/f33 2101817 0 2024-04-16T02:35:32.484 INFO:tasks.workunit.client.1.smithi153.stdout:8/217: mkdir d2/d1c/d1f/d27/d2d/d4b 0 2024-04-16T02:35:32.486 INFO:tasks.workunit.client.1.smithi153.stdout:7/237: mkdir d0/d4/db/d34/d4b 0 2024-04-16T02:35:32.489 INFO:tasks.workunit.client.1.smithi153.stdout:2/220: mknod d9/d2b/d18/d3f/d40/c57 0 2024-04-16T02:35:32.489 INFO:tasks.workunit.client.1.smithi153.stdout:0/244: write dd/ff [808329,86557] 0 2024-04-16T02:35:32.490 INFO:tasks.workunit.client.1.smithi153.stdout:0/245: readlink dd/d32/d33/l3d 0 2024-04-16T02:35:32.493 INFO:tasks.workunit.client.1.smithi153.stdout:7/238: dwrite d0/d4/db/d26/f46 [0,4194304] 0 2024-04-16T02:35:32.493 INFO:tasks.workunit.client.1.smithi153.stdout:3/233: symlink d6/d10/d22/d2c/l4f 0 2024-04-16T02:35:32.495 INFO:tasks.workunit.client.1.smithi153.stdout:8/218: creat d2/d13/d3a/f4c x:0 0 0 2024-04-16T02:35:32.498 INFO:tasks.workunit.client.1.smithi153.stdout:2/221: dwrite d9/d2b/d1c/f28 [0,4194304] 0 2024-04-16T02:35:32.500 INFO:tasks.workunit.client.1.smithi153.stdout:4/259: rename d3/d9/db/d17/d28/c34 to d3/d9/db/de/c5b 0 2024-04-16T02:35:32.502 INFO:tasks.workunit.client.1.smithi153.stdout:0/246: creat dd/f47 x:0 0 0 2024-04-16T02:35:32.502 INFO:tasks.workunit.client.1.smithi153.stdout:4/260: chown d3/d9/db/d17/d28/c37 8622 1 2024-04-16T02:35:32.504 INFO:tasks.workunit.client.1.smithi153.stdout:3/234: symlink d6/d10/d3b/l50 0 2024-04-16T02:35:32.506 INFO:tasks.workunit.client.1.smithi153.stdout:1/221: getdents d4/dd/d13/d26 0 2024-04-16T02:35:32.506 INFO:tasks.workunit.client.1.smithi153.stdout:9/257: rename d4/d6/d1b/d2a/l2e to d4/d33/d3a/l4c 0 2024-04-16T02:35:32.507 INFO:tasks.workunit.client.1.smithi153.stdout:2/222: creat d9/d2b/d18/d3f/f58 x:0 0 0 2024-04-16T02:35:32.509 INFO:tasks.workunit.client.1.smithi153.stdout:3/235: symlink d6/l51 0 2024-04-16T02:35:32.510 INFO:tasks.workunit.client.1.smithi153.stdout:3/236: chown d6/d10/d22/f4c 11 1 2024-04-16T02:35:32.511 INFO:tasks.workunit.client.1.smithi153.stdout:9/258: write d4/f27 [193498,32584] 0 2024-04-16T02:35:32.512 INFO:tasks.workunit.client.1.smithi153.stdout:6/178: rename dc/d10/c2b to dc/d12/c3d 0 2024-04-16T02:35:32.513 INFO:tasks.workunit.client.1.smithi153.stdout:1/222: symlink d4/dd/d18/d29/l48 0 2024-04-16T02:35:32.516 INFO:tasks.workunit.client.1.smithi153.stdout:3/237: truncate d6/d10/d22/f31 674432 0 2024-04-16T02:35:32.516 INFO:tasks.workunit.client.1.smithi153.stdout:9/259: mknod d4/d6/d1b/c4d 0 2024-04-16T02:35:32.518 INFO:tasks.workunit.client.1.smithi153.stdout:5/218: rename d8/df/d2e/d37 to d8/d1f/d20/d39 0 2024-04-16T02:35:32.519 INFO:tasks.workunit.client.1.smithi153.stdout:6/179: symlink dc/d12/d15/l3e 0 2024-04-16T02:35:32.521 INFO:tasks.workunit.client.1.smithi153.stdout:0/247: creat dd/d14/d19/d1c/f48 x:0 0 0 2024-04-16T02:35:32.522 INFO:tasks.workunit.client.1.smithi153.stdout:4/261: creat d3/d9/db/de/f5c x:0 0 0 2024-04-16T02:35:32.523 INFO:tasks.workunit.client.1.smithi153.stdout:2/223: mknod d9/d2b/d1c/d20/d45/c59 0 2024-04-16T02:35:32.523 INFO:tasks.workunit.client.1.smithi153.stdout:0/248: dread dd/d14/d19/f1b [0,4194304] 0 2024-04-16T02:35:32.524 INFO:tasks.workunit.client.1.smithi153.stdout:9/260: mkdir d4/d6/dc/d24/d4e 0 2024-04-16T02:35:32.526 INFO:tasks.workunit.client.1.smithi153.stdout:3/238: rename d6/d10/d3b/f41 to d6/f52 0 2024-04-16T02:35:32.527 INFO:tasks.workunit.client.1.smithi153.stdout:5/219: read d8/df/f11 [86246,59269] 0 2024-04-16T02:35:32.529 INFO:tasks.workunit.client.1.smithi153.stdout:4/262: chown d3/l49 786707 1 2024-04-16T02:35:32.530 INFO:tasks.workunit.client.1.smithi153.stdout:2/224: mknod d9/d24/d27/d2c/c5a 0 2024-04-16T02:35:32.532 INFO:tasks.workunit.client.1.smithi153.stdout:9/261: creat d4/d33/f4f x:0 0 0 2024-04-16T02:35:32.532 INFO:tasks.workunit.client.1.smithi153.stdout:4/263: truncate d3/d9/f58 400008 0 2024-04-16T02:35:32.533 INFO:tasks.workunit.client.1.smithi153.stdout:1/223: creat d4/f49 x:0 0 0 2024-04-16T02:35:32.534 INFO:tasks.workunit.client.1.smithi153.stdout:2/225: write d9/d24/d27/d2c/d41/f44 [2854732,51575] 0 2024-04-16T02:35:32.535 INFO:tasks.workunit.client.1.smithi153.stdout:3/239: read d6/d10/f19 [120039,39759] 0 2024-04-16T02:35:32.536 INFO:tasks.workunit.client.1.smithi153.stdout:1/224: chown d4/l5 15586 1 2024-04-16T02:35:32.537 INFO:tasks.workunit.client.1.smithi153.stdout:0/249: creat dd/d14/f49 x:0 0 0 2024-04-16T02:35:32.537 INFO:tasks.workunit.client.1.smithi153.stdout:5/220: dwrite d8/df/f24 [0,4194304] 0 2024-04-16T02:35:32.537 INFO:tasks.workunit.client.1.smithi153.stdout:4/264: creat d3/d9/db/d17/f5d x:0 0 0 2024-04-16T02:35:32.538 INFO:tasks.workunit.client.1.smithi153.stdout:8/219: dread d2/d13/f1b [4194304,4194304] 0 2024-04-16T02:35:32.539 INFO:tasks.workunit.client.1.smithi153.stdout:5/221: read f4 [410101,102343] 0 2024-04-16T02:35:32.543 INFO:tasks.workunit.client.1.smithi153.stdout:1/225: creat d4/dd/d18/d29/f4a x:0 0 0 2024-04-16T02:35:32.544 INFO:tasks.workunit.client.1.smithi153.stdout:3/240: mknod d6/d10/d22/d2c/c53 0 2024-04-16T02:35:32.545 INFO:tasks.workunit.client.1.smithi153.stdout:1/226: truncate d4/f40 5299 0 2024-04-16T02:35:32.545 INFO:tasks.workunit.client.1.smithi153.stdout:8/220: creat d2/d1c/f4d x:0 0 0 2024-04-16T02:35:32.546 INFO:tasks.workunit.client.1.smithi153.stdout:9/262: dwrite d4/d6/dc/d24/f30 [0,4194304] 0 2024-04-16T02:35:32.547 INFO:tasks.workunit.client.1.smithi153.stdout:8/221: write d2/f7 [5224332,104129] 0 2024-04-16T02:35:32.548 INFO:tasks.workunit.client.1.smithi153.stdout:1/227: write d4/dd/d13/d26/f3d [1045241,93846] 0 2024-04-16T02:35:32.548 INFO:tasks.workunit.client.1.smithi153.stdout:0/250: mkdir dd/d14/d19/d20/d4a 0 2024-04-16T02:35:32.551 INFO:tasks.workunit.client.1.smithi153.stdout:1/228: readlink d4/dd/d13/l17 0 2024-04-16T02:35:32.553 INFO:tasks.workunit.client.1.smithi153.stdout:4/265: rename d3/d9/db/d17/f47 to d3/d9/db/d17/d28/f5e 0 2024-04-16T02:35:32.554 INFO:tasks.workunit.client.1.smithi153.stdout:3/241: creat d6/d10/d22/d2e/f54 x:0 0 0 2024-04-16T02:35:32.554 INFO:tasks.workunit.client.1.smithi153.stdout:1/229: truncate d4/dd/d13/d23/f27 1241736 0 2024-04-16T02:35:32.556 INFO:tasks.workunit.client.1.smithi153.stdout:5/222: dread d8/df/f1e [0,4194304] 0 2024-04-16T02:35:32.556 INFO:tasks.workunit.client.1.smithi153.stdout:5/223: stat f4 0 2024-04-16T02:35:32.558 INFO:tasks.workunit.client.1.smithi153.stdout:9/263: mknod d4/d33/d3a/d40/d4a/c50 0 2024-04-16T02:35:32.562 INFO:tasks.workunit.client.1.smithi153.stdout:0/251: truncate dd/f22 1123954 0 2024-04-16T02:35:32.562 INFO:tasks.workunit.client.1.smithi153.stdout:0/252: write dd/d14/d19/d1c/f36 [671207,4889] 0 2024-04-16T02:35:32.564 INFO:tasks.workunit.client.1.smithi153.stdout:9/264: mknod d4/d33/d3a/d40/c51 0 2024-04-16T02:35:32.566 INFO:tasks.workunit.client.1.smithi153.stdout:4/266: rename d3/d9/db/c22 to d3/d9/d24/d35/c5f 0 2024-04-16T02:35:32.567 INFO:tasks.workunit.client.1.smithi153.stdout:4/267: readlink d3/l4 0 2024-04-16T02:35:32.568 INFO:tasks.workunit.client.1.smithi153.stdout:4/268: write d3/d9/f58 [1441750,114598] 0 2024-04-16T02:35:32.569 INFO:tasks.workunit.client.1.smithi153.stdout:2/226: getdents d9/d2b/d1c/d20/d45 0 2024-04-16T02:35:32.570 INFO:tasks.workunit.client.1.smithi153.stdout:0/253: dwrite f9 [0,4194304] 0 2024-04-16T02:35:32.572 INFO:tasks.workunit.client.1.smithi153.stdout:2/227: truncate d9/d2b/d18/d3f/f48 971811 0 2024-04-16T02:35:32.574 INFO:tasks.workunit.client.1.smithi153.stdout:9/265: mknod d4/d6/dc/d46/c52 0 2024-04-16T02:35:32.575 INFO:tasks.workunit.client.1.smithi153.stdout:9/266: write d4/d6/d1b/f37 [278499,18091] 0 2024-04-16T02:35:32.576 INFO:tasks.workunit.client.1.smithi153.stdout:9/267: write d4/f13 [4459251,14894] 0 2024-04-16T02:35:32.577 INFO:tasks.workunit.client.1.smithi153.stdout:8/222: write d2/f9 [708553,120705] 0 2024-04-16T02:35:32.581 INFO:tasks.workunit.client.1.smithi153.stdout:2/228: mknod d9/d24/d34/d36/c5b 0 2024-04-16T02:35:32.582 INFO:tasks.workunit.client.1.smithi153.stdout:2/229: read - d9/d24/d34/d36/f3b zero size 2024-04-16T02:35:32.582 INFO:tasks.workunit.client.1.smithi153.stdout:2/230: stat d9/c22 0 2024-04-16T02:35:32.583 INFO:tasks.workunit.client.1.smithi153.stdout:3/242: truncate d6/d10/f1c 2268701 0 2024-04-16T02:35:32.584 INFO:tasks.workunit.client.1.smithi153.stdout:7/239: sync 2024-04-16T02:35:32.584 INFO:tasks.workunit.client.1.smithi153.stdout:8/223: symlink d2/de/d3f/l4e 0 2024-04-16T02:35:32.584 INFO:tasks.workunit.client.1.smithi153.stdout:7/240: chown d0/d13 0 1 2024-04-16T02:35:32.584 INFO:tasks.workunit.client.1.smithi153.stdout:1/230: truncate d4/dd/d13/d23/f27 23414 0 2024-04-16T02:35:32.585 INFO:tasks.workunit.client.1.smithi153.stdout:8/224: write d2/d13/d3a/f4c [880369,49613] 0 2024-04-16T02:35:32.585 INFO:tasks.workunit.client.1.smithi153.stdout:9/268: creat d4/d6/dc/d2d/f53 x:0 0 0 2024-04-16T02:35:32.586 INFO:tasks.workunit.client.1.smithi153.stdout:7/241: write d0/d4/db/d2b/f38 [199883,61496] 0 2024-04-16T02:35:32.587 INFO:tasks.workunit.client.1.smithi153.stdout:9/269: dread - d4/d6/dc/d24/f43 zero size 2024-04-16T02:35:32.588 INFO:tasks.workunit.client.1.smithi153.stdout:2/231: stat f0 0 2024-04-16T02:35:32.589 INFO:tasks.workunit.client.1.smithi153.stdout:9/270: fsync d4/d6/d1b/f3e 0 2024-04-16T02:35:32.589 INFO:tasks.workunit.client.1.smithi153.stdout:3/243: mkdir d6/d10/d3b/d55 0 2024-04-16T02:35:32.591 INFO:tasks.workunit.client.1.smithi153.stdout:2/232: dread d9/d2b/d1c/d20/d45/f54 [0,4194304] 0 2024-04-16T02:35:32.591 INFO:tasks.workunit.client.1.smithi153.stdout:0/254: creat dd/d14/f4b x:0 0 0 2024-04-16T02:35:32.592 INFO:tasks.workunit.client.1.smithi153.stdout:4/269: dwrite d3/d9/f14 [0,4194304] 0 2024-04-16T02:35:32.593 INFO:tasks.workunit.client.1.smithi153.stdout:8/225: unlink d2/d1c/d1f/l25 0 2024-04-16T02:35:32.594 INFO:tasks.workunit.client.1.smithi153.stdout:7/242: creat d0/d13/f4c x:0 0 0 2024-04-16T02:35:32.594 INFO:tasks.workunit.client.1.smithi153.stdout:2/233: read f0 [401608,125888] 0 2024-04-16T02:35:32.596 INFO:tasks.workunit.client.1.smithi153.stdout:7/243: chown d0/d4/db/d2b/f3e 24096887 1 2024-04-16T02:35:32.596 INFO:tasks.workunit.client.1.smithi153.stdout:2/234: dread - d9/d24/d34/f4f zero size 2024-04-16T02:35:32.602 INFO:tasks.workunit.client.1.smithi153.stdout:0/255: link dd/d14/f49 dd/d28/f4c 0 2024-04-16T02:35:32.604 INFO:tasks.workunit.client.1.smithi153.stdout:8/226: rename d2/de/d3f/c44 to d2/de/c4f 0 2024-04-16T02:35:32.605 INFO:tasks.workunit.client.1.smithi153.stdout:8/227: read - d2/d13/d2c/f48 zero size 2024-04-16T02:35:32.610 INFO:tasks.workunit.client.1.smithi153.stdout:2/235: mkdir d9/d2b/d5c 0 2024-04-16T02:35:32.611 INFO:tasks.workunit.client.1.smithi153.stdout:4/270: link d3/d9/d24/f2b d3/d9/db/de/f60 0 2024-04-16T02:35:32.611 INFO:tasks.workunit.client.1.smithi153.stdout:4/271: dread - d3/d9/d24/f4d zero size 2024-04-16T02:35:32.612 INFO:tasks.workunit.client.1.smithi153.stdout:4/272: fdatasync d3/d9/f29 0 2024-04-16T02:35:32.616 INFO:tasks.workunit.client.1.smithi153.stdout:0/256: rename dd/d14/d19/f1a to dd/d14/d19/d1c/f4d 0 2024-04-16T02:35:32.616 INFO:tasks.workunit.client.1.smithi153.stdout:0/257: readlink dd/d14/d19/l2d 0 2024-04-16T02:35:32.617 INFO:tasks.workunit.client.1.smithi153.stdout:0/258: write dd/d28/f41 [1171552,62984] 0 2024-04-16T02:35:32.618 INFO:tasks.workunit.client.1.smithi153.stdout:0/259: write dd/d14/f3b [740254,27958] 0 2024-04-16T02:35:32.619 INFO:tasks.workunit.client.1.smithi153.stdout:1/231: truncate d4/dd/d18/d29/f39 69597 0 2024-04-16T02:35:32.620 INFO:tasks.workunit.client.1.smithi153.stdout:1/232: chown d4/f22 631 1 2024-04-16T02:35:32.621 INFO:tasks.workunit.client.1.smithi153.stdout:4/273: mknod d3/d9/db/d45/c61 0 2024-04-16T02:35:32.624 INFO:tasks.workunit.client.1.smithi153.stdout:8/228: rename d2/c2f to d2/d13/d3a/c50 0 2024-04-16T02:35:32.626 INFO:tasks.workunit.client.1.smithi153.stdout:0/260: creat dd/d32/d33/f4e x:0 0 0 2024-04-16T02:35:32.626 INFO:tasks.workunit.client.1.smithi153.stdout:0/261: readlink dd/d14/d19/l2d 0 2024-04-16T02:35:32.628 INFO:tasks.workunit.client.1.smithi153.stdout:4/274: mknod d3/d9/db/de/d2c/c62 0 2024-04-16T02:35:32.630 INFO:tasks.workunit.client.1.smithi153.stdout:8/229: symlink d2/d13/d3a/l51 0 2024-04-16T02:35:32.633 INFO:tasks.workunit.client.1.smithi153.stdout:1/233: link d4/dd/d18/f42 d4/dd/d18/f4b 0 2024-04-16T02:35:32.634 INFO:tasks.workunit.client.1.smithi153.stdout:2/236: write d9/d2b/f17 [1147524,97121] 0 2024-04-16T02:35:32.635 INFO:tasks.workunit.client.1.smithi153.stdout:2/237: readlink d9/d24/d27/l38 0 2024-04-16T02:35:32.635 INFO:tasks.workunit.client.1.smithi153.stdout:2/238: chown d9/d24/d34 17 1 2024-04-16T02:35:32.636 INFO:tasks.workunit.client.1.smithi153.stdout:4/275: rename d3/l4 to d3/d9/db/de/d2c/d38/l63 0 2024-04-16T02:35:32.641 INFO:tasks.workunit.client.1.smithi153.stdout:4/276: mknod d3/d9/d24/d3e/c64 0 2024-04-16T02:35:32.642 INFO:tasks.workunit.client.1.smithi153.stdout:0/262: creat dd/d14/d19/d1c/f4f x:0 0 0 2024-04-16T02:35:32.642 INFO:tasks.workunit.client.1.smithi153.stdout:2/239: dwrite d9/d24/d27/d2c/d41/f44 [0,4194304] 0 2024-04-16T02:35:32.644 INFO:tasks.workunit.client.1.smithi153.stdout:8/230: getdents d2/d1c/d1f 0 2024-04-16T02:35:32.645 INFO:tasks.workunit.client.1.smithi153.stdout:4/277: dread d3/d9/f29 [0,4194304] 0 2024-04-16T02:35:32.645 INFO:tasks.workunit.client.1.smithi153.stdout:0/263: unlink dd/fe 0 2024-04-16T02:35:32.646 INFO:tasks.workunit.client.1.smithi153.stdout:4/278: readlink d3/d9/db/d17/d28/l54 0 2024-04-16T02:35:32.646 INFO:tasks.workunit.client.1.smithi153.stdout:2/240: rmdir d9/d24/d27/d2c 39 2024-04-16T02:35:32.647 INFO:tasks.workunit.client.1.smithi153.stdout:4/279: write d3/d9/f14 [4138378,41480] 0 2024-04-16T02:35:32.649 INFO:tasks.workunit.client.1.smithi153.stdout:8/231: unlink d2/de/d3f/f43 0 2024-04-16T02:35:32.650 INFO:tasks.workunit.client.1.smithi153.stdout:0/264: creat dd/d26/d44/f50 x:0 0 0 2024-04-16T02:35:32.651 INFO:tasks.workunit.client.1.smithi153.stdout:0/265: dread - dd/d14/d19/d1c/f4f zero size 2024-04-16T02:35:32.654 INFO:tasks.workunit.client.1.smithi153.stdout:4/280: stat d3/d9/db/de/l1f 0 2024-04-16T02:35:32.656 INFO:tasks.workunit.client.1.smithi153.stdout:4/281: dread - d3/d9/db/d17/f5d zero size 2024-04-16T02:35:32.656 INFO:tasks.workunit.client.1.smithi153.stdout:8/232: mknod d2/d13/d2c/c52 0 2024-04-16T02:35:32.657 INFO:tasks.workunit.client.1.smithi153.stdout:2/241: mkdir d9/d24/d27/d2c/d5d 0 2024-04-16T02:35:32.658 INFO:tasks.workunit.client.1.smithi153.stdout:2/242: write d9/d2b/f2a [577049,93043] 0 2024-04-16T02:35:32.659 INFO:tasks.workunit.client.1.smithi153.stdout:0/266: mkdir dd/d14/d51 0 2024-04-16T02:35:32.660 INFO:tasks.workunit.client.1.smithi153.stdout:4/282: creat d3/d9/db/f65 x:0 0 0 2024-04-16T02:35:32.662 INFO:tasks.workunit.client.1.smithi153.stdout:8/233: creat d2/d1c/d1f/f53 x:0 0 0 2024-04-16T02:35:32.663 INFO:tasks.workunit.client.1.smithi153.stdout:2/243: symlink d9/d24/d34/l5e 0 2024-04-16T02:35:32.663 INFO:tasks.workunit.client.1.smithi153.stdout:2/244: fdatasync d9/d2b/f2a 0 2024-04-16T02:35:32.665 INFO:tasks.workunit.client.1.smithi153.stdout:4/283: creat d3/d9/db/de/d2c/d38/f66 x:0 0 0 2024-04-16T02:35:32.666 INFO:tasks.workunit.client.1.smithi153.stdout:4/284: write d3/d9/db/d17/f55 [43641,55665] 0 2024-04-16T02:35:32.667 INFO:tasks.workunit.client.1.smithi153.stdout:0/267: mknod dd/d14/d19/d1c/c52 0 2024-04-16T02:35:32.667 INFO:tasks.workunit.client.1.smithi153.stdout:1/234: write d4/dd/d13/d23/f27 [983246,23953] 0 2024-04-16T02:35:32.670 INFO:tasks.workunit.client.1.smithi153.stdout:0/268: chown dd/d14/d19/d1c/f36 11432245 1 2024-04-16T02:35:32.670 INFO:tasks.workunit.client.1.smithi153.stdout:2/245: mkdir d9/d2b/d5c/d5f 0 2024-04-16T02:35:32.670 INFO:tasks.workunit.client.1.smithi153.stdout:1/235: write d4/dd/d13/f1a [1858273,72625] 0 2024-04-16T02:35:32.674 INFO:tasks.workunit.client.1.smithi153.stdout:1/236: rename d4/dd to d4/dd/d4c 22 2024-04-16T02:35:32.675 INFO:tasks.workunit.client.1.smithi153.stdout:4/285: dwrite d3/d9/d24/f4d [0,4194304] 0 2024-04-16T02:35:32.675 INFO:tasks.workunit.client.1.smithi153.stdout:0/269: rmdir dd/d14/d19/d1c 39 2024-04-16T02:35:32.681 INFO:tasks.workunit.client.1.smithi153.stdout:4/286: creat d3/d9/db/d17/f67 x:0 0 0 2024-04-16T02:35:32.685 INFO:tasks.workunit.client.1.smithi153.stdout:2/246: dread d9/d24/d27/f1d [4194304,4194304] 0 2024-04-16T02:35:32.685 INFO:tasks.workunit.client.1.smithi153.stdout:0/270: getdents dd/d14/d19/d1c 0 2024-04-16T02:35:32.685 INFO:tasks.workunit.client.1.smithi153.stdout:8/234: dwrite d2/f17 [0,4194304] 0 2024-04-16T02:35:32.687 INFO:tasks.workunit.client.1.smithi153.stdout:2/247: dread - d9/d2b/f46 zero size 2024-04-16T02:35:32.688 INFO:tasks.workunit.client.1.smithi153.stdout:8/235: write d2/f7 [3831007,66802] 0 2024-04-16T02:35:32.690 INFO:tasks.workunit.client.1.smithi153.stdout:2/248: chown d9/d24/d27 845468 1 2024-04-16T02:35:32.690 INFO:tasks.workunit.client.1.smithi153.stdout:1/237: dwrite d4/f22 [0,4194304] 0 2024-04-16T02:35:32.691 INFO:tasks.workunit.client.1.smithi153.stdout:4/287: rename d3/d9/f23 to d3/d9/f68 0 2024-04-16T02:35:32.692 INFO:tasks.workunit.client.1.smithi153.stdout:2/249: chown d9/d2b/d1c/f28 3675 1 2024-04-16T02:35:32.695 INFO:tasks.workunit.client.1.smithi153.stdout:4/288: write d3/d9/db/f4a [1189608,70466] 0 2024-04-16T02:35:32.695 INFO:tasks.workunit.client.1.smithi153.stdout:8/236: truncate d2/d13/d3a/f46 886730 0 2024-04-16T02:35:32.696 INFO:tasks.workunit.client.1.smithi153.stdout:6/180: sync 2024-04-16T02:35:32.697 INFO:tasks.workunit.client.1.smithi153.stdout:8/237: write d2/f7 [4609614,122085] 0 2024-04-16T02:35:32.700 INFO:tasks.workunit.client.1.smithi153.stdout:1/238: dread d4/dd/d13/d26/f34 [0,4194304] 0 2024-04-16T02:35:32.702 INFO:tasks.workunit.client.1.smithi153.stdout:6/181: creat dc/d1a/d2d/f3f x:0 0 0 2024-04-16T02:35:32.704 INFO:tasks.workunit.client.1.smithi153.stdout:4/289: mknod d3/d9/db/de/d2c/d52/c69 0 2024-04-16T02:35:32.704 INFO:tasks.workunit.client.1.smithi153.stdout:1/239: rmdir d4/dd/d13/d23 39 2024-04-16T02:35:32.708 INFO:tasks.workunit.client.1.smithi153.stdout:8/238: dwrite d2/f9 [0,4194304] 0 2024-04-16T02:35:32.710 INFO:tasks.workunit.client.1.smithi153.stdout:1/240: symlink d4/dd/d13/d23/d31/l4d 0 2024-04-16T02:35:32.711 INFO:tasks.workunit.client.1.smithi153.stdout:6/182: dread dc/d12/f24 [0,4194304] 0 2024-04-16T02:35:32.712 INFO:tasks.workunit.client.1.smithi153.stdout:4/290: dread d3/d9/db/de/f26 [0,4194304] 0 2024-04-16T02:35:32.712 INFO:tasks.workunit.client.1.smithi153.stdout:1/241: dread - d4/dd/d18/f42 zero size 2024-04-16T02:35:32.716 INFO:tasks.workunit.client.1.smithi153.stdout:4/291: mknod d3/d9/d24/c6a 0 2024-04-16T02:35:32.719 INFO:tasks.workunit.client.1.smithi153.stdout:1/242: symlink d4/dd/l4e 0 2024-04-16T02:35:32.722 INFO:tasks.workunit.client.1.smithi153.stdout:2/250: dread f1 [0,4194304] 0 2024-04-16T02:35:32.724 INFO:tasks.workunit.client.1.smithi153.stdout:2/251: mkdir d9/d24/d34/d60 0 2024-04-16T02:35:32.725 INFO:tasks.workunit.client.1.smithi153.stdout:8/239: write d2/de/f24 [98312,125450] 0 2024-04-16T02:35:32.725 INFO:tasks.workunit.client.1.smithi153.stdout:0/271: dread dd/d14/d19/f46 [4194304,4194304] 0 2024-04-16T02:35:32.727 INFO:tasks.workunit.client.1.smithi153.stdout:0/272: creat dd/f53 x:0 0 0 2024-04-16T02:35:32.728 INFO:tasks.workunit.client.1.smithi153.stdout:8/240: symlink d2/d1c/d1f/d27/d2d/d47/l54 0 2024-04-16T02:35:32.729 INFO:tasks.workunit.client.1.smithi153.stdout:0/273: chown dd/d28/f4c 1 1 2024-04-16T02:35:32.731 INFO:tasks.workunit.client.1.smithi153.stdout:0/274: chown dd/d14/d19/d20/d4a 7179 1 2024-04-16T02:35:32.731 INFO:tasks.workunit.client.1.smithi153.stdout:8/241: rmdir d2/de/d34 39 2024-04-16T02:35:32.732 INFO:tasks.workunit.client.1.smithi153.stdout:2/252: dwrite d9/d2b/d1c/f28 [0,4194304] 0 2024-04-16T02:35:32.738 INFO:tasks.workunit.client.1.smithi153.stdout:8/242: dread d2/d1c/f26 [0,4194304] 0 2024-04-16T02:35:32.738 INFO:tasks.workunit.client.1.smithi153.stdout:8/243: stat d2/d13/d2c/f48 0 2024-04-16T02:35:32.824 INFO:tasks.workunit.client.1.smithi153.stdout:5/224: sync 2024-04-16T02:35:32.875 INFO:tasks.workunit.client.1.smithi153.stdout:9/271: sync 2024-04-16T02:35:32.875 INFO:tasks.workunit.client.1.smithi153.stdout:3/244: sync 2024-04-16T02:35:32.876 INFO:tasks.workunit.client.1.smithi153.stdout:3/245: chown d6/d10/l1f 2 1 2024-04-16T02:35:32.908 INFO:tasks.workunit.client.1.smithi153.stdout:6/183: truncate dc/d12/d15/f2f 2521597 0 2024-04-16T02:35:32.909 INFO:tasks.workunit.client.1.smithi153.stdout:1/243: rmdir d4 39 2024-04-16T02:35:32.911 INFO:tasks.workunit.client.1.smithi153.stdout:8/244: mknod d2/d1c/d1f/d27/d2d/d47/c55 0 2024-04-16T02:35:32.912 INFO:tasks.workunit.client.1.smithi153.stdout:8/245: fdatasync d2/d1c/d1f/f3c 0 2024-04-16T02:35:32.912 INFO:tasks.workunit.client.1.smithi153.stdout:5/225: rmdir d8/df/d2f 39 2024-04-16T02:35:32.912 INFO:tasks.workunit.client.1.smithi153.stdout:8/246: read d2/f9 [1418453,88211] 0 2024-04-16T02:35:32.912 INFO:tasks.workunit.client.1.smithi153.stdout:5/226: stat d8/df/d23/c26 0 2024-04-16T02:35:32.913 INFO:tasks.workunit.client.1.smithi153.stdout:8/247: fsync d2/d1c/d1f/d27/f2a 0 2024-04-16T02:35:32.916 INFO:tasks.workunit.client.1.smithi153.stdout:9/272: rename d4/d6/d1b/d2a/f32 to d4/d6/dc/d46/f54 0 2024-04-16T02:35:32.918 INFO:tasks.workunit.client.1.smithi153.stdout:0/275: mkdir dd/d14/d51/d54 0 2024-04-16T02:35:32.919 INFO:tasks.workunit.client.1.smithi153.stdout:0/276: dread - dd/d26/d44/f50 zero size 2024-04-16T02:35:32.922 INFO:tasks.workunit.client.1.smithi153.stdout:2/253: creat d9/d2b/d5c/d5f/f61 x:0 0 0 2024-04-16T02:35:32.922 INFO:tasks.workunit.client.1.smithi153.stdout:2/254: chown d9/d24/d34 52 1 2024-04-16T02:35:32.924 INFO:tasks.workunit.client.1.smithi153.stdout:2/255: read - d9/d24/d27/d2c/d41/f56 zero size 2024-04-16T02:35:32.924 INFO:tasks.workunit.client.1.smithi153.stdout:4/292: sync 2024-04-16T02:35:32.924 INFO:tasks.workunit.client.1.smithi153.stdout:7/244: sync 2024-04-16T02:35:32.931 INFO:tasks.workunit.client.1.smithi153.stdout:1/244: creat d4/dd/d18/f4f x:0 0 0 2024-04-16T02:35:32.934 INFO:tasks.workunit.client.1.smithi153.stdout:1/245: readlink d4/dd/d13/d23/d31/l4d 0 2024-04-16T02:35:32.934 INFO:tasks.workunit.client.1.smithi153.stdout:9/273: dread d4/d6/dc/f11 [0,4194304] 0 2024-04-16T02:35:32.936 INFO:tasks.workunit.client.1.smithi153.stdout:4/293: mkdir d3/d9/db/d17/d3f/d6b 0 2024-04-16T02:35:32.937 INFO:tasks.workunit.client.1.smithi153.stdout:2/256: read d9/d24/d27/f1f [207743,90466] 0 2024-04-16T02:35:32.939 INFO:tasks.workunit.client.1.smithi153.stdout:6/184: dwrite dc/f31 [4194304,4194304] 0 2024-04-16T02:35:32.940 INFO:tasks.workunit.client.1.smithi153.stdout:3/246: link d6/d10/c16 d6/d10/d22/d2e/c56 0 2024-04-16T02:35:32.940 INFO:tasks.workunit.client.1.smithi153.stdout:5/227: creat d8/f3a x:0 0 0 2024-04-16T02:35:32.942 INFO:tasks.workunit.client.1.smithi153.stdout:9/274: creat d4/d6/dc/d46/f55 x:0 0 0 2024-04-16T02:35:32.946 INFO:tasks.workunit.client.1.smithi153.stdout:0/277: dread dd/d28/f3c [0,4194304] 0 2024-04-16T02:35:32.948 INFO:tasks.workunit.client.1.smithi153.stdout:0/278: stat dd/d14/d51 0 2024-04-16T02:35:32.948 INFO:tasks.workunit.client.1.smithi153.stdout:5/228: fdatasync f4 0 2024-04-16T02:35:32.948 INFO:tasks.workunit.client.1.smithi153.stdout:6/185: dread dc/d12/d15/f19 [0,4194304] 0 2024-04-16T02:35:32.949 INFO:tasks.workunit.client.1.smithi153.stdout:3/247: dwrite d6/d10/d22/d2e/f35 [0,4194304] 0 2024-04-16T02:35:32.950 INFO:tasks.workunit.client.1.smithi153.stdout:0/279: chown dd/l13 10 1 2024-04-16T02:35:32.952 INFO:tasks.workunit.client.1.smithi153.stdout:2/257: rename d9/d24/d34/l5e to d9/d2b/d18/d3f/l62 0 2024-04-16T02:35:32.952 INFO:tasks.workunit.client.1.smithi153.stdout:8/248: truncate d2/f12 2842686 0 2024-04-16T02:35:32.956 INFO:tasks.workunit.client.1.smithi153.stdout:7/245: getdents d0/d4/db/d34 0 2024-04-16T02:35:32.957 INFO:tasks.workunit.client.1.smithi153.stdout:6/186: dwrite f3 [0,4194304] 0 2024-04-16T02:35:32.960 INFO:tasks.workunit.client.1.smithi153.stdout:6/187: write dc/fe [289163,3561] 0 2024-04-16T02:35:32.962 INFO:tasks.workunit.client.1.smithi153.stdout:3/248: rename d6/d10/f27 to d6/d10/d22/d43/f57 0 2024-04-16T02:35:32.963 INFO:tasks.workunit.client.1.smithi153.stdout:1/246: dwrite d4/dd/d18/f42 [0,4194304] 0 2024-04-16T02:35:32.964 INFO:tasks.workunit.client.1.smithi153.stdout:3/249: fdatasync d6/f1a 0 2024-04-16T02:35:32.967 INFO:tasks.workunit.client.1.smithi153.stdout:0/280: mkdir dd/d14/d19/d55 0 2024-04-16T02:35:32.967 INFO:tasks.workunit.client.1.smithi153.stdout:1/247: dread - d4/dd/d18/f4f zero size 2024-04-16T02:35:32.968 INFO:tasks.workunit.client.1.smithi153.stdout:5/229: symlink d8/d1f/d20/d39/l3b 0 2024-04-16T02:35:32.968 INFO:tasks.workunit.client.1.smithi153.stdout:7/246: dread d0/d4/db/d26/f46 [0,4194304] 0 2024-04-16T02:35:32.972 INFO:tasks.workunit.client.1.smithi153.stdout:2/258: creat d9/d24/d27/d2c/d5d/f63 x:0 0 0 2024-04-16T02:35:32.972 INFO:tasks.workunit.client.1.smithi153.stdout:8/249: dread d2/f14 [0,4194304] 0 2024-04-16T02:35:32.973 INFO:tasks.workunit.client.1.smithi153.stdout:6/188: creat dc/d1a/f40 x:0 0 0 2024-04-16T02:35:32.974 INFO:tasks.workunit.client.1.smithi153.stdout:9/275: creat d4/d6/f56 x:0 0 0 2024-04-16T02:35:32.974 INFO:tasks.workunit.client.1.smithi153.stdout:3/250: creat d6/d10/d22/d2c/f58 x:0 0 0 2024-04-16T02:35:32.975 INFO:tasks.workunit.client.1.smithi153.stdout:0/281: unlink dd/c11 0 2024-04-16T02:35:32.977 INFO:tasks.workunit.client.1.smithi153.stdout:1/248: creat d4/dd/d18/d29/f50 x:0 0 0 2024-04-16T02:35:32.977 INFO:tasks.workunit.client.1.smithi153.stdout:6/189: write dc/d1a/d2d/f3f [756318,47386] 0 2024-04-16T02:35:32.977 INFO:tasks.workunit.client.1.smithi153.stdout:0/282: stat dd/d14/d51 0 2024-04-16T02:35:32.978 INFO:tasks.workunit.client.1.smithi153.stdout:8/250: unlink d2/d1c/f4d 0 2024-04-16T02:35:32.978 INFO:tasks.workunit.client.1.smithi153.stdout:3/251: write d6/d10/d22/f4c [690042,121217] 0 2024-04-16T02:35:32.981 INFO:tasks.workunit.client.1.smithi153.stdout:2/259: unlink d9/d2b/d1c/d20/f43 0 2024-04-16T02:35:32.982 INFO:tasks.workunit.client.1.smithi153.stdout:5/230: dwrite d8/f3a [0,4194304] 0 2024-04-16T02:35:32.985 INFO:tasks.workunit.client.1.smithi153.stdout:2/260: read - d9/d24/d27/d2c/d5d/f63 zero size 2024-04-16T02:35:32.986 INFO:tasks.workunit.client.1.smithi153.stdout:9/276: mknod d4/c57 0 2024-04-16T02:35:32.991 INFO:tasks.workunit.client.1.smithi153.stdout:4/294: dread d3/d9/f1a [0,4194304] 0 2024-04-16T02:35:32.991 INFO:tasks.workunit.client.1.smithi153.stdout:6/190: symlink dc/l41 0 2024-04-16T02:35:32.993 INFO:tasks.workunit.client.1.smithi153.stdout:5/231: stat d8/df/f11 0 2024-04-16T02:35:32.994 INFO:tasks.workunit.client.1.smithi153.stdout:6/191: dread - dc/f21 zero size 2024-04-16T02:35:32.995 INFO:tasks.workunit.client.1.smithi153.stdout:1/249: dwrite d4/dd/d13/d26/f33 [4194304,4194304] 0 2024-04-16T02:35:32.996 INFO:tasks.workunit.client.1.smithi153.stdout:2/261: mknod d9/d2b/d1c/d20/c64 0 2024-04-16T02:35:32.996 INFO:tasks.workunit.client.1.smithi153.stdout:7/247: dwrite d0/d4/db/d26/f46 [0,4194304] 0 2024-04-16T02:35:32.997 INFO:tasks.workunit.client.1.smithi153.stdout:9/277: creat d4/d6/dc/d2d/f58 x:0 0 0 2024-04-16T02:35:33.000 INFO:tasks.workunit.client.1.smithi153.stdout:4/295: unlink d3/d9/f58 0 2024-04-16T02:35:33.001 INFO:tasks.workunit.client.1.smithi153.stdout:2/262: truncate d9/d2b/f46 987168 0 2024-04-16T02:35:33.001 INFO:tasks.workunit.client.1.smithi153.stdout:4/296: readlink d3/d9/db/de/l10 0 2024-04-16T02:35:33.002 INFO:tasks.workunit.client.1.smithi153.stdout:5/232: truncate d8/df/f11 910581 0 2024-04-16T02:35:33.002 INFO:tasks.workunit.client.1.smithi153.stdout:9/278: read d4/d6/d1b/f37 [18285,74469] 0 2024-04-16T02:35:33.004 INFO:tasks.workunit.client.1.smithi153.stdout:4/297: read - d3/d9/db/d17/d3f/f4f zero size 2024-04-16T02:35:33.004 INFO:tasks.workunit.client.1.smithi153.stdout:9/279: fsync d4/d6/dc/f11 0 2024-04-16T02:35:33.004 INFO:tasks.workunit.client.1.smithi153.stdout:2/263: write d9/d2b/d18/d3f/f49 [78002,112414] 0 2024-04-16T02:35:33.008 INFO:tasks.workunit.client.1.smithi153.stdout:1/250: rename d4/fe to d4/dd/d18/f51 0 2024-04-16T02:35:33.009 INFO:tasks.workunit.client.1.smithi153.stdout:6/192: dread dc/d1a/d2d/f3f [0,4194304] 0 2024-04-16T02:35:33.011 INFO:tasks.workunit.client.1.smithi153.stdout:6/193: chown dc/d10/f1b 1283 1 2024-04-16T02:35:33.011 INFO:tasks.workunit.client.1.smithi153.stdout:6/194: fsync dc/f21 0 2024-04-16T02:35:33.013 INFO:tasks.workunit.client.1.smithi153.stdout:9/280: dwrite d4/d6/dc/d24/f25 [0,4194304] 0 2024-04-16T02:35:33.016 INFO:tasks.workunit.client.1.smithi153.stdout:1/251: creat d4/dd/d18/f52 x:0 0 0 2024-04-16T02:35:33.016 INFO:tasks.workunit.client.1.smithi153.stdout:5/233: symlink d8/df/l3c 0 2024-04-16T02:35:33.016 INFO:tasks.workunit.client.1.smithi153.stdout:8/251: stat d2/f12 0 2024-04-16T02:35:33.017 INFO:tasks.workunit.client.1.smithi153.stdout:1/252: readlink d4/dd/d18/l1e 0 2024-04-16T02:35:33.018 INFO:tasks.workunit.client.1.smithi153.stdout:7/248: rmdir d0/d4/d3f 0 2024-04-16T02:35:33.019 INFO:tasks.workunit.client.1.smithi153.stdout:2/264: symlink d9/d2b/l65 0 2024-04-16T02:35:33.020 INFO:tasks.workunit.client.1.smithi153.stdout:0/283: dwrite dd/d28/f3c [0,4194304] 0 2024-04-16T02:35:33.020 INFO:tasks.workunit.client.1.smithi153.stdout:6/195: symlink dc/d10/l42 0 2024-04-16T02:35:33.022 INFO:tasks.workunit.client.1.smithi153.stdout:1/253: dread - d4/dd/d18/f4f zero size 2024-04-16T02:35:33.022 INFO:tasks.workunit.client.1.smithi153.stdout:6/196: fsync dc/f17 0 2024-04-16T02:35:33.024 INFO:tasks.workunit.client.1.smithi153.stdout:7/249: read d0/f36 [239133,117950] 0 2024-04-16T02:35:33.025 INFO:tasks.workunit.client.1.smithi153.stdout:9/281: dread d4/d33/f38 [0,4194304] 0 2024-04-16T02:35:33.026 INFO:tasks.workunit.client.1.smithi153.stdout:4/298: getdents d3/d9/db/d45 0 2024-04-16T02:35:33.029 INFO:tasks.workunit.client.1.smithi153.stdout:2/265: creat d9/d24/d27/d2c/d5d/f66 x:0 0 0 2024-04-16T02:35:33.030 INFO:tasks.workunit.client.1.smithi153.stdout:1/254: write d4/dd/d13/d26/f34 [3101945,116748] 0 2024-04-16T02:35:33.032 INFO:tasks.workunit.client.1.smithi153.stdout:4/299: read - d3/d9/db/f65 zero size 2024-04-16T02:35:33.032 INFO:tasks.workunit.client.1.smithi153.stdout:5/234: dread d8/f3a [0,4194304] 0 2024-04-16T02:35:33.034 INFO:tasks.workunit.client.1.smithi153.stdout:8/252: dread d2/de/f3e [0,4194304] 0 2024-04-16T02:35:33.035 INFO:tasks.workunit.client.1.smithi153.stdout:7/250: mkdir d0/d4/db/d34/d4b/d4d 0 2024-04-16T02:35:33.035 INFO:tasks.workunit.client.1.smithi153.stdout:2/266: creat d9/d2b/d1c/d20/f67 x:0 0 0 2024-04-16T02:35:33.036 INFO:tasks.workunit.client.1.smithi153.stdout:9/282: rename d4/d6/d1b/f37 to d4/d6/d1b/f59 0 2024-04-16T02:35:33.038 INFO:tasks.workunit.client.1.smithi153.stdout:0/284: dwrite dd/d14/d19/f1e [0,4194304] 0 2024-04-16T02:35:33.038 INFO:tasks.workunit.client.1.smithi153.stdout:1/255: creat d4/dd/d18/f53 x:0 0 0 2024-04-16T02:35:33.039 INFO:tasks.workunit.client.1.smithi153.stdout:2/267: chown d9/d2b/d1c/d20/d45/f54 6880 1 2024-04-16T02:35:33.039 INFO:tasks.workunit.client.1.smithi153.stdout:5/235: creat d8/df/d2e/f3d x:0 0 0 2024-04-16T02:35:33.039 INFO:tasks.workunit.client.1.smithi153.stdout:6/197: rmdir dc/d10 39 2024-04-16T02:35:33.040 INFO:tasks.workunit.client.1.smithi153.stdout:2/268: chown d9 96864541 1 2024-04-16T02:35:33.040 INFO:tasks.workunit.client.1.smithi153.stdout:0/285: write dd/f10 [2108143,90082] 0 2024-04-16T02:35:33.040 INFO:tasks.workunit.client.1.smithi153.stdout:9/283: truncate d4/d33/f4f 734906 0 2024-04-16T02:35:33.041 INFO:tasks.workunit.client.1.smithi153.stdout:8/253: fdatasync d2/ff 0 2024-04-16T02:35:33.043 INFO:tasks.workunit.client.1.smithi153.stdout:7/251: creat d0/d4/db/d2b/f4e x:0 0 0 2024-04-16T02:35:33.043 INFO:tasks.workunit.client.1.smithi153.stdout:0/286: fdatasync dd/d32/d33/f42 0 2024-04-16T02:35:33.046 INFO:tasks.workunit.client.1.smithi153.stdout:6/198: creat dc/d1a/d2d/f43 x:0 0 0 2024-04-16T02:35:33.047 INFO:tasks.workunit.client.1.smithi153.stdout:9/284: truncate d4/d6/dc/d2d/f58 1040379 0 2024-04-16T02:35:33.049 INFO:tasks.workunit.client.1.smithi153.stdout:4/300: rename d3/d9/db/de/d2c/d38/f66 to d3/d9/f6c 0 2024-04-16T02:35:33.050 INFO:tasks.workunit.client.1.smithi153.stdout:4/301: write d3/d9/f29 [1596199,67829] 0 2024-04-16T02:35:33.051 INFO:tasks.workunit.client.1.smithi153.stdout:1/256: creat d4/dd/d13/f54 x:0 0 0 2024-04-16T02:35:33.052 INFO:tasks.workunit.client.1.smithi153.stdout:0/287: symlink dd/d14/d19/d1c/l56 0 2024-04-16T02:35:33.053 INFO:tasks.workunit.client.1.smithi153.stdout:2/269: unlink d9/d24/d27/d2c/d5d/f66 0 2024-04-16T02:35:33.053 INFO:tasks.workunit.client.1.smithi153.stdout:0/288: write dd/d32/d33/f42 [304622,39512] 0 2024-04-16T02:35:33.058 INFO:tasks.workunit.client.1.smithi153.stdout:6/199: creat dc/d1a/d2d/f44 x:0 0 0 2024-04-16T02:35:33.058 INFO:tasks.workunit.client.1.smithi153.stdout:9/285: symlink d4/d6/dc/d2d/l5a 0 2024-04-16T02:35:33.058 INFO:tasks.workunit.client.1.smithi153.stdout:5/236: truncate d8/df/d2f/f33 386065 0 2024-04-16T02:35:33.058 INFO:tasks.workunit.client.1.smithi153.stdout:2/270: dread d9/d2b/d18/d3f/f49 [0,4194304] 0 2024-04-16T02:35:33.058 INFO:tasks.workunit.client.1.smithi153.stdout:4/302: stat d3/d9/d24/d35/c5f 0 2024-04-16T02:35:33.059 INFO:tasks.workunit.client.1.smithi153.stdout:5/237: fsync d8/df/d2e/f3d 0 2024-04-16T02:35:33.061 INFO:tasks.workunit.client.1.smithi153.stdout:1/257: dwrite d4/dd/d13/f54 [0,4194304] 0 2024-04-16T02:35:33.061 INFO:tasks.workunit.client.1.smithi153.stdout:9/286: creat d4/d33/f5b x:0 0 0 2024-04-16T02:35:33.062 INFO:tasks.workunit.client.1.smithi153.stdout:7/252: rmdir d0/d4/db/d16/d41 0 2024-04-16T02:35:33.063 INFO:tasks.workunit.client.1.smithi153.stdout:1/258: write d4/dd/d13/d26/f3d [588655,5114] 0 2024-04-16T02:35:33.063 INFO:tasks.workunit.client.1.smithi153.stdout:8/254: link d2/d13/l1a d2/l56 0 2024-04-16T02:35:33.064 INFO:tasks.workunit.client.1.smithi153.stdout:7/253: dread - d0/d13/f4a zero size 2024-04-16T02:35:33.064 INFO:tasks.workunit.client.1.smithi153.stdout:8/255: fdatasync d2/de/f15 0 2024-04-16T02:35:33.066 INFO:tasks.workunit.client.1.smithi153.stdout:1/259: chown d4/c37 15 1 2024-04-16T02:35:33.067 INFO:tasks.workunit.client.1.smithi153.stdout:5/238: symlink d8/df/d23/l3e 0 2024-04-16T02:35:33.072 INFO:tasks.workunit.client.1.smithi153.stdout:7/254: rmdir d0/d4/db/d34 39 2024-04-16T02:35:33.073 INFO:tasks.workunit.client.1.smithi153.stdout:8/256: mkdir d2/d1c/d1f/d57 0 2024-04-16T02:35:33.074 INFO:tasks.workunit.client.1.smithi153.stdout:0/289: dread dd/d14/d19/d1c/f3e [0,4194304] 0 2024-04-16T02:35:33.075 INFO:tasks.workunit.client.1.smithi153.stdout:5/239: creat d8/d1f/f3f x:0 0 0 2024-04-16T02:35:33.075 INFO:tasks.workunit.client.1.smithi153.stdout:0/290: readlink dd/d14/d19/l2d 0 2024-04-16T02:35:33.076 INFO:tasks.workunit.client.1.smithi153.stdout:0/291: fsync dd/f10 0 2024-04-16T02:35:33.077 INFO:tasks.workunit.client.1.smithi153.stdout:6/200: getdents dc/d12 0 2024-04-16T02:35:33.079 INFO:tasks.workunit.client.1.smithi153.stdout:6/201: fsync dc/f17 0 2024-04-16T02:35:33.079 INFO:tasks.workunit.client.1.smithi153.stdout:1/260: rename d4/dd/d18/d29/f39 to d4/dd/d20/f55 0 2024-04-16T02:35:33.080 INFO:tasks.workunit.client.1.smithi153.stdout:2/271: getdents d9/d2b/d1c 0 2024-04-16T02:35:33.084 INFO:tasks.workunit.client.1.smithi153.stdout:0/292: fdatasync dd/ff 0 2024-04-16T02:35:33.084 INFO:tasks.workunit.client.1.smithi153.stdout:5/240: dwrite d8/f30 [0,4194304] 0 2024-04-16T02:35:33.086 INFO:tasks.workunit.client.1.smithi153.stdout:6/202: truncate dc/d10/f37 1037818 0 2024-04-16T02:35:33.087 INFO:tasks.workunit.client.1.smithi153.stdout:5/241: write d8/df/f24 [971998,49667] 0 2024-04-16T02:35:33.087 INFO:tasks.workunit.client.1.smithi153.stdout:9/287: truncate d4/d6/dc/d24/f42 1233184 0 2024-04-16T02:35:33.087 INFO:tasks.workunit.client.1.smithi153.stdout:1/261: symlink d4/dd/d18/l56 0 2024-04-16T02:35:33.089 INFO:tasks.workunit.client.1.smithi153.stdout:6/203: stat dc/d12/c1d 0 2024-04-16T02:35:33.091 INFO:tasks.workunit.client.1.smithi153.stdout:5/242: write d8/df/f1e [4966189,108227] 0 2024-04-16T02:35:33.095 INFO:tasks.workunit.client.1.smithi153.stdout:7/255: getdents d0/d4 0 2024-04-16T02:35:33.096 INFO:tasks.workunit.client.1.smithi153.stdout:0/293: dread f5 [0,4194304] 0 2024-04-16T02:35:33.098 INFO:tasks.workunit.client.1.smithi153.stdout:0/294: dread - dd/d26/d44/f50 zero size 2024-04-16T02:35:33.098 INFO:tasks.workunit.client.1.smithi153.stdout:2/272: creat d9/d24/d27/f68 x:0 0 0 2024-04-16T02:35:33.100 INFO:tasks.workunit.client.1.smithi153.stdout:9/288: link d4/d6/dc/l41 d4/d33/d3a/l5c 0 2024-04-16T02:35:33.103 INFO:tasks.workunit.client.1.smithi153.stdout:0/295: chown dd/d14/d19/d1c/f3e 2335 1 2024-04-16T02:35:33.104 INFO:tasks.workunit.client.1.smithi153.stdout:8/257: dread d2/d13/f1d [0,4194304] 0 2024-04-16T02:35:33.104 INFO:tasks.workunit.client.1.smithi153.stdout:0/296: fdatasync dd/d32/f35 0 2024-04-16T02:35:33.107 INFO:tasks.workunit.client.1.smithi153.stdout:5/243: dwrite d8/f3a [0,4194304] 0 2024-04-16T02:35:33.108 INFO:tasks.workunit.client.1.smithi153.stdout:5/244: chown d8/f30 66325303 1 2024-04-16T02:35:33.109 INFO:tasks.workunit.client.1.smithi153.stdout:7/256: dwrite d0/d4/db/d2b/f22 [4194304,4194304] 0 2024-04-16T02:35:33.110 INFO:tasks.workunit.client.1.smithi153.stdout:7/257: chown d0/d4/db/d2b/f4e 346983 1 2024-04-16T02:35:33.167 INFO:tasks.workunit.client.1.smithi153.stdout:3/252: sync 2024-04-16T02:35:33.175 INFO:tasks.workunit.client.1.smithi153.stdout:9/289: unlink d4/c57 0 2024-04-16T02:35:33.177 INFO:tasks.workunit.client.1.smithi153.stdout:2/273: creat d9/d2b/d5c/d5f/f69 x:0 0 0 2024-04-16T02:35:33.179 INFO:tasks.workunit.client.1.smithi153.stdout:1/262: write d4/dd/d13/d26/f2b [759910,24330] 0 2024-04-16T02:35:33.180 INFO:tasks.workunit.client.1.smithi153.stdout:9/290: dread d4/d6/d1b/f3e [0,4194304] 0 2024-04-16T02:35:33.180 INFO:tasks.workunit.client.1.smithi153.stdout:5/245: creat d8/d1f/d20/f40 x:0 0 0 2024-04-16T02:35:33.182 INFO:tasks.workunit.client.1.smithi153.stdout:7/258: unlink d0/d13/f4c 0 2024-04-16T02:35:33.182 INFO:tasks.workunit.client.1.smithi153.stdout:6/204: truncate dc/f31 3885290 0 2024-04-16T02:35:33.183 INFO:tasks.workunit.client.1.smithi153.stdout:6/205: read - dc/d1a/f26 zero size 2024-04-16T02:35:33.183 INFO:tasks.workunit.client.1.smithi153.stdout:3/253: mkdir d6/d10/d3b/d59 0 2024-04-16T02:35:33.184 INFO:tasks.workunit.client.1.smithi153.stdout:6/206: fdatasync f3 0 2024-04-16T02:35:33.184 INFO:tasks.workunit.client.1.smithi153.stdout:3/254: write d6/d10/d22/f31 [118553,114683] 0 2024-04-16T02:35:33.185 INFO:tasks.workunit.client.1.smithi153.stdout:2/274: unlink d9/d24/d27/d2c/c3e 0 2024-04-16T02:35:33.187 INFO:tasks.workunit.client.1.smithi153.stdout:1/263: mkdir d4/dd/d18/d57 0 2024-04-16T02:35:33.188 INFO:tasks.workunit.client.1.smithi153.stdout:0/297: write dd/d28/f4c [524212,13573] 0 2024-04-16T02:35:33.189 INFO:tasks.workunit.client.1.smithi153.stdout:0/298: dread - dd/d28/f40 zero size 2024-04-16T02:35:33.190 INFO:tasks.workunit.client.1.smithi153.stdout:9/291: unlink d4/d6/d1b/f2f 0 2024-04-16T02:35:33.192 INFO:tasks.workunit.client.1.smithi153.stdout:7/259: rename d0/d4/db/f12 to d0/d13/f4f 0 2024-04-16T02:35:33.193 INFO:tasks.workunit.client.1.smithi153.stdout:7/260: dread - d0/d4/db/d2b/f28 zero size 2024-04-16T02:35:33.195 INFO:tasks.workunit.client.1.smithi153.stdout:9/292: dwrite d4/d6/f56 [0,4194304] 0 2024-04-16T02:35:33.198 INFO:tasks.workunit.client.1.smithi153.stdout:1/264: mkdir d4/dd/d13/d26/d58 0 2024-04-16T02:35:33.199 INFO:tasks.workunit.client.1.smithi153.stdout:5/246: mkdir d8/d41 0 2024-04-16T02:35:33.199 INFO:tasks.workunit.client.1.smithi153.stdout:1/265: chown d4/dd/d18/d29/l35 6948 1 2024-04-16T02:35:33.202 INFO:tasks.workunit.client.1.smithi153.stdout:0/299: rename dd/d32/d33 to dd/d32/d57 0 2024-04-16T02:35:33.204 INFO:tasks.workunit.client.1.smithi153.stdout:6/207: symlink dc/l45 0 2024-04-16T02:35:33.206 INFO:tasks.workunit.client.1.smithi153.stdout:2/275: mknod d9/d24/d34/d60/c6a 0 2024-04-16T02:35:33.208 INFO:tasks.workunit.client.1.smithi153.stdout:7/261: stat d0/d13/c35 0 2024-04-16T02:35:33.208 INFO:tasks.workunit.client.1.smithi153.stdout:7/262: readlink d0/d4/l2d 0 2024-04-16T02:35:33.209 INFO:tasks.workunit.client.1.smithi153.stdout:7/263: write d0/d4/db/d26/f27 [2445634,54552] 0 2024-04-16T02:35:33.211 INFO:tasks.workunit.client.1.smithi153.stdout:9/293: mkdir d4/d6/dc/d24/d5d 0 2024-04-16T02:35:33.211 INFO:tasks.workunit.client.1.smithi153.stdout:9/294: chown d4 0 1 2024-04-16T02:35:33.213 INFO:tasks.workunit.client.1.smithi153.stdout:1/266: creat d4/dd/d18/d29/f59 x:0 0 0 2024-04-16T02:35:33.213 INFO:tasks.workunit.client.1.smithi153.stdout:1/267: fdatasync d4/dd/d18/f52 0 2024-04-16T02:35:33.215 INFO:tasks.workunit.client.1.smithi153.stdout:4/303: sync 2024-04-16T02:35:33.215 INFO:tasks.workunit.client.1.smithi153.stdout:6/208: rmdir dc/d12/d15 39 2024-04-16T02:35:33.216 INFO:tasks.workunit.client.1.smithi153.stdout:4/304: fsync d3/d9/f14 0 2024-04-16T02:35:33.217 INFO:tasks.workunit.client.1.smithi153.stdout:4/305: chown d3/d9/db/de/f3b 60045164 1 2024-04-16T02:35:33.217 INFO:tasks.workunit.client.1.smithi153.stdout:4/306: chown d3/d9/db/d17/d28/l48 3406574 1 2024-04-16T02:35:33.217 INFO:tasks.workunit.client.1.smithi153.stdout:3/255: link d6/c17 d6/d10/d22/d43/c5a 0 2024-04-16T02:35:33.218 INFO:tasks.workunit.client.1.smithi153.stdout:3/256: chown c4 7 1 2024-04-16T02:35:33.219 INFO:tasks.workunit.client.1.smithi153.stdout:9/295: rmdir d4/d33/d3a/d40 39 2024-04-16T02:35:33.219 INFO:tasks.workunit.client.1.smithi153.stdout:8/258: link d2/d1c/d1f/d27/l29 d2/de/l58 0 2024-04-16T02:35:33.221 INFO:tasks.workunit.client.1.smithi153.stdout:1/268: creat d4/dd/d18/d29/f5a x:0 0 0 2024-04-16T02:35:33.222 INFO:tasks.workunit.client.1.smithi153.stdout:1/269: chown d4/dd/d18/d29/l48 43 1 2024-04-16T02:35:33.225 INFO:tasks.workunit.client.1.smithi153.stdout:8/259: rmdir d2/d1c/d1f/d27/d2d 39 2024-04-16T02:35:33.226 INFO:tasks.workunit.client.1.smithi153.stdout:5/247: link d8/df/d2e/c34 d8/df/d28/c42 0 2024-04-16T02:35:33.228 INFO:tasks.workunit.client.1.smithi153.stdout:7/264: creat d0/d4/db/f50 x:0 0 0 2024-04-16T02:35:33.229 INFO:tasks.workunit.client.1.smithi153.stdout:3/257: mkdir d6/d10/d5b 0 2024-04-16T02:35:33.231 INFO:tasks.workunit.client.1.smithi153.stdout:9/296: unlink d4/d6/dc/l41 0 2024-04-16T02:35:33.233 INFO:tasks.workunit.client.1.smithi153.stdout:2/276: getdents d9/d24/d27/d2c/d5d 0 2024-04-16T02:35:33.235 INFO:tasks.workunit.client.1.smithi153.stdout:4/307: rename d3/d9/db/de/d2c/f50 to d3/f6d 0 2024-04-16T02:35:33.237 INFO:tasks.workunit.client.1.smithi153.stdout:8/260: creat d2/d1c/d1f/d27/d2d/f59 x:0 0 0 2024-04-16T02:35:33.237 INFO:tasks.workunit.client.1.smithi153.stdout:5/248: mknod d8/c43 0 2024-04-16T02:35:33.238 INFO:tasks.workunit.client.1.smithi153.stdout:5/249: chown d8/l17 3466 1 2024-04-16T02:35:33.238 INFO:tasks.workunit.client.1.smithi153.stdout:7/265: mkdir d0/d4/db/d34/d51 0 2024-04-16T02:35:33.239 INFO:tasks.workunit.client.1.smithi153.stdout:7/266: stat d0/d4/fc 0 2024-04-16T02:35:33.240 INFO:tasks.workunit.client.1.smithi153.stdout:4/308: dread d3/d9/f1a [0,4194304] 0 2024-04-16T02:35:33.240 INFO:tasks.workunit.client.1.smithi153.stdout:7/267: chown d0/d4/db/d34/d4b/d4d 6 1 2024-04-16T02:35:33.240 INFO:tasks.workunit.client.1.smithi153.stdout:1/270: rename d4/dd/d13/f1a to d4/dd/d20/f5b 0 2024-04-16T02:35:33.241 INFO:tasks.workunit.client.1.smithi153.stdout:8/261: creat d2/d1c/d1f/f5a x:0 0 0 2024-04-16T02:35:33.242 INFO:tasks.workunit.client.1.smithi153.stdout:3/258: rmdir d6/d10/d3b/d55 0 2024-04-16T02:35:33.242 INFO:tasks.workunit.client.1.smithi153.stdout:4/309: write d3/d9/d24/d3e/f5a [546859,4062] 0 2024-04-16T02:35:33.243 INFO:tasks.workunit.client.1.smithi153.stdout:5/250: write d8/df/f2c [552675,38348] 0 2024-04-16T02:35:33.243 INFO:tasks.workunit.client.1.smithi153.stdout:8/262: dread - d2/d1c/d1f/f53 zero size 2024-04-16T02:35:33.245 INFO:tasks.workunit.client.1.smithi153.stdout:9/297: getdents d4/d6/dc/d2d 0 2024-04-16T02:35:33.246 INFO:tasks.workunit.client.1.smithi153.stdout:5/251: chown d8/df/d2e/f3d 503340281 1 2024-04-16T02:35:33.247 INFO:tasks.workunit.client.1.smithi153.stdout:4/310: mknod d3/d9/db/d17/c6e 0 2024-04-16T02:35:33.248 INFO:tasks.workunit.client.1.smithi153.stdout:3/259: mknod d6/d10/d22/c5c 0 2024-04-16T02:35:33.249 INFO:tasks.workunit.client.1.smithi153.stdout:2/277: dread d9/d2b/d18/f29 [0,4194304] 0 2024-04-16T02:35:33.249 INFO:tasks.workunit.client.1.smithi153.stdout:4/311: dread - d3/d9/f6c zero size 2024-04-16T02:35:33.250 INFO:tasks.workunit.client.1.smithi153.stdout:9/298: chown d4/d6/d1b/c1f 36 1 2024-04-16T02:35:33.251 INFO:tasks.workunit.client.1.smithi153.stdout:4/312: fsync d3/ff 0 2024-04-16T02:35:33.251 INFO:tasks.workunit.client.1.smithi153.stdout:3/260: dread - d6/d10/d22/d2e/f54 zero size 2024-04-16T02:35:33.254 INFO:tasks.workunit.client.1.smithi153.stdout:9/299: write d4/d6/f56 [497744,51098] 0 2024-04-16T02:35:33.255 INFO:tasks.workunit.client.1.smithi153.stdout:1/271: dwrite d4/f7 [4194304,4194304] 0 2024-04-16T02:35:33.256 INFO:tasks.workunit.client.1.smithi153.stdout:4/313: unlink d3/d9/c57 0 2024-04-16T02:35:33.259 INFO:tasks.workunit.client.1.smithi153.stdout:3/261: read d6/d10/f18 [3036311,79700] 0 2024-04-16T02:35:33.261 INFO:tasks.workunit.client.1.smithi153.stdout:5/252: dwrite d8/df/d28/f35 [0,4194304] 0 2024-04-16T02:35:33.271 INFO:tasks.workunit.client.1.smithi153.stdout:2/278: dwrite d9/d2b/d18/d3f/f48 [0,4194304] 0 2024-04-16T02:35:33.273 INFO:tasks.workunit.client.1.smithi153.stdout:8/263: dwrite d2/d1c/f36 [0,4194304] 0 2024-04-16T02:35:33.275 INFO:tasks.workunit.client.1.smithi153.stdout:9/300: getdents d4/d6 0 2024-04-16T02:35:33.276 INFO:tasks.workunit.client.1.smithi153.stdout:5/253: rename d8/df/d23/c26 to d8/df/d2f/c44 0 2024-04-16T02:35:33.277 INFO:tasks.workunit.client.1.smithi153.stdout:1/272: symlink d4/dd/d13/d26/l5c 0 2024-04-16T02:35:33.280 INFO:tasks.workunit.client.1.smithi153.stdout:3/262: link d6/d10/c49 d6/d10/d22/d2e/c5d 0 2024-04-16T02:35:33.281 INFO:tasks.workunit.client.1.smithi153.stdout:5/254: fsync d8/f30 0 2024-04-16T02:35:33.284 INFO:tasks.workunit.client.1.smithi153.stdout:9/301: unlink d4/d6/dc/f21 0 2024-04-16T02:35:33.284 INFO:tasks.workunit.client.1.smithi153.stdout:1/273: creat d4/dd/d13/d23/f5d x:0 0 0 2024-04-16T02:35:33.285 INFO:tasks.workunit.client.1.smithi153.stdout:3/263: truncate d6/f52 203706 0 2024-04-16T02:35:33.288 INFO:tasks.workunit.client.1.smithi153.stdout:2/279: dwrite d9/d24/d27/d2c/d41/f4c [0,4194304] 0 2024-04-16T02:35:33.288 INFO:tasks.workunit.client.1.smithi153.stdout:1/274: chown d4/dd/l10 7 1 2024-04-16T02:35:33.295 INFO:tasks.workunit.client.1.smithi153.stdout:1/275: creat d4/f5e x:0 0 0 2024-04-16T02:35:33.295 INFO:tasks.workunit.client.1.smithi153.stdout:9/302: dwrite d4/d6/d1b/f3e [0,4194304] 0 2024-04-16T02:35:33.295 INFO:tasks.workunit.client.1.smithi153.stdout:2/280: creat d9/d2b/f6b x:0 0 0 2024-04-16T02:35:33.297 INFO:tasks.workunit.client.1.smithi153.stdout:9/303: write d4/d6/d1b/f3e [2143075,75120] 0 2024-04-16T02:35:33.299 INFO:tasks.workunit.client.1.smithi153.stdout:4/314: dread d3/ff [4194304,4194304] 0 2024-04-16T02:35:33.299 INFO:tasks.workunit.client.1.smithi153.stdout:2/281: write d9/d24/d27/d2c/d41/f44 [2537815,53422] 0 2024-04-16T02:35:33.299 INFO:tasks.workunit.client.1.smithi153.stdout:1/276: mkdir d4/dd/d13/d23/d5f 0 2024-04-16T02:35:33.300 INFO:tasks.workunit.client.1.smithi153.stdout:9/304: write d4/f48 [1674373,4761] 0 2024-04-16T02:35:33.300 INFO:tasks.workunit.client.1.smithi153.stdout:1/277: chown d4/l9 92147608 1 2024-04-16T02:35:33.302 INFO:tasks.workunit.client.1.smithi153.stdout:9/305: read d4/d6/f22 [378409,60311] 0 2024-04-16T02:35:33.302 INFO:tasks.workunit.client.1.smithi153.stdout:2/282: creat d9/d2b/d5c/f6c x:0 0 0 2024-04-16T02:35:33.305 INFO:tasks.workunit.client.1.smithi153.stdout:2/283: dread - d9/d24/d27/d2c/d41/f53 zero size 2024-04-16T02:35:33.306 INFO:tasks.workunit.client.1.smithi153.stdout:1/278: rename l1 to d4/dd/d13/d23/d31/l60 0 2024-04-16T02:35:33.310 INFO:tasks.workunit.client.1.smithi153.stdout:8/264: write d2/f9 [458781,20872] 0 2024-04-16T02:35:33.311 INFO:tasks.workunit.client.1.smithi153.stdout:2/284: rename d9/d24/c25 to d9/d24/d34/d36/c6d 0 2024-04-16T02:35:33.312 INFO:tasks.workunit.client.1.smithi153.stdout:5/255: chown d8/df/d2f/c44 8 1 2024-04-16T02:35:33.314 INFO:tasks.workunit.client.1.smithi153.stdout:5/256: write d8/f30 [2617517,130128] 0 2024-04-16T02:35:33.314 INFO:tasks.workunit.client.1.smithi153.stdout:8/265: mkdir d2/d13/d3a/d5b 0 2024-04-16T02:35:33.315 INFO:tasks.workunit.client.1.smithi153.stdout:9/306: creat d4/d33/d3a/d40/f5e x:0 0 0 2024-04-16T02:35:33.316 INFO:tasks.workunit.client.1.smithi153.stdout:2/285: creat d9/d2b/d18/d3f/f6e x:0 0 0 2024-04-16T02:35:33.318 INFO:tasks.workunit.client.1.smithi153.stdout:3/264: truncate d6/d10/d22/d2e/f35 3593902 0 2024-04-16T02:35:33.318 INFO:tasks.workunit.client.1.smithi153.stdout:5/257: write d8/df/d2e/f3d [92022,122474] 0 2024-04-16T02:35:33.321 INFO:tasks.workunit.client.1.smithi153.stdout:2/286: read - d9/d24/d27/d2c/d41/f56 zero size 2024-04-16T02:35:33.321 INFO:tasks.workunit.client.1.smithi153.stdout:8/266: mkdir d2/de/d5c 0 2024-04-16T02:35:33.324 INFO:tasks.workunit.client.1.smithi153.stdout:9/307: truncate d4/d6/dc/f11 886159 0 2024-04-16T02:35:33.324 INFO:tasks.workunit.client.1.smithi153.stdout:2/287: readlink d9/d24/l4b 0 2024-04-16T02:35:33.324 INFO:tasks.workunit.client.1.smithi153.stdout:5/258: symlink d8/df/l45 0 2024-04-16T02:35:33.325 INFO:tasks.workunit.client.1.smithi153.stdout:1/279: dwrite d4/dd/f1d [0,4194304] 0 2024-04-16T02:35:33.327 INFO:tasks.workunit.client.1.smithi153.stdout:2/288: chown d9/d2b/d18/l31 10 1 2024-04-16T02:35:33.327 INFO:tasks.workunit.client.1.smithi153.stdout:8/267: rmdir d2/d1c/d1f/d27/d2d/d47 39 2024-04-16T02:35:33.329 INFO:tasks.workunit.client.1.smithi153.stdout:9/308: write d4/d33/f3f [235650,124940] 0 2024-04-16T02:35:33.330 INFO:tasks.workunit.client.1.smithi153.stdout:5/259: symlink d8/df/d28/l46 0 2024-04-16T02:35:33.333 INFO:tasks.workunit.client.1.smithi153.stdout:9/309: readlink d4/d33/l45 0 2024-04-16T02:35:33.333 INFO:tasks.workunit.client.1.smithi153.stdout:0/300: sync 2024-04-16T02:35:33.333 INFO:tasks.workunit.client.1.smithi153.stdout:8/268: dread - d2/d1c/d1f/d27/d2d/f59 zero size 2024-04-16T02:35:33.335 INFO:tasks.workunit.client.1.smithi153.stdout:4/315: dwrite d3/d9/db/de/f60 [0,4194304] 0 2024-04-16T02:35:33.337 INFO:tasks.workunit.client.1.smithi153.stdout:9/310: dread - d4/d6/dc/d24/f43 zero size 2024-04-16T02:35:33.337 INFO:tasks.workunit.client.1.smithi153.stdout:2/289: creat d9/d24/d27/f6f x:0 0 0 2024-04-16T02:35:33.337 INFO:tasks.workunit.client.1.smithi153.stdout:8/269: chown d2/d13/f1d 32356346 1 2024-04-16T02:35:33.337 INFO:tasks.workunit.client.1.smithi153.stdout:0/301: mknod dd/d32/d57/c58 0 2024-04-16T02:35:33.339 INFO:tasks.workunit.client.1.smithi153.stdout:8/270: dread - d2/d1c/d1f/f53 zero size 2024-04-16T02:35:33.339 INFO:tasks.workunit.client.1.smithi153.stdout:2/290: stat d9/d24/d27/d2c/d41/f44 0 2024-04-16T02:35:33.340 INFO:tasks.workunit.client.1.smithi153.stdout:5/260: getdents d8/df/d2f 0 2024-04-16T02:35:33.341 INFO:tasks.workunit.client.1.smithi153.stdout:0/302: creat dd/d14/d51/f59 x:0 0 0 2024-04-16T02:35:33.341 INFO:tasks.workunit.client.1.smithi153.stdout:9/311: chown d4/d33/d3a/l4c 3084943 1 2024-04-16T02:35:33.344 INFO:tasks.workunit.client.1.smithi153.stdout:0/303: fdatasync dd/d14/f4b 0 2024-04-16T02:35:33.344 INFO:tasks.workunit.client.1.smithi153.stdout:8/271: chown d2/d1c/d1f/d27/d2d/d47 4264603 1 2024-04-16T02:35:33.346 INFO:tasks.workunit.client.1.smithi153.stdout:5/261: symlink d8/l47 0 2024-04-16T02:35:33.346 INFO:tasks.workunit.client.1.smithi153.stdout:9/312: write d4/d6/f20 [4662057,52347] 0 2024-04-16T02:35:33.348 INFO:tasks.workunit.client.1.smithi153.stdout:2/291: creat d9/d2b/d18/d3f/d40/f70 x:0 0 0 2024-04-16T02:35:33.351 INFO:tasks.workunit.client.1.smithi153.stdout:0/304: mknod dd/d14/d19/d20/c5a 0 2024-04-16T02:35:33.351 INFO:tasks.workunit.client.1.smithi153.stdout:4/316: dwrite d3/d9/d24/d3e/f5a [0,4194304] 0 2024-04-16T02:35:33.352 INFO:tasks.workunit.client.1.smithi153.stdout:8/272: write d2/d1c/d1f/d27/d2d/f59 [959888,3590] 0 2024-04-16T02:35:33.353 INFO:tasks.workunit.client.1.smithi153.stdout:2/292: mkdir d9/d71 0 2024-04-16T02:35:33.354 INFO:tasks.workunit.client.1.smithi153.stdout:5/262: mknod d8/df/d2f/c48 0 2024-04-16T02:35:33.354 INFO:tasks.workunit.client.1.smithi153.stdout:3/265: dwrite d6/d10/d22/d2e/f4a [0,4194304] 0 2024-04-16T02:35:33.355 INFO:tasks.workunit.client.1.smithi153.stdout:3/266: stat d6/c7 0 2024-04-16T02:35:33.357 INFO:tasks.workunit.client.1.smithi153.stdout:9/313: creat d4/d33/d3a/f5f x:0 0 0 2024-04-16T02:35:33.358 INFO:tasks.workunit.client.1.smithi153.stdout:4/317: creat d3/d9/db/d17/d28/f6f x:0 0 0 2024-04-16T02:35:33.358 INFO:tasks.workunit.client.1.smithi153.stdout:3/267: write d6/d10/f32 [309195,87334] 0 2024-04-16T02:35:33.359 INFO:tasks.workunit.client.1.smithi153.stdout:8/273: symlink d2/d13/d3a/l5d 0 2024-04-16T02:35:33.360 INFO:tasks.workunit.client.1.smithi153.stdout:1/280: dread d4/dd/d20/f5b [0,4194304] 0 2024-04-16T02:35:33.360 INFO:tasks.workunit.client.1.smithi153.stdout:4/318: readlink d3/d9/db/d17/l1e 0 2024-04-16T02:35:33.361 INFO:tasks.workunit.client.1.smithi153.stdout:1/281: stat d4/dd/d18/f4b 0 2024-04-16T02:35:33.363 INFO:tasks.workunit.client.1.smithi153.stdout:3/268: write d6/f52 [1048776,27666] 0 2024-04-16T02:35:33.363 INFO:tasks.workunit.client.1.smithi153.stdout:4/319: write d3/d9/db/de/f3b [939174,60956] 0 2024-04-16T02:35:33.433 INFO:tasks.workunit.client.1.smithi153.stdout:7/268: sync 2024-04-16T02:35:33.438 INFO:tasks.workunit.client.1.smithi153.stdout:7/269: dwrite d0/d4/db/d2b/f40 [0,4194304] 0 2024-04-16T02:35:33.452 INFO:tasks.workunit.client.1.smithi153.stdout:6/209: sync 2024-04-16T02:35:33.452 INFO:tasks.workunit.client.1.smithi153.stdout:5/263: sync 2024-04-16T02:35:33.453 INFO:tasks.workunit.client.1.smithi153.stdout:5/264: truncate d8/df/d23/f32 560889 0 2024-04-16T02:35:33.511 INFO:tasks.workunit.client.1.smithi153.stdout:9/314: mkdir d4/d6/dc/d24/d60 0 2024-04-16T02:35:33.511 INFO:tasks.workunit.client.1.smithi153.stdout:9/315: stat d4/d6/c18 0 2024-04-16T02:35:33.515 INFO:tasks.workunit.client.1.smithi153.stdout:4/320: rmdir d3/d9/db/de 39 2024-04-16T02:35:33.515 INFO:tasks.workunit.client.1.smithi153.stdout:4/321: fsync d3/d9/d24/f4d 0 2024-04-16T02:35:33.520 INFO:tasks.workunit.client.1.smithi153.stdout:7/270: symlink d0/d4/db/d16/l52 0 2024-04-16T02:35:33.520 INFO:tasks.workunit.client.1.smithi153.stdout:4/322: dread d3/d9/d24/f4d [0,4194304] 0 2024-04-16T02:35:33.521 INFO:tasks.workunit.client.1.smithi153.stdout:4/323: write d3/d9/f3d [834563,129120] 0 2024-04-16T02:35:33.529 INFO:tasks.workunit.client.1.smithi153.stdout:0/305: dwrite dd/f22 [0,4194304] 0 2024-04-16T02:35:33.529 INFO:tasks.workunit.client.1.smithi153.stdout:7/271: rmdir d0/d4/db/d34 39 2024-04-16T02:35:33.531 INFO:tasks.workunit.client.1.smithi153.stdout:7/272: dread - d0/d4/db/d2b/f28 zero size 2024-04-16T02:35:33.531 INFO:tasks.workunit.client.1.smithi153.stdout:6/210: chown dc/d12/d15/c29 7154446 1 2024-04-16T02:35:33.532 INFO:tasks.workunit.client.1.smithi153.stdout:5/265: mknod d8/c49 0 2024-04-16T02:35:33.534 INFO:tasks.workunit.client.1.smithi153.stdout:1/282: truncate d4/dd/d18/f4b 1395973 0 2024-04-16T02:35:33.535 INFO:tasks.workunit.client.1.smithi153.stdout:7/273: dread - d0/d4/db/d2b/f3e zero size 2024-04-16T02:35:33.535 INFO:tasks.workunit.client.1.smithi153.stdout:8/274: rmdir d2/de/d5c 0 2024-04-16T02:35:33.536 INFO:tasks.workunit.client.1.smithi153.stdout:9/316: symlink d4/d6/dc/d24/d4e/l61 0 2024-04-16T02:35:33.537 INFO:tasks.workunit.client.1.smithi153.stdout:2/293: mknod d9/d24/d27/d2c/d41/c72 0 2024-04-16T02:35:33.539 INFO:tasks.workunit.client.1.smithi153.stdout:3/269: dread d6/d10/f18 [0,4194304] 0 2024-04-16T02:35:33.540 INFO:tasks.workunit.client.1.smithi153.stdout:5/266: creat d8/d1f/d20/d39/f4a x:0 0 0 2024-04-16T02:35:33.540 INFO:tasks.workunit.client.1.smithi153.stdout:1/283: dread d4/dd/f1d [0,4194304] 0 2024-04-16T02:35:33.542 INFO:tasks.workunit.client.1.smithi153.stdout:9/317: read d4/d6/f1a [88450,124142] 0 2024-04-16T02:35:33.543 INFO:tasks.workunit.client.1.smithi153.stdout:9/318: readlink d4/d33/l45 0 2024-04-16T02:35:33.543 INFO:tasks.workunit.client.1.smithi153.stdout:7/274: rename d0/d4/db/d26 to d0/d4/db/d2b/d53 0 2024-04-16T02:35:33.544 INFO:tasks.workunit.client.1.smithi153.stdout:9/319: read d4/d6/f56 [2647902,117961] 0 2024-04-16T02:35:33.547 INFO:tasks.workunit.client.1.smithi153.stdout:4/324: creat d3/d9/f70 x:0 0 0 2024-04-16T02:35:33.549 INFO:tasks.workunit.client.1.smithi153.stdout:5/267: symlink d8/df/d23/l4b 0 2024-04-16T02:35:33.550 INFO:tasks.workunit.client.1.smithi153.stdout:4/325: write d3/d9/db/d17/f67 [757952,55196] 0 2024-04-16T02:35:33.551 INFO:tasks.workunit.client.1.smithi153.stdout:8/275: mkdir d2/d1c/d1f/d27/d5e 0 2024-04-16T02:35:33.551 INFO:tasks.workunit.client.1.smithi153.stdout:5/268: fsync d8/f3a 0 2024-04-16T02:35:33.553 INFO:tasks.workunit.client.1.smithi153.stdout:1/284: rename d4/dd/d18/d29/f5a to d4/dd/d13/d23/d5f/f61 0 2024-04-16T02:35:33.554 INFO:tasks.workunit.client.1.smithi153.stdout:2/294: dwrite d9/d24/d27/f1d [4194304,4194304] 0 2024-04-16T02:35:33.555 INFO:tasks.workunit.client.1.smithi153.stdout:1/285: fsync d4/dd/d18/d29/f50 0 2024-04-16T02:35:33.558 INFO:tasks.workunit.client.1.smithi153.stdout:0/306: dread dd/d14/d19/d1c/f2f [0,4194304] 0 2024-04-16T02:35:33.561 INFO:tasks.workunit.client.1.smithi153.stdout:5/269: dwrite d8/f30 [0,4194304] 0 2024-04-16T02:35:33.564 INFO:tasks.workunit.client.1.smithi153.stdout:2/295: fsync d9/d2b/d18/d3f/d40/f70 0 2024-04-16T02:35:33.564 INFO:tasks.workunit.client.1.smithi153.stdout:7/275: dread d0/d4/fc [0,4194304] 0 2024-04-16T02:35:33.565 INFO:tasks.workunit.client.1.smithi153.stdout:9/320: mknod d4/c62 0 2024-04-16T02:35:33.566 INFO:tasks.workunit.client.1.smithi153.stdout:6/211: link dc/d12/c3d dc/d12/c46 0 2024-04-16T02:35:33.571 INFO:tasks.workunit.client.1.smithi153.stdout:0/307: creat dd/d26/f5b x:0 0 0 2024-04-16T02:35:33.572 INFO:tasks.workunit.client.1.smithi153.stdout:5/270: creat d8/d1f/d20/d39/f4c x:0 0 0 2024-04-16T02:35:33.573 INFO:tasks.workunit.client.1.smithi153.stdout:3/270: truncate d6/d10/f32 3435226 0 2024-04-16T02:35:33.574 INFO:tasks.workunit.client.1.smithi153.stdout:9/321: dwrite d4/f48 [0,4194304] 0 2024-04-16T02:35:33.575 INFO:tasks.workunit.client.1.smithi153.stdout:9/322: write d4/d33/f5b [138732,45144] 0 2024-04-16T02:35:33.577 INFO:tasks.workunit.client.1.smithi153.stdout:2/296: mkdir d9/d24/d73 0 2024-04-16T02:35:33.578 INFO:tasks.workunit.client.1.smithi153.stdout:8/276: link d2/f14 d2/de/f5f 0 2024-04-16T02:35:33.579 INFO:tasks.workunit.client.1.smithi153.stdout:2/297: dread - d9/d2b/d18/d3f/f58 zero size 2024-04-16T02:35:33.579 INFO:tasks.workunit.client.1.smithi153.stdout:0/308: mknod dd/d14/d51/c5c 0 2024-04-16T02:35:33.579 INFO:tasks.workunit.client.1.smithi153.stdout:7/276: symlink d0/l54 0 2024-04-16T02:35:33.581 INFO:tasks.workunit.client.1.smithi153.stdout:8/277: write d2/d13/d2c/f32 [21552,26061] 0 2024-04-16T02:35:33.582 INFO:tasks.workunit.client.1.smithi153.stdout:3/271: symlink d6/l5e 0 2024-04-16T02:35:33.583 INFO:tasks.workunit.client.1.smithi153.stdout:5/271: dwrite d8/df/d28/f35 [0,4194304] 0 2024-04-16T02:35:33.584 INFO:tasks.workunit.client.1.smithi153.stdout:5/272: chown d8 752898 1 2024-04-16T02:35:33.584 INFO:tasks.workunit.client.1.smithi153.stdout:9/323: fdatasync d4/d33/f38 0 2024-04-16T02:35:33.585 INFO:tasks.workunit.client.1.smithi153.stdout:1/286: creat d4/dd/d13/f62 x:0 0 0 2024-04-16T02:35:33.585 INFO:tasks.workunit.client.1.smithi153.stdout:4/326: creat d3/d9/db/de/f71 x:0 0 0 2024-04-16T02:35:33.586 INFO:tasks.workunit.client.1.smithi153.stdout:6/212: mkdir dc/d47 0 2024-04-16T02:35:33.587 INFO:tasks.workunit.client.1.smithi153.stdout:6/213: stat dc/d10/l32 0 2024-04-16T02:35:33.590 INFO:tasks.workunit.client.1.smithi153.stdout:0/309: creat dd/d32/d57/f5d x:0 0 0 2024-04-16T02:35:33.591 INFO:tasks.workunit.client.1.smithi153.stdout:0/310: write dd/f10 [2150412,9475] 0 2024-04-16T02:35:33.592 INFO:tasks.workunit.client.1.smithi153.stdout:8/278: mkdir d2/d1c/d1f/d27/d2d/d47/d60 0 2024-04-16T02:35:33.594 INFO:tasks.workunit.client.1.smithi153.stdout:3/272: creat d6/d10/d22/d2c/f5f x:0 0 0 2024-04-16T02:35:33.595 INFO:tasks.workunit.client.1.smithi153.stdout:6/214: dwrite dc/d12/d15/f23 [4194304,4194304] 0 2024-04-16T02:35:33.597 INFO:tasks.workunit.client.1.smithi153.stdout:6/215: write dc/d10/f37 [544091,94360] 0 2024-04-16T02:35:33.601 INFO:tasks.workunit.client.1.smithi153.stdout:4/327: symlink d3/d9/d24/d35/l72 0 2024-04-16T02:35:33.601 INFO:tasks.workunit.client.1.smithi153.stdout:4/328: fdatasync d3/d9/db/de/f60 0 2024-04-16T02:35:33.603 INFO:tasks.workunit.client.1.smithi153.stdout:2/298: mknod d9/d24/d27/c74 0 2024-04-16T02:35:33.604 INFO:tasks.workunit.client.1.smithi153.stdout:2/299: write d9/d2b/f17 [1980337,98661] 0 2024-04-16T02:35:33.607 INFO:tasks.workunit.client.1.smithi153.stdout:0/311: creat dd/d26/f5e x:0 0 0 2024-04-16T02:35:33.608 INFO:tasks.workunit.client.1.smithi153.stdout:0/312: chown dd/d28/f40 777148 1 2024-04-16T02:35:33.613 INFO:tasks.workunit.client.1.smithi153.stdout:6/216: mknod dc/d12/d15/c48 0 2024-04-16T02:35:33.613 INFO:tasks.workunit.client.1.smithi153.stdout:9/324: mknod d4/d6/dc/d24/d60/c63 0 2024-04-16T02:35:33.616 INFO:tasks.workunit.client.1.smithi153.stdout:2/300: creat d9/d2b/d5c/f75 x:0 0 0 2024-04-16T02:35:33.616 INFO:tasks.workunit.client.1.smithi153.stdout:7/277: mknod d0/d4/db/d34/d51/c55 0 2024-04-16T02:35:33.617 INFO:tasks.workunit.client.1.smithi153.stdout:2/301: fsync d9/d2b/f16 0 2024-04-16T02:35:33.617 INFO:tasks.workunit.client.1.smithi153.stdout:0/313: mknod dd/d32/d57/c5f 0 2024-04-16T02:35:33.618 INFO:tasks.workunit.client.1.smithi153.stdout:0/314: stat dd/d32/d57/f4e 0 2024-04-16T02:35:33.619 INFO:tasks.workunit.client.1.smithi153.stdout:3/273: link d6/f52 d6/d10/d22/d2c/f60 0 2024-04-16T02:35:33.619 INFO:tasks.workunit.client.1.smithi153.stdout:9/325: dwrite d4/d6/dc/d24/f30 [0,4194304] 0 2024-04-16T02:35:33.619 INFO:tasks.workunit.client.1.smithi153.stdout:0/315: write dd/d14/d19/d20/f31 [765217,95999] 0 2024-04-16T02:35:33.620 INFO:tasks.workunit.client.1.smithi153.stdout:0/316: read - dd/f47 zero size 2024-04-16T02:35:33.622 INFO:tasks.workunit.client.1.smithi153.stdout:0/317: stat dd/d14/d51 0 2024-04-16T02:35:33.622 INFO:tasks.workunit.client.1.smithi153.stdout:1/287: write d4/dd/f16 [1422000,34273] 0 2024-04-16T02:35:33.623 INFO:tasks.workunit.client.1.smithi153.stdout:5/273: creat d8/f4d x:0 0 0 2024-04-16T02:35:33.625 INFO:tasks.workunit.client.1.smithi153.stdout:3/274: mknod d6/c61 0 2024-04-16T02:35:33.627 INFO:tasks.workunit.client.1.smithi153.stdout:9/326: mkdir d4/d6/dc/d2d/d64 0 2024-04-16T02:35:33.627 INFO:tasks.workunit.client.1.smithi153.stdout:4/329: creat d3/f73 x:0 0 0 2024-04-16T02:35:33.629 INFO:tasks.workunit.client.1.smithi153.stdout:6/217: mknod dc/d12/c49 0 2024-04-16T02:35:33.630 INFO:tasks.workunit.client.1.smithi153.stdout:4/330: write d3/d9/db/d17/f5d [475103,82988] 0 2024-04-16T02:35:33.630 INFO:tasks.workunit.client.1.smithi153.stdout:1/288: readlink d4/dd/d18/l3f 0 2024-04-16T02:35:33.633 INFO:tasks.workunit.client.1.smithi153.stdout:4/331: write d3/d9/f29 [876021,32429] 0 2024-04-16T02:35:33.638 INFO:tasks.workunit.client.1.smithi153.stdout:9/327: dwrite d4/d33/f5b [0,4194304] 0 2024-04-16T02:35:33.638 INFO:tasks.workunit.client.1.smithi153.stdout:2/302: dread d9/fb [0,4194304] 0 2024-04-16T02:35:33.642 INFO:tasks.workunit.client.1.smithi153.stdout:6/218: dwrite dc/f17 [0,4194304] 0 2024-04-16T02:35:33.643 INFO:tasks.workunit.client.1.smithi153.stdout:4/332: rename d3/d9/db/d17/f4b to d3/d9/db/de/d2c/f74 0 2024-04-16T02:35:33.643 INFO:tasks.workunit.client.1.smithi153.stdout:6/219: readlink dc/l45 0 2024-04-16T02:35:33.643 INFO:tasks.workunit.client.1.smithi153.stdout:1/289: symlink d4/dd/d13/d26/l63 0 2024-04-16T02:35:33.645 INFO:tasks.workunit.client.1.smithi153.stdout:1/290: chown d4/dd/d13/d23/l3e 51 1 2024-04-16T02:35:33.648 INFO:tasks.workunit.client.1.smithi153.stdout:9/328: symlink d4/d33/d3a/l65 0 2024-04-16T02:35:33.649 INFO:tasks.workunit.client.1.smithi153.stdout:9/329: chown d4/d6/dc 123022739 1 2024-04-16T02:35:33.649 INFO:tasks.workunit.client.1.smithi153.stdout:5/274: dread d8/df/f1e [0,4194304] 0 2024-04-16T02:35:33.650 INFO:tasks.workunit.client.1.smithi153.stdout:6/220: creat dc/d47/f4a x:0 0 0 2024-04-16T02:35:33.653 INFO:tasks.workunit.client.1.smithi153.stdout:5/275: truncate d8/d1f/d20/d39/f38 587831 0 2024-04-16T02:35:33.655 INFO:tasks.workunit.client.1.smithi153.stdout:4/333: getdents d3/d9/d24 0 2024-04-16T02:35:33.657 INFO:tasks.workunit.client.1.smithi153.stdout:5/276: mkdir d8/df/d2f/d4e 0 2024-04-16T02:35:33.657 INFO:tasks.workunit.client.1.smithi153.stdout:9/330: dwrite d4/f48 [0,4194304] 0 2024-04-16T02:35:33.657 INFO:tasks.workunit.client.1.smithi153.stdout:4/334: readlink d3/d9/db/de/l53 0 2024-04-16T02:35:33.659 INFO:tasks.workunit.client.1.smithi153.stdout:6/221: dwrite f3 [0,4194304] 0 2024-04-16T02:35:33.659 INFO:tasks.workunit.client.1.smithi153.stdout:4/335: dread - d3/d9/db/de/f71 zero size 2024-04-16T02:35:33.665 INFO:tasks.workunit.client.1.smithi153.stdout:6/222: unlink dc/d10/l32 0 2024-04-16T02:35:33.667 INFO:tasks.workunit.client.1.smithi153.stdout:6/223: write dc/d1a/f26 [391671,53325] 0 2024-04-16T02:35:33.667 INFO:tasks.workunit.client.1.smithi153.stdout:4/336: dwrite d3/d9/db/d17/f5d [0,4194304] 0 2024-04-16T02:35:33.668 INFO:tasks.workunit.client.1.smithi153.stdout:4/337: stat d3/d9/d24/d35 0 2024-04-16T02:35:33.669 INFO:tasks.workunit.client.1.smithi153.stdout:6/224: write f9 [3121535,66392] 0 2024-04-16T02:35:33.674 INFO:tasks.workunit.client.1.smithi153.stdout:4/338: mkdir d3/d9/d24/d75 0 2024-04-16T02:35:33.675 INFO:tasks.workunit.client.1.smithi153.stdout:1/291: mknod d4/dd/d13/d26/d58/c64 0 2024-04-16T02:35:33.675 INFO:tasks.workunit.client.1.smithi153.stdout:5/277: getdents d8/d1f/d20/d39 0 2024-04-16T02:35:33.676 INFO:tasks.workunit.client.1.smithi153.stdout:6/225: mknod dc/d47/c4b 0 2024-04-16T02:35:33.677 INFO:tasks.workunit.client.1.smithi153.stdout:1/292: stat d4/dd/c41 0 2024-04-16T02:35:33.677 INFO:tasks.workunit.client.1.smithi153.stdout:5/278: rename d8/df/d2e to d8/df/d2e/d4f 22 2024-04-16T02:35:33.677 INFO:tasks.workunit.client.1.smithi153.stdout:6/226: read dc/d12/f24 [2275040,43414] 0 2024-04-16T02:35:33.678 INFO:tasks.workunit.client.1.smithi153.stdout:5/279: fsync d8/f3a 0 2024-04-16T02:35:33.679 INFO:tasks.workunit.client.1.smithi153.stdout:5/280: write d8/f4d [998424,83702] 0 2024-04-16T02:35:33.680 INFO:tasks.workunit.client.1.smithi153.stdout:1/293: write d4/dd/d13/d26/f3d [1268774,80953] 0 2024-04-16T02:35:33.681 INFO:tasks.workunit.client.1.smithi153.stdout:6/227: rmdir dc/d12 39 2024-04-16T02:35:33.681 INFO:tasks.workunit.client.1.smithi153.stdout:5/281: truncate d8/d1f/f3f 417747 0 2024-04-16T02:35:33.682 INFO:tasks.workunit.client.1.smithi153.stdout:5/282: rename d8/d1f/d20 to d8/d1f/d20/d50 22 2024-04-16T02:35:33.683 INFO:tasks.workunit.client.1.smithi153.stdout:6/228: symlink dc/d1a/l4c 0 2024-04-16T02:35:33.684 INFO:tasks.workunit.client.1.smithi153.stdout:6/229: readlink l0 0 2024-04-16T02:35:33.685 INFO:tasks.workunit.client.1.smithi153.stdout:5/283: mkdir d8/df/d2f/d4e/d51 0 2024-04-16T02:35:33.687 INFO:tasks.workunit.client.1.smithi153.stdout:6/230: chown dc/d12/d15 9389327 1 2024-04-16T02:35:33.687 INFO:tasks.workunit.client.1.smithi153.stdout:1/294: dwrite d4/dd/d13/f54 [4194304,4194304] 0 2024-04-16T02:35:33.689 INFO:tasks.workunit.client.1.smithi153.stdout:5/284: readlink d8/l1a 0 2024-04-16T02:35:33.695 INFO:tasks.workunit.client.1.smithi153.stdout:1/295: mkdir d4/dd/d18/d57/d65 0 2024-04-16T02:35:33.695 INFO:tasks.workunit.client.1.smithi153.stdout:4/339: write d3/d9/d24/d3e/f5a [5134287,111313] 0 2024-04-16T02:35:33.695 INFO:tasks.workunit.client.1.smithi153.stdout:6/231: link c7 dc/d12/d15/c4d 0 2024-04-16T02:35:33.696 INFO:tasks.workunit.client.1.smithi153.stdout:4/340: fsync d3/d9/f1a 0 2024-04-16T02:35:33.697 INFO:tasks.workunit.client.1.smithi153.stdout:6/232: fdatasync f9 0 2024-04-16T02:35:33.697 INFO:tasks.workunit.client.1.smithi153.stdout:5/285: dwrite d8/f3a [0,4194304] 0 2024-04-16T02:35:33.698 INFO:tasks.workunit.client.1.smithi153.stdout:4/341: fdatasync d3/d9/f1a 0 2024-04-16T02:35:33.698 INFO:tasks.workunit.client.1.smithi153.stdout:6/233: fdatasync dc/d1a/f1c 0 2024-04-16T02:35:33.699 INFO:tasks.workunit.client.1.smithi153.stdout:5/286: write d8/d1f/f21 [539087,122678] 0 2024-04-16T02:35:33.699 INFO:tasks.workunit.client.1.smithi153.stdout:1/296: creat d4/dd/d18/f66 x:0 0 0 2024-04-16T02:35:33.700 INFO:tasks.workunit.client.1.smithi153.stdout:4/342: fdatasync d3/d9/f6c 0 2024-04-16T02:35:33.702 INFO:tasks.workunit.client.1.smithi153.stdout:4/343: chown d3/d9/db/d17/d28/l48 931509090 1 2024-04-16T02:35:33.703 INFO:tasks.workunit.client.1.smithi153.stdout:6/234: creat dc/d12/d15/f4e x:0 0 0 2024-04-16T02:35:33.704 INFO:tasks.workunit.client.1.smithi153.stdout:4/344: write d3/d9/db/d17/f55 [212317,88772] 0 2024-04-16T02:35:33.705 INFO:tasks.workunit.client.1.smithi153.stdout:5/287: mknod d8/df/d2f/d4e/d51/c52 0 2024-04-16T02:35:33.705 INFO:tasks.workunit.client.1.smithi153.stdout:6/235: truncate dc/d12/d15/f4e 559608 0 2024-04-16T02:35:33.706 INFO:tasks.workunit.client.1.smithi153.stdout:8/279: sync 2024-04-16T02:35:33.706 INFO:tasks.workunit.client.1.smithi153.stdout:7/278: sync 2024-04-16T02:35:33.706 INFO:tasks.workunit.client.1.smithi153.stdout:1/297: rename d4/dd/d13/d23/l3e to d4/dd/l67 0 2024-04-16T02:35:33.707 INFO:tasks.workunit.client.1.smithi153.stdout:1/298: fsync d4/f40 0 2024-04-16T02:35:33.709 INFO:tasks.workunit.client.1.smithi153.stdout:6/236: mknod dc/d10/c4f 0 2024-04-16T02:35:33.710 INFO:tasks.workunit.client.1.smithi153.stdout:4/345: symlink d3/d9/d24/d75/l76 0 2024-04-16T02:35:33.712 INFO:tasks.workunit.client.1.smithi153.stdout:7/279: mknod d0/d4/db/d34/c56 0 2024-04-16T02:35:33.712 INFO:tasks.workunit.client.1.smithi153.stdout:8/280: fdatasync d2/f23 0 2024-04-16T02:35:33.713 INFO:tasks.workunit.client.1.smithi153.stdout:4/346: dread - d3/d9/f6c zero size 2024-04-16T02:35:33.714 INFO:tasks.workunit.client.1.smithi153.stdout:5/288: dwrite d8/d1f/d20/d39/f4a [0,4194304] 0 2024-04-16T02:35:33.714 INFO:tasks.workunit.client.1.smithi153.stdout:6/237: symlink dc/d12/d15/l50 0 2024-04-16T02:35:33.719 INFO:tasks.workunit.client.1.smithi153.stdout:5/289: readlink d8/df/d23/l31 0 2024-04-16T02:35:33.720 INFO:tasks.workunit.client.1.smithi153.stdout:1/299: dwrite d4/dd/d13/f2a [0,4194304] 0 2024-04-16T02:35:33.722 INFO:tasks.workunit.client.1.smithi153.stdout:7/280: creat d0/d4/db/f57 x:0 0 0 2024-04-16T02:35:33.790 INFO:tasks.workunit.client.1.smithi153.stdout:1/300: write d4/dd/d18/f52 [239203,99579] 0 2024-04-16T02:35:33.791 INFO:tasks.workunit.client.1.smithi153.stdout:4/347: creat d3/d9/f77 x:0 0 0 2024-04-16T02:35:33.791 INFO:tasks.workunit.client.1.smithi153.stdout:5/290: creat d8/df/d2e/f53 x:0 0 0 2024-04-16T02:35:33.793 INFO:tasks.workunit.client.1.smithi153.stdout:8/281: dread d2/f9 [0,4194304] 0 2024-04-16T02:35:33.794 INFO:tasks.workunit.client.1.smithi153.stdout:4/348: chown d3/d9/c15 1489387 1 2024-04-16T02:35:33.795 INFO:tasks.workunit.client.1.smithi153.stdout:7/281: readlink d0/d4/db/d2b/l15 0 2024-04-16T02:35:33.795 INFO:tasks.workunit.client.1.smithi153.stdout:5/291: chown d8/df/d23/f32 345 1 2024-04-16T02:35:33.796 INFO:tasks.workunit.client.1.smithi153.stdout:6/238: dwrite dc/d1a/f1c [0,4194304] 0 2024-04-16T02:35:33.796 INFO:tasks.workunit.client.1.smithi153.stdout:1/301: write d4/dd/d18/f51 [1011244,30380] 0 2024-04-16T02:35:33.797 INFO:tasks.workunit.client.1.smithi153.stdout:8/282: mknod d2/de/c61 0 2024-04-16T02:35:33.799 INFO:tasks.workunit.client.1.smithi153.stdout:7/282: creat d0/d4/f58 x:0 0 0 2024-04-16T02:35:33.799 INFO:tasks.workunit.client.1.smithi153.stdout:8/283: write d2/de/f15 [799653,37924] 0 2024-04-16T02:35:33.800 INFO:tasks.workunit.client.1.smithi153.stdout:6/239: truncate dc/d1a/d2d/f3f 1733982 0 2024-04-16T02:35:33.805 INFO:tasks.workunit.client.1.smithi153.stdout:8/284: creat d2/d13/f62 x:0 0 0 2024-04-16T02:35:33.806 INFO:tasks.workunit.client.1.smithi153.stdout:6/240: rmdir dc/d1a 39 2024-04-16T02:35:33.807 INFO:tasks.workunit.client.1.smithi153.stdout:1/302: dwrite d4/dd/d13/d26/f2b [0,4194304] 0 2024-04-16T02:35:33.808 INFO:tasks.workunit.client.1.smithi153.stdout:8/285: write d2/f23 [4639433,92852] 0 2024-04-16T02:35:33.808 INFO:tasks.workunit.client.1.smithi153.stdout:1/303: write d4/dd/d13/f2a [2402832,117347] 0 2024-04-16T02:35:33.809 INFO:tasks.workunit.client.1.smithi153.stdout:4/349: link d3/d9/c2f d3/d9/d24/d3e/c78 0 2024-04-16T02:35:33.811 INFO:tasks.workunit.client.1.smithi153.stdout:4/350: chown d3/d9/db/de/l1f 0 1 2024-04-16T02:35:33.813 INFO:tasks.workunit.client.1.smithi153.stdout:5/292: link d8/c15 d8/df/d2e/c54 0 2024-04-16T02:35:33.818 INFO:tasks.workunit.client.1.smithi153.stdout:6/241: mknod dc/d1a/d2d/c51 0 2024-04-16T02:35:33.819 INFO:tasks.workunit.client.1.smithi153.stdout:8/286: mkdir d2/d1c/d1f/d27/d5e/d63 0 2024-04-16T02:35:33.820 INFO:tasks.workunit.client.1.smithi153.stdout:7/283: mkdir d0/d4/db/d2b/d53/d59 0 2024-04-16T02:35:33.821 INFO:tasks.workunit.client.1.smithi153.stdout:1/304: symlink d4/dd/d13/d26/l68 0 2024-04-16T02:35:33.821 INFO:tasks.workunit.client.1.smithi153.stdout:4/351: mkdir d3/d9/d79 0 2024-04-16T02:35:33.823 INFO:tasks.workunit.client.1.smithi153.stdout:6/242: dread dc/d1a/f26 [0,4194304] 0 2024-04-16T02:35:33.823 INFO:tasks.workunit.client.1.smithi153.stdout:5/293: dwrite d8/d1f/f3f [0,4194304] 0 2024-04-16T02:35:33.825 INFO:tasks.workunit.client.1.smithi153.stdout:7/284: chown d0/d4/db/d2b/f38 3 1 2024-04-16T02:35:33.825 INFO:tasks.workunit.client.1.smithi153.stdout:4/352: chown d3/d9/db/de/f60 50955 1 2024-04-16T02:35:33.827 INFO:tasks.workunit.client.1.smithi153.stdout:0/318: sync 2024-04-16T02:35:33.829 INFO:tasks.workunit.client.1.smithi153.stdout:1/305: creat d4/dd/d13/d23/d5f/f69 x:0 0 0 2024-04-16T02:35:33.829 INFO:tasks.workunit.client.1.smithi153.stdout:6/243: write dc/d12/d15/f22 [1380129,58039] 0 2024-04-16T02:35:33.833 INFO:tasks.workunit.client.1.smithi153.stdout:0/319: creat dd/d14/d51/f60 x:0 0 0 2024-04-16T02:35:33.833 INFO:tasks.workunit.client.1.smithi153.stdout:6/244: chown f9 1199 1 2024-04-16T02:35:33.835 INFO:tasks.workunit.client.1.smithi153.stdout:8/287: dwrite d2/d13/f62 [0,4194304] 0 2024-04-16T02:35:33.835 INFO:tasks.workunit.client.1.smithi153.stdout:4/353: symlink d3/d9/db/de/d2c/l7a 0 2024-04-16T02:35:33.840 INFO:tasks.workunit.client.1.smithi153.stdout:4/354: fdatasync d3/d9/f70 0 2024-04-16T02:35:33.840 INFO:tasks.workunit.client.1.smithi153.stdout:6/245: mknod dc/d10/c52 0 2024-04-16T02:35:33.840 INFO:tasks.workunit.client.1.smithi153.stdout:5/294: rename d8/d1f/d20/d39/l3b to d8/d1f/d20/d39/l55 0 2024-04-16T02:35:33.841 INFO:tasks.workunit.client.1.smithi153.stdout:8/288: chown d2/d13/d2c/l4a 5980 1 2024-04-16T02:35:33.844 INFO:tasks.workunit.client.1.smithi153.stdout:8/289: fsync d2/d13/d3a/f4c 0 2024-04-16T02:35:33.845 INFO:tasks.workunit.client.1.smithi153.stdout:6/246: write dc/d1a/d2d/f44 [821045,68070] 0 2024-04-16T02:35:33.845 INFO:tasks.workunit.client.1.smithi153.stdout:1/306: truncate d4/f1f 53291 0 2024-04-16T02:35:33.847 INFO:tasks.workunit.client.1.smithi153.stdout:7/285: dwrite d0/d4/db/d2b/d53/f27 [0,4194304] 0 2024-04-16T02:35:33.849 INFO:tasks.workunit.client.1.smithi153.stdout:0/320: dread dd/f16 [0,4194304] 0 2024-04-16T02:35:33.849 INFO:tasks.workunit.client.1.smithi153.stdout:6/247: chown dc/fe 3 1 2024-04-16T02:35:33.849 INFO:tasks.workunit.client.1.smithi153.stdout:7/286: fdatasync d0/f3d 0 2024-04-16T02:35:33.850 INFO:tasks.workunit.client.1.smithi153.stdout:8/290: write d2/de/f3e [5076913,21077] 0 2024-04-16T02:35:33.851 INFO:tasks.workunit.client.1.smithi153.stdout:4/355: getdents d3/d9/db/d17/d3f 0 2024-04-16T02:35:33.853 INFO:tasks.workunit.client.1.smithi153.stdout:6/248: write dc/d1a/f40 [217615,37228] 0 2024-04-16T02:35:33.856 INFO:tasks.workunit.client.1.smithi153.stdout:0/321: creat dd/d32/d57/f61 x:0 0 0 2024-04-16T02:35:33.858 INFO:tasks.workunit.client.1.smithi153.stdout:8/291: fsync d2/d13/f1d 0 2024-04-16T02:35:33.858 INFO:tasks.workunit.client.1.smithi153.stdout:4/356: write d3/d9/db/de/f3b [564926,88462] 0 2024-04-16T02:35:33.862 INFO:tasks.workunit.client.1.smithi153.stdout:3/275: sync 2024-04-16T02:35:33.864 INFO:tasks.workunit.client.1.smithi153.stdout:0/322: creat dd/d32/d3f/f62 x:0 0 0 2024-04-16T02:35:33.864 INFO:tasks.workunit.client.1.smithi153.stdout:5/295: dwrite d8/d1f/d20/f40 [0,4194304] 0 2024-04-16T02:35:33.866 INFO:tasks.workunit.client.1.smithi153.stdout:3/276: dread - d6/d10/d22/d2c/f44 zero size 2024-04-16T02:35:33.868 INFO:tasks.workunit.client.1.smithi153.stdout:1/307: dwrite d4/dd/d18/f66 [0,4194304] 0 2024-04-16T02:35:33.872 INFO:tasks.workunit.client.1.smithi153.stdout:7/287: creat d0/d4/db/d2b/d53/d59/f5a x:0 0 0 2024-04-16T02:35:33.872 INFO:tasks.workunit.client.1.smithi153.stdout:7/288: fsync d0/d13/f4a 0 2024-04-16T02:35:33.872 INFO:tasks.workunit.client.1.smithi153.stdout:7/289: fdatasync d0/d13/f32 0 2024-04-16T02:35:33.874 INFO:tasks.workunit.client.1.smithi153.stdout:4/357: mknod d3/d9/db/d17/d3f/d6b/c7b 0 2024-04-16T02:35:33.874 INFO:tasks.workunit.client.1.smithi153.stdout:0/323: unlink dd/d14/f49 0 2024-04-16T02:35:33.875 INFO:tasks.workunit.client.1.smithi153.stdout:5/296: dread d8/f30 [0,4194304] 0 2024-04-16T02:35:33.877 INFO:tasks.workunit.client.1.smithi153.stdout:0/324: fdatasync dd/d14/d19/f27 0 2024-04-16T02:35:33.877 INFO:tasks.workunit.client.1.smithi153.stdout:1/308: dread d4/dd/f1d [0,4194304] 0 2024-04-16T02:35:33.879 INFO:tasks.workunit.client.1.smithi153.stdout:0/325: write dd/f22 [4031172,467] 0 2024-04-16T02:35:33.879 INFO:tasks.workunit.client.1.smithi153.stdout:3/277: mknod d6/d10/c62 0 2024-04-16T02:35:33.880 INFO:tasks.workunit.client.1.smithi153.stdout:6/249: truncate dc/f11 3853116 0 2024-04-16T02:35:33.881 INFO:tasks.workunit.client.1.smithi153.stdout:0/326: read - dd/d28/f40 zero size 2024-04-16T02:35:33.883 INFO:tasks.workunit.client.1.smithi153.stdout:5/297: creat d8/df/d23/f56 x:0 0 0 2024-04-16T02:35:33.885 INFO:tasks.workunit.client.1.smithi153.stdout:8/292: dwrite d2/f14 [0,4194304] 0 2024-04-16T02:35:33.886 INFO:tasks.workunit.client.1.smithi153.stdout:5/298: stat d8/df/l45 0 2024-04-16T02:35:33.888 INFO:tasks.workunit.client.1.smithi153.stdout:8/293: dread - d2/d1c/d1f/f53 zero size 2024-04-16T02:35:33.890 INFO:tasks.workunit.client.1.smithi153.stdout:7/290: creat d0/f5b x:0 0 0 2024-04-16T02:35:33.891 INFO:tasks.workunit.client.1.smithi153.stdout:3/278: dwrite d6/f11 [0,4194304] 0 2024-04-16T02:35:33.892 INFO:tasks.workunit.client.1.smithi153.stdout:6/250: symlink dc/d12/d15/l53 0 2024-04-16T02:35:33.895 INFO:tasks.workunit.client.1.smithi153.stdout:4/358: mknod d3/d9/db/d17/d28/d4c/c7c 0 2024-04-16T02:35:33.896 INFO:tasks.workunit.client.1.smithi153.stdout:4/359: dread - d3/f1c zero size 2024-04-16T02:35:33.899 INFO:tasks.workunit.client.1.smithi153.stdout:8/294: dread d2/d1c/d1f/d27/d2d/f59 [0,4194304] 0 2024-04-16T02:35:33.899 INFO:tasks.workunit.client.1.smithi153.stdout:7/291: mkdir d0/d4/db/d2b/d53/d59/d5c 0 2024-04-16T02:35:33.901 INFO:tasks.workunit.client.1.smithi153.stdout:3/279: readlink d6/d10/d22/l3c 0 2024-04-16T02:35:33.904 INFO:tasks.workunit.client.1.smithi153.stdout:4/360: dread d3/d9/f1a [0,4194304] 0 2024-04-16T02:35:33.905 INFO:tasks.workunit.client.1.smithi153.stdout:7/292: rmdir d0/d13 39 2024-04-16T02:35:33.908 INFO:tasks.workunit.client.1.smithi153.stdout:3/280: creat d6/d10/d22/d2c/f63 x:0 0 0 2024-04-16T02:35:33.908 INFO:tasks.workunit.client.1.smithi153.stdout:0/327: link dd/d28/l45 dd/d32/d57/l63 0 2024-04-16T02:35:33.909 INFO:tasks.workunit.client.1.smithi153.stdout:1/309: getdents d4/dd/d18/d57 0 2024-04-16T02:35:33.910 INFO:tasks.workunit.client.1.smithi153.stdout:9/331: sync 2024-04-16T02:35:33.910 INFO:tasks.workunit.client.1.smithi153.stdout:2/303: sync 2024-04-16T02:35:33.911 INFO:tasks.workunit.client.1.smithi153.stdout:2/304: fsync d9/d2b/d18/d3f/d40/f70 0 2024-04-16T02:35:33.944 INFO:tasks.workunit.client.1.smithi153.stdout:7/293: creat d0/d4/db/d2b/d53/f5d x:0 0 0 2024-04-16T02:35:33.947 INFO:tasks.workunit.client.1.smithi153.stdout:6/251: write dc/d12/d15/f2f [3100968,17275] 0 2024-04-16T02:35:33.949 INFO:tasks.workunit.client.1.smithi153.stdout:7/294: dwrite d0/f3d [0,4194304] 0 2024-04-16T02:35:33.949 INFO:tasks.workunit.client.1.smithi153.stdout:5/299: getdents d8/df/d2f 0 2024-04-16T02:35:33.953 INFO:tasks.workunit.client.1.smithi153.stdout:3/281: rename d6/cd to d6/d10/d22/d2e/c64 0 2024-04-16T02:35:33.954 INFO:tasks.workunit.client.1.smithi153.stdout:3/282: write d6/d10/d22/d2c/f58 [437251,124461] 0 2024-04-16T02:35:33.955 INFO:tasks.workunit.client.1.smithi153.stdout:3/283: chown d6/lf 43 1 2024-04-16T02:35:33.956 INFO:tasks.workunit.client.1.smithi153.stdout:1/310: symlink d4/dd/d18/d29/l6a 0 2024-04-16T02:35:33.958 INFO:tasks.workunit.client.1.smithi153.stdout:2/305: symlink d9/d2b/d18/d3f/l76 0 2024-04-16T02:35:33.958 INFO:tasks.workunit.client.1.smithi153.stdout:2/306: chown d9/d24/d27/f1d 2572 1 2024-04-16T02:35:33.959 INFO:tasks.workunit.client.1.smithi153.stdout:2/307: fdatasync d9/d2b/d18/d3f/f6e 0 2024-04-16T02:35:33.959 INFO:tasks.workunit.client.1.smithi153.stdout:8/295: rmdir d2/d1c/d1f/d57 0 2024-04-16T02:35:33.962 INFO:tasks.workunit.client.1.smithi153.stdout:7/295: mknod d0/d4/db/d2b/d53/d59/c5e 0 2024-04-16T02:35:33.962 INFO:tasks.workunit.client.1.smithi153.stdout:5/300: write d8/df/f22 [3568363,78863] 0 2024-04-16T02:35:33.963 INFO:tasks.workunit.client.1.smithi153.stdout:4/361: link d3/c32 d3/d9/db/d17/d28/c7d 0 2024-04-16T02:35:33.964 INFO:tasks.workunit.client.1.smithi153.stdout:3/284: mknod d6/d10/d22/d2c/c65 0 2024-04-16T02:35:33.964 INFO:tasks.workunit.client.1.smithi153.stdout:4/362: stat d3/d9/db/de/f60 0 2024-04-16T02:35:33.965 INFO:tasks.workunit.client.1.smithi153.stdout:9/332: mkdir d4/d6/dc/d24/d5d/d66 0 2024-04-16T02:35:33.966 INFO:tasks.workunit.client.1.smithi153.stdout:6/252: symlink dc/l54 0 2024-04-16T02:35:33.967 INFO:tasks.workunit.client.1.smithi153.stdout:6/253: chown f9 314 1 2024-04-16T02:35:33.968 INFO:tasks.workunit.client.1.smithi153.stdout:6/254: write dc/d1a/f40 [1199379,37492] 0 2024-04-16T02:35:33.968 INFO:tasks.workunit.client.1.smithi153.stdout:0/328: symlink dd/d28/l64 0 2024-04-16T02:35:33.972 INFO:tasks.workunit.client.1.smithi153.stdout:1/311: unlink d4/dd/d13/d26/f3d 0 2024-04-16T02:35:33.973 INFO:tasks.workunit.client.1.smithi153.stdout:4/363: dwrite d3/d9/d24/d3e/f5a [0,4194304] 0 2024-04-16T02:35:33.973 INFO:tasks.workunit.client.1.smithi153.stdout:9/333: rmdir d4/d6/d1b/d2a 39 2024-04-16T02:35:33.975 INFO:tasks.workunit.client.1.smithi153.stdout:9/334: fsync d4/d6/dc/d2d/f53 0 2024-04-16T02:35:33.975 INFO:tasks.workunit.client.1.smithi153.stdout:0/329: dread dd/d14/d19/d20/f31 [0,4194304] 0 2024-04-16T02:35:33.977 INFO:tasks.workunit.client.1.smithi153.stdout:9/335: write d4/f27 [1096067,59471] 0 2024-04-16T02:35:33.982 INFO:tasks.workunit.client.1.smithi153.stdout:1/312: dwrite d4/dd/d18/d29/f4a [0,4194304] 0 2024-04-16T02:35:33.982 INFO:tasks.workunit.client.1.smithi153.stdout:4/364: rename d3/d9/d24/d75/l76 to d3/d9/db/de/d2c/d38/l7e 0 2024-04-16T02:35:33.984 INFO:tasks.workunit.client.1.smithi153.stdout:2/308: rmdir d9/d2b/d18/d3f 39 2024-04-16T02:35:33.985 INFO:tasks.workunit.client.1.smithi153.stdout:0/330: dwrite dd/d32/d57/f5d [0,4194304] 0 2024-04-16T02:35:33.986 INFO:tasks.workunit.client.1.smithi153.stdout:6/255: link dc/d1a/d2d/f44 dc/d47/f55 0 2024-04-16T02:35:33.987 INFO:tasks.workunit.client.1.smithi153.stdout:9/336: symlink d4/d33/l67 0 2024-04-16T02:35:33.990 INFO:tasks.workunit.client.1.smithi153.stdout:0/331: dread dd/d32/d57/f5d [0,4194304] 0 2024-04-16T02:35:33.990 INFO:tasks.workunit.client.1.smithi153.stdout:3/285: getdents d6/d10/d22/d2e 0 2024-04-16T02:35:33.994 INFO:tasks.workunit.client.1.smithi153.stdout:4/365: creat d3/d9/db/d45/f7f x:0 0 0 2024-04-16T02:35:33.994 INFO:tasks.workunit.client.1.smithi153.stdout:4/366: chown d3/d9/lc 972093714 1 2024-04-16T02:35:33.996 INFO:tasks.workunit.client.1.smithi153.stdout:1/313: rename d4/dd/d13/d23/d31/c46 to d4/dd/d13/d26/d58/c6b 0 2024-04-16T02:35:34.000 INFO:tasks.workunit.client.1.smithi153.stdout:6/256: symlink dc/d47/l56 0 2024-04-16T02:35:34.001 INFO:tasks.workunit.client.1.smithi153.stdout:6/257: write dc/d10/f37 [266856,47684] 0 2024-04-16T02:35:34.003 INFO:tasks.workunit.client.1.smithi153.stdout:0/332: unlink dd/f16 0 2024-04-16T02:35:34.004 INFO:tasks.workunit.client.1.smithi153.stdout:5/301: truncate d8/df/f22 3166875 0 2024-04-16T02:35:34.005 INFO:tasks.workunit.client.1.smithi153.stdout:4/367: creat d3/d9/d24/d3e/f80 x:0 0 0 2024-04-16T02:35:34.005 INFO:tasks.workunit.client.1.smithi153.stdout:5/302: chown d8/d1f/d20/d39/f38 732010641 1 2024-04-16T02:35:34.006 INFO:tasks.workunit.client.1.smithi153.stdout:1/314: mkdir d4/dd/d18/d6c 0 2024-04-16T02:35:34.006 INFO:tasks.workunit.client.1.smithi153.stdout:5/303: truncate d8/df/d23/f56 500070 0 2024-04-16T02:35:34.007 INFO:tasks.workunit.client.1.smithi153.stdout:5/304: chown d8/df 172635 1 2024-04-16T02:35:34.007 INFO:tasks.workunit.client.1.smithi153.stdout:9/337: dread d4/d6/d1b/f59 [0,4194304] 0 2024-04-16T02:35:34.012 INFO:tasks.workunit.client.1.smithi153.stdout:6/258: mkdir dc/d47/d57 0 2024-04-16T02:35:34.013 INFO:tasks.workunit.client.1.smithi153.stdout:9/338: dwrite d4/d6/dc/d46/f49 [0,4194304] 0 2024-04-16T02:35:34.016 INFO:tasks.workunit.client.1.smithi153.stdout:4/368: symlink d3/d9/db/de/d2c/d52/l81 0 2024-04-16T02:35:34.016 INFO:tasks.workunit.client.1.smithi153.stdout:4/369: stat d3/d9/db/d17/d3f 0 2024-04-16T02:35:34.017 INFO:tasks.workunit.client.1.smithi153.stdout:1/315: creat d4/dd/d13/d23/f6d x:0 0 0 2024-04-16T02:35:34.017 INFO:tasks.workunit.client.1.smithi153.stdout:4/370: truncate d3/f6d 557216 0 2024-04-16T02:35:34.019 INFO:tasks.workunit.client.1.smithi153.stdout:2/309: link d9/d24/d34/d36/c5b d9/d2b/d1c/d20/c77 0 2024-04-16T02:35:34.024 INFO:tasks.workunit.client.1.smithi153.stdout:1/316: dwrite d4/dd/f16 [0,4194304] 0 2024-04-16T02:35:34.026 INFO:tasks.workunit.client.1.smithi153.stdout:4/371: dwrite d3/d9/f1a [4194304,4194304] 0 2024-04-16T02:35:34.026 INFO:tasks.workunit.client.1.smithi153.stdout:9/339: rename d4/d6/dc/d46/f54 to d4/d33/d3a/d40/d4a/f68 0 2024-04-16T02:35:34.027 INFO:tasks.workunit.client.1.smithi153.stdout:4/372: stat d3/d9/d79 0 2024-04-16T02:35:34.029 INFO:tasks.workunit.client.1.smithi153.stdout:3/286: link d6/d10/d3b/l48 d6/d10/d22/d2c/l66 0 2024-04-16T02:35:34.030 INFO:tasks.workunit.client.1.smithi153.stdout:3/287: fdatasync d6/d10/d22/d2e/f4e 0 2024-04-16T02:35:34.037 INFO:tasks.workunit.client.1.smithi153.stdout:1/317: mkdir d4/dd/d18/d29/d6e 0 2024-04-16T02:35:34.037 INFO:tasks.workunit.client.1.smithi153.stdout:0/333: dwrite dd/d14/d19/f1b [0,4194304] 0 2024-04-16T02:35:34.039 INFO:tasks.workunit.client.1.smithi153.stdout:1/318: write d4/dd/d13/d23/f5d [352857,97904] 0 2024-04-16T02:35:34.040 INFO:tasks.workunit.client.1.smithi153.stdout:9/340: rmdir d4/d6/dc/d46 39 2024-04-16T02:35:34.041 INFO:tasks.workunit.client.1.smithi153.stdout:9/341: read d4/d6/f1a [572965,1362] 0 2024-04-16T02:35:34.041 INFO:tasks.workunit.client.1.smithi153.stdout:4/373: symlink d3/d9/d24/d35/l82 0 2024-04-16T02:35:34.043 INFO:tasks.workunit.client.1.smithi153.stdout:4/374: chown d3/d9/db/de/d2c/d52/l81 45660165 1 2024-04-16T02:35:34.043 INFO:tasks.workunit.client.1.smithi153.stdout:5/305: unlink d8/df/f22 0 2024-04-16T02:35:34.044 INFO:tasks.workunit.client.1.smithi153.stdout:6/259: truncate dc/fe 3402990 0 2024-04-16T02:35:34.045 INFO:tasks.workunit.client.1.smithi153.stdout:4/375: write d3/d9/db/de/d2c/f56 [711035,74376] 0 2024-04-16T02:35:34.046 INFO:tasks.workunit.client.1.smithi153.stdout:5/306: write d8/f4d [385306,52948] 0 2024-04-16T02:35:34.048 INFO:tasks.workunit.client.1.smithi153.stdout:3/288: unlink d6/d10/d22/l46 0 2024-04-16T02:35:34.050 INFO:tasks.workunit.client.1.smithi153.stdout:4/376: chown d3/d9/db/d17/l18 7 1 2024-04-16T02:35:34.050 INFO:tasks.workunit.client.1.smithi153.stdout:3/289: read d6/d10/f18 [353967,78888] 0 2024-04-16T02:35:34.051 INFO:tasks.workunit.client.1.smithi153.stdout:9/342: dwrite d4/d33/d3a/f5f [0,4194304] 0 2024-04-16T02:35:34.053 INFO:tasks.workunit.client.1.smithi153.stdout:0/334: mkdir dd/d14/d51/d54/d65 0 2024-04-16T02:35:34.054 INFO:tasks.workunit.client.1.smithi153.stdout:2/310: getdents d9 0 2024-04-16T02:35:34.056 INFO:tasks.workunit.client.1.smithi153.stdout:6/260: fsync dc/d12/d15/f19 0 2024-04-16T02:35:34.056 INFO:tasks.workunit.client.1.smithi153.stdout:0/335: fdatasync dd/f10 0 2024-04-16T02:35:34.061 INFO:tasks.workunit.client.1.smithi153.stdout:6/261: readlink dc/l18 0 2024-04-16T02:35:34.061 INFO:tasks.workunit.client.1.smithi153.stdout:5/307: dread d8/df/f16 [0,4194304] 0 2024-04-16T02:35:34.063 INFO:tasks.workunit.client.1.smithi153.stdout:6/262: fdatasync dc/d12/d15/f20 0 2024-04-16T02:35:34.063 INFO:tasks.workunit.client.1.smithi153.stdout:9/343: dread d4/d6/dc/d2d/f58 [0,4194304] 0 2024-04-16T02:35:34.063 INFO:tasks.workunit.client.1.smithi153.stdout:2/311: creat d9/d24/d27/d2c/d41/f78 x:0 0 0 2024-04-16T02:35:34.068 INFO:tasks.workunit.client.1.smithi153.stdout:6/263: write dc/f17 [3123822,22553] 0 2024-04-16T02:35:34.068 INFO:tasks.workunit.client.1.smithi153.stdout:3/290: dwrite d6/d10/f1e [0,4194304] 0 2024-04-16T02:35:34.068 INFO:tasks.workunit.client.1.smithi153.stdout:9/344: chown d4/d6/d1b/f59 116 1 2024-04-16T02:35:34.070 INFO:tasks.workunit.client.1.smithi153.stdout:1/319: dwrite d4/dd/d13/d26/f2b [0,4194304] 0 2024-04-16T02:35:34.074 INFO:tasks.workunit.client.1.smithi153.stdout:0/336: dwrite dd/d26/d44/f50 [0,4194304] 0 2024-04-16T02:35:34.076 INFO:tasks.workunit.client.1.smithi153.stdout:6/264: dread f6 [0,4194304] 0 2024-04-16T02:35:34.078 INFO:tasks.workunit.client.1.smithi153.stdout:2/312: rename d9/d24/d34/f4f to d9/d2b/d1c/d20/f79 0 2024-04-16T02:35:34.081 INFO:tasks.workunit.client.1.smithi153.stdout:6/265: rename dc/d1a/f40 to dc/d1a/f58 0 2024-04-16T02:35:34.082 INFO:tasks.workunit.client.1.smithi153.stdout:6/266: chown dc/d12/d15/l53 24098915 1 2024-04-16T02:35:34.082 INFO:tasks.workunit.client.1.smithi153.stdout:6/267: stat dc/d1a/f26 0 2024-04-16T02:35:34.084 INFO:tasks.workunit.client.1.smithi153.stdout:3/291: rename d6/c9 to d6/d10/d22/d2e/c67 0 2024-04-16T02:35:34.085 INFO:tasks.workunit.client.1.smithi153.stdout:9/345: mkdir d4/d33/d3a/d69 0 2024-04-16T02:35:34.087 INFO:tasks.workunit.client.1.smithi153.stdout:3/292: dread - d6/d10/d22/d2c/f5f zero size 2024-04-16T02:35:34.087 INFO:tasks.workunit.client.1.smithi153.stdout:1/320: creat d4/f6f x:0 0 0 2024-04-16T02:35:34.089 INFO:tasks.workunit.client.1.smithi153.stdout:9/346: dread - d4/d6/dc/d24/f43 zero size 2024-04-16T02:35:34.089 INFO:tasks.workunit.client.1.smithi153.stdout:2/313: read d9/d2b/d18/d3f/f49 [12198,41029] 0 2024-04-16T02:35:34.092 INFO:tasks.workunit.client.1.smithi153.stdout:1/321: mknod d4/dd/d13/d23/c70 0 2024-04-16T02:35:34.093 INFO:tasks.workunit.client.1.smithi153.stdout:6/268: dwrite dc/d12/d15/f22 [0,4194304] 0 2024-04-16T02:35:34.093 INFO:tasks.workunit.client.1.smithi153.stdout:9/347: write d4/d6/d1b/f3e [687836,58909] 0 2024-04-16T02:35:34.097 INFO:tasks.workunit.client.1.smithi153.stdout:1/322: chown d4/dd/d20/l44 0 1 2024-04-16T02:35:34.097 INFO:tasks.workunit.client.1.smithi153.stdout:6/269: mkdir dc/d59 0 2024-04-16T02:35:34.098 INFO:tasks.workunit.client.1.smithi153.stdout:0/337: dwrite dd/d28/f3c [0,4194304] 0 2024-04-16T02:35:34.099 INFO:tasks.workunit.client.1.smithi153.stdout:9/348: mknod d4/d33/d3a/d69/c6a 0 2024-04-16T02:35:34.099 INFO:tasks.workunit.client.1.smithi153.stdout:1/323: write d4/f40 [751032,84014] 0 2024-04-16T02:35:34.101 INFO:tasks.workunit.client.1.smithi153.stdout:9/349: fsync d4/d33/f4f 0 2024-04-16T02:35:34.101 INFO:tasks.workunit.client.1.smithi153.stdout:0/338: write dd/ff [4639179,2684] 0 2024-04-16T02:35:34.109 INFO:tasks.workunit.client.1.smithi153.stdout:2/314: dwrite d9/d2b/f17 [0,4194304] 0 2024-04-16T02:35:34.112 INFO:tasks.workunit.client.1.smithi153.stdout:6/270: rename dc/d10/f37 to dc/d12/d15/f5a 0 2024-04-16T02:35:34.114 INFO:tasks.workunit.client.1.smithi153.stdout:6/271: chown dc/d59 21467 1 2024-04-16T02:35:34.114 INFO:tasks.workunit.client.1.smithi153.stdout:2/315: write d9/d2b/f6b [391655,111481] 0 2024-04-16T02:35:34.115 INFO:tasks.workunit.client.1.smithi153.stdout:1/324: mkdir d4/dd/d18/d29/d71 0 2024-04-16T02:35:34.116 INFO:tasks.workunit.client.1.smithi153.stdout:9/350: mknod d4/d33/d3a/d40/d4a/c6b 0 2024-04-16T02:35:34.116 INFO:tasks.workunit.client.1.smithi153.stdout:6/272: chown f9 4265 1 2024-04-16T02:35:34.118 INFO:tasks.workunit.client.1.smithi153.stdout:0/339: dwrite dd/d14/d19/f1b [0,4194304] 0 2024-04-16T02:35:34.118 INFO:tasks.workunit.client.1.smithi153.stdout:1/325: readlink d4/dd/d13/l17 0 2024-04-16T02:35:34.120 INFO:tasks.workunit.client.1.smithi153.stdout:1/326: chown d4/dd/d18/f66 3455945 1 2024-04-16T02:35:34.121 INFO:tasks.workunit.client.1.smithi153.stdout:2/316: mkdir d9/d24/d27/d2c/d41/d7a 0 2024-04-16T02:35:34.123 INFO:tasks.workunit.client.1.smithi153.stdout:3/293: dwrite d6/f20 [0,4194304] 0 2024-04-16T02:35:34.124 INFO:tasks.workunit.client.1.smithi153.stdout:1/327: chown d4/dd/d13/d23/f6d 98 1 2024-04-16T02:35:34.125 INFO:tasks.workunit.client.1.smithi153.stdout:3/294: read - d6/d10/d22/d2e/f42 zero size 2024-04-16T02:35:34.126 INFO:tasks.workunit.client.1.smithi153.stdout:1/328: write d4/f49 [194286,17616] 0 2024-04-16T02:35:34.127 INFO:tasks.workunit.client.1.smithi153.stdout:6/273: mkdir dc/d59/d5b 0 2024-04-16T02:35:34.131 INFO:tasks.workunit.client.1.smithi153.stdout:0/340: dwrite dd/f22 [0,4194304] 0 2024-04-16T02:35:34.131 INFO:tasks.workunit.client.1.smithi153.stdout:6/274: mknod dc/d1a/c5c 0 2024-04-16T02:35:34.131 INFO:tasks.workunit.client.1.smithi153.stdout:3/295: mknod d6/d10/c68 0 2024-04-16T02:35:34.133 INFO:tasks.workunit.client.1.smithi153.stdout:2/317: getdents d9/d2b/d1c 0 2024-04-16T02:35:34.136 INFO:tasks.workunit.client.1.smithi153.stdout:6/275: fdatasync dc/d10/f1b 0 2024-04-16T02:35:34.136 INFO:tasks.workunit.client.1.smithi153.stdout:0/341: mknod dd/d32/d3f/c66 0 2024-04-16T02:35:34.136 INFO:tasks.workunit.client.1.smithi153.stdout:2/318: read - d9/d24/d27/f6f zero size 2024-04-16T02:35:34.138 INFO:tasks.workunit.client.1.smithi153.stdout:2/319: readlink d9/d2b/d18/l31 0 2024-04-16T02:35:34.139 INFO:tasks.workunit.client.1.smithi153.stdout:6/276: mknod dc/d12/d15/c5d 0 2024-04-16T02:35:34.139 INFO:tasks.workunit.client.1.smithi153.stdout:0/342: symlink dd/d26/l67 0 2024-04-16T02:35:34.141 INFO:tasks.workunit.client.1.smithi153.stdout:0/343: fsync dd/d14/d19/d1c/f48 0 2024-04-16T02:35:34.141 INFO:tasks.workunit.client.1.smithi153.stdout:1/329: read d4/dd/d20/f55 [51178,127800] 0 2024-04-16T02:35:34.143 INFO:tasks.workunit.client.1.smithi153.stdout:3/296: link d6/d10/d22/d2e/f42 d6/d10/d22/d43/f69 0 2024-04-16T02:35:34.143 INFO:tasks.workunit.client.1.smithi153.stdout:1/330: read d4/dd/f1d [287530,90071] 0 2024-04-16T02:35:34.144 INFO:tasks.workunit.client.1.smithi153.stdout:3/297: read d6/d10/d22/d2c/f58 [196715,62278] 0 2024-04-16T02:35:34.146 INFO:tasks.workunit.client.1.smithi153.stdout:2/320: getdents d9/d2b 0 2024-04-16T02:35:34.147 INFO:tasks.workunit.client.1.smithi153.stdout:1/331: creat d4/dd/d20/f72 x:0 0 0 2024-04-16T02:35:34.148 INFO:tasks.workunit.client.1.smithi153.stdout:0/344: mkdir dd/d14/d51/d54/d65/d68 0 2024-04-16T02:35:34.149 INFO:tasks.workunit.client.1.smithi153.stdout:2/321: creat d9/d24/d34/d36/f7b x:0 0 0 2024-04-16T02:35:34.149 INFO:tasks.workunit.client.1.smithi153.stdout:8/296: sync 2024-04-16T02:35:34.150 INFO:tasks.workunit.client.1.smithi153.stdout:2/322: write d9/d2b/f46 [1514401,80992] 0 2024-04-16T02:35:34.150 INFO:tasks.workunit.client.1.smithi153.stdout:3/298: symlink d6/d10/d3b/d59/l6a 0 2024-04-16T02:35:34.151 INFO:tasks.workunit.client.1.smithi153.stdout:1/332: symlink d4/l73 0 2024-04-16T02:35:34.152 INFO:tasks.workunit.client.1.smithi153.stdout:2/323: dread - d9/d24/d34/d36/f7b zero size 2024-04-16T02:35:34.152 INFO:tasks.workunit.client.1.smithi153.stdout:8/297: symlink d2/d1c/d1f/l64 0 2024-04-16T02:35:34.153 INFO:tasks.workunit.client.1.smithi153.stdout:8/298: fsync d2/de/f5f 0 2024-04-16T02:35:34.154 INFO:tasks.workunit.client.1.smithi153.stdout:1/333: dread d4/dd/d20/f55 [0,4194304] 0 2024-04-16T02:35:34.157 INFO:tasks.workunit.client.1.smithi153.stdout:3/299: link d6/d10/d22/d2c/c65 d6/d10/d5b/c6b 0 2024-04-16T02:35:34.158 INFO:tasks.workunit.client.1.smithi153.stdout:8/299: symlink d2/l65 0 2024-04-16T02:35:34.160 INFO:tasks.workunit.client.1.smithi153.stdout:1/334: unlink d4/dd/d13/l21 0 2024-04-16T02:35:34.160 INFO:tasks.workunit.client.1.smithi153.stdout:0/345: dread dd/f10 [0,4194304] 0 2024-04-16T02:35:34.161 INFO:tasks.workunit.client.1.smithi153.stdout:2/324: dwrite d9/d2b/d18/d3f/f6e [0,4194304] 0 2024-04-16T02:35:34.163 INFO:tasks.workunit.client.1.smithi153.stdout:8/300: stat d2/l3 0 2024-04-16T02:35:34.163 INFO:tasks.workunit.client.1.smithi153.stdout:3/300: mknod d6/d10/c6c 0 2024-04-16T02:35:34.164 INFO:tasks.workunit.client.1.smithi153.stdout:0/346: readlink dd/d32/d57/l63 0 2024-04-16T02:35:34.166 INFO:tasks.workunit.client.1.smithi153.stdout:0/347: dread - dd/d32/d57/f4e zero size 2024-04-16T02:35:34.167 INFO:tasks.workunit.client.1.smithi153.stdout:2/325: mkdir d9/d24/d34/d60/d7c 0 2024-04-16T02:35:34.167 INFO:tasks.workunit.client.1.smithi153.stdout:8/301: mkdir d2/d13/d3a/d66 0 2024-04-16T02:35:34.169 INFO:tasks.workunit.client.1.smithi153.stdout:3/301: creat d6/d10/d3b/f6d x:0 0 0 2024-04-16T02:35:34.172 INFO:tasks.workunit.client.1.smithi153.stdout:8/302: mkdir d2/d1c/d1f/d27/d2d/d47/d60/d67 0 2024-04-16T02:35:34.173 INFO:tasks.workunit.client.1.smithi153.stdout:3/302: rename d6/d10/f1e to d6/d10/d22/d4d/f6e 0 2024-04-16T02:35:34.175 INFO:tasks.workunit.client.1.smithi153.stdout:6/277: getdents dc/d12/d15 0 2024-04-16T02:35:34.175 INFO:tasks.workunit.client.1.smithi153.stdout:9/351: dwrite d4/d6/f56 [0,4194304] 0 2024-04-16T02:35:34.178 INFO:tasks.workunit.client.1.smithi153.stdout:6/278: write dc/d1a/f1c [3008044,81710] 0 2024-04-16T02:35:34.180 INFO:tasks.workunit.client.1.smithi153.stdout:2/326: dread d9/d2b/d18/d3f/f49 [0,4194304] 0 2024-04-16T02:35:34.180 INFO:tasks.workunit.client.1.smithi153.stdout:0/348: dwrite dd/d14/d19/f46 [0,4194304] 0 2024-04-16T02:35:34.182 INFO:tasks.workunit.client.1.smithi153.stdout:3/303: truncate d6/d10/d22/d43/f57 4700366 0 2024-04-16T02:35:34.186 INFO:tasks.workunit.client.1.smithi153.stdout:9/352: creat d4/d33/d3a/d69/f6c x:0 0 0 2024-04-16T02:35:34.186 INFO:tasks.workunit.client.1.smithi153.stdout:2/327: readlink d9/d2b/l65 0 2024-04-16T02:35:34.189 INFO:tasks.workunit.client.1.smithi153.stdout:3/304: rmdir d6/d10 39 2024-04-16T02:35:34.190 INFO:tasks.workunit.client.1.smithi153.stdout:2/328: truncate d9/d24/d27/f6f 1023770 0 2024-04-16T02:35:34.191 INFO:tasks.workunit.client.1.smithi153.stdout:8/303: dwrite d2/de/d34/f40 [0,4194304] 0 2024-04-16T02:35:34.194 INFO:tasks.workunit.client.1.smithi153.stdout:9/353: dwrite d4/f47 [0,4194304] 0 2024-04-16T02:35:34.198 INFO:tasks.workunit.client.1.smithi153.stdout:8/304: creat d2/de/d34/f68 x:0 0 0 2024-04-16T02:35:34.200 INFO:tasks.workunit.client.1.smithi153.stdout:3/305: creat d6/d10/d22/d2e/f6f x:0 0 0 2024-04-16T02:35:34.202 INFO:tasks.workunit.client.1.smithi153.stdout:0/349: read dd/d28/f41 [1204441,106444] 0 2024-04-16T02:35:34.203 INFO:tasks.workunit.client.1.smithi153.stdout:7/296: sync 2024-04-16T02:35:34.206 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:33 smithi151 ceph-mon[29003]: pgmap v14: 65 pgs: 65 active+clean; 637 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.1 MiB/s rd, 78 MiB/s wr, 431 op/s 2024-04-16T02:35:34.207 INFO:tasks.workunit.client.1.smithi153.stdout:7/297: write d0/f5b [734091,1166] 0 2024-04-16T02:35:34.209 INFO:tasks.workunit.client.1.smithi153.stdout:9/354: dwrite d4/d6/dc/d2d/f53 [0,4194304] 0 2024-04-16T02:35:34.209 INFO:tasks.workunit.client.1.smithi153.stdout:8/305: creat d2/d13/d3a/d66/f69 x:0 0 0 2024-04-16T02:35:34.211 INFO:tasks.workunit.client.1.smithi153.stdout:0/350: link dd/d14/d51/f60 dd/d14/d51/d54/d65/f69 0 2024-04-16T02:35:34.212 INFO:tasks.workunit.client.1.smithi153.stdout:6/279: truncate dc/d1a/f58 789462 0 2024-04-16T02:35:34.212 INFO:tasks.workunit.client.1.smithi153.stdout:1/335: dread d4/dd/d13/f54 [0,4194304] 0 2024-04-16T02:35:34.214 INFO:tasks.workunit.client.1.smithi153.stdout:9/355: readlink d4/d6/d1b/l35 0 2024-04-16T02:35:34.214 INFO:tasks.workunit.client.1.smithi153.stdout:8/306: write d2/f9 [4541783,74998] 0 2024-04-16T02:35:34.214 INFO:tasks.workunit.client.1.smithi153.stdout:1/336: stat d4/dd/d13/d26 0 2024-04-16T02:35:34.216 INFO:tasks.workunit.client.1.smithi153.stdout:9/356: write d4/d6/dc/d24/f43 [81827,118737] 0 2024-04-16T02:35:34.218 INFO:tasks.workunit.client.1.smithi153.stdout:0/351: symlink dd/d14/d19/d20/d4a/l6a 0 2024-04-16T02:35:34.223 INFO:tasks.workunit.client.1.smithi153.stdout:0/352: read - dd/d26/f5b zero size 2024-04-16T02:35:34.224 INFO:tasks.workunit.client.1.smithi153.stdout:1/337: link d4/dd/d18/d29/c38 d4/dd/d18/d29/d6e/c74 0 2024-04-16T02:35:34.227 INFO:tasks.workunit.client.1.smithi153.stdout:6/280: dread dc/d1a/d2d/f3f [0,4194304] 0 2024-04-16T02:35:34.229 INFO:tasks.workunit.client.1.smithi153.stdout:8/307: dwrite d2/de/f5f [0,4194304] 0 2024-04-16T02:35:34.231 INFO:tasks.workunit.client.1.smithi153.stdout:1/338: symlink d4/dd/d13/d23/l75 0 2024-04-16T02:35:34.233 INFO:tasks.workunit.client.1.smithi153.stdout:9/357: dwrite d4/f47 [0,4194304] 0 2024-04-16T02:35:34.233 INFO:tasks.workunit.client.1.smithi153.stdout:6/281: chown dc/ff 255475 1 2024-04-16T02:35:34.233 INFO:tasks.workunit.client.1.smithi153.stdout:6/282: chown dc/d1a 206429343 1 2024-04-16T02:35:34.236 INFO:tasks.workunit.client.1.smithi153.stdout:0/353: rename dd/d32/d57/l3d to dd/d14/d19/d20/l6b 0 2024-04-16T02:35:34.236 INFO:tasks.workunit.client.1.smithi153.stdout:2/329: dwrite d9/d2b/d18/d3f/f49 [0,4194304] 0 2024-04-16T02:35:34.237 INFO:tasks.workunit.client.1.smithi153.stdout:4/377: sync 2024-04-16T02:35:34.237 INFO:tasks.workunit.client.1.smithi153.stdout:5/308: sync 2024-04-16T02:35:34.239 INFO:tasks.workunit.client.1.smithi153.stdout:7/298: truncate d0/d4/db/d2b/f38 2271445 0 2024-04-16T02:35:34.239 INFO:tasks.workunit.client.1.smithi153.stdout:3/306: dread d6/d10/f32 [0,4194304] 0 2024-04-16T02:35:34.239 INFO:tasks.workunit.client.1.smithi153.stdout:8/308: fsync d2/d13/d2c/f32 0 2024-04-16T02:35:34.242 INFO:tasks.workunit.client.1.smithi153.stdout:9/358: chown d4/d33/d3a/l65 13643 1 2024-04-16T02:35:34.243 INFO:tasks.workunit.client.1.smithi153.stdout:7/299: readlink d0/d4/db/d2b/l11 0 2024-04-16T02:35:34.245 INFO:tasks.workunit.client.1.smithi153.stdout:8/309: read d2/f14 [1347866,105653] 0 2024-04-16T02:35:34.245 INFO:tasks.workunit.client.1.smithi153.stdout:4/378: dread - d3/d9/db/d17/d28/f5e zero size 2024-04-16T02:35:34.246 INFO:tasks.workunit.client.1.smithi153.stdout:8/310: fsync d2/f9 0 2024-04-16T02:35:34.246 INFO:tasks.workunit.client.1.smithi153.stdout:3/307: symlink d6/d10/d22/d43/l70 0 2024-04-16T02:35:34.247 INFO:tasks.workunit.client.1.smithi153.stdout:5/309: chown d8/df/c19 1656472763 1 2024-04-16T02:35:34.247 INFO:tasks.workunit.client.1.smithi153.stdout:0/354: mkdir dd/d32/d57/d6c 0 2024-04-16T02:35:34.248 INFO:tasks.workunit.client.1.smithi153.stdout:6/283: symlink dc/d47/d57/l5e 0 2024-04-16T02:35:34.249 INFO:tasks.workunit.client.1.smithi153.stdout:0/355: chown dd/d26 2 1 2024-04-16T02:35:34.249 INFO:tasks.workunit.client.1.smithi153.stdout:2/330: creat d9/d24/d34/f7d x:0 0 0 2024-04-16T02:35:34.251 INFO:tasks.workunit.client.1.smithi153.stdout:1/339: dread d4/dd/d18/d29/f4a [0,4194304] 0 2024-04-16T02:35:34.251 INFO:tasks.workunit.client.1.smithi153.stdout:4/379: unlink d3/d9/f70 0 2024-04-16T02:35:34.252 INFO:tasks.workunit.client.1.smithi153.stdout:7/300: mkdir d0/d4/db/d34/d51/d5f 0 2024-04-16T02:35:34.253 INFO:tasks.workunit.client.1.smithi153.stdout:8/311: write d2/d1c/d1f/d27/d2d/f59 [579301,94091] 0 2024-04-16T02:35:34.254 INFO:tasks.workunit.client.1.smithi153.stdout:5/310: creat d8/df/d2e/f57 x:0 0 0 2024-04-16T02:35:34.256 INFO:tasks.workunit.client.1.smithi153.stdout:0/356: creat dd/d14/d51/d54/d65/f6d x:0 0 0 2024-04-16T02:35:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:33 smithi153 ceph-mon[36747]: pgmap v14: 65 pgs: 65 active+clean; 637 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.1 MiB/s rd, 78 MiB/s wr, 431 op/s 2024-04-16T02:35:34.284 INFO:tasks.workunit.client.1.smithi153.stdout:8/312: fsync d2/d1c/d1f/f3c 0 2024-04-16T02:35:34.285 INFO:tasks.workunit.client.1.smithi153.stdout:4/380: mknod d3/d9/db/d17/d28/c83 0 2024-04-16T02:35:34.287 INFO:tasks.workunit.client.1.smithi153.stdout:9/359: dwrite d4/d6/f22 [0,4194304] 0 2024-04-16T02:35:34.291 INFO:tasks.workunit.client.1.smithi153.stdout:9/360: chown d4/d6/c39 3138920 1 2024-04-16T02:35:34.296 INFO:tasks.workunit.client.1.smithi153.stdout:5/311: dwrite d8/d1f/d20/f40 [0,4194304] 0 2024-04-16T02:35:34.299 INFO:tasks.workunit.client.1.smithi153.stdout:0/357: dwrite dd/f10 [0,4194304] 0 2024-04-16T02:35:34.302 INFO:tasks.workunit.client.1.smithi153.stdout:4/381: dwrite d3/d9/db/d17/f55 [0,4194304] 0 2024-04-16T02:35:34.305 INFO:tasks.workunit.client.1.smithi153.stdout:6/284: dread dc/f17 [0,4194304] 0 2024-04-16T02:35:34.354 INFO:tasks.workunit.client.1.smithi153.stdout:1/340: sync 2024-04-16T02:35:34.405 INFO:tasks.workunit.client.1.smithi153.stdout:2/331: write d9/d2b/f19 [128417,94303] 0 2024-04-16T02:35:34.414 INFO:tasks.workunit.client.1.smithi153.stdout:5/312: mknod d8/d1f/d20/d39/c58 0 2024-04-16T02:35:34.416 INFO:tasks.workunit.client.1.smithi153.stdout:6/285: symlink dc/d10/l5f 0 2024-04-16T02:35:34.417 INFO:tasks.workunit.client.1.smithi153.stdout:6/286: write dc/d1a/f1c [817534,72579] 0 2024-04-16T02:35:34.418 INFO:tasks.workunit.client.1.smithi153.stdout:1/341: rmdir d4/dd/d13/d23 39 2024-04-16T02:35:34.418 INFO:tasks.workunit.client.1.smithi153.stdout:6/287: dread dc/d1a/f26 [0,4194304] 0 2024-04-16T02:35:34.421 INFO:tasks.workunit.client.1.smithi153.stdout:1/342: dread d4/f36 [0,4194304] 0 2024-04-16T02:35:34.421 INFO:tasks.workunit.client.1.smithi153.stdout:1/343: fsync d4/dd/d18/f51 0 2024-04-16T02:35:34.422 INFO:tasks.workunit.client.1.smithi153.stdout:1/344: read - d4/dd/d18/d29/f59 zero size 2024-04-16T02:35:34.427 INFO:tasks.workunit.client.1.smithi153.stdout:5/313: unlink d8/d1f/d20/d39/f38 0 2024-04-16T02:35:34.433 INFO:tasks.workunit.client.1.smithi153.stdout:2/332: mknod d9/d24/d34/d60/d7c/c7e 0 2024-04-16T02:35:34.433 INFO:tasks.workunit.client.1.smithi153.stdout:2/333: read d9/d2b/f17 [3043436,83913] 0 2024-04-16T02:35:34.433 INFO:tasks.workunit.client.1.smithi153.stdout:2/334: chown d9/d2b 5830076 1 2024-04-16T02:35:34.434 INFO:tasks.workunit.client.1.smithi153.stdout:1/345: creat d4/dd/d18/f76 x:0 0 0 2024-04-16T02:35:34.439 INFO:tasks.workunit.client.1.smithi153.stdout:2/335: dwrite d9/d24/d27/f6f [0,4194304] 0 2024-04-16T02:35:34.441 INFO:tasks.workunit.client.1.smithi153.stdout:2/336: dread - d9/d2b/d1c/d20/f79 zero size 2024-04-16T02:35:34.441 INFO:tasks.workunit.client.1.smithi153.stdout:6/288: fsync dc/d1a/f58 0 2024-04-16T02:35:34.442 INFO:tasks.workunit.client.1.smithi153.stdout:6/289: write dc/f21 [68836,43647] 0 2024-04-16T02:35:34.447 INFO:tasks.workunit.client.1.smithi153.stdout:3/308: link d6/d10/c16 d6/d10/c71 0 2024-04-16T02:35:34.453 INFO:tasks.workunit.client.1.smithi153.stdout:2/337: mknod d9/d2b/d5c/d5f/c7f 0 2024-04-16T02:35:34.453 INFO:tasks.workunit.client.1.smithi153.stdout:3/309: dwrite d6/d10/d22/d2c/f3d [0,4194304] 0 2024-04-16T02:35:34.454 INFO:tasks.workunit.client.1.smithi153.stdout:6/290: write f6 [899903,30018] 0 2024-04-16T02:35:34.456 INFO:tasks.workunit.client.1.smithi153.stdout:6/291: write f6 [5232058,77757] 0 2024-04-16T02:35:34.458 INFO:tasks.workunit.client.1.smithi153.stdout:1/346: mkdir d4/dd/d77 0 2024-04-16T02:35:34.459 INFO:tasks.workunit.client.1.smithi153.stdout:5/314: creat d8/f59 x:0 0 0 2024-04-16T02:35:34.460 INFO:tasks.workunit.client.1.smithi153.stdout:5/315: fsync d8/d1f/f21 0 2024-04-16T02:35:34.462 INFO:tasks.workunit.client.1.smithi153.stdout:1/347: write d4/dd/d20/f5b [4539330,61912] 0 2024-04-16T02:35:34.466 INFO:tasks.workunit.client.1.smithi153.stdout:5/316: mkdir d8/df/d28/d5a 0 2024-04-16T02:35:34.468 INFO:tasks.workunit.client.1.smithi153.stdout:5/317: creat d8/df/d2e/f5b x:0 0 0 2024-04-16T02:35:34.470 INFO:tasks.workunit.client.1.smithi153.stdout:5/318: write d8/df/d2e/f53 [93827,95124] 0 2024-04-16T02:35:34.471 INFO:tasks.workunit.client.1.smithi153.stdout:1/348: creat d4/dd/d13/d26/f78 x:0 0 0 2024-04-16T02:35:34.476 INFO:tasks.workunit.client.1.smithi153.stdout:7/301: truncate d0/d4/db/d2b/f22 4479797 0 2024-04-16T02:35:34.476 INFO:tasks.workunit.client.1.smithi153.stdout:6/292: dread dc/d12/d15/f19 [0,4194304] 0 2024-04-16T02:35:34.476 INFO:tasks.workunit.client.1.smithi153.stdout:7/302: read - d0/d4/db/d2b/f3e zero size 2024-04-16T02:35:34.477 INFO:tasks.workunit.client.1.smithi153.stdout:8/313: rename d2/l65 to d2/d1c/d1f/d27/l6a 0 2024-04-16T02:35:34.478 INFO:tasks.workunit.client.1.smithi153.stdout:6/293: truncate dc/d47/f55 1249631 0 2024-04-16T02:35:34.478 INFO:tasks.workunit.client.1.smithi153.stdout:9/361: truncate d4/d6/f22 4076408 0 2024-04-16T02:35:34.478 INFO:tasks.workunit.client.1.smithi153.stdout:6/294: chown f3 1603 1 2024-04-16T02:35:34.480 INFO:tasks.workunit.client.1.smithi153.stdout:9/362: dread d4/d6/d1b/f59 [0,4194304] 0 2024-04-16T02:35:34.480 INFO:tasks.workunit.client.1.smithi153.stdout:0/358: rename dd/d14/d51/d54/d65/f6d to dd/d32/f6e 0 2024-04-16T02:35:34.482 INFO:tasks.workunit.client.1.smithi153.stdout:7/303: creat d0/d13/f60 x:0 0 0 2024-04-16T02:35:34.485 INFO:tasks.workunit.client.1.smithi153.stdout:6/295: chown dc/d12/c46 3 1 2024-04-16T02:35:34.486 INFO:tasks.workunit.client.1.smithi153.stdout:0/359: dwrite dd/f47 [0,4194304] 0 2024-04-16T02:35:34.488 INFO:tasks.workunit.client.1.smithi153.stdout:3/310: rename d6/d10/d22/d2e/f4a to d6/d10/d22/d4d/f72 0 2024-04-16T02:35:34.489 INFO:tasks.workunit.client.1.smithi153.stdout:8/314: symlink d2/d1c/d1f/d27/d2d/d4b/l6b 0 2024-04-16T02:35:34.492 INFO:tasks.workunit.client.1.smithi153.stdout:6/296: readlink la 0 2024-04-16T02:35:34.496 INFO:tasks.workunit.client.1.smithi153.stdout:3/311: dwrite d6/f20 [0,4194304] 0 2024-04-16T02:35:34.497 INFO:tasks.workunit.client.1.smithi153.stdout:8/315: dread d2/de/d34/f40 [0,4194304] 0 2024-04-16T02:35:34.499 INFO:tasks.workunit.client.1.smithi153.stdout:6/297: chown dc/f11 628 1 2024-04-16T02:35:34.503 INFO:tasks.workunit.client.1.smithi153.stdout:6/298: write dc/d1a/f1c [124914,57007] 0 2024-04-16T02:35:34.503 INFO:tasks.workunit.client.1.smithi153.stdout:4/382: mkdir d3/d9/db/d84 0 2024-04-16T02:35:34.505 INFO:tasks.workunit.client.1.smithi153.stdout:0/360: dread f5 [0,4194304] 0 2024-04-16T02:35:34.507 INFO:tasks.workunit.client.1.smithi153.stdout:6/299: creat dc/d47/f60 x:0 0 0 2024-04-16T02:35:34.507 INFO:tasks.workunit.client.1.smithi153.stdout:4/383: read d3/d9/db/de/f13 [947059,75410] 0 2024-04-16T02:35:34.507 INFO:tasks.workunit.client.1.smithi153.stdout:0/361: creat dd/f6f x:0 0 0 2024-04-16T02:35:34.510 INFO:tasks.workunit.client.1.smithi153.stdout:4/384: mkdir d3/d9/db/de/d2c/d52/d85 0 2024-04-16T02:35:34.512 INFO:tasks.workunit.client.1.smithi153.stdout:2/338: truncate d9/d2b/f46 1448140 0 2024-04-16T02:35:34.514 INFO:tasks.workunit.client.1.smithi153.stdout:9/363: rename d4/d6/f1a to d4/d33/d3a/d40/f6d 0 2024-04-16T02:35:34.514 INFO:tasks.workunit.client.1.smithi153.stdout:2/339: read d9/d2b/f19 [114284,123376] 0 2024-04-16T02:35:34.515 INFO:tasks.workunit.client.1.smithi153.stdout:2/340: fdatasync d9/d2b/d18/d3f/f48 0 2024-04-16T02:35:34.516 INFO:tasks.workunit.client.1.smithi153.stdout:7/304: rename d0/d4/db/d47/c48 to d0/d4/db/d34/d4b/c61 0 2024-04-16T02:35:34.516 INFO:tasks.workunit.client.1.smithi153.stdout:2/341: truncate d9/d2b/f2a 936946 0 2024-04-16T02:35:34.521 INFO:tasks.workunit.client.1.smithi153.stdout:5/319: truncate d8/df/d2e/f53 63350 0 2024-04-16T02:35:34.522 INFO:tasks.workunit.client.1.smithi153.stdout:5/320: truncate d8/df/d2e/f5b 815633 0 2024-04-16T02:35:34.522 INFO:tasks.workunit.client.1.smithi153.stdout:2/342: creat d9/d2b/d5c/f80 x:0 0 0 2024-04-16T02:35:34.523 INFO:tasks.workunit.client.1.smithi153.stdout:5/321: dread - d8/d1f/d20/d39/f4c zero size 2024-04-16T02:35:34.528 INFO:tasks.workunit.client.1.smithi153.stdout:0/362: rename dd/d28/f40 to dd/d14/d19/d20/f70 0 2024-04-16T02:35:34.528 INFO:tasks.workunit.client.1.smithi153.stdout:0/363: write dd/f53 [623049,58314] 0 2024-04-16T02:35:34.529 INFO:tasks.workunit.client.1.smithi153.stdout:2/343: symlink d9/d2b/l81 0 2024-04-16T02:35:34.530 INFO:tasks.workunit.client.1.smithi153.stdout:7/305: mknod d0/d4/db/d34/d4b/d4d/c62 0 2024-04-16T02:35:34.533 INFO:tasks.workunit.client.1.smithi153.stdout:5/322: creat d8/df/d2f/d4e/d51/f5c x:0 0 0 2024-04-16T02:35:34.535 INFO:tasks.workunit.client.1.smithi153.stdout:4/385: rename d3/d9/db/d17/d28/c37 to d3/d9/db/de/d2c/d52/c86 0 2024-04-16T02:35:34.536 INFO:tasks.workunit.client.1.smithi153.stdout:0/364: mknod dd/d14/d51/d54/d65/c71 0 2024-04-16T02:35:34.539 INFO:tasks.workunit.client.1.smithi153.stdout:0/365: dread dd/f53 [0,4194304] 0 2024-04-16T02:35:34.540 INFO:tasks.workunit.client.1.smithi153.stdout:5/323: symlink d8/df/d2f/d4e/d51/l5d 0 2024-04-16T02:35:34.540 INFO:tasks.workunit.client.1.smithi153.stdout:0/366: readlink dd/d14/d19/d1c/l56 0 2024-04-16T02:35:34.540 INFO:tasks.workunit.client.1.smithi153.stdout:5/324: readlink d8/d1f/l27 0 2024-04-16T02:35:34.542 INFO:tasks.workunit.client.1.smithi153.stdout:4/386: rename d3/d9/db/d17/c6e to d3/d9/db/de/d2c/c87 0 2024-04-16T02:35:34.544 INFO:tasks.workunit.client.1.smithi153.stdout:7/306: creat d0/d4/db/d34/d51/d5f/f63 x:0 0 0 2024-04-16T02:35:34.545 INFO:tasks.workunit.client.1.smithi153.stdout:0/367: creat dd/d14/d51/d54/d65/f72 x:0 0 0 2024-04-16T02:35:34.547 INFO:tasks.workunit.client.1.smithi153.stdout:3/312: dwrite d6/d10/d22/d2c/f33 [0,4194304] 0 2024-04-16T02:35:34.547 INFO:tasks.workunit.client.1.smithi153.stdout:5/325: fdatasync d8/df/d2f/f33 0 2024-04-16T02:35:34.547 INFO:tasks.workunit.client.1.smithi153.stdout:8/316: truncate d2/f7 3213156 0 2024-04-16T02:35:34.548 INFO:tasks.workunit.client.1.smithi153.stdout:5/326: fsync d8/f3a 0 2024-04-16T02:35:34.548 INFO:tasks.workunit.client.0.smithi151.stdout:1/4: sync 2024-04-16T02:35:34.548 INFO:tasks.workunit.client.0.smithi151.stdout:3/2: sync 2024-04-16T02:35:34.548 INFO:tasks.workunit.client.0.smithi151.stdout:3/3: chown . 2441 1 2024-04-16T02:35:34.549 INFO:tasks.workunit.client.0.smithi151.stdout:3/4: write - no filename 2024-04-16T02:35:34.549 INFO:tasks.workunit.client.0.smithi151.stdout:3/5: read - no filename 2024-04-16T02:35:34.549 INFO:tasks.workunit.client.0.smithi151.stdout:3/6: rmdir - no directory 2024-04-16T02:35:34.549 INFO:tasks.workunit.client.0.smithi151.stdout:3/7: dread - no filename 2024-04-16T02:35:34.550 INFO:tasks.workunit.client.1.smithi153.stdout:4/387: symlink d3/d9/db/d17/d3f/d6b/l88 0 2024-04-16T02:35:34.551 INFO:tasks.workunit.client.0.smithi151.stdout:1/5: rename f0 to f1 0 2024-04-16T02:35:34.551 INFO:tasks.workunit.client.0.smithi151.stdout:3/8: symlink l0 0 2024-04-16T02:35:34.552 INFO:tasks.workunit.client.1.smithi153.stdout:6/300: write dc/d12/d15/f19 [2906649,42432] 0 2024-04-16T02:35:34.552 INFO:tasks.workunit.client.0.smithi151.stdout:3/9: chown l0 87 1 2024-04-16T02:35:34.553 INFO:tasks.workunit.client.0.smithi151.stdout:1/6: creat f2 x:0 0 0 2024-04-16T02:35:34.554 INFO:tasks.workunit.client.0.smithi151.stdout:3/10: mkdir d1 0 2024-04-16T02:35:34.555 INFO:tasks.workunit.client.1.smithi153.stdout:0/368: unlink dd/d32/f6e 0 2024-04-16T02:35:34.555 INFO:tasks.workunit.client.0.smithi151.stdout:1/7: mkdir d3 0 2024-04-16T02:35:34.556 INFO:tasks.workunit.client.0.smithi151.stdout:1/8: read - f2 zero size 2024-04-16T02:35:34.557 INFO:tasks.workunit.client.0.smithi151.stdout:3/11: getdents d1 0 2024-04-16T02:35:34.557 INFO:tasks.workunit.client.0.smithi151.stdout:1/9: chown f1 6750 1 2024-04-16T02:35:34.557 INFO:tasks.workunit.client.1.smithi153.stdout:4/388: dwrite d3/d9/db/de/f3b [0,4194304] 0 2024-04-16T02:35:34.558 INFO:tasks.workunit.client.1.smithi153.stdout:4/389: fsync d3/d9/f6c 0 2024-04-16T02:35:34.559 INFO:tasks.workunit.client.1.smithi153.stdout:4/390: chown d3/d9/db/de/d2c/d38 2434 1 2024-04-16T02:35:34.559 INFO:tasks.workunit.client.0.smithi151.stdout:1/10: unlink f2 0 2024-04-16T02:35:34.560 INFO:tasks.workunit.client.1.smithi153.stdout:4/391: write d3/d9/f77 [424670,62653] 0 2024-04-16T02:35:34.562 INFO:tasks.workunit.client.0.smithi151.stdout:3/12: rename l0 to d1/l2 0 2024-04-16T02:35:34.563 INFO:tasks.workunit.client.0.smithi151.stdout:1/11: fdatasync f1 0 2024-04-16T02:35:34.567 INFO:tasks.workunit.client.1.smithi153.stdout:6/301: fdatasync dc/d12/f24 0 2024-04-16T02:35:34.567 INFO:tasks.workunit.client.0.smithi151.stdout:1/12: mkdir d3/d4 0 2024-04-16T02:35:34.568 INFO:tasks.workunit.client.0.smithi151.stdout:1/13: read - f1 zero size 2024-04-16T02:35:34.569 INFO:tasks.workunit.client.1.smithi153.stdout:9/364: dwrite d4/d6/dc/d2d/f58 [0,4194304] 0 2024-04-16T02:35:34.570 INFO:tasks.workunit.client.0.smithi151.stdout:1/14: write f1 [131295,98181] 0 2024-04-16T02:35:34.570 INFO:tasks.workunit.client.1.smithi153.stdout:4/392: symlink d3/d9/db/de/d2c/d38/l89 0 2024-04-16T02:35:34.572 INFO:tasks.workunit.client.1.smithi153.stdout:1/349: sync 2024-04-16T02:35:34.573 INFO:tasks.workunit.client.1.smithi153.stdout:7/307: creat d0/d4/db/f64 x:0 0 0 2024-04-16T02:35:34.576 INFO:tasks.workunit.client.1.smithi153.stdout:7/308: write d0/d4/db/f64 [615409,68433] 0 2024-04-16T02:35:34.576 INFO:tasks.workunit.client.0.smithi151.stdout:9/30: sync 2024-04-16T02:35:34.576 INFO:tasks.workunit.client.0.smithi151.stdout:7/4: sync 2024-04-16T02:35:34.576 INFO:tasks.workunit.client.0.smithi151.stdout:5/50: sync 2024-04-16T02:35:34.577 INFO:tasks.workunit.client.0.smithi151.stdout:4/57: sync 2024-04-16T02:35:34.577 INFO:tasks.workunit.client.0.smithi151.stdout:8/46: sync 2024-04-16T02:35:34.577 INFO:tasks.workunit.client.0.smithi151.stdout:6/2: sync 2024-04-16T02:35:34.577 INFO:tasks.workunit.client.0.smithi151.stdout:6/3: truncate - no filename 2024-04-16T02:35:34.577 INFO:tasks.workunit.client.0.smithi151.stdout:6/4: dwrite - no filename 2024-04-16T02:35:34.577 INFO:tasks.workunit.client.0.smithi151.stdout:6/5: read - no filename 2024-04-16T02:35:34.577 INFO:tasks.workunit.client.0.smithi151.stdout:6/6: readlink - no filename 2024-04-16T02:35:34.577 INFO:tasks.workunit.client.0.smithi151.stdout:6/7: stat - no entries 2024-04-16T02:35:34.577 INFO:tasks.workunit.client.0.smithi151.stdout:2/46: sync 2024-04-16T02:35:34.578 INFO:tasks.workunit.client.1.smithi153.stdout:7/309: write d0/d4/f58 [293652,101849] 0 2024-04-16T02:35:34.578 INFO:tasks.workunit.client.1.smithi153.stdout:8/317: dread d2/d1c/f36 [0,4194304] 0 2024-04-16T02:35:34.578 INFO:tasks.workunit.client.1.smithi153.stdout:8/318: truncate d2/de/d34/f68 269736 0 2024-04-16T02:35:34.579 INFO:tasks.workunit.client.0.smithi151.stdout:0/150: sync 2024-04-16T02:35:34.579 INFO:tasks.workunit.client.1.smithi153.stdout:8/319: fdatasync d2/d13/d3a/f4c 0 2024-04-16T02:35:34.580 INFO:tasks.workunit.client.0.smithi151.stdout:1/15: dwrite f1 [0,4194304] 0 2024-04-16T02:35:34.582 INFO:tasks.workunit.client.0.smithi151.stdout:4/58: rename d0/lf to d0/l13 0 2024-04-16T02:35:34.582 INFO:tasks.workunit.client.0.smithi151.stdout:1/16: chown d3 16505 1 2024-04-16T02:35:34.582 INFO:tasks.workunit.client.1.smithi153.stdout:5/327: dread d8/df/f10 [0,4194304] 0 2024-04-16T02:35:34.583 INFO:tasks.workunit.client.1.smithi153.stdout:5/328: chown d8/df/d23/c2b 0 1 2024-04-16T02:35:34.584 INFO:tasks.workunit.client.1.smithi153.stdout:5/329: stat d8/f3a 0 2024-04-16T02:35:34.586 INFO:tasks.workunit.client.1.smithi153.stdout:8/320: dwrite d2/d13/d2c/f32 [0,4194304] 0 2024-04-16T02:35:34.586 INFO:tasks.workunit.client.0.smithi151.stdout:5/51: write d2/d6/f8 [75210,65045] 0 2024-04-16T02:35:34.587 INFO:tasks.workunit.client.0.smithi151.stdout:4/59: chown d0/l1 13 1 2024-04-16T02:35:34.588 INFO:tasks.workunit.client.0.smithi151.stdout:1/17: chown d3/d4 4712864 1 2024-04-16T02:35:34.588 INFO:tasks.workunit.client.0.smithi151.stdout:5/52: chown d2/ff 7688946 1 2024-04-16T02:35:34.590 INFO:tasks.workunit.client.0.smithi151.stdout:0/151: rename c8 to dc/d20/d2c/c33 0 2024-04-16T02:35:34.590 INFO:tasks.workunit.client.0.smithi151.stdout:5/53: dread d2/f5 [0,4194304] 0 2024-04-16T02:35:34.590 INFO:tasks.workunit.client.0.smithi151.stdout:9/31: dread d1/f5 [0,4194304] 0 2024-04-16T02:35:34.592 INFO:tasks.workunit.client.0.smithi151.stdout:8/47: creat ff x:0 0 0 2024-04-16T02:35:34.594 INFO:tasks.workunit.client.0.smithi151.stdout:6/8: creat f0 x:0 0 0 2024-04-16T02:35:34.595 INFO:tasks.workunit.client.1.smithi153.stdout:1/350: unlink d4/f6f 0 2024-04-16T02:35:34.596 INFO:tasks.workunit.client.0.smithi151.stdout:6/9: chown f0 40806135 1 2024-04-16T02:35:34.597 INFO:tasks.workunit.client.0.smithi151.stdout:2/47: mknod de/c10 0 2024-04-16T02:35:34.598 INFO:tasks.workunit.client.1.smithi153.stdout:7/310: mknod d0/d4/db/d47/c65 0 2024-04-16T02:35:34.599 INFO:tasks.workunit.client.1.smithi153.stdout:3/313: getdents d6/d10/d3b 0 2024-04-16T02:35:34.600 INFO:tasks.workunit.client.0.smithi151.stdout:1/18: mkdir d3/d4/d5 0 2024-04-16T02:35:34.602 INFO:tasks.workunit.client.0.smithi151.stdout:4/60: symlink d0/d11/l14 0 2024-04-16T02:35:34.604 INFO:tasks.workunit.client.1.smithi153.stdout:2/344: dwrite d9/d2b/d5c/d5f/f69 [0,4194304] 0 2024-04-16T02:35:34.605 INFO:tasks.workunit.client.1.smithi153.stdout:9/365: mkdir d4/d6/d1b/d6e 0 2024-04-16T02:35:34.606 INFO:tasks.workunit.client.0.smithi151.stdout:5/54: dread d2/fd [0,4194304] 0 2024-04-16T02:35:34.607 INFO:tasks.workunit.client.0.smithi151.stdout:0/152: dwrite dc/d15/f23 [0,4194304] 0 2024-04-16T02:35:34.612 INFO:tasks.workunit.client.1.smithi153.stdout:5/330: getdents d8/df/d28/d5a 0 2024-04-16T02:35:34.613 INFO:tasks.workunit.client.0.smithi151.stdout:4/61: creat d0/f15 x:0 0 0 2024-04-16T02:35:34.613 INFO:tasks.workunit.client.0.smithi151.stdout:5/55: dread d2/f5 [0,4194304] 0 2024-04-16T02:35:34.617 INFO:tasks.workunit.client.1.smithi153.stdout:7/311: rename d0/d13/f60 to d0/d4/db/d34/d4b/d4d/f66 0 2024-04-16T02:35:34.618 INFO:tasks.workunit.client.1.smithi153.stdout:7/312: chown d0/d4/db/d2b/f40 14 1 2024-04-16T02:35:34.618 INFO:tasks.workunit.client.1.smithi153.stdout:6/302: link dc/ff dc/d47/f61 0 2024-04-16T02:35:34.620 INFO:tasks.workunit.client.0.smithi151.stdout:2/48: dread f7 [0,4194304] 0 2024-04-16T02:35:34.623 INFO:tasks.workunit.client.0.smithi151.stdout:9/32: dwrite d1/f5 [4194304,4194304] 0 2024-04-16T02:35:34.625 INFO:tasks.workunit.client.0.smithi151.stdout:1/19: symlink d3/d4/d5/l6 0 2024-04-16T02:35:34.625 INFO:tasks.workunit.client.0.smithi151.stdout:6/10: dwrite f0 [0,4194304] 0 2024-04-16T02:35:34.626 INFO:tasks.workunit.client.0.smithi151.stdout:2/49: mkdir de/d11 0 2024-04-16T02:35:34.626 INFO:tasks.workunit.client.0.smithi151.stdout:1/20: readlink d3/d4/d5/l6 0 2024-04-16T02:35:34.627 INFO:tasks.workunit.client.1.smithi153.stdout:0/369: truncate dd/f10 431463 0 2024-04-16T02:35:34.627 INFO:tasks.workunit.client.0.smithi151.stdout:5/56: symlink d2/l10 0 2024-04-16T02:35:34.628 INFO:tasks.workunit.client.1.smithi153.stdout:0/370: write dd/f6f [345676,80388] 0 2024-04-16T02:35:34.634 INFO:tasks.workunit.client.0.smithi151.stdout:9/33: creat d1/d3/fa x:0 0 0 2024-04-16T02:35:34.634 INFO:tasks.workunit.client.0.smithi151.stdout:6/11: rename f0 to f1 0 2024-04-16T02:35:34.635 INFO:tasks.workunit.client.1.smithi153.stdout:6/303: mkdir dc/d10/d62 0 2024-04-16T02:35:34.636 INFO:tasks.workunit.client.0.smithi151.stdout:1/21: mknod d3/c7 0 2024-04-16T02:35:34.638 INFO:tasks.workunit.client.0.smithi151.stdout:2/50: rename cd to de/d11/c12 0 2024-04-16T02:35:34.639 INFO:tasks.workunit.client.1.smithi153.stdout:4/393: getdents d3/d9/db 0 2024-04-16T02:35:34.639 INFO:tasks.workunit.client.0.smithi151.stdout:2/51: fsync de/ff 0 2024-04-16T02:35:34.640 INFO:tasks.workunit.client.0.smithi151.stdout:9/34: dread d1/f5 [0,4194304] 0 2024-04-16T02:35:34.640 INFO:tasks.workunit.client.1.smithi153.stdout:5/331: mkdir d8/df/d28/d5a/d5e 0 2024-04-16T02:35:34.642 INFO:tasks.workunit.client.0.smithi151.stdout:9/35: write d1/f5 [7066884,34259] 0 2024-04-16T02:35:34.642 INFO:tasks.workunit.client.1.smithi153.stdout:5/332: write d8/df/d28/f35 [1315803,3712] 0 2024-04-16T02:35:34.644 INFO:tasks.workunit.client.0.smithi151.stdout:2/52: dread de/ff [0,4194304] 0 2024-04-16T02:35:34.645 INFO:tasks.workunit.client.1.smithi153.stdout:0/371: unlink dd/d14/d19/l24 0 2024-04-16T02:35:34.645 INFO:tasks.workunit.client.0.smithi151.stdout:2/53: truncate f8 4673095 0 2024-04-16T02:35:34.645 INFO:tasks.workunit.client.1.smithi153.stdout:0/372: chown dd/d28/l64 0 1 2024-04-16T02:35:34.646 INFO:tasks.workunit.client.0.smithi151.stdout:1/22: creat d3/f8 x:0 0 0 2024-04-16T02:35:34.646 INFO:tasks.workunit.client.0.smithi151.stdout:1/23: dread - d3/f8 zero size 2024-04-16T02:35:34.647 INFO:tasks.workunit.client.1.smithi153.stdout:0/373: read - dd/d14/d19/d1c/f48 zero size 2024-04-16T02:35:34.647 INFO:tasks.workunit.client.0.smithi151.stdout:1/24: stat f1 0 2024-04-16T02:35:34.648 INFO:tasks.workunit.client.1.smithi153.stdout:4/394: creat d3/d9/db/de/d2c/f8a x:0 0 0 2024-04-16T02:35:34.649 INFO:tasks.workunit.client.0.smithi151.stdout:5/57: rename d2/ff to d2/d6/f11 0 2024-04-16T02:35:34.650 INFO:tasks.workunit.client.1.smithi153.stdout:3/314: rename d6/d10/d22/d2c/l40 to d6/d10/l73 0 2024-04-16T02:35:34.651 INFO:tasks.workunit.client.1.smithi153.stdout:2/345: dread d9/d24/d27/d2c/d41/f4c [0,4194304] 0 2024-04-16T02:35:34.652 INFO:tasks.workunit.client.1.smithi153.stdout:3/315: stat d6/d10/d22/d2e/f35 0 2024-04-16T02:35:34.654 INFO:tasks.workunit.client.0.smithi151.stdout:9/36: creat d1/d3/fb x:0 0 0 2024-04-16T02:35:34.654 INFO:tasks.workunit.client.1.smithi153.stdout:1/351: dwrite d4/f1f [0,4194304] 0 2024-04-16T02:35:34.655 INFO:tasks.workunit.client.1.smithi153.stdout:2/346: fdatasync d9/d24/d27/d2c/d41/f44 0 2024-04-16T02:35:34.655 INFO:tasks.workunit.client.1.smithi153.stdout:5/333: symlink d8/df/d2f/d4e/d51/l5f 0 2024-04-16T02:35:34.655 INFO:tasks.workunit.client.1.smithi153.stdout:9/366: truncate d4/f13 2904695 0 2024-04-16T02:35:34.656 INFO:tasks.workunit.client.0.smithi151.stdout:9/37: readlink d1/l4 0 2024-04-16T02:35:34.657 INFO:tasks.workunit.client.1.smithi153.stdout:5/334: stat d8/d1f/d20/d39/c58 0 2024-04-16T02:35:34.657 INFO:tasks.workunit.client.1.smithi153.stdout:1/352: write d4/dd/d18/f51 [2018507,30199] 0 2024-04-16T02:35:34.658 INFO:tasks.workunit.client.0.smithi151.stdout:2/54: dwrite f8 [0,4194304] 0 2024-04-16T02:35:34.659 INFO:tasks.workunit.client.1.smithi153.stdout:7/313: dread d0/d4/db/d2b/f25 [0,4194304] 0 2024-04-16T02:35:34.660 INFO:tasks.workunit.client.1.smithi153.stdout:6/304: truncate dc/d10/f1b 6878051 0 2024-04-16T02:35:34.662 INFO:tasks.workunit.client.1.smithi153.stdout:6/305: truncate dc/d1a/f26 1002767 0 2024-04-16T02:35:34.664 INFO:tasks.workunit.client.0.smithi151.stdout:1/25: dwrite f1 [0,4194304] 0 2024-04-16T02:35:34.665 INFO:tasks.workunit.client.1.smithi153.stdout:4/395: creat d3/d9/db/de/d2c/d52/f8b x:0 0 0 2024-04-16T02:35:34.668 INFO:tasks.workunit.client.1.smithi153.stdout:9/367: symlink d4/d33/d3a/d40/l6f 0 2024-04-16T02:35:34.669 INFO:tasks.workunit.client.0.smithi151.stdout:9/38: dread d1/f5 [0,4194304] 0 2024-04-16T02:35:34.671 INFO:tasks.workunit.client.1.smithi153.stdout:2/347: mknod d9/d24/d34/c82 0 2024-04-16T02:35:34.673 INFO:tasks.workunit.client.1.smithi153.stdout:5/335: creat d8/df/d2f/f60 x:0 0 0 2024-04-16T02:35:34.673 INFO:tasks.workunit.client.1.smithi153.stdout:0/374: mkdir dd/d14/d19/d1c/d73 0 2024-04-16T02:35:34.674 INFO:tasks.workunit.client.0.smithi151.stdout:5/58: rename d2/f7 to d2/f12 0 2024-04-16T02:35:34.674 INFO:tasks.workunit.client.0.smithi151.stdout:2/55: dread f8 [0,4194304] 0 2024-04-16T02:35:34.674 INFO:tasks.workunit.client.0.smithi151.stdout:9/39: truncate d1/d3/fa 48823 0 2024-04-16T02:35:34.675 INFO:tasks.workunit.client.1.smithi153.stdout:6/306: rmdir dc/d12/d15 39 2024-04-16T02:35:34.677 INFO:tasks.workunit.client.0.smithi151.stdout:5/59: write d2/f5 [249361,57360] 0 2024-04-16T02:35:34.677 INFO:tasks.workunit.client.1.smithi153.stdout:9/368: dwrite d4/d6/f56 [4194304,4194304] 0 2024-04-16T02:35:34.678 INFO:tasks.workunit.client.0.smithi151.stdout:9/40: write d1/f5 [6040092,87077] 0 2024-04-16T02:35:34.678 INFO:tasks.workunit.client.0.smithi151.stdout:1/26: dread f1 [0,4194304] 0 2024-04-16T02:35:34.679 INFO:tasks.workunit.client.1.smithi153.stdout:4/396: creat d3/d9/db/d17/f8c x:0 0 0 2024-04-16T02:35:34.683 INFO:tasks.workunit.client.1.smithi153.stdout:3/316: rename f1 to d6/d10/d22/d2c/f74 0 2024-04-16T02:35:34.689 INFO:tasks.workunit.client.0.smithi151.stdout:9/41: unlink d1/c9 0 2024-04-16T02:35:34.690 INFO:tasks.workunit.client.1.smithi153.stdout:1/353: mknod d4/dd/d13/c79 0 2024-04-16T02:35:34.691 INFO:tasks.workunit.client.0.smithi151.stdout:2/56: dwrite f8 [0,4194304] 0 2024-04-16T02:35:34.691 INFO:tasks.workunit.client.0.smithi151.stdout:9/42: write d1/d3/fa [544693,94727] 0 2024-04-16T02:35:34.691 INFO:tasks.workunit.client.0.smithi151.stdout:1/27: link d3/c7 d3/d4/d5/c9 0 2024-04-16T02:35:34.692 INFO:tasks.workunit.client.1.smithi153.stdout:7/314: link d0/d4/db/d34/d4b/d4d/f66 d0/d4/db/d2b/d53/d59/f67 0 2024-04-16T02:35:34.695 INFO:tasks.workunit.client.0.smithi151.stdout:2/57: mknod de/c13 0 2024-04-16T02:35:34.696 INFO:tasks.workunit.client.1.smithi153.stdout:5/336: unlink d8/df/d2e/f53 0 2024-04-16T02:35:34.698 INFO:tasks.workunit.client.0.smithi151.stdout:2/58: rmdir de/d11 39 2024-04-16T02:35:34.698 INFO:tasks.workunit.client.0.smithi151.stdout:2/59: chown lc 7 1 2024-04-16T02:35:34.699 INFO:tasks.workunit.client.1.smithi153.stdout:0/375: rmdir dd/d32 39 2024-04-16T02:35:34.700 INFO:tasks.workunit.client.1.smithi153.stdout:0/376: stat ca 0 2024-04-16T02:35:34.701 INFO:tasks.workunit.client.0.smithi151.stdout:9/43: link d1/d3/fa d1/fc 0 2024-04-16T02:35:34.701 INFO:tasks.workunit.client.1.smithi153.stdout:6/307: mkdir dc/d59/d63 0 2024-04-16T02:35:34.701 INFO:tasks.workunit.client.1.smithi153.stdout:1/354: dwrite d4/dd/d13/f14 [0,4194304] 0 2024-04-16T02:35:34.704 INFO:tasks.workunit.client.1.smithi153.stdout:0/377: write dd/d28/f3c [1983134,43132] 0 2024-04-16T02:35:34.705 INFO:tasks.workunit.client.1.smithi153.stdout:4/397: unlink d3/l16 0 2024-04-16T02:35:34.705 INFO:tasks.workunit.client.1.smithi153.stdout:8/321: sync 2024-04-16T02:35:34.706 INFO:tasks.workunit.client.1.smithi153.stdout:9/369: read - d4/d6/dc/d46/f55 zero size 2024-04-16T02:35:34.709 INFO:tasks.workunit.client.1.smithi153.stdout:1/355: write d4/dd/d18/f52 [1185675,110939] 0 2024-04-16T02:35:34.710 INFO:tasks.workunit.client.1.smithi153.stdout:7/315: dwrite d0/d13/f32 [0,4194304] 0 2024-04-16T02:35:34.716 INFO:tasks.workunit.client.1.smithi153.stdout:1/356: fsync d4/dd/d20/f5b 0 2024-04-16T02:35:34.717 INFO:tasks.workunit.client.1.smithi153.stdout:7/316: dread - d0/d4/db/f50 zero size 2024-04-16T02:35:34.717 INFO:tasks.workunit.client.1.smithi153.stdout:3/317: creat d6/d10/d5b/f75 x:0 0 0 2024-04-16T02:35:34.723 INFO:tasks.workunit.client.1.smithi153.stdout:0/378: dwrite dd/d26/f5b [0,4194304] 0 2024-04-16T02:35:34.723 INFO:tasks.workunit.client.1.smithi153.stdout:6/308: mknod dc/d10/d62/c64 0 2024-04-16T02:35:34.724 INFO:tasks.workunit.client.1.smithi153.stdout:1/357: mkdir d4/dd/d13/d26/d58/d7a 0 2024-04-16T02:35:34.726 INFO:tasks.workunit.client.1.smithi153.stdout:3/318: mkdir d6/d10/d3b/d76 0 2024-04-16T02:35:34.729 INFO:tasks.workunit.client.1.smithi153.stdout:9/370: creat d4/d6/d1b/d2a/f70 x:0 0 0 2024-04-16T02:35:34.729 INFO:tasks.workunit.client.1.smithi153.stdout:2/348: dread d9/d24/d27/d2c/d41/f44 [0,4194304] 0 2024-04-16T02:35:34.730 INFO:tasks.workunit.client.1.smithi153.stdout:7/317: fdatasync d0/d4/f20 0 2024-04-16T02:35:34.731 INFO:tasks.workunit.client.1.smithi153.stdout:8/322: dwrite d2/d1c/f36 [0,4194304] 0 2024-04-16T02:35:34.731 INFO:tasks.workunit.client.1.smithi153.stdout:6/309: fdatasync dc/f17 0 2024-04-16T02:35:34.733 INFO:tasks.workunit.client.1.smithi153.stdout:5/337: getdents d8/df 0 2024-04-16T02:35:34.734 INFO:tasks.workunit.client.1.smithi153.stdout:3/319: stat d6/d10/d3b/l48 0 2024-04-16T02:35:34.735 INFO:tasks.workunit.client.1.smithi153.stdout:5/338: chown d8/df/f11 0 1 2024-04-16T02:35:34.736 INFO:tasks.workunit.client.1.smithi153.stdout:4/398: getdents d3/d9/db/de/d2c/d38 0 2024-04-16T02:35:34.736 INFO:tasks.workunit.client.1.smithi153.stdout:3/320: stat d6/d10/c68 0 2024-04-16T02:35:34.739 INFO:tasks.workunit.client.1.smithi153.stdout:1/358: creat d4/dd/d18/d29/d71/f7b x:0 0 0 2024-04-16T02:35:34.740 INFO:tasks.workunit.client.1.smithi153.stdout:3/321: chown d6/d10/f13 27 1 2024-04-16T02:35:34.741 INFO:tasks.workunit.client.1.smithi153.stdout:2/349: mknod d9/d71/c83 0 2024-04-16T02:35:34.741 INFO:tasks.workunit.client.1.smithi153.stdout:5/339: mknod d8/d1f/d20/c61 0 2024-04-16T02:35:34.741 INFO:tasks.workunit.client.1.smithi153.stdout:4/399: write d3/d9/d24/f2b [2823826,13855] 0 2024-04-16T02:35:34.742 INFO:tasks.workunit.client.1.smithi153.stdout:1/359: fdatasync d4/dd/d18/d29/f50 0 2024-04-16T02:35:34.744 INFO:tasks.workunit.client.1.smithi153.stdout:7/318: truncate d0/d4/db/d2b/f22 3601796 0 2024-04-16T02:35:34.746 INFO:tasks.workunit.client.1.smithi153.stdout:9/371: creat d4/d6/d1b/f71 x:0 0 0 2024-04-16T02:35:34.748 INFO:tasks.workunit.client.1.smithi153.stdout:3/322: read d6/f11 [2822854,54864] 0 2024-04-16T02:35:34.749 INFO:tasks.workunit.client.1.smithi153.stdout:5/340: mkdir d8/df/d23/d62 0 2024-04-16T02:35:34.751 INFO:tasks.workunit.client.1.smithi153.stdout:5/341: write d8/df/d2e/f3d [1053778,54849] 0 2024-04-16T02:35:34.754 INFO:tasks.workunit.client.1.smithi153.stdout:7/319: mkdir d0/d4/db/d34/d51/d5f/d68 0 2024-04-16T02:35:34.755 INFO:tasks.workunit.client.1.smithi153.stdout:9/372: fsync d4/d33/d3a/d40/d4a/f68 0 2024-04-16T02:35:34.756 INFO:tasks.workunit.client.1.smithi153.stdout:7/320: chown d0/d4/db/d34 30339 1 2024-04-16T02:35:34.756 INFO:tasks.workunit.client.1.smithi153.stdout:3/323: dwrite d6/d10/d22/d2c/f63 [0,4194304] 0 2024-04-16T02:35:34.757 INFO:tasks.workunit.client.1.smithi153.stdout:6/310: link dc/d12/c46 dc/d10/d62/c65 0 2024-04-16T02:35:34.759 INFO:tasks.workunit.client.1.smithi153.stdout:7/321: write d0/fa [273246,28650] 0 2024-04-16T02:35:34.759 INFO:tasks.workunit.client.1.smithi153.stdout:3/324: stat d6/d10/d22/d2c/f3d 0 2024-04-16T02:35:34.760 INFO:tasks.workunit.client.1.smithi153.stdout:2/350: symlink d9/d2b/d18/l84 0 2024-04-16T02:35:34.761 INFO:tasks.workunit.client.1.smithi153.stdout:5/342: dread d8/d1f/d20/f40 [0,4194304] 0 2024-04-16T02:35:34.761 INFO:tasks.workunit.client.1.smithi153.stdout:9/373: creat d4/d33/d3a/d40/d4a/f72 x:0 0 0 2024-04-16T02:35:34.762 INFO:tasks.workunit.client.1.smithi153.stdout:3/325: chown d6/d10/d3b 1869910087 1 2024-04-16T02:35:34.765 INFO:tasks.workunit.client.1.smithi153.stdout:8/323: dwrite d2/f12 [0,4194304] 0 2024-04-16T02:35:34.770 INFO:tasks.workunit.client.1.smithi153.stdout:2/351: chown d9/d24/d27/l1e 77 1 2024-04-16T02:35:34.772 INFO:tasks.workunit.client.1.smithi153.stdout:7/322: dread d0/d4/db/f64 [0,4194304] 0 2024-04-16T02:35:34.773 INFO:tasks.workunit.client.1.smithi153.stdout:5/343: creat d8/df/d28/f63 x:0 0 0 2024-04-16T02:35:34.773 INFO:tasks.workunit.client.1.smithi153.stdout:7/323: truncate d0/d4/db/f57 232627 0 2024-04-16T02:35:34.774 INFO:tasks.workunit.client.1.smithi153.stdout:3/326: chown d6/d10/c30 3224048 1 2024-04-16T02:35:34.774 INFO:tasks.workunit.client.1.smithi153.stdout:3/327: chown d6/d10/l1f 3177 1 2024-04-16T02:35:34.776 INFO:tasks.workunit.client.1.smithi153.stdout:9/374: mknod d4/d6/d1b/d2a/c73 0 2024-04-16T02:35:34.778 INFO:tasks.workunit.client.1.smithi153.stdout:8/324: rmdir d2/d13/d2c 39 2024-04-16T02:35:34.778 INFO:tasks.workunit.client.1.smithi153.stdout:3/328: dread d6/d10/f32 [0,4194304] 0 2024-04-16T02:35:34.779 INFO:tasks.workunit.client.1.smithi153.stdout:6/311: creat dc/d59/d63/f66 x:0 0 0 2024-04-16T02:35:34.783 INFO:tasks.workunit.client.1.smithi153.stdout:5/344: mkdir d8/df/d28/d64 0 2024-04-16T02:35:34.784 INFO:tasks.workunit.client.1.smithi153.stdout:1/360: link d4/dd/d13/d26/d58/c6b d4/dd/d13/d26/c7c 0 2024-04-16T02:35:34.787 INFO:tasks.workunit.client.1.smithi153.stdout:6/312: symlink dc/d47/d57/l67 0 2024-04-16T02:35:34.788 INFO:tasks.workunit.client.1.smithi153.stdout:7/324: dwrite d0/d13/f4f [0,4194304] 0 2024-04-16T02:35:34.789 INFO:tasks.workunit.client.1.smithi153.stdout:3/329: creat d6/d10/d22/d43/f77 x:0 0 0 2024-04-16T02:35:34.791 INFO:tasks.workunit.client.1.smithi153.stdout:9/375: dread d4/d6/f22 [0,4194304] 0 2024-04-16T02:35:34.791 INFO:tasks.workunit.client.1.smithi153.stdout:2/352: symlink d9/d24/d27/l85 0 2024-04-16T02:35:34.793 INFO:tasks.workunit.client.1.smithi153.stdout:7/325: read d0/d4/db/d2b/f25 [1075360,36446] 0 2024-04-16T02:35:34.793 INFO:tasks.workunit.client.1.smithi153.stdout:1/361: rmdir d4/dd/d13/d26 39 2024-04-16T02:35:34.793 INFO:tasks.workunit.client.1.smithi153.stdout:5/345: chown d8/df/d2e/c54 5764629 1 2024-04-16T02:35:34.795 INFO:tasks.workunit.client.1.smithi153.stdout:5/346: dread - d8/f59 zero size 2024-04-16T02:35:34.796 INFO:tasks.workunit.client.1.smithi153.stdout:6/313: creat dc/d1a/f68 x:0 0 0 2024-04-16T02:35:34.797 INFO:tasks.workunit.client.1.smithi153.stdout:3/330: mknod d6/d10/d5b/c78 0 2024-04-16T02:35:34.797 INFO:tasks.workunit.client.1.smithi153.stdout:8/325: dwrite d2/d13/d2c/f32 [0,4194304] 0 2024-04-16T02:35:34.798 INFO:tasks.workunit.client.1.smithi153.stdout:8/326: stat d2/d13/f1b 0 2024-04-16T02:35:34.800 INFO:tasks.workunit.client.1.smithi153.stdout:9/376: rename d4/d6/dc/d24/f29 to d4/d6/d1b/d6e/f74 0 2024-04-16T02:35:34.801 INFO:tasks.workunit.client.1.smithi153.stdout:9/377: fsync d4/d33/d3a/f5f 0 2024-04-16T02:35:34.802 INFO:tasks.workunit.client.1.smithi153.stdout:7/326: creat d0/d4/db/d34/d4b/f69 x:0 0 0 2024-04-16T02:35:34.803 INFO:tasks.workunit.client.1.smithi153.stdout:3/331: read d6/f1a [3088664,70667] 0 2024-04-16T02:35:34.804 INFO:tasks.workunit.client.1.smithi153.stdout:5/347: symlink d8/d1f/d20/l65 0 2024-04-16T02:35:34.864 INFO:tasks.workunit.client.1.smithi153.stdout:8/327: creat d2/d1c/d1f/d27/d2d/d47/f6c x:0 0 0 2024-04-16T02:35:34.864 INFO:tasks.workunit.client.1.smithi153.stdout:9/378: mkdir d4/d6/dc/d2d/d75 0 2024-04-16T02:35:34.865 INFO:tasks.workunit.client.1.smithi153.stdout:7/327: unlink d0/d4/db/c42 0 2024-04-16T02:35:34.866 INFO:tasks.workunit.client.1.smithi153.stdout:1/362: write d4/dd/d13/d26/f34 [510991,15865] 0 2024-04-16T02:35:34.867 INFO:tasks.workunit.client.1.smithi153.stdout:2/353: creat d9/d2b/f86 x:0 0 0 2024-04-16T02:35:34.868 INFO:tasks.workunit.client.1.smithi153.stdout:5/348: mkdir d8/df/d28/d5a/d66 0 2024-04-16T02:35:34.868 INFO:tasks.workunit.client.1.smithi153.stdout:1/363: chown d4/dd/d13/f2a 480459 1 2024-04-16T02:35:34.870 INFO:tasks.workunit.client.1.smithi153.stdout:2/354: chown d9/d71/c83 714333 1 2024-04-16T02:35:34.870 INFO:tasks.workunit.client.1.smithi153.stdout:8/328: symlink d2/d13/d3a/d5b/l6d 0 2024-04-16T02:35:34.875 INFO:tasks.workunit.client.1.smithi153.stdout:3/332: dwrite d6/d10/d22/d2c/f63 [4194304,4194304] 0 2024-04-16T02:35:34.875 INFO:tasks.workunit.client.1.smithi153.stdout:2/355: fsync d9/d2b/f16 0 2024-04-16T02:35:34.876 INFO:tasks.workunit.client.1.smithi153.stdout:3/333: readlink d6/lf 0 2024-04-16T02:35:34.878 INFO:tasks.workunit.client.1.smithi153.stdout:1/364: symlink d4/dd/d13/d23/d5f/l7d 0 2024-04-16T02:35:34.880 INFO:tasks.workunit.client.1.smithi153.stdout:1/365: write d4/dd/d13/f14 [1395944,129038] 0 2024-04-16T02:35:34.882 INFO:tasks.workunit.client.1.smithi153.stdout:1/366: write d4/dd/d13/d23/f27 [1157135,82155] 0 2024-04-16T02:35:34.883 INFO:tasks.workunit.client.1.smithi153.stdout:3/334: rmdir d6/d10/d3b 39 2024-04-16T02:35:34.884 INFO:tasks.workunit.client.1.smithi153.stdout:9/379: truncate d4/d6/dc/d2d/f58 1019204 0 2024-04-16T02:35:34.886 INFO:tasks.workunit.client.1.smithi153.stdout:8/329: truncate d2/d13/f1d 2376629 0 2024-04-16T02:35:34.888 INFO:tasks.workunit.client.1.smithi153.stdout:7/328: dwrite d0/d4/db/d2b/f22 [0,4194304] 0 2024-04-16T02:35:34.888 INFO:tasks.workunit.client.1.smithi153.stdout:2/356: rename d9/d2b/d18/l31 to d9/d24/d34/l87 0 2024-04-16T02:35:34.890 INFO:tasks.workunit.client.1.smithi153.stdout:7/329: dread - d0/d4/db/d2b/f28 zero size 2024-04-16T02:35:34.894 INFO:tasks.workunit.client.1.smithi153.stdout:1/367: mkdir d4/dd/d13/d7e 0 2024-04-16T02:35:34.894 INFO:tasks.workunit.client.1.smithi153.stdout:3/335: dread d6/d10/d22/d2c/f3d [0,4194304] 0 2024-04-16T02:35:34.894 INFO:tasks.workunit.client.1.smithi153.stdout:5/349: dread d8/f30 [0,4194304] 0 2024-04-16T02:35:34.895 INFO:tasks.workunit.client.1.smithi153.stdout:1/368: dread - d4/f5e zero size 2024-04-16T02:35:34.897 INFO:tasks.workunit.client.1.smithi153.stdout:3/336: chown d6/d10/c3f 0 1 2024-04-16T02:35:34.900 INFO:tasks.workunit.client.1.smithi153.stdout:8/330: dwrite d2/d1c/d1f/d27/d2d/f31 [0,4194304] 0 2024-04-16T02:35:34.902 INFO:tasks.workunit.client.1.smithi153.stdout:2/357: mkdir d9/d2b/d1c/d20/d88 0 2024-04-16T02:35:34.904 INFO:tasks.workunit.client.1.smithi153.stdout:0/379: sync 2024-04-16T02:35:34.907 INFO:tasks.workunit.client.1.smithi153.stdout:5/350: creat d8/df/d28/d5a/f67 x:0 0 0 2024-04-16T02:35:34.907 INFO:tasks.workunit.client.1.smithi153.stdout:5/351: chown d8/f4d 1 1 2024-04-16T02:35:34.908 INFO:tasks.workunit.client.1.smithi153.stdout:9/380: creat d4/d6/d1b/f76 x:0 0 0 2024-04-16T02:35:34.909 INFO:tasks.workunit.client.1.smithi153.stdout:5/352: chown d8/df/d2f/d4e/d51/c52 6703623 1 2024-04-16T02:35:34.910 INFO:tasks.workunit.client.1.smithi153.stdout:3/337: dread - d6/d10/d3b/f6d zero size 2024-04-16T02:35:34.911 INFO:tasks.workunit.client.1.smithi153.stdout:1/369: rmdir d4 39 2024-04-16T02:35:34.912 INFO:tasks.workunit.client.1.smithi153.stdout:7/330: symlink d0/d4/db/d2b/d43/l6a 0 2024-04-16T02:35:34.915 INFO:tasks.workunit.client.1.smithi153.stdout:5/353: creat d8/df/d28/d5a/f68 x:0 0 0 2024-04-16T02:35:34.916 INFO:tasks.workunit.client.1.smithi153.stdout:3/338: mknod d6/d10/d3b/d59/c79 0 2024-04-16T02:35:34.917 INFO:tasks.workunit.client.1.smithi153.stdout:5/354: readlink d8/df/l45 0 2024-04-16T02:35:34.920 INFO:tasks.workunit.client.1.smithi153.stdout:0/380: write dd/d14/d19/d1c/f4d [588806,121760] 0 2024-04-16T02:35:34.921 INFO:tasks.workunit.client.1.smithi153.stdout:8/331: creat d2/d1c/d1f/d27/f6e x:0 0 0 2024-04-16T02:35:34.923 INFO:tasks.workunit.client.1.smithi153.stdout:1/370: dread d4/f40 [0,4194304] 0 2024-04-16T02:35:34.923 INFO:tasks.workunit.client.1.smithi153.stdout:2/358: dwrite d9/d24/d27/d2c/d41/f4c [0,4194304] 0 2024-04-16T02:35:34.924 INFO:tasks.workunit.client.1.smithi153.stdout:3/339: truncate d6/d10/d22/d2c/f3d 4545616 0 2024-04-16T02:35:34.926 INFO:tasks.workunit.client.1.smithi153.stdout:5/355: dread d8/df/f16 [0,4194304] 0 2024-04-16T02:35:34.928 INFO:tasks.workunit.client.1.smithi153.stdout:7/331: getdents d0/d4/db/d34/d4b/d4d 0 2024-04-16T02:35:34.929 INFO:tasks.workunit.client.1.smithi153.stdout:1/371: write d4/dd/d13/d23/d5f/f69 [401040,75960] 0 2024-04-16T02:35:34.933 INFO:tasks.workunit.client.1.smithi153.stdout:3/340: unlink d6/d10/f13 0 2024-04-16T02:35:34.935 INFO:tasks.workunit.client.1.smithi153.stdout:9/381: dwrite d4/d6/d1b/f59 [0,4194304] 0 2024-04-16T02:35:34.937 INFO:tasks.workunit.client.1.smithi153.stdout:0/381: creat dd/d28/f74 x:0 0 0 2024-04-16T02:35:34.938 INFO:tasks.workunit.client.1.smithi153.stdout:8/332: dread d2/f12 [0,4194304] 0 2024-04-16T02:35:34.938 INFO:tasks.workunit.client.1.smithi153.stdout:3/341: read d6/f52 [903403,77819] 0 2024-04-16T02:35:34.938 INFO:tasks.workunit.client.1.smithi153.stdout:9/382: chown d4/d33/f4f 5515426 1 2024-04-16T02:35:34.939 INFO:tasks.workunit.client.1.smithi153.stdout:7/332: creat d0/d4/db/d2b/d43/f6b x:0 0 0 2024-04-16T02:35:34.939 INFO:tasks.workunit.client.1.smithi153.stdout:5/356: mknod d8/d41/c69 0 2024-04-16T02:35:34.942 INFO:tasks.workunit.client.1.smithi153.stdout:8/333: fdatasync d2/d13/f62 0 2024-04-16T02:35:34.945 INFO:tasks.workunit.client.1.smithi153.stdout:3/342: chown d6/d10/d22/d2c/f63 6836 1 2024-04-16T02:35:34.946 INFO:tasks.workunit.client.1.smithi153.stdout:7/333: rmdir d0/d13 39 2024-04-16T02:35:34.947 INFO:tasks.workunit.client.1.smithi153.stdout:9/383: creat d4/d6/f77 x:0 0 0 2024-04-16T02:35:34.947 INFO:tasks.workunit.client.1.smithi153.stdout:8/334: unlink d2/c18 0 2024-04-16T02:35:34.948 INFO:tasks.workunit.client.1.smithi153.stdout:0/382: creat dd/d14/d51/d54/d65/d68/f75 x:0 0 0 2024-04-16T02:35:34.949 INFO:tasks.workunit.client.1.smithi153.stdout:4/400: sync 2024-04-16T02:35:34.951 INFO:tasks.workunit.client.1.smithi153.stdout:8/335: fsync d2/f14 0 2024-04-16T02:35:34.955 INFO:tasks.workunit.client.1.smithi153.stdout:2/359: dwrite d9/d2b/f46 [0,4194304] 0 2024-04-16T02:35:34.956 INFO:tasks.workunit.client.1.smithi153.stdout:1/372: dread d4/dd/d18/f4b [0,4194304] 0 2024-04-16T02:35:34.959 INFO:tasks.workunit.client.1.smithi153.stdout:5/357: dwrite d8/df/d2f/d4e/d51/f5c [0,4194304] 0 2024-04-16T02:35:34.961 INFO:tasks.workunit.client.1.smithi153.stdout:3/343: rename d6/d10/d22/d2e/f54 to d6/d10/d22/d4d/f7a 0 2024-04-16T02:35:34.964 INFO:tasks.workunit.client.1.smithi153.stdout:0/383: read dd/d14/d19/d1c/f36 [611470,114628] 0 2024-04-16T02:35:34.964 INFO:tasks.workunit.client.1.smithi153.stdout:7/334: mkdir d0/d4/db/d34/d51/d5f/d68/d6c 0 2024-04-16T02:35:34.966 INFO:tasks.workunit.client.1.smithi153.stdout:8/336: dread d2/de/f24 [0,4194304] 0 2024-04-16T02:35:34.966 INFO:tasks.workunit.client.1.smithi153.stdout:0/384: chown dd/d14/d51/d54/d65 249710 1 2024-04-16T02:35:34.971 INFO:tasks.workunit.client.1.smithi153.stdout:4/401: creat d3/d9/f8d x:0 0 0 2024-04-16T02:35:34.972 INFO:tasks.workunit.client.1.smithi153.stdout:4/402: truncate d3/d9/db/de/d2c/f56 1411612 0 2024-04-16T02:35:34.972 INFO:tasks.workunit.client.1.smithi153.stdout:5/358: rename d8/df/c1c to d8/df/d23/c6a 0 2024-04-16T02:35:34.974 INFO:tasks.workunit.client.1.smithi153.stdout:3/344: read d6/d10/d22/f2a [2660357,3752] 0 2024-04-16T02:35:34.976 INFO:tasks.workunit.client.1.smithi153.stdout:3/345: readlink d6/d10/d3b/l50 0 2024-04-16T02:35:34.977 INFO:tasks.workunit.client.1.smithi153.stdout:0/385: read dd/d32/f39 [361881,60985] 0 2024-04-16T02:35:34.977 INFO:tasks.workunit.client.1.smithi153.stdout:9/384: dread d4/d6/dc/d24/f25 [0,4194304] 0 2024-04-16T02:35:34.978 INFO:tasks.workunit.client.1.smithi153.stdout:5/359: creat d8/d1f/d20/d39/f6b x:0 0 0 2024-04-16T02:35:34.980 INFO:tasks.workunit.client.1.smithi153.stdout:4/403: rename d3/d9/db/d17/d3f/d6b/l88 to d3/d9/d24/d35/l8e 0 2024-04-16T02:35:34.982 INFO:tasks.workunit.client.1.smithi153.stdout:3/346: unlink d6/c61 0 2024-04-16T02:35:34.987 INFO:tasks.workunit.client.1.smithi153.stdout:9/385: rename d4/d33/d3a/f5f to d4/d33/d3a/d40/f78 0 2024-04-16T02:35:34.989 INFO:tasks.workunit.client.1.smithi153.stdout:9/386: truncate d4/d6/dc/d46/f55 369755 0 2024-04-16T02:35:34.991 INFO:tasks.workunit.client.1.smithi153.stdout:3/347: getdents d6/d10/d3b/d76 0 2024-04-16T02:35:34.993 INFO:tasks.workunit.client.1.smithi153.stdout:7/335: dread d0/d4/db/ff [0,4194304] 0 2024-04-16T02:35:34.995 INFO:tasks.workunit.client.1.smithi153.stdout:0/386: dread dd/d28/f3c [0,4194304] 0 2024-04-16T02:35:34.996 INFO:tasks.workunit.client.1.smithi153.stdout:3/348: creat d6/d10/d3b/f7b x:0 0 0 2024-04-16T02:35:34.999 INFO:tasks.workunit.client.1.smithi153.stdout:8/337: dread d2/f14 [0,4194304] 0 2024-04-16T02:35:35.002 INFO:tasks.workunit.client.1.smithi153.stdout:7/336: rename d0/d4/db/d34/d51/d5f to d0/d4/db/d2b/d43/d6d 0 2024-04-16T02:35:35.003 INFO:tasks.workunit.client.1.smithi153.stdout:0/387: write dd/d14/d19/d1c/f36 [1433791,70746] 0 2024-04-16T02:35:35.004 INFO:tasks.workunit.client.1.smithi153.stdout:3/349: mkdir d6/d10/d3b/d59/d7c 0 2024-04-16T02:35:35.005 INFO:tasks.workunit.client.1.smithi153.stdout:4/404: getdents d3/d9/d24/d35 0 2024-04-16T02:35:35.009 INFO:tasks.workunit.client.1.smithi153.stdout:0/388: mknod dd/d26/c76 0 2024-04-16T02:35:35.009 INFO:tasks.workunit.client.1.smithi153.stdout:0/389: stat dd/d28/f3c 0 2024-04-16T02:35:35.012 INFO:tasks.workunit.client.1.smithi153.stdout:9/387: truncate d4/d33/d3a/d40/f78 855438 0 2024-04-16T02:35:35.012 INFO:tasks.workunit.client.1.smithi153.stdout:8/338: truncate d2/d1c/d1f/f53 776375 0 2024-04-16T02:35:35.013 INFO:tasks.workunit.client.1.smithi153.stdout:3/350: link d6/f11 d6/d10/d3b/d76/f7d 0 2024-04-16T02:35:35.068 INFO:tasks.workunit.client.0.smithi151.stdout:3/13: chown d1/l2 344633 1 2024-04-16T02:35:35.068 INFO:tasks.workunit.client.0.smithi151.stdout:3/14: write - no filename 2024-04-16T02:35:35.070 INFO:tasks.workunit.client.0.smithi151.stdout:3/15: symlink d1/l3 0 2024-04-16T02:35:35.070 INFO:tasks.workunit.client.0.smithi151.stdout:3/16: fsync - no filename 2024-04-16T02:35:35.070 INFO:tasks.workunit.client.0.smithi151.stdout:3/17: fdatasync - no filename 2024-04-16T02:35:35.070 INFO:tasks.workunit.client.0.smithi151.stdout:3/18: dread - no filename 2024-04-16T02:35:35.070 INFO:tasks.workunit.client.0.smithi151.stdout:3/19: read - no filename 2024-04-16T02:35:35.072 INFO:tasks.workunit.client.0.smithi151.stdout:3/20: rename d1/l2 to d1/l4 0 2024-04-16T02:35:35.072 INFO:tasks.workunit.client.0.smithi151.stdout:3/21: fdatasync - no filename 2024-04-16T02:35:35.074 INFO:tasks.workunit.client.0.smithi151.stdout:3/22: mknod d1/c5 0 2024-04-16T02:35:35.076 INFO:tasks.workunit.client.0.smithi151.stdout:8/48: getdents . 0 2024-04-16T02:35:35.076 INFO:tasks.workunit.client.0.smithi151.stdout:8/49: read - ff zero size 2024-04-16T02:35:35.078 INFO:tasks.workunit.client.0.smithi151.stdout:8/50: symlink l10 0 2024-04-16T02:35:35.082 INFO:tasks.workunit.client.0.smithi151.stdout:8/51: dread fb [0,4194304] 0 2024-04-16T02:35:35.083 INFO:tasks.workunit.client.1.smithi153.stdout:6/314: sync 2024-04-16T02:35:35.084 INFO:tasks.workunit.client.0.smithi151.stdout:8/52: unlink l1 0 2024-04-16T02:35:35.085 INFO:tasks.workunit.client.0.smithi151.stdout:8/53: write fb [5329290,74425] 0 2024-04-16T02:35:35.086 INFO:tasks.workunit.client.0.smithi151.stdout:8/54: write f3 [5023487,2127] 0 2024-04-16T02:35:35.087 INFO:tasks.workunit.client.0.smithi151.stdout:6/12: fsync f1 0 2024-04-16T02:35:35.089 INFO:tasks.workunit.client.0.smithi151.stdout:8/55: mkdir d11 0 2024-04-16T02:35:35.090 INFO:tasks.workunit.client.0.smithi151.stdout:8/56: fdatasync fe 0 2024-04-16T02:35:35.091 INFO:tasks.workunit.client.0.smithi151.stdout:6/13: mkdir d2 0 2024-04-16T02:35:35.092 INFO:tasks.workunit.client.1.smithi153.stdout:9/388: dwrite d4/d6/f56 [4194304,4194304] 0 2024-04-16T02:35:35.092 INFO:tasks.workunit.client.1.smithi153.stdout:0/390: mkdir dd/d14/d19/d77 0 2024-04-16T02:35:35.092 INFO:tasks.workunit.client.1.smithi153.stdout:4/405: link d3/d9/db/d17/l41 d3/d9/db/l8f 0 2024-04-16T02:35:35.092 INFO:tasks.workunit.client.1.smithi153.stdout:5/360: rename d8/df/f10 to d8/f6c 0 2024-04-16T02:35:35.093 INFO:tasks.workunit.client.1.smithi153.stdout:8/339: creat d2/de/d34/f6f x:0 0 0 2024-04-16T02:35:35.095 INFO:tasks.workunit.client.0.smithi151.stdout:8/57: dread f2 [0,4194304] 0 2024-04-16T02:35:35.098 INFO:tasks.workunit.client.1.smithi153.stdout:3/351: mknod d6/d10/d3b/d59/d7c/c7e 0 2024-04-16T02:35:35.098 INFO:tasks.workunit.client.1.smithi153.stdout:9/389: chown d4/d33/d3a/d69/c6a 524023774 1 2024-04-16T02:35:35.098 INFO:tasks.workunit.client.1.smithi153.stdout:8/340: dread - d2/d13/d2c/f48 zero size 2024-04-16T02:35:35.098 INFO:tasks.workunit.client.0.smithi151.stdout:8/58: write fe [819942,28442] 0 2024-04-16T02:35:35.098 INFO:tasks.workunit.client.1.smithi153.stdout:4/406: dread d3/f6d [0,4194304] 0 2024-04-16T02:35:35.099 INFO:tasks.workunit.client.1.smithi153.stdout:0/391: rename dd/d32/f39 to dd/d14/d19/d20/d4a/f78 0 2024-04-16T02:35:35.101 INFO:tasks.workunit.client.0.smithi151.stdout:6/14: symlink d2/l3 0 2024-04-16T02:35:35.103 INFO:tasks.workunit.client.1.smithi153.stdout:5/361: mkdir d8/d1f/d20/d39/d6d 0 2024-04-16T02:35:35.103 INFO:tasks.workunit.client.1.smithi153.stdout:6/315: write dc/d47/f55 [20130,93457] 0 2024-04-16T02:35:35.104 INFO:tasks.workunit.client.0.smithi151.stdout:8/59: write f2 [1103203,47842] 0 2024-04-16T02:35:35.105 INFO:tasks.workunit.client.1.smithi153.stdout:5/362: write d8/d1f/d20/d39/f6b [910498,96392] 0 2024-04-16T02:35:35.105 INFO:tasks.workunit.client.0.smithi151.stdout:5/60: unlink d2/d6/f11 0 2024-04-16T02:35:35.107 INFO:tasks.workunit.client.1.smithi153.stdout:8/341: symlink d2/d13/d3a/d5b/l70 0 2024-04-16T02:35:35.108 INFO:tasks.workunit.client.0.smithi151.stdout:9/44: getdents d1/d3 0 2024-04-16T02:35:35.109 INFO:tasks.workunit.client.1.smithi153.stdout:8/342: write d2/de/d34/f68 [615471,28603] 0 2024-04-16T02:35:35.109 INFO:tasks.workunit.client.0.smithi151.stdout:6/15: dwrite f1 [0,4194304] 0 2024-04-16T02:35:35.109 INFO:tasks.workunit.client.1.smithi153.stdout:0/392: mkdir dd/d14/d51/d54/d79 0 2024-04-16T02:35:35.109 INFO:tasks.workunit.client.1.smithi153.stdout:8/343: chown d2/d13/d3a/f46 259287715 1 2024-04-16T02:35:35.111 INFO:tasks.workunit.client.0.smithi151.stdout:9/45: write d1/f5 [816147,2432] 0 2024-04-16T02:35:35.114 INFO:tasks.workunit.client.0.smithi151.stdout:6/16: dread f1 [0,4194304] 0 2024-04-16T02:35:35.117 INFO:tasks.workunit.client.0.smithi151.stdout:9/46: mkdir d1/dd 0 2024-04-16T02:35:35.118 INFO:tasks.workunit.client.1.smithi153.stdout:6/316: symlink dc/d47/d57/l69 0 2024-04-16T02:35:35.118 INFO:tasks.workunit.client.1.smithi153.stdout:3/352: creat d6/d10/d3b/d76/f7f x:0 0 0 2024-04-16T02:35:35.118 INFO:tasks.workunit.client.0.smithi151.stdout:9/47: dread - d1/f7 zero size 2024-04-16T02:35:35.119 INFO:tasks.workunit.client.1.smithi153.stdout:6/317: fsync dc/d47/f55 0 2024-04-16T02:35:35.120 INFO:tasks.workunit.client.0.smithi151.stdout:5/61: write d2/fb [477443,129266] 0 2024-04-16T02:35:35.120 INFO:tasks.workunit.client.0.smithi151.stdout:1/28: rmdir d3 39 2024-04-16T02:35:35.123 INFO:tasks.workunit.client.0.smithi151.stdout:6/17: dwrite f1 [0,4194304] 0 2024-04-16T02:35:35.123 INFO:tasks.workunit.client.0.smithi151.stdout:2/60: getdents de 0 2024-04-16T02:35:35.125 INFO:tasks.workunit.client.1.smithi153.stdout:8/344: creat d2/d1c/d1f/d27/d2d/d47/d60/f71 x:0 0 0 2024-04-16T02:35:35.128 INFO:tasks.workunit.client.1.smithi153.stdout:4/407: dwrite d3/d9/db/de/f5c [0,4194304] 0 2024-04-16T02:35:35.129 INFO:tasks.workunit.client.0.smithi151.stdout:9/48: mknod d1/d3/ce 0 2024-04-16T02:35:35.130 INFO:tasks.workunit.client.1.smithi153.stdout:8/345: write d2/de/f3e [4221818,84774] 0 2024-04-16T02:35:35.131 INFO:tasks.workunit.client.0.smithi151.stdout:9/49: truncate d1/d3/fb 108888 0 2024-04-16T02:35:35.132 INFO:tasks.workunit.client.1.smithi153.stdout:3/353: dread d6/d10/d22/d2c/f33 [0,4194304] 0 2024-04-16T02:35:35.133 INFO:tasks.workunit.client.1.smithi153.stdout:6/318: truncate dc/d10/f1b 1198822 0 2024-04-16T02:35:35.134 INFO:tasks.workunit.client.0.smithi151.stdout:5/62: fsync d2/fd 0 2024-04-16T02:35:35.135 INFO:tasks.workunit.client.0.smithi151.stdout:5/63: read d2/f12 [430066,87478] 0 2024-04-16T02:35:35.135 INFO:tasks.workunit.client.1.smithi153.stdout:4/408: write d3/d9/d24/f2b [3742784,10546] 0 2024-04-16T02:35:35.137 INFO:tasks.workunit.client.1.smithi153.stdout:6/319: stat dc/d10/d62/c64 0 2024-04-16T02:35:35.138 INFO:tasks.workunit.client.1.smithi153.stdout:0/393: dwrite dd/d28/f41 [0,4194304] 0 2024-04-16T02:35:35.138 INFO:tasks.workunit.client.1.smithi153.stdout:5/363: rename d8/df/d2f/f33 to d8/df/d2e/f6e 0 2024-04-16T02:35:35.141 INFO:tasks.workunit.client.1.smithi153.stdout:3/354: creat d6/d10/d22/d2e/f80 x:0 0 0 2024-04-16T02:35:35.142 INFO:tasks.workunit.client.1.smithi153.stdout:0/394: truncate dd/d14/d19/d20/f70 689835 0 2024-04-16T02:35:35.143 INFO:tasks.workunit.client.0.smithi151.stdout:2/61: creat de/f14 x:0 0 0 2024-04-16T02:35:35.143 INFO:tasks.workunit.client.0.smithi151.stdout:1/29: symlink d3/d4/la 0 2024-04-16T02:35:35.145 INFO:tasks.workunit.client.1.smithi153.stdout:3/355: chown d6/d10/d22/d2c/l4f 337 1 2024-04-16T02:35:35.145 INFO:tasks.workunit.client.1.smithi153.stdout:0/395: chown dd/d26/c76 8361404 1 2024-04-16T02:35:35.146 INFO:tasks.workunit.client.1.smithi153.stdout:8/346: mknod d2/d1c/d1f/d27/c72 0 2024-04-16T02:35:35.147 INFO:tasks.workunit.client.0.smithi151.stdout:5/64: creat d2/d6/f13 x:0 0 0 2024-04-16T02:35:35.150 INFO:tasks.workunit.client.0.smithi151.stdout:9/50: mknod d1/dd/cf 0 2024-04-16T02:35:35.150 INFO:tasks.workunit.client.1.smithi153.stdout:5/364: dwrite d8/d1f/d20/d39/f6b [0,4194304] 0 2024-04-16T02:35:35.151 INFO:tasks.workunit.client.1.smithi153.stdout:9/390: dwrite d4/f13 [0,4194304] 0 2024-04-16T02:35:35.152 INFO:tasks.workunit.client.0.smithi151.stdout:1/30: mkdir d3/d4/db 0 2024-04-16T02:35:35.152 INFO:tasks.workunit.client.0.smithi151.stdout:1/31: readlink d3/d4/d5/l6 0 2024-04-16T02:35:35.156 INFO:tasks.workunit.client.0.smithi151.stdout:9/51: mkdir d1/d3/d10 0 2024-04-16T02:35:35.157 INFO:tasks.workunit.client.1.smithi153.stdout:8/347: creat d2/d1c/d1f/d27/d5e/f73 x:0 0 0 2024-04-16T02:35:35.159 INFO:tasks.workunit.client.0.smithi151.stdout:2/62: dwrite f7 [0,4194304] 0 2024-04-16T02:35:35.160 INFO:tasks.workunit.client.1.smithi153.stdout:0/396: dwrite dd/d14/d19/d20/f70 [0,4194304] 0 2024-04-16T02:35:35.163 INFO:tasks.workunit.client.1.smithi153.stdout:5/365: write d8/df/d23/f56 [1124196,27771] 0 2024-04-16T02:35:35.163 INFO:tasks.workunit.client.1.smithi153.stdout:4/409: getdents d3/d9/db/de/d2c/d52 0 2024-04-16T02:35:35.166 INFO:tasks.workunit.client.1.smithi153.stdout:4/410: write d3/d9/f8d [136473,106299] 0 2024-04-16T02:35:35.168 INFO:tasks.workunit.client.1.smithi153.stdout:3/356: getdents d6/d10/d3b/d59/d7c 0 2024-04-16T02:35:35.168 INFO:tasks.workunit.client.0.smithi151.stdout:2/63: mkdir de/d15 0 2024-04-16T02:35:35.171 INFO:tasks.workunit.client.1.smithi153.stdout:0/397: read dd/d14/d19/d1c/f3e [2098881,29240] 0 2024-04-16T02:35:35.172 INFO:tasks.workunit.client.0.smithi151.stdout:9/52: dwrite d1/d3/fa [0,4194304] 0 2024-04-16T02:35:35.175 INFO:tasks.workunit.client.1.smithi153.stdout:4/411: creat d3/d9/db/de/d2c/d52/d85/f90 x:0 0 0 2024-04-16T02:35:35.179 INFO:tasks.workunit.client.1.smithi153.stdout:0/398: dread dd/f10 [0,4194304] 0 2024-04-16T02:35:35.179 INFO:tasks.workunit.client.0.smithi151.stdout:2/64: link de/c13 de/d15/c16 0 2024-04-16T02:35:35.183 INFO:tasks.workunit.client.0.smithi151.stdout:3/23: getdents d1 0 2024-04-16T02:35:35.183 INFO:tasks.workunit.client.0.smithi151.stdout:3/24: dwrite - no filename 2024-04-16T02:35:35.185 INFO:tasks.workunit.client.0.smithi151.stdout:3/25: mkdir d1/d6 0 2024-04-16T02:35:35.185 INFO:tasks.workunit.client.1.smithi153.stdout:4/412: unlink d3/d9/db/c2e 0 2024-04-16T02:35:35.186 INFO:tasks.workunit.client.1.smithi153.stdout:0/399: mknod dd/d14/c7a 0 2024-04-16T02:35:35.187 INFO:tasks.workunit.client.1.smithi153.stdout:0/400: chown dd/d32/d3f/c66 9134 1 2024-04-16T02:35:35.189 INFO:tasks.workunit.client.0.smithi151.stdout:3/26: creat d1/d6/f7 x:0 0 0 2024-04-16T02:35:35.190 INFO:tasks.workunit.client.0.smithi151.stdout:3/27: truncate d1/d6/f7 942749 0 2024-04-16T02:35:35.191 INFO:tasks.workunit.client.1.smithi153.stdout:0/401: mknod dd/d14/d19/d1c/d73/c7b 0 2024-04-16T02:35:35.192 INFO:tasks.workunit.client.1.smithi153.stdout:4/413: getdents d3/d9/db/d17/d28/d4c 0 2024-04-16T02:35:35.192 INFO:tasks.workunit.client.1.smithi153.stdout:6/320: dread f3 [0,4194304] 0 2024-04-16T02:35:35.192 INFO:tasks.workunit.client.1.smithi153.stdout:0/402: dread - dd/d32/d57/f4e zero size 2024-04-16T02:35:35.192 INFO:tasks.workunit.client.0.smithi151.stdout:3/28: write d1/d6/f7 [877575,63473] 0 2024-04-16T02:35:35.195 INFO:tasks.workunit.client.1.smithi153.stdout:6/321: unlink dc/d47/f55 0 2024-04-16T02:35:35.195 INFO:tasks.workunit.client.0.smithi151.stdout:3/29: unlink d1/c5 0 2024-04-16T02:35:35.195 INFO:tasks.workunit.client.1.smithi153.stdout:4/414: symlink d3/d9/db/d17/l91 0 2024-04-16T02:35:35.196 INFO:tasks.workunit.client.0.smithi151.stdout:8/60: truncate f3 466599 0 2024-04-16T02:35:35.197 INFO:tasks.workunit.client.0.smithi151.stdout:3/30: mknod d1/d6/c8 0 2024-04-16T02:35:35.199 INFO:tasks.workunit.client.1.smithi153.stdout:4/415: stat d3/d9/db/de/d2c/c87 0 2024-04-16T02:35:35.199 INFO:tasks.workunit.client.1.smithi153.stdout:0/403: mknod dd/d14/d19/d55/c7c 0 2024-04-16T02:35:35.201 INFO:tasks.workunit.client.1.smithi153.stdout:4/416: fsync d3/d9/db/de/f26 0 2024-04-16T02:35:35.203 INFO:tasks.workunit.client.1.smithi153.stdout:6/322: dread dc/d12/d15/f5a [0,4194304] 0 2024-04-16T02:35:35.203 INFO:tasks.workunit.client.1.smithi153.stdout:5/366: dread d8/f4d [0,4194304] 0 2024-04-16T02:35:35.203 INFO:tasks.workunit.client.1.smithi153.stdout:0/404: unlink dd/d14/d51/d54/d65/d68/f75 0 2024-04-16T02:35:35.205 INFO:tasks.workunit.client.0.smithi151.stdout:5/65: getdents d2/d6 0 2024-04-16T02:35:35.205 INFO:tasks.workunit.client.0.smithi151.stdout:6/18: fsync f1 0 2024-04-16T02:35:35.206 INFO:tasks.workunit.client.1.smithi153.stdout:9/391: dread d4/f48 [0,4194304] 0 2024-04-16T02:35:35.206 INFO:tasks.workunit.client.0.smithi151.stdout:6/19: write f1 [948827,111096] 0 2024-04-16T02:35:35.208 INFO:tasks.workunit.client.1.smithi153.stdout:0/405: mkdir dd/d14/d51/d7d 0 2024-04-16T02:35:35.209 INFO:tasks.workunit.client.1.smithi153.stdout:3/357: write d6/d10/d22/d2e/f45 [970948,24045] 0 2024-04-16T02:35:35.209 INFO:tasks.workunit.client.0.smithi151.stdout:6/20: symlink d2/l4 0 2024-04-16T02:35:35.210 INFO:tasks.workunit.client.1.smithi153.stdout:4/417: creat d3/d9/d79/f92 x:0 0 0 2024-04-16T02:35:35.210 INFO:tasks.workunit.client.0.smithi151.stdout:6/21: write f1 [5219130,87153] 0 2024-04-16T02:35:35.211 INFO:tasks.workunit.client.0.smithi151.stdout:5/66: truncate d2/fd 3357966 0 2024-04-16T02:35:35.211 INFO:tasks.workunit.client.1.smithi153.stdout:6/323: truncate dc/d1a/f36 342273 0 2024-04-16T02:35:35.212 INFO:tasks.workunit.client.1.smithi153.stdout:6/324: fdatasync dc/d12/d15/f2f 0 2024-04-16T02:35:35.213 INFO:tasks.workunit.client.1.smithi153.stdout:0/406: mknod dd/d14/d51/c7e 0 2024-04-16T02:35:35.213 INFO:tasks.workunit.client.0.smithi151.stdout:6/22: fsync f1 0 2024-04-16T02:35:35.214 INFO:tasks.workunit.client.1.smithi153.stdout:6/325: read dc/d1a/f58 [34124,20178] 0 2024-04-16T02:35:35.214 INFO:tasks.workunit.client.1.smithi153.stdout:3/358: mkdir d6/d10/d5b/d81 0 2024-04-16T02:35:35.215 INFO:tasks.workunit.client.0.smithi151.stdout:5/67: symlink d2/d6/l14 0 2024-04-16T02:35:35.215 INFO:tasks.workunit.client.1.smithi153.stdout:4/418: symlink d3/d9/db/de/d2c/d38/l93 0 2024-04-16T02:35:35.216 INFO:tasks.workunit.client.0.smithi151.stdout:6/23: mkdir d2/d5 0 2024-04-16T02:35:35.216 INFO:tasks.workunit.client.1.smithi153.stdout:5/367: creat d8/f6f x:0 0 0 2024-04-16T02:35:35.217 INFO:tasks.workunit.client.1.smithi153.stdout:5/368: fdatasync d8/df/d2e/f5b 0 2024-04-16T02:35:35.217 INFO:tasks.workunit.client.0.smithi151.stdout:5/68: symlink d2/d6/l15 0 2024-04-16T02:35:35.219 INFO:tasks.workunit.client.1.smithi153.stdout:3/359: creat d6/d10/d5b/f82 x:0 0 0 2024-04-16T02:35:35.219 INFO:tasks.workunit.client.1.smithi153.stdout:0/407: dwrite dd/d28/f74 [0,4194304] 0 2024-04-16T02:35:35.221 INFO:tasks.workunit.client.0.smithi151.stdout:5/69: dread d2/f12 [0,4194304] 0 2024-04-16T02:35:35.222 INFO:tasks.workunit.client.1.smithi153.stdout:0/408: write dd/d26/d44/f50 [3328930,101964] 0 2024-04-16T02:35:35.224 INFO:tasks.workunit.client.1.smithi153.stdout:5/369: read - d8/df/d28/d5a/f68 zero size 2024-04-16T02:35:35.224 INFO:tasks.workunit.client.0.smithi151.stdout:6/24: dwrite f1 [4194304,4194304] 0 2024-04-16T02:35:35.225 INFO:tasks.workunit.client.1.smithi153.stdout:5/370: chown d8/d41/c69 58 1 2024-04-16T02:35:35.226 INFO:tasks.workunit.client.1.smithi153.stdout:3/360: symlink d6/d10/d22/d4d/l83 0 2024-04-16T02:35:35.227 INFO:tasks.workunit.client.0.smithi151.stdout:3/31: rmdir d1 39 2024-04-16T02:35:35.228 INFO:tasks.workunit.client.1.smithi153.stdout:0/409: fdatasync dd/f10 0 2024-04-16T02:35:35.228 INFO:tasks.workunit.client.1.smithi153.stdout:4/419: dwrite d3/d9/db/de/f12 [0,4194304] 0 2024-04-16T02:35:35.232 INFO:tasks.workunit.client.0.smithi151.stdout:6/25: creat d2/d5/f6 x:0 0 0 2024-04-16T02:35:35.232 INFO:tasks.workunit.client.0.smithi151.stdout:6/26: write f1 [7214315,1512] 0 2024-04-16T02:35:35.232 INFO:tasks.workunit.client.0.smithi151.stdout:5/70: creat d2/f16 x:0 0 0 2024-04-16T02:35:35.233 INFO:tasks.workunit.client.0.smithi151.stdout:3/32: rename d1/d6/f7 to d1/d6/f9 0 2024-04-16T02:35:35.235 INFO:tasks.workunit.client.1.smithi153.stdout:2/360: sync 2024-04-16T02:35:35.237 INFO:tasks.workunit.client.1.smithi153.stdout:5/371: dwrite d8/df/d2f/d4e/d51/f5c [4194304,4194304] 0 2024-04-16T02:35:35.240 INFO:tasks.workunit.client.1.smithi153.stdout:0/410: mkdir dd/d14/d51/d7f 0 2024-04-16T02:35:35.241 INFO:tasks.workunit.client.1.smithi153.stdout:4/420: creat d3/d9/db/d17/d3f/f94 x:0 0 0 2024-04-16T02:35:35.242 INFO:tasks.workunit.client.1.smithi153.stdout:4/421: dread - d3/d9/db/d17/d3f/f4f zero size 2024-04-16T02:35:35.243 INFO:tasks.workunit.client.0.smithi151.stdout:5/71: symlink d2/l17 0 2024-04-16T02:35:35.244 INFO:tasks.workunit.client.0.smithi151.stdout:5/72: write d2/d6/fa [1010550,26452] 0 2024-04-16T02:35:35.246 INFO:tasks.workunit.client.0.smithi151.stdout:5/73: stat d2/l10 0 2024-04-16T02:35:35.248 INFO:tasks.workunit.client.0.smithi151.stdout:5/74: chown d2/d6/f9 5 1 2024-04-16T02:35:35.249 INFO:tasks.workunit.client.1.smithi153.stdout:6/326: rename dc/ff to dc/d10/d62/f6a 0 2024-04-16T02:35:35.250 INFO:tasks.workunit.client.1.smithi153.stdout:5/372: mkdir d8/df/d23/d70 0 2024-04-16T02:35:35.250 INFO:tasks.workunit.client.0.smithi151.stdout:5/75: readlink d2/l17 0 2024-04-16T02:35:35.251 INFO:tasks.workunit.client.0.smithi151.stdout:5/76: chown d2/d6/l15 67052256 1 2024-04-16T02:35:35.252 INFO:tasks.workunit.client.1.smithi153.stdout:0/411: mkdir dd/d32/d80 0 2024-04-16T02:35:35.252 INFO:tasks.workunit.client.0.smithi151.stdout:3/33: rename d1/d6/f9 to d1/fa 0 2024-04-16T02:35:35.253 INFO:tasks.workunit.client.0.smithi151.stdout:5/77: fdatasync d2/f5 0 2024-04-16T02:35:35.253 INFO:tasks.workunit.client.1.smithi153.stdout:4/422: creat d3/d9/d79/f95 x:0 0 0 2024-04-16T02:35:35.253 INFO:tasks.workunit.client.0.smithi151.stdout:8/61: dwrite f3 [0,4194304] 0 2024-04-16T02:35:35.256 INFO:tasks.workunit.client.1.smithi153.stdout:2/361: symlink d9/d2b/d18/l89 0 2024-04-16T02:35:35.258 INFO:tasks.workunit.client.0.smithi151.stdout:8/62: mknod d11/c12 0 2024-04-16T02:35:35.258 INFO:tasks.workunit.client.0.smithi151.stdout:3/34: dread d1/fa [0,4194304] 0 2024-04-16T02:35:35.259 INFO:tasks.workunit.client.0.smithi151.stdout:5/78: creat d2/f18 x:0 0 0 2024-04-16T02:35:35.260 INFO:tasks.workunit.client.1.smithi153.stdout:0/412: unlink dd/d14/c29 0 2024-04-16T02:35:35.261 INFO:tasks.workunit.client.1.smithi153.stdout:4/423: creat d3/d9/db/d17/d3f/f96 x:0 0 0 2024-04-16T02:35:35.262 INFO:tasks.workunit.client.1.smithi153.stdout:3/361: link d6/d10/d22/d2e/c67 d6/d10/d3b/d76/c84 0 2024-04-16T02:35:35.262 INFO:tasks.workunit.client.1.smithi153.stdout:0/413: read dd/ff [1564087,15839] 0 2024-04-16T02:35:35.263 INFO:tasks.workunit.client.1.smithi153.stdout:3/362: chown d6/d10/l1f 988970 1 2024-04-16T02:35:35.263 INFO:tasks.workunit.client.0.smithi151.stdout:5/79: mkdir d2/d6/d19 0 2024-04-16T02:35:35.264 INFO:tasks.workunit.client.1.smithi153.stdout:2/362: mknod d9/d24/d34/d60/c8a 0 2024-04-16T02:35:35.264 INFO:tasks.workunit.client.0.smithi151.stdout:8/63: link ld d11/l13 0 2024-04-16T02:35:35.265 INFO:tasks.workunit.client.1.smithi153.stdout:2/363: stat d9/d24/d34/d36/c3c 0 2024-04-16T02:35:35.265 INFO:tasks.workunit.client.1.smithi153.stdout:5/373: creat d8/d1f/d20/d39/d6d/f71 x:0 0 0 2024-04-16T02:35:35.266 INFO:tasks.workunit.client.1.smithi153.stdout:5/374: readlink d8/l17 0 2024-04-16T02:35:35.266 INFO:tasks.workunit.client.0.smithi151.stdout:5/80: chown d2/d6/f13 1365 1 2024-04-16T02:35:35.269 INFO:tasks.workunit.client.0.smithi151.stdout:3/35: dwrite d1/fa [0,4194304] 0 2024-04-16T02:35:35.271 INFO:tasks.workunit.client.1.smithi153.stdout:3/363: rename d6/f1a to d6/d10/d3b/f85 0 2024-04-16T02:35:35.272 INFO:tasks.workunit.client.1.smithi153.stdout:3/364: dread - d6/d10/d22/d2e/f80 zero size 2024-04-16T02:35:35.273 INFO:tasks.workunit.client.1.smithi153.stdout:5/375: mkdir d8/df/d28/d5a/d72 0 2024-04-16T02:35:35.275 INFO:tasks.workunit.client.1.smithi153.stdout:4/424: mknod d3/d9/db/c97 0 2024-04-16T02:35:35.278 INFO:tasks.workunit.client.0.smithi151.stdout:3/36: getdents d1 0 2024-04-16T02:35:35.279 INFO:tasks.workunit.client.0.smithi151.stdout:8/64: truncate f2 3234318 0 2024-04-16T02:35:35.280 INFO:tasks.workunit.client.0.smithi151.stdout:3/37: symlink d1/lb 0 2024-04-16T02:35:35.281 INFO:tasks.workunit.client.1.smithi153.stdout:5/376: creat d8/d1f/f73 x:0 0 0 2024-04-16T02:35:35.282 INFO:tasks.workunit.client.0.smithi151.stdout:3/38: mknod d1/d6/cc 0 2024-04-16T02:35:35.283 INFO:tasks.workunit.client.1.smithi153.stdout:5/377: write d8/d1f/d20/d39/d6d/f71 [488267,52211] 0 2024-04-16T02:35:35.283 INFO:tasks.workunit.client.1.smithi153.stdout:4/425: mkdir d3/d9/db/de/d2c/d98 0 2024-04-16T02:35:35.285 INFO:tasks.workunit.client.0.smithi151.stdout:8/65: link ff d11/f14 0 2024-04-16T02:35:35.285 INFO:tasks.workunit.client.1.smithi153.stdout:4/426: dread - d3/d9/db/d45/f7f zero size 2024-04-16T02:35:35.287 INFO:tasks.workunit.client.0.smithi151.stdout:3/39: rename d1/l3 to d1/ld 0 2024-04-16T02:35:35.288 INFO:tasks.workunit.client.0.smithi151.stdout:8/66: creat d11/f15 x:0 0 0 2024-04-16T02:35:35.289 INFO:tasks.workunit.client.1.smithi153.stdout:4/427: chown d3/d9/d24/d3e/c78 63 1 2024-04-16T02:35:35.291 INFO:tasks.workunit.client.0.smithi151.stdout:8/67: write d11/f14 [955093,91482] 0 2024-04-16T02:35:35.292 INFO:tasks.workunit.client.0.smithi151.stdout:3/40: dread d1/fa [0,4194304] 0 2024-04-16T02:35:35.293 INFO:tasks.workunit.client.1.smithi153.stdout:4/428: unlink d3/d9/db/d17/d3f/f96 0 2024-04-16T02:35:35.294 INFO:tasks.workunit.client.0.smithi151.stdout:3/41: fsync d1/fa 0 2024-04-16T02:35:35.296 INFO:tasks.workunit.client.0.smithi151.stdout:8/68: dread fe [0,4194304] 0 2024-04-16T02:35:35.297 INFO:tasks.workunit.client.1.smithi153.stdout:0/414: dwrite dd/f53 [0,4194304] 0 2024-04-16T02:35:35.297 INFO:tasks.workunit.client.1.smithi153.stdout:2/364: dread d9/d24/d27/f30 [0,4194304] 0 2024-04-16T02:35:35.298 INFO:tasks.workunit.client.0.smithi151.stdout:3/42: creat d1/d6/fe x:0 0 0 2024-04-16T02:35:35.302 INFO:tasks.workunit.client.0.smithi151.stdout:8/69: write fe [112659,45127] 0 2024-04-16T02:35:35.302 INFO:tasks.workunit.client.0.smithi151.stdout:8/70: chown f3 28041 1 2024-04-16T02:35:35.303 INFO:tasks.workunit.client.0.smithi151.stdout:3/43: creat d1/d6/ff x:0 0 0 2024-04-16T02:35:35.303 INFO:tasks.workunit.client.0.smithi151.stdout:3/44: chown d1/d6/fe 2344064 1 2024-04-16T02:35:35.303 INFO:tasks.workunit.client.0.smithi151.stdout:8/71: creat d11/f16 x:0 0 0 2024-04-16T02:35:35.303 INFO:tasks.workunit.client.1.smithi153.stdout:5/378: dread d8/d1f/f3f [0,4194304] 0 2024-04-16T02:35:35.303 INFO:tasks.workunit.client.1.smithi153.stdout:2/365: creat d9/f8b x:0 0 0 2024-04-16T02:35:35.304 INFO:tasks.workunit.client.0.smithi151.stdout:3/45: mkdir d1/d6/d10 0 2024-04-16T02:35:35.305 INFO:tasks.workunit.client.0.smithi151.stdout:8/72: creat d11/f17 x:0 0 0 2024-04-16T02:35:35.307 INFO:tasks.workunit.client.0.smithi151.stdout:3/46: creat d1/d6/f11 x:0 0 0 2024-04-16T02:35:35.307 INFO:tasks.workunit.client.1.smithi153.stdout:5/379: write d8/f4d [1959092,33871] 0 2024-04-16T02:35:35.308 INFO:tasks.workunit.client.0.smithi151.stdout:3/47: write d1/d6/fe [959979,22288] 0 2024-04-16T02:35:35.308 INFO:tasks.workunit.client.1.smithi153.stdout:4/429: dwrite d3/d9/db/d17/d3f/f4f [0,4194304] 0 2024-04-16T02:35:35.309 INFO:tasks.workunit.client.1.smithi153.stdout:3/365: dwrite d6/d10/d22/d43/f69 [0,4194304] 0 2024-04-16T02:35:35.311 INFO:tasks.workunit.client.1.smithi153.stdout:2/366: getdents d9/d71 0 2024-04-16T02:35:35.311 INFO:tasks.workunit.client.0.smithi151.stdout:8/73: link fb d11/f18 0 2024-04-16T02:35:35.313 INFO:tasks.workunit.client.1.smithi153.stdout:5/380: rmdir d8/d1f/d20 39 2024-04-16T02:35:35.314 INFO:tasks.workunit.client.0.smithi151.stdout:3/48: dread d1/fa [0,4194304] 0 2024-04-16T02:35:35.316 INFO:tasks.workunit.client.1.smithi153.stdout:4/430: symlink d3/d9/db/d45/l99 0 2024-04-16T02:35:35.317 INFO:tasks.workunit.client.0.smithi151.stdout:8/74: rename l4 to d11/l19 0 2024-04-16T02:35:35.319 INFO:tasks.workunit.client.0.smithi151.stdout:3/49: creat d1/d6/f12 x:0 0 0 2024-04-16T02:35:35.319 INFO:tasks.workunit.client.1.smithi153.stdout:2/367: creat d9/d24/d27/d2c/d5d/f8c x:0 0 0 2024-04-16T02:35:35.320 INFO:tasks.workunit.client.1.smithi153.stdout:4/431: symlink d3/d9/db/de/d2c/l9a 0 2024-04-16T02:35:35.321 INFO:tasks.workunit.client.1.smithi153.stdout:5/381: mknod d8/df/d23/d62/c74 0 2024-04-16T02:35:35.324 INFO:tasks.workunit.client.1.smithi153.stdout:5/382: write d8/d1f/f3f [1293412,29199] 0 2024-04-16T02:35:35.325 INFO:tasks.workunit.client.1.smithi153.stdout:4/432: mknod d3/d9/c9b 0 2024-04-16T02:35:35.325 INFO:tasks.workunit.client.0.smithi151.stdout:3/50: dwrite d1/fa [0,4194304] 0 2024-04-16T02:35:35.326 INFO:tasks.workunit.client.1.smithi153.stdout:5/383: fsync d8/d1f/d20/d39/f4c 0 2024-04-16T02:35:35.327 INFO:tasks.workunit.client.1.smithi153.stdout:4/433: creat d3/d9/db/d17/d28/f9c x:0 0 0 2024-04-16T02:35:35.327 INFO:tasks.workunit.client.1.smithi153.stdout:4/434: dread - d3/d9/d79/f92 zero size 2024-04-16T02:35:35.330 INFO:tasks.workunit.client.0.smithi151.stdout:3/51: symlink d1/d6/l13 0 2024-04-16T02:35:35.330 INFO:tasks.workunit.client.1.smithi153.stdout:3/366: write d6/d10/d22/f2a [28354,102903] 0 2024-04-16T02:35:35.330 INFO:tasks.workunit.client.1.smithi153.stdout:2/368: dread d9/d2b/f2a [0,4194304] 0 2024-04-16T02:35:35.333 INFO:tasks.workunit.client.0.smithi151.stdout:3/52: mknod d1/d6/c14 0 2024-04-16T02:35:35.334 INFO:tasks.workunit.client.1.smithi153.stdout:1/373: sync 2024-04-16T02:35:35.334 INFO:tasks.workunit.client.1.smithi153.stdout:7/337: sync 2024-04-16T02:35:35.334 INFO:tasks.workunit.client.1.smithi153.stdout:3/367: rmdir d6/d10/d22 39 2024-04-16T02:35:35.335 INFO:tasks.workunit.client.1.smithi153.stdout:5/384: dwrite d8/d1f/f73 [0,4194304] 0 2024-04-16T02:35:35.335 INFO:tasks.workunit.client.0.smithi151.stdout:3/53: rmdir d1 39 2024-04-16T02:35:35.335 INFO:tasks.workunit.client.1.smithi153.stdout:2/369: mkdir d9/d71/d8d 0 2024-04-16T02:35:35.336 INFO:tasks.workunit.client.1.smithi153.stdout:7/338: fsync d0/d4/db/d34/d4b/f69 0 2024-04-16T02:35:35.338 INFO:tasks.workunit.client.0.smithi151.stdout:3/54: unlink d1/d6/f11 0 2024-04-16T02:35:35.339 INFO:tasks.workunit.client.1.smithi153.stdout:5/385: mknod d8/d1f/c75 0 2024-04-16T02:35:35.339 INFO:tasks.workunit.client.1.smithi153.stdout:3/368: write d6/d10/d22/f31 [1598265,117072] 0 2024-04-16T02:35:35.340 INFO:tasks.workunit.client.1.smithi153.stdout:2/370: creat d9/d2b/d18/d3f/d40/f8e x:0 0 0 2024-04-16T02:35:35.340 INFO:tasks.workunit.client.1.smithi153.stdout:4/435: dread d3/d9/db/d17/f67 [0,4194304] 0 2024-04-16T02:35:35.341 INFO:tasks.workunit.client.0.smithi151.stdout:3/55: link d1/d6/ff d1/d6/d10/f15 0 2024-04-16T02:35:35.341 INFO:tasks.workunit.client.0.smithi151.stdout:3/56: read - d1/d6/f12 zero size 2024-04-16T02:35:35.342 INFO:tasks.workunit.client.1.smithi153.stdout:5/386: stat d8/l1a 0 2024-04-16T02:35:35.344 INFO:tasks.workunit.client.1.smithi153.stdout:7/339: creat d0/d4/db/d2b/d53/f6e x:0 0 0 2024-04-16T02:35:35.345 INFO:tasks.workunit.client.1.smithi153.stdout:4/436: chown d3/d9/db/d17/d28/f5e 1 1 2024-04-16T02:35:35.346 INFO:tasks.workunit.client.1.smithi153.stdout:3/369: creat d6/d10/d3b/d59/f86 x:0 0 0 2024-04-16T02:35:35.346 INFO:tasks.workunit.client.1.smithi153.stdout:7/340: write d0/d4/db/f57 [1158878,22192] 0 2024-04-16T02:35:35.348 INFO:tasks.workunit.client.0.smithi151.stdout:3/57: dwrite d1/fa [0,4194304] 0 2024-04-16T02:35:35.349 INFO:tasks.workunit.client.0.smithi151.stdout:3/58: stat d1 0 2024-04-16T02:35:35.349 INFO:tasks.workunit.client.1.smithi153.stdout:1/374: dwrite d4/dd/d13/d23/d5f/f61 [0,4194304] 0 2024-04-16T02:35:35.349 INFO:tasks.workunit.client.1.smithi153.stdout:4/437: write d3/d9/d24/f2b [4703779,36144] 0 2024-04-16T02:35:35.350 INFO:tasks.workunit.client.1.smithi153.stdout:1/375: readlink d4/dd/d13/l17 0 2024-04-16T02:35:35.350 INFO:tasks.workunit.client.1.smithi153.stdout:7/341: dread - d0/d4/db/d2b/f4e zero size 2024-04-16T02:35:35.351 INFO:tasks.workunit.client.1.smithi153.stdout:4/438: truncate d3/f1c 998051 0 2024-04-16T02:35:35.352 INFO:tasks.workunit.client.1.smithi153.stdout:7/342: write d0/d4/db/d2b/f28 [830299,110294] 0 2024-04-16T02:35:35.354 INFO:tasks.workunit.client.0.smithi151.stdout:3/59: dread d1/fa [0,4194304] 0 2024-04-16T02:35:35.354 INFO:tasks.workunit.client.1.smithi153.stdout:7/343: chown d0/d4/db/d2b/d53/d59/f5a 65933402 1 2024-04-16T02:35:35.355 INFO:tasks.workunit.client.1.smithi153.stdout:3/370: creat d6/d10/d3b/d59/f87 x:0 0 0 2024-04-16T02:35:35.355 INFO:tasks.workunit.client.0.smithi151.stdout:3/60: write d1/d6/fe [739984,37158] 0 2024-04-16T02:35:35.358 INFO:tasks.workunit.client.1.smithi153.stdout:1/376: creat d4/dd/d13/d23/d5f/f7f x:0 0 0 2024-04-16T02:35:35.358 INFO:tasks.workunit.client.1.smithi153.stdout:3/371: dread - d6/d10/d22/d43/f77 zero size 2024-04-16T02:35:35.359 INFO:tasks.workunit.client.0.smithi151.stdout:3/61: rename d1/d6/f12 to d1/d6/d10/f16 0 2024-04-16T02:35:35.360 INFO:tasks.workunit.client.1.smithi153.stdout:4/439: creat d3/d9/db/de/d2c/d38/f9d x:0 0 0 2024-04-16T02:35:35.360 INFO:tasks.workunit.client.0.smithi151.stdout:3/62: dread - d1/d6/ff zero size 2024-04-16T02:35:35.362 INFO:tasks.workunit.client.1.smithi153.stdout:3/372: truncate d6/d10/d5b/f82 344104 0 2024-04-16T02:35:35.362 INFO:tasks.workunit.client.1.smithi153.stdout:7/344: creat d0/d4/db/d2b/d43/d6d/f6f x:0 0 0 2024-04-16T02:35:35.363 INFO:tasks.workunit.client.1.smithi153.stdout:2/371: getdents d9/d24/d34/d60/d7c 0 2024-04-16T02:35:35.365 INFO:tasks.workunit.client.1.smithi153.stdout:5/387: dwrite d8/df/f1e [0,4194304] 0 2024-04-16T02:35:35.365 INFO:tasks.workunit.client.1.smithi153.stdout:7/345: chown d0/d4/db/d2b/f3e 356243896 1 2024-04-16T02:35:35.365 INFO:tasks.workunit.client.1.smithi153.stdout:2/372: readlink d9/d24/d27/l38 0 2024-04-16T02:35:35.366 INFO:tasks.workunit.client.1.smithi153.stdout:4/440: mkdir d3/d9/d24/d75/d9e 0 2024-04-16T02:35:35.366 INFO:tasks.workunit.client.1.smithi153.stdout:2/373: fdatasync d9/d24/d27/d2c/d41/f4c 0 2024-04-16T02:35:35.367 INFO:tasks.workunit.client.1.smithi153.stdout:1/377: mkdir d4/dd/d13/d7e/d80 0 2024-04-16T02:35:35.368 INFO:tasks.workunit.client.1.smithi153.stdout:3/373: rename d6/d10/d3b/d59/c79 to d6/d10/d3b/c88 0 2024-04-16T02:35:35.370 INFO:tasks.workunit.client.1.smithi153.stdout:7/346: unlink d0/f3d 0 2024-04-16T02:35:35.375 INFO:tasks.workunit.client.1.smithi153.stdout:4/441: rmdir d3/d9/db/d45 39 2024-04-16T02:35:35.375 INFO:tasks.workunit.client.1.smithi153.stdout:2/374: dread d9/d2b/f2a [0,4194304] 0 2024-04-16T02:35:35.375 INFO:tasks.workunit.client.1.smithi153.stdout:4/442: readlink d3/l49 0 2024-04-16T02:35:35.376 INFO:tasks.workunit.client.1.smithi153.stdout:1/378: symlink d4/dd/d13/d7e/l81 0 2024-04-16T02:35:35.377 INFO:tasks.workunit.client.1.smithi153.stdout:7/347: dwrite d0/d4/f58 [0,4194304] 0 2024-04-16T02:35:35.378 INFO:tasks.workunit.client.1.smithi153.stdout:4/443: write d3/d9/d24/d3e/f80 [17691,21720] 0 2024-04-16T02:35:35.378 INFO:tasks.workunit.client.1.smithi153.stdout:2/375: write d9/d24/d27/d2c/d5d/f8c [626953,95929] 0 2024-04-16T02:35:35.379 INFO:tasks.workunit.client.1.smithi153.stdout:3/374: unlink d6/f12 0 2024-04-16T02:35:35.380 INFO:tasks.workunit.client.1.smithi153.stdout:1/379: write d4/dd/d13/d23/f6d [207507,83717] 0 2024-04-16T02:35:35.382 INFO:tasks.workunit.client.1.smithi153.stdout:3/375: fsync d6/d10/d22/d2e/f4e 0 2024-04-16T02:35:35.382 INFO:tasks.workunit.client.0.smithi151.stdout:3/63: getdents d1/d6 0 2024-04-16T02:35:35.382 INFO:tasks.workunit.client.1.smithi153.stdout:7/348: truncate d0/d4/db/d2b/d53/f6e 425371 0 2024-04-16T02:35:35.383 INFO:tasks.workunit.client.0.smithi151.stdout:3/64: chown d1/d6/d10/f16 3201296 1 2024-04-16T02:35:35.384 INFO:tasks.workunit.client.0.smithi151.stdout:3/65: chown d1/d6/fe 507720496 1 2024-04-16T02:35:35.384 INFO:tasks.workunit.client.1.smithi153.stdout:7/349: dread - d0/d4/db/d2b/d43/d6d/f63 zero size 2024-04-16T02:35:35.384 INFO:tasks.workunit.client.1.smithi153.stdout:4/444: write d3/d9/db/de/d2c/f74 [882962,11470] 0 2024-04-16T02:35:35.385 INFO:tasks.workunit.client.0.smithi151.stdout:3/66: dread - d1/d6/d10/f15 zero size 2024-04-16T02:35:35.387 INFO:tasks.workunit.client.0.smithi151.stdout:3/67: creat d1/f17 x:0 0 0 2024-04-16T02:35:35.388 INFO:tasks.workunit.client.0.smithi151.stdout:3/68: stat d1/d6/c14 0 2024-04-16T02:35:35.389 INFO:tasks.workunit.client.1.smithi153.stdout:3/376: mknod d6/d10/d3b/d59/d7c/c89 0 2024-04-16T02:35:35.389 INFO:tasks.workunit.client.1.smithi153.stdout:5/388: write d8/f6c [298846,27224] 0 2024-04-16T02:35:35.390 INFO:tasks.workunit.client.1.smithi153.stdout:7/350: stat d0/d4/c3c 0 2024-04-16T02:35:35.392 INFO:tasks.workunit.client.1.smithi153.stdout:4/445: unlink d3/d9/db/de/d2c/d52/f8b 0 2024-04-16T02:35:35.393 INFO:tasks.workunit.client.1.smithi153.stdout:2/376: symlink d9/d2b/d5c/l8f 0 2024-04-16T02:35:35.394 INFO:tasks.workunit.client.1.smithi153.stdout:1/380: mkdir d4/dd/d18/d57/d65/d82 0 2024-04-16T02:35:35.395 INFO:tasks.workunit.client.1.smithi153.stdout:1/381: readlink d4/dd/d18/d29/l35 0 2024-04-16T02:35:35.397 INFO:tasks.workunit.client.1.smithi153.stdout:9/392: sync 2024-04-16T02:35:35.397 INFO:tasks.workunit.client.1.smithi153.stdout:8/348: sync 2024-04-16T02:35:35.397 INFO:tasks.workunit.client.1.smithi153.stdout:6/327: sync 2024-04-16T02:35:35.397 INFO:tasks.workunit.client.1.smithi153.stdout:4/446: symlink d3/d9/d24/d35/l9f 0 2024-04-16T02:35:35.398 INFO:tasks.workunit.client.1.smithi153.stdout:2/377: creat d9/d2b/d1c/f90 x:0 0 0 2024-04-16T02:35:35.398 INFO:tasks.workunit.client.1.smithi153.stdout:6/328: readlink dc/d1a/l2e 0 2024-04-16T02:35:35.400 INFO:tasks.workunit.client.1.smithi153.stdout:4/447: dread d3/f6d [0,4194304] 0 2024-04-16T02:35:35.400 INFO:tasks.workunit.client.1.smithi153.stdout:7/351: symlink d0/d4/db/d2b/d53/d59/d5c/l70 0 2024-04-16T02:35:35.403 INFO:tasks.workunit.client.1.smithi153.stdout:3/377: dwrite d6/d10/d22/d2e/f35 [0,4194304] 0 2024-04-16T02:35:35.404 INFO:tasks.workunit.client.1.smithi153.stdout:9/393: mkdir d4/d6/dc/d2d/d79 0 2024-04-16T02:35:35.404 INFO:tasks.workunit.client.1.smithi153.stdout:4/448: rename d3/d9/db/d17/d3f/d6b to d3/d9/db/d17/d3f/d6b/da0 22 2024-04-16T02:35:35.407 INFO:tasks.workunit.client.1.smithi153.stdout:6/329: dread dc/d10/d62/f6a [0,4194304] 0 2024-04-16T02:35:35.407 INFO:tasks.workunit.client.1.smithi153.stdout:1/382: creat d4/dd/d13/d26/d58/d7a/f83 x:0 0 0 2024-04-16T02:35:35.408 INFO:tasks.workunit.client.1.smithi153.stdout:6/330: truncate dc/d1a/d2d/f43 564407 0 2024-04-16T02:35:35.409 INFO:tasks.workunit.client.1.smithi153.stdout:7/352: mkdir d0/d4/db/d34/d4b/d4d/d71 0 2024-04-16T02:35:35.410 INFO:tasks.workunit.client.1.smithi153.stdout:9/394: symlink d4/d6/d1b/d2a/l7a 0 2024-04-16T02:35:35.412 INFO:tasks.workunit.client.1.smithi153.stdout:9/395: write d4/d6/d1b/f71 [85351,120891] 0 2024-04-16T02:35:35.415 INFO:tasks.workunit.client.1.smithi153.stdout:1/383: rename d4/dd/d77 to d4/dd/d18/d57/d65/d84 0 2024-04-16T02:35:35.419 INFO:tasks.workunit.client.1.smithi153.stdout:5/389: truncate d8/d1f/d20/d39/d6d/f71 138101 0 2024-04-16T02:35:35.421 INFO:tasks.workunit.client.1.smithi153.stdout:8/349: dread d2/de/d34/f40 [0,4194304] 0 2024-04-16T02:35:35.422 INFO:tasks.workunit.client.1.smithi153.stdout:9/396: rename d4/d6/dc/d46/c52 to d4/d33/d3a/d69/c7b 0 2024-04-16T02:35:35.423 INFO:tasks.workunit.client.1.smithi153.stdout:2/378: getdents d9/d24/d27/d2c/d5d 0 2024-04-16T02:35:35.423 INFO:tasks.workunit.client.1.smithi153.stdout:1/384: symlink d4/dd/d13/d7e/d80/l85 0 2024-04-16T02:35:35.425 INFO:tasks.workunit.client.1.smithi153.stdout:8/350: truncate d2/d1c/f26 1321202 0 2024-04-16T02:35:35.426 INFO:tasks.workunit.client.1.smithi153.stdout:2/379: write d9/d24/d27/f6f [2677547,24496] 0 2024-04-16T02:35:35.428 INFO:tasks.workunit.client.1.smithi153.stdout:5/390: dwrite d8/df/d2e/f57 [0,4194304] 0 2024-04-16T02:35:35.428 INFO:tasks.workunit.client.1.smithi153.stdout:7/353: dread d0/f1e [0,4194304] 0 2024-04-16T02:35:35.429 INFO:tasks.workunit.client.1.smithi153.stdout:1/385: truncate d4/dd/d18/d29/f4a 4658119 0 2024-04-16T02:35:35.430 INFO:tasks.workunit.client.1.smithi153.stdout:8/351: write d2/f23 [1541109,27532] 0 2024-04-16T02:35:35.433 INFO:tasks.workunit.client.1.smithi153.stdout:2/380: getdents d9/d24/d27/d2c/d41/d7a 0 2024-04-16T02:35:35.435 INFO:tasks.workunit.client.1.smithi153.stdout:7/354: mkdir d0/d4/db/d2b/d43/d6d/d68/d72 0 2024-04-16T02:35:35.435 INFO:tasks.workunit.client.1.smithi153.stdout:2/381: dread - d9/d24/d27/f68 zero size 2024-04-16T02:35:35.435 INFO:tasks.workunit.client.1.smithi153.stdout:8/352: dread - d2/d1c/d1f/d27/d2d/d47/f6c zero size 2024-04-16T02:35:35.436 INFO:tasks.workunit.client.1.smithi153.stdout:5/391: creat d8/df/d28/d5a/d5e/f76 x:0 0 0 2024-04-16T02:35:35.437 INFO:tasks.workunit.client.1.smithi153.stdout:6/331: dwrite dc/d1a/f36 [0,4194304] 0 2024-04-16T02:35:35.437 INFO:tasks.workunit.client.1.smithi153.stdout:8/353: readlink d2/d13/l19 0 2024-04-16T02:35:35.438 INFO:tasks.workunit.client.1.smithi153.stdout:5/392: creat d8/df/d2e/f77 x:0 0 0 2024-04-16T02:35:35.439 INFO:tasks.workunit.client.1.smithi153.stdout:1/386: link d4/dd/d13/d23/c32 d4/dd/d13/d23/c86 0 2024-04-16T02:35:35.440 INFO:tasks.workunit.client.1.smithi153.stdout:7/355: mknod d0/d4/db/d34/d4b/d4d/c73 0 2024-04-16T02:35:35.441 INFO:tasks.workunit.client.1.smithi153.stdout:8/354: mkdir d2/de/d74 0 2024-04-16T02:35:35.442 INFO:tasks.workunit.client.1.smithi153.stdout:7/356: write d0/d4/db/d2b/d53/f46 [2071762,23867] 0 2024-04-16T02:35:35.443 INFO:tasks.workunit.client.1.smithi153.stdout:2/382: rename d9/d2b/d18/d3f/f48 to d9/d71/d8d/f91 0 2024-04-16T02:35:35.444 INFO:tasks.workunit.client.1.smithi153.stdout:8/355: chown d2/d13/d3a/d5b/l6d 320 1 2024-04-16T02:35:35.444 INFO:tasks.workunit.client.1.smithi153.stdout:6/332: mkdir dc/d6b 0 2024-04-16T02:35:35.446 INFO:tasks.workunit.client.1.smithi153.stdout:1/387: mknod d4/dd/d18/c87 0 2024-04-16T02:35:35.446 INFO:tasks.workunit.client.1.smithi153.stdout:6/333: stat dc/d10/c35 0 2024-04-16T02:35:35.446 INFO:tasks.workunit.client.1.smithi153.stdout:7/357: stat d0/d4/db/d2b/d43/d6d/d68 0 2024-04-16T02:35:35.447 INFO:tasks.workunit.client.1.smithi153.stdout:2/383: chown d9/d24/d27/c74 0 1 2024-04-16T02:35:35.447 INFO:tasks.workunit.client.1.smithi153.stdout:8/356: creat d2/de/d3f/f75 x:0 0 0 2024-04-16T02:35:35.447 INFO:tasks.workunit.client.1.smithi153.stdout:5/393: dread d8/df/d2f/d4e/d51/f5c [0,4194304] 0 2024-04-16T02:35:35.448 INFO:tasks.workunit.client.1.smithi153.stdout:1/388: stat d4/dd/d13/d26/f33 0 2024-04-16T02:35:35.450 INFO:tasks.workunit.client.1.smithi153.stdout:6/334: rename dc/d12/d15/f4e to dc/d10/f6c 0 2024-04-16T02:35:35.450 INFO:tasks.workunit.client.1.smithi153.stdout:2/384: truncate f0 2151410 0 2024-04-16T02:35:35.452 INFO:tasks.workunit.client.1.smithi153.stdout:5/394: symlink d8/df/d2e/l78 0 2024-04-16T02:35:35.453 INFO:tasks.workunit.client.1.smithi153.stdout:1/389: creat d4/dd/d13/d23/d31/f88 x:0 0 0 2024-04-16T02:35:35.454 INFO:tasks.workunit.client.1.smithi153.stdout:6/335: write dc/d12/d15/f5a [1234121,37271] 0 2024-04-16T02:35:35.456 INFO:tasks.workunit.client.1.smithi153.stdout:5/395: creat d8/df/d2e/f79 x:0 0 0 2024-04-16T02:35:35.457 INFO:tasks.workunit.client.1.smithi153.stdout:7/358: dwrite d0/d4/db/f2a [0,4194304] 0 2024-04-16T02:35:35.459 INFO:tasks.workunit.client.1.smithi153.stdout:8/357: dwrite d2/d1c/d1f/f5a [0,4194304] 0 2024-04-16T02:35:35.465 INFO:tasks.workunit.client.1.smithi153.stdout:1/390: mknod d4/dd/d13/d23/d31/c89 0 2024-04-16T02:35:35.466 INFO:tasks.workunit.client.1.smithi153.stdout:6/336: symlink dc/d47/d57/l6d 0 2024-04-16T02:35:35.468 INFO:tasks.workunit.client.1.smithi153.stdout:5/396: symlink d8/l7a 0 2024-04-16T02:35:35.469 INFO:tasks.workunit.client.1.smithi153.stdout:5/397: stat d8/df/d28/d5a/d66 0 2024-04-16T02:35:35.470 INFO:tasks.workunit.client.1.smithi153.stdout:2/385: rename d9/d24/d27/d2c/d41/f53 to d9/d24/d27/f92 0 2024-04-16T02:35:35.471 INFO:tasks.workunit.client.1.smithi153.stdout:5/398: fsync d8/df/d2f/f60 0 2024-04-16T02:35:35.471 INFO:tasks.workunit.client.1.smithi153.stdout:7/359: rename d0/d4/db/d2b to d0/d4/db/d2b/d43/d6d/d68/d6c/d74 22 2024-04-16T02:35:35.474 INFO:tasks.workunit.client.1.smithi153.stdout:2/386: write d9/d24/d27/d2c/d41/f44 [1968874,74380] 0 2024-04-16T02:35:35.474 INFO:tasks.workunit.client.1.smithi153.stdout:1/391: symlink d4/dd/d13/d26/l8a 0 2024-04-16T02:35:35.476 INFO:tasks.workunit.client.1.smithi153.stdout:6/337: dwrite f6 [0,4194304] 0 2024-04-16T02:35:35.477 INFO:tasks.workunit.client.1.smithi153.stdout:7/360: creat d0/d4/db/d34/d4b/f75 x:0 0 0 2024-04-16T02:35:35.479 INFO:tasks.workunit.client.1.smithi153.stdout:8/358: link d2/d1c/d1f/d27/l29 d2/d13/d3a/l76 0 2024-04-16T02:35:35.479 INFO:tasks.workunit.client.1.smithi153.stdout:1/392: write d4/dd/d13/d26/f33 [3029639,70648] 0 2024-04-16T02:35:35.486 INFO:tasks.workunit.client.1.smithi153.stdout:5/399: dwrite d8/d1f/d20/d39/f4c [0,4194304] 0 2024-04-16T02:35:35.489 INFO:tasks.workunit.client.1.smithi153.stdout:0/415: sync 2024-04-16T02:35:35.491 INFO:tasks.workunit.client.1.smithi153.stdout:2/387: dwrite d9/d2b/d5c/f75 [0,4194304] 0 2024-04-16T02:35:35.491 INFO:tasks.workunit.client.1.smithi153.stdout:7/361: rename d0/d4/db/d34/d4b/d4d/d71 to d0/d4/db/d2b/d53/d59/d76 0 2024-04-16T02:35:35.492 INFO:tasks.workunit.client.1.smithi153.stdout:0/416: write dd/d14/d19/d20/f70 [1035583,30430] 0 2024-04-16T02:35:35.495 INFO:tasks.workunit.client.1.smithi153.stdout:6/338: getdents dc/d12/d15 0 2024-04-16T02:35:35.496 INFO:tasks.workunit.client.1.smithi153.stdout:7/362: dread - d0/d4/db/d2b/d43/f6b zero size 2024-04-16T02:35:35.542 INFO:tasks.workunit.client.1.smithi153.stdout:8/359: write d2/d1c/f26 [1787363,60780] 0 2024-04-16T02:35:35.543 INFO:tasks.workunit.client.1.smithi153.stdout:7/363: mknod d0/d4/db/d2b/c77 0 2024-04-16T02:35:35.543 INFO:tasks.workunit.client.1.smithi153.stdout:5/400: truncate d8/df/d2f/d4e/d51/f5c 6781165 0 2024-04-16T02:35:35.544 INFO:tasks.workunit.client.1.smithi153.stdout:8/360: fdatasync d2/f23 0 2024-04-16T02:35:35.546 INFO:tasks.workunit.client.1.smithi153.stdout:6/339: link dc/d10/l27 dc/d1a/l6e 0 2024-04-16T02:35:35.546 INFO:tasks.workunit.client.1.smithi153.stdout:2/388: dwrite d9/d2b/d1c/f90 [0,4194304] 0 2024-04-16T02:35:35.547 INFO:tasks.workunit.client.1.smithi153.stdout:7/364: mknod d0/d4/db/d47/c78 0 2024-04-16T02:35:35.548 INFO:tasks.workunit.client.1.smithi153.stdout:1/393: dread d4/dd/f1d [0,4194304] 0 2024-04-16T02:35:35.548 INFO:tasks.workunit.client.1.smithi153.stdout:0/417: rename dd/d14/d51/d54/d79 to dd/d14/d19/d81 0 2024-04-16T02:35:35.550 INFO:tasks.workunit.client.1.smithi153.stdout:0/418: chown dd/d14/d19/d20/d4a/f78 127757 1 2024-04-16T02:35:35.552 INFO:tasks.workunit.client.1.smithi153.stdout:8/361: unlink d2/de/d34/f68 0 2024-04-16T02:35:35.553 INFO:tasks.workunit.client.1.smithi153.stdout:8/362: chown d2/d1c/d1f/d27/c2b 3006 1 2024-04-16T02:35:35.553 INFO:tasks.workunit.client.1.smithi153.stdout:8/363: fsync d2/d1c/d1f/d27/f2a 0 2024-04-16T02:35:35.556 INFO:tasks.workunit.client.1.smithi153.stdout:7/365: stat d0/d4/l9 0 2024-04-16T02:35:35.559 INFO:tasks.workunit.client.1.smithi153.stdout:0/419: creat dd/d26/d44/f82 x:0 0 0 2024-04-16T02:35:35.560 INFO:tasks.workunit.client.1.smithi153.stdout:5/401: creat d8/df/d28/d5a/d66/f7b x:0 0 0 2024-04-16T02:35:35.561 INFO:tasks.workunit.client.1.smithi153.stdout:8/364: truncate d2/f14 2798125 0 2024-04-16T02:35:35.562 INFO:tasks.workunit.client.1.smithi153.stdout:7/366: creat d0/d4/db/d34/f79 x:0 0 0 2024-04-16T02:35:35.564 INFO:tasks.workunit.client.1.smithi153.stdout:5/402: symlink d8/d1f/d20/d39/d6d/l7c 0 2024-04-16T02:35:35.564 INFO:tasks.workunit.client.1.smithi153.stdout:7/367: dread - d0/d4/db/d34/f79 zero size 2024-04-16T02:35:35.564 INFO:tasks.workunit.client.1.smithi153.stdout:8/365: mkdir d2/d1c/d1f/d27/d2d/d47/d77 0 2024-04-16T02:35:35.566 INFO:tasks.workunit.client.1.smithi153.stdout:2/389: rename d9/d24/d34/d60/c8a to d9/d24/d27/c93 0 2024-04-16T02:35:35.567 INFO:tasks.workunit.client.1.smithi153.stdout:6/340: dwrite dc/d1a/f58 [0,4194304] 0 2024-04-16T02:35:35.568 INFO:tasks.workunit.client.1.smithi153.stdout:1/394: getdents d4/dd/d13 0 2024-04-16T02:35:35.569 INFO:tasks.workunit.client.1.smithi153.stdout:2/390: write d9/d2b/d18/d3f/d40/f8e [603234,98696] 0 2024-04-16T02:35:35.570 INFO:tasks.workunit.client.1.smithi153.stdout:0/420: creat dd/d28/f83 x:0 0 0 2024-04-16T02:35:35.570 INFO:tasks.workunit.client.1.smithi153.stdout:7/368: write d0/d4/db/f64 [1346917,129964] 0 2024-04-16T02:35:35.572 INFO:tasks.workunit.client.1.smithi153.stdout:5/403: symlink d8/df/d28/d64/l7d 0 2024-04-16T02:35:35.572 INFO:tasks.workunit.client.1.smithi153.stdout:8/366: creat d2/d13/d3a/d66/f78 x:0 0 0 2024-04-16T02:35:35.573 INFO:tasks.workunit.client.1.smithi153.stdout:3/378: sync 2024-04-16T02:35:35.574 INFO:tasks.workunit.client.1.smithi153.stdout:5/404: write d8/d1f/f73 [576989,69105] 0 2024-04-16T02:35:35.574 INFO:tasks.workunit.client.1.smithi153.stdout:1/395: creat d4/dd/d13/d23/d31/f8b x:0 0 0 2024-04-16T02:35:35.574 INFO:tasks.workunit.client.1.smithi153.stdout:6/341: creat dc/d6b/f6f x:0 0 0 2024-04-16T02:35:35.575 INFO:tasks.workunit.client.1.smithi153.stdout:2/391: symlink d9/d71/l94 0 2024-04-16T02:35:35.580 INFO:tasks.workunit.client.1.smithi153.stdout:0/421: creat dd/d14/d51/d54/f84 x:0 0 0 2024-04-16T02:35:35.582 INFO:tasks.workunit.client.1.smithi153.stdout:6/342: creat dc/d47/d57/f70 x:0 0 0 2024-04-16T02:35:35.582 INFO:tasks.workunit.client.1.smithi153.stdout:5/405: read d8/f6c [20572,2500] 0 2024-04-16T02:35:35.584 INFO:tasks.workunit.client.1.smithi153.stdout:7/369: dwrite d0/d13/f4f [4194304,4194304] 0 2024-04-16T02:35:35.587 INFO:tasks.workunit.client.1.smithi153.stdout:0/422: fsync dd/d14/f3b 0 2024-04-16T02:35:35.588 INFO:tasks.workunit.client.1.smithi153.stdout:5/406: readlink d8/la 0 2024-04-16T02:35:35.589 INFO:tasks.workunit.client.1.smithi153.stdout:6/343: unlink dc/d47/d57/l69 0 2024-04-16T02:35:35.591 INFO:tasks.workunit.client.1.smithi153.stdout:3/379: write d6/d10/d22/d2c/f3d [3926277,78815] 0 2024-04-16T02:35:35.596 INFO:tasks.workunit.client.1.smithi153.stdout:2/392: write d9/d2b/d1c/d20/d45/f54 [784987,101541] 0 2024-04-16T02:35:35.597 INFO:tasks.workunit.client.1.smithi153.stdout:0/423: creat dd/d32/f85 x:0 0 0 2024-04-16T02:35:35.598 INFO:tasks.workunit.client.1.smithi153.stdout:0/424: dread - dd/d26/d44/f82 zero size 2024-04-16T02:35:35.598 INFO:tasks.workunit.client.1.smithi153.stdout:1/396: dread d4/dd/d13/f14 [0,4194304] 0 2024-04-16T02:35:35.598 INFO:tasks.workunit.client.1.smithi153.stdout:6/344: unlink dc/l18 0 2024-04-16T02:35:35.601 INFO:tasks.workunit.client.1.smithi153.stdout:3/380: chown d6/d10/d22/d2c/c65 38111155 1 2024-04-16T02:35:35.603 INFO:tasks.workunit.client.1.smithi153.stdout:3/381: write d6/d10/d22/d2c/f3d [1672259,50207] 0 2024-04-16T02:35:35.603 INFO:tasks.workunit.client.1.smithi153.stdout:2/393: mknod d9/d24/d27/d2c/d5d/c95 0 2024-04-16T02:35:35.604 INFO:tasks.workunit.client.1.smithi153.stdout:7/370: truncate d0/d4/db/f2a 1367089 0 2024-04-16T02:35:35.606 INFO:tasks.workunit.client.1.smithi153.stdout:1/397: creat d4/dd/d18/d57/d65/d84/f8c x:0 0 0 2024-04-16T02:35:35.606 INFO:tasks.workunit.client.1.smithi153.stdout:4/449: sync 2024-04-16T02:35:35.608 INFO:tasks.workunit.client.1.smithi153.stdout:5/407: creat d8/f7e x:0 0 0 2024-04-16T02:35:35.610 INFO:tasks.workunit.client.1.smithi153.stdout:2/394: dwrite d9/d2b/d18/d3f/d40/f70 [0,4194304] 0 2024-04-16T02:35:35.611 INFO:tasks.workunit.client.1.smithi153.stdout:8/367: dread d2/de/f5f [0,4194304] 0 2024-04-16T02:35:35.612 INFO:tasks.workunit.client.1.smithi153.stdout:2/395: write d9/d2b/f16 [5140590,22319] 0 2024-04-16T02:35:35.613 INFO:tasks.workunit.client.1.smithi153.stdout:1/398: mkdir d4/dd/d13/d8d 0 2024-04-16T02:35:35.617 INFO:tasks.workunit.client.1.smithi153.stdout:4/450: symlink d3/d9/d79/la1 0 2024-04-16T02:35:35.623 INFO:tasks.workunit.client.1.smithi153.stdout:0/425: creat dd/d14/f86 x:0 0 0 2024-04-16T02:35:35.624 INFO:tasks.workunit.client.1.smithi153.stdout:6/345: link dc/d10/l42 dc/d1a/l71 0 2024-04-16T02:35:35.625 INFO:tasks.workunit.client.1.smithi153.stdout:8/368: creat d2/d1c/d1f/d27/d2d/d47/d60/f79 x:0 0 0 2024-04-16T02:35:35.628 INFO:tasks.workunit.client.1.smithi153.stdout:2/396: dwrite d9/d24/d34/d36/f3b [0,4194304] 0 2024-04-16T02:35:35.632 INFO:tasks.workunit.client.1.smithi153.stdout:7/371: dwrite d0/d4/db/d34/d4b/d4d/f66 [0,4194304] 0 2024-04-16T02:35:35.639 INFO:tasks.workunit.client.1.smithi153.stdout:7/372: dwrite d0/d4/db/d2b/f3e [0,4194304] 0 2024-04-16T02:35:35.640 INFO:tasks.workunit.client.1.smithi153.stdout:7/373: chown d0/d4/db/d34/l3a 43757491 1 2024-04-16T02:35:35.656 INFO:tasks.workunit.client.1.smithi153.stdout:9/397: sync 2024-04-16T02:35:35.658 INFO:tasks.workunit.client.1.smithi153.stdout:7/374: dread d0/d4/db/d2b/d53/f27 [0,4194304] 0 2024-04-16T02:35:35.661 INFO:tasks.workunit.client.1.smithi153.stdout:7/375: dread d0/d4/db/d34/d4b/d4d/f66 [0,4194304] 0 2024-04-16T02:35:35.696 INFO:tasks.workunit.client.0.smithi151.stdout:7/5: sync 2024-04-16T02:35:35.696 INFO:tasks.workunit.client.1.smithi153.stdout:4/451: write d3/d9/db/f65 [695219,58271] 0 2024-04-16T02:35:35.698 INFO:tasks.workunit.client.1.smithi153.stdout:6/346: mkdir dc/d6b/d72 0 2024-04-16T02:35:35.701 INFO:tasks.workunit.client.0.smithi151.stdout:7/6: mkdir d0 0 2024-04-16T02:35:35.701 INFO:tasks.workunit.client.0.smithi151.stdout:7/7: dwrite - no filename 2024-04-16T02:35:35.701 INFO:tasks.workunit.client.0.smithi151.stdout:7/8: dwrite - no filename 2024-04-16T02:35:35.701 INFO:tasks.workunit.client.0.smithi151.stdout:7/9: rename d0 to d0/d1 22 2024-04-16T02:35:35.701 INFO:tasks.workunit.client.0.smithi151.stdout:7/10: unlink - no file 2024-04-16T02:35:35.702 INFO:tasks.workunit.client.0.smithi151.stdout:7/11: link - no file 2024-04-16T02:35:35.702 INFO:tasks.workunit.client.0.smithi151.stdout:7/12: chown d0 2674 1 2024-04-16T02:35:35.702 INFO:tasks.workunit.client.0.smithi151.stdout:7/13: truncate - no filename 2024-04-16T02:35:35.702 INFO:tasks.workunit.client.0.smithi151.stdout:7/14: dread - no filename 2024-04-16T02:35:35.702 INFO:tasks.workunit.client.0.smithi151.stdout:7/15: rename d0 to d0/d2 22 2024-04-16T02:35:35.704 INFO:tasks.workunit.client.1.smithi153.stdout:2/397: write d9/d2b/d18/f29 [3896650,45169] 0 2024-04-16T02:35:35.705 INFO:tasks.workunit.client.0.smithi151.stdout:7/16: creat d0/f3 x:0 0 0 2024-04-16T02:35:35.705 INFO:tasks.workunit.client.0.smithi151.stdout:7/17: dread - d0/f3 zero size 2024-04-16T02:35:35.706 INFO:tasks.workunit.client.1.smithi153.stdout:8/369: dwrite d2/f9 [4194304,4194304] 0 2024-04-16T02:35:35.706 INFO:tasks.workunit.client.1.smithi153.stdout:8/370: readlink d2/de/l38 0 2024-04-16T02:35:35.707 INFO:tasks.workunit.client.0.smithi151.stdout:7/18: creat d0/f4 x:0 0 0 2024-04-16T02:35:35.708 INFO:tasks.workunit.client.0.smithi151.stdout:7/19: chown d0/f4 0 1 2024-04-16T02:35:35.708 INFO:tasks.workunit.client.0.smithi151.stdout:7/20: readlink - no filename 2024-04-16T02:35:35.708 INFO:tasks.workunit.client.1.smithi153.stdout:9/398: unlink d4/d6/d1b/f3e 0 2024-04-16T02:35:35.709 INFO:tasks.workunit.client.1.smithi153.stdout:3/382: truncate d6/d10/f18 3806182 0 2024-04-16T02:35:35.709 INFO:tasks.workunit.client.1.smithi153.stdout:9/399: chown d4/d33/f38 79237 1 2024-04-16T02:35:35.711 INFO:tasks.workunit.client.1.smithi153.stdout:7/376: write d0/f1e [559322,11113] 0 2024-04-16T02:35:35.713 INFO:tasks.workunit.client.1.smithi153.stdout:4/452: creat d3/d9/db/de/d2c/d52/fa2 x:0 0 0 2024-04-16T02:35:35.714 INFO:tasks.workunit.client.1.smithi153.stdout:4/453: stat d3/d9/db/de/d2c/f74 0 2024-04-16T02:35:35.714 INFO:tasks.workunit.client.1.smithi153.stdout:0/426: mkdir dd/d14/d19/d77/d87 0 2024-04-16T02:35:35.716 INFO:tasks.workunit.client.0.smithi151.stdout:7/21: dwrite d0/f3 [0,4194304] 0 2024-04-16T02:35:35.716 INFO:tasks.workunit.client.0.smithi151.stdout:7/22: dread - d0/f4 zero size 2024-04-16T02:35:35.719 INFO:tasks.workunit.client.1.smithi153.stdout:2/398: read d9/d2b/f17 [29771,91906] 0 2024-04-16T02:35:35.720 INFO:tasks.workunit.client.1.smithi153.stdout:2/399: write d9/d24/d27/d2c/d41/f4c [327081,15692] 0 2024-04-16T02:35:35.723 INFO:tasks.workunit.client.1.smithi153.stdout:3/383: chown d6/c17 435796871 1 2024-04-16T02:35:35.723 INFO:tasks.workunit.client.1.smithi153.stdout:3/384: dread - d6/d10/d5b/f75 zero size 2024-04-16T02:35:35.724 INFO:tasks.workunit.client.1.smithi153.stdout:7/377: mkdir d0/d4/db/d2b/d53/d59/d76/d7a 0 2024-04-16T02:35:35.724 INFO:tasks.workunit.client.1.smithi153.stdout:5/408: link d8/l1a d8/d1f/l7f 0 2024-04-16T02:35:35.726 INFO:tasks.workunit.client.1.smithi153.stdout:0/427: rename dd/d32/f35 to dd/d14/d19/d20/f88 0 2024-04-16T02:35:35.726 INFO:tasks.workunit.client.1.smithi153.stdout:6/347: truncate dc/d1a/d2d/f44 2044151 0 2024-04-16T02:35:35.727 INFO:tasks.workunit.client.1.smithi153.stdout:1/399: mknod d4/dd/d13/d23/d31/c8e 0 2024-04-16T02:35:35.728 INFO:tasks.workunit.client.1.smithi153.stdout:2/400: creat d9/d2b/d1c/d20/d45/f96 x:0 0 0 2024-04-16T02:35:35.729 INFO:tasks.workunit.client.1.smithi153.stdout:5/409: dread d8/df/d23/f32 [0,4194304] 0 2024-04-16T02:35:35.729 INFO:tasks.workunit.client.0.smithi151.stdout:7/23: rmdir d0 39 2024-04-16T02:35:35.730 INFO:tasks.workunit.client.1.smithi153.stdout:2/401: stat d9/f8b 0 2024-04-16T02:35:35.731 INFO:tasks.workunit.client.0.smithi151.stdout:7/24: write d0/f3 [87132,37147] 0 2024-04-16T02:35:35.734 INFO:tasks.workunit.client.1.smithi153.stdout:6/348: creat dc/d6b/f73 x:0 0 0 2024-04-16T02:35:35.734 INFO:tasks.workunit.client.1.smithi153.stdout:8/371: creat d2/d1c/d1f/d27/f7a x:0 0 0 2024-04-16T02:35:35.735 INFO:tasks.workunit.client.0.smithi151.stdout:7/25: link d0/f3 d0/f5 0 2024-04-16T02:35:35.736 INFO:tasks.workunit.client.1.smithi153.stdout:0/428: dwrite dd/d14/f86 [0,4194304] 0 2024-04-16T02:35:35.736 INFO:tasks.workunit.client.1.smithi153.stdout:9/400: creat d4/d33/d3a/f7c x:0 0 0 2024-04-16T02:35:35.736 INFO:tasks.workunit.client.1.smithi153.stdout:1/400: dread d4/dd/d13/d23/f27 [0,4194304] 0 2024-04-16T02:35:35.738 INFO:tasks.workunit.client.1.smithi153.stdout:9/401: stat d4/d6/dc/l1c 0 2024-04-16T02:35:35.738 INFO:tasks.workunit.client.1.smithi153.stdout:4/454: creat d3/d9/db/de/fa3 x:0 0 0 2024-04-16T02:35:35.738 INFO:tasks.workunit.client.1.smithi153.stdout:2/402: mknod d9/d24/c97 0 2024-04-16T02:35:35.739 INFO:tasks.workunit.client.0.smithi151.stdout:7/26: link d0/f5 d0/f6 0 2024-04-16T02:35:35.740 INFO:tasks.workunit.client.0.smithi151.stdout:7/27: chown d0/f6 1313644 1 2024-04-16T02:35:35.741 INFO:tasks.workunit.client.1.smithi153.stdout:5/410: creat d8/d1f/d20/d39/f80 x:0 0 0 2024-04-16T02:35:35.741 INFO:tasks.workunit.client.1.smithi153.stdout:0/429: truncate dd/d14/d51/d54/f84 437041 0 2024-04-16T02:35:35.741 INFO:tasks.workunit.client.1.smithi153.stdout:2/403: dread - d9/d24/d34/f7d zero size 2024-04-16T02:35:35.742 INFO:tasks.workunit.client.1.smithi153.stdout:4/455: chown d3/d9/db/de/d2c/d52/c86 678590411 1 2024-04-16T02:35:35.742 INFO:tasks.workunit.client.1.smithi153.stdout:2/404: chown d9/d2b/d1c/f90 30996 1 2024-04-16T02:35:35.743 INFO:tasks.workunit.client.1.smithi153.stdout:4/456: fsync d3/d9/db/de/d2c/f8a 0 2024-04-16T02:35:35.743 INFO:tasks.workunit.client.1.smithi153.stdout:7/378: dread d0/d4/db/d2b/f25 [0,4194304] 0 2024-04-16T02:35:35.744 INFO:tasks.workunit.client.1.smithi153.stdout:2/405: truncate d9/d2b/d1c/f28 4951733 0 2024-04-16T02:35:35.745 INFO:tasks.workunit.client.1.smithi153.stdout:6/349: creat dc/d12/d15/f74 x:0 0 0 2024-04-16T02:35:35.746 INFO:tasks.workunit.client.1.smithi153.stdout:6/350: stat l0 0 2024-04-16T02:35:35.746 INFO:tasks.workunit.client.1.smithi153.stdout:2/406: dread - d9/d2b/d1c/d20/f79 zero size 2024-04-16T02:35:35.746 INFO:tasks.workunit.client.1.smithi153.stdout:7/379: write d0/d4/db/d2b/d53/f46 [5230586,80651] 0 2024-04-16T02:35:35.746 INFO:tasks.workunit.client.1.smithi153.stdout:8/372: creat d2/d1c/d1f/d27/d2d/d47/d60/f7b x:0 0 0 2024-04-16T02:35:35.747 INFO:tasks.workunit.client.0.smithi151.stdout:7/28: getdents d0 0 2024-04-16T02:35:35.747 INFO:tasks.workunit.client.1.smithi153.stdout:8/373: chown d2 974264572 1 2024-04-16T02:35:35.748 INFO:tasks.workunit.client.1.smithi153.stdout:1/401: symlink d4/dd/d18/d57/l8f 0 2024-04-16T02:35:35.748 INFO:tasks.workunit.client.1.smithi153.stdout:7/380: readlink d0/d4/l9 0 2024-04-16T02:35:35.749 INFO:tasks.workunit.client.0.smithi151.stdout:7/29: creat d0/f7 x:0 0 0 2024-04-16T02:35:35.749 INFO:tasks.workunit.client.1.smithi153.stdout:2/407: write d9/d24/d27/f68 [638293,45325] 0 2024-04-16T02:35:35.749 INFO:tasks.workunit.client.0.smithi151.stdout:7/30: fdatasync d0/f7 0 2024-04-16T02:35:35.750 INFO:tasks.workunit.client.0.smithi151.stdout:7/31: fsync d0/f4 0 2024-04-16T02:35:35.751 INFO:tasks.workunit.client.1.smithi153.stdout:0/430: rmdir dd/d14/d19/d77 39 2024-04-16T02:35:35.751 INFO:tasks.workunit.client.0.smithi151.stdout:7/32: symlink d0/l8 0 2024-04-16T02:35:35.752 INFO:tasks.workunit.client.1.smithi153.stdout:1/402: truncate d4/dd/d13/d23/d5f/f7f 821265 0 2024-04-16T02:35:35.753 INFO:tasks.workunit.client.1.smithi153.stdout:0/431: chown dd/d32/d57/c5f 58322 1 2024-04-16T02:35:35.754 INFO:tasks.workunit.client.1.smithi153.stdout:1/403: dread - d4/dd/d18/d29/d71/f7b zero size 2024-04-16T02:35:35.754 INFO:tasks.workunit.client.1.smithi153.stdout:6/351: dread dc/d47/f61 [0,4194304] 0 2024-04-16T02:35:35.754 INFO:tasks.workunit.client.1.smithi153.stdout:1/404: chown d4/dd/d13 1423910 1 2024-04-16T02:35:35.755 INFO:tasks.workunit.client.0.smithi151.stdout:7/33: rename d0/f4 to d0/f9 0 2024-04-16T02:35:35.755 INFO:tasks.workunit.client.0.smithi151.stdout:7/34: chown d0 11269970 1 2024-04-16T02:35:35.755 INFO:tasks.workunit.client.1.smithi153.stdout:8/374: creat d2/d1c/d1f/f7c x:0 0 0 2024-04-16T02:35:35.756 INFO:tasks.workunit.client.1.smithi153.stdout:9/402: getdents d4/d6/dc/d24/d5d/d66 0 2024-04-16T02:35:35.758 INFO:tasks.workunit.client.1.smithi153.stdout:3/385: getdents d6/d10/d3b 0 2024-04-16T02:35:35.761 INFO:tasks.workunit.client.1.smithi153.stdout:7/381: creat d0/d4/db/d2b/d53/d59/d76/f7b x:0 0 0 2024-04-16T02:35:35.763 INFO:tasks.workunit.client.1.smithi153.stdout:0/432: creat dd/d32/d57/f89 x:0 0 0 2024-04-16T02:35:35.763 INFO:tasks.workunit.client.1.smithi153.stdout:2/408: dwrite d9/d71/d8d/f91 [0,4194304] 0 2024-04-16T02:35:35.763 INFO:tasks.workunit.client.1.smithi153.stdout:6/352: unlink dc/c1f 0 2024-04-16T02:35:35.765 INFO:tasks.workunit.client.1.smithi153.stdout:6/353: chown dc/d12/d15/c5d 77124835 1 2024-04-16T02:35:35.765 INFO:tasks.workunit.client.0.smithi151.stdout:7/35: dwrite d0/f3 [0,4194304] 0 2024-04-16T02:35:35.766 INFO:tasks.workunit.client.0.smithi151.stdout:7/36: chown d0/f9 65 1 2024-04-16T02:35:35.767 INFO:tasks.workunit.client.0.smithi151.stdout:7/37: chown d0/l8 108437 1 2024-04-16T02:35:35.769 INFO:tasks.workunit.client.0.smithi151.stdout:7/38: symlink d0/la 0 2024-04-16T02:35:35.770 INFO:tasks.workunit.client.0.smithi151.stdout:7/39: dread - d0/f7 zero size 2024-04-16T02:35:35.770 INFO:tasks.workunit.client.1.smithi153.stdout:6/354: read dc/d12/d15/f1e [392511,90912] 0 2024-04-16T02:35:35.770 INFO:tasks.workunit.client.1.smithi153.stdout:3/386: mkdir d6/d10/d22/d2e/d8a 0 2024-04-16T02:35:35.770 INFO:tasks.workunit.client.1.smithi153.stdout:7/382: fdatasync d0/d4/db/d34/d4b/d4d/f66 0 2024-04-16T02:35:35.770 INFO:tasks.workunit.client.1.smithi153.stdout:4/457: truncate d3/d9/f1a 1951992 0 2024-04-16T02:35:35.771 INFO:tasks.workunit.client.0.smithi151.stdout:7/40: creat d0/fb x:0 0 0 2024-04-16T02:35:35.771 INFO:tasks.workunit.client.1.smithi153.stdout:6/355: read dc/d10/f34 [676113,108278] 0 2024-04-16T02:35:35.772 INFO:tasks.workunit.client.0.smithi151.stdout:7/41: chown d0/f5 75215023 1 2024-04-16T02:35:35.772 INFO:tasks.workunit.client.0.smithi151.stdout:7/42: write d0/f7 [242371,66679] 0 2024-04-16T02:35:35.773 INFO:tasks.workunit.client.1.smithi153.stdout:3/387: truncate d6/d10/d5b/f82 613627 0 2024-04-16T02:35:35.773 INFO:tasks.workunit.client.1.smithi153.stdout:1/405: unlink d4/dd/d13/f54 0 2024-04-16T02:35:35.773 INFO:tasks.workunit.client.0.smithi151.stdout:7/43: read - d0/fb zero size 2024-04-16T02:35:35.774 INFO:tasks.workunit.client.1.smithi153.stdout:7/383: write d0/d4/db/f39 [1190193,130240] 0 2024-04-16T02:35:35.774 INFO:tasks.workunit.client.0.smithi151.stdout:7/44: write d0/fb [343086,1059] 0 2024-04-16T02:35:35.775 INFO:tasks.workunit.client.1.smithi153.stdout:2/409: rename d9/d24/d27/c74 to d9/d2b/d18/d3f/d40/c98 0 2024-04-16T02:35:35.776 INFO:tasks.workunit.client.1.smithi153.stdout:7/384: fdatasync d0/d4/db/f57 0 2024-04-16T02:35:35.777 INFO:tasks.workunit.client.0.smithi151.stdout:7/45: creat d0/fc x:0 0 0 2024-04-16T02:35:35.778 INFO:tasks.workunit.client.1.smithi153.stdout:7/385: dread - d0/d4/db/d34/d4b/f75 zero size 2024-04-16T02:35:35.779 INFO:tasks.workunit.client.0.smithi151.stdout:7/46: truncate d0/fc 49666 0 2024-04-16T02:35:35.779 INFO:tasks.workunit.client.0.smithi151.stdout:7/47: rename d0 to d0/dd 22 2024-04-16T02:35:35.780 INFO:tasks.workunit.client.0.smithi151.stdout:7/48: truncate d0/fb 1175807 0 2024-04-16T02:35:35.782 INFO:tasks.workunit.client.1.smithi153.stdout:0/433: symlink dd/d14/d51/d7f/l8a 0 2024-04-16T02:35:35.782 INFO:tasks.workunit.client.1.smithi153.stdout:3/388: symlink d6/d10/d22/d2c/l8b 0 2024-04-16T02:35:35.783 INFO:tasks.workunit.client.1.smithi153.stdout:1/406: stat d4/dd/d13/d26/f78 0 2024-04-16T02:35:35.784 INFO:tasks.workunit.client.1.smithi153.stdout:2/410: mkdir d9/d24/d34/d99 0 2024-04-16T02:35:35.784 INFO:tasks.workunit.client.1.smithi153.stdout:3/389: read d6/d10/d22/d43/f69 [717968,102355] 0 2024-04-16T02:35:35.786 INFO:tasks.workunit.client.1.smithi153.stdout:0/434: read dd/f22 [2243646,39828] 0 2024-04-16T02:35:35.787 INFO:tasks.workunit.client.1.smithi153.stdout:7/386: dwrite d0/d4/db/f39 [0,4194304] 0 2024-04-16T02:35:35.788 INFO:tasks.workunit.client.0.smithi151.stdout:7/49: dwrite d0/f9 [0,4194304] 0 2024-04-16T02:35:35.789 INFO:tasks.workunit.client.1.smithi153.stdout:2/411: write f1 [4818678,75246] 0 2024-04-16T02:35:35.791 INFO:tasks.workunit.client.1.smithi153.stdout:1/407: creat d4/dd/d18/f90 x:0 0 0 2024-04-16T02:35:35.794 INFO:tasks.workunit.client.0.smithi151.stdout:7/50: dread d0/f6 [0,4194304] 0 2024-04-16T02:35:35.797 INFO:tasks.workunit.client.0.smithi151.stdout:7/51: creat d0/fe x:0 0 0 2024-04-16T02:35:35.798 INFO:tasks.workunit.client.0.smithi151.stdout:7/52: truncate d0/fe 787802 0 2024-04-16T02:35:35.800 INFO:tasks.workunit.client.0.smithi151.stdout:7/53: truncate d0/fb 1550864 0 2024-04-16T02:35:35.802 INFO:tasks.workunit.client.1.smithi153.stdout:6/356: rename dc/d6b/d72 to dc/d12/d75 0 2024-04-16T02:35:35.802 INFO:tasks.workunit.client.1.smithi153.stdout:6/357: read - dc/d1a/f3a zero size 2024-04-16T02:35:35.804 INFO:tasks.workunit.client.0.smithi151.stdout:7/54: dread d0/fb [0,4194304] 0 2024-04-16T02:35:35.805 INFO:tasks.workunit.client.1.smithi153.stdout:8/375: truncate d2/d13/d2c/f32 1131596 0 2024-04-16T02:35:35.805 INFO:tasks.workunit.client.1.smithi153.stdout:3/390: mkdir d6/d10/d22/d2e/d8a/d8c 0 2024-04-16T02:35:35.806 INFO:tasks.workunit.client.1.smithi153.stdout:9/403: write d4/d6/dc/d24/f42 [1930800,114239] 0 2024-04-16T02:35:35.808 INFO:tasks.workunit.client.1.smithi153.stdout:2/412: mkdir d9/d24/d27/d9a 0 2024-04-16T02:35:35.809 INFO:tasks.workunit.client.1.smithi153.stdout:8/376: symlink d2/d13/d3a/d5b/l7d 0 2024-04-16T02:35:35.810 INFO:tasks.workunit.client.1.smithi153.stdout:3/391: unlink d6/d10/d22/c5c 0 2024-04-16T02:35:35.811 INFO:tasks.workunit.client.1.smithi153.stdout:0/435: creat dd/d14/f8b x:0 0 0 2024-04-16T02:35:35.812 INFO:tasks.workunit.client.1.smithi153.stdout:7/387: rename d0/d13/c21 to d0/d4/db/d2b/d53/d59/c7c 0 2024-04-16T02:35:35.815 INFO:tasks.workunit.client.1.smithi153.stdout:0/436: readlink dd/d14/d19/d20/l6b 0 2024-04-16T02:35:35.815 INFO:tasks.workunit.client.1.smithi153.stdout:8/377: chown d2/d13/l1a 774536242 1 2024-04-16T02:35:35.816 INFO:tasks.workunit.client.1.smithi153.stdout:4/458: dread d3/d9/f14 [0,4194304] 0 2024-04-16T02:35:35.817 INFO:tasks.workunit.client.1.smithi153.stdout:0/437: dread - dd/d26/d44/f82 zero size 2024-04-16T02:35:35.817 INFO:tasks.workunit.client.1.smithi153.stdout:7/388: dread d0/d4/db/ff [0,4194304] 0 2024-04-16T02:35:35.818 INFO:tasks.workunit.client.1.smithi153.stdout:8/378: fdatasync d2/d1c/d1f/d27/d2d/d47/d60/f7b 0 2024-04-16T02:35:35.819 INFO:tasks.workunit.client.1.smithi153.stdout:7/389: write d0/d4/db/f57 [1358571,116598] 0 2024-04-16T02:35:35.819 INFO:tasks.workunit.client.1.smithi153.stdout:4/459: creat d3/d9/d24/fa4 x:0 0 0 2024-04-16T02:35:35.821 INFO:tasks.workunit.client.1.smithi153.stdout:0/438: symlink dd/d14/d51/d54/d65/d68/l8c 0 2024-04-16T02:35:35.821 INFO:tasks.workunit.client.1.smithi153.stdout:7/390: write d0/d4/db/d2b/d43/d6d/f6f [1045665,40920] 0 2024-04-16T02:35:35.822 INFO:tasks.workunit.client.1.smithi153.stdout:0/439: dread - dd/d14/f4b zero size 2024-04-16T02:35:35.823 INFO:tasks.workunit.client.1.smithi153.stdout:7/391: write d0/d13/f4a [137605,29888] 0 2024-04-16T02:35:35.824 INFO:tasks.workunit.client.1.smithi153.stdout:8/379: fdatasync d2/d13/f1d 0 2024-04-16T02:35:35.824 INFO:tasks.workunit.client.1.smithi153.stdout:7/392: read d0/d4/db/f2a [841734,46595] 0 2024-04-16T02:35:35.825 INFO:tasks.workunit.client.1.smithi153.stdout:4/460: mkdir d3/da5 0 2024-04-16T02:35:35.827 INFO:tasks.workunit.client.1.smithi153.stdout:7/393: write d0/d4/db/d2b/d53/d59/d76/f7b [649674,24142] 0 2024-04-16T02:35:35.827 INFO:tasks.workunit.client.1.smithi153.stdout:4/461: fdatasync d3/d9/db/de/d2c/f8a 0 2024-04-16T02:35:35.828 INFO:tasks.workunit.client.1.smithi153.stdout:0/440: getdents dd/d32/d80 0 2024-04-16T02:35:35.829 INFO:tasks.workunit.client.1.smithi153.stdout:4/462: dread - d3/d9/db/de/f71 zero size 2024-04-16T02:35:35.833 INFO:tasks.workunit.client.1.smithi153.stdout:4/463: truncate d3/d9/db/de/d2c/d52/fa2 275812 0 2024-04-16T02:35:35.835 INFO:tasks.workunit.client.1.smithi153.stdout:9/404: dread d4/d6/f22 [0,4194304] 0 2024-04-16T02:35:35.836 INFO:tasks.workunit.client.1.smithi153.stdout:4/464: chown d3/d9/db/d17/d28/l39 186 1 2024-04-16T02:35:35.837 INFO:tasks.workunit.client.1.smithi153.stdout:0/441: dwrite dd/d14/d19/d1c/f4d [0,4194304] 0 2024-04-16T02:35:35.839 INFO:tasks.workunit.client.1.smithi153.stdout:2/413: dread d9/d24/d27/f1d [4194304,4194304] 0 2024-04-16T02:35:35.844 INFO:tasks.workunit.client.1.smithi153.stdout:8/380: truncate d2/d1c/d1f/d27/d2d/f59 376947 0 2024-04-16T02:35:35.846 INFO:tasks.workunit.client.1.smithi153.stdout:4/465: getdents d3/d9/db/de/d2c/d98 0 2024-04-16T02:35:35.847 INFO:tasks.workunit.client.1.smithi153.stdout:6/358: dwrite dc/d1a/d2d/f44 [0,4194304] 0 2024-04-16T02:35:35.848 INFO:tasks.workunit.client.1.smithi153.stdout:9/405: dwrite d4/d33/d3a/d69/f6c [0,4194304] 0 2024-04-16T02:35:35.848 INFO:tasks.workunit.client.1.smithi153.stdout:7/394: truncate d0/d4/db/d2b/d43/d6d/f6f 721337 0 2024-04-16T02:35:35.851 INFO:tasks.workunit.client.1.smithi153.stdout:2/414: creat d9/d2b/d1c/d20/d88/f9b x:0 0 0 2024-04-16T02:35:35.853 INFO:tasks.workunit.client.1.smithi153.stdout:2/415: write d9/d24/d27/f68 [1082492,43961] 0 2024-04-16T02:35:35.853 INFO:tasks.workunit.client.1.smithi153.stdout:4/466: creat d3/d9/d79/fa6 x:0 0 0 2024-04-16T02:35:35.854 INFO:tasks.workunit.client.1.smithi153.stdout:4/467: chown d3/d9/l19 413042670 1 2024-04-16T02:35:35.854 INFO:tasks.workunit.client.1.smithi153.stdout:6/359: rename dc/d47/l56 to dc/d6b/l76 0 2024-04-16T02:35:35.856 INFO:tasks.workunit.client.1.smithi153.stdout:6/360: write dc/f21 [157240,62827] 0 2024-04-16T02:35:35.856 INFO:tasks.workunit.client.1.smithi153.stdout:2/416: write d9/d24/d34/d36/f7b [486016,24143] 0 2024-04-16T02:35:35.857 INFO:tasks.workunit.client.1.smithi153.stdout:6/361: stat dc/d1a/f3a 0 2024-04-16T02:35:35.858 INFO:tasks.workunit.client.1.smithi153.stdout:6/362: write f4 [359591,54327] 0 2024-04-16T02:35:35.859 INFO:tasks.workunit.client.1.smithi153.stdout:0/442: link c7 dd/d14/d19/d20/c8d 0 2024-04-16T02:35:35.859 INFO:tasks.workunit.client.1.smithi153.stdout:0/443: readlink dd/l13 0 2024-04-16T02:35:35.862 INFO:tasks.workunit.client.1.smithi153.stdout:7/395: mknod d0/d4/db/d34/d4b/d4d/c7d 0 2024-04-16T02:35:35.864 INFO:tasks.workunit.client.1.smithi153.stdout:4/468: fsync d3/f6d 0 2024-04-16T02:35:35.867 INFO:tasks.workunit.client.1.smithi153.stdout:0/444: dwrite dd/d26/d44/f50 [0,4194304] 0 2024-04-16T02:35:35.867 INFO:tasks.workunit.client.1.smithi153.stdout:6/363: rename dc/d12 to dc/d47/d57/d77 0 2024-04-16T02:35:35.871 INFO:tasks.workunit.client.1.smithi153.stdout:6/364: symlink dc/d1a/d2d/l78 0 2024-04-16T02:35:35.871 INFO:tasks.workunit.client.1.smithi153.stdout:9/406: getdents d4/d6 0 2024-04-16T02:35:35.872 INFO:tasks.workunit.client.1.smithi153.stdout:0/445: dread dd/d28/f41 [0,4194304] 0 2024-04-16T02:35:35.873 INFO:tasks.workunit.client.1.smithi153.stdout:7/396: mknod d0/d4/db/c7e 0 2024-04-16T02:35:35.874 INFO:tasks.workunit.client.1.smithi153.stdout:4/469: mknod d3/d9/d24/d75/d9e/ca7 0 2024-04-16T02:35:35.876 INFO:tasks.workunit.client.1.smithi153.stdout:7/397: truncate d0/d4/db/d2b/d53/d59/f5a 323021 0 2024-04-16T02:35:35.877 INFO:tasks.workunit.client.1.smithi153.stdout:2/417: getdents d9/d24/d27/d2c/d41 0 2024-04-16T02:35:35.878 INFO:tasks.workunit.client.1.smithi153.stdout:2/418: chown d9/d24/d34/f7d 208158027 1 2024-04-16T02:35:35.878 INFO:tasks.workunit.client.1.smithi153.stdout:9/407: mkdir d4/d33/d7d 0 2024-04-16T02:35:35.880 INFO:tasks.workunit.client.1.smithi153.stdout:9/408: dread - d4/d33/d3a/d40/f5e zero size 2024-04-16T02:35:35.881 INFO:tasks.workunit.client.1.smithi153.stdout:6/365: fsync dc/d10/f1b 0 2024-04-16T02:35:35.881 INFO:tasks.workunit.client.1.smithi153.stdout:9/409: fsync d4/d33/d3a/f7c 0 2024-04-16T02:35:35.882 INFO:tasks.workunit.client.1.smithi153.stdout:4/470: mknod d3/d9/db/de/d2c/d38/ca8 0 2024-04-16T02:35:35.883 INFO:tasks.workunit.client.1.smithi153.stdout:7/398: symlink d0/d4/db/d16/l7f 0 2024-04-16T02:35:35.888 INFO:tasks.workunit.client.1.smithi153.stdout:9/410: dread d4/d6/dc/d46/f55 [0,4194304] 0 2024-04-16T02:35:35.889 INFO:tasks.workunit.client.1.smithi153.stdout:7/399: write d0/f36 [762015,39776] 0 2024-04-16T02:35:35.892 INFO:tasks.workunit.client.1.smithi153.stdout:2/419: dwrite d9/d24/d27/d2c/d5d/f63 [0,4194304] 0 2024-04-16T02:35:35.893 INFO:tasks.workunit.client.1.smithi153.stdout:0/446: write dd/f10 [1217223,27950] 0 2024-04-16T02:35:35.894 INFO:tasks.workunit.client.1.smithi153.stdout:6/366: dwrite dc/d1a/d2d/f44 [0,4194304] 0 2024-04-16T02:35:35.894 INFO:tasks.workunit.client.1.smithi153.stdout:7/400: readlink d0/d4/db/d2b/d53/d59/d5c/l70 0 2024-04-16T02:35:35.895 INFO:tasks.workunit.client.1.smithi153.stdout:9/411: read d4/d33/f5b [3151273,115663] 0 2024-04-16T02:35:35.897 INFO:tasks.workunit.client.1.smithi153.stdout:9/412: stat d4/d33/d3a/d40/c51 0 2024-04-16T02:35:35.898 INFO:tasks.workunit.client.1.smithi153.stdout:9/413: readlink d4/d33/l45 0 2024-04-16T02:35:35.900 INFO:tasks.workunit.client.1.smithi153.stdout:0/447: rename c2 to dd/d32/c8e 0 2024-04-16T02:35:35.901 INFO:tasks.workunit.client.1.smithi153.stdout:6/367: symlink dc/d47/d57/d77/d15/l79 0 2024-04-16T02:35:35.902 INFO:tasks.workunit.client.1.smithi153.stdout:9/414: mknod d4/d6/dc/d2d/c7e 0 2024-04-16T02:35:35.902 INFO:tasks.workunit.client.1.smithi153.stdout:0/448: read - dd/d26/f5e zero size 2024-04-16T02:35:35.905 INFO:tasks.workunit.client.1.smithi153.stdout:9/415: write d4/d6/f20 [3826770,43743] 0 2024-04-16T02:35:35.906 INFO:tasks.workunit.client.1.smithi153.stdout:0/449: fdatasync dd/d14/d19/f1b 0 2024-04-16T02:35:35.907 INFO:tasks.workunit.client.1.smithi153.stdout:0/450: write dd/f53 [741854,110190] 0 2024-04-16T02:35:35.910 INFO:tasks.workunit.client.1.smithi153.stdout:9/416: mkdir d4/d6/dc/d7f 0 2024-04-16T02:35:35.912 INFO:tasks.workunit.client.1.smithi153.stdout:0/451: link dd/d14/d51/d54/d65/f72 dd/d14/d19/d1c/d73/f8f 0 2024-04-16T02:35:35.913 INFO:tasks.workunit.client.1.smithi153.stdout:0/452: chown dd/d14/d51/d54/d65/d68/l8c 702705 1 2024-04-16T02:35:35.913 INFO:tasks.workunit.client.1.smithi153.stdout:0/453: write dd/f10 [1351716,122818] 0 2024-04-16T02:35:35.916 INFO:tasks.workunit.client.1.smithi153.stdout:0/454: mknod dd/d14/d51/d54/c90 0 2024-04-16T02:35:35.918 INFO:tasks.workunit.client.1.smithi153.stdout:9/417: read d4/d33/d3a/d40/f6d [445682,59713] 0 2024-04-16T02:35:35.919 INFO:tasks.workunit.client.1.smithi153.stdout:0/455: creat dd/d14/d19/d77/d87/f91 x:0 0 0 2024-04-16T02:35:35.921 INFO:tasks.workunit.client.1.smithi153.stdout:6/368: fsync dc/d1a/f26 0 2024-04-16T02:35:35.922 INFO:tasks.workunit.client.1.smithi153.stdout:0/456: write dd/d14/d19/f1e [4511596,99444] 0 2024-04-16T02:35:35.927 INFO:tasks.workunit.client.1.smithi153.stdout:9/418: dread d4/d33/d3a/d40/f6d [0,4194304] 0 2024-04-16T02:35:35.929 INFO:tasks.workunit.client.1.smithi153.stdout:2/420: dwrite d9/d2b/f14 [0,4194304] 0 2024-04-16T02:35:35.935 INFO:tasks.workunit.client.1.smithi153.stdout:9/419: unlink d4/d6/dc/d24/d60/c63 0 2024-04-16T02:35:35.937 INFO:tasks.workunit.client.1.smithi153.stdout:4/471: dread d3/d9/db/de/f3b [0,4194304] 0 2024-04-16T02:35:35.937 INFO:tasks.workunit.client.1.smithi153.stdout:0/457: dwrite dd/d26/f5b [4194304,4194304] 0 2024-04-16T02:35:35.937 INFO:tasks.workunit.client.1.smithi153.stdout:6/369: dwrite dc/f21 [0,4194304] 0 2024-04-16T02:35:35.941 INFO:tasks.workunit.client.1.smithi153.stdout:6/370: write dc/d1a/d2d/f44 [2891480,79254] 0 2024-04-16T02:35:35.942 INFO:tasks.workunit.client.1.smithi153.stdout:0/458: mkdir dd/d26/d92 0 2024-04-16T02:35:35.943 INFO:tasks.workunit.client.1.smithi153.stdout:9/420: creat d4/d33/d3a/d40/d4a/f80 x:0 0 0 2024-04-16T02:35:35.947 INFO:tasks.workunit.client.1.smithi153.stdout:2/421: dwrite d9/d24/d27/d2c/d41/f78 [0,4194304] 0 2024-04-16T02:35:35.949 INFO:tasks.workunit.client.1.smithi153.stdout:0/459: truncate dd/d14/f8b 21108 0 2024-04-16T02:35:35.952 INFO:tasks.workunit.client.1.smithi153.stdout:0/460: write dd/d14/d19/f1e [4330486,16944] 0 2024-04-16T02:35:35.953 INFO:tasks.workunit.client.1.smithi153.stdout:4/472: dwrite d3/d9/d24/fa4 [0,4194304] 0 2024-04-16T02:35:35.954 INFO:tasks.workunit.client.1.smithi153.stdout:2/422: chown d9/d24/d34/l87 3357 1 2024-04-16T02:35:35.958 INFO:tasks.workunit.client.1.smithi153.stdout:2/423: symlink d9/d71/l9c 0 2024-04-16T02:35:35.959 INFO:tasks.workunit.client.1.smithi153.stdout:0/461: creat dd/d32/d80/f93 x:0 0 0 2024-04-16T02:35:35.962 INFO:tasks.workunit.client.1.smithi153.stdout:2/424: write d9/d2b/f19 [783993,17856] 0 2024-04-16T02:35:35.963 INFO:tasks.workunit.client.1.smithi153.stdout:2/425: read - d9/d2b/d1c/d20/f79 zero size 2024-04-16T02:35:35.965 INFO:tasks.workunit.client.1.smithi153.stdout:4/473: creat d3/d9/db/de/fa9 x:0 0 0 2024-04-16T02:35:35.965 INFO:tasks.workunit.client.1.smithi153.stdout:2/426: dread d9/d2b/f2a [0,4194304] 0 2024-04-16T02:35:35.967 INFO:tasks.workunit.client.1.smithi153.stdout:9/421: rmdir d4/d33/d3a 39 2024-04-16T02:35:35.969 INFO:tasks.workunit.client.1.smithi153.stdout:4/474: mknod d3/d9/db/de/d2c/d38/caa 0 2024-04-16T02:35:35.970 INFO:tasks.workunit.client.1.smithi153.stdout:2/427: symlink d9/d24/d34/d36/l9d 0 2024-04-16T02:35:35.971 INFO:tasks.workunit.client.1.smithi153.stdout:6/371: truncate f3 2238754 0 2024-04-16T02:35:35.972 INFO:tasks.workunit.client.1.smithi153.stdout:6/372: stat dc/d47/d57/l6d 0 2024-04-16T02:35:35.972 INFO:tasks.workunit.client.1.smithi153.stdout:4/475: fsync d3/d9/db/d17/d28/f5e 0 2024-04-16T02:35:35.973 INFO:tasks.workunit.client.1.smithi153.stdout:9/422: getdents d4/d33/d7d 0 2024-04-16T02:35:35.973 INFO:tasks.workunit.client.1.smithi153.stdout:6/373: chown dc/d10/f34 1198420 1 2024-04-16T02:35:35.974 INFO:tasks.workunit.client.1.smithi153.stdout:2/428: rename f0 to d9/f9e 0 2024-04-16T02:35:35.976 INFO:tasks.workunit.client.1.smithi153.stdout:9/423: creat d4/d6/dc/d24/d4e/f81 x:0 0 0 2024-04-16T02:35:35.977 INFO:tasks.workunit.client.1.smithi153.stdout:2/429: unlink d9/d24/d27/f4a 0 2024-04-16T02:35:35.978 INFO:tasks.workunit.client.1.smithi153.stdout:6/374: mkdir dc/d47/d57/d77/d7a 0 2024-04-16T02:35:35.979 INFO:tasks.workunit.client.1.smithi153.stdout:2/430: chown d9/d24/d27/d2c/d41/f4c 4413174 1 2024-04-16T02:35:35.982 INFO:tasks.workunit.client.1.smithi153.stdout:4/476: dread d3/d9/f29 [0,4194304] 0 2024-04-16T02:35:35.982 INFO:tasks.workunit.client.1.smithi153.stdout:6/375: unlink dc/f21 0 2024-04-16T02:35:35.984 INFO:tasks.workunit.client.1.smithi153.stdout:9/424: write d4/d33/d3a/d40/d4a/f80 [110596,99814] 0 2024-04-16T02:35:35.984 INFO:tasks.workunit.client.1.smithi153.stdout:2/431: write d9/d24/d27/d2c/d41/f4c [3346383,119044] 0 2024-04-16T02:35:35.985 INFO:tasks.workunit.client.1.smithi153.stdout:5/411: sync 2024-04-16T02:35:35.987 INFO:tasks.workunit.client.1.smithi153.stdout:5/412: fdatasync d8/d1f/d20/d39/f6b 0 2024-04-16T02:35:35.988 INFO:tasks.workunit.client.1.smithi153.stdout:6/376: dread - dc/d47/d57/d77/d15/f74 zero size 2024-04-16T02:35:35.992 INFO:tasks.workunit.client.1.smithi153.stdout:2/432: write d9/d2b/f6b [877426,91640] 0 2024-04-16T02:35:35.994 INFO:tasks.workunit.client.1.smithi153.stdout:9/425: link d4/d33/l44 d4/d6/dc/d7f/l82 0 2024-04-16T02:35:35.994 INFO:tasks.workunit.client.1.smithi153.stdout:5/413: fdatasync d8/df/d23/f32 0 2024-04-16T02:35:35.996 INFO:tasks.workunit.client.1.smithi153.stdout:6/377: unlink dc/d10/l42 0 2024-04-16T02:35:35.998 INFO:tasks.workunit.client.1.smithi153.stdout:4/477: dwrite d3/d9/db/de/d2c/f74 [0,4194304] 0 2024-04-16T02:35:35.998 INFO:tasks.workunit.client.1.smithi153.stdout:2/433: read - d9/d24/d27/d2c/d41/f56 zero size 2024-04-16T02:35:36.003 INFO:tasks.workunit.client.1.smithi153.stdout:5/414: dread d8/df/d2e/f5b [0,4194304] 0 2024-04-16T02:35:36.003 INFO:tasks.workunit.client.1.smithi153.stdout:9/426: rename d4/d6/dc/d24/f42 to d4/d6/dc/d24/f83 0 2024-04-16T02:35:36.004 INFO:tasks.workunit.client.1.smithi153.stdout:0/462: dread dd/d14/d19/d1c/f3e [0,4194304] 0 2024-04-16T02:35:36.006 INFO:tasks.workunit.client.1.smithi153.stdout:4/478: chown d3/d9/db/d45 40276 1 2024-04-16T02:35:36.008 INFO:tasks.workunit.client.1.smithi153.stdout:2/434: creat d9/d24/d27/d9a/f9f x:0 0 0 2024-04-16T02:35:36.018 INFO:tasks.workunit.client.1.smithi153.stdout:5/415: dwrite d8/df/f16 [0,4194304] 0 2024-04-16T02:35:36.018 INFO:tasks.workunit.client.1.smithi153.stdout:9/427: dread d4/f48 [0,4194304] 0 2024-04-16T02:35:36.020 INFO:tasks.workunit.client.1.smithi153.stdout:4/479: getdents d3/d9/db/de/d2c 0 2024-04-16T02:35:36.023 INFO:tasks.workunit.client.1.smithi153.stdout:6/378: dread f6 [0,4194304] 0 2024-04-16T02:35:36.024 INFO:tasks.workunit.client.1.smithi153.stdout:9/428: creat d4/d6/dc/d46/f84 x:0 0 0 2024-04-16T02:35:36.025 INFO:tasks.workunit.client.1.smithi153.stdout:4/480: symlink d3/da5/lab 0 2024-04-16T02:35:36.026 INFO:tasks.workunit.client.1.smithi153.stdout:6/379: readlink dc/d10/l27 0 2024-04-16T02:35:36.027 INFO:tasks.workunit.client.1.smithi153.stdout:6/380: truncate dc/d10/f33 277801 0 2024-04-16T02:35:36.029 INFO:tasks.workunit.client.1.smithi153.stdout:9/429: creat d4/d33/d7d/f85 x:0 0 0 2024-04-16T02:35:36.029 INFO:tasks.workunit.client.1.smithi153.stdout:4/481: mknod d3/d9/db/de/d2c/cac 0 2024-04-16T02:35:36.031 INFO:tasks.workunit.client.1.smithi153.stdout:0/463: dread dd/ff [0,4194304] 0 2024-04-16T02:35:36.031 INFO:tasks.workunit.client.1.smithi153.stdout:6/381: truncate dc/d47/d57/d77/d15/f20 1366366 0 2024-04-16T02:35:36.033 INFO:tasks.workunit.client.1.smithi153.stdout:9/430: write d4/d33/d7d/f85 [925182,21313] 0 2024-04-16T02:35:36.035 INFO:tasks.workunit.client.1.smithi153.stdout:4/482: write d3/d9/db/de/d2c/f74 [451180,85568] 0 2024-04-16T02:35:36.035 INFO:tasks.workunit.client.1.smithi153.stdout:6/382: dread - dc/d47/d57/f70 zero size 2024-04-16T02:35:36.036 INFO:tasks.workunit.client.1.smithi153.stdout:9/431: dread - d4/d33/d3a/f7c zero size 2024-04-16T02:35:36.036 INFO:tasks.workunit.client.1.smithi153.stdout:0/464: truncate f9 3113883 0 2024-04-16T02:35:36.037 INFO:tasks.workunit.client.1.smithi153.stdout:5/416: dread d8/df/d28/f35 [0,4194304] 0 2024-04-16T02:35:36.038 INFO:tasks.workunit.client.1.smithi153.stdout:4/483: unlink d3/d9/f29 0 2024-04-16T02:35:36.038 INFO:tasks.workunit.client.1.smithi153.stdout:6/383: read dc/d47/d57/d77/f24 [455295,15057] 0 2024-04-16T02:35:36.040 INFO:tasks.workunit.client.1.smithi153.stdout:5/417: creat d8/d1f/d20/f81 x:0 0 0 2024-04-16T02:35:36.041 INFO:tasks.workunit.client.1.smithi153.stdout:4/484: truncate d3/d9/db/de/fa9 516530 0 2024-04-16T02:35:36.041 INFO:tasks.workunit.client.1.smithi153.stdout:0/465: fdatasync f9 0 2024-04-16T02:35:36.042 INFO:tasks.workunit.client.1.smithi153.stdout:0/466: stat dd/f22 0 2024-04-16T02:35:36.043 INFO:tasks.workunit.client.1.smithi153.stdout:0/467: write dd/d14/f8b [968761,29330] 0 2024-04-16T02:35:36.044 INFO:tasks.workunit.client.1.smithi153.stdout:9/432: creat d4/d33/d3a/f86 x:0 0 0 2024-04-16T02:35:36.046 INFO:tasks.workunit.client.1.smithi153.stdout:5/418: creat d8/df/d28/f82 x:0 0 0 2024-04-16T02:35:36.046 INFO:tasks.workunit.client.1.smithi153.stdout:5/419: dread - d8/f59 zero size 2024-04-16T02:35:36.047 INFO:tasks.workunit.client.1.smithi153.stdout:4/485: unlink d3/d9/db/d17/l91 0 2024-04-16T02:35:36.050 INFO:tasks.workunit.client.1.smithi153.stdout:0/468: symlink dd/d32/d80/l94 0 2024-04-16T02:35:36.052 INFO:tasks.workunit.client.1.smithi153.stdout:5/420: dread d8/df/d2e/f3d [0,4194304] 0 2024-04-16T02:35:36.053 INFO:tasks.workunit.client.1.smithi153.stdout:6/384: dread dc/d47/d57/d77/d15/f19 [0,4194304] 0 2024-04-16T02:35:36.053 INFO:tasks.workunit.client.1.smithi153.stdout:4/486: mkdir d3/dad 0 2024-04-16T02:35:36.054 INFO:tasks.workunit.client.1.smithi153.stdout:9/433: creat d4/d33/d3a/f87 x:0 0 0 2024-04-16T02:35:36.054 INFO:tasks.workunit.client.1.smithi153.stdout:4/487: fsync d3/d9/f8d 0 2024-04-16T02:35:36.055 INFO:tasks.workunit.client.1.smithi153.stdout:9/434: fdatasync d4/d33/d3a/d40/f5e 0 2024-04-16T02:35:36.057 INFO:tasks.workunit.client.1.smithi153.stdout:4/488: getdents d3/dad 0 2024-04-16T02:35:36.057 INFO:tasks.workunit.client.1.smithi153.stdout:9/435: chown d4/d6/dc/d2d/l4b 108094 1 2024-04-16T02:35:36.059 INFO:tasks.workunit.client.1.smithi153.stdout:0/469: truncate dd/f47 2796217 0 2024-04-16T02:35:36.060 INFO:tasks.workunit.client.1.smithi153.stdout:5/421: dwrite d8/f6f [0,4194304] 0 2024-04-16T02:35:36.065 INFO:tasks.workunit.client.1.smithi153.stdout:0/470: rename dd/c12 to dd/d14/d19/c95 0 2024-04-16T02:35:36.068 INFO:tasks.workunit.client.1.smithi153.stdout:5/422: getdents d8/d1f 0 2024-04-16T02:35:36.070 INFO:tasks.workunit.client.1.smithi153.stdout:4/489: rename d3/d9/d24/d35/l72 to d3/d9/db/d84/lae 0 2024-04-16T02:35:36.071 INFO:tasks.workunit.client.1.smithi153.stdout:0/471: creat dd/d32/f96 x:0 0 0 2024-04-16T02:35:36.071 INFO:tasks.workunit.client.1.smithi153.stdout:9/436: truncate d4/d6/d1b/f59 2782009 0 2024-04-16T02:35:36.072 INFO:tasks.workunit.client.1.smithi153.stdout:4/490: readlink d3/d9/db/d17/d28/l54 0 2024-04-16T02:35:36.074 INFO:tasks.workunit.client.1.smithi153.stdout:0/472: chown dd/d14/d51/d54/f84 11325 1 2024-04-16T02:35:36.076 INFO:tasks.workunit.client.1.smithi153.stdout:5/423: dwrite d8/f3a [4194304,4194304] 0 2024-04-16T02:35:36.079 INFO:tasks.workunit.client.1.smithi153.stdout:6/385: dread dc/d47/d57/d77/d15/f23 [0,4194304] 0 2024-04-16T02:35:36.079 INFO:tasks.workunit.client.1.smithi153.stdout:5/424: mkdir d8/d1f/d20/d83 0 2024-04-16T02:35:36.083 INFO:tasks.workunit.client.1.smithi153.stdout:9/437: write d4/d6/dc/f11 [1213245,122761] 0 2024-04-16T02:35:36.084 INFO:tasks.workunit.client.1.smithi153.stdout:4/491: dwrite d3/d9/d24/fa4 [0,4194304] 0 2024-04-16T02:35:36.085 INFO:tasks.workunit.client.1.smithi153.stdout:6/386: symlink dc/d47/l7b 0 2024-04-16T02:35:36.088 INFO:tasks.workunit.client.1.smithi153.stdout:4/492: dread - d3/d9/d79/f92 zero size 2024-04-16T02:35:36.088 INFO:tasks.workunit.client.1.smithi153.stdout:0/473: dwrite dd/d14/d19/f27 [0,4194304] 0 2024-04-16T02:35:36.089 INFO:tasks.workunit.client.1.smithi153.stdout:6/387: truncate dc/d10/f33 1038470 0 2024-04-16T02:35:36.090 INFO:tasks.workunit.client.1.smithi153.stdout:0/474: write dd/d32/d3f/f62 [324369,102962] 0 2024-04-16T02:35:36.091 INFO:tasks.workunit.client.1.smithi153.stdout:0/475: read - dd/d14/d51/f60 zero size 2024-04-16T02:35:36.094 INFO:tasks.workunit.client.1.smithi153.stdout:0/476: write dd/d26/f5b [2021571,53842] 0 2024-04-16T02:35:36.094 INFO:tasks.workunit.client.1.smithi153.stdout:6/388: rename dc/c2c to dc/d47/d57/d77/d15/c7c 0 2024-04-16T02:35:36.095 INFO:tasks.workunit.client.1.smithi153.stdout:9/438: read d4/d33/d3a/d40/d4a/f68 [1256573,65605] 0 2024-04-16T02:35:36.096 INFO:tasks.workunit.client.1.smithi153.stdout:4/493: creat d3/dad/faf x:0 0 0 2024-04-16T02:35:36.098 INFO:tasks.workunit.client.1.smithi153.stdout:4/494: write d3/d9/db/de/f13 [1470992,114618] 0 2024-04-16T02:35:36.099 INFO:tasks.workunit.client.1.smithi153.stdout:6/389: mkdir dc/d47/d7d 0 2024-04-16T02:35:36.101 INFO:tasks.workunit.client.1.smithi153.stdout:0/477: symlink dd/d14/d19/d20/d4a/l97 0 2024-04-16T02:35:36.103 INFO:tasks.workunit.client.1.smithi153.stdout:0/478: write dd/f6f [1161108,49936] 0 2024-04-16T02:35:36.103 INFO:tasks.workunit.client.1.smithi153.stdout:4/495: creat d3/d9/db/d17/d28/fb0 x:0 0 0 2024-04-16T02:35:36.106 INFO:tasks.workunit.client.1.smithi153.stdout:0/479: fdatasync dd/d26/d44/f82 0 2024-04-16T02:35:36.106 INFO:tasks.workunit.client.1.smithi153.stdout:9/439: dwrite d4/d33/f3f [0,4194304] 0 2024-04-16T02:35:36.109 INFO:tasks.workunit.client.1.smithi153.stdout:6/390: dread dc/d47/d57/d77/d15/f19 [0,4194304] 0 2024-04-16T02:35:36.110 INFO:tasks.workunit.client.1.smithi153.stdout:0/480: unlink dd/d14/f8b 0 2024-04-16T02:35:36.111 INFO:tasks.workunit.client.1.smithi153.stdout:0/481: dread - dd/d28/f83 zero size 2024-04-16T02:35:36.114 INFO:tasks.workunit.client.1.smithi153.stdout:5/425: dread d8/d1f/d20/d39/f6b [0,4194304] 0 2024-04-16T02:35:36.119 INFO:tasks.workunit.client.1.smithi153.stdout:4/496: dwrite d3/d9/d79/fa6 [0,4194304] 0 2024-04-16T02:35:36.119 INFO:tasks.workunit.client.1.smithi153.stdout:0/482: write dd/d14/d19/f27 [5075539,42989] 0 2024-04-16T02:35:36.119 INFO:tasks.workunit.client.1.smithi153.stdout:9/440: creat d4/d33/d3a/f88 x:0 0 0 2024-04-16T02:35:36.119 INFO:tasks.workunit.client.1.smithi153.stdout:5/426: fsync d8/df/d2e/f3d 0 2024-04-16T02:35:36.122 INFO:tasks.workunit.client.1.smithi153.stdout:9/441: creat d4/d6/dc/d24/f89 x:0 0 0 2024-04-16T02:35:36.123 INFO:tasks.workunit.client.1.smithi153.stdout:9/442: dread - d4/d6/f77 zero size 2024-04-16T02:35:36.124 INFO:tasks.workunit.client.1.smithi153.stdout:4/497: link d3/d9/db/d17/l18 d3/d9/d24/d35/lb1 0 2024-04-16T02:35:36.125 INFO:tasks.workunit.client.1.smithi153.stdout:6/391: write f3 [1784868,44223] 0 2024-04-16T02:35:36.127 INFO:tasks.workunit.client.1.smithi153.stdout:4/498: readlink d3/d9/db/de/d2c/d38/l93 0 2024-04-16T02:35:36.128 INFO:tasks.workunit.client.1.smithi153.stdout:9/443: rename d4/d33/f3f to d4/d6/dc/d2d/d64/f8a 0 2024-04-16T02:35:36.129 INFO:tasks.workunit.client.1.smithi153.stdout:6/392: creat dc/d47/f7e x:0 0 0 2024-04-16T02:35:36.129 INFO:tasks.workunit.client.1.smithi153.stdout:5/427: dwrite d8/f6f [0,4194304] 0 2024-04-16T02:35:36.133 INFO:tasks.workunit.client.1.smithi153.stdout:0/483: dwrite dd/d14/d19/f1b [0,4194304] 0 2024-04-16T02:35:36.136 INFO:tasks.workunit.client.1.smithi153.stdout:5/428: truncate d8/df/d28/d5a/d66/f7b 740417 0 2024-04-16T02:35:36.138 INFO:tasks.workunit.client.1.smithi153.stdout:9/444: rename d4/d6/f77 to d4/d6/dc/d24/d5d/f8b 0 2024-04-16T02:35:36.138 INFO:tasks.workunit.client.1.smithi153.stdout:6/393: dread dc/d47/d57/d77/d15/f20 [0,4194304] 0 2024-04-16T02:35:36.140 INFO:tasks.workunit.client.1.smithi153.stdout:4/499: getdents d3/d9/db/de/d2c/d52 0 2024-04-16T02:35:36.144 INFO:tasks.workunit.client.1.smithi153.stdout:9/445: mkdir d4/d6/d8c 0 2024-04-16T02:35:36.144 INFO:tasks.workunit.client.1.smithi153.stdout:0/484: dwrite dd/d14/d19/d77/d87/f91 [0,4194304] 0 2024-04-16T02:35:36.147 INFO:tasks.workunit.client.1.smithi153.stdout:5/429: mkdir d8/df/d84 0 2024-04-16T02:35:36.149 INFO:tasks.workunit.client.1.smithi153.stdout:5/430: read - d8/f59 zero size 2024-04-16T02:35:36.149 INFO:tasks.workunit.client.1.smithi153.stdout:4/500: mknod d3/d9/db/de/d2c/d38/cb2 0 2024-04-16T02:35:36.151 INFO:tasks.workunit.client.1.smithi153.stdout:4/501: chown d3/d9/d79/fa6 7942281 1 2024-04-16T02:35:36.153 INFO:tasks.workunit.client.1.smithi153.stdout:5/431: getdents d8/df/d23/d70 0 2024-04-16T02:35:36.154 INFO:tasks.workunit.client.1.smithi153.stdout:0/485: dwrite dd/d28/f83 [0,4194304] 0 2024-04-16T02:35:36.156 INFO:tasks.workunit.client.1.smithi153.stdout:0/486: chown dd/d32/d57/f61 192 1 2024-04-16T02:35:36.161 INFO:tasks.workunit.client.1.smithi153.stdout:4/502: dwrite d3/d9/db/de/fa9 [0,4194304] 0 2024-04-16T02:35:36.163 INFO:tasks.workunit.client.1.smithi153.stdout:9/446: rename d4/d6/dc/d24/d5d/f8b to d4/d6/dc/d2d/d64/f8d 0 2024-04-16T02:35:36.163 INFO:tasks.workunit.client.1.smithi153.stdout:6/394: write dc/d47/d57/d77/d15/f23 [1074161,69464] 0 2024-04-16T02:35:36.164 INFO:tasks.workunit.client.1.smithi153.stdout:4/503: fdatasync d3/d9/db/f4a 0 2024-04-16T02:35:36.165 INFO:tasks.workunit.client.1.smithi153.stdout:0/487: creat dd/d14/d19/d1c/f98 x:0 0 0 2024-04-16T02:35:36.166 INFO:tasks.workunit.client.1.smithi153.stdout:4/504: read - d3/d9/d79/f95 zero size 2024-04-16T02:35:36.166 INFO:tasks.workunit.client.1.smithi153.stdout:0/488: read - dd/d14/d51/d54/d65/f72 zero size 2024-04-16T02:35:36.167 INFO:tasks.workunit.client.1.smithi153.stdout:4/505: read - d3/d9/db/de/d2c/d52/d85/f90 zero size 2024-04-16T02:35:36.170 INFO:tasks.workunit.client.1.smithi153.stdout:6/395: symlink dc/d47/d57/l7f 0 2024-04-16T02:35:36.171 INFO:tasks.workunit.client.1.smithi153.stdout:6/396: dread - dc/d47/f4a zero size 2024-04-16T02:35:36.173 INFO:tasks.workunit.client.1.smithi153.stdout:9/447: symlink d4/d33/d3a/d40/d4a/l8e 0 2024-04-16T02:35:36.175 INFO:tasks.workunit.client.1.smithi153.stdout:0/489: readlink dd/d28/l45 0 2024-04-16T02:35:36.175 INFO:tasks.workunit.client.1.smithi153.stdout:4/506: mknod d3/d9/db/d17/d28/d4c/cb3 0 2024-04-16T02:35:36.176 INFO:tasks.workunit.client.1.smithi153.stdout:6/397: mkdir dc/d47/d80 0 2024-04-16T02:35:36.180 INFO:tasks.workunit.client.1.smithi153.stdout:3/392: sync 2024-04-16T02:35:36.180 INFO:tasks.workunit.client.1.smithi153.stdout:1/408: sync 2024-04-16T02:35:36.182 INFO:tasks.workunit.client.1.smithi153.stdout:4/507: creat d3/d9/d24/d75/fb4 x:0 0 0 2024-04-16T02:35:36.182 INFO:tasks.workunit.client.1.smithi153.stdout:9/448: creat d4/d6/dc/d2d/d79/f8f x:0 0 0 2024-04-16T02:35:36.185 INFO:tasks.workunit.client.1.smithi153.stdout:1/409: fdatasync d4/dd/d13/f2a 0 2024-04-16T02:35:36.187 INFO:tasks.workunit.client.1.smithi153.stdout:9/449: rmdir d4/d6/dc/d24/d5d 39 2024-04-16T02:35:36.188 INFO:tasks.workunit.client.1.smithi153.stdout:1/410: readlink d4/dd/d20/l44 0 2024-04-16T02:35:36.189 INFO:tasks.workunit.client.1.smithi153.stdout:9/450: stat d4/c26 0 2024-04-16T02:35:36.189 INFO:tasks.workunit.client.1.smithi153.stdout:5/432: dread d8/df/d2f/d4e/d51/f5c [4194304,4194304] 0 2024-04-16T02:35:36.191 INFO:tasks.workunit.client.1.smithi153.stdout:0/490: dwrite dd/d26/f5e [0,4194304] 0 2024-04-16T02:35:36.192 INFO:tasks.workunit.client.1.smithi153.stdout:1/411: chown d4/dd/d18/d29/d6e 541289483 1 2024-04-16T02:35:36.195 INFO:tasks.workunit.client.1.smithi153.stdout:6/398: dread dc/d1a/d2d/f3f [0,4194304] 0 2024-04-16T02:35:36.196 INFO:tasks.workunit.client.1.smithi153.stdout:3/393: write d6/d10/f1c [2576118,129899] 0 2024-04-16T02:35:36.197 INFO:tasks.workunit.client.1.smithi153.stdout:4/508: dwrite d3/d9/f77 [0,4194304] 0 2024-04-16T02:35:36.198 INFO:tasks.workunit.client.1.smithi153.stdout:5/433: mknod d8/df/d28/d5a/d72/c85 0 2024-04-16T02:35:36.202 INFO:tasks.workunit.client.1.smithi153.stdout:0/491: write dd/d14/d19/d1c/f48 [683058,93728] 0 2024-04-16T02:35:36.202 INFO:tasks.workunit.client.1.smithi153.stdout:9/451: getdents d4/d33/d3a 0 2024-04-16T02:35:36.203 INFO:tasks.workunit.client.1.smithi153.stdout:6/399: fsync dc/d47/d57/d77/d15/f22 0 2024-04-16T02:35:36.203 INFO:tasks.workunit.client.1.smithi153.stdout:5/434: fdatasync d8/d1f/f3f 0 2024-04-16T02:35:36.207 INFO:tasks.workunit.client.1.smithi153.stdout:9/452: dread - d4/d33/d3a/f7c zero size 2024-04-16T02:35:36.207 INFO:tasks.workunit.client.1.smithi153.stdout:5/435: fsync d8/f30 0 2024-04-16T02:35:36.208 INFO:tasks.workunit.client.1.smithi153.stdout:4/509: getdents d3/da5 0 2024-04-16T02:35:36.209 INFO:tasks.workunit.client.1.smithi153.stdout:0/492: creat dd/d14/d51/d54/d65/f99 x:0 0 0 2024-04-16T02:35:36.209 INFO:tasks.workunit.client.1.smithi153.stdout:1/412: dwrite d4/f5e [0,4194304] 0 2024-04-16T02:35:36.211 INFO:tasks.workunit.client.1.smithi153.stdout:0/493: fdatasync dd/d14/f4b 0 2024-04-16T02:35:36.213 INFO:tasks.workunit.client.1.smithi153.stdout:6/400: truncate dc/d47/d57/d77/f24 1252657 0 2024-04-16T02:35:36.214 INFO:tasks.workunit.client.1.smithi153.stdout:9/453: rmdir d4/d6/dc/d2d/d75 0 2024-04-16T02:35:36.215 INFO:tasks.workunit.client.1.smithi153.stdout:0/494: write dd/d32/d57/f89 [673943,27418] 0 2024-04-16T02:35:36.216 INFO:tasks.workunit.client.1.smithi153.stdout:1/413: creat d4/dd/d13/d26/d58/f91 x:0 0 0 2024-04-16T02:35:36.219 INFO:tasks.workunit.client.1.smithi153.stdout:1/414: dread - d4/dd/d18/d29/f59 zero size 2024-04-16T02:35:36.220 INFO:tasks.workunit.client.1.smithi153.stdout:5/436: dwrite d8/df/d2e/f3d [0,4194304] 0 2024-04-16T02:35:36.222 INFO:tasks.workunit.client.1.smithi153.stdout:1/415: stat d4/dd/d13/d26/l68 0 2024-04-16T02:35:36.224 INFO:tasks.workunit.client.1.smithi153.stdout:6/401: mknod dc/d47/d57/d77/d75/c81 0 2024-04-16T02:35:36.224 INFO:tasks.workunit.client.1.smithi153.stdout:4/510: link d3/d9/db/de/d2c/d38/c40 d3/d9/db/d17/d28/d4c/cb5 0 2024-04-16T02:35:36.230 INFO:tasks.workunit.client.1.smithi153.stdout:4/511: stat d3/d9/db/de/d2c/d38/f9d 0 2024-04-16T02:35:36.231 INFO:tasks.workunit.client.1.smithi153.stdout:0/495: dwrite dd/d26/d44/f82 [0,4194304] 0 2024-04-16T02:35:36.231 INFO:tasks.workunit.client.1.smithi153.stdout:9/454: dwrite d4/d6/dc/d24/f43 [0,4194304] 0 2024-04-16T02:35:36.232 INFO:tasks.workunit.client.1.smithi153.stdout:1/416: readlink d4/dd/d13/d7e/d80/l85 0 2024-04-16T02:35:36.232 INFO:tasks.workunit.client.1.smithi153.stdout:6/402: chown dc/d1a/d2d/c3b 4 1 2024-04-16T02:35:36.241 INFO:tasks.workunit.client.1.smithi153.stdout:9/455: chown d4/d6/l34 3384 1 2024-04-16T02:35:36.241 INFO:tasks.workunit.client.1.smithi153.stdout:5/437: dread d8/d1f/d20/d39/f6b [0,4194304] 0 2024-04-16T02:35:36.241 INFO:tasks.workunit.client.1.smithi153.stdout:6/403: mknod dc/d1a/d2d/c82 0 2024-04-16T02:35:36.242 INFO:tasks.workunit.client.1.smithi153.stdout:4/512: write d3/d9/db/de/f26 [558207,73878] 0 2024-04-16T02:35:36.242 INFO:tasks.workunit.client.1.smithi153.stdout:0/496: write dd/d26/f5e [466767,20611] 0 2024-04-16T02:35:36.244 INFO:tasks.workunit.client.1.smithi153.stdout:0/497: readlink dd/d28/l45 0 2024-04-16T02:35:36.244 INFO:tasks.workunit.client.1.smithi153.stdout:5/438: fsync d8/df/d2e/f57 0 2024-04-16T02:35:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:35 smithi151 ceph-mon[29003]: pgmap v15: 65 pgs: 65 active+clean; 637 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 3.3 MiB/s rd, 76 MiB/s wr, 295 op/s 2024-04-16T02:35:36.246 INFO:tasks.workunit.client.1.smithi153.stdout:9/456: chown d4/d6/dc/d2d/f58 0 1 2024-04-16T02:35:36.248 INFO:tasks.workunit.client.1.smithi153.stdout:9/457: fsync d4/d6/d1b/f71 0 2024-04-16T02:35:36.248 INFO:tasks.workunit.client.1.smithi153.stdout:5/439: chown d8/df/d2e/c54 88431665 1 2024-04-16T02:35:36.251 INFO:tasks.workunit.client.1.smithi153.stdout:6/404: dwrite dc/d47/d57/f70 [0,4194304] 0 2024-04-16T02:35:36.253 INFO:tasks.workunit.client.1.smithi153.stdout:5/440: unlink d8/d1f/d20/c61 0 2024-04-16T02:35:36.253 INFO:tasks.workunit.client.1.smithi153.stdout:5/441: fdatasync d8/df/d2e/f77 0 2024-04-16T02:35:36.255 INFO:tasks.workunit.client.1.smithi153.stdout:6/405: readlink dc/d6b/l76 0 2024-04-16T02:35:36.256 INFO:tasks.workunit.client.1.smithi153.stdout:0/498: getdents dd/d26/d44 0 2024-04-16T02:35:36.257 INFO:tasks.workunit.client.1.smithi153.stdout:9/458: creat d4/d6/d1b/f90 x:0 0 0 2024-04-16T02:35:36.258 INFO:tasks.workunit.client.1.smithi153.stdout:5/442: truncate d8/df/d23/f32 162863 0 2024-04-16T02:35:36.261 INFO:tasks.workunit.client.1.smithi153.stdout:0/499: mknod dd/d14/d19/d1c/d73/c9a 0 2024-04-16T02:35:36.263 INFO:tasks.workunit.client.1.smithi153.stdout:5/443: write d8/df/d2e/f3d [959006,91352] 0 2024-04-16T02:35:36.266 INFO:tasks.workunit.client.1.smithi153.stdout:0/500: chown dd/d32/d3f/c66 3889 1 2024-04-16T02:35:36.266 INFO:tasks.workunit.client.1.smithi153.stdout:1/417: write d4/dd/d18/f4b [1752695,127619] 0 2024-04-16T02:35:36.268 INFO:tasks.workunit.client.1.smithi153.stdout:9/459: dwrite d4/d33/d3a/f87 [0,4194304] 0 2024-04-16T02:35:36.269 INFO:tasks.workunit.client.1.smithi153.stdout:0/501: mknod dd/d14/d19/d55/c9b 0 2024-04-16T02:35:36.270 INFO:tasks.workunit.client.1.smithi153.stdout:6/406: dread dc/d47/d57/d77/d15/f5a [0,4194304] 0 2024-04-16T02:35:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:35 smithi153 ceph-mon[36747]: pgmap v15: 65 pgs: 65 active+clean; 637 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 3.3 MiB/s rd, 76 MiB/s wr, 295 op/s 2024-04-16T02:35:36.314 INFO:tasks.workunit.client.0.smithi151.stdout:0/153: sync 2024-04-16T02:35:36.314 INFO:tasks.workunit.client.0.smithi151.stdout:4/62: sync 2024-04-16T02:35:36.314 INFO:tasks.workunit.client.0.smithi151.stdout:0/154: stat dc/d20/l26 0 2024-04-16T02:35:36.315 INFO:tasks.workunit.client.1.smithi153.stdout:8/381: sync 2024-04-16T02:35:36.315 INFO:tasks.workunit.client.0.smithi151.stdout:4/63: creat d0/f16 x:0 0 0 2024-04-16T02:35:36.316 INFO:tasks.workunit.client.0.smithi151.stdout:0/155: mkdir dc/d15/d34 0 2024-04-16T02:35:36.317 INFO:tasks.workunit.client.0.smithi151.stdout:0/156: fdatasync dc/d15/f23 0 2024-04-16T02:35:36.317 INFO:tasks.workunit.client.0.smithi151.stdout:4/64: mkdir d0/d17 0 2024-04-16T02:35:36.318 INFO:tasks.workunit.client.0.smithi151.stdout:4/65: truncate d0/fd 821905 0 2024-04-16T02:35:36.322 INFO:tasks.workunit.client.0.smithi151.stdout:4/66: symlink d0/d11/l18 0 2024-04-16T02:35:36.322 INFO:tasks.workunit.client.0.smithi151.stdout:0/157: dread dc/d15/f19 [0,4194304] 0 2024-04-16T02:35:36.322 INFO:tasks.workunit.client.0.smithi151.stdout:4/67: truncate d0/f15 806451 0 2024-04-16T02:35:36.327 INFO:tasks.workunit.client.1.smithi153.stdout:9/460: creat d4/d33/d3a/d40/d4a/f91 x:0 0 0 2024-04-16T02:35:36.327 INFO:tasks.workunit.client.0.smithi151.stdout:4/68: dwrite d0/fe [0,4194304] 0 2024-04-16T02:35:36.328 INFO:tasks.workunit.client.1.smithi153.stdout:0/502: write dd/d32/d57/f42 [712399,43825] 0 2024-04-16T02:35:36.328 INFO:tasks.workunit.client.0.smithi151.stdout:4/69: chown d0/c9 2032 1 2024-04-16T02:35:36.328 INFO:tasks.workunit.client.1.smithi153.stdout:8/382: stat d2/d13/d3a/d5b 0 2024-04-16T02:35:36.329 INFO:tasks.workunit.client.1.smithi153.stdout:9/461: read - d4/d6/d1b/f76 zero size 2024-04-16T02:35:36.330 INFO:tasks.workunit.client.1.smithi153.stdout:4/513: dread d3/d9/db/d17/f55 [0,4194304] 0 2024-04-16T02:35:36.331 INFO:tasks.workunit.client.1.smithi153.stdout:1/418: dwrite d4/f5e [0,4194304] 0 2024-04-16T02:35:36.333 INFO:tasks.workunit.client.1.smithi153.stdout:9/462: fsync d4/d6/dc/d46/f49 0 2024-04-16T02:35:36.334 INFO:tasks.workunit.client.1.smithi153.stdout:1/419: chown d4/dd/d18/d29/l6a 298266 1 2024-04-16T02:35:36.336 INFO:tasks.workunit.client.1.smithi153.stdout:1/420: write d4/dd/d13/d26/f33 [4874211,63858] 0 2024-04-16T02:35:36.337 INFO:tasks.workunit.client.1.smithi153.stdout:6/407: creat dc/d59/d5b/f83 x:0 0 0 2024-04-16T02:35:36.339 INFO:tasks.workunit.client.0.smithi151.stdout:4/70: dwrite d0/fe [0,4194304] 0 2024-04-16T02:35:36.340 INFO:tasks.workunit.client.1.smithi153.stdout:8/383: unlink d2/d1c/d1f/l21 0 2024-04-16T02:35:36.340 INFO:tasks.workunit.client.1.smithi153.stdout:9/463: read d4/d33/d3a/d40/f78 [209182,265] 0 2024-04-16T02:35:36.346 INFO:tasks.workunit.client.1.smithi153.stdout:8/384: dwrite d2/de/d34/f6f [0,4194304] 0 2024-04-16T02:35:36.348 INFO:tasks.workunit.client.1.smithi153.stdout:8/385: write d2/de/f3e [4263001,57807] 0 2024-04-16T02:35:36.349 INFO:tasks.workunit.client.0.smithi151.stdout:4/71: truncate d0/f8 1251960 0 2024-04-16T02:35:36.350 INFO:tasks.workunit.client.1.smithi153.stdout:6/408: mknod dc/c84 0 2024-04-16T02:35:36.351 INFO:tasks.workunit.client.0.smithi151.stdout:4/72: mkdir d0/d11/d19 0 2024-04-16T02:35:36.352 INFO:tasks.workunit.client.1.smithi153.stdout:9/464: unlink d4/l15 0 2024-04-16T02:35:36.352 INFO:tasks.workunit.client.1.smithi153.stdout:0/503: write dd/d14/d51/d54/d65/f72 [852929,103738] 0 2024-04-16T02:35:36.354 INFO:tasks.workunit.client.0.smithi151.stdout:4/73: mknod d0/c1a 0 2024-04-16T02:35:36.354 INFO:tasks.workunit.client.1.smithi153.stdout:9/465: read d4/d6/dc/d24/f43 [3099677,9457] 0 2024-04-16T02:35:36.355 INFO:tasks.workunit.client.1.smithi153.stdout:4/514: getdents d3/d9/d24/d35 0 2024-04-16T02:35:36.356 INFO:tasks.workunit.client.1.smithi153.stdout:8/386: rmdir d2/d13/d3a/d5b 39 2024-04-16T02:35:36.359 INFO:tasks.workunit.client.1.smithi153.stdout:6/409: creat dc/d47/d57/d77/f85 x:0 0 0 2024-04-16T02:35:36.361 INFO:tasks.workunit.client.1.smithi153.stdout:9/466: rename d4/d6/d1b/d2a/l3c to d4/d6/dc/d24/d4e/l92 0 2024-04-16T02:35:36.361 INFO:tasks.workunit.client.1.smithi153.stdout:8/387: symlink d2/d1c/d1f/d27/d2d/d47/d60/l7e 0 2024-04-16T02:35:36.363 INFO:tasks.workunit.client.1.smithi153.stdout:6/410: dread - dc/d1a/f3a zero size 2024-04-16T02:35:36.363 INFO:tasks.workunit.client.1.smithi153.stdout:9/467: chown d4/d6/dc 27043652 1 2024-04-16T02:35:36.367 INFO:tasks.workunit.client.1.smithi153.stdout:8/388: chown d2/d1c/d1f/d27/f2a 90940 1 2024-04-16T02:35:36.367 INFO:tasks.workunit.client.1.smithi153.stdout:0/504: dwrite dd/d14/d19/d1c/f3e [0,4194304] 0 2024-04-16T02:35:36.370 INFO:tasks.workunit.client.0.smithi151.stdout:4/74: write d0/f3 [2426049,69785] 0 2024-04-16T02:35:36.370 INFO:tasks.workunit.client.1.smithi153.stdout:4/515: dwrite d3/d9/f68 [0,4194304] 0 2024-04-16T02:35:36.371 INFO:tasks.workunit.client.1.smithi153.stdout:4/516: dread - d3/d9/db/d17/d28/f9c zero size 2024-04-16T02:35:36.373 INFO:tasks.workunit.client.1.smithi153.stdout:9/468: unlink d4/d6/f20 0 2024-04-16T02:35:36.374 INFO:tasks.workunit.client.1.smithi153.stdout:6/411: creat dc/d47/d57/d77/d15/f86 x:0 0 0 2024-04-16T02:35:36.377 INFO:tasks.workunit.client.1.smithi153.stdout:4/517: write d3/d9/db/de/f26 [607635,54690] 0 2024-04-16T02:35:36.382 INFO:tasks.workunit.client.1.smithi153.stdout:0/505: write dd/d14/d19/d20/d4a/f78 [944414,33716] 0 2024-04-16T02:35:36.382 INFO:tasks.workunit.client.1.smithi153.stdout:4/518: write d3/d9/f14 [442468,99915] 0 2024-04-16T02:35:36.384 INFO:tasks.workunit.client.1.smithi153.stdout:9/469: write d4/d33/d3a/d40/d4a/f68 [2958825,25616] 0 2024-04-16T02:35:36.384 INFO:tasks.workunit.client.1.smithi153.stdout:8/389: dwrite d2/de/f15 [0,4194304] 0 2024-04-16T02:35:36.386 INFO:tasks.workunit.client.1.smithi153.stdout:0/506: fsync dd/d14/d51/f59 0 2024-04-16T02:35:36.386 INFO:tasks.workunit.client.1.smithi153.stdout:1/421: dread d4/dd/d13/d26/f2b [0,4194304] 0 2024-04-16T02:35:36.387 INFO:tasks.workunit.client.0.smithi151.stdout:4/75: dwrite d0/f8 [0,4194304] 0 2024-04-16T02:35:36.387 INFO:tasks.workunit.client.0.smithi151.stdout:4/76: stat d0/f15 0 2024-04-16T02:35:36.388 INFO:tasks.workunit.client.1.smithi153.stdout:4/519: dread - d3/d9/db/d17/d3f/f94 zero size 2024-04-16T02:35:36.388 INFO:tasks.workunit.client.0.smithi151.stdout:4/77: readlink d0/d11/l18 0 2024-04-16T02:35:36.392 INFO:tasks.workunit.client.0.smithi151.stdout:4/78: creat d0/d11/d19/f1b x:0 0 0 2024-04-16T02:35:36.393 INFO:tasks.workunit.client.0.smithi151.stdout:4/79: fsync d0/fd 0 2024-04-16T02:35:36.394 INFO:tasks.workunit.client.1.smithi153.stdout:6/412: dwrite dc/d59/d63/f66 [0,4194304] 0 2024-04-16T02:35:36.395 INFO:tasks.workunit.client.1.smithi153.stdout:8/390: write d2/d1c/d1f/d27/d5e/f73 [552037,76065] 0 2024-04-16T02:35:36.398 INFO:tasks.workunit.client.1.smithi153.stdout:0/507: symlink dd/d26/d92/l9c 0 2024-04-16T02:35:36.398 INFO:tasks.workunit.client.0.smithi151.stdout:4/80: dread d0/fe [0,4194304] 0 2024-04-16T02:35:36.400 INFO:tasks.workunit.client.1.smithi153.stdout:2/435: sync 2024-04-16T02:35:36.401 INFO:tasks.workunit.client.1.smithi153.stdout:3/394: sync 2024-04-16T02:35:36.401 INFO:tasks.workunit.client.1.smithi153.stdout:7/401: sync 2024-04-16T02:35:36.401 INFO:tasks.workunit.client.1.smithi153.stdout:9/470: dwrite d4/d6/d1b/f71 [0,4194304] 0 2024-04-16T02:35:36.401 INFO:tasks.workunit.client.1.smithi153.stdout:6/413: creat dc/d47/f87 x:0 0 0 2024-04-16T02:35:36.403 INFO:tasks.workunit.client.1.smithi153.stdout:4/520: creat d3/d9/db/fb6 x:0 0 0 2024-04-16T02:35:36.403 INFO:tasks.workunit.client.0.smithi151.stdout:4/81: rename d0/l13 to d0/l1c 0 2024-04-16T02:35:36.404 INFO:tasks.workunit.client.1.smithi153.stdout:4/521: dread - d3/d9/d24/d75/fb4 zero size 2024-04-16T02:35:36.405 INFO:tasks.workunit.client.1.smithi153.stdout:1/422: creat d4/dd/d13/d26/f92 x:0 0 0 2024-04-16T02:35:36.407 INFO:tasks.workunit.client.1.smithi153.stdout:8/391: mknod d2/d13/c7f 0 2024-04-16T02:35:36.407 INFO:tasks.workunit.client.1.smithi153.stdout:1/423: truncate d4/dd/d20/f72 1043707 0 2024-04-16T02:35:36.409 INFO:tasks.workunit.client.0.smithi151.stdout:4/82: getdents d0/d11 0 2024-04-16T02:35:36.410 INFO:tasks.workunit.client.0.smithi151.stdout:4/83: fsync d0/fb 0 2024-04-16T02:35:36.416 INFO:tasks.workunit.client.0.smithi151.stdout:4/84: dwrite d0/fb [0,4194304] 0 2024-04-16T02:35:36.417 INFO:tasks.workunit.client.0.smithi151.stdout:4/85: truncate d0/f10 4806131 0 2024-04-16T02:35:36.421 INFO:tasks.workunit.client.0.smithi151.stdout:4/86: dread d0/f3 [0,4194304] 0 2024-04-16T02:35:36.423 INFO:tasks.workunit.client.0.smithi151.stdout:4/87: rmdir d0/d11 39 2024-04-16T02:35:36.424 INFO:tasks.workunit.client.0.smithi151.stdout:4/88: read d0/f8 [2609112,99374] 0 2024-04-16T02:35:36.427 INFO:tasks.workunit.client.0.smithi151.stdout:4/89: creat d0/d17/f1d x:0 0 0 2024-04-16T02:35:36.430 INFO:tasks.workunit.client.0.smithi151.stdout:4/90: dread d0/fb [0,4194304] 0 2024-04-16T02:35:36.431 INFO:tasks.workunit.client.0.smithi151.stdout:4/91: write d0/f16 [342967,53440] 0 2024-04-16T02:35:36.433 INFO:tasks.workunit.client.0.smithi151.stdout:2/65: sync 2024-04-16T02:35:36.433 INFO:tasks.workunit.client.0.smithi151.stdout:9/53: sync 2024-04-16T02:35:36.433 INFO:tasks.workunit.client.0.smithi151.stdout:3/69: sync 2024-04-16T02:35:36.433 INFO:tasks.workunit.client.0.smithi151.stdout:7/55: sync 2024-04-16T02:35:36.433 INFO:tasks.workunit.client.0.smithi151.stdout:6/27: sync 2024-04-16T02:35:36.434 INFO:tasks.workunit.client.0.smithi151.stdout:1/32: sync 2024-04-16T02:35:36.434 INFO:tasks.workunit.client.0.smithi151.stdout:5/81: sync 2024-04-16T02:35:36.434 INFO:tasks.workunit.client.0.smithi151.stdout:8/75: sync 2024-04-16T02:35:36.434 INFO:tasks.workunit.client.0.smithi151.stdout:4/92: write d0/d11/d19/f1b [592108,127345] 0 2024-04-16T02:35:36.435 INFO:tasks.workunit.client.0.smithi151.stdout:8/76: chown d11/f18 0 1 2024-04-16T02:35:36.435 INFO:tasks.workunit.client.0.smithi151.stdout:5/82: chown d2/f18 25643893 1 2024-04-16T02:35:36.435 INFO:tasks.workunit.client.0.smithi151.stdout:6/28: dread - d2/d5/f6 zero size 2024-04-16T02:35:36.435 INFO:tasks.workunit.client.0.smithi151.stdout:3/70: dread - d1/d6/d10/f15 zero size 2024-04-16T02:35:36.436 INFO:tasks.workunit.client.0.smithi151.stdout:9/54: chown d1/d3/fb 579018386 1 2024-04-16T02:35:36.436 INFO:tasks.workunit.client.0.smithi151.stdout:1/33: mkdir d3/dc 0 2024-04-16T02:35:36.438 INFO:tasks.workunit.client.0.smithi151.stdout:4/93: write d0/fb [4919799,108364] 0 2024-04-16T02:35:36.439 INFO:tasks.workunit.client.0.smithi151.stdout:9/55: truncate d1/f7 697578 0 2024-04-16T02:35:36.439 INFO:tasks.workunit.client.0.smithi151.stdout:7/56: rename d0/fc to d0/ff 0 2024-04-16T02:35:36.441 INFO:tasks.workunit.client.0.smithi151.stdout:6/29: symlink d2/d5/l7 0 2024-04-16T02:35:36.442 INFO:tasks.workunit.client.0.smithi151.stdout:3/71: symlink d1/d6/l18 0 2024-04-16T02:35:36.444 INFO:tasks.workunit.client.0.smithi151.stdout:1/34: write f1 [4054873,93621] 0 2024-04-16T02:35:36.444 INFO:tasks.workunit.client.0.smithi151.stdout:5/83: dwrite d2/d6/f9 [0,4194304] 0 2024-04-16T02:35:36.450 INFO:tasks.workunit.client.1.smithi153.stdout:0/508: mkdir dd/d14/d51/d7f/d9d 0 2024-04-16T02:35:36.451 INFO:tasks.workunit.client.1.smithi153.stdout:9/471: rmdir d4/d6/dc/d7f 39 2024-04-16T02:35:36.451 INFO:tasks.workunit.client.1.smithi153.stdout:7/402: symlink d0/d4/db/d2b/d53/d59/d5c/l80 0 2024-04-16T02:35:36.452 INFO:tasks.workunit.client.1.smithi153.stdout:4/522: mknod d3/d9/db/de/d2c/d52/d85/cb7 0 2024-04-16T02:35:36.454 INFO:tasks.workunit.client.1.smithi153.stdout:1/424: symlink d4/dd/d18/d29/d71/l93 0 2024-04-16T02:35:36.454 INFO:tasks.workunit.client.1.smithi153.stdout:4/523: fsync d3/d9/f77 0 2024-04-16T02:35:36.455 INFO:tasks.workunit.client.1.smithi153.stdout:8/392: fdatasync d2/d1c/d1f/d27/d2d/d47/d60/f7b 0 2024-04-16T02:35:36.455 INFO:tasks.workunit.client.1.smithi153.stdout:0/509: unlink dd/d14/d19/d1c/f2f 0 2024-04-16T02:35:36.460 INFO:tasks.workunit.client.1.smithi153.stdout:2/436: link d9/d2b/d18/d3f/l62 d9/d24/d34/d60/d7c/la0 0 2024-04-16T02:35:36.461 INFO:tasks.workunit.client.1.smithi153.stdout:1/425: mknod d4/dd/d18/d6c/c94 0 2024-04-16T02:35:36.462 INFO:tasks.workunit.client.1.smithi153.stdout:7/403: dwrite d0/d4/db/f64 [0,4194304] 0 2024-04-16T02:35:36.462 INFO:tasks.workunit.client.1.smithi153.stdout:4/524: unlink d3/d9/d24/d35/lb1 0 2024-04-16T02:35:36.462 INFO:tasks.workunit.client.1.smithi153.stdout:8/393: truncate d2/d1c/d1f/d27/f2a 739231 0 2024-04-16T02:35:36.464 INFO:tasks.workunit.client.1.smithi153.stdout:4/525: fdatasync d3/d9/db/d17/d3f/f94 0 2024-04-16T02:35:36.464 INFO:tasks.workunit.client.1.smithi153.stdout:7/404: chown d0/d4/db/d16/l7f 28541702 1 2024-04-16T02:35:36.466 INFO:tasks.workunit.client.1.smithi153.stdout:7/405: truncate d0/f1e 5563519 0 2024-04-16T02:35:36.467 INFO:tasks.workunit.client.1.smithi153.stdout:6/414: dread dc/d47/f61 [0,4194304] 0 2024-04-16T02:35:36.467 INFO:tasks.workunit.client.1.smithi153.stdout:8/394: unlink d2/d1c/d1f/l64 0 2024-04-16T02:35:36.470 INFO:tasks.workunit.client.1.smithi153.stdout:0/510: getdents dd/d32 0 2024-04-16T02:35:36.472 INFO:tasks.workunit.client.1.smithi153.stdout:1/426: mkdir d4/dd/d13/d8d/d95 0 2024-04-16T02:35:36.473 INFO:tasks.workunit.client.1.smithi153.stdout:7/406: creat d0/d13/f81 x:0 0 0 2024-04-16T02:35:36.475 INFO:tasks.workunit.client.1.smithi153.stdout:4/526: rename d3/d9/db/de/d2c/c62 to d3/d9/db/de/d2c/d98/cb8 0 2024-04-16T02:35:36.475 INFO:tasks.workunit.client.1.smithi153.stdout:7/407: truncate d0/d4/db/d34/d4b/f75 727419 0 2024-04-16T02:35:36.476 INFO:tasks.workunit.client.1.smithi153.stdout:2/437: dread d9/d24/d34/d36/f3b [0,4194304] 0 2024-04-16T02:35:36.476 INFO:tasks.workunit.client.1.smithi153.stdout:6/415: creat dc/d10/f88 x:0 0 0 2024-04-16T02:35:36.479 INFO:tasks.workunit.client.1.smithi153.stdout:0/511: mkdir dd/d14/d51/d9e 0 2024-04-16T02:35:36.479 INFO:tasks.workunit.client.1.smithi153.stdout:4/527: dread d3/d9/db/de/f26 [0,4194304] 0 2024-04-16T02:35:36.480 INFO:tasks.workunit.client.1.smithi153.stdout:1/427: chown d4/dd/d13/d23/f6d 1 1 2024-04-16T02:35:36.481 INFO:tasks.workunit.client.1.smithi153.stdout:2/438: symlink d9/d71/la1 0 2024-04-16T02:35:36.482 INFO:tasks.workunit.client.1.smithi153.stdout:1/428: write d4/dd/d13/d23/d5f/f69 [1466482,24640] 0 2024-04-16T02:35:36.482 INFO:tasks.workunit.client.1.smithi153.stdout:7/408: mknod d0/d4/c82 0 2024-04-16T02:35:36.485 INFO:tasks.workunit.client.1.smithi153.stdout:8/395: link d2/d1c/d1f/d27/f6e d2/d1c/d1f/d27/d2d/d47/d60/d67/f80 0 2024-04-16T02:35:36.486 INFO:tasks.workunit.client.1.smithi153.stdout:0/512: creat dd/d14/d51/d7f/d9d/f9f x:0 0 0 2024-04-16T02:35:36.486 INFO:tasks.workunit.client.1.smithi153.stdout:2/439: mkdir d9/d2b/d5c/da2 0 2024-04-16T02:35:36.490 INFO:tasks.workunit.client.1.smithi153.stdout:2/440: write d9/d24/d27/d2c/d41/f4c [4206988,54063] 0 2024-04-16T02:35:36.490 INFO:tasks.workunit.client.1.smithi153.stdout:7/409: mknod d0/d4/db/d16/c83 0 2024-04-16T02:35:36.491 INFO:tasks.workunit.client.1.smithi153.stdout:5/444: sync 2024-04-16T02:35:36.492 INFO:tasks.workunit.client.1.smithi153.stdout:4/528: getdents d3/d9/db/d17/d3f/d6b 0 2024-04-16T02:35:36.494 INFO:tasks.workunit.client.1.smithi153.stdout:5/445: chown d8/f4d 11 1 2024-04-16T02:35:36.499 INFO:tasks.workunit.client.1.smithi153.stdout:8/396: dwrite d2/d1c/d1f/d27/d5e/f73 [0,4194304] 0 2024-04-16T02:35:36.500 INFO:tasks.workunit.client.1.smithi153.stdout:4/529: mknod d3/d9/db/d84/cb9 0 2024-04-16T02:35:36.501 INFO:tasks.workunit.client.1.smithi153.stdout:5/446: symlink d8/df/d28/l86 0 2024-04-16T02:35:36.501 INFO:tasks.workunit.client.1.smithi153.stdout:1/429: dwrite d4/f40 [0,4194304] 0 2024-04-16T02:35:36.501 INFO:tasks.workunit.client.1.smithi153.stdout:0/513: dwrite dd/d14/d19/d1c/f4d [0,4194304] 0 2024-04-16T02:35:36.503 INFO:tasks.workunit.client.1.smithi153.stdout:5/447: chown d8/f7e 53949382 1 2024-04-16T02:35:36.503 INFO:tasks.workunit.client.1.smithi153.stdout:0/514: chown dd/d14/d19/d55/c9b 821387 1 2024-04-16T02:35:36.505 INFO:tasks.workunit.client.0.smithi151.stdout:9/56: mknod d1/c11 0 2024-04-16T02:35:36.507 INFO:tasks.workunit.client.0.smithi151.stdout:7/57: mknod d0/c10 0 2024-04-16T02:35:36.508 INFO:tasks.workunit.client.0.smithi151.stdout:8/77: link d11/f17 d11/f1a 0 2024-04-16T02:35:36.508 INFO:tasks.workunit.client.0.smithi151.stdout:5/84: fdatasync d2/d6/f8 0 2024-04-16T02:35:36.509 INFO:tasks.workunit.client.1.smithi153.stdout:5/448: mknod d8/df/d28/d5a/d72/c87 0 2024-04-16T02:35:36.510 INFO:tasks.workunit.client.0.smithi151.stdout:5/85: dread - d2/f18 zero size 2024-04-16T02:35:36.513 INFO:tasks.workunit.client.0.smithi151.stdout:5/86: fdatasync d2/f5 0 2024-04-16T02:35:36.513 INFO:tasks.workunit.client.1.smithi153.stdout:7/410: rename d0/d4/l2d to d0/d4/db/l84 0 2024-04-16T02:35:36.515 INFO:tasks.workunit.client.0.smithi151.stdout:1/35: rmdir d3 39 2024-04-16T02:35:36.516 INFO:tasks.workunit.client.1.smithi153.stdout:4/530: creat d3/d9/fba x:0 0 0 2024-04-16T02:35:36.517 INFO:tasks.workunit.client.0.smithi151.stdout:6/30: dread f1 [0,4194304] 0 2024-04-16T02:35:36.518 INFO:tasks.workunit.client.0.smithi151.stdout:3/72: rename d1/d6/c14 to d1/d6/d10/c19 0 2024-04-16T02:35:36.519 INFO:tasks.workunit.client.0.smithi151.stdout:9/57: mknod d1/d3/c12 0 2024-04-16T02:35:36.519 INFO:tasks.workunit.client.0.smithi151.stdout:3/73: chown d1/l4 1 1 2024-04-16T02:35:36.520 INFO:tasks.workunit.client.1.smithi153.stdout:1/430: rename d4/dd/f16 to d4/dd/d18/d57/f96 0 2024-04-16T02:35:36.520 INFO:tasks.workunit.client.0.smithi151.stdout:9/58: chown d1/d3 487 1 2024-04-16T02:35:36.520 INFO:tasks.workunit.client.0.smithi151.stdout:9/59: fdatasync d1/fc 0 2024-04-16T02:35:36.521 INFO:tasks.workunit.client.0.smithi151.stdout:2/66: creat de/d15/f17 x:0 0 0 2024-04-16T02:35:36.526 INFO:tasks.workunit.client.1.smithi153.stdout:8/397: link d2/f17 d2/d1c/d1f/d27/d2d/f81 0 2024-04-16T02:35:36.526 INFO:tasks.workunit.client.1.smithi153.stdout:2/441: truncate d9/d24/d27/f30 2581199 0 2024-04-16T02:35:36.526 INFO:tasks.workunit.client.0.smithi151.stdout:7/58: rename d0/ff to d0/f11 0 2024-04-16T02:35:36.526 INFO:tasks.workunit.client.0.smithi151.stdout:5/87: truncate d2/fb 321848 0 2024-04-16T02:35:36.527 INFO:tasks.workunit.client.1.smithi153.stdout:0/515: truncate dd/d14/d19/d77/d87/f91 2486818 0 2024-04-16T02:35:36.529 INFO:tasks.workunit.client.1.smithi153.stdout:1/431: mknod d4/dd/d18/d29/d71/c97 0 2024-04-16T02:35:36.531 INFO:tasks.workunit.client.1.smithi153.stdout:2/442: dwrite d9/d2b/d5c/f80 [0,4194304] 0 2024-04-16T02:35:36.531 INFO:tasks.workunit.client.1.smithi153.stdout:0/516: write dd/d14/f86 [4696399,20353] 0 2024-04-16T02:35:36.531 INFO:tasks.workunit.client.1.smithi153.stdout:5/449: creat d8/df/f88 x:0 0 0 2024-04-16T02:35:36.531 INFO:tasks.workunit.client.0.smithi151.stdout:7/59: rename d0/la to d0/l12 0 2024-04-16T02:35:36.532 INFO:tasks.workunit.client.1.smithi153.stdout:0/517: chown dd/d14/d19/d77/d87 359 1 2024-04-16T02:35:36.533 INFO:tasks.workunit.client.1.smithi153.stdout:5/450: truncate d8/d1f/f73 5203510 0 2024-04-16T02:35:36.535 INFO:tasks.workunit.client.0.smithi151.stdout:8/78: rmdir d11 39 2024-04-16T02:35:36.536 INFO:tasks.workunit.client.1.smithi153.stdout:1/432: unlink d4/dd/d18/f66 0 2024-04-16T02:35:36.536 INFO:tasks.workunit.client.0.smithi151.stdout:6/31: dwrite f1 [4194304,4194304] 0 2024-04-16T02:35:36.538 INFO:tasks.workunit.client.1.smithi153.stdout:8/398: mknod d2/d13/d3a/d5b/c82 0 2024-04-16T02:35:36.539 INFO:tasks.workunit.client.0.smithi151.stdout:7/60: dread d0/fe [0,4194304] 0 2024-04-16T02:35:36.543 INFO:tasks.workunit.client.1.smithi153.stdout:2/443: mkdir d9/d24/d27/d2c/d5d/da3 0 2024-04-16T02:35:36.543 INFO:tasks.workunit.client.1.smithi153.stdout:1/433: dread d4/dd/d13/d23/f5d [0,4194304] 0 2024-04-16T02:35:36.543 INFO:tasks.workunit.client.0.smithi151.stdout:2/67: creat de/d11/f18 x:0 0 0 2024-04-16T02:35:36.543 INFO:tasks.workunit.client.0.smithi151.stdout:2/68: chown de/d15 235 1 2024-04-16T02:35:36.544 INFO:tasks.workunit.client.1.smithi153.stdout:1/434: write d4/f49 [479428,63571] 0 2024-04-16T02:35:36.545 INFO:tasks.workunit.client.0.smithi151.stdout:2/69: chown de 952013 1 2024-04-16T02:35:36.545 INFO:tasks.workunit.client.0.smithi151.stdout:5/88: fdatasync d2/f12 0 2024-04-16T02:35:36.545 INFO:tasks.workunit.client.1.smithi153.stdout:7/411: getdents d0 0 2024-04-16T02:35:36.546 INFO:tasks.workunit.client.0.smithi151.stdout:1/36: mknod d3/dc/cd 0 2024-04-16T02:35:36.548 INFO:tasks.workunit.client.0.smithi151.stdout:8/79: fsync d11/f14 0 2024-04-16T02:35:36.551 INFO:tasks.workunit.client.1.smithi153.stdout:7/412: read d0/d4/db/d2b/d53/d59/d76/f7b [480666,29066] 0 2024-04-16T02:35:36.554 INFO:tasks.workunit.client.1.smithi153.stdout:4/531: getdents d3/d9/db/d17 0 2024-04-16T02:35:36.554 INFO:tasks.workunit.client.0.smithi151.stdout:6/32: rename f1 to d2/d5/f8 0 2024-04-16T02:35:36.555 INFO:tasks.workunit.client.0.smithi151.stdout:7/61: fdatasync d0/f6 0 2024-04-16T02:35:36.558 INFO:tasks.workunit.client.0.smithi151.stdout:8/80: dread ff [0,4194304] 0 2024-04-16T02:35:36.559 INFO:tasks.workunit.client.0.smithi151.stdout:1/37: dwrite f1 [0,4194304] 0 2024-04-16T02:35:36.559 INFO:tasks.workunit.client.1.smithi153.stdout:4/532: dread d3/d9/d24/d3e/f80 [0,4194304] 0 2024-04-16T02:35:36.560 INFO:tasks.workunit.client.1.smithi153.stdout:4/533: chown d3/d9/db/de/d2c/d38/l89 3 1 2024-04-16T02:35:36.561 INFO:tasks.workunit.client.0.smithi151.stdout:9/60: truncate d1/fc 1471364 0 2024-04-16T02:35:36.561 INFO:tasks.workunit.client.0.smithi151.stdout:1/38: dread - d3/f8 zero size 2024-04-16T02:35:36.562 INFO:tasks.workunit.client.1.smithi153.stdout:5/451: dread d8/d1f/d20/d39/f4c [0,4194304] 0 2024-04-16T02:35:36.563 INFO:tasks.workunit.client.1.smithi153.stdout:5/452: truncate d8/df/d2e/f3d 4348837 0 2024-04-16T02:35:36.564 INFO:tasks.workunit.client.1.smithi153.stdout:8/399: mknod d2/d13/c83 0 2024-04-16T02:35:36.566 INFO:tasks.workunit.client.0.smithi151.stdout:1/39: readlink d3/d4/la 0 2024-04-16T02:35:36.567 INFO:tasks.workunit.client.0.smithi151.stdout:1/40: write d3/f8 [162192,32847] 0 2024-04-16T02:35:36.571 INFO:tasks.workunit.client.0.smithi151.stdout:2/70: dread de/ff [0,4194304] 0 2024-04-16T02:35:36.571 INFO:tasks.workunit.client.0.smithi151.stdout:3/74: dwrite d1/d6/ff [0,4194304] 0 2024-04-16T02:35:36.571 INFO:tasks.workunit.client.1.smithi153.stdout:0/518: rename dd/d14/d51/d7f/d9d/f9f to dd/d14/d19/fa0 0 2024-04-16T02:35:36.572 INFO:tasks.workunit.client.0.smithi151.stdout:6/33: creat d2/f9 x:0 0 0 2024-04-16T02:35:36.574 INFO:tasks.workunit.client.0.smithi151.stdout:2/71: write f7 [5156143,48221] 0 2024-04-16T02:35:36.574 INFO:tasks.workunit.client.1.smithi153.stdout:5/453: creat d8/df/d2f/d4e/f89 x:0 0 0 2024-04-16T02:35:36.575 INFO:tasks.workunit.client.1.smithi153.stdout:5/454: chown d8/c43 204 1 2024-04-16T02:35:36.576 INFO:tasks.workunit.client.1.smithi153.stdout:7/413: mknod d0/d4/db/d2b/d43/d6d/d68/d72/c85 0 2024-04-16T02:35:36.577 INFO:tasks.workunit.client.1.smithi153.stdout:5/455: read - d8/df/d28/d5a/f68 zero size 2024-04-16T02:35:36.577 INFO:tasks.workunit.client.1.smithi153.stdout:3/395: sync 2024-04-16T02:35:36.578 INFO:tasks.workunit.client.1.smithi153.stdout:5/456: dread - d8/f59 zero size 2024-04-16T02:35:36.580 INFO:tasks.workunit.client.1.smithi153.stdout:0/519: truncate dd/d14/d51/f60 598190 0 2024-04-16T02:35:36.581 INFO:tasks.workunit.client.1.smithi153.stdout:8/400: mknod d2/d1c/d1f/d27/d2d/d47/d77/c84 0 2024-04-16T02:35:36.581 INFO:tasks.workunit.client.0.smithi151.stdout:1/41: rename d3/dc/cd to d3/d4/ce 0 2024-04-16T02:35:36.583 INFO:tasks.workunit.client.1.smithi153.stdout:2/444: dread d9/d24/d27/f6f [0,4194304] 0 2024-04-16T02:35:36.584 INFO:tasks.workunit.client.1.smithi153.stdout:7/414: creat d0/d4/db/d2b/d43/d6d/d68/f86 x:0 0 0 2024-04-16T02:35:36.584 INFO:tasks.workunit.client.1.smithi153.stdout:1/435: link d4/lb d4/dd/d18/d57/l98 0 2024-04-16T02:35:36.585 INFO:tasks.workunit.client.0.smithi151.stdout:9/61: creat d1/d3/d6/f13 x:0 0 0 2024-04-16T02:35:36.586 INFO:tasks.workunit.client.0.smithi151.stdout:9/62: chown d1/f5 1739320 1 2024-04-16T02:35:36.587 INFO:tasks.workunit.client.1.smithi153.stdout:7/415: fdatasync d0/d4/f58 0 2024-04-16T02:35:36.587 INFO:tasks.workunit.client.1.smithi153.stdout:8/401: creat d2/d1c/f85 x:0 0 0 2024-04-16T02:35:36.587 INFO:tasks.workunit.client.1.smithi153.stdout:4/534: link d3/d9/db/de/l1f d3/d9/db/de/lbb 0 2024-04-16T02:35:36.587 INFO:tasks.workunit.client.0.smithi151.stdout:3/75: rename d1/d6/fe to d1/d6/d10/f1a 0 2024-04-16T02:35:36.588 INFO:tasks.workunit.client.1.smithi153.stdout:0/520: dread dd/d32/d57/f89 [0,4194304] 0 2024-04-16T02:35:36.589 INFO:tasks.workunit.client.1.smithi153.stdout:2/445: symlink d9/d24/d27/d9a/la4 0 2024-04-16T02:35:36.590 INFO:tasks.workunit.client.0.smithi151.stdout:2/72: mknod de/c19 0 2024-04-16T02:35:36.590 INFO:tasks.workunit.client.1.smithi153.stdout:7/416: write d0/f36 [413817,32350] 0 2024-04-16T02:35:36.591 INFO:tasks.workunit.client.1.smithi153.stdout:8/402: dread - d2/d13/d2c/f48 zero size 2024-04-16T02:35:36.592 INFO:tasks.workunit.client.0.smithi151.stdout:5/89: getdents d2/d6 0 2024-04-16T02:35:36.593 INFO:tasks.workunit.client.0.smithi151.stdout:7/62: getdents d0 0 2024-04-16T02:35:36.594 INFO:tasks.workunit.client.1.smithi153.stdout:3/396: dwrite d6/d10/d5b/f75 [0,4194304] 0 2024-04-16T02:35:36.595 INFO:tasks.workunit.client.0.smithi151.stdout:9/63: symlink d1/d3/l14 0 2024-04-16T02:35:36.596 INFO:tasks.workunit.client.1.smithi153.stdout:4/535: mkdir d3/dbc 0 2024-04-16T02:35:36.597 INFO:tasks.workunit.client.0.smithi151.stdout:3/76: rmdir d1/d6/d10 39 2024-04-16T02:35:36.599 INFO:tasks.workunit.client.0.smithi151.stdout:1/42: mkdir d3/d4/db/df 0 2024-04-16T02:35:36.602 INFO:tasks.workunit.client.1.smithi153.stdout:3/397: chown d6/f11 283042446 1 2024-04-16T02:35:36.602 INFO:tasks.workunit.client.1.smithi153.stdout:5/457: dread d8/f4d [0,4194304] 0 2024-04-16T02:35:36.602 INFO:tasks.workunit.client.0.smithi151.stdout:7/63: mknod d0/c13 0 2024-04-16T02:35:36.602 INFO:tasks.workunit.client.1.smithi153.stdout:1/436: dwrite d4/dd/d13/d26/d58/f91 [0,4194304] 0 2024-04-16T02:35:36.606 INFO:tasks.workunit.client.0.smithi151.stdout:2/73: dwrite de/f14 [0,4194304] 0 2024-04-16T02:35:36.606 INFO:tasks.workunit.client.0.smithi151.stdout:9/64: symlink d1/d3/l15 0 2024-04-16T02:35:36.607 INFO:tasks.workunit.client.1.smithi153.stdout:4/536: rename d3/d9/db/d84/cb9 to d3/d9/db/d17/d28/cbd 0 2024-04-16T02:35:36.608 INFO:tasks.workunit.client.0.smithi151.stdout:5/90: mkdir d2/d6/d19/d1a 0 2024-04-16T02:35:36.609 INFO:tasks.workunit.client.1.smithi153.stdout:8/403: mkdir d2/d1c/d1f/d27/d2d/d47/d60/d67/d86 0 2024-04-16T02:35:36.611 INFO:tasks.workunit.client.0.smithi151.stdout:1/43: mkdir d3/d4/d5/d10 0 2024-04-16T02:35:36.612 INFO:tasks.workunit.client.0.smithi151.stdout:5/91: chown d2/f5 1205773 1 2024-04-16T02:35:36.612 INFO:tasks.workunit.client.1.smithi153.stdout:5/458: dwrite d8/df/d2e/f3d [4194304,4194304] 0 2024-04-16T02:35:36.613 INFO:tasks.workunit.client.0.smithi151.stdout:7/64: dread d0/f7 [0,4194304] 0 2024-04-16T02:35:36.615 INFO:tasks.workunit.client.1.smithi153.stdout:3/398: unlink d6/d10/d22/d2e/c64 0 2024-04-16T02:35:36.616 INFO:tasks.workunit.client.0.smithi151.stdout:8/81: dwrite d11/f1a [0,4194304] 0 2024-04-16T02:35:36.616 INFO:tasks.workunit.client.0.smithi151.stdout:2/74: creat de/d15/f1a x:0 0 0 2024-04-16T02:35:36.618 INFO:tasks.workunit.client.1.smithi153.stdout:8/404: mknod d2/d1c/d1f/d27/d2d/d47/d77/c87 0 2024-04-16T02:35:36.619 INFO:tasks.workunit.client.1.smithi153.stdout:3/399: truncate d6/d10/d22/f4c 1798132 0 2024-04-16T02:35:36.620 INFO:tasks.workunit.client.0.smithi151.stdout:3/77: dwrite d1/d6/ff [0,4194304] 0 2024-04-16T02:35:36.620 INFO:tasks.workunit.client.1.smithi153.stdout:5/459: creat d8/df/d28/f8a x:0 0 0 2024-04-16T02:35:36.622 INFO:tasks.workunit.client.1.smithi153.stdout:4/537: dwrite d3/d9/d79/fa6 [0,4194304] 0 2024-04-16T02:35:36.622 INFO:tasks.workunit.client.1.smithi153.stdout:5/460: dread - d8/df/f88 zero size 2024-04-16T02:35:36.623 INFO:tasks.workunit.client.0.smithi151.stdout:3/78: chown d1/lb 5235 1 2024-04-16T02:35:36.623 INFO:tasks.workunit.client.0.smithi151.stdout:9/65: dread d1/fc [0,4194304] 0 2024-04-16T02:35:36.626 INFO:tasks.workunit.client.0.smithi151.stdout:9/66: fdatasync d1/d3/fb 0 2024-04-16T02:35:36.628 INFO:tasks.workunit.client.0.smithi151.stdout:7/65: creat d0/f14 x:0 0 0 2024-04-16T02:35:36.629 INFO:tasks.workunit.client.0.smithi151.stdout:8/82: creat d11/f1b x:0 0 0 2024-04-16T02:35:36.631 INFO:tasks.workunit.client.0.smithi151.stdout:7/66: truncate d0/f14 677239 0 2024-04-16T02:35:36.631 INFO:tasks.workunit.client.0.smithi151.stdout:8/83: chown ff 86964978 1 2024-04-16T02:35:36.632 INFO:tasks.workunit.client.1.smithi153.stdout:7/417: dread d0/d4/db/d2b/d53/d59/d76/f7b [0,4194304] 0 2024-04-16T02:35:36.633 INFO:tasks.workunit.client.0.smithi151.stdout:5/92: dread d2/d6/f9 [0,4194304] 0 2024-04-16T02:35:36.633 INFO:tasks.workunit.client.0.smithi151.stdout:2/75: truncate de/ff 4402944 0 2024-04-16T02:35:36.633 INFO:tasks.workunit.client.1.smithi153.stdout:5/461: chown d8/df/d23/d62/c74 322 1 2024-04-16T02:35:36.636 INFO:tasks.workunit.client.0.smithi151.stdout:1/44: getdents d3/d4/db/df 0 2024-04-16T02:35:36.636 INFO:tasks.workunit.client.0.smithi151.stdout:3/79: write d1/fa [1401200,54748] 0 2024-04-16T02:35:36.641 INFO:tasks.workunit.client.0.smithi151.stdout:8/84: rename d11/f14 to d11/f1c 0 2024-04-16T02:35:36.642 INFO:tasks.workunit.client.0.smithi151.stdout:7/67: dwrite d0/f3 [0,4194304] 0 2024-04-16T02:35:36.643 INFO:tasks.workunit.client.0.smithi151.stdout:8/85: read d11/f1a [4192429,11264] 0 2024-04-16T02:35:36.645 INFO:tasks.workunit.client.0.smithi151.stdout:1/45: mknod d3/d4/c11 0 2024-04-16T02:35:36.646 INFO:tasks.workunit.client.0.smithi151.stdout:1/46: readlink d3/d4/la 0 2024-04-16T02:35:36.647 INFO:tasks.workunit.client.0.smithi151.stdout:2/76: symlink de/l1b 0 2024-04-16T02:35:36.650 INFO:tasks.workunit.client.0.smithi151.stdout:2/77: chown de/f14 336 1 2024-04-16T02:35:36.652 INFO:tasks.workunit.client.0.smithi151.stdout:7/68: dread d0/f5 [0,4194304] 0 2024-04-16T02:35:36.653 INFO:tasks.workunit.client.0.smithi151.stdout:5/93: link d2/fb d2/d6/d19/d1a/f1b 0 2024-04-16T02:35:36.655 INFO:tasks.workunit.client.0.smithi151.stdout:2/78: creat de/d15/f1c x:0 0 0 2024-04-16T02:35:36.656 INFO:tasks.workunit.client.0.smithi151.stdout:8/86: dwrite f3 [4194304,4194304] 0 2024-04-16T02:35:36.656 INFO:tasks.workunit.client.0.smithi151.stdout:7/69: fdatasync d0/f7 0 2024-04-16T02:35:36.658 INFO:tasks.workunit.client.0.smithi151.stdout:1/47: dread f1 [0,4194304] 0 2024-04-16T02:35:36.660 INFO:tasks.workunit.client.0.smithi151.stdout:2/79: symlink de/d15/l1d 0 2024-04-16T02:35:36.661 INFO:tasks.workunit.client.0.smithi151.stdout:7/70: dread d0/fe [0,4194304] 0 2024-04-16T02:35:36.663 INFO:tasks.workunit.client.0.smithi151.stdout:2/80: dread - de/d15/f1c zero size 2024-04-16T02:35:36.664 INFO:tasks.workunit.client.0.smithi151.stdout:1/48: creat d3/d4/d5/f12 x:0 0 0 2024-04-16T02:35:36.664 INFO:tasks.workunit.client.0.smithi151.stdout:3/80: dwrite d1/d6/d10/f1a [0,4194304] 0 2024-04-16T02:35:36.664 INFO:tasks.workunit.client.0.smithi151.stdout:7/71: read d0/f7 [225523,40358] 0 2024-04-16T02:35:36.665 INFO:tasks.workunit.client.0.smithi151.stdout:2/81: truncate f7 5769049 0 2024-04-16T02:35:36.665 INFO:tasks.workunit.client.0.smithi151.stdout:3/81: fdatasync d1/f17 0 2024-04-16T02:35:36.668 INFO:tasks.workunit.client.0.smithi151.stdout:1/49: symlink d3/d4/l13 0 2024-04-16T02:35:36.669 INFO:tasks.workunit.client.0.smithi151.stdout:3/82: dread d1/d6/d10/f15 [0,4194304] 0 2024-04-16T02:35:36.669 INFO:tasks.workunit.client.0.smithi151.stdout:7/72: mknod d0/c15 0 2024-04-16T02:35:36.671 INFO:tasks.workunit.client.0.smithi151.stdout:2/82: mkdir de/d15/d1e 0 2024-04-16T02:35:36.674 INFO:tasks.workunit.client.0.smithi151.stdout:2/83: write de/d15/f1a [802644,126890] 0 2024-04-16T02:35:36.674 INFO:tasks.workunit.client.0.smithi151.stdout:1/50: symlink d3/d4/d5/l14 0 2024-04-16T02:35:36.675 INFO:tasks.workunit.client.0.smithi151.stdout:2/84: stat c4 0 2024-04-16T02:35:36.676 INFO:tasks.workunit.client.0.smithi151.stdout:2/85: chown de/f14 7586070 1 2024-04-16T02:35:36.679 INFO:tasks.workunit.client.0.smithi151.stdout:3/83: dwrite d1/fa [0,4194304] 0 2024-04-16T02:35:36.679 INFO:tasks.workunit.client.0.smithi151.stdout:2/86: write de/d15/f17 [156581,41257] 0 2024-04-16T02:35:36.682 INFO:tasks.workunit.client.1.smithi153.stdout:6/416: sync 2024-04-16T02:35:36.682 INFO:tasks.workunit.client.1.smithi153.stdout:9/472: sync 2024-04-16T02:35:36.683 INFO:tasks.workunit.client.0.smithi151.stdout:2/87: mknod de/d15/c1f 0 2024-04-16T02:35:36.684 INFO:tasks.workunit.client.1.smithi153.stdout:0/521: dread dd/d32/d57/f5d [0,4194304] 0 2024-04-16T02:35:36.685 INFO:tasks.workunit.client.1.smithi153.stdout:9/473: dread - d4/d6/d1b/f90 zero size 2024-04-16T02:35:36.685 INFO:tasks.workunit.client.1.smithi153.stdout:4/538: rename d3/d9/db/fb6 to d3/d9/db/de/d2c/d52/fbe 0 2024-04-16T02:35:36.686 INFO:tasks.workunit.client.1.smithi153.stdout:7/418: write d0/d4/db/f50 [400148,40280] 0 2024-04-16T02:35:36.686 INFO:tasks.workunit.client.0.smithi151.stdout:3/84: write d1/d6/d10/f15 [584838,18587] 0 2024-04-16T02:35:36.689 INFO:tasks.workunit.client.1.smithi153.stdout:6/417: symlink dc/d47/d57/d77/d75/l89 0 2024-04-16T02:35:36.689 INFO:tasks.workunit.client.1.smithi153.stdout:5/462: symlink d8/df/d28/d5a/d5e/l8b 0 2024-04-16T02:35:36.691 INFO:tasks.workunit.client.0.smithi151.stdout:3/85: creat d1/d6/f1b x:0 0 0 2024-04-16T02:35:36.691 INFO:tasks.workunit.client.1.smithi153.stdout:0/522: write dd/d14/d19/d20/f88 [494292,10001] 0 2024-04-16T02:35:36.692 INFO:tasks.workunit.client.1.smithi153.stdout:2/446: dwrite d9/f9e [0,4194304] 0 2024-04-16T02:35:36.693 INFO:tasks.workunit.client.1.smithi153.stdout:9/474: unlink d4/d6/dc/d2d/d64/f8a 0 2024-04-16T02:35:36.693 INFO:tasks.workunit.client.1.smithi153.stdout:3/400: link d6/d10/c71 d6/d10/d5b/c8d 0 2024-04-16T02:35:36.694 INFO:tasks.workunit.client.1.smithi153.stdout:7/419: mkdir d0/d4/db/d87 0 2024-04-16T02:35:36.696 INFO:tasks.workunit.client.1.smithi153.stdout:0/523: write dd/d26/f5e [782118,77282] 0 2024-04-16T02:35:36.696 INFO:tasks.workunit.client.0.smithi151.stdout:3/86: fdatasync d1/d6/ff 0 2024-04-16T02:35:36.697 INFO:tasks.workunit.client.0.smithi151.stdout:1/51: fsync d3/d4/d5/f12 0 2024-04-16T02:35:36.699 INFO:tasks.workunit.client.1.smithi153.stdout:2/447: truncate d9/d2b/d1c/d20/f67 1006082 0 2024-04-16T02:35:36.700 INFO:tasks.workunit.client.0.smithi151.stdout:7/73: getdents d0 0 2024-04-16T02:35:36.700 INFO:tasks.workunit.client.1.smithi153.stdout:5/463: mknod d8/df/d84/c8c 0 2024-04-16T02:35:36.701 INFO:tasks.workunit.client.1.smithi153.stdout:0/524: stat dd/d32/d80/f93 0 2024-04-16T02:35:36.703 INFO:tasks.workunit.client.1.smithi153.stdout:2/448: mknod d9/d24/d34/d60/d7c/ca5 0 2024-04-16T02:35:36.706 INFO:tasks.workunit.client.0.smithi151.stdout:7/74: mknod d0/c16 0 2024-04-16T02:35:36.706 INFO:tasks.workunit.client.0.smithi151.stdout:3/87: dread d1/d6/d10/f15 [0,4194304] 0 2024-04-16T02:35:36.707 INFO:tasks.workunit.client.1.smithi153.stdout:2/449: read d9/d24/d34/d36/f3b [2786738,35932] 0 2024-04-16T02:35:36.707 INFO:tasks.workunit.client.1.smithi153.stdout:5/464: mkdir d8/d1f/d20/d8d 0 2024-04-16T02:35:36.707 INFO:tasks.workunit.client.1.smithi153.stdout:6/418: dread dc/d47/d57/f70 [0,4194304] 0 2024-04-16T02:35:36.707 INFO:tasks.workunit.client.1.smithi153.stdout:6/419: fsync dc/d1a/d2d/f43 0 2024-04-16T02:35:36.708 INFO:tasks.workunit.client.1.smithi153.stdout:0/525: symlink dd/d32/la1 0 2024-04-16T02:35:36.708 INFO:tasks.workunit.client.1.smithi153.stdout:0/526: chown dd/d32/d3f/f62 306717 1 2024-04-16T02:35:36.709 INFO:tasks.workunit.client.1.smithi153.stdout:0/527: chown dd/d14/d19/d55/c9b 814904530 1 2024-04-16T02:35:36.710 INFO:tasks.workunit.client.0.smithi151.stdout:1/52: truncate f1 51720 0 2024-04-16T02:35:36.711 INFO:tasks.workunit.client.1.smithi153.stdout:5/465: creat d8/df/d28/d5a/d72/f8e x:0 0 0 2024-04-16T02:35:36.712 INFO:tasks.workunit.client.0.smithi151.stdout:3/88: symlink d1/d6/l1c 0 2024-04-16T02:35:36.712 INFO:tasks.workunit.client.1.smithi153.stdout:5/466: chown d8/d1f/d20/d39/f4a 118940118 1 2024-04-16T02:35:36.713 INFO:tasks.workunit.client.1.smithi153.stdout:5/467: dread - d8/d1f/d20/f81 zero size 2024-04-16T02:35:36.714 INFO:tasks.workunit.client.0.smithi151.stdout:1/53: symlink d3/l15 0 2024-04-16T02:35:36.715 INFO:tasks.workunit.client.0.smithi151.stdout:3/89: creat d1/f1d x:0 0 0 2024-04-16T02:35:36.716 INFO:tasks.workunit.client.1.smithi153.stdout:4/539: dread d3/d9/f3d [0,4194304] 0 2024-04-16T02:35:36.716 INFO:tasks.workunit.client.0.smithi151.stdout:7/75: dwrite d0/f3 [0,4194304] 0 2024-04-16T02:35:36.716 INFO:tasks.workunit.client.1.smithi153.stdout:0/528: rmdir dd/d26/d92 39 2024-04-16T02:35:36.717 INFO:tasks.workunit.client.1.smithi153.stdout:0/529: chown dd/d14/d19/d20/d4a/f78 132925029 1 2024-04-16T02:35:36.720 INFO:tasks.workunit.client.0.smithi151.stdout:2/88: dwrite de/ff [0,4194304] 0 2024-04-16T02:35:36.720 INFO:tasks.workunit.client.1.smithi153.stdout:7/420: rename d0/d4/db/d16/c3b to d0/d4/db/c88 0 2024-04-16T02:35:36.720 INFO:tasks.workunit.client.1.smithi153.stdout:9/475: write d4/d6/dc/d2d/f58 [992633,108018] 0 2024-04-16T02:35:36.721 INFO:tasks.workunit.client.1.smithi153.stdout:9/476: readlink d4/d6/d1b/l36 0 2024-04-16T02:35:36.721 INFO:tasks.workunit.client.1.smithi153.stdout:7/421: write d0/d13/f81 [684795,62281] 0 2024-04-16T02:35:36.722 INFO:tasks.workunit.client.1.smithi153.stdout:5/468: mknod d8/df/d2e/c8f 0 2024-04-16T02:35:36.725 INFO:tasks.workunit.client.0.smithi151.stdout:2/89: dread f8 [0,4194304] 0 2024-04-16T02:35:36.726 INFO:tasks.workunit.client.0.smithi151.stdout:7/76: getdents d0 0 2024-04-16T02:35:36.727 INFO:tasks.workunit.client.0.smithi151.stdout:7/77: write d0/f7 [41777,31505] 0 2024-04-16T02:35:36.727 INFO:tasks.workunit.client.1.smithi153.stdout:9/477: write d4/d33/d3a/d40/f6d [764953,44838] 0 2024-04-16T02:35:36.727 INFO:tasks.workunit.client.1.smithi153.stdout:9/478: chown d4/d33/f38 47242 1 2024-04-16T02:35:36.729 INFO:tasks.workunit.client.0.smithi151.stdout:2/90: unlink lc 0 2024-04-16T02:35:36.730 INFO:tasks.workunit.client.1.smithi153.stdout:4/540: rmdir d3/d9/db/de/d2c/d38 39 2024-04-16T02:35:36.732 INFO:tasks.workunit.client.1.smithi153.stdout:3/401: dwrite d6/d10/d22/d43/f69 [0,4194304] 0 2024-04-16T02:35:36.732 INFO:tasks.workunit.client.0.smithi151.stdout:7/78: unlink d0/fe 0 2024-04-16T02:35:36.732 INFO:tasks.workunit.client.1.smithi153.stdout:7/422: creat d0/d4/db/d2b/f89 x:0 0 0 2024-04-16T02:35:36.739 INFO:tasks.workunit.client.1.smithi153.stdout:7/423: write d0/d4/db/f50 [1005721,118371] 0 2024-04-16T02:35:36.741 INFO:tasks.workunit.client.0.smithi151.stdout:2/91: dwrite de/d11/f18 [0,4194304] 0 2024-04-16T02:35:36.741 INFO:tasks.workunit.client.1.smithi153.stdout:2/450: dread d9/d2b/f17 [0,4194304] 0 2024-04-16T02:35:36.742 INFO:tasks.workunit.client.1.smithi153.stdout:9/479: chown d4/d33/d3a/d40/d4a/f72 2327282 1 2024-04-16T02:35:36.742 INFO:tasks.workunit.client.1.smithi153.stdout:3/402: chown d6/lf 419 1 2024-04-16T02:35:36.743 INFO:tasks.workunit.client.0.smithi151.stdout:7/79: dwrite d0/f7 [0,4194304] 0 2024-04-16T02:35:36.745 INFO:tasks.workunit.client.1.smithi153.stdout:0/530: dread dd/f10 [0,4194304] 0 2024-04-16T02:35:36.745 INFO:tasks.workunit.client.1.smithi153.stdout:4/541: rename d3/d9/db/d17/d3f/d6b/c7b to d3/d9/db/d84/cbf 0 2024-04-16T02:35:36.745 INFO:tasks.workunit.client.1.smithi153.stdout:7/424: chown d0/d4/db/d2b/d43/d6d/d68/d6c 30119 1 2024-04-16T02:35:36.747 INFO:tasks.workunit.client.1.smithi153.stdout:4/542: write d3/d9/fba [67515,609] 0 2024-04-16T02:35:36.748 INFO:tasks.workunit.client.1.smithi153.stdout:5/469: unlink d8/df/f24 0 2024-04-16T02:35:36.752 INFO:tasks.workunit.client.1.smithi153.stdout:1/437: sync 2024-04-16T02:35:36.754 INFO:tasks.workunit.client.0.smithi151.stdout:2/92: dread f8 [0,4194304] 0 2024-04-16T02:35:36.754 INFO:tasks.workunit.client.1.smithi153.stdout:9/480: rename d4/d33/d3a to d4/d6/dc/d46/d93 0 2024-04-16T02:35:36.756 INFO:tasks.workunit.client.0.smithi151.stdout:1/54: dwrite f1 [0,4194304] 0 2024-04-16T02:35:36.757 INFO:tasks.workunit.client.1.smithi153.stdout:6/420: dwrite dc/d47/d57/d77/f24 [0,4194304] 0 2024-04-16T02:35:36.758 INFO:tasks.workunit.client.0.smithi151.stdout:2/93: write f8 [529668,8242] 0 2024-04-16T02:35:36.759 INFO:tasks.workunit.client.0.smithi151.stdout:2/94: fsync de/d15/f17 0 2024-04-16T02:35:36.759 INFO:tasks.workunit.client.1.smithi153.stdout:5/470: mkdir d8/d41/d90 0 2024-04-16T02:35:36.764 INFO:tasks.workunit.client.1.smithi153.stdout:5/471: chown d8/d1f/d20/d83 555294247 1 2024-04-16T02:35:36.765 INFO:tasks.workunit.client.1.smithi153.stdout:7/425: link d0/d4/db/c7e d0/d13/c8a 0 2024-04-16T02:35:36.770 INFO:tasks.workunit.client.0.smithi151.stdout:1/55: dwrite d3/d4/d5/f12 [0,4194304] 0 2024-04-16T02:35:36.770 INFO:tasks.workunit.client.1.smithi153.stdout:1/438: creat d4/f99 x:0 0 0 2024-04-16T02:35:36.772 INFO:tasks.workunit.client.0.smithi151.stdout:1/56: readlink d3/d4/la 0 2024-04-16T02:35:36.774 INFO:tasks.workunit.client.1.smithi153.stdout:2/451: dwrite d9/d2b/d1c/f90 [0,4194304] 0 2024-04-16T02:35:36.774 INFO:tasks.workunit.client.0.smithi151.stdout:2/95: dwrite de/d15/f1c [0,4194304] 0 2024-04-16T02:35:36.778 INFO:tasks.workunit.client.1.smithi153.stdout:0/531: getdents dd/d14/d51/d54 0 2024-04-16T02:35:36.779 INFO:tasks.workunit.client.1.smithi153.stdout:3/403: dwrite d6/d10/d22/d2e/f35 [4194304,4194304] 0 2024-04-16T02:35:36.780 INFO:tasks.workunit.client.1.smithi153.stdout:9/481: dwrite d4/d6/dc/d24/f30 [0,4194304] 0 2024-04-16T02:35:36.781 INFO:tasks.workunit.client.1.smithi153.stdout:8/405: sync 2024-04-16T02:35:36.782 INFO:tasks.workunit.client.1.smithi153.stdout:5/472: rmdir d8/df/d2f/d4e 39 2024-04-16T02:35:36.785 INFO:tasks.workunit.client.0.smithi151.stdout:1/57: dread d3/d4/d5/f12 [0,4194304] 0 2024-04-16T02:35:36.788 INFO:tasks.workunit.client.1.smithi153.stdout:5/473: readlink d8/df/d23/l3e 0 2024-04-16T02:35:36.788 INFO:tasks.workunit.client.0.smithi151.stdout:2/96: rename de/d11/c12 to de/c20 0 2024-04-16T02:35:36.790 INFO:tasks.workunit.client.0.smithi151.stdout:2/97: chown c4 1 1 2024-04-16T02:35:36.791 INFO:tasks.workunit.client.0.smithi151.stdout:7/80: dwrite d0/f7 [4194304,4194304] 0 2024-04-16T02:35:36.791 INFO:tasks.workunit.client.0.smithi151.stdout:1/58: rename d3/d4/d5 to d3/d4/db/d16 0 2024-04-16T02:35:36.792 INFO:tasks.workunit.client.1.smithi153.stdout:2/452: fdatasync d9/d24/d34/d36/f3b 0 2024-04-16T02:35:36.793 INFO:tasks.workunit.client.1.smithi153.stdout:5/474: stat d8/d41 0 2024-04-16T02:35:36.793 INFO:tasks.workunit.client.1.smithi153.stdout:4/543: getdents d3/d9/db/de/d2c/d52 0 2024-04-16T02:35:36.799 INFO:tasks.workunit.client.1.smithi153.stdout:7/426: dwrite d0/d13/f4a [0,4194304] 0 2024-04-16T02:35:36.802 INFO:tasks.workunit.client.1.smithi153.stdout:2/453: chown d9/d2b/d1c/d20/d88/f9b 103780 1 2024-04-16T02:35:36.803 INFO:tasks.workunit.client.1.smithi153.stdout:3/404: chown d6/f2d 381508 1 2024-04-16T02:35:36.805 INFO:tasks.workunit.client.1.smithi153.stdout:2/454: fdatasync d9/d71/d8d/f91 0 2024-04-16T02:35:36.806 INFO:tasks.workunit.client.1.smithi153.stdout:0/532: dwrite dd/d14/d19/d20/f70 [0,4194304] 0 2024-04-16T02:35:36.807 INFO:tasks.workunit.client.1.smithi153.stdout:2/455: write d9/d2b/d18/d3f/f6e [4532535,7003] 0 2024-04-16T02:35:36.807 INFO:tasks.workunit.client.1.smithi153.stdout:0/533: truncate dd/f6f 1680670 0 2024-04-16T02:35:36.808 INFO:tasks.workunit.client.1.smithi153.stdout:0/534: dread - dd/d32/d80/f93 zero size 2024-04-16T02:35:36.810 INFO:tasks.workunit.client.0.smithi151.stdout:0/158: sync 2024-04-16T02:35:36.813 INFO:tasks.workunit.client.0.smithi151.stdout:2/98: mknod de/d11/c21 0 2024-04-16T02:35:36.815 INFO:tasks.workunit.client.1.smithi153.stdout:9/482: write d4/d6/d1b/d6e/f74 [10107982,42425] 0 2024-04-16T02:35:36.817 INFO:tasks.workunit.client.0.smithi151.stdout:1/59: read d3/f8 [164483,28584] 0 2024-04-16T02:35:36.817 INFO:tasks.workunit.client.0.smithi151.stdout:7/81: write d0/f3 [1126777,69768] 0 2024-04-16T02:35:36.818 INFO:tasks.workunit.client.0.smithi151.stdout:7/82: readlink d0/l8 0 2024-04-16T02:35:36.819 INFO:tasks.workunit.client.0.smithi151.stdout:0/159: read - dc/d15/f29 zero size 2024-04-16T02:35:36.820 INFO:tasks.workunit.client.1.smithi153.stdout:7/427: chown d0/d4/db/l84 446 1 2024-04-16T02:35:36.820 INFO:tasks.workunit.client.1.smithi153.stdout:5/475: mknod d8/df/d28/d5a/d5e/c91 0 2024-04-16T02:35:36.822 INFO:tasks.workunit.client.1.smithi153.stdout:7/428: write d0/d13/f32 [2326327,46738] 0 2024-04-16T02:35:36.822 INFO:tasks.workunit.client.1.smithi153.stdout:5/476: stat d8/df/d28/d5a/d72/c85 0 2024-04-16T02:35:36.823 INFO:tasks.workunit.client.1.smithi153.stdout:4/544: dread d3/d9/db/de/f26 [0,4194304] 0 2024-04-16T02:35:36.824 INFO:tasks.workunit.client.0.smithi151.stdout:2/99: getdents de/d15/d1e 0 2024-04-16T02:35:36.824 INFO:tasks.workunit.client.1.smithi153.stdout:4/545: fdatasync d3/d9/db/d17/d3f/f94 0 2024-04-16T02:35:36.827 INFO:tasks.workunit.client.0.smithi151.stdout:1/60: rename d3/d4/la to d3/d4/db/d16/l17 0 2024-04-16T02:35:36.829 INFO:tasks.workunit.client.1.smithi153.stdout:1/439: truncate d4/dd/d20/f5b 514672 0 2024-04-16T02:35:36.832 INFO:tasks.workunit.client.0.smithi151.stdout:2/100: mknod de/c22 0 2024-04-16T02:35:36.833 INFO:tasks.workunit.client.0.smithi151.stdout:2/101: chown c4 106957253 1 2024-04-16T02:35:36.834 INFO:tasks.workunit.client.0.smithi151.stdout:7/83: dwrite d0/f5 [0,4194304] 0 2024-04-16T02:35:36.835 INFO:tasks.workunit.client.1.smithi153.stdout:7/429: mkdir d0/d4/db/d2b/d53/d59/d5c/d8b 0 2024-04-16T02:35:36.835 INFO:tasks.workunit.client.0.smithi151.stdout:0/160: dwrite dc/f2f [0,4194304] 0 2024-04-16T02:35:36.836 INFO:tasks.workunit.client.0.smithi151.stdout:1/61: getdents d3/dc 0 2024-04-16T02:35:36.838 INFO:tasks.workunit.client.0.smithi151.stdout:7/84: mknod d0/c17 0 2024-04-16T02:35:36.838 INFO:tasks.workunit.client.1.smithi153.stdout:2/456: truncate d9/d24/d34/d36/f3b 4056908 0 2024-04-16T02:35:36.839 INFO:tasks.workunit.client.1.smithi153.stdout:9/483: mkdir d4/d94 0 2024-04-16T02:35:36.840 INFO:tasks.workunit.client.0.smithi151.stdout:2/102: symlink de/d15/l23 0 2024-04-16T02:35:36.841 INFO:tasks.workunit.client.0.smithi151.stdout:2/103: write de/d15/f1a [458549,37652] 0 2024-04-16T02:35:36.842 INFO:tasks.workunit.client.1.smithi153.stdout:8/406: getdents d2/d13 0 2024-04-16T02:35:36.843 INFO:tasks.workunit.client.1.smithi153.stdout:3/405: rmdir d6/d10/d5b/d81 0 2024-04-16T02:35:36.845 INFO:tasks.workunit.client.0.smithi151.stdout:2/104: dread de/d15/f17 [0,4194304] 0 2024-04-16T02:35:36.846 INFO:tasks.workunit.client.1.smithi153.stdout:7/430: mkdir d0/d4/db/d2b/d8c 0 2024-04-16T02:35:36.848 INFO:tasks.workunit.client.1.smithi153.stdout:1/440: creat d4/dd/d18/d57/d65/d82/f9a x:0 0 0 2024-04-16T02:35:36.848 INFO:tasks.workunit.client.1.smithi153.stdout:2/457: dread d9/d2b/f17 [0,4194304] 0 2024-04-16T02:35:36.848 INFO:tasks.workunit.client.1.smithi153.stdout:8/407: rmdir d2/d13/d3a/d5b 39 2024-04-16T02:35:36.849 INFO:tasks.workunit.client.0.smithi151.stdout:2/105: creat de/d15/d1e/f24 x:0 0 0 2024-04-16T02:35:36.852 INFO:tasks.workunit.client.0.smithi151.stdout:2/106: chown de/c10 707609 1 2024-04-16T02:35:36.852 INFO:tasks.workunit.client.1.smithi153.stdout:9/484: link d4/d6/dc/d24/d4e/f81 d4/d6/d8c/f95 0 2024-04-16T02:35:36.853 INFO:tasks.workunit.client.1.smithi153.stdout:2/458: stat d9/d2b/l11 0 2024-04-16T02:35:36.854 INFO:tasks.workunit.client.0.smithi151.stdout:2/107: write de/f14 [3186061,5284] 0 2024-04-16T02:35:36.854 INFO:tasks.workunit.client.1.smithi153.stdout:5/477: rmdir d8/df/d28/d5a 39 2024-04-16T02:35:36.855 INFO:tasks.workunit.client.1.smithi153.stdout:9/485: fdatasync d4/d6/dc/d46/d93/d40/d4a/f80 0 2024-04-16T02:35:36.855 INFO:tasks.workunit.client.1.smithi153.stdout:2/459: fsync d9/d24/d27/d9a/f9f 0 2024-04-16T02:35:36.856 INFO:tasks.workunit.client.1.smithi153.stdout:4/546: getdents d3/d9/db/d17 0 2024-04-16T02:35:36.857 INFO:tasks.workunit.client.0.smithi151.stdout:2/108: mkdir de/d15/d1e/d25 0 2024-04-16T02:35:36.857 INFO:tasks.workunit.client.1.smithi153.stdout:7/431: dwrite d0/d13/f81 [0,4194304] 0 2024-04-16T02:35:36.858 INFO:tasks.workunit.client.1.smithi153.stdout:1/441: mknod d4/dd/d13/d23/d31/c9b 0 2024-04-16T02:35:36.859 INFO:tasks.workunit.client.1.smithi153.stdout:8/408: symlink d2/d1c/d1f/d27/d2d/d47/d60/d67/l88 0 2024-04-16T02:35:36.859 INFO:tasks.workunit.client.1.smithi153.stdout:7/432: stat d0/d4/db/d2b/f3e 0 2024-04-16T02:35:36.861 INFO:tasks.workunit.client.0.smithi151.stdout:2/109: dread f8 [0,4194304] 0 2024-04-16T02:35:36.863 INFO:tasks.workunit.client.0.smithi151.stdout:2/110: write de/d15/d1e/f24 [336788,91970] 0 2024-04-16T02:35:36.864 INFO:tasks.workunit.client.1.smithi153.stdout:4/547: write d3/d9/f3d [390064,90011] 0 2024-04-16T02:35:36.864 INFO:tasks.workunit.client.0.smithi151.stdout:2/111: chown de/d15/f1c 0 1 2024-04-16T02:35:36.866 INFO:tasks.workunit.client.1.smithi153.stdout:9/486: dread d4/d6/dc/d24/f30 [0,4194304] 0 2024-04-16T02:35:36.866 INFO:tasks.workunit.client.0.smithi151.stdout:2/112: write de/d15/f1c [3658602,74787] 0 2024-04-16T02:35:36.867 INFO:tasks.workunit.client.1.smithi153.stdout:5/478: mkdir d8/df/d28/d5a/d66/d92 0 2024-04-16T02:35:36.869 INFO:tasks.workunit.client.1.smithi153.stdout:1/442: write d4/dd/d13/d26/f2b [2041531,2132] 0 2024-04-16T02:35:36.869 INFO:tasks.workunit.client.1.smithi153.stdout:5/479: readlink d8/df/d23/l31 0 2024-04-16T02:35:36.880 INFO:tasks.workunit.client.0.smithi151.stdout:2/113: getdents de/d15 0 2024-04-16T02:35:36.884 INFO:tasks.workunit.client.0.smithi151.stdout:7/85: write d0/fb [1611883,64822] 0 2024-04-16T02:35:36.884 INFO:tasks.workunit.client.0.smithi151.stdout:1/62: truncate f1 1319396 0 2024-04-16T02:35:36.885 INFO:tasks.workunit.client.0.smithi151.stdout:2/114: chown de/ff 62762 1 2024-04-16T02:35:36.888 INFO:tasks.workunit.client.0.smithi151.stdout:1/63: dread d3/f8 [0,4194304] 0 2024-04-16T02:35:36.889 INFO:tasks.workunit.client.0.smithi151.stdout:2/115: rename de/d15/l1d to de/d11/l26 0 2024-04-16T02:35:36.890 INFO:tasks.workunit.client.0.smithi151.stdout:2/116: stat de/c10 0 2024-04-16T02:35:36.891 INFO:tasks.workunit.client.0.smithi151.stdout:0/161: dwrite fa [0,4194304] 0 2024-04-16T02:35:36.892 INFO:tasks.workunit.client.0.smithi151.stdout:2/117: write de/d11/f18 [1178396,106298] 0 2024-04-16T02:35:36.897 INFO:tasks.workunit.client.0.smithi151.stdout:1/64: rmdir d3/dc 0 2024-04-16T02:35:36.898 INFO:tasks.workunit.client.0.smithi151.stdout:0/162: dread dc/f1f [0,4194304] 0 2024-04-16T02:35:36.901 INFO:tasks.workunit.client.0.smithi151.stdout:0/163: dread dc/d15/f19 [0,4194304] 0 2024-04-16T02:35:36.901 INFO:tasks.workunit.client.0.smithi151.stdout:7/86: truncate d0/f9 1782865 0 2024-04-16T02:35:36.904 INFO:tasks.workunit.client.0.smithi151.stdout:1/65: creat d3/d4/db/d16/d10/f18 x:0 0 0 2024-04-16T02:35:36.907 INFO:tasks.workunit.client.0.smithi151.stdout:2/118: getdents de/d15/d1e 0 2024-04-16T02:35:36.908 INFO:tasks.workunit.client.1.smithi153.stdout:7/433: mkdir d0/d4/db/d47/d8d 0 2024-04-16T02:35:36.908 INFO:tasks.workunit.client.1.smithi153.stdout:2/460: dread d9/d24/d27/f68 [0,4194304] 0 2024-04-16T02:35:36.911 INFO:tasks.workunit.client.0.smithi151.stdout:7/87: rename d0/f9 to d0/f18 0 2024-04-16T02:35:36.912 INFO:tasks.workunit.client.1.smithi153.stdout:6/421: dread dc/d1a/f58 [0,4194304] 0 2024-04-16T02:35:36.912 INFO:tasks.workunit.client.0.smithi151.stdout:7/88: stat d0/c17 0 2024-04-16T02:35:36.914 INFO:tasks.workunit.client.0.smithi151.stdout:7/89: chown d0/f11 1 1 2024-04-16T02:35:36.915 INFO:tasks.workunit.client.1.smithi153.stdout:3/406: truncate d6/d10/d5b/f75 1598034 0 2024-04-16T02:35:36.916 INFO:tasks.workunit.client.0.smithi151.stdout:1/66: dwrite d3/d4/db/d16/d10/f18 [0,4194304] 0 2024-04-16T02:35:36.916 INFO:tasks.workunit.client.1.smithi153.stdout:4/548: dwrite d3/d9/f8d [0,4194304] 0 2024-04-16T02:35:36.916 INFO:tasks.workunit.client.1.smithi153.stdout:1/443: dread d4/dd/d13/d23/f6d [0,4194304] 0 2024-04-16T02:35:36.917 INFO:tasks.workunit.client.0.smithi151.stdout:2/119: rename f8 to de/f27 0 2024-04-16T02:35:36.917 INFO:tasks.workunit.client.1.smithi153.stdout:3/407: chown d6/d10/d22/d4d 18 1 2024-04-16T02:35:36.917 INFO:tasks.workunit.client.1.smithi153.stdout:9/487: write d4/d6/d8c/f95 [16922,104533] 0 2024-04-16T02:35:36.919 INFO:tasks.workunit.client.1.smithi153.stdout:2/461: fdatasync d9/d2b/ff 0 2024-04-16T02:35:36.919 INFO:tasks.workunit.client.1.smithi153.stdout:1/444: fsync d4/f22 0 2024-04-16T02:35:36.922 INFO:tasks.workunit.client.1.smithi153.stdout:6/422: unlink dc/d47/d57/d77/d15/f5a 0 2024-04-16T02:35:36.923 INFO:tasks.workunit.client.0.smithi151.stdout:1/67: mkdir d3/d4/db/d19 0 2024-04-16T02:35:36.923 INFO:tasks.workunit.client.0.smithi151.stdout:0/164: dwrite dc/f11 [4194304,4194304] 0 2024-04-16T02:35:36.925 INFO:tasks.workunit.client.1.smithi153.stdout:5/480: unlink d8/df/l45 0 2024-04-16T02:35:36.927 INFO:tasks.workunit.client.1.smithi153.stdout:9/488: write d4/d6/dc/d2d/f53 [3239891,101445] 0 2024-04-16T02:35:36.928 INFO:tasks.workunit.client.0.smithi151.stdout:0/165: readlink dc/l14 0 2024-04-16T02:35:36.928 INFO:tasks.workunit.client.1.smithi153.stdout:6/423: truncate dc/d6b/f6f 497434 0 2024-04-16T02:35:36.929 INFO:tasks.workunit.client.1.smithi153.stdout:1/445: truncate d4/dd/d18/f76 888238 0 2024-04-16T02:35:36.930 INFO:tasks.workunit.client.0.smithi151.stdout:2/120: rmdir de/d15/d1e/d25 0 2024-04-16T02:35:36.930 INFO:tasks.workunit.client.1.smithi153.stdout:5/481: write d8/df/d2e/f77 [755641,53832] 0 2024-04-16T02:35:36.932 INFO:tasks.workunit.client.1.smithi153.stdout:1/446: dread - d4/dd/d18/d29/f59 zero size 2024-04-16T02:35:36.932 INFO:tasks.workunit.client.0.smithi151.stdout:1/68: symlink d3/d4/db/d19/l1a 0 2024-04-16T02:35:36.933 INFO:tasks.workunit.client.1.smithi153.stdout:4/549: dwrite d3/d9/d24/f2b [0,4194304] 0 2024-04-16T02:35:36.933 INFO:tasks.workunit.client.1.smithi153.stdout:1/447: dread - d4/dd/d18/f53 zero size 2024-04-16T02:35:36.937 INFO:tasks.workunit.client.1.smithi153.stdout:5/482: dwrite d8/d1f/f73 [0,4194304] 0 2024-04-16T02:35:36.938 INFO:tasks.workunit.client.1.smithi153.stdout:3/408: rename d6/d10/d3b/d59/f86 to d6/d10/d3b/d59/d7c/f8e 0 2024-04-16T02:35:36.938 INFO:tasks.workunit.client.1.smithi153.stdout:2/462: symlink d9/d24/d73/la6 0 2024-04-16T02:35:36.939 INFO:tasks.workunit.client.1.smithi153.stdout:3/409: stat d6/c17 0 2024-04-16T02:35:36.942 INFO:tasks.workunit.client.1.smithi153.stdout:9/489: dread d4/d6/dc/d46/f55 [0,4194304] 0 2024-04-16T02:35:36.942 INFO:tasks.workunit.client.0.smithi151.stdout:1/69: getdents d3/d4 0 2024-04-16T02:35:36.943 INFO:tasks.workunit.client.1.smithi153.stdout:5/483: creat d8/df/d23/f93 x:0 0 0 2024-04-16T02:35:36.946 INFO:tasks.workunit.client.0.smithi151.stdout:2/121: dwrite de/ff [4194304,4194304] 0 2024-04-16T02:35:36.948 INFO:tasks.workunit.client.1.smithi153.stdout:3/410: fdatasync d6/d10/d3b/d59/d7c/f8e 0 2024-04-16T02:35:36.951 INFO:tasks.workunit.client.0.smithi151.stdout:1/70: rmdir d3/d4/db/d16/d10 39 2024-04-16T02:35:36.951 INFO:tasks.workunit.client.1.smithi153.stdout:4/550: link d3/d9/db/f65 d3/d9/db/de/d2c/d52/d85/fc0 0 2024-04-16T02:35:36.953 INFO:tasks.workunit.client.1.smithi153.stdout:4/551: dread - d3/d9/db/d17/d28/f9c zero size 2024-04-16T02:35:36.958 INFO:tasks.workunit.client.1.smithi153.stdout:2/463: creat d9/d24/d27/d2c/d5d/da3/fa7 x:0 0 0 2024-04-16T02:35:36.959 INFO:tasks.workunit.client.1.smithi153.stdout:7/434: dread d0/d4/db/d2b/d53/d59/f67 [0,4194304] 0 2024-04-16T02:35:36.960 INFO:tasks.workunit.client.0.smithi151.stdout:2/122: dwrite de/f27 [8388608,4194304] 0 2024-04-16T02:35:36.963 INFO:tasks.workunit.client.1.smithi153.stdout:9/490: mknod d4/c96 0 2024-04-16T02:35:36.963 INFO:tasks.workunit.client.0.smithi151.stdout:1/71: write d3/d4/db/d16/f12 [3763069,128508] 0 2024-04-16T02:35:36.963 INFO:tasks.workunit.client.1.smithi153.stdout:0/535: sync 2024-04-16T02:35:36.965 INFO:tasks.workunit.client.0.smithi151.stdout:7/90: dread d0/f18 [0,4194304] 0 2024-04-16T02:35:36.966 INFO:tasks.workunit.client.1.smithi153.stdout:4/552: dwrite d3/d9/f3d [4194304,4194304] 0 2024-04-16T02:35:36.966 INFO:tasks.workunit.client.1.smithi153.stdout:5/484: rename d8/df/d28/d5a/d5e/f76 to d8/df/d2f/d4e/f94 0 2024-04-16T02:35:36.967 INFO:tasks.workunit.client.1.smithi153.stdout:9/491: dread - d4/d6/dc/d24/f89 zero size 2024-04-16T02:35:36.968 INFO:tasks.workunit.client.0.smithi151.stdout:1/72: chown d3/d4/db/d16/l17 3020767 1 2024-04-16T02:35:36.968 INFO:tasks.workunit.client.0.smithi151.stdout:2/123: mkdir de/d15/d1e/d28 0 2024-04-16T02:35:36.969 INFO:tasks.workunit.client.0.smithi151.stdout:0/166: dwrite dc/ff [0,4194304] 0 2024-04-16T02:35:36.969 INFO:tasks.workunit.client.0.smithi151.stdout:1/73: stat d3/c7 0 2024-04-16T02:35:36.971 INFO:tasks.workunit.client.0.smithi151.stdout:2/124: chown l5 22 1 2024-04-16T02:35:36.972 INFO:tasks.workunit.client.0.smithi151.stdout:0/167: readlink dc/d15/l31 0 2024-04-16T02:35:36.972 INFO:tasks.workunit.client.1.smithi153.stdout:9/492: write d4/d33/f4f [27775,128417] 0 2024-04-16T02:35:36.972 INFO:tasks.workunit.client.1.smithi153.stdout:0/536: write dd/d14/d19/d20/d4a/f78 [869857,27493] 0 2024-04-16T02:35:36.978 INFO:tasks.workunit.client.1.smithi153.stdout:2/464: dwrite d9/d2b/f16 [4194304,4194304] 0 2024-04-16T02:35:36.978 INFO:tasks.workunit.client.1.smithi153.stdout:5/485: chown d8/d1f/d20/d39/f4a 373632918 1 2024-04-16T02:35:36.980 INFO:tasks.workunit.client.0.smithi151.stdout:0/168: rename dc/f27 to dc/d15/d34/f35 0 2024-04-16T02:35:36.982 INFO:tasks.workunit.client.1.smithi153.stdout:6/424: truncate dc/d1a/d2d/f44 1495382 0 2024-04-16T02:35:36.983 INFO:tasks.workunit.client.0.smithi151.stdout:0/169: symlink dc/l36 0 2024-04-16T02:35:36.983 INFO:tasks.workunit.client.1.smithi153.stdout:2/465: truncate d9/d2b/f46 4659519 0 2024-04-16T02:35:36.983 INFO:tasks.workunit.client.1.smithi153.stdout:6/425: stat dc/d47/d7d 0 2024-04-16T02:35:36.986 INFO:tasks.workunit.client.0.smithi151.stdout:0/170: dread dc/ff [0,4194304] 0 2024-04-16T02:35:36.987 INFO:tasks.workunit.client.1.smithi153.stdout:1/448: dwrite d4/dd/d20/f55 [0,4194304] 0 2024-04-16T02:35:36.987 INFO:tasks.workunit.client.1.smithi153.stdout:7/435: rename d0/d4/c3c to d0/d4/db/d2b/d8c/c8e 0 2024-04-16T02:35:36.990 INFO:tasks.workunit.client.1.smithi153.stdout:6/426: rmdir dc/d1a 39 2024-04-16T02:35:36.990 INFO:tasks.workunit.client.1.smithi153.stdout:4/553: dread d3/d9/db/de/f60 [0,4194304] 0 2024-04-16T02:35:36.991 INFO:tasks.workunit.client.0.smithi151.stdout:0/171: getdents dc/d15 0 2024-04-16T02:35:36.991 INFO:tasks.workunit.client.1.smithi153.stdout:1/449: stat d4/dd/d20/f72 0 2024-04-16T02:35:36.993 INFO:tasks.workunit.client.1.smithi153.stdout:2/466: mkdir d9/d71/d8d/da8 0 2024-04-16T02:35:36.993 INFO:tasks.workunit.client.1.smithi153.stdout:0/537: getdents dd/d14/d51/d9e 0 2024-04-16T02:35:36.994 INFO:tasks.workunit.client.1.smithi153.stdout:8/409: sync 2024-04-16T02:35:36.996 INFO:tasks.workunit.client.1.smithi153.stdout:7/436: chown d0/d4/f58 33679483 1 2024-04-16T02:35:36.996 INFO:tasks.workunit.client.1.smithi153.stdout:1/450: readlink d4/dd/d18/d29/l35 0 2024-04-16T02:35:36.996 INFO:tasks.workunit.client.1.smithi153.stdout:6/427: unlink dc/f17 0 2024-04-16T02:35:36.998 INFO:tasks.workunit.client.0.smithi151.stdout:4/94: sync 2024-04-16T02:35:36.998 INFO:tasks.workunit.client.1.smithi153.stdout:3/411: dwrite d6/d10/d3b/d76/f7d [0,4194304] 0 2024-04-16T02:35:36.999 INFO:tasks.workunit.client.0.smithi151.stdout:4/95: creat d0/d11/d19/f1e x:0 0 0 2024-04-16T02:35:37.000 INFO:tasks.workunit.client.1.smithi153.stdout:2/467: read d9/d2b/d18/d3f/d40/f70 [2868664,27923] 0 2024-04-16T02:35:37.003 INFO:tasks.workunit.client.1.smithi153.stdout:0/538: truncate dd/d14/d19/d20/f31 1547067 0 2024-04-16T02:35:37.005 INFO:tasks.workunit.client.1.smithi153.stdout:3/412: symlink d6/d10/d3b/d59/d7c/l8f 0 2024-04-16T02:35:37.005 INFO:tasks.workunit.client.1.smithi153.stdout:7/437: rename d0/d4/db/d2b/d43/d6d/d68/d72/c85 to d0/c8f 0 2024-04-16T02:35:37.006 INFO:tasks.workunit.client.0.smithi151.stdout:4/96: dwrite d0/f15 [0,4194304] 0 2024-04-16T02:35:37.007 INFO:tasks.workunit.client.1.smithi153.stdout:0/539: write dd/d14/d19/d1c/f98 [750837,64375] 0 2024-04-16T02:35:37.008 INFO:tasks.workunit.client.0.smithi151.stdout:2/125: fdatasync de/f27 0 2024-04-16T02:35:37.009 INFO:tasks.workunit.client.1.smithi153.stdout:4/554: dwrite d3/d9/db/d17/d28/f9c [0,4194304] 0 2024-04-16T02:35:37.010 INFO:tasks.workunit.client.1.smithi153.stdout:0/540: write f9 [1215443,79472] 0 2024-04-16T02:35:37.010 INFO:tasks.workunit.client.0.smithi151.stdout:4/97: fsync d0/f3 0 2024-04-16T02:35:37.057 INFO:tasks.workunit.client.0.smithi151.stdout:2/126: chown de/d15/f1a 59984 1 2024-04-16T02:35:37.058 INFO:tasks.workunit.client.0.smithi151.stdout:0/172: write dc/f1f [637976,69100] 0 2024-04-16T02:35:37.059 INFO:tasks.workunit.client.1.smithi153.stdout:8/410: creat d2/d1c/d1f/d27/d2d/d47/d60/d67/d86/f89 x:0 0 0 2024-04-16T02:35:37.062 INFO:tasks.workunit.client.1.smithi153.stdout:6/428: symlink dc/d47/d57/d77/d7a/l8a 0 2024-04-16T02:35:37.062 INFO:tasks.workunit.client.0.smithi151.stdout:4/98: creat d0/f1f x:0 0 0 2024-04-16T02:35:37.064 INFO:tasks.workunit.client.0.smithi151.stdout:1/74: dwrite f1 [0,4194304] 0 2024-04-16T02:35:37.065 INFO:tasks.workunit.client.0.smithi151.stdout:0/173: fdatasync dc/f28 0 2024-04-16T02:35:37.066 INFO:tasks.workunit.client.1.smithi153.stdout:2/468: creat d9/d24/d34/d60/fa9 x:0 0 0 2024-04-16T02:35:37.066 INFO:tasks.workunit.client.0.smithi151.stdout:1/75: chown d3/f8 292303116 1 2024-04-16T02:35:37.067 INFO:tasks.workunit.client.0.smithi151.stdout:0/174: write dc/f2f [4655026,22642] 0 2024-04-16T02:35:37.068 INFO:tasks.workunit.client.0.smithi151.stdout:4/99: unlink d0/d11/l18 0 2024-04-16T02:35:37.069 INFO:tasks.workunit.client.1.smithi153.stdout:3/413: write d6/f20 [5009302,6402] 0 2024-04-16T02:35:37.069 INFO:tasks.workunit.client.0.smithi151.stdout:0/175: read - dc/d15/f25 zero size 2024-04-16T02:35:37.071 INFO:tasks.workunit.client.0.smithi151.stdout:2/127: dread de/d15/f1c [0,4194304] 0 2024-04-16T02:35:37.072 INFO:tasks.workunit.client.0.smithi151.stdout:2/128: stat de/l1b 0 2024-04-16T02:35:37.074 INFO:tasks.workunit.client.1.smithi153.stdout:4/555: mknod d3/d9/d24/cc1 0 2024-04-16T02:35:37.075 INFO:tasks.workunit.client.1.smithi153.stdout:0/541: write dd/f22 [781380,46953] 0 2024-04-16T02:35:37.077 INFO:tasks.workunit.client.1.smithi153.stdout:4/556: chown d3/d9/db/d17/d28/d4c/c7c 557 1 2024-04-16T02:35:37.079 INFO:tasks.workunit.client.1.smithi153.stdout:1/451: getdents d4/dd/d13/d23/d31 0 2024-04-16T02:35:37.080 INFO:tasks.workunit.client.0.smithi151.stdout:1/76: dwrite d3/d4/db/d16/f12 [4194304,4194304] 0 2024-04-16T02:35:37.080 INFO:tasks.workunit.client.1.smithi153.stdout:1/452: chown d4/dd/d13/d23/d5f/f69 66 1 2024-04-16T02:35:37.081 INFO:tasks.workunit.client.1.smithi153.stdout:2/469: mknod d9/d24/caa 0 2024-04-16T02:35:37.083 INFO:tasks.workunit.client.1.smithi153.stdout:9/493: write d4/d6/dc/d46/f55 [714440,3539] 0 2024-04-16T02:35:37.084 INFO:tasks.workunit.client.1.smithi153.stdout:3/414: unlink d6/d10/d22/d2e/f42 0 2024-04-16T02:35:37.085 INFO:tasks.workunit.client.0.smithi151.stdout:1/77: dread f1 [0,4194304] 0 2024-04-16T02:35:37.086 INFO:tasks.workunit.client.1.smithi153.stdout:5/486: truncate d8/d1f/d20/d39/f4c 472455 0 2024-04-16T02:35:37.088 INFO:tasks.workunit.client.1.smithi153.stdout:5/487: write d8/df/d2e/f5b [1861030,131029] 0 2024-04-16T02:35:37.090 INFO:tasks.workunit.client.1.smithi153.stdout:1/453: mknod d4/dd/d13/d26/d58/c9c 0 2024-04-16T02:35:37.093 INFO:tasks.workunit.client.1.smithi153.stdout:2/470: rename d9/d2b/d5c/f6c to d9/d2b/d18/d3f/fab 0 2024-04-16T02:35:37.095 INFO:tasks.workunit.client.1.smithi153.stdout:9/494: mknod d4/d6/dc/d2d/d64/c97 0 2024-04-16T02:35:37.097 INFO:tasks.workunit.client.1.smithi153.stdout:8/411: rmdir d2/de/d74 0 2024-04-16T02:35:37.097 INFO:tasks.workunit.client.1.smithi153.stdout:8/412: readlink d2/de/l22 0 2024-04-16T02:35:37.099 INFO:tasks.workunit.client.1.smithi153.stdout:8/413: truncate d2/d1c/d1f/d27/f6e 713269 0 2024-04-16T02:35:37.100 INFO:tasks.workunit.client.1.smithi153.stdout:5/488: symlink d8/df/d28/d5a/d5e/l95 0 2024-04-16T02:35:37.100 INFO:tasks.workunit.client.1.smithi153.stdout:6/429: write dc/d10/f34 [3230656,53002] 0 2024-04-16T02:35:37.101 INFO:tasks.workunit.client.1.smithi153.stdout:1/454: rmdir d4/dd/d18/d6c 39 2024-04-16T02:35:37.103 INFO:tasks.workunit.client.1.smithi153.stdout:9/495: rmdir d4/d6/dc/d24/d4e 39 2024-04-16T02:35:37.105 INFO:tasks.workunit.client.1.smithi153.stdout:4/557: link d3/d9/d24/c6a d3/d9/db/d17/d28/d4c/cc2 0 2024-04-16T02:35:37.105 INFO:tasks.workunit.client.1.smithi153.stdout:0/542: getdents dd/d14/d51/d54 0 2024-04-16T02:35:37.108 INFO:tasks.workunit.client.1.smithi153.stdout:5/489: fdatasync d8/f4d 0 2024-04-16T02:35:37.109 INFO:tasks.workunit.client.1.smithi153.stdout:2/471: dwrite d9/d2b/d1c/d20/f79 [0,4194304] 0 2024-04-16T02:35:37.116 INFO:tasks.workunit.client.1.smithi153.stdout:6/430: creat dc/d10/d62/f8b x:0 0 0 2024-04-16T02:35:37.118 INFO:tasks.workunit.client.1.smithi153.stdout:7/438: mknod d0/c90 0 2024-04-16T02:35:37.118 INFO:tasks.workunit.client.1.smithi153.stdout:6/431: readlink dc/l28 0 2024-04-16T02:35:37.121 INFO:tasks.workunit.client.1.smithi153.stdout:3/415: link d6/d10/d3b/l48 d6/d10/d3b/d59/d7c/l90 0 2024-04-16T02:35:37.121 INFO:tasks.workunit.client.1.smithi153.stdout:1/455: dread d4/dd/d18/f42 [0,4194304] 0 2024-04-16T02:35:37.121 INFO:tasks.workunit.client.1.smithi153.stdout:8/414: symlink d2/d13/d3a/d5b/l8a 0 2024-04-16T02:35:37.122 INFO:tasks.workunit.client.1.smithi153.stdout:5/490: write d8/df/d23/f56 [140896,11421] 0 2024-04-16T02:35:37.125 INFO:tasks.workunit.client.1.smithi153.stdout:1/456: stat d4/dd/d18/f76 0 2024-04-16T02:35:37.127 INFO:tasks.workunit.client.1.smithi153.stdout:0/543: dwrite dd/d14/d51/d54/f84 [0,4194304] 0 2024-04-16T02:35:37.128 INFO:tasks.workunit.client.1.smithi153.stdout:6/432: chown dc/d1a/f58 614091703 1 2024-04-16T02:35:37.128 INFO:tasks.workunit.client.1.smithi153.stdout:0/544: chown dd/d14/d51 4 1 2024-04-16T02:35:37.129 INFO:tasks.workunit.client.1.smithi153.stdout:2/472: write d9/d2b/d18/d3f/fab [863112,60691] 0 2024-04-16T02:35:37.129 INFO:tasks.workunit.client.1.smithi153.stdout:8/415: stat d2/de/c4f 0 2024-04-16T02:35:37.130 INFO:tasks.workunit.client.1.smithi153.stdout:3/416: creat d6/d10/d22/d2e/d8a/f91 x:0 0 0 2024-04-16T02:35:37.133 INFO:tasks.workunit.client.1.smithi153.stdout:5/491: mknod d8/df/d28/d5a/d66/c96 0 2024-04-16T02:35:37.134 INFO:tasks.workunit.client.1.smithi153.stdout:7/439: write d0/d4/db/f2a [17653,4584] 0 2024-04-16T02:35:37.134 INFO:tasks.workunit.client.1.smithi153.stdout:8/416: read d2/d1c/d1f/d27/d2d/d47/d60/d67/f80 [140299,104093] 0 2024-04-16T02:35:37.135 INFO:tasks.workunit.client.1.smithi153.stdout:1/457: readlink d4/dd/l67 0 2024-04-16T02:35:37.137 INFO:tasks.workunit.client.1.smithi153.stdout:0/545: dread dd/d14/d19/d1c/d73/f8f [0,4194304] 0 2024-04-16T02:35:37.137 INFO:tasks.workunit.client.1.smithi153.stdout:4/558: dread d3/d9/db/de/d2c/f74 [0,4194304] 0 2024-04-16T02:35:37.138 INFO:tasks.workunit.client.1.smithi153.stdout:4/559: write d3/ff [4888575,78090] 0 2024-04-16T02:35:37.140 INFO:tasks.workunit.client.1.smithi153.stdout:6/433: creat dc/d47/d80/f8c x:0 0 0 2024-04-16T02:35:37.141 INFO:tasks.workunit.client.1.smithi153.stdout:7/440: mknod d0/d4/db/d2b/d53/c91 0 2024-04-16T02:35:37.142 INFO:tasks.workunit.client.1.smithi153.stdout:7/441: stat d0/d4/db/ff 0 2024-04-16T02:35:37.142 INFO:tasks.workunit.client.1.smithi153.stdout:0/546: dread f9 [0,4194304] 0 2024-04-16T02:35:37.146 INFO:tasks.workunit.client.1.smithi153.stdout:0/547: dread dd/d14/d19/f1b [4194304,4194304] 0 2024-04-16T02:35:37.147 INFO:tasks.workunit.client.1.smithi153.stdout:0/548: write dd/d32/f96 [417669,44454] 0 2024-04-16T02:35:37.150 INFO:tasks.workunit.client.1.smithi153.stdout:5/492: creat d8/df/f97 x:0 0 0 2024-04-16T02:35:37.153 INFO:tasks.workunit.client.1.smithi153.stdout:6/434: mknod dc/d47/d57/d77/c8d 0 2024-04-16T02:35:37.154 INFO:tasks.workunit.client.1.smithi153.stdout:3/417: dwrite d6/d10/d22/d2c/f58 [0,4194304] 0 2024-04-16T02:35:37.154 INFO:tasks.workunit.client.1.smithi153.stdout:1/458: creat d4/dd/d13/d26/f9d x:0 0 0 2024-04-16T02:35:37.158 INFO:tasks.workunit.client.0.smithi151.stdout:2/129: rmdir de 39 2024-04-16T02:35:37.158 INFO:tasks.workunit.client.1.smithi153.stdout:7/442: rename d0/d4/db/d34/d51/c55 to d0/d13/c92 0 2024-04-16T02:35:37.161 INFO:tasks.workunit.client.0.smithi151.stdout:1/78: rename d3/d4/db/d19 to d3/d4/db/d1b 0 2024-04-16T02:35:37.163 INFO:tasks.workunit.client.1.smithi153.stdout:8/417: dwrite d2/d1c/d1f/d27/d2d/f59 [0,4194304] 0 2024-04-16T02:35:37.163 INFO:tasks.workunit.client.0.smithi151.stdout:4/100: mkdir d0/d20 0 2024-04-16T02:35:37.163 INFO:tasks.workunit.client.0.smithi151.stdout:4/101: stat d0/f2 0 2024-04-16T02:35:37.165 INFO:tasks.workunit.client.1.smithi153.stdout:4/560: link d3/d9/db/d17/f67 d3/d9/db/de/d2c/d38/fc3 0 2024-04-16T02:35:37.167 INFO:tasks.workunit.client.1.smithi153.stdout:6/435: mknod dc/d47/d57/d77/d7a/c8e 0 2024-04-16T02:35:37.168 INFO:tasks.workunit.client.1.smithi153.stdout:3/418: creat d6/d10/d3b/f92 x:0 0 0 2024-04-16T02:35:37.168 INFO:tasks.workunit.client.1.smithi153.stdout:6/436: read dc/d10/f33 [189087,26548] 0 2024-04-16T02:35:37.170 INFO:tasks.workunit.client.1.smithi153.stdout:6/437: write dc/d10/f33 [618616,107359] 0 2024-04-16T02:35:37.170 INFO:tasks.workunit.client.1.smithi153.stdout:0/549: dread dd/f18 [0,4194304] 0 2024-04-16T02:35:37.172 INFO:tasks.workunit.client.1.smithi153.stdout:5/493: rmdir d8 39 2024-04-16T02:35:37.172 INFO:tasks.workunit.client.1.smithi153.stdout:6/438: dread dc/d6b/f6f [0,4194304] 0 2024-04-16T02:35:37.173 INFO:tasks.workunit.client.0.smithi151.stdout:1/79: write d3/d4/db/d16/d10/f18 [1820152,58638] 0 2024-04-16T02:35:37.173 INFO:tasks.workunit.client.1.smithi153.stdout:6/439: write dc/d47/f60 [120842,5760] 0 2024-04-16T02:35:37.174 INFO:tasks.workunit.client.1.smithi153.stdout:6/440: dread - dc/d47/d80/f8c zero size 2024-04-16T02:35:37.175 INFO:tasks.workunit.client.0.smithi151.stdout:0/176: dwrite dc/f28 [0,4194304] 0 2024-04-16T02:35:37.175 INFO:tasks.workunit.client.0.smithi151.stdout:4/102: unlink d0/lc 0 2024-04-16T02:35:37.179 INFO:tasks.workunit.client.0.smithi151.stdout:2/130: symlink de/l29 0 2024-04-16T02:35:37.181 INFO:tasks.workunit.client.0.smithi151.stdout:4/103: creat d0/d20/f21 x:0 0 0 2024-04-16T02:35:37.182 INFO:tasks.workunit.client.0.smithi151.stdout:2/131: chown de/d15/d1e/f24 335 1 2024-04-16T02:35:37.184 INFO:tasks.workunit.client.0.smithi151.stdout:4/104: stat d0/l1c 0 2024-04-16T02:35:37.185 INFO:tasks.workunit.client.0.smithi151.stdout:0/177: getdents dc/d20 0 2024-04-16T02:35:37.187 INFO:tasks.workunit.client.1.smithi153.stdout:7/443: truncate d0/d4/db/d2b/d53/d59/d76/f7b 997117 0 2024-04-16T02:35:37.188 INFO:tasks.workunit.client.0.smithi151.stdout:2/132: rename ca to de/d15/d1e/c2a 0 2024-04-16T02:35:37.188 INFO:tasks.workunit.client.1.smithi153.stdout:0/550: creat dd/d14/d51/fa2 x:0 0 0 2024-04-16T02:35:37.188 INFO:tasks.workunit.client.1.smithi153.stdout:1/459: mkdir d4/dd/d18/d29/d6e/d9e 0 2024-04-16T02:35:37.189 INFO:tasks.workunit.client.0.smithi151.stdout:1/80: dwrite d3/d4/db/d16/f12 [8388608,4194304] 0 2024-04-16T02:35:37.189 INFO:tasks.workunit.client.0.smithi151.stdout:4/105: rmdir d0/d11/d19 39 2024-04-16T02:35:37.190 INFO:tasks.workunit.client.0.smithi151.stdout:4/106: chown d0/d17/f1d 15177312 1 2024-04-16T02:35:37.191 INFO:tasks.workunit.client.1.smithi153.stdout:6/441: mknod dc/d47/d57/d77/d75/c8f 0 2024-04-16T02:35:37.191 INFO:tasks.workunit.client.0.smithi151.stdout:4/107: stat d0/d20 0 2024-04-16T02:35:37.194 INFO:tasks.workunit.client.0.smithi151.stdout:2/133: creat de/d11/f2b x:0 0 0 2024-04-16T02:35:37.195 INFO:tasks.workunit.client.0.smithi151.stdout:4/108: truncate d0/fd 1170806 0 2024-04-16T02:35:37.195 INFO:tasks.workunit.client.0.smithi151.stdout:2/134: fsync de/d15/f17 0 2024-04-16T02:35:37.196 INFO:tasks.workunit.client.1.smithi153.stdout:0/551: creat dd/d14/d51/d54/d65/fa3 x:0 0 0 2024-04-16T02:35:37.197 INFO:tasks.workunit.client.1.smithi153.stdout:0/552: write dd/d14/f86 [4486872,127187] 0 2024-04-16T02:35:37.198 INFO:tasks.workunit.client.1.smithi153.stdout:5/494: rename d8/df/d2e/f77 to d8/df/d28/d64/f98 0 2024-04-16T02:35:37.199 INFO:tasks.workunit.client.0.smithi151.stdout:1/81: dread d3/d4/db/d16/d10/f18 [0,4194304] 0 2024-04-16T02:35:37.199 INFO:tasks.workunit.client.1.smithi153.stdout:5/495: write d8/df/d2e/f57 [372512,69047] 0 2024-04-16T02:35:37.200 INFO:tasks.workunit.client.1.smithi153.stdout:8/418: creat d2/d1c/d1f/d27/d2d/f8b x:0 0 0 2024-04-16T02:35:37.201 INFO:tasks.workunit.client.0.smithi151.stdout:0/178: rename dc/f2d to dc/f37 0 2024-04-16T02:35:37.202 INFO:tasks.workunit.client.1.smithi153.stdout:3/419: creat d6/d10/d22/f93 x:0 0 0 2024-04-16T02:35:37.202 INFO:tasks.workunit.client.0.smithi151.stdout:2/135: creat de/d15/f2c x:0 0 0 2024-04-16T02:35:37.203 INFO:tasks.workunit.client.1.smithi153.stdout:1/460: mknod d4/dd/d18/d6c/c9f 0 2024-04-16T02:35:37.205 INFO:tasks.workunit.client.0.smithi151.stdout:0/179: unlink dc/l16 0 2024-04-16T02:35:37.206 INFO:tasks.workunit.client.1.smithi153.stdout:5/496: mkdir d8/df/d28/d5a/d66/d99 0 2024-04-16T02:35:37.206 INFO:tasks.workunit.client.1.smithi153.stdout:6/442: dwrite dc/d1a/f58 [0,4194304] 0 2024-04-16T02:35:37.207 INFO:tasks.workunit.client.0.smithi151.stdout:2/136: symlink de/d15/l2d 0 2024-04-16T02:35:37.208 INFO:tasks.workunit.client.0.smithi151.stdout:4/109: write d0/f8 [3666252,69050] 0 2024-04-16T02:35:37.209 INFO:tasks.workunit.client.1.smithi153.stdout:0/553: mkdir dd/d14/d19/d1c/da4 0 2024-04-16T02:35:37.210 INFO:tasks.workunit.client.1.smithi153.stdout:3/420: write d6/d10/d22/d2c/f60 [422345,56991] 0 2024-04-16T02:35:37.211 INFO:tasks.workunit.client.1.smithi153.stdout:6/443: mknod dc/d47/d57/c90 0 2024-04-16T02:35:37.211 INFO:tasks.workunit.client.0.smithi151.stdout:4/110: mknod d0/d20/c22 0 2024-04-16T02:35:37.211 INFO:tasks.workunit.client.0.smithi151.stdout:0/180: dread dc/d15/f23 [0,4194304] 0 2024-04-16T02:35:37.213 INFO:tasks.workunit.client.1.smithi153.stdout:1/461: rename d4/dd/d13/l17 to d4/dd/d13/d23/d31/la0 0 2024-04-16T02:35:37.213 INFO:tasks.workunit.client.1.smithi153.stdout:6/444: fdatasync dc/d47/d57/d77/d15/f74 0 2024-04-16T02:35:37.214 INFO:tasks.workunit.client.1.smithi153.stdout:5/497: mknod d8/df/d23/d70/c9a 0 2024-04-16T02:35:37.215 INFO:tasks.workunit.client.1.smithi153.stdout:3/421: mknod d6/d10/d22/d2e/c94 0 2024-04-16T02:35:37.216 INFO:tasks.workunit.client.1.smithi153.stdout:6/445: write dc/d6b/f73 [631459,87218] 0 2024-04-16T02:35:37.218 INFO:tasks.workunit.client.1.smithi153.stdout:6/446: dread - dc/d47/f4a zero size 2024-04-16T02:35:37.219 INFO:tasks.workunit.client.1.smithi153.stdout:1/462: mknod d4/dd/d13/d7e/ca1 0 2024-04-16T02:35:37.219 INFO:tasks.workunit.client.0.smithi151.stdout:4/111: dwrite d0/f1f [0,4194304] 0 2024-04-16T02:35:37.220 INFO:tasks.workunit.client.1.smithi153.stdout:6/447: stat dc/d47/d57/d77/d15/l53 0 2024-04-16T02:35:37.220 INFO:tasks.workunit.client.1.smithi153.stdout:0/554: dwrite dd/f53 [0,4194304] 0 2024-04-16T02:35:37.221 INFO:tasks.workunit.client.1.smithi153.stdout:3/422: mknod d6/d10/d22/d2e/c95 0 2024-04-16T02:35:37.221 INFO:tasks.workunit.client.0.smithi151.stdout:0/181: fsync dc/f37 0 2024-04-16T02:35:37.224 INFO:tasks.workunit.client.1.smithi153.stdout:7/444: dread d0/d4/db/d2b/f2f [0,4194304] 0 2024-04-16T02:35:37.225 INFO:tasks.workunit.client.0.smithi151.stdout:0/182: mknod dc/d15/c38 0 2024-04-16T02:35:37.227 INFO:tasks.workunit.client.0.smithi151.stdout:4/112: creat d0/d11/f23 x:0 0 0 2024-04-16T02:35:37.228 INFO:tasks.workunit.client.1.smithi153.stdout:7/445: fsync d0/d4/db/d34/d4b/d4d/f66 0 2024-04-16T02:35:37.229 INFO:tasks.workunit.client.1.smithi153.stdout:7/446: fdatasync d0/d4/db/d2b/f3e 0 2024-04-16T02:35:37.230 INFO:tasks.workunit.client.1.smithi153.stdout:3/423: unlink d6/f52 0 2024-04-16T02:35:37.231 INFO:tasks.workunit.client.1.smithi153.stdout:5/498: getdents d8/df/d23 0 2024-04-16T02:35:37.231 INFO:tasks.workunit.client.0.smithi151.stdout:6/34: sync 2024-04-16T02:35:37.232 INFO:tasks.workunit.client.0.smithi151.stdout:9/67: sync 2024-04-16T02:35:37.232 INFO:tasks.workunit.client.0.smithi151.stdout:4/113: dread d0/f8 [0,4194304] 0 2024-04-16T02:35:37.232 INFO:tasks.workunit.client.1.smithi153.stdout:5/499: fsync d8/df/d28/d5a/d66/f7b 0 2024-04-16T02:35:37.232 INFO:tasks.workunit.client.0.smithi151.stdout:4/114: chown d0/l4 2472952 1 2024-04-16T02:35:37.235 INFO:tasks.workunit.client.1.smithi153.stdout:1/463: rename d4/dd/d18/d57 to d4/da2 0 2024-04-16T02:35:37.235 INFO:tasks.workunit.client.0.smithi151.stdout:4/115: creat d0/d20/f24 x:0 0 0 2024-04-16T02:35:37.237 INFO:tasks.workunit.client.0.smithi151.stdout:4/116: rmdir d0/d17 39 2024-04-16T02:35:37.238 INFO:tasks.workunit.client.0.smithi151.stdout:4/117: read d0/f15 [2412498,96044] 0 2024-04-16T02:35:37.240 INFO:tasks.workunit.client.0.smithi151.stdout:6/35: dwrite d2/d5/f8 [4194304,4194304] 0 2024-04-16T02:35:37.242 INFO:tasks.workunit.client.0.smithi151.stdout:4/118: creat d0/d11/d19/f25 x:0 0 0 2024-04-16T02:35:37.244 INFO:tasks.workunit.client.1.smithi153.stdout:6/448: rename dc/d1a/d2d/f3f to dc/d47/d57/f91 0 2024-04-16T02:35:37.246 INFO:tasks.workunit.client.1.smithi153.stdout:5/500: mknod d8/df/d28/d5a/d66/d92/c9b 0 2024-04-16T02:35:37.247 INFO:tasks.workunit.client.1.smithi153.stdout:5/501: chown d8/f30 39297347 1 2024-04-16T02:35:37.248 INFO:tasks.workunit.client.1.smithi153.stdout:1/464: rename d4/dd/d18/l56 to d4/dd/d13/d7e/d80/la3 0 2024-04-16T02:35:37.249 INFO:tasks.workunit.client.1.smithi153.stdout:3/424: mknod d6/d10/d22/d43/c96 0 2024-04-16T02:35:37.250 INFO:tasks.workunit.client.1.smithi153.stdout:1/465: fdatasync d4/dd/d13/d23/d31/f88 0 2024-04-16T02:35:37.250 INFO:tasks.workunit.client.1.smithi153.stdout:6/449: mknod dc/c92 0 2024-04-16T02:35:37.250 INFO:tasks.workunit.client.1.smithi153.stdout:7/447: dwrite d0/d4/fc [0,4194304] 0 2024-04-16T02:35:37.251 INFO:tasks.workunit.client.1.smithi153.stdout:5/502: creat d8/df/d23/d70/f9c x:0 0 0 2024-04-16T02:35:37.252 INFO:tasks.workunit.client.1.smithi153.stdout:3/425: unlink d6/d10/d22/d4d/f7a 0 2024-04-16T02:35:37.254 INFO:tasks.workunit.client.1.smithi153.stdout:7/448: read d0/d4/db/d2b/d53/f46 [2366380,89027] 0 2024-04-16T02:35:37.254 INFO:tasks.workunit.client.1.smithi153.stdout:6/450: rename dc/d47/d57/d77/d75 to dc/d47/d93 0 2024-04-16T02:35:37.256 INFO:tasks.workunit.client.1.smithi153.stdout:6/451: write dc/d47/d57/d77/d15/f74 [464597,40563] 0 2024-04-16T02:35:37.257 INFO:tasks.workunit.client.1.smithi153.stdout:1/466: rename d4/dd/d13/d23/f27 to d4/dd/d20/fa4 0 2024-04-16T02:35:37.259 INFO:tasks.workunit.client.1.smithi153.stdout:1/467: write d4/dd/d13/d23/d5f/f69 [2446228,105769] 0 2024-04-16T02:35:37.259 INFO:tasks.workunit.client.1.smithi153.stdout:0/555: write f9 [5168756,89959] 0 2024-04-16T02:35:37.260 INFO:tasks.workunit.client.1.smithi153.stdout:6/452: creat dc/d59/d63/f94 x:0 0 0 2024-04-16T02:35:37.261 INFO:tasks.workunit.client.1.smithi153.stdout:5/503: rename d8/df/d23/d62 to d8/d41/d90/d9d 0 2024-04-16T02:35:37.262 INFO:tasks.workunit.client.1.smithi153.stdout:3/426: creat d6/f97 x:0 0 0 2024-04-16T02:35:37.262 INFO:tasks.workunit.client.1.smithi153.stdout:1/468: rmdir d4/da2/d65/d82 39 2024-04-16T02:35:37.263 INFO:tasks.workunit.client.1.smithi153.stdout:1/469: dread - d4/dd/d13/d26/f9d zero size 2024-04-16T02:35:37.264 INFO:tasks.workunit.client.1.smithi153.stdout:7/449: rename d0/d4/db/d2b/d53/d59/d5c to d0/d4/db/d47/d8d/d93 0 2024-04-16T02:35:37.265 INFO:tasks.workunit.client.1.smithi153.stdout:3/427: creat d6/d10/d22/d2c/f98 x:0 0 0 2024-04-16T02:35:37.266 INFO:tasks.workunit.client.1.smithi153.stdout:3/428: fsync d6/d10/f32 0 2024-04-16T02:35:37.267 INFO:tasks.workunit.client.1.smithi153.stdout:1/470: unlink d4/f36 0 2024-04-16T02:35:37.268 INFO:tasks.workunit.client.1.smithi153.stdout:7/450: creat d0/d4/db/d47/f94 x:0 0 0 2024-04-16T02:35:37.270 INFO:tasks.workunit.client.1.smithi153.stdout:6/453: rename dc/d1a/f36 to dc/d59/d63/f95 0 2024-04-16T02:35:37.271 INFO:tasks.workunit.client.1.smithi153.stdout:6/454: dread - dc/d10/f88 zero size 2024-04-16T02:35:37.271 INFO:tasks.workunit.client.1.smithi153.stdout:3/429: symlink d6/l99 0 2024-04-16T02:35:37.272 INFO:tasks.workunit.client.1.smithi153.stdout:7/451: mknod d0/d4/db/d34/d4b/c95 0 2024-04-16T02:35:37.272 INFO:tasks.workunit.client.1.smithi153.stdout:3/430: chown d6/f2d 384 1 2024-04-16T02:35:37.273 INFO:tasks.workunit.client.1.smithi153.stdout:1/471: rename d4/dd/d13/d7e/d80/l85 to d4/da2/d65/d82/la5 0 2024-04-16T02:35:37.273 INFO:tasks.workunit.client.1.smithi153.stdout:6/455: symlink dc/d6b/l96 0 2024-04-16T02:35:37.274 INFO:tasks.workunit.client.1.smithi153.stdout:1/472: read d4/f22 [1486286,34249] 0 2024-04-16T02:35:37.275 INFO:tasks.workunit.client.1.smithi153.stdout:1/473: write d4/da2/d65/d84/f8c [557216,20610] 0 2024-04-16T02:35:37.276 INFO:tasks.workunit.client.1.smithi153.stdout:3/431: dread d6/d10/d22/d2c/f58 [0,4194304] 0 2024-04-16T02:35:37.276 INFO:tasks.workunit.client.1.smithi153.stdout:1/474: fsync d4/da2/d65/d82/f9a 0 2024-04-16T02:35:37.277 INFO:tasks.workunit.client.1.smithi153.stdout:1/475: dread - d4/dd/d18/d29/d71/f7b zero size 2024-04-16T02:35:37.277 INFO:tasks.workunit.client.1.smithi153.stdout:1/476: readlink d4/dd/d13/d7e/l81 0 2024-04-16T02:35:37.277 INFO:tasks.workunit.client.1.smithi153.stdout:3/432: symlink d6/d10/d22/d2c/l9a 0 2024-04-16T02:35:37.295 INFO:tasks.workunit.client.0.smithi151.stdout:6/36: getdents d2/d5 0 2024-04-16T02:35:37.296 INFO:tasks.workunit.client.0.smithi151.stdout:4/119: creat d0/d11/f26 x:0 0 0 2024-04-16T02:35:37.299 INFO:tasks.workunit.client.0.smithi151.stdout:6/37: link d2/d5/l7 d2/d5/la 0 2024-04-16T02:35:37.300 INFO:tasks.workunit.client.0.smithi151.stdout:6/38: dread - d2/f9 zero size 2024-04-16T02:35:37.312 INFO:tasks.workunit.client.0.smithi151.stdout:6/39: rmdir d2 39 2024-04-16T02:35:37.313 INFO:tasks.workunit.client.0.smithi151.stdout:4/120: truncate d0/fe 2457346 0 2024-04-16T02:35:37.315 INFO:tasks.workunit.client.0.smithi151.stdout:6/40: write d2/d5/f6 [218455,104743] 0 2024-04-16T02:35:37.315 INFO:tasks.workunit.client.0.smithi151.stdout:4/121: creat d0/d20/f27 x:0 0 0 2024-04-16T02:35:37.319 INFO:tasks.workunit.client.0.smithi151.stdout:6/41: creat d2/d5/fb x:0 0 0 2024-04-16T02:35:37.327 INFO:tasks.workunit.client.1.smithi153.stdout:9/496: sync 2024-04-16T02:35:37.327 INFO:tasks.workunit.client.0.smithi151.stdout:6/42: dwrite d2/d5/fb [0,4194304] 0 2024-04-16T02:35:37.327 INFO:tasks.workunit.client.0.smithi151.stdout:4/122: dwrite d0/f15 [0,4194304] 0 2024-04-16T02:35:37.329 INFO:tasks.workunit.client.1.smithi153.stdout:0/556: write dd/d14/d51/d54/d65/f69 [683449,98903] 0 2024-04-16T02:35:37.329 INFO:tasks.workunit.client.0.smithi151.stdout:6/43: write d2/d5/fb [973765,66056] 0 2024-04-16T02:35:37.330 INFO:tasks.workunit.client.0.smithi151.stdout:6/44: chown d2/d5 216108 1 2024-04-16T02:35:37.330 INFO:tasks.workunit.client.1.smithi153.stdout:6/456: write dc/f11 [2450115,49651] 0 2024-04-16T02:35:37.332 INFO:tasks.workunit.client.0.smithi151.stdout:6/45: chown d2/l4 139 1 2024-04-16T02:35:37.332 INFO:tasks.workunit.client.1.smithi153.stdout:1/477: creat d4/da2/d65/d84/fa6 x:0 0 0 2024-04-16T02:35:37.333 INFO:tasks.workunit.client.1.smithi153.stdout:6/457: dread - dc/d47/d80/f8c zero size 2024-04-16T02:35:37.334 INFO:tasks.workunit.client.0.smithi151.stdout:4/123: dread d0/f15 [0,4194304] 0 2024-04-16T02:35:37.334 INFO:tasks.workunit.client.0.smithi151.stdout:6/46: creat d2/d5/fc x:0 0 0 2024-04-16T02:35:37.337 INFO:tasks.workunit.client.0.smithi151.stdout:4/124: mknod d0/d17/c28 0 2024-04-16T02:35:37.340 INFO:tasks.workunit.client.1.smithi153.stdout:3/433: dwrite d6/d10/d3b/d76/f7f [0,4194304] 0 2024-04-16T02:35:37.340 INFO:tasks.workunit.client.1.smithi153.stdout:0/557: mknod dd/d14/d19/ca5 0 2024-04-16T02:35:37.341 INFO:tasks.workunit.client.0.smithi151.stdout:4/125: creat d0/d11/d19/f29 x:0 0 0 2024-04-16T02:35:37.341 INFO:tasks.workunit.client.0.smithi151.stdout:6/47: link d2/d5/l7 d2/ld 0 2024-04-16T02:35:37.341 INFO:tasks.workunit.client.1.smithi153.stdout:9/497: rename d4/d6/dc/d2d/f58 to d4/d6/dc/d24/d5d/d66/f98 0 2024-04-16T02:35:37.342 INFO:tasks.workunit.client.1.smithi153.stdout:6/458: mkdir dc/d47/d57/d77/d7a/d97 0 2024-04-16T02:35:37.344 INFO:tasks.workunit.client.1.smithi153.stdout:9/498: readlink d4/d6/dc/d46/d93/d40/l6f 0 2024-04-16T02:35:37.345 INFO:tasks.workunit.client.1.smithi153.stdout:6/459: fsync dc/d47/f7e 0 2024-04-16T02:35:37.345 INFO:tasks.workunit.client.1.smithi153.stdout:0/558: write dd/d32/d57/f42 [571754,4127] 0 2024-04-16T02:35:37.348 INFO:tasks.workunit.client.1.smithi153.stdout:9/499: write d4/d6/d1b/d6e/f74 [5699972,3870] 0 2024-04-16T02:35:37.348 INFO:tasks.workunit.client.1.smithi153.stdout:0/559: write dd/d14/d19/d20/f88 [1188456,1644] 0 2024-04-16T02:35:37.350 INFO:tasks.workunit.client.1.smithi153.stdout:0/560: read - dd/d14/d51/f59 zero size 2024-04-16T02:35:37.350 INFO:tasks.workunit.client.1.smithi153.stdout:0/561: read - dd/d14/d51/d54/d65/f99 zero size 2024-04-16T02:35:37.351 INFO:tasks.workunit.client.1.smithi153.stdout:1/478: getdents d4/dd/d13/d7e/d80 0 2024-04-16T02:35:37.352 INFO:tasks.workunit.client.0.smithi151.stdout:4/126: dwrite d0/f2 [0,4194304] 0 2024-04-16T02:35:37.353 INFO:tasks.workunit.client.0.smithi151.stdout:4/127: stat d0/l12 0 2024-04-16T02:35:37.354 INFO:tasks.workunit.client.0.smithi151.stdout:6/48: dwrite d2/d5/f8 [8388608,4194304] 0 2024-04-16T02:35:37.354 INFO:tasks.workunit.client.1.smithi153.stdout:6/460: unlink dc/d1a/f26 0 2024-04-16T02:35:37.356 INFO:tasks.workunit.client.0.smithi151.stdout:4/128: fdatasync d0/d11/d19/f25 0 2024-04-16T02:35:37.356 INFO:tasks.workunit.client.1.smithi153.stdout:9/500: mkdir d4/d6/d1b/d2a/d99 0 2024-04-16T02:35:37.356 INFO:tasks.workunit.client.1.smithi153.stdout:3/434: link d6/d10/d22/d4d/l83 d6/d10/d22/d2e/l9b 0 2024-04-16T02:35:37.356 INFO:tasks.workunit.client.0.smithi151.stdout:4/129: fsync d0/d20/f24 0 2024-04-16T02:35:37.357 INFO:tasks.workunit.client.1.smithi153.stdout:3/435: chown d6/d10/c3f 0 1 2024-04-16T02:35:37.357 INFO:tasks.workunit.client.0.smithi151.stdout:6/49: read d2/d5/f6 [163170,91952] 0 2024-04-16T02:35:37.358 INFO:tasks.workunit.client.0.smithi151.stdout:4/130: read d0/d11/d19/f1b [688796,66658] 0 2024-04-16T02:35:37.361 INFO:tasks.workunit.client.0.smithi151.stdout:4/131: symlink d0/d20/l2a 0 2024-04-16T02:35:37.362 INFO:tasks.workunit.client.1.smithi153.stdout:1/479: creat d4/dd/d13/d26/d58/d7a/fa7 x:0 0 0 2024-04-16T02:35:37.362 INFO:tasks.workunit.client.1.smithi153.stdout:6/461: unlink la 0 2024-04-16T02:35:37.362 INFO:tasks.workunit.client.1.smithi153.stdout:9/501: mknod d4/d6/dc/d2d/d64/c9a 0 2024-04-16T02:35:37.365 INFO:tasks.workunit.client.1.smithi153.stdout:1/480: truncate d4/dd/d18/d29/f50 866489 0 2024-04-16T02:35:37.367 INFO:tasks.workunit.client.1.smithi153.stdout:0/562: rename dd/d14/d19/d1c to dd/d14/da6 0 2024-04-16T02:35:37.367 INFO:tasks.workunit.client.0.smithi151.stdout:4/132: creat d0/d17/f2b x:0 0 0 2024-04-16T02:35:37.369 INFO:tasks.workunit.client.0.smithi151.stdout:6/50: dwrite d2/d5/fc [0,4194304] 0 2024-04-16T02:35:37.369 INFO:tasks.workunit.client.0.smithi151.stdout:4/133: chown d0/f3 32346445 1 2024-04-16T02:35:37.370 INFO:tasks.workunit.client.1.smithi153.stdout:0/563: write dd/d14/d51/d54/f84 [4318819,25404] 0 2024-04-16T02:35:37.370 INFO:tasks.workunit.client.0.smithi151.stdout:4/134: chown d0/fb 425 1 2024-04-16T02:35:37.370 INFO:tasks.workunit.client.0.smithi151.stdout:6/51: dread - d2/f9 zero size 2024-04-16T02:35:37.370 INFO:tasks.workunit.client.0.smithi151.stdout:6/52: chown d2/l4 11 1 2024-04-16T02:35:37.371 INFO:tasks.workunit.client.1.smithi153.stdout:9/502: dwrite d4/d33/d7d/f85 [0,4194304] 0 2024-04-16T02:35:37.374 INFO:tasks.workunit.client.1.smithi153.stdout:1/481: dwrite d4/f22 [0,4194304] 0 2024-04-16T02:35:37.374 INFO:tasks.workunit.client.1.smithi153.stdout:9/503: readlink d4/d6/l34 0 2024-04-16T02:35:37.375 INFO:tasks.workunit.client.0.smithi151.stdout:4/135: dread d0/d11/d19/f1b [0,4194304] 0 2024-04-16T02:35:37.377 INFO:tasks.workunit.client.1.smithi153.stdout:1/482: write d4/dd/d13/d26/f9d [879040,57253] 0 2024-04-16T02:35:37.380 INFO:tasks.workunit.client.1.smithi153.stdout:0/564: read dd/d28/f41 [1269450,90650] 0 2024-04-16T02:35:37.380 INFO:tasks.workunit.client.1.smithi153.stdout:9/504: mkdir d4/d33/d9b 0 2024-04-16T02:35:37.384 INFO:tasks.workunit.client.0.smithi151.stdout:6/53: write d2/d5/fc [4241862,51154] 0 2024-04-16T02:35:37.386 INFO:tasks.workunit.client.1.smithi153.stdout:0/565: rename f9 to dd/d32/d57/fa7 0 2024-04-16T02:35:37.387 INFO:tasks.workunit.client.0.smithi151.stdout:6/54: write d2/d5/fb [793415,918] 0 2024-04-16T02:35:37.387 INFO:tasks.workunit.client.1.smithi153.stdout:0/566: readlink dd/d32/l43 0 2024-04-16T02:35:37.387 INFO:tasks.workunit.client.0.smithi151.stdout:6/55: fsync d2/f9 0 2024-04-16T02:35:37.389 INFO:tasks.workunit.client.0.smithi151.stdout:6/56: symlink d2/le 0 2024-04-16T02:35:37.396 INFO:tasks.workunit.client.1.smithi153.stdout:9/505: creat d4/d6/dc/d24/d4e/f9c x:0 0 0 2024-04-16T02:35:37.396 INFO:tasks.workunit.client.0.smithi151.stdout:6/57: dwrite d2/d5/f8 [4194304,4194304] 0 2024-04-16T02:35:37.397 INFO:tasks.workunit.client.1.smithi153.stdout:1/483: rmdir d4/dd/d18/d29/d6e/d9e 0 2024-04-16T02:35:37.398 INFO:tasks.workunit.client.1.smithi153.stdout:3/436: dwrite d6/d10/f19 [0,4194304] 0 2024-04-16T02:35:37.400 INFO:tasks.workunit.client.0.smithi151.stdout:6/58: mkdir d2/d5/df 0 2024-04-16T02:35:37.401 INFO:tasks.workunit.client.1.smithi153.stdout:1/484: dread d4/dd/d20/f72 [0,4194304] 0 2024-04-16T02:35:37.407 INFO:tasks.workunit.client.1.smithi153.stdout:1/485: dread - d4/dd/d13/f62 zero size 2024-04-16T02:35:37.408 INFO:tasks.workunit.client.0.smithi151.stdout:6/59: dwrite d2/d5/fc [0,4194304] 0 2024-04-16T02:35:37.411 INFO:tasks.workunit.client.0.smithi151.stdout:6/60: write d2/d5/f6 [463405,51005] 0 2024-04-16T02:35:37.415 INFO:tasks.workunit.client.0.smithi151.stdout:6/61: dread d2/d5/f8 [0,4194304] 0 2024-04-16T02:35:37.416 INFO:tasks.workunit.client.0.smithi151.stdout:6/62: chown d2/f9 57358006 1 2024-04-16T02:35:37.416 INFO:tasks.workunit.client.0.smithi151.stdout:6/63: rename d2 to d2/d10 22 2024-04-16T02:35:37.418 INFO:tasks.workunit.client.1.smithi153.stdout:0/567: dread dd/d14/d19/f46 [4194304,4194304] 0 2024-04-16T02:35:37.418 INFO:tasks.workunit.client.1.smithi153.stdout:1/486: dwrite d4/dd/d13/d26/f9d [0,4194304] 0 2024-04-16T02:35:37.418 INFO:tasks.workunit.client.1.smithi153.stdout:0/568: fsync f5 0 2024-04-16T02:35:37.419 INFO:tasks.workunit.client.0.smithi151.stdout:6/64: write d2/d5/f8 [8994251,67852] 0 2024-04-16T02:35:37.420 INFO:tasks.workunit.client.0.smithi151.stdout:6/65: write d2/d5/f8 [8874616,15635] 0 2024-04-16T02:35:37.422 INFO:tasks.workunit.client.0.smithi151.stdout:6/66: getdents d2/d5/df 0 2024-04-16T02:35:37.427 INFO:tasks.workunit.client.1.smithi153.stdout:1/487: getdents d4/dd/d13/d7e 0 2024-04-16T02:35:37.427 INFO:tasks.workunit.client.0.smithi151.stdout:6/67: link d2/d5/l7 d2/d5/df/l11 0 2024-04-16T02:35:37.428 INFO:tasks.workunit.client.0.smithi151.stdout:6/68: chown d2/d5/fc 671 1 2024-04-16T02:35:37.429 INFO:tasks.workunit.client.1.smithi153.stdout:0/569: dwrite dd/d14/d51/f59 [0,4194304] 0 2024-04-16T02:35:37.430 INFO:tasks.workunit.client.0.smithi151.stdout:6/69: rmdir d2/d5/df 39 2024-04-16T02:35:37.431 INFO:tasks.workunit.client.1.smithi153.stdout:9/506: dread d4/d33/f38 [0,4194304] 0 2024-04-16T02:35:37.431 INFO:tasks.workunit.client.0.smithi151.stdout:6/70: mknod d2/c12 0 2024-04-16T02:35:37.432 INFO:tasks.workunit.client.1.smithi153.stdout:3/437: truncate d6/d10/d3b/f85 1235368 0 2024-04-16T02:35:37.432 INFO:tasks.workunit.client.1.smithi153.stdout:0/570: fdatasync dd/d14/da6/f3e 0 2024-04-16T02:35:37.433 INFO:tasks.workunit.client.1.smithi153.stdout:9/507: readlink d4/d6/dc/d46/d93/d40/d4a/l8e 0 2024-04-16T02:35:37.437 INFO:tasks.workunit.client.1.smithi153.stdout:0/571: read dd/d14/d51/f60 [384878,96291] 0 2024-04-16T02:35:37.437 INFO:tasks.workunit.client.0.smithi151.stdout:6/71: dwrite d2/d5/fc [0,4194304] 0 2024-04-16T02:35:37.438 INFO:tasks.workunit.client.1.smithi153.stdout:0/572: readlink dd/d32/la1 0 2024-04-16T02:35:37.438 INFO:tasks.workunit.client.1.smithi153.stdout:3/438: mknod d6/d10/d22/c9c 0 2024-04-16T02:35:37.440 INFO:tasks.workunit.client.0.smithi151.stdout:6/72: mkdir d2/d13 0 2024-04-16T02:35:37.440 INFO:tasks.workunit.client.1.smithi153.stdout:3/439: creat d6/d10/d3b/f9d x:0 0 0 2024-04-16T02:35:37.441 INFO:tasks.workunit.client.1.smithi153.stdout:3/440: chown d6/d10/d22/d4d 70796351 1 2024-04-16T02:35:37.441 INFO:tasks.workunit.client.0.smithi151.stdout:6/73: mknod d2/d5/df/c14 0 2024-04-16T02:35:37.442 INFO:tasks.workunit.client.1.smithi153.stdout:0/573: creat dd/d14/da6/da4/fa8 x:0 0 0 2024-04-16T02:35:37.442 INFO:tasks.workunit.client.1.smithi153.stdout:3/441: chown d6/d10/d5b/f82 106 1 2024-04-16T02:35:37.443 INFO:tasks.workunit.client.0.smithi151.stdout:6/74: creat d2/d13/f15 x:0 0 0 2024-04-16T02:35:37.443 INFO:tasks.workunit.client.1.smithi153.stdout:1/488: dread d4/dd/d13/d26/f2b [0,4194304] 0 2024-04-16T02:35:37.446 INFO:tasks.workunit.client.1.smithi153.stdout:3/442: chown d6/d10/d22/d43/f77 27 1 2024-04-16T02:35:37.446 INFO:tasks.workunit.client.1.smithi153.stdout:1/489: symlink d4/la8 0 2024-04-16T02:35:37.449 INFO:tasks.workunit.client.1.smithi153.stdout:3/443: symlink d6/l9e 0 2024-04-16T02:35:37.451 INFO:tasks.workunit.client.0.smithi151.stdout:6/75: fdatasync d2/d5/f8 0 2024-04-16T02:35:37.451 INFO:tasks.workunit.client.1.smithi153.stdout:0/574: dwrite dd/d14/d51/fa2 [0,4194304] 0 2024-04-16T02:35:37.453 INFO:tasks.workunit.client.1.smithi153.stdout:3/444: write d6/d10/d3b/d59/d7c/f8e [1011153,34441] 0 2024-04-16T02:35:37.453 INFO:tasks.workunit.client.1.smithi153.stdout:2/473: sync 2024-04-16T02:35:37.453 INFO:tasks.workunit.client.1.smithi153.stdout:4/561: sync 2024-04-16T02:35:37.455 INFO:tasks.workunit.client.1.smithi153.stdout:0/575: creat dd/d14/d19/d55/fa9 x:0 0 0 2024-04-16T02:35:37.455 INFO:tasks.workunit.client.1.smithi153.stdout:4/562: fdatasync d3/d9/db/d45/f7f 0 2024-04-16T02:35:37.455 INFO:tasks.workunit.client.1.smithi153.stdout:2/474: symlink d9/d2b/d1c/d20/lac 0 2024-04-16T02:35:37.456 INFO:tasks.workunit.client.1.smithi153.stdout:4/563: fsync d3/f73 0 2024-04-16T02:35:37.457 INFO:tasks.workunit.client.1.smithi153.stdout:0/576: read dd/ff [1166416,18588] 0 2024-04-16T02:35:37.457 INFO:tasks.workunit.client.1.smithi153.stdout:3/445: rename d6/d10/d3b/d59/d7c/f8e to d6/d10/d3b/f9f 0 2024-04-16T02:35:37.461 INFO:tasks.workunit.client.1.smithi153.stdout:4/564: dwrite d3/d9/d24/f4d [0,4194304] 0 2024-04-16T02:35:37.469 INFO:tasks.workunit.client.0.smithi151.stdout:6/76: rename d2/le to d2/d5/l16 0 2024-04-16T02:35:37.470 INFO:tasks.workunit.client.0.smithi151.stdout:6/77: fsync d2/d5/fb 0 2024-04-16T02:35:37.473 INFO:tasks.workunit.client.1.smithi153.stdout:3/446: mkdir d6/d10/d22/d4d/da0 0 2024-04-16T02:35:37.474 INFO:tasks.workunit.client.1.smithi153.stdout:3/447: write d6/f11 [125052,33465] 0 2024-04-16T02:35:37.474 INFO:tasks.workunit.client.1.smithi153.stdout:4/565: creat d3/d9/d79/fc4 x:0 0 0 2024-04-16T02:35:37.475 INFO:tasks.workunit.client.0.smithi151.stdout:6/78: dread d2/d5/f8 [0,4194304] 0 2024-04-16T02:35:37.475 INFO:tasks.workunit.client.1.smithi153.stdout:4/566: write d3/d9/f8d [3552905,4993] 0 2024-04-16T02:35:37.476 INFO:tasks.workunit.client.1.smithi153.stdout:3/448: dread - d6/d10/d22/d2e/f6f zero size 2024-04-16T02:35:37.476 INFO:tasks.workunit.client.1.smithi153.stdout:2/475: mknod d9/d71/d8d/da8/cad 0 2024-04-16T02:35:37.478 INFO:tasks.workunit.client.0.smithi151.stdout:6/79: chown d2/ld 37 1 2024-04-16T02:35:37.478 INFO:tasks.workunit.client.1.smithi153.stdout:0/577: symlink dd/d14/d51/d9e/laa 0 2024-04-16T02:35:37.478 INFO:tasks.workunit.client.1.smithi153.stdout:3/449: chown d6/d10/d22/d2e/c38 1506630 1 2024-04-16T02:35:37.478 INFO:tasks.workunit.client.1.smithi153.stdout:2/476: stat d9/d2b/d1c/d20/d45/f96 0 2024-04-16T02:35:37.480 INFO:tasks.workunit.client.1.smithi153.stdout:0/578: truncate dd/d32/d3f/f62 757616 0 2024-04-16T02:35:37.481 INFO:tasks.workunit.client.0.smithi151.stdout:6/80: dread d2/d5/f6 [0,4194304] 0 2024-04-16T02:35:37.481 INFO:tasks.workunit.client.1.smithi153.stdout:4/567: mkdir d3/d9/d24/d75/dc5 0 2024-04-16T02:35:37.482 INFO:tasks.workunit.client.0.smithi151.stdout:6/81: chown d2/d5/f8 200761123 1 2024-04-16T02:35:37.483 INFO:tasks.workunit.client.0.smithi151.stdout:6/82: truncate d2/d5/fc 5340859 0 2024-04-16T02:35:37.484 INFO:tasks.workunit.client.1.smithi153.stdout:9/508: dwrite d4/d33/d7d/f85 [4194304,4194304] 0 2024-04-16T02:35:37.484 INFO:tasks.workunit.client.1.smithi153.stdout:2/477: mkdir d9/d24/d27/d9a/dae 0 2024-04-16T02:35:37.484 INFO:tasks.workunit.client.1.smithi153.stdout:4/568: write d3/d9/db/f4a [2215710,114294] 0 2024-04-16T02:35:37.485 INFO:tasks.workunit.client.1.smithi153.stdout:0/579: mknod dd/d14/d19/d55/cab 0 2024-04-16T02:35:37.486 INFO:tasks.workunit.client.1.smithi153.stdout:3/450: rename d6 to d6/d10/d22/d2c/da1 22 2024-04-16T02:35:37.486 INFO:tasks.workunit.client.1.smithi153.stdout:2/478: stat d9/d24/d27/d2c/d5d/da3 0 2024-04-16T02:35:37.487 INFO:tasks.workunit.client.1.smithi153.stdout:1/490: truncate d4/dd/d20/f55 3831518 0 2024-04-16T02:35:37.487 INFO:tasks.workunit.client.1.smithi153.stdout:8/419: sync 2024-04-16T02:35:37.487 INFO:tasks.workunit.client.1.smithi153.stdout:5/504: sync 2024-04-16T02:35:37.487 INFO:tasks.workunit.client.1.smithi153.stdout:7/452: sync 2024-04-16T02:35:37.488 INFO:tasks.workunit.client.1.smithi153.stdout:6/462: sync 2024-04-16T02:35:37.488 INFO:tasks.workunit.client.1.smithi153.stdout:2/479: fdatasync d9/d24/d34/d60/fa9 0 2024-04-16T02:35:37.490 INFO:tasks.workunit.client.0.smithi151.stdout:6/83: dwrite d2/d13/f15 [0,4194304] 0 2024-04-16T02:35:37.491 INFO:tasks.workunit.client.1.smithi153.stdout:5/505: readlink d8/df/d28/d64/l7d 0 2024-04-16T02:35:37.491 INFO:tasks.workunit.client.1.smithi153.stdout:9/509: creat d4/d6/f9d x:0 0 0 2024-04-16T02:35:37.493 INFO:tasks.workunit.client.1.smithi153.stdout:1/491: dread d4/f22 [0,4194304] 0 2024-04-16T02:35:37.494 INFO:tasks.workunit.client.1.smithi153.stdout:4/569: write d3/d9/db/d17/f55 [1385693,31525] 0 2024-04-16T02:35:37.494 INFO:tasks.workunit.client.0.smithi151.stdout:6/84: symlink d2/l17 0 2024-04-16T02:35:37.501 INFO:tasks.workunit.client.1.smithi153.stdout:0/580: symlink dd/d26/lac 0 2024-04-16T02:35:37.503 INFO:tasks.workunit.client.1.smithi153.stdout:2/480: creat d9/d2b/d1c/d20/d45/faf x:0 0 0 2024-04-16T02:35:37.503 INFO:tasks.workunit.client.0.smithi151.stdout:6/85: dwrite d2/d5/fb [0,4194304] 0 2024-04-16T02:35:37.506 INFO:tasks.workunit.client.1.smithi153.stdout:8/420: rename d2/d1c/d1f/f5a to d2/de/d34/f8c 0 2024-04-16T02:35:37.508 INFO:tasks.workunit.client.1.smithi153.stdout:8/421: chown d2/d1c/d1f/d27/d2d/d47/l54 121550 1 2024-04-16T02:35:37.508 INFO:tasks.workunit.client.0.smithi151.stdout:6/86: getdents d2/d5/df 0 2024-04-16T02:35:37.510 INFO:tasks.workunit.client.0.smithi151.stdout:6/87: write d2/d5/fb [3551404,1415] 0 2024-04-16T02:35:37.512 INFO:tasks.workunit.client.0.smithi151.stdout:6/88: creat d2/d13/f18 x:0 0 0 2024-04-16T02:35:37.512 INFO:tasks.workunit.client.1.smithi153.stdout:4/570: creat d3/da5/fc6 x:0 0 0 2024-04-16T02:35:37.512 INFO:tasks.workunit.client.0.smithi151.stdout:6/89: chown d2/d5/df 4138213 1 2024-04-16T02:35:37.513 INFO:tasks.workunit.client.1.smithi153.stdout:1/492: creat d4/dd/d13/d26/d58/d7a/fa9 x:0 0 0 2024-04-16T02:35:37.516 INFO:tasks.workunit.client.1.smithi153.stdout:4/571: dread d3/d9/d24/f4d [0,4194304] 0 2024-04-16T02:35:37.517 INFO:tasks.workunit.client.0.smithi151.stdout:6/90: rename d2/d5/l16 to d2/d13/l19 0 2024-04-16T02:35:37.521 INFO:tasks.workunit.client.0.smithi151.stdout:6/91: rename d2/d13/f15 to d2/f1a 0 2024-04-16T02:35:37.521 INFO:tasks.workunit.client.1.smithi153.stdout:7/453: rename d0/d13/l37 to d0/d4/db/d2b/d43/d6d/d68/d72/l96 0 2024-04-16T02:35:37.522 INFO:tasks.workunit.client.0.smithi151.stdout:6/92: fdatasync d2/d5/fb 0 2024-04-16T02:35:37.522 INFO:tasks.workunit.client.1.smithi153.stdout:7/454: read - d0/d4/db/d2b/d43/d6d/d68/f86 zero size 2024-04-16T02:35:37.523 INFO:tasks.workunit.client.0.smithi151.stdout:6/93: chown d2/d5/df 890 1 2024-04-16T02:35:37.523 INFO:tasks.workunit.client.0.smithi151.stdout:6/94: chown d2/d5 1 1 2024-04-16T02:35:37.523 INFO:tasks.workunit.client.1.smithi153.stdout:5/506: write d8/df/f11 [1157958,12444] 0 2024-04-16T02:35:37.523 INFO:tasks.workunit.client.1.smithi153.stdout:7/455: read - d0/d4/db/d34/d4b/f69 zero size 2024-04-16T02:35:37.526 INFO:tasks.workunit.client.0.smithi151.stdout:6/95: creat d2/f1b x:0 0 0 2024-04-16T02:35:37.530 INFO:tasks.workunit.client.1.smithi153.stdout:1/493: write d4/f22 [2033886,46069] 0 2024-04-16T02:35:37.531 INFO:tasks.workunit.client.1.smithi153.stdout:6/463: mknod dc/d10/c98 0 2024-04-16T02:35:37.531 INFO:tasks.workunit.client.1.smithi153.stdout:8/422: dread d2/f9 [0,4194304] 0 2024-04-16T02:35:37.531 INFO:tasks.workunit.client.1.smithi153.stdout:1/494: dread - d4/dd/d13/f62 zero size 2024-04-16T02:35:37.532 INFO:tasks.workunit.client.1.smithi153.stdout:4/572: mknod d3/d9/db/d84/cc7 0 2024-04-16T02:35:37.533 INFO:tasks.workunit.client.1.smithi153.stdout:0/581: mknod dd/cad 0 2024-04-16T02:35:37.534 INFO:tasks.workunit.client.1.smithi153.stdout:2/481: mknod d9/d2b/cb0 0 2024-04-16T02:35:37.537 INFO:tasks.workunit.client.1.smithi153.stdout:3/451: getdents d6/d10/d22/d4d 0 2024-04-16T02:35:37.539 INFO:tasks.workunit.client.1.smithi153.stdout:8/423: creat d2/d13/d2c/f8d x:0 0 0 2024-04-16T02:35:37.539 INFO:tasks.workunit.client.1.smithi153.stdout:4/573: mkdir d3/d9/db/d17/d28/dc8 0 2024-04-16T02:35:37.540 INFO:tasks.workunit.client.1.smithi153.stdout:6/464: chown dc/d47/d57/d77/d15/f2f 881 1 2024-04-16T02:35:37.542 INFO:tasks.workunit.client.1.smithi153.stdout:5/507: fsync d8/df/d2e/f6e 0 2024-04-16T02:35:37.543 INFO:tasks.workunit.client.1.smithi153.stdout:2/482: mkdir d9/d24/d34/d36/db1 0 2024-04-16T02:35:37.544 INFO:tasks.workunit.client.1.smithi153.stdout:6/465: read dc/d10/f33 [632617,39166] 0 2024-04-16T02:35:37.544 INFO:tasks.workunit.client.1.smithi153.stdout:9/510: dwrite d4/d6/d1b/f59 [0,4194304] 0 2024-04-16T02:35:37.545 INFO:tasks.workunit.client.1.smithi153.stdout:6/466: chown dc/d47/d57 24474 1 2024-04-16T02:35:37.546 INFO:tasks.workunit.client.1.smithi153.stdout:1/495: dread d4/dd/f1d [0,4194304] 0 2024-04-16T02:35:37.547 INFO:tasks.workunit.client.1.smithi153.stdout:8/424: mkdir d2/d1c/d1f/d27/d2d/d47/d77/d8e 0 2024-04-16T02:35:37.548 INFO:tasks.workunit.client.1.smithi153.stdout:7/456: link d0/d4/db/d2b/c17 d0/d4/db/d47/c97 0 2024-04-16T02:35:37.553 INFO:tasks.workunit.client.1.smithi153.stdout:6/467: symlink dc/d10/l99 0 2024-04-16T02:35:37.556 INFO:tasks.workunit.client.1.smithi153.stdout:0/582: dwrite dd/d14/d51/d54/d65/f72 [0,4194304] 0 2024-04-16T02:35:37.557 INFO:tasks.workunit.client.1.smithi153.stdout:5/508: unlink d8/df/f2c 0 2024-04-16T02:35:37.558 INFO:tasks.workunit.client.1.smithi153.stdout:6/468: chown dc/d59/d63/f66 890026421 1 2024-04-16T02:35:37.559 INFO:tasks.workunit.client.1.smithi153.stdout:1/496: mknod d4/da2/d65/caa 0 2024-04-16T02:35:37.561 INFO:tasks.workunit.client.1.smithi153.stdout:8/425: creat d2/d1c/d1f/d27/d5e/d63/f8f x:0 0 0 2024-04-16T02:35:37.565 INFO:tasks.workunit.client.1.smithi153.stdout:9/511: write d4/d6/f22 [1299543,51581] 0 2024-04-16T02:35:37.566 INFO:tasks.workunit.client.1.smithi153.stdout:5/509: creat d8/df/d28/d5a/d72/f9e x:0 0 0 2024-04-16T02:35:37.567 INFO:tasks.workunit.client.1.smithi153.stdout:6/469: symlink dc/d47/d93/l9a 0 2024-04-16T02:35:37.567 INFO:tasks.workunit.client.1.smithi153.stdout:9/512: write d4/d6/f9d [912537,42386] 0 2024-04-16T02:35:37.569 INFO:tasks.workunit.client.1.smithi153.stdout:8/426: symlink d2/d13/l90 0 2024-04-16T02:35:37.570 INFO:tasks.workunit.client.1.smithi153.stdout:4/574: dwrite d3/d9/db/de/d2c/d38/fc3 [0,4194304] 0 2024-04-16T02:35:37.571 INFO:tasks.workunit.client.1.smithi153.stdout:1/497: truncate d4/dd/d13/f2a 3600376 0 2024-04-16T02:35:37.572 INFO:tasks.workunit.client.1.smithi153.stdout:0/583: creat dd/d14/d19/fae x:0 0 0 2024-04-16T02:35:37.572 INFO:tasks.workunit.client.1.smithi153.stdout:1/498: readlink d4/dd/d13/d26/l63 0 2024-04-16T02:35:37.574 INFO:tasks.workunit.client.1.smithi153.stdout:5/510: symlink d8/df/d2f/d4e/d51/l9f 0 2024-04-16T02:35:37.575 INFO:tasks.workunit.client.1.smithi153.stdout:8/427: creat d2/de/d3f/f91 x:0 0 0 2024-04-16T02:35:37.575 INFO:tasks.workunit.client.1.smithi153.stdout:7/457: truncate d0/d4/db/d2b/f22 16266 0 2024-04-16T02:35:37.576 INFO:tasks.workunit.client.1.smithi153.stdout:5/511: write d8/df/d28/f82 [775032,29866] 0 2024-04-16T02:35:37.580 INFO:tasks.workunit.client.1.smithi153.stdout:9/513: dread d4/d6/dc/d24/d5d/d66/f98 [0,4194304] 0 2024-04-16T02:35:37.580 INFO:tasks.workunit.client.1.smithi153.stdout:1/499: creat d4/dd/d13/d23/fab x:0 0 0 2024-04-16T02:35:37.581 INFO:tasks.workunit.client.1.smithi153.stdout:0/584: dwrite dd/d26/f5b [4194304,4194304] 0 2024-04-16T02:35:37.582 INFO:tasks.workunit.client.1.smithi153.stdout:9/514: write d4/d33/f4f [882481,88183] 0 2024-04-16T02:35:37.585 INFO:tasks.workunit.client.1.smithi153.stdout:7/458: creat d0/d4/db/d47/d8d/f98 x:0 0 0 2024-04-16T02:35:37.586 INFO:tasks.workunit.client.1.smithi153.stdout:5/512: creat d8/d1f/d20/d39/d6d/fa0 x:0 0 0 2024-04-16T02:35:37.587 INFO:tasks.workunit.client.1.smithi153.stdout:9/515: write d4/d6/dc/d46/d93/f87 [3867845,65582] 0 2024-04-16T02:35:37.590 INFO:tasks.workunit.client.1.smithi153.stdout:0/585: mknod dd/d32/d57/d6c/caf 0 2024-04-16T02:35:37.590 INFO:tasks.workunit.client.1.smithi153.stdout:4/575: getdents d3 0 2024-04-16T02:35:37.591 INFO:tasks.workunit.client.1.smithi153.stdout:7/459: write d0/d4/db/d2b/f3e [3625127,98296] 0 2024-04-16T02:35:37.591 INFO:tasks.workunit.client.1.smithi153.stdout:6/470: rename dc/d6b/l76 to dc/d1a/l9b 0 2024-04-16T02:35:37.594 INFO:tasks.workunit.client.1.smithi153.stdout:4/576: stat d3/d9/d24/d75/d9e/ca7 0 2024-04-16T02:35:37.594 INFO:tasks.workunit.client.1.smithi153.stdout:9/516: symlink d4/l9e 0 2024-04-16T02:35:37.594 INFO:tasks.workunit.client.1.smithi153.stdout:6/471: readlink dc/d47/d57/l67 0 2024-04-16T02:35:37.595 INFO:tasks.workunit.client.1.smithi153.stdout:0/586: mknod dd/d32/d80/cb0 0 2024-04-16T02:35:37.596 INFO:tasks.workunit.client.1.smithi153.stdout:7/460: creat d0/d4/db/d2b/d43/f99 x:0 0 0 2024-04-16T02:35:37.596 INFO:tasks.workunit.client.1.smithi153.stdout:1/500: dwrite d4/dd/d13/d26/d58/d7a/fa9 [0,4194304] 0 2024-04-16T02:35:37.598 INFO:tasks.workunit.client.1.smithi153.stdout:9/517: chown d4/d6/d1b/d2a/f70 3 1 2024-04-16T02:35:37.599 INFO:tasks.workunit.client.1.smithi153.stdout:0/587: write dd/d28/f3c [1891834,108509] 0 2024-04-16T02:35:37.600 INFO:tasks.workunit.client.1.smithi153.stdout:5/513: dread d8/df/d2e/f5b [0,4194304] 0 2024-04-16T02:35:37.601 INFO:tasks.workunit.client.1.smithi153.stdout:8/428: rmdir d2/de 39 2024-04-16T02:35:37.604 INFO:tasks.workunit.client.1.smithi153.stdout:5/514: write d8/d1f/f3f [1490847,120541] 0 2024-04-16T02:35:37.605 INFO:tasks.workunit.client.1.smithi153.stdout:6/472: mknod dc/d1a/c9c 0 2024-04-16T02:35:37.607 INFO:tasks.workunit.client.1.smithi153.stdout:5/515: readlink d8/df/d23/l31 0 2024-04-16T02:35:37.609 INFO:tasks.workunit.client.1.smithi153.stdout:1/501: dread d4/dd/d13/d26/f9d [0,4194304] 0 2024-04-16T02:35:37.610 INFO:tasks.workunit.client.1.smithi153.stdout:6/473: read dc/d10/f34 [2499953,105976] 0 2024-04-16T02:35:37.614 INFO:tasks.workunit.client.1.smithi153.stdout:8/429: dread d2/d1c/f26 [0,4194304] 0 2024-04-16T02:35:37.614 INFO:tasks.workunit.client.1.smithi153.stdout:0/588: dread dd/d14/da6/f48 [0,4194304] 0 2024-04-16T02:35:37.615 INFO:tasks.workunit.client.1.smithi153.stdout:0/589: fdatasync dd/f53 0 2024-04-16T02:35:37.616 INFO:tasks.workunit.client.1.smithi153.stdout:4/577: creat d3/d9/db/de/d2c/fc9 x:0 0 0 2024-04-16T02:35:37.616 INFO:tasks.workunit.client.1.smithi153.stdout:9/518: dwrite d4/d6/dc/d46/d93/d40/f6d [0,4194304] 0 2024-04-16T02:35:37.618 INFO:tasks.workunit.client.1.smithi153.stdout:5/516: rename d8/f59 to d8/df/d28/d5a/fa1 0 2024-04-16T02:35:37.623 INFO:tasks.workunit.client.1.smithi153.stdout:0/590: mkdir dd/d14/da6/da4/db1 0 2024-04-16T02:35:37.626 INFO:tasks.workunit.client.1.smithi153.stdout:9/519: unlink d4/d6/dc/d46/d93/d40/f6d 0 2024-04-16T02:35:37.627 INFO:tasks.workunit.client.1.smithi153.stdout:9/520: write d4/d6/d1b/f90 [286078,120084] 0 2024-04-16T02:35:37.634 INFO:tasks.workunit.client.1.smithi153.stdout:0/591: stat dd/d28/f4c 0 2024-04-16T02:35:37.636 INFO:tasks.workunit.client.1.smithi153.stdout:8/430: truncate d2/d13/d3a/f46 932974 0 2024-04-16T02:35:37.636 INFO:tasks.workunit.client.1.smithi153.stdout:7/461: dwrite d0/d4/f20 [0,4194304] 0 2024-04-16T02:35:37.639 INFO:tasks.workunit.client.1.smithi153.stdout:7/462: read d0/d4/db/d2b/d53/d59/f5a [145461,130968] 0 2024-04-16T02:35:37.640 INFO:tasks.workunit.client.1.smithi153.stdout:7/463: chown d0/d4/db/d2b/c77 238960 1 2024-04-16T02:35:37.641 INFO:tasks.workunit.client.1.smithi153.stdout:9/521: dwrite d4/d6/dc/d24/f30 [0,4194304] 0 2024-04-16T02:35:37.642 INFO:tasks.workunit.client.1.smithi153.stdout:6/474: getdents dc/d59 0 2024-04-16T02:35:37.644 INFO:tasks.workunit.client.1.smithi153.stdout:0/592: mknod dd/d32/d57/d6c/cb2 0 2024-04-16T02:35:37.645 INFO:tasks.workunit.client.1.smithi153.stdout:1/502: truncate d4/dd/d13/d26/f2b 1982336 0 2024-04-16T02:35:37.648 INFO:tasks.workunit.client.1.smithi153.stdout:9/522: dread d4/d6/f9d [0,4194304] 0 2024-04-16T02:35:37.648 INFO:tasks.workunit.client.1.smithi153.stdout:4/578: getdents d3/d9/db/d17/d3f/d6b 0 2024-04-16T02:35:37.650 INFO:tasks.workunit.client.1.smithi153.stdout:4/579: chown d3/d9/db/de/d2c/cac 1122232 1 2024-04-16T02:35:37.650 INFO:tasks.workunit.client.1.smithi153.stdout:1/503: mknod d4/dd/d13/d23/d5f/cac 0 2024-04-16T02:35:37.653 INFO:tasks.workunit.client.1.smithi153.stdout:8/431: getdents d2/de/d3f 0 2024-04-16T02:35:37.654 INFO:tasks.workunit.client.1.smithi153.stdout:7/464: link d0/d4/db/d2b/d8c/c8e d0/d4/db/d87/c9a 0 2024-04-16T02:35:37.656 INFO:tasks.workunit.client.1.smithi153.stdout:9/523: getdents d4/d6/dc/d24 0 2024-04-16T02:35:37.657 INFO:tasks.workunit.client.1.smithi153.stdout:1/504: getdents d4/dd/d13/d26/d58 0 2024-04-16T02:35:37.657 INFO:tasks.workunit.client.1.smithi153.stdout:7/465: chown d0/f1e 10772 1 2024-04-16T02:35:37.658 INFO:tasks.workunit.client.1.smithi153.stdout:4/580: dwrite d3/d9/d79/fc4 [0,4194304] 0 2024-04-16T02:35:37.661 INFO:tasks.workunit.client.1.smithi153.stdout:8/432: rename d2/d13/d3a/d66 to d2/d92 0 2024-04-16T02:35:37.665 INFO:tasks.workunit.client.1.smithi153.stdout:1/505: chown d4/c2d 148 1 2024-04-16T02:35:37.666 INFO:tasks.workunit.client.1.smithi153.stdout:9/524: creat d4/d6/dc/d24/f9f x:0 0 0 2024-04-16T02:35:37.667 INFO:tasks.workunit.client.1.smithi153.stdout:7/466: dwrite d0/d4/db/f64 [0,4194304] 0 2024-04-16T02:35:37.668 INFO:tasks.workunit.client.1.smithi153.stdout:1/506: dread - d4/dd/d13/d26/f92 zero size 2024-04-16T02:35:37.668 INFO:tasks.workunit.client.1.smithi153.stdout:9/525: fdatasync d4/d6/d1b/f90 0 2024-04-16T02:35:37.669 INFO:tasks.workunit.client.1.smithi153.stdout:4/581: rename d3/d9/db/d17/d28/f9c to d3/d9/db/de/d2c/d98/fca 0 2024-04-16T02:35:37.669 INFO:tasks.workunit.client.0.smithi151.stdout:5/94: sync 2024-04-16T02:35:37.670 INFO:tasks.workunit.client.1.smithi153.stdout:4/582: chown d3/d9/db/d84 87111 1 2024-04-16T02:35:37.670 INFO:tasks.workunit.client.1.smithi153.stdout:8/433: fsync d2/d13/d2c/f8d 0 2024-04-16T02:35:37.671 INFO:tasks.workunit.client.0.smithi151.stdout:5/95: rmdir d2/d6 39 2024-04-16T02:35:37.673 INFO:tasks.workunit.client.1.smithi153.stdout:3/452: sync 2024-04-16T02:35:37.675 INFO:tasks.workunit.client.1.smithi153.stdout:6/475: dread dc/d47/d57/d77/d15/f22 [0,4194304] 0 2024-04-16T02:35:37.676 INFO:tasks.workunit.client.1.smithi153.stdout:1/507: symlink d4/dd/d18/d29/d71/lad 0 2024-04-16T02:35:37.676 INFO:tasks.workunit.client.1.smithi153.stdout:9/526: dwrite d4/d6/dc/d46/f49 [0,4194304] 0 2024-04-16T02:35:37.676 INFO:tasks.workunit.client.0.smithi151.stdout:5/96: creat d2/d6/f1c x:0 0 0 2024-04-16T02:35:37.678 INFO:tasks.workunit.client.1.smithi153.stdout:4/583: chown d3/d9/db/d17/d28/c7d 56347 1 2024-04-16T02:35:37.679 INFO:tasks.workunit.client.1.smithi153.stdout:4/584: dread - d3/d9/db/d17/d28/f5e zero size 2024-04-16T02:35:37.683 INFO:tasks.workunit.client.1.smithi153.stdout:3/453: mknod d6/ca2 0 2024-04-16T02:35:37.685 INFO:tasks.workunit.client.1.smithi153.stdout:3/454: write d6/d10/d22/f4c [1956902,48901] 0 2024-04-16T02:35:37.688 INFO:tasks.workunit.client.0.smithi151.stdout:5/97: write d2/fd [3218726,120941] 0 2024-04-16T02:35:37.688 INFO:tasks.workunit.client.0.smithi151.stdout:5/98: readlink d2/l10 0 2024-04-16T02:35:37.690 INFO:tasks.workunit.client.0.smithi151.stdout:5/99: write d2/d6/f8 [600622,57878] 0 2024-04-16T02:35:37.691 INFO:tasks.workunit.client.0.smithi151.stdout:5/100: fdatasync d2/d6/f9 0 2024-04-16T02:35:37.691 INFO:tasks.workunit.client.0.smithi151.stdout:5/101: chown d2/d6 3807 1 2024-04-16T02:35:37.695 INFO:tasks.workunit.client.0.smithi151.stdout:5/102: getdents d2/d6/d19/d1a 0 2024-04-16T02:35:37.701 INFO:tasks.workunit.client.0.smithi151.stdout:5/103: dwrite d2/d6/fa [0,4194304] 0 2024-04-16T02:35:37.701 INFO:tasks.workunit.client.0.smithi151.stdout:5/104: stat d2/d6/fa 0 2024-04-16T02:35:37.704 INFO:tasks.workunit.client.0.smithi151.stdout:5/105: creat d2/d6/d19/f1d x:0 0 0 2024-04-16T02:35:37.708 INFO:tasks.workunit.client.0.smithi151.stdout:5/106: dread d2/d6/f8 [0,4194304] 0 2024-04-16T02:35:37.709 INFO:tasks.workunit.client.0.smithi151.stdout:5/107: creat d2/d6/f1e x:0 0 0 2024-04-16T02:35:37.714 INFO:tasks.workunit.client.1.smithi153.stdout:6/476: symlink dc/l9d 0 2024-04-16T02:35:37.714 INFO:tasks.workunit.client.1.smithi153.stdout:4/585: creat d3/da5/fcb x:0 0 0 2024-04-16T02:35:37.715 INFO:tasks.workunit.client.0.smithi151.stdout:5/108: dwrite d2/d6/fa [0,4194304] 0 2024-04-16T02:35:37.716 INFO:tasks.workunit.client.1.smithi153.stdout:6/477: chown dc/d6b/f73 1 1 2024-04-16T02:35:37.716 INFO:tasks.workunit.client.1.smithi153.stdout:9/527: dwrite d4/d6/dc/d2d/d64/f8d [0,4194304] 0 2024-04-16T02:35:37.716 INFO:tasks.workunit.client.1.smithi153.stdout:3/455: write d6/d10/d22/d2e/f45 [3888687,124009] 0 2024-04-16T02:35:37.716 INFO:tasks.workunit.client.1.smithi153.stdout:8/434: truncate d2/d1c/d1f/d27/d5e/f73 1241034 0 2024-04-16T02:35:37.718 INFO:tasks.workunit.client.1.smithi153.stdout:1/508: creat d4/fae x:0 0 0 2024-04-16T02:35:37.719 INFO:tasks.workunit.client.1.smithi153.stdout:8/435: stat d2/d13/d3a/d5b/l6d 0 2024-04-16T02:35:37.721 INFO:tasks.workunit.client.1.smithi153.stdout:9/528: write d4/d6/dc/d46/d93/f7c [214646,96450] 0 2024-04-16T02:35:37.722 INFO:tasks.workunit.client.1.smithi153.stdout:3/456: mkdir d6/d10/d22/d4d/da0/da3 0 2024-04-16T02:35:37.725 INFO:tasks.workunit.client.0.smithi151.stdout:5/109: dwrite d2/d6/d19/f1d [0,4194304] 0 2024-04-16T02:35:37.727 INFO:tasks.workunit.client.1.smithi153.stdout:4/586: dwrite d3/da5/fcb [0,4194304] 0 2024-04-16T02:35:37.730 INFO:tasks.workunit.client.1.smithi153.stdout:9/529: rename d4/d6/dc/d46/d93 to d4/d6/dc/d2d/da0 0 2024-04-16T02:35:37.731 INFO:tasks.workunit.client.0.smithi151.stdout:5/110: creat d2/f1f x:0 0 0 2024-04-16T02:35:37.732 INFO:tasks.workunit.client.0.smithi151.stdout:5/111: readlink d2/l17 0 2024-04-16T02:35:37.733 INFO:tasks.workunit.client.1.smithi153.stdout:1/509: getdents d4/da2/d65/d84 0 2024-04-16T02:35:37.736 INFO:tasks.workunit.client.1.smithi153.stdout:4/587: link d3/d9/db/d17/f5d d3/d9/db/d45/fcc 0 2024-04-16T02:35:37.737 INFO:tasks.workunit.client.0.smithi151.stdout:5/112: rename d2/fd to d2/f20 0 2024-04-16T02:35:37.738 INFO:tasks.workunit.client.1.smithi153.stdout:1/510: getdents d4/dd/d13/d7e/d80 0 2024-04-16T02:35:37.740 INFO:tasks.workunit.client.0.smithi151.stdout:5/113: mknod d2/c21 0 2024-04-16T02:35:37.741 INFO:tasks.workunit.client.1.smithi153.stdout:9/530: read d4/f48 [3917183,87452] 0 2024-04-16T02:35:37.741 INFO:tasks.workunit.client.1.smithi153.stdout:1/511: mkdir d4/dd/d18/d6c/daf 0 2024-04-16T02:35:37.741 INFO:tasks.workunit.client.1.smithi153.stdout:2/483: sync 2024-04-16T02:35:37.743 INFO:tasks.workunit.client.0.smithi151.stdout:5/114: rename d2/d6/l14 to d2/d6/d19/l22 0 2024-04-16T02:35:37.743 INFO:tasks.workunit.client.1.smithi153.stdout:4/588: creat d3/d9/db/de/fcd x:0 0 0 2024-04-16T02:35:37.743 INFO:tasks.workunit.client.1.smithi153.stdout:9/531: chown d4/f27 63 1 2024-04-16T02:35:37.743 INFO:tasks.workunit.client.1.smithi153.stdout:8/436: dread d2/f23 [0,4194304] 0 2024-04-16T02:35:37.744 INFO:tasks.workunit.client.1.smithi153.stdout:2/484: chown d9/d2b/ff 2 1 2024-04-16T02:35:37.744 INFO:tasks.workunit.client.0.smithi151.stdout:5/115: mknod d2/d6/d19/c23 0 2024-04-16T02:35:37.746 INFO:tasks.workunit.client.1.smithi153.stdout:2/485: read d9/d24/d27/f68 [982202,11064] 0 2024-04-16T02:35:37.746 INFO:tasks.workunit.client.1.smithi153.stdout:3/457: dread d6/d10/d22/f31 [0,4194304] 0 2024-04-16T02:35:37.748 INFO:tasks.workunit.client.0.smithi151.stdout:5/116: dread d2/d6/fa [0,4194304] 0 2024-04-16T02:35:37.748 INFO:tasks.workunit.client.1.smithi153.stdout:9/532: rename d4/d6/d1b/c4d to d4/d6/dc/d7f/ca1 0 2024-04-16T02:35:37.751 INFO:tasks.workunit.client.1.smithi153.stdout:4/589: dwrite d3/d9/db/d17/f8c [0,4194304] 0 2024-04-16T02:35:37.754 INFO:tasks.workunit.client.1.smithi153.stdout:1/512: link d4/f5e d4/da2/d65/d82/fb0 0 2024-04-16T02:35:37.754 INFO:tasks.workunit.client.1.smithi153.stdout:4/590: write d3/d9/db/de/f12 [2824352,19016] 0 2024-04-16T02:35:37.754 INFO:tasks.workunit.client.1.smithi153.stdout:6/478: dread dc/d1a/d2d/f44 [0,4194304] 0 2024-04-16T02:35:37.757 INFO:tasks.workunit.client.1.smithi153.stdout:4/591: readlink d3/d9/db/de/l53 0 2024-04-16T02:35:37.757 INFO:tasks.workunit.client.1.smithi153.stdout:2/486: dwrite d9/d2b/d18/f29 [0,4194304] 0 2024-04-16T02:35:37.766 INFO:tasks.workunit.client.1.smithi153.stdout:2/487: rmdir d9/d2b/d5c/d5f 39 2024-04-16T02:35:37.769 INFO:tasks.workunit.client.1.smithi153.stdout:4/592: creat d3/d9/d24/d75/d9e/fce x:0 0 0 2024-04-16T02:35:37.770 INFO:tasks.workunit.client.1.smithi153.stdout:8/437: write d2/f7 [3094614,27064] 0 2024-04-16T02:35:37.771 INFO:tasks.workunit.client.0.smithi151.stdout:5/117: dwrite d2/fb [0,4194304] 0 2024-04-16T02:35:37.775 INFO:tasks.workunit.client.1.smithi153.stdout:5/517: sync 2024-04-16T02:35:37.775 INFO:tasks.workunit.client.0.smithi151.stdout:5/118: creat d2/f24 x:0 0 0 2024-04-16T02:35:37.775 INFO:tasks.workunit.client.1.smithi153.stdout:5/518: dread - d8/df/d2f/f60 zero size 2024-04-16T02:35:37.776 INFO:tasks.workunit.client.1.smithi153.stdout:2/488: rename d9/d24/d27/f6f to d9/d24/d27/d2c/d41/d7a/fb2 0 2024-04-16T02:35:37.777 INFO:tasks.workunit.client.1.smithi153.stdout:4/593: stat d3/d9/db/de/c3c 0 2024-04-16T02:35:37.778 INFO:tasks.workunit.client.1.smithi153.stdout:5/519: dread - d8/d1f/d20/d39/d6d/fa0 zero size 2024-04-16T02:35:37.779 INFO:tasks.workunit.client.0.smithi151.stdout:5/119: dread d2/d6/f9 [0,4194304] 0 2024-04-16T02:35:37.784 INFO:tasks.workunit.client.1.smithi153.stdout:8/438: mkdir d2/d93 0 2024-04-16T02:35:37.785 INFO:tasks.workunit.client.1.smithi153.stdout:4/594: creat d3/d9/d24/d3e/fcf x:0 0 0 2024-04-16T02:35:37.786 INFO:tasks.workunit.client.1.smithi153.stdout:1/513: truncate d4/dd/d13/d23/d5f/f61 1683982 0 2024-04-16T02:35:37.787 INFO:tasks.workunit.client.0.smithi151.stdout:5/120: truncate d2/fb 2331854 0 2024-04-16T02:35:37.787 INFO:tasks.workunit.client.1.smithi153.stdout:2/489: dwrite d9/d2b/f16 [4194304,4194304] 0 2024-04-16T02:35:37.787 INFO:tasks.workunit.client.1.smithi153.stdout:5/520: write d8/df/d2e/f5b [1654247,40492] 0 2024-04-16T02:35:37.789 INFO:tasks.workunit.client.0.smithi151.stdout:5/121: creat d2/d6/f25 x:0 0 0 2024-04-16T02:35:37.789 INFO:tasks.workunit.client.1.smithi153.stdout:1/514: fsync d4/f40 0 2024-04-16T02:35:37.789 INFO:tasks.workunit.client.0.smithi151.stdout:5/122: readlink d2/d6/l15 0 2024-04-16T02:35:37.790 INFO:tasks.workunit.client.1.smithi153.stdout:3/458: dwrite d6/f39 [0,4194304] 0 2024-04-16T02:35:37.790 INFO:tasks.workunit.client.0.smithi151.stdout:5/123: read - d2/f1f zero size 2024-04-16T02:35:37.792 INFO:tasks.workunit.client.1.smithi153.stdout:6/479: dwrite dc/d6b/f6f [0,4194304] 0 2024-04-16T02:35:37.795 INFO:tasks.workunit.client.1.smithi153.stdout:3/459: chown d6/d10/d3b/d76 407915859 1 2024-04-16T02:35:37.795 INFO:tasks.workunit.client.1.smithi153.stdout:9/533: truncate d4/d6/dc/d24/f43 905785 0 2024-04-16T02:35:37.802 INFO:tasks.workunit.client.0.smithi151.stdout:5/124: rmdir d2 39 2024-04-16T02:35:37.803 INFO:tasks.workunit.client.1.smithi153.stdout:2/490: symlink d9/d2b/d1c/lb3 0 2024-04-16T02:35:37.803 INFO:tasks.workunit.client.1.smithi153.stdout:5/521: fdatasync d8/df/d23/f32 0 2024-04-16T02:35:37.805 INFO:tasks.workunit.client.1.smithi153.stdout:6/480: mknod dc/d47/d93/c9e 0 2024-04-16T02:35:37.805 INFO:tasks.workunit.client.0.smithi151.stdout:5/125: truncate d2/f24 1023993 0 2024-04-16T02:35:37.805 INFO:tasks.workunit.client.1.smithi153.stdout:1/515: creat d4/dd/d13/d7e/fb1 x:0 0 0 2024-04-16T02:35:37.805 INFO:tasks.workunit.client.1.smithi153.stdout:5/522: fdatasync d8/df/d28/d5a/f68 0 2024-04-16T02:35:37.807 INFO:tasks.workunit.client.1.smithi153.stdout:2/491: chown d9/d2b/f86 31735432 1 2024-04-16T02:35:37.807 INFO:tasks.workunit.client.1.smithi153.stdout:4/595: dwrite d3/d9/db/de/d2c/f74 [4194304,4194304] 0 2024-04-16T02:35:37.807 INFO:tasks.workunit.client.0.smithi151.stdout:5/126: mknod d2/c26 0 2024-04-16T02:35:37.808 INFO:tasks.workunit.client.1.smithi153.stdout:8/439: creat d2/d13/f94 x:0 0 0 2024-04-16T02:35:37.808 INFO:tasks.workunit.client.1.smithi153.stdout:3/460: write d6/d10/d22/d2c/f5f [590032,130749] 0 2024-04-16T02:35:37.809 INFO:tasks.workunit.client.1.smithi153.stdout:6/481: stat dc/d1a/d2d/f44 0 2024-04-16T02:35:37.810 INFO:tasks.workunit.client.1.smithi153.stdout:9/534: link d4/d6/dc/d2d/d79/f8f d4/d6/dc/d2d/fa2 0 2024-04-16T02:35:37.811 INFO:tasks.workunit.client.1.smithi153.stdout:2/492: readlink d9/d24/l4b 0 2024-04-16T02:35:37.811 INFO:tasks.workunit.client.0.smithi151.stdout:5/127: rename d2/f20 to d2/d6/f27 0 2024-04-16T02:35:37.812 INFO:tasks.workunit.client.1.smithi153.stdout:5/523: truncate d8/df/d28/d5a/fa1 435058 0 2024-04-16T02:35:37.812 INFO:tasks.workunit.client.1.smithi153.stdout:8/440: truncate d2/d1c/f85 322558 0 2024-04-16T02:35:37.812 INFO:tasks.workunit.client.1.smithi153.stdout:6/482: stat dc/d1a/f1c 0 2024-04-16T02:35:37.813 INFO:tasks.workunit.client.0.smithi151.stdout:5/128: symlink d2/d6/d19/l28 0 2024-04-16T02:35:37.813 INFO:tasks.workunit.client.1.smithi153.stdout:1/516: creat d4/dd/d18/d29/d6e/fb2 x:0 0 0 2024-04-16T02:35:37.815 INFO:tasks.workunit.client.0.smithi151.stdout:5/129: readlink d2/d6/d19/l22 0 2024-04-16T02:35:37.815 INFO:tasks.workunit.client.1.smithi153.stdout:5/524: write d8/df/d28/f82 [82787,60381] 0 2024-04-16T02:35:37.816 INFO:tasks.workunit.client.1.smithi153.stdout:0/593: sync 2024-04-16T02:35:37.816 INFO:tasks.workunit.client.1.smithi153.stdout:7/467: sync 2024-04-16T02:35:37.818 INFO:tasks.workunit.client.0.smithi151.stdout:5/130: creat d2/d6/f29 x:0 0 0 2024-04-16T02:35:37.818 INFO:tasks.workunit.client.1.smithi153.stdout:1/517: truncate d4/da2/d65/d84/fa6 555887 0 2024-04-16T02:35:37.819 INFO:tasks.workunit.client.0.smithi151.stdout:5/131: write d2/d6/f9 [3863146,120582] 0 2024-04-16T02:35:37.820 INFO:tasks.workunit.client.0.smithi151.stdout:5/132: read - d2/f16 zero size 2024-04-16T02:35:37.821 INFO:tasks.workunit.client.1.smithi153.stdout:0/594: stat dd/d14/d19/d55/c7c 0 2024-04-16T02:35:37.821 INFO:tasks.workunit.client.1.smithi153.stdout:2/493: read d9/f9e [2385842,44427] 0 2024-04-16T02:35:37.822 INFO:tasks.workunit.client.1.smithi153.stdout:4/596: dwrite d3/d9/f3d [8388608,4194304] 0 2024-04-16T02:35:37.826 INFO:tasks.workunit.client.1.smithi153.stdout:5/525: write d8/df/d28/f35 [4760581,51207] 0 2024-04-16T02:35:37.827 INFO:tasks.workunit.client.1.smithi153.stdout:0/595: dread - dd/d14/d19/fa0 zero size 2024-04-16T02:35:37.827 INFO:tasks.workunit.client.1.smithi153.stdout:7/468: creat d0/d4/db/d2b/d8c/f9b x:0 0 0 2024-04-16T02:35:37.827 INFO:tasks.workunit.client.0.smithi151.stdout:5/133: dwrite d2/f1f [0,4194304] 0 2024-04-16T02:35:37.828 INFO:tasks.workunit.client.1.smithi153.stdout:2/494: truncate d9/d2b/d1c/d20/f79 5083822 0 2024-04-16T02:35:37.830 INFO:tasks.workunit.client.1.smithi153.stdout:9/535: rmdir d4/d6/dc 39 2024-04-16T02:35:37.830 INFO:tasks.workunit.client.1.smithi153.stdout:2/495: fdatasync d9/d2b/f6b 0 2024-04-16T02:35:37.831 INFO:tasks.workunit.client.0.smithi151.stdout:5/134: write d2/d6/f27 [1737709,72087] 0 2024-04-16T02:35:37.833 INFO:tasks.workunit.client.0.smithi151.stdout:5/135: creat d2/d6/f2a x:0 0 0 2024-04-16T02:35:37.836 INFO:tasks.workunit.client.1.smithi153.stdout:5/526: mknod d8/d41/d90/d9d/ca2 0 2024-04-16T02:35:37.838 INFO:tasks.workunit.client.0.smithi151.stdout:5/136: symlink d2/l2b 0 2024-04-16T02:35:37.841 INFO:tasks.workunit.client.1.smithi153.stdout:0/596: mknod dd/d14/d51/d7d/cb3 0 2024-04-16T02:35:37.842 INFO:tasks.workunit.client.1.smithi153.stdout:0/597: fdatasync dd/d14/d51/fa2 0 2024-04-16T02:35:37.844 INFO:tasks.workunit.client.1.smithi153.stdout:9/536: fdatasync d4/d6/dc/d2d/fa2 0 2024-04-16T02:35:37.844 INFO:tasks.workunit.client.1.smithi153.stdout:5/527: dread d8/df/d28/d64/f98 [0,4194304] 0 2024-04-16T02:35:37.844 INFO:tasks.workunit.client.1.smithi153.stdout:9/537: stat d4/d6 0 2024-04-16T02:35:37.845 INFO:tasks.workunit.client.0.smithi151.stdout:5/137: dwrite d2/d6/f9 [0,4194304] 0 2024-04-16T02:35:37.847 INFO:tasks.workunit.client.1.smithi153.stdout:7/469: link d0/d4/db/f50 d0/d4/db/d2b/d43/d6d/d68/f9c 0 2024-04-16T02:35:37.847 INFO:tasks.workunit.client.0.smithi151.stdout:5/138: write d2/d6/f1c [1013574,15804] 0 2024-04-16T02:35:37.847 INFO:tasks.workunit.client.1.smithi153.stdout:3/461: truncate d6/d10/d22/d2c/f58 3482358 0 2024-04-16T02:35:37.851 INFO:tasks.workunit.client.0.smithi151.stdout:5/139: dread d2/f24 [0,4194304] 0 2024-04-16T02:35:37.851 INFO:tasks.workunit.client.1.smithi153.stdout:4/597: getdents d3/d9/db/de/d2c/d38 0 2024-04-16T02:35:37.853 INFO:tasks.workunit.client.1.smithi153.stdout:9/538: dread d4/d6/dc/d2d/d64/f8d [0,4194304] 0 2024-04-16T02:35:37.853 INFO:tasks.workunit.client.0.smithi151.stdout:5/140: symlink d2/d6/d19/l2c 0 2024-04-16T02:35:37.855 INFO:tasks.workunit.client.0.smithi151.stdout:5/141: fsync d2/d6/f8 0 2024-04-16T02:35:37.856 INFO:tasks.workunit.client.1.smithi153.stdout:5/528: dwrite d8/df/d28/f35 [0,4194304] 0 2024-04-16T02:35:37.858 INFO:tasks.workunit.client.1.smithi153.stdout:0/598: rename dd/d32/la1 to dd/d26/d92/lb4 0 2024-04-16T02:35:37.858 INFO:tasks.workunit.client.1.smithi153.stdout:8/441: dread d2/ff [0,4194304] 0 2024-04-16T02:35:37.859 INFO:tasks.workunit.client.0.smithi151.stdout:5/142: creat d2/f2d x:0 0 0 2024-04-16T02:35:37.859 INFO:tasks.workunit.client.1.smithi153.stdout:9/539: unlink d4/d6/dc/d2d/da0/l4c 0 2024-04-16T02:35:37.862 INFO:tasks.workunit.client.1.smithi153.stdout:5/529: rename d8/df/d23/l4b to d8/df/d84/la3 0 2024-04-16T02:35:37.862 INFO:tasks.workunit.client.1.smithi153.stdout:0/599: readlink dd/d14/d19/d20/d4a/l97 0 2024-04-16T02:35:37.863 INFO:tasks.workunit.client.1.smithi153.stdout:3/462: link d6/c17 d6/d10/d22/d4d/da0/ca4 0 2024-04-16T02:35:37.864 INFO:tasks.workunit.client.1.smithi153.stdout:9/540: readlink d4/d33/l44 0 2024-04-16T02:35:37.865 INFO:tasks.workunit.client.1.smithi153.stdout:2/496: dread d9/d24/d27/d2c/d41/f44 [0,4194304] 0 2024-04-16T02:35:37.865 INFO:tasks.workunit.client.0.smithi151.stdout:5/143: dwrite d2/f1f [0,4194304] 0 2024-04-16T02:35:37.866 INFO:tasks.workunit.client.1.smithi153.stdout:1/518: dread d4/dd/d13/d26/f34 [0,4194304] 0 2024-04-16T02:35:37.867 INFO:tasks.workunit.client.1.smithi153.stdout:4/598: dwrite d3/d9/f6c [0,4194304] 0 2024-04-16T02:35:37.870 INFO:tasks.workunit.client.0.smithi151.stdout:5/144: creat d2/d6/d19/d1a/f2e x:0 0 0 2024-04-16T02:35:37.872 INFO:tasks.workunit.client.1.smithi153.stdout:5/530: chown d8/df/c1b 9577 1 2024-04-16T02:35:37.872 INFO:tasks.workunit.client.1.smithi153.stdout:4/599: dread - d3/d9/db/d17/d3f/f94 zero size 2024-04-16T02:35:37.873 INFO:tasks.workunit.client.0.smithi151.stdout:5/145: creat d2/d6/f2f x:0 0 0 2024-04-16T02:35:37.873 INFO:tasks.workunit.client.0.smithi151.stdout:5/146: stat d2/l2b 0 2024-04-16T02:35:37.876 INFO:tasks.workunit.client.1.smithi153.stdout:0/600: dwrite dd/d26/f5e [0,4194304] 0 2024-04-16T02:35:37.876 INFO:tasks.workunit.client.1.smithi153.stdout:3/463: truncate d6/d10/d22/f28 2267360 0 2024-04-16T02:35:37.877 INFO:tasks.workunit.client.1.smithi153.stdout:9/541: creat d4/d6/dc/d7f/fa3 x:0 0 0 2024-04-16T02:35:37.879 INFO:tasks.workunit.client.1.smithi153.stdout:8/442: creat d2/d1c/d1f/d27/d2d/d47/d77/d8e/f95 x:0 0 0 2024-04-16T02:35:37.879 INFO:tasks.workunit.client.1.smithi153.stdout:5/531: mkdir d8/df/d84/da4 0 2024-04-16T02:35:37.879 INFO:tasks.workunit.client.1.smithi153.stdout:4/600: rename d3 to d3/d9/db/de/dd0 22 2024-04-16T02:35:37.881 INFO:tasks.workunit.client.1.smithi153.stdout:5/532: chown d8/df/d23 0 1 2024-04-16T02:35:37.881 INFO:tasks.workunit.client.1.smithi153.stdout:0/601: rmdir dd/d28 39 2024-04-16T02:35:37.881 INFO:tasks.workunit.client.0.smithi151.stdout:5/147: dwrite d2/d6/f8 [0,4194304] 0 2024-04-16T02:35:37.884 INFO:tasks.workunit.client.1.smithi153.stdout:3/464: symlink d6/d10/d5b/la5 0 2024-04-16T02:35:37.884 INFO:tasks.workunit.client.1.smithi153.stdout:3/465: fdatasync d6/d10/d3b/d59/f87 0 2024-04-16T02:35:37.907 INFO:tasks.workunit.client.1.smithi153.stdout:1/519: creat d4/dd/d13/fb3 x:0 0 0 2024-04-16T02:35:37.907 INFO:tasks.workunit.client.1.smithi153.stdout:5/533: symlink d8/df/d2e/la5 0 2024-04-16T02:35:37.908 INFO:tasks.workunit.client.1.smithi153.stdout:5/534: chown d8/d1f/c75 16127 1 2024-04-16T02:35:37.910 INFO:tasks.workunit.client.0.smithi151.stdout:5/148: getdents d2/d6/d19/d1a 0 2024-04-16T02:35:37.910 INFO:tasks.workunit.client.1.smithi153.stdout:4/601: rename d3/d9/db/de/c5b to d3/d9/db/d17/d28/dc8/cd1 0 2024-04-16T02:35:37.910 INFO:tasks.workunit.client.0.smithi151.stdout:5/149: readlink d2/d6/l15 0 2024-04-16T02:35:37.911 INFO:tasks.workunit.client.1.smithi153.stdout:0/602: creat dd/d14/da6/da4/fb5 x:0 0 0 2024-04-16T02:35:37.912 INFO:tasks.workunit.client.0.smithi151.stdout:5/150: mkdir d2/d6/d19/d30 0 2024-04-16T02:35:37.912 INFO:tasks.workunit.client.1.smithi153.stdout:1/520: dread d4/dd/d13/d26/f34 [0,4194304] 0 2024-04-16T02:35:37.913 INFO:tasks.workunit.client.1.smithi153.stdout:0/603: truncate dd/d14/da6/da4/fa8 267104 0 2024-04-16T02:35:37.913 INFO:tasks.workunit.client.1.smithi153.stdout:8/443: mkdir d2/d13/d96 0 2024-04-16T02:35:37.913 INFO:tasks.workunit.client.1.smithi153.stdout:1/521: readlink d4/dd/d13/d26/l68 0 2024-04-16T02:35:37.914 INFO:tasks.workunit.client.0.smithi151.stdout:5/151: symlink d2/d6/l31 0 2024-04-16T02:35:37.914 INFO:tasks.workunit.client.1.smithi153.stdout:1/522: fdatasync d4/da2/d65/d84/fa6 0 2024-04-16T02:35:37.914 INFO:tasks.workunit.client.0.smithi151.stdout:5/152: dread - d2/d6/f2a zero size 2024-04-16T02:35:37.915 INFO:tasks.workunit.client.1.smithi153.stdout:8/444: chown d2/d1c/d1f/f3c 0 1 2024-04-16T02:35:37.916 INFO:tasks.workunit.client.0.smithi151.stdout:5/153: fdatasync d2/d6/f1c 0 2024-04-16T02:35:37.916 INFO:tasks.workunit.client.1.smithi153.stdout:5/535: mknod d8/d41/d90/d9d/ca6 0 2024-04-16T02:35:37.917 INFO:tasks.workunit.client.1.smithi153.stdout:4/602: mkdir d3/da5/dd2 0 2024-04-16T02:35:37.918 INFO:tasks.workunit.client.1.smithi153.stdout:2/497: truncate d9/d2b/d1c/f90 1109514 0 2024-04-16T02:35:37.921 INFO:tasks.workunit.client.1.smithi153.stdout:0/604: write dd/f22 [2578684,113380] 0 2024-04-16T02:35:37.921 INFO:tasks.workunit.client.1.smithi153.stdout:0/605: read - dd/d32/f85 zero size 2024-04-16T02:35:37.922 INFO:tasks.workunit.client.0.smithi151.stdout:5/154: truncate d2/d6/f27 521197 0 2024-04-16T02:35:37.922 INFO:tasks.workunit.client.0.smithi151.stdout:5/155: stat d2/d6/d19/d1a/f2e 0 2024-04-16T02:35:37.923 INFO:tasks.workunit.client.1.smithi153.stdout:5/536: rmdir d8/df/d28/d64 39 2024-04-16T02:35:37.923 INFO:tasks.workunit.client.0.smithi151.stdout:5/156: readlink d2/d6/d19/l22 0 2024-04-16T02:35:37.925 INFO:tasks.workunit.client.0.smithi151.stdout:5/157: creat d2/d6/f32 x:0 0 0 2024-04-16T02:35:37.925 INFO:tasks.workunit.client.1.smithi153.stdout:1/523: truncate d4/dd/d13/d23/f5d 415808 0 2024-04-16T02:35:37.925 INFO:tasks.workunit.client.1.smithi153.stdout:4/603: creat d3/d9/d79/fd3 x:0 0 0 2024-04-16T02:35:37.926 INFO:tasks.workunit.client.0.smithi151.stdout:5/158: truncate d2/d6/f2a 209997 0 2024-04-16T02:35:37.926 INFO:tasks.workunit.client.0.smithi151.stdout:5/159: chown d2/l17 148562411 1 2024-04-16T02:35:37.928 INFO:tasks.workunit.client.1.smithi153.stdout:8/445: mkdir d2/d93/d97 0 2024-04-16T02:35:37.929 INFO:tasks.workunit.client.1.smithi153.stdout:0/606: mkdir dd/d14/d19/d77/db6 0 2024-04-16T02:35:37.932 INFO:tasks.workunit.client.1.smithi153.stdout:5/537: rename d8/df/d23/l31 to d8/df/d23/d70/la7 0 2024-04-16T02:35:37.932 INFO:tasks.workunit.client.1.smithi153.stdout:1/524: dwrite d4/dd/d13/d26/d58/d7a/fa9 [0,4194304] 0 2024-04-16T02:35:37.933 INFO:tasks.workunit.client.0.smithi151.stdout:5/160: write d2/d6/f27 [944163,47735] 0 2024-04-16T02:35:37.934 INFO:tasks.workunit.client.1.smithi153.stdout:5/538: readlink d8/df/d28/l86 0 2024-04-16T02:35:37.935 INFO:tasks.workunit.client.0.smithi151.stdout:5/161: mkdir d2/d6/d19/d30/d33 0 2024-04-16T02:35:37.940 INFO:tasks.workunit.client.1.smithi153.stdout:3/466: fsync d6/d10/d22/d43/f57 0 2024-04-16T02:35:37.941 INFO:tasks.workunit.client.1.smithi153.stdout:4/604: mknod d3/da5/dd2/cd4 0 2024-04-16T02:35:37.941 INFO:tasks.workunit.client.1.smithi153.stdout:8/446: write d2/d13/f62 [2434982,94429] 0 2024-04-16T02:35:37.943 INFO:tasks.workunit.client.1.smithi153.stdout:3/467: truncate d6/d10/d22/d2e/f80 570445 0 2024-04-16T02:35:37.944 INFO:tasks.workunit.client.0.smithi151.stdout:5/162: dwrite d2/d6/f2a [0,4194304] 0 2024-04-16T02:35:37.944 INFO:tasks.workunit.client.1.smithi153.stdout:5/539: mkdir d8/df/d28/d5a/d66/da8 0 2024-04-16T02:35:37.944 INFO:tasks.workunit.client.0.smithi151.stdout:5/163: chown d2/d6/d19/l28 3 1 2024-04-16T02:35:37.946 INFO:tasks.workunit.client.1.smithi153.stdout:4/605: mknod d3/d9/d24/d3e/cd5 0 2024-04-16T02:35:37.947 INFO:tasks.workunit.client.1.smithi153.stdout:5/540: write d8/df/d2e/f5b [3005568,95133] 0 2024-04-16T02:35:37.948 INFO:tasks.workunit.client.1.smithi153.stdout:6/483: sync 2024-04-16T02:35:37.949 INFO:tasks.workunit.client.1.smithi153.stdout:0/607: dwrite dd/d14/d51/d54/d65/f69 [0,4194304] 0 2024-04-16T02:35:37.949 INFO:tasks.workunit.client.1.smithi153.stdout:8/447: mkdir d2/d1c/d1f/d27/d2d/d47/d60/d67/d98 0 2024-04-16T02:35:37.951 INFO:tasks.workunit.client.1.smithi153.stdout:8/448: truncate d2/d1c/d1f/f53 1207928 0 2024-04-16T02:35:37.953 INFO:tasks.workunit.client.0.smithi151.stdout:5/164: write d2/f24 [1896624,103662] 0 2024-04-16T02:35:37.954 INFO:tasks.workunit.client.1.smithi153.stdout:5/541: symlink d8/d41/d90/la9 0 2024-04-16T02:35:37.954 INFO:tasks.workunit.client.1.smithi153.stdout:2/498: truncate d9/d2b/f16 5774744 0 2024-04-16T02:35:37.957 INFO:tasks.workunit.client.1.smithi153.stdout:5/542: readlink d8/df/d2e/la5 0 2024-04-16T02:35:37.959 INFO:tasks.workunit.client.1.smithi153.stdout:5/543: write d8/d1f/f3f [1698620,75230] 0 2024-04-16T02:35:37.959 INFO:tasks.workunit.client.1.smithi153.stdout:3/468: symlink d6/d10/la6 0 2024-04-16T02:35:37.960 INFO:tasks.workunit.client.1.smithi153.stdout:3/469: write d6/f39 [1459101,108963] 0 2024-04-16T02:35:37.961 INFO:tasks.workunit.client.0.smithi151.stdout:5/165: dwrite d2/d6/f27 [0,4194304] 0 2024-04-16T02:35:37.962 INFO:tasks.workunit.client.0.smithi151.stdout:5/166: chown d2/d6/f1e 0 1 2024-04-16T02:35:37.972 INFO:tasks.workunit.client.1.smithi153.stdout:5/544: creat d8/df/d2f/d4e/faa x:0 0 0 2024-04-16T02:35:37.972 INFO:tasks.workunit.client.0.smithi151.stdout:5/167: dwrite d2/f24 [0,4194304] 0 2024-04-16T02:35:37.972 INFO:tasks.workunit.client.0.smithi151.stdout:5/168: stat d2/d6/f2f 0 2024-04-16T02:35:37.974 INFO:tasks.workunit.client.1.smithi153.stdout:1/525: dread d4/f5e [0,4194304] 0 2024-04-16T02:35:37.974 INFO:tasks.workunit.client.1.smithi153.stdout:2/499: fsync d9/d24/d34/d36/f3b 0 2024-04-16T02:35:37.975 INFO:tasks.workunit.client.0.smithi151.stdout:5/169: creat d2/d6/f34 x:0 0 0 2024-04-16T02:35:37.976 INFO:tasks.workunit.client.1.smithi153.stdout:6/484: rename dc/c84 to dc/d1a/c9f 0 2024-04-16T02:35:37.978 INFO:tasks.workunit.client.1.smithi153.stdout:3/470: fsync d6/d10/d5b/f75 0 2024-04-16T02:35:37.980 INFO:tasks.workunit.client.1.smithi153.stdout:5/545: mknod d8/df/d2f/d4e/d51/cab 0 2024-04-16T02:35:37.980 INFO:tasks.workunit.client.1.smithi153.stdout:5/546: dread - d8/df/d2e/f79 zero size 2024-04-16T02:35:37.983 INFO:tasks.workunit.client.0.smithi151.stdout:5/170: dwrite d2/f2d [0,4194304] 0 2024-04-16T02:35:37.984 INFO:tasks.workunit.client.1.smithi153.stdout:5/547: write d8/df/d28/d5a/f67 [862526,40835] 0 2024-04-16T02:35:37.986 INFO:tasks.workunit.client.0.smithi151.stdout:5/171: stat d2/d6/d19/d30/d33 0 2024-04-16T02:35:37.987 INFO:tasks.workunit.client.0.smithi151.stdout:5/172: write d2/d6/f25 [450376,121833] 0 2024-04-16T02:35:37.988 INFO:tasks.workunit.client.0.smithi151.stdout:5/173: dread - d2/d6/d19/d1a/f2e zero size 2024-04-16T02:35:37.989 INFO:tasks.workunit.client.0.smithi151.stdout:5/174: chown d2/d6/d19/d1a/f1b 68 1 2024-04-16T02:35:37.991 INFO:tasks.workunit.client.1.smithi153.stdout:3/471: dwrite d6/d10/d22/d2e/f6f [0,4194304] 0 2024-04-16T02:35:37.992 INFO:tasks.workunit.client.1.smithi153.stdout:0/608: rename dd/d32/d57/f89 to dd/d26/d44/fb7 0 2024-04-16T02:35:37.993 INFO:tasks.workunit.client.0.smithi151.stdout:5/175: link d2/d6/f34 d2/d6/f35 0 2024-04-16T02:35:37.998 INFO:tasks.workunit.client.1.smithi153.stdout:5/548: chown d8/df/d28/c42 431936 1 2024-04-16T02:35:37.999 INFO:tasks.workunit.client.1.smithi153.stdout:5/549: chown d8/df/d2e/c54 24559 1 2024-04-16T02:35:38.000 INFO:tasks.workunit.client.1.smithi153.stdout:8/449: truncate d2/de/d34/f8c 3842645 0 2024-04-16T02:35:38.000 INFO:tasks.workunit.client.1.smithi153.stdout:5/550: chown d8/df/d2e/f5b 32344827 1 2024-04-16T02:35:38.002 INFO:tasks.workunit.client.0.smithi151.stdout:5/176: dwrite d2/f18 [0,4194304] 0 2024-04-16T02:35:38.003 INFO:tasks.workunit.client.1.smithi153.stdout:6/485: symlink dc/d47/d57/d77/d7a/d97/la0 0 2024-04-16T02:35:38.003 INFO:tasks.workunit.client.0.smithi151.stdout:5/177: readlink d2/d6/d19/l2c 0 2024-04-16T02:35:38.003 INFO:tasks.workunit.client.1.smithi153.stdout:5/551: dread d8/df/d28/d5a/d66/f7b [0,4194304] 0 2024-04-16T02:35:38.004 INFO:tasks.workunit.client.1.smithi153.stdout:0/609: creat dd/d14/d19/d77/fb8 x:0 0 0 2024-04-16T02:35:38.007 INFO:tasks.workunit.client.1.smithi153.stdout:0/610: write dd/d14/d51/fa2 [376068,32615] 0 2024-04-16T02:35:38.008 INFO:tasks.workunit.client.1.smithi153.stdout:8/450: creat d2/de/d34/f99 x:0 0 0 2024-04-16T02:35:38.010 INFO:tasks.workunit.client.1.smithi153.stdout:3/472: dwrite d6/d10/d22/f31 [0,4194304] 0 2024-04-16T02:35:38.010 INFO:tasks.workunit.client.1.smithi153.stdout:5/552: mknod d8/df/d28/d5a/d72/cac 0 2024-04-16T02:35:38.011 INFO:tasks.workunit.client.1.smithi153.stdout:0/611: write dd/d14/d51/d54/f84 [2474454,42995] 0 2024-04-16T02:35:38.012 INFO:tasks.workunit.client.0.smithi151.stdout:5/178: dwrite d2/f5 [0,4194304] 0 2024-04-16T02:35:38.013 INFO:tasks.workunit.client.1.smithi153.stdout:3/473: write d6/d10/d5b/f82 [599408,28441] 0 2024-04-16T02:35:38.014 INFO:tasks.workunit.client.1.smithi153.stdout:3/474: readlink d6/l9e 0 2024-04-16T02:35:38.015 INFO:tasks.workunit.client.0.smithi151.stdout:5/179: rmdir d2/d6/d19/d30 39 2024-04-16T02:35:38.015 INFO:tasks.workunit.client.1.smithi153.stdout:6/486: dread dc/d6b/f73 [0,4194304] 0 2024-04-16T02:35:38.016 INFO:tasks.workunit.client.0.smithi151.stdout:5/180: truncate d2/d6/fa 4230556 0 2024-04-16T02:35:38.017 INFO:tasks.workunit.client.1.smithi153.stdout:3/475: read - d6/d10/d3b/f9d zero size 2024-04-16T02:35:38.017 INFO:tasks.workunit.client.1.smithi153.stdout:8/451: readlink d2/l10 0 2024-04-16T02:35:38.017 INFO:tasks.workunit.client.1.smithi153.stdout:6/487: chown dc/d1a/f1c 0 1 2024-04-16T02:35:38.019 INFO:tasks.workunit.client.1.smithi153.stdout:3/476: write d6/d10/d22/d2e/f35 [2679498,67324] 0 2024-04-16T02:35:38.021 INFO:tasks.workunit.client.0.smithi151.stdout:5/181: dread d2/f12 [0,4194304] 0 2024-04-16T02:35:38.025 INFO:tasks.workunit.client.1.smithi153.stdout:8/452: dwrite d2/d1c/d1f/d27/d2d/d47/d60/f79 [0,4194304] 0 2024-04-16T02:35:38.026 INFO:tasks.workunit.client.1.smithi153.stdout:0/612: creat dd/d32/fb9 x:0 0 0 2024-04-16T02:35:38.026 INFO:tasks.workunit.client.0.smithi151.stdout:5/182: dread d2/d6/d19/f1d [0,4194304] 0 2024-04-16T02:35:38.027 INFO:tasks.workunit.client.1.smithi153.stdout:8/453: write d2/d1c/d1f/d27/d2d/f31 [27096,48007] 0 2024-04-16T02:35:38.027 INFO:tasks.workunit.client.0.smithi151.stdout:5/183: stat d2/d6/f1e 0 2024-04-16T02:35:38.028 INFO:tasks.workunit.client.1.smithi153.stdout:0/613: chown dd/d14/d51/f60 492995345 1 2024-04-16T02:35:38.030 INFO:tasks.workunit.client.1.smithi153.stdout:6/488: rename dc/l9d to dc/d47/d57/d77/d15/la1 0 2024-04-16T02:35:38.031 INFO:tasks.workunit.client.0.smithi151.stdout:5/184: mkdir d2/d6/d19/d30/d36 0 2024-04-16T02:35:38.031 INFO:tasks.workunit.client.1.smithi153.stdout:6/489: chown dc/d10/l5f 288351 1 2024-04-16T02:35:38.031 INFO:tasks.workunit.client.0.smithi151.stdout:5/185: chown d2/d6/f1e 150946157 1 2024-04-16T02:35:38.035 INFO:tasks.workunit.client.0.smithi151.stdout:5/186: creat d2/f37 x:0 0 0 2024-04-16T02:35:38.037 INFO:tasks.workunit.client.1.smithi153.stdout:0/614: creat dd/d14/d19/d77/d87/fba x:0 0 0 2024-04-16T02:35:38.037 INFO:tasks.workunit.client.1.smithi153.stdout:2/500: dwrite d9/d2b/f17 [0,4194304] 0 2024-04-16T02:35:38.038 INFO:tasks.workunit.client.1.smithi153.stdout:3/477: mknod d6/d10/d22/d2e/d8a/d8c/ca7 0 2024-04-16T02:35:38.038 INFO:tasks.workunit.client.1.smithi153.stdout:1/526: write d4/dd/d20/f5b [688806,47767] 0 2024-04-16T02:35:38.039 INFO:tasks.workunit.client.1.smithi153.stdout:0/615: readlink dd/d14/d19/d20/d4a/l97 0 2024-04-16T02:35:38.040 INFO:tasks.workunit.client.1.smithi153.stdout:6/490: symlink dc/d47/d57/d77/d15/la2 0 2024-04-16T02:35:38.040 INFO:tasks.workunit.client.1.smithi153.stdout:5/553: link d8/df/d2f/d4e/d51/f5c d8/df/d28/d5a/d5e/fad 0 2024-04-16T02:35:38.043 INFO:tasks.workunit.client.1.smithi153.stdout:5/554: chown d8/df/d84/la3 0 1 2024-04-16T02:35:38.045 INFO:tasks.workunit.client.1.smithi153.stdout:0/616: creat dd/d32/d57/d6c/fbb x:0 0 0 2024-04-16T02:35:38.047 INFO:tasks.workunit.client.1.smithi153.stdout:2/501: rename d9/d2b/d1c/d20/lac to d9/d2b/d5c/da2/lb4 0 2024-04-16T02:35:38.047 INFO:tasks.workunit.client.1.smithi153.stdout:0/617: rename dd to dd/d26/d44/dbc 22 2024-04-16T02:35:38.048 INFO:tasks.workunit.client.1.smithi153.stdout:1/527: mkdir d4/dd/d13/d26/db4 0 2024-04-16T02:35:38.048 INFO:tasks.workunit.client.1.smithi153.stdout:5/555: rmdir d8/df/d28/d5a/d66 39 2024-04-16T02:35:38.050 INFO:tasks.workunit.client.1.smithi153.stdout:0/618: write dd/d14/da6/da4/fa8 [648734,92988] 0 2024-04-16T02:35:38.050 INFO:tasks.workunit.client.1.smithi153.stdout:1/528: write d4/dd/d13/d26/f78 [664479,80005] 0 2024-04-16T02:35:38.052 INFO:tasks.workunit.client.1.smithi153.stdout:5/556: chown d8/df/d2f/d4e/faa 5678 1 2024-04-16T02:35:38.053 INFO:tasks.workunit.client.1.smithi153.stdout:0/619: symlink dd/d14/da6/d73/lbd 0 2024-04-16T02:35:38.054 INFO:tasks.workunit.client.1.smithi153.stdout:1/529: truncate d4/dd/d13/d26/d58/d7a/fa7 362353 0 2024-04-16T02:35:38.055 INFO:tasks.workunit.client.1.smithi153.stdout:0/620: dread - dd/d14/d51/d54/d65/f99 zero size 2024-04-16T02:35:38.057 INFO:tasks.workunit.client.1.smithi153.stdout:5/557: rename d8/df/d28/d5a/d72/f8e to d8/d1f/d20/d83/fae 0 2024-04-16T02:35:38.058 INFO:tasks.workunit.client.1.smithi153.stdout:5/558: fsync d8/df/d2f/d4e/faa 0 2024-04-16T02:35:38.058 INFO:tasks.workunit.client.0.smithi151.stdout:5/187: truncate d2/f2d 84072 0 2024-04-16T02:35:38.059 INFO:tasks.workunit.client.1.smithi153.stdout:8/454: unlink d2/de/d34/f8c 0 2024-04-16T02:35:38.060 INFO:tasks.workunit.client.0.smithi151.stdout:5/188: write d2/f12 [1191416,100823] 0 2024-04-16T02:35:38.060 INFO:tasks.workunit.client.1.smithi153.stdout:1/530: symlink d4/da2/d65/d82/lb5 0 2024-04-16T02:35:38.061 INFO:tasks.workunit.client.0.smithi151.stdout:5/189: chown d2/f37 2 1 2024-04-16T02:35:38.061 INFO:tasks.workunit.client.1.smithi153.stdout:1/531: write d4/f7 [1915712,118561] 0 2024-04-16T02:35:38.062 INFO:tasks.workunit.client.0.smithi151.stdout:5/190: write d2/d6/f8 [5213699,81629] 0 2024-04-16T02:35:38.063 INFO:tasks.workunit.client.1.smithi153.stdout:7/470: sync 2024-04-16T02:35:38.065 INFO:tasks.workunit.client.0.smithi151.stdout:5/191: creat d2/d6/d19/d30/d36/f38 x:0 0 0 2024-04-16T02:35:38.065 INFO:tasks.workunit.client.0.smithi151.stdout:5/192: readlink d2/l10 0 2024-04-16T02:35:38.066 INFO:tasks.workunit.client.0.smithi151.stdout:8/87: sync 2024-04-16T02:35:38.066 INFO:tasks.workunit.client.0.smithi151.stdout:8/88: readlink la 0 2024-04-16T02:35:38.069 INFO:tasks.workunit.client.1.smithi153.stdout:8/455: creat d2/d93/f9a x:0 0 0 2024-04-16T02:35:38.072 INFO:tasks.workunit.client.1.smithi153.stdout:7/471: fsync d0/d4/db/f50 0 2024-04-16T02:35:38.074 INFO:tasks.workunit.client.1.smithi153.stdout:1/532: dread d4/dd/d18/f42 [0,4194304] 0 2024-04-16T02:35:38.074 INFO:tasks.workunit.client.1.smithi153.stdout:8/456: truncate d2/f14 1271174 0 2024-04-16T02:35:38.075 INFO:tasks.workunit.client.0.smithi151.stdout:5/193: truncate d2/f12 909674 0 2024-04-16T02:35:38.075 INFO:tasks.workunit.client.1.smithi153.stdout:6/491: dread dc/d1a/f1c [0,4194304] 0 2024-04-16T02:35:38.075 INFO:tasks.workunit.client.1.smithi153.stdout:7/472: dread d0/d4/db/d2b/d53/f6e [0,4194304] 0 2024-04-16T02:35:38.076 INFO:tasks.workunit.client.0.smithi151.stdout:5/194: fdatasync d2/d6/d19/f1d 0 2024-04-16T02:35:38.076 INFO:tasks.workunit.client.0.smithi151.stdout:5/195: chown d2/f37 1950488 1 2024-04-16T02:35:38.077 INFO:tasks.workunit.client.1.smithi153.stdout:3/478: dread d6/f2d [0,4194304] 0 2024-04-16T02:35:38.077 INFO:tasks.workunit.client.1.smithi153.stdout:6/492: write dc/d59/d63/f66 [2690397,23996] 0 2024-04-16T02:35:38.078 INFO:tasks.workunit.client.1.smithi153.stdout:5/559: getdents d8/df/d2f/d4e/d51 0 2024-04-16T02:35:38.078 INFO:tasks.workunit.client.0.smithi151.stdout:5/196: creat d2/d6/d19/d30/f39 x:0 0 0 2024-04-16T02:35:38.079 INFO:tasks.workunit.client.1.smithi153.stdout:1/533: dread d4/dd/d18/f42 [0,4194304] 0 2024-04-16T02:35:38.080 INFO:tasks.workunit.client.1.smithi153.stdout:8/457: chown d2/d1c/d1f/d27/c72 11216415 1 2024-04-16T02:35:38.081 INFO:tasks.workunit.client.0.smithi151.stdout:5/197: creat d2/d6/d19/d30/f3a x:0 0 0 2024-04-16T02:35:38.083 INFO:tasks.workunit.client.0.smithi151.stdout:5/198: creat d2/d6/d19/d1a/f3b x:0 0 0 2024-04-16T02:35:38.086 INFO:tasks.workunit.client.1.smithi153.stdout:6/493: symlink dc/d47/d80/la3 0 2024-04-16T02:35:38.086 INFO:tasks.workunit.client.0.smithi151.stdout:5/199: mknod d2/d6/d19/c3c 0 2024-04-16T02:35:38.088 INFO:tasks.workunit.client.1.smithi153.stdout:3/479: creat d6/d10/d3b/d59/d7c/fa8 x:0 0 0 2024-04-16T02:35:38.088 INFO:tasks.workunit.client.1.smithi153.stdout:5/560: dwrite d8/df/d28/f8a [0,4194304] 0 2024-04-16T02:35:38.088 INFO:tasks.workunit.client.1.smithi153.stdout:1/534: mkdir d4/dd/d13/d23/db6 0 2024-04-16T02:35:38.090 INFO:tasks.workunit.client.1.smithi153.stdout:8/458: dwrite d2/de/f15 [0,4194304] 0 2024-04-16T02:35:38.090 INFO:tasks.workunit.client.1.smithi153.stdout:0/621: dread dd/d28/f41 [0,4194304] 0 2024-04-16T02:35:38.091 INFO:tasks.workunit.client.1.smithi153.stdout:3/480: stat d6/d10/d3b/d59/d7c/c89 0 2024-04-16T02:35:38.092 INFO:tasks.workunit.client.1.smithi153.stdout:6/494: creat dc/d47/d93/fa4 x:0 0 0 2024-04-16T02:35:38.093 INFO:tasks.workunit.client.1.smithi153.stdout:2/502: truncate d9/d24/d27/d2c/d41/f44 823476 0 2024-04-16T02:35:38.094 INFO:tasks.workunit.client.1.smithi153.stdout:6/495: stat dc/d47/d57/d77/d7a 0 2024-04-16T02:35:38.095 INFO:tasks.workunit.client.1.smithi153.stdout:5/561: mkdir d8/d1f/d20/daf 0 2024-04-16T02:35:38.096 INFO:tasks.workunit.client.1.smithi153.stdout:5/562: chown d8/df/d2e 5 1 2024-04-16T02:35:38.097 INFO:tasks.workunit.client.1.smithi153.stdout:3/481: dread d6/d10/d5b/f82 [0,4194304] 0 2024-04-16T02:35:38.097 INFO:tasks.workunit.client.1.smithi153.stdout:9/542: sync 2024-04-16T02:35:38.097 INFO:tasks.workunit.client.1.smithi153.stdout:4/606: sync 2024-04-16T02:35:38.098 INFO:tasks.workunit.client.1.smithi153.stdout:5/563: truncate d8/d1f/f21 1364443 0 2024-04-16T02:35:38.099 INFO:tasks.workunit.client.1.smithi153.stdout:5/564: fdatasync d8/df/f97 0 2024-04-16T02:35:38.102 INFO:tasks.workunit.client.1.smithi153.stdout:0/622: rename dd/d28/f74 to dd/d14/da6/da4/fbe 0 2024-04-16T02:35:38.105 INFO:tasks.workunit.client.1.smithi153.stdout:1/535: mkdir d4/dd/d13/d26/db7 0 2024-04-16T02:35:38.105 INFO:tasks.workunit.client.1.smithi153.stdout:6/496: mknod dc/d1a/d2d/ca5 0 2024-04-16T02:35:38.107 INFO:tasks.workunit.client.1.smithi153.stdout:2/503: creat d9/d24/d27/d2c/d41/fb5 x:0 0 0 2024-04-16T02:35:38.107 INFO:tasks.workunit.client.1.smithi153.stdout:4/607: dread d3/d9/db/f65 [0,4194304] 0 2024-04-16T02:35:38.108 INFO:tasks.workunit.client.1.smithi153.stdout:8/459: dwrite d2/f9 [0,4194304] 0 2024-04-16T02:35:38.114 INFO:tasks.workunit.client.1.smithi153.stdout:3/482: creat d6/d10/d3b/d59/fa9 x:0 0 0 2024-04-16T02:35:38.116 INFO:tasks.workunit.client.1.smithi153.stdout:4/608: write d3/d9/d79/fd3 [68368,72278] 0 2024-04-16T02:35:38.116 INFO:tasks.workunit.client.1.smithi153.stdout:0/623: rmdir dd/d14/d51/d54 39 2024-04-16T02:35:38.117 INFO:tasks.workunit.client.1.smithi153.stdout:6/497: mkdir dc/d1a/d2d/da6 0 2024-04-16T02:35:38.117 INFO:tasks.workunit.client.1.smithi153.stdout:5/565: dwrite d8/df/d23/f56 [0,4194304] 0 2024-04-16T02:35:38.125 INFO:tasks.workunit.client.1.smithi153.stdout:8/460: dread - d2/d1c/d1f/f2e zero size 2024-04-16T02:35:38.125 INFO:tasks.workunit.client.1.smithi153.stdout:0/624: chown dd/d32/d57/d6c 91 1 2024-04-16T02:35:38.128 INFO:tasks.workunit.client.1.smithi153.stdout:5/566: readlink d8/df/d2e/l78 0 2024-04-16T02:35:38.128 INFO:tasks.workunit.client.1.smithi153.stdout:1/536: dwrite d4/dd/d13/d23/d31/f88 [0,4194304] 0 2024-04-16T02:35:38.129 INFO:tasks.workunit.client.1.smithi153.stdout:9/543: dread d4/d6/dc/d2d/d64/f8d [0,4194304] 0 2024-04-16T02:35:38.132 INFO:tasks.workunit.client.1.smithi153.stdout:8/461: chown d2/d13/d3a/d5b/c82 112 1 2024-04-16T02:35:38.133 INFO:tasks.workunit.client.1.smithi153.stdout:4/609: rmdir d3/d9/db/de 39 2024-04-16T02:35:38.135 INFO:tasks.workunit.client.1.smithi153.stdout:3/483: read d6/d10/d22/d2c/f74 [83202,30148] 0 2024-04-16T02:35:38.136 INFO:tasks.workunit.client.1.smithi153.stdout:7/473: dwrite d0/d4/db/d2b/d53/f46 [0,4194304] 0 2024-04-16T02:35:38.139 INFO:tasks.workunit.client.1.smithi153.stdout:0/625: mknod dd/d14/d51/d7f/d9d/cbf 0 2024-04-16T02:35:38.143 INFO:tasks.workunit.client.1.smithi153.stdout:6/498: dwrite dc/d47/d57/d77/d15/f74 [0,4194304] 0 2024-04-16T02:35:38.145 INFO:tasks.workunit.client.1.smithi153.stdout:2/504: truncate d9/d2b/f19 187428 0 2024-04-16T02:35:38.147 INFO:tasks.workunit.client.1.smithi153.stdout:3/484: write d6/d10/d22/d43/f57 [1875631,43723] 0 2024-04-16T02:35:38.151 INFO:tasks.workunit.client.1.smithi153.stdout:6/499: unlink dc/d47/f60 0 2024-04-16T02:35:38.153 INFO:tasks.workunit.client.1.smithi153.stdout:8/462: creat d2/f9b x:0 0 0 2024-04-16T02:35:38.153 INFO:tasks.workunit.client.1.smithi153.stdout:5/567: dread d8/f4d [0,4194304] 0 2024-04-16T02:35:38.153 INFO:tasks.workunit.client.1.smithi153.stdout:7/474: creat d0/d4/db/d2b/d43/f9d x:0 0 0 2024-04-16T02:35:38.154 INFO:tasks.workunit.client.1.smithi153.stdout:4/610: symlink d3/d9/db/de/d2c/d98/ld6 0 2024-04-16T02:35:38.154 INFO:tasks.workunit.client.1.smithi153.stdout:1/537: rename d4/dd/d13/c3b to d4/da2/d65/d84/cb8 0 2024-04-16T02:35:38.155 INFO:tasks.workunit.client.1.smithi153.stdout:8/463: rmdir d2/de 39 2024-04-16T02:35:38.156 INFO:tasks.workunit.client.1.smithi153.stdout:7/475: fsync d0/d4/db/f64 0 2024-04-16T02:35:38.158 INFO:tasks.workunit.client.1.smithi153.stdout:4/611: fsync d3/d9/db/d17/f55 0 2024-04-16T02:35:38.161 INFO:tasks.workunit.client.1.smithi153.stdout:9/544: rename d4/d6/dc/d2d/d64/f8d to d4/d6/d1b/d2a/d99/fa4 0 2024-04-16T02:35:38.162 INFO:tasks.workunit.client.1.smithi153.stdout:9/545: write d4/d6/d1b/f59 [2048583,7699] 0 2024-04-16T02:35:38.163 INFO:tasks.workunit.client.1.smithi153.stdout:6/500: dwrite dc/d47/d57/d77/d15/f22 [0,4194304] 0 2024-04-16T02:35:38.166 INFO:tasks.workunit.client.1.smithi153.stdout:3/485: rename d6/d10/d22/d2c/c53 to d6/d10/d3b/caa 0 2024-04-16T02:35:38.170 INFO:tasks.workunit.client.1.smithi153.stdout:4/612: dwrite d3/d9/d79/fc4 [0,4194304] 0 2024-04-16T02:35:38.171 INFO:tasks.workunit.client.1.smithi153.stdout:6/501: symlink dc/d47/d57/d77/la7 0 2024-04-16T02:35:38.173 INFO:tasks.workunit.client.1.smithi153.stdout:7/476: read d0/d13/f81 [2013861,34828] 0 2024-04-16T02:35:38.175 INFO:tasks.workunit.client.1.smithi153.stdout:7/477: chown d0/d4/db/d87 468 1 2024-04-16T02:35:38.176 INFO:tasks.workunit.client.1.smithi153.stdout:8/464: rename d2/d1c/d1f to d2/d93/d97/d9c 0 2024-04-16T02:35:38.178 INFO:tasks.workunit.client.1.smithi153.stdout:4/613: creat d3/d9/db/de/d2c/d98/fd7 x:0 0 0 2024-04-16T02:35:38.183 INFO:tasks.workunit.client.1.smithi153.stdout:7/478: symlink d0/d4/db/d2b/d43/d6d/d68/l9e 0 2024-04-16T02:35:38.183 INFO:tasks.workunit.client.1.smithi153.stdout:2/505: write f6 [300362,48184] 0 2024-04-16T02:35:38.185 INFO:tasks.workunit.client.1.smithi153.stdout:1/538: rename d4/dd/d18/f42 to d4/dd/d13/d23/d5f/fb9 0 2024-04-16T02:35:38.190 INFO:tasks.workunit.client.1.smithi153.stdout:9/546: link d4/d6/dc/d2d/da0/d40/d4a/c6b d4/d6/dc/d2d/da0/ca5 0 2024-04-16T02:35:38.192 INFO:tasks.workunit.client.1.smithi153.stdout:9/547: write d4/d33/d7d/f85 [4525320,69264] 0 2024-04-16T02:35:38.192 INFO:tasks.workunit.client.1.smithi153.stdout:7/479: rename d0/d4/db/d2b/d53/d59/d76/d7a to d0/d4/db/d2b/d8c/d9f 0 2024-04-16T02:35:38.193 INFO:tasks.workunit.client.1.smithi153.stdout:1/539: dwrite d4/da2/d65/d82/f9a [0,4194304] 0 2024-04-16T02:35:38.194 INFO:tasks.workunit.client.1.smithi153.stdout:8/465: link d2/d93/d97/d9c/d27/d2d/f8b d2/d13/d3a/d5b/f9d 0 2024-04-16T02:35:38.194 INFO:tasks.workunit.client.1.smithi153.stdout:7/480: chown d0/d4/db/d2b/d53/d59/d76 16905509 1 2024-04-16T02:35:38.195 INFO:tasks.workunit.client.1.smithi153.stdout:4/614: symlink d3/dbc/ld8 0 2024-04-16T02:35:38.196 INFO:tasks.workunit.client.1.smithi153.stdout:6/502: link dc/d47/d57/d77/la7 dc/d10/d62/la8 0 2024-04-16T02:35:38.197 INFO:tasks.workunit.client.1.smithi153.stdout:6/503: chown dc/d10/c2a 208866475 1 2024-04-16T02:35:38.201 INFO:tasks.workunit.client.1.smithi153.stdout:1/540: rename d4/fae to d4/dd/d18/d6c/fba 0 2024-04-16T02:35:38.202 INFO:tasks.workunit.client.1.smithi153.stdout:3/486: write d6/d10/d22/f28 [1611950,48061] 0 2024-04-16T02:35:38.203 INFO:tasks.workunit.client.1.smithi153.stdout:2/506: creat d9/d2b/fb6 x:0 0 0 2024-04-16T02:35:38.204 INFO:tasks.workunit.client.1.smithi153.stdout:9/548: mknod d4/ca6 0 2024-04-16T02:35:38.206 INFO:tasks.workunit.client.1.smithi153.stdout:6/504: dread dc/d47/d57/f91 [0,4194304] 0 2024-04-16T02:35:38.206 INFO:tasks.workunit.client.1.smithi153.stdout:5/568: dread d8/d1f/d20/d39/f4a [0,4194304] 0 2024-04-16T02:35:38.208 INFO:tasks.workunit.client.1.smithi153.stdout:1/541: truncate d4/dd/d20/f5b 1101300 0 2024-04-16T02:35:38.210 INFO:tasks.workunit.client.1.smithi153.stdout:4/615: dwrite d3/d9/db/d17/d3f/f94 [0,4194304] 0 2024-04-16T02:35:38.210 INFO:tasks.workunit.client.1.smithi153.stdout:3/487: chown d6/d10/d22/d2e/c67 519327 1 2024-04-16T02:35:38.212 INFO:tasks.workunit.client.1.smithi153.stdout:7/481: rename d0/d4/db/d2b/c24 to d0/d4/db/d2b/d43/d6d/d68/d72/ca0 0 2024-04-16T02:35:38.213 INFO:tasks.workunit.client.1.smithi153.stdout:9/549: mknod d4/d6/dc/d2d/d79/ca7 0 2024-04-16T02:35:38.213 INFO:tasks.workunit.client.1.smithi153.stdout:7/482: stat d0/d4/db/d2b/d43/d6d/d68/d6c 0 2024-04-16T02:35:38.214 INFO:tasks.workunit.client.1.smithi153.stdout:6/505: mkdir dc/d47/d93/da9 0 2024-04-16T02:35:38.215 INFO:tasks.workunit.client.1.smithi153.stdout:2/507: dread d9/d24/d27/f68 [0,4194304] 0 2024-04-16T02:35:38.217 INFO:tasks.workunit.client.1.smithi153.stdout:2/508: readlink d9/d2b/d1c/l3a 0 2024-04-16T02:35:38.218 INFO:tasks.workunit.client.1.smithi153.stdout:4/616: mknod d3/d9/d79/cd9 0 2024-04-16T02:35:38.220 INFO:tasks.workunit.client.1.smithi153.stdout:2/509: readlink d9/d2b/d1c/lb3 0 2024-04-16T02:35:38.220 INFO:tasks.workunit.client.1.smithi153.stdout:3/488: chown d6/d10/d5b/c8d 1 1 2024-04-16T02:35:38.221 INFO:tasks.workunit.client.1.smithi153.stdout:5/569: mknod d8/df/d2f/cb0 0 2024-04-16T02:35:38.222 INFO:tasks.workunit.client.1.smithi153.stdout:7/483: rename d0/d4/db/d34/d4b/d4d/c62 to d0/d4/db/d2b/d43/d6d/d68/d72/ca1 0 2024-04-16T02:35:38.223 INFO:tasks.workunit.client.1.smithi153.stdout:4/617: chown d3/d9/db/de/f12 7 1 2024-04-16T02:35:38.223 INFO:tasks.workunit.client.1.smithi153.stdout:9/550: dwrite d4/f47 [0,4194304] 0 2024-04-16T02:35:38.225 INFO:tasks.workunit.client.1.smithi153.stdout:1/542: creat d4/da2/d65/fbb x:0 0 0 2024-04-16T02:35:38.229 INFO:tasks.workunit.client.1.smithi153.stdout:6/506: dwrite dc/d59/d63/f94 [0,4194304] 0 2024-04-16T02:35:38.230 INFO:tasks.workunit.client.1.smithi153.stdout:9/551: creat d4/d6/dc/d2d/fa8 x:0 0 0 2024-04-16T02:35:38.231 INFO:tasks.workunit.client.1.smithi153.stdout:3/489: fdatasync d6/d10/d22/d2c/f60 0 2024-04-16T02:35:38.232 INFO:tasks.workunit.client.1.smithi153.stdout:8/466: dread d2/de/f5f [0,4194304] 0 2024-04-16T02:35:38.232 INFO:tasks.workunit.client.1.smithi153.stdout:5/570: creat d8/df/d2f/d4e/d51/fb1 x:0 0 0 2024-04-16T02:35:38.233 INFO:tasks.workunit.client.1.smithi153.stdout:1/543: fdatasync d4/dd/d13/d23/f6d 0 2024-04-16T02:35:38.235 INFO:tasks.workunit.client.1.smithi153.stdout:4/618: creat d3/d9/db/de/d2c/d52/d85/fda x:0 0 0 2024-04-16T02:35:38.235 INFO:tasks.workunit.client.1.smithi153.stdout:1/544: readlink d4/dd/d13/d23/l75 0 2024-04-16T02:35:38.235 INFO:tasks.workunit.client.1.smithi153.stdout:5/571: readlink d8/df/d2f/d4e/d51/l9f 0 2024-04-16T02:35:38.236 INFO:tasks.workunit.client.1.smithi153.stdout:7/484: symlink d0/d4/db/d2b/d43/d6d/d68/la2 0 2024-04-16T02:35:38.236 INFO:tasks.workunit.client.1.smithi153.stdout:1/545: fsync d4/da2/d65/d84/f8c 0 2024-04-16T02:35:38.238 INFO:tasks.workunit.client.1.smithi153.stdout:4/619: dread - d3/d9/db/de/d2c/d52/fbe zero size 2024-04-16T02:35:38.241 INFO:tasks.workunit.client.1.smithi153.stdout:3/490: read d6/d10/d3b/f85 [277539,8501] 0 2024-04-16T02:35:38.242 INFO:tasks.workunit.client.1.smithi153.stdout:8/467: dwrite d2/d93/d97/d9c/d27/d2d/d47/f6c [0,4194304] 0 2024-04-16T02:35:38.243 INFO:tasks.workunit.client.1.smithi153.stdout:5/572: symlink d8/d1f/d20/d83/lb2 0 2024-04-16T02:35:38.244 INFO:tasks.workunit.client.1.smithi153.stdout:9/552: symlink d4/d94/la9 0 2024-04-16T02:35:38.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:37 smithi151 ceph-mon[29003]: pgmap v16: 65 pgs: 65 active+clean; 768 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 9.9 MiB/s rd, 97 MiB/s wr, 323 op/s 2024-04-16T02:35:38.245 INFO:tasks.workunit.client.1.smithi153.stdout:9/553: readlink d4/d6/l34 0 2024-04-16T02:35:38.248 INFO:tasks.workunit.client.1.smithi153.stdout:7/485: rename d0/d4/db/d2b/d53/f6e to d0/d4/db/d2b/d53/fa3 0 2024-04-16T02:35:38.249 INFO:tasks.workunit.client.1.smithi153.stdout:6/507: truncate dc/d47/d57/d77/d15/f2f 2862161 0 2024-04-16T02:35:38.250 INFO:tasks.workunit.client.1.smithi153.stdout:3/491: creat d6/d10/d3b/fab x:0 0 0 2024-04-16T02:35:38.251 INFO:tasks.workunit.client.1.smithi153.stdout:5/573: creat d8/d41/fb3 x:0 0 0 2024-04-16T02:35:38.251 INFO:tasks.workunit.client.1.smithi153.stdout:5/574: fsync d8/df/f16 0 2024-04-16T02:35:38.253 INFO:tasks.workunit.client.1.smithi153.stdout:4/620: creat d3/d9/d24/d75/dc5/fdb x:0 0 0 2024-04-16T02:35:38.253 INFO:tasks.workunit.client.1.smithi153.stdout:7/486: symlink d0/d4/db/d2b/d53/la4 0 2024-04-16T02:35:38.254 INFO:tasks.workunit.client.1.smithi153.stdout:6/508: creat dc/d47/d93/faa x:0 0 0 2024-04-16T02:35:38.262 INFO:tasks.workunit.client.1.smithi153.stdout:3/492: dread d6/d10/d22/d43/f57 [0,4194304] 0 2024-04-16T02:35:38.270 INFO:tasks.workunit.client.0.smithi151.stdout:3/90: sync 2024-04-16T02:35:38.270 INFO:tasks.workunit.client.0.smithi151.stdout:7/91: sync 2024-04-16T02:35:38.270 INFO:tasks.workunit.client.0.smithi151.stdout:7/92: fdatasync d0/f14 0 2024-04-16T02:35:38.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:37 smithi153 ceph-mon[36747]: pgmap v16: 65 pgs: 65 active+clean; 768 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 9.9 MiB/s rd, 97 MiB/s wr, 323 op/s 2024-04-16T02:35:38.276 INFO:tasks.workunit.client.0.smithi151.stdout:3/91: dread d1/d6/d10/f1a [0,4194304] 0 2024-04-16T02:35:38.277 INFO:tasks.workunit.client.1.smithi153.stdout:5/575: creat d8/d1f/d20/d39/fb4 x:0 0 0 2024-04-16T02:35:38.277 INFO:tasks.workunit.client.1.smithi153.stdout:4/621: creat d3/fdc x:0 0 0 2024-04-16T02:35:38.278 INFO:tasks.workunit.client.0.smithi151.stdout:7/93: rmdir d0 39 2024-04-16T02:35:38.278 INFO:tasks.workunit.client.1.smithi153.stdout:1/546: getdents d4/dd/d13/d26/d58/d7a 0 2024-04-16T02:35:38.279 INFO:tasks.workunit.client.1.smithi153.stdout:6/509: creat dc/d47/d93/fab x:0 0 0 2024-04-16T02:35:38.279 INFO:tasks.workunit.client.1.smithi153.stdout:5/576: chown d8/df/d2f/d4e/f94 5459119 1 2024-04-16T02:35:38.284 INFO:tasks.workunit.client.0.smithi151.stdout:3/92: dwrite d1/d6/f1b [0,4194304] 0 2024-04-16T02:35:38.286 INFO:tasks.workunit.client.0.smithi151.stdout:7/94: dread d0/f5 [0,4194304] 0 2024-04-16T02:35:38.288 INFO:tasks.workunit.client.1.smithi153.stdout:6/510: unlink dc/d47/d93/l9a 0 2024-04-16T02:35:38.288 INFO:tasks.workunit.client.0.smithi151.stdout:3/93: write d1/d6/d10/f16 [339298,73240] 0 2024-04-16T02:35:38.292 INFO:tasks.workunit.client.0.smithi151.stdout:3/94: fsync d1/d6/d10/f1a 0 2024-04-16T02:35:38.292 INFO:tasks.workunit.client.1.smithi153.stdout:3/493: write d6/d10/d22/d2c/f33 [1420627,123845] 0 2024-04-16T02:35:38.292 INFO:tasks.workunit.client.1.smithi153.stdout:9/554: link d4/d6/dc/d2d/da0/ca5 d4/d6/d8c/caa 0 2024-04-16T02:35:38.293 INFO:tasks.workunit.client.1.smithi153.stdout:1/547: mknod d4/dd/cbc 0 2024-04-16T02:35:38.294 INFO:tasks.workunit.client.0.smithi151.stdout:7/95: write d0/f11 [651422,69893] 0 2024-04-16T02:35:38.294 INFO:tasks.workunit.client.1.smithi153.stdout:9/555: write d4/d6/f56 [1895938,6251] 0 2024-04-16T02:35:38.294 INFO:tasks.workunit.client.1.smithi153.stdout:7/487: link d0/d4/db/d2b/d43/l6a d0/d4/db/d2b/d8c/d9f/la5 0 2024-04-16T02:35:38.297 INFO:tasks.workunit.client.0.smithi151.stdout:7/96: mkdir d0/d19 0 2024-04-16T02:35:38.299 INFO:tasks.workunit.client.0.smithi151.stdout:3/95: creat d1/d6/f1e x:0 0 0 2024-04-16T02:35:38.299 INFO:tasks.workunit.client.1.smithi153.stdout:5/577: mknod d8/df/d28/d5a/cb5 0 2024-04-16T02:35:38.301 INFO:tasks.workunit.client.0.smithi151.stdout:3/96: write d1/d6/f1e [868550,115016] 0 2024-04-16T02:35:38.302 INFO:tasks.workunit.client.0.smithi151.stdout:7/97: dread d0/f3 [0,4194304] 0 2024-04-16T02:35:38.304 INFO:tasks.workunit.client.1.smithi153.stdout:6/511: rmdir dc/d47/d57/d77 39 2024-04-16T02:35:38.305 INFO:tasks.workunit.client.0.smithi151.stdout:7/98: dread d0/f14 [0,4194304] 0 2024-04-16T02:35:38.306 INFO:tasks.workunit.client.1.smithi153.stdout:3/494: dwrite d6/d10/d22/d2c/f63 [0,4194304] 0 2024-04-16T02:35:38.306 INFO:tasks.workunit.client.0.smithi151.stdout:3/97: rename d1/l4 to d1/d6/l1f 0 2024-04-16T02:35:38.306 INFO:tasks.workunit.client.1.smithi153.stdout:7/488: creat d0/d4/db/d2b/d8c/fa6 x:0 0 0 2024-04-16T02:35:38.307 INFO:tasks.workunit.client.1.smithi153.stdout:7/489: chown d0/d4/db/d2b/d53/f27 980839 1 2024-04-16T02:35:38.307 INFO:tasks.workunit.client.0.smithi151.stdout:7/99: write d0/f6 [4128692,64041] 0 2024-04-16T02:35:38.308 INFO:tasks.workunit.client.1.smithi153.stdout:7/490: chown d0/d4/db/d34/d4b/f69 433 1 2024-04-16T02:35:38.310 INFO:tasks.workunit.client.0.smithi151.stdout:3/98: mkdir d1/d6/d20 0 2024-04-16T02:35:38.312 INFO:tasks.workunit.client.1.smithi153.stdout:9/556: creat d4/d6/dc/d24/d5d/fab x:0 0 0 2024-04-16T02:35:38.316 INFO:tasks.workunit.client.0.smithi151.stdout:7/100: dwrite d0/f5 [4194304,4194304] 0 2024-04-16T02:35:38.318 INFO:tasks.workunit.client.1.smithi153.stdout:8/468: truncate d2/d93/d97/d9c/d27/d2d/d47/f6c 1676790 0 2024-04-16T02:35:38.319 INFO:tasks.workunit.client.1.smithi153.stdout:5/578: symlink d8/d1f/d20/daf/lb6 0 2024-04-16T02:35:38.319 INFO:tasks.workunit.client.1.smithi153.stdout:3/495: stat c3 0 2024-04-16T02:35:38.320 INFO:tasks.workunit.client.0.smithi151.stdout:3/99: rmdir d1/d6 39 2024-04-16T02:35:38.321 INFO:tasks.workunit.client.1.smithi153.stdout:7/491: creat d0/d4/db/d47/fa7 x:0 0 0 2024-04-16T02:35:38.321 INFO:tasks.workunit.client.1.smithi153.stdout:6/512: dwrite dc/d47/d57/d77/f85 [0,4194304] 0 2024-04-16T02:35:38.321 INFO:tasks.workunit.client.0.smithi151.stdout:3/100: chown d1/lb 317520601 1 2024-04-16T02:35:38.325 INFO:tasks.workunit.client.0.smithi151.stdout:7/101: dread d0/fb [0,4194304] 0 2024-04-16T02:35:38.328 INFO:tasks.workunit.client.0.smithi151.stdout:7/102: mknod d0/c1a 0 2024-04-16T02:35:38.328 INFO:tasks.workunit.client.1.smithi153.stdout:1/548: getdents d4/dd/d13/d8d 0 2024-04-16T02:35:38.328 INFO:tasks.workunit.client.0.smithi151.stdout:3/101: dread d1/d6/ff [0,4194304] 0 2024-04-16T02:35:38.330 INFO:tasks.workunit.client.1.smithi153.stdout:5/579: chown d8/c14 176959978 1 2024-04-16T02:35:38.331 INFO:tasks.workunit.client.0.smithi151.stdout:7/103: symlink d0/d19/l1b 0 2024-04-16T02:35:38.332 INFO:tasks.workunit.client.0.smithi151.stdout:7/104: fdatasync d0/fb 0 2024-04-16T02:35:38.333 INFO:tasks.workunit.client.1.smithi153.stdout:9/557: symlink d4/d6/lac 0 2024-04-16T02:35:38.334 INFO:tasks.workunit.client.0.smithi151.stdout:3/102: creat d1/d6/f21 x:0 0 0 2024-04-16T02:35:38.335 INFO:tasks.workunit.client.0.smithi151.stdout:3/103: fsync d1/d6/f1e 0 2024-04-16T02:35:38.335 INFO:tasks.workunit.client.1.smithi153.stdout:6/513: mkdir dc/d47/d57/d77/d15/dac 0 2024-04-16T02:35:38.335 INFO:tasks.workunit.client.1.smithi153.stdout:5/580: dwrite d8/d1f/f3f [0,4194304] 0 2024-04-16T02:35:38.336 INFO:tasks.workunit.client.0.smithi151.stdout:3/104: readlink d1/lb 0 2024-04-16T02:35:38.336 INFO:tasks.workunit.client.0.smithi151.stdout:7/105: dread d0/f11 [0,4194304] 0 2024-04-16T02:35:38.338 INFO:tasks.workunit.client.0.smithi151.stdout:3/105: mkdir d1/d6/d22 0 2024-04-16T02:35:38.338 INFO:tasks.workunit.client.1.smithi153.stdout:8/469: unlink d2/d93/d97/d9c/d27/l29 0 2024-04-16T02:35:38.340 INFO:tasks.workunit.client.1.smithi153.stdout:9/558: write d4/f48 [4278593,44632] 0 2024-04-16T02:35:38.340 INFO:tasks.workunit.client.0.smithi151.stdout:3/106: mkdir d1/d6/d22/d23 0 2024-04-16T02:35:38.341 INFO:tasks.workunit.client.0.smithi151.stdout:3/107: write d1/f17 [433661,62351] 0 2024-04-16T02:35:38.343 INFO:tasks.workunit.client.1.smithi153.stdout:5/581: rmdir d8/d1f/d20/d39 39 2024-04-16T02:35:38.343 INFO:tasks.workunit.client.0.smithi151.stdout:3/108: write d1/d6/d10/f15 [4077736,24549] 0 2024-04-16T02:35:38.345 INFO:tasks.workunit.client.1.smithi153.stdout:4/622: creat d3/fdd x:0 0 0 2024-04-16T02:35:38.347 INFO:tasks.workunit.client.1.smithi153.stdout:3/496: link d6/d10/d22/d2c/f60 d6/d10/d22/d4d/da0/fac 0 2024-04-16T02:35:38.351 INFO:tasks.workunit.client.1.smithi153.stdout:8/470: rename d2/d13/l1a to d2/d13/d2c/l9e 0 2024-04-16T02:35:38.351 INFO:tasks.workunit.client.0.smithi151.stdout:3/109: fsync d1/f17 0 2024-04-16T02:35:38.352 INFO:tasks.workunit.client.1.smithi153.stdout:3/497: readlink d6/d10/d22/d2c/l9a 0 2024-04-16T02:35:38.352 INFO:tasks.workunit.client.0.smithi151.stdout:3/110: chown d1/d6/c8 238326 1 2024-04-16T02:35:38.353 INFO:tasks.workunit.client.1.smithi153.stdout:9/559: symlink d4/lad 0 2024-04-16T02:35:38.354 INFO:tasks.workunit.client.1.smithi153.stdout:9/560: dread - d4/d6/dc/d2d/da0/d40/f5e zero size 2024-04-16T02:35:38.354 INFO:tasks.workunit.client.0.smithi151.stdout:3/111: creat d1/d6/f24 x:0 0 0 2024-04-16T02:35:38.356 INFO:tasks.workunit.client.0.smithi151.stdout:3/112: symlink d1/d6/d10/l25 0 2024-04-16T02:35:38.357 INFO:tasks.workunit.client.1.smithi153.stdout:4/623: symlink d3/d9/db/de/d2c/d52/lde 0 2024-04-16T02:35:38.357 INFO:tasks.workunit.client.0.smithi151.stdout:3/113: readlink d1/d6/l18 0 2024-04-16T02:35:38.359 INFO:tasks.workunit.client.1.smithi153.stdout:5/582: rename d8/c15 to d8/df/d28/d5a/d5e/cb7 0 2024-04-16T02:35:38.360 INFO:tasks.workunit.client.0.smithi151.stdout:3/114: dread d1/d6/d10/f15 [0,4194304] 0 2024-04-16T02:35:38.360 INFO:tasks.workunit.client.1.smithi153.stdout:5/583: truncate d8/df/f97 101192 0 2024-04-16T02:35:38.362 INFO:tasks.workunit.client.1.smithi153.stdout:5/584: write d8/df/d28/d5a/f68 [367433,130359] 0 2024-04-16T02:35:38.365 INFO:tasks.workunit.client.1.smithi153.stdout:7/492: dread d0/d4/db/d2b/f38 [0,4194304] 0 2024-04-16T02:35:38.365 INFO:tasks.workunit.client.1.smithi153.stdout:5/585: read d8/f6f [3632622,96919] 0 2024-04-16T02:35:38.367 INFO:tasks.workunit.client.1.smithi153.stdout:6/514: link dc/d47/d57/d77/c38 dc/d47/d57/cad 0 2024-04-16T02:35:38.368 INFO:tasks.workunit.client.0.smithi151.stdout:3/115: rmdir d1/d6 39 2024-04-16T02:35:38.368 INFO:tasks.workunit.client.1.smithi153.stdout:1/549: dread d4/dd/d20/f55 [0,4194304] 0 2024-04-16T02:35:38.369 INFO:tasks.workunit.client.1.smithi153.stdout:1/550: readlink d4/dd/l4e 0 2024-04-16T02:35:38.369 INFO:tasks.workunit.client.1.smithi153.stdout:9/561: rmdir d4 39 2024-04-16T02:35:38.369 INFO:tasks.workunit.client.0.smithi151.stdout:3/116: readlink d1/d6/l1f 0 2024-04-16T02:35:38.371 INFO:tasks.workunit.client.0.smithi151.stdout:3/117: creat d1/f26 x:0 0 0 2024-04-16T02:35:38.372 INFO:tasks.workunit.client.0.smithi151.stdout:3/118: write d1/d6/f1e [1935543,6151] 0 2024-04-16T02:35:38.373 INFO:tasks.workunit.client.0.smithi151.stdout:3/119: read d1/d6/d10/f15 [3965454,118316] 0 2024-04-16T02:35:38.374 INFO:tasks.workunit.client.1.smithi153.stdout:0/626: sync 2024-04-16T02:35:38.374 INFO:tasks.workunit.client.1.smithi153.stdout:2/510: sync 2024-04-16T02:35:38.376 INFO:tasks.workunit.client.1.smithi153.stdout:1/551: dwrite d4/dd/d13/d23/d5f/f7f [0,4194304] 0 2024-04-16T02:35:38.377 INFO:tasks.workunit.client.0.smithi151.stdout:3/120: link d1/d6/d10/l25 d1/l27 0 2024-04-16T02:35:38.378 INFO:tasks.workunit.client.1.smithi153.stdout:6/515: fsync dc/d1a/d2d/f44 0 2024-04-16T02:35:38.378 INFO:tasks.workunit.client.1.smithi153.stdout:8/471: dwrite d2/f12 [0,4194304] 0 2024-04-16T02:35:38.383 INFO:tasks.workunit.client.1.smithi153.stdout:7/493: getdents d0/d4/db/d47/d8d/d93/d8b 0 2024-04-16T02:35:38.386 INFO:tasks.workunit.client.0.smithi151.stdout:3/121: dwrite d1/d6/d10/f15 [0,4194304] 0 2024-04-16T02:35:38.387 INFO:tasks.workunit.client.1.smithi153.stdout:2/511: symlink d9/d2b/d18/d3f/d40/lb7 0 2024-04-16T02:35:38.387 INFO:tasks.workunit.client.1.smithi153.stdout:8/472: dwrite d2/d13/d3a/f46 [0,4194304] 0 2024-04-16T02:35:38.388 INFO:tasks.workunit.client.1.smithi153.stdout:2/512: read - d9/f8b zero size 2024-04-16T02:35:38.388 INFO:tasks.workunit.client.1.smithi153.stdout:4/624: creat d3/fdf x:0 0 0 2024-04-16T02:35:38.394 INFO:tasks.workunit.client.1.smithi153.stdout:3/498: getdents d6/d10/d3b/d59 0 2024-04-16T02:35:38.398 INFO:tasks.workunit.client.1.smithi153.stdout:4/625: truncate d3/d9/db/de/d2c/d52/fa2 1214756 0 2024-04-16T02:35:38.399 INFO:tasks.workunit.client.1.smithi153.stdout:3/499: write d6/d10/d22/f2a [2898695,92676] 0 2024-04-16T02:35:38.400 INFO:tasks.workunit.client.1.smithi153.stdout:9/562: mkdir d4/d33/dae 0 2024-04-16T02:35:38.403 INFO:tasks.workunit.client.1.smithi153.stdout:8/473: creat d2/d13/f9f x:0 0 0 2024-04-16T02:35:38.403 INFO:tasks.workunit.client.1.smithi153.stdout:4/626: mkdir d3/d9/db/d84/de0 0 2024-04-16T02:35:38.405 INFO:tasks.workunit.client.1.smithi153.stdout:5/586: fsync d8/df/d28/d5a/fa1 0 2024-04-16T02:35:38.407 INFO:tasks.workunit.client.1.smithi153.stdout:1/552: dwrite d4/dd/d18/d29/f4a [0,4194304] 0 2024-04-16T02:35:38.407 INFO:tasks.workunit.client.1.smithi153.stdout:3/500: rmdir d6/d10/d22/d2e/d8a/d8c 39 2024-04-16T02:35:38.408 INFO:tasks.workunit.client.1.smithi153.stdout:4/627: fdatasync d3/d9/fba 0 2024-04-16T02:35:38.409 INFO:tasks.workunit.client.1.smithi153.stdout:8/474: mknod d2/d93/d97/d9c/d27/d2d/d47/d60/d67/ca0 0 2024-04-16T02:35:38.409 INFO:tasks.workunit.client.1.smithi153.stdout:5/587: truncate d8/df/d2f/d4e/faa 158605 0 2024-04-16T02:35:38.410 INFO:tasks.workunit.client.1.smithi153.stdout:0/627: dread dd/f18 [0,4194304] 0 2024-04-16T02:35:38.439 INFO:tasks.workunit.client.1.smithi153.stdout:2/513: dread d9/d2b/d1c/d20/f79 [0,4194304] 0 2024-04-16T02:35:38.440 INFO:tasks.workunit.client.1.smithi153.stdout:2/514: write d9/d24/d27/d2c/d41/f56 [1021754,15071] 0 2024-04-16T02:35:38.441 INFO:tasks.workunit.client.1.smithi153.stdout:2/515: truncate d9/d2b/fb6 927413 0 2024-04-16T02:35:38.455 INFO:tasks.workunit.client.1.smithi153.stdout:1/553: creat d4/da2/d65/d84/fbd x:0 0 0 2024-04-16T02:35:38.455 INFO:tasks.workunit.client.1.smithi153.stdout:6/516: dread dc/d10/f1b [0,4194304] 0 2024-04-16T02:35:38.456 INFO:tasks.workunit.client.1.smithi153.stdout:6/517: read dc/d10/f1b [254201,94887] 0 2024-04-16T02:35:38.456 INFO:tasks.workunit.client.1.smithi153.stdout:1/554: fdatasync d4/dd/d18/d29/d6e/fb2 0 2024-04-16T02:35:38.456 INFO:tasks.workunit.client.1.smithi153.stdout:3/501: creat d6/d10/d22/d2e/fad x:0 0 0 2024-04-16T02:35:38.456 INFO:tasks.workunit.client.1.smithi153.stdout:1/555: chown d4/dd/d13/d26/db7 822195 1 2024-04-16T02:35:38.458 INFO:tasks.workunit.client.1.smithi153.stdout:4/628: mkdir d3/d9/d24/d3e/de1 0 2024-04-16T02:35:38.459 INFO:tasks.workunit.client.1.smithi153.stdout:0/628: symlink dd/d26/d44/lc0 0 2024-04-16T02:35:38.461 INFO:tasks.workunit.client.1.smithi153.stdout:7/494: truncate d0/d4/db/d2b/f3e 1638819 0 2024-04-16T02:35:38.461 INFO:tasks.workunit.client.1.smithi153.stdout:2/516: rmdir d9/d2b/d18/d3f 39 2024-04-16T02:35:38.463 INFO:tasks.workunit.client.1.smithi153.stdout:6/518: fdatasync dc/d47/d57/d77/d15/f19 0 2024-04-16T02:35:38.463 INFO:tasks.workunit.client.1.smithi153.stdout:9/563: write d4/d6/dc/d24/f83 [2684550,65108] 0 2024-04-16T02:35:38.465 INFO:tasks.workunit.client.1.smithi153.stdout:1/556: dwrite d4/dd/d13/d26/d58/d7a/fa7 [0,4194304] 0 2024-04-16T02:35:38.466 INFO:tasks.workunit.client.1.smithi153.stdout:8/475: symlink d2/d93/d97/d9c/d27/d2d/la1 0 2024-04-16T02:35:38.470 INFO:tasks.workunit.client.1.smithi153.stdout:0/629: dread dd/d14/d19/d20/f88 [0,4194304] 0 2024-04-16T02:35:38.470 INFO:tasks.workunit.client.1.smithi153.stdout:7/495: symlink d0/d4/db/d2b/la8 0 2024-04-16T02:35:38.472 INFO:tasks.workunit.client.1.smithi153.stdout:0/630: write dd/d14/da6/d73/f8f [4482063,96603] 0 2024-04-16T02:35:38.473 INFO:tasks.workunit.client.1.smithi153.stdout:5/588: creat d8/df/d28/d5a/fb8 x:0 0 0 2024-04-16T02:35:38.474 INFO:tasks.workunit.client.1.smithi153.stdout:5/589: write d8/d1f/f3f [5121793,109802] 0 2024-04-16T02:35:38.474 INFO:tasks.workunit.client.1.smithi153.stdout:4/629: mkdir d3/d9/db/de/d2c/de2 0 2024-04-16T02:35:38.478 INFO:tasks.workunit.client.1.smithi153.stdout:8/476: truncate d2/d93/d97/d9c/f2e 620687 0 2024-04-16T02:35:38.483 INFO:tasks.workunit.client.1.smithi153.stdout:0/631: creat dd/d14/d51/d7f/fc1 x:0 0 0 2024-04-16T02:35:38.485 INFO:tasks.workunit.client.1.smithi153.stdout:2/517: write d9/d24/d27/f1f [288327,108985] 0 2024-04-16T02:35:38.488 INFO:tasks.workunit.client.1.smithi153.stdout:6/519: mkdir dc/d47/d57/d77/d15/dac/dae 0 2024-04-16T02:35:38.488 INFO:tasks.workunit.client.1.smithi153.stdout:7/496: dread d0/d13/f4f [0,4194304] 0 2024-04-16T02:35:38.489 INFO:tasks.workunit.client.1.smithi153.stdout:3/502: getdents d6 0 2024-04-16T02:35:38.491 INFO:tasks.workunit.client.1.smithi153.stdout:9/564: creat d4/d33/faf x:0 0 0 2024-04-16T02:35:38.492 INFO:tasks.workunit.client.1.smithi153.stdout:0/632: dwrite dd/d14/d19/d20/f70 [0,4194304] 0 2024-04-16T02:35:38.492 INFO:tasks.workunit.client.1.smithi153.stdout:1/557: creat d4/dd/fbe x:0 0 0 2024-04-16T02:35:38.495 INFO:tasks.workunit.client.1.smithi153.stdout:2/518: write d9/d2b/d18/d3f/fab [787860,29367] 0 2024-04-16T02:35:38.495 INFO:tasks.workunit.client.1.smithi153.stdout:5/590: rename d8/d1f/d20 to d8/df/d28/d5a/d66/d99/db9 0 2024-04-16T02:35:38.497 INFO:tasks.workunit.client.1.smithi153.stdout:2/519: fsync d9/d2b/fb6 0 2024-04-16T02:35:38.497 INFO:tasks.workunit.client.1.smithi153.stdout:6/520: symlink dc/d47/d57/d77/d15/laf 0 2024-04-16T02:35:38.498 INFO:tasks.workunit.client.1.smithi153.stdout:4/630: dread d3/d9/d24/fa4 [0,4194304] 0 2024-04-16T02:35:38.498 INFO:tasks.workunit.client.1.smithi153.stdout:1/558: mknod d4/dd/d13/d7e/d80/cbf 0 2024-04-16T02:35:38.500 INFO:tasks.workunit.client.1.smithi153.stdout:3/503: rename d6/d10/d5b/c78 to d6/d10/d22/d4d/da0/cae 0 2024-04-16T02:35:38.501 INFO:tasks.workunit.client.1.smithi153.stdout:0/633: symlink dd/d14/d51/lc2 0 2024-04-16T02:35:38.502 INFO:tasks.workunit.client.1.smithi153.stdout:4/631: chown d3/d9/db/de/d2c/d98/fd7 79681659 1 2024-04-16T02:35:38.502 INFO:tasks.workunit.client.1.smithi153.stdout:3/504: stat d6/d10/d22/d2e/fad 0 2024-04-16T02:35:38.502 INFO:tasks.workunit.client.1.smithi153.stdout:1/559: creat d4/dd/d13/d23/d5f/fc0 x:0 0 0 2024-04-16T02:35:38.503 INFO:tasks.workunit.client.1.smithi153.stdout:0/634: chown dd/d32/d80/l94 1 1 2024-04-16T02:35:38.503 INFO:tasks.workunit.client.1.smithi153.stdout:7/497: rename d0/d4/db/d34/l3a to d0/d4/db/d2b/d43/d6d/d68/d72/la9 0 2024-04-16T02:35:38.504 INFO:tasks.workunit.client.1.smithi153.stdout:1/560: chown d4/dd/d13/d23/d31/c89 612166 1 2024-04-16T02:35:38.504 INFO:tasks.workunit.client.1.smithi153.stdout:6/521: creat dc/d47/d57/d77/d15/dac/dae/fb0 x:0 0 0 2024-04-16T02:35:38.506 INFO:tasks.workunit.client.1.smithi153.stdout:5/591: rmdir d8/df/d28/d5a/d66/da8 0 2024-04-16T02:35:38.507 INFO:tasks.workunit.client.1.smithi153.stdout:3/505: write d6/d10/d22/d43/f57 [4481325,83501] 0 2024-04-16T02:35:38.509 INFO:tasks.workunit.client.1.smithi153.stdout:2/520: rename d9/d2b/d18/d3f/d40/lb7 to d9/d2b/d1c/d20/d45/lb8 0 2024-04-16T02:35:38.509 INFO:tasks.workunit.client.1.smithi153.stdout:0/635: dread dd/d14/da6/da4/fa8 [0,4194304] 0 2024-04-16T02:35:38.510 INFO:tasks.workunit.client.1.smithi153.stdout:6/522: unlink dc/d47/d93/c81 0 2024-04-16T02:35:38.511 INFO:tasks.workunit.client.1.smithi153.stdout:1/561: symlink d4/dd/d13/d23/d5f/lc1 0 2024-04-16T02:35:38.512 INFO:tasks.workunit.client.1.smithi153.stdout:2/521: read d9/d24/d27/d2c/d41/f56 [759213,27148] 0 2024-04-16T02:35:38.512 INFO:tasks.workunit.client.1.smithi153.stdout:3/506: mkdir d6/daf 0 2024-04-16T02:35:38.514 INFO:tasks.workunit.client.1.smithi153.stdout:6/523: write dc/d10/d62/f8b [162784,69530] 0 2024-04-16T02:35:38.515 INFO:tasks.workunit.client.1.smithi153.stdout:1/562: chown d4/dd/d13/d26/f9d 31 1 2024-04-16T02:35:38.515 INFO:tasks.workunit.client.1.smithi153.stdout:7/498: dwrite d0/d4/db/d47/f94 [0,4194304] 0 2024-04-16T02:35:38.520 INFO:tasks.workunit.client.1.smithi153.stdout:6/524: rename dc/d1a/d2d/c82 to dc/d59/d63/cb1 0 2024-04-16T02:35:38.521 INFO:tasks.workunit.client.1.smithi153.stdout:1/563: dread - d4/da2/d65/d84/fbd zero size 2024-04-16T02:35:38.523 INFO:tasks.workunit.client.1.smithi153.stdout:7/499: mkdir d0/d4/db/d34/d4b/daa 0 2024-04-16T02:35:38.528 INFO:tasks.workunit.client.1.smithi153.stdout:6/525: chown dc/d47/d57/d77/d15/f74 5402 1 2024-04-16T02:35:38.529 INFO:tasks.workunit.client.1.smithi153.stdout:5/592: dwrite d8/df/d28/d5a/d66/d99/db9/d39/fb4 [0,4194304] 0 2024-04-16T02:35:38.530 INFO:tasks.workunit.client.1.smithi153.stdout:7/500: mknod d0/d4/db/d2b/d53/cab 0 2024-04-16T02:35:38.530 INFO:tasks.workunit.client.1.smithi153.stdout:2/522: dwrite d9/d2b/d18/d3f/f49 [0,4194304] 0 2024-04-16T02:35:38.532 INFO:tasks.workunit.client.1.smithi153.stdout:8/477: dread d2/d13/d3a/f4c [0,4194304] 0 2024-04-16T02:35:38.535 INFO:tasks.workunit.client.1.smithi153.stdout:8/478: fsync d2/d13/f9f 0 2024-04-16T02:35:38.542 INFO:tasks.workunit.client.1.smithi153.stdout:2/523: write d9/d24/d27/f68 [1691947,1675] 0 2024-04-16T02:35:38.544 INFO:tasks.workunit.client.1.smithi153.stdout:6/526: dwrite dc/d47/d57/d77/d15/dac/dae/fb0 [0,4194304] 0 2024-04-16T02:35:38.546 INFO:tasks.workunit.client.1.smithi153.stdout:0/636: dread dd/d14/f3b [0,4194304] 0 2024-04-16T02:35:38.547 INFO:tasks.workunit.client.1.smithi153.stdout:2/524: stat d9/d71/d8d/da8 0 2024-04-16T02:35:38.548 INFO:tasks.workunit.client.1.smithi153.stdout:6/527: write dc/d10/d62/f8b [895112,83935] 0 2024-04-16T02:35:38.550 INFO:tasks.workunit.client.1.smithi153.stdout:2/525: chown d9/d2b/d18/d3f/d40/f70 706 1 2024-04-16T02:35:38.552 INFO:tasks.workunit.client.1.smithi153.stdout:5/593: creat d8/df/fba x:0 0 0 2024-04-16T02:35:38.557 INFO:tasks.workunit.client.1.smithi153.stdout:0/637: dwrite dd/d32/d57/f42 [0,4194304] 0 2024-04-16T02:35:38.559 INFO:tasks.workunit.client.1.smithi153.stdout:2/526: creat d9/d24/d27/d9a/fb9 x:0 0 0 2024-04-16T02:35:38.559 INFO:tasks.workunit.client.1.smithi153.stdout:7/501: getdents d0/d4/db/d47 0 2024-04-16T02:35:38.561 INFO:tasks.workunit.client.1.smithi153.stdout:3/507: readlink d6/d10/d3b/l48 0 2024-04-16T02:35:38.561 INFO:tasks.workunit.client.1.smithi153.stdout:1/564: truncate d4/dd/f1d 1640833 0 2024-04-16T02:35:38.563 INFO:tasks.workunit.client.1.smithi153.stdout:8/479: write d2/f14 [1597483,46792] 0 2024-04-16T02:35:38.564 INFO:tasks.workunit.client.1.smithi153.stdout:0/638: symlink dd/d32/d3f/lc3 0 2024-04-16T02:35:38.565 INFO:tasks.workunit.client.1.smithi153.stdout:7/502: creat d0/d4/db/d2b/fac x:0 0 0 2024-04-16T02:35:38.565 INFO:tasks.workunit.client.1.smithi153.stdout:0/639: stat dd/d32/d57/l63 0 2024-04-16T02:35:38.565 INFO:tasks.workunit.client.1.smithi153.stdout:8/480: write d2/d13/d2c/f48 [523179,101426] 0 2024-04-16T02:35:38.566 INFO:tasks.workunit.client.1.smithi153.stdout:2/527: rename d9/d2b/d18/d3f/fab to d9/fba 0 2024-04-16T02:35:38.569 INFO:tasks.workunit.client.1.smithi153.stdout:7/503: chown d0/d4/db/d34/d4b/f75 443496307 1 2024-04-16T02:35:38.569 INFO:tasks.workunit.client.1.smithi153.stdout:3/508: write d6/d10/d22/d2c/f44 [792114,22472] 0 2024-04-16T02:35:38.569 INFO:tasks.workunit.client.1.smithi153.stdout:1/565: mkdir d4/dd/d13/d23/db6/dc2 0 2024-04-16T02:35:38.604 INFO:tasks.workunit.client.0.smithi151.stdout:1/82: sync 2024-04-16T02:35:38.605 INFO:tasks.workunit.client.0.smithi151.stdout:1/83: readlink d3/d4/db/d16/l14 0 2024-04-16T02:35:38.607 INFO:tasks.workunit.client.0.smithi151.stdout:1/84: mkdir d3/d4/db/d1b/d1c 0 2024-04-16T02:35:38.609 INFO:tasks.workunit.client.0.smithi151.stdout:1/85: symlink d3/d4/db/d16/d10/l1d 0 2024-04-16T02:35:38.610 INFO:tasks.workunit.client.0.smithi151.stdout:1/86: creat d3/d4/db/d16/d10/f1e x:0 0 0 2024-04-16T02:35:38.612 INFO:tasks.workunit.client.0.smithi151.stdout:1/87: mkdir d3/d1f 0 2024-04-16T02:35:38.613 INFO:tasks.workunit.client.0.smithi151.stdout:1/88: truncate d3/d4/db/d16/d10/f1e 481029 0 2024-04-16T02:35:38.615 INFO:tasks.workunit.client.0.smithi151.stdout:1/89: dread d3/d4/db/d16/d10/f1e [0,4194304] 0 2024-04-16T02:35:38.616 INFO:tasks.workunit.client.0.smithi151.stdout:1/90: read d3/d4/db/d16/d10/f18 [656247,47074] 0 2024-04-16T02:35:38.617 INFO:tasks.workunit.client.0.smithi151.stdout:1/91: symlink d3/d1f/l20 0 2024-04-16T02:35:38.620 INFO:tasks.workunit.client.0.smithi151.stdout:1/92: dread f1 [0,4194304] 0 2024-04-16T02:35:38.622 INFO:tasks.workunit.client.0.smithi151.stdout:1/93: dread d3/d4/db/d16/d10/f1e [0,4194304] 0 2024-04-16T02:35:38.623 INFO:tasks.workunit.client.0.smithi151.stdout:1/94: dread d3/d4/db/d16/d10/f1e [0,4194304] 0 2024-04-16T02:35:38.626 INFO:tasks.workunit.client.0.smithi151.stdout:1/95: link d3/d4/c11 d3/d4/db/d1b/d1c/c21 0 2024-04-16T02:35:38.626 INFO:tasks.workunit.client.0.smithi151.stdout:1/96: stat d3/d4/db/d16/d10/f18 0 2024-04-16T02:35:38.627 INFO:tasks.workunit.client.0.smithi151.stdout:1/97: readlink d3/d4/db/d16/l14 0 2024-04-16T02:35:38.627 INFO:tasks.workunit.client.0.smithi151.stdout:1/98: rename d3/d4/db to d3/d4/db/d1b/d22 22 2024-04-16T02:35:38.630 INFO:tasks.workunit.client.0.smithi151.stdout:1/99: link d3/d4/db/d16/l14 d3/d4/l23 0 2024-04-16T02:35:38.631 INFO:tasks.workunit.client.0.smithi151.stdout:1/100: write d3/f8 [727952,32101] 0 2024-04-16T02:35:38.638 INFO:tasks.workunit.client.0.smithi151.stdout:1/101: dwrite d3/d4/db/d16/d10/f1e [0,4194304] 0 2024-04-16T02:35:38.639 INFO:tasks.workunit.client.0.smithi151.stdout:1/102: readlink d3/d4/db/d1b/l1a 0 2024-04-16T02:35:38.642 INFO:tasks.workunit.client.0.smithi151.stdout:1/103: symlink d3/d4/db/d16/d10/l24 0 2024-04-16T02:35:38.649 INFO:tasks.workunit.client.0.smithi151.stdout:1/104: dwrite d3/d4/db/d16/d10/f1e [0,4194304] 0 2024-04-16T02:35:38.652 INFO:tasks.workunit.client.0.smithi151.stdout:1/105: mknod d3/d4/db/df/c25 0 2024-04-16T02:35:38.656 INFO:tasks.workunit.client.0.smithi151.stdout:1/106: dread d3/d4/db/d16/f12 [4194304,4194304] 0 2024-04-16T02:35:38.661 INFO:tasks.workunit.client.1.smithi153.stdout:0/640: getdents dd/d14/da6/da4 0 2024-04-16T02:35:38.661 INFO:tasks.workunit.client.1.smithi153.stdout:3/509: write d6/d10/f32 [1222205,79182] 0 2024-04-16T02:35:38.662 INFO:tasks.workunit.client.0.smithi151.stdout:1/107: link d3/d4/l13 d3/d4/db/d16/l26 0 2024-04-16T02:35:38.663 INFO:tasks.workunit.client.1.smithi153.stdout:6/528: truncate f6 1032160 0 2024-04-16T02:35:38.666 INFO:tasks.workunit.client.0.smithi151.stdout:1/108: dread d3/d4/db/d16/d10/f18 [0,4194304] 0 2024-04-16T02:35:38.667 INFO:tasks.workunit.client.1.smithi153.stdout:7/504: unlink d0/d4/db/d2b/d43/f6b 0 2024-04-16T02:35:38.667 INFO:tasks.workunit.client.1.smithi153.stdout:5/594: dwrite d8/d1f/f73 [4194304,4194304] 0 2024-04-16T02:35:38.667 INFO:tasks.workunit.client.1.smithi153.stdout:8/481: dwrite d2/d13/f94 [0,4194304] 0 2024-04-16T02:35:38.667 INFO:tasks.workunit.client.0.smithi151.stdout:1/109: fdatasync f1 0 2024-04-16T02:35:38.668 INFO:tasks.workunit.client.1.smithi153.stdout:6/529: chown dc/d59/d63/f94 79266985 1 2024-04-16T02:35:38.671 INFO:tasks.workunit.client.1.smithi153.stdout:5/595: write d8/df/d2f/f60 [168635,69731] 0 2024-04-16T02:35:38.672 INFO:tasks.workunit.client.1.smithi153.stdout:1/566: dread d4/dd/d20/f55 [0,4194304] 0 2024-04-16T02:35:38.672 INFO:tasks.workunit.client.0.smithi151.stdout:1/110: dread f1 [0,4194304] 0 2024-04-16T02:35:38.673 INFO:tasks.workunit.client.1.smithi153.stdout:1/567: readlink d4/dd/d13/d7e/d80/la3 0 2024-04-16T02:35:38.674 INFO:tasks.workunit.client.1.smithi153.stdout:1/568: stat d4/dd/l67 0 2024-04-16T02:35:38.675 INFO:tasks.workunit.client.0.smithi151.stdout:1/111: stat d3/d4/c11 0 2024-04-16T02:35:38.676 INFO:tasks.workunit.client.1.smithi153.stdout:3/510: creat d6/d10/d22/d2e/d8a/d8c/fb0 x:0 0 0 2024-04-16T02:35:38.676 INFO:tasks.workunit.client.1.smithi153.stdout:3/511: chown d6/d10/d3b/d59/l6a 522 1 2024-04-16T02:35:38.677 INFO:tasks.workunit.client.1.smithi153.stdout:6/530: mknod dc/d59/d63/cb2 0 2024-04-16T02:35:38.678 INFO:tasks.workunit.client.1.smithi153.stdout:5/596: rmdir d8 39 2024-04-16T02:35:38.680 INFO:tasks.workunit.client.1.smithi153.stdout:0/641: creat dd/d14/d19/fc4 x:0 0 0 2024-04-16T02:35:38.682 INFO:tasks.workunit.client.1.smithi153.stdout:8/482: rename d2/d93/d97/d9c/d27/d2d/f31 to d2/d93/d97/d9c/d27/d2d/d47/d60/d67/fa2 0 2024-04-16T02:35:38.683 INFO:tasks.workunit.client.1.smithi153.stdout:1/569: mknod d4/da2/d65/d84/cc3 0 2024-04-16T02:35:38.685 INFO:tasks.workunit.client.1.smithi153.stdout:5/597: dread - d8/df/d28/d5a/d66/d99/db9/d83/fae zero size 2024-04-16T02:35:38.685 INFO:tasks.workunit.client.0.smithi151.stdout:1/112: truncate d3/d4/db/d16/d10/f1e 3647349 0 2024-04-16T02:35:38.686 INFO:tasks.workunit.client.1.smithi153.stdout:0/642: mknod dd/d32/d57/cc5 0 2024-04-16T02:35:38.687 INFO:tasks.workunit.client.0.smithi151.stdout:1/113: fsync d3/d4/db/d16/d10/f18 0 2024-04-16T02:35:38.687 INFO:tasks.workunit.client.1.smithi153.stdout:6/531: rename dc/d47/d57/d77/d7a/c8e to dc/d47/d57/d77/d15/dac/dae/cb3 0 2024-04-16T02:35:38.688 INFO:tasks.workunit.client.1.smithi153.stdout:7/505: link d0/d4/db/d34/d4b/c95 d0/d4/db/d2b/cad 0 2024-04-16T02:35:38.691 INFO:tasks.workunit.client.1.smithi153.stdout:5/598: mknod d8/d41/d90/d9d/cbb 0 2024-04-16T02:35:38.691 INFO:tasks.workunit.client.0.smithi151.stdout:1/114: dread d3/d4/db/d16/d10/f18 [0,4194304] 0 2024-04-16T02:35:38.692 INFO:tasks.workunit.client.1.smithi153.stdout:3/512: creat d6/d10/d22/fb1 x:0 0 0 2024-04-16T02:35:38.692 INFO:tasks.workunit.client.1.smithi153.stdout:9/565: sync 2024-04-16T02:35:38.694 INFO:tasks.workunit.client.0.smithi151.stdout:1/115: mknod d3/d4/db/d16/c27 0 2024-04-16T02:35:38.695 INFO:tasks.workunit.client.1.smithi153.stdout:7/506: creat d0/d4/db/d2b/d8c/fae x:0 0 0 2024-04-16T02:35:38.698 INFO:tasks.workunit.client.1.smithi153.stdout:9/566: dwrite d4/d33/f4f [0,4194304] 0 2024-04-16T02:35:38.699 INFO:tasks.workunit.client.0.smithi151.stdout:1/116: write f1 [2976074,121663] 0 2024-04-16T02:35:38.699 INFO:tasks.workunit.client.1.smithi153.stdout:3/513: mknod d6/d10/d3b/d76/cb2 0 2024-04-16T02:35:38.704 INFO:tasks.workunit.client.0.smithi151.stdout:1/117: link d3/d4/db/df/c25 d3/d4/db/d16/d10/c28 0 2024-04-16T02:35:38.704 INFO:tasks.workunit.client.0.smithi151.stdout:1/118: chown d3/d4/db/d16 54500 1 2024-04-16T02:35:38.705 INFO:tasks.workunit.client.0.smithi151.stdout:1/119: rmdir d3/d4 39 2024-04-16T02:35:38.709 INFO:tasks.workunit.client.0.smithi151.stdout:1/120: chown d3/d4/db/d16/d10/f1e 19011 1 2024-04-16T02:35:38.710 INFO:tasks.workunit.client.0.smithi151.stdout:1/121: mknod d3/d4/c29 0 2024-04-16T02:35:38.715 INFO:tasks.workunit.client.0.smithi151.stdout:1/122: dwrite d3/d4/db/d16/d10/f18 [0,4194304] 0 2024-04-16T02:35:38.724 INFO:tasks.workunit.client.0.smithi151.stdout:1/123: dwrite d3/d4/db/d16/d10/f1e [0,4194304] 0 2024-04-16T02:35:38.729 INFO:tasks.workunit.client.0.smithi151.stdout:1/124: dread d3/f8 [0,4194304] 0 2024-04-16T02:35:38.730 INFO:tasks.workunit.client.0.smithi151.stdout:1/125: fdatasync d3/f8 0 2024-04-16T02:35:38.731 INFO:tasks.workunit.client.0.smithi151.stdout:1/126: chown d3/d4/db/d16/f12 157956059 1 2024-04-16T02:35:38.734 INFO:tasks.workunit.client.0.smithi151.stdout:1/127: dread d3/d4/db/d16/f12 [0,4194304] 0 2024-04-16T02:35:38.736 INFO:tasks.workunit.client.0.smithi151.stdout:1/128: write d3/d4/db/d16/d10/f18 [3332270,59697] 0 2024-04-16T02:35:38.739 INFO:tasks.workunit.client.0.smithi151.stdout:2/137: sync 2024-04-16T02:35:38.740 INFO:tasks.workunit.client.0.smithi151.stdout:1/129: dread d3/d4/db/d16/d10/f1e [0,4194304] 0 2024-04-16T02:35:38.740 INFO:tasks.workunit.client.0.smithi151.stdout:2/138: truncate de/ff 12965313 0 2024-04-16T02:35:38.740 INFO:tasks.workunit.client.0.smithi151.stdout:2/139: readlink l5 0 2024-04-16T02:35:38.742 INFO:tasks.workunit.client.0.smithi151.stdout:1/130: mkdir d3/d4/db/d1b/d2a 0 2024-04-16T02:35:38.745 INFO:tasks.workunit.client.0.smithi151.stdout:1/131: write d3/d4/db/d16/d10/f1e [1149086,5879] 0 2024-04-16T02:35:38.746 INFO:tasks.workunit.client.0.smithi151.stdout:2/140: dwrite de/d15/f2c [0,4194304] 0 2024-04-16T02:35:38.749 INFO:tasks.workunit.client.0.smithi151.stdout:2/141: creat de/d11/f2e x:0 0 0 2024-04-16T02:35:38.751 INFO:tasks.workunit.client.1.smithi153.stdout:1/570: rename d4/dd/d13/d23/d5f/fb9 to d4/fc4 0 2024-04-16T02:35:38.751 INFO:tasks.workunit.client.0.smithi151.stdout:1/132: link d3/d4/db/d16/l14 d3/d4/db/d1b/d1c/l2b 0 2024-04-16T02:35:38.752 INFO:tasks.workunit.client.1.smithi153.stdout:8/483: write d2/d93/d97/d9c/d27/d2d/f81 [3140228,97768] 0 2024-04-16T02:35:38.758 INFO:tasks.workunit.client.1.smithi153.stdout:1/571: creat d4/dd/d18/d29/d71/fc5 x:0 0 0 2024-04-16T02:35:38.758 INFO:tasks.workunit.client.0.smithi151.stdout:2/142: dwrite de/d15/d1e/f24 [0,4194304] 0 2024-04-16T02:35:38.759 INFO:tasks.workunit.client.1.smithi153.stdout:3/514: getdents d6 0 2024-04-16T02:35:38.760 INFO:tasks.workunit.client.1.smithi153.stdout:8/484: creat d2/d93/d97/d9c/d27/d5e/fa3 x:0 0 0 2024-04-16T02:35:38.763 INFO:tasks.workunit.client.1.smithi153.stdout:9/567: dread d4/d6/dc/d2d/da0/f7c [0,4194304] 0 2024-04-16T02:35:38.763 INFO:tasks.workunit.client.1.smithi153.stdout:8/485: fdatasync d2/ff 0 2024-04-16T02:35:38.763 INFO:tasks.workunit.client.1.smithi153.stdout:1/572: symlink d4/dd/d20/lc6 0 2024-04-16T02:35:38.767 INFO:tasks.workunit.client.1.smithi153.stdout:8/486: chown d2/f12 2361 1 2024-04-16T02:35:38.768 INFO:tasks.workunit.client.1.smithi153.stdout:0/643: dwrite dd/d28/f4c [0,4194304] 0 2024-04-16T02:35:38.770 INFO:tasks.workunit.client.1.smithi153.stdout:9/568: mkdir d4/d6/dc/d24/d5d/db0 0 2024-04-16T02:35:38.770 INFO:tasks.workunit.client.1.smithi153.stdout:6/532: dwrite dc/d47/d57/d77/d15/f1e [0,4194304] 0 2024-04-16T02:35:38.772 INFO:tasks.workunit.client.1.smithi153.stdout:1/573: mkdir d4/dd/d13/d8d/d95/dc7 0 2024-04-16T02:35:38.772 INFO:tasks.workunit.client.1.smithi153.stdout:8/487: dread - d2/d93/d97/d9c/f3c zero size 2024-04-16T02:35:38.773 INFO:tasks.workunit.client.1.smithi153.stdout:0/644: readlink dd/d14/d51/d7f/l8a 0 2024-04-16T02:35:38.774 INFO:tasks.workunit.client.0.smithi151.stdout:1/133: truncate d3/d4/db/d16/d10/f1e 1615099 0 2024-04-16T02:35:38.776 INFO:tasks.workunit.client.1.smithi153.stdout:3/515: dwrite d6/d10/d22/f4c [0,4194304] 0 2024-04-16T02:35:38.777 INFO:tasks.workunit.client.1.smithi153.stdout:3/516: write d6/d10/d22/d2e/f45 [2878376,63732] 0 2024-04-16T02:35:38.778 INFO:tasks.workunit.client.0.smithi151.stdout:1/134: creat d3/d4/db/d1b/d1c/f2c x:0 0 0 2024-04-16T02:35:38.781 INFO:tasks.workunit.client.1.smithi153.stdout:6/533: creat dc/d47/d93/fb4 x:0 0 0 2024-04-16T02:35:38.782 INFO:tasks.workunit.client.1.smithi153.stdout:9/569: mkdir d4/d6/db1 0 2024-04-16T02:35:38.783 INFO:tasks.workunit.client.0.smithi151.stdout:1/135: rename d3/d4/db/d1b/l1a to d3/d4/db/d1b/l2d 0 2024-04-16T02:35:38.784 INFO:tasks.workunit.client.0.smithi151.stdout:1/136: stat d3/d1f/l20 0 2024-04-16T02:35:38.785 INFO:tasks.workunit.client.0.smithi151.stdout:1/137: dread - d3/d4/db/d1b/d1c/f2c zero size 2024-04-16T02:35:38.786 INFO:tasks.workunit.client.1.smithi153.stdout:7/507: dread d0/d4/db/d2b/d53/f27 [0,4194304] 0 2024-04-16T02:35:38.788 INFO:tasks.workunit.client.1.smithi153.stdout:7/508: write d0/f1e [5322292,103361] 0 2024-04-16T02:35:38.788 INFO:tasks.workunit.client.1.smithi153.stdout:3/517: rename d6/l99 to d6/d10/d22/d43/lb3 0 2024-04-16T02:35:38.788 INFO:tasks.workunit.client.0.smithi151.stdout:1/138: creat d3/d1f/f2e x:0 0 0 2024-04-16T02:35:38.789 INFO:tasks.workunit.client.1.smithi153.stdout:7/509: chown d0/d4/db/d2b/f25 396057 1 2024-04-16T02:35:38.791 INFO:tasks.workunit.client.1.smithi153.stdout:4/632: sync 2024-04-16T02:35:38.793 INFO:tasks.workunit.client.1.smithi153.stdout:4/633: fdatasync d3/d9/db/de/d2c/d38/f9d 0 2024-04-16T02:35:38.793 INFO:tasks.workunit.client.1.smithi153.stdout:0/645: rename dd/d14/d51/d54/d65/f69 to dd/d26/d44/fc6 0 2024-04-16T02:35:38.794 INFO:tasks.workunit.client.1.smithi153.stdout:7/510: symlink d0/d4/db/d34/d51/laf 0 2024-04-16T02:35:38.798 INFO:tasks.workunit.client.1.smithi153.stdout:0/646: unlink dd/d32/fb9 0 2024-04-16T02:35:38.798 INFO:tasks.workunit.client.1.smithi153.stdout:7/511: rmdir d0/d4/db/d2b/d8c/d9f 39 2024-04-16T02:35:38.798 INFO:tasks.workunit.client.1.smithi153.stdout:1/574: dread d4/dd/d18/f51 [0,4194304] 0 2024-04-16T02:35:38.799 INFO:tasks.workunit.client.0.smithi151.stdout:2/143: dread f7 [0,4194304] 0 2024-04-16T02:35:38.800 INFO:tasks.workunit.client.1.smithi153.stdout:7/512: write d0/d4/db/d2b/d53/f46 [2396776,48216] 0 2024-04-16T02:35:38.802 INFO:tasks.workunit.client.1.smithi153.stdout:1/575: symlink d4/da2/d65/d84/lc8 0 2024-04-16T02:35:38.802 INFO:tasks.workunit.client.1.smithi153.stdout:9/570: dread d4/f27 [0,4194304] 0 2024-04-16T02:35:38.803 INFO:tasks.workunit.client.1.smithi153.stdout:9/571: dread - d4/d6/dc/d7f/fa3 zero size 2024-04-16T02:35:38.804 INFO:tasks.workunit.client.1.smithi153.stdout:1/576: write d4/dd/fbe [152081,66860] 0 2024-04-16T02:35:38.804 INFO:tasks.workunit.client.1.smithi153.stdout:7/513: creat d0/d4/db/d47/d8d/d93/d8b/fb0 x:0 0 0 2024-04-16T02:35:38.805 INFO:tasks.workunit.client.1.smithi153.stdout:0/647: getdents dd/d14/d19/d20/d4a 0 2024-04-16T02:35:38.808 INFO:tasks.workunit.client.1.smithi153.stdout:9/572: fdatasync d4/d6/dc/d24/d4e/f81 0 2024-04-16T02:35:38.808 INFO:tasks.workunit.client.0.smithi151.stdout:1/139: chown d3/d4/db/d1b/l2d 0 1 2024-04-16T02:35:38.809 INFO:tasks.workunit.client.1.smithi153.stdout:9/573: dread - d4/d6/dc/d24/d5d/fab zero size 2024-04-16T02:35:38.810 INFO:tasks.workunit.client.0.smithi151.stdout:2/144: truncate de/d15/f1c 1908317 0 2024-04-16T02:35:38.810 INFO:tasks.workunit.client.1.smithi153.stdout:0/648: creat dd/d14/d51/d54/fc7 x:0 0 0 2024-04-16T02:35:38.811 INFO:tasks.workunit.client.1.smithi153.stdout:0/649: truncate dd/f22 4726557 0 2024-04-16T02:35:38.816 INFO:tasks.workunit.client.0.smithi151.stdout:2/145: link de/d15/l23 de/d15/d1e/l2f 0 2024-04-16T02:35:38.818 INFO:tasks.workunit.client.1.smithi153.stdout:6/534: dwrite dc/d47/f61 [4194304,4194304] 0 2024-04-16T02:35:38.819 INFO:tasks.workunit.client.1.smithi153.stdout:1/577: truncate d4/f40 1875220 0 2024-04-16T02:35:38.820 INFO:tasks.workunit.client.0.smithi151.stdout:1/140: rename d3/d4/db/d16/d10/f1e to d3/f2f 0 2024-04-16T02:35:38.821 INFO:tasks.workunit.client.1.smithi153.stdout:4/634: write d3/d9/db/d45/fcc [505250,113988] 0 2024-04-16T02:35:38.822 INFO:tasks.workunit.client.0.smithi151.stdout:2/146: mknod de/d11/c30 0 2024-04-16T02:35:38.826 INFO:tasks.workunit.client.0.smithi151.stdout:1/141: dread f1 [0,4194304] 0 2024-04-16T02:35:38.827 INFO:tasks.workunit.client.0.smithi151.stdout:2/147: readlink de/d11/l26 0 2024-04-16T02:35:38.828 INFO:tasks.workunit.client.0.smithi151.stdout:2/148: mknod de/d11/c31 0 2024-04-16T02:35:38.829 INFO:tasks.workunit.client.1.smithi153.stdout:6/535: fsync dc/d10/f1b 0 2024-04-16T02:35:38.830 INFO:tasks.workunit.client.0.smithi151.stdout:0/183: sync 2024-04-16T02:35:38.830 INFO:tasks.workunit.client.0.smithi151.stdout:4/136: sync 2024-04-16T02:35:38.830 INFO:tasks.workunit.client.0.smithi151.stdout:9/68: sync 2024-04-16T02:35:38.830 INFO:tasks.workunit.client.0.smithi151.stdout:1/142: creat d3/d4/db/d1b/d2a/f30 x:0 0 0 2024-04-16T02:35:38.831 INFO:tasks.workunit.client.1.smithi153.stdout:7/514: link d0/d4/db/d2b/f22 d0/d4/db/d16/fb1 0 2024-04-16T02:35:38.832 INFO:tasks.workunit.client.0.smithi151.stdout:1/143: rmdir d3 39 2024-04-16T02:35:38.832 INFO:tasks.workunit.client.0.smithi151.stdout:9/69: truncate d1/d3/d6/f13 419824 0 2024-04-16T02:35:38.833 INFO:tasks.workunit.client.0.smithi151.stdout:4/137: creat d0/d20/f2c x:0 0 0 2024-04-16T02:35:38.834 INFO:tasks.workunit.client.0.smithi151.stdout:4/138: fdatasync d0/fd 0 2024-04-16T02:35:38.834 INFO:tasks.workunit.client.1.smithi153.stdout:0/650: dwrite dd/d14/d51/f60 [4194304,4194304] 0 2024-04-16T02:35:38.837 INFO:tasks.workunit.client.1.smithi153.stdout:4/635: dwrite d3/d9/db/de/d2c/d38/f9d [0,4194304] 0 2024-04-16T02:35:38.838 INFO:tasks.workunit.client.1.smithi153.stdout:1/578: dread d4/dd/d20/fa4 [0,4194304] 0 2024-04-16T02:35:38.838 INFO:tasks.workunit.client.0.smithi151.stdout:4/139: dread d0/f2 [4194304,4194304] 0 2024-04-16T02:35:38.844 INFO:tasks.workunit.client.1.smithi153.stdout:4/636: truncate d3/fdc 708172 0 2024-04-16T02:35:38.846 INFO:tasks.workunit.client.0.smithi151.stdout:1/144: dwrite d3/d4/db/d1b/d1c/f2c [0,4194304] 0 2024-04-16T02:35:38.848 INFO:tasks.workunit.client.1.smithi153.stdout:3/518: dread d6/d10/d22/d2e/f6f [0,4194304] 0 2024-04-16T02:35:38.848 INFO:tasks.workunit.client.1.smithi153.stdout:0/651: rename dd/d14/d19/d77/d87/fba to dd/d32/fc8 0 2024-04-16T02:35:38.850 INFO:tasks.workunit.client.0.smithi151.stdout:1/145: chown d3/f2f 21445069 1 2024-04-16T02:35:38.851 INFO:tasks.workunit.client.0.smithi151.stdout:0/184: getdents dc/d15/d34 0 2024-04-16T02:35:38.853 INFO:tasks.workunit.client.0.smithi151.stdout:0/185: write dc/ff [4362585,23082] 0 2024-04-16T02:35:38.854 INFO:tasks.workunit.client.0.smithi151.stdout:1/146: unlink d3/d1f/f2e 0 2024-04-16T02:35:38.855 INFO:tasks.workunit.client.1.smithi153.stdout:9/574: dwrite d4/d6/dc/d2d/da0/d40/f78 [0,4194304] 0 2024-04-16T02:35:38.855 INFO:tasks.workunit.client.0.smithi151.stdout:9/70: dread d1/f5 [4194304,4194304] 0 2024-04-16T02:35:38.856 INFO:tasks.workunit.client.0.smithi151.stdout:4/140: creat d0/f2d x:0 0 0 2024-04-16T02:35:38.856 INFO:tasks.workunit.client.0.smithi151.stdout:4/141: chown d0 183056377 1 2024-04-16T02:35:38.861 INFO:tasks.workunit.client.1.smithi153.stdout:6/536: dwrite dc/d6b/f73 [0,4194304] 0 2024-04-16T02:35:38.861 INFO:tasks.workunit.client.1.smithi153.stdout:7/515: dwrite d0/d4/db/d2b/d53/f46 [0,4194304] 0 2024-04-16T02:35:38.861 INFO:tasks.workunit.client.0.smithi151.stdout:9/71: creat d1/dd/f16 x:0 0 0 2024-04-16T02:35:38.861 INFO:tasks.workunit.client.1.smithi153.stdout:3/519: symlink d6/d10/d22/d43/lb4 0 2024-04-16T02:35:38.865 INFO:tasks.workunit.client.0.smithi151.stdout:0/186: rename dc/f2f to dc/f39 0 2024-04-16T02:35:38.866 INFO:tasks.workunit.client.0.smithi151.stdout:1/147: dwrite d3/d4/db/d16/f12 [4194304,4194304] 0 2024-04-16T02:35:38.868 INFO:tasks.workunit.client.1.smithi153.stdout:2/528: sync 2024-04-16T02:35:38.870 INFO:tasks.workunit.client.1.smithi153.stdout:0/652: dread dd/d14/d19/d20/d4a/f78 [0,4194304] 0 2024-04-16T02:35:38.871 INFO:tasks.workunit.client.1.smithi153.stdout:9/575: dread d4/d6/dc/d24/d4e/f81 [0,4194304] 0 2024-04-16T02:35:38.872 INFO:tasks.workunit.client.1.smithi153.stdout:3/520: symlink d6/d10/d5b/lb5 0 2024-04-16T02:35:38.874 INFO:tasks.workunit.client.0.smithi151.stdout:4/142: link d0/d20/l2a d0/l2e 0 2024-04-16T02:35:38.875 INFO:tasks.workunit.client.1.smithi153.stdout:7/516: unlink d0/d4/db/d47/c97 0 2024-04-16T02:35:38.876 INFO:tasks.workunit.client.1.smithi153.stdout:3/521: write d6/d10/d22/f2a [4862630,41598] 0 2024-04-16T02:35:38.876 INFO:tasks.workunit.client.1.smithi153.stdout:6/537: rmdir dc/d1a/d2d/da6 0 2024-04-16T02:35:38.876 INFO:tasks.workunit.client.1.smithi153.stdout:2/529: dread d9/d2b/d1c/d20/f67 [0,4194304] 0 2024-04-16T02:35:38.878 INFO:tasks.workunit.client.1.smithi153.stdout:2/530: chown d9/d71/l9c 514764 1 2024-04-16T02:35:38.879 INFO:tasks.workunit.client.0.smithi151.stdout:2/149: dwrite de/d15/f1c [0,4194304] 0 2024-04-16T02:35:38.879 INFO:tasks.workunit.client.0.smithi151.stdout:9/72: dwrite d1/dd/f16 [0,4194304] 0 2024-04-16T02:35:38.879 INFO:tasks.workunit.client.1.smithi153.stdout:2/531: write d9/d2b/d18/d3f/f6e [5219885,31864] 0 2024-04-16T02:35:38.880 INFO:tasks.workunit.client.0.smithi151.stdout:2/150: dread - de/d11/f2b zero size 2024-04-16T02:35:38.881 INFO:tasks.workunit.client.0.smithi151.stdout:4/143: unlink d0/d20/f27 0 2024-04-16T02:35:38.882 INFO:tasks.workunit.client.0.smithi151.stdout:1/148: dread f1 [0,4194304] 0 2024-04-16T02:35:38.884 INFO:tasks.workunit.client.0.smithi151.stdout:4/144: truncate d0/f3 3279900 0 2024-04-16T02:35:38.884 INFO:tasks.workunit.client.1.smithi153.stdout:7/517: symlink d0/lb2 0 2024-04-16T02:35:38.885 INFO:tasks.workunit.client.0.smithi151.stdout:4/145: dread - d0/d11/d19/f1e zero size 2024-04-16T02:35:38.887 INFO:tasks.workunit.client.0.smithi151.stdout:2/151: rename de/ff to de/d15/d1e/d28/f32 0 2024-04-16T02:35:38.888 INFO:tasks.workunit.client.0.smithi151.stdout:1/149: mknod d3/d4/db/d1b/d2a/c31 0 2024-04-16T02:35:38.889 INFO:tasks.workunit.client.0.smithi151.stdout:4/146: creat d0/d20/f2f x:0 0 0 2024-04-16T02:35:38.890 INFO:tasks.workunit.client.1.smithi153.stdout:2/532: read d9/f9e [3623411,35917] 0 2024-04-16T02:35:38.891 INFO:tasks.workunit.client.1.smithi153.stdout:9/576: link d4/d6/dc/d2d/da0/f88 d4/d6/dc/d24/d5d/d66/fb2 0 2024-04-16T02:35:38.891 INFO:tasks.workunit.client.1.smithi153.stdout:7/518: rmdir d0/d4/db/d34/d4b/d4d 39 2024-04-16T02:35:38.892 INFO:tasks.workunit.client.0.smithi151.stdout:4/147: write d0/d20/f2f [36145,39947] 0 2024-04-16T02:35:38.893 INFO:tasks.workunit.client.0.smithi151.stdout:4/148: read - d0/d20/f21 zero size 2024-04-16T02:35:38.893 INFO:tasks.workunit.client.1.smithi153.stdout:3/522: dwrite d6/f11 [0,4194304] 0 2024-04-16T02:35:38.895 INFO:tasks.workunit.client.1.smithi153.stdout:3/523: write d6/d10/d22/d43/f57 [5390993,15327] 0 2024-04-16T02:35:38.896 INFO:tasks.workunit.client.0.smithi151.stdout:1/150: dread d3/d4/db/d16/f12 [8388608,4194304] 0 2024-04-16T02:35:38.896 INFO:tasks.workunit.client.1.smithi153.stdout:2/533: creat d9/d24/d27/d2c/d5d/fbb x:0 0 0 2024-04-16T02:35:38.897 INFO:tasks.workunit.client.1.smithi153.stdout:9/577: rmdir d4/d6/dc/d24/d5d/d66 39 2024-04-16T02:35:38.897 INFO:tasks.workunit.client.1.smithi153.stdout:3/524: fsync d6/d10/d22/d2e/f4e 0 2024-04-16T02:35:38.897 INFO:tasks.workunit.client.1.smithi153.stdout:2/534: stat d9/d24/d73 0 2024-04-16T02:35:38.898 INFO:tasks.workunit.client.0.smithi151.stdout:1/151: read d3/d4/db/d1b/d1c/f2c [1852352,98621] 0 2024-04-16T02:35:38.902 INFO:tasks.workunit.client.1.smithi153.stdout:5/599: sync 2024-04-16T02:35:38.902 INFO:tasks.workunit.client.1.smithi153.stdout:8/488: sync 2024-04-16T02:35:38.902 INFO:tasks.workunit.client.1.smithi153.stdout:9/578: creat d4/d6/dc/d7f/fb3 x:0 0 0 2024-04-16T02:35:38.903 INFO:tasks.workunit.client.1.smithi153.stdout:1/579: dread d4/dd/d13/d26/f34 [0,4194304] 0 2024-04-16T02:35:38.903 INFO:tasks.workunit.client.1.smithi153.stdout:9/579: readlink d4/l9e 0 2024-04-16T02:35:38.903 INFO:tasks.workunit.client.0.smithi151.stdout:4/149: dwrite d0/d11/d19/f1e [0,4194304] 0 2024-04-16T02:35:38.903 INFO:tasks.workunit.client.0.smithi151.stdout:1/152: mkdir d3/d4/db/df/d32 0 2024-04-16T02:35:38.906 INFO:tasks.workunit.client.1.smithi153.stdout:5/600: symlink d8/df/d2e/lbc 0 2024-04-16T02:35:38.910 INFO:tasks.workunit.client.1.smithi153.stdout:2/535: dread f6 [0,4194304] 0 2024-04-16T02:35:38.914 INFO:tasks.workunit.client.1.smithi153.stdout:3/525: creat d6/d10/d22/d2e/d8a/fb6 x:0 0 0 2024-04-16T02:35:38.916 INFO:tasks.workunit.client.1.smithi153.stdout:0/653: dread dd/d32/d57/f5d [0,4194304] 0 2024-04-16T02:35:38.916 INFO:tasks.workunit.client.1.smithi153.stdout:2/536: symlink d9/d24/d27/lbc 0 2024-04-16T02:35:38.918 INFO:tasks.workunit.client.1.smithi153.stdout:4/637: truncate d3/f1c 814801 0 2024-04-16T02:35:38.919 INFO:tasks.workunit.client.1.smithi153.stdout:3/526: mkdir d6/d10/d3b/d59/d7c/db7 0 2024-04-16T02:35:38.922 INFO:tasks.workunit.client.1.smithi153.stdout:5/601: dwrite d8/df/d2f/d4e/faa [0,4194304] 0 2024-04-16T02:35:38.926 INFO:tasks.workunit.client.1.smithi153.stdout:8/489: dwrite d2/de/d3f/f91 [0,4194304] 0 2024-04-16T02:35:38.926 INFO:tasks.workunit.client.0.smithi151.stdout:4/150: rmdir d0/d20 39 2024-04-16T02:35:38.927 INFO:tasks.workunit.client.1.smithi153.stdout:5/602: write d8/df/d28/d5a/fb8 [862612,82926] 0 2024-04-16T02:35:38.930 INFO:tasks.workunit.client.0.smithi151.stdout:1/153: truncate f1 1076905 0 2024-04-16T02:35:38.932 INFO:tasks.workunit.client.0.smithi151.stdout:1/154: fsync d3/d4/db/d1b/d2a/f30 0 2024-04-16T02:35:38.932 INFO:tasks.workunit.client.1.smithi153.stdout:3/527: symlink d6/d10/d22/d2e/lb8 0 2024-04-16T02:35:38.934 INFO:tasks.workunit.client.1.smithi153.stdout:9/580: dread d4/f48 [0,4194304] 0 2024-04-16T02:35:38.934 INFO:tasks.workunit.client.0.smithi151.stdout:4/151: write d0/f2 [2876360,61774] 0 2024-04-16T02:35:38.935 INFO:tasks.workunit.client.1.smithi153.stdout:5/603: readlink d8/d1f/l7f 0 2024-04-16T02:35:38.935 INFO:tasks.workunit.client.1.smithi153.stdout:8/490: mkdir d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/da4 0 2024-04-16T02:35:38.936 INFO:tasks.workunit.client.1.smithi153.stdout:5/604: write d8/df/f97 [513049,110120] 0 2024-04-16T02:35:38.937 INFO:tasks.workunit.client.1.smithi153.stdout:4/638: getdents d3/d9/db/de/d2c/de2 0 2024-04-16T02:35:38.938 INFO:tasks.workunit.client.1.smithi153.stdout:3/528: mknod d6/d10/d22/d4d/cb9 0 2024-04-16T02:35:38.938 INFO:tasks.workunit.client.0.smithi151.stdout:4/152: symlink d0/d20/l30 0 2024-04-16T02:35:38.940 INFO:tasks.workunit.client.1.smithi153.stdout:7/519: write d0/d4/db/d2b/d43/d6d/d68/f9c [895271,67196] 0 2024-04-16T02:35:38.940 INFO:tasks.workunit.client.0.smithi151.stdout:4/153: mkdir d0/d11/d19/d31 0 2024-04-16T02:35:38.942 INFO:tasks.workunit.client.1.smithi153.stdout:4/639: chown d3/d9/f77 7074764 1 2024-04-16T02:35:38.942 INFO:tasks.workunit.client.0.smithi151.stdout:4/154: symlink d0/d11/d19/d31/l32 0 2024-04-16T02:35:38.943 INFO:tasks.workunit.client.1.smithi153.stdout:1/580: write d4/dd/d20/f72 [1686101,83890] 0 2024-04-16T02:35:38.943 INFO:tasks.workunit.client.1.smithi153.stdout:6/538: dwrite f6 [0,4194304] 0 2024-04-16T02:35:38.946 INFO:tasks.workunit.client.0.smithi151.stdout:4/155: rename d0/d11/d19/d31/l32 to d0/d17/l33 0 2024-04-16T02:35:38.947 INFO:tasks.workunit.client.0.smithi151.stdout:4/156: read - d0/d11/d19/f29 zero size 2024-04-16T02:35:38.950 INFO:tasks.workunit.client.0.smithi151.stdout:4/157: mkdir d0/d11/d19/d31/d34 0 2024-04-16T02:35:38.951 INFO:tasks.workunit.client.1.smithi153.stdout:3/529: write d6/d10/d22/d2e/f6f [4274318,33739] 0 2024-04-16T02:35:38.953 INFO:tasks.workunit.client.1.smithi153.stdout:4/640: chown d3/f73 4 1 2024-04-16T02:35:38.954 INFO:tasks.workunit.client.1.smithi153.stdout:1/581: write d4/dd/d13/d23/d5f/f7f [3302806,14310] 0 2024-04-16T02:35:38.954 INFO:tasks.workunit.client.0.smithi151.stdout:1/155: truncate d3/d4/db/d16/d10/f18 3389016 0 2024-04-16T02:35:38.955 INFO:tasks.workunit.client.1.smithi153.stdout:5/605: dwrite d8/df/f88 [0,4194304] 0 2024-04-16T02:35:38.957 INFO:tasks.workunit.client.1.smithi153.stdout:7/520: unlink d0/d4/db/c7e 0 2024-04-16T02:35:38.958 INFO:tasks.workunit.client.0.smithi151.stdout:4/158: dwrite d0/f2d [0,4194304] 0 2024-04-16T02:35:38.960 INFO:tasks.workunit.client.0.smithi151.stdout:1/156: read d3/d4/db/d1b/d1c/f2c [2579888,65184] 0 2024-04-16T02:35:38.961 INFO:tasks.workunit.client.1.smithi153.stdout:2/537: write d9/d2b/f19 [141429,110193] 0 2024-04-16T02:35:38.962 INFO:tasks.workunit.client.1.smithi153.stdout:8/491: dwrite d2/d1c/f36 [0,4194304] 0 2024-04-16T02:35:38.964 INFO:tasks.workunit.client.1.smithi153.stdout:5/606: write d8/df/d23/f56 [439627,124690] 0 2024-04-16T02:35:38.965 INFO:tasks.workunit.client.1.smithi153.stdout:4/641: unlink d3/d9/d24/d3e/f80 0 2024-04-16T02:35:38.965 INFO:tasks.workunit.client.0.smithi151.stdout:1/157: fdatasync d3/f2f 0 2024-04-16T02:35:38.965 INFO:tasks.workunit.client.1.smithi153.stdout:9/581: dwrite d4/d6/dc/d46/f84 [0,4194304] 0 2024-04-16T02:35:38.965 INFO:tasks.workunit.client.0.smithi151.stdout:4/159: dread d0/f1f [0,4194304] 0 2024-04-16T02:35:38.966 INFO:tasks.workunit.client.1.smithi153.stdout:3/530: write d6/d10/d22/d2e/d8a/f91 [911357,24053] 0 2024-04-16T02:35:38.970 INFO:tasks.workunit.client.0.smithi151.stdout:4/160: mknod d0/d11/d19/d31/d34/c35 0 2024-04-16T02:35:38.972 INFO:tasks.workunit.client.1.smithi153.stdout:5/607: fsync d8/df/d23/f93 0 2024-04-16T02:35:38.973 INFO:tasks.workunit.client.1.smithi153.stdout:0/654: dread dd/d26/f5e [0,4194304] 0 2024-04-16T02:35:38.977 INFO:tasks.workunit.client.0.smithi151.stdout:4/161: dread d0/f15 [0,4194304] 0 2024-04-16T02:35:38.978 INFO:tasks.workunit.client.1.smithi153.stdout:9/582: stat d4/d6/dc/d24/d60 0 2024-04-16T02:35:38.979 INFO:tasks.workunit.client.1.smithi153.stdout:1/582: truncate d4/f40 922398 0 2024-04-16T02:35:38.980 INFO:tasks.workunit.client.1.smithi153.stdout:9/583: stat d4/d33/l45 0 2024-04-16T02:35:38.980 INFO:tasks.workunit.client.0.smithi151.stdout:4/162: creat d0/d20/f36 x:0 0 0 2024-04-16T02:35:38.980 INFO:tasks.workunit.client.0.smithi151.stdout:4/163: read - d0/d20/f36 zero size 2024-04-16T02:35:38.981 INFO:tasks.workunit.client.0.smithi151.stdout:4/164: chown d0/d11/d19/d31/d34 34 1 2024-04-16T02:35:38.981 INFO:tasks.workunit.client.1.smithi153.stdout:8/492: dread - d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/f89 zero size 2024-04-16T02:35:38.984 INFO:tasks.workunit.client.0.smithi151.stdout:4/165: creat d0/d11/f37 x:0 0 0 2024-04-16T02:35:38.985 INFO:tasks.workunit.client.1.smithi153.stdout:4/642: mkdir d3/d9/db/d84/de0/de3 0 2024-04-16T02:35:38.988 INFO:tasks.workunit.client.1.smithi153.stdout:8/493: write d2/d13/d2c/f48 [977189,5208] 0 2024-04-16T02:35:38.988 INFO:tasks.workunit.client.1.smithi153.stdout:7/521: dwrite d0/d4/db/d34/d4b/f69 [0,4194304] 0 2024-04-16T02:35:38.989 INFO:tasks.workunit.client.1.smithi153.stdout:2/538: truncate d9/d2b/d5c/d5f/f69 5172389 0 2024-04-16T02:35:38.989 INFO:tasks.workunit.client.1.smithi153.stdout:5/608: rename d8/df/d28/d5a/d66/d99/db9 to d8/d41/dbd 0 2024-04-16T02:35:38.997 INFO:tasks.workunit.client.1.smithi153.stdout:0/655: creat dd/d14/da6/da4/db1/fc9 x:0 0 0 2024-04-16T02:35:38.998 INFO:tasks.workunit.client.1.smithi153.stdout:8/494: rename d2/d13 to d2/d13/d96/da5 22 2024-04-16T02:35:38.999 INFO:tasks.workunit.client.1.smithi153.stdout:9/584: creat d4/d6/db1/fb4 x:0 0 0 2024-04-16T02:35:39.001 INFO:tasks.workunit.client.1.smithi153.stdout:3/531: dwrite d6/d10/d22/d2c/f98 [0,4194304] 0 2024-04-16T02:35:39.003 INFO:tasks.workunit.client.1.smithi153.stdout:4/643: dread d3/d9/db/f65 [0,4194304] 0 2024-04-16T02:35:39.003 INFO:tasks.workunit.client.1.smithi153.stdout:7/522: creat d0/d4/db/d34/d51/fb3 x:0 0 0 2024-04-16T02:35:39.004 INFO:tasks.workunit.client.1.smithi153.stdout:5/609: mknod d8/d41/d90/d9d/cbe 0 2024-04-16T02:35:39.009 INFO:tasks.workunit.client.1.smithi153.stdout:8/495: mknod d2/d1c/ca6 0 2024-04-16T02:35:39.010 INFO:tasks.workunit.client.1.smithi153.stdout:8/496: write d2/d93/d97/d9c/d27/d2d/d47/d60/f79 [1823993,64210] 0 2024-04-16T02:35:39.011 INFO:tasks.workunit.client.0.smithi151.stdout:1/158: dwrite d3/f2f [0,4194304] 0 2024-04-16T02:35:39.014 INFO:tasks.workunit.client.1.smithi153.stdout:4/644: mkdir d3/d9/db/d17/d28/dc8/de4 0 2024-04-16T02:35:39.016 INFO:tasks.workunit.client.0.smithi151.stdout:1/159: dread d3/d4/db/d1b/d1c/f2c [0,4194304] 0 2024-04-16T02:35:39.019 INFO:tasks.workunit.client.0.smithi151.stdout:1/160: write d3/d4/db/d16/f12 [7576772,73304] 0 2024-04-16T02:35:39.019 INFO:tasks.workunit.client.1.smithi153.stdout:8/497: symlink d2/d93/d97/d9c/d27/d5e/la7 0 2024-04-16T02:35:39.021 INFO:tasks.workunit.client.0.smithi151.stdout:1/161: stat d3/d4/l13 0 2024-04-16T02:35:39.023 INFO:tasks.workunit.client.1.smithi153.stdout:2/539: getdents d9/d24/d27/d9a 0 2024-04-16T02:35:39.029 INFO:tasks.workunit.client.1.smithi153.stdout:1/583: dread d4/f22 [0,4194304] 0 2024-04-16T02:35:39.030 INFO:tasks.workunit.client.0.smithi151.stdout:6/96: sync 2024-04-16T02:35:39.030 INFO:tasks.workunit.client.0.smithi151.stdout:7/106: sync 2024-04-16T02:35:39.030 INFO:tasks.workunit.client.0.smithi151.stdout:3/122: sync 2024-04-16T02:35:39.030 INFO:tasks.workunit.client.0.smithi151.stdout:5/200: sync 2024-04-16T02:35:39.030 INFO:tasks.workunit.client.0.smithi151.stdout:4/166: dwrite d0/d11/d19/f1b [0,4194304] 0 2024-04-16T02:35:39.030 INFO:tasks.workunit.client.1.smithi153.stdout:2/540: chown d9/d24/d34/d60/d7c/la0 5519031 1 2024-04-16T02:35:39.031 INFO:tasks.workunit.client.1.smithi153.stdout:2/541: readlink d9/d24/d34/d36/l9d 0 2024-04-16T02:35:39.032 INFO:tasks.workunit.client.0.smithi151.stdout:5/201: write d2/d6/f32 [1036184,104928] 0 2024-04-16T02:35:39.033 INFO:tasks.workunit.client.1.smithi153.stdout:5/610: getdents d8/d41/dbd 0 2024-04-16T02:35:39.034 INFO:tasks.workunit.client.0.smithi151.stdout:7/107: rmdir d0/d19 39 2024-04-16T02:35:39.034 INFO:tasks.workunit.client.0.smithi151.stdout:5/202: dread - d2/d6/f35 zero size 2024-04-16T02:35:39.037 INFO:tasks.workunit.client.1.smithi153.stdout:1/584: dwrite d4/dd/d13/d26/f92 [0,4194304] 0 2024-04-16T02:35:39.038 INFO:tasks.workunit.client.1.smithi153.stdout:8/498: dread d2/de/f5f [0,4194304] 0 2024-04-16T02:35:39.039 INFO:tasks.workunit.client.0.smithi151.stdout:8/89: sync 2024-04-16T02:35:39.046 INFO:tasks.workunit.client.1.smithi153.stdout:0/656: dread dd/d14/da6/f3e [0,4194304] 0 2024-04-16T02:35:39.046 INFO:tasks.workunit.client.0.smithi151.stdout:1/162: truncate d3/d4/db/d16/f12 2288131 0 2024-04-16T02:35:39.046 INFO:tasks.workunit.client.0.smithi151.stdout:3/123: dwrite d1/d6/f1e [0,4194304] 0 2024-04-16T02:35:39.047 INFO:tasks.workunit.client.1.smithi153.stdout:8/499: symlink d2/la8 0 2024-04-16T02:35:39.047 INFO:tasks.workunit.client.1.smithi153.stdout:9/585: write d4/d6/dc/d24/f43 [1439832,127075] 0 2024-04-16T02:35:39.049 INFO:tasks.workunit.client.1.smithi153.stdout:1/585: getdents d4/dd/d18/d6c/daf 0 2024-04-16T02:35:39.049 INFO:tasks.workunit.client.1.smithi153.stdout:9/586: write d4/d6/db1/fb4 [727597,84228] 0 2024-04-16T02:35:39.050 INFO:tasks.workunit.client.1.smithi153.stdout:7/523: write d0/d4/db/d2b/d43/d6d/f6f [1237849,14354] 0 2024-04-16T02:35:39.051 INFO:tasks.workunit.client.1.smithi153.stdout:5/611: link d8/df/fba d8/df/fbf 0 2024-04-16T02:35:39.051 INFO:tasks.workunit.client.1.smithi153.stdout:9/587: chown d4/d33/l45 2281 1 2024-04-16T02:35:39.051 INFO:tasks.workunit.client.1.smithi153.stdout:1/586: chown d4/dd/d13/d23/db6 127445 1 2024-04-16T02:35:39.052 INFO:tasks.workunit.client.1.smithi153.stdout:3/532: dread d6/f2d [0,4194304] 0 2024-04-16T02:35:39.084 INFO:tasks.workunit.client.1.smithi153.stdout:8/500: rmdir d2/d93/d97/d9c/d27/d2d/d47 39 2024-04-16T02:35:39.087 INFO:tasks.workunit.client.0.smithi151.stdout:7/108: rename d0/d19/l1b to d0/l1c 0 2024-04-16T02:35:39.088 INFO:tasks.workunit.client.0.smithi151.stdout:4/167: rename d0/l1c to d0/l38 0 2024-04-16T02:35:39.089 INFO:tasks.workunit.client.1.smithi153.stdout:5/612: mkdir d8/df/d28/d5a/d72/dc0 0 2024-04-16T02:35:39.090 INFO:tasks.workunit.client.1.smithi153.stdout:0/657: mkdir dd/d14/dca 0 2024-04-16T02:35:39.091 INFO:tasks.workunit.client.1.smithi153.stdout:0/658: write dd/d28/f4c [2827647,1866] 0 2024-04-16T02:35:39.091 INFO:tasks.workunit.client.0.smithi151.stdout:5/203: dwrite d2/f16 [0,4194304] 0 2024-04-16T02:35:39.091 INFO:tasks.workunit.client.0.smithi151.stdout:6/97: truncate d2/d5/fc 2629707 0 2024-04-16T02:35:39.092 INFO:tasks.workunit.client.1.smithi153.stdout:0/659: chown dd/d32/d57/f61 1068235442 1 2024-04-16T02:35:39.092 INFO:tasks.workunit.client.1.smithi153.stdout:9/588: creat d4/d6/dc/d2d/da0/fb5 x:0 0 0 2024-04-16T02:35:39.094 INFO:tasks.workunit.client.1.smithi153.stdout:8/501: write d2/de/f15 [2945453,78978] 0 2024-04-16T02:35:39.098 INFO:tasks.workunit.client.1.smithi153.stdout:9/589: dread d4/d6/dc/d46/f55 [0,4194304] 0 2024-04-16T02:35:39.100 INFO:tasks.workunit.client.1.smithi153.stdout:5/613: rename d8/df/d23/d70/f9c to d8/d41/dbd/d83/fc1 0 2024-04-16T02:35:39.100 INFO:tasks.workunit.client.1.smithi153.stdout:0/660: rmdir dd/d14/d19/d77/d87 39 2024-04-16T02:35:39.101 INFO:tasks.workunit.client.1.smithi153.stdout:0/661: readlink dd/d14/da6/l56 0 2024-04-16T02:35:39.102 INFO:tasks.workunit.client.0.smithi151.stdout:1/163: dwrite d3/d4/db/d1b/d2a/f30 [0,4194304] 0 2024-04-16T02:35:39.103 INFO:tasks.workunit.client.0.smithi151.stdout:4/168: fdatasync d0/fd 0 2024-04-16T02:35:39.105 INFO:tasks.workunit.client.0.smithi151.stdout:5/204: mknod d2/d6/c3d 0 2024-04-16T02:35:39.109 INFO:tasks.workunit.client.1.smithi153.stdout:8/502: write d2/de/f3e [1486400,117411] 0 2024-04-16T02:35:39.110 INFO:tasks.workunit.client.1.smithi153.stdout:9/590: symlink d4/d6/dc/d2d/da0/d40/d4a/lb6 0 2024-04-16T02:35:39.111 INFO:tasks.workunit.client.1.smithi153.stdout:5/614: stat d8/d1f/f21 0 2024-04-16T02:35:39.113 INFO:tasks.workunit.client.0.smithi151.stdout:3/124: dwrite d1/d6/f1e [0,4194304] 0 2024-04-16T02:35:39.115 INFO:tasks.workunit.client.1.smithi153.stdout:8/503: dwrite d2/de/d3f/f91 [0,4194304] 0 2024-04-16T02:35:39.115 INFO:tasks.workunit.client.0.smithi151.stdout:7/109: dread d0/f3 [0,4194304] 0 2024-04-16T02:35:39.117 INFO:tasks.workunit.client.1.smithi153.stdout:8/504: write d2/d93/d97/d9c/d27/d5e/fa3 [288729,64305] 0 2024-04-16T02:35:39.118 INFO:tasks.workunit.client.1.smithi153.stdout:9/591: creat d4/d6/dc/d2d/da0/d69/fb7 x:0 0 0 2024-04-16T02:35:39.120 INFO:tasks.workunit.client.0.smithi151.stdout:6/98: dwrite d2/d5/f8 [4194304,4194304] 0 2024-04-16T02:35:39.121 INFO:tasks.workunit.client.0.smithi151.stdout:3/125: write d1/f17 [711774,25769] 0 2024-04-16T02:35:39.122 INFO:tasks.workunit.client.1.smithi153.stdout:0/662: creat dd/d26/fcb x:0 0 0 2024-04-16T02:35:39.126 INFO:tasks.workunit.client.0.smithi151.stdout:7/110: write d0/f3 [7219116,35556] 0 2024-04-16T02:35:39.128 INFO:tasks.workunit.client.0.smithi151.stdout:6/99: symlink d2/l1c 0 2024-04-16T02:35:39.131 INFO:tasks.workunit.client.1.smithi153.stdout:8/505: dwrite d2/f9b [0,4194304] 0 2024-04-16T02:35:39.132 INFO:tasks.workunit.client.0.smithi151.stdout:8/90: dread d11/f18 [0,4194304] 0 2024-04-16T02:35:39.132 INFO:tasks.workunit.client.0.smithi151.stdout:4/169: dwrite d0/d11/d19/f25 [0,4194304] 0 2024-04-16T02:35:39.132 INFO:tasks.workunit.client.1.smithi153.stdout:6/539: sync 2024-04-16T02:35:39.134 INFO:tasks.workunit.client.1.smithi153.stdout:9/592: symlink d4/d6/dc/d24/d4e/lb8 0 2024-04-16T02:35:39.134 INFO:tasks.workunit.client.0.smithi151.stdout:1/164: dread d3/d4/db/d16/f12 [0,4194304] 0 2024-04-16T02:35:39.136 INFO:tasks.workunit.client.1.smithi153.stdout:0/663: dwrite dd/d14/d19/f27 [0,4194304] 0 2024-04-16T02:35:39.137 INFO:tasks.workunit.client.0.smithi151.stdout:5/205: dwrite d2/d6/d19/d30/f3a [0,4194304] 0 2024-04-16T02:35:39.140 INFO:tasks.workunit.client.0.smithi151.stdout:7/111: creat d0/f1d x:0 0 0 2024-04-16T02:35:39.141 INFO:tasks.workunit.client.0.smithi151.stdout:8/91: read d11/f1c [254641,59077] 0 2024-04-16T02:35:39.141 INFO:tasks.workunit.client.1.smithi153.stdout:3/533: write d6/d10/d22/d2c/f58 [4194761,87662] 0 2024-04-16T02:35:39.141 INFO:tasks.workunit.client.1.smithi153.stdout:5/615: rmdir d8/df/d28/d5a/d66/d99 0 2024-04-16T02:35:39.147 INFO:tasks.workunit.client.0.smithi151.stdout:5/206: write d2/f16 [1353771,90607] 0 2024-04-16T02:35:39.147 INFO:tasks.workunit.client.1.smithi153.stdout:6/540: symlink dc/d6b/lb5 0 2024-04-16T02:35:39.148 INFO:tasks.workunit.client.0.smithi151.stdout:7/112: write d0/fb [245692,120546] 0 2024-04-16T02:35:39.149 INFO:tasks.workunit.client.0.smithi151.stdout:6/100: dwrite d2/f1b [0,4194304] 0 2024-04-16T02:35:39.149 INFO:tasks.workunit.client.0.smithi151.stdout:5/207: chown d2/f16 377869 1 2024-04-16T02:35:39.150 INFO:tasks.workunit.client.0.smithi151.stdout:7/113: readlink d0/l8 0 2024-04-16T02:35:39.150 INFO:tasks.workunit.client.0.smithi151.stdout:8/92: mknod d11/c1d 0 2024-04-16T02:35:39.152 INFO:tasks.workunit.client.1.smithi153.stdout:3/534: dwrite d6/d10/d22/f2a [0,4194304] 0 2024-04-16T02:35:39.154 INFO:tasks.workunit.client.1.smithi153.stdout:0/664: symlink dd/d14/da6/lcc 0 2024-04-16T02:35:39.155 INFO:tasks.workunit.client.0.smithi151.stdout:1/165: rename d3/d4/db/d16 to d3/d33 0 2024-04-16T02:35:39.161 INFO:tasks.workunit.client.0.smithi151.stdout:5/208: creat d2/d6/d19/d30/d36/f3e x:0 0 0 2024-04-16T02:35:39.165 INFO:tasks.workunit.client.0.smithi151.stdout:7/114: dwrite d0/f1d [0,4194304] 0 2024-04-16T02:35:39.168 INFO:tasks.workunit.client.0.smithi151.stdout:8/93: rename l10 to d11/l1e 0 2024-04-16T02:35:39.169 INFO:tasks.workunit.client.0.smithi151.stdout:1/166: unlink d3/f8 0 2024-04-16T02:35:39.169 INFO:tasks.workunit.client.1.smithi153.stdout:6/541: unlink dc/d59/d63/cb1 0 2024-04-16T02:35:39.170 INFO:tasks.workunit.client.1.smithi153.stdout:6/542: stat dc/d47/d57/d77/d15/c48 0 2024-04-16T02:35:39.171 INFO:tasks.workunit.client.1.smithi153.stdout:6/543: write dc/d47/d57/d77/d15/f74 [2443372,34048] 0 2024-04-16T02:35:39.172 INFO:tasks.workunit.client.0.smithi151.stdout:7/115: dread d0/fb [0,4194304] 0 2024-04-16T02:35:39.173 INFO:tasks.workunit.client.1.smithi153.stdout:5/616: getdents d8 0 2024-04-16T02:35:39.174 INFO:tasks.workunit.client.1.smithi153.stdout:5/617: write d8/d41/fb3 [1034984,99790] 0 2024-04-16T02:35:39.175 INFO:tasks.workunit.client.1.smithi153.stdout:5/618: chown d8/d1f/l7f 27870 1 2024-04-16T02:35:39.175 INFO:tasks.workunit.client.0.smithi151.stdout:1/167: mknod d3/d4/db/d1b/d2a/c34 0 2024-04-16T02:35:39.176 INFO:tasks.workunit.client.1.smithi153.stdout:3/535: creat d6/d10/d22/d2c/fba x:0 0 0 2024-04-16T02:35:39.178 INFO:tasks.workunit.client.0.smithi151.stdout:7/116: dread d0/f5 [0,4194304] 0 2024-04-16T02:35:39.182 INFO:tasks.workunit.client.0.smithi151.stdout:7/117: stat d0/c15 0 2024-04-16T02:35:39.182 INFO:tasks.workunit.client.0.smithi151.stdout:7/118: stat d0/l8 0 2024-04-16T02:35:39.183 INFO:tasks.workunit.client.0.smithi151.stdout:1/168: unlink d3/d4/db/d1b/l2d 0 2024-04-16T02:35:39.183 INFO:tasks.workunit.client.1.smithi153.stdout:0/665: getdents dd/d14/d19 0 2024-04-16T02:35:39.183 INFO:tasks.workunit.client.1.smithi153.stdout:8/506: read d2/f23 [4083826,62270] 0 2024-04-16T02:35:39.183 INFO:tasks.workunit.client.1.smithi153.stdout:6/544: creat dc/d47/d7d/fb6 x:0 0 0 2024-04-16T02:35:39.183 INFO:tasks.workunit.client.1.smithi153.stdout:3/536: mkdir d6/d10/d22/d4d/dbb 0 2024-04-16T02:35:39.184 INFO:tasks.workunit.client.1.smithi153.stdout:8/507: fsync d2/d13/d3a/f4c 0 2024-04-16T02:35:39.184 INFO:tasks.workunit.client.0.smithi151.stdout:6/101: getdents d2/d13 0 2024-04-16T02:35:39.184 INFO:tasks.workunit.client.0.smithi151.stdout:7/119: dread d0/f6 [4194304,4194304] 0 2024-04-16T02:35:39.186 INFO:tasks.workunit.client.0.smithi151.stdout:7/120: chown d0 235 1 2024-04-16T02:35:39.186 INFO:tasks.workunit.client.1.smithi153.stdout:8/508: chown d2/de/l22 12088 1 2024-04-16T02:35:39.186 INFO:tasks.workunit.client.1.smithi153.stdout:6/545: mkdir dc/d47/d57/d77/d7a/d97/db7 0 2024-04-16T02:35:39.187 INFO:tasks.workunit.client.0.smithi151.stdout:6/102: getdents d2/d13 0 2024-04-16T02:35:39.187 INFO:tasks.workunit.client.1.smithi153.stdout:0/666: dwrite dd/d32/d57/f61 [0,4194304] 0 2024-04-16T02:35:39.188 INFO:tasks.workunit.client.1.smithi153.stdout:6/546: chown dc/d47/d57/d77/c1d 1284378265 1 2024-04-16T02:35:39.190 INFO:tasks.workunit.client.1.smithi153.stdout:3/537: rename d6/d10/d22/d2e/c56 to d6/d10/d3b/d76/cbc 0 2024-04-16T02:35:39.192 INFO:tasks.workunit.client.0.smithi151.stdout:7/121: write d0/f6 [6380723,128640] 0 2024-04-16T02:35:39.195 INFO:tasks.workunit.client.0.smithi151.stdout:1/169: dwrite d3/f2f [0,4194304] 0 2024-04-16T02:35:39.197 INFO:tasks.workunit.client.1.smithi153.stdout:7/524: dread d0/d13/f32 [0,4194304] 0 2024-04-16T02:35:39.199 INFO:tasks.workunit.client.1.smithi153.stdout:8/509: link d2/d13/d2c/f8d d2/d93/d97/fa9 0 2024-04-16T02:35:39.199 INFO:tasks.workunit.client.0.smithi151.stdout:6/103: rename d2/d5/l7 to d2/l1d 0 2024-04-16T02:35:39.202 INFO:tasks.workunit.client.0.smithi151.stdout:1/170: symlink d3/d4/db/df/l35 0 2024-04-16T02:35:39.204 INFO:tasks.workunit.client.1.smithi153.stdout:3/538: dwrite d6/f11 [0,4194304] 0 2024-04-16T02:35:39.205 INFO:tasks.workunit.client.1.smithi153.stdout:8/510: creat d2/d92/faa x:0 0 0 2024-04-16T02:35:39.208 INFO:tasks.workunit.client.0.smithi151.stdout:6/104: symlink d2/d5/l1e 0 2024-04-16T02:35:39.210 INFO:tasks.workunit.client.0.smithi151.stdout:6/105: write d2/d5/f8 [12188866,64943] 0 2024-04-16T02:35:39.212 INFO:tasks.workunit.client.0.smithi151.stdout:1/171: dwrite d3/d33/f12 [0,4194304] 0 2024-04-16T02:35:39.213 INFO:tasks.workunit.client.1.smithi153.stdout:9/593: truncate d4/d6/d1b/d6e/f74 4224517 0 2024-04-16T02:35:39.214 INFO:tasks.workunit.client.1.smithi153.stdout:3/539: creat d6/d10/d22/d4d/fbd x:0 0 0 2024-04-16T02:35:39.215 INFO:tasks.workunit.client.1.smithi153.stdout:7/525: dwrite d0/d4/db/d2b/d43/f99 [0,4194304] 0 2024-04-16T02:35:39.215 INFO:tasks.workunit.client.1.smithi153.stdout:9/594: fdatasync d4/d6/dc/d7f/fa3 0 2024-04-16T02:35:39.216 INFO:tasks.workunit.client.0.smithi151.stdout:8/94: dread fe [0,4194304] 0 2024-04-16T02:35:39.216 INFO:tasks.workunit.client.0.smithi151.stdout:6/106: mkdir d2/d1f 0 2024-04-16T02:35:39.217 INFO:tasks.workunit.client.1.smithi153.stdout:7/526: dread - d0/d4/db/d2b/d53/f5d zero size 2024-04-16T02:35:39.217 INFO:tasks.workunit.client.1.smithi153.stdout:8/511: creat d2/d13/d96/fab x:0 0 0 2024-04-16T02:35:39.218 INFO:tasks.workunit.client.1.smithi153.stdout:8/512: stat d2/l10 0 2024-04-16T02:35:39.220 INFO:tasks.workunit.client.0.smithi151.stdout:6/107: mkdir d2/d5/df/d20 0 2024-04-16T02:35:39.221 INFO:tasks.workunit.client.0.smithi151.stdout:8/95: creat d11/f1f x:0 0 0 2024-04-16T02:35:39.222 INFO:tasks.workunit.client.1.smithi153.stdout:9/595: dread - d4/d6/dc/d2d/fa8 zero size 2024-04-16T02:35:39.223 INFO:tasks.workunit.client.0.smithi151.stdout:1/172: creat d3/d33/f36 x:0 0 0 2024-04-16T02:35:39.226 INFO:tasks.workunit.client.0.smithi151.stdout:8/96: mknod d11/c20 0 2024-04-16T02:35:39.226 INFO:tasks.workunit.client.0.smithi151.stdout:8/97: read fe [593157,104421] 0 2024-04-16T02:35:39.227 INFO:tasks.workunit.client.0.smithi151.stdout:8/98: fsync d11/f1b 0 2024-04-16T02:35:39.228 INFO:tasks.workunit.client.0.smithi151.stdout:6/108: truncate d2/d5/fb 3089624 0 2024-04-16T02:35:39.228 INFO:tasks.workunit.client.1.smithi153.stdout:8/513: write d2/d93/d97/d9c/d27/d2d/f59 [3451088,73886] 0 2024-04-16T02:35:39.229 INFO:tasks.workunit.client.0.smithi151.stdout:6/109: stat d2/d5/df 0 2024-04-16T02:35:39.229 INFO:tasks.workunit.client.1.smithi153.stdout:7/527: rmdir d0/d4/db/d2b/d43/d6d 39 2024-04-16T02:35:39.232 INFO:tasks.workunit.client.0.smithi151.stdout:1/173: rename d3/d33/f12 to d3/d4/db/df/d32/f37 0 2024-04-16T02:35:39.233 INFO:tasks.workunit.client.0.smithi151.stdout:5/209: write d2/fb [1889353,98352] 0 2024-04-16T02:35:39.233 INFO:tasks.workunit.client.1.smithi153.stdout:3/540: link d6/d10/f32 d6/d10/d3b/d59/fbe 0 2024-04-16T02:35:39.235 INFO:tasks.workunit.client.1.smithi153.stdout:0/667: dread dd/d14/da6/d73/f8f [0,4194304] 0 2024-04-16T02:35:39.235 INFO:tasks.workunit.client.1.smithi153.stdout:3/541: write d6/d10/d22/f93 [677397,88413] 0 2024-04-16T02:35:39.237 INFO:tasks.workunit.client.1.smithi153.stdout:9/596: rename d4/d94 to d4/d6/dc/d2d/d64/db9 0 2024-04-16T02:35:39.237 INFO:tasks.workunit.client.1.smithi153.stdout:8/514: readlink d2/ld 0 2024-04-16T02:35:39.238 INFO:tasks.workunit.client.0.smithi151.stdout:5/210: rmdir d2/d6/d19 39 2024-04-16T02:35:39.239 INFO:tasks.workunit.client.1.smithi153.stdout:7/528: rmdir d0/d4/db/d47/d8d 39 2024-04-16T02:35:39.240 INFO:tasks.workunit.client.1.smithi153.stdout:7/529: stat d0/d4/db/d2b/d53/f5d 0 2024-04-16T02:35:39.241 INFO:tasks.workunit.client.1.smithi153.stdout:3/542: creat d6/d10/d22/d4d/fbf x:0 0 0 2024-04-16T02:35:39.241 INFO:tasks.workunit.client.1.smithi153.stdout:7/530: fsync d0/d4/db/f50 0 2024-04-16T02:35:39.241 INFO:tasks.workunit.client.0.smithi151.stdout:8/99: dwrite d11/f1c [0,4194304] 0 2024-04-16T02:35:39.242 INFO:tasks.workunit.client.1.smithi153.stdout:3/543: fdatasync d6/d10/d22/f31 0 2024-04-16T02:35:39.243 INFO:tasks.workunit.client.0.smithi151.stdout:1/174: dread d3/d33/d10/f18 [0,4194304] 0 2024-04-16T02:35:39.244 INFO:tasks.workunit.client.1.smithi153.stdout:0/668: mkdir dd/d14/dca/dcd 0 2024-04-16T02:35:39.245 INFO:tasks.workunit.client.0.smithi151.stdout:5/211: creat d2/d6/d19/d30/f3f x:0 0 0 2024-04-16T02:35:39.245 INFO:tasks.workunit.client.0.smithi151.stdout:1/175: mkdir d3/d4/db/df/d38 0 2024-04-16T02:35:39.247 INFO:tasks.workunit.client.0.smithi151.stdout:5/212: creat d2/d6/d19/d1a/f40 x:0 0 0 2024-04-16T02:35:39.251 INFO:tasks.workunit.client.0.smithi151.stdout:5/213: rename d2/f37 to d2/d6/d19/d1a/f41 0 2024-04-16T02:35:39.252 INFO:tasks.workunit.client.1.smithi153.stdout:9/597: creat d4/d33/fba x:0 0 0 2024-04-16T02:35:39.253 INFO:tasks.workunit.client.0.smithi151.stdout:1/176: creat d3/d33/f39 x:0 0 0 2024-04-16T02:35:39.254 INFO:tasks.workunit.client.1.smithi153.stdout:7/531: creat d0/d4/db/d2b/d43/d6d/d68/d72/fb4 x:0 0 0 2024-04-16T02:35:39.254 INFO:tasks.workunit.client.0.smithi151.stdout:1/177: write d3/d33/f36 [196544,72446] 0 2024-04-16T02:35:39.256 INFO:tasks.workunit.client.0.smithi151.stdout:5/214: mkdir d2/d6/d19/d30/d36/d42 0 2024-04-16T02:35:39.256 INFO:tasks.workunit.client.0.smithi151.stdout:1/178: write d3/d33/f36 [291072,44349] 0 2024-04-16T02:35:39.256 INFO:tasks.workunit.client.1.smithi153.stdout:9/598: chown d4/d6/dc/d46/f49 33 1 2024-04-16T02:35:39.259 INFO:tasks.workunit.client.1.smithi153.stdout:6/547: dread dc/d47/d57/d77/f85 [0,4194304] 0 2024-04-16T02:35:39.260 INFO:tasks.workunit.client.1.smithi153.stdout:7/532: mkdir d0/d4/db/d2b/d8c/db5 0 2024-04-16T02:35:39.261 INFO:tasks.workunit.client.1.smithi153.stdout:9/599: mkdir d4/d6/dc/d2d/d79/dbb 0 2024-04-16T02:35:39.265 INFO:tasks.workunit.client.1.smithi153.stdout:7/533: write d0/d4/db/d34/d4b/f75 [1491957,39398] 0 2024-04-16T02:35:39.265 INFO:tasks.workunit.client.0.smithi151.stdout:7/122: truncate d0/f3 1497033 0 2024-04-16T02:35:39.266 INFO:tasks.workunit.client.1.smithi153.stdout:0/669: write dd/d14/d19/f1b [5173743,59549] 0 2024-04-16T02:35:39.267 INFO:tasks.workunit.client.0.smithi151.stdout:7/123: write d0/f7 [3119902,109509] 0 2024-04-16T02:35:39.267 INFO:tasks.workunit.client.1.smithi153.stdout:6/548: unlink dc/d10/d62/la8 0 2024-04-16T02:35:39.269 INFO:tasks.workunit.client.0.smithi151.stdout:7/124: fsync d0/fb 0 2024-04-16T02:35:39.270 INFO:tasks.workunit.client.1.smithi153.stdout:8/515: dwrite d2/d13/d3a/d5b/f9d [0,4194304] 0 2024-04-16T02:35:39.270 INFO:tasks.workunit.client.0.smithi151.stdout:7/125: fdatasync d0/f1d 0 2024-04-16T02:35:39.271 INFO:tasks.workunit.client.1.smithi153.stdout:9/600: unlink d4/d6/dc/d24/d5d/fab 0 2024-04-16T02:35:39.273 INFO:tasks.workunit.client.1.smithi153.stdout:7/534: mknod d0/cb6 0 2024-04-16T02:35:39.273 INFO:tasks.workunit.client.0.smithi151.stdout:7/126: dread d0/f14 [0,4194304] 0 2024-04-16T02:35:39.273 INFO:tasks.workunit.client.1.smithi153.stdout:9/601: chown d4/d6/db1/fb4 131276 1 2024-04-16T02:35:39.274 INFO:tasks.workunit.client.1.smithi153.stdout:4/645: sync 2024-04-16T02:35:39.275 INFO:tasks.workunit.client.0.smithi151.stdout:7/127: dread d0/fb [0,4194304] 0 2024-04-16T02:35:39.277 INFO:tasks.workunit.client.1.smithi153.stdout:8/516: readlink d2/d13/d3a/l76 0 2024-04-16T02:35:39.279 INFO:tasks.workunit.client.0.smithi151.stdout:7/128: creat d0/d19/f1e x:0 0 0 2024-04-16T02:35:39.280 INFO:tasks.workunit.client.1.smithi153.stdout:7/535: unlink d0/d4/db/d16/c83 0 2024-04-16T02:35:39.283 INFO:tasks.workunit.client.1.smithi153.stdout:6/549: mkdir dc/d47/d57/d77/db8 0 2024-04-16T02:35:39.283 INFO:tasks.workunit.client.0.smithi151.stdout:1/179: write f1 [1207401,128037] 0 2024-04-16T02:35:39.284 INFO:tasks.workunit.client.1.smithi153.stdout:3/544: dread d6/f20 [0,4194304] 0 2024-04-16T02:35:39.285 INFO:tasks.workunit.client.0.smithi151.stdout:6/110: truncate d2/d5/fb 1997991 0 2024-04-16T02:35:39.285 INFO:tasks.workunit.client.1.smithi153.stdout:9/602: dwrite d4/d6/f22 [0,4194304] 0 2024-04-16T02:35:39.286 INFO:tasks.workunit.client.0.smithi151.stdout:6/111: truncate d2/f9 507446 0 2024-04-16T02:35:39.287 INFO:tasks.workunit.client.1.smithi153.stdout:5/619: dread d8/d1f/f3f [0,4194304] 0 2024-04-16T02:35:39.287 INFO:tasks.workunit.client.1.smithi153.stdout:4/646: rename d3/d9/d79/la1 to d3/d9/d24/d35/le5 0 2024-04-16T02:35:39.287 INFO:tasks.workunit.client.1.smithi153.stdout:9/603: fdatasync d4/d6/dc/d24/f43 0 2024-04-16T02:35:39.288 INFO:tasks.workunit.client.0.smithi151.stdout:7/129: dwrite d0/f11 [0,4194304] 0 2024-04-16T02:35:39.291 INFO:tasks.workunit.client.1.smithi153.stdout:8/517: symlink d2/d13/d3a/lac 0 2024-04-16T02:35:39.291 INFO:tasks.workunit.client.0.smithi151.stdout:7/130: chown d0/f5 188761939 1 2024-04-16T02:35:39.292 INFO:tasks.workunit.client.1.smithi153.stdout:8/518: readlink d2/d13/d3a/d5b/l6d 0 2024-04-16T02:35:39.292 INFO:tasks.workunit.client.1.smithi153.stdout:6/550: creat dc/d47/d57/d77/d15/dac/fb9 x:0 0 0 2024-04-16T02:35:39.293 INFO:tasks.workunit.client.1.smithi153.stdout:8/519: write d2/d92/faa [75828,124345] 0 2024-04-16T02:35:39.294 INFO:tasks.workunit.client.0.smithi151.stdout:7/131: fdatasync d0/fb 0 2024-04-16T02:35:39.294 INFO:tasks.workunit.client.1.smithi153.stdout:8/520: write d2/d93/d97/d9c/d27/f2a [1437272,121837] 0 2024-04-16T02:35:39.296 INFO:tasks.workunit.client.1.smithi153.stdout:7/536: mkdir d0/d4/db/d34/d4b/db7 0 2024-04-16T02:35:39.297 INFO:tasks.workunit.client.1.smithi153.stdout:6/551: mkdir dc/d6b/dba 0 2024-04-16T02:35:39.298 INFO:tasks.workunit.client.0.smithi151.stdout:6/112: dwrite d2/f9 [0,4194304] 0 2024-04-16T02:35:39.299 INFO:tasks.workunit.client.1.smithi153.stdout:0/670: dread dd/d14/d19/d77/d87/f91 [0,4194304] 0 2024-04-16T02:35:39.300 INFO:tasks.workunit.client.1.smithi153.stdout:3/545: creat d6/d10/d22/d4d/dbb/fc0 x:0 0 0 2024-04-16T02:35:39.300 INFO:tasks.workunit.client.1.smithi153.stdout:5/620: symlink d8/df/lc2 0 2024-04-16T02:35:39.303 INFO:tasks.workunit.client.1.smithi153.stdout:6/552: readlink dc/d1a/l9b 0 2024-04-16T02:35:39.305 INFO:tasks.workunit.client.1.smithi153.stdout:6/553: fsync dc/d47/d57/d77/d15/f22 0 2024-04-16T02:35:39.308 INFO:tasks.workunit.client.1.smithi153.stdout:4/647: link d3/d9/l19 d3/da5/le6 0 2024-04-16T02:35:39.308 INFO:tasks.workunit.client.1.smithi153.stdout:8/521: mknod d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/da4/cad 0 2024-04-16T02:35:39.308 INFO:tasks.workunit.client.1.smithi153.stdout:9/604: getdents d4/d6/dc/d7f 0 2024-04-16T02:35:39.309 INFO:tasks.workunit.client.1.smithi153.stdout:5/621: dread d8/df/f88 [0,4194304] 0 2024-04-16T02:35:39.312 INFO:tasks.workunit.client.1.smithi153.stdout:6/554: getdents dc/d47/d57/d77/d15 0 2024-04-16T02:35:39.313 INFO:tasks.workunit.client.1.smithi153.stdout:5/622: write d8/d41/dbd/d83/fc1 [23491,44045] 0 2024-04-16T02:35:39.313 INFO:tasks.workunit.client.1.smithi153.stdout:9/605: creat d4/d6/fbc x:0 0 0 2024-04-16T02:35:39.313 INFO:tasks.workunit.client.1.smithi153.stdout:8/522: truncate d2/de/d34/f40 4212297 0 2024-04-16T02:35:39.314 INFO:tasks.workunit.client.1.smithi153.stdout:9/606: stat d4/d6/dc/d7f/fa3 0 2024-04-16T02:35:39.314 INFO:tasks.workunit.client.1.smithi153.stdout:5/623: chown d8/l17 73 1 2024-04-16T02:35:39.315 INFO:tasks.workunit.client.1.smithi153.stdout:6/555: rename dc/d47/l7b to dc/d6b/dba/lbb 0 2024-04-16T02:35:39.315 INFO:tasks.workunit.client.1.smithi153.stdout:9/607: read - d4/d6/dc/d24/d4e/f9c zero size 2024-04-16T02:35:39.316 INFO:tasks.workunit.client.0.smithi151.stdout:7/132: dread d0/f18 [0,4194304] 0 2024-04-16T02:35:39.317 INFO:tasks.workunit.client.1.smithi153.stdout:2/542: sync 2024-04-16T02:35:39.317 INFO:tasks.workunit.client.1.smithi153.stdout:1/587: sync 2024-04-16T02:35:39.318 INFO:tasks.workunit.client.1.smithi153.stdout:9/608: stat d4/d6/dc/d2d/da0/l5c 0 2024-04-16T02:35:39.318 INFO:tasks.workunit.client.1.smithi153.stdout:1/588: dread - d4/dd/d13/d23/d5f/fc0 zero size 2024-04-16T02:35:39.319 INFO:tasks.workunit.client.1.smithi153.stdout:8/523: rename d2/de/c4f to d2/d93/d97/d9c/cae 0 2024-04-16T02:35:39.320 INFO:tasks.workunit.client.0.smithi151.stdout:0/187: sync 2024-04-16T02:35:39.325 INFO:tasks.workunit.client.0.smithi151.stdout:0/188: write dc/f11 [7293434,99940] 0 2024-04-16T02:35:39.325 INFO:tasks.workunit.client.1.smithi153.stdout:1/589: rmdir d4/da2/d65 39 2024-04-16T02:35:39.328 INFO:tasks.workunit.client.1.smithi153.stdout:9/609: dread - d4/d6/dc/d24/d5d/d66/fb2 zero size 2024-04-16T02:35:39.328 INFO:tasks.workunit.client.1.smithi153.stdout:5/624: link d8/df/d2e/c34 d8/df/d2f/cc3 0 2024-04-16T02:35:39.329 INFO:tasks.workunit.client.1.smithi153.stdout:8/524: write d2/f14 [1236680,108965] 0 2024-04-16T02:35:39.330 INFO:tasks.workunit.client.1.smithi153.stdout:5/625: chown d8/df/d23/c2d 111963 1 2024-04-16T02:35:39.331 INFO:tasks.workunit.client.1.smithi153.stdout:1/590: symlink d4/dd/d13/d7e/lc9 0 2024-04-16T02:35:39.334 INFO:tasks.workunit.client.1.smithi153.stdout:5/626: mknod d8/df/d2e/cc4 0 2024-04-16T02:35:39.334 INFO:tasks.workunit.client.1.smithi153.stdout:3/546: dread d6/d10/f18 [0,4194304] 0 2024-04-16T02:35:39.341 INFO:tasks.workunit.client.0.smithi151.stdout:6/113: truncate d2/d5/f8 3010794 0 2024-04-16T02:35:39.342 INFO:tasks.workunit.client.0.smithi151.stdout:7/133: truncate d0/f7 111023 0 2024-04-16T02:35:39.344 INFO:tasks.workunit.client.0.smithi151.stdout:6/114: mknod d2/d5/df/d20/c21 0 2024-04-16T02:35:39.345 INFO:tasks.workunit.client.0.smithi151.stdout:0/189: truncate dc/f39 2248982 0 2024-04-16T02:35:39.350 INFO:tasks.workunit.client.0.smithi151.stdout:0/190: rename dc/l14 to dc/l3a 0 2024-04-16T02:35:39.355 INFO:tasks.workunit.client.0.smithi151.stdout:6/115: dwrite d2/d5/f6 [0,4194304] 0 2024-04-16T02:35:39.356 INFO:tasks.workunit.client.0.smithi151.stdout:6/116: fsync d2/d13/f18 0 2024-04-16T02:35:39.362 INFO:tasks.workunit.client.0.smithi151.stdout:6/117: mkdir d2/d1f/d22 0 2024-04-16T02:35:39.365 INFO:tasks.workunit.client.0.smithi151.stdout:0/191: write f3 [8695106,75762] 0 2024-04-16T02:35:39.366 INFO:tasks.workunit.client.0.smithi151.stdout:0/192: chown c7 1763661 1 2024-04-16T02:35:39.368 INFO:tasks.workunit.client.1.smithi153.stdout:5/627: creat d8/df/d84/fc5 x:0 0 0 2024-04-16T02:35:39.370 INFO:tasks.workunit.client.0.smithi151.stdout:0/193: mkdir dc/d15/d34/d3b 0 2024-04-16T02:35:39.370 INFO:tasks.workunit.client.0.smithi151.stdout:7/134: dwrite d0/f7 [0,4194304] 0 2024-04-16T02:35:39.370 INFO:tasks.workunit.client.1.smithi153.stdout:6/556: getdents dc/d47/d57/d77 0 2024-04-16T02:35:39.370 INFO:tasks.workunit.client.1.smithi153.stdout:4/648: dread d3/d9/db/de/f13 [0,4194304] 0 2024-04-16T02:35:39.376 INFO:tasks.workunit.client.1.smithi153.stdout:5/628: rmdir d8/df/d2e 39 2024-04-16T02:35:39.377 INFO:tasks.workunit.client.0.smithi151.stdout:7/135: rename d0/d19 to d0/d1f 0 2024-04-16T02:35:39.378 INFO:tasks.workunit.client.1.smithi153.stdout:6/557: rename dc/d10/f1b to dc/d10/fbc 0 2024-04-16T02:35:39.378 INFO:tasks.workunit.client.0.smithi151.stdout:6/118: truncate d2/f1a 3189783 0 2024-04-16T02:35:39.378 INFO:tasks.workunit.client.0.smithi151.stdout:7/136: write d0/f7 [2286563,103735] 0 2024-04-16T02:35:39.379 INFO:tasks.workunit.client.0.smithi151.stdout:7/137: write d0/f14 [1110979,100951] 0 2024-04-16T02:35:39.380 INFO:tasks.workunit.client.0.smithi151.stdout:6/119: write d2/d13/f18 [972765,14248] 0 2024-04-16T02:35:39.382 INFO:tasks.workunit.client.0.smithi151.stdout:0/194: write dc/d15/f2b [763694,774] 0 2024-04-16T02:35:39.385 INFO:tasks.workunit.client.0.smithi151.stdout:0/195: fsync dc/ff 0 2024-04-16T02:35:39.385 INFO:tasks.workunit.client.0.smithi151.stdout:6/120: mkdir d2/d5/df/d23 0 2024-04-16T02:35:39.385 INFO:tasks.workunit.client.1.smithi153.stdout:0/671: dwrite dd/d14/d19/d77/d87/f91 [0,4194304] 0 2024-04-16T02:35:39.387 INFO:tasks.workunit.client.0.smithi151.stdout:7/138: symlink d0/l20 0 2024-04-16T02:35:39.387 INFO:tasks.workunit.client.1.smithi153.stdout:7/537: dwrite d0/d4/db/d2b/f40 [0,4194304] 0 2024-04-16T02:35:39.389 INFO:tasks.workunit.client.0.smithi151.stdout:7/139: write d0/f11 [2271067,73329] 0 2024-04-16T02:35:39.392 INFO:tasks.workunit.client.1.smithi153.stdout:2/543: dwrite d9/d2b/d1c/d20/f67 [0,4194304] 0 2024-04-16T02:35:39.392 INFO:tasks.workunit.client.1.smithi153.stdout:1/591: dwrite d4/dd/d13/f2a [4194304,4194304] 0 2024-04-16T02:35:39.394 INFO:tasks.workunit.client.0.smithi151.stdout:6/121: creat d2/d5/df/d20/f24 x:0 0 0 2024-04-16T02:35:39.396 INFO:tasks.workunit.client.0.smithi151.stdout:0/196: mkdir dc/d3c 0 2024-04-16T02:35:39.396 INFO:tasks.workunit.client.1.smithi153.stdout:2/544: fdatasync d9/d2b/d5c/f80 0 2024-04-16T02:35:39.398 INFO:tasks.workunit.client.1.smithi153.stdout:8/525: dread d2/f17 [0,4194304] 0 2024-04-16T02:35:39.399 INFO:tasks.workunit.client.0.smithi151.stdout:6/122: creat d2/d1f/d22/f25 x:0 0 0 2024-04-16T02:35:39.399 INFO:tasks.workunit.client.1.smithi153.stdout:1/592: dread d4/dd/d20/fa4 [0,4194304] 0 2024-04-16T02:35:39.400 INFO:tasks.workunit.client.0.smithi151.stdout:0/197: truncate dc/d15/f17 3188793 0 2024-04-16T02:35:39.405 INFO:tasks.workunit.client.0.smithi151.stdout:6/123: unlink d2/c12 0 2024-04-16T02:35:39.406 INFO:tasks.workunit.client.0.smithi151.stdout:0/198: creat dc/d20/f3d x:0 0 0 2024-04-16T02:35:39.408 INFO:tasks.workunit.client.0.smithi151.stdout:6/124: creat d2/d1f/f26 x:0 0 0 2024-04-16T02:35:39.409 INFO:tasks.workunit.client.0.smithi151.stdout:0/199: read dc/d15/f2b [737294,102405] 0 2024-04-16T02:35:39.417 INFO:tasks.workunit.client.0.smithi151.stdout:6/125: creat d2/d5/f27 x:0 0 0 2024-04-16T02:35:39.417 INFO:tasks.workunit.client.0.smithi151.stdout:6/126: stat d2/d5/f27 0 2024-04-16T02:35:39.419 INFO:tasks.workunit.client.0.smithi151.stdout:6/127: symlink d2/l28 0 2024-04-16T02:35:39.421 INFO:tasks.workunit.client.0.smithi151.stdout:6/128: chown d2/ld 31677756 1 2024-04-16T02:35:39.425 INFO:tasks.workunit.client.0.smithi151.stdout:6/129: dread d2/f9 [0,4194304] 0 2024-04-16T02:35:39.429 INFO:tasks.workunit.client.0.smithi151.stdout:6/130: dread d2/f1b [0,4194304] 0 2024-04-16T02:35:39.432 INFO:tasks.workunit.client.0.smithi151.stdout:6/131: creat d2/d1f/f29 x:0 0 0 2024-04-16T02:35:39.433 INFO:tasks.workunit.client.0.smithi151.stdout:6/132: dread - d2/d1f/f29 zero size 2024-04-16T02:35:39.438 INFO:tasks.workunit.client.0.smithi151.stdout:6/133: dread d2/d5/f8 [0,4194304] 0 2024-04-16T02:35:39.442 INFO:tasks.workunit.client.0.smithi151.stdout:6/134: dread d2/d5/fb [0,4194304] 0 2024-04-16T02:35:39.443 INFO:tasks.workunit.client.0.smithi151.stdout:6/135: fsync d2/d5/f27 0 2024-04-16T02:35:39.444 INFO:tasks.workunit.client.1.smithi153.stdout:4/649: symlink d3/d9/db/d84/de0/de3/le7 0 2024-04-16T02:35:39.445 INFO:tasks.workunit.client.1.smithi153.stdout:3/547: rename d6/d10/d22/d2e/d8a/d8c/fb0 to d6/d10/d22/d2e/d8a/fc1 0 2024-04-16T02:35:39.449 INFO:tasks.workunit.client.0.smithi151.stdout:6/136: dwrite d2/d13/f18 [0,4194304] 0 2024-04-16T02:35:39.452 INFO:tasks.workunit.client.1.smithi153.stdout:3/548: dwrite d6/d10/d22/d2e/f35 [8388608,4194304] 0 2024-04-16T02:35:39.454 INFO:tasks.workunit.client.1.smithi153.stdout:3/549: fsync d6/d10/d22/fb1 0 2024-04-16T02:35:39.457 INFO:tasks.workunit.client.1.smithi153.stdout:0/672: creat dd/d14/d51/d7d/fce x:0 0 0 2024-04-16T02:35:39.459 INFO:tasks.workunit.client.1.smithi153.stdout:5/629: dwrite d8/d41/dbd/f40 [0,4194304] 0 2024-04-16T02:35:39.461 INFO:tasks.workunit.client.1.smithi153.stdout:9/610: sync 2024-04-16T02:35:39.461 INFO:tasks.workunit.client.0.smithi151.stdout:6/137: write d2/f1a [1802168,105787] 0 2024-04-16T02:35:39.466 INFO:tasks.workunit.client.0.smithi151.stdout:6/138: dread d2/d5/f8 [0,4194304] 0 2024-04-16T02:35:39.466 INFO:tasks.workunit.client.1.smithi153.stdout:5/630: read d8/f30 [205120,21928] 0 2024-04-16T02:35:39.468 INFO:tasks.workunit.client.1.smithi153.stdout:6/558: rename dc/d59/d5b/f83 to dc/d47/d93/da9/fbd 0 2024-04-16T02:35:39.470 INFO:tasks.workunit.client.0.smithi151.stdout:6/139: fsync d2/f1a 0 2024-04-16T02:35:39.471 INFO:tasks.workunit.client.1.smithi153.stdout:7/538: mkdir d0/d4/db/d2b/d43/d6d/d68/d6c/db8 0 2024-04-16T02:35:39.473 INFO:tasks.workunit.client.1.smithi153.stdout:0/673: creat dd/d14/d19/d55/fcf x:0 0 0 2024-04-16T02:35:39.476 INFO:tasks.workunit.client.1.smithi153.stdout:7/539: read d0/d4/db/f39 [4009571,84962] 0 2024-04-16T02:35:39.479 INFO:tasks.workunit.client.0.smithi151.stdout:6/140: dwrite d2/f9 [0,4194304] 0 2024-04-16T02:35:39.481 INFO:tasks.workunit.client.0.smithi151.stdout:6/141: chown d2/d5/df/d20/f24 85705616 1 2024-04-16T02:35:39.482 INFO:tasks.workunit.client.1.smithi153.stdout:4/650: mkdir d3/dbc/de8 0 2024-04-16T02:35:39.482 INFO:tasks.workunit.client.0.smithi151.stdout:6/142: mknod d2/d5/df/d23/c2a 0 2024-04-16T02:35:39.485 INFO:tasks.workunit.client.0.smithi151.stdout:6/143: creat d2/d5/df/d20/f2b x:0 0 0 2024-04-16T02:35:39.490 INFO:tasks.workunit.client.0.smithi151.stdout:6/144: write d2/d5/fb [2980813,127016] 0 2024-04-16T02:35:39.491 INFO:tasks.workunit.client.0.smithi151.stdout:6/145: readlink d2/d5/l1e 0 2024-04-16T02:35:39.492 INFO:tasks.workunit.client.0.smithi151.stdout:6/146: dread - d2/d5/df/d20/f2b zero size 2024-04-16T02:35:39.492 INFO:tasks.workunit.client.0.smithi151.stdout:6/147: chown d2/d1f/d22 435903006 1 2024-04-16T02:35:39.493 INFO:tasks.workunit.client.0.smithi151.stdout:6/148: chown d2/d5 840 1 2024-04-16T02:35:39.494 INFO:tasks.workunit.client.0.smithi151.stdout:6/149: chown d2/d5/f6 1719 1 2024-04-16T02:35:39.496 INFO:tasks.workunit.client.1.smithi153.stdout:0/674: creat dd/d26/d44/fd0 x:0 0 0 2024-04-16T02:35:39.498 INFO:tasks.workunit.client.0.smithi151.stdout:6/150: truncate d2/f1a 3043887 0 2024-04-16T02:35:39.498 INFO:tasks.workunit.client.0.smithi151.stdout:6/151: stat d2/d5/fb 0 2024-04-16T02:35:39.500 INFO:tasks.workunit.client.0.smithi151.stdout:6/152: symlink d2/d1f/l2c 0 2024-04-16T02:35:39.500 INFO:tasks.workunit.client.1.smithi153.stdout:4/651: fsync d3/d9/d24/f4d 0 2024-04-16T02:35:39.502 INFO:tasks.workunit.client.1.smithi153.stdout:4/652: chown d3/d9/db/d17/d28/l48 0 1 2024-04-16T02:35:39.503 INFO:tasks.workunit.client.0.smithi151.stdout:6/153: creat d2/d5/df/f2d x:0 0 0 2024-04-16T02:35:39.503 INFO:tasks.workunit.client.1.smithi153.stdout:5/631: rename d8/df/d23/d70/la7 to d8/df/d28/d5a/d72/dc0/lc6 0 2024-04-16T02:35:39.503 INFO:tasks.workunit.client.1.smithi153.stdout:0/675: dwrite dd/f6f [0,4194304] 0 2024-04-16T02:35:39.504 INFO:tasks.workunit.client.1.smithi153.stdout:5/632: write d8/df/d23/f56 [3859894,112825] 0 2024-04-16T02:35:39.505 INFO:tasks.workunit.client.1.smithi153.stdout:4/653: write d3/d9/db/de/d2c/f56 [309350,33312] 0 2024-04-16T02:35:39.505 INFO:tasks.workunit.client.0.smithi151.stdout:6/154: mkdir d2/d5/d2e 0 2024-04-16T02:35:39.506 INFO:tasks.workunit.client.1.smithi153.stdout:7/540: fdatasync d0/d13/f81 0 2024-04-16T02:35:39.507 INFO:tasks.workunit.client.0.smithi151.stdout:6/155: mknod d2/d5/df/d23/c2f 0 2024-04-16T02:35:39.509 INFO:tasks.workunit.client.0.smithi151.stdout:6/156: fdatasync d2/d1f/f26 0 2024-04-16T02:35:39.511 INFO:tasks.workunit.client.0.smithi151.stdout:6/157: creat d2/f30 x:0 0 0 2024-04-16T02:35:39.512 INFO:tasks.workunit.client.1.smithi153.stdout:6/559: creat dc/d1a/fbe x:0 0 0 2024-04-16T02:35:39.513 INFO:tasks.workunit.client.0.smithi151.stdout:6/158: mkdir d2/d5/df/d23/d31 0 2024-04-16T02:35:39.514 INFO:tasks.workunit.client.1.smithi153.stdout:5/633: write d8/d41/dbd/d83/fae [336809,48631] 0 2024-04-16T02:35:39.515 INFO:tasks.workunit.client.0.smithi151.stdout:6/159: mknod d2/d5/df/d23/c32 0 2024-04-16T02:35:39.516 INFO:tasks.workunit.client.1.smithi153.stdout:7/541: stat d0/l1f 0 2024-04-16T02:35:39.517 INFO:tasks.workunit.client.1.smithi153.stdout:6/560: creat dc/d59/d63/fbf x:0 0 0 2024-04-16T02:35:39.517 INFO:tasks.workunit.client.0.smithi151.stdout:6/160: truncate d2/d5/df/d20/f2b 182139 0 2024-04-16T02:35:39.518 INFO:tasks.workunit.client.0.smithi151.stdout:6/161: stat d2/l1d 0 2024-04-16T02:35:39.519 INFO:tasks.workunit.client.0.smithi151.stdout:6/162: creat d2/f33 x:0 0 0 2024-04-16T02:35:39.521 INFO:tasks.workunit.client.1.smithi153.stdout:6/561: dread dc/d10/f33 [0,4194304] 0 2024-04-16T02:35:39.521 INFO:tasks.workunit.client.0.smithi151.stdout:6/163: creat d2/d5/f34 x:0 0 0 2024-04-16T02:35:39.522 INFO:tasks.workunit.client.0.smithi151.stdout:6/164: read - d2/f30 zero size 2024-04-16T02:35:39.522 INFO:tasks.workunit.client.0.smithi151.stdout:6/165: fdatasync d2/d5/df/d20/f24 0 2024-04-16T02:35:39.522 INFO:tasks.workunit.client.0.smithi151.stdout:6/166: chown d2/d1f 15366620 1 2024-04-16T02:35:39.523 INFO:tasks.workunit.client.1.smithi153.stdout:5/634: unlink d8/df/d2e/c54 0 2024-04-16T02:35:39.524 INFO:tasks.workunit.client.1.smithi153.stdout:5/635: dread - d8/df/d28/d5a/d72/f9e zero size 2024-04-16T02:35:39.524 INFO:tasks.workunit.client.1.smithi153.stdout:7/542: creat d0/d4/db/d34/d4b/fb9 x:0 0 0 2024-04-16T02:35:39.525 INFO:tasks.workunit.client.0.smithi151.stdout:6/167: rename d2/d1f/l2c to d2/d5/df/d23/d31/l35 0 2024-04-16T02:35:39.526 INFO:tasks.workunit.client.1.smithi153.stdout:6/562: mknod dc/d47/d93/da9/cc0 0 2024-04-16T02:35:39.528 INFO:tasks.workunit.client.0.smithi151.stdout:6/168: dread d2/d5/f8 [0,4194304] 0 2024-04-16T02:35:39.531 INFO:tasks.workunit.client.1.smithi153.stdout:6/563: symlink dc/d47/d57/d77/d7a/d97/lc1 0 2024-04-16T02:35:39.532 INFO:tasks.workunit.client.1.smithi153.stdout:7/543: rmdir d0/d4/db/d2b/d8c/d9f 39 2024-04-16T02:35:39.534 INFO:tasks.workunit.client.0.smithi151.stdout:6/169: dwrite d2/d5/df/d20/f2b [0,4194304] 0 2024-04-16T02:35:39.536 INFO:tasks.workunit.client.1.smithi153.stdout:5/636: dread d8/df/d28/f35 [0,4194304] 0 2024-04-16T02:35:39.538 INFO:tasks.workunit.client.1.smithi153.stdout:6/564: dwrite dc/d59/d63/f66 [0,4194304] 0 2024-04-16T02:35:39.541 INFO:tasks.workunit.client.1.smithi153.stdout:2/545: truncate d9/d2b/d18/f29 2812593 0 2024-04-16T02:35:39.543 INFO:tasks.workunit.client.1.smithi153.stdout:1/593: truncate d4/f49 160400 0 2024-04-16T02:35:39.544 INFO:tasks.workunit.client.1.smithi153.stdout:2/546: write d9/d2b/d5c/f75 [3645314,94845] 0 2024-04-16T02:35:39.545 INFO:tasks.workunit.client.1.smithi153.stdout:3/550: truncate d6/d10/d22/d2e/f35 9178654 0 2024-04-16T02:35:39.547 INFO:tasks.workunit.client.1.smithi153.stdout:1/594: rmdir d4/dd/d18/d29/d71 39 2024-04-16T02:35:39.547 INFO:tasks.workunit.client.1.smithi153.stdout:2/547: truncate d9/d2b/d1c/d20/d45/faf 942093 0 2024-04-16T02:35:39.548 INFO:tasks.workunit.client.1.smithi153.stdout:7/544: dread d0/d13/f4f [0,4194304] 0 2024-04-16T02:35:39.550 INFO:tasks.workunit.client.1.smithi153.stdout:7/545: write d0/d4/db/d2b/d53/d59/f67 [231861,88073] 0 2024-04-16T02:35:39.550 INFO:tasks.workunit.client.1.smithi153.stdout:3/551: creat d6/d10/d3b/d76/fc2 x:0 0 0 2024-04-16T02:35:39.552 INFO:tasks.workunit.client.1.smithi153.stdout:8/526: symlink d2/d13/laf 0 2024-04-16T02:35:39.552 INFO:tasks.workunit.client.1.smithi153.stdout:3/552: write d6/d10/d3b/d76/f7d [2104149,75562] 0 2024-04-16T02:35:39.553 INFO:tasks.workunit.client.1.smithi153.stdout:3/553: stat d6/d10/d3b/f7b 0 2024-04-16T02:35:39.553 INFO:tasks.workunit.client.1.smithi153.stdout:9/611: truncate d4/d6/f22 1811193 0 2024-04-16T02:35:39.555 INFO:tasks.workunit.client.1.smithi153.stdout:1/595: unlink d4/dd/d18/f52 0 2024-04-16T02:35:39.557 INFO:tasks.workunit.client.1.smithi153.stdout:8/527: symlink d2/d92/lb0 0 2024-04-16T02:35:39.558 INFO:tasks.workunit.client.1.smithi153.stdout:7/546: creat d0/d4/db/d2b/d8c/db5/fba x:0 0 0 2024-04-16T02:35:39.560 INFO:tasks.workunit.client.1.smithi153.stdout:9/612: dwrite d4/d6/dc/d24/d4e/f9c [0,4194304] 0 2024-04-16T02:35:39.560 INFO:tasks.workunit.client.1.smithi153.stdout:3/554: creat d6/d10/d22/d2e/d8a/fc3 x:0 0 0 2024-04-16T02:35:39.561 INFO:tasks.workunit.client.1.smithi153.stdout:7/547: write d0/d13/f32 [1881133,126619] 0 2024-04-16T02:35:39.593 INFO:tasks.workunit.client.0.smithi151.stdout:6/170: truncate d2/f1a 3799614 0 2024-04-16T02:35:39.594 INFO:tasks.workunit.client.0.smithi151.stdout:6/171: chown d2/d5/l1e 23 1 2024-04-16T02:35:39.595 INFO:tasks.workunit.client.0.smithi151.stdout:6/172: write d2/d5/f6 [1163324,867] 0 2024-04-16T02:35:39.597 INFO:tasks.workunit.client.0.smithi151.stdout:6/173: symlink d2/d5/d2e/l36 0 2024-04-16T02:35:39.599 INFO:tasks.workunit.client.0.smithi151.stdout:6/174: symlink d2/d5/l37 0 2024-04-16T02:35:39.600 INFO:tasks.workunit.client.0.smithi151.stdout:6/175: mkdir d2/d5/df/d38 0 2024-04-16T02:35:39.602 INFO:tasks.workunit.client.0.smithi151.stdout:6/176: symlink d2/d5/l39 0 2024-04-16T02:35:39.602 INFO:tasks.workunit.client.0.smithi151.stdout:6/177: dread - d2/f30 zero size 2024-04-16T02:35:39.606 INFO:tasks.workunit.client.0.smithi151.stdout:6/178: dread d2/d5/fb [0,4194304] 0 2024-04-16T02:35:39.617 INFO:tasks.workunit.client.0.smithi151.stdout:6/179: dread d2/f1a [0,4194304] 0 2024-04-16T02:35:39.619 INFO:tasks.workunit.client.0.smithi151.stdout:6/180: mkdir d2/d13/d3a 0 2024-04-16T02:35:39.622 INFO:tasks.workunit.client.0.smithi151.stdout:6/181: link d2/d5/f27 d2/d5/df/d38/f3b 0 2024-04-16T02:35:39.622 INFO:tasks.workunit.client.0.smithi151.stdout:6/182: chown d2/l4 680031708 1 2024-04-16T02:35:39.622 INFO:tasks.workunit.client.0.smithi151.stdout:6/183: chown d2/d5/f6 337095464 1 2024-04-16T02:35:39.624 INFO:tasks.workunit.client.0.smithi151.stdout:6/184: creat d2/d5/df/f3c x:0 0 0 2024-04-16T02:35:39.628 INFO:tasks.workunit.client.0.smithi151.stdout:6/185: write d2/f1a [2982054,60877] 0 2024-04-16T02:35:39.630 INFO:tasks.workunit.client.0.smithi151.stdout:6/186: mkdir d2/d3d 0 2024-04-16T02:35:39.631 INFO:tasks.workunit.client.1.smithi153.stdout:1/596: getdents d4/dd/d13/d23 0 2024-04-16T02:35:39.631 INFO:tasks.workunit.client.1.smithi153.stdout:3/555: write d6/d10/d22/d43/f77 [40597,34478] 0 2024-04-16T02:35:39.632 INFO:tasks.workunit.client.1.smithi153.stdout:9/613: rmdir d4/d6/dc/d2d/d64/db9 39 2024-04-16T02:35:39.632 INFO:tasks.workunit.client.0.smithi151.stdout:6/187: getdents d2/d5/df/d23/d31 0 2024-04-16T02:35:39.633 INFO:tasks.workunit.client.1.smithi153.stdout:7/548: fsync d0/d4/db/d47/f94 0 2024-04-16T02:35:39.633 INFO:tasks.workunit.client.1.smithi153.stdout:2/548: dread d9/d24/d34/d36/f7b [0,4194304] 0 2024-04-16T02:35:39.633 INFO:tasks.workunit.client.0.smithi151.stdout:6/188: read - d2/d5/f34 zero size 2024-04-16T02:35:39.634 INFO:tasks.workunit.client.1.smithi153.stdout:5/637: write f4 [1837936,94816] 0 2024-04-16T02:35:39.634 INFO:tasks.workunit.client.0.smithi151.stdout:6/189: truncate d2/f30 298370 0 2024-04-16T02:35:39.636 INFO:tasks.workunit.client.1.smithi153.stdout:2/549: fsync d9/d24/d27/d2c/d41/f4c 0 2024-04-16T02:35:39.636 INFO:tasks.workunit.client.1.smithi153.stdout:7/549: chown d0/d4/db/d2b/d53/d59/f67 134082573 1 2024-04-16T02:35:39.636 INFO:tasks.workunit.client.1.smithi153.stdout:2/550: readlink d9/d71/l94 0 2024-04-16T02:35:39.638 INFO:tasks.workunit.client.1.smithi153.stdout:1/597: write d4/dd/d13/d26/f34 [1696043,29988] 0 2024-04-16T02:35:39.638 INFO:tasks.workunit.client.1.smithi153.stdout:6/565: dread dc/d47/d57/d77/d15/f2f [0,4194304] 0 2024-04-16T02:35:39.639 INFO:tasks.workunit.client.0.smithi151.stdout:6/190: rename d2/d5/f34 to d2/d5/f3e 0 2024-04-16T02:35:39.641 INFO:tasks.workunit.client.1.smithi153.stdout:3/556: symlink d6/d10/d22/d43/lc4 0 2024-04-16T02:35:39.644 INFO:tasks.workunit.client.1.smithi153.stdout:6/566: creat dc/d47/d57/d77/d15/fc2 x:0 0 0 2024-04-16T02:35:39.645 INFO:tasks.workunit.client.0.smithi151.stdout:6/191: truncate d2/f1a 3068827 0 2024-04-16T02:35:39.645 INFO:tasks.workunit.client.1.smithi153.stdout:7/550: dread d0/d4/db/d2b/f25 [0,4194304] 0 2024-04-16T02:35:39.645 INFO:tasks.workunit.client.1.smithi153.stdout:3/557: creat d6/d10/d3b/fc5 x:0 0 0 2024-04-16T02:35:39.647 INFO:tasks.workunit.client.0.smithi151.stdout:6/192: mkdir d2/d5/df/d38/d3f 0 2024-04-16T02:35:39.647 INFO:tasks.workunit.client.0.smithi151.stdout:6/193: fdatasync d2/d1f/d22/f25 0 2024-04-16T02:35:39.649 INFO:tasks.workunit.client.0.smithi151.stdout:6/194: readlink d2/d5/df/d23/d31/l35 0 2024-04-16T02:35:39.650 INFO:tasks.workunit.client.1.smithi153.stdout:3/558: dread d6/d10/d22/d2e/f6f [0,4194304] 0 2024-04-16T02:35:39.651 INFO:tasks.workunit.client.1.smithi153.stdout:3/559: stat d6/d10/d3b/f85 0 2024-04-16T02:35:39.652 INFO:tasks.workunit.client.1.smithi153.stdout:7/551: mknod d0/d4/db/d2b/d43/d6d/d68/d6c/cbb 0 2024-04-16T02:35:39.652 INFO:tasks.workunit.client.1.smithi153.stdout:2/551: truncate d9/d2b/d18/d3f/f49 3736574 0 2024-04-16T02:35:39.653 INFO:tasks.workunit.client.1.smithi153.stdout:1/598: write d4/dd/d13/d23/d5f/f61 [259857,90741] 0 2024-04-16T02:35:39.654 INFO:tasks.workunit.client.1.smithi153.stdout:5/638: dwrite d8/df/d28/d5a/d5e/fad [0,4194304] 0 2024-04-16T02:35:39.656 INFO:tasks.workunit.client.0.smithi151.stdout:6/195: rename d2/d5/df/d38 to d2/d1f/d22/d40 0 2024-04-16T02:35:39.656 INFO:tasks.workunit.client.1.smithi153.stdout:5/639: chown d8/d41/dbd/d39/f6b 2005 1 2024-04-16T02:35:39.659 INFO:tasks.workunit.client.1.smithi153.stdout:1/599: symlink d4/dd/d18/d6c/lca 0 2024-04-16T02:35:39.660 INFO:tasks.workunit.client.1.smithi153.stdout:2/552: dread d9/fba [0,4194304] 0 2024-04-16T02:35:39.661 INFO:tasks.workunit.client.1.smithi153.stdout:6/567: write dc/fe [4009931,53380] 0 2024-04-16T02:35:39.664 INFO:tasks.workunit.client.0.smithi151.stdout:6/196: dwrite d2/d1f/f29 [0,4194304] 0 2024-04-16T02:35:39.666 INFO:tasks.workunit.client.0.smithi151.stdout:6/197: truncate d2/d5/df/f2d 986196 0 2024-04-16T02:35:39.666 INFO:tasks.workunit.client.1.smithi153.stdout:5/640: fsync d8/df/d2e/f6e 0 2024-04-16T02:35:39.667 INFO:tasks.workunit.client.0.smithi151.stdout:6/198: dread - d2/d5/f27 zero size 2024-04-16T02:35:39.667 INFO:tasks.workunit.client.1.smithi153.stdout:1/600: write d4/dd/d18/d29/f4a [2805566,53696] 0 2024-04-16T02:35:39.667 INFO:tasks.workunit.client.1.smithi153.stdout:3/560: write d6/d10/d3b/f85 [1836683,4958] 0 2024-04-16T02:35:39.668 INFO:tasks.workunit.client.1.smithi153.stdout:6/568: readlink dc/d47/d57/d77/d15/l79 0 2024-04-16T02:35:39.669 INFO:tasks.workunit.client.1.smithi153.stdout:6/569: chown dc/d1a 5759 1 2024-04-16T02:35:39.672 INFO:tasks.workunit.client.1.smithi153.stdout:2/553: creat d9/d71/d8d/fbd x:0 0 0 2024-04-16T02:35:39.674 INFO:tasks.workunit.client.1.smithi153.stdout:7/552: getdents d0/d13 0 2024-04-16T02:35:39.675 INFO:tasks.workunit.client.1.smithi153.stdout:2/554: read d9/d2b/d18/d3f/d40/f8e [377443,40097] 0 2024-04-16T02:35:39.676 INFO:tasks.workunit.client.0.smithi151.stdout:6/199: dwrite d2/d1f/d22/f25 [0,4194304] 0 2024-04-16T02:35:39.676 INFO:tasks.workunit.client.1.smithi153.stdout:1/601: symlink d4/dd/d13/d8d/d95/lcb 0 2024-04-16T02:35:39.678 INFO:tasks.workunit.client.1.smithi153.stdout:5/641: rename d8/df/d2f/d4e/faa to d8/d41/dbd/d83/fc7 0 2024-04-16T02:35:39.679 INFO:tasks.workunit.client.0.smithi151.stdout:6/200: creat d2/d5/df/d23/f41 x:0 0 0 2024-04-16T02:35:39.680 INFO:tasks.workunit.client.1.smithi153.stdout:5/642: write d8/d1f/f21 [1872451,47075] 0 2024-04-16T02:35:39.681 INFO:tasks.workunit.client.1.smithi153.stdout:5/643: chown f4 1 1 2024-04-16T02:35:39.681 INFO:tasks.workunit.client.0.smithi151.stdout:6/201: dread d2/f30 [0,4194304] 0 2024-04-16T02:35:39.683 INFO:tasks.workunit.client.1.smithi153.stdout:0/676: sync 2024-04-16T02:35:39.683 INFO:tasks.workunit.client.1.smithi153.stdout:7/553: creat d0/d13/fbc x:0 0 0 2024-04-16T02:35:39.684 INFO:tasks.workunit.client.0.smithi151.stdout:6/202: symlink d2/d5/df/d20/l42 0 2024-04-16T02:35:39.685 INFO:tasks.workunit.client.1.smithi153.stdout:1/602: dwrite d4/dd/d13/d23/d5f/fc0 [0,4194304] 0 2024-04-16T02:35:39.687 INFO:tasks.workunit.client.1.smithi153.stdout:1/603: write d4/dd/d20/f5b [1896128,25433] 0 2024-04-16T02:35:39.687 INFO:tasks.workunit.client.1.smithi153.stdout:6/570: rename dc/d47/d57/d77/d15/c7c to dc/d1a/d2d/cc3 0 2024-04-16T02:35:39.687 INFO:tasks.workunit.client.0.smithi151.stdout:6/203: creat d2/d5/df/d23/d31/f43 x:0 0 0 2024-04-16T02:35:39.688 INFO:tasks.workunit.client.1.smithi153.stdout:6/571: dread - dc/d47/d7d/fb6 zero size 2024-04-16T02:35:39.690 INFO:tasks.workunit.client.0.smithi151.stdout:6/204: mknod d2/d3d/c44 0 2024-04-16T02:35:39.693 INFO:tasks.workunit.client.0.smithi151.stdout:6/205: mkdir d2/d5/df/d23/d45 0 2024-04-16T02:35:39.693 INFO:tasks.workunit.client.1.smithi153.stdout:3/561: dwrite d6/d10/d3b/d59/fbe [4194304,4194304] 0 2024-04-16T02:35:39.694 INFO:tasks.workunit.client.0.smithi151.stdout:6/206: stat d2/d5/df/d23/d31 0 2024-04-16T02:35:39.695 INFO:tasks.workunit.client.1.smithi153.stdout:7/554: mkdir d0/d4/db/d2b/d53/dbd 0 2024-04-16T02:35:39.695 INFO:tasks.workunit.client.1.smithi153.stdout:3/562: chown d6/d10/d22/d2e/d8a/f91 117898855 1 2024-04-16T02:35:39.696 INFO:tasks.workunit.client.1.smithi153.stdout:7/555: dread - d0/d4/db/d2b/f89 zero size 2024-04-16T02:35:39.696 INFO:tasks.workunit.client.0.smithi151.stdout:6/207: mkdir d2/d13/d46 0 2024-04-16T02:35:39.697 INFO:tasks.workunit.client.1.smithi153.stdout:2/555: rename d9/d2b/d5c/d5f/f69 to d9/d71/d8d/da8/fbe 0 2024-04-16T02:35:39.698 INFO:tasks.workunit.client.1.smithi153.stdout:2/556: write d9/d2b/d1c/d20/d88/f9b [669396,80261] 0 2024-04-16T02:35:39.699 INFO:tasks.workunit.client.1.smithi153.stdout:1/604: creat d4/dd/d13/d7e/d80/fcc x:0 0 0 2024-04-16T02:35:39.705 INFO:tasks.workunit.client.0.smithi151.stdout:6/208: dwrite d2/f30 [0,4194304] 0 2024-04-16T02:35:39.706 INFO:tasks.workunit.client.1.smithi153.stdout:2/557: dread d9/d2b/d1c/d20/d45/f54 [0,4194304] 0 2024-04-16T02:35:39.707 INFO:tasks.workunit.client.1.smithi153.stdout:6/572: mkdir dc/d47/d57/d77/dc4 0 2024-04-16T02:35:39.710 INFO:tasks.workunit.client.1.smithi153.stdout:0/677: dread f5 [0,4194304] 0 2024-04-16T02:35:39.711 INFO:tasks.workunit.client.0.smithi151.stdout:6/209: unlink d2/d5/df/d23/f41 0 2024-04-16T02:35:39.713 INFO:tasks.workunit.client.1.smithi153.stdout:5/644: link l2 d8/d41/d90/lc8 0 2024-04-16T02:35:39.714 INFO:tasks.workunit.client.1.smithi153.stdout:7/556: fdatasync d0/d4/db/d2b/f3e 0 2024-04-16T02:35:39.716 INFO:tasks.workunit.client.1.smithi153.stdout:0/678: mkdir dd/d32/d57/d6c/dd1 0 2024-04-16T02:35:39.716 INFO:tasks.workunit.client.0.smithi151.stdout:6/210: truncate d2/d5/fc 70707 0 2024-04-16T02:35:39.718 INFO:tasks.workunit.client.1.smithi153.stdout:6/573: mknod dc/d47/d57/d77/cc5 0 2024-04-16T02:35:39.718 INFO:tasks.workunit.client.0.smithi151.stdout:6/211: creat d2/d5/df/d20/f47 x:0 0 0 2024-04-16T02:35:39.719 INFO:tasks.workunit.client.0.smithi151.stdout:6/212: creat d2/d5/df/f48 x:0 0 0 2024-04-16T02:35:39.720 INFO:tasks.workunit.client.1.smithi153.stdout:2/558: mknod d9/d2b/d18/cbf 0 2024-04-16T02:35:39.720 INFO:tasks.workunit.client.0.smithi151.stdout:6/213: readlink d2/d5/df/d20/l42 0 2024-04-16T02:35:39.722 INFO:tasks.workunit.client.0.smithi151.stdout:6/214: mkdir d2/d5/df/d23/d49 0 2024-04-16T02:35:39.724 INFO:tasks.workunit.client.1.smithi153.stdout:0/679: mkdir dd/d26/d44/dd2 0 2024-04-16T02:35:39.725 INFO:tasks.workunit.client.1.smithi153.stdout:1/605: link d4/dd/d20/f55 d4/dd/d13/d26/db4/fcd 0 2024-04-16T02:35:39.726 INFO:tasks.workunit.client.0.smithi151.stdout:6/215: dread d2/f30 [0,4194304] 0 2024-04-16T02:35:39.726 INFO:tasks.workunit.client.1.smithi153.stdout:6/574: truncate dc/d10/f33 484936 0 2024-04-16T02:35:39.729 INFO:tasks.workunit.client.1.smithi153.stdout:7/557: rename d0/d4/db/d2b/d43/d6d/d68/d72/l96 to d0/d4/db/d2b/d43/d6d/d68/lbe 0 2024-04-16T02:35:39.729 INFO:tasks.workunit.client.0.smithi151.stdout:6/216: symlink d2/l4a 0 2024-04-16T02:35:39.730 INFO:tasks.workunit.client.1.smithi153.stdout:0/680: creat dd/d14/da6/da4/fd3 x:0 0 0 2024-04-16T02:35:39.730 INFO:tasks.workunit.client.1.smithi153.stdout:0/681: stat dd/d14/da6/lcc 0 2024-04-16T02:35:39.731 INFO:tasks.workunit.client.1.smithi153.stdout:7/558: read d0/d4/f58 [1277941,93028] 0 2024-04-16T02:35:39.731 INFO:tasks.workunit.client.1.smithi153.stdout:1/606: fsync d4/da2/d65/d82/f9a 0 2024-04-16T02:35:39.732 INFO:tasks.workunit.client.1.smithi153.stdout:5/645: getdents d8/df/d28/d5a/d72/dc0 0 2024-04-16T02:35:39.734 INFO:tasks.workunit.client.0.smithi151.stdout:6/217: dread d2/d13/f18 [0,4194304] 0 2024-04-16T02:35:39.734 INFO:tasks.workunit.client.1.smithi153.stdout:0/682: unlink dd/d26/d92/l9c 0 2024-04-16T02:35:39.734 INFO:tasks.workunit.client.1.smithi153.stdout:0/683: chown dd/d28/f83 1549289 1 2024-04-16T02:35:39.735 INFO:tasks.workunit.client.0.smithi151.stdout:6/218: creat d2/d1f/f4b x:0 0 0 2024-04-16T02:35:39.736 INFO:tasks.workunit.client.1.smithi153.stdout:6/575: rmdir dc/d47/d57/d77/d15/dac/dae 39 2024-04-16T02:35:39.737 INFO:tasks.workunit.client.0.smithi151.stdout:6/219: symlink d2/d13/d3a/l4c 0 2024-04-16T02:35:39.738 INFO:tasks.workunit.client.0.smithi151.stdout:6/220: read d2/f30 [2772345,85468] 0 2024-04-16T02:35:39.739 INFO:tasks.workunit.client.1.smithi153.stdout:7/559: rmdir d0/d4/db/d2b/d43/d6d/d68/d6c 39 2024-04-16T02:35:39.741 INFO:tasks.workunit.client.1.smithi153.stdout:5/646: write d8/d41/dbd/d83/fc7 [3231391,10831] 0 2024-04-16T02:35:39.742 INFO:tasks.workunit.client.1.smithi153.stdout:0/684: rename dd/d14/d19/ca5 to dd/d14/d51/d9e/cd4 0 2024-04-16T02:35:39.742 INFO:tasks.workunit.client.1.smithi153.stdout:6/576: dwrite f3 [0,4194304] 0 2024-04-16T02:35:39.742 INFO:tasks.workunit.client.0.smithi151.stdout:6/221: dread d2/f1a [0,4194304] 0 2024-04-16T02:35:39.744 INFO:tasks.workunit.client.1.smithi153.stdout:0/685: truncate dd/d26/d44/f82 5068291 0 2024-04-16T02:35:39.750 INFO:tasks.workunit.client.1.smithi153.stdout:0/686: write dd/d14/da6/da4/fbe [2589308,18173] 0 2024-04-16T02:35:39.750 INFO:tasks.workunit.client.1.smithi153.stdout:1/607: dwrite d4/dd/d13/f14 [0,4194304] 0 2024-04-16T02:35:39.751 INFO:tasks.workunit.client.0.smithi151.stdout:6/222: dwrite d2/d1f/f26 [0,4194304] 0 2024-04-16T02:35:39.755 INFO:tasks.workunit.client.1.smithi153.stdout:6/577: truncate dc/d47/d57/d77/d15/f20 1410234 0 2024-04-16T02:35:39.757 INFO:tasks.workunit.client.1.smithi153.stdout:7/560: mkdir d0/d4/db/d2b/d43/dbf 0 2024-04-16T02:35:39.759 INFO:tasks.workunit.client.1.smithi153.stdout:6/578: truncate dc/d47/d93/da9/fbd 498192 0 2024-04-16T02:35:39.762 INFO:tasks.workunit.client.1.smithi153.stdout:7/561: mkdir d0/d4/db/d87/dc0 0 2024-04-16T02:35:39.766 INFO:tasks.workunit.client.1.smithi153.stdout:7/562: fsync d0/d4/db/d2b/d53/d59/f5a 0 2024-04-16T02:35:39.767 INFO:tasks.workunit.client.1.smithi153.stdout:5/647: dread d8/df/f16 [0,4194304] 0 2024-04-16T02:35:39.768 INFO:tasks.workunit.client.1.smithi153.stdout:5/648: chown d8/df/d2e/c8f 15681059 1 2024-04-16T02:35:39.768 INFO:tasks.workunit.client.1.smithi153.stdout:0/687: link dd/d14/d19/c2b dd/d14/d51/d7f/cd5 0 2024-04-16T02:35:39.769 INFO:tasks.workunit.client.1.smithi153.stdout:7/563: creat d0/d4/db/d2b/fc1 x:0 0 0 2024-04-16T02:35:39.779 INFO:tasks.workunit.client.0.smithi151.stdout:6/223: truncate d2/d5/fb 938344 0 2024-04-16T02:35:39.779 INFO:tasks.workunit.client.0.smithi151.stdout:6/224: truncate d2/f33 223934 0 2024-04-16T02:35:39.780 INFO:tasks.workunit.client.1.smithi153.stdout:1/608: dread d4/dd/d13/d23/d5f/f7f [0,4194304] 0 2024-04-16T02:35:39.780 INFO:tasks.workunit.client.1.smithi153.stdout:1/609: chown d4/c37 456 1 2024-04-16T02:35:39.781 INFO:tasks.workunit.client.0.smithi151.stdout:6/225: rmdir d2/d5/df/d20 39 2024-04-16T02:35:39.783 INFO:tasks.workunit.client.0.smithi151.stdout:6/226: rename d2/d5/df/l11 to d2/l4d 0 2024-04-16T02:35:39.784 INFO:tasks.workunit.client.0.smithi151.stdout:6/227: creat d2/d1f/d22/f4e x:0 0 0 2024-04-16T02:35:39.788 INFO:tasks.workunit.client.1.smithi153.stdout:4/654: sync 2024-04-16T02:35:39.811 INFO:tasks.workunit.client.1.smithi153.stdout:6/579: link dc/l28 dc/d59/d5b/lc6 0 2024-04-16T02:35:39.812 INFO:tasks.workunit.client.0.smithi151.stdout:3/126: sync 2024-04-16T02:35:39.812 INFO:tasks.workunit.client.0.smithi151.stdout:9/73: sync 2024-04-16T02:35:39.813 INFO:tasks.workunit.client.0.smithi151.stdout:2/152: sync 2024-04-16T02:35:39.813 INFO:tasks.workunit.client.1.smithi153.stdout:5/649: readlink d8/d41/d90/lc8 0 2024-04-16T02:35:39.814 INFO:tasks.workunit.client.0.smithi151.stdout:3/127: chown d1/d6/l18 1349 1 2024-04-16T02:35:39.814 INFO:tasks.workunit.client.0.smithi151.stdout:2/153: truncate de/d15/f1c 4259288 0 2024-04-16T02:35:39.820 INFO:tasks.workunit.client.0.smithi151.stdout:6/228: mkdir d2/d5/df/d20/d4f 0 2024-04-16T02:35:39.821 INFO:tasks.workunit.client.0.smithi151.stdout:6/229: read - d2/d5/df/d20/f47 zero size 2024-04-16T02:35:39.821 INFO:tasks.workunit.client.1.smithi153.stdout:6/580: truncate dc/d10/f34 721996 0 2024-04-16T02:35:39.822 INFO:tasks.workunit.client.0.smithi151.stdout:2/154: dread de/d15/f1a [0,4194304] 0 2024-04-16T02:35:39.823 INFO:tasks.workunit.client.0.smithi151.stdout:6/230: truncate d2/d1f/f4b 590085 0 2024-04-16T02:35:39.824 INFO:tasks.workunit.client.1.smithi153.stdout:5/650: rename d8/df/d28/d5a/d72/c87 to d8/d41/dbd/d39/cc9 0 2024-04-16T02:35:39.825 INFO:tasks.workunit.client.1.smithi153.stdout:7/564: rmdir d0/d4/db/d34/d4b/d4d 39 2024-04-16T02:35:39.827 INFO:tasks.workunit.client.1.smithi153.stdout:1/610: mknod d4/dd/d13/d8d/d95/dc7/cce 0 2024-04-16T02:35:39.828 INFO:tasks.workunit.client.1.smithi153.stdout:1/611: readlink d4/dd/d13/d23/d31/l43 0 2024-04-16T02:35:39.830 INFO:tasks.workunit.client.0.smithi151.stdout:9/74: mkdir d1/d3/d6/d17 0 2024-04-16T02:35:39.832 INFO:tasks.workunit.client.0.smithi151.stdout:6/231: dwrite d2/d1f/f29 [4194304,4194304] 0 2024-04-16T02:35:39.833 INFO:tasks.workunit.client.1.smithi153.stdout:1/612: dread d4/f22 [0,4194304] 0 2024-04-16T02:35:39.833 INFO:tasks.workunit.client.0.smithi151.stdout:6/232: chown d2/d5/df/d23 6299815 1 2024-04-16T02:35:39.835 INFO:tasks.workunit.client.1.smithi153.stdout:4/655: dread d3/d9/db/de/f5c [0,4194304] 0 2024-04-16T02:35:39.836 INFO:tasks.workunit.client.1.smithi153.stdout:0/688: getdents dd/d14/dca 0 2024-04-16T02:35:39.839 INFO:tasks.workunit.client.0.smithi151.stdout:9/75: symlink d1/dd/l18 0 2024-04-16T02:35:39.839 INFO:tasks.workunit.client.1.smithi153.stdout:1/613: write d4/da2/f96 [3807647,79457] 0 2024-04-16T02:35:39.842 INFO:tasks.workunit.client.1.smithi153.stdout:4/656: symlink d3/d9/db/de/d2c/d52/le9 0 2024-04-16T02:35:39.843 INFO:tasks.workunit.client.1.smithi153.stdout:0/689: symlink dd/d14/d51/d54/ld6 0 2024-04-16T02:35:39.843 INFO:tasks.workunit.client.1.smithi153.stdout:5/651: link l2 d8/df/d84/da4/lca 0 2024-04-16T02:35:39.844 INFO:tasks.workunit.client.0.smithi151.stdout:6/233: mknod d2/d1f/c50 0 2024-04-16T02:35:39.846 INFO:tasks.workunit.client.1.smithi153.stdout:0/690: mknod dd/d32/cd7 0 2024-04-16T02:35:39.848 INFO:tasks.workunit.client.0.smithi151.stdout:6/234: unlink d2/d1f/d22/f25 0 2024-04-16T02:35:39.849 INFO:tasks.workunit.client.1.smithi153.stdout:5/652: mkdir d8/df/d28/d5a/d72/dc0/dcb 0 2024-04-16T02:35:39.849 INFO:tasks.workunit.client.1.smithi153.stdout:1/614: dwrite d4/dd/d18/f90 [0,4194304] 0 2024-04-16T02:35:39.850 INFO:tasks.workunit.client.0.smithi151.stdout:9/76: symlink d1/d3/d10/l19 0 2024-04-16T02:35:39.851 INFO:tasks.workunit.client.1.smithi153.stdout:4/657: rmdir d3/dbc/de8 0 2024-04-16T02:35:39.853 INFO:tasks.workunit.client.1.smithi153.stdout:8/528: sync 2024-04-16T02:35:39.853 INFO:tasks.workunit.client.1.smithi153.stdout:9/614: sync 2024-04-16T02:35:39.854 INFO:tasks.workunit.client.1.smithi153.stdout:0/691: mknod dd/d14/d19/d20/cd8 0 2024-04-16T02:35:39.854 INFO:tasks.workunit.client.1.smithi153.stdout:4/658: read d3/d9/db/d17/f8c [2212482,69665] 0 2024-04-16T02:35:39.854 INFO:tasks.workunit.client.0.smithi151.stdout:9/77: dread d1/f5 [4194304,4194304] 0 2024-04-16T02:35:39.854 INFO:tasks.workunit.client.0.smithi151.stdout:6/235: readlink d2/l1d 0 2024-04-16T02:35:39.855 INFO:tasks.workunit.client.1.smithi153.stdout:1/615: mknod d4/da2/ccf 0 2024-04-16T02:35:39.855 INFO:tasks.workunit.client.1.smithi153.stdout:9/615: write d4/d6/fbc [254871,23402] 0 2024-04-16T02:35:39.855 INFO:tasks.workunit.client.0.smithi151.stdout:2/155: dread de/f27 [8388608,4194304] 0 2024-04-16T02:35:39.856 INFO:tasks.workunit.client.1.smithi153.stdout:4/659: chown d3/d9/db/de/f3b 111081088 1 2024-04-16T02:35:39.856 INFO:tasks.workunit.client.0.smithi151.stdout:9/78: write d1/d3/d6/f13 [1188069,60227] 0 2024-04-16T02:35:39.857 INFO:tasks.workunit.client.1.smithi153.stdout:0/692: chown dd/d14/d19/d55 3361302 1 2024-04-16T02:35:39.857 INFO:tasks.workunit.client.0.smithi151.stdout:9/79: chown d1/d3/l15 395 1 2024-04-16T02:35:39.858 INFO:tasks.workunit.client.1.smithi153.stdout:0/693: chown dd/d14/d51/d54/d65 2254 1 2024-04-16T02:35:39.859 INFO:tasks.workunit.client.1.smithi153.stdout:8/529: creat d2/d93/d97/d9c/d27/d5e/d63/fb1 x:0 0 0 2024-04-16T02:35:39.859 INFO:tasks.workunit.client.1.smithi153.stdout:4/660: read d3/d9/db/de/f3b [2725578,101753] 0 2024-04-16T02:35:39.860 INFO:tasks.workunit.client.0.smithi151.stdout:6/236: mknod d2/d1f/c51 0 2024-04-16T02:35:39.860 INFO:tasks.workunit.client.1.smithi153.stdout:4/661: truncate d3/d9/d79/fd3 425235 0 2024-04-16T02:35:39.861 INFO:tasks.workunit.client.1.smithi153.stdout:4/662: dread - d3/fdf zero size 2024-04-16T02:35:39.861 INFO:tasks.workunit.client.1.smithi153.stdout:1/616: truncate d4/dd/d18/d29/d71/fc5 283784 0 2024-04-16T02:35:39.862 INFO:tasks.workunit.client.1.smithi153.stdout:5/653: getdents d8/d41/dbd/d83 0 2024-04-16T02:35:39.864 INFO:tasks.workunit.client.1.smithi153.stdout:8/530: rename d2/d13/d3a/d5b/l8a to d2/d13/d3a/d5b/lb2 0 2024-04-16T02:35:39.866 INFO:tasks.workunit.client.0.smithi151.stdout:9/80: dwrite d1/d3/fb [0,4194304] 0 2024-04-16T02:35:39.868 INFO:tasks.workunit.client.1.smithi153.stdout:5/654: mknod d8/df/d23/d70/ccc 0 2024-04-16T02:35:39.869 INFO:tasks.workunit.client.1.smithi153.stdout:4/663: mknod d3/d9/d24/d75/d9e/cea 0 2024-04-16T02:35:39.869 INFO:tasks.workunit.client.0.smithi151.stdout:6/237: dwrite d2/d1f/f29 [4194304,4194304] 0 2024-04-16T02:35:39.871 INFO:tasks.workunit.client.0.smithi151.stdout:6/238: fsync d2/d5/df/f3c 0 2024-04-16T02:35:39.872 INFO:tasks.workunit.client.1.smithi153.stdout:1/617: rename d4/f40 to d4/dd/d13/d26/db7/fd0 0 2024-04-16T02:35:39.872 INFO:tasks.workunit.client.1.smithi153.stdout:8/531: dwrite d2/d93/d97/d9c/d27/d2d/d47/d60/f71 [0,4194304] 0 2024-04-16T02:35:39.874 INFO:tasks.workunit.client.1.smithi153.stdout:1/618: write d4/dd/fbe [741494,72118] 0 2024-04-16T02:35:39.875 INFO:tasks.workunit.client.1.smithi153.stdout:4/664: symlink d3/d9/d24/d75/dc5/leb 0 2024-04-16T02:35:39.876 INFO:tasks.workunit.client.0.smithi151.stdout:9/81: dwrite d1/d3/fb [0,4194304] 0 2024-04-16T02:35:39.876 INFO:tasks.workunit.client.1.smithi153.stdout:8/532: creat d2/d13/d96/fb3 x:0 0 0 2024-04-16T02:35:39.877 INFO:tasks.workunit.client.0.smithi151.stdout:2/156: truncate de/f27 736046 0 2024-04-16T02:35:39.878 INFO:tasks.workunit.client.1.smithi153.stdout:8/533: chown d2/d1c/l33 0 1 2024-04-16T02:35:39.882 INFO:tasks.workunit.client.1.smithi153.stdout:5/655: rename d8/d41/dbd/d39/f4a to d8/d1f/fcd 0 2024-04-16T02:35:39.882 INFO:tasks.workunit.client.0.smithi151.stdout:9/82: creat d1/d3/f1a x:0 0 0 2024-04-16T02:35:39.884 INFO:tasks.workunit.client.1.smithi153.stdout:1/619: dwrite d4/dd/d18/d29/f50 [0,4194304] 0 2024-04-16T02:35:39.884 INFO:tasks.workunit.client.0.smithi151.stdout:9/83: fdatasync d1/dd/f16 0 2024-04-16T02:35:39.887 INFO:tasks.workunit.client.0.smithi151.stdout:2/157: dwrite de/d11/f18 [4194304,4194304] 0 2024-04-16T02:35:39.887 INFO:tasks.workunit.client.1.smithi153.stdout:4/665: mkdir d3/d9/db/d17/d28/dc8/de4/dec 0 2024-04-16T02:35:39.888 INFO:tasks.workunit.client.0.smithi151.stdout:9/84: creat d1/d3/d6/f1b x:0 0 0 2024-04-16T02:35:39.889 INFO:tasks.workunit.client.0.smithi151.stdout:9/85: write d1/d3/f1a [322395,109377] 0 2024-04-16T02:35:39.890 INFO:tasks.workunit.client.0.smithi151.stdout:2/158: creat de/d15/d1e/d28/f33 x:0 0 0 2024-04-16T02:35:39.891 INFO:tasks.workunit.client.1.smithi153.stdout:4/666: write d3/d9/db/d17/f5d [4342977,118183] 0 2024-04-16T02:35:39.892 INFO:tasks.workunit.client.1.smithi153.stdout:9/616: dread d4/d6/d1b/d6e/f74 [0,4194304] 0 2024-04-16T02:35:39.892 INFO:tasks.workunit.client.0.smithi151.stdout:9/86: mknod d1/d3/d6/c1c 0 2024-04-16T02:35:39.894 INFO:tasks.workunit.client.1.smithi153.stdout:1/620: link d4/dd/cbc d4/dd/d18/d29/d71/cd1 0 2024-04-16T02:35:39.895 INFO:tasks.workunit.client.0.smithi151.stdout:2/159: read de/d15/f17 [181166,95519] 0 2024-04-16T02:35:39.896 INFO:tasks.workunit.client.0.smithi151.stdout:9/87: mkdir d1/d3/d6/d1d 0 2024-04-16T02:35:39.897 INFO:tasks.workunit.client.0.smithi151.stdout:6/239: write d2/d5/fb [918818,16790] 0 2024-04-16T02:35:39.898 INFO:tasks.workunit.client.1.smithi153.stdout:5/656: dwrite d8/df/d28/f63 [0,4194304] 0 2024-04-16T02:35:39.898 INFO:tasks.workunit.client.1.smithi153.stdout:4/667: write d3/d9/f68 [769877,125477] 0 2024-04-16T02:35:39.898 INFO:tasks.workunit.client.0.smithi151.stdout:2/160: creat de/d15/f34 x:0 0 0 2024-04-16T02:35:39.898 INFO:tasks.workunit.client.0.smithi151.stdout:6/240: stat d2/l4 0 2024-04-16T02:35:39.899 INFO:tasks.workunit.client.0.smithi151.stdout:2/161: chown de/d15/d1e/d28 8010720 1 2024-04-16T02:35:39.900 INFO:tasks.workunit.client.0.smithi151.stdout:6/241: chown d2/d5/df/d20/f24 5233 1 2024-04-16T02:35:39.901 INFO:tasks.workunit.client.1.smithi153.stdout:9/617: unlink d4/d6/dc/d2d/da0/f7c 0 2024-04-16T02:35:39.901 INFO:tasks.workunit.client.1.smithi153.stdout:4/668: dread - d3/d9/db/de/d2c/fc9 zero size 2024-04-16T02:35:39.901 INFO:tasks.workunit.client.0.smithi151.stdout:9/88: unlink d1/dd/cf 0 2024-04-16T02:35:39.901 INFO:tasks.workunit.client.0.smithi151.stdout:6/242: chown d2/f1b 51859 1 2024-04-16T02:35:39.902 INFO:tasks.workunit.client.0.smithi151.stdout:6/243: write d2/f9 [3140367,115982] 0 2024-04-16T02:35:39.903 INFO:tasks.workunit.client.1.smithi153.stdout:4/669: truncate d3/d9/db/de/fcd 705317 0 2024-04-16T02:35:39.904 INFO:tasks.workunit.client.0.smithi151.stdout:9/89: creat d1/d3/f1e x:0 0 0 2024-04-16T02:35:39.904 INFO:tasks.workunit.client.0.smithi151.stdout:6/244: truncate d2/d1f/d22/f4e 931892 0 2024-04-16T02:35:39.906 INFO:tasks.workunit.client.0.smithi151.stdout:6/245: dread - d2/d5/f3e zero size 2024-04-16T02:35:39.906 INFO:tasks.workunit.client.1.smithi153.stdout:1/621: rename d4/f99 to d4/dd/d18/d29/fd2 0 2024-04-16T02:35:39.907 INFO:tasks.workunit.client.0.smithi151.stdout:9/90: mknod d1/d3/d6/c1f 0 2024-04-16T02:35:39.907 INFO:tasks.workunit.client.1.smithi153.stdout:5/657: mknod d8/df/d28/cce 0 2024-04-16T02:35:39.909 INFO:tasks.workunit.client.0.smithi151.stdout:6/246: mkdir d2/d5/df/d23/d31/d52 0 2024-04-16T02:35:39.910 INFO:tasks.workunit.client.1.smithi153.stdout:1/622: creat d4/dd/d13/d23/fd3 x:0 0 0 2024-04-16T02:35:39.910 INFO:tasks.workunit.client.1.smithi153.stdout:9/618: dwrite d4/d6/dc/d2d/da0/d40/d4a/f68 [0,4194304] 0 2024-04-16T02:35:39.911 INFO:tasks.workunit.client.0.smithi151.stdout:6/247: dread - d2/d5/df/d23/d31/f43 zero size 2024-04-16T02:35:39.912 INFO:tasks.workunit.client.1.smithi153.stdout:4/670: getdents d3/d9/db/de/d2c/d38 0 2024-04-16T02:35:39.916 INFO:tasks.workunit.client.1.smithi153.stdout:5/658: dread d8/f4d [0,4194304] 0 2024-04-16T02:35:39.916 INFO:tasks.workunit.client.1.smithi153.stdout:9/619: mkdir d4/d6/dc/d24/d5d/d66/dbd 0 2024-04-16T02:35:39.918 INFO:tasks.workunit.client.1.smithi153.stdout:4/671: mknod d3/d9/d79/ced 0 2024-04-16T02:35:39.918 INFO:tasks.workunit.client.0.smithi151.stdout:9/91: dwrite d1/d3/d6/f13 [0,4194304] 0 2024-04-16T02:35:39.920 INFO:tasks.workunit.client.0.smithi151.stdout:6/248: rename d2/d5/df/f2d to d2/d13/d46/f53 0 2024-04-16T02:35:39.921 INFO:tasks.workunit.client.1.smithi153.stdout:9/620: read d4/d33/f5b [3113604,61616] 0 2024-04-16T02:35:39.922 INFO:tasks.workunit.client.1.smithi153.stdout:1/623: rename d4/dd/d18/d6c/daf to d4/dd/dd4 0 2024-04-16T02:35:39.922 INFO:tasks.workunit.client.0.smithi151.stdout:6/249: write d2/d5/df/d20/f2b [5160791,53277] 0 2024-04-16T02:35:39.922 INFO:tasks.workunit.client.1.smithi153.stdout:9/621: write d4/d6/dc/d2d/da0/f86 [978426,4274] 0 2024-04-16T02:35:39.923 INFO:tasks.workunit.client.1.smithi153.stdout:4/672: creat d3/da5/dd2/fee x:0 0 0 2024-04-16T02:35:39.924 INFO:tasks.workunit.client.0.smithi151.stdout:9/92: mkdir d1/d3/d10/d20 0 2024-04-16T02:35:39.926 INFO:tasks.workunit.client.0.smithi151.stdout:9/93: write d1/f7 [1100159,18001] 0 2024-04-16T02:35:39.926 INFO:tasks.workunit.client.1.smithi153.stdout:5/659: rename d8/d41/dbd/d83/lb2 to d8/d41/dbd/daf/lcf 0 2024-04-16T02:35:39.927 INFO:tasks.workunit.client.0.smithi151.stdout:9/94: truncate d1/d3/f1e 269285 0 2024-04-16T02:35:39.928 INFO:tasks.workunit.client.1.smithi153.stdout:5/660: write d8/f3a [5642395,109741] 0 2024-04-16T02:35:39.929 INFO:tasks.workunit.client.0.smithi151.stdout:6/250: creat d2/d13/d3a/f54 x:0 0 0 2024-04-16T02:35:39.929 INFO:tasks.workunit.client.0.smithi151.stdout:9/95: write d1/d3/f1a [1274766,96412] 0 2024-04-16T02:35:39.929 INFO:tasks.workunit.client.1.smithi153.stdout:5/661: truncate d8/f30 4365313 0 2024-04-16T02:35:39.930 INFO:tasks.workunit.client.0.smithi151.stdout:6/251: readlink d2/d5/df/d20/l42 0 2024-04-16T02:35:39.931 INFO:tasks.workunit.client.1.smithi153.stdout:5/662: chown d8/df/d23/f32 6731 1 2024-04-16T02:35:39.934 INFO:tasks.workunit.client.1.smithi153.stdout:9/622: creat d4/d6/d1b/fbe x:0 0 0 2024-04-16T02:35:39.934 INFO:tasks.workunit.client.0.smithi151.stdout:9/96: rmdir d1/dd 39 2024-04-16T02:35:39.936 INFO:tasks.workunit.client.1.smithi153.stdout:5/663: mkdir d8/df/d28/d5a/d72/dd0 0 2024-04-16T02:35:39.937 INFO:tasks.workunit.client.0.smithi151.stdout:6/252: creat d2/d3d/f55 x:0 0 0 2024-04-16T02:35:39.938 INFO:tasks.workunit.client.0.smithi151.stdout:6/253: readlink d2/l28 0 2024-04-16T02:35:39.940 INFO:tasks.workunit.client.1.smithi153.stdout:9/623: creat d4/d6/dc/d2d/d79/dbb/fbf x:0 0 0 2024-04-16T02:35:39.941 INFO:tasks.workunit.client.0.smithi151.stdout:6/254: rmdir d2/d5/d2e 39 2024-04-16T02:35:39.941 INFO:tasks.workunit.client.0.smithi151.stdout:6/255: chown d2/d13/f18 7777648 1 2024-04-16T02:35:39.943 INFO:tasks.workunit.client.1.smithi153.stdout:9/624: symlink d4/d6/dc/d2d/d64/lc0 0 2024-04-16T02:35:39.944 INFO:tasks.workunit.client.0.smithi151.stdout:2/162: read de/f27 [681557,20806] 0 2024-04-16T02:35:39.944 INFO:tasks.workunit.client.0.smithi151.stdout:6/256: dread - d2/d5/df/d20/f24 zero size 2024-04-16T02:35:39.945 INFO:tasks.workunit.client.1.smithi153.stdout:9/625: write d4/d6/dc/d24/d4e/f9c [1444655,116651] 0 2024-04-16T02:35:39.949 INFO:tasks.workunit.client.0.smithi151.stdout:6/257: mkdir d2/d5/df/d23/d56 0 2024-04-16T02:35:39.959 INFO:tasks.workunit.client.0.smithi151.stdout:6/258: dwrite d2/d1f/f29 [8388608,4194304] 0 2024-04-16T02:35:39.965 INFO:tasks.workunit.client.1.smithi153.stdout:0/694: truncate dd/d26/d44/f50 3529009 0 2024-04-16T02:35:39.965 INFO:tasks.workunit.client.1.smithi153.stdout:0/695: chown dd/d28/f4c 37332 1 2024-04-16T02:35:39.966 INFO:tasks.workunit.client.0.smithi151.stdout:2/163: dwrite de/f27 [0,4194304] 0 2024-04-16T02:35:39.969 INFO:tasks.workunit.client.1.smithi153.stdout:6/581: dread dc/d1a/f1c [0,4194304] 0 2024-04-16T02:35:39.969 INFO:tasks.workunit.client.0.smithi151.stdout:2/164: mkdir de/d15/d1e/d28/d35 0 2024-04-16T02:35:39.970 INFO:tasks.workunit.client.1.smithi153.stdout:0/696: unlink dd/d26/d44/c2e 0 2024-04-16T02:35:39.972 INFO:tasks.workunit.client.0.smithi151.stdout:2/165: write de/f27 [1722710,44920] 0 2024-04-16T02:35:39.973 INFO:tasks.workunit.client.0.smithi151.stdout:2/166: chown de/d15/f17 38058965 1 2024-04-16T02:35:39.973 INFO:tasks.workunit.client.1.smithi153.stdout:6/582: dread dc/d47/d57/d77/d15/f2f [0,4194304] 0 2024-04-16T02:35:39.975 INFO:tasks.workunit.client.0.smithi151.stdout:2/167: write de/f27 [1276350,48336] 0 2024-04-16T02:35:39.975 INFO:tasks.workunit.client.1.smithi153.stdout:0/697: write dd/d14/d51/d54/d65/f72 [5620602,29274] 0 2024-04-16T02:35:39.979 INFO:tasks.workunit.client.0.smithi151.stdout:2/168: creat de/d15/f36 x:0 0 0 2024-04-16T02:35:39.980 INFO:tasks.workunit.client.0.smithi151.stdout:9/97: dread d1/f5 [0,4194304] 0 2024-04-16T02:35:39.982 INFO:tasks.workunit.client.1.smithi153.stdout:3/563: sync 2024-04-16T02:35:39.982 INFO:tasks.workunit.client.1.smithi153.stdout:2/559: sync 2024-04-16T02:35:39.982 INFO:tasks.workunit.client.0.smithi151.stdout:2/169: truncate de/d15/f36 132496 0 2024-04-16T02:35:39.987 INFO:tasks.workunit.client.0.smithi151.stdout:9/98: link d1/c11 d1/d3/d10/c21 0 2024-04-16T02:35:39.989 INFO:tasks.workunit.client.0.smithi151.stdout:6/259: dwrite d2/f1a [0,4194304] 0 2024-04-16T02:35:39.989 INFO:tasks.workunit.client.0.smithi151.stdout:2/170: dread de/d15/f1c [0,4194304] 0 2024-04-16T02:35:39.990 INFO:tasks.workunit.client.0.smithi151.stdout:6/260: read - d2/d3d/f55 zero size 2024-04-16T02:35:39.994 INFO:tasks.workunit.client.0.smithi151.stdout:9/99: mkdir d1/d3/d6/d1d/d22 0 2024-04-16T02:35:39.997 INFO:tasks.workunit.client.0.smithi151.stdout:4/170: sync 2024-04-16T02:35:39.998 INFO:tasks.workunit.client.1.smithi153.stdout:3/564: write d6/d10/d22/d4d/f6e [4787285,115872] 0 2024-04-16T02:35:39.998 INFO:tasks.workunit.client.0.smithi151.stdout:6/261: creat d2/d5/df/d23/d49/f57 x:0 0 0 2024-04-16T02:35:39.999 INFO:tasks.workunit.client.0.smithi151.stdout:2/171: getdents de/d15/d1e 0 2024-04-16T02:35:39.999 INFO:tasks.workunit.client.1.smithi153.stdout:2/560: mkdir d9/d24/d27/d2c/d5d/da3/dc0 0 2024-04-16T02:35:40.001 INFO:tasks.workunit.client.1.smithi153.stdout:2/561: dread - d9/d2b/d1c/d20/d45/f96 zero size 2024-04-16T02:35:40.001 INFO:tasks.workunit.client.0.smithi151.stdout:9/100: rename d1/f7 to d1/d3/d6/d1d/d22/f23 0 2024-04-16T02:35:40.002 INFO:tasks.workunit.client.1.smithi153.stdout:4/673: dread d3/ff [0,4194304] 0 2024-04-16T02:35:40.002 INFO:tasks.workunit.client.1.smithi153.stdout:6/583: link dc/d10/f34 dc/d1a/d2d/fc7 0 2024-04-16T02:35:40.005 INFO:tasks.workunit.client.0.smithi151.stdout:9/101: creat d1/d3/d10/f24 x:0 0 0 2024-04-16T02:35:40.006 INFO:tasks.workunit.client.0.smithi151.stdout:6/262: creat d2/d5/df/d23/d56/f58 x:0 0 0 2024-04-16T02:35:40.006 INFO:tasks.workunit.client.0.smithi151.stdout:9/102: chown d1/d3/d10/l19 998727 1 2024-04-16T02:35:40.007 INFO:tasks.workunit.client.1.smithi153.stdout:4/674: creat d3/d9/db/de/d2c/d52/d85/fef x:0 0 0 2024-04-16T02:35:40.009 INFO:tasks.workunit.client.1.smithi153.stdout:4/675: fsync d3/d9/db/de/f26 0 2024-04-16T02:35:40.013 INFO:tasks.workunit.client.1.smithi153.stdout:4/676: mkdir d3/d9/db/d84/de0/de3/df0 0 2024-04-16T02:35:40.014 INFO:tasks.workunit.client.1.smithi153.stdout:1/624: getdents d4/dd/d18/d29/d71 0 2024-04-16T02:35:40.017 INFO:tasks.workunit.client.0.smithi151.stdout:4/171: dread d0/fd [0,4194304] 0 2024-04-16T02:35:40.018 INFO:tasks.workunit.client.1.smithi153.stdout:7/565: dread d0/d13/f4a [0,4194304] 0 2024-04-16T02:35:40.019 INFO:tasks.workunit.client.1.smithi153.stdout:1/625: mkdir d4/da2/dd5 0 2024-04-16T02:35:40.022 INFO:tasks.workunit.client.1.smithi153.stdout:1/626: creat d4/dd/d18/d6c/fd6 x:0 0 0 2024-04-16T02:35:40.022 INFO:tasks.workunit.client.1.smithi153.stdout:3/565: dread d6/d10/d22/f31 [0,4194304] 0 2024-04-16T02:35:40.023 INFO:tasks.workunit.client.1.smithi153.stdout:1/627: write d4/dd/d18/d6c/fd6 [830182,19049] 0 2024-04-16T02:35:40.023 INFO:tasks.workunit.client.0.smithi151.stdout:2/172: write de/d15/f1a [1403052,126969] 0 2024-04-16T02:35:40.024 INFO:tasks.workunit.client.1.smithi153.stdout:1/628: chown d4/da2/d65/caa 43089145 1 2024-04-16T02:35:40.025 INFO:tasks.workunit.client.1.smithi153.stdout:1/629: stat d4/dd/d20/l44 0 2024-04-16T02:35:40.026 INFO:tasks.workunit.client.0.smithi151.stdout:2/173: creat de/d15/d1e/d28/f37 x:0 0 0 2024-04-16T02:35:40.027 INFO:tasks.workunit.client.0.smithi151.stdout:4/172: link d0/fe d0/d11/f39 0 2024-04-16T02:35:40.028 INFO:tasks.workunit.client.1.smithi153.stdout:5/664: rename d8/d41/dbd/daf to d8/df/d2e/dd1 0 2024-04-16T02:35:40.028 INFO:tasks.workunit.client.1.smithi153.stdout:7/566: getdents d0/d4/db/d2b/d43/d6d/d68/d6c/db8 0 2024-04-16T02:35:40.029 INFO:tasks.workunit.client.1.smithi153.stdout:5/665: chown d8/f4d 88663949 1 2024-04-16T02:35:40.029 INFO:tasks.workunit.client.0.smithi151.stdout:4/173: dread - d0/d20/f2c zero size 2024-04-16T02:35:40.029 INFO:tasks.workunit.client.0.smithi151.stdout:9/103: write d1/d3/fa [2491209,85743] 0 2024-04-16T02:35:40.030 INFO:tasks.workunit.client.0.smithi151.stdout:6/263: write d2/d5/fc [335386,38231] 0 2024-04-16T02:35:40.030 INFO:tasks.workunit.client.1.smithi153.stdout:3/566: creat d6/d10/d22/d4d/da0/da3/fc6 x:0 0 0 2024-04-16T02:35:40.030 INFO:tasks.workunit.client.0.smithi151.stdout:4/174: chown d0/f1f 87003790 1 2024-04-16T02:35:40.031 INFO:tasks.workunit.client.0.smithi151.stdout:2/174: getdents de/d15/d1e 0 2024-04-16T02:35:40.032 INFO:tasks.workunit.client.0.smithi151.stdout:2/175: chown de/d15 21 1 2024-04-16T02:35:40.033 INFO:tasks.workunit.client.0.smithi151.stdout:4/175: write d0/d11/d19/f29 [904097,96430] 0 2024-04-16T02:35:40.034 INFO:tasks.workunit.client.1.smithi153.stdout:0/698: rename dd/d32/c8e to dd/d14/d51/d54/d65/d68/cd9 0 2024-04-16T02:35:40.034 INFO:tasks.workunit.client.0.smithi151.stdout:9/104: mknod d1/d3/d6/d17/c25 0 2024-04-16T02:35:40.036 INFO:tasks.workunit.client.1.smithi153.stdout:5/666: symlink d8/df/d2e/dd1/ld2 0 2024-04-16T02:35:40.036 INFO:tasks.workunit.client.1.smithi153.stdout:3/567: fsync d6/d10/d22/d4d/f6e 0 2024-04-16T02:35:40.036 INFO:tasks.workunit.client.1.smithi153.stdout:1/630: dwrite d4/dd/d13/d26/d58/d7a/fa7 [0,4194304] 0 2024-04-16T02:35:40.037 INFO:tasks.workunit.client.1.smithi153.stdout:4/677: dread d3/d9/db/d17/f55 [0,4194304] 0 2024-04-16T02:35:40.039 INFO:tasks.workunit.client.0.smithi151.stdout:4/176: dread d0/d11/d19/f1b [0,4194304] 0 2024-04-16T02:35:40.041 INFO:tasks.workunit.client.0.smithi151.stdout:6/264: write d2/d5/f27 [331696,111084] 0 2024-04-16T02:35:40.044 INFO:tasks.workunit.client.1.smithi153.stdout:6/584: rename dc/f31 to dc/d47/d57/d77/d7a/d97/fc8 0 2024-04-16T02:35:40.045 INFO:tasks.workunit.client.0.smithi151.stdout:9/105: rename d1/d3/d10/c21 to d1/d3/d6/d1d/d22/c26 0 2024-04-16T02:35:40.046 INFO:tasks.workunit.client.1.smithi153.stdout:5/667: mkdir d8/d41/dbd/dd3 0 2024-04-16T02:35:40.048 INFO:tasks.workunit.client.1.smithi153.stdout:3/568: dwrite d6/d10/d3b/d59/fbe [8388608,4194304] 0 2024-04-16T02:35:40.048 INFO:tasks.workunit.client.0.smithi151.stdout:2/176: truncate de/d11/f18 4685543 0 2024-04-16T02:35:40.049 INFO:tasks.workunit.client.0.smithi151.stdout:6/265: creat d2/d1f/f59 x:0 0 0 2024-04-16T02:35:40.050 INFO:tasks.workunit.client.1.smithi153.stdout:4/678: creat d3/d9/db/d17/d28/dc8/ff1 x:0 0 0 2024-04-16T02:35:40.050 INFO:tasks.workunit.client.1.smithi153.stdout:3/569: fsync d6/d10/d22/d2e/d8a/fc3 0 2024-04-16T02:35:40.050 INFO:tasks.workunit.client.0.smithi151.stdout:9/106: creat d1/d3/d6/d17/f27 x:0 0 0 2024-04-16T02:35:40.051 INFO:tasks.workunit.client.1.smithi153.stdout:3/570: write d6/d10/d22/d2e/f4e [83011,2603] 0 2024-04-16T02:35:40.051 INFO:tasks.workunit.client.0.smithi151.stdout:4/177: dwrite d0/f3 [0,4194304] 0 2024-04-16T02:35:40.052 INFO:tasks.workunit.client.0.smithi151.stdout:2/177: creat de/d15/d1e/d28/f38 x:0 0 0 2024-04-16T02:35:40.053 INFO:tasks.workunit.client.1.smithi153.stdout:2/562: write d9/d2b/f16 [2747040,86894] 0 2024-04-16T02:35:40.054 INFO:tasks.workunit.client.0.smithi151.stdout:4/178: dread - d0/d20/f21 zero size 2024-04-16T02:35:40.055 INFO:tasks.workunit.client.0.smithi151.stdout:9/107: write d1/fc [2527799,26175] 0 2024-04-16T02:35:40.061 INFO:tasks.workunit.client.1.smithi153.stdout:7/567: rename d0/d4/db/d2b/d8c/db5/fba to d0/d4/db/d34/d4b/fc2 0 2024-04-16T02:35:40.062 INFO:tasks.workunit.client.0.smithi151.stdout:9/108: mkdir d1/d3/d6/d28 0 2024-04-16T02:35:40.066 INFO:tasks.workunit.client.1.smithi153.stdout:6/585: dread dc/d47/d57/d77/d15/f23 [0,4194304] 0 2024-04-16T02:35:40.066 INFO:tasks.workunit.client.1.smithi153.stdout:4/679: dwrite d3/d9/db/de/d2c/d52/fa2 [0,4194304] 0 2024-04-16T02:35:40.066 INFO:tasks.workunit.client.0.smithi151.stdout:6/266: dwrite d2/d1f/d22/d40/f3b [0,4194304] 0 2024-04-16T02:35:40.069 INFO:tasks.workunit.client.0.smithi151.stdout:2/178: dwrite de/d15/f2c [4194304,4194304] 0 2024-04-16T02:35:40.071 INFO:tasks.workunit.client.0.smithi151.stdout:4/179: dread d0/d11/d19/f1b [0,4194304] 0 2024-04-16T02:35:40.072 INFO:tasks.workunit.client.0.smithi151.stdout:9/109: symlink d1/d3/d6/d1d/d22/l29 0 2024-04-16T02:35:40.073 INFO:tasks.workunit.client.1.smithi153.stdout:2/563: dwrite d9/d24/d27/d2c/d41/f4c [0,4194304] 0 2024-04-16T02:35:40.073 INFO:tasks.workunit.client.0.smithi151.stdout:4/180: stat d0/d11 0 2024-04-16T02:35:40.074 INFO:tasks.workunit.client.1.smithi153.stdout:6/586: write dc/d47/d57/d77/d15/f1e [1976509,56166] 0 2024-04-16T02:35:40.074 INFO:tasks.workunit.client.1.smithi153.stdout:2/564: read d9/d2b/d1c/d20/f67 [1156454,4567] 0 2024-04-16T02:35:40.074 INFO:tasks.workunit.client.1.smithi153.stdout:0/699: rename dd/d14/d19/d81 to dd/d14/d19/d77/d87/dda 0 2024-04-16T02:35:40.076 INFO:tasks.workunit.client.0.smithi151.stdout:9/110: creat d1/d3/d6/f2a x:0 0 0 2024-04-16T02:35:40.076 INFO:tasks.workunit.client.1.smithi153.stdout:4/680: chown d3/d9/db/d17/d28/cbd 1 1 2024-04-16T02:35:40.077 INFO:tasks.workunit.client.0.smithi151.stdout:6/267: dread d2/d1f/f4b [0,4194304] 0 2024-04-16T02:35:40.078 INFO:tasks.workunit.client.0.smithi151.stdout:6/268: fsync d2/f1a 0 2024-04-16T02:35:40.079 INFO:tasks.workunit.client.0.smithi151.stdout:6/269: chown d2/l17 9608834 1 2024-04-16T02:35:40.080 INFO:tasks.workunit.client.1.smithi153.stdout:4/681: write d3/d9/d24/d75/dc5/fdb [880512,34662] 0 2024-04-16T02:35:40.080 INFO:tasks.workunit.client.1.smithi153.stdout:0/700: write dd/d14/d51/d54/fc7 [535044,68099] 0 2024-04-16T02:35:40.080 INFO:tasks.workunit.client.0.smithi151.stdout:9/111: symlink d1/dd/l2b 0 2024-04-16T02:35:40.082 INFO:tasks.workunit.client.0.smithi151.stdout:4/181: dwrite d0/d11/d19/f1e [0,4194304] 0 2024-04-16T02:35:40.083 INFO:tasks.workunit.client.0.smithi151.stdout:6/270: creat d2/d5/f5a x:0 0 0 2024-04-16T02:35:40.085 INFO:tasks.workunit.client.0.smithi151.stdout:9/112: mkdir d1/d3/d6/d1d/d22/d2c 0 2024-04-16T02:35:40.089 INFO:tasks.workunit.client.0.smithi151.stdout:4/182: mkdir d0/d11/d19/d31/d34/d3a 0 2024-04-16T02:35:40.091 INFO:tasks.workunit.client.0.smithi151.stdout:4/183: readlink d0/d11/l14 0 2024-04-16T02:35:40.093 INFO:tasks.workunit.client.0.smithi151.stdout:9/113: creat d1/d3/d6/d1d/d22/d2c/f2d x:0 0 0 2024-04-16T02:35:40.097 INFO:tasks.workunit.client.0.smithi151.stdout:6/271: dwrite d2/d5/df/f3c [0,4194304] 0 2024-04-16T02:35:40.097 INFO:tasks.workunit.client.1.smithi153.stdout:5/668: rename d8/l47 to d8/d41/dbd/d83/ld4 0 2024-04-16T02:35:40.099 INFO:tasks.workunit.client.0.smithi151.stdout:9/114: unlink d1/d3/d10/l19 0 2024-04-16T02:35:40.099 INFO:tasks.workunit.client.1.smithi153.stdout:2/565: creat d9/d2b/d5c/da2/fc1 x:0 0 0 2024-04-16T02:35:40.100 INFO:tasks.workunit.client.0.smithi151.stdout:4/184: mknod d0/d11/d19/d31/d34/d3a/c3b 0 2024-04-16T02:35:40.100 INFO:tasks.workunit.client.1.smithi153.stdout:2/566: truncate d9/d2b/d5c/f80 5180250 0 2024-04-16T02:35:40.103 INFO:tasks.workunit.client.1.smithi153.stdout:0/701: symlink dd/d14/d51/d7d/ldb 0 2024-04-16T02:35:40.104 INFO:tasks.workunit.client.0.smithi151.stdout:9/115: creat d1/f2e x:0 0 0 2024-04-16T02:35:40.107 INFO:tasks.workunit.client.0.smithi151.stdout:4/185: creat d0/f3c x:0 0 0 2024-04-16T02:35:40.107 INFO:tasks.workunit.client.1.smithi153.stdout:4/682: link d3/d9/db/de/fa3 d3/d9/d24/d75/dc5/ff2 0 2024-04-16T02:35:40.107 INFO:tasks.workunit.client.0.smithi151.stdout:6/272: dwrite d2/d1f/f59 [0,4194304] 0 2024-04-16T02:35:40.107 INFO:tasks.workunit.client.0.smithi151.stdout:9/116: symlink d1/d3/d6/d17/l2f 0 2024-04-16T02:35:40.109 INFO:tasks.workunit.client.0.smithi151.stdout:9/117: fdatasync d1/d3/fa 0 2024-04-16T02:35:40.111 INFO:tasks.workunit.client.0.smithi151.stdout:9/118: read d1/d3/d6/d1d/d22/f23 [331792,65432] 0 2024-04-16T02:35:40.113 INFO:tasks.workunit.client.0.smithi151.stdout:9/119: write d1/d3/f1e [670396,105779] 0 2024-04-16T02:35:40.113 INFO:tasks.workunit.client.1.smithi153.stdout:4/683: rename d3/d9/d79 to d3/d9/db/d45/df3 0 2024-04-16T02:35:40.113 INFO:tasks.workunit.client.0.smithi151.stdout:6/273: creat d2/d5/df/d23/d31/d52/f5b x:0 0 0 2024-04-16T02:35:40.114 INFO:tasks.workunit.client.1.smithi153.stdout:0/702: symlink dd/d14/ldc 0 2024-04-16T02:35:40.115 INFO:tasks.workunit.client.0.smithi151.stdout:4/186: creat d0/d11/f3d x:0 0 0 2024-04-16T02:35:40.116 INFO:tasks.workunit.client.1.smithi153.stdout:4/684: symlink d3/d9/db/d84/lf4 0 2024-04-16T02:35:40.118 INFO:tasks.workunit.client.1.smithi153.stdout:0/703: mkdir dd/d14/d51/ddd 0 2024-04-16T02:35:40.119 INFO:tasks.workunit.client.1.smithi153.stdout:0/704: readlink dd/d14/d51/d9e/laa 0 2024-04-16T02:35:40.119 INFO:tasks.workunit.client.1.smithi153.stdout:0/705: dread - dd/d26/fcb zero size 2024-04-16T02:35:40.120 INFO:tasks.workunit.client.1.smithi153.stdout:4/685: mknod d3/d9/d24/d75/d9e/cf5 0 2024-04-16T02:35:40.120 INFO:tasks.workunit.client.1.smithi153.stdout:4/686: fdatasync d3/f73 0 2024-04-16T02:35:40.123 INFO:tasks.workunit.client.0.smithi151.stdout:6/274: dwrite d2/d5/df/d23/d31/d52/f5b [0,4194304] 0 2024-04-16T02:35:40.123 INFO:tasks.workunit.client.0.smithi151.stdout:4/187: dread d0/f15 [0,4194304] 0 2024-04-16T02:35:40.124 INFO:tasks.workunit.client.1.smithi153.stdout:8/534: sync 2024-04-16T02:35:40.124 INFO:tasks.workunit.client.0.smithi151.stdout:4/188: chown d0/d11/f26 414039 1 2024-04-16T02:35:40.126 INFO:tasks.workunit.client.0.smithi151.stdout:6/275: creat d2/d13/f5c x:0 0 0 2024-04-16T02:35:40.127 INFO:tasks.workunit.client.1.smithi153.stdout:0/706: link dd/f47 dd/d32/fde 0 2024-04-16T02:35:40.129 INFO:tasks.workunit.client.0.smithi151.stdout:6/276: mknod d2/d5/df/d23/d56/c5d 0 2024-04-16T02:35:40.130 INFO:tasks.workunit.client.0.smithi151.stdout:6/277: chown d2/d13/d3a/f54 787634 1 2024-04-16T02:35:40.131 INFO:tasks.workunit.client.1.smithi153.stdout:8/535: rename d2/d13/d3a/l5d to d2/d93/d97/d9c/d27/d5e/d63/lb4 0 2024-04-16T02:35:40.131 INFO:tasks.workunit.client.0.smithi151.stdout:4/189: rename d0/d17/l33 to d0/l3e 0 2024-04-16T02:35:40.131 INFO:tasks.workunit.client.0.smithi151.stdout:6/278: stat d2/d5/df/d23/c2a 0 2024-04-16T02:35:40.132 INFO:tasks.workunit.client.1.smithi153.stdout:4/687: link d3/d9/db/de/d2c/d38/caa d3/d9/db/d17/d3f/d6b/cf6 0 2024-04-16T02:35:40.132 INFO:tasks.workunit.client.0.smithi151.stdout:6/279: write d2/d3d/f55 [223588,104557] 0 2024-04-16T02:35:40.133 INFO:tasks.workunit.client.1.smithi153.stdout:4/688: truncate d3/d9/db/d45/df3/fa6 5134997 0 2024-04-16T02:35:40.134 INFO:tasks.workunit.client.1.smithi153.stdout:8/536: creat d2/d13/d96/fb5 x:0 0 0 2024-04-16T02:35:40.135 INFO:tasks.workunit.client.1.smithi153.stdout:4/689: dread - d3/d9/d24/d75/dc5/ff2 zero size 2024-04-16T02:35:40.136 INFO:tasks.workunit.client.0.smithi151.stdout:6/280: creat d2/d5/df/d20/d4f/f5e x:0 0 0 2024-04-16T02:35:40.136 INFO:tasks.workunit.client.1.smithi153.stdout:8/537: mknod d2/d1c/cb6 0 2024-04-16T02:35:40.136 INFO:tasks.workunit.client.0.smithi151.stdout:6/281: chown d2/d5/l39 37518469 1 2024-04-16T02:35:40.137 INFO:tasks.workunit.client.1.smithi153.stdout:8/538: chown d2/d93/d97/d9c/d27/d5e/fa3 233495 1 2024-04-16T02:35:40.138 INFO:tasks.workunit.client.1.smithi153.stdout:8/539: read - d2/d93/d97/d9c/d27/d5e/d63/fb1 zero size 2024-04-16T02:35:40.139 INFO:tasks.workunit.client.0.smithi151.stdout:6/282: unlink d2/d5/df/d23/d31/d52/f5b 0 2024-04-16T02:35:40.142 INFO:tasks.workunit.client.0.smithi151.stdout:6/283: truncate d2/d5/df/d23/d49/f57 140620 0 2024-04-16T02:35:40.143 INFO:tasks.workunit.client.1.smithi153.stdout:4/690: mkdir d3/d9/d24/d75/d9e/df7 0 2024-04-16T02:35:40.143 INFO:tasks.workunit.client.1.smithi153.stdout:4/691: chown d3/d9/c15 30 1 2024-04-16T02:35:40.144 INFO:tasks.workunit.client.1.smithi153.stdout:2/567: rmdir d9/d2b 39 2024-04-16T02:35:40.146 INFO:tasks.workunit.client.1.smithi153.stdout:8/540: mkdir d2/d93/d97/d9c/d27/d2d/d47/d60/d67/db7 0 2024-04-16T02:35:40.147 INFO:tasks.workunit.client.1.smithi153.stdout:8/541: dread - d2/d93/d97/d9c/d27/d2d/d47/d77/d8e/f95 zero size 2024-04-16T02:35:40.148 INFO:tasks.workunit.client.1.smithi153.stdout:4/692: creat d3/d9/d24/d3e/ff8 x:0 0 0 2024-04-16T02:35:40.148 INFO:tasks.workunit.client.1.smithi153.stdout:8/542: dread - d2/d93/d97/d9c/f7c zero size 2024-04-16T02:35:40.149 INFO:tasks.workunit.client.1.smithi153.stdout:4/693: write d3/d9/db/d45/df3/f92 [1012970,92183] 0 2024-04-16T02:35:40.149 INFO:tasks.workunit.client.1.smithi153.stdout:0/707: creat dd/d14/d19/d55/fdf x:0 0 0 2024-04-16T02:35:40.150 INFO:tasks.workunit.client.0.smithi151.stdout:2/179: chown de/d11/f18 0 1 2024-04-16T02:35:40.152 INFO:tasks.workunit.client.1.smithi153.stdout:2/568: chown d9/d2b/d1c/f28 78817 1 2024-04-16T02:35:40.154 INFO:tasks.workunit.client.0.smithi151.stdout:6/284: truncate d2/d5/f27 2443941 0 2024-04-16T02:35:40.156 INFO:tasks.workunit.client.1.smithi153.stdout:0/708: dwrite dd/d26/f5b [4194304,4194304] 0 2024-04-16T02:35:40.157 INFO:tasks.workunit.client.1.smithi153.stdout:5/669: symlink d8/df/d84/da4/ld5 0 2024-04-16T02:35:40.157 INFO:tasks.workunit.client.0.smithi151.stdout:2/180: dwrite de/d15/d1e/d28/f38 [0,4194304] 0 2024-04-16T02:35:40.161 INFO:tasks.workunit.client.1.smithi153.stdout:4/694: mknod d3/d9/db/cf9 0 2024-04-16T02:35:40.162 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:39 smithi151 ceph-mon[29003]: pgmap v17: 65 pgs: 65 active+clean; 1.2 GiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 25 MiB/s rd, 166 MiB/s wr, 420 op/s 2024-04-16T02:35:40.164 INFO:tasks.workunit.client.1.smithi153.stdout:0/709: dwrite dd/d14/da6/f36 [0,4194304] 0 2024-04-16T02:35:40.166 INFO:tasks.workunit.client.0.smithi151.stdout:6/285: dwrite d2/d5/df/d20/d4f/f5e [0,4194304] 0 2024-04-16T02:35:40.167 INFO:tasks.workunit.client.0.smithi151.stdout:6/286: chown d2/d5/l37 992713 1 2024-04-16T02:35:40.169 INFO:tasks.workunit.client.0.smithi151.stdout:6/287: write d2/d5/f6 [884402,87653] 0 2024-04-16T02:35:40.170 INFO:tasks.workunit.client.1.smithi153.stdout:8/543: creat d2/d13/fb8 x:0 0 0 2024-04-16T02:35:40.172 INFO:tasks.workunit.client.1.smithi153.stdout:4/695: symlink d3/d9/db/d17/d3f/d6b/lfa 0 2024-04-16T02:35:40.174 INFO:tasks.workunit.client.0.smithi151.stdout:6/288: mkdir d2/d13/d5f 0 2024-04-16T02:35:40.174 INFO:tasks.workunit.client.1.smithi153.stdout:5/670: creat d8/d41/d90/d9d/fd6 x:0 0 0 2024-04-16T02:35:40.176 INFO:tasks.workunit.client.0.smithi151.stdout:4/190: dread d0/f10 [0,4194304] 0 2024-04-16T02:35:40.176 INFO:tasks.workunit.client.0.smithi151.stdout:6/289: chown d2/d5/l1e 81734 1 2024-04-16T02:35:40.177 INFO:tasks.workunit.client.1.smithi153.stdout:5/671: write d8/df/d2e/f57 [839258,94050] 0 2024-04-16T02:35:40.177 INFO:tasks.workunit.client.1.smithi153.stdout:0/710: mkdir dd/d14/da6/de0 0 2024-04-16T02:35:40.179 INFO:tasks.workunit.client.1.smithi153.stdout:5/672: read d8/df/f88 [2333909,9861] 0 2024-04-16T02:35:40.182 INFO:tasks.workunit.client.0.smithi151.stdout:4/191: rmdir d0/d17 39 2024-04-16T02:35:40.183 INFO:tasks.workunit.client.0.smithi151.stdout:4/192: truncate d0/fd 2043988 0 2024-04-16T02:35:40.184 INFO:tasks.workunit.client.1.smithi153.stdout:4/696: dwrite d3/d9/db/de/d2c/d52/d85/f90 [0,4194304] 0 2024-04-16T02:35:40.184 INFO:tasks.workunit.client.0.smithi151.stdout:6/290: write d2/f1b [3320267,118658] 0 2024-04-16T02:35:40.185 INFO:tasks.workunit.client.0.smithi151.stdout:2/181: getdents de 0 2024-04-16T02:35:40.185 INFO:tasks.workunit.client.1.smithi153.stdout:0/711: creat dd/d14/dca/dcd/fe1 x:0 0 0 2024-04-16T02:35:40.186 INFO:tasks.workunit.client.0.smithi151.stdout:2/182: chown de/d11 4420 1 2024-04-16T02:35:40.186 INFO:tasks.workunit.client.1.smithi153.stdout:0/712: dread - dd/d32/f85 zero size 2024-04-16T02:35:40.189 INFO:tasks.workunit.client.0.smithi151.stdout:6/291: unlink d2/ld 0 2024-04-16T02:35:40.192 INFO:tasks.workunit.client.0.smithi151.stdout:2/183: creat de/d15/f39 x:0 0 0 2024-04-16T02:35:40.192 INFO:tasks.workunit.client.1.smithi153.stdout:0/713: symlink dd/d28/le2 0 2024-04-16T02:35:40.193 INFO:tasks.workunit.client.0.smithi151.stdout:4/193: rename d0/d20/f24 to d0/d11/d19/d31/d34/f3f 0 2024-04-16T02:35:40.193 INFO:tasks.workunit.client.1.smithi153.stdout:0/714: chown dd/d14/d51/d54/d65/d68/l8c 260575 1 2024-04-16T02:35:40.195 INFO:tasks.workunit.client.1.smithi153.stdout:4/697: getdents d3/d9/db/d45/df3 0 2024-04-16T02:35:40.196 INFO:tasks.workunit.client.0.smithi151.stdout:4/194: mknod d0/d11/d19/c40 0 2024-04-16T02:35:40.198 INFO:tasks.workunit.client.0.smithi151.stdout:6/292: dwrite d2/d1f/f26 [0,4194304] 0 2024-04-16T02:35:40.199 INFO:tasks.workunit.client.1.smithi153.stdout:8/544: dread d2/d13/f62 [0,4194304] 0 2024-04-16T02:35:40.200 INFO:tasks.workunit.client.1.smithi153.stdout:0/715: mkdir dd/d14/d19/d77/de3 0 2024-04-16T02:35:40.200 INFO:tasks.workunit.client.0.smithi151.stdout:4/195: unlink d0/d20/f2f 0 2024-04-16T02:35:40.201 INFO:tasks.workunit.client.1.smithi153.stdout:4/698: getdents d3/d9/d24 0 2024-04-16T02:35:40.203 INFO:tasks.workunit.client.1.smithi153.stdout:4/699: chown d3/d9/db/de/f3b 241810375 1 2024-04-16T02:35:40.204 INFO:tasks.workunit.client.1.smithi153.stdout:0/716: write dd/d14/d19/fae [680460,60936] 0 2024-04-16T02:35:40.205 INFO:tasks.workunit.client.0.smithi151.stdout:6/293: dread d2/d5/df/d20/f2b [0,4194304] 0 2024-04-16T02:35:40.205 INFO:tasks.workunit.client.1.smithi153.stdout:4/700: truncate d3/d9/db/de/fcd 793442 0 2024-04-16T02:35:40.206 INFO:tasks.workunit.client.1.smithi153.stdout:8/545: creat d2/d93/d97/d9c/d27/fb9 x:0 0 0 2024-04-16T02:35:40.209 INFO:tasks.workunit.client.1.smithi153.stdout:8/546: chown d2/d13/c7f 7289265 1 2024-04-16T02:35:40.209 INFO:tasks.workunit.client.1.smithi153.stdout:8/547: fsync d2/d1c/f85 0 2024-04-16T02:35:40.210 INFO:tasks.workunit.client.1.smithi153.stdout:4/701: creat d3/d9/db/de/d2c/d52/d85/ffb x:0 0 0 2024-04-16T02:35:40.210 INFO:tasks.workunit.client.0.smithi151.stdout:6/294: link d2/d5/fc d2/d5/f60 0 2024-04-16T02:35:40.211 INFO:tasks.workunit.client.0.smithi151.stdout:6/295: fdatasync d2/d3d/f55 0 2024-04-16T02:35:40.213 INFO:tasks.workunit.client.0.smithi151.stdout:6/296: creat d2/d5/df/d23/d56/f61 x:0 0 0 2024-04-16T02:35:40.214 INFO:tasks.workunit.client.1.smithi153.stdout:4/702: creat d3/d9/db/d45/df3/ffc x:0 0 0 2024-04-16T02:35:40.215 INFO:tasks.workunit.client.1.smithi153.stdout:2/569: truncate d9/d24/d27/d2c/d41/d7a/fb2 3758541 0 2024-04-16T02:35:40.216 INFO:tasks.workunit.client.1.smithi153.stdout:4/703: chown d3/d9/db/d17/l33 58185015 1 2024-04-16T02:35:40.216 INFO:tasks.workunit.client.1.smithi153.stdout:5/673: dread d8/d1f/f73 [0,4194304] 0 2024-04-16T02:35:40.217 INFO:tasks.workunit.client.1.smithi153.stdout:8/548: mkdir d2/dba 0 2024-04-16T02:35:40.217 INFO:tasks.workunit.client.1.smithi153.stdout:5/674: stat d8/d41/dbd/d83/fc7 0 2024-04-16T02:35:40.218 INFO:tasks.workunit.client.0.smithi151.stdout:2/184: truncate de/d15/f2c 6118080 0 2024-04-16T02:35:40.218 INFO:tasks.workunit.client.1.smithi153.stdout:5/675: fsync d8/df/d84/fc5 0 2024-04-16T02:35:40.218 INFO:tasks.workunit.client.1.smithi153.stdout:9/626: sync 2024-04-16T02:35:40.219 INFO:tasks.workunit.client.1.smithi153.stdout:8/549: truncate d2/d93/d97/d9c/d27/f2a 2106710 0 2024-04-16T02:35:40.220 INFO:tasks.workunit.client.1.smithi153.stdout:9/627: chown d4/f27 6128664 1 2024-04-16T02:35:40.221 INFO:tasks.workunit.client.1.smithi153.stdout:2/570: rename d9/d2b/f14 to d9/d2b/d1c/d20/fc2 0 2024-04-16T02:35:40.222 INFO:tasks.workunit.client.1.smithi153.stdout:8/550: write d2/d1c/f85 [1089526,125105] 0 2024-04-16T02:35:40.224 INFO:tasks.workunit.client.1.smithi153.stdout:2/571: fdatasync d9/d24/d27/f68 0 2024-04-16T02:35:40.226 INFO:tasks.workunit.client.0.smithi151.stdout:2/185: dwrite de/d15/d1e/d28/f37 [0,4194304] 0 2024-04-16T02:35:40.228 INFO:tasks.workunit.client.1.smithi153.stdout:5/676: dwrite d8/df/d2e/f79 [0,4194304] 0 2024-04-16T02:35:40.231 INFO:tasks.workunit.client.0.smithi151.stdout:2/186: creat de/f3a x:0 0 0 2024-04-16T02:35:40.234 INFO:tasks.workunit.client.1.smithi153.stdout:8/551: creat d2/d93/d97/d9c/d27/d2d/d47/d77/d8e/fbb x:0 0 0 2024-04-16T02:35:40.235 INFO:tasks.workunit.client.1.smithi153.stdout:5/677: unlink d8/df/d28/l86 0 2024-04-16T02:35:40.239 INFO:tasks.workunit.client.1.smithi153.stdout:0/717: dread dd/d26/f5b [0,4194304] 0 2024-04-16T02:35:40.240 INFO:tasks.workunit.client.1.smithi153.stdout:0/718: fsync dd/ff 0 2024-04-16T02:35:40.240 INFO:tasks.workunit.client.1.smithi153.stdout:8/552: chown d2/de/l58 80766434 1 2024-04-16T02:35:40.245 INFO:tasks.workunit.client.1.smithi153.stdout:2/572: dwrite d9/d2b/d1c/d20/fc2 [0,4194304] 0 2024-04-16T02:35:40.245 INFO:tasks.workunit.client.1.smithi153.stdout:5/678: rename d8/f3a to d8/df/d2e/dd1/fd7 0 2024-04-16T02:35:40.246 INFO:tasks.workunit.client.1.smithi153.stdout:8/553: truncate d2/d93/f9a 394511 0 2024-04-16T02:35:40.248 INFO:tasks.workunit.client.1.smithi153.stdout:0/719: creat dd/d28/fe4 x:0 0 0 2024-04-16T02:35:40.248 INFO:tasks.workunit.client.1.smithi153.stdout:1/631: sync 2024-04-16T02:35:40.249 INFO:tasks.workunit.client.1.smithi153.stdout:6/587: sync 2024-04-16T02:35:40.249 INFO:tasks.workunit.client.1.smithi153.stdout:3/571: sync 2024-04-16T02:35:40.249 INFO:tasks.workunit.client.1.smithi153.stdout:7/568: sync 2024-04-16T02:35:40.251 INFO:tasks.workunit.client.1.smithi153.stdout:4/704: dwrite d3/d9/d24/d75/dc5/ff2 [0,4194304] 0 2024-04-16T02:35:40.252 INFO:tasks.workunit.client.1.smithi153.stdout:0/720: stat dd/d26/d44 0 2024-04-16T02:35:40.253 INFO:tasks.workunit.client.1.smithi153.stdout:2/573: symlink d9/d2b/d18/lc3 0 2024-04-16T02:35:40.255 INFO:tasks.workunit.client.1.smithi153.stdout:6/588: chown dc/d59/d63/f66 1 1 2024-04-16T02:35:40.256 INFO:tasks.workunit.client.1.smithi153.stdout:1/632: write d4/dd/d18/d29/f50 [2478471,129416] 0 2024-04-16T02:35:40.257 INFO:tasks.workunit.client.1.smithi153.stdout:9/628: dwrite d4/d6/dc/d2d/fa2 [0,4194304] 0 2024-04-16T02:35:40.259 INFO:tasks.workunit.client.0.smithi151.stdout:2/187: fdatasync de/d15/f1a 0 2024-04-16T02:35:40.259 INFO:tasks.workunit.client.1.smithi153.stdout:4/705: truncate d3/f6d 1244165 0 2024-04-16T02:35:40.261 INFO:tasks.workunit.client.1.smithi153.stdout:9/629: chown d4/d6/dc/d2d/da0/d40/f5e 0 1 2024-04-16T02:35:40.263 INFO:tasks.workunit.client.0.smithi151.stdout:2/188: mknod de/c3b 0 2024-04-16T02:35:40.264 INFO:tasks.workunit.client.1.smithi153.stdout:4/706: truncate d3/d9/db/de/d2c/d98/fca 4597518 0 2024-04-16T02:35:40.265 INFO:tasks.workunit.client.1.smithi153.stdout:4/707: dread - d3/d9/db/de/d2c/f8a zero size 2024-04-16T02:35:40.266 INFO:tasks.workunit.client.1.smithi153.stdout:7/569: mkdir d0/d4/db/d2b/d53/d59/dc3 0 2024-04-16T02:35:40.266 INFO:tasks.workunit.client.1.smithi153.stdout:8/554: mknod d2/d93/d97/d9c/d27/d2d/d47/d60/cbc 0 2024-04-16T02:35:40.266 INFO:tasks.workunit.client.0.smithi151.stdout:2/189: creat de/d15/d1e/f3c x:0 0 0 2024-04-16T02:35:40.267 INFO:tasks.workunit.client.1.smithi153.stdout:1/633: fsync d4/f5e 0 2024-04-16T02:35:40.268 INFO:tasks.workunit.client.1.smithi153.stdout:6/589: rmdir dc/d47/d57/d77/d15/dac 39 2024-04-16T02:35:40.270 INFO:tasks.workunit.client.0.smithi151.stdout:2/190: unlink de/d11/f2e 0 2024-04-16T02:35:40.270 INFO:tasks.workunit.client.1.smithi153.stdout:8/555: write d2/d93/d97/d9c/d27/d5e/d63/f8f [952163,30752] 0 2024-04-16T02:35:40.271 INFO:tasks.workunit.client.0.smithi151.stdout:2/191: chown de/d15/d1e/d28/f32 168 1 2024-04-16T02:35:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:39 smithi153 ceph-mon[36747]: pgmap v17: 65 pgs: 65 active+clean; 1.2 GiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 25 MiB/s rd, 166 MiB/s wr, 420 op/s 2024-04-16T02:35:40.272 INFO:tasks.workunit.client.0.smithi151.stdout:2/192: truncate de/d15/f34 380253 0 2024-04-16T02:35:40.273 INFO:tasks.workunit.client.1.smithi153.stdout:2/574: symlink d9/d24/d34/d99/lc4 0 2024-04-16T02:35:40.274 INFO:tasks.workunit.client.0.smithi151.stdout:2/193: dread de/d15/f34 [0,4194304] 0 2024-04-16T02:35:40.275 INFO:tasks.workunit.client.1.smithi153.stdout:1/634: symlink d4/dd/d18/d29/d71/ld7 0 2024-04-16T02:35:40.275 INFO:tasks.workunit.client.1.smithi153.stdout:2/575: chown d9/d2b/d1c/d20/d88/f9b 22865812 1 2024-04-16T02:35:40.276 INFO:tasks.workunit.client.1.smithi153.stdout:4/708: dwrite d3/d9/db/de/fa3 [4194304,4194304] 0 2024-04-16T02:35:40.276 INFO:tasks.workunit.client.1.smithi153.stdout:9/630: mkdir d4/d6/dc/d2d/d64/db9/dc1 0 2024-04-16T02:35:40.278 INFO:tasks.workunit.client.1.smithi153.stdout:1/635: chown d4/dd/d18/d29/l45 172 1 2024-04-16T02:35:40.278 INFO:tasks.workunit.client.1.smithi153.stdout:7/570: symlink d0/d4/db/d2b/d8c/lc4 0 2024-04-16T02:35:40.279 INFO:tasks.workunit.client.1.smithi153.stdout:6/590: mkdir dc/d47/d57/d77/dc4/dc9 0 2024-04-16T02:35:40.279 INFO:tasks.workunit.client.1.smithi153.stdout:8/556: mknod d2/d13/cbd 0 2024-04-16T02:35:40.282 INFO:tasks.workunit.client.1.smithi153.stdout:9/631: chown d4/d6/dc/d2d/da0/l5c 87188 1 2024-04-16T02:35:40.284 INFO:tasks.workunit.client.1.smithi153.stdout:4/709: fdatasync d3/d9/d24/d3e/f5a 0 2024-04-16T02:35:40.284 INFO:tasks.workunit.client.1.smithi153.stdout:6/591: write dc/d6b/f6f [2530614,79014] 0 2024-04-16T02:35:40.286 INFO:tasks.workunit.client.1.smithi153.stdout:0/721: dread dd/d32/fde [0,4194304] 0 2024-04-16T02:35:40.288 INFO:tasks.workunit.client.0.smithi151.stdout:2/194: truncate de/d15/d1e/d28/f37 3772815 0 2024-04-16T02:35:40.288 INFO:tasks.workunit.client.1.smithi153.stdout:2/576: rename d9/d2b/d18/d3f/d40/c57 to d9/d24/d27/d2c/d5d/da3/dc0/cc5 0 2024-04-16T02:35:40.289 INFO:tasks.workunit.client.1.smithi153.stdout:4/710: readlink d3/d9/db/de/d2c/d38/l93 0 2024-04-16T02:35:40.290 INFO:tasks.workunit.client.1.smithi153.stdout:9/632: rmdir d4/d6/dc/d2d/d64/db9 39 2024-04-16T02:35:40.293 INFO:tasks.workunit.client.1.smithi153.stdout:6/592: truncate dc/d59/d63/f95 8663280 0 2024-04-16T02:35:40.294 INFO:tasks.workunit.client.0.smithi151.stdout:2/195: write de/d11/f18 [3641728,123177] 0 2024-04-16T02:35:40.295 INFO:tasks.workunit.client.1.smithi153.stdout:7/571: dread d0/d4/db/d2b/d43/d6d/f6f [0,4194304] 0 2024-04-16T02:35:40.299 INFO:tasks.workunit.client.1.smithi153.stdout:0/722: dread dd/d14/d19/fae [0,4194304] 0 2024-04-16T02:35:40.300 INFO:tasks.workunit.client.0.smithi151.stdout:2/196: dwrite de/d15/f36 [0,4194304] 0 2024-04-16T02:35:40.301 INFO:tasks.workunit.client.1.smithi153.stdout:2/577: rmdir d9/d24/d34/d60 39 2024-04-16T02:35:40.301 INFO:tasks.workunit.client.1.smithi153.stdout:8/557: dwrite d2/d1c/f36 [0,4194304] 0 2024-04-16T02:35:40.301 INFO:tasks.workunit.client.0.smithi151.stdout:2/197: fdatasync de/d15/f1a 0 2024-04-16T02:35:40.302 INFO:tasks.workunit.client.1.smithi153.stdout:3/572: dwrite d6/d10/d22/d4d/da0/fac [0,4194304] 0 2024-04-16T02:35:40.304 INFO:tasks.workunit.client.0.smithi151.stdout:2/198: fdatasync de/d15/f1c 0 2024-04-16T02:35:40.306 INFO:tasks.workunit.client.1.smithi153.stdout:6/593: write dc/d47/d57/d77/d15/f23 [1106697,62973] 0 2024-04-16T02:35:40.308 INFO:tasks.workunit.client.0.smithi151.stdout:2/199: unlink de/d11/l26 0 2024-04-16T02:35:40.308 INFO:tasks.workunit.client.1.smithi153.stdout:1/636: truncate d4/dd/d18/f51 925752 0 2024-04-16T02:35:40.310 INFO:tasks.workunit.client.1.smithi153.stdout:2/578: dread - d9/d24/d27/f92 zero size 2024-04-16T02:35:40.311 INFO:tasks.workunit.client.1.smithi153.stdout:8/558: write d2/d13/d3a/f4c [931981,35417] 0 2024-04-16T02:35:40.312 INFO:tasks.workunit.client.1.smithi153.stdout:4/711: rename d3/d9/db/d84/de0 to d3/d9/db/d17/d28/dc8/de4/dfd 0 2024-04-16T02:35:40.313 INFO:tasks.workunit.client.1.smithi153.stdout:8/559: chown d2/d92 1166562 1 2024-04-16T02:35:40.313 INFO:tasks.workunit.client.1.smithi153.stdout:6/594: chown dc/d1a/d2d/f44 220205070 1 2024-04-16T02:35:40.314 INFO:tasks.workunit.client.1.smithi153.stdout:9/633: dwrite d4/d6/dc/d2d/d79/f8f [0,4194304] 0 2024-04-16T02:35:40.316 INFO:tasks.workunit.client.0.smithi151.stdout:2/200: dwrite de/d11/f2b [0,4194304] 0 2024-04-16T02:35:40.319 INFO:tasks.workunit.client.1.smithi153.stdout:0/723: truncate dd/f53 1738701 0 2024-04-16T02:35:40.321 INFO:tasks.workunit.client.0.smithi151.stdout:2/201: mkdir de/d3d 0 2024-04-16T02:35:40.322 INFO:tasks.workunit.client.0.smithi151.stdout:2/202: chown de/c13 464238 1 2024-04-16T02:35:40.323 INFO:tasks.workunit.client.0.smithi151.stdout:2/203: chown de/d15/f34 113673473 1 2024-04-16T02:35:40.324 INFO:tasks.workunit.client.1.smithi153.stdout:7/572: link d0/d13/fbc d0/d4/db/d2b/d53/d59/d76/fc5 0 2024-04-16T02:35:40.325 INFO:tasks.workunit.client.0.smithi151.stdout:2/204: creat de/f3e x:0 0 0 2024-04-16T02:35:40.326 INFO:tasks.workunit.client.1.smithi153.stdout:2/579: unlink d9/d2b/c52 0 2024-04-16T02:35:40.326 INFO:tasks.workunit.client.1.smithi153.stdout:8/560: rmdir d2/d13/d3a/d5b 39 2024-04-16T02:35:40.327 INFO:tasks.workunit.client.0.smithi151.stdout:2/205: chown de/f14 1 1 2024-04-16T02:35:40.329 INFO:tasks.workunit.client.1.smithi153.stdout:3/573: dread d6/d10/d5b/f82 [0,4194304] 0 2024-04-16T02:35:40.329 INFO:tasks.workunit.client.1.smithi153.stdout:1/637: dwrite d4/dd/d20/f5b [0,4194304] 0 2024-04-16T02:35:40.329 INFO:tasks.workunit.client.1.smithi153.stdout:9/634: symlink d4/d6/dc/d46/lc2 0 2024-04-16T02:35:40.334 INFO:tasks.workunit.client.0.smithi151.stdout:2/206: dwrite de/d15/f1a [0,4194304] 0 2024-04-16T02:35:40.334 INFO:tasks.workunit.client.1.smithi153.stdout:2/580: rmdir d9/d71/d8d 39 2024-04-16T02:35:40.335 INFO:tasks.workunit.client.0.smithi151.stdout:2/207: read - de/f3e zero size 2024-04-16T02:35:40.338 INFO:tasks.workunit.client.0.smithi151.stdout:2/208: unlink l5 0 2024-04-16T02:35:40.339 INFO:tasks.workunit.client.0.smithi151.stdout:2/209: write de/d15/f39 [556037,47828] 0 2024-04-16T02:35:40.340 INFO:tasks.workunit.client.1.smithi153.stdout:0/724: symlink dd/d14/da6/le5 0 2024-04-16T02:35:40.341 INFO:tasks.workunit.client.0.smithi151.stdout:2/210: creat de/d15/d1e/f3f x:0 0 0 2024-04-16T02:35:40.342 INFO:tasks.workunit.client.1.smithi153.stdout:6/595: creat dc/d47/d57/d77/dc4/dc9/fca x:0 0 0 2024-04-16T02:35:40.343 INFO:tasks.workunit.client.1.smithi153.stdout:8/561: write d2/d93/d97/d9c/d27/d2d/d47/f6c [2697151,110693] 0 2024-04-16T02:35:40.343 INFO:tasks.workunit.client.0.smithi151.stdout:2/211: read de/d15/d1e/f24 [2166766,118083] 0 2024-04-16T02:35:40.344 INFO:tasks.workunit.client.0.smithi151.stdout:2/212: truncate de/f3e 679515 0 2024-04-16T02:35:40.344 INFO:tasks.workunit.client.1.smithi153.stdout:7/573: read d0/d4/db/d2b/f28 [726150,106295] 0 2024-04-16T02:35:40.346 INFO:tasks.workunit.client.1.smithi153.stdout:1/638: rename d4/dd/d13/d26/db7 to d4/dd/d18/d6c/dd8 0 2024-04-16T02:35:40.348 INFO:tasks.workunit.client.0.smithi151.stdout:2/213: rename de/d15/f2c to de/f40 0 2024-04-16T02:35:40.349 INFO:tasks.workunit.client.1.smithi153.stdout:2/581: readlink d9/d2b/d18/lc3 0 2024-04-16T02:35:40.349 INFO:tasks.workunit.client.1.smithi153.stdout:9/635: dwrite d4/d33/f4f [0,4194304] 0 2024-04-16T02:35:40.349 INFO:tasks.workunit.client.1.smithi153.stdout:6/596: chown dc/d47/f4a 17695560 1 2024-04-16T02:35:40.350 INFO:tasks.workunit.client.1.smithi153.stdout:4/712: dwrite d3/ff [4194304,4194304] 0 2024-04-16T02:35:40.354 INFO:tasks.workunit.client.1.smithi153.stdout:1/639: stat d4/dd/d13/d23/f6d 0 2024-04-16T02:35:40.355 INFO:tasks.workunit.client.1.smithi153.stdout:3/574: rename d6/d10/d3b/d59/d7c/l90 to d6/d10/d22/d4d/da0/lc7 0 2024-04-16T02:35:40.358 INFO:tasks.workunit.client.1.smithi153.stdout:0/725: creat dd/d14/da6/de0/fe6 x:0 0 0 2024-04-16T02:35:40.359 INFO:tasks.workunit.client.1.smithi153.stdout:2/582: creat d9/d2b/d5c/da2/fc6 x:0 0 0 2024-04-16T02:35:40.360 INFO:tasks.workunit.client.1.smithi153.stdout:8/562: creat d2/d13/fbe x:0 0 0 2024-04-16T02:35:40.362 INFO:tasks.workunit.client.1.smithi153.stdout:4/713: unlink d3/d9/db/de/fcd 0 2024-04-16T02:35:40.366 INFO:tasks.workunit.client.1.smithi153.stdout:3/575: symlink d6/daf/lc8 0 2024-04-16T02:35:40.367 INFO:tasks.workunit.client.1.smithi153.stdout:0/726: mkdir dd/d14/d19/d77/d87/de7 0 2024-04-16T02:35:40.367 INFO:tasks.workunit.client.1.smithi153.stdout:2/583: fdatasync d9/d24/d27/d2c/d41/f4c 0 2024-04-16T02:35:40.368 INFO:tasks.workunit.client.1.smithi153.stdout:5/679: sync 2024-04-16T02:35:40.370 INFO:tasks.workunit.client.1.smithi153.stdout:9/636: dwrite d4/d6/d1b/fbe [0,4194304] 0 2024-04-16T02:35:40.371 INFO:tasks.workunit.client.1.smithi153.stdout:1/640: dwrite d4/dd/d20/f72 [0,4194304] 0 2024-04-16T02:35:40.375 INFO:tasks.workunit.client.1.smithi153.stdout:6/597: link dc/d1a/l71 dc/d47/d57/d77/d15/lcb 0 2024-04-16T02:35:40.375 INFO:tasks.workunit.client.1.smithi153.stdout:0/727: readlink dd/d14/d19/d20/d4a/l97 0 2024-04-16T02:35:40.383 INFO:tasks.workunit.client.1.smithi153.stdout:4/714: dwrite d3/d9/db/d45/df3/fd3 [0,4194304] 0 2024-04-16T02:35:40.383 INFO:tasks.workunit.client.1.smithi153.stdout:7/574: rmdir d0/d4/db/d2b/d53/d59 39 2024-04-16T02:35:40.384 INFO:tasks.workunit.client.1.smithi153.stdout:9/637: mknod d4/d6/d1b/d2a/cc3 0 2024-04-16T02:35:40.385 INFO:tasks.workunit.client.1.smithi153.stdout:1/641: creat d4/dd/d13/d23/fd9 x:0 0 0 2024-04-16T02:35:40.390 INFO:tasks.workunit.client.1.smithi153.stdout:6/598: creat dc/d47/d57/d77/d7a/d97/fcc x:0 0 0 2024-04-16T02:35:40.392 INFO:tasks.workunit.client.1.smithi153.stdout:0/728: truncate dd/d32/fde 3422538 0 2024-04-16T02:35:40.396 INFO:tasks.workunit.client.1.smithi153.stdout:4/715: dwrite d3/da5/dd2/fee [0,4194304] 0 2024-04-16T02:35:40.400 INFO:tasks.workunit.client.1.smithi153.stdout:1/642: symlink d4/dd/d13/d8d/d95/lda 0 2024-04-16T02:35:40.401 INFO:tasks.workunit.client.1.smithi153.stdout:6/599: dread f4 [0,4194304] 0 2024-04-16T02:35:40.401 INFO:tasks.workunit.client.1.smithi153.stdout:9/638: symlink d4/d6/lc4 0 2024-04-16T02:35:40.402 INFO:tasks.workunit.client.1.smithi153.stdout:4/716: chown d3/d9/db/de/d2c/d98/fd7 47880683 1 2024-04-16T02:35:40.403 INFO:tasks.workunit.client.1.smithi153.stdout:6/600: stat dc/d47/d57/c90 0 2024-04-16T02:35:40.404 INFO:tasks.workunit.client.1.smithi153.stdout:0/729: dwrite dd/d26/fcb [0,4194304] 0 2024-04-16T02:35:40.406 INFO:tasks.workunit.client.1.smithi153.stdout:7/575: mkdir d0/d4/db/d87/dc0/dc6 0 2024-04-16T02:35:40.406 INFO:tasks.workunit.client.0.smithi151.stdout:2/214: dwrite de/d15/f1c [0,4194304] 0 2024-04-16T02:35:40.412 INFO:tasks.workunit.client.0.smithi151.stdout:2/215: rename de/d15/d1e/d28 to de/d11/d41 0 2024-04-16T02:35:40.414 INFO:tasks.workunit.client.1.smithi153.stdout:6/601: readlink dc/d59/d5b/lc6 0 2024-04-16T02:35:40.417 INFO:tasks.workunit.client.1.smithi153.stdout:6/602: chown dc/d47/d57/d77/d15/c5d 4 1 2024-04-16T02:35:40.419 INFO:tasks.workunit.client.1.smithi153.stdout:1/643: mknod d4/dd/d18/d29/cdb 0 2024-04-16T02:35:40.420 INFO:tasks.workunit.client.1.smithi153.stdout:0/730: read dd/d14/d19/f46 [111529,72954] 0 2024-04-16T02:35:40.421 INFO:tasks.workunit.client.1.smithi153.stdout:3/576: dwrite d6/d10/d22/d43/f69 [0,4194304] 0 2024-04-16T02:35:40.421 INFO:tasks.workunit.client.1.smithi153.stdout:8/563: truncate d2/d93/d97/d9c/d27/d2d/d47/f6c 1419233 0 2024-04-16T02:35:40.424 INFO:tasks.workunit.client.1.smithi153.stdout:5/680: dread d8/df/f97 [0,4194304] 0 2024-04-16T02:35:40.424 INFO:tasks.workunit.client.1.smithi153.stdout:6/603: rmdir dc/d6b 39 2024-04-16T02:35:40.426 INFO:tasks.workunit.client.1.smithi153.stdout:7/576: symlink d0/d4/db/d2b/d43/lc7 0 2024-04-16T02:35:40.427 INFO:tasks.workunit.client.1.smithi153.stdout:5/681: write d8/df/d2f/f60 [716710,28814] 0 2024-04-16T02:35:40.427 INFO:tasks.workunit.client.1.smithi153.stdout:1/644: rmdir d4/dd/d13/d8d/d95/dc7 39 2024-04-16T02:35:40.428 INFO:tasks.workunit.client.1.smithi153.stdout:1/645: chown d4/dd/d13/d23/fd3 115792846 1 2024-04-16T02:35:40.430 INFO:tasks.workunit.client.1.smithi153.stdout:3/577: symlink d6/d10/d22/d4d/dbb/lc9 0 2024-04-16T02:35:40.431 INFO:tasks.workunit.client.1.smithi153.stdout:8/564: creat d2/d93/d97/d9c/d27/d2d/d47/d60/d67/fbf x:0 0 0 2024-04-16T02:35:40.433 INFO:tasks.workunit.client.1.smithi153.stdout:6/604: write dc/d47/d57/d77/d15/f19 [301756,6959] 0 2024-04-16T02:35:40.435 INFO:tasks.workunit.client.1.smithi153.stdout:3/578: fdatasync d6/d10/d22/d43/f77 0 2024-04-16T02:35:40.437 INFO:tasks.workunit.client.1.smithi153.stdout:8/565: mknod d2/d93/d97/cc0 0 2024-04-16T02:35:40.438 INFO:tasks.workunit.client.1.smithi153.stdout:7/577: dwrite d0/d4/db/d2b/f89 [0,4194304] 0 2024-04-16T02:35:40.438 INFO:tasks.workunit.client.1.smithi153.stdout:3/579: chown d6/d10/d22/d4d/f6e 2087 1 2024-04-16T02:35:40.438 INFO:tasks.workunit.client.1.smithi153.stdout:5/682: rename d8/df/d2f/c48 to d8/df/d28/d5a/cd8 0 2024-04-16T02:35:40.439 INFO:tasks.workunit.client.1.smithi153.stdout:8/566: chown d2/d13/d96/fb5 0 1 2024-04-16T02:35:40.440 INFO:tasks.workunit.client.1.smithi153.stdout:7/578: stat d0/d4/db/f50 0 2024-04-16T02:35:40.443 INFO:tasks.workunit.client.1.smithi153.stdout:8/567: chown d2/l56 0 1 2024-04-16T02:35:40.446 INFO:tasks.workunit.client.1.smithi153.stdout:6/605: getdents dc/d47/d57/d77/d15 0 2024-04-16T02:35:40.446 INFO:tasks.workunit.client.1.smithi153.stdout:8/568: write d2/d1c/f85 [1271638,53043] 0 2024-04-16T02:35:40.448 INFO:tasks.workunit.client.1.smithi153.stdout:9/639: write d4/d6/d1b/d6e/f74 [5065727,95290] 0 2024-04-16T02:35:40.448 INFO:tasks.workunit.client.1.smithi153.stdout:4/717: truncate d3/d9/db/d17/f67 2667009 0 2024-04-16T02:35:40.450 INFO:tasks.workunit.client.0.smithi151.stdout:2/216: dread de/f14 [0,4194304] 0 2024-04-16T02:35:40.452 INFO:tasks.workunit.client.0.smithi151.stdout:2/217: truncate de/d15/d1e/f3f 251972 0 2024-04-16T02:35:40.452 INFO:tasks.workunit.client.1.smithi153.stdout:5/683: dread d8/df/f88 [0,4194304] 0 2024-04-16T02:35:40.453 INFO:tasks.workunit.client.1.smithi153.stdout:0/731: truncate dd/d14/da6/da4/fbe 2184510 0 2024-04-16T02:35:40.454 INFO:tasks.workunit.client.1.smithi153.stdout:2/584: dread d9/d24/d27/d2c/d5d/f63 [0,4194304] 0 2024-04-16T02:35:40.455 INFO:tasks.workunit.client.1.smithi153.stdout:6/606: write dc/d47/d57/d77/d15/f1e [2964394,128793] 0 2024-04-16T02:35:40.456 INFO:tasks.workunit.client.1.smithi153.stdout:7/579: dwrite d0/f36 [0,4194304] 0 2024-04-16T02:35:40.457 INFO:tasks.workunit.client.1.smithi153.stdout:9/640: write d4/d6/dc/d2d/da0/d69/f6c [2924858,32955] 0 2024-04-16T02:35:40.457 INFO:tasks.workunit.client.0.smithi151.stdout:2/218: dwrite de/d11/f18 [4194304,4194304] 0 2024-04-16T02:35:40.458 INFO:tasks.workunit.client.1.smithi153.stdout:8/569: mknod d2/d93/d97/d9c/d27/d2d/d47/cc1 0 2024-04-16T02:35:40.460 INFO:tasks.workunit.client.0.smithi151.stdout:2/219: mkdir de/d11/d42 0 2024-04-16T02:35:40.461 INFO:tasks.workunit.client.1.smithi153.stdout:3/580: rename d6/d10/d22/d2c/c65 to d6/d10/d22/d2e/d8a/cca 0 2024-04-16T02:35:40.462 INFO:tasks.workunit.client.0.smithi151.stdout:2/220: chown de/d15/c16 247149332 1 2024-04-16T02:35:40.463 INFO:tasks.workunit.client.1.smithi153.stdout:6/607: fdatasync dc/d47/d57/d77/d7a/d97/fc8 0 2024-04-16T02:35:40.465 INFO:tasks.workunit.client.0.smithi151.stdout:2/221: rename de/d15/f1c to de/d15/f43 0 2024-04-16T02:35:40.469 INFO:tasks.workunit.client.0.smithi151.stdout:2/222: rename de/c19 to de/d11/c44 0 2024-04-16T02:35:40.470 INFO:tasks.workunit.client.0.smithi151.stdout:2/223: write de/d15/f17 [39631,53811] 0 2024-04-16T02:35:40.471 INFO:tasks.workunit.client.1.smithi153.stdout:5/684: creat d8/df/d23/d70/fd9 x:0 0 0 2024-04-16T02:35:40.471 INFO:tasks.workunit.client.1.smithi153.stdout:6/608: symlink dc/d47/d93/da9/lcd 0 2024-04-16T02:35:40.472 INFO:tasks.workunit.client.1.smithi153.stdout:8/570: chown d2/d93/d97/d9c/d27/d5e/d63/fb1 1368715137 1 2024-04-16T02:35:40.472 INFO:tasks.workunit.client.0.smithi151.stdout:2/224: fdatasync de/f27 0 2024-04-16T02:35:40.473 INFO:tasks.workunit.client.1.smithi153.stdout:3/581: creat d6/d10/d22/d4d/dbb/fcb x:0 0 0 2024-04-16T02:35:40.476 INFO:tasks.workunit.client.0.smithi151.stdout:2/225: creat de/f45 x:0 0 0 2024-04-16T02:35:40.477 INFO:tasks.workunit.client.1.smithi153.stdout:1/646: dwrite d4/dd/d13/d26/db4/fcd [0,4194304] 0 2024-04-16T02:35:40.477 INFO:tasks.workunit.client.1.smithi153.stdout:6/609: creat dc/d10/d62/fce x:0 0 0 2024-04-16T02:35:40.478 INFO:tasks.workunit.client.1.smithi153.stdout:4/718: dwrite d3/d9/db/de/d2c/f74 [8388608,4194304] 0 2024-04-16T02:35:40.478 INFO:tasks.workunit.client.0.smithi151.stdout:2/226: creat de/d15/f46 x:0 0 0 2024-04-16T02:35:40.480 INFO:tasks.workunit.client.0.smithi151.stdout:2/227: creat de/d15/d1e/f47 x:0 0 0 2024-04-16T02:35:40.480 INFO:tasks.workunit.client.1.smithi153.stdout:3/582: read - d6/d10/d3b/d59/fa9 zero size 2024-04-16T02:35:40.481 INFO:tasks.workunit.client.1.smithi153.stdout:8/571: dread - d2/d92/f69 zero size 2024-04-16T02:35:40.482 INFO:tasks.workunit.client.1.smithi153.stdout:2/585: write d9/d2b/d1c/d20/f67 [1909610,105331] 0 2024-04-16T02:35:40.485 INFO:tasks.workunit.client.0.smithi151.stdout:2/228: dread de/f27 [0,4194304] 0 2024-04-16T02:35:40.486 INFO:tasks.workunit.client.1.smithi153.stdout:1/647: chown d4/dd/d18/d29/d71 21 1 2024-04-16T02:35:40.488 INFO:tasks.workunit.client.1.smithi153.stdout:4/719: chown d3/d9/db/d17/f8c 648332560 1 2024-04-16T02:35:40.490 INFO:tasks.workunit.client.1.smithi153.stdout:5/685: getdents d8/d41/dbd/d83 0 2024-04-16T02:35:40.491 INFO:tasks.workunit.client.1.smithi153.stdout:7/580: dwrite d0/d4/db/d2b/f40 [0,4194304] 0 2024-04-16T02:35:40.491 INFO:tasks.workunit.client.1.smithi153.stdout:9/641: dwrite d4/d6/dc/d2d/da0/d40/d4a/f80 [0,4194304] 0 2024-04-16T02:35:40.494 INFO:tasks.workunit.client.0.smithi151.stdout:2/229: dwrite de/f3e [0,4194304] 0 2024-04-16T02:35:40.498 INFO:tasks.workunit.client.1.smithi153.stdout:1/648: creat d4/dd/d13/d26/d58/fdc x:0 0 0 2024-04-16T02:35:40.498 INFO:tasks.workunit.client.1.smithi153.stdout:4/720: chown d3/l51 1 1 2024-04-16T02:35:40.498 INFO:tasks.workunit.client.1.smithi153.stdout:5/686: readlink d8/df/d28/d5a/d5e/l8b 0 2024-04-16T02:35:40.498 INFO:tasks.workunit.client.0.smithi151.stdout:2/230: symlink de/d11/d42/l48 0 2024-04-16T02:35:40.500 INFO:tasks.workunit.client.0.smithi151.stdout:2/231: truncate de/d15/f39 1064025 0 2024-04-16T02:35:40.500 INFO:tasks.workunit.client.1.smithi153.stdout:3/583: dwrite d6/d10/f1c [0,4194304] 0 2024-04-16T02:35:40.501 INFO:tasks.workunit.client.1.smithi153.stdout:6/610: getdents dc/d6b/dba 0 2024-04-16T02:35:40.502 INFO:tasks.workunit.client.0.smithi151.stdout:2/232: stat de/d11 0 2024-04-16T02:35:40.503 INFO:tasks.workunit.client.0.smithi151.stdout:2/233: write de/d15/f36 [4644608,112522] 0 2024-04-16T02:35:40.506 INFO:tasks.workunit.client.1.smithi153.stdout:9/642: creat d4/d6/dc/d24/d5d/d66/dbd/fc5 x:0 0 0 2024-04-16T02:35:40.507 INFO:tasks.workunit.client.0.smithi151.stdout:2/234: chown de/c20 938 1 2024-04-16T02:35:40.508 INFO:tasks.workunit.client.1.smithi153.stdout:4/721: write d3/d9/db/de/d2c/f56 [900018,120577] 0 2024-04-16T02:35:40.509 INFO:tasks.workunit.client.0.smithi151.stdout:2/235: truncate de/d15/d1e/f3f 887202 0 2024-04-16T02:35:40.512 INFO:tasks.workunit.client.1.smithi153.stdout:1/649: creat d4/dd/d13/d23/fdd x:0 0 0 2024-04-16T02:35:40.518 INFO:tasks.workunit.client.1.smithi153.stdout:8/572: dwrite d2/d93/d97/d9c/d27/d2d/d47/d77/d8e/fbb [0,4194304] 0 2024-04-16T02:35:40.519 INFO:tasks.workunit.client.0.smithi151.stdout:2/236: dwrite de/f45 [0,4194304] 0 2024-04-16T02:35:40.520 INFO:tasks.workunit.client.1.smithi153.stdout:3/584: rmdir d6/d10/d3b/d59 39 2024-04-16T02:35:40.520 INFO:tasks.workunit.client.0.smithi151.stdout:2/237: fdatasync de/d15/d1e/f3f 0 2024-04-16T02:35:40.522 INFO:tasks.workunit.client.0.smithi151.stdout:2/238: truncate de/d15/f39 1275338 0 2024-04-16T02:35:40.522 INFO:tasks.workunit.client.1.smithi153.stdout:1/650: dread - d4/dd/d13/d26/d58/d7a/f83 zero size 2024-04-16T02:35:40.524 INFO:tasks.workunit.client.1.smithi153.stdout:2/586: link d9/d24/d27/d2c/d5d/da3/dc0/cc5 d9/d2b/cc7 0 2024-04-16T02:35:40.525 INFO:tasks.workunit.client.1.smithi153.stdout:8/573: read - d2/d93/d97/d9c/f7c zero size 2024-04-16T02:35:40.528 INFO:tasks.workunit.client.1.smithi153.stdout:2/587: chown d9/d2b/d5c/f80 482 1 2024-04-16T02:35:40.529 INFO:tasks.workunit.client.0.smithi151.stdout:2/239: rename de/d15/d1e/f47 to de/f49 0 2024-04-16T02:35:40.530 INFO:tasks.workunit.client.1.smithi153.stdout:3/585: fdatasync d6/f20 0 2024-04-16T02:35:40.531 INFO:tasks.workunit.client.1.smithi153.stdout:9/643: dwrite d4/d33/faf [0,4194304] 0 2024-04-16T02:35:40.534 INFO:tasks.workunit.client.1.smithi153.stdout:7/581: rename d0/d4/db/d2b/d43/d6d/d68/d72 to d0/d4/db/d2b/d53/d59/d76/dc8 0 2024-04-16T02:35:40.535 INFO:tasks.workunit.client.0.smithi151.stdout:2/240: dwrite de/d11/d41/f38 [0,4194304] 0 2024-04-16T02:35:40.535 INFO:tasks.workunit.client.1.smithi153.stdout:0/732: dread dd/d14/d51/f59 [0,4194304] 0 2024-04-16T02:35:40.536 INFO:tasks.workunit.client.1.smithi153.stdout:6/611: dwrite f3 [0,4194304] 0 2024-04-16T02:35:40.536 INFO:tasks.workunit.client.1.smithi153.stdout:5/687: link d8/df/d84/c8c d8/d41/dbd/d39/d6d/cda 0 2024-04-16T02:35:40.539 INFO:tasks.workunit.client.1.smithi153.stdout:9/644: fsync d4/d6/dc/d2d/f53 0 2024-04-16T02:35:40.541 INFO:tasks.workunit.client.1.smithi153.stdout:2/588: creat d9/d24/d73/fc8 x:0 0 0 2024-04-16T02:35:40.543 INFO:tasks.workunit.client.1.smithi153.stdout:3/586: creat d6/d10/d22/d4d/da0/da3/fcc x:0 0 0 2024-04-16T02:35:40.543 INFO:tasks.workunit.client.1.smithi153.stdout:9/645: chown d4/d6 144004 1 2024-04-16T02:35:40.545 INFO:tasks.workunit.client.0.smithi151.stdout:2/241: dwrite de/d15/d1e/f3c [0,4194304] 0 2024-04-16T02:35:40.546 INFO:tasks.workunit.client.0.smithi151.stdout:2/242: rename de/d11 to de/d11/d4a 22 2024-04-16T02:35:40.547 INFO:tasks.workunit.client.1.smithi153.stdout:5/688: creat d8/d41/dbd/d83/fdb x:0 0 0 2024-04-16T02:35:40.547 INFO:tasks.workunit.client.1.smithi153.stdout:0/733: read dd/d14/d19/d20/d4a/f78 [300592,25883] 0 2024-04-16T02:35:40.553 INFO:tasks.workunit.client.1.smithi153.stdout:0/734: read - dd/d14/d19/d55/fcf zero size 2024-04-16T02:35:40.553 INFO:tasks.workunit.client.1.smithi153.stdout:1/651: dwrite d4/dd/d13/d26/d58/d7a/fa7 [4194304,4194304] 0 2024-04-16T02:35:40.557 INFO:tasks.workunit.client.1.smithi153.stdout:7/582: rename d0/d4/db/d2b/d53/d59/d76/fc5 to d0/d4/db/d34/d4b/daa/fc9 0 2024-04-16T02:35:40.562 INFO:tasks.workunit.client.1.smithi153.stdout:3/587: truncate d6/d10/d22/d4d/da0/da3/fcc 700249 0 2024-04-16T02:35:40.563 INFO:tasks.workunit.client.1.smithi153.stdout:0/735: write dd/d32/d57/f61 [3224860,63796] 0 2024-04-16T02:35:40.563 INFO:tasks.workunit.client.1.smithi153.stdout:5/689: mknod d8/d41/dbd/dd3/cdc 0 2024-04-16T02:35:40.564 INFO:tasks.workunit.client.1.smithi153.stdout:8/574: dwrite d2/d13/d3a/d5b/f9d [4194304,4194304] 0 2024-04-16T02:35:40.564 INFO:tasks.workunit.client.1.smithi153.stdout:9/646: dwrite d4/f13 [0,4194304] 0 2024-04-16T02:35:40.566 INFO:tasks.workunit.client.1.smithi153.stdout:8/575: stat d2/f17 0 2024-04-16T02:35:40.568 INFO:tasks.workunit.client.1.smithi153.stdout:3/588: rename d6/d10/d3b/f9f to d6/d10/d22/d4d/dbb/fcd 0 2024-04-16T02:35:40.573 INFO:tasks.workunit.client.1.smithi153.stdout:8/576: chown d2/d13/d3a/d5b/f9d 18087 1 2024-04-16T02:35:40.580 INFO:tasks.workunit.client.1.smithi153.stdout:1/652: dwrite d4/da2/d65/d84/fbd [0,4194304] 0 2024-04-16T02:35:40.580 INFO:tasks.workunit.client.1.smithi153.stdout:3/589: creat d6/fce x:0 0 0 2024-04-16T02:35:40.581 INFO:tasks.workunit.client.1.smithi153.stdout:7/583: dwrite d0/d13/f4a [4194304,4194304] 0 2024-04-16T02:35:40.581 INFO:tasks.workunit.client.1.smithi153.stdout:9/647: rename d4/d33/d7d to d4/d6/dc/d24/dc6 0 2024-04-16T02:35:40.581 INFO:tasks.workunit.client.1.smithi153.stdout:3/590: stat d6/daf/lc8 0 2024-04-16T02:35:40.583 INFO:tasks.workunit.client.1.smithi153.stdout:7/584: dread - d0/d4/db/d2b/fc1 zero size 2024-04-16T02:35:40.584 INFO:tasks.workunit.client.1.smithi153.stdout:0/736: dread dd/d26/d44/fb7 [0,4194304] 0 2024-04-16T02:35:40.584 INFO:tasks.workunit.client.1.smithi153.stdout:7/585: readlink d0/l1f 0 2024-04-16T02:35:40.585 INFO:tasks.workunit.client.1.smithi153.stdout:3/591: chown d6/d10/d22/l37 550 1 2024-04-16T02:35:40.586 INFO:tasks.workunit.client.1.smithi153.stdout:3/592: stat d6/d10/d22/d4d/fbd 0 2024-04-16T02:35:40.590 INFO:tasks.workunit.client.1.smithi153.stdout:7/586: symlink d0/d4/db/d2b/d43/d6d/d68/d6c/lca 0 2024-04-16T02:35:40.591 INFO:tasks.workunit.client.1.smithi153.stdout:1/653: rename d4/dd/d20/f72 to d4/dd/d18/d29/fde 0 2024-04-16T02:35:40.594 INFO:tasks.workunit.client.1.smithi153.stdout:3/593: creat d6/d10/d22/d2e/fcf x:0 0 0 2024-04-16T02:35:40.595 INFO:tasks.workunit.client.1.smithi153.stdout:3/594: chown d6/d10/d22/d2c/f74 7482832 1 2024-04-16T02:35:40.598 INFO:tasks.workunit.client.0.smithi151.stdout:8/100: sync 2024-04-16T02:35:40.598 INFO:tasks.workunit.client.1.smithi153.stdout:2/589: write d9/d2b/d18/f29 [1603919,63799] 0 2024-04-16T02:35:40.599 INFO:tasks.workunit.client.1.smithi153.stdout:2/590: dread - d9/d24/d27/d2c/d5d/fbb zero size 2024-04-16T02:35:40.600 INFO:tasks.workunit.client.1.smithi153.stdout:1/654: symlink d4/dd/d13/d7e/d80/ldf 0 2024-04-16T02:35:40.600 INFO:tasks.workunit.client.1.smithi153.stdout:0/737: link dd/d26/c76 dd/d14/da6/de0/ce8 0 2024-04-16T02:35:40.601 INFO:tasks.workunit.client.1.smithi153.stdout:7/587: mknod d0/d4/db/d47/d8d/d93/d8b/ccb 0 2024-04-16T02:35:40.602 INFO:tasks.workunit.client.0.smithi151.stdout:8/101: rename d11/l1e to d11/l21 0 2024-04-16T02:35:40.605 INFO:tasks.workunit.client.1.smithi153.stdout:9/648: rename d4/d6/dc/d24/d5d to d4/d33/dae/dc7 0 2024-04-16T02:35:40.606 INFO:tasks.workunit.client.1.smithi153.stdout:5/690: truncate d8/df/d28/f63 2363195 0 2024-04-16T02:35:40.608 INFO:tasks.workunit.client.0.smithi151.stdout:2/243: dread de/f40 [0,4194304] 0 2024-04-16T02:35:40.608 INFO:tasks.workunit.client.1.smithi153.stdout:0/738: mknod dd/d14/d51/d7f/ce9 0 2024-04-16T02:35:40.608 INFO:tasks.workunit.client.0.smithi151.stdout:8/102: write d11/f17 [5004154,15771] 0 2024-04-16T02:35:40.609 INFO:tasks.workunit.client.0.smithi151.stdout:2/244: chown de/d15/f46 1779 1 2024-04-16T02:35:40.611 INFO:tasks.workunit.client.0.smithi151.stdout:2/245: mkdir de/d11/d41/d4b 0 2024-04-16T02:35:40.613 INFO:tasks.workunit.client.1.smithi153.stdout:2/591: mknod d9/d24/d34/d60/cc9 0 2024-04-16T02:35:40.613 INFO:tasks.workunit.client.1.smithi153.stdout:2/592: dread - d9/d24/d27/d2c/d5d/fbb zero size 2024-04-16T02:35:40.614 INFO:tasks.workunit.client.1.smithi153.stdout:1/655: creat d4/dd/d13/d26/db4/fe0 x:0 0 0 2024-04-16T02:35:40.615 INFO:tasks.workunit.client.1.smithi153.stdout:2/593: fsync d9/d2b/d5c/f75 0 2024-04-16T02:35:40.615 INFO:tasks.workunit.client.0.smithi151.stdout:8/103: dwrite f3 [4194304,4194304] 0 2024-04-16T02:35:40.616 INFO:tasks.workunit.client.0.smithi151.stdout:8/104: fsync d11/f1c 0 2024-04-16T02:35:40.618 INFO:tasks.workunit.client.1.smithi153.stdout:9/649: rename d4/d6/dc/d2d/l5a to d4/d6/dc/d2d/d79/dbb/lc8 0 2024-04-16T02:35:40.618 INFO:tasks.workunit.client.0.smithi151.stdout:8/105: truncate d11/f1c 4983269 0 2024-04-16T02:35:40.619 INFO:tasks.workunit.client.1.smithi153.stdout:8/577: dread d2/de/d3f/f91 [0,4194304] 0 2024-04-16T02:35:40.621 INFO:tasks.workunit.client.0.smithi151.stdout:2/246: dwrite de/d15/d1e/f3c [0,4194304] 0 2024-04-16T02:35:40.622 INFO:tasks.workunit.client.0.smithi151.stdout:2/247: fsync de/d15/d1e/f3f 0 2024-04-16T02:35:40.624 INFO:tasks.workunit.client.0.smithi151.stdout:2/248: write de/d15/f39 [225120,76761] 0 2024-04-16T02:35:40.624 INFO:tasks.workunit.client.1.smithi153.stdout:9/650: dwrite d4/d6/dc/d2d/da0/f87 [0,4194304] 0 2024-04-16T02:35:40.627 INFO:tasks.workunit.client.0.smithi151.stdout:5/215: sync 2024-04-16T02:35:40.628 INFO:tasks.workunit.client.0.smithi151.stdout:5/216: readlink d2/l2b 0 2024-04-16T02:35:40.629 INFO:tasks.workunit.client.0.smithi151.stdout:5/217: dread - d2/d6/d19/d1a/f2e zero size 2024-04-16T02:35:40.631 INFO:tasks.workunit.client.0.smithi151.stdout:5/218: write d2/d6/f32 [1049709,105927] 0 2024-04-16T02:35:40.631 INFO:tasks.workunit.client.1.smithi153.stdout:9/651: dwrite d4/f27 [0,4194304] 0 2024-04-16T02:35:40.631 INFO:tasks.workunit.client.0.smithi151.stdout:2/249: dread de/d15/f17 [0,4194304] 0 2024-04-16T02:35:40.632 INFO:tasks.workunit.client.1.smithi153.stdout:1/656: creat d4/dd/d18/d6c/dd8/fe1 x:0 0 0 2024-04-16T02:35:40.632 INFO:tasks.workunit.client.1.smithi153.stdout:9/652: fdatasync d4/d33/faf 0 2024-04-16T02:35:40.632 INFO:tasks.workunit.client.0.smithi151.stdout:5/219: write d2/f24 [547245,17108] 0 2024-04-16T02:35:40.634 INFO:tasks.workunit.client.0.smithi151.stdout:8/106: truncate d11/f17 1891369 0 2024-04-16T02:35:40.639 INFO:tasks.workunit.client.0.smithi151.stdout:5/220: unlink d2/d6/d19/c3c 0 2024-04-16T02:35:40.640 INFO:tasks.workunit.client.1.smithi153.stdout:0/739: symlink dd/d14/d19/d77/de3/lea 0 2024-04-16T02:35:40.640 INFO:tasks.workunit.client.0.smithi151.stdout:2/250: mknod de/d11/d41/d35/c4c 0 2024-04-16T02:35:40.644 INFO:tasks.workunit.client.0.smithi151.stdout:8/107: dwrite d11/f1b [0,4194304] 0 2024-04-16T02:35:40.646 INFO:tasks.workunit.client.0.smithi151.stdout:2/251: symlink de/d11/d42/l4d 0 2024-04-16T02:35:40.648 INFO:tasks.workunit.client.0.smithi151.stdout:2/252: truncate de/d15/f17 428847 0 2024-04-16T02:35:40.648 INFO:tasks.workunit.client.1.smithi153.stdout:7/588: write d0/d4/db/d2b/f22 [383047,84612] 0 2024-04-16T02:35:40.649 INFO:tasks.workunit.client.0.smithi151.stdout:5/221: mkdir d2/d6/d43 0 2024-04-16T02:35:40.651 INFO:tasks.workunit.client.0.smithi151.stdout:2/253: readlink de/l1b 0 2024-04-16T02:35:40.653 INFO:tasks.workunit.client.0.smithi151.stdout:8/108: read fb [1267756,100797] 0 2024-04-16T02:35:40.653 INFO:tasks.workunit.client.1.smithi153.stdout:2/594: unlink d9/d24/d27/f30 0 2024-04-16T02:35:40.654 INFO:tasks.workunit.client.1.smithi153.stdout:9/653: creat d4/d6/d1b/d2a/fc9 x:0 0 0 2024-04-16T02:35:40.654 INFO:tasks.workunit.client.1.smithi153.stdout:9/654: readlink d4/d6/dc/d7f/l82 0 2024-04-16T02:35:40.655 INFO:tasks.workunit.client.1.smithi153.stdout:8/578: mkdir d2/d13/d3a/dc2 0 2024-04-16T02:35:40.656 INFO:tasks.workunit.client.0.smithi151.stdout:8/109: mkdir d11/d22 0 2024-04-16T02:35:40.656 INFO:tasks.workunit.client.1.smithi153.stdout:7/589: stat d0/d4/db/l45 0 2024-04-16T02:35:40.658 INFO:tasks.workunit.client.1.smithi153.stdout:0/740: symlink dd/d26/leb 0 2024-04-16T02:35:40.661 INFO:tasks.workunit.client.0.smithi151.stdout:8/110: link fe d11/d22/f23 0 2024-04-16T02:35:40.662 INFO:tasks.workunit.client.0.smithi151.stdout:2/254: dwrite de/f49 [0,4194304] 0 2024-04-16T02:35:40.668 INFO:tasks.workunit.client.0.smithi151.stdout:8/111: rename c7 to d11/d22/c24 0 2024-04-16T02:35:40.669 INFO:tasks.workunit.client.0.smithi151.stdout:2/255: dread de/d15/f39 [0,4194304] 0 2024-04-16T02:35:40.671 INFO:tasks.workunit.client.1.smithi153.stdout:7/590: symlink d0/d4/db/d2b/d53/dbd/lcc 0 2024-04-16T02:35:40.671 INFO:tasks.workunit.client.1.smithi153.stdout:2/595: creat d9/d71/fca x:0 0 0 2024-04-16T02:35:40.672 INFO:tasks.workunit.client.1.smithi153.stdout:0/741: truncate dd/d32/fde 203812 0 2024-04-16T02:35:40.672 INFO:tasks.workunit.client.1.smithi153.stdout:9/655: creat d4/d6/d1b/fca x:0 0 0 2024-04-16T02:35:40.673 INFO:tasks.workunit.client.1.smithi153.stdout:8/579: creat d2/d93/d97/d9c/d27/fc3 x:0 0 0 2024-04-16T02:35:40.673 INFO:tasks.workunit.client.1.smithi153.stdout:0/742: chown dd/d14/d19/d55/c9b 30111 1 2024-04-16T02:35:40.675 INFO:tasks.workunit.client.0.smithi151.stdout:2/256: rename de/l29 to de/d3d/l4e 0 2024-04-16T02:35:40.676 INFO:tasks.workunit.client.0.smithi151.stdout:2/257: stat de/d11/d41/f32 0 2024-04-16T02:35:40.676 INFO:tasks.workunit.client.1.smithi153.stdout:7/591: dread d0/d4/db/d2b/f40 [0,4194304] 0 2024-04-16T02:35:40.677 INFO:tasks.workunit.client.0.smithi151.stdout:2/258: chown l9 0 1 2024-04-16T02:35:40.677 INFO:tasks.workunit.client.0.smithi151.stdout:5/222: dread d2/f18 [0,4194304] 0 2024-04-16T02:35:40.677 INFO:tasks.workunit.client.0.smithi151.stdout:5/223: chown d2/d6/f2a 65803 1 2024-04-16T02:35:40.680 INFO:tasks.workunit.client.0.smithi151.stdout:2/259: rename de/d15/l2d to de/d11/d41/d35/l4f 0 2024-04-16T02:35:40.682 INFO:tasks.workunit.client.0.smithi151.stdout:5/224: unlink d2/d6/f2f 0 2024-04-16T02:35:40.683 INFO:tasks.workunit.client.0.smithi151.stdout:5/225: fsync d2/d6/d19/d30/f3f 0 2024-04-16T02:35:40.684 INFO:tasks.workunit.client.1.smithi153.stdout:1/657: link d4/dd/d13/d26/f2b d4/da2/d65/fe2 0 2024-04-16T02:35:40.684 INFO:tasks.workunit.client.0.smithi151.stdout:5/226: write d2/d6/d19/d30/f3a [1170413,117977] 0 2024-04-16T02:35:40.686 INFO:tasks.workunit.client.1.smithi153.stdout:2/596: symlink d9/d24/d27/d2c/d5d/da3/lcb 0 2024-04-16T02:35:40.687 INFO:tasks.workunit.client.1.smithi153.stdout:2/597: stat d9/d24/d27/d2c/d41/f78 0 2024-04-16T02:35:40.687 INFO:tasks.workunit.client.0.smithi151.stdout:8/112: write d11/f16 [383637,61544] 0 2024-04-16T02:35:40.689 INFO:tasks.workunit.client.1.smithi153.stdout:9/656: mkdir d4/d33/dae/dc7/d66/dcb 0 2024-04-16T02:35:40.689 INFO:tasks.workunit.client.1.smithi153.stdout:1/658: symlink d4/dd/d20/le3 0 2024-04-16T02:35:40.690 INFO:tasks.workunit.client.1.smithi153.stdout:1/659: read - d4/dd/d13/d7e/fb1 zero size 2024-04-16T02:35:40.691 INFO:tasks.workunit.client.0.smithi151.stdout:8/113: rename la to d11/l25 0 2024-04-16T02:35:40.691 INFO:tasks.workunit.client.1.smithi153.stdout:2/598: write f6 [2197876,18868] 0 2024-04-16T02:35:40.691 INFO:tasks.workunit.client.1.smithi153.stdout:8/580: mkdir d2/d13/d3a/dc2/dc4 0 2024-04-16T02:35:40.692 INFO:tasks.workunit.client.1.smithi153.stdout:0/743: creat dd/d14/d19/d77/db6/fec x:0 0 0 2024-04-16T02:35:40.694 INFO:tasks.workunit.client.1.smithi153.stdout:2/599: rmdir d9/d2b/d18 39 2024-04-16T02:35:40.694 INFO:tasks.workunit.client.0.smithi151.stdout:2/260: getdents de 0 2024-04-16T02:35:40.696 INFO:tasks.workunit.client.0.smithi151.stdout:2/261: read de/d15/d1e/f3c [2928953,42668] 0 2024-04-16T02:35:40.697 INFO:tasks.workunit.client.0.smithi151.stdout:5/227: rmdir d2/d6/d19/d30/d33 0 2024-04-16T02:35:40.699 INFO:tasks.workunit.client.1.smithi153.stdout:1/660: dwrite d4/dd/d13/d23/d5f/f7f [0,4194304] 0 2024-04-16T02:35:40.699 INFO:tasks.workunit.client.1.smithi153.stdout:8/581: rename d2/d13/d2c/l4a to d2/d93/d97/d9c/d27/d2d/d47/d60/d67/db7/lc5 0 2024-04-16T02:35:40.700 INFO:tasks.workunit.client.0.smithi151.stdout:8/114: dwrite d11/f16 [0,4194304] 0 2024-04-16T02:35:40.702 INFO:tasks.workunit.client.0.smithi151.stdout:5/228: creat d2/d6/d19/d30/f44 x:0 0 0 2024-04-16T02:35:40.704 INFO:tasks.workunit.client.0.smithi151.stdout:5/229: chown d2/d6/d19/d30/f39 16346679 1 2024-04-16T02:35:40.704 INFO:tasks.workunit.client.0.smithi151.stdout:5/230: fdatasync d2/d6/f2a 0 2024-04-16T02:35:40.705 INFO:tasks.workunit.client.0.smithi151.stdout:5/231: readlink d2/d6/d19/l2c 0 2024-04-16T02:35:40.706 INFO:tasks.workunit.client.1.smithi153.stdout:2/600: dread d9/d24/d27/d2c/d41/f56 [0,4194304] 0 2024-04-16T02:35:40.706 INFO:tasks.workunit.client.0.smithi151.stdout:5/232: fsync d2/f24 0 2024-04-16T02:35:40.708 INFO:tasks.workunit.client.1.smithi153.stdout:9/657: link d4/d6/dc/d2d/d64/c97 d4/d6/dc/d24/d4e/ccc 0 2024-04-16T02:35:40.709 INFO:tasks.workunit.client.1.smithi153.stdout:8/582: dwrite d2/d13/d96/fb5 [0,4194304] 0 2024-04-16T02:35:40.709 INFO:tasks.workunit.client.1.smithi153.stdout:1/661: dwrite d4/f7 [0,4194304] 0 2024-04-16T02:35:40.713 INFO:tasks.workunit.client.1.smithi153.stdout:1/662: truncate d4/dd/d18/f76 941772 0 2024-04-16T02:35:40.714 INFO:tasks.workunit.client.1.smithi153.stdout:9/658: truncate d4/d33/dae/dc7/d66/dbd/fc5 603417 0 2024-04-16T02:35:40.716 INFO:tasks.workunit.client.1.smithi153.stdout:2/601: rename f1 to d9/d2b/d5c/da2/fcc 0 2024-04-16T02:35:40.718 INFO:tasks.workunit.client.1.smithi153.stdout:1/663: dread d4/dd/d18/d29/d71/fc5 [0,4194304] 0 2024-04-16T02:35:40.721 INFO:tasks.workunit.client.1.smithi153.stdout:8/583: dwrite d2/d93/d97/d9c/d27/d2d/d47/d77/d8e/fbb [0,4194304] 0 2024-04-16T02:35:40.722 INFO:tasks.workunit.client.1.smithi153.stdout:9/659: write d4/f48 [1126061,21422] 0 2024-04-16T02:35:40.722 INFO:tasks.workunit.client.0.smithi151.stdout:5/233: dwrite d2/d6/d19/d30/f44 [0,4194304] 0 2024-04-16T02:35:40.722 INFO:tasks.workunit.client.0.smithi151.stdout:2/262: dwrite de/d15/f34 [0,4194304] 0 2024-04-16T02:35:40.723 INFO:tasks.workunit.client.1.smithi153.stdout:8/584: readlink d2/de/d34/l3b 0 2024-04-16T02:35:40.724 INFO:tasks.workunit.client.0.smithi151.stdout:4/196: sync 2024-04-16T02:35:40.724 INFO:tasks.workunit.client.0.smithi151.stdout:9/120: sync 2024-04-16T02:35:40.724 INFO:tasks.workunit.client.0.smithi151.stdout:7/140: sync 2024-04-16T02:35:40.724 INFO:tasks.workunit.client.0.smithi151.stdout:1/180: sync 2024-04-16T02:35:40.724 INFO:tasks.workunit.client.0.smithi151.stdout:6/297: sync 2024-04-16T02:35:40.726 INFO:tasks.workunit.client.0.smithi151.stdout:3/128: sync 2024-04-16T02:35:40.726 INFO:tasks.workunit.client.0.smithi151.stdout:0/200: sync 2024-04-16T02:35:40.730 INFO:tasks.workunit.client.1.smithi153.stdout:2/602: unlink d9/d24/d27/lbc 0 2024-04-16T02:35:40.738 INFO:tasks.workunit.client.1.smithi153.stdout:9/660: write d4/d33/f5b [4690605,77503] 0 2024-04-16T02:35:40.738 INFO:tasks.workunit.client.0.smithi151.stdout:3/129: chown d1/d6 4385793 1 2024-04-16T02:35:40.738 INFO:tasks.workunit.client.1.smithi153.stdout:9/661: write d4/d6/dc/d2d/da0/d40/d4a/f91 [288123,102836] 0 2024-04-16T02:35:40.739 INFO:tasks.workunit.client.1.smithi153.stdout:9/662: stat d4/d6/dc/d46/f84 0 2024-04-16T02:35:40.741 INFO:tasks.workunit.client.1.smithi153.stdout:9/663: readlink d4/d6/dc/d2d/da0/d40/d4a/lb6 0 2024-04-16T02:35:40.742 INFO:tasks.workunit.client.1.smithi153.stdout:8/585: unlink d2/d93/d97/d9c/d27/d2d/d47/cc1 0 2024-04-16T02:35:40.742 INFO:tasks.workunit.client.1.smithi153.stdout:1/664: truncate d4/f5e 1592979 0 2024-04-16T02:35:40.742 INFO:tasks.workunit.client.0.smithi151.stdout:7/141: unlink d0/fb 0 2024-04-16T02:35:40.744 INFO:tasks.workunit.client.0.smithi151.stdout:8/115: dwrite d11/f15 [0,4194304] 0 2024-04-16T02:35:40.744 INFO:tasks.workunit.client.1.smithi153.stdout:0/744: write dd/f6f [4353071,94527] 0 2024-04-16T02:35:40.745 INFO:tasks.workunit.client.1.smithi153.stdout:9/664: creat d4/d6/dc/d2d/d79/dbb/fcd x:0 0 0 2024-04-16T02:35:40.747 INFO:tasks.workunit.client.1.smithi153.stdout:8/586: mkdir d2/d93/d97/d9c/d27/d5e/d63/dc6 0 2024-04-16T02:35:40.749 INFO:tasks.workunit.client.0.smithi151.stdout:4/197: dwrite d0/f2d [0,4194304] 0 2024-04-16T02:35:40.750 INFO:tasks.workunit.client.1.smithi153.stdout:0/745: mknod dd/d14/da6/d73/ced 0 2024-04-16T02:35:40.750 INFO:tasks.workunit.client.1.smithi153.stdout:0/746: write dd/d14/da6/da4/fb5 [832184,77249] 0 2024-04-16T02:35:40.751 INFO:tasks.workunit.client.0.smithi151.stdout:7/142: readlink d0/l20 0 2024-04-16T02:35:40.752 INFO:tasks.workunit.client.1.smithi153.stdout:8/587: mkdir d2/d13/d96/dc7 0 2024-04-16T02:35:40.754 INFO:tasks.workunit.client.1.smithi153.stdout:9/665: link d4/d33/f4f d4/d6/d1b/d2a/d99/fce 0 2024-04-16T02:35:40.755 INFO:tasks.workunit.client.1.smithi153.stdout:0/747: read - dd/d14/d19/fc4 zero size 2024-04-16T02:35:40.755 INFO:tasks.workunit.client.1.smithi153.stdout:4/722: sync 2024-04-16T02:35:40.756 INFO:tasks.workunit.client.0.smithi151.stdout:6/298: dread d2/d3d/f55 [0,4194304] 0 2024-04-16T02:35:40.756 INFO:tasks.workunit.client.0.smithi151.stdout:1/181: mkdir d3/d4/db/df/d32/d3a 0 2024-04-16T02:35:40.757 INFO:tasks.workunit.client.1.smithi153.stdout:0/748: chown dd/d32/d80/f93 1016248 1 2024-04-16T02:35:40.758 INFO:tasks.workunit.client.0.smithi151.stdout:7/143: fdatasync d0/f6 0 2024-04-16T02:35:40.759 INFO:tasks.workunit.client.1.smithi153.stdout:8/588: mkdir d2/d93/d97/d9c/d27/d2d/d47/d60/dc8 0 2024-04-16T02:35:40.761 INFO:tasks.workunit.client.1.smithi153.stdout:9/666: write d4/d33/dae/dc7/d66/f98 [1372939,51733] 0 2024-04-16T02:35:40.764 INFO:tasks.workunit.client.1.smithi153.stdout:8/589: chown d2/d13/d2c/f32 23234525 1 2024-04-16T02:35:40.767 INFO:tasks.workunit.client.1.smithi153.stdout:4/723: unlink d3/d9/db/d17/f67 0 2024-04-16T02:35:40.767 INFO:tasks.workunit.client.0.smithi151.stdout:9/121: dwrite d1/d3/f1a [0,4194304] 0 2024-04-16T02:35:40.769 INFO:tasks.workunit.client.1.smithi153.stdout:4/724: dread - d3/d9/db/de/d2c/d52/d85/fef zero size 2024-04-16T02:35:40.770 INFO:tasks.workunit.client.1.smithi153.stdout:8/590: fsync d2/de/f24 0 2024-04-16T02:35:40.771 INFO:tasks.workunit.client.1.smithi153.stdout:4/725: chown d3/d9/db/d17/d28/c83 806 1 2024-04-16T02:35:40.772 INFO:tasks.workunit.client.0.smithi151.stdout:2/263: dwrite de/d11/f2b [4194304,4194304] 0 2024-04-16T02:35:40.772 INFO:tasks.workunit.client.0.smithi151.stdout:3/130: dwrite d1/d6/d10/f15 [4194304,4194304] 0 2024-04-16T02:35:40.773 INFO:tasks.workunit.client.1.smithi153.stdout:9/667: creat d4/d6/dc/d2d/d64/db9/fcf x:0 0 0 2024-04-16T02:35:40.774 INFO:tasks.workunit.client.0.smithi151.stdout:3/131: dread - d1/d6/f24 zero size 2024-04-16T02:35:40.780 INFO:tasks.workunit.client.0.smithi151.stdout:0/201: dwrite dc/d15/f17 [0,4194304] 0 2024-04-16T02:35:40.783 INFO:tasks.workunit.client.0.smithi151.stdout:4/198: symlink d0/l41 0 2024-04-16T02:35:40.783 INFO:tasks.workunit.client.0.smithi151.stdout:7/144: mknod d0/c21 0 2024-04-16T02:35:40.785 INFO:tasks.workunit.client.0.smithi151.stdout:6/299: mkdir d2/d5/df/d23/d45/d62 0 2024-04-16T02:35:40.785 INFO:tasks.workunit.client.0.smithi151.stdout:1/182: creat d3/d4/db/df/d38/f3b x:0 0 0 2024-04-16T02:35:40.785 INFO:tasks.workunit.client.1.smithi153.stdout:1/665: dread d4/dd/d13/d26/f33 [0,4194304] 0 2024-04-16T02:35:40.785 INFO:tasks.workunit.client.1.smithi153.stdout:9/668: creat d4/d6/dc/d24/dc6/fd0 x:0 0 0 2024-04-16T02:35:40.786 INFO:tasks.workunit.client.0.smithi151.stdout:4/199: fsync d0/d11/f23 0 2024-04-16T02:35:40.786 INFO:tasks.workunit.client.0.smithi151.stdout:1/183: fdatasync f1 0 2024-04-16T02:35:40.786 INFO:tasks.workunit.client.1.smithi153.stdout:8/591: creat d2/dba/fc9 x:0 0 0 2024-04-16T02:35:40.786 INFO:tasks.workunit.client.1.smithi153.stdout:1/666: chown d4/dd/d18/d6c/c94 91255 1 2024-04-16T02:35:40.787 INFO:tasks.workunit.client.0.smithi151.stdout:2/264: creat de/d15/f50 x:0 0 0 2024-04-16T02:35:40.787 INFO:tasks.workunit.client.0.smithi151.stdout:9/122: mknod d1/d3/d6/d1d/d22/d2c/c30 0 2024-04-16T02:35:40.788 INFO:tasks.workunit.client.0.smithi151.stdout:6/300: read - d2/d5/df/d20/f24 zero size 2024-04-16T02:35:40.788 INFO:tasks.workunit.client.1.smithi153.stdout:4/726: link d3/d9/db/de/d2c/d52/fa2 d3/d9/db/de/d2c/d52/d85/ffe 0 2024-04-16T02:35:40.789 INFO:tasks.workunit.client.0.smithi151.stdout:2/265: chown de/d15/f46 4 1 2024-04-16T02:35:40.789 INFO:tasks.workunit.client.1.smithi153.stdout:8/592: symlink d2/d1c/lca 0 2024-04-16T02:35:40.789 INFO:tasks.workunit.client.0.smithi151.stdout:3/132: creat d1/d6/d10/f28 x:0 0 0 2024-04-16T02:35:40.790 INFO:tasks.workunit.client.0.smithi151.stdout:7/145: truncate d0/d1f/f1e 591344 0 2024-04-16T02:35:40.790 INFO:tasks.workunit.client.0.smithi151.stdout:8/116: write d11/f1a [1342547,36815] 0 2024-04-16T02:35:40.793 INFO:tasks.workunit.client.0.smithi151.stdout:1/184: symlink d3/d4/l3c 0 2024-04-16T02:35:40.793 INFO:tasks.workunit.client.0.smithi151.stdout:6/301: mkdir d2/d13/d63 0 2024-04-16T02:35:40.793 INFO:tasks.workunit.client.0.smithi151.stdout:4/200: creat d0/d20/f42 x:0 0 0 2024-04-16T02:35:40.795 INFO:tasks.workunit.client.0.smithi151.stdout:1/185: write d3/d33/f36 [801401,33988] 0 2024-04-16T02:35:40.796 INFO:tasks.workunit.client.1.smithi153.stdout:8/593: symlink d2/d13/lcb 0 2024-04-16T02:35:40.796 INFO:tasks.workunit.client.1.smithi153.stdout:9/669: dwrite d4/d6/dc/d7f/fa3 [0,4194304] 0 2024-04-16T02:35:40.796 INFO:tasks.workunit.client.0.smithi151.stdout:2/266: creat de/d11/d42/f51 x:0 0 0 2024-04-16T02:35:40.797 INFO:tasks.workunit.client.1.smithi153.stdout:2/603: dread d9/d2b/f17 [0,4194304] 0 2024-04-16T02:35:40.797 INFO:tasks.workunit.client.1.smithi153.stdout:8/594: fdatasync d2/d93/d97/d9c/f7c 0 2024-04-16T02:35:40.798 INFO:tasks.workunit.client.0.smithi151.stdout:1/186: write d3/d33/f36 [868135,33638] 0 2024-04-16T02:35:40.798 INFO:tasks.workunit.client.0.smithi151.stdout:8/117: rename d11/d22/f23 to d11/d22/f26 0 2024-04-16T02:35:40.801 INFO:tasks.workunit.client.0.smithi151.stdout:5/234: dread d2/d6/d19/d1a/f1b [0,4194304] 0 2024-04-16T02:35:40.803 INFO:tasks.workunit.client.1.smithi153.stdout:9/670: creat d4/d33/fd1 x:0 0 0 2024-04-16T02:35:40.805 INFO:tasks.workunit.client.0.smithi151.stdout:9/123: dread d1/d3/d6/d1d/d22/f23 [0,4194304] 0 2024-04-16T02:35:40.806 INFO:tasks.workunit.client.0.smithi151.stdout:2/267: creat de/d11/d42/f52 x:0 0 0 2024-04-16T02:35:40.806 INFO:tasks.workunit.client.0.smithi151.stdout:7/146: creat d0/f22 x:0 0 0 2024-04-16T02:35:40.809 INFO:tasks.workunit.client.0.smithi151.stdout:3/133: getdents d1/d6/d22/d23 0 2024-04-16T02:35:40.810 INFO:tasks.workunit.client.0.smithi151.stdout:6/302: mknod d2/d5/df/d23/d45/d62/c64 0 2024-04-16T02:35:40.810 INFO:tasks.workunit.client.0.smithi151.stdout:7/147: mknod d0/d1f/c23 0 2024-04-16T02:35:40.810 INFO:tasks.workunit.client.0.smithi151.stdout:5/235: symlink d2/d6/d19/l45 0 2024-04-16T02:35:40.810 INFO:tasks.workunit.client.1.smithi153.stdout:9/671: rename d4/d33/f5b to d4/fd2 0 2024-04-16T02:35:40.811 INFO:tasks.workunit.client.0.smithi151.stdout:3/134: stat d1/d6/f1b 0 2024-04-16T02:35:40.811 INFO:tasks.workunit.client.0.smithi151.stdout:4/201: creat d0/d11/f43 x:0 0 0 2024-04-16T02:35:40.812 INFO:tasks.workunit.client.0.smithi151.stdout:9/124: creat d1/d3/d6/d17/f31 x:0 0 0 2024-04-16T02:35:40.812 INFO:tasks.workunit.client.0.smithi151.stdout:9/125: fsync d1/d3/d6/f2a 0 2024-04-16T02:35:40.814 INFO:tasks.workunit.client.0.smithi151.stdout:4/202: write d0/f8 [714568,46015] 0 2024-04-16T02:35:40.814 INFO:tasks.workunit.client.0.smithi151.stdout:9/126: stat d1/d3/d6/d17/f31 0 2024-04-16T02:35:40.815 INFO:tasks.workunit.client.0.smithi151.stdout:4/203: chown d0/fd 55177693 1 2024-04-16T02:35:40.816 INFO:tasks.workunit.client.0.smithi151.stdout:5/236: unlink d2/d6/c3d 0 2024-04-16T02:35:40.818 INFO:tasks.workunit.client.1.smithi153.stdout:9/672: dwrite d4/d6/dc/d24/d4e/f81 [0,4194304] 0 2024-04-16T02:35:40.820 INFO:tasks.workunit.client.0.smithi151.stdout:7/148: mknod d0/c24 0 2024-04-16T02:35:40.822 INFO:tasks.workunit.client.0.smithi151.stdout:6/303: mkdir d2/d1f/d22/d40/d3f/d65 0 2024-04-16T02:35:40.823 INFO:tasks.workunit.client.0.smithi151.stdout:3/135: mknod d1/d6/d22/d23/c29 0 2024-04-16T02:35:40.824 INFO:tasks.workunit.client.1.smithi153.stdout:9/673: creat d4/d33/dae/dc7/fd3 x:0 0 0 2024-04-16T02:35:40.825 INFO:tasks.workunit.client.1.smithi153.stdout:9/674: chown d4/d6/dc/d2d/da0/d69/fb7 3547 1 2024-04-16T02:35:40.827 INFO:tasks.workunit.client.0.smithi151.stdout:4/204: creat d0/d11/d19/f44 x:0 0 0 2024-04-16T02:35:40.830 INFO:tasks.workunit.client.0.smithi151.stdout:7/149: unlink d0/c15 0 2024-04-16T02:35:40.830 INFO:tasks.workunit.client.0.smithi151.stdout:6/304: chown d2/d5/df/d23/d31/d52 1023 1 2024-04-16T02:35:40.832 INFO:tasks.workunit.client.0.smithi151.stdout:8/118: write fe [307135,81101] 0 2024-04-16T02:35:40.836 INFO:tasks.workunit.client.0.smithi151.stdout:5/237: dwrite d2/d6/d19/d30/d36/f38 [0,4194304] 0 2024-04-16T02:35:40.838 INFO:tasks.workunit.client.0.smithi151.stdout:9/127: dwrite d1/d3/f1e [0,4194304] 0 2024-04-16T02:35:40.840 INFO:tasks.workunit.client.1.smithi153.stdout:9/675: dread d4/d6/d1b/d6e/f74 [0,4194304] 0 2024-04-16T02:35:40.840 INFO:tasks.workunit.client.1.smithi153.stdout:8/595: truncate d2/d13/f94 2604374 0 2024-04-16T02:35:40.843 INFO:tasks.workunit.client.1.smithi153.stdout:2/604: dwrite d9/d24/d27/d2c/d41/f56 [0,4194304] 0 2024-04-16T02:35:40.843 INFO:tasks.workunit.client.0.smithi151.stdout:3/136: rename d1/d6/d10 to d1/d6/d20/d2a 0 2024-04-16T02:35:40.843 INFO:tasks.workunit.client.0.smithi151.stdout:9/128: write d1/d3/fb [1836337,105179] 0 2024-04-16T02:35:40.846 INFO:tasks.workunit.client.0.smithi151.stdout:4/205: dread d0/fd [0,4194304] 0 2024-04-16T02:35:40.847 INFO:tasks.workunit.client.0.smithi151.stdout:1/187: dread d3/d33/d10/f18 [0,4194304] 0 2024-04-16T02:35:40.848 INFO:tasks.workunit.client.1.smithi153.stdout:9/676: dread d4/d6/d1b/d6e/f74 [0,4194304] 0 2024-04-16T02:35:40.850 INFO:tasks.workunit.client.1.smithi153.stdout:9/677: fsync d4/d33/fd1 0 2024-04-16T02:35:40.850 INFO:tasks.workunit.client.1.smithi153.stdout:8/596: mkdir d2/dcc 0 2024-04-16T02:35:40.853 INFO:tasks.workunit.client.0.smithi151.stdout:3/137: creat d1/d6/d22/f2b x:0 0 0 2024-04-16T02:35:40.854 INFO:tasks.workunit.client.1.smithi153.stdout:9/678: write d4/d6/dc/d24/f30 [890378,80664] 0 2024-04-16T02:35:40.854 INFO:tasks.workunit.client.0.smithi151.stdout:5/238: mknod d2/d6/c46 0 2024-04-16T02:35:40.856 INFO:tasks.workunit.client.0.smithi151.stdout:4/206: readlink d0/l38 0 2024-04-16T02:35:40.858 INFO:tasks.workunit.client.1.smithi153.stdout:8/597: dwrite d2/d13/d96/fb3 [0,4194304] 0 2024-04-16T02:35:40.861 INFO:tasks.workunit.client.1.smithi153.stdout:9/679: link d4/d6/dc/d2d/d64/db9/la9 d4/d6/dc/d2d/ld4 0 2024-04-16T02:35:40.862 INFO:tasks.workunit.client.0.smithi151.stdout:7/150: getdents d0 0 2024-04-16T02:35:40.864 INFO:tasks.workunit.client.1.smithi153.stdout:6/612: sync 2024-04-16T02:35:40.865 INFO:tasks.workunit.client.1.smithi153.stdout:9/680: truncate d4/d6/dc/d24/dc6/fd0 792154 0 2024-04-16T02:35:40.866 INFO:tasks.workunit.client.1.smithi153.stdout:8/598: dwrite d2/d93/d97/d9c/d27/d2d/f59 [0,4194304] 0 2024-04-16T02:35:40.869 INFO:tasks.workunit.client.1.smithi153.stdout:2/605: truncate d9/d24/d27/d2c/d41/f56 2070452 0 2024-04-16T02:35:40.874 INFO:tasks.workunit.client.1.smithi153.stdout:9/681: mkdir d4/d33/dae/dc7/dd5 0 2024-04-16T02:35:40.875 INFO:tasks.workunit.client.1.smithi153.stdout:2/606: fsync d9/d2b/f19 0 2024-04-16T02:35:40.876 INFO:tasks.workunit.client.0.smithi151.stdout:7/151: dwrite d0/f5 [0,4194304] 0 2024-04-16T02:35:40.876 INFO:tasks.workunit.client.1.smithi153.stdout:6/613: dwrite dc/d10/f6c [0,4194304] 0 2024-04-16T02:35:40.878 INFO:tasks.workunit.client.0.smithi151.stdout:6/305: dwrite d2/d5/fc [0,4194304] 0 2024-04-16T02:35:40.878 INFO:tasks.workunit.client.1.smithi153.stdout:9/682: mknod d4/d6/dc/d24/dc6/cd6 0 2024-04-16T02:35:40.880 INFO:tasks.workunit.client.1.smithi153.stdout:2/607: truncate d9/fb 351678 0 2024-04-16T02:35:40.880 INFO:tasks.workunit.client.0.smithi151.stdout:3/138: getdents d1/d6 0 2024-04-16T02:35:40.882 INFO:tasks.workunit.client.1.smithi153.stdout:6/614: symlink dc/d47/d93/lcf 0 2024-04-16T02:35:40.882 INFO:tasks.workunit.client.0.smithi151.stdout:7/152: chown d0/f6 291 1 2024-04-16T02:35:40.882 INFO:tasks.workunit.client.0.smithi151.stdout:5/239: write d2/d6/f34 [474048,107666] 0 2024-04-16T02:35:40.883 INFO:tasks.workunit.client.0.smithi151.stdout:6/306: stat d2/d5/la 0 2024-04-16T02:35:40.884 INFO:tasks.workunit.client.0.smithi151.stdout:3/139: creat d1/d6/d22/f2c x:0 0 0 2024-04-16T02:35:40.884 INFO:tasks.workunit.client.0.smithi151.stdout:6/307: chown d2/d5/df/d20/l42 18 1 2024-04-16T02:35:40.885 INFO:tasks.workunit.client.0.smithi151.stdout:7/153: creat d0/f25 x:0 0 0 2024-04-16T02:35:40.886 INFO:tasks.workunit.client.1.smithi153.stdout:2/608: mknod d9/d24/d34/d36/ccd 0 2024-04-16T02:35:40.888 INFO:tasks.workunit.client.0.smithi151.stdout:6/308: dread - d2/d5/f3e zero size 2024-04-16T02:35:40.889 INFO:tasks.workunit.client.0.smithi151.stdout:3/140: creat d1/d6/d22/d23/f2d x:0 0 0 2024-04-16T02:35:40.891 INFO:tasks.workunit.client.0.smithi151.stdout:7/154: dread - d0/f22 zero size 2024-04-16T02:35:40.892 INFO:tasks.workunit.client.1.smithi153.stdout:9/683: dwrite d4/d6/dc/d2d/da0/d69/fb7 [0,4194304] 0 2024-04-16T02:35:40.894 INFO:tasks.workunit.client.0.smithi151.stdout:6/309: creat d2/d1f/d22/d40/d3f/f66 x:0 0 0 2024-04-16T02:35:40.895 INFO:tasks.workunit.client.0.smithi151.stdout:4/207: dread d0/fe [0,4194304] 0 2024-04-16T02:35:40.896 INFO:tasks.workunit.client.0.smithi151.stdout:4/208: chown d0/f16 1617368 1 2024-04-16T02:35:40.897 INFO:tasks.workunit.client.0.smithi151.stdout:6/310: chown d2/d5/df/d23/c2f 2111733 1 2024-04-16T02:35:40.897 INFO:tasks.workunit.client.0.smithi151.stdout:3/141: creat d1/f2e x:0 0 0 2024-04-16T02:35:40.898 INFO:tasks.workunit.client.0.smithi151.stdout:6/311: write d2/d3d/f55 [1152710,37097] 0 2024-04-16T02:35:40.899 INFO:tasks.workunit.client.1.smithi153.stdout:6/615: dread dc/d1a/d2d/f44 [0,4194304] 0 2024-04-16T02:35:40.900 INFO:tasks.workunit.client.0.smithi151.stdout:7/155: unlink d0/f14 0 2024-04-16T02:35:40.901 INFO:tasks.workunit.client.0.smithi151.stdout:4/209: rmdir d0/d20 39 2024-04-16T02:35:40.902 INFO:tasks.workunit.client.1.smithi153.stdout:2/609: dread d9/d2b/d1c/d20/d88/f9b [0,4194304] 0 2024-04-16T02:35:40.904 INFO:tasks.workunit.client.0.smithi151.stdout:6/312: creat d2/d13/f67 x:0 0 0 2024-04-16T02:35:40.905 INFO:tasks.workunit.client.1.smithi153.stdout:6/616: rename dc/d10/fbc to dc/d47/d57/d77/dc4/fd0 0 2024-04-16T02:35:40.907 INFO:tasks.workunit.client.1.smithi153.stdout:6/617: creat dc/d47/d93/fd1 x:0 0 0 2024-04-16T02:35:40.909 INFO:tasks.workunit.client.1.smithi153.stdout:6/618: truncate dc/d10/d62/fce 662109 0 2024-04-16T02:35:40.910 INFO:tasks.workunit.client.0.smithi151.stdout:6/313: creat d2/d1f/d22/f68 x:0 0 0 2024-04-16T02:35:40.911 INFO:tasks.workunit.client.1.smithi153.stdout:2/610: creat d9/d24/d34/fce x:0 0 0 2024-04-16T02:35:40.911 INFO:tasks.workunit.client.0.smithi151.stdout:7/156: dread d0/f3 [0,4194304] 0 2024-04-16T02:35:40.912 INFO:tasks.workunit.client.1.smithi153.stdout:8/599: dread d2/d13/f1b [0,4194304] 0 2024-04-16T02:35:40.912 INFO:tasks.workunit.client.1.smithi153.stdout:9/684: link d4/d6/d1b/d2a/d99/fa4 d4/d6/dc/fd7 0 2024-04-16T02:35:40.912 INFO:tasks.workunit.client.0.smithi151.stdout:3/142: link d1/l27 d1/d6/d20/l2f 0 2024-04-16T02:35:40.913 INFO:tasks.workunit.client.0.smithi151.stdout:6/314: chown d2/d1f/d22/d40/d3f/f66 761 1 2024-04-16T02:35:40.913 INFO:tasks.workunit.client.0.smithi151.stdout:4/210: dwrite d0/f2 [0,4194304] 0 2024-04-16T02:35:40.914 INFO:tasks.workunit.client.1.smithi153.stdout:6/619: symlink dc/d47/d57/d77/d7a/d97/db7/ld2 0 2024-04-16T02:35:40.914 INFO:tasks.workunit.client.1.smithi153.stdout:7/592: sync 2024-04-16T02:35:40.914 INFO:tasks.workunit.client.1.smithi153.stdout:5/691: sync 2024-04-16T02:35:40.914 INFO:tasks.workunit.client.1.smithi153.stdout:3/595: sync 2024-04-16T02:35:40.915 INFO:tasks.workunit.client.1.smithi153.stdout:8/600: creat d2/d93/d97/d9c/d27/d5e/fcd x:0 0 0 2024-04-16T02:35:40.915 INFO:tasks.workunit.client.1.smithi153.stdout:9/685: rmdir d4/d6/dc/d2d/da0/d40 39 2024-04-16T02:35:40.916 INFO:tasks.workunit.client.1.smithi153.stdout:5/692: stat d8/d41/dbd/dd3 0 2024-04-16T02:35:40.916 INFO:tasks.workunit.client.0.smithi151.stdout:4/211: write d0/d11/f23 [272642,35694] 0 2024-04-16T02:35:40.916 INFO:tasks.workunit.client.1.smithi153.stdout:3/596: readlink d6/d10/d22/d2c/l9a 0 2024-04-16T02:35:40.917 INFO:tasks.workunit.client.0.smithi151.stdout:7/157: symlink d0/l26 0 2024-04-16T02:35:40.918 INFO:tasks.workunit.client.1.smithi153.stdout:2/611: creat d9/d2b/d5c/fcf x:0 0 0 2024-04-16T02:35:40.918 INFO:tasks.workunit.client.0.smithi151.stdout:6/315: chown d2/d5/d2e 0 1 2024-04-16T02:35:40.918 INFO:tasks.workunit.client.1.smithi153.stdout:6/620: symlink dc/d1a/d2d/ld3 0 2024-04-16T02:35:40.919 INFO:tasks.workunit.client.0.smithi151.stdout:4/212: mkdir d0/d11/d19/d31/d45 0 2024-04-16T02:35:40.920 INFO:tasks.workunit.client.1.smithi153.stdout:8/601: mknod d2/de/d3f/cce 0 2024-04-16T02:35:40.921 INFO:tasks.workunit.client.0.smithi151.stdout:7/158: chown d0/f11 7114372 1 2024-04-16T02:35:40.921 INFO:tasks.workunit.client.0.smithi151.stdout:6/316: truncate d2/d5/f8 1833267 0 2024-04-16T02:35:40.922 INFO:tasks.workunit.client.0.smithi151.stdout:6/317: readlink d2/d5/d2e/l36 0 2024-04-16T02:35:40.923 INFO:tasks.workunit.client.1.smithi153.stdout:5/693: creat d8/df/d84/da4/fdd x:0 0 0 2024-04-16T02:35:40.924 INFO:tasks.workunit.client.1.smithi153.stdout:7/593: dread d0/d4/db/d16/fb1 [0,4194304] 0 2024-04-16T02:35:40.924 INFO:tasks.workunit.client.0.smithi151.stdout:4/213: mkdir d0/d11/d19/d46 0 2024-04-16T02:35:40.926 INFO:tasks.workunit.client.1.smithi153.stdout:9/686: write d4/d6/dc/d24/f89 [550012,20054] 0 2024-04-16T02:35:40.927 INFO:tasks.workunit.client.1.smithi153.stdout:6/621: symlink dc/d10/d62/ld4 0 2024-04-16T02:35:40.928 INFO:tasks.workunit.client.1.smithi153.stdout:9/687: stat d4/d6/d1b/f71 0 2024-04-16T02:35:40.929 INFO:tasks.workunit.client.1.smithi153.stdout:6/622: truncate dc/d47/d93/fab 867866 0 2024-04-16T02:35:40.929 INFO:tasks.workunit.client.0.smithi151.stdout:5/240: dread d2/d6/fa [0,4194304] 0 2024-04-16T02:35:40.930 INFO:tasks.workunit.client.0.smithi151.stdout:7/159: truncate d0/f1d 3655914 0 2024-04-16T02:35:40.931 INFO:tasks.workunit.client.1.smithi153.stdout:0/749: sync 2024-04-16T02:35:40.931 INFO:tasks.workunit.client.1.smithi153.stdout:1/667: sync 2024-04-16T02:35:40.931 INFO:tasks.workunit.client.1.smithi153.stdout:4/727: sync 2024-04-16T02:35:40.931 INFO:tasks.workunit.client.1.smithi153.stdout:8/602: creat d2/d93/d97/d9c/d27/d2d/d47/d60/fcf x:0 0 0 2024-04-16T02:35:40.932 INFO:tasks.workunit.client.1.smithi153.stdout:8/603: fdatasync d2/f14 0 2024-04-16T02:35:40.933 INFO:tasks.workunit.client.0.smithi151.stdout:6/318: dwrite d2/d5/fc [0,4194304] 0 2024-04-16T02:35:40.933 INFO:tasks.workunit.client.0.smithi151.stdout:6/319: chown d2/f9 109325 1 2024-04-16T02:35:40.934 INFO:tasks.workunit.client.1.smithi153.stdout:3/597: rename d6/d10/d22/d43/c96 to d6/d10/d3b/d59/cd0 0 2024-04-16T02:35:40.935 INFO:tasks.workunit.client.1.smithi153.stdout:9/688: dread d4/d33/faf [0,4194304] 0 2024-04-16T02:35:40.936 INFO:tasks.workunit.client.0.smithi151.stdout:5/241: rename d2/f18 to d2/d6/d19/d1a/f47 0 2024-04-16T02:35:40.939 INFO:tasks.workunit.client.1.smithi153.stdout:0/750: symlink dd/d32/d57/lee 0 2024-04-16T02:35:40.941 INFO:tasks.workunit.client.1.smithi153.stdout:7/594: dwrite d0/d4/db/d2b/d43/d6d/f6f [0,4194304] 0 2024-04-16T02:35:40.941 INFO:tasks.workunit.client.1.smithi153.stdout:0/751: stat dd/d14/f4b 0 2024-04-16T02:35:40.943 INFO:tasks.workunit.client.0.smithi151.stdout:5/242: dwrite d2/d6/f9 [0,4194304] 0 2024-04-16T02:35:40.943 INFO:tasks.workunit.client.1.smithi153.stdout:3/598: chown d6/l15 72711 1 2024-04-16T02:35:40.943 INFO:tasks.workunit.client.1.smithi153.stdout:0/752: readlink dd/d14/ldc 0 2024-04-16T02:35:40.944 INFO:tasks.workunit.client.1.smithi153.stdout:6/623: symlink dc/d1a/ld5 0 2024-04-16T02:35:40.945 INFO:tasks.workunit.client.0.smithi151.stdout:6/320: symlink d2/d13/d5f/l69 0 2024-04-16T02:35:40.946 INFO:tasks.workunit.client.0.smithi151.stdout:5/243: chown d2/d6/f35 81308151 1 2024-04-16T02:35:40.948 INFO:tasks.workunit.client.0.smithi151.stdout:5/244: chown d2/d6/d19/d1a/f47 252 1 2024-04-16T02:35:40.949 INFO:tasks.workunit.client.1.smithi153.stdout:3/599: dwrite d6/d10/d22/d4d/fbd [0,4194304] 0 2024-04-16T02:35:40.949 INFO:tasks.workunit.client.0.smithi151.stdout:7/160: getdents d0/d1f 0 2024-04-16T02:35:40.950 INFO:tasks.workunit.client.0.smithi151.stdout:7/161: dread - d0/f22 zero size 2024-04-16T02:35:40.952 INFO:tasks.workunit.client.0.smithi151.stdout:4/214: rmdir d0/d11 39 2024-04-16T02:35:40.956 INFO:tasks.workunit.client.0.smithi151.stdout:6/321: creat d2/d1f/d22/d40/f6a x:0 0 0 2024-04-16T02:35:40.958 INFO:tasks.workunit.client.0.smithi151.stdout:5/245: creat d2/d6/d19/d1a/f48 x:0 0 0 2024-04-16T02:35:40.958 INFO:tasks.workunit.client.1.smithi153.stdout:1/668: rename d4/dd/d13/d23/c86 to d4/dd/d18/d29/d71/ce4 0 2024-04-16T02:35:40.959 INFO:tasks.workunit.client.1.smithi153.stdout:5/694: truncate d8/df/f97 556685 0 2024-04-16T02:35:40.962 INFO:tasks.workunit.client.0.smithi151.stdout:7/162: mkdir d0/d27 0 2024-04-16T02:35:40.963 INFO:tasks.workunit.client.1.smithi153.stdout:1/669: dread d4/dd/d20/f5b [0,4194304] 0 2024-04-16T02:35:40.964 INFO:tasks.workunit.client.0.smithi151.stdout:4/215: creat d0/d11/d19/d31/f47 x:0 0 0 2024-04-16T02:35:40.964 INFO:tasks.workunit.client.0.smithi151.stdout:5/246: mkdir d2/d6/d19/d30/d49 0 2024-04-16T02:35:40.965 INFO:tasks.workunit.client.1.smithi153.stdout:4/728: creat d3/d9/db/fff x:0 0 0 2024-04-16T02:35:40.966 INFO:tasks.workunit.client.1.smithi153.stdout:2/612: dwrite d9/d2b/d18/d3f/d40/f8e [0,4194304] 0 2024-04-16T02:35:40.967 INFO:tasks.workunit.client.1.smithi153.stdout:4/729: chown d3/d9/d24/d75/dc5/leb 4019 1 2024-04-16T02:35:40.968 INFO:tasks.workunit.client.1.smithi153.stdout:4/730: stat d3/d9/c9b 0 2024-04-16T02:35:40.968 INFO:tasks.workunit.client.0.smithi151.stdout:7/163: unlink d0/c21 0 2024-04-16T02:35:40.968 INFO:tasks.workunit.client.1.smithi153.stdout:9/689: rename d4/d33/f38 to d4/d6/dc/d24/d4e/fd8 0 2024-04-16T02:35:40.969 INFO:tasks.workunit.client.0.smithi151.stdout:5/247: dread d2/d6/f34 [0,4194304] 0 2024-04-16T02:35:40.970 INFO:tasks.workunit.client.0.smithi151.stdout:4/216: rename d0/d11/d19/f1e to d0/d11/d19/d31/d34/f48 0 2024-04-16T02:35:40.970 INFO:tasks.workunit.client.1.smithi153.stdout:0/753: symlink dd/d26/d44/dd2/lef 0 2024-04-16T02:35:40.974 INFO:tasks.workunit.client.1.smithi153.stdout:8/604: getdents d2 0 2024-04-16T02:35:40.975 INFO:tasks.workunit.client.1.smithi153.stdout:8/605: read - d2/d13/d2c/f8d zero size 2024-04-16T02:35:40.977 INFO:tasks.workunit.client.0.smithi151.stdout:5/248: link d2/f24 d2/d6/d43/f4a 0 2024-04-16T02:35:40.978 INFO:tasks.workunit.client.1.smithi153.stdout:4/731: mknod d3/d9/db/d17/d28/dc8/de4/dfd/c100 0 2024-04-16T02:35:40.978 INFO:tasks.workunit.client.0.smithi151.stdout:7/164: truncate d0/f18 976873 0 2024-04-16T02:35:40.979 INFO:tasks.workunit.client.1.smithi153.stdout:4/732: write d3/da5/dd2/fee [3843893,30008] 0 2024-04-16T02:35:40.980 INFO:tasks.workunit.client.1.smithi153.stdout:4/733: dread - d3/d9/db/d45/df3/f95 zero size 2024-04-16T02:35:40.980 INFO:tasks.workunit.client.0.smithi151.stdout:4/217: creat d0/d20/f49 x:0 0 0 2024-04-16T02:35:40.980 INFO:tasks.workunit.client.1.smithi153.stdout:0/754: mknod dd/d14/d19/d55/cf0 0 2024-04-16T02:35:40.981 INFO:tasks.workunit.client.1.smithi153.stdout:4/734: chown d3/d9/c20 21 1 2024-04-16T02:35:40.981 INFO:tasks.workunit.client.1.smithi153.stdout:4/735: fdatasync d3/fdc 0 2024-04-16T02:35:40.982 INFO:tasks.workunit.client.1.smithi153.stdout:4/736: chown d3/d9/db/d84/cc7 64356069 1 2024-04-16T02:35:40.982 INFO:tasks.workunit.client.0.smithi151.stdout:4/218: readlink d0/d20/l2a 0 2024-04-16T02:35:40.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:40 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:35:41.028 INFO:tasks.workunit.client.0.smithi151.stdout:7/165: dwrite d0/f3 [4194304,4194304] 0 2024-04-16T02:35:41.030 INFO:tasks.workunit.client.1.smithi153.stdout:8/606: unlink d2/de/d3f/f91 0 2024-04-16T02:35:41.031 INFO:tasks.workunit.client.0.smithi151.stdout:4/219: mkdir d0/d11/d19/d31/d4a 0 2024-04-16T02:35:41.032 INFO:tasks.workunit.client.1.smithi153.stdout:8/607: chown d2/de/l58 4765 1 2024-04-16T02:35:41.035 INFO:tasks.workunit.client.0.smithi151.stdout:7/166: creat d0/d1f/f28 x:0 0 0 2024-04-16T02:35:41.036 INFO:tasks.workunit.client.1.smithi153.stdout:5/695: creat d8/fde x:0 0 0 2024-04-16T02:35:41.038 INFO:tasks.workunit.client.1.smithi153.stdout:6/624: rename dc/d47/d57/d77/f24 to dc/d47/d7d/fd6 0 2024-04-16T02:35:41.039 INFO:tasks.workunit.client.0.smithi151.stdout:7/167: creat d0/d1f/f29 x:0 0 0 2024-04-16T02:35:41.040 INFO:tasks.workunit.client.1.smithi153.stdout:6/625: read dc/d47/d57/f91 [860897,77788] 0 2024-04-16T02:35:41.042 INFO:tasks.workunit.client.0.smithi151.stdout:7/168: creat d0/d1f/f2a x:0 0 0 2024-04-16T02:35:41.043 INFO:tasks.workunit.client.0.smithi151.stdout:4/220: dwrite d0/fb [4194304,4194304] 0 2024-04-16T02:35:41.045 INFO:tasks.workunit.client.0.smithi151.stdout:4/221: read - d0/d20/f21 zero size 2024-04-16T02:35:41.045 INFO:tasks.workunit.client.0.smithi151.stdout:6/322: dread d2/f9 [0,4194304] 0 2024-04-16T02:35:41.047 INFO:tasks.workunit.client.0.smithi151.stdout:5/249: dread d2/d6/d19/d30/f3a [0,4194304] 0 2024-04-16T02:35:41.049 INFO:tasks.workunit.client.0.smithi151.stdout:5/250: write d2/f1f [2328789,118010] 0 2024-04-16T02:35:41.050 INFO:tasks.workunit.client.0.smithi151.stdout:4/222: truncate d0/d11/d19/d31/f47 1031664 0 2024-04-16T02:35:41.050 INFO:tasks.workunit.client.0.smithi151.stdout:6/323: mknod d2/d5/df/d23/d56/c6b 0 2024-04-16T02:35:41.051 INFO:tasks.workunit.client.1.smithi153.stdout:4/737: fsync d3/d9/d24/f2b 0 2024-04-16T02:35:41.051 INFO:tasks.workunit.client.1.smithi153.stdout:7/595: dwrite d0/d4/db/d34/f79 [0,4194304] 0 2024-04-16T02:35:41.052 INFO:tasks.workunit.client.0.smithi151.stdout:6/324: chown d2/d5/f5a 99 1 2024-04-16T02:35:41.052 INFO:tasks.workunit.client.1.smithi153.stdout:2/613: creat d9/d24/d27/fd0 x:0 0 0 2024-04-16T02:35:41.053 INFO:tasks.workunit.client.0.smithi151.stdout:4/223: rmdir d0/d11 39 2024-04-16T02:35:41.053 INFO:tasks.workunit.client.1.smithi153.stdout:4/738: chown d3/d9/db/de/d2c/d98 14881323 1 2024-04-16T02:35:41.053 INFO:tasks.workunit.client.0.smithi151.stdout:7/169: dwrite d0/f11 [0,4194304] 0 2024-04-16T02:35:41.053 INFO:tasks.workunit.client.1.smithi153.stdout:2/614: fdatasync d9/d2b/d1c/f28 0 2024-04-16T02:35:41.054 INFO:tasks.workunit.client.0.smithi151.stdout:6/325: write d2/d5/f5a [279301,21802] 0 2024-04-16T02:35:41.054 INFO:tasks.workunit.client.1.smithi153.stdout:8/608: symlink d2/d93/d97/d9c/d27/d2d/d47/d60/d67/db7/ld0 0 2024-04-16T02:35:41.058 INFO:tasks.workunit.client.1.smithi153.stdout:3/600: truncate d6/d10/d22/d43/f69 1286111 0 2024-04-16T02:35:41.059 INFO:tasks.workunit.client.1.smithi153.stdout:6/626: readlink dc/d1a/l71 0 2024-04-16T02:35:41.062 INFO:tasks.workunit.client.0.smithi151.stdout:7/170: creat d0/d27/f2b x:0 0 0 2024-04-16T02:35:41.064 INFO:tasks.workunit.client.1.smithi153.stdout:5/696: dread d8/d41/dbd/d39/f4c [0,4194304] 0 2024-04-16T02:35:41.064 INFO:tasks.workunit.client.1.smithi153.stdout:3/601: dwrite d6/d10/d22/d2c/f5f [0,4194304] 0 2024-04-16T02:35:41.065 INFO:tasks.workunit.client.0.smithi151.stdout:6/326: link d2/d5/df/f3c d2/d1f/d22/d40/d3f/f6c 0 2024-04-16T02:35:41.066 INFO:tasks.workunit.client.1.smithi153.stdout:1/670: link d4/dd/d13/d8d/d95/lda d4/da2/d65/le5 0 2024-04-16T02:35:41.071 INFO:tasks.workunit.client.0.smithi151.stdout:4/224: rename d0/d20/f36 to d0/d11/f4b 0 2024-04-16T02:35:41.072 INFO:tasks.workunit.client.0.smithi151.stdout:7/171: mkdir d0/d2c 0 2024-04-16T02:35:41.072 INFO:tasks.workunit.client.1.smithi153.stdout:2/615: creat d9/d24/d27/d2c/d5d/da3/dc0/fd1 x:0 0 0 2024-04-16T02:35:41.073 INFO:tasks.workunit.client.0.smithi151.stdout:7/172: write d0/d27/f2b [821268,28034] 0 2024-04-16T02:35:41.074 INFO:tasks.workunit.client.1.smithi153.stdout:8/609: creat d2/d93/d97/d9c/d27/d2d/d47/fd1 x:0 0 0 2024-04-16T02:35:41.076 INFO:tasks.workunit.client.0.smithi151.stdout:4/225: unlink d0/fd 0 2024-04-16T02:35:41.077 INFO:tasks.workunit.client.0.smithi151.stdout:4/226: chown d0/d11/d19/f29 14614 1 2024-04-16T02:35:41.077 INFO:tasks.workunit.client.1.smithi153.stdout:9/690: getdents d4/d6/d1b 0 2024-04-16T02:35:41.078 INFO:tasks.workunit.client.0.smithi151.stdout:7/173: unlink d0/f25 0 2024-04-16T02:35:41.079 INFO:tasks.workunit.client.0.smithi151.stdout:7/174: read - d0/f22 zero size 2024-04-16T02:35:41.079 INFO:tasks.workunit.client.1.smithi153.stdout:9/691: write d4/d6/dc/d2d/d79/f8f [4748994,116926] 0 2024-04-16T02:35:41.079 INFO:tasks.workunit.client.1.smithi153.stdout:5/697: creat d8/df/d2e/fdf x:0 0 0 2024-04-16T02:35:41.080 INFO:tasks.workunit.client.1.smithi153.stdout:5/698: dread - d8/df/fba zero size 2024-04-16T02:35:41.081 INFO:tasks.workunit.client.1.smithi153.stdout:1/671: fdatasync d4/dd/d18/d29/fd2 0 2024-04-16T02:35:41.084 INFO:tasks.workunit.client.1.smithi153.stdout:2/616: write d9/d2b/f2a [1378572,67305] 0 2024-04-16T02:35:41.086 INFO:tasks.workunit.client.1.smithi153.stdout:1/672: dread d4/dd/d13/d26/db4/fcd [0,4194304] 0 2024-04-16T02:35:41.087 INFO:tasks.workunit.client.0.smithi151.stdout:4/227: dwrite d0/f1f [0,4194304] 0 2024-04-16T02:35:41.087 INFO:tasks.workunit.client.1.smithi153.stdout:6/627: mkdir dc/d1a/dd7 0 2024-04-16T02:35:41.088 INFO:tasks.workunit.client.0.smithi151.stdout:7/175: dwrite d0/f6 [4194304,4194304] 0 2024-04-16T02:35:41.091 INFO:tasks.workunit.client.0.smithi151.stdout:7/176: chown d0/l20 18 1 2024-04-16T02:35:41.092 INFO:tasks.workunit.client.0.smithi151.stdout:7/177: read - d0/d1f/f29 zero size 2024-04-16T02:35:41.092 INFO:tasks.workunit.client.0.smithi151.stdout:7/178: readlink d0/l20 0 2024-04-16T02:35:41.094 INFO:tasks.workunit.client.0.smithi151.stdout:4/228: rename d0/f2 to d0/f4c 0 2024-04-16T02:35:41.096 INFO:tasks.workunit.client.1.smithi153.stdout:3/602: unlink d6/d10/d3b/caa 0 2024-04-16T02:35:41.099 INFO:tasks.workunit.client.0.smithi151.stdout:7/179: rename d0/l12 to d0/d1f/l2d 0 2024-04-16T02:35:41.099 INFO:tasks.workunit.client.0.smithi151.stdout:4/229: mknod d0/d11/d19/d31/d4a/c4d 0 2024-04-16T02:35:41.103 INFO:tasks.workunit.client.0.smithi151.stdout:7/180: mkdir d0/d1f/d2e 0 2024-04-16T02:35:41.106 INFO:tasks.workunit.client.0.smithi151.stdout:7/181: symlink d0/d2c/l2f 0 2024-04-16T02:35:41.110 INFO:tasks.workunit.client.0.smithi151.stdout:7/182: creat d0/d1f/d2e/f30 x:0 0 0 2024-04-16T02:35:41.110 INFO:tasks.workunit.client.0.smithi151.stdout:4/230: dwrite d0/d11/d19/d31/f47 [0,4194304] 0 2024-04-16T02:35:41.120 INFO:tasks.workunit.client.0.smithi151.stdout:4/231: dwrite d0/d11/d19/f1b [4194304,4194304] 0 2024-04-16T02:35:41.121 INFO:tasks.workunit.client.0.smithi151.stdout:4/232: stat d0/d20/c22 0 2024-04-16T02:35:41.125 INFO:tasks.workunit.client.0.smithi151.stdout:4/233: fdatasync d0/d11/f39 0 2024-04-16T02:35:41.136 INFO:tasks.workunit.client.1.smithi153.stdout:4/739: unlink d3/d9/db/de/d2c/d38/fc3 0 2024-04-16T02:35:41.137 INFO:tasks.workunit.client.1.smithi153.stdout:7/596: dread d0/d4/f58 [0,4194304] 0 2024-04-16T02:35:41.137 INFO:tasks.workunit.client.1.smithi153.stdout:1/673: rmdir d4/dd/d18/d29 39 2024-04-16T02:35:41.138 INFO:tasks.workunit.client.1.smithi153.stdout:6/628: symlink dc/d6b/ld8 0 2024-04-16T02:35:41.139 INFO:tasks.workunit.client.0.smithi151.stdout:7/183: rename d0/d1f/d2e to d0/d1f/d31 0 2024-04-16T02:35:41.139 INFO:tasks.workunit.client.1.smithi153.stdout:6/629: dread - dc/d47/d93/fd1 zero size 2024-04-16T02:35:41.140 INFO:tasks.workunit.client.1.smithi153.stdout:8/610: write d2/d13/d2c/f32 [1795312,36364] 0 2024-04-16T02:35:41.140 INFO:tasks.workunit.client.1.smithi153.stdout:1/674: dread - d4/dd/d18/d6c/dd8/fe1 zero size 2024-04-16T02:35:41.140 INFO:tasks.workunit.client.0.smithi151.stdout:7/184: creat d0/d27/f32 x:0 0 0 2024-04-16T02:35:41.141 INFO:tasks.workunit.client.0.smithi151.stdout:7/185: fsync d0/d1f/f28 0 2024-04-16T02:35:41.142 INFO:tasks.workunit.client.1.smithi153.stdout:3/603: creat d6/d10/d3b/d76/fd1 x:0 0 0 2024-04-16T02:35:41.143 INFO:tasks.workunit.client.1.smithi153.stdout:9/692: creat d4/d6/dc/fd9 x:0 0 0 2024-04-16T02:35:41.144 INFO:tasks.workunit.client.1.smithi153.stdout:2/617: creat d9/d2b/d5c/fd2 x:0 0 0 2024-04-16T02:35:41.146 INFO:tasks.workunit.client.1.smithi153.stdout:0/755: dwrite dd/f53 [0,4194304] 0 2024-04-16T02:35:41.147 INFO:tasks.workunit.client.1.smithi153.stdout:4/740: creat d3/d9/db/de/f101 x:0 0 0 2024-04-16T02:35:41.147 INFO:tasks.workunit.client.0.smithi151.stdout:7/186: dread d0/f5 [4194304,4194304] 0 2024-04-16T02:35:41.149 INFO:tasks.workunit.client.0.smithi151.stdout:7/187: mkdir d0/d1f/d33 0 2024-04-16T02:35:41.150 INFO:tasks.workunit.client.0.smithi151.stdout:7/188: write d0/d1f/f28 [187209,19325] 0 2024-04-16T02:35:41.151 INFO:tasks.workunit.client.1.smithi153.stdout:2/618: creat d9/d24/d27/d2c/d5d/da3/fd3 x:0 0 0 2024-04-16T02:35:41.152 INFO:tasks.workunit.client.1.smithi153.stdout:3/604: rmdir d6/d10/d3b/d76 39 2024-04-16T02:35:41.154 INFO:tasks.workunit.client.1.smithi153.stdout:1/675: write d4/dd/d13/d23/f5d [1153774,117574] 0 2024-04-16T02:35:41.154 INFO:tasks.workunit.client.1.smithi153.stdout:4/741: creat d3/d9/db/de/d2c/d52/d85/f102 x:0 0 0 2024-04-16T02:35:41.155 INFO:tasks.workunit.client.1.smithi153.stdout:7/597: dwrite d0/d4/db/f2a [0,4194304] 0 2024-04-16T02:35:41.155 INFO:tasks.workunit.client.1.smithi153.stdout:6/630: getdents dc/d47/d57/d77/d7a/d97/db7 0 2024-04-16T02:35:41.159 INFO:tasks.workunit.client.1.smithi153.stdout:7/598: mknod d0/d4/db/d2b/d43/d6d/ccd 0 2024-04-16T02:35:41.163 INFO:tasks.workunit.client.1.smithi153.stdout:1/676: readlink d4/dd/d13/d23/d31/l60 0 2024-04-16T02:35:41.165 INFO:tasks.workunit.client.1.smithi153.stdout:0/756: getdents dd/d14/d19/d77/de3 0 2024-04-16T02:35:41.168 INFO:tasks.workunit.client.1.smithi153.stdout:8/611: truncate d2/d93/d97/d9c/d27/d2d/d47/d77/d8e/fbb 3116355 0 2024-04-16T02:35:41.170 INFO:tasks.workunit.client.1.smithi153.stdout:9/693: dwrite d4/d6/f9d [0,4194304] 0 2024-04-16T02:35:41.173 INFO:tasks.workunit.client.1.smithi153.stdout:0/757: truncate dd/d14/d19/d77/db6/fec 82531 0 2024-04-16T02:35:41.174 INFO:tasks.workunit.client.1.smithi153.stdout:6/631: link dc/d47/d57/d77/d15/f1e dc/d6b/fd9 0 2024-04-16T02:35:41.174 INFO:tasks.workunit.client.1.smithi153.stdout:0/758: fsync dd/d28/fe4 0 2024-04-16T02:35:41.175 INFO:tasks.workunit.client.1.smithi153.stdout:9/694: chown d4/d6 73796 1 2024-04-16T02:35:41.175 INFO:tasks.workunit.client.1.smithi153.stdout:3/605: dwrite d6/d10/d3b/fab [0,4194304] 0 2024-04-16T02:35:41.179 INFO:tasks.workunit.client.1.smithi153.stdout:3/606: write d6/d10/d22/d4d/dbb/fc0 [803272,16484] 0 2024-04-16T02:35:41.180 INFO:tasks.workunit.client.1.smithi153.stdout:6/632: stat dc/d1a/d2d/f44 0 2024-04-16T02:35:41.180 INFO:tasks.workunit.client.1.smithi153.stdout:7/599: creat d0/d4/db/d34/d4b/d4d/fce x:0 0 0 2024-04-16T02:35:41.182 INFO:tasks.workunit.client.1.smithi153.stdout:4/742: dwrite d3/d9/d24/d3e/ff8 [0,4194304] 0 2024-04-16T02:35:41.183 INFO:tasks.workunit.client.1.smithi153.stdout:9/695: write d4/d6/dc/d2d/da0/fb5 [1020052,55915] 0 2024-04-16T02:35:41.183 INFO:tasks.workunit.client.1.smithi153.stdout:1/677: getdents d4/dd/d13/d23/db6/dc2 0 2024-04-16T02:35:41.184 INFO:tasks.workunit.client.1.smithi153.stdout:6/633: readlink dc/d6b/lb5 0 2024-04-16T02:35:41.186 INFO:tasks.workunit.client.1.smithi153.stdout:9/696: truncate d4/fd2 4833468 0 2024-04-16T02:35:41.186 INFO:tasks.workunit.client.1.smithi153.stdout:8/612: chown d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/da4 14 1 2024-04-16T02:35:41.189 INFO:tasks.workunit.client.1.smithi153.stdout:8/613: stat d2/de/f24 0 2024-04-16T02:35:41.189 INFO:tasks.workunit.client.1.smithi153.stdout:4/743: mkdir d3/d9/d24/d3e/d103 0 2024-04-16T02:35:41.192 INFO:tasks.workunit.client.1.smithi153.stdout:3/607: fdatasync d6/d10/d22/d43/f69 0 2024-04-16T02:35:41.192 INFO:tasks.workunit.client.1.smithi153.stdout:7/600: dread d0/d4/db/d2b/f89 [0,4194304] 0 2024-04-16T02:35:41.192 INFO:tasks.workunit.client.1.smithi153.stdout:4/744: dread - d3/d9/d24/d75/d9e/fce zero size 2024-04-16T02:35:41.193 INFO:tasks.workunit.client.1.smithi153.stdout:0/759: creat dd/d14/d19/d77/ff1 x:0 0 0 2024-04-16T02:35:41.195 INFO:tasks.workunit.client.1.smithi153.stdout:8/614: write d2/d93/d97/d9c/d27/d2d/d47/d60/f79 [1893321,95011] 0 2024-04-16T02:35:41.200 INFO:tasks.workunit.client.1.smithi153.stdout:0/760: symlink dd/d32/d57/lf2 0 2024-04-16T02:35:41.201 INFO:tasks.workunit.client.1.smithi153.stdout:4/745: symlink d3/d9/db/de/d2c/d52/l104 0 2024-04-16T02:35:41.203 INFO:tasks.workunit.client.1.smithi153.stdout:9/697: dwrite d4/d33/dae/dc7/d66/dbd/fc5 [0,4194304] 0 2024-04-16T02:35:41.208 INFO:tasks.workunit.client.1.smithi153.stdout:7/601: dwrite d0/d13/f4a [8388608,4194304] 0 2024-04-16T02:35:41.208 INFO:tasks.workunit.client.1.smithi153.stdout:4/746: symlink d3/dad/l105 0 2024-04-16T02:35:41.210 INFO:tasks.workunit.client.1.smithi153.stdout:3/608: link d6/c7 d6/d10/d22/cd2 0 2024-04-16T02:35:41.211 INFO:tasks.workunit.client.1.smithi153.stdout:1/678: dwrite d4/dd/f1d [0,4194304] 0 2024-04-16T02:35:41.214 INFO:tasks.workunit.client.1.smithi153.stdout:4/747: mknod d3/dbc/c106 0 2024-04-16T02:35:41.219 INFO:tasks.workunit.client.1.smithi153.stdout:7/602: creat d0/d4/db/d47/d8d/d93/fcf x:0 0 0 2024-04-16T02:35:41.223 INFO:tasks.workunit.client.1.smithi153.stdout:9/698: getdents d4/d6/d1b/d2a/d99 0 2024-04-16T02:35:41.224 INFO:tasks.workunit.client.1.smithi153.stdout:1/679: fsync d4/dd/d18/f51 0 2024-04-16T02:35:41.225 INFO:tasks.workunit.client.1.smithi153.stdout:4/748: creat d3/d9/db/de/d2c/d52/d85/f107 x:0 0 0 2024-04-16T02:35:41.226 INFO:tasks.workunit.client.1.smithi153.stdout:8/615: dwrite d2/d13/f1d [0,4194304] 0 2024-04-16T02:35:41.226 INFO:tasks.workunit.client.1.smithi153.stdout:7/603: write d0/d4/db/d34/f79 [3690462,16566] 0 2024-04-16T02:35:41.227 INFO:tasks.workunit.client.1.smithi153.stdout:3/609: dwrite d6/d10/d22/d2c/f33 [4194304,4194304] 0 2024-04-16T02:35:41.228 INFO:tasks.workunit.client.1.smithi153.stdout:6/634: dread dc/d59/d63/f95 [0,4194304] 0 2024-04-16T02:35:41.231 INFO:tasks.workunit.client.1.smithi153.stdout:9/699: rmdir d4/d6/dc/d2d/d79/dbb 39 2024-04-16T02:35:41.232 INFO:tasks.workunit.client.1.smithi153.stdout:1/680: mkdir d4/dd/d13/d7e/de6 0 2024-04-16T02:35:41.234 INFO:tasks.workunit.client.1.smithi153.stdout:4/749: read d3/d9/db/d45/fcc [3307528,118133] 0 2024-04-16T02:35:41.237 INFO:tasks.workunit.client.1.smithi153.stdout:8/616: mknod d2/d93/d97/d9c/d27/d5e/d63/dc6/cd2 0 2024-04-16T02:35:41.239 INFO:tasks.workunit.client.1.smithi153.stdout:9/700: symlink d4/d33/dae/dc7/dd5/lda 0 2024-04-16T02:35:41.243 INFO:tasks.workunit.client.1.smithi153.stdout:3/610: rename d6/d10/d22/d2e/f6f to d6/d10/d22/d2c/fd3 0 2024-04-16T02:35:41.243 INFO:tasks.workunit.client.1.smithi153.stdout:8/617: fdatasync d2/d93/d97/d9c/d27/d2d/d47/d60/f79 0 2024-04-16T02:35:41.245 INFO:tasks.workunit.client.1.smithi153.stdout:6/635: dwrite dc/d6b/f73 [4194304,4194304] 0 2024-04-16T02:35:41.248 INFO:tasks.workunit.client.1.smithi153.stdout:1/681: dwrite d4/dd/d18/f76 [0,4194304] 0 2024-04-16T02:35:41.248 INFO:tasks.workunit.client.1.smithi153.stdout:9/701: unlink d4/d6/dc/d2d/d79/ca7 0 2024-04-16T02:35:41.250 INFO:tasks.workunit.client.1.smithi153.stdout:3/611: fsync d6/d10/d22/d2c/fba 0 2024-04-16T02:35:41.252 INFO:tasks.workunit.client.1.smithi153.stdout:8/618: unlink d2/c28 0 2024-04-16T02:35:41.257 INFO:tasks.workunit.client.1.smithi153.stdout:4/750: dwrite d3/d9/db/de/d2c/d52/d85/ffb [0,4194304] 0 2024-04-16T02:35:41.259 INFO:tasks.workunit.client.1.smithi153.stdout:4/751: dread - d3/d9/db/d45/df3/ffc zero size 2024-04-16T02:35:41.263 INFO:tasks.workunit.client.1.smithi153.stdout:1/682: dwrite d4/dd/d13/d23/fd9 [0,4194304] 0 2024-04-16T02:35:41.264 INFO:tasks.workunit.client.1.smithi153.stdout:0/761: dread dd/d14/d51/d54/f84 [0,4194304] 0 2024-04-16T02:35:41.265 INFO:tasks.workunit.client.1.smithi153.stdout:0/762: read - dd/d32/f85 zero size 2024-04-16T02:35:41.265 INFO:tasks.workunit.client.1.smithi153.stdout:4/752: truncate d3/d9/db/de/d2c/d52/d85/f107 472718 0 2024-04-16T02:35:41.266 INFO:tasks.workunit.client.1.smithi153.stdout:4/753: dread - d3/d9/db/d17/d28/f5e zero size 2024-04-16T02:35:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:40 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:35:41.279 INFO:tasks.workunit.client.1.smithi153.stdout:5/699: sync 2024-04-16T02:35:41.300 INFO:tasks.workunit.client.1.smithi153.stdout:3/612: write d6/d10/d22/d2e/f35 [7150990,92617] 0 2024-04-16T02:35:41.306 INFO:tasks.workunit.client.1.smithi153.stdout:3/613: dwrite d6/d10/d22/d2c/f58 [4194304,4194304] 0 2024-04-16T02:35:41.308 INFO:tasks.workunit.client.1.smithi153.stdout:6/636: creat dc/d47/d57/d77/d15/dac/dae/fda x:0 0 0 2024-04-16T02:35:41.309 INFO:tasks.workunit.client.1.smithi153.stdout:1/683: getdents d4/dd/d13/d7e/de6 0 2024-04-16T02:35:41.313 INFO:tasks.workunit.client.1.smithi153.stdout:9/702: rename d4/d6/dc/d2d/da0/d40 to d4/d6/ddb 0 2024-04-16T02:35:41.314 INFO:tasks.workunit.client.1.smithi153.stdout:5/700: getdents d8/df/d23/d70 0 2024-04-16T02:35:41.316 INFO:tasks.workunit.client.1.smithi153.stdout:6/637: dwrite dc/d10/f6c [4194304,4194304] 0 2024-04-16T02:35:41.317 INFO:tasks.workunit.client.1.smithi153.stdout:9/703: symlink d4/d6/ldc 0 2024-04-16T02:35:41.319 INFO:tasks.workunit.client.1.smithi153.stdout:1/684: dwrite d4/dd/d13/d23/fd3 [0,4194304] 0 2024-04-16T02:35:41.320 INFO:tasks.workunit.client.1.smithi153.stdout:9/704: chown d4/d6/ddb/d4a/f91 315 1 2024-04-16T02:35:41.322 INFO:tasks.workunit.client.1.smithi153.stdout:1/685: mknod d4/dd/dd4/ce7 0 2024-04-16T02:35:41.323 INFO:tasks.workunit.client.1.smithi153.stdout:9/705: rmdir d4/d6/ddb/d4a 39 2024-04-16T02:35:41.324 INFO:tasks.workunit.client.1.smithi153.stdout:6/638: truncate dc/d47/d57/d77/d15/f20 786531 0 2024-04-16T02:35:41.325 INFO:tasks.workunit.client.1.smithi153.stdout:5/701: dwrite d8/df/d23/f93 [0,4194304] 0 2024-04-16T02:35:41.328 INFO:tasks.workunit.client.1.smithi153.stdout:1/686: unlink d4/c24 0 2024-04-16T02:35:41.331 INFO:tasks.workunit.client.1.smithi153.stdout:5/702: rename d8/df/d28/f35 to d8/df/d28/d5a/d72/dd0/fe0 0 2024-04-16T02:35:41.333 INFO:tasks.workunit.client.1.smithi153.stdout:5/703: fsync d8/df/d23/f56 0 2024-04-16T02:35:41.333 INFO:tasks.workunit.client.1.smithi153.stdout:1/687: creat d4/dd/d13/d8d/fe8 x:0 0 0 2024-04-16T02:35:41.334 INFO:tasks.workunit.client.1.smithi153.stdout:9/706: creat d4/d6/dc/d2d/d64/fdd x:0 0 0 2024-04-16T02:35:41.335 INFO:tasks.workunit.client.1.smithi153.stdout:6/639: dwrite dc/d1a/d2d/f43 [0,4194304] 0 2024-04-16T02:35:41.336 INFO:tasks.workunit.client.1.smithi153.stdout:1/688: fdatasync d4/dd/d13/d26/d58/d7a/f83 0 2024-04-16T02:35:41.337 INFO:tasks.workunit.client.1.smithi153.stdout:9/707: truncate d4/d6/dc/d24/d4e/fd8 4229170 0 2024-04-16T02:35:41.338 INFO:tasks.workunit.client.1.smithi153.stdout:1/689: fsync d4/dd/d13/d23/d31/f88 0 2024-04-16T02:35:41.341 INFO:tasks.workunit.client.1.smithi153.stdout:6/640: rename dc/d6b/l96 to dc/d47/d57/ldb 0 2024-04-16T02:35:41.341 INFO:tasks.workunit.client.1.smithi153.stdout:1/690: truncate d4/dd/d13/d26/d58/f91 4745384 0 2024-04-16T02:35:41.346 INFO:tasks.workunit.client.1.smithi153.stdout:9/708: dwrite d4/d6/d8c/f95 [0,4194304] 0 2024-04-16T02:35:41.349 INFO:tasks.workunit.client.1.smithi153.stdout:9/709: getdents d4/d6/dc/d2d/d64/db9/dc1 0 2024-04-16T02:35:41.352 INFO:tasks.workunit.client.1.smithi153.stdout:9/710: creat d4/d33/dae/dc7/fde x:0 0 0 2024-04-16T02:35:41.358 INFO:tasks.workunit.client.1.smithi153.stdout:9/711: read d4/d6/d1b/f59 [2920890,28092] 0 2024-04-16T02:35:41.365 INFO:tasks.workunit.client.1.smithi153.stdout:7/604: write d0/fa [1602548,106539] 0 2024-04-16T02:35:41.367 INFO:tasks.workunit.client.1.smithi153.stdout:7/605: read d0/d13/f81 [2999948,116348] 0 2024-04-16T02:35:41.391 INFO:tasks.workunit.client.1.smithi153.stdout:8/619: truncate d2/d13/d2c/f32 480298 0 2024-04-16T02:35:41.392 INFO:tasks.workunit.client.1.smithi153.stdout:4/754: truncate d3/d9/f6c 412583 0 2024-04-16T02:35:41.394 INFO:tasks.workunit.client.1.smithi153.stdout:3/614: truncate d6/d10/d22/d2e/f35 5333159 0 2024-04-16T02:35:41.395 INFO:tasks.workunit.client.1.smithi153.stdout:3/615: truncate d6/d10/d22/d2e/d8a/fc3 510820 0 2024-04-16T02:35:41.397 INFO:tasks.workunit.client.1.smithi153.stdout:3/616: readlink d6/d10/d3b/l48 0 2024-04-16T02:35:41.400 INFO:tasks.workunit.client.1.smithi153.stdout:3/617: mknod d6/d10/d22/d4d/da0/cd4 0 2024-04-16T02:35:41.402 INFO:tasks.workunit.client.1.smithi153.stdout:3/618: truncate d6/d10/d22/d4d/fbf 665155 0 2024-04-16T02:35:41.402 INFO:tasks.workunit.client.1.smithi153.stdout:6/641: write f9 [1155695,9069] 0 2024-04-16T02:35:41.403 INFO:tasks.workunit.client.1.smithi153.stdout:9/712: write d4/d6/dc/d46/f55 [586016,58421] 0 2024-04-16T02:35:41.405 INFO:tasks.workunit.client.1.smithi153.stdout:3/619: mknod d6/d10/d22/d4d/cd5 0 2024-04-16T02:35:41.407 INFO:tasks.workunit.client.1.smithi153.stdout:9/713: mkdir d4/d6/d1b/ddf 0 2024-04-16T02:35:41.409 INFO:tasks.workunit.client.1.smithi153.stdout:2/619: sync 2024-04-16T02:35:41.409 INFO:tasks.workunit.client.1.smithi153.stdout:4/755: write d3/d9/db/d17/d28/f6f [565316,5099] 0 2024-04-16T02:35:41.411 INFO:tasks.workunit.client.1.smithi153.stdout:6/642: rename dc/d47/d57/d77/d15/dac/dae/fda to dc/d47/d57/d77/dc4/dc9/fdc 0 2024-04-16T02:35:41.414 INFO:tasks.workunit.client.1.smithi153.stdout:9/714: mknod d4/d6/dc/d2d/d64/db9/dc1/ce0 0 2024-04-16T02:35:41.417 INFO:tasks.workunit.client.1.smithi153.stdout:6/643: creat dc/d47/d7d/fdd x:0 0 0 2024-04-16T02:35:41.422 INFO:tasks.workunit.client.1.smithi153.stdout:6/644: write dc/d10/d62/f8b [1331747,4445] 0 2024-04-16T02:35:41.422 INFO:tasks.workunit.client.1.smithi153.stdout:4/756: rename d3/d9/db/de/d2c/d98/ld6 to d3/d9/db/l108 0 2024-04-16T02:35:41.422 INFO:tasks.workunit.client.1.smithi153.stdout:8/620: dwrite d2/d93/d97/d9c/d27/d2d/d47/f6c [0,4194304] 0 2024-04-16T02:35:41.423 INFO:tasks.workunit.client.1.smithi153.stdout:2/620: dwrite d9/d2b/d5c/da2/fc1 [0,4194304] 0 2024-04-16T02:35:41.425 INFO:tasks.workunit.client.1.smithi153.stdout:2/621: readlink d9/d2b/d18/l84 0 2024-04-16T02:35:41.426 INFO:tasks.workunit.client.1.smithi153.stdout:9/715: readlink d4/d6/d1b/l35 0 2024-04-16T02:35:41.426 INFO:tasks.workunit.client.1.smithi153.stdout:4/757: fdatasync d3/d9/db/de/d2c/f56 0 2024-04-16T02:35:41.431 INFO:tasks.workunit.client.1.smithi153.stdout:2/622: dread - d9/d24/d73/fc8 zero size 2024-04-16T02:35:41.432 INFO:tasks.workunit.client.1.smithi153.stdout:3/620: dread d6/d10/d22/f2a [0,4194304] 0 2024-04-16T02:35:41.433 INFO:tasks.workunit.client.1.smithi153.stdout:9/716: mknod d4/d6/d1b/d2a/d99/ce1 0 2024-04-16T02:35:41.433 INFO:tasks.workunit.client.1.smithi153.stdout:8/621: creat d2/d93/d97/d9c/d27/d2d/d47/fd3 x:0 0 0 2024-04-16T02:35:41.436 INFO:tasks.workunit.client.1.smithi153.stdout:8/622: chown d2/d93/d97/d9c/d27/fc3 23614 1 2024-04-16T02:35:41.436 INFO:tasks.workunit.client.1.smithi153.stdout:6/645: rename dc/d47/d57/d77/d7a/d97/fc8 to dc/d47/d57/d77/fde 0 2024-04-16T02:35:41.437 INFO:tasks.workunit.client.1.smithi153.stdout:6/646: dread - dc/d10/f88 zero size 2024-04-16T02:35:41.438 INFO:tasks.workunit.client.1.smithi153.stdout:9/717: chown d4/d6/d8c/caa 206 1 2024-04-16T02:35:41.439 INFO:tasks.workunit.client.1.smithi153.stdout:8/623: dread - d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/f89 zero size 2024-04-16T02:35:41.443 INFO:tasks.workunit.client.1.smithi153.stdout:9/718: read d4/d6/dc/d24/f89 [160943,123827] 0 2024-04-16T02:35:41.443 INFO:tasks.workunit.client.1.smithi153.stdout:6/647: symlink dc/d47/d57/d77/d7a/ldf 0 2024-04-16T02:35:41.443 INFO:tasks.workunit.client.1.smithi153.stdout:8/624: symlink d2/d13/d96/ld4 0 2024-04-16T02:35:41.444 INFO:tasks.workunit.client.1.smithi153.stdout:3/621: dwrite d6/d10/d22/d2c/fba [0,4194304] 0 2024-04-16T02:35:41.448 INFO:tasks.workunit.client.1.smithi153.stdout:6/648: creat dc/d59/d5b/fe0 x:0 0 0 2024-04-16T02:35:41.450 INFO:tasks.workunit.client.1.smithi153.stdout:3/622: rename d6/d10/d22/c9c to d6/d10/d3b/d59/cd6 0 2024-04-16T02:35:41.452 INFO:tasks.workunit.client.1.smithi153.stdout:8/625: creat d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d98/fd5 x:0 0 0 2024-04-16T02:35:41.453 INFO:tasks.workunit.client.1.smithi153.stdout:8/626: dread - d2/d13/fbe zero size 2024-04-16T02:35:41.454 INFO:tasks.workunit.client.1.smithi153.stdout:8/627: stat d2/d93/d97/fa9 0 2024-04-16T02:35:41.456 INFO:tasks.workunit.client.1.smithi153.stdout:8/628: read d2/d93/f9a [360183,111497] 0 2024-04-16T02:35:41.459 INFO:tasks.workunit.client.1.smithi153.stdout:8/629: mkdir d2/d13/d3a/dd6 0 2024-04-16T02:35:41.460 INFO:tasks.workunit.client.1.smithi153.stdout:4/758: dread d3/d9/d24/f4d [0,4194304] 0 2024-04-16T02:35:41.462 INFO:tasks.workunit.client.1.smithi153.stdout:4/759: creat d3/d9/db/d17/d3f/f109 x:0 0 0 2024-04-16T02:35:41.463 INFO:tasks.workunit.client.1.smithi153.stdout:8/630: rename d2/d13/d2c/f32 to d2/d93/d97/d9c/d27/d2d/d47/d60/fd7 0 2024-04-16T02:35:41.464 INFO:tasks.workunit.client.1.smithi153.stdout:4/760: creat d3/d9/db/de/f10a x:0 0 0 2024-04-16T02:35:41.465 INFO:tasks.workunit.client.1.smithi153.stdout:8/631: creat d2/d13/d3a/d5b/fd8 x:0 0 0 2024-04-16T02:35:41.465 INFO:tasks.workunit.client.1.smithi153.stdout:4/761: mknod d3/dbc/c10b 0 2024-04-16T02:35:41.470 INFO:tasks.workunit.client.1.smithi153.stdout:4/762: getdents d3/d9/db/d17/d28/dc8/de4 0 2024-04-16T02:35:41.470 INFO:tasks.workunit.client.1.smithi153.stdout:3/623: dread d6/d10/d22/d4d/f6e [0,4194304] 0 2024-04-16T02:35:41.471 INFO:tasks.workunit.client.1.smithi153.stdout:4/763: fdatasync d3/d9/db/d45/df3/fd3 0 2024-04-16T02:35:41.471 INFO:tasks.workunit.client.1.smithi153.stdout:3/624: write d6/d10/d22/d2c/f60 [407513,117263] 0 2024-04-16T02:35:41.472 INFO:tasks.workunit.client.1.smithi153.stdout:4/764: read - d3/d9/db/de/d2c/d98/fd7 zero size 2024-04-16T02:35:41.473 INFO:tasks.workunit.client.1.smithi153.stdout:4/765: fsync d3/d9/f77 0 2024-04-16T02:35:41.477 INFO:tasks.workunit.client.1.smithi153.stdout:3/625: dwrite d6/d10/d22/d2c/f33 [0,4194304] 0 2024-04-16T02:35:41.522 INFO:tasks.workunit.client.1.smithi153.stdout:0/763: sync 2024-04-16T02:35:41.543 INFO:tasks.workunit.client.1.smithi153.stdout:4/766: creat d3/d9/db/d17/d28/dc8/de4/dec/f10c x:0 0 0 2024-04-16T02:35:41.544 INFO:tasks.workunit.client.1.smithi153.stdout:8/632: truncate d2/f9b 897822 0 2024-04-16T02:35:41.545 INFO:tasks.workunit.client.1.smithi153.stdout:3/626: chown d6/d10/c49 1 1 2024-04-16T02:35:41.545 INFO:tasks.workunit.client.1.smithi153.stdout:8/633: chown d2/de/d34/f6f 3933 1 2024-04-16T02:35:41.546 INFO:tasks.workunit.client.1.smithi153.stdout:4/767: mkdir d3/dbc/d10d 0 2024-04-16T02:35:41.546 INFO:tasks.workunit.client.1.smithi153.stdout:8/634: chown d2/de/f5f 11713 1 2024-04-16T02:35:41.546 INFO:tasks.workunit.client.1.smithi153.stdout:0/764: getdents dd/d14/d19/d77/d87/dda 0 2024-04-16T02:35:41.547 INFO:tasks.workunit.client.1.smithi153.stdout:3/627: dread - d6/d10/d3b/d76/fc2 zero size 2024-04-16T02:35:41.547 INFO:tasks.workunit.client.1.smithi153.stdout:8/635: fdatasync d2/d93/d97/d9c/d27/d2d/d47/fd3 0 2024-04-16T02:35:41.548 INFO:tasks.workunit.client.1.smithi153.stdout:0/765: rename dd to dd/d14/d51/ddd/df3 22 2024-04-16T02:35:41.548 INFO:tasks.workunit.client.1.smithi153.stdout:4/768: mknod d3/d9/db/d17/c10e 0 2024-04-16T02:35:41.554 INFO:tasks.workunit.client.1.smithi153.stdout:8/636: dread d2/d93/d97/d9c/d27/f2a [0,4194304] 0 2024-04-16T02:35:41.555 INFO:tasks.workunit.client.1.smithi153.stdout:0/766: dwrite dd/d14/da6/da4/db1/fc9 [0,4194304] 0 2024-04-16T02:35:41.556 INFO:tasks.workunit.client.1.smithi153.stdout:8/637: truncate d2/d13/f9f 741635 0 2024-04-16T02:35:41.625 INFO:tasks.workunit.client.1.smithi153.stdout:7/606: sync 2024-04-16T02:35:41.625 INFO:tasks.workunit.client.1.smithi153.stdout:5/704: sync 2024-04-16T02:35:41.625 INFO:tasks.workunit.client.1.smithi153.stdout:1/691: sync 2024-04-16T02:35:41.627 INFO:tasks.workunit.client.1.smithi153.stdout:5/705: truncate d8/d41/dbd/d83/fc1 390704 0 2024-04-16T02:35:41.629 INFO:tasks.workunit.client.1.smithi153.stdout:4/769: rmdir d3/d9/db/d17/d28/dc8/de4/dec 39 2024-04-16T02:35:41.630 INFO:tasks.workunit.client.1.smithi153.stdout:8/638: creat d2/dba/fd9 x:0 0 0 2024-04-16T02:35:41.630 INFO:tasks.workunit.client.1.smithi153.stdout:4/770: fsync d3/d9/db/de/d2c/d52/d85/fef 0 2024-04-16T02:35:41.632 INFO:tasks.workunit.client.1.smithi153.stdout:0/767: rename dd/d14/d51/d54/d65/fa3 to dd/d14/d51/d7f/d9d/ff4 0 2024-04-16T02:35:41.632 INFO:tasks.workunit.client.1.smithi153.stdout:7/607: dwrite d0/d4/db/f2a [0,4194304] 0 2024-04-16T02:35:41.633 INFO:tasks.workunit.client.1.smithi153.stdout:7/608: chown d0/d4/db/d2b/l11 44 1 2024-04-16T02:35:41.633 INFO:tasks.workunit.client.1.smithi153.stdout:1/692: unlink d4/dd/d13/d8d/d95/lcb 0 2024-04-16T02:35:41.634 INFO:tasks.workunit.client.1.smithi153.stdout:3/628: truncate d6/d10/d22/d4d/dbb/fcd 108630 0 2024-04-16T02:35:41.638 INFO:tasks.workunit.client.1.smithi153.stdout:4/771: chown d3/d9/f6c 1 1 2024-04-16T02:35:41.638 INFO:tasks.workunit.client.1.smithi153.stdout:0/768: fsync dd/d26/f5b 0 2024-04-16T02:35:41.639 INFO:tasks.workunit.client.1.smithi153.stdout:3/629: write d6/d10/d3b/d59/fa9 [876132,75601] 0 2024-04-16T02:35:41.642 INFO:tasks.workunit.client.1.smithi153.stdout:7/609: truncate d0/d4/db/d2b/f4e 695520 0 2024-04-16T02:35:41.644 INFO:tasks.workunit.client.1.smithi153.stdout:4/772: truncate d3/d9/db/de/d2c/d52/d85/f102 790914 0 2024-04-16T02:35:41.645 INFO:tasks.workunit.client.1.smithi153.stdout:1/693: dwrite d4/dd/d13/d26/f34 [0,4194304] 0 2024-04-16T02:35:41.646 INFO:tasks.workunit.client.1.smithi153.stdout:8/639: mkdir d2/d93/d97/d9c/d27/dda 0 2024-04-16T02:35:41.649 INFO:tasks.workunit.client.1.smithi153.stdout:1/694: write d4/dd/d13/d26/f34 [3856589,14138] 0 2024-04-16T02:35:41.651 INFO:tasks.workunit.client.1.smithi153.stdout:6/649: sync 2024-04-16T02:35:41.651 INFO:tasks.workunit.client.1.smithi153.stdout:9/719: sync 2024-04-16T02:35:41.651 INFO:tasks.workunit.client.1.smithi153.stdout:2/623: sync 2024-04-16T02:35:41.651 INFO:tasks.workunit.client.1.smithi153.stdout:0/769: creat dd/d32/d57/d6c/dd1/ff5 x:0 0 0 2024-04-16T02:35:41.653 INFO:tasks.workunit.client.1.smithi153.stdout:7/610: unlink d0/d4/db/d2b/d53/d59/d76/dc8/la9 0 2024-04-16T02:35:41.654 INFO:tasks.workunit.client.1.smithi153.stdout:4/773: mknod d3/d9/db/d17/d28/dc8/de4/dfd/de3/c10f 0 2024-04-16T02:35:41.654 INFO:tasks.workunit.client.1.smithi153.stdout:1/695: mkdir d4/da2/d65/d82/de9 0 2024-04-16T02:35:41.657 INFO:tasks.workunit.client.1.smithi153.stdout:6/650: unlink dc/d47/d7d/fb6 0 2024-04-16T02:35:41.657 INFO:tasks.workunit.client.1.smithi153.stdout:0/770: write dd/d32/f85 [199220,11572] 0 2024-04-16T02:35:41.658 INFO:tasks.workunit.client.1.smithi153.stdout:4/774: chown d3/d9/d24/d75/d9e/df7 1 1 2024-04-16T02:35:41.659 INFO:tasks.workunit.client.1.smithi153.stdout:5/706: dwrite d8/df/d23/f32 [0,4194304] 0 2024-04-16T02:35:41.662 INFO:tasks.workunit.client.1.smithi153.stdout:8/640: rename d2/d13/d3a/dc2/dc4 to d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/da4/ddb 0 2024-04-16T02:35:41.666 INFO:tasks.workunit.client.1.smithi153.stdout:0/771: read - dd/d14/d19/d55/fa9 zero size 2024-04-16T02:35:41.669 INFO:tasks.workunit.client.1.smithi153.stdout:1/696: creat d4/dd/d13/d23/d31/fea x:0 0 0 2024-04-16T02:35:41.670 INFO:tasks.workunit.client.1.smithi153.stdout:6/651: unlink dc/d1a/d2d/ca5 0 2024-04-16T02:35:41.671 INFO:tasks.workunit.client.1.smithi153.stdout:9/720: dwrite d4/d6/ddb/f5e [0,4194304] 0 2024-04-16T02:35:41.676 INFO:tasks.workunit.client.1.smithi153.stdout:3/630: dread d6/d10/d3b/d76/f7d [0,4194304] 0 2024-04-16T02:35:41.676 INFO:tasks.workunit.client.1.smithi153.stdout:0/772: fdatasync dd/d14/d51/d54/fc7 0 2024-04-16T02:35:41.676 INFO:tasks.workunit.client.1.smithi153.stdout:2/624: dwrite d9/d2b/f46 [0,4194304] 0 2024-04-16T02:35:41.676 INFO:tasks.workunit.client.1.smithi153.stdout:5/707: rmdir d8/df/d84 39 2024-04-16T02:35:41.677 INFO:tasks.workunit.client.1.smithi153.stdout:7/611: symlink d0/d4/db/d2b/d53/ld0 0 2024-04-16T02:35:41.678 INFO:tasks.workunit.client.1.smithi153.stdout:7/612: stat d0 0 2024-04-16T02:35:41.678 INFO:tasks.workunit.client.1.smithi153.stdout:8/641: creat d2/d93/d97/d9c/fdc x:0 0 0 2024-04-16T02:35:41.679 INFO:tasks.workunit.client.1.smithi153.stdout:5/708: readlink d8/df/d2e/dd1/ld2 0 2024-04-16T02:35:41.680 INFO:tasks.workunit.client.1.smithi153.stdout:8/642: fdatasync d2/d13/d2c/f48 0 2024-04-16T02:35:41.681 INFO:tasks.workunit.client.1.smithi153.stdout:6/652: fsync dc/d47/d57/d77/d7a/d97/fcc 0 2024-04-16T02:35:41.682 INFO:tasks.workunit.client.1.smithi153.stdout:7/613: fsync d0/d4/db/d2b/d43/f9d 0 2024-04-16T02:35:41.682 INFO:tasks.workunit.client.1.smithi153.stdout:2/625: write d9/d24/d27/d2c/d5d/da3/fd3 [12642,28302] 0 2024-04-16T02:35:41.683 INFO:tasks.workunit.client.1.smithi153.stdout:7/614: chown d0 0 1 2024-04-16T02:35:41.683 INFO:tasks.workunit.client.1.smithi153.stdout:8/643: fdatasync d2/d93/d97/d9c/d27/d2d/f59 0 2024-04-16T02:35:41.685 INFO:tasks.workunit.client.1.smithi153.stdout:2/626: write d9/d2b/d1c/d20/d45/faf [738772,17172] 0 2024-04-16T02:35:41.687 INFO:tasks.workunit.client.1.smithi153.stdout:2/627: write d9/d2b/d1c/d20/fc2 [2988121,15930] 0 2024-04-16T02:35:41.689 INFO:tasks.workunit.client.1.smithi153.stdout:4/775: mkdir d3/d9/db/de/d2c/de2/d110 0 2024-04-16T02:35:41.691 INFO:tasks.workunit.client.0.smithi151.stdout:0/202: sync 2024-04-16T02:35:41.691 INFO:tasks.workunit.client.0.smithi151.stdout:0/203: stat dc/ff 0 2024-04-16T02:35:41.691 INFO:tasks.workunit.client.1.smithi153.stdout:1/697: rename d4/dd/d18/l3f to d4/dd/d18/d6c/leb 0 2024-04-16T02:35:41.693 INFO:tasks.workunit.client.0.smithi151.stdout:0/204: creat dc/d20/d2c/f3e x:0 0 0 2024-04-16T02:35:41.696 INFO:tasks.workunit.client.1.smithi153.stdout:2/628: write d9/d2b/d5c/da2/fcc [2166032,69718] 0 2024-04-16T02:35:41.698 INFO:tasks.workunit.client.1.smithi153.stdout:3/631: mkdir d6/d10/d3b/dd7 0 2024-04-16T02:35:41.698 INFO:tasks.workunit.client.1.smithi153.stdout:8/644: dread d2/d93/d97/d9c/d27/f2a [0,4194304] 0 2024-04-16T02:35:41.698 INFO:tasks.workunit.client.1.smithi153.stdout:4/776: symlink d3/d9/db/d17/d28/dc8/de4/dfd/de3/l111 0 2024-04-16T02:35:41.700 INFO:tasks.workunit.client.1.smithi153.stdout:9/721: creat d4/d6/dc/fe2 x:0 0 0 2024-04-16T02:35:41.700 INFO:tasks.workunit.client.1.smithi153.stdout:0/773: rmdir dd/d14/d19/d77/d87/de7 0 2024-04-16T02:35:41.700 INFO:tasks.workunit.client.1.smithi153.stdout:8/645: readlink d2/d13/d3a/d5b/l7d 0 2024-04-16T02:35:41.701 INFO:tasks.workunit.client.1.smithi153.stdout:1/698: mkdir d4/dd/d13/d8d/d95/dec 0 2024-04-16T02:35:41.703 INFO:tasks.workunit.client.1.smithi153.stdout:4/777: unlink d3/d9/db/de/d2c/d52/fa2 0 2024-04-16T02:35:41.704 INFO:tasks.workunit.client.1.smithi153.stdout:3/632: dread d6/d10/d22/d4d/da0/da3/fcc [0,4194304] 0 2024-04-16T02:35:41.705 INFO:tasks.workunit.client.1.smithi153.stdout:8/646: write d2/d93/d97/d9c/d27/d2d/d47/d60/fcf [351072,84301] 0 2024-04-16T02:35:41.705 INFO:tasks.workunit.client.1.smithi153.stdout:9/722: unlink d4/d6/d1b/d2a/cc3 0 2024-04-16T02:35:41.705 INFO:tasks.workunit.client.1.smithi153.stdout:0/774: write dd/d14/da6/de0/fe6 [24096,104908] 0 2024-04-16T02:35:41.708 INFO:tasks.workunit.client.1.smithi153.stdout:8/647: stat d2/dcc 0 2024-04-16T02:35:41.710 INFO:tasks.workunit.client.1.smithi153.stdout:7/615: dread d0/d4/db/d2b/f38 [0,4194304] 0 2024-04-16T02:35:41.710 INFO:tasks.workunit.client.1.smithi153.stdout:8/648: dread - d2/d92/f69 zero size 2024-04-16T02:35:41.710 INFO:tasks.workunit.client.1.smithi153.stdout:7/616: stat d0/d4/db/l84 0 2024-04-16T02:35:41.712 INFO:tasks.workunit.client.1.smithi153.stdout:2/629: link d9/d24/d27/d2c/d5d/c95 d9/d24/d34/cd4 0 2024-04-16T02:35:41.712 INFO:tasks.workunit.client.1.smithi153.stdout:8/649: chown d2/d93/d97/d9c/d27/d2d/d47/d60/cbc 155566 1 2024-04-16T02:35:41.712 INFO:tasks.workunit.client.1.smithi153.stdout:8/650: stat d2/f7 0 2024-04-16T02:35:41.712 INFO:tasks.workunit.client.0.smithi151.stdout:0/205: dread f3 [0,4194304] 0 2024-04-16T02:35:41.714 INFO:tasks.workunit.client.1.smithi153.stdout:8/651: stat d2/de/d34/l3b 0 2024-04-16T02:35:41.714 INFO:tasks.workunit.client.1.smithi153.stdout:9/723: rmdir d4/d6/dc/d2d/d64/db9/dc1 39 2024-04-16T02:35:41.715 INFO:tasks.workunit.client.1.smithi153.stdout:5/709: dread d8/d1f/f3f [0,4194304] 0 2024-04-16T02:35:41.715 INFO:tasks.workunit.client.0.smithi151.stdout:0/206: creat dc/d3c/f3f x:0 0 0 2024-04-16T02:35:41.717 INFO:tasks.workunit.client.0.smithi151.stdout:0/207: creat dc/d15/f40 x:0 0 0 2024-04-16T02:35:41.717 INFO:tasks.workunit.client.0.smithi151.stdout:0/208: chown dc/d20/d2c/f3e 3280 1 2024-04-16T02:35:41.718 INFO:tasks.workunit.client.1.smithi153.stdout:2/630: creat d9/d2b/d1c/fd5 x:0 0 0 2024-04-16T02:35:41.719 INFO:tasks.workunit.client.1.smithi153.stdout:2/631: fsync d9/d2b/d18/d3f/f6e 0 2024-04-16T02:35:41.721 INFO:tasks.workunit.client.1.smithi153.stdout:9/724: dwrite d4/d6/f9d [0,4194304] 0 2024-04-16T02:35:41.721 INFO:tasks.workunit.client.1.smithi153.stdout:8/652: fdatasync d2/d93/f9a 0 2024-04-16T02:35:41.722 INFO:tasks.workunit.client.1.smithi153.stdout:9/725: stat d4/d6/d8c 0 2024-04-16T02:35:41.722 INFO:tasks.workunit.client.1.smithi153.stdout:9/726: readlink d4/lad 0 2024-04-16T02:35:41.724 INFO:tasks.workunit.client.1.smithi153.stdout:4/778: rename d3/d9/db/de/d2c/d52/c86 to d3/d9/db/de/c112 0 2024-04-16T02:35:41.725 INFO:tasks.workunit.client.0.smithi151.stdout:0/209: dwrite f3 [4194304,4194304] 0 2024-04-16T02:35:41.726 INFO:tasks.workunit.client.0.smithi151.stdout:0/210: write dc/f11 [4756892,30714] 0 2024-04-16T02:35:41.727 INFO:tasks.workunit.client.1.smithi153.stdout:4/779: fdatasync d3/d9/db/de/d2c/d52/d85/f102 0 2024-04-16T02:35:41.729 INFO:tasks.workunit.client.1.smithi153.stdout:7/617: link d0/d4/db/d47/f94 d0/d4/db/d2b/d53/d59/d76/fd1 0 2024-04-16T02:35:41.730 INFO:tasks.workunit.client.0.smithi151.stdout:0/211: rename dc/l36 to dc/d3c/l41 0 2024-04-16T02:35:41.732 INFO:tasks.workunit.client.0.smithi151.stdout:0/212: creat dc/d20/f42 x:0 0 0 2024-04-16T02:35:41.732 INFO:tasks.workunit.client.0.smithi151.stdout:0/213: readlink dc/d15/l2a 0 2024-04-16T02:35:41.734 INFO:tasks.workunit.client.0.smithi151.stdout:0/214: read dc/d15/f2b [104386,44225] 0 2024-04-16T02:35:41.735 INFO:tasks.workunit.client.1.smithi153.stdout:0/775: getdents dd/d32/d57 0 2024-04-16T02:35:41.736 INFO:tasks.workunit.client.1.smithi153.stdout:0/776: write dd/d32/d57/f42 [2941120,124942] 0 2024-04-16T02:35:41.737 INFO:tasks.workunit.client.0.smithi151.stdout:0/215: creat dc/f43 x:0 0 0 2024-04-16T02:35:41.740 INFO:tasks.workunit.client.1.smithi153.stdout:5/710: creat d8/df/d28/d5a/fe1 x:0 0 0 2024-04-16T02:35:41.743 INFO:tasks.workunit.client.1.smithi153.stdout:0/777: dwrite dd/d28/f4c [0,4194304] 0 2024-04-16T02:35:41.747 INFO:tasks.workunit.client.1.smithi153.stdout:5/711: rename d8/df/d2e/la5 to d8/d41/dbd/dd3/le2 0 2024-04-16T02:35:41.747 INFO:tasks.workunit.client.1.smithi153.stdout:3/633: rmdir d6 39 2024-04-16T02:35:41.749 INFO:tasks.workunit.client.1.smithi153.stdout:4/780: getdents d3/d9/db/d17/d28/dc8/de4/dfd/de3 0 2024-04-16T02:35:41.751 INFO:tasks.workunit.client.1.smithi153.stdout:3/634: stat d6/d10/d22/d2c/f44 0 2024-04-16T02:35:41.751 INFO:tasks.workunit.client.1.smithi153.stdout:0/778: dwrite dd/d14/da6/f3e [4194304,4194304] 0 2024-04-16T02:35:41.753 INFO:tasks.workunit.client.1.smithi153.stdout:4/781: mknod d3/da5/c113 0 2024-04-16T02:35:41.753 INFO:tasks.workunit.client.1.smithi153.stdout:7/618: dread d0/d4/db/d2b/d43/d6d/d68/f9c [0,4194304] 0 2024-04-16T02:35:41.755 INFO:tasks.workunit.client.1.smithi153.stdout:4/782: chown d3/d9/db/d17/d28/c7d 13 1 2024-04-16T02:35:41.756 INFO:tasks.workunit.client.1.smithi153.stdout:5/712: mknod d8/df/d28/d5a/d66/ce3 0 2024-04-16T02:35:41.756 INFO:tasks.workunit.client.1.smithi153.stdout:5/713: fdatasync d8/d41/fb3 0 2024-04-16T02:35:41.757 INFO:tasks.workunit.client.1.smithi153.stdout:5/714: chown d8/l17 0 1 2024-04-16T02:35:41.758 INFO:tasks.workunit.client.1.smithi153.stdout:3/635: creat d6/d10/d22/d4d/da0/fd8 x:0 0 0 2024-04-16T02:35:41.758 INFO:tasks.workunit.client.1.smithi153.stdout:3/636: chown c4 55784097 1 2024-04-16T02:35:41.759 INFO:tasks.workunit.client.0.smithi151.stdout:0/216: truncate f3 4133971 0 2024-04-16T02:35:41.759 INFO:tasks.workunit.client.1.smithi153.stdout:3/637: chown d6/d10/d22/f28 1697225916 1 2024-04-16T02:35:41.760 INFO:tasks.workunit.client.1.smithi153.stdout:3/638: truncate d6/d10/d22/fb1 980589 0 2024-04-16T02:35:41.763 INFO:tasks.workunit.client.0.smithi151.stdout:0/217: creat dc/d15/d34/d3b/f44 x:0 0 0 2024-04-16T02:35:41.766 INFO:tasks.workunit.client.1.smithi153.stdout:7/619: fdatasync d0/d4/db/ff 0 2024-04-16T02:35:41.766 INFO:tasks.workunit.client.1.smithi153.stdout:1/699: write d4/dd/d13/d26/f2b [2878091,75820] 0 2024-04-16T02:35:41.767 INFO:tasks.workunit.client.1.smithi153.stdout:7/620: chown d0/d4/db/d2b/d43/d6d/ccd 166768 1 2024-04-16T02:35:41.768 INFO:tasks.workunit.client.1.smithi153.stdout:4/783: mkdir d3/d9/db/d17/d28/dc8/de4/dfd/d114 0 2024-04-16T02:35:41.776 INFO:tasks.workunit.client.1.smithi153.stdout:2/632: write d9/d2b/d1c/f90 [977635,16436] 0 2024-04-16T02:35:41.777 INFO:tasks.workunit.client.1.smithi153.stdout:7/621: mknod d0/d4/db/d2b/cd2 0 2024-04-16T02:35:41.778 INFO:tasks.workunit.client.1.smithi153.stdout:2/633: stat d9/d2b/d1c/d20/fc2 0 2024-04-16T02:35:41.779 INFO:tasks.workunit.client.1.smithi153.stdout:2/634: readlink d9/d2b/d18/l89 0 2024-04-16T02:35:41.779 INFO:tasks.workunit.client.1.smithi153.stdout:8/653: truncate d2/d13/d3a/f46 1229330 0 2024-04-16T02:35:41.780 INFO:tasks.workunit.client.0.smithi151.stdout:0/218: truncate dc/d15/f17 389466 0 2024-04-16T02:35:41.781 INFO:tasks.workunit.client.0.smithi151.stdout:0/219: chown dc/f37 10 1 2024-04-16T02:35:41.784 INFO:tasks.workunit.client.1.smithi153.stdout:0/779: rename dd/d14/d51/lc2 to dd/d14/d19/d77/lf6 0 2024-04-16T02:35:41.785 INFO:tasks.workunit.client.0.smithi151.stdout:0/220: rmdir dc/d15 39 2024-04-16T02:35:41.786 INFO:tasks.workunit.client.1.smithi153.stdout:1/700: mkdir d4/dd/d13/ded 0 2024-04-16T02:35:41.787 INFO:tasks.workunit.client.0.smithi151.stdout:0/221: stat dc/d15/d34/d3b/f44 0 2024-04-16T02:35:41.790 INFO:tasks.workunit.client.1.smithi153.stdout:2/635: creat d9/d24/d27/d2c/d41/fd6 x:0 0 0 2024-04-16T02:35:41.790 INFO:tasks.workunit.client.1.smithi153.stdout:2/636: stat d9/c22 0 2024-04-16T02:35:41.790 INFO:tasks.workunit.client.1.smithi153.stdout:6/653: sync 2024-04-16T02:35:41.791 INFO:tasks.workunit.client.1.smithi153.stdout:3/639: creat d6/d10/d22/d2c/fd9 x:0 0 0 2024-04-16T02:35:41.792 INFO:tasks.workunit.client.1.smithi153.stdout:8/654: rename d2/ld to d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/ldd 0 2024-04-16T02:35:41.794 INFO:tasks.workunit.client.1.smithi153.stdout:5/715: write d8/df/fbf [744846,118673] 0 2024-04-16T02:35:41.794 INFO:tasks.workunit.client.1.smithi153.stdout:1/701: mknod d4/da2/d65/d84/cee 0 2024-04-16T02:35:41.801 INFO:tasks.workunit.client.1.smithi153.stdout:1/702: write d4/dd/d13/fb3 [390605,106515] 0 2024-04-16T02:35:41.802 INFO:tasks.workunit.client.1.smithi153.stdout:2/637: dwrite d9/d2b/fb6 [0,4194304] 0 2024-04-16T02:35:41.802 INFO:tasks.workunit.client.1.smithi153.stdout:6/654: rename dc/d1a/c9f to dc/d59/ce1 0 2024-04-16T02:35:41.802 INFO:tasks.workunit.client.1.smithi153.stdout:0/780: dread dd/d14/da6/d73/f8f [0,4194304] 0 2024-04-16T02:35:41.804 INFO:tasks.workunit.client.1.smithi153.stdout:6/655: readlink dc/d47/d57/d77/d15/la1 0 2024-04-16T02:35:41.805 INFO:tasks.workunit.client.1.smithi153.stdout:5/716: creat d8/d41/dbd/d39/d6d/fe4 x:0 0 0 2024-04-16T02:35:41.805 INFO:tasks.workunit.client.1.smithi153.stdout:7/622: dread d0/d4/db/d34/d4b/f69 [0,4194304] 0 2024-04-16T02:35:41.805 INFO:tasks.workunit.client.1.smithi153.stdout:1/703: write d4/f7 [2578626,90024] 0 2024-04-16T02:35:41.806 INFO:tasks.workunit.client.1.smithi153.stdout:6/656: readlink dc/d10/l27 0 2024-04-16T02:35:41.806 INFO:tasks.workunit.client.1.smithi153.stdout:4/784: write d3/d9/f1a [2000314,87501] 0 2024-04-16T02:35:41.807 INFO:tasks.workunit.client.1.smithi153.stdout:3/640: rename d6/d10/d3b/f92 to d6/d10/d22/d4d/fda 0 2024-04-16T02:35:41.816 INFO:tasks.workunit.client.1.smithi153.stdout:9/727: sync 2024-04-16T02:35:41.816 INFO:tasks.workunit.client.1.smithi153.stdout:7/623: chown d0/d4/c10 37623559 1 2024-04-16T02:35:41.817 INFO:tasks.workunit.client.1.smithi153.stdout:8/655: rename d2/d92/f78 to d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/fde 0 2024-04-16T02:35:41.818 INFO:tasks.workunit.client.1.smithi153.stdout:4/785: dwrite d3/d9/db/d17/d3f/f94 [0,4194304] 0 2024-04-16T02:35:41.819 INFO:tasks.workunit.client.1.smithi153.stdout:3/641: dwrite d6/d10/d22/d2c/fd9 [0,4194304] 0 2024-04-16T02:35:41.821 INFO:tasks.workunit.client.1.smithi153.stdout:6/657: dread dc/d10/f33 [0,4194304] 0 2024-04-16T02:35:41.823 INFO:tasks.workunit.client.1.smithi153.stdout:5/717: creat d8/df/d28/d64/fe5 x:0 0 0 2024-04-16T02:35:41.825 INFO:tasks.workunit.client.1.smithi153.stdout:8/656: dread d2/d92/faa [0,4194304] 0 2024-04-16T02:35:41.826 INFO:tasks.workunit.client.1.smithi153.stdout:5/718: chown d8/df/fbf 7391935 1 2024-04-16T02:35:41.827 INFO:tasks.workunit.client.1.smithi153.stdout:6/658: chown dc/d1a/d2d/f43 6629640 1 2024-04-16T02:35:41.827 INFO:tasks.workunit.client.1.smithi153.stdout:1/704: rmdir d4/dd/d13/d23/db6/dc2 0 2024-04-16T02:35:41.827 INFO:tasks.workunit.client.1.smithi153.stdout:9/728: chown d4/d6/dc/d2d/da0/d69/c7b 2411 1 2024-04-16T02:35:41.829 INFO:tasks.workunit.client.1.smithi153.stdout:4/786: creat d3/d9/db/de/d2c/d52/f115 x:0 0 0 2024-04-16T02:35:41.831 INFO:tasks.workunit.client.1.smithi153.stdout:8/657: write d2/d93/d97/d9c/d27/fb9 [77992,74620] 0 2024-04-16T02:35:41.833 INFO:tasks.workunit.client.1.smithi153.stdout:4/787: chown d3/d9/db/de/fa9 2 1 2024-04-16T02:35:41.835 INFO:tasks.workunit.client.1.smithi153.stdout:2/638: dwrite d9/d24/d27/d2c/d41/f78 [0,4194304] 0 2024-04-16T02:35:41.837 INFO:tasks.workunit.client.1.smithi153.stdout:8/658: chown d2/d93/d97/d9c/d27/d2d/d4b 3252 1 2024-04-16T02:35:41.842 INFO:tasks.workunit.client.1.smithi153.stdout:3/642: link d6/d10/d5b/lb5 d6/d10/d22/d4d/dbb/ldb 0 2024-04-16T02:35:41.844 INFO:tasks.workunit.client.1.smithi153.stdout:0/781: dread dd/d14/d19/f27 [0,4194304] 0 2024-04-16T02:35:41.844 INFO:tasks.workunit.client.1.smithi153.stdout:9/729: creat d4/d6/dc/fe3 x:0 0 0 2024-04-16T02:35:41.846 INFO:tasks.workunit.client.1.smithi153.stdout:1/705: truncate d4/dd/d18/d29/d71/f7b 185236 0 2024-04-16T02:35:41.846 INFO:tasks.workunit.client.1.smithi153.stdout:6/659: dread dc/d47/d57/d77/d15/f23 [0,4194304] 0 2024-04-16T02:35:41.846 INFO:tasks.workunit.client.1.smithi153.stdout:7/624: getdents d0/d4/db/d34/d4b/daa 0 2024-04-16T02:35:41.847 INFO:tasks.workunit.client.1.smithi153.stdout:5/719: dwrite d8/d41/fb3 [0,4194304] 0 2024-04-16T02:35:41.848 INFO:tasks.workunit.client.1.smithi153.stdout:3/643: unlink d6/d10/d22/d43/lc4 0 2024-04-16T02:35:41.851 INFO:tasks.workunit.client.1.smithi153.stdout:7/625: chown d0/d4/db/d2b/d43 2 1 2024-04-16T02:35:41.852 INFO:tasks.workunit.client.1.smithi153.stdout:1/706: chown d4/dd/d18/d29/l6a 7634789 1 2024-04-16T02:35:41.852 INFO:tasks.workunit.client.1.smithi153.stdout:9/730: chown d4/d6/d1b/f71 97 1 2024-04-16T02:35:41.854 INFO:tasks.workunit.client.1.smithi153.stdout:8/659: dread d2/d93/d97/d9c/d27/f2a [0,4194304] 0 2024-04-16T02:35:41.854 INFO:tasks.workunit.client.1.smithi153.stdout:3/644: dread - d6/d10/d22/d4d/fda zero size 2024-04-16T02:35:41.855 INFO:tasks.workunit.client.1.smithi153.stdout:1/707: stat d4/dd/d18/d29/d71 0 2024-04-16T02:35:41.857 INFO:tasks.workunit.client.1.smithi153.stdout:4/788: dwrite d3/d9/d24/d75/dc5/ff2 [0,4194304] 0 2024-04-16T02:35:41.858 INFO:tasks.workunit.client.1.smithi153.stdout:3/645: chown d6/d10/d3b/dd7 670577 1 2024-04-16T02:35:41.860 INFO:tasks.workunit.client.1.smithi153.stdout:1/708: dread - d4/dd/d13/d23/fab zero size 2024-04-16T02:35:41.862 INFO:tasks.workunit.client.1.smithi153.stdout:4/789: read - d3/d9/db/d17/d3f/f109 zero size 2024-04-16T02:35:41.862 INFO:tasks.workunit.client.1.smithi153.stdout:3/646: chown d6/d10/d22/d4d/fbd 15394559 1 2024-04-16T02:35:41.862 INFO:tasks.workunit.client.1.smithi153.stdout:6/660: dwrite dc/d1a/d2d/f43 [0,4194304] 0 2024-04-16T02:35:41.863 INFO:tasks.workunit.client.1.smithi153.stdout:4/790: chown d3/d9 31 1 2024-04-16T02:35:41.864 INFO:tasks.workunit.client.1.smithi153.stdout:5/720: dwrite d8/d41/dbd/d39/fb4 [0,4194304] 0 2024-04-16T02:35:41.867 INFO:tasks.workunit.client.1.smithi153.stdout:3/647: write d6/d10/d3b/fc5 [181926,115364] 0 2024-04-16T02:35:41.870 INFO:tasks.workunit.client.1.smithi153.stdout:4/791: dread - d3/d9/db/de/d2c/fc9 zero size 2024-04-16T02:35:41.871 INFO:tasks.workunit.client.1.smithi153.stdout:9/731: dwrite d4/d6/dc/d2d/da0/f87 [0,4194304] 0 2024-04-16T02:35:41.889 INFO:tasks.workunit.client.0.smithi151.stdout:8/119: sync 2024-04-16T02:35:41.890 INFO:tasks.workunit.client.0.smithi151.stdout:2/268: sync 2024-04-16T02:35:41.890 INFO:tasks.workunit.client.0.smithi151.stdout:8/120: fsync ff 0 2024-04-16T02:35:41.890 INFO:tasks.workunit.client.0.smithi151.stdout:8/121: readlink l8 0 2024-04-16T02:35:41.890 INFO:tasks.workunit.client.0.smithi151.stdout:8/122: fsync f3 0 2024-04-16T02:35:41.896 INFO:tasks.workunit.client.0.smithi151.stdout:8/123: creat d11/d22/f27 x:0 0 0 2024-04-16T02:35:41.897 INFO:tasks.workunit.client.1.smithi153.stdout:5/721: dread d8/df/d28/f8a [0,4194304] 0 2024-04-16T02:35:41.899 INFO:tasks.workunit.client.0.smithi151.stdout:2/269: getdents de/d15/d1e 0 2024-04-16T02:35:41.900 INFO:tasks.workunit.client.0.smithi151.stdout:2/270: chown de/d15/c1f 13570349 1 2024-04-16T02:35:41.901 INFO:tasks.workunit.client.0.smithi151.stdout:8/124: rename f3 to d11/f28 0 2024-04-16T02:35:41.902 INFO:tasks.workunit.client.0.smithi151.stdout:2/271: truncate de/f49 4827096 0 2024-04-16T02:35:41.903 INFO:tasks.workunit.client.1.smithi153.stdout:7/626: read d0/d4/db/d2b/d43/d6d/f6f [994002,3980] 0 2024-04-16T02:35:41.907 INFO:tasks.workunit.client.0.smithi151.stdout:2/272: symlink de/d11/d41/d35/l53 0 2024-04-16T02:35:41.907 INFO:tasks.workunit.client.1.smithi153.stdout:8/660: dread - d2/de/d3f/f75 zero size 2024-04-16T02:35:41.910 INFO:tasks.workunit.client.1.smithi153.stdout:6/661: dread - dc/d47/d57/d77/d15/f86 zero size 2024-04-16T02:35:41.913 INFO:tasks.workunit.client.1.smithi153.stdout:7/627: dwrite d0/d4/db/d2b/d53/f5d [0,4194304] 0 2024-04-16T02:35:41.913 INFO:tasks.workunit.client.1.smithi153.stdout:4/792: unlink d3/dad/faf 0 2024-04-16T02:35:41.916 INFO:tasks.workunit.client.1.smithi153.stdout:3/648: unlink d6/d10/d22/f93 0 2024-04-16T02:35:41.918 INFO:tasks.workunit.client.1.smithi153.stdout:0/782: creat dd/d14/d51/ff7 x:0 0 0 2024-04-16T02:35:41.921 INFO:tasks.workunit.client.1.smithi153.stdout:8/661: mkdir d2/d93/d97/d9c/d27/d2d/d47/d77/ddf 0 2024-04-16T02:35:41.921 INFO:tasks.workunit.client.1.smithi153.stdout:3/649: read - d6/d10/d3b/d59/f87 zero size 2024-04-16T02:35:41.923 INFO:tasks.workunit.client.1.smithi153.stdout:6/662: dwrite dc/d47/d93/fa4 [0,4194304] 0 2024-04-16T02:35:41.923 INFO:tasks.workunit.client.1.smithi153.stdout:0/783: chown dd/d14/da6/le5 4284648 1 2024-04-16T02:35:41.928 INFO:tasks.workunit.client.1.smithi153.stdout:4/793: creat d3/d9/db/d17/f116 x:0 0 0 2024-04-16T02:35:41.929 INFO:tasks.workunit.client.1.smithi153.stdout:9/732: dwrite d4/d6/ddb/d4a/f80 [0,4194304] 0 2024-04-16T02:35:41.929 INFO:tasks.workunit.client.1.smithi153.stdout:2/639: truncate d9/d2b/f46 4062383 0 2024-04-16T02:35:41.932 INFO:tasks.workunit.client.1.smithi153.stdout:3/650: creat d6/d10/d22/d4d/da0/fdc x:0 0 0 2024-04-16T02:35:41.933 INFO:tasks.workunit.client.1.smithi153.stdout:4/794: creat d3/d9/d24/d3e/f117 x:0 0 0 2024-04-16T02:35:41.934 INFO:tasks.workunit.client.1.smithi153.stdout:6/663: fsync dc/d47/d57/d77/d15/f86 0 2024-04-16T02:35:41.934 INFO:tasks.workunit.client.1.smithi153.stdout:0/784: truncate dd/d26/d44/fd0 485028 0 2024-04-16T02:35:41.935 INFO:tasks.workunit.client.1.smithi153.stdout:1/709: getdents d4/dd/d13/d8d/d95 0 2024-04-16T02:35:41.936 INFO:tasks.workunit.client.1.smithi153.stdout:7/628: creat d0/d4/db/fd3 x:0 0 0 2024-04-16T02:35:41.936 INFO:tasks.workunit.client.0.smithi151.stdout:8/125: rmdir d11/d22 39 2024-04-16T02:35:41.937 INFO:tasks.workunit.client.1.smithi153.stdout:5/722: getdents d8/d41/dbd/d39 0 2024-04-16T02:35:41.937 INFO:tasks.workunit.client.1.smithi153.stdout:6/664: chown dc/d47/c4b 0 1 2024-04-16T02:35:41.938 INFO:tasks.workunit.client.1.smithi153.stdout:3/651: write d6/d10/d22/d4d/da0/fac [1999533,121441] 0 2024-04-16T02:35:41.939 INFO:tasks.workunit.client.1.smithi153.stdout:0/785: symlink dd/d26/d44/dd2/lf8 0 2024-04-16T02:35:41.939 INFO:tasks.workunit.client.1.smithi153.stdout:9/733: symlink d4/d6/d1b/ddf/le4 0 2024-04-16T02:35:41.941 INFO:tasks.workunit.client.1.smithi153.stdout:5/723: chown d8/d1f/l7f 0 1 2024-04-16T02:35:41.942 INFO:tasks.workunit.client.1.smithi153.stdout:1/710: symlink d4/dd/d18/d6c/dd8/lef 0 2024-04-16T02:35:41.942 INFO:tasks.workunit.client.0.smithi151.stdout:8/126: mknod d11/d22/c29 0 2024-04-16T02:35:41.942 INFO:tasks.workunit.client.0.smithi151.stdout:2/273: write de/d15/d1e/f3c [3039624,2426] 0 2024-04-16T02:35:41.943 INFO:tasks.workunit.client.1.smithi153.stdout:9/734: fdatasync d4/d33/dae/dc7/d66/f98 0 2024-04-16T02:35:41.944 INFO:tasks.workunit.client.0.smithi151.stdout:8/127: mkdir d11/d2a 0 2024-04-16T02:35:41.944 INFO:tasks.workunit.client.0.smithi151.stdout:8/128: chown d11/f1a 88 1 2024-04-16T02:35:41.945 INFO:tasks.workunit.client.0.smithi151.stdout:2/274: mknod de/d11/d41/c54 0 2024-04-16T02:35:41.947 INFO:tasks.workunit.client.0.smithi151.stdout:2/275: chown de/d11/d41/d35/l4f 18742132 1 2024-04-16T02:35:41.948 INFO:tasks.workunit.client.1.smithi153.stdout:5/724: fdatasync d8/f6c 0 2024-04-16T02:35:41.949 INFO:tasks.workunit.client.0.smithi151.stdout:8/129: dread d11/f28 [4194304,4194304] 0 2024-04-16T02:35:41.949 INFO:tasks.workunit.client.1.smithi153.stdout:2/640: getdents d9/d24/d27/d2c/d5d/da3 0 2024-04-16T02:35:41.949 INFO:tasks.workunit.client.0.smithi151.stdout:2/276: creat de/d11/d41/d4b/f55 x:0 0 0 2024-04-16T02:35:41.950 INFO:tasks.workunit.client.0.smithi151.stdout:2/277: chown de/d15/f46 30 1 2024-04-16T02:35:41.951 INFO:tasks.workunit.client.0.smithi151.stdout:8/130: mknod d11/c2b 0 2024-04-16T02:35:41.952 INFO:tasks.workunit.client.0.smithi151.stdout:2/278: truncate de/d11/d41/d4b/f55 180529 0 2024-04-16T02:35:41.952 INFO:tasks.workunit.client.1.smithi153.stdout:0/786: creat dd/d14/d19/d77/d87/dda/ff9 x:0 0 0 2024-04-16T02:35:41.953 INFO:tasks.workunit.client.1.smithi153.stdout:7/629: dwrite d0/d4/db/f2a [4194304,4194304] 0 2024-04-16T02:35:41.956 INFO:tasks.workunit.client.0.smithi151.stdout:2/279: dread de/d11/f2b [4194304,4194304] 0 2024-04-16T02:35:41.958 INFO:tasks.workunit.client.1.smithi153.stdout:5/725: symlink d8/df/d28/d5a/d72/dc0/le6 0 2024-04-16T02:35:41.959 INFO:tasks.workunit.client.0.smithi151.stdout:2/280: creat de/d11/d41/f56 x:0 0 0 2024-04-16T02:35:41.959 INFO:tasks.workunit.client.1.smithi153.stdout:0/787: write dd/d14/d19/d55/fa9 [364830,48875] 0 2024-04-16T02:35:41.960 INFO:tasks.workunit.client.0.smithi151.stdout:2/281: dread - de/d11/d42/f52 zero size 2024-04-16T02:35:41.961 INFO:tasks.workunit.client.1.smithi153.stdout:3/652: rename d6/d10/d22/d4d/fbd to d6/d10/d22/fdd 0 2024-04-16T02:35:41.961 INFO:tasks.workunit.client.1.smithi153.stdout:1/711: dwrite d4/da2/d65/fe2 [0,4194304] 0 2024-04-16T02:35:41.962 INFO:tasks.workunit.client.0.smithi151.stdout:8/131: getdents d11/d22 0 2024-04-16T02:35:41.962 INFO:tasks.workunit.client.1.smithi153.stdout:9/735: rename d4/d6 to d4/d6/d1b/d6e/de5 22 2024-04-16T02:35:41.963 INFO:tasks.workunit.client.0.smithi151.stdout:8/132: fsync fe 0 2024-04-16T02:35:41.963 INFO:tasks.workunit.client.0.smithi151.stdout:2/282: truncate f7 4418506 0 2024-04-16T02:35:41.964 INFO:tasks.workunit.client.1.smithi153.stdout:8/662: dwrite d2/d92/f69 [0,4194304] 0 2024-04-16T02:35:41.964 INFO:tasks.workunit.client.1.smithi153.stdout:0/788: write dd/d14/f86 [5703851,38744] 0 2024-04-16T02:35:41.965 INFO:tasks.workunit.client.0.smithi151.stdout:2/283: write de/d15/f46 [137980,42626] 0 2024-04-16T02:35:41.966 INFO:tasks.workunit.client.0.smithi151.stdout:2/284: chown de/d11/d41/d35/l53 47463898 1 2024-04-16T02:35:41.975 INFO:tasks.workunit.client.1.smithi153.stdout:1/712: write d4/dd/d18/d29/fde [738038,40516] 0 2024-04-16T02:35:41.976 INFO:tasks.workunit.client.1.smithi153.stdout:3/653: mknod d6/d10/d3b/d59/d7c/cde 0 2024-04-16T02:35:41.977 INFO:tasks.workunit.client.1.smithi153.stdout:5/726: dread d8/f6c [0,4194304] 0 2024-04-16T02:35:41.977 INFO:tasks.workunit.client.0.smithi151.stdout:8/133: write f2 [2520795,9549] 0 2024-04-16T02:35:41.978 INFO:tasks.workunit.client.0.smithi151.stdout:2/285: dwrite de/d15/f43 [0,4194304] 0 2024-04-16T02:35:41.980 INFO:tasks.workunit.client.1.smithi153.stdout:6/665: dwrite dc/d47/d57/d77/d15/f2f [0,4194304] 0 2024-04-16T02:35:41.980 INFO:tasks.workunit.client.0.smithi151.stdout:2/286: symlink de/d15/l57 0 2024-04-16T02:35:41.981 INFO:tasks.workunit.client.0.smithi151.stdout:8/134: getdents d11/d2a 0 2024-04-16T02:35:41.982 INFO:tasks.workunit.client.1.smithi153.stdout:6/666: dread - dc/d47/d93/faa zero size 2024-04-16T02:35:41.982 INFO:tasks.workunit.client.1.smithi153.stdout:2/641: link d9/d2b/d1c/c3d d9/d2b/d1c/d20/d45/cd7 0 2024-04-16T02:35:41.988 INFO:tasks.workunit.client.0.smithi151.stdout:2/287: dwrite de/d15/f43 [0,4194304] 0 2024-04-16T02:35:41.990 INFO:tasks.workunit.client.0.smithi151.stdout:8/135: truncate d11/f16 495195 0 2024-04-16T02:35:41.990 INFO:tasks.workunit.client.1.smithi153.stdout:7/630: creat d0/d4/db/d2b/d43/d6d/d68/fd4 x:0 0 0 2024-04-16T02:35:41.992 INFO:tasks.workunit.client.0.smithi151.stdout:2/288: symlink de/d11/d41/l58 0 2024-04-16T02:35:41.992 INFO:tasks.workunit.client.1.smithi153.stdout:1/713: rename d4/dd/d13/d23/d31/c89 to d4/dd/d13/d8d/d95/cf0 0 2024-04-16T02:35:41.993 INFO:tasks.workunit.client.0.smithi151.stdout:2/289: write de/d15/f17 [124213,75998] 0 2024-04-16T02:35:41.994 INFO:tasks.workunit.client.1.smithi153.stdout:6/667: symlink dc/d1a/d2d/le2 0 2024-04-16T02:35:41.995 INFO:tasks.workunit.client.1.smithi153.stdout:6/668: dread - dc/d47/d80/f8c zero size 2024-04-16T02:35:41.997 INFO:tasks.workunit.client.1.smithi153.stdout:6/669: write f6 [276249,65627] 0 2024-04-16T02:35:41.997 INFO:tasks.workunit.client.1.smithi153.stdout:8/663: rmdir d2/dcc 0 2024-04-16T02:35:42.000 INFO:tasks.workunit.client.1.smithi153.stdout:5/727: mknod d8/df/d28/d5a/d72/dc0/dcb/ce7 0 2024-04-16T02:35:42.001 INFO:tasks.workunit.client.0.smithi151.stdout:2/290: dwrite de/d11/d41/d4b/f55 [0,4194304] 0 2024-04-16T02:35:42.001 INFO:tasks.workunit.client.1.smithi153.stdout:1/714: creat d4/dd/d18/d6c/dd8/ff1 x:0 0 0 2024-04-16T02:35:42.002 INFO:tasks.workunit.client.0.smithi151.stdout:2/291: write de/f27 [5150564,6525] 0 2024-04-16T02:35:42.003 INFO:tasks.workunit.client.1.smithi153.stdout:7/631: mknod d0/d4/db/d2b/d8c/d9f/cd5 0 2024-04-16T02:35:42.004 INFO:tasks.workunit.client.1.smithi153.stdout:9/736: dwrite d4/d6/d1b/f76 [0,4194304] 0 2024-04-16T02:35:42.004 INFO:tasks.workunit.client.1.smithi153.stdout:2/642: dread d9/fb [0,4194304] 0 2024-04-16T02:35:42.006 INFO:tasks.workunit.client.1.smithi153.stdout:0/789: dread dd/d28/f83 [0,4194304] 0 2024-04-16T02:35:42.006 INFO:tasks.workunit.client.0.smithi151.stdout:2/292: rmdir de/d15 39 2024-04-16T02:35:42.010 INFO:tasks.workunit.client.1.smithi153.stdout:3/654: rename d6/d10/d22/d2e/c5d to d6/d10/d22/d43/cdf 0 2024-04-16T02:35:42.011 INFO:tasks.workunit.client.0.smithi151.stdout:2/293: write de/d11/f2b [7302627,18727] 0 2024-04-16T02:35:42.012 INFO:tasks.workunit.client.1.smithi153.stdout:1/715: write d4/f1f [5903909,67155] 0 2024-04-16T02:35:42.012 INFO:tasks.workunit.client.1.smithi153.stdout:6/670: creat dc/d47/d57/fe3 x:0 0 0 2024-04-16T02:35:42.015 INFO:tasks.workunit.client.1.smithi153.stdout:9/737: symlink d4/d33/dae/dc7/le6 0 2024-04-16T02:35:42.015 INFO:tasks.workunit.client.0.smithi151.stdout:8/136: dread d11/f1c [0,4194304] 0 2024-04-16T02:35:42.015 INFO:tasks.workunit.client.1.smithi153.stdout:9/738: dread - d4/d33/fba zero size 2024-04-16T02:35:42.016 INFO:tasks.workunit.client.0.smithi151.stdout:2/294: rename de/d11/d41/f56 to de/d11/d41/d35/f59 0 2024-04-16T02:35:42.016 INFO:tasks.workunit.client.1.smithi153.stdout:2/643: creat d9/d24/d34/d36/db1/fd8 x:0 0 0 2024-04-16T02:35:42.017 INFO:tasks.workunit.client.0.smithi151.stdout:2/295: chown c4 0 1 2024-04-16T02:35:42.017 INFO:tasks.workunit.client.1.smithi153.stdout:9/739: chown d4/d6/dc/d24/f30 695 1 2024-04-16T02:35:42.018 INFO:tasks.workunit.client.1.smithi153.stdout:8/664: rename d2/d93/d97/d9c/d27/c72 to d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d98/ce0 0 2024-04-16T02:35:42.018 INFO:tasks.workunit.client.0.smithi151.stdout:8/137: creat d11/d2a/f2c x:0 0 0 2024-04-16T02:35:42.019 INFO:tasks.workunit.client.1.smithi153.stdout:9/740: write d4/d6/db1/fb4 [1375233,114850] 0 2024-04-16T02:35:42.020 INFO:tasks.workunit.client.0.smithi151.stdout:8/138: truncate d11/f1b 4810951 0 2024-04-16T02:35:42.020 INFO:tasks.workunit.client.0.smithi151.stdout:2/296: symlink de/d3d/l5a 0 2024-04-16T02:35:42.021 INFO:tasks.workunit.client.1.smithi153.stdout:0/790: link dd/d14/d19/d55/fcf dd/d26/d92/ffa 0 2024-04-16T02:35:42.021 INFO:tasks.workunit.client.0.smithi151.stdout:8/139: mknod d11/d2a/c2d 0 2024-04-16T02:35:42.022 INFO:tasks.workunit.client.1.smithi153.stdout:1/716: dwrite d4/da2/d65/d84/f8c [0,4194304] 0 2024-04-16T02:35:42.023 INFO:tasks.workunit.client.1.smithi153.stdout:0/791: dread - dd/d14/da6/da4/fd3 zero size 2024-04-16T02:35:42.023 INFO:tasks.workunit.client.0.smithi151.stdout:8/140: mkdir d11/d22/d2e 0 2024-04-16T02:35:42.024 INFO:tasks.workunit.client.1.smithi153.stdout:3/655: mknod d6/d10/d22/ce0 0 2024-04-16T02:35:42.025 INFO:tasks.workunit.client.1.smithi153.stdout:1/717: write d4/dd/d18/f53 [622915,100105] 0 2024-04-16T02:35:42.025 INFO:tasks.workunit.client.0.smithi151.stdout:2/297: creat de/d15/f5b x:0 0 0 2024-04-16T02:35:42.027 INFO:tasks.workunit.client.1.smithi153.stdout:7/632: rename d0/cb6 to d0/d4/db/d47/d8d/d93/d8b/cd6 0 2024-04-16T02:35:42.027 INFO:tasks.workunit.client.1.smithi153.stdout:2/644: creat d9/d71/d8d/fd9 x:0 0 0 2024-04-16T02:35:42.029 INFO:tasks.workunit.client.1.smithi153.stdout:3/656: read - d6/d10/d22/d2e/d8a/fb6 zero size 2024-04-16T02:35:42.032 INFO:tasks.workunit.client.0.smithi151.stdout:8/141: truncate d11/f17 576719 0 2024-04-16T02:35:42.032 INFO:tasks.workunit.client.1.smithi153.stdout:7/633: write d0/d4/db/d2b/d53/d59/f67 [862910,74445] 0 2024-04-16T02:35:42.035 INFO:tasks.workunit.client.1.smithi153.stdout:8/665: dwrite d2/d93/d97/d9c/d27/d2d/d47/d60/f79 [0,4194304] 0 2024-04-16T02:35:42.035 INFO:tasks.workunit.client.0.smithi151.stdout:8/142: dread d11/f15 [0,4194304] 0 2024-04-16T02:35:42.037 INFO:tasks.workunit.client.0.smithi151.stdout:8/143: write d11/d22/f26 [965908,86536] 0 2024-04-16T02:35:42.038 INFO:tasks.workunit.client.0.smithi151.stdout:8/144: write d11/d2a/f2c [316522,23033] 0 2024-04-16T02:35:42.038 INFO:tasks.workunit.client.1.smithi153.stdout:6/671: rename dc/d47/d57/d77/c46 to dc/d59/d5b/ce4 0 2024-04-16T02:35:42.039 INFO:tasks.workunit.client.0.smithi151.stdout:8/145: fdatasync fb 0 2024-04-16T02:35:42.042 INFO:tasks.workunit.client.0.smithi151.stdout:8/146: dread ff [0,4194304] 0 2024-04-16T02:35:42.043 INFO:tasks.workunit.client.1.smithi153.stdout:1/718: dwrite d4/dd/d13/d26/d58/d7a/fa9 [0,4194304] 0 2024-04-16T02:35:42.046 INFO:tasks.workunit.client.1.smithi153.stdout:8/666: chown d2/d93/d97/d9c/d27/f7a 304800 1 2024-04-16T02:35:42.046 INFO:tasks.workunit.client.1.smithi153.stdout:5/728: write d8/f7e [746761,121027] 0 2024-04-16T02:35:42.046 INFO:tasks.workunit.client.1.smithi153.stdout:7/634: symlink d0/d13/ld7 0 2024-04-16T02:35:42.048 INFO:tasks.workunit.client.0.smithi151.stdout:8/147: dwrite d11/f18 [0,4194304] 0 2024-04-16T02:35:42.049 INFO:tasks.workunit.client.1.smithi153.stdout:2/645: dwrite d9/d24/d34/d60/fa9 [0,4194304] 0 2024-04-16T02:35:42.050 INFO:tasks.workunit.client.1.smithi153.stdout:6/672: mknod dc/d59/ce5 0 2024-04-16T02:35:42.051 INFO:tasks.workunit.client.0.smithi151.stdout:8/148: creat d11/d2a/f2f x:0 0 0 2024-04-16T02:35:42.053 INFO:tasks.workunit.client.1.smithi153.stdout:7/635: chown d0/d4/db/d47/d8d/d93/l70 1 1 2024-04-16T02:35:42.054 INFO:tasks.workunit.client.1.smithi153.stdout:3/657: getdents d6/daf 0 2024-04-16T02:35:42.056 INFO:tasks.workunit.client.1.smithi153.stdout:1/719: mkdir d4/dd/d13/d26/db4/df2 0 2024-04-16T02:35:42.056 INFO:tasks.workunit.client.1.smithi153.stdout:2/646: chown d9/d2b/d1c/d20/d45/f96 32 1 2024-04-16T02:35:42.058 INFO:tasks.workunit.client.1.smithi153.stdout:0/792: write dd/d14/d19/d20/f31 [2483888,108362] 0 2024-04-16T02:35:42.061 INFO:tasks.workunit.client.1.smithi153.stdout:5/729: mknod d8/d41/dbd/d39/ce8 0 2024-04-16T02:35:42.063 INFO:tasks.workunit.client.1.smithi153.stdout:9/741: rename d4/d6/dc/d24/f25 to d4/fe7 0 2024-04-16T02:35:42.064 INFO:tasks.workunit.client.1.smithi153.stdout:4/795: sync 2024-04-16T02:35:42.065 INFO:tasks.workunit.client.1.smithi153.stdout:2/647: rmdir d9/d2b/d1c 39 2024-04-16T02:35:42.068 INFO:tasks.workunit.client.1.smithi153.stdout:6/673: dwrite dc/fe [0,4194304] 0 2024-04-16T02:35:42.068 INFO:tasks.workunit.client.1.smithi153.stdout:4/796: chown d3/d9/db/de/f3b 2 1 2024-04-16T02:35:42.070 INFO:tasks.workunit.client.1.smithi153.stdout:1/720: truncate d4/dd/d13/d7e/fb1 1043477 0 2024-04-16T02:35:42.074 INFO:tasks.workunit.client.1.smithi153.stdout:3/658: mkdir d6/d10/d3b/d59/d7c/de1 0 2024-04-16T02:35:42.076 INFO:tasks.workunit.client.1.smithi153.stdout:0/793: chown dd/d14/d19/fa0 820043768 1 2024-04-16T02:35:42.077 INFO:tasks.workunit.client.1.smithi153.stdout:7/636: dwrite d0/d4/db/f2a [0,4194304] 0 2024-04-16T02:35:42.077 INFO:tasks.workunit.client.1.smithi153.stdout:9/742: mknod d4/d6/dc/d46/ce8 0 2024-04-16T02:35:42.078 INFO:tasks.workunit.client.1.smithi153.stdout:1/721: write d4/dd/d18/d6c/dd8/ff1 [147821,111219] 0 2024-04-16T02:35:42.083 INFO:tasks.workunit.client.1.smithi153.stdout:4/797: mkdir d3/d9/d24/d75/d118 0 2024-04-16T02:35:42.086 INFO:tasks.workunit.client.1.smithi153.stdout:6/674: dread dc/d47/d93/fa4 [0,4194304] 0 2024-04-16T02:35:42.088 INFO:tasks.workunit.client.1.smithi153.stdout:5/730: mkdir d8/d41/dbd/d8d/de9 0 2024-04-16T02:35:42.091 INFO:tasks.workunit.client.1.smithi153.stdout:0/794: creat dd/d14/da6/da4/ffb x:0 0 0 2024-04-16T02:35:42.091 INFO:tasks.workunit.client.1.smithi153.stdout:6/675: write dc/d10/d62/fce [225373,127972] 0 2024-04-16T02:35:42.092 INFO:tasks.workunit.client.1.smithi153.stdout:2/648: dwrite d9/d2b/d5c/da2/fc1 [0,4194304] 0 2024-04-16T02:35:42.095 INFO:tasks.workunit.client.1.smithi153.stdout:7/637: rename d0/d4/db/f57 to d0/d4/db/d2b/d8c/d9f/fd8 0 2024-04-16T02:35:42.095 INFO:tasks.workunit.client.1.smithi153.stdout:4/798: write d3/d9/d24/d75/dc5/ff2 [8014261,9145] 0 2024-04-16T02:35:42.097 INFO:tasks.workunit.client.1.smithi153.stdout:1/722: creat d4/dd/d13/d23/ff3 x:0 0 0 2024-04-16T02:35:42.098 INFO:tasks.workunit.client.1.smithi153.stdout:9/743: fsync d4/d6/d1b/f90 0 2024-04-16T02:35:42.101 INFO:tasks.workunit.client.1.smithi153.stdout:7/638: chown d0/d4/db/d34/d51/laf 7871318 1 2024-04-16T02:35:42.101 INFO:tasks.workunit.client.1.smithi153.stdout:5/731: chown d8/df/d2f/cc3 56 1 2024-04-16T02:35:42.104 INFO:tasks.workunit.client.1.smithi153.stdout:2/649: chown d9/d2b/d1c/f28 3 1 2024-04-16T02:35:42.105 INFO:tasks.workunit.client.0.smithi151.stdout:1/188: sync 2024-04-16T02:35:42.105 INFO:tasks.workunit.client.0.smithi151.stdout:1/189: dread - d3/d4/db/df/d38/f3b zero size 2024-04-16T02:35:42.106 INFO:tasks.workunit.client.1.smithi153.stdout:3/659: dwrite d6/f39 [0,4194304] 0 2024-04-16T02:35:42.106 INFO:tasks.workunit.client.0.smithi151.stdout:1/190: fdatasync f1 0 2024-04-16T02:35:42.111 INFO:tasks.workunit.client.1.smithi153.stdout:5/732: chown d8/c14 5 1 2024-04-16T02:35:42.112 INFO:tasks.workunit.client.0.smithi151.stdout:9/129: sync 2024-04-16T02:35:42.112 INFO:tasks.workunit.client.1.smithi153.stdout:7/639: symlink d0/d4/db/ld9 0 2024-04-16T02:35:42.113 INFO:tasks.workunit.client.1.smithi153.stdout:4/799: fdatasync d3/d9/db/de/d2c/d52/d85/f90 0 2024-04-16T02:35:42.115 INFO:tasks.workunit.client.0.smithi151.stdout:9/130: symlink d1/d3/d6/d28/l32 0 2024-04-16T02:35:42.116 INFO:tasks.workunit.client.0.smithi151.stdout:9/131: dread - d1/d3/d6/f2a zero size 2024-04-16T02:35:42.117 INFO:tasks.workunit.client.1.smithi153.stdout:2/650: symlink d9/d24/d34/d60/lda 0 2024-04-16T02:35:42.117 INFO:tasks.workunit.client.1.smithi153.stdout:8/667: dread d2/d13/d3a/d5b/f9d [4194304,4194304] 0 2024-04-16T02:35:42.118 INFO:tasks.workunit.client.0.smithi151.stdout:9/132: fsync d1/d3/d6/f2a 0 2024-04-16T02:35:42.118 INFO:tasks.workunit.client.0.smithi151.stdout:8/149: dread f2 [0,4194304] 0 2024-04-16T02:35:42.119 INFO:tasks.workunit.client.1.smithi153.stdout:6/676: dwrite dc/d47/d57/d77/dc4/fd0 [0,4194304] 0 2024-04-16T02:35:42.120 INFO:tasks.workunit.client.0.smithi151.stdout:8/150: creat d11/d22/f30 x:0 0 0 2024-04-16T02:35:42.122 INFO:tasks.workunit.client.0.smithi151.stdout:8/151: truncate d11/f1f 974347 0 2024-04-16T02:35:42.122 INFO:tasks.workunit.client.0.smithi151.stdout:8/152: chown d11/d22/d2e 2536827 1 2024-04-16T02:35:42.123 INFO:tasks.workunit.client.1.smithi153.stdout:0/795: dwrite dd/d14/d51/d7f/d9d/ff4 [0,4194304] 0 2024-04-16T02:35:42.124 INFO:tasks.workunit.client.1.smithi153.stdout:7/640: creat d0/d4/db/d34/d51/fda x:0 0 0 2024-04-16T02:35:42.127 INFO:tasks.workunit.client.1.smithi153.stdout:1/723: rename d4/dd/d13/d8d/d95/dc7/cce to d4/dd/d13/d23/db6/cf4 0 2024-04-16T02:35:42.128 INFO:tasks.workunit.client.1.smithi153.stdout:7/641: chown d0/d4/db/d2b/d53/ld0 55683965 1 2024-04-16T02:35:42.130 INFO:tasks.workunit.client.0.smithi151.stdout:1/191: dwrite d3/d4/db/d1b/d1c/f2c [0,4194304] 0 2024-04-16T02:35:42.136 INFO:tasks.workunit.client.1.smithi153.stdout:4/800: chown d3/d9/db/de/d2c/d52/d85/ffe 0 1 2024-04-16T02:35:42.136 INFO:tasks.workunit.client.1.smithi153.stdout:2/651: creat d9/fdb x:0 0 0 2024-04-16T02:35:42.137 INFO:tasks.workunit.client.1.smithi153.stdout:0/796: rename dd/d14/da6/f3e to dd/d14/dca/dcd/ffc 0 2024-04-16T02:35:42.138 INFO:tasks.workunit.client.0.smithi151.stdout:1/192: symlink d3/d33/l3d 0 2024-04-16T02:35:42.141 INFO:tasks.workunit.client.0.smithi151.stdout:8/153: dwrite d11/f16 [0,4194304] 0 2024-04-16T02:35:42.142 INFO:tasks.workunit.client.1.smithi153.stdout:8/668: dwrite d2/d13/d3a/d5b/f9d [4194304,4194304] 0 2024-04-16T02:35:42.143 INFO:tasks.workunit.client.1.smithi153.stdout:1/724: symlink d4/dd/d13/d26/d58/d7a/lf5 0 2024-04-16T02:35:42.145 INFO:tasks.workunit.client.1.smithi153.stdout:6/677: dwrite dc/d59/d63/f94 [0,4194304] 0 2024-04-16T02:35:42.146 INFO:tasks.workunit.client.0.smithi151.stdout:8/154: dread f2 [0,4194304] 0 2024-04-16T02:35:42.147 INFO:tasks.workunit.client.1.smithi153.stdout:0/797: read dd/d14/da6/f98 [624864,4916] 0 2024-04-16T02:35:42.147 INFO:tasks.workunit.client.0.smithi151.stdout:1/193: link d3/d33/d10/c28 d3/d4/db/df/d32/c3e 0 2024-04-16T02:35:42.148 INFO:tasks.workunit.client.1.smithi153.stdout:7/642: symlink d0/d4/db/d2b/d8c/db5/ldb 0 2024-04-16T02:35:42.150 INFO:tasks.workunit.client.1.smithi153.stdout:2/652: mkdir d9/d24/d34/d60/ddc 0 2024-04-16T02:35:42.150 INFO:tasks.workunit.client.0.smithi151.stdout:1/194: fsync d3/d33/d10/f18 0 2024-04-16T02:35:42.153 INFO:tasks.workunit.client.1.smithi153.stdout:4/801: rename d3/d9/d24/d35 to d3/d9/db/de/d2c/de2/d119 0 2024-04-16T02:35:42.153 INFO:tasks.workunit.client.1.smithi153.stdout:3/660: write d6/d10/d22/d2c/f3d [5551312,14864] 0 2024-04-16T02:35:42.154 INFO:tasks.workunit.client.1.smithi153.stdout:2/653: dread - d9/d24/d27/d2c/d5d/da3/dc0/fd1 zero size 2024-04-16T02:35:42.158 INFO:tasks.workunit.client.1.smithi153.stdout:6/678: chown dc/d1a/c5c 391039 1 2024-04-16T02:35:42.158 INFO:tasks.workunit.client.1.smithi153.stdout:2/654: chown d9/d2b/d18/c37 2891 1 2024-04-16T02:35:42.158 INFO:tasks.workunit.client.0.smithi151.stdout:1/195: dwrite d3/f2f [0,4194304] 0 2024-04-16T02:35:42.160 INFO:tasks.workunit.client.1.smithi153.stdout:1/725: mknod d4/dd/d13/d26/d58/cf6 0 2024-04-16T02:35:42.162 INFO:tasks.workunit.client.0.smithi151.stdout:1/196: mkdir d3/d4/db/d3f 0 2024-04-16T02:35:42.163 INFO:tasks.workunit.client.1.smithi153.stdout:6/679: write dc/d47/f4a [803290,46427] 0 2024-04-16T02:35:42.167 INFO:tasks.workunit.client.0.smithi151.stdout:1/197: fdatasync d3/d4/db/d1b/d1c/f2c 0 2024-04-16T02:35:42.168 INFO:tasks.workunit.client.1.smithi153.stdout:1/726: stat d4/dd/d13/f2a 0 2024-04-16T02:35:42.168 INFO:tasks.workunit.client.1.smithi153.stdout:7/643: dwrite d0/d4/db/d2b/d53/f5d [4194304,4194304] 0 2024-04-16T02:35:42.168 INFO:tasks.workunit.client.0.smithi151.stdout:1/198: dread - d3/d33/f39 zero size 2024-04-16T02:35:42.172 INFO:tasks.workunit.client.1.smithi153.stdout:6/680: chown dc/d1a/d2d/le2 3800 1 2024-04-16T02:35:42.173 INFO:tasks.workunit.client.0.smithi151.stdout:1/199: getdents d3/d4/db/d1b/d2a 0 2024-04-16T02:35:42.176 INFO:tasks.workunit.client.1.smithi153.stdout:3/661: dwrite d6/d10/d22/d2e/d8a/fc3 [0,4194304] 0 2024-04-16T02:35:42.176 INFO:tasks.workunit.client.0.smithi151.stdout:1/200: mkdir d3/d4/db/d1b/d40 0 2024-04-16T02:35:42.178 INFO:tasks.workunit.client.1.smithi153.stdout:9/744: dread d4/d6/d1b/fbe [0,4194304] 0 2024-04-16T02:35:42.179 INFO:tasks.workunit.client.1.smithi153.stdout:2/655: mkdir d9/d24/ddd 0 2024-04-16T02:35:42.180 INFO:tasks.workunit.client.0.smithi151.stdout:1/201: rmdir d3/d4/db/d3f 0 2024-04-16T02:35:42.182 INFO:tasks.workunit.client.0.smithi151.stdout:9/133: dread d1/dd/f16 [0,4194304] 0 2024-04-16T02:35:42.182 INFO:tasks.workunit.client.1.smithi153.stdout:5/733: dwrite d8/f6f [4194304,4194304] 0 2024-04-16T02:35:42.183 INFO:tasks.workunit.client.0.smithi151.stdout:9/134: readlink d1/d3/d6/d17/l2f 0 2024-04-16T02:35:42.183 INFO:tasks.workunit.client.1.smithi153.stdout:3/662: write d6/d10/d22/d4d/fbf [1333362,63010] 0 2024-04-16T02:35:42.184 INFO:tasks.workunit.client.1.smithi153.stdout:1/727: mkdir d4/dd/d18/df7 0 2024-04-16T02:35:42.184 INFO:tasks.workunit.client.0.smithi151.stdout:9/135: stat d1/d3/d6 0 2024-04-16T02:35:42.190 INFO:tasks.workunit.client.0.smithi151.stdout:9/136: rename d1/d3/fb to d1/d3/f33 0 2024-04-16T02:35:42.190 INFO:tasks.workunit.client.1.smithi153.stdout:2/656: creat d9/d2b/d1c/d20/fde x:0 0 0 2024-04-16T02:35:42.192 INFO:tasks.workunit.client.0.smithi151.stdout:1/202: dwrite d3/d4/db/d1b/d1c/f2c [0,4194304] 0 2024-04-16T02:35:42.194 INFO:tasks.workunit.client.0.smithi151.stdout:9/137: write d1/d3/f33 [261979,68369] 0 2024-04-16T02:35:42.195 INFO:tasks.workunit.client.0.smithi151.stdout:9/138: write d1/d3/f33 [920395,99530] 0 2024-04-16T02:35:42.196 INFO:tasks.workunit.client.0.smithi151.stdout:1/203: symlink d3/d4/db/d1b/d2a/l41 0 2024-04-16T02:35:42.197 INFO:tasks.workunit.client.1.smithi153.stdout:4/802: dwrite d3/d9/db/d45/fcc [0,4194304] 0 2024-04-16T02:35:42.199 INFO:tasks.workunit.client.0.smithi151.stdout:1/204: mknod d3/d4/c42 0 2024-04-16T02:35:42.199 INFO:tasks.workunit.client.1.smithi153.stdout:4/803: chown d3/d9/db/d17/d28/dc8/de4 991997081 1 2024-04-16T02:35:42.201 INFO:tasks.workunit.client.0.smithi151.stdout:1/205: mknod d3/d4/db/df/c43 0 2024-04-16T02:35:42.201 INFO:tasks.workunit.client.1.smithi153.stdout:1/728: creat d4/dd/d13/d26/d58/d7a/ff8 x:0 0 0 2024-04-16T02:35:42.204 INFO:tasks.workunit.client.0.smithi151.stdout:1/206: creat d3/d4/db/d1b/d1c/f44 x:0 0 0 2024-04-16T02:35:42.205 INFO:tasks.workunit.client.1.smithi153.stdout:2/657: rmdir d9/d71/d8d 39 2024-04-16T02:35:42.205 INFO:tasks.workunit.client.1.smithi153.stdout:9/745: link d4/d6/dc/d2d/da0/f88 d4/d6/dc/d7f/fe9 0 2024-04-16T02:35:42.206 INFO:tasks.workunit.client.1.smithi153.stdout:5/734: link d8/df/d28/d5a/d5e/fad d8/d41/dbd/d39/d6d/fea 0 2024-04-16T02:35:42.211 INFO:tasks.workunit.client.1.smithi153.stdout:6/681: dwrite dc/d47/d57/d77/dc4/dc9/fdc [0,4194304] 0 2024-04-16T02:35:42.212 INFO:tasks.workunit.client.0.smithi151.stdout:1/207: dwrite d3/d4/db/df/d38/f3b [0,4194304] 0 2024-04-16T02:35:42.213 INFO:tasks.workunit.client.1.smithi153.stdout:3/663: dwrite d6/d10/d22/d4d/da0/fac [0,4194304] 0 2024-04-16T02:35:42.213 INFO:tasks.workunit.client.1.smithi153.stdout:8/669: write d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/fde [533805,2145] 0 2024-04-16T02:35:42.215 INFO:tasks.workunit.client.1.smithi153.stdout:4/804: fsync d3/d9/db/de/fa9 0 2024-04-16T02:35:42.215 INFO:tasks.workunit.client.1.smithi153.stdout:1/729: readlink d4/dd/d13/d23/d31/la0 0 2024-04-16T02:35:42.215 INFO:tasks.workunit.client.1.smithi153.stdout:9/746: mkdir d4/d6/ddb/d4a/dea 0 2024-04-16T02:35:42.216 INFO:tasks.workunit.client.1.smithi153.stdout:2/658: rmdir d9/d24/d27/d2c/d5d/da3/dc0 39 2024-04-16T02:35:42.218 INFO:tasks.workunit.client.0.smithi151.stdout:1/208: rmdir d3/d4/db 39 2024-04-16T02:35:42.222 INFO:tasks.workunit.client.1.smithi153.stdout:3/664: symlink d6/d10/d22/d2e/le2 0 2024-04-16T02:35:42.225 INFO:tasks.workunit.client.1.smithi153.stdout:5/735: creat d8/d1f/feb x:0 0 0 2024-04-16T02:35:42.225 INFO:tasks.workunit.client.1.smithi153.stdout:5/736: stat d8/d1f 0 2024-04-16T02:35:42.227 INFO:tasks.workunit.client.1.smithi153.stdout:1/730: mknod d4/dd/d18/d6c/dd8/cf9 0 2024-04-16T02:35:42.232 INFO:tasks.workunit.client.0.smithi151.stdout:1/209: getdents d3/d4 0 2024-04-16T02:35:42.232 INFO:tasks.workunit.client.1.smithi153.stdout:9/747: mknod d4/d33/dae/dc7/d66/ceb 0 2024-04-16T02:35:42.232 INFO:tasks.workunit.client.1.smithi153.stdout:7/644: dread d0/d4/db/d47/f94 [0,4194304] 0 2024-04-16T02:35:42.233 INFO:tasks.workunit.client.1.smithi153.stdout:3/665: dwrite d6/d10/d3b/d76/fd1 [0,4194304] 0 2024-04-16T02:35:42.234 INFO:tasks.workunit.client.1.smithi153.stdout:9/748: chown d4/d33/dae/dc7/le6 37192 1 2024-04-16T02:35:42.235 INFO:tasks.workunit.client.0.smithi151.stdout:1/210: symlink d3/d4/db/d1b/d1c/l45 0 2024-04-16T02:35:42.235 INFO:tasks.workunit.client.1.smithi153.stdout:7/645: write d0/d4/db/f64 [3231824,14332] 0 2024-04-16T02:35:42.236 INFO:tasks.workunit.client.1.smithi153.stdout:8/670: rename d2/d93/d97/d9c/d27/d2d/d47/c55 to d2/d13/d3a/dc2/ce1 0 2024-04-16T02:35:42.238 INFO:tasks.workunit.client.1.smithi153.stdout:5/737: unlink d8/df/d28/d5a/f67 0 2024-04-16T02:35:42.241 INFO:tasks.workunit.client.1.smithi153.stdout:8/671: dread - d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/f89 zero size 2024-04-16T02:35:42.242 INFO:tasks.workunit.client.1.smithi153.stdout:1/731: mknod d4/dd/d18/d6c/cfa 0 2024-04-16T02:35:42.243 INFO:tasks.workunit.client.0.smithi151.stdout:9/139: dread d1/fc [0,4194304] 0 2024-04-16T02:35:42.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:41 smithi151 ceph-mon[29003]: pgmap v18: 65 pgs: 65 active+clean; 1.2 GiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 25 MiB/s rd, 164 MiB/s wr, 318 op/s 2024-04-16T02:35:42.245 INFO:tasks.workunit.client.0.smithi151.stdout:9/140: mknod d1/d3/d6/d17/c34 0 2024-04-16T02:35:42.252 INFO:tasks.workunit.client.0.smithi151.stdout:9/141: dwrite d1/d3/d6/d17/f27 [0,4194304] 0 2024-04-16T02:35:42.253 INFO:tasks.workunit.client.0.smithi151.stdout:9/142: write d1/d3/f1e [2265716,83002] 0 2024-04-16T02:35:42.256 INFO:tasks.workunit.client.0.smithi151.stdout:9/143: creat d1/dd/f35 x:0 0 0 2024-04-16T02:35:42.259 INFO:tasks.workunit.client.0.smithi151.stdout:9/144: unlink d1/d3/d6/c1f 0 2024-04-16T02:35:42.261 INFO:tasks.workunit.client.0.smithi151.stdout:1/211: dread d3/d4/db/df/d32/f37 [0,4194304] 0 2024-04-16T02:35:42.262 INFO:tasks.workunit.client.0.smithi151.stdout:9/145: mkdir d1/d3/d6/d1d/d22/d36 0 2024-04-16T02:35:42.263 INFO:tasks.workunit.client.0.smithi151.stdout:1/212: mknod d3/d4/db/d1b/d2a/c46 0 2024-04-16T02:35:42.267 INFO:tasks.workunit.client.0.smithi151.stdout:1/213: creat d3/d4/db/df/f47 x:0 0 0 2024-04-16T02:35:42.271 INFO:tasks.workunit.client.0.smithi151.stdout:1/214: rmdir d3/d4/db/df/d38 39 2024-04-16T02:35:42.271 INFO:tasks.workunit.client.0.smithi151.stdout:9/146: dwrite d1/d3/d6/d17/f27 [0,4194304] 0 2024-04-16T02:35:42.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:41 smithi153 ceph-mon[36747]: pgmap v18: 65 pgs: 65 active+clean; 1.2 GiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 25 MiB/s rd, 164 MiB/s wr, 318 op/s 2024-04-16T02:35:42.273 INFO:tasks.workunit.client.0.smithi151.stdout:1/215: mkdir d3/d1f/d48 0 2024-04-16T02:35:42.279 INFO:tasks.workunit.client.0.smithi151.stdout:9/147: dwrite d1/d3/d10/f24 [0,4194304] 0 2024-04-16T02:35:42.281 INFO:tasks.workunit.client.0.smithi151.stdout:9/148: readlink d1/d3/d6/d1d/d22/l29 0 2024-04-16T02:35:42.284 INFO:tasks.workunit.client.0.smithi151.stdout:9/149: symlink d1/d3/l37 0 2024-04-16T02:35:42.285 INFO:tasks.workunit.client.0.smithi151.stdout:1/216: dread f1 [0,4194304] 0 2024-04-16T02:35:42.286 INFO:tasks.workunit.client.0.smithi151.stdout:1/217: readlink d3/d4/db/df/l35 0 2024-04-16T02:35:42.292 INFO:tasks.workunit.client.0.smithi151.stdout:1/218: rename d3/d33/d10/c28 to d3/c49 0 2024-04-16T02:35:42.293 INFO:tasks.workunit.client.0.smithi151.stdout:1/219: chown d3/d33/l6 10768830 1 2024-04-16T02:35:42.293 INFO:tasks.workunit.client.0.smithi151.stdout:1/220: readlink d3/l15 0 2024-04-16T02:35:42.296 INFO:tasks.workunit.client.0.smithi151.stdout:1/221: symlink d3/d1f/l4a 0 2024-04-16T02:35:42.297 INFO:tasks.workunit.client.0.smithi151.stdout:9/150: truncate d1/d3/d10/f24 1940817 0 2024-04-16T02:35:42.298 INFO:tasks.workunit.client.0.smithi151.stdout:9/151: fsync d1/d3/f1e 0 2024-04-16T02:35:42.299 INFO:tasks.workunit.client.0.smithi151.stdout:9/152: mkdir d1/dd/d38 0 2024-04-16T02:35:42.301 INFO:tasks.workunit.client.0.smithi151.stdout:9/153: mkdir d1/d3/d6/d28/d39 0 2024-04-16T02:35:42.302 INFO:tasks.workunit.client.0.smithi151.stdout:9/154: read d1/d3/f1a [1928176,26636] 0 2024-04-16T02:35:42.306 INFO:tasks.workunit.client.0.smithi151.stdout:9/155: write d1/d3/d10/f24 [2133138,36403] 0 2024-04-16T02:35:42.307 INFO:tasks.workunit.client.0.smithi151.stdout:1/222: truncate d3/d4/db/df/d38/f3b 3445318 0 2024-04-16T02:35:42.308 INFO:tasks.workunit.client.0.smithi151.stdout:1/223: stat d3/c7 0 2024-04-16T02:35:42.308 INFO:tasks.workunit.client.1.smithi153.stdout:6/682: rename dc/l41 to dc/le6 0 2024-04-16T02:35:42.309 INFO:tasks.workunit.client.1.smithi153.stdout:5/738: write d8/df/d2e/fdf [942381,32269] 0 2024-04-16T02:35:42.313 INFO:tasks.workunit.client.1.smithi153.stdout:6/683: chown dc/d10/d62/f6a 6 1 2024-04-16T02:35:42.313 INFO:tasks.workunit.client.1.smithi153.stdout:4/805: truncate d3/d9/db/de/d2c/d98/fca 855948 0 2024-04-16T02:35:42.313 INFO:tasks.workunit.client.0.smithi151.stdout:1/224: dwrite d3/d33/f39 [0,4194304] 0 2024-04-16T02:35:42.314 INFO:tasks.workunit.client.1.smithi153.stdout:1/732: creat d4/dd/d13/d26/db4/ffb x:0 0 0 2024-04-16T02:35:42.315 INFO:tasks.workunit.client.1.smithi153.stdout:5/739: mkdir d8/d41/d90/d9d/dec 0 2024-04-16T02:35:42.320 INFO:tasks.workunit.client.1.smithi153.stdout:7/646: dwrite d0/d4/db/d2b/d43/f9d [0,4194304] 0 2024-04-16T02:35:42.327 INFO:tasks.workunit.client.1.smithi153.stdout:7/647: stat d0/d4/db/d87/dc0 0 2024-04-16T02:35:42.329 INFO:tasks.workunit.client.1.smithi153.stdout:2/659: dread d9/d2b/d1c/d20/fc2 [0,4194304] 0 2024-04-16T02:35:42.329 INFO:tasks.workunit.client.1.smithi153.stdout:8/672: dwrite d2/de/d34/f99 [0,4194304] 0 2024-04-16T02:35:42.333 INFO:tasks.workunit.client.1.smithi153.stdout:8/673: write d2/de/f5f [1901999,37294] 0 2024-04-16T02:35:42.336 INFO:tasks.workunit.client.1.smithi153.stdout:4/806: dwrite d3/d9/d24/d3e/fcf [0,4194304] 0 2024-04-16T02:35:42.336 INFO:tasks.workunit.client.1.smithi153.stdout:5/740: dwrite d8/df/d28/d64/fe5 [0,4194304] 0 2024-04-16T02:35:42.338 INFO:tasks.workunit.client.0.smithi151.stdout:9/156: mknod d1/d3/d6/d1d/d22/d2c/c3a 0 2024-04-16T02:35:42.342 INFO:tasks.workunit.client.0.smithi151.stdout:9/157: mknod d1/d3/d6/d1d/d22/d2c/c3b 0 2024-04-16T02:35:42.343 INFO:tasks.workunit.client.1.smithi153.stdout:6/684: mkdir dc/d47/d57/d77/d15/dac/dae/de7 0 2024-04-16T02:35:42.345 INFO:tasks.workunit.client.0.smithi151.stdout:1/225: rename d3/d33/l6 to d3/l4b 0 2024-04-16T02:35:42.346 INFO:tasks.workunit.client.0.smithi151.stdout:9/158: symlink d1/dd/d38/l3c 0 2024-04-16T02:35:42.347 INFO:tasks.workunit.client.1.smithi153.stdout:2/660: mknod d9/d24/d27/d2c/d41/cdf 0 2024-04-16T02:35:42.347 INFO:tasks.workunit.client.0.smithi151.stdout:1/226: mknod d3/d1f/c4c 0 2024-04-16T02:35:42.348 INFO:tasks.workunit.client.1.smithi153.stdout:8/674: unlink d2/d93/d97/d9c/d27/d2d/d4b/l6b 0 2024-04-16T02:35:42.349 INFO:tasks.workunit.client.0.smithi151.stdout:9/159: rmdir d1/dd/d38 39 2024-04-16T02:35:42.350 INFO:tasks.workunit.client.1.smithi153.stdout:8/675: write d2/d13/d3a/d5b/fd8 [259606,97859] 0 2024-04-16T02:35:42.350 INFO:tasks.workunit.client.1.smithi153.stdout:5/741: rmdir d8/df/d28/d5a/d72/dd0 39 2024-04-16T02:35:42.350 INFO:tasks.workunit.client.0.smithi151.stdout:9/160: fsync d1/d3/d6/f2a 0 2024-04-16T02:35:42.351 INFO:tasks.workunit.client.1.smithi153.stdout:0/798: sync 2024-04-16T02:35:42.352 INFO:tasks.workunit.client.0.smithi151.stdout:1/227: rename d3/d4/ce to d3/d4/db/d1b/d2a/c4d 0 2024-04-16T02:35:42.354 INFO:tasks.workunit.client.1.smithi153.stdout:6/685: rename dc/d47/d93/da9 to dc/d47/d57/d77/d15/de8 0 2024-04-16T02:35:42.354 INFO:tasks.workunit.client.1.smithi153.stdout:4/807: mknod d3/d9/d24/d3e/d103/c11a 0 2024-04-16T02:35:42.355 INFO:tasks.workunit.client.0.smithi151.stdout:1/228: creat d3/d33/d10/f4e x:0 0 0 2024-04-16T02:35:42.355 INFO:tasks.workunit.client.1.smithi153.stdout:5/742: mkdir d8/d41/d90/ded 0 2024-04-16T02:35:42.356 INFO:tasks.workunit.client.1.smithi153.stdout:4/808: chown d3/d9/d24/d3e/de1 1503790022 1 2024-04-16T02:35:42.356 INFO:tasks.workunit.client.1.smithi153.stdout:6/686: truncate dc/d47/d93/fd1 460743 0 2024-04-16T02:35:42.357 INFO:tasks.workunit.client.1.smithi153.stdout:5/743: chown d8/df/fbf 2 1 2024-04-16T02:35:42.357 INFO:tasks.workunit.client.0.smithi151.stdout:1/229: mkdir d3/d33/d10/d4f 0 2024-04-16T02:35:42.358 INFO:tasks.workunit.client.1.smithi153.stdout:9/749: dread d4/d6/d1b/d2a/d99/fce [0,4194304] 0 2024-04-16T02:35:42.360 INFO:tasks.workunit.client.1.smithi153.stdout:0/799: write dd/d14/d19/f1e [2706089,75588] 0 2024-04-16T02:35:42.360 INFO:tasks.workunit.client.0.smithi151.stdout:9/161: dwrite d1/d3/d6/d1d/d22/f23 [0,4194304] 0 2024-04-16T02:35:42.360 INFO:tasks.workunit.client.0.smithi151.stdout:1/230: readlink d3/d33/l14 0 2024-04-16T02:35:42.361 INFO:tasks.workunit.client.0.smithi151.stdout:1/231: chown d3/d1f/c4c 3635 1 2024-04-16T02:35:42.361 INFO:tasks.workunit.client.1.smithi153.stdout:0/800: readlink dd/d14/d19/d77/de3/lea 0 2024-04-16T02:35:42.361 INFO:tasks.workunit.client.1.smithi153.stdout:6/687: creat dc/d10/fe9 x:0 0 0 2024-04-16T02:35:42.361 INFO:tasks.workunit.client.1.smithi153.stdout:5/744: dread - d8/df/d28/d5a/d72/f9e zero size 2024-04-16T02:35:42.362 INFO:tasks.workunit.client.1.smithi153.stdout:0/801: stat dd/d14/d19/d77/de3 0 2024-04-16T02:35:42.362 INFO:tasks.workunit.client.1.smithi153.stdout:5/745: chown d8/df/d23/c6a 8130 1 2024-04-16T02:35:42.366 INFO:tasks.workunit.client.1.smithi153.stdout:1/733: dread d4/dd/d13/d23/d5f/f7f [0,4194304] 0 2024-04-16T02:35:42.367 INFO:tasks.workunit.client.1.smithi153.stdout:4/809: link d3/d9/db/de/f26 d3/d9/db/d17/d28/d4c/f11b 0 2024-04-16T02:35:42.368 INFO:tasks.workunit.client.1.smithi153.stdout:1/734: readlink d4/dd/d13/d26/l63 0 2024-04-16T02:35:42.368 INFO:tasks.workunit.client.1.smithi153.stdout:0/802: read dd/d14/d51/f59 [4174049,39010] 0 2024-04-16T02:35:42.368 INFO:tasks.workunit.client.0.smithi151.stdout:9/162: creat d1/d3/d6/d1d/d22/d36/f3d x:0 0 0 2024-04-16T02:35:42.370 INFO:tasks.workunit.client.1.smithi153.stdout:6/688: dread dc/d10/f34 [0,4194304] 0 2024-04-16T02:35:42.370 INFO:tasks.workunit.client.1.smithi153.stdout:9/750: link d4/d33/dae/dc7/le6 d4/d6/ddb/lec 0 2024-04-16T02:35:42.371 INFO:tasks.workunit.client.1.smithi153.stdout:1/735: chown d4/dd/d13/d23/d31/c8e 14111 1 2024-04-16T02:35:42.371 INFO:tasks.workunit.client.0.smithi151.stdout:9/163: symlink d1/d3/d6/l3e 0 2024-04-16T02:35:42.371 INFO:tasks.workunit.client.0.smithi151.stdout:9/164: chown d1 15 1 2024-04-16T02:35:42.372 INFO:tasks.workunit.client.0.smithi151.stdout:9/165: readlink d1/d3/l14 0 2024-04-16T02:35:42.372 INFO:tasks.workunit.client.1.smithi153.stdout:6/689: truncate dc/d59/d63/f94 5079725 0 2024-04-16T02:35:42.372 INFO:tasks.workunit.client.1.smithi153.stdout:0/803: symlink dd/d32/d80/lfd 0 2024-04-16T02:35:42.373 INFO:tasks.workunit.client.0.smithi151.stdout:9/166: dread - d1/d3/d6/f2a zero size 2024-04-16T02:35:42.375 INFO:tasks.workunit.client.1.smithi153.stdout:4/810: creat d3/d9/db/d17/d28/dc8/de4/dfd/de3/f11c x:0 0 0 2024-04-16T02:35:42.378 INFO:tasks.workunit.client.1.smithi153.stdout:8/676: dread d2/d1c/f26 [0,4194304] 0 2024-04-16T02:35:42.379 INFO:tasks.workunit.client.0.smithi151.stdout:9/167: dwrite d1/d3/f33 [0,4194304] 0 2024-04-16T02:35:42.380 INFO:tasks.workunit.client.1.smithi153.stdout:0/804: symlink dd/d14/d19/d77/d87/lfe 0 2024-04-16T02:35:42.382 INFO:tasks.workunit.client.1.smithi153.stdout:9/751: creat d4/d6/dc/d24/d60/fed x:0 0 0 2024-04-16T02:35:42.383 INFO:tasks.workunit.client.0.smithi151.stdout:4/234: sync 2024-04-16T02:35:42.383 INFO:tasks.workunit.client.0.smithi151.stdout:5/251: sync 2024-04-16T02:35:42.383 INFO:tasks.workunit.client.0.smithi151.stdout:3/143: sync 2024-04-16T02:35:42.383 INFO:tasks.workunit.client.0.smithi151.stdout:9/168: unlink d1/d3/c12 0 2024-04-16T02:35:42.384 INFO:tasks.workunit.client.1.smithi153.stdout:1/736: mkdir d4/dd/d13/ded/dfc 0 2024-04-16T02:35:42.385 INFO:tasks.workunit.client.1.smithi153.stdout:8/677: dwrite d2/ff [0,4194304] 0 2024-04-16T02:35:42.385 INFO:tasks.workunit.client.0.smithi151.stdout:4/235: stat d0/d11/d19/f1b 0 2024-04-16T02:35:42.386 INFO:tasks.workunit.client.0.smithi151.stdout:9/169: fsync d1/d3/d6/f13 0 2024-04-16T02:35:42.386 INFO:tasks.workunit.client.0.smithi151.stdout:4/236: write d0/f1f [189949,17194] 0 2024-04-16T02:35:42.387 INFO:tasks.workunit.client.0.smithi151.stdout:4/237: fdatasync d0/d20/f42 0 2024-04-16T02:35:42.388 INFO:tasks.workunit.client.1.smithi153.stdout:4/811: dread d3/d9/db/d17/f55 [0,4194304] 0 2024-04-16T02:35:42.390 INFO:tasks.workunit.client.0.smithi151.stdout:3/144: mkdir d1/d6/d22/d23/d30 0 2024-04-16T02:35:42.393 INFO:tasks.workunit.client.1.smithi153.stdout:0/805: rmdir dd/d14/d51/d54/d65/d68 39 2024-04-16T02:35:42.393 INFO:tasks.workunit.client.1.smithi153.stdout:7/648: truncate d0/d13/f4a 6898721 0 2024-04-16T02:35:42.393 INFO:tasks.workunit.client.1.smithi153.stdout:2/661: truncate d9/d2b/d1c/d20/fc2 3772627 0 2024-04-16T02:35:42.393 INFO:tasks.workunit.client.1.smithi153.stdout:3/666: sync 2024-04-16T02:35:42.394 INFO:tasks.workunit.client.1.smithi153.stdout:0/806: truncate dd/d14/d19/d77/db6/fec 419282 0 2024-04-16T02:35:42.395 INFO:tasks.workunit.client.1.smithi153.stdout:8/678: creat d2/d93/d97/d9c/d27/d2d/d47/d77/fe2 x:0 0 0 2024-04-16T02:35:42.396 INFO:tasks.workunit.client.0.smithi151.stdout:4/238: creat d0/d11/d19/d31/d34/d3a/f4e x:0 0 0 2024-04-16T02:35:42.398 INFO:tasks.workunit.client.0.smithi151.stdout:5/252: truncate d2/f2d 544482 0 2024-04-16T02:35:42.399 INFO:tasks.workunit.client.1.smithi153.stdout:4/812: dread - d3/d9/db/de/d2c/d52/d85/fda zero size 2024-04-16T02:35:42.399 INFO:tasks.workunit.client.1.smithi153.stdout:9/752: symlink d4/d6/d1b/lee 0 2024-04-16T02:35:42.400 INFO:tasks.workunit.client.0.smithi151.stdout:1/232: creat d3/d4/db/d1b/d1c/f50 x:0 0 0 2024-04-16T02:35:42.401 INFO:tasks.workunit.client.0.smithi151.stdout:1/233: readlink d3/d4/db/df/l35 0 2024-04-16T02:35:42.402 INFO:tasks.workunit.client.0.smithi151.stdout:1/234: readlink d3/d4/db/d1b/d1c/l2b 0 2024-04-16T02:35:42.403 INFO:tasks.workunit.client.1.smithi153.stdout:1/737: dwrite d4/fc4 [0,4194304] 0 2024-04-16T02:35:42.403 INFO:tasks.workunit.client.0.smithi151.stdout:1/235: chown d3/d4/db/df/d32 720111870 1 2024-04-16T02:35:42.406 INFO:tasks.workunit.client.0.smithi151.stdout:3/145: truncate d1/d6/d20/d2a/f1a 782614 0 2024-04-16T02:35:42.406 INFO:tasks.workunit.client.1.smithi153.stdout:6/690: write dc/d1a/d2d/f44 [2129167,87592] 0 2024-04-16T02:35:42.408 INFO:tasks.workunit.client.1.smithi153.stdout:3/667: dread d6/d10/d3b/d76/fd1 [0,4194304] 0 2024-04-16T02:35:42.408 INFO:tasks.workunit.client.0.smithi151.stdout:1/236: rename f1 to d3/d1f/d48/f51 0 2024-04-16T02:35:42.410 INFO:tasks.workunit.client.1.smithi153.stdout:8/679: link d2/d93/d97/d9c/d27/d5e/d63/dc6/cd2 d2/d93/ce3 0 2024-04-16T02:35:42.411 INFO:tasks.workunit.client.1.smithi153.stdout:4/813: link d3/d9/db/d17/f116 d3/f11d 0 2024-04-16T02:35:42.411 INFO:tasks.workunit.client.1.smithi153.stdout:0/807: dread dd/d14/d51/d54/d65/f72 [4194304,4194304] 0 2024-04-16T02:35:42.412 INFO:tasks.workunit.client.0.smithi151.stdout:5/253: creat d2/d6/d19/f4b x:0 0 0 2024-04-16T02:35:42.414 INFO:tasks.workunit.client.0.smithi151.stdout:3/146: getdents d1/d6/d22/d23 0 2024-04-16T02:35:42.415 INFO:tasks.workunit.client.1.smithi153.stdout:5/746: dwrite d8/d41/dbd/d39/f4c [0,4194304] 0 2024-04-16T02:35:42.416 INFO:tasks.workunit.client.1.smithi153.stdout:4/814: dread - d3/d9/db/d17/f116 zero size 2024-04-16T02:35:42.417 INFO:tasks.workunit.client.1.smithi153.stdout:3/668: write d6/d10/d22/f2a [3052368,91561] 0 2024-04-16T02:35:42.418 INFO:tasks.workunit.client.0.smithi151.stdout:3/147: creat d1/d6/f31 x:0 0 0 2024-04-16T02:35:42.420 INFO:tasks.workunit.client.0.smithi151.stdout:1/237: dwrite d3/d33/d10/f4e [0,4194304] 0 2024-04-16T02:35:42.421 INFO:tasks.workunit.client.1.smithi153.stdout:4/815: chown d3/d9/d24/d75/d9e/cf5 6266234 1 2024-04-16T02:35:42.423 INFO:tasks.workunit.client.1.smithi153.stdout:6/691: rename dc/d59/d5b/ce4 to dc/cea 0 2024-04-16T02:35:42.424 INFO:tasks.workunit.client.1.smithi153.stdout:8/680: dwrite d2/d93/d97/d9c/d27/fb9 [0,4194304] 0 2024-04-16T02:35:42.427 INFO:tasks.workunit.client.1.smithi153.stdout:5/747: mknod d8/df/d2f/cee 0 2024-04-16T02:35:42.427 INFO:tasks.workunit.client.0.smithi151.stdout:5/254: dwrite d2/d6/f29 [0,4194304] 0 2024-04-16T02:35:42.428 INFO:tasks.workunit.client.0.smithi151.stdout:3/148: symlink d1/d6/d20/l32 0 2024-04-16T02:35:42.429 INFO:tasks.workunit.client.0.smithi151.stdout:5/255: chown d2/d6/f13 1894 1 2024-04-16T02:35:42.429 INFO:tasks.workunit.client.0.smithi151.stdout:3/149: fsync d1/f1d 0 2024-04-16T02:35:42.432 INFO:tasks.workunit.client.0.smithi151.stdout:5/256: write d2/d6/d19/d30/f39 [490223,108461] 0 2024-04-16T02:35:42.433 INFO:tasks.workunit.client.0.smithi151.stdout:1/238: mkdir d3/d4/db/d1b/d2a/d52 0 2024-04-16T02:35:42.433 INFO:tasks.workunit.client.1.smithi153.stdout:0/808: rename dd/d14/d51/d54/d65/d68/l8c to dd/d14/d51/d7f/d9d/lff 0 2024-04-16T02:35:42.436 INFO:tasks.workunit.client.1.smithi153.stdout:6/692: dwrite dc/d59/d63/f66 [4194304,4194304] 0 2024-04-16T02:35:42.440 INFO:tasks.workunit.client.0.smithi151.stdout:5/257: mkdir d2/d6/d19/d1a/d4c 0 2024-04-16T02:35:42.445 INFO:tasks.workunit.client.0.smithi151.stdout:3/150: creat d1/d6/f33 x:0 0 0 2024-04-16T02:35:42.447 INFO:tasks.workunit.client.1.smithi153.stdout:9/753: dread d4/d6/ddb/d4a/f68 [0,4194304] 0 2024-04-16T02:35:42.447 INFO:tasks.workunit.client.1.smithi153.stdout:9/754: write d4/d33/fd1 [277049,90228] 0 2024-04-16T02:35:42.449 INFO:tasks.workunit.client.0.smithi151.stdout:3/151: fdatasync d1/d6/d20/d2a/f16 0 2024-04-16T02:35:42.449 INFO:tasks.workunit.client.0.smithi151.stdout:1/239: link d3/l15 d3/l53 0 2024-04-16T02:35:42.451 INFO:tasks.workunit.client.1.smithi153.stdout:8/681: mkdir d2/d93/d97/d9c/d27/d5e/de4 0 2024-04-16T02:35:42.451 INFO:tasks.workunit.client.0.smithi151.stdout:5/258: dwrite d2/d6/d19/f4b [0,4194304] 0 2024-04-16T02:35:42.451 INFO:tasks.workunit.client.1.smithi153.stdout:3/669: creat d6/d10/d3b/d59/d7c/de1/fe3 x:0 0 0 2024-04-16T02:35:42.453 INFO:tasks.workunit.client.0.smithi151.stdout:3/152: chown d1/ld 6 1 2024-04-16T02:35:42.453 INFO:tasks.workunit.client.1.smithi153.stdout:5/748: symlink d8/d41/d90/d9d/lef 0 2024-04-16T02:35:42.456 INFO:tasks.workunit.client.0.smithi151.stdout:5/259: rename d2/c26 to d2/d6/d19/d1a/c4d 0 2024-04-16T02:35:42.456 INFO:tasks.workunit.client.0.smithi151.stdout:1/240: write d3/d4/db/d1b/d1c/f2c [4450335,56375] 0 2024-04-16T02:35:42.459 INFO:tasks.workunit.client.0.smithi151.stdout:1/241: write d3/d4/db/d1b/d1c/f2c [2132556,29467] 0 2024-04-16T02:35:42.459 INFO:tasks.workunit.client.0.smithi151.stdout:5/260: write d2/f12 [2465264,65447] 0 2024-04-16T02:35:42.459 INFO:tasks.workunit.client.0.smithi151.stdout:4/239: dread d0/d11/d19/f25 [0,4194304] 0 2024-04-16T02:35:42.463 INFO:tasks.workunit.client.0.smithi151.stdout:4/240: fsync d0/f4c 0 2024-04-16T02:35:42.464 INFO:tasks.workunit.client.1.smithi153.stdout:0/809: symlink dd/d14/d19/d77/l100 0 2024-04-16T02:35:42.465 INFO:tasks.workunit.client.0.smithi151.stdout:4/241: truncate d0/d20/f2c 831670 0 2024-04-16T02:35:42.466 INFO:tasks.workunit.client.1.smithi153.stdout:7/649: dwrite d0/d4/db/d2b/f25 [0,4194304] 0 2024-04-16T02:35:42.466 INFO:tasks.workunit.client.1.smithi153.stdout:5/749: dwrite f4 [0,4194304] 0 2024-04-16T02:35:42.466 INFO:tasks.workunit.client.1.smithi153.stdout:4/816: creat d3/d9/db/de/d2c/d52/d85/f11e x:0 0 0 2024-04-16T02:35:42.466 INFO:tasks.workunit.client.0.smithi151.stdout:5/261: getdents d2/d6/d19/d30/d36/d42 0 2024-04-16T02:35:42.467 INFO:tasks.workunit.client.0.smithi151.stdout:1/242: dread d3/d4/db/d1b/d1c/f2c [0,4194304] 0 2024-04-16T02:35:42.469 INFO:tasks.workunit.client.1.smithi153.stdout:9/755: mkdir d4/d6/def 0 2024-04-16T02:35:42.469 INFO:tasks.workunit.client.0.smithi151.stdout:4/242: unlink d0/l2e 0 2024-04-16T02:35:42.473 INFO:tasks.workunit.client.0.smithi151.stdout:4/243: rmdir d0/d17 39 2024-04-16T02:35:42.477 INFO:tasks.workunit.client.0.smithi151.stdout:4/244: creat d0/d20/f4f x:0 0 0 2024-04-16T02:35:42.478 INFO:tasks.workunit.client.1.smithi153.stdout:6/693: getdents dc/d47/d57/d77/db8 0 2024-04-16T02:35:42.479 INFO:tasks.workunit.client.0.smithi151.stdout:4/245: write d0/d11/d19/f1b [270555,98444] 0 2024-04-16T02:35:42.481 INFO:tasks.workunit.client.1.smithi153.stdout:6/694: dread - dc/d47/d57/d77/dc4/dc9/fca zero size 2024-04-16T02:35:42.481 INFO:tasks.workunit.client.1.smithi153.stdout:7/650: rename d0/f1e to d0/d4/db/d2b/d53/d59/dc3/fdc 0 2024-04-16T02:35:42.482 INFO:tasks.workunit.client.0.smithi151.stdout:1/243: dread d3/d33/f36 [0,4194304] 0 2024-04-16T02:35:42.483 INFO:tasks.workunit.client.0.smithi151.stdout:4/246: write d0/d11/d19/f25 [2359882,9269] 0 2024-04-16T02:35:42.484 INFO:tasks.workunit.client.1.smithi153.stdout:0/810: dwrite dd/d14/d19/d20/f70 [0,4194304] 0 2024-04-16T02:35:42.484 INFO:tasks.workunit.client.1.smithi153.stdout:3/670: creat d6/fe4 x:0 0 0 2024-04-16T02:35:42.484 INFO:tasks.workunit.client.1.smithi153.stdout:9/756: dwrite d4/d6/dc/d24/f30 [0,4194304] 0 2024-04-16T02:35:42.486 INFO:tasks.workunit.client.1.smithi153.stdout:8/682: dread d2/d93/d97/d9c/d27/d2d/d47/d60/d67/fa2 [0,4194304] 0 2024-04-16T02:35:42.486 INFO:tasks.workunit.client.1.smithi153.stdout:4/817: creat d3/d9/db/d17/d28/dc8/f11f x:0 0 0 2024-04-16T02:35:42.486 INFO:tasks.workunit.client.0.smithi151.stdout:1/244: creat d3/d4/db/f54 x:0 0 0 2024-04-16T02:35:42.487 INFO:tasks.workunit.client.0.smithi151.stdout:4/247: rmdir d0/d11 39 2024-04-16T02:35:42.487 INFO:tasks.workunit.client.1.smithi153.stdout:0/811: write dd/d14/da6/da4/db1/fc9 [4822595,122357] 0 2024-04-16T02:35:42.488 INFO:tasks.workunit.client.0.smithi151.stdout:1/245: creat d3/d33/d10/f55 x:0 0 0 2024-04-16T02:35:42.491 INFO:tasks.workunit.client.0.smithi151.stdout:1/246: stat d3/d4/db/df/c25 0 2024-04-16T02:35:42.493 INFO:tasks.workunit.client.0.smithi151.stdout:1/247: dread - d3/d4/db/d1b/d1c/f50 zero size 2024-04-16T02:35:42.494 INFO:tasks.workunit.client.0.smithi151.stdout:4/248: dread d0/d11/d19/f1b [4194304,4194304] 0 2024-04-16T02:35:42.494 INFO:tasks.workunit.client.0.smithi151.stdout:1/248: dread - d3/d4/db/d1b/d1c/f50 zero size 2024-04-16T02:35:42.496 INFO:tasks.workunit.client.1.smithi153.stdout:7/651: write d0/d4/db/d2b/d53/d59/d76/dc8/fb4 [138147,115159] 0 2024-04-16T02:35:42.496 INFO:tasks.workunit.client.1.smithi153.stdout:3/671: chown d6/d10/d22/cd2 320819078 1 2024-04-16T02:35:42.496 INFO:tasks.workunit.client.1.smithi153.stdout:0/812: creat dd/d32/d3f/f101 x:0 0 0 2024-04-16T02:35:42.497 INFO:tasks.workunit.client.0.smithi151.stdout:1/249: unlink d3/d1f/l20 0 2024-04-16T02:35:42.502 INFO:tasks.workunit.client.1.smithi153.stdout:6/695: write dc/d47/d57/d77/d15/fc2 [1010565,39036] 0 2024-04-16T02:35:42.502 INFO:tasks.workunit.client.1.smithi153.stdout:5/750: dwrite d8/d1f/f73 [0,4194304] 0 2024-04-16T02:35:42.502 INFO:tasks.workunit.client.1.smithi153.stdout:2/662: sync 2024-04-16T02:35:42.503 INFO:tasks.workunit.client.1.smithi153.stdout:7/652: write d0/d4/db/d2b/d8c/fae [888726,94536] 0 2024-04-16T02:35:42.504 INFO:tasks.workunit.client.0.smithi151.stdout:1/250: rename d3/d4/db/d1b/d1c/l2b to d3/d1f/l56 0 2024-04-16T02:35:42.504 INFO:tasks.workunit.client.1.smithi153.stdout:3/672: symlink d6/d10/d22/d4d/dbb/le5 0 2024-04-16T02:35:42.507 INFO:tasks.workunit.client.0.smithi151.stdout:4/249: dwrite d0/d11/d19/d31/d34/f3f [0,4194304] 0 2024-04-16T02:35:42.507 INFO:tasks.workunit.client.1.smithi153.stdout:7/653: fsync d0/d4/db/f64 0 2024-04-16T02:35:42.508 INFO:tasks.workunit.client.1.smithi153.stdout:7/654: write d0/fa [215981,22773] 0 2024-04-16T02:35:42.510 INFO:tasks.workunit.client.0.smithi151.stdout:4/250: mkdir d0/d11/d19/d31/d34/d3a/d50 0 2024-04-16T02:35:42.511 INFO:tasks.workunit.client.1.smithi153.stdout:4/818: getdents d3/d9/d24 0 2024-04-16T02:35:42.514 INFO:tasks.workunit.client.0.smithi151.stdout:5/262: write d2/d6/d43/f4a [2423317,44257] 0 2024-04-16T02:35:42.516 INFO:tasks.workunit.client.0.smithi151.stdout:4/251: creat d0/d11/d19/d31/d34/d3a/d50/f51 x:0 0 0 2024-04-16T02:35:42.518 INFO:tasks.workunit.client.0.smithi151.stdout:5/263: write d2/d6/d19/d30/f39 [1634060,12835] 0 2024-04-16T02:35:42.518 INFO:tasks.workunit.client.1.smithi153.stdout:4/819: mkdir d3/d9/db/d17/d28/dc8/de4/dfd/d120 0 2024-04-16T02:35:42.520 INFO:tasks.workunit.client.1.smithi153.stdout:3/673: mkdir d6/de6 0 2024-04-16T02:35:42.521 INFO:tasks.workunit.client.1.smithi153.stdout:4/820: truncate d3/d9/db/d45/df3/f95 141324 0 2024-04-16T02:35:42.522 INFO:tasks.workunit.client.1.smithi153.stdout:6/696: creat dc/d47/d57/feb x:0 0 0 2024-04-16T02:35:42.524 INFO:tasks.workunit.client.1.smithi153.stdout:2/663: creat d9/d2b/d18/fe0 x:0 0 0 2024-04-16T02:35:42.524 INFO:tasks.workunit.client.1.smithi153.stdout:3/674: creat d6/d10/d3b/d59/d7c/de1/fe7 x:0 0 0 2024-04-16T02:35:42.526 INFO:tasks.workunit.client.1.smithi153.stdout:6/697: mknod dc/d10/cec 0 2024-04-16T02:35:42.527 INFO:tasks.workunit.client.0.smithi151.stdout:5/264: mknod d2/d6/d19/d1a/c4e 0 2024-04-16T02:35:42.527 INFO:tasks.workunit.client.1.smithi153.stdout:8/683: write d2/d93/d97/fa9 [475356,29204] 0 2024-04-16T02:35:42.527 INFO:tasks.workunit.client.1.smithi153.stdout:6/698: chown f9 99284 1 2024-04-16T02:35:42.527 INFO:tasks.workunit.client.1.smithi153.stdout:2/664: dread - d9/d24/d27/d2c/d41/f4e zero size 2024-04-16T02:35:42.528 INFO:tasks.workunit.client.0.smithi151.stdout:5/265: fsync d2/d6/f29 0 2024-04-16T02:35:42.530 INFO:tasks.workunit.client.1.smithi153.stdout:6/699: chown dc/d47/d93/faa 26 1 2024-04-16T02:35:42.530 INFO:tasks.workunit.client.0.smithi151.stdout:5/266: symlink d2/d6/d19/d1a/d4c/l4f 0 2024-04-16T02:35:42.531 INFO:tasks.workunit.client.1.smithi153.stdout:3/675: symlink d6/d10/d22/d2e/d8a/d8c/le8 0 2024-04-16T02:35:42.531 INFO:tasks.workunit.client.1.smithi153.stdout:0/813: write dd/d14/da6/f98 [1726464,26402] 0 2024-04-16T02:35:42.531 INFO:tasks.workunit.client.1.smithi153.stdout:8/684: unlink d2/d13/d2c/c52 0 2024-04-16T02:35:42.533 INFO:tasks.workunit.client.1.smithi153.stdout:4/821: creat d3/d9/db/de/d2c/d52/d85/f121 x:0 0 0 2024-04-16T02:35:42.535 INFO:tasks.workunit.client.1.smithi153.stdout:4/822: stat d3/d9/d24/d75/d9e/ca7 0 2024-04-16T02:35:42.535 INFO:tasks.workunit.client.1.smithi153.stdout:6/700: mkdir dc/d47/d57/d77/dc4/dc9/ded 0 2024-04-16T02:35:42.536 INFO:tasks.workunit.client.1.smithi153.stdout:9/757: dwrite d4/d6/dc/fd7 [0,4194304] 0 2024-04-16T02:35:42.539 INFO:tasks.workunit.client.1.smithi153.stdout:3/676: creat d6/d10/d3b/d59/fe9 x:0 0 0 2024-04-16T02:35:42.539 INFO:tasks.workunit.client.1.smithi153.stdout:8/685: mkdir d2/d93/d97/de5 0 2024-04-16T02:35:42.539 INFO:tasks.workunit.client.1.smithi153.stdout:0/814: mknod dd/d26/c102 0 2024-04-16T02:35:42.539 INFO:tasks.workunit.client.1.smithi153.stdout:6/701: chown dc/d6b/f73 29 1 2024-04-16T02:35:42.541 INFO:tasks.workunit.client.0.smithi151.stdout:5/267: dwrite d2/d6/f1e [0,4194304] 0 2024-04-16T02:35:42.541 INFO:tasks.workunit.client.1.smithi153.stdout:2/665: write d9/d71/d8d/da8/fbe [4691574,128553] 0 2024-04-16T02:35:42.542 INFO:tasks.workunit.client.1.smithi153.stdout:9/758: write d4/d6/dc/d2d/d64/db9/fcf [973182,20930] 0 2024-04-16T02:35:42.542 INFO:tasks.workunit.client.1.smithi153.stdout:8/686: rmdir d2/d1c 39 2024-04-16T02:35:42.543 INFO:tasks.workunit.client.1.smithi153.stdout:5/751: dwrite d8/df/d2e/f3d [0,4194304] 0 2024-04-16T02:35:42.548 INFO:tasks.workunit.client.0.smithi151.stdout:4/252: dwrite d0/fe [0,4194304] 0 2024-04-16T02:35:42.550 INFO:tasks.workunit.client.1.smithi153.stdout:7/655: dwrite d0/d4/db/d2b/f38 [0,4194304] 0 2024-04-16T02:35:42.551 INFO:tasks.workunit.client.1.smithi153.stdout:5/752: dread d8/df/f11 [0,4194304] 0 2024-04-16T02:35:42.552 INFO:tasks.workunit.client.0.smithi151.stdout:4/253: fdatasync d0/d11/d19/d31/d34/d3a/f4e 0 2024-04-16T02:35:42.554 INFO:tasks.workunit.client.1.smithi153.stdout:5/753: fdatasync d8/d41/dbd/d83/fdb 0 2024-04-16T02:35:42.555 INFO:tasks.workunit.client.0.smithi151.stdout:5/268: creat d2/d6/f50 x:0 0 0 2024-04-16T02:35:42.557 INFO:tasks.workunit.client.1.smithi153.stdout:4/823: creat d3/d9/db/de/d2c/d38/f122 x:0 0 0 2024-04-16T02:35:42.558 INFO:tasks.workunit.client.1.smithi153.stdout:8/687: symlink d2/d13/d3a/dc2/le6 0 2024-04-16T02:35:42.558 INFO:tasks.workunit.client.1.smithi153.stdout:5/754: dread - d8/fde zero size 2024-04-16T02:35:42.559 INFO:tasks.workunit.client.1.smithi153.stdout:7/656: creat d0/d4/db/d47/d8d/d93/fdd x:0 0 0 2024-04-16T02:35:42.561 INFO:tasks.workunit.client.0.smithi151.stdout:4/254: creat d0/d17/f52 x:0 0 0 2024-04-16T02:35:42.562 INFO:tasks.workunit.client.0.smithi151.stdout:5/269: mknod d2/d6/d43/c51 0 2024-04-16T02:35:42.562 INFO:tasks.workunit.client.1.smithi153.stdout:7/657: chown d0/d4/db/d47 214870100 1 2024-04-16T02:35:42.563 INFO:tasks.workunit.client.0.smithi151.stdout:5/270: write d2/f5 [3745807,54704] 0 2024-04-16T02:35:42.563 INFO:tasks.workunit.client.1.smithi153.stdout:9/759: dwrite d4/d6/dc/d24/dc6/fd0 [0,4194304] 0 2024-04-16T02:35:42.564 INFO:tasks.workunit.client.1.smithi153.stdout:4/824: write d3/d9/db/de/d2c/f56 [1858184,78585] 0 2024-04-16T02:35:42.565 INFO:tasks.workunit.client.1.smithi153.stdout:5/755: readlink d8/df/d84/da4/ld5 0 2024-04-16T02:35:42.566 INFO:tasks.workunit.client.0.smithi151.stdout:5/271: creat d2/d6/d19/d1a/f52 x:0 0 0 2024-04-16T02:35:42.567 INFO:tasks.workunit.client.0.smithi151.stdout:4/255: link d0/d11/f43 d0/d11/f53 0 2024-04-16T02:35:42.568 INFO:tasks.workunit.client.0.smithi151.stdout:5/272: creat d2/d6/d19/d1a/d4c/f53 x:0 0 0 2024-04-16T02:35:42.570 INFO:tasks.workunit.client.1.smithi153.stdout:4/825: symlink d3/d9/d24/d3e/d103/l123 0 2024-04-16T02:35:42.570 INFO:tasks.workunit.client.0.smithi151.stdout:4/256: symlink d0/d11/d19/d31/d34/l54 0 2024-04-16T02:35:42.571 INFO:tasks.workunit.client.0.smithi151.stdout:5/273: symlink d2/d6/d19/d30/d49/l54 0 2024-04-16T02:35:42.571 INFO:tasks.workunit.client.1.smithi153.stdout:5/756: symlink d8/d41/dbd/dd3/lf0 0 2024-04-16T02:35:42.572 INFO:tasks.workunit.client.0.smithi151.stdout:5/274: dread - d2/d6/d19/d1a/f48 zero size 2024-04-16T02:35:42.572 INFO:tasks.workunit.client.1.smithi153.stdout:2/666: getdents d9/d24/d27/d2c/d41/d7a 0 2024-04-16T02:35:42.573 INFO:tasks.workunit.client.1.smithi153.stdout:7/658: creat d0/d4/db/d34/d51/fde x:0 0 0 2024-04-16T02:35:42.574 INFO:tasks.workunit.client.1.smithi153.stdout:8/688: mknod d2/d93/d97/d9c/d27/ce7 0 2024-04-16T02:35:42.575 INFO:tasks.workunit.client.0.smithi151.stdout:4/257: link d0/d11/f23 d0/f55 0 2024-04-16T02:35:42.576 INFO:tasks.workunit.client.0.smithi151.stdout:5/275: symlink d2/d6/d43/l55 0 2024-04-16T02:35:42.577 INFO:tasks.workunit.client.0.smithi151.stdout:5/276: dread - d2/d6/d19/d1a/f40 zero size 2024-04-16T02:35:42.577 INFO:tasks.workunit.client.1.smithi153.stdout:5/757: chown d8/df/d2f/d4e/d51/l5f 29955 1 2024-04-16T02:35:42.578 INFO:tasks.workunit.client.1.smithi153.stdout:3/677: dread d6/d10/d3b/d76/f7d [0,4194304] 0 2024-04-16T02:35:42.579 INFO:tasks.workunit.client.0.smithi151.stdout:4/258: rename d0/c7 to d0/d11/d19/d31/d34/d3a/c56 0 2024-04-16T02:35:42.579 INFO:tasks.workunit.client.1.smithi153.stdout:8/689: mkdir d2/d13/de8 0 2024-04-16T02:35:42.580 INFO:tasks.workunit.client.1.smithi153.stdout:7/659: truncate d0/d4/db/f64 4461453 0 2024-04-16T02:35:42.580 INFO:tasks.workunit.client.0.smithi151.stdout:5/277: mknod d2/d6/c56 0 2024-04-16T02:35:42.581 INFO:tasks.workunit.client.1.smithi153.stdout:0/815: dwrite dd/d14/f4b [0,4194304] 0 2024-04-16T02:35:42.582 INFO:tasks.workunit.client.0.smithi151.stdout:4/259: unlink d0/d20/l30 0 2024-04-16T02:35:42.582 INFO:tasks.workunit.client.1.smithi153.stdout:2/667: write d9/d24/d27/d2c/d5d/da3/dc0/fd1 [549927,24322] 0 2024-04-16T02:35:42.583 INFO:tasks.workunit.client.1.smithi153.stdout:4/826: dwrite d3/d9/f3d [0,4194304] 0 2024-04-16T02:35:42.583 INFO:tasks.workunit.client.0.smithi151.stdout:5/278: mknod d2/d6/d43/c57 0 2024-04-16T02:35:42.583 INFO:tasks.workunit.client.1.smithi153.stdout:3/678: write d6/f97 [87587,107387] 0 2024-04-16T02:35:42.584 INFO:tasks.workunit.client.0.smithi151.stdout:5/279: chown d2/d6/d19/d30/f3f 0 1 2024-04-16T02:35:42.587 INFO:tasks.workunit.client.1.smithi153.stdout:5/758: read d8/df/d2e/f57 [1928282,9870] 0 2024-04-16T02:35:42.588 INFO:tasks.workunit.client.1.smithi153.stdout:4/827: chown d3/dbc 609 1 2024-04-16T02:35:42.591 INFO:tasks.workunit.client.1.smithi153.stdout:8/690: creat d2/d93/d97/d9c/d27/d2d/d47/d60/d67/d86/fe9 x:0 0 0 2024-04-16T02:35:42.595 INFO:tasks.workunit.client.1.smithi153.stdout:0/816: truncate dd/ff 3855069 0 2024-04-16T02:35:42.598 INFO:tasks.workunit.client.1.smithi153.stdout:7/660: mkdir d0/d4/db/d2b/d43/d6d/d68/d6c/db8/ddf 0 2024-04-16T02:35:42.600 INFO:tasks.workunit.client.1.smithi153.stdout:2/668: dwrite d9/d2b/ff [0,4194304] 0 2024-04-16T02:35:42.600 INFO:tasks.workunit.client.1.smithi153.stdout:7/661: readlink d0/d4/db/d2b/d43/lc7 0 2024-04-16T02:35:42.602 INFO:tasks.workunit.client.1.smithi153.stdout:9/760: truncate d4/d6/f56 1128590 0 2024-04-16T02:35:42.604 INFO:tasks.workunit.client.1.smithi153.stdout:9/761: write d4/d6/dc/d2d/fa8 [931503,59937] 0 2024-04-16T02:35:42.605 INFO:tasks.workunit.client.1.smithi153.stdout:4/828: creat d3/d9/db/d17/d28/dc8/de4/dfd/de3/df0/f124 x:0 0 0 2024-04-16T02:35:42.607 INFO:tasks.workunit.client.0.smithi151.stdout:5/280: dwrite d2/d6/f13 [0,4194304] 0 2024-04-16T02:35:42.608 INFO:tasks.workunit.client.1.smithi153.stdout:9/762: truncate d4/d33/f4f 4425532 0 2024-04-16T02:35:42.608 INFO:tasks.workunit.client.1.smithi153.stdout:2/669: dwrite d9/d2b/ff [4194304,4194304] 0 2024-04-16T02:35:42.610 INFO:tasks.workunit.client.1.smithi153.stdout:9/763: readlink d4/d6/d1b/l35 0 2024-04-16T02:35:42.610 INFO:tasks.workunit.client.1.smithi153.stdout:5/759: dread d8/df/d2e/f6e [0,4194304] 0 2024-04-16T02:35:42.610 INFO:tasks.workunit.client.1.smithi153.stdout:7/662: mkdir d0/d4/db/d87/dc0/de0 0 2024-04-16T02:35:42.613 INFO:tasks.workunit.client.0.smithi151.stdout:5/281: creat d2/d6/d19/d30/d36/d42/f58 x:0 0 0 2024-04-16T02:35:42.613 INFO:tasks.workunit.client.1.smithi153.stdout:8/691: dread d2/d93/d97/d9c/d27/d5e/d63/f8f [0,4194304] 0 2024-04-16T02:35:42.615 INFO:tasks.workunit.client.1.smithi153.stdout:4/829: symlink d3/d9/db/de/d2c/d38/l125 0 2024-04-16T02:35:42.616 INFO:tasks.workunit.client.0.smithi151.stdout:5/282: write d2/d6/f1e [4607609,102557] 0 2024-04-16T02:35:42.616 INFO:tasks.workunit.client.0.smithi151.stdout:5/283: chown d2/d6/f9 0 1 2024-04-16T02:35:42.617 INFO:tasks.workunit.client.1.smithi153.stdout:5/760: truncate d8/d41/dbd/d39/d6d/fe4 354306 0 2024-04-16T02:35:42.618 INFO:tasks.workunit.client.1.smithi153.stdout:2/670: dread - d9/d24/d27/d2c/d41/f4e zero size 2024-04-16T02:35:42.620 INFO:tasks.workunit.client.1.smithi153.stdout:5/761: write d8/d41/fb3 [2899132,110704] 0 2024-04-16T02:35:42.620 INFO:tasks.workunit.client.0.smithi151.stdout:5/284: mknod d2/d6/d19/d30/c59 0 2024-04-16T02:35:42.622 INFO:tasks.workunit.client.1.smithi153.stdout:5/762: fsync d8/d41/dbd/d39/f4c 0 2024-04-16T02:35:42.622 INFO:tasks.workunit.client.1.smithi153.stdout:4/830: read d3/d9/d24/fa4 [3557106,75414] 0 2024-04-16T02:35:42.623 INFO:tasks.workunit.client.0.smithi151.stdout:5/285: stat d2/d6/d19/d1a/c4d 0 2024-04-16T02:35:42.624 INFO:tasks.workunit.client.1.smithi153.stdout:2/671: creat d9/d2b/d5c/da2/fe1 x:0 0 0 2024-04-16T02:35:42.625 INFO:tasks.workunit.client.0.smithi151.stdout:5/286: creat d2/d6/d19/d30/d36/d42/f5a x:0 0 0 2024-04-16T02:35:42.625 INFO:tasks.workunit.client.0.smithi151.stdout:4/260: dread d0/d11/d19/d31/d34/f48 [0,4194304] 0 2024-04-16T02:35:42.626 INFO:tasks.workunit.client.1.smithi153.stdout:5/763: chown d8/d41/dbd/d83/fdb 77 1 2024-04-16T02:35:42.626 INFO:tasks.workunit.client.0.smithi151.stdout:4/261: chown d0/d11/d19/d31/d34/d3a/d50/f51 266040376 1 2024-04-16T02:35:42.627 INFO:tasks.workunit.client.1.smithi153.stdout:8/692: link d2/d13/f9f d2/d13/d2c/fea 0 2024-04-16T02:35:42.628 INFO:tasks.workunit.client.0.smithi151.stdout:4/262: rmdir d0/d20 39 2024-04-16T02:35:42.629 INFO:tasks.workunit.client.1.smithi153.stdout:2/672: write d9/d2b/f6b [713600,107871] 0 2024-04-16T02:35:42.629 INFO:tasks.workunit.client.0.smithi151.stdout:4/263: write d0/f10 [4209747,15837] 0 2024-04-16T02:35:42.629 INFO:tasks.workunit.client.1.smithi153.stdout:7/663: getdents d0/d4/db/d34 0 2024-04-16T02:35:42.631 INFO:tasks.workunit.client.1.smithi153.stdout:7/664: chown d0/d4/db/d87/dc0/dc6 8985 1 2024-04-16T02:35:42.631 INFO:tasks.workunit.client.1.smithi153.stdout:8/693: write d2/d93/d97/d9c/d27/fc3 [217160,29581] 0 2024-04-16T02:35:42.631 INFO:tasks.workunit.client.1.smithi153.stdout:5/764: unlink d8/d41/dbd/d39/ce8 0 2024-04-16T02:35:42.632 INFO:tasks.workunit.client.0.smithi151.stdout:4/264: symlink d0/d11/d19/d31/d34/d3a/d50/l57 0 2024-04-16T02:35:42.632 INFO:tasks.workunit.client.0.smithi151.stdout:4/265: chown d0/d11/d19/f29 385759 1 2024-04-16T02:35:42.635 INFO:tasks.workunit.client.1.smithi153.stdout:3/679: dread d6/d10/d22/f2a [0,4194304] 0 2024-04-16T02:35:42.636 INFO:tasks.workunit.client.1.smithi153.stdout:5/765: mknod d8/d41/d90/cf1 0 2024-04-16T02:35:42.636 INFO:tasks.workunit.client.1.smithi153.stdout:0/817: dread dd/d28/f3c [0,4194304] 0 2024-04-16T02:35:42.637 INFO:tasks.workunit.client.0.smithi151.stdout:4/266: rename d0/d11/f39 to d0/d11/d19/d46/f58 0 2024-04-16T02:35:42.638 INFO:tasks.workunit.client.0.smithi151.stdout:4/267: chown d0/f55 2680 1 2024-04-16T02:35:42.638 INFO:tasks.workunit.client.1.smithi153.stdout:3/680: write d6/d10/d22/d2e/f4e [187210,37076] 0 2024-04-16T02:35:42.638 INFO:tasks.workunit.client.0.smithi151.stdout:4/268: fsync d0/d11/d19/d31/d34/f3f 0 2024-04-16T02:35:42.641 INFO:tasks.workunit.client.1.smithi153.stdout:1/738: sync 2024-04-16T02:35:42.644 INFO:tasks.workunit.client.0.smithi151.stdout:5/287: rmdir d2/d6/d19/d30/d36/d42 39 2024-04-16T02:35:42.644 INFO:tasks.workunit.client.1.smithi153.stdout:0/818: creat dd/d32/d57/d6c/dd1/f103 x:0 0 0 2024-04-16T02:35:42.644 INFO:tasks.workunit.client.1.smithi153.stdout:5/766: dread d8/f6c [0,4194304] 0 2024-04-16T02:35:42.645 INFO:tasks.workunit.client.1.smithi153.stdout:1/739: write d4/f7 [8926176,83120] 0 2024-04-16T02:35:42.645 INFO:tasks.workunit.client.0.smithi151.stdout:4/269: getdents d0/d11 0 2024-04-16T02:35:42.646 INFO:tasks.workunit.client.0.smithi151.stdout:5/288: rmdir d2/d6/d19/d30/d49 39 2024-04-16T02:35:42.646 INFO:tasks.workunit.client.0.smithi151.stdout:5/289: write d2/fb [1757362,13097] 0 2024-04-16T02:35:42.647 INFO:tasks.workunit.client.1.smithi153.stdout:8/694: dwrite d2/d93/d97/d9c/d27/d5e/d63/fb1 [0,4194304] 0 2024-04-16T02:35:42.647 INFO:tasks.workunit.client.1.smithi153.stdout:0/819: dread - dd/d14/d19/fa0 zero size 2024-04-16T02:35:42.654 INFO:tasks.workunit.client.1.smithi153.stdout:5/767: dwrite d8/d1f/feb [0,4194304] 0 2024-04-16T02:35:42.656 INFO:tasks.workunit.client.1.smithi153.stdout:8/695: dwrite d2/dba/fd9 [0,4194304] 0 2024-04-16T02:35:42.657 INFO:tasks.workunit.client.1.smithi153.stdout:3/681: dread d6/d10/d22/d4d/fbf [0,4194304] 0 2024-04-16T02:35:42.658 INFO:tasks.workunit.client.1.smithi153.stdout:3/682: chown d6/d10/d3b/d59/d7c/c89 10288 1 2024-04-16T02:35:42.659 INFO:tasks.workunit.client.1.smithi153.stdout:3/683: chown d6/fe4 1953149 1 2024-04-16T02:35:42.661 INFO:tasks.workunit.client.1.smithi153.stdout:3/684: truncate d6/d10/d22/d2e/fad 790684 0 2024-04-16T02:35:42.671 INFO:tasks.workunit.client.1.smithi153.stdout:9/764: truncate d4/d6/ddb/f78 2158631 0 2024-04-16T02:35:42.674 INFO:tasks.workunit.client.0.smithi151.stdout:5/290: link d2/d6/f8 d2/d6/d19/d30/d36/f5b 0 2024-04-16T02:35:42.675 INFO:tasks.workunit.client.0.smithi151.stdout:5/291: dread - d2/d6/d19/d30/d36/d42/f58 zero size 2024-04-16T02:35:42.682 INFO:tasks.workunit.client.1.smithi153.stdout:8/696: rmdir d2/d13 39 2024-04-16T02:35:42.682 INFO:tasks.workunit.client.1.smithi153.stdout:4/831: write d3/d9/db/de/f26 [1509828,31286] 0 2024-04-16T02:35:42.682 INFO:tasks.workunit.client.1.smithi153.stdout:7/665: link d0/d4/db/d2b/d53/cab d0/d4/ce1 0 2024-04-16T02:35:42.692 INFO:tasks.workunit.client.0.smithi151.stdout:5/292: dread d2/f16 [0,4194304] 0 2024-04-16T02:35:42.692 INFO:tasks.workunit.client.1.smithi153.stdout:2/673: dwrite d9/d2b/d18/d3f/f49 [4194304,4194304] 0 2024-04-16T02:35:42.694 INFO:tasks.workunit.client.1.smithi153.stdout:0/820: dread dd/d14/da6/da4/fbe [0,4194304] 0 2024-04-16T02:35:42.695 INFO:tasks.workunit.client.1.smithi153.stdout:6/702: sync 2024-04-16T02:35:42.695 INFO:tasks.workunit.client.1.smithi153.stdout:3/685: dwrite d6/d10/d22/d2e/d8a/fc1 [0,4194304] 0 2024-04-16T02:35:42.702 INFO:tasks.workunit.client.1.smithi153.stdout:9/765: dread d4/d6/dc/d24/f83 [0,4194304] 0 2024-04-16T02:35:42.702 INFO:tasks.workunit.client.1.smithi153.stdout:9/766: stat d4/d6/d1b/d6e 0 2024-04-16T02:35:42.706 INFO:tasks.workunit.client.1.smithi153.stdout:7/666: unlink d0/d4/db/d2b/d53/d59/c5e 0 2024-04-16T02:35:42.707 INFO:tasks.workunit.client.1.smithi153.stdout:4/832: creat d3/d9/db/d17/d3f/d6b/f126 x:0 0 0 2024-04-16T02:35:42.707 INFO:tasks.workunit.client.1.smithi153.stdout:7/667: dread - d0/d4/db/d34/d4b/fc2 zero size 2024-04-16T02:35:42.710 INFO:tasks.workunit.client.1.smithi153.stdout:0/821: readlink dd/d26/d92/lb4 0 2024-04-16T02:35:42.711 INFO:tasks.workunit.client.1.smithi153.stdout:6/703: write dc/d47/d57/d77/d15/f23 [4835323,6021] 0 2024-04-16T02:35:42.712 INFO:tasks.workunit.client.1.smithi153.stdout:1/740: rename d4/dd/d13/d26/d58/d7a to d4/dd/dfd 0 2024-04-16T02:35:42.713 INFO:tasks.workunit.client.1.smithi153.stdout:6/704: fdatasync dc/d6b/f6f 0 2024-04-16T02:35:42.714 INFO:tasks.workunit.client.1.smithi153.stdout:3/686: creat d6/de6/fea x:0 0 0 2024-04-16T02:35:42.716 INFO:tasks.workunit.client.1.smithi153.stdout:9/767: symlink d4/lf0 0 2024-04-16T02:35:42.718 INFO:tasks.workunit.client.1.smithi153.stdout:0/822: symlink dd/d14/d51/d7f/d9d/l104 0 2024-04-16T02:35:42.720 INFO:tasks.workunit.client.1.smithi153.stdout:7/668: mknod d0/d4/db/d2b/d43/dbf/ce2 0 2024-04-16T02:35:42.722 INFO:tasks.workunit.client.1.smithi153.stdout:1/741: unlink d4/dd/d13/d26/d58/fdc 0 2024-04-16T02:35:42.723 INFO:tasks.workunit.client.1.smithi153.stdout:3/687: write d6/d10/d3b/d76/f7d [265531,69740] 0 2024-04-16T02:35:42.724 INFO:tasks.workunit.client.1.smithi153.stdout:9/768: unlink d4/d33/fba 0 2024-04-16T02:35:42.727 INFO:tasks.workunit.client.1.smithi153.stdout:7/669: symlink d0/d4/db/d2b/d8c/db5/le3 0 2024-04-16T02:35:42.728 INFO:tasks.workunit.client.1.smithi153.stdout:7/670: stat d0/d4/db/d2b/d53/f27 0 2024-04-16T02:35:42.729 INFO:tasks.workunit.client.1.smithi153.stdout:0/823: symlink dd/l105 0 2024-04-16T02:35:42.730 INFO:tasks.workunit.client.1.smithi153.stdout:0/824: chown dd/d14/d19/d77 131039358 1 2024-04-16T02:35:42.731 INFO:tasks.workunit.client.1.smithi153.stdout:5/768: rename d8/df/d2e/dd1/fd7 to d8/d1f/ff2 0 2024-04-16T02:35:42.732 INFO:tasks.workunit.client.1.smithi153.stdout:9/769: unlink d4/d6/dc/d46/lc2 0 2024-04-16T02:35:42.734 INFO:tasks.workunit.client.1.smithi153.stdout:3/688: unlink d6/d10/d22/d2c/l4b 0 2024-04-16T02:35:42.734 INFO:tasks.workunit.client.1.smithi153.stdout:3/689: fsync d6/d10/d22/fb1 0 2024-04-16T02:35:42.735 INFO:tasks.workunit.client.1.smithi153.stdout:0/825: fsync dd/d14/d19/d55/fcf 0 2024-04-16T02:35:42.739 INFO:tasks.workunit.client.1.smithi153.stdout:0/826: dread dd/d14/d19/f1e [0,4194304] 0 2024-04-16T02:35:42.739 INFO:tasks.workunit.client.1.smithi153.stdout:8/697: rename d2/d93/d97/d9c/d27/d2d to d2/d92/deb 0 2024-04-16T02:35:42.739 INFO:tasks.workunit.client.1.smithi153.stdout:0/827: readlink dd/l15 0 2024-04-16T02:35:42.741 INFO:tasks.workunit.client.1.smithi153.stdout:1/742: creat d4/dd/d13/d26/ffe x:0 0 0 2024-04-16T02:35:42.742 INFO:tasks.workunit.client.1.smithi153.stdout:0/828: write dd/d32/d57/f61 [2533277,31663] 0 2024-04-16T02:35:42.746 INFO:tasks.workunit.client.1.smithi153.stdout:9/770: mknod d4/d6/dc/d24/d4e/cf1 0 2024-04-16T02:35:42.746 INFO:tasks.workunit.client.1.smithi153.stdout:2/674: dwrite d9/d24/d27/d2c/d41/f4e [0,4194304] 0 2024-04-16T02:35:42.748 INFO:tasks.workunit.client.1.smithi153.stdout:2/675: dread - d9/d2b/d1c/d20/d45/f96 zero size 2024-04-16T02:35:42.749 INFO:tasks.workunit.client.1.smithi153.stdout:7/671: symlink d0/d4/db/d2b/d43/d6d/d68/le4 0 2024-04-16T02:35:42.749 INFO:tasks.workunit.client.1.smithi153.stdout:0/829: dread dd/d14/d19/d20/f70 [0,4194304] 0 2024-04-16T02:35:42.753 INFO:tasks.workunit.client.1.smithi153.stdout:3/690: mkdir d6/d10/d22/d4d/da0/deb 0 2024-04-16T02:35:42.754 INFO:tasks.workunit.client.1.smithi153.stdout:7/672: write d0/d4/db/d34/d4b/f75 [1937355,65283] 0 2024-04-16T02:35:42.754 INFO:tasks.workunit.client.1.smithi153.stdout:5/769: rename d8/df/d84/da4/fdd to d8/df/d23/ff3 0 2024-04-16T02:35:42.757 INFO:tasks.workunit.client.0.smithi151.stdout:4/270: write d0/d17/f2b [19945,38052] 0 2024-04-16T02:35:42.760 INFO:tasks.workunit.client.0.smithi151.stdout:4/271: mkdir d0/d20/d59 0 2024-04-16T02:35:42.761 INFO:tasks.workunit.client.1.smithi153.stdout:4/833: truncate d3/d9/db/d45/df3/fa6 4908822 0 2024-04-16T02:35:42.761 INFO:tasks.workunit.client.1.smithi153.stdout:2/676: chown d9/d2b/d1c/d20/fc2 191288 1 2024-04-16T02:35:42.762 INFO:tasks.workunit.client.0.smithi151.stdout:4/272: creat d0/d11/d19/d31/d34/f5a x:0 0 0 2024-04-16T02:35:42.762 INFO:tasks.workunit.client.1.smithi153.stdout:0/830: mknod dd/d14/d51/d9e/c106 0 2024-04-16T02:35:42.763 INFO:tasks.workunit.client.1.smithi153.stdout:6/705: dwrite dc/d10/f33 [0,4194304] 0 2024-04-16T02:35:42.763 INFO:tasks.workunit.client.1.smithi153.stdout:3/691: unlink d6/d10/d22/d4d/da0/da3/fcc 0 2024-04-16T02:35:42.764 INFO:tasks.workunit.client.1.smithi153.stdout:5/770: dwrite d8/df/d28/d64/fe5 [0,4194304] 0 2024-04-16T02:35:42.764 INFO:tasks.workunit.client.0.smithi151.stdout:4/273: mknod d0/c5b 0 2024-04-16T02:35:42.765 INFO:tasks.workunit.client.0.smithi151.stdout:4/274: stat d0/l3e 0 2024-04-16T02:35:42.766 INFO:tasks.workunit.client.1.smithi153.stdout:2/677: chown d9/d24/d27/d2c/d41/f4e 3761443 1 2024-04-16T02:35:42.766 INFO:tasks.workunit.client.0.smithi151.stdout:4/275: symlink d0/d17/l5c 0 2024-04-16T02:35:42.768 INFO:tasks.workunit.client.1.smithi153.stdout:3/692: write d6/d10/d22/d4d/da0/fd8 [68119,120657] 0 2024-04-16T02:35:42.768 INFO:tasks.workunit.client.1.smithi153.stdout:7/673: creat d0/d4/db/d2b/d43/d6d/d68/d6c/db8/fe5 x:0 0 0 2024-04-16T02:35:42.769 INFO:tasks.workunit.client.0.smithi151.stdout:4/276: symlink d0/d11/d19/d46/l5d 0 2024-04-16T02:35:42.770 INFO:tasks.workunit.client.0.smithi151.stdout:4/277: unlink d0/c9 0 2024-04-16T02:35:42.771 INFO:tasks.workunit.client.0.smithi151.stdout:4/278: readlink d0/l38 0 2024-04-16T02:35:42.772 INFO:tasks.workunit.client.1.smithi153.stdout:4/834: creat d3/da5/dd2/f127 x:0 0 0 2024-04-16T02:35:42.773 INFO:tasks.workunit.client.0.smithi151.stdout:4/279: write d0/d11/d19/f25 [2684606,6163] 0 2024-04-16T02:35:42.774 INFO:tasks.workunit.client.0.smithi151.stdout:4/280: chown d0/d11/d19/d46 317349695 1 2024-04-16T02:35:42.774 INFO:tasks.workunit.client.1.smithi153.stdout:5/771: symlink d8/df/d23/lf4 0 2024-04-16T02:35:42.775 INFO:tasks.workunit.client.0.smithi151.stdout:4/281: dread - d0/d20/f49 zero size 2024-04-16T02:35:42.775 INFO:tasks.workunit.client.1.smithi153.stdout:4/835: fsync d3/d9/db/de/d2c/f56 0 2024-04-16T02:35:42.776 INFO:tasks.workunit.client.0.smithi151.stdout:4/282: chown d0/d11/d19/d31/f47 1027844 1 2024-04-16T02:35:42.776 INFO:tasks.workunit.client.1.smithi153.stdout:6/706: mknod dc/d47/d57/d77/d7a/d97/cee 0 2024-04-16T02:35:42.777 INFO:tasks.workunit.client.1.smithi153.stdout:6/707: write dc/d10/f6c [7297892,130295] 0 2024-04-16T02:35:42.778 INFO:tasks.workunit.client.0.smithi151.stdout:4/283: write d0/fe [2023899,57477] 0 2024-04-16T02:35:42.778 INFO:tasks.workunit.client.0.smithi151.stdout:5/293: truncate d2/d6/d19/d30/f39 439187 0 2024-04-16T02:35:42.779 INFO:tasks.workunit.client.1.smithi153.stdout:2/678: fdatasync d9/d2b/d5c/d5f/f61 0 2024-04-16T02:35:42.779 INFO:tasks.workunit.client.1.smithi153.stdout:4/836: fsync d3/d9/db/d17/d28/dc8/de4/dfd/de3/df0/f124 0 2024-04-16T02:35:42.780 INFO:tasks.workunit.client.0.smithi151.stdout:5/294: chown d2/d6/d19/d1a/f3b 21998348 1 2024-04-16T02:35:42.780 INFO:tasks.workunit.client.0.smithi151.stdout:4/284: write d0/d11/d19/d31/d34/f5a [336355,9023] 0 2024-04-16T02:35:42.780 INFO:tasks.workunit.client.0.smithi151.stdout:4/285: readlink d0/l4 0 2024-04-16T02:35:42.782 INFO:tasks.workunit.client.1.smithi153.stdout:1/743: rename d4/dd/d18/d29/c38 to d4/dd/d13/d7e/cff 0 2024-04-16T02:35:42.783 INFO:tasks.workunit.client.0.smithi151.stdout:5/295: creat d2/d6/d19/d30/d49/f5c x:0 0 0 2024-04-16T02:35:42.784 INFO:tasks.workunit.client.1.smithi153.stdout:8/698: rmdir d2/d92/deb/d47/d60/d67/d86/da4/ddb 0 2024-04-16T02:35:42.785 INFO:tasks.workunit.client.1.smithi153.stdout:8/699: fsync d2/d92/deb/d47/d77/fe2 0 2024-04-16T02:35:42.785 INFO:tasks.workunit.client.0.smithi151.stdout:5/296: creat d2/d6/d19/d30/d36/d42/f5d x:0 0 0 2024-04-16T02:35:42.786 INFO:tasks.workunit.client.1.smithi153.stdout:5/772: symlink d8/df/d2e/lf5 0 2024-04-16T02:35:42.787 INFO:tasks.workunit.client.0.smithi151.stdout:5/297: mkdir d2/d5e 0 2024-04-16T02:35:42.790 INFO:tasks.workunit.client.0.smithi151.stdout:5/298: creat d2/d6/d19/f5f x:0 0 0 2024-04-16T02:35:42.790 INFO:tasks.workunit.client.1.smithi153.stdout:6/708: unlink dc/d10/l5f 0 2024-04-16T02:35:42.792 INFO:tasks.workunit.client.1.smithi153.stdout:2/679: unlink d9/d24/d27/d9a/la4 0 2024-04-16T02:35:42.793 INFO:tasks.workunit.client.1.smithi153.stdout:4/837: creat d3/d9/d24/d75/d9e/f128 x:0 0 0 2024-04-16T02:35:42.793 INFO:tasks.workunit.client.1.smithi153.stdout:6/709: dread - dc/d47/d57/d77/dc4/dc9/fca zero size 2024-04-16T02:35:42.794 INFO:tasks.workunit.client.1.smithi153.stdout:3/693: mkdir d6/d10/dec 0 2024-04-16T02:35:42.795 INFO:tasks.workunit.client.1.smithi153.stdout:5/773: dwrite f4 [4194304,4194304] 0 2024-04-16T02:35:42.797 INFO:tasks.workunit.client.0.smithi151.stdout:5/299: dwrite d2/d6/f50 [0,4194304] 0 2024-04-16T02:35:42.797 INFO:tasks.workunit.client.1.smithi153.stdout:1/744: mknod d4/da2/d65/d82/c100 0 2024-04-16T02:35:42.797 INFO:tasks.workunit.client.1.smithi153.stdout:4/838: truncate d3/d9/db/de/f10a 369835 0 2024-04-16T02:35:42.799 INFO:tasks.workunit.client.1.smithi153.stdout:4/839: fsync d3/d9/d24/d75/dc5/ff2 0 2024-04-16T02:35:42.799 INFO:tasks.workunit.client.1.smithi153.stdout:8/700: dread - d2/d92/deb/d47/d60/f7b zero size 2024-04-16T02:35:42.799 INFO:tasks.workunit.client.1.smithi153.stdout:5/774: truncate d8/df/d23/f56 4605399 0 2024-04-16T02:35:42.799 INFO:tasks.workunit.client.0.smithi151.stdout:5/300: truncate d2/d6/d19/d1a/f52 806729 0 2024-04-16T02:35:42.800 INFO:tasks.workunit.client.1.smithi153.stdout:8/701: chown d2/d93/d97/d9c/f3c 684 1 2024-04-16T02:35:42.800 INFO:tasks.workunit.client.0.smithi151.stdout:5/301: write d2/d6/d19/d1a/f40 [91745,49792] 0 2024-04-16T02:35:42.801 INFO:tasks.workunit.client.0.smithi151.stdout:5/302: stat d2/f24 0 2024-04-16T02:35:42.801 INFO:tasks.workunit.client.1.smithi153.stdout:4/840: truncate d3/d9/d24/d3e/f117 201085 0 2024-04-16T02:35:42.801 INFO:tasks.workunit.client.1.smithi153.stdout:8/702: write d2/f12 [2709489,30884] 0 2024-04-16T02:35:42.806 INFO:tasks.workunit.client.1.smithi153.stdout:2/680: creat d9/d2b/d1c/fe2 x:0 0 0 2024-04-16T02:35:42.806 INFO:tasks.workunit.client.1.smithi153.stdout:6/710: unlink dc/d10/f33 0 2024-04-16T02:35:42.806 INFO:tasks.workunit.client.0.smithi151.stdout:5/303: mknod d2/d6/d19/d30/d36/c60 0 2024-04-16T02:35:42.808 INFO:tasks.workunit.client.1.smithi153.stdout:1/745: creat d4/dd/d13/d8d/f101 x:0 0 0 2024-04-16T02:35:42.811 INFO:tasks.workunit.client.0.smithi151.stdout:5/304: mknod d2/d6/d19/d30/c61 0 2024-04-16T02:35:42.813 INFO:tasks.workunit.client.1.smithi153.stdout:6/711: dwrite dc/d47/d93/faa [0,4194304] 0 2024-04-16T02:35:42.813 INFO:tasks.workunit.client.1.smithi153.stdout:4/841: creat d3/d9/d24/d75/dc5/f129 x:0 0 0 2024-04-16T02:35:42.814 INFO:tasks.workunit.client.1.smithi153.stdout:8/703: mkdir d2/d93/d97/d9c/d27/d5e/d63/dec 0 2024-04-16T02:35:42.814 INFO:tasks.workunit.client.1.smithi153.stdout:4/842: read d3/f6d [254565,30612] 0 2024-04-16T02:35:42.819 INFO:tasks.workunit.client.1.smithi153.stdout:3/694: mkdir d6/d10/dec/ded 0 2024-04-16T02:35:42.821 INFO:tasks.workunit.client.1.smithi153.stdout:0/831: rename dd/d26/d92/ffa to dd/d14/f107 0 2024-04-16T02:35:42.821 INFO:tasks.workunit.client.1.smithi153.stdout:4/843: read d3/d9/db/d45/df3/fc4 [504181,14719] 0 2024-04-16T02:35:42.823 INFO:tasks.workunit.client.1.smithi153.stdout:5/775: mknod d8/df/d28/cf6 0 2024-04-16T02:35:42.828 INFO:tasks.workunit.client.1.smithi153.stdout:3/695: rmdir d6/d10/d22/d2c 39 2024-04-16T02:35:42.831 INFO:tasks.workunit.client.1.smithi153.stdout:8/704: rename d2/d93/d97/cc0 to d2/d92/deb/d47/d60/d67/d86/ced 0 2024-04-16T02:35:42.832 INFO:tasks.workunit.client.1.smithi153.stdout:0/832: fsync dd/d14/d19/d20/d4a/f78 0 2024-04-16T02:35:42.833 INFO:tasks.workunit.client.1.smithi153.stdout:6/712: fsync dc/d1a/f68 0 2024-04-16T02:35:42.837 INFO:tasks.workunit.client.1.smithi153.stdout:8/705: mkdir d2/d93/dee 0 2024-04-16T02:35:42.838 INFO:tasks.workunit.client.1.smithi153.stdout:5/776: symlink d8/lf7 0 2024-04-16T02:35:42.839 INFO:tasks.workunit.client.1.smithi153.stdout:0/833: mknod dd/d14/da6/da4/db1/c108 0 2024-04-16T02:35:42.840 INFO:tasks.workunit.client.1.smithi153.stdout:2/681: dwrite d9/d24/d27/d9a/f9f [0,4194304] 0 2024-04-16T02:35:42.841 INFO:tasks.workunit.client.1.smithi153.stdout:6/713: chown dc/d47/d57/d77/d15/f86 15664753 1 2024-04-16T02:35:42.842 INFO:tasks.workunit.client.1.smithi153.stdout:1/746: getdents d4/dd/d20 0 2024-04-16T02:35:42.843 INFO:tasks.workunit.client.1.smithi153.stdout:6/714: read f6 [3859585,53950] 0 2024-04-16T02:35:42.846 INFO:tasks.workunit.client.1.smithi153.stdout:8/706: unlink d2/d92/deb/d47/d60/f79 0 2024-04-16T02:35:42.847 INFO:tasks.workunit.client.1.smithi153.stdout:5/777: creat d8/df/d28/d5a/d72/dc0/ff8 x:0 0 0 2024-04-16T02:35:42.848 INFO:tasks.workunit.client.1.smithi153.stdout:0/834: truncate dd/d32/d3f/f62 1115426 0 2024-04-16T02:35:42.850 INFO:tasks.workunit.client.1.smithi153.stdout:3/696: link d6/d10/d22/d2c/f63 d6/d10/dec/ded/fee 0 2024-04-16T02:35:42.854 INFO:tasks.workunit.client.1.smithi153.stdout:0/835: truncate dd/d32/d80/f93 164975 0 2024-04-16T02:35:42.855 INFO:tasks.workunit.client.1.smithi153.stdout:3/697: write d6/d10/dec/ded/fee [2524359,94015] 0 2024-04-16T02:35:42.857 INFO:tasks.workunit.client.1.smithi153.stdout:1/747: creat d4/da2/d65/f102 x:0 0 0 2024-04-16T02:35:42.860 INFO:tasks.workunit.client.1.smithi153.stdout:8/707: rename d2/d13/d3a/d5b/fd8 to d2/d1c/fef 0 2024-04-16T02:35:42.862 INFO:tasks.workunit.client.1.smithi153.stdout:0/836: dwrite dd/d32/d57/d6c/fbb [0,4194304] 0 2024-04-16T02:35:42.870 INFO:tasks.workunit.client.1.smithi153.stdout:3/698: getdents d6/d10/d22/d2e 0 2024-04-16T02:35:42.872 INFO:tasks.workunit.client.1.smithi153.stdout:0/837: link dd/d32/cd7 dd/d14/c109 0 2024-04-16T02:35:42.873 INFO:tasks.workunit.client.1.smithi153.stdout:3/699: dread d6/f97 [0,4194304] 0 2024-04-16T02:35:42.873 INFO:tasks.workunit.client.1.smithi153.stdout:6/715: dwrite dc/f11 [0,4194304] 0 2024-04-16T02:35:42.874 INFO:tasks.workunit.client.1.smithi153.stdout:3/700: stat d6/f11 0 2024-04-16T02:35:42.877 INFO:tasks.workunit.client.1.smithi153.stdout:6/716: mknod dc/d47/d57/d77/d7a/d97/cef 0 2024-04-16T02:35:42.878 INFO:tasks.workunit.client.1.smithi153.stdout:3/701: mkdir d6/d10/d22/d4d/da0/da3/def 0 2024-04-16T02:35:42.880 INFO:tasks.workunit.client.1.smithi153.stdout:6/717: creat dc/d6b/dba/ff0 x:0 0 0 2024-04-16T02:35:42.881 INFO:tasks.workunit.client.1.smithi153.stdout:1/748: dread d4/dd/d18/d29/fde [0,4194304] 0 2024-04-16T02:35:42.881 INFO:tasks.workunit.client.1.smithi153.stdout:0/838: write dd/d14/d19/d20/f70 [766315,93717] 0 2024-04-16T02:35:42.881 INFO:tasks.workunit.client.1.smithi153.stdout:6/718: stat dc/d47/d80 0 2024-04-16T02:35:42.883 INFO:tasks.workunit.client.1.smithi153.stdout:1/749: stat d4/f5e 0 2024-04-16T02:35:42.891 INFO:tasks.workunit.client.1.smithi153.stdout:3/702: rename d6/d10/d22/d4d/da0/da3 to d6/d10/d3b/d59/d7c/de1/df0 0 2024-04-16T02:35:42.892 INFO:tasks.workunit.client.1.smithi153.stdout:3/703: truncate d6/d10/d22/d2e/fad 894588 0 2024-04-16T02:35:42.892 INFO:tasks.workunit.client.1.smithi153.stdout:6/719: rename dc/d6b/lb5 to dc/d47/d57/d77/db8/lf1 0 2024-04-16T02:35:42.893 INFO:tasks.workunit.client.1.smithi153.stdout:3/704: stat d6/c17 0 2024-04-16T02:35:42.893 INFO:tasks.workunit.client.1.smithi153.stdout:6/720: write dc/d10/fe9 [447601,93464] 0 2024-04-16T02:35:42.894 INFO:tasks.workunit.client.1.smithi153.stdout:0/839: rename dd/d14/d19/d55/c7c to dd/d14/d19/d77/d87/dda/c10a 0 2024-04-16T02:35:42.902 INFO:tasks.workunit.client.1.smithi153.stdout:6/721: dwrite dc/d47/d57/fe3 [0,4194304] 0 2024-04-16T02:35:42.903 INFO:tasks.workunit.client.1.smithi153.stdout:0/840: dwrite dd/d14/d19/d77/ff1 [0,4194304] 0 2024-04-16T02:35:42.905 INFO:tasks.workunit.client.1.smithi153.stdout:6/722: rmdir dc/d47/d93 39 2024-04-16T02:35:42.906 INFO:tasks.workunit.client.1.smithi153.stdout:0/841: symlink dd/d14/d19/d77/de3/l10b 0 2024-04-16T02:35:42.907 INFO:tasks.workunit.client.1.smithi153.stdout:9/771: sync 2024-04-16T02:35:42.912 INFO:tasks.workunit.client.1.smithi153.stdout:3/705: dread d6/d10/d22/d2c/f63 [4194304,4194304] 0 2024-04-16T02:35:42.929 INFO:tasks.workunit.client.1.smithi153.stdout:3/706: dread d6/d10/d5b/f75 [0,4194304] 0 2024-04-16T02:35:43.019 INFO:tasks.workunit.client.1.smithi153.stdout:0/842: rmdir dd/d14/d51/d7f/d9d 39 2024-04-16T02:35:43.022 INFO:tasks.workunit.client.1.smithi153.stdout:7/674: sync 2024-04-16T02:35:43.026 INFO:tasks.workunit.client.1.smithi153.stdout:7/675: write d0/d13/f4f [5789916,112728] 0 2024-04-16T02:35:43.026 INFO:tasks.workunit.client.1.smithi153.stdout:3/707: symlink d6/d10/d3b/d59/d7c/db7/lf1 0 2024-04-16T02:35:43.027 INFO:tasks.workunit.client.1.smithi153.stdout:6/723: dwrite dc/d1a/d2d/fc7 [0,4194304] 0 2024-04-16T02:35:43.032 INFO:tasks.workunit.client.1.smithi153.stdout:7/676: truncate d0/d4/db/d2b/d43/d6d/f63 280164 0 2024-04-16T02:35:43.034 INFO:tasks.workunit.client.1.smithi153.stdout:9/772: truncate d4/d6/dc/d24/d4e/fd8 319860 0 2024-04-16T02:35:43.035 INFO:tasks.workunit.client.1.smithi153.stdout:9/773: dread - d4/d6/dc/d2d/d64/fdd zero size 2024-04-16T02:35:43.037 INFO:tasks.workunit.client.1.smithi153.stdout:6/724: dread dc/d47/d57/d77/d15/fc2 [0,4194304] 0 2024-04-16T02:35:43.037 INFO:tasks.workunit.client.1.smithi153.stdout:9/774: creat d4/d6/d1b/d2a/d99/ff2 x:0 0 0 2024-04-16T02:35:43.039 INFO:tasks.workunit.client.1.smithi153.stdout:6/725: rmdir dc/d47/d7d 39 2024-04-16T02:35:43.040 INFO:tasks.workunit.client.1.smithi153.stdout:7/677: link d0/d4/db/d2b/f40 d0/d4/db/d2b/d53/d59/dc3/fe6 0 2024-04-16T02:35:43.041 INFO:tasks.workunit.client.1.smithi153.stdout:7/678: chown d0 650243 1 2024-04-16T02:35:43.041 INFO:tasks.workunit.client.1.smithi153.stdout:9/775: mkdir d4/d6/def/df3 0 2024-04-16T02:35:43.042 INFO:tasks.workunit.client.1.smithi153.stdout:6/726: unlink dc/d47/d57/d77/d7a/ldf 0 2024-04-16T02:35:43.043 INFO:tasks.workunit.client.1.smithi153.stdout:7/679: chown d0/d4/db/d2b/cad 20271 1 2024-04-16T02:35:43.044 INFO:tasks.workunit.client.1.smithi153.stdout:9/776: mkdir d4/d6/dc/d24/d4e/df4 0 2024-04-16T02:35:43.044 INFO:tasks.workunit.client.1.smithi153.stdout:3/708: write d6/f2d [2360796,97050] 0 2024-04-16T02:35:43.045 INFO:tasks.workunit.client.1.smithi153.stdout:9/777: stat d4/d6/dc/d24/dc6/f85 0 2024-04-16T02:35:43.047 INFO:tasks.workunit.client.1.smithi153.stdout:7/680: mkdir d0/d4/db/d2b/d43/d6d/de7 0 2024-04-16T02:35:43.048 INFO:tasks.workunit.client.0.smithi151.stdout:6/327: sync 2024-04-16T02:35:43.050 INFO:tasks.workunit.client.0.smithi151.stdout:6/328: chown d2/d5/df/d23/d31/f43 1258920 1 2024-04-16T02:35:43.050 INFO:tasks.workunit.client.1.smithi153.stdout:3/709: dread d6/d10/d3b/d59/fa9 [0,4194304] 0 2024-04-16T02:35:43.053 INFO:tasks.workunit.client.1.smithi153.stdout:9/778: dwrite d4/d6/dc/fd7 [0,4194304] 0 2024-04-16T02:35:43.053 INFO:tasks.workunit.client.0.smithi151.stdout:6/329: mkdir d2/d1f/d22/d6d 0 2024-04-16T02:35:43.055 INFO:tasks.workunit.client.1.smithi153.stdout:7/681: mkdir d0/de8 0 2024-04-16T02:35:43.056 INFO:tasks.workunit.client.1.smithi153.stdout:3/710: fdatasync d6/d10/d22/d2c/f63 0 2024-04-16T02:35:43.058 INFO:tasks.workunit.client.1.smithi153.stdout:6/727: truncate dc/d47/d93/faa 3217376 0 2024-04-16T02:35:43.060 INFO:tasks.workunit.client.1.smithi153.stdout:7/682: fdatasync d0/d4/f20 0 2024-04-16T02:35:43.062 INFO:tasks.workunit.client.1.smithi153.stdout:2/682: sync 2024-04-16T02:35:43.062 INFO:tasks.workunit.client.1.smithi153.stdout:1/750: sync 2024-04-16T02:35:43.062 INFO:tasks.workunit.client.1.smithi153.stdout:4/844: sync 2024-04-16T02:35:43.062 INFO:tasks.workunit.client.1.smithi153.stdout:8/708: sync 2024-04-16T02:35:43.062 INFO:tasks.workunit.client.1.smithi153.stdout:5/778: sync 2024-04-16T02:35:43.062 INFO:tasks.workunit.client.1.smithi153.stdout:8/709: stat d2/dba 0 2024-04-16T02:35:43.064 INFO:tasks.workunit.client.1.smithi153.stdout:9/779: creat d4/d33/ff5 x:0 0 0 2024-04-16T02:35:43.066 INFO:tasks.workunit.client.1.smithi153.stdout:7/683: mknod d0/d4/db/d2b/d43/d6d/d68/d6c/ce9 0 2024-04-16T02:35:43.068 INFO:tasks.workunit.client.1.smithi153.stdout:2/683: unlink d9/d24/d73/la6 0 2024-04-16T02:35:43.068 INFO:tasks.workunit.client.1.smithi153.stdout:7/684: dread - d0/d13/fbc zero size 2024-04-16T02:35:43.069 INFO:tasks.workunit.client.1.smithi153.stdout:9/780: fsync d4/d6/d1b/f71 0 2024-04-16T02:35:43.071 INFO:tasks.workunit.client.1.smithi153.stdout:3/711: getdents d6/d10/d3b 0 2024-04-16T02:35:43.071 INFO:tasks.workunit.client.1.smithi153.stdout:6/728: rename dc/d47/d57/d77/d15/c29 to dc/d1a/cf2 0 2024-04-16T02:35:43.073 INFO:tasks.workunit.client.1.smithi153.stdout:9/781: symlink d4/d6/def/lf6 0 2024-04-16T02:35:43.073 INFO:tasks.workunit.client.1.smithi153.stdout:7/685: stat d0/d4/db/d2b/d53/d59/d76/dc8/fb4 0 2024-04-16T02:35:43.073 INFO:tasks.workunit.client.1.smithi153.stdout:8/710: dwrite d2/d92/deb/d47/fd1 [0,4194304] 0 2024-04-16T02:35:43.077 INFO:tasks.workunit.client.1.smithi153.stdout:5/779: rename d8/df/d2e/lbc to d8/df/d2e/dd1/lf9 0 2024-04-16T02:35:43.078 INFO:tasks.workunit.client.1.smithi153.stdout:3/712: mknod d6/d10/d3b/d59/d7c/de1/cf2 0 2024-04-16T02:35:43.081 INFO:tasks.workunit.client.1.smithi153.stdout:5/780: rmdir d8/df/d28 39 2024-04-16T02:35:43.085 INFO:tasks.workunit.client.1.smithi153.stdout:1/751: rename d4/dd/d18/d29/d6e/c74 to d4/da2/d65/d84/c103 0 2024-04-16T02:35:43.085 INFO:tasks.workunit.client.1.smithi153.stdout:6/729: dwrite dc/d47/d93/fb4 [0,4194304] 0 2024-04-16T02:35:43.085 INFO:tasks.workunit.client.1.smithi153.stdout:3/713: mknod d6/d10/d3b/d59/cf3 0 2024-04-16T02:35:43.088 INFO:tasks.workunit.client.1.smithi153.stdout:2/684: rename d9/d24/d34/d60/d7c/c7e to d9/d24/d34/d99/ce3 0 2024-04-16T02:35:43.091 INFO:tasks.workunit.client.1.smithi153.stdout:3/714: stat d6/d10/d22/d2c/f33 0 2024-04-16T02:35:43.094 INFO:tasks.workunit.client.1.smithi153.stdout:7/686: dwrite d0/d4/db/d34/d4b/fc2 [0,4194304] 0 2024-04-16T02:35:43.094 INFO:tasks.workunit.client.1.smithi153.stdout:6/730: symlink dc/d47/d57/d77/d15/lf3 0 2024-04-16T02:35:43.103 INFO:tasks.workunit.client.1.smithi153.stdout:1/752: dwrite d4/dd/d18/d6c/fba [0,4194304] 0 2024-04-16T02:35:43.103 INFO:tasks.workunit.client.1.smithi153.stdout:8/711: truncate d2/d92/deb/d47/d60/f71 2126344 0 2024-04-16T02:35:43.110 INFO:tasks.workunit.client.1.smithi153.stdout:7/687: mkdir d0/d4/db/d87/dc0/dea 0 2024-04-16T02:35:43.110 INFO:tasks.workunit.client.1.smithi153.stdout:1/753: readlink d4/dd/d18/d29/l6a 0 2024-04-16T02:35:43.110 INFO:tasks.workunit.client.1.smithi153.stdout:3/715: dwrite d6/f11 [0,4194304] 0 2024-04-16T02:35:43.111 INFO:tasks.workunit.client.1.smithi153.stdout:9/782: dwrite d4/d6/dc/d7f/fe9 [0,4194304] 0 2024-04-16T02:35:43.113 INFO:tasks.workunit.client.1.smithi153.stdout:5/781: truncate d8/df/d28/d5a/f68 49284 0 2024-04-16T02:35:43.113 INFO:tasks.workunit.client.1.smithi153.stdout:8/712: fdatasync d2/d92/deb/d47/d60/d67/d86/fe9 0 2024-04-16T02:35:43.113 INFO:tasks.workunit.client.1.smithi153.stdout:7/688: chown d0/lb2 115 1 2024-04-16T02:35:43.115 INFO:tasks.workunit.client.0.smithi151.stdout:6/330: dread d2/d1f/f29 [8388608,4194304] 0 2024-04-16T02:35:43.115 INFO:tasks.workunit.client.1.smithi153.stdout:5/782: fsync d8/df/d28/d5a/d66/f7b 0 2024-04-16T02:35:43.116 INFO:tasks.workunit.client.0.smithi151.stdout:6/331: readlink d2/d5/la 0 2024-04-16T02:35:43.117 INFO:tasks.workunit.client.0.smithi151.stdout:6/332: dread - d2/d13/d3a/f54 zero size 2024-04-16T02:35:43.119 INFO:tasks.workunit.client.1.smithi153.stdout:1/754: symlink d4/dd/d13/d23/l104 0 2024-04-16T02:35:43.119 INFO:tasks.workunit.client.1.smithi153.stdout:5/783: write d8/df/f11 [439841,72909] 0 2024-04-16T02:35:43.121 INFO:tasks.workunit.client.0.smithi151.stdout:6/333: rename d2/f30 to d2/d5/df/d20/d4f/f6e 0 2024-04-16T02:35:43.122 INFO:tasks.workunit.client.0.smithi151.stdout:6/334: chown d2/d13/d3a/l4c 26432468 1 2024-04-16T02:35:43.122 INFO:tasks.workunit.client.0.smithi151.stdout:6/335: chown d2/d5/l1e 106782 1 2024-04-16T02:35:43.124 INFO:tasks.workunit.client.0.smithi151.stdout:6/336: fsync d2/d1f/f29 0 2024-04-16T02:35:43.124 INFO:tasks.workunit.client.1.smithi153.stdout:8/713: unlink d2/d13/d3a/f46 0 2024-04-16T02:35:43.127 INFO:tasks.workunit.client.1.smithi153.stdout:3/716: symlink d6/d10/lf4 0 2024-04-16T02:35:43.127 INFO:tasks.workunit.client.0.smithi151.stdout:6/337: mkdir d2/d13/d6f 0 2024-04-16T02:35:43.128 INFO:tasks.workunit.client.1.smithi153.stdout:6/731: link dc/d1a/l4c dc/d47/d57/d77/d7a/d97/db7/lf4 0 2024-04-16T02:35:43.128 INFO:tasks.workunit.client.1.smithi153.stdout:2/685: dread d9/f9e [0,4194304] 0 2024-04-16T02:35:43.128 INFO:tasks.workunit.client.1.smithi153.stdout:9/783: creat d4/d6/dc/d24/d4e/df4/ff7 x:0 0 0 2024-04-16T02:35:43.129 INFO:tasks.workunit.client.1.smithi153.stdout:2/686: write d9/d2b/d5c/fcf [470143,70184] 0 2024-04-16T02:35:43.131 INFO:tasks.workunit.client.0.smithi151.stdout:6/338: rename d2/d5/df/d23/d49 to d2/d13/d70 0 2024-04-16T02:35:43.131 INFO:tasks.workunit.client.1.smithi153.stdout:5/784: rename d8/d41/dbd/d83/fdb to d8/d41/d90/ded/ffa 0 2024-04-16T02:35:43.133 INFO:tasks.workunit.client.1.smithi153.stdout:7/689: rmdir d0/d4/db/d87/dc0/dc6 0 2024-04-16T02:35:43.133 INFO:tasks.workunit.client.0.smithi151.stdout:6/339: getdents d2/d13/d70 0 2024-04-16T02:35:43.135 INFO:tasks.workunit.client.1.smithi153.stdout:6/732: mkdir dc/d47/d57/d77/d7a/d97/db7/df5 0 2024-04-16T02:35:43.135 INFO:tasks.workunit.client.1.smithi153.stdout:9/784: dwrite d4/f48 [4194304,4194304] 0 2024-04-16T02:35:43.136 INFO:tasks.workunit.client.1.smithi153.stdout:9/785: chown d4/d6/d1b/ddf/le4 2230868 1 2024-04-16T02:35:43.137 INFO:tasks.workunit.client.1.smithi153.stdout:8/714: mknod d2/d93/dee/cf0 0 2024-04-16T02:35:43.137 INFO:tasks.workunit.client.1.smithi153.stdout:9/786: chown d4/d6/dc/d24/f9f 65 1 2024-04-16T02:35:43.137 INFO:tasks.workunit.client.1.smithi153.stdout:8/715: stat d2/l56 0 2024-04-16T02:35:43.138 INFO:tasks.workunit.client.1.smithi153.stdout:8/716: chown d2/de 126851038 1 2024-04-16T02:35:43.140 INFO:tasks.workunit.client.1.smithi153.stdout:7/690: fsync d0/d4/db/d2b/f89 0 2024-04-16T02:35:43.140 INFO:tasks.workunit.client.1.smithi153.stdout:0/843: sync 2024-04-16T02:35:43.141 INFO:tasks.workunit.client.1.smithi153.stdout:6/733: symlink dc/d59/d63/lf6 0 2024-04-16T02:35:43.143 INFO:tasks.workunit.client.1.smithi153.stdout:9/787: rename d4/d6/d1b/d2a/f70 to d4/d6/d1b/d6e/ff8 0 2024-04-16T02:35:43.144 INFO:tasks.workunit.client.1.smithi153.stdout:2/687: creat d9/d2b/d18/fe4 x:0 0 0 2024-04-16T02:35:43.145 INFO:tasks.workunit.client.1.smithi153.stdout:7/691: dread - d0/d4/db/d2b/d43/d6d/d68/d6c/db8/fe5 zero size 2024-04-16T02:35:43.145 INFO:tasks.workunit.client.1.smithi153.stdout:0/844: mknod dd/d14/d51/d54/c10c 0 2024-04-16T02:35:43.145 INFO:tasks.workunit.client.1.smithi153.stdout:9/788: read - d4/d6/dc/fd9 zero size 2024-04-16T02:35:43.146 INFO:tasks.workunit.client.1.smithi153.stdout:6/734: creat dc/d47/d57/d77/dc4/ff7 x:0 0 0 2024-04-16T02:35:43.147 INFO:tasks.workunit.client.1.smithi153.stdout:2/688: mkdir d9/d24/d27/d2c/d5d/de5 0 2024-04-16T02:35:43.148 INFO:tasks.workunit.client.1.smithi153.stdout:7/692: write d0/d4/db/d34/d51/fde [530834,129415] 0 2024-04-16T02:35:43.150 INFO:tasks.workunit.client.1.smithi153.stdout:6/735: write dc/d47/d57/d77/d15/dac/dae/fb0 [5178064,2249] 0 2024-04-16T02:35:43.151 INFO:tasks.workunit.client.1.smithi153.stdout:9/789: rename d4/d6/ddb/d4a/dea to d4/df9 0 2024-04-16T02:35:43.153 INFO:tasks.workunit.client.1.smithi153.stdout:7/693: write d0/d4/db/d2b/d8c/f9b [43521,65872] 0 2024-04-16T02:35:43.156 INFO:tasks.workunit.client.1.smithi153.stdout:7/694: chown d0/d4/db/d34/c56 1201 1 2024-04-16T02:35:43.156 INFO:tasks.workunit.client.1.smithi153.stdout:0/845: dwrite dd/d26/d44/fc6 [4194304,4194304] 0 2024-04-16T02:35:43.156 INFO:tasks.workunit.client.1.smithi153.stdout:6/736: chown dc/d1a/d2d/c3b 245 1 2024-04-16T02:35:43.157 INFO:tasks.workunit.client.1.smithi153.stdout:9/790: creat d4/d33/dae/ffa x:0 0 0 2024-04-16T02:35:43.158 INFO:tasks.workunit.client.1.smithi153.stdout:1/755: write d4/dd/d13/d23/f6d [643092,61366] 0 2024-04-16T02:35:43.158 INFO:tasks.workunit.client.1.smithi153.stdout:0/846: readlink dd/l15 0 2024-04-16T02:35:43.159 INFO:tasks.workunit.client.1.smithi153.stdout:0/847: fsync dd/f53 0 2024-04-16T02:35:43.160 INFO:tasks.workunit.client.1.smithi153.stdout:2/689: rename c5 to d9/d2b/d5c/ce6 0 2024-04-16T02:35:43.160 INFO:tasks.workunit.client.1.smithi153.stdout:9/791: chown d4/d6/dc/d2d/da0/d69/fb7 89020 1 2024-04-16T02:35:43.161 INFO:tasks.workunit.client.1.smithi153.stdout:1/756: write d4/dd/d13/d23/f5d [1465342,3417] 0 2024-04-16T02:35:43.162 INFO:tasks.workunit.client.1.smithi153.stdout:9/792: fdatasync d4/d33/dae/dc7/fd3 0 2024-04-16T02:35:43.164 INFO:tasks.workunit.client.0.smithi151.stdout:6/340: dread d2/d5/f27 [0,4194304] 0 2024-04-16T02:35:43.166 INFO:tasks.workunit.client.1.smithi153.stdout:6/737: creat dc/d47/d57/d77/dc4/dc9/ff8 x:0 0 0 2024-04-16T02:35:43.168 INFO:tasks.workunit.client.0.smithi151.stdout:6/341: getdents d2/d5/df/d23/d45/d62 0 2024-04-16T02:35:43.169 INFO:tasks.workunit.client.1.smithi153.stdout:3/717: dread d6/d10/d22/f31 [0,4194304] 0 2024-04-16T02:35:43.170 INFO:tasks.workunit.client.1.smithi153.stdout:5/785: write d8/df/d28/f8a [1412827,40092] 0 2024-04-16T02:35:43.171 INFO:tasks.workunit.client.1.smithi153.stdout:3/718: write d6/d10/d3b/fc5 [51642,125189] 0 2024-04-16T02:35:43.171 INFO:tasks.workunit.client.1.smithi153.stdout:4/845: sync 2024-04-16T02:35:43.173 INFO:tasks.workunit.client.1.smithi153.stdout:8/717: write d2/de/d3f/f75 [808338,113960] 0 2024-04-16T02:35:43.175 INFO:tasks.workunit.client.1.smithi153.stdout:6/738: symlink dc/d59/lf9 0 2024-04-16T02:35:43.175 INFO:tasks.workunit.client.0.smithi151.stdout:6/342: dwrite d2/d5/f60 [0,4194304] 0 2024-04-16T02:35:43.178 INFO:tasks.workunit.client.0.smithi151.stdout:7/189: sync 2024-04-16T02:35:43.178 INFO:tasks.workunit.client.0.smithi151.stdout:2/298: sync 2024-04-16T02:35:43.178 INFO:tasks.workunit.client.0.smithi151.stdout:0/222: sync 2024-04-16T02:35:43.183 INFO:tasks.workunit.client.0.smithi151.stdout:7/190: mknod d0/c34 0 2024-04-16T02:35:43.183 INFO:tasks.workunit.client.1.smithi153.stdout:2/690: truncate d9/d2b/d5c/f75 1031956 0 2024-04-16T02:35:43.183 INFO:tasks.workunit.client.1.smithi153.stdout:1/757: dread d4/dd/fbe [0,4194304] 0 2024-04-16T02:35:43.184 INFO:tasks.workunit.client.0.smithi151.stdout:6/343: dwrite d2/d5/f6 [0,4194304] 0 2024-04-16T02:35:43.184 INFO:tasks.workunit.client.0.smithi151.stdout:7/191: write d0/d1f/d31/f30 [813192,41893] 0 2024-04-16T02:35:43.185 INFO:tasks.workunit.client.1.smithi153.stdout:2/691: write d9/d2b/d1c/fd5 [923314,68497] 0 2024-04-16T02:35:43.187 INFO:tasks.workunit.client.0.smithi151.stdout:2/299: creat de/f5c x:0 0 0 2024-04-16T02:35:43.188 INFO:tasks.workunit.client.0.smithi151.stdout:6/344: creat d2/f71 x:0 0 0 2024-04-16T02:35:43.188 INFO:tasks.workunit.client.1.smithi153.stdout:4/846: stat d3/d9/db/de/d2c/d38/l89 0 2024-04-16T02:35:43.189 INFO:tasks.workunit.client.1.smithi153.stdout:5/786: dwrite d8/d1f/feb [0,4194304] 0 2024-04-16T02:35:43.189 INFO:tasks.workunit.client.0.smithi151.stdout:2/300: truncate de/d11/d41/f33 787045 0 2024-04-16T02:35:43.191 INFO:tasks.workunit.client.1.smithi153.stdout:1/758: fdatasync d4/dd/dfd/f83 0 2024-04-16T02:35:43.192 INFO:tasks.workunit.client.1.smithi153.stdout:8/718: mkdir d2/d92/deb/d47/d60/d67/db7/df1 0 2024-04-16T02:35:43.193 INFO:tasks.workunit.client.1.smithi153.stdout:5/787: stat d8/df/d2f/f60 0 2024-04-16T02:35:43.195 INFO:tasks.workunit.client.1.smithi153.stdout:6/739: creat dc/d47/d57/d77/db8/ffa x:0 0 0 2024-04-16T02:35:43.195 INFO:tasks.workunit.client.0.smithi151.stdout:2/301: rename de/c20 to de/d11/d42/c5d 0 2024-04-16T02:35:43.196 INFO:tasks.workunit.client.0.smithi151.stdout:2/302: fsync de/d11/d42/f52 0 2024-04-16T02:35:43.198 INFO:tasks.workunit.client.1.smithi153.stdout:8/719: read d2/de/f24 [1497306,16132] 0 2024-04-16T02:35:43.199 INFO:tasks.workunit.client.0.smithi151.stdout:0/223: write dc/f37 [633414,84112] 0 2024-04-16T02:35:43.200 INFO:tasks.workunit.client.1.smithi153.stdout:2/692: unlink d9/d2b/f2a 0 2024-04-16T02:35:43.200 INFO:tasks.workunit.client.0.smithi151.stdout:2/303: chown de/c22 100684 1 2024-04-16T02:35:43.201 INFO:tasks.workunit.client.0.smithi151.stdout:0/224: chown dc/d15/f29 3921 1 2024-04-16T02:35:43.201 INFO:tasks.workunit.client.1.smithi153.stdout:3/719: dwrite d6/d10/d22/d2c/fd9 [0,4194304] 0 2024-04-16T02:35:43.203 INFO:tasks.workunit.client.0.smithi151.stdout:2/304: mkdir de/d11/d42/d5e 0 2024-04-16T02:35:43.204 INFO:tasks.workunit.client.1.smithi153.stdout:0/848: dread dd/d32/d57/f61 [0,4194304] 0 2024-04-16T02:35:43.204 INFO:tasks.workunit.client.1.smithi153.stdout:1/759: symlink d4/dd/d13/d8d/l105 0 2024-04-16T02:35:43.206 INFO:tasks.workunit.client.0.smithi151.stdout:0/225: rename dc/f28 to dc/d15/f45 0 2024-04-16T02:35:43.206 INFO:tasks.workunit.client.1.smithi153.stdout:5/788: symlink d8/d41/dbd/d8d/lfb 0 2024-04-16T02:35:43.207 INFO:tasks.workunit.client.0.smithi151.stdout:2/305: symlink de/d15/d1e/l5f 0 2024-04-16T02:35:43.207 INFO:tasks.workunit.client.0.smithi151.stdout:0/226: rmdir dc/d15/d34/d3b 39 2024-04-16T02:35:43.209 INFO:tasks.workunit.client.1.smithi153.stdout:9/793: dread d4/d6/dc/d2d/da0/d69/fb7 [0,4194304] 0 2024-04-16T02:35:43.210 INFO:tasks.workunit.client.1.smithi153.stdout:2/693: truncate d9/d71/fca 892617 0 2024-04-16T02:35:43.210 INFO:tasks.workunit.client.0.smithi151.stdout:2/306: mkdir de/d11/d41/d4b/d60 0 2024-04-16T02:35:43.212 INFO:tasks.workunit.client.0.smithi151.stdout:0/227: fdatasync dc/d15/f19 0 2024-04-16T02:35:43.212 INFO:tasks.workunit.client.1.smithi153.stdout:3/720: read d6/d10/d22/d43/f57 [4654289,63034] 0 2024-04-16T02:35:43.213 INFO:tasks.workunit.client.1.smithi153.stdout:4/847: dwrite d3/d9/db/de/d2c/f74 [0,4194304] 0 2024-04-16T02:35:43.215 INFO:tasks.workunit.client.1.smithi153.stdout:7/695: dwrite d0/d4/db/d47/d8d/f98 [0,4194304] 0 2024-04-16T02:35:43.216 INFO:tasks.workunit.client.0.smithi151.stdout:0/228: symlink dc/l46 0 2024-04-16T02:35:43.218 INFO:tasks.workunit.client.0.smithi151.stdout:0/229: write dc/f11 [1857954,56827] 0 2024-04-16T02:35:43.220 INFO:tasks.workunit.client.0.smithi151.stdout:0/230: fsync dc/f43 0 2024-04-16T02:35:43.221 INFO:tasks.workunit.client.0.smithi151.stdout:2/307: dwrite de/d11/d42/f52 [0,4194304] 0 2024-04-16T02:35:43.222 INFO:tasks.workunit.client.0.smithi151.stdout:7/192: dread d0/f18 [0,4194304] 0 2024-04-16T02:35:43.222 INFO:tasks.workunit.client.1.smithi153.stdout:8/720: symlink d2/d92/deb/d47/d60/d67/d98/lf2 0 2024-04-16T02:35:43.223 INFO:tasks.workunit.client.1.smithi153.stdout:4/848: dread d3/fdc [0,4194304] 0 2024-04-16T02:35:43.224 INFO:tasks.workunit.client.1.smithi153.stdout:2/694: rmdir d9/d2b/d1c 39 2024-04-16T02:35:43.224 INFO:tasks.workunit.client.0.smithi151.stdout:2/308: fdatasync de/d11/f18 0 2024-04-16T02:35:43.228 INFO:tasks.workunit.client.1.smithi153.stdout:1/760: read d4/dd/d18/d29/d71/fc5 [47067,31199] 0 2024-04-16T02:35:43.229 INFO:tasks.workunit.client.0.smithi151.stdout:0/231: dread f3 [0,4194304] 0 2024-04-16T02:35:43.230 INFO:tasks.workunit.client.0.smithi151.stdout:2/309: symlink de/d15/d1e/l61 0 2024-04-16T02:35:43.233 INFO:tasks.workunit.client.0.smithi151.stdout:6/345: dread d2/d5/df/d20/d4f/f6e [0,4194304] 0 2024-04-16T02:35:43.233 INFO:tasks.workunit.client.1.smithi153.stdout:9/794: symlink d4/d6/dc/lfb 0 2024-04-16T02:35:43.233 INFO:tasks.workunit.client.0.smithi151.stdout:7/193: dwrite d0/f22 [0,4194304] 0 2024-04-16T02:35:43.234 INFO:tasks.workunit.client.1.smithi153.stdout:1/761: read - d4/dd/d13/d8d/f101 zero size 2024-04-16T02:35:43.236 INFO:tasks.workunit.client.0.smithi151.stdout:6/346: write d2/d1f/f59 [1102508,48242] 0 2024-04-16T02:35:43.237 INFO:tasks.workunit.client.1.smithi153.stdout:8/721: rename d2/d92/deb/d47/d77/d8e to d2/d93/d97/d9c/d27/d5e/de4/df3 0 2024-04-16T02:35:43.238 INFO:tasks.workunit.client.1.smithi153.stdout:5/789: getdents d8/df 0 2024-04-16T02:35:43.238 INFO:tasks.workunit.client.0.smithi151.stdout:6/347: truncate d2/d13/f67 684497 0 2024-04-16T02:35:43.240 INFO:tasks.workunit.client.0.smithi151.stdout:6/348: fdatasync d2/d5/fb 0 2024-04-16T02:35:43.240 INFO:tasks.workunit.client.1.smithi153.stdout:5/790: chown d8/df/d28/d5a/d66/ce3 853 1 2024-04-16T02:35:43.244 INFO:tasks.workunit.client.1.smithi153.stdout:8/722: rename d2/d92/deb/d47/d60/d67/d98/fd5 to d2/d13/d96/ff4 0 2024-04-16T02:35:43.244 INFO:tasks.workunit.client.1.smithi153.stdout:7/696: link d0/d4/db/d2b/d43/d6d/d68/d6c/ce9 d0/d4/db/d2b/d8c/ceb 0 2024-04-16T02:35:43.244 INFO:tasks.workunit.client.0.smithi151.stdout:6/349: mkdir d2/d5/df/d23/d56/d72 0 2024-04-16T02:35:43.245 INFO:tasks.workunit.client.1.smithi153.stdout:5/791: creat d8/df/d28/d5a/d66/ffc x:0 0 0 2024-04-16T02:35:43.246 INFO:tasks.workunit.client.0.smithi151.stdout:7/194: dwrite d0/f6 [8388608,4194304] 0 2024-04-16T02:35:43.246 INFO:tasks.workunit.client.1.smithi153.stdout:6/740: dread dc/d47/d57/f70 [0,4194304] 0 2024-04-16T02:35:43.248 INFO:tasks.workunit.client.0.smithi151.stdout:6/350: truncate d2/d1f/d22/d40/d3f/f66 456447 0 2024-04-16T02:35:43.248 INFO:tasks.workunit.client.1.smithi153.stdout:8/723: mknod d2/d93/d97/d9c/cf5 0 2024-04-16T02:35:43.249 INFO:tasks.workunit.client.1.smithi153.stdout:6/741: creat dc/d59/d63/ffb x:0 0 0 2024-04-16T02:35:43.251 INFO:tasks.workunit.client.0.smithi151.stdout:7/195: dread d0/d1f/f28 [0,4194304] 0 2024-04-16T02:35:43.252 INFO:tasks.workunit.client.0.smithi151.stdout:7/196: fsync d0/d27/f2b 0 2024-04-16T02:35:43.252 INFO:tasks.workunit.client.1.smithi153.stdout:8/724: creat d2/d92/deb/d47/d77/ff6 x:0 0 0 2024-04-16T02:35:43.253 INFO:tasks.workunit.client.1.smithi153.stdout:7/697: creat d0/d4/db/d2b/d43/fec x:0 0 0 2024-04-16T02:35:43.255 INFO:tasks.workunit.client.1.smithi153.stdout:6/742: creat dc/d47/d57/d77/d15/dac/dae/ffc x:0 0 0 2024-04-16T02:35:43.255 INFO:tasks.workunit.client.0.smithi151.stdout:6/351: dread d2/d5/f6 [0,4194304] 0 2024-04-16T02:35:43.255 INFO:tasks.workunit.client.0.smithi151.stdout:6/352: fdatasync d2/d13/f5c 0 2024-04-16T02:35:43.256 INFO:tasks.workunit.client.1.smithi153.stdout:5/792: creat d8/d1f/ffd x:0 0 0 2024-04-16T02:35:43.256 INFO:tasks.workunit.client.1.smithi153.stdout:6/743: chown dc/d1a/f68 0 1 2024-04-16T02:35:43.256 INFO:tasks.workunit.client.1.smithi153.stdout:8/725: mknod d2/d13/d3a/d5b/cf7 0 2024-04-16T02:35:43.257 INFO:tasks.workunit.client.0.smithi151.stdout:6/353: mkdir d2/d1f/d22/d40/d3f/d73 0 2024-04-16T02:35:43.258 INFO:tasks.workunit.client.1.smithi153.stdout:5/793: creat d8/d41/dbd/d83/ffe x:0 0 0 2024-04-16T02:35:43.258 INFO:tasks.workunit.client.1.smithi153.stdout:7/698: symlink d0/d4/db/d2b/d43/d6d/d68/led 0 2024-04-16T02:35:43.262 INFO:tasks.workunit.client.1.smithi153.stdout:5/794: dread - d8/d41/dbd/d39/d6d/fa0 zero size 2024-04-16T02:35:43.262 INFO:tasks.workunit.client.0.smithi151.stdout:6/354: link d2/d13/d3a/f54 d2/d5/d2e/f74 0 2024-04-16T02:35:43.262 INFO:tasks.workunit.client.0.smithi151.stdout:6/355: truncate d2/d5/df/d23/d56/f61 680298 0 2024-04-16T02:35:43.263 INFO:tasks.workunit.client.1.smithi153.stdout:6/744: dwrite dc/d59/d63/ffb [0,4194304] 0 2024-04-16T02:35:43.264 INFO:tasks.workunit.client.1.smithi153.stdout:6/745: chown dc/d47/d93 17 1 2024-04-16T02:35:43.265 INFO:tasks.workunit.client.0.smithi151.stdout:6/356: rename d2/d13/d3a/l4c to d2/d13/d5f/l75 0 2024-04-16T02:35:43.266 INFO:tasks.workunit.client.1.smithi153.stdout:5/795: creat d8/df/d28/d5a/d72/fff x:0 0 0 2024-04-16T02:35:43.268 INFO:tasks.workunit.client.1.smithi153.stdout:6/746: creat dc/d47/d57/d77/d7a/d97/db7/ffd x:0 0 0 2024-04-16T02:35:43.268 INFO:tasks.workunit.client.1.smithi153.stdout:5/796: stat d8/d41/dbd/d39/d6d 0 2024-04-16T02:35:43.270 INFO:tasks.workunit.client.0.smithi151.stdout:0/232: write fa [442837,48819] 0 2024-04-16T02:35:43.271 INFO:tasks.workunit.client.1.smithi153.stdout:8/726: read d2/f23 [6618504,129123] 0 2024-04-16T02:35:43.272 INFO:tasks.workunit.client.0.smithi151.stdout:7/197: write d0/f18 [713173,92309] 0 2024-04-16T02:35:43.274 INFO:tasks.workunit.client.0.smithi151.stdout:0/233: dread - dc/d20/d2c/f3e zero size 2024-04-16T02:35:43.275 INFO:tasks.workunit.client.0.smithi151.stdout:0/234: chown dc/f11 2489 1 2024-04-16T02:35:43.277 INFO:tasks.workunit.client.0.smithi151.stdout:0/235: write dc/d20/f42 [512823,105967] 0 2024-04-16T02:35:43.277 INFO:tasks.workunit.client.1.smithi153.stdout:1/762: dread d4/dd/d13/d26/f9d [0,4194304] 0 2024-04-16T02:35:43.277 INFO:tasks.workunit.client.1.smithi153.stdout:6/747: fsync dc/d47/d57/f91 0 2024-04-16T02:35:43.277 INFO:tasks.workunit.client.1.smithi153.stdout:0/849: write dd/d14/da6/f48 [711497,15393] 0 2024-04-16T02:35:43.278 INFO:tasks.workunit.client.0.smithi151.stdout:7/198: mkdir d0/d35 0 2024-04-16T02:35:43.278 INFO:tasks.workunit.client.1.smithi153.stdout:3/721: write d6/d10/d3b/f7b [871220,84140] 0 2024-04-16T02:35:43.281 INFO:tasks.workunit.client.1.smithi153.stdout:8/727: rmdir d2/d93/d97/d9c/d27/d5e 39 2024-04-16T02:35:43.282 INFO:tasks.workunit.client.0.smithi151.stdout:7/199: rename d0/f6 to d0/d1f/d33/f36 0 2024-04-16T02:35:43.282 INFO:tasks.workunit.client.1.smithi153.stdout:1/763: creat d4/dd/d13/d23/d31/f106 x:0 0 0 2024-04-16T02:35:43.283 INFO:tasks.workunit.client.1.smithi153.stdout:6/748: truncate f3 110520 0 2024-04-16T02:35:43.285 INFO:tasks.workunit.client.0.smithi151.stdout:0/236: creat dc/f47 x:0 0 0 2024-04-16T02:35:43.287 INFO:tasks.workunit.client.1.smithi153.stdout:6/749: write dc/d47/d93/fab [168185,9581] 0 2024-04-16T02:35:43.287 INFO:tasks.workunit.client.0.smithi151.stdout:7/200: dread d0/f3 [4194304,4194304] 0 2024-04-16T02:35:43.288 INFO:tasks.workunit.client.0.smithi151.stdout:0/237: creat dc/d3c/f48 x:0 0 0 2024-04-16T02:35:43.290 INFO:tasks.workunit.client.0.smithi151.stdout:0/238: readlink dc/d3c/l41 0 2024-04-16T02:35:43.292 INFO:tasks.workunit.client.1.smithi153.stdout:1/764: symlink d4/dd/d13/d23/d31/l107 0 2024-04-16T02:35:43.293 INFO:tasks.workunit.client.0.smithi151.stdout:0/239: creat dc/d20/f49 x:0 0 0 2024-04-16T02:35:43.293 INFO:tasks.workunit.client.1.smithi153.stdout:4/849: dwrite d3/f6d [0,4194304] 0 2024-04-16T02:35:43.294 INFO:tasks.workunit.client.1.smithi153.stdout:2/695: dwrite d9/d24/d27/d2c/d41/d7a/fb2 [0,4194304] 0 2024-04-16T02:35:43.295 INFO:tasks.workunit.client.0.smithi151.stdout:0/240: dread - dc/d20/d2c/f3e zero size 2024-04-16T02:35:43.296 INFO:tasks.workunit.client.1.smithi153.stdout:0/850: unlink dd/d14/d19/d55/fcf 0 2024-04-16T02:35:43.296 INFO:tasks.workunit.client.1.smithi153.stdout:1/765: read - d4/dd/d13/d8d/f101 zero size 2024-04-16T02:35:43.297 INFO:tasks.workunit.client.0.smithi151.stdout:0/241: readlink dc/d20/l26 0 2024-04-16T02:35:43.297 INFO:tasks.workunit.client.1.smithi153.stdout:1/766: fdatasync d4/dd/d13/d26/f2b 0 2024-04-16T02:35:43.298 INFO:tasks.workunit.client.0.smithi151.stdout:7/201: dwrite d0/f22 [4194304,4194304] 0 2024-04-16T02:35:43.298 INFO:tasks.workunit.client.1.smithi153.stdout:8/728: dwrite d2/d92/f69 [0,4194304] 0 2024-04-16T02:35:43.298 INFO:tasks.workunit.client.1.smithi153.stdout:0/851: truncate dd/d14/da6/da4/ffb 730624 0 2024-04-16T02:35:43.299 INFO:tasks.workunit.client.0.smithi151.stdout:6/357: dread d2/d5/df/d20/d4f/f5e [0,4194304] 0 2024-04-16T02:35:43.300 INFO:tasks.workunit.client.1.smithi153.stdout:6/750: readlink dc/le6 0 2024-04-16T02:35:43.300 INFO:tasks.workunit.client.1.smithi153.stdout:7/699: dread d0/d4/db/f2a [4194304,4194304] 0 2024-04-16T02:35:43.302 INFO:tasks.workunit.client.1.smithi153.stdout:7/700: dread - d0/d4/db/d47/d8d/d93/fcf zero size 2024-04-16T02:35:43.305 INFO:tasks.workunit.client.1.smithi153.stdout:3/722: rename d6/d10/d22/d4d/da0/fdc to d6/ff5 0 2024-04-16T02:35:43.305 INFO:tasks.workunit.client.1.smithi153.stdout:8/729: dwrite d2/d13/d96/fb5 [0,4194304] 0 2024-04-16T02:35:43.309 INFO:tasks.workunit.client.0.smithi151.stdout:0/242: unlink dc/c13 0 2024-04-16T02:35:43.309 INFO:tasks.workunit.client.0.smithi151.stdout:6/358: dread d2/d1f/d22/f4e [0,4194304] 0 2024-04-16T02:35:43.311 INFO:tasks.workunit.client.1.smithi153.stdout:0/852: chown dd/d14/d19/c95 1737 1 2024-04-16T02:35:43.312 INFO:tasks.workunit.client.0.smithi151.stdout:7/202: mknod d0/d35/c37 0 2024-04-16T02:35:43.313 INFO:tasks.workunit.client.0.smithi151.stdout:7/203: fdatasync d0/f18 0 2024-04-16T02:35:43.315 INFO:tasks.workunit.client.0.smithi151.stdout:0/243: creat dc/d20/f4a x:0 0 0 2024-04-16T02:35:43.316 INFO:tasks.workunit.client.0.smithi151.stdout:7/204: chown d0/d2c/l2f 3771 1 2024-04-16T02:35:43.318 INFO:tasks.workunit.client.0.smithi151.stdout:6/359: getdents d2/d1f/d22/d40/d3f/d73 0 2024-04-16T02:35:43.321 INFO:tasks.workunit.client.1.smithi153.stdout:1/767: dwrite d4/dd/d18/d29/fde [0,4194304] 0 2024-04-16T02:35:43.322 INFO:tasks.workunit.client.1.smithi153.stdout:2/696: creat d9/d2b/d1c/d20/fe7 x:0 0 0 2024-04-16T02:35:43.322 INFO:tasks.workunit.client.0.smithi151.stdout:6/360: symlink d2/d1f/d22/d40/d3f/d73/l76 0 2024-04-16T02:35:43.323 INFO:tasks.workunit.client.0.smithi151.stdout:7/205: dread d0/f22 [0,4194304] 0 2024-04-16T02:35:43.326 INFO:tasks.workunit.client.0.smithi151.stdout:7/206: mkdir d0/d2c/d38 0 2024-04-16T02:35:43.328 INFO:tasks.workunit.client.1.smithi153.stdout:6/751: write dc/d47/f87 [777762,103279] 0 2024-04-16T02:35:43.329 INFO:tasks.workunit.client.1.smithi153.stdout:6/752: chown dc/d59/d5b/lc6 876811574 1 2024-04-16T02:35:43.330 INFO:tasks.workunit.client.0.smithi151.stdout:6/361: rename d2/l28 to d2/d13/d63/l77 0 2024-04-16T02:35:43.330 INFO:tasks.workunit.client.1.smithi153.stdout:4/850: rename d3/d9/db/de/d2c/d52/d85/ffb to d3/da5/dd2/f12a 0 2024-04-16T02:35:43.331 INFO:tasks.workunit.client.0.smithi151.stdout:7/207: creat d0/d35/f39 x:0 0 0 2024-04-16T02:35:43.332 INFO:tasks.workunit.client.1.smithi153.stdout:4/851: fsync d3/d9/f3d 0 2024-04-16T02:35:43.333 INFO:tasks.workunit.client.1.smithi153.stdout:2/697: dwrite d9/d24/d27/d2c/d41/d7a/fb2 [0,4194304] 0 2024-04-16T02:35:43.333 INFO:tasks.workunit.client.0.smithi151.stdout:6/362: unlink d2/d5/l37 0 2024-04-16T02:35:43.338 INFO:tasks.workunit.client.0.smithi151.stdout:6/363: rename d2/d13/d5f/l69 to d2/d5/df/d23/d56/d72/l78 0 2024-04-16T02:35:43.340 INFO:tasks.workunit.client.0.smithi151.stdout:7/208: dwrite d0/f7 [4194304,4194304] 0 2024-04-16T02:35:43.344 INFO:tasks.workunit.client.0.smithi151.stdout:6/364: link d2/d1f/f59 d2/d13/d46/f79 0 2024-04-16T02:35:43.348 INFO:tasks.workunit.client.0.smithi151.stdout:7/209: rename d0/d1f/d31/f30 to d0/d1f/d33/f3a 0 2024-04-16T02:35:43.351 INFO:tasks.workunit.client.0.smithi151.stdout:7/210: creat d0/d2c/f3b x:0 0 0 2024-04-16T02:35:43.351 INFO:tasks.workunit.client.0.smithi151.stdout:7/211: read d0/f18 [532408,55544] 0 2024-04-16T02:35:43.354 INFO:tasks.workunit.client.0.smithi151.stdout:7/212: rename d0/l26 to d0/l3c 0 2024-04-16T02:35:43.355 INFO:tasks.workunit.client.0.smithi151.stdout:7/213: stat d0/d1f/l2d 0 2024-04-16T02:35:43.356 INFO:tasks.workunit.client.0.smithi151.stdout:7/214: chown d0/d27/f2b 27586 1 2024-04-16T02:35:43.358 INFO:tasks.workunit.client.0.smithi151.stdout:7/215: creat d0/d2c/d38/f3d x:0 0 0 2024-04-16T02:35:43.359 INFO:tasks.workunit.client.0.smithi151.stdout:7/216: readlink d0/l8 0 2024-04-16T02:35:43.360 INFO:tasks.workunit.client.0.smithi151.stdout:0/244: dread dc/d15/f45 [0,4194304] 0 2024-04-16T02:35:43.363 INFO:tasks.workunit.client.0.smithi151.stdout:0/245: mkdir dc/d20/d2c/d4b 0 2024-04-16T02:35:43.367 INFO:tasks.workunit.client.0.smithi151.stdout:7/217: dwrite d0/d27/f32 [0,4194304] 0 2024-04-16T02:35:43.367 INFO:tasks.workunit.client.0.smithi151.stdout:0/246: write dc/d15/d34/d3b/f44 [129777,31258] 0 2024-04-16T02:35:43.370 INFO:tasks.workunit.client.0.smithi151.stdout:7/218: write d0/f5 [3428197,86604] 0 2024-04-16T02:35:43.375 INFO:tasks.workunit.client.0.smithi151.stdout:9/170: sync 2024-04-16T02:35:43.375 INFO:tasks.workunit.client.0.smithi151.stdout:8/155: sync 2024-04-16T02:35:43.375 INFO:tasks.workunit.client.0.smithi151.stdout:6/365: dread d2/f1a [0,4194304] 0 2024-04-16T02:35:43.375 INFO:tasks.workunit.client.0.smithi151.stdout:0/247: mkdir dc/d4c 0 2024-04-16T02:35:43.379 INFO:tasks.workunit.client.0.smithi151.stdout:6/366: symlink d2/d5/df/d20/l7a 0 2024-04-16T02:35:43.380 INFO:tasks.workunit.client.0.smithi151.stdout:8/156: symlink d11/l31 0 2024-04-16T02:35:43.380 INFO:tasks.workunit.client.0.smithi151.stdout:7/219: fsync d0/d2c/f3b 0 2024-04-16T02:35:43.382 INFO:tasks.workunit.client.0.smithi151.stdout:6/367: unlink d2/d5/df/d23/c32 0 2024-04-16T02:35:43.384 INFO:tasks.workunit.client.0.smithi151.stdout:0/248: dread dc/d15/f19 [0,4194304] 0 2024-04-16T02:35:43.384 INFO:tasks.workunit.client.0.smithi151.stdout:8/157: chown d11/f1a 1 1 2024-04-16T02:35:43.385 INFO:tasks.workunit.client.0.smithi151.stdout:9/171: mknod d1/d3/d6/d28/d39/c3f 0 2024-04-16T02:35:43.385 INFO:tasks.workunit.client.0.smithi151.stdout:9/172: chown d1/d3/d6/d1d/d22 0 1 2024-04-16T02:35:43.386 INFO:tasks.workunit.client.1.smithi153.stdout:7/701: rmdir d0/d4/db/d34/d4b/db7 0 2024-04-16T02:35:43.386 INFO:tasks.workunit.client.0.smithi151.stdout:7/220: mkdir d0/d2c/d38/d3e 0 2024-04-16T02:35:43.387 INFO:tasks.workunit.client.1.smithi153.stdout:6/753: rmdir dc/d47/d57/d77/d7a 39 2024-04-16T02:35:43.387 INFO:tasks.workunit.client.1.smithi153.stdout:3/723: creat d6/d10/d22/ff6 x:0 0 0 2024-04-16T02:35:43.388 INFO:tasks.workunit.client.0.smithi151.stdout:6/368: rename d2/l4a to d2/d1f/d22/d6d/l7b 0 2024-04-16T02:35:43.388 INFO:tasks.workunit.client.0.smithi151.stdout:6/369: stat d2/d5/f5a 0 2024-04-16T02:35:43.388 INFO:tasks.workunit.client.0.smithi151.stdout:8/158: mknod d11/d2a/c32 0 2024-04-16T02:35:43.389 INFO:tasks.workunit.client.1.smithi153.stdout:0/853: rename c7 to dd/d14/d19/d55/c10d 0 2024-04-16T02:35:43.390 INFO:tasks.workunit.client.1.smithi153.stdout:8/730: getdents d2/d13/d96 0 2024-04-16T02:35:43.392 INFO:tasks.workunit.client.1.smithi153.stdout:2/698: creat d9/d24/d27/d9a/fe8 x:0 0 0 2024-04-16T02:35:43.394 INFO:tasks.workunit.client.0.smithi151.stdout:7/221: creat d0/d1f/d31/f3f x:0 0 0 2024-04-16T02:35:43.397 INFO:tasks.workunit.client.1.smithi153.stdout:6/754: dwrite dc/d10/d62/f8b [0,4194304] 0 2024-04-16T02:35:43.397 INFO:tasks.workunit.client.1.smithi153.stdout:4/852: creat d3/d9/db/de/d2c/de2/d110/f12b x:0 0 0 2024-04-16T02:35:43.398 INFO:tasks.workunit.client.0.smithi151.stdout:0/249: link dc/d15/f19 dc/d15/d34/f4d 0 2024-04-16T02:35:43.398 INFO:tasks.workunit.client.1.smithi153.stdout:7/702: mkdir d0/d4/db/d87/dc0/de0/dee 0 2024-04-16T02:35:43.399 INFO:tasks.workunit.client.0.smithi151.stdout:8/159: dwrite fe [0,4194304] 0 2024-04-16T02:35:43.399 INFO:tasks.workunit.client.1.smithi153.stdout:8/731: mknod d2/d92/deb/d47/d60/d67/d86/da4/cf8 0 2024-04-16T02:35:43.402 INFO:tasks.workunit.client.0.smithi151.stdout:6/370: getdents d2/d5/df/d20/d4f 0 2024-04-16T02:35:43.404 INFO:tasks.workunit.client.1.smithi153.stdout:0/854: dwrite dd/d32/d57/d6c/fbb [0,4194304] 0 2024-04-16T02:35:43.404 INFO:tasks.workunit.client.1.smithi153.stdout:2/699: rename d9/d24/d34/d36/ccd to d9/d24/d27/d9a/dae/ce9 0 2024-04-16T02:35:43.404 INFO:tasks.workunit.client.0.smithi151.stdout:8/160: mkdir d11/d2a/d33 0 2024-04-16T02:35:43.406 INFO:tasks.workunit.client.1.smithi153.stdout:6/755: mkdir dc/d59/dfe 0 2024-04-16T02:35:43.406 INFO:tasks.workunit.client.1.smithi153.stdout:2/700: fdatasync d9/d71/fca 0 2024-04-16T02:35:43.407 INFO:tasks.workunit.client.0.smithi151.stdout:6/371: mknod d2/d1f/d22/c7c 0 2024-04-16T02:35:43.408 INFO:tasks.workunit.client.0.smithi151.stdout:6/372: write d2/d5/df/d23/d56/f58 [630285,116848] 0 2024-04-16T02:35:43.410 INFO:tasks.workunit.client.0.smithi151.stdout:7/222: getdents d0/d27 0 2024-04-16T02:35:43.411 INFO:tasks.workunit.client.1.smithi153.stdout:3/724: creat d6/d10/d22/d2e/d8a/ff7 x:0 0 0 2024-04-16T02:35:43.414 INFO:tasks.workunit.client.1.smithi153.stdout:2/701: mknod d9/d24/d27/d2c/d41/cea 0 2024-04-16T02:35:43.414 INFO:tasks.workunit.client.1.smithi153.stdout:7/703: chown d0/d4/db/d2b/d43/d6d/d68/d6c/ce9 91 1 2024-04-16T02:35:43.414 INFO:tasks.workunit.client.1.smithi153.stdout:3/725: mkdir d6/d10/dec/df8 0 2024-04-16T02:35:43.416 INFO:tasks.workunit.client.1.smithi153.stdout:2/702: mknod d9/d2b/d1c/d20/ceb 0 2024-04-16T02:35:43.418 INFO:tasks.workunit.client.1.smithi153.stdout:7/704: symlink d0/d4/db/d2b/d43/d6d/d68/lef 0 2024-04-16T02:35:43.419 INFO:tasks.workunit.client.1.smithi153.stdout:2/703: write d9/fba [771844,93664] 0 2024-04-16T02:35:43.420 INFO:tasks.workunit.client.0.smithi151.stdout:3/153: sync 2024-04-16T02:35:43.420 INFO:tasks.workunit.client.0.smithi151.stdout:5/305: sync 2024-04-16T02:35:43.420 INFO:tasks.workunit.client.0.smithi151.stdout:1/251: sync 2024-04-16T02:35:43.420 INFO:tasks.workunit.client.0.smithi151.stdout:2/310: sync 2024-04-16T02:35:43.420 INFO:tasks.workunit.client.0.smithi151.stdout:4/286: sync 2024-04-16T02:35:43.421 INFO:tasks.workunit.client.1.smithi153.stdout:2/704: fdatasync d9/d2b/f46 0 2024-04-16T02:35:43.422 INFO:tasks.workunit.client.1.smithi153.stdout:7/705: creat d0/d4/db/d2b/d43/d6d/d68/d6c/db8/ddf/ff0 x:0 0 0 2024-04-16T02:35:43.424 INFO:tasks.workunit.client.1.smithi153.stdout:2/705: write d9/d24/d27/d2c/d5d/da3/fd3 [699167,98452] 0 2024-04-16T02:35:43.427 INFO:tasks.workunit.client.1.smithi153.stdout:2/706: rename d9/d24/d34/d36/f7b to d9/d2b/d18/d3f/fec 0 2024-04-16T02:35:43.428 INFO:tasks.workunit.client.1.smithi153.stdout:2/707: chown d9/d24/d27/d2c/d41/f4c 0 1 2024-04-16T02:35:43.429 INFO:tasks.workunit.client.1.smithi153.stdout:8/732: dread d2/d93/d97/d9c/d27/d5e/de4/df3/fbb [0,4194304] 0 2024-04-16T02:35:43.431 INFO:tasks.workunit.client.1.smithi153.stdout:8/733: write d2/d92/deb/d47/d60/d67/d86/f89 [442481,126352] 0 2024-04-16T02:35:43.438 INFO:tasks.workunit.client.1.smithi153.stdout:0/855: truncate dd/d14/da6/da4/fa8 714927 0 2024-04-16T02:35:43.439 INFO:tasks.workunit.client.1.smithi153.stdout:7/706: write d0/d4/db/d2b/d53/d59/f5a [1296030,106256] 0 2024-04-16T02:35:43.441 INFO:tasks.workunit.client.1.smithi153.stdout:2/708: write d9/d2b/d18/d3f/d40/f70 [957504,122928] 0 2024-04-16T02:35:43.441 INFO:tasks.workunit.client.1.smithi153.stdout:9/795: sync 2024-04-16T02:35:43.443 INFO:tasks.workunit.client.1.smithi153.stdout:0/856: link dd/d26/l67 dd/d14/d19/d77/d87/dda/l10e 0 2024-04-16T02:35:43.445 INFO:tasks.workunit.client.1.smithi153.stdout:7/707: dread d0/f5b [0,4194304] 0 2024-04-16T02:35:43.450 INFO:tasks.workunit.client.1.smithi153.stdout:0/857: creat dd/d14/d51/d54/f10f x:0 0 0 2024-04-16T02:35:43.451 INFO:tasks.workunit.client.1.smithi153.stdout:7/708: symlink d0/lf1 0 2024-04-16T02:35:43.453 INFO:tasks.workunit.client.1.smithi153.stdout:8/734: dwrite d2/de/f24 [0,4194304] 0 2024-04-16T02:35:43.456 INFO:tasks.workunit.client.1.smithi153.stdout:2/709: dwrite d9/d24/d73/fc8 [0,4194304] 0 2024-04-16T02:35:43.456 INFO:tasks.workunit.client.1.smithi153.stdout:7/709: mknod d0/de8/cf2 0 2024-04-16T02:35:43.457 INFO:tasks.workunit.client.1.smithi153.stdout:9/796: dwrite d4/d6/dc/d2d/da0/f88 [0,4194304] 0 2024-04-16T02:35:43.459 INFO:tasks.workunit.client.1.smithi153.stdout:8/735: dread - d2/d92/deb/d47/d60/f7b zero size 2024-04-16T02:35:43.460 INFO:tasks.workunit.client.1.smithi153.stdout:0/858: mknod dd/d14/d19/d77/d87/c110 0 2024-04-16T02:35:43.460 INFO:tasks.workunit.client.1.smithi153.stdout:2/710: read - d9/d2b/f86 zero size 2024-04-16T02:35:43.467 INFO:tasks.workunit.client.1.smithi153.stdout:8/736: fdatasync d2/d13/f94 0 2024-04-16T02:35:43.469 INFO:tasks.workunit.client.0.smithi151.stdout:7/223: creat d0/d1f/f40 x:0 0 0 2024-04-16T02:35:43.470 INFO:tasks.workunit.client.1.smithi153.stdout:2/711: mkdir d9/d24/ddd/ded 0 2024-04-16T02:35:43.470 INFO:tasks.workunit.client.0.smithi151.stdout:6/373: link d2/d1f/d22/d40/d3f/f6c d2/d5/df/d23/d31/f7d 0 2024-04-16T02:35:43.472 INFO:tasks.workunit.client.1.smithi153.stdout:9/797: dwrite d4/d6/dc/d2d/fa2 [0,4194304] 0 2024-04-16T02:35:43.472 INFO:tasks.workunit.client.1.smithi153.stdout:7/710: dread d0/d4/db/d2b/d53/d59/d76/dc8/fb4 [0,4194304] 0 2024-04-16T02:35:43.472 INFO:tasks.workunit.client.0.smithi151.stdout:4/287: write d0/d20/f42 [706657,55782] 0 2024-04-16T02:35:43.474 INFO:tasks.workunit.client.1.smithi153.stdout:2/712: fdatasync d9/d24/d27/f1f 0 2024-04-16T02:35:43.474 INFO:tasks.workunit.client.0.smithi151.stdout:9/173: dwrite d1/d3/f1a [0,4194304] 0 2024-04-16T02:35:43.475 INFO:tasks.workunit.client.0.smithi151.stdout:3/154: stat d1/d6/d20/d2a/c19 0 2024-04-16T02:35:43.477 INFO:tasks.workunit.client.1.smithi153.stdout:7/711: write d0/d4/db/d2b/d8c/f9b [1057890,105961] 0 2024-04-16T02:35:43.477 INFO:tasks.workunit.client.1.smithi153.stdout:9/798: mkdir d4/d6/d1b/d2a/d99/dfc 0 2024-04-16T02:35:43.480 INFO:tasks.workunit.client.0.smithi151.stdout:2/311: rename de/d3d/l5a to de/d11/l62 0 2024-04-16T02:35:43.480 INFO:tasks.workunit.client.1.smithi153.stdout:6/756: dread dc/d47/f4a [0,4194304] 0 2024-04-16T02:35:43.484 INFO:tasks.workunit.client.1.smithi153.stdout:8/737: dread d2/de/f5f [0,4194304] 0 2024-04-16T02:35:43.484 INFO:tasks.workunit.client.0.smithi151.stdout:1/252: unlink d3/d4/db/df/d38/f3b 0 2024-04-16T02:35:43.487 INFO:tasks.workunit.client.0.smithi151.stdout:9/174: dwrite d1/f2e [0,4194304] 0 2024-04-16T02:35:43.488 INFO:tasks.workunit.client.1.smithi153.stdout:7/712: creat d0/d4/db/d2b/d43/d6d/d68/d6c/ff3 x:0 0 0 2024-04-16T02:35:43.489 INFO:tasks.workunit.client.0.smithi151.stdout:3/155: symlink d1/d6/d22/d23/l34 0 2024-04-16T02:35:43.489 INFO:tasks.workunit.client.0.smithi151.stdout:9/175: write d1/d3/fa [2892153,69173] 0 2024-04-16T02:35:43.492 INFO:tasks.workunit.client.0.smithi151.stdout:7/224: rename d0/c13 to d0/d27/c41 0 2024-04-16T02:35:43.493 INFO:tasks.workunit.client.1.smithi153.stdout:4/853: dread d3/d9/db/de/d2c/d52/d85/ffe [0,4194304] 0 2024-04-16T02:35:43.496 INFO:tasks.workunit.client.0.smithi151.stdout:3/156: creat d1/d6/d20/d2a/f35 x:0 0 0 2024-04-16T02:35:43.497 INFO:tasks.workunit.client.1.smithi153.stdout:7/713: chown d0/d4/db/d34/d4b/f75 933639 1 2024-04-16T02:35:43.498 INFO:tasks.workunit.client.1.smithi153.stdout:6/757: dwrite dc/f11 [0,4194304] 0 2024-04-16T02:35:43.498 INFO:tasks.workunit.client.0.smithi151.stdout:8/161: rmdir d11/d2a 39 2024-04-16T02:35:43.499 INFO:tasks.workunit.client.1.smithi153.stdout:2/713: link d9/d2b/d1c/d20/d45/cd7 d9/d71/cee 0 2024-04-16T02:35:43.499 INFO:tasks.workunit.client.0.smithi151.stdout:2/312: dwrite de/d15/d1e/f24 [0,4194304] 0 2024-04-16T02:35:43.500 INFO:tasks.workunit.client.1.smithi153.stdout:4/854: stat d3/d9/f77 0 2024-04-16T02:35:43.504 INFO:tasks.workunit.client.1.smithi153.stdout:9/799: dwrite d4/d6/dc/d7f/fb3 [0,4194304] 0 2024-04-16T02:35:43.506 INFO:tasks.workunit.client.0.smithi151.stdout:9/176: dwrite d1/d3/f1a [4194304,4194304] 0 2024-04-16T02:35:43.508 INFO:tasks.workunit.client.1.smithi153.stdout:8/738: dwrite d2/d93/d97/d9c/f53 [0,4194304] 0 2024-04-16T02:35:43.508 INFO:tasks.workunit.client.1.smithi153.stdout:6/758: mknod dc/d59/dfe/cff 0 2024-04-16T02:35:43.508 INFO:tasks.workunit.client.0.smithi151.stdout:9/177: dread - d1/d3/d6/f2a zero size 2024-04-16T02:35:43.510 INFO:tasks.workunit.client.0.smithi151.stdout:9/178: chown d1/d3/d6/d1d/d22/d36 2 1 2024-04-16T02:35:43.511 INFO:tasks.workunit.client.1.smithi153.stdout:6/759: stat dc/d47/d57/d77/d15/dac 0 2024-04-16T02:35:43.512 INFO:tasks.workunit.client.1.smithi153.stdout:8/739: write d2/d92/deb/d47/d77/ff6 [212158,123286] 0 2024-04-16T02:35:43.513 INFO:tasks.workunit.client.0.smithi151.stdout:1/253: link d3/d4/l13 d3/d4/db/l57 0 2024-04-16T02:35:43.513 INFO:tasks.workunit.client.1.smithi153.stdout:7/714: rename d0/d4/db/d2b/f28 to d0/d4/db/d34/d4b/d4d/ff4 0 2024-04-16T02:35:43.513 INFO:tasks.workunit.client.1.smithi153.stdout:8/740: fsync d2/d93/d97/d9c/fdc 0 2024-04-16T02:35:43.514 INFO:tasks.workunit.client.1.smithi153.stdout:9/800: readlink d4/d6/d1b/l36 0 2024-04-16T02:35:43.515 INFO:tasks.workunit.client.0.smithi151.stdout:9/179: unlink d1/f5 0 2024-04-16T02:35:43.515 INFO:tasks.workunit.client.1.smithi153.stdout:2/714: getdents d9/d24/d27/d2c/d41/d7a 0 2024-04-16T02:35:43.520 INFO:tasks.workunit.client.0.smithi151.stdout:9/180: fdatasync d1/d3/d6/f13 0 2024-04-16T02:35:43.521 INFO:tasks.workunit.client.0.smithi151.stdout:3/157: mknod d1/d6/d22/d23/d30/c36 0 2024-04-16T02:35:43.522 INFO:tasks.workunit.client.0.smithi151.stdout:7/225: getdents d0/d1f/d31 0 2024-04-16T02:35:43.523 INFO:tasks.workunit.client.1.smithi153.stdout:7/715: symlink d0/d4/db/d2b/d43/d6d/d68/d6c/lf5 0 2024-04-16T02:35:43.524 INFO:tasks.workunit.client.1.smithi153.stdout:9/801: dwrite d4/d6/db1/fb4 [0,4194304] 0 2024-04-16T02:35:43.525 INFO:tasks.workunit.client.1.smithi153.stdout:2/715: rename d9/d2b/d18/d3f/fec to d9/d24/d27/d2c/fef 0 2024-04-16T02:35:43.526 INFO:tasks.workunit.client.1.smithi153.stdout:3/726: dread d6/d10/d22/d2e/f45 [4194304,4194304] 0 2024-04-16T02:35:43.528 INFO:tasks.workunit.client.1.smithi153.stdout:7/716: chown d0/d4/c49 89 1 2024-04-16T02:35:43.531 INFO:tasks.workunit.client.1.smithi153.stdout:9/802: chown d4/d33/dae/dc7/le6 15003 1 2024-04-16T02:35:43.532 INFO:tasks.workunit.client.1.smithi153.stdout:7/717: chown d0/d4/db/d2b/d53/d59/d76/dc8/fb4 9390 1 2024-04-16T02:35:43.533 INFO:tasks.workunit.client.1.smithi153.stdout:6/760: getdents dc/d47/d93 0 2024-04-16T02:35:43.535 INFO:tasks.workunit.client.1.smithi153.stdout:7/718: write d0/d4/db/d34/d4b/d4d/f66 [2963123,120552] 0 2024-04-16T02:35:43.535 INFO:tasks.workunit.client.0.smithi151.stdout:9/181: dread d1/dd/f16 [0,4194304] 0 2024-04-16T02:35:43.537 INFO:tasks.workunit.client.1.smithi153.stdout:2/716: dwrite d9/d2b/d5c/da2/fc6 [0,4194304] 0 2024-04-16T02:35:43.539 INFO:tasks.workunit.client.0.smithi151.stdout:2/313: rename de/d15/f43 to de/f63 0 2024-04-16T02:35:43.539 INFO:tasks.workunit.client.0.smithi151.stdout:6/374: dwrite d2/d1f/f59 [0,4194304] 0 2024-04-16T02:35:43.540 INFO:tasks.workunit.client.1.smithi153.stdout:8/741: getdents d2/d92/deb/d47/d77 0 2024-04-16T02:35:43.540 INFO:tasks.workunit.client.0.smithi151.stdout:6/375: dread - d2/d5/f3e zero size 2024-04-16T02:35:43.542 INFO:tasks.workunit.client.1.smithi153.stdout:3/727: creat d6/d10/d3b/d59/d7c/de1/df0/def/ff9 x:0 0 0 2024-04-16T02:35:43.542 INFO:tasks.workunit.client.0.smithi151.stdout:7/226: mkdir d0/d1f/d31/d42 0 2024-04-16T02:35:43.542 INFO:tasks.workunit.client.1.smithi153.stdout:7/719: symlink d0/d4/db/d87/dc0/de0/lf6 0 2024-04-16T02:35:43.543 INFO:tasks.workunit.client.0.smithi151.stdout:3/158: mkdir d1/d37 0 2024-04-16T02:35:43.544 INFO:tasks.workunit.client.0.smithi151.stdout:8/162: link d11/l31 d11/l34 0 2024-04-16T02:35:43.544 INFO:tasks.workunit.client.1.smithi153.stdout:2/717: fdatasync d9/d2b/f86 0 2024-04-16T02:35:43.545 INFO:tasks.workunit.client.0.smithi151.stdout:5/306: dwrite d2/d6/d19/d30/f39 [0,4194304] 0 2024-04-16T02:35:43.545 INFO:tasks.workunit.client.0.smithi151.stdout:9/182: mkdir d1/d3/d6/d17/d40 0 2024-04-16T02:35:43.546 INFO:tasks.workunit.client.0.smithi151.stdout:6/376: truncate d2/d13/f5c 549072 0 2024-04-16T02:35:43.550 INFO:tasks.workunit.client.1.smithi153.stdout:5/797: sync 2024-04-16T02:35:43.551 INFO:tasks.workunit.client.1.smithi153.stdout:2/718: mknod d9/d71/d8d/cf0 0 2024-04-16T02:35:43.553 INFO:tasks.workunit.client.1.smithi153.stdout:3/728: unlink d6/d10/d5b/c8d 0 2024-04-16T02:35:43.554 INFO:tasks.workunit.client.1.smithi153.stdout:8/742: creat d2/d93/d97/d9c/d27/ff9 x:0 0 0 2024-04-16T02:35:43.556 INFO:tasks.workunit.client.1.smithi153.stdout:2/719: chown d9/d24/d27/d2c/c2d 99313414 1 2024-04-16T02:35:43.558 INFO:tasks.workunit.client.0.smithi151.stdout:7/227: chown d0/c16 219566 1 2024-04-16T02:35:43.559 INFO:tasks.workunit.client.1.smithi153.stdout:3/729: creat d6/d10/d22/d2e/ffa x:0 0 0 2024-04-16T02:35:43.559 INFO:tasks.workunit.client.1.smithi153.stdout:2/720: chown d9/d24/d27/d2c/d41/fb5 24 1 2024-04-16T02:35:43.560 INFO:tasks.workunit.client.1.smithi153.stdout:3/730: readlink d6/d10/d22/l3c 0 2024-04-16T02:35:43.560 INFO:tasks.workunit.client.0.smithi151.stdout:1/254: rename d3/d4/db to d3/d58 0 2024-04-16T02:35:43.561 INFO:tasks.workunit.client.1.smithi153.stdout:2/721: fsync d9/d24/d34/fce 0 2024-04-16T02:35:43.561 INFO:tasks.workunit.client.1.smithi153.stdout:3/731: write d6/d10/d3b/d76/f7d [3546020,103410] 0 2024-04-16T02:35:43.562 INFO:tasks.workunit.client.0.smithi151.stdout:8/163: dwrite d11/f18 [0,4194304] 0 2024-04-16T02:35:43.563 INFO:tasks.workunit.client.1.smithi153.stdout:5/798: dread d8/df/d28/d5a/fb8 [0,4194304] 0 2024-04-16T02:35:43.564 INFO:tasks.workunit.client.0.smithi151.stdout:9/183: dwrite d1/d3/d6/f2a [0,4194304] 0 2024-04-16T02:35:43.565 INFO:tasks.workunit.client.0.smithi151.stdout:6/377: creat d2/d5/df/d23/d56/f7e x:0 0 0 2024-04-16T02:35:43.565 INFO:tasks.workunit.client.1.smithi153.stdout:5/799: fsync d8/d41/dbd/d39/f4c 0 2024-04-16T02:35:43.567 INFO:tasks.workunit.client.0.smithi151.stdout:2/314: getdents de/d11 0 2024-04-16T02:35:43.567 INFO:tasks.workunit.client.1.smithi153.stdout:2/722: creat d9/d24/d34/d60/ddc/ff1 x:0 0 0 2024-04-16T02:35:43.567 INFO:tasks.workunit.client.0.smithi151.stdout:5/307: rmdir d2/d6/d19/d30/d36 39 2024-04-16T02:35:43.570 INFO:tasks.workunit.client.0.smithi151.stdout:7/228: mkdir d0/d2c/d38/d43 0 2024-04-16T02:35:43.571 INFO:tasks.workunit.client.0.smithi151.stdout:6/378: stat d2/d13/d46/f53 0 2024-04-16T02:35:43.573 INFO:tasks.workunit.client.0.smithi151.stdout:3/159: dwrite d1/d6/ff [4194304,4194304] 0 2024-04-16T02:35:43.573 INFO:tasks.workunit.client.0.smithi151.stdout:1/255: truncate d3/d58/d1b/d1c/f50 471672 0 2024-04-16T02:35:43.574 INFO:tasks.workunit.client.1.smithi153.stdout:2/723: rename d9/d24/ddd/ded to d9/d71/df2 0 2024-04-16T02:35:43.575 INFO:tasks.workunit.client.0.smithi151.stdout:2/315: unlink de/d11/d42/f52 0 2024-04-16T02:35:43.576 INFO:tasks.workunit.client.1.smithi153.stdout:4/855: write d3/d9/d24/d75/fb4 [423864,117255] 0 2024-04-16T02:35:43.577 INFO:tasks.workunit.client.0.smithi151.stdout:9/184: chown d1/dd/d38/l3c 83913515 1 2024-04-16T02:35:43.579 INFO:tasks.workunit.client.0.smithi151.stdout:2/316: fdatasync de/d15/f17 0 2024-04-16T02:35:43.579 INFO:tasks.workunit.client.1.smithi153.stdout:2/724: unlink d9/d2b/f46 0 2024-04-16T02:35:43.581 INFO:tasks.workunit.client.0.smithi151.stdout:6/379: read d2/d1f/f4b [242456,27550] 0 2024-04-16T02:35:43.581 INFO:tasks.workunit.client.1.smithi153.stdout:4/856: dread d3/fdc [0,4194304] 0 2024-04-16T02:35:43.584 INFO:tasks.workunit.client.1.smithi153.stdout:6/761: write dc/d47/d57/d77/d15/f1e [986086,57850] 0 2024-04-16T02:35:43.586 INFO:tasks.workunit.client.0.smithi151.stdout:6/380: stat d2/d1f/d22/d40/d3f/d73/l76 0 2024-04-16T02:35:43.587 INFO:tasks.workunit.client.0.smithi151.stdout:3/160: creat d1/d37/f38 x:0 0 0 2024-04-16T02:35:43.589 INFO:tasks.workunit.client.1.smithi153.stdout:8/743: write d2/de/d34/f40 [2134568,36579] 0 2024-04-16T02:35:43.589 INFO:tasks.workunit.client.1.smithi153.stdout:5/800: truncate d8/f30 3752213 0 2024-04-16T02:35:43.590 INFO:tasks.workunit.client.1.smithi153.stdout:4/857: truncate d3/d9/db/de/f71 673020 0 2024-04-16T02:35:43.591 INFO:tasks.workunit.client.1.smithi153.stdout:2/725: symlink d9/lf3 0 2024-04-16T02:35:43.591 INFO:tasks.workunit.client.0.smithi151.stdout:9/185: dwrite d1/d3/d6/d1d/d22/f23 [0,4194304] 0 2024-04-16T02:35:43.593 INFO:tasks.workunit.client.1.smithi153.stdout:6/762: creat dc/d47/d57/d77/dc4/dc9/ded/f100 x:0 0 0 2024-04-16T02:35:43.596 INFO:tasks.workunit.client.0.smithi151.stdout:6/381: chown d2/d13/d63/l77 5 1 2024-04-16T02:35:43.598 INFO:tasks.workunit.client.1.smithi153.stdout:3/732: dwrite d6/d10/d22/d2c/fd3 [4194304,4194304] 0 2024-04-16T02:35:43.599 INFO:tasks.workunit.client.0.smithi151.stdout:9/186: dread d1/d3/d6/d1d/d22/f23 [0,4194304] 0 2024-04-16T02:35:43.600 INFO:tasks.workunit.client.0.smithi151.stdout:8/164: fsync fb 0 2024-04-16T02:35:43.601 INFO:tasks.workunit.client.0.smithi151.stdout:3/161: creat d1/d6/d22/d23/d30/f39 x:0 0 0 2024-04-16T02:35:43.601 INFO:tasks.workunit.client.0.smithi151.stdout:1/256: link d3/d58/d1b/d1c/c21 d3/d1f/d48/c59 0 2024-04-16T02:35:43.602 INFO:tasks.workunit.client.0.smithi151.stdout:3/162: dread - d1/d6/d22/d23/f2d zero size 2024-04-16T02:35:43.603 INFO:tasks.workunit.client.0.smithi151.stdout:1/257: readlink d3/d58/d1b/d2a/l41 0 2024-04-16T02:35:43.603 INFO:tasks.workunit.client.1.smithi153.stdout:3/733: write d6/de6/fea [430280,63069] 0 2024-04-16T02:35:43.607 INFO:tasks.workunit.client.1.smithi153.stdout:5/801: dwrite d8/df/d2f/d4e/f94 [0,4194304] 0 2024-04-16T02:35:43.609 INFO:tasks.workunit.client.0.smithi151.stdout:6/382: rename d2/d5/f60 to d2/d1f/d22/d40/d3f/f7f 0 2024-04-16T02:35:43.609 INFO:tasks.workunit.client.1.smithi153.stdout:5/802: dread - d8/d41/dbd/d39/f80 zero size 2024-04-16T02:35:43.610 INFO:tasks.workunit.client.1.smithi153.stdout:5/803: write d8/df/d2f/d4e/f94 [2164098,330] 0 2024-04-16T02:35:43.610 INFO:tasks.workunit.client.1.smithi153.stdout:4/858: creat d3/d9/f12c x:0 0 0 2024-04-16T02:35:43.611 INFO:tasks.workunit.client.1.smithi153.stdout:7/720: dread d0/d4/db/d2b/d53/f46 [0,4194304] 0 2024-04-16T02:35:43.611 INFO:tasks.workunit.client.0.smithi151.stdout:9/187: dwrite d1/d3/d6/d1d/d22/f23 [0,4194304] 0 2024-04-16T02:35:43.612 INFO:tasks.workunit.client.1.smithi153.stdout:5/804: chown d8/df/d2f/cb0 3454 1 2024-04-16T02:35:43.613 INFO:tasks.workunit.client.0.smithi151.stdout:3/163: symlink d1/d37/l3a 0 2024-04-16T02:35:43.614 INFO:tasks.workunit.client.0.smithi151.stdout:1/258: creat d3/d58/df/d38/f5a x:0 0 0 2024-04-16T02:35:43.616 INFO:tasks.workunit.client.1.smithi153.stdout:2/726: dwrite d9/d24/d27/d9a/f9f [4194304,4194304] 0 2024-04-16T02:35:43.616 INFO:tasks.workunit.client.0.smithi151.stdout:6/383: read d2/d5/f5a [267384,66015] 0 2024-04-16T02:35:43.616 INFO:tasks.workunit.client.0.smithi151.stdout:2/317: getdents de/d15 0 2024-04-16T02:35:43.617 INFO:tasks.workunit.client.1.smithi153.stdout:2/727: dread - d9/d24/d27/f92 zero size 2024-04-16T02:35:43.618 INFO:tasks.workunit.client.0.smithi151.stdout:8/165: mkdir d11/d2a/d35 0 2024-04-16T02:35:43.619 INFO:tasks.workunit.client.0.smithi151.stdout:2/318: chown l9 2117 1 2024-04-16T02:35:43.619 INFO:tasks.workunit.client.1.smithi153.stdout:4/859: symlink d3/d9/db/d17/d3f/l12d 0 2024-04-16T02:35:43.620 INFO:tasks.workunit.client.0.smithi151.stdout:1/259: readlink d3/d58/d1b/d1c/l45 0 2024-04-16T02:35:43.620 INFO:tasks.workunit.client.0.smithi151.stdout:2/319: dread - de/f5c zero size 2024-04-16T02:35:43.621 INFO:tasks.workunit.client.0.smithi151.stdout:7/229: dread d0/f7 [0,4194304] 0 2024-04-16T02:35:43.622 INFO:tasks.workunit.client.0.smithi151.stdout:5/308: dread d2/d6/f9 [0,4194304] 0 2024-04-16T02:35:43.622 INFO:tasks.workunit.client.0.smithi151.stdout:9/188: rmdir d1/d3/d10 39 2024-04-16T02:35:43.623 INFO:tasks.workunit.client.1.smithi153.stdout:7/721: rmdir d0/d4/db/d47/d8d/d93 39 2024-04-16T02:35:43.623 INFO:tasks.workunit.client.0.smithi151.stdout:8/166: write ff [5653619,130902] 0 2024-04-16T02:35:43.624 INFO:tasks.workunit.client.0.smithi151.stdout:3/164: mkdir d1/d6/d20/d3b 0 2024-04-16T02:35:43.626 INFO:tasks.workunit.client.1.smithi153.stdout:7/722: chown d0/d4/db/d2b/d8c/lc4 328203 1 2024-04-16T02:35:43.629 INFO:tasks.workunit.client.1.smithi153.stdout:3/734: dread d6/d10/d22/d4d/dbb/fc0 [0,4194304] 0 2024-04-16T02:35:43.630 INFO:tasks.workunit.client.1.smithi153.stdout:2/728: rename d9/d24/d27/d2c/d5d/f8c to d9/d2b/d1c/d20/d45/ff4 0 2024-04-16T02:35:43.630 INFO:tasks.workunit.client.0.smithi151.stdout:3/165: write d1/d6/d22/f2b [1004518,105614] 0 2024-04-16T02:35:43.631 INFO:tasks.workunit.client.1.smithi153.stdout:4/860: dwrite d3/d9/db/d45/df3/f95 [0,4194304] 0 2024-04-16T02:35:43.631 INFO:tasks.workunit.client.0.smithi151.stdout:9/189: fsync d1/dd/f35 0 2024-04-16T02:35:43.633 INFO:tasks.workunit.client.1.smithi153.stdout:6/763: write dc/d1a/f58 [660222,25036] 0 2024-04-16T02:35:43.638 INFO:tasks.workunit.client.0.smithi151.stdout:1/260: mknod d3/d58/d1b/d40/c5b 0 2024-04-16T02:35:43.640 INFO:tasks.workunit.client.1.smithi153.stdout:5/805: dwrite d8/df/f1e [0,4194304] 0 2024-04-16T02:35:43.640 INFO:tasks.workunit.client.1.smithi153.stdout:7/723: truncate d0/d4/db/d2b/f40 1927850 0 2024-04-16T02:35:43.641 INFO:tasks.workunit.client.1.smithi153.stdout:7/724: chown d0/f5b 28408580 1 2024-04-16T02:35:43.641 INFO:tasks.workunit.client.0.smithi151.stdout:2/320: unlink de/c10 0 2024-04-16T02:35:43.642 INFO:tasks.workunit.client.1.smithi153.stdout:5/806: write d8/f6c [948492,55081] 0 2024-04-16T02:35:43.643 INFO:tasks.workunit.client.1.smithi153.stdout:5/807: chown d8/df/d28/d5a/d72/f9e 0 1 2024-04-16T02:35:43.645 INFO:tasks.workunit.client.1.smithi153.stdout:2/729: link d9/f8b d9/d71/d8d/da8/ff5 0 2024-04-16T02:35:43.646 INFO:tasks.workunit.client.1.smithi153.stdout:2/730: dread - d9/d2b/f86 zero size 2024-04-16T02:35:43.647 INFO:tasks.workunit.client.0.smithi151.stdout:8/167: mknod d11/d2a/d35/c36 0 2024-04-16T02:35:43.648 INFO:tasks.workunit.client.0.smithi151.stdout:3/166: chown d1/d6/d20/d2a/f1a 261565732 1 2024-04-16T02:35:43.652 INFO:tasks.workunit.client.1.smithi153.stdout:4/861: link d3/d9/db/de/d2c/d38/l89 d3/d9/d24/d75/l12e 0 2024-04-16T02:35:43.652 INFO:tasks.workunit.client.0.smithi151.stdout:9/190: dwrite d1/d3/d6/f2a [0,4194304] 0 2024-04-16T02:35:43.653 INFO:tasks.workunit.client.0.smithi151.stdout:3/167: chown d1/d6/ff 4264094 1 2024-04-16T02:35:43.654 INFO:tasks.workunit.client.1.smithi153.stdout:2/731: creat d9/d24/d34/d60/ff6 x:0 0 0 2024-04-16T02:35:43.656 INFO:tasks.workunit.client.0.smithi151.stdout:3/168: creat d1/d37/f3c x:0 0 0 2024-04-16T02:35:43.657 INFO:tasks.workunit.client.1.smithi153.stdout:6/764: rename dc/d59/d63 to dc/d47/d57/d101 0 2024-04-16T02:35:43.658 INFO:tasks.workunit.client.0.smithi151.stdout:2/321: write de/f14 [1927005,87335] 0 2024-04-16T02:35:43.659 INFO:tasks.workunit.client.0.smithi151.stdout:5/309: dwrite d2/d6/d19/d30/d36/d42/f58 [0,4194304] 0 2024-04-16T02:35:43.660 INFO:tasks.workunit.client.0.smithi151.stdout:2/322: chown de/d15/c16 173 1 2024-04-16T02:35:43.660 INFO:tasks.workunit.client.0.smithi151.stdout:5/310: write d2/f5 [4674000,55258] 0 2024-04-16T02:35:43.662 INFO:tasks.workunit.client.1.smithi153.stdout:2/732: dwrite d9/d2b/d1c/d20/fde [0,4194304] 0 2024-04-16T02:35:43.663 INFO:tasks.workunit.client.0.smithi151.stdout:1/261: rename d3/l15 to d3/d58/d1b/d1c/l5c 0 2024-04-16T02:35:43.665 INFO:tasks.workunit.client.1.smithi153.stdout:4/862: mknod d3/d9/db/d17/d28/dc8/c12f 0 2024-04-16T02:35:43.667 INFO:tasks.workunit.client.1.smithi153.stdout:7/725: creat d0/d4/db/d2b/d43/d6d/d68/ff7 x:0 0 0 2024-04-16T02:35:43.668 INFO:tasks.workunit.client.1.smithi153.stdout:5/808: link d8/df/d28/d5a/d72/cac d8/df/d28/d5a/d66/c100 0 2024-04-16T02:35:43.669 INFO:tasks.workunit.client.1.smithi153.stdout:5/809: dread - d8/df/d28/d5a/d72/fff zero size 2024-04-16T02:35:43.669 INFO:tasks.workunit.client.0.smithi151.stdout:7/230: dread d0/d1f/d33/f36 [0,4194304] 0 2024-04-16T02:35:43.669 INFO:tasks.workunit.client.0.smithi151.stdout:5/311: mkdir d2/d6/d19/d62 0 2024-04-16T02:35:43.675 INFO:tasks.workunit.client.0.smithi151.stdout:3/169: rmdir d1/d6/d20/d3b 0 2024-04-16T02:35:43.677 INFO:tasks.workunit.client.1.smithi153.stdout:4/863: creat d3/d9/d24/d75/d9e/f130 x:0 0 0 2024-04-16T02:35:43.678 INFO:tasks.workunit.client.0.smithi151.stdout:9/191: link d1/dd/f35 d1/d3/f41 0 2024-04-16T02:35:43.680 INFO:tasks.workunit.client.1.smithi153.stdout:7/726: rename d0/d4/db/d34/d4b/d4d/f66 to d0/d4/db/d2b/d53/dbd/ff8 0 2024-04-16T02:35:43.681 INFO:tasks.workunit.client.0.smithi151.stdout:7/231: symlink d0/d1f/d33/l44 0 2024-04-16T02:35:43.681 INFO:tasks.workunit.client.0.smithi151.stdout:2/323: dwrite de/f14 [4194304,4194304] 0 2024-04-16T02:35:43.684 INFO:tasks.workunit.client.0.smithi151.stdout:1/262: dwrite d3/d33/d10/f4e [4194304,4194304] 0 2024-04-16T02:35:43.688 INFO:tasks.workunit.client.0.smithi151.stdout:2/324: write de/d11/d41/f38 [441468,87974] 0 2024-04-16T02:35:43.695 INFO:tasks.workunit.client.0.smithi151.stdout:6/384: dwrite d2/d5/df/f3c [0,4194304] 0 2024-04-16T02:35:43.695 INFO:tasks.workunit.client.1.smithi153.stdout:2/733: dwrite d9/f8b [0,4194304] 0 2024-04-16T02:35:43.697 INFO:tasks.workunit.client.1.smithi153.stdout:1/768: sync 2024-04-16T02:35:43.697 INFO:tasks.workunit.client.1.smithi153.stdout:0/859: sync 2024-04-16T02:35:43.700 INFO:tasks.workunit.client.0.smithi151.stdout:7/232: unlink d0/d1f/d31/f3f 0 2024-04-16T02:35:43.701 INFO:tasks.workunit.client.1.smithi153.stdout:7/727: rmdir d0/d4/db/d34/d4b 39 2024-04-16T02:35:43.702 INFO:tasks.workunit.client.0.smithi151.stdout:9/192: dread d1/d3/d10/f24 [0,4194304] 0 2024-04-16T02:35:43.703 INFO:tasks.workunit.client.0.smithi151.stdout:2/325: mknod de/d3d/c64 0 2024-04-16T02:35:43.703 INFO:tasks.workunit.client.0.smithi151.stdout:3/170: dwrite d1/d6/d22/f2b [0,4194304] 0 2024-04-16T02:35:43.713 INFO:tasks.workunit.client.1.smithi153.stdout:6/765: truncate dc/d1a/f58 3623750 0 2024-04-16T02:35:43.713 INFO:tasks.workunit.client.0.smithi151.stdout:3/171: mkdir d1/d6/d20/d3d 0 2024-04-16T02:35:43.715 INFO:tasks.workunit.client.0.smithi151.stdout:1/263: dwrite d3/d33/f36 [0,4194304] 0 2024-04-16T02:35:43.716 INFO:tasks.workunit.client.0.smithi151.stdout:8/168: dread d11/f15 [0,4194304] 0 2024-04-16T02:35:43.717 INFO:tasks.workunit.client.0.smithi151.stdout:2/326: mkdir de/d65 0 2024-04-16T02:35:43.719 INFO:tasks.workunit.client.0.smithi151.stdout:7/233: rename d0/d27/f32 to d0/d1f/d31/d42/f45 0 2024-04-16T02:35:43.720 INFO:tasks.workunit.client.0.smithi151.stdout:2/327: chown de/d3d 18542 1 2024-04-16T02:35:43.724 INFO:tasks.workunit.client.0.smithi151.stdout:9/193: dread d1/d3/d6/d17/f27 [0,4194304] 0 2024-04-16T02:35:43.725 INFO:tasks.workunit.client.0.smithi151.stdout:2/328: readlink de/d11/d41/l58 0 2024-04-16T02:35:43.725 INFO:tasks.workunit.client.0.smithi151.stdout:3/172: readlink d1/d6/l18 0 2024-04-16T02:35:43.725 INFO:tasks.workunit.client.0.smithi151.stdout:9/194: dread - d1/d3/d6/d17/f31 zero size 2024-04-16T02:35:43.726 INFO:tasks.workunit.client.0.smithi151.stdout:9/195: chown d1/d3/d6/d17/d40 845858 1 2024-04-16T02:35:43.727 INFO:tasks.workunit.client.0.smithi151.stdout:8/169: dread d11/f15 [0,4194304] 0 2024-04-16T02:35:43.728 INFO:tasks.workunit.client.0.smithi151.stdout:1/264: readlink d3/d58/l57 0 2024-04-16T02:35:43.729 INFO:tasks.workunit.client.0.smithi151.stdout:8/170: write d11/d2a/f2f [855479,83032] 0 2024-04-16T02:35:43.730 INFO:tasks.workunit.client.0.smithi151.stdout:9/196: symlink d1/d3/d6/l42 0 2024-04-16T02:35:43.732 INFO:tasks.workunit.client.0.smithi151.stdout:1/265: rename d3/d33/d10/f18 to d3/d33/d10/f5d 0 2024-04-16T02:35:43.733 INFO:tasks.workunit.client.0.smithi151.stdout:8/171: symlink d11/d2a/d33/l37 0 2024-04-16T02:35:43.735 INFO:tasks.workunit.client.0.smithi151.stdout:8/172: chown d11/d2a/f2f 99707 1 2024-04-16T02:35:43.737 INFO:tasks.workunit.client.0.smithi151.stdout:8/173: fsync d11/f17 0 2024-04-16T02:35:43.739 INFO:tasks.workunit.client.0.smithi151.stdout:7/234: dwrite d0/f3 [8388608,4194304] 0 2024-04-16T02:35:43.739 INFO:tasks.workunit.client.1.smithi153.stdout:4/864: dread d3/d9/db/de/fa3 [0,4194304] 0 2024-04-16T02:35:43.739 INFO:tasks.workunit.client.0.smithi151.stdout:3/173: dread d1/d6/d20/d2a/f1a [0,4194304] 0 2024-04-16T02:35:43.740 INFO:tasks.workunit.client.0.smithi151.stdout:9/197: dread d1/d3/d10/f24 [0,4194304] 0 2024-04-16T02:35:43.743 INFO:tasks.workunit.client.0.smithi151.stdout:8/174: unlink fe 0 2024-04-16T02:35:43.746 INFO:tasks.workunit.client.0.smithi151.stdout:1/266: dwrite d3/d33/d10/f55 [0,4194304] 0 2024-04-16T02:35:43.746 INFO:tasks.workunit.client.0.smithi151.stdout:7/235: dread d0/d1f/f28 [0,4194304] 0 2024-04-16T02:35:43.746 INFO:tasks.workunit.client.0.smithi151.stdout:3/174: chown d1/d6/d22/d23/d30 740 1 2024-04-16T02:35:43.748 INFO:tasks.workunit.client.0.smithi151.stdout:3/175: dread - d1/f1d zero size 2024-04-16T02:35:43.749 INFO:tasks.workunit.client.0.smithi151.stdout:5/312: truncate d2/d6/d43/f4a 3622052 0 2024-04-16T02:35:43.752 INFO:tasks.workunit.client.0.smithi151.stdout:9/198: mkdir d1/d3/d6/d28/d39/d43 0 2024-04-16T02:35:43.754 INFO:tasks.workunit.client.0.smithi151.stdout:9/199: chown d1/dd/f35 443 1 2024-04-16T02:35:43.755 INFO:tasks.workunit.client.0.smithi151.stdout:7/236: mkdir d0/d1f/d46 0 2024-04-16T02:35:43.758 INFO:tasks.workunit.client.0.smithi151.stdout:1/267: dwrite d3/d58/d1b/d2a/f30 [0,4194304] 0 2024-04-16T02:35:43.761 INFO:tasks.workunit.client.0.smithi151.stdout:3/176: fdatasync d1/d6/f1b 0 2024-04-16T02:35:43.762 INFO:tasks.workunit.client.0.smithi151.stdout:6/385: dwrite d2/d5/f5a [0,4194304] 0 2024-04-16T02:35:43.763 INFO:tasks.workunit.client.0.smithi151.stdout:9/200: chown d1/d3/d6/d1d/d22/c26 1811 1 2024-04-16T02:35:43.764 INFO:tasks.workunit.client.0.smithi151.stdout:9/201: readlink d1/d3/l14 0 2024-04-16T02:35:43.765 INFO:tasks.workunit.client.0.smithi151.stdout:6/386: truncate d2/d5/df/d23/d31/f43 906955 0 2024-04-16T02:35:43.766 INFO:tasks.workunit.client.0.smithi151.stdout:9/202: write d1/d3/d6/d1d/d22/f23 [658156,73325] 0 2024-04-16T02:35:43.768 INFO:tasks.workunit.client.0.smithi151.stdout:8/175: creat d11/f38 x:0 0 0 2024-04-16T02:35:43.772 INFO:tasks.workunit.client.0.smithi151.stdout:7/237: mknod d0/c47 0 2024-04-16T02:35:43.776 INFO:tasks.workunit.client.0.smithi151.stdout:8/176: dwrite d11/d22/f27 [0,4194304] 0 2024-04-16T02:35:43.776 INFO:tasks.workunit.client.0.smithi151.stdout:9/203: mkdir d1/d3/d6/d17/d44 0 2024-04-16T02:35:43.778 INFO:tasks.workunit.client.0.smithi151.stdout:8/177: chown d11/f1b 161 1 2024-04-16T02:35:43.780 INFO:tasks.workunit.client.0.smithi151.stdout:1/268: getdents d3/d33/d10/d4f 0 2024-04-16T02:35:43.782 INFO:tasks.workunit.client.0.smithi151.stdout:1/269: truncate d3/d58/df/d38/f5a 1048002 0 2024-04-16T02:35:43.782 INFO:tasks.workunit.client.0.smithi151.stdout:9/204: rmdir d1/d3/d6/d1d 39 2024-04-16T02:35:43.787 INFO:tasks.workunit.client.0.smithi151.stdout:8/178: symlink d11/d2a/d33/l39 0 2024-04-16T02:35:43.788 INFO:tasks.workunit.client.0.smithi151.stdout:5/313: truncate d2/fb 1764590 0 2024-04-16T02:35:43.788 INFO:tasks.workunit.client.1.smithi153.stdout:8/744: sync 2024-04-16T02:35:43.788 INFO:tasks.workunit.client.1.smithi153.stdout:9/803: sync 2024-04-16T02:35:43.790 INFO:tasks.workunit.client.1.smithi153.stdout:9/804: read d4/d33/f4f [567074,32928] 0 2024-04-16T02:35:43.790 INFO:tasks.workunit.client.0.smithi151.stdout:1/270: creat d3/d58/f5e x:0 0 0 2024-04-16T02:35:43.792 INFO:tasks.workunit.client.0.smithi151.stdout:8/179: symlink d11/d2a/d35/l3a 0 2024-04-16T02:35:43.793 INFO:tasks.workunit.client.0.smithi151.stdout:7/238: link d0/l20 d0/d1f/d31/l48 0 2024-04-16T02:35:43.794 INFO:tasks.workunit.client.0.smithi151.stdout:9/205: symlink d1/dd/l45 0 2024-04-16T02:35:43.794 INFO:tasks.workunit.client.0.smithi151.stdout:8/180: dread d11/f1f [0,4194304] 0 2024-04-16T02:35:43.795 INFO:tasks.workunit.client.1.smithi153.stdout:9/805: dwrite d4/d6/fbc [0,4194304] 0 2024-04-16T02:35:43.796 INFO:tasks.workunit.client.0.smithi151.stdout:9/206: dread - d1/d3/d6/f1b zero size 2024-04-16T02:35:43.797 INFO:tasks.workunit.client.0.smithi151.stdout:5/314: mkdir d2/d6/d19/d30/d36/d63 0 2024-04-16T02:35:43.798 INFO:tasks.workunit.client.0.smithi151.stdout:7/239: symlink d0/d2c/d38/l49 0 2024-04-16T02:35:43.799 INFO:tasks.workunit.client.0.smithi151.stdout:8/181: fdatasync d11/f15 0 2024-04-16T02:35:43.800 INFO:tasks.workunit.client.0.smithi151.stdout:7/240: write d0/d1f/f2a [416221,116332] 0 2024-04-16T02:35:43.806 INFO:tasks.workunit.client.0.smithi151.stdout:5/315: fsync d2/d6/f8 0 2024-04-16T02:35:43.809 INFO:tasks.workunit.client.0.smithi151.stdout:7/241: unlink d0/d35/f39 0 2024-04-16T02:35:43.810 INFO:tasks.workunit.client.0.smithi151.stdout:1/271: dread d3/d58/df/d32/f37 [0,4194304] 0 2024-04-16T02:35:43.812 INFO:tasks.workunit.client.0.smithi151.stdout:6/387: dread d2/d5/df/d20/f2b [0,4194304] 0 2024-04-16T02:35:43.812 INFO:tasks.workunit.client.0.smithi151.stdout:1/272: read d3/d58/df/d38/f5a [155500,5327] 0 2024-04-16T02:35:43.816 INFO:tasks.workunit.client.0.smithi151.stdout:6/388: dread d2/d1f/f59 [0,4194304] 0 2024-04-16T02:35:43.816 INFO:tasks.workunit.client.0.smithi151.stdout:6/389: readlink d2/l1c 0 2024-04-16T02:35:43.818 INFO:tasks.workunit.client.0.smithi151.stdout:7/242: mknod d0/d1f/d31/d42/c4a 0 2024-04-16T02:35:43.822 INFO:tasks.workunit.client.0.smithi151.stdout:9/207: dread d1/fc [0,4194304] 0 2024-04-16T02:35:43.822 INFO:tasks.workunit.client.1.smithi153.stdout:3/735: sync 2024-04-16T02:35:43.824 INFO:tasks.workunit.client.0.smithi151.stdout:6/390: rename d2/f1b to d2/d1f/f80 0 2024-04-16T02:35:43.827 INFO:tasks.workunit.client.1.smithi153.stdout:3/736: dwrite d6/d10/d3b/d59/d7c/de1/df0/def/ff9 [0,4194304] 0 2024-04-16T02:35:43.829 INFO:tasks.workunit.client.0.smithi151.stdout:1/273: symlink d3/d58/df/d32/d3a/l5f 0 2024-04-16T02:35:43.829 INFO:tasks.workunit.client.0.smithi151.stdout:6/391: dwrite d2/f1a [0,4194304] 0 2024-04-16T02:35:43.838 INFO:tasks.workunit.client.0.smithi151.stdout:7/243: symlink d0/l4b 0 2024-04-16T02:35:43.838 INFO:tasks.workunit.client.0.smithi151.stdout:8/182: mknod d11/d22/d2e/c3b 0 2024-04-16T02:35:43.839 INFO:tasks.workunit.client.0.smithi151.stdout:6/392: mkdir d2/d5/df/d20/d4f/d81 0 2024-04-16T02:35:43.839 INFO:tasks.workunit.client.0.smithi151.stdout:7/244: chown d0/d1f/d46 12900833 1 2024-04-16T02:35:43.840 INFO:tasks.workunit.client.0.smithi151.stdout:6/393: chown d2/d5/fb 761 1 2024-04-16T02:35:43.841 INFO:tasks.workunit.client.0.smithi151.stdout:7/245: write d0/f3 [11854908,106537] 0 2024-04-16T02:35:43.842 INFO:tasks.workunit.client.0.smithi151.stdout:8/183: write f2 [3545140,105959] 0 2024-04-16T02:35:43.842 INFO:tasks.workunit.client.1.smithi153.stdout:5/810: link d8/df/d2e/f3d d8/d41/d90/ded/f101 0 2024-04-16T02:35:43.843 INFO:tasks.workunit.client.1.smithi153.stdout:0/860: creat dd/d26/d44/dd2/f111 x:0 0 0 2024-04-16T02:35:43.843 INFO:tasks.workunit.client.1.smithi153.stdout:2/734: read - d9/d24/d27/d2c/d5d/da3/fa7 zero size 2024-04-16T02:35:43.843 INFO:tasks.workunit.client.1.smithi153.stdout:1/769: symlink d4/dd/d18/d6c/l108 0 2024-04-16T02:35:43.843 INFO:tasks.workunit.client.0.smithi151.stdout:7/246: mkdir d0/d1f/d4c 0 2024-04-16T02:35:43.844 INFO:tasks.workunit.client.0.smithi151.stdout:7/247: stat d0/d1f/f29 0 2024-04-16T02:35:43.845 INFO:tasks.workunit.client.0.smithi151.stdout:8/184: symlink d11/d2a/l3c 0 2024-04-16T02:35:43.845 INFO:tasks.workunit.client.0.smithi151.stdout:7/248: chown d0/f5 1 1 2024-04-16T02:35:43.845 INFO:tasks.workunit.client.1.smithi153.stdout:5/811: fdatasync d8/df/d28/d5a/d66/ffc 0 2024-04-16T02:35:43.846 INFO:tasks.workunit.client.0.smithi151.stdout:6/394: getdents d2/d5/df/d23 0 2024-04-16T02:35:43.847 INFO:tasks.workunit.client.1.smithi153.stdout:9/806: unlink d4/d6/d1b/d2a/l2b 0 2024-04-16T02:35:43.848 INFO:tasks.workunit.client.0.smithi151.stdout:8/185: unlink d11/c20 0 2024-04-16T02:35:43.848 INFO:tasks.workunit.client.1.smithi153.stdout:9/807: chown d4/d6/dc/d2d/da0/d69/fb7 13 1 2024-04-16T02:35:43.849 INFO:tasks.workunit.client.0.smithi151.stdout:7/249: rename d0/d1f/c23 to d0/d1f/d31/d42/c4d 0 2024-04-16T02:35:43.850 INFO:tasks.workunit.client.1.smithi153.stdout:6/766: symlink dc/d47/d57/d77/d15/dac/l102 0 2024-04-16T02:35:43.850 INFO:tasks.workunit.client.0.smithi151.stdout:6/395: mkdir d2/d5/df/d23/d82 0 2024-04-16T02:35:43.851 INFO:tasks.workunit.client.1.smithi153.stdout:4/865: creat d3/d9/d24/d3e/de1/f131 x:0 0 0 2024-04-16T02:35:43.854 INFO:tasks.workunit.client.0.smithi151.stdout:7/250: read d0/f1d [3429258,27236] 0 2024-04-16T02:35:43.854 INFO:tasks.workunit.client.1.smithi153.stdout:3/737: dread d6/d10/d22/d2c/f44 [0,4194304] 0 2024-04-16T02:35:43.854 INFO:tasks.workunit.client.1.smithi153.stdout:2/735: rmdir d9/d24/d73 39 2024-04-16T02:35:43.855 INFO:tasks.workunit.client.1.smithi153.stdout:8/745: creat d2/d13/d96/dc7/ffa x:0 0 0 2024-04-16T02:35:43.857 INFO:tasks.workunit.client.1.smithi153.stdout:1/770: creat d4/dd/d13/d26/db4/f109 x:0 0 0 2024-04-16T02:35:43.858 INFO:tasks.workunit.client.0.smithi151.stdout:8/186: link d11/f1f d11/d22/d2e/f3d 0 2024-04-16T02:35:43.859 INFO:tasks.workunit.client.0.smithi151.stdout:8/187: chown d11/f38 1 1 2024-04-16T02:35:43.859 INFO:tasks.workunit.client.0.smithi151.stdout:6/396: dwrite d2/d1f/f80 [0,4194304] 0 2024-04-16T02:35:43.862 INFO:tasks.workunit.client.1.smithi153.stdout:8/746: dwrite d2/d13/fbe [0,4194304] 0 2024-04-16T02:35:43.862 INFO:tasks.workunit.client.0.smithi151.stdout:7/251: symlink d0/d2c/d38/d3e/l4e 0 2024-04-16T02:35:43.864 INFO:tasks.workunit.client.1.smithi153.stdout:0/861: mknod dd/d14/dca/dcd/c112 0 2024-04-16T02:35:43.866 INFO:tasks.workunit.client.1.smithi153.stdout:6/767: chown dc/d47/d57/d77/d7a/d97/lc1 2067564 1 2024-04-16T02:35:43.866 INFO:tasks.workunit.client.0.smithi151.stdout:6/397: symlink d2/l83 0 2024-04-16T02:35:43.868 INFO:tasks.workunit.client.0.smithi151.stdout:9/208: truncate d1/d3/d6/f2a 2503858 0 2024-04-16T02:35:43.871 INFO:tasks.workunit.client.1.smithi153.stdout:7/728: link d0/d4/db/d34/d51/fb3 d0/d4/db/d34/d4b/d4d/ff9 0 2024-04-16T02:35:43.872 INFO:tasks.workunit.client.0.smithi151.stdout:7/252: mkdir d0/d1f/d46/d4f 0 2024-04-16T02:35:43.875 INFO:tasks.workunit.client.0.smithi151.stdout:8/188: dread d11/d22/d2e/f3d [0,4194304] 0 2024-04-16T02:35:43.878 INFO:tasks.workunit.client.0.smithi151.stdout:9/209: link d1/d3/d6/d17/c25 d1/d3/d6/d28/c46 0 2024-04-16T02:35:43.878 INFO:tasks.workunit.client.0.smithi151.stdout:6/398: dwrite d2/d5/fb [0,4194304] 0 2024-04-16T02:35:43.878 INFO:tasks.workunit.client.1.smithi153.stdout:1/771: mknod d4/dd/d13/d23/d5f/c10a 0 2024-04-16T02:35:43.879 INFO:tasks.workunit.client.0.smithi151.stdout:9/210: readlink d1/dd/d38/l3c 0 2024-04-16T02:35:43.880 INFO:tasks.workunit.client.0.smithi151.stdout:7/253: creat d0/d1f/d46/d4f/f50 x:0 0 0 2024-04-16T02:35:43.885 INFO:tasks.workunit.client.1.smithi153.stdout:6/768: stat dc/cea 0 2024-04-16T02:35:43.885 INFO:tasks.workunit.client.0.smithi151.stdout:9/211: dread d1/d3/f1a [4194304,4194304] 0 2024-04-16T02:35:43.886 INFO:tasks.workunit.client.0.smithi151.stdout:6/399: truncate d2/d5/f8 1941792 0 2024-04-16T02:35:43.886 INFO:tasks.workunit.client.1.smithi153.stdout:6/769: dread - dc/d6b/dba/ff0 zero size 2024-04-16T02:35:43.887 INFO:tasks.workunit.client.0.smithi151.stdout:9/212: write d1/d3/f1e [2899223,120190] 0 2024-04-16T02:35:43.887 INFO:tasks.workunit.client.0.smithi151.stdout:7/254: getdents d0/d1f/d4c 0 2024-04-16T02:35:43.889 INFO:tasks.workunit.client.0.smithi151.stdout:8/189: link c9 d11/c3e 0 2024-04-16T02:35:43.889 INFO:tasks.workunit.client.1.smithi153.stdout:3/738: symlink d6/d10/d22/d43/lfb 0 2024-04-16T02:35:43.892 INFO:tasks.workunit.client.0.smithi151.stdout:7/255: fdatasync d0/d1f/d46/d4f/f50 0 2024-04-16T02:35:43.892 INFO:tasks.workunit.client.0.smithi151.stdout:6/400: dread d2/d5/f27 [0,4194304] 0 2024-04-16T02:35:43.893 INFO:tasks.workunit.client.1.smithi153.stdout:9/808: creat d4/d6/dc/d2d/d64/ffd x:0 0 0 2024-04-16T02:35:43.893 INFO:tasks.workunit.client.1.smithi153.stdout:1/772: dread d4/da2/d65/d84/fa6 [0,4194304] 0 2024-04-16T02:35:43.894 INFO:tasks.workunit.client.1.smithi153.stdout:9/809: readlink d4/d6/lc4 0 2024-04-16T02:35:43.894 INFO:tasks.workunit.client.0.smithi151.stdout:8/190: creat d11/d2a/d35/f3f x:0 0 0 2024-04-16T02:35:43.894 INFO:tasks.workunit.client.0.smithi151.stdout:9/213: dread d1/f2e [0,4194304] 0 2024-04-16T02:35:43.895 INFO:tasks.workunit.client.1.smithi153.stdout:5/812: rmdir d8 39 2024-04-16T02:35:43.895 INFO:tasks.workunit.client.1.smithi153.stdout:1/773: write d4/dd/d13/d26/db4/ffb [736157,18610] 0 2024-04-16T02:35:43.895 INFO:tasks.workunit.client.0.smithi151.stdout:9/214: stat d1/d3/c8 0 2024-04-16T02:35:43.895 INFO:tasks.workunit.client.1.smithi153.stdout:1/774: chown d4/fc4 3 1 2024-04-16T02:35:43.896 INFO:tasks.workunit.client.0.smithi151.stdout:6/401: creat d2/d5/df/d23/d56/f84 x:0 0 0 2024-04-16T02:35:43.897 INFO:tasks.workunit.client.1.smithi153.stdout:0/862: symlink dd/d14/d51/d7f/d9d/l113 0 2024-04-16T02:35:43.897 INFO:tasks.workunit.client.0.smithi151.stdout:6/402: stat d2/d5/df/d23/d56/d72 0 2024-04-16T02:35:43.897 INFO:tasks.workunit.client.0.smithi151.stdout:8/191: creat d11/d2a/d35/f40 x:0 0 0 2024-04-16T02:35:43.898 INFO:tasks.workunit.client.1.smithi153.stdout:7/729: getdents d0/d4/db/d87/dc0/dea 0 2024-04-16T02:35:43.899 INFO:tasks.workunit.client.0.smithi151.stdout:8/192: write d11/f16 [4540227,103093] 0 2024-04-16T02:35:43.899 INFO:tasks.workunit.client.0.smithi151.stdout:6/403: fsync d2/d5/fb 0 2024-04-16T02:35:43.900 INFO:tasks.workunit.client.0.smithi151.stdout:7/256: mknod d0/d1f/d31/d42/c51 0 2024-04-16T02:35:43.901 INFO:tasks.workunit.client.1.smithi153.stdout:2/736: creat d9/d24/d34/ff7 x:0 0 0 2024-04-16T02:35:43.901 INFO:tasks.workunit.client.0.smithi151.stdout:8/193: chown d11/d2a/d33/l37 833 1 2024-04-16T02:35:43.902 INFO:tasks.workunit.client.0.smithi151.stdout:9/215: write d1/f2e [811605,40258] 0 2024-04-16T02:35:43.902 INFO:tasks.workunit.client.0.smithi151.stdout:8/194: fdatasync d11/d2a/d35/f3f 0 2024-04-16T02:35:43.903 INFO:tasks.workunit.client.0.smithi151.stdout:8/195: dread - d11/f38 zero size 2024-04-16T02:35:43.906 INFO:tasks.workunit.client.0.smithi151.stdout:7/257: creat d0/d1f/f52 x:0 0 0 2024-04-16T02:35:43.910 INFO:tasks.workunit.client.0.smithi151.stdout:7/258: readlink d0/d1f/d31/l48 0 2024-04-16T02:35:43.915 INFO:tasks.workunit.client.0.smithi151.stdout:8/196: getdents d11/d22/d2e 0 2024-04-16T02:35:43.917 INFO:tasks.workunit.client.0.smithi151.stdout:8/197: read d11/d2a/f2f [682407,95238] 0 2024-04-16T02:35:43.919 INFO:tasks.workunit.client.0.smithi151.stdout:7/259: getdents d0 0 2024-04-16T02:35:43.919 INFO:tasks.workunit.client.0.smithi151.stdout:9/216: truncate d1/d3/f1a 3104605 0 2024-04-16T02:35:43.920 INFO:tasks.workunit.client.0.smithi151.stdout:8/198: readlink d11/l19 0 2024-04-16T02:35:43.920 INFO:tasks.workunit.client.0.smithi151.stdout:6/404: dwrite d2/d1f/f59 [0,4194304] 0 2024-04-16T02:35:43.922 INFO:tasks.workunit.client.0.smithi151.stdout:8/199: write d11/f16 [5194681,6328] 0 2024-04-16T02:35:43.922 INFO:tasks.workunit.client.0.smithi151.stdout:6/405: chown d2/d13/d5f 5 1 2024-04-16T02:35:43.923 INFO:tasks.workunit.client.0.smithi151.stdout:6/406: chown d2/f71 1111974935 1 2024-04-16T02:35:43.923 INFO:tasks.workunit.client.0.smithi151.stdout:9/217: fsync d1/d3/d6/d1d/d22/f23 0 2024-04-16T02:35:43.924 INFO:tasks.workunit.client.0.smithi151.stdout:9/218: chown d1/l4 675303 1 2024-04-16T02:35:43.924 INFO:tasks.workunit.client.0.smithi151.stdout:6/407: chown d2/d5/df/d23/d31 9 1 2024-04-16T02:35:43.925 INFO:tasks.workunit.client.0.smithi151.stdout:9/219: fsync d1/d3/d6/d1d/d22/d2c/f2d 0 2024-04-16T02:35:43.927 INFO:tasks.workunit.client.0.smithi151.stdout:8/200: symlink d11/d22/l41 0 2024-04-16T02:35:43.928 INFO:tasks.workunit.client.0.smithi151.stdout:9/220: dread d1/fc [0,4194304] 0 2024-04-16T02:35:43.929 INFO:tasks.workunit.client.0.smithi151.stdout:7/260: link d0/d2c/d38/d3e/l4e d0/d1f/l53 0 2024-04-16T02:35:43.929 INFO:tasks.workunit.client.0.smithi151.stdout:9/221: read d1/fc [1816330,113308] 0 2024-04-16T02:35:43.930 INFO:tasks.workunit.client.0.smithi151.stdout:7/261: stat d0/d1f/d31/d42/f45 0 2024-04-16T02:35:43.932 INFO:tasks.workunit.client.0.smithi151.stdout:8/201: rename d11/f18 to d11/d2a/f42 0 2024-04-16T02:35:43.935 INFO:tasks.workunit.client.0.smithi151.stdout:7/262: unlink d0/d1f/d31/d42/c51 0 2024-04-16T02:35:43.936 INFO:tasks.workunit.client.0.smithi151.stdout:6/408: getdents d2/d5/df/d23/d31/d52 0 2024-04-16T02:35:43.937 INFO:tasks.workunit.client.0.smithi151.stdout:9/222: mknod d1/d3/d6/c47 0 2024-04-16T02:35:43.938 INFO:tasks.workunit.client.0.smithi151.stdout:6/409: mkdir d2/d1f/d22/d40/d3f/d85 0 2024-04-16T02:35:43.940 INFO:tasks.workunit.client.0.smithi151.stdout:8/202: creat d11/f43 x:0 0 0 2024-04-16T02:35:43.942 INFO:tasks.workunit.client.0.smithi151.stdout:9/223: fdatasync d1/d3/d10/f24 0 2024-04-16T02:35:43.943 INFO:tasks.workunit.client.0.smithi151.stdout:9/224: fdatasync d1/dd/f16 0 2024-04-16T02:35:43.943 INFO:tasks.workunit.client.0.smithi151.stdout:6/410: symlink d2/d5/df/d23/d31/d52/l86 0 2024-04-16T02:35:43.944 INFO:tasks.workunit.client.0.smithi151.stdout:9/225: chown d1/d3/d6/d17/l2f 29 1 2024-04-16T02:35:43.946 INFO:tasks.workunit.client.0.smithi151.stdout:9/226: creat d1/d3/d6/d28/d39/f48 x:0 0 0 2024-04-16T02:35:43.948 INFO:tasks.workunit.client.1.smithi153.stdout:1/775: mknod d4/dd/dd4/c10b 0 2024-04-16T02:35:43.949 INFO:tasks.workunit.client.1.smithi153.stdout:1/776: stat d4/dd/d18/d29/d71/f7b 0 2024-04-16T02:35:43.949 INFO:tasks.workunit.client.0.smithi151.stdout:9/227: chown d1/d3/d6/d17/l2f 42 1 2024-04-16T02:35:43.951 INFO:tasks.workunit.client.0.smithi151.stdout:6/411: dwrite d2/f71 [0,4194304] 0 2024-04-16T02:35:43.954 INFO:tasks.workunit.client.1.smithi153.stdout:9/810: mknod d4/d6/dc/d2d/d79/dbb/cfe 0 2024-04-16T02:35:43.955 INFO:tasks.workunit.client.1.smithi153.stdout:9/811: write d4/d6/dc/d2d/d64/ffd [660238,63075] 0 2024-04-16T02:35:43.956 INFO:tasks.workunit.client.0.smithi151.stdout:6/412: unlink d2/d5/df/d23/c2a 0 2024-04-16T02:35:43.956 INFO:tasks.workunit.client.1.smithi153.stdout:5/813: creat d8/df/d28/d5a/d66/d92/f102 x:0 0 0 2024-04-16T02:35:43.957 INFO:tasks.workunit.client.0.smithi151.stdout:7/263: truncate d0/d1f/f1e 100895 0 2024-04-16T02:35:43.960 INFO:tasks.workunit.client.1.smithi153.stdout:2/737: symlink d9/d2b/d5c/lf8 0 2024-04-16T02:35:43.960 INFO:tasks.workunit.client.1.smithi153.stdout:1/777: write d4/dd/d13/d26/db4/fcd [5022403,109262] 0 2024-04-16T02:35:43.962 INFO:tasks.workunit.client.1.smithi153.stdout:7/730: creat d0/d4/ffa x:0 0 0 2024-04-16T02:35:43.963 INFO:tasks.workunit.client.1.smithi153.stdout:9/812: symlink d4/d6/dc/d24/dc6/lff 0 2024-04-16T02:35:43.966 INFO:tasks.workunit.client.1.smithi153.stdout:1/778: write d4/da2/d65/fe2 [1010937,23326] 0 2024-04-16T02:35:43.967 INFO:tasks.workunit.client.1.smithi153.stdout:5/814: symlink d8/df/d28/d5a/l103 0 2024-04-16T02:35:43.968 INFO:tasks.workunit.client.1.smithi153.stdout:0/863: write dd/d14/d51/d54/d65/f99 [832096,3474] 0 2024-04-16T02:35:43.968 INFO:tasks.workunit.client.1.smithi153.stdout:8/747: dwrite d2/f9 [4194304,4194304] 0 2024-04-16T02:35:43.974 INFO:tasks.workunit.client.1.smithi153.stdout:5/815: chown d8/l7a 6826501 1 2024-04-16T02:35:43.975 INFO:tasks.workunit.client.1.smithi153.stdout:4/866: dwrite d3/f1c [0,4194304] 0 2024-04-16T02:35:43.977 INFO:tasks.workunit.client.1.smithi153.stdout:2/738: write d9/d24/d34/f7d [957124,125792] 0 2024-04-16T02:35:43.978 INFO:tasks.workunit.client.1.smithi153.stdout:7/731: rename d0/d4/db/d2b/f38 to d0/d4/db/d87/dc0/ffb 0 2024-04-16T02:35:43.981 INFO:tasks.workunit.client.1.smithi153.stdout:3/739: dwrite d6/d10/d22/f28 [0,4194304] 0 2024-04-16T02:35:43.982 INFO:tasks.workunit.client.1.smithi153.stdout:5/816: chown d8/c43 3190094 1 2024-04-16T02:35:43.984 INFO:tasks.workunit.client.1.smithi153.stdout:8/748: dread d2/d93/d97/fa9 [0,4194304] 0 2024-04-16T02:35:43.984 INFO:tasks.workunit.client.1.smithi153.stdout:0/864: rename dd/d14/da6/da4/ffb to dd/d26/d44/dd2/f114 0 2024-04-16T02:35:43.987 INFO:tasks.workunit.client.1.smithi153.stdout:4/867: symlink d3/d9/db/de/d2c/l132 0 2024-04-16T02:35:43.992 INFO:tasks.workunit.client.1.smithi153.stdout:3/740: symlink d6/d10/dec/ded/lfc 0 2024-04-16T02:35:43.993 INFO:tasks.workunit.client.1.smithi153.stdout:7/732: creat d0/d4/db/d34/d4b/ffc x:0 0 0 2024-04-16T02:35:43.994 INFO:tasks.workunit.client.1.smithi153.stdout:0/865: creat dd/f115 x:0 0 0 2024-04-16T02:35:43.996 INFO:tasks.workunit.client.1.smithi153.stdout:7/733: rmdir d0/d4/db/d2b/d53/dbd 39 2024-04-16T02:35:43.997 INFO:tasks.workunit.client.1.smithi153.stdout:0/866: truncate dd/d14/d19/d55/fdf 303860 0 2024-04-16T02:35:43.997 INFO:tasks.workunit.client.1.smithi153.stdout:4/868: dread d3/d9/db/de/f13 [0,4194304] 0 2024-04-16T02:35:43.997 INFO:tasks.workunit.client.1.smithi153.stdout:3/741: rename d6/d10/d22/d2e/c36 to d6/d10/d3b/dd7/cfd 0 2024-04-16T02:35:43.998 INFO:tasks.workunit.client.1.smithi153.stdout:7/734: mknod d0/d4/db/d34/d4b/cfd 0 2024-04-16T02:35:43.999 INFO:tasks.workunit.client.1.smithi153.stdout:0/867: dread dd/d26/d44/fd0 [0,4194304] 0 2024-04-16T02:35:44.000 INFO:tasks.workunit.client.1.smithi153.stdout:7/735: truncate d0/d4/db/d34/d4b/ffc 82190 0 2024-04-16T02:35:44.001 INFO:tasks.workunit.client.1.smithi153.stdout:0/868: mknod dd/d14/d19/d20/d4a/c116 0 2024-04-16T02:35:44.003 INFO:tasks.workunit.client.1.smithi153.stdout:4/869: getdents d3/d9/db/de/d2c/d52 0 2024-04-16T02:35:44.004 INFO:tasks.workunit.client.1.smithi153.stdout:0/869: symlink dd/d14/l117 0 2024-04-16T02:35:44.006 INFO:tasks.workunit.client.1.smithi153.stdout:4/870: rmdir d3/d9/db/d17/d28/dc8/de4/dfd/de3 39 2024-04-16T02:35:44.009 INFO:tasks.workunit.client.1.smithi153.stdout:7/736: dwrite d0/d4/db/d16/fb1 [0,4194304] 0 2024-04-16T02:35:44.014 INFO:tasks.workunit.client.1.smithi153.stdout:7/737: unlink d0/d4/db/d47/d8d/d93/l70 0 2024-04-16T02:35:44.018 INFO:tasks.workunit.client.1.smithi153.stdout:1/779: write d4/dd/d18/d6c/dd8/fd0 [1339548,107813] 0 2024-04-16T02:35:44.020 INFO:tasks.workunit.client.1.smithi153.stdout:9/813: truncate d4/d6/dc/d24/d4e/f9c 1967302 0 2024-04-16T02:35:44.020 INFO:tasks.workunit.client.1.smithi153.stdout:1/780: readlink d4/da2/d65/d82/la5 0 2024-04-16T02:35:44.021 INFO:tasks.workunit.client.1.smithi153.stdout:9/814: chown d4/d6/d1b/f90 155307 1 2024-04-16T02:35:44.023 INFO:tasks.workunit.client.1.smithi153.stdout:2/739: truncate d9/d2b/f16 3137437 0 2024-04-16T02:35:44.025 INFO:tasks.workunit.client.1.smithi153.stdout:8/749: truncate d2/d13/d96/fb3 2497581 0 2024-04-16T02:35:44.026 INFO:tasks.workunit.client.1.smithi153.stdout:2/740: truncate d9/d2b/d5c/fd2 675912 0 2024-04-16T02:35:44.057 INFO:tasks.workunit.client.0.smithi151.stdout:9/228: symlink d1/d3/d6/d17/d40/l49 0 2024-04-16T02:35:44.059 INFO:tasks.workunit.client.0.smithi151.stdout:6/413: mkdir d2/d5/df/d87 0 2024-04-16T02:35:44.060 INFO:tasks.workunit.client.0.smithi151.stdout:6/414: write d2/d3d/f55 [1139933,65263] 0 2024-04-16T02:35:44.060 INFO:tasks.workunit.client.0.smithi151.stdout:6/415: stat d2/d1f/f80 0 2024-04-16T02:35:44.062 INFO:tasks.workunit.client.0.smithi151.stdout:6/416: mknod d2/d1f/d22/c88 0 2024-04-16T02:35:44.065 INFO:tasks.workunit.client.0.smithi151.stdout:6/417: rename d2/d5/df/d23/d56/c6b to d2/d13/d5f/c89 0 2024-04-16T02:35:44.067 INFO:tasks.workunit.client.1.smithi153.stdout:6/770: sync 2024-04-16T02:35:44.067 INFO:tasks.workunit.client.0.smithi151.stdout:6/418: creat d2/d5/df/d23/d56/f8a x:0 0 0 2024-04-16T02:35:44.068 INFO:tasks.workunit.client.0.smithi151.stdout:9/229: dread d1/d3/f1a [0,4194304] 0 2024-04-16T02:35:44.068 INFO:tasks.workunit.client.1.smithi153.stdout:3/742: truncate d6/f39 3835414 0 2024-04-16T02:35:44.070 INFO:tasks.workunit.client.1.smithi153.stdout:0/870: truncate dd/d14/d19/f46 1329639 0 2024-04-16T02:35:44.070 INFO:tasks.workunit.client.1.smithi153.stdout:9/815: dwrite d4/d6/dc/d7f/fe9 [0,4194304] 0 2024-04-16T02:35:44.071 INFO:tasks.workunit.client.1.smithi153.stdout:4/871: truncate d3/d9/d24/d75/dc5/fdb 18651 0 2024-04-16T02:35:44.073 INFO:tasks.workunit.client.0.smithi151.stdout:6/419: rename d2/d5/fb to d2/d5/df/d23/d56/f8b 0 2024-04-16T02:35:44.073 INFO:tasks.workunit.client.1.smithi153.stdout:4/872: fsync d3/da5/dd2/f127 0 2024-04-16T02:35:44.074 INFO:tasks.workunit.client.0.smithi151.stdout:9/230: mkdir d1/dd/d4a 0 2024-04-16T02:35:44.074 INFO:tasks.workunit.client.0.smithi151.stdout:6/420: write d2/d5/df/d23/d31/f43 [1049951,15769] 0 2024-04-16T02:35:44.076 INFO:tasks.workunit.client.1.smithi153.stdout:8/750: mknod d2/d92/deb/d47/d60/d67/d98/cfb 0 2024-04-16T02:35:44.076 INFO:tasks.workunit.client.1.smithi153.stdout:4/873: write d3/d9/db/d45/df3/ffc [396195,49041] 0 2024-04-16T02:35:44.078 INFO:tasks.workunit.client.0.smithi151.stdout:6/421: creat d2/d5/df/d23/f8c x:0 0 0 2024-04-16T02:35:44.079 INFO:tasks.workunit.client.1.smithi153.stdout:8/751: readlink d2/d13/d3a/l76 0 2024-04-16T02:35:44.079 INFO:tasks.workunit.client.1.smithi153.stdout:2/741: unlink d9/d24/d27/d2c/d5d/da3/lcb 0 2024-04-16T02:35:44.079 INFO:tasks.workunit.client.1.smithi153.stdout:0/871: truncate dd/d32/f96 872102 0 2024-04-16T02:35:44.083 INFO:tasks.workunit.client.0.smithi151.stdout:6/422: rename d2/d1f to d2/d5/df/d23/d82/d8d 0 2024-04-16T02:35:44.083 INFO:tasks.workunit.client.1.smithi153.stdout:1/781: dread d4/dd/d13/d23/fd9 [0,4194304] 0 2024-04-16T02:35:44.084 INFO:tasks.workunit.client.1.smithi153.stdout:1/782: fdatasync d4/dd/d13/d23/fab 0 2024-04-16T02:35:44.085 INFO:tasks.workunit.client.1.smithi153.stdout:1/783: write d4/dd/d18/d6c/dd8/fe1 [989722,85896] 0 2024-04-16T02:35:44.085 INFO:tasks.workunit.client.0.smithi151.stdout:6/423: mknod d2/d5/df/d87/c8e 0 2024-04-16T02:35:44.088 INFO:tasks.workunit.client.0.smithi151.stdout:9/231: write d1/d3/f1a [3635254,108939] 0 2024-04-16T02:35:44.092 INFO:tasks.workunit.client.0.smithi151.stdout:6/424: dwrite d2/d5/df/d23/d56/f58 [0,4194304] 0 2024-04-16T02:35:44.093 INFO:tasks.workunit.client.0.smithi151.stdout:9/232: dread d1/dd/f16 [0,4194304] 0 2024-04-16T02:35:44.094 INFO:tasks.workunit.client.1.smithi153.stdout:3/743: dread d6/d10/d22/f2a [0,4194304] 0 2024-04-16T02:35:44.095 INFO:tasks.workunit.client.0.smithi151.stdout:9/233: creat d1/d3/d10/f4b x:0 0 0 2024-04-16T02:35:44.095 INFO:tasks.workunit.client.0.smithi151.stdout:9/234: dread - d1/dd/f35 zero size 2024-04-16T02:35:44.098 INFO:tasks.workunit.client.0.smithi151.stdout:9/235: truncate d1/dd/f16 1466535 0 2024-04-16T02:35:44.099 INFO:tasks.workunit.client.0.smithi151.stdout:9/236: stat d1/d3/d10/d20 0 2024-04-16T02:35:44.101 INFO:tasks.workunit.client.0.smithi151.stdout:9/237: mknod d1/dd/d38/c4c 0 2024-04-16T02:35:44.102 INFO:tasks.workunit.client.0.smithi151.stdout:9/238: creat d1/d3/d10/f4d x:0 0 0 2024-04-16T02:35:44.106 INFO:tasks.workunit.client.0.smithi151.stdout:9/239: getdents d1/d3/d6/d1d/d22/d36 0 2024-04-16T02:35:44.108 INFO:tasks.workunit.client.0.smithi151.stdout:9/240: symlink d1/d3/d6/d1d/l4e 0 2024-04-16T02:35:44.112 INFO:tasks.workunit.client.0.smithi151.stdout:9/241: write d1/dd/f35 [750671,70854] 0 2024-04-16T02:35:44.115 INFO:tasks.workunit.client.0.smithi151.stdout:0/250: sync 2024-04-16T02:35:44.115 INFO:tasks.workunit.client.0.smithi151.stdout:4/288: sync 2024-04-16T02:35:44.116 INFO:tasks.workunit.client.0.smithi151.stdout:9/242: rmdir d1/d3 39 2024-04-16T02:35:44.116 INFO:tasks.workunit.client.0.smithi151.stdout:0/251: fdatasync dc/d20/d2c/f3e 0 2024-04-16T02:35:44.117 INFO:tasks.workunit.client.0.smithi151.stdout:4/289: mknod d0/d11/d19/d31/c5e 0 2024-04-16T02:35:44.117 INFO:tasks.workunit.client.0.smithi151.stdout:9/243: readlink d1/d3/l15 0 2024-04-16T02:35:44.118 INFO:tasks.workunit.client.0.smithi151.stdout:0/252: mkdir dc/d15/d34/d4e 0 2024-04-16T02:35:44.119 INFO:tasks.workunit.client.0.smithi151.stdout:0/253: dread - dc/f47 zero size 2024-04-16T02:35:44.146 INFO:tasks.workunit.client.1.smithi153.stdout:5/817: sync 2024-04-16T02:35:44.158 INFO:tasks.workunit.client.0.smithi151.stdout:4/290: symlink d0/d11/d19/d31/d45/l5f 0 2024-04-16T02:35:44.158 INFO:tasks.workunit.client.0.smithi151.stdout:9/244: getdents d1/d3/d6/d1d/d22/d36 0 2024-04-16T02:35:44.158 INFO:tasks.workunit.client.0.smithi151.stdout:0/254: mknod dc/d20/c4f 0 2024-04-16T02:35:44.159 INFO:tasks.workunit.client.0.smithi151.stdout:0/255: write dc/d3c/f48 [536662,78715] 0 2024-04-16T02:35:44.159 INFO:tasks.workunit.client.0.smithi151.stdout:4/291: mkdir d0/d11/d19/d31/d4a/d60 0 2024-04-16T02:35:44.161 INFO:tasks.workunit.client.1.smithi153.stdout:9/816: rename d4/d6/dc/d7f/fa3 to d4/d6/d1b/d6e/f100 0 2024-04-16T02:35:44.162 INFO:tasks.workunit.client.0.smithi151.stdout:0/256: rename dc/f37 to dc/d15/f50 0 2024-04-16T02:35:44.163 INFO:tasks.workunit.client.0.smithi151.stdout:4/292: unlink d0/d11/d19/d31/d34/d3a/f4e 0 2024-04-16T02:35:44.164 INFO:tasks.workunit.client.1.smithi153.stdout:6/771: symlink dc/d47/d57/d77/d15/l103 0 2024-04-16T02:35:44.165 INFO:tasks.workunit.client.1.smithi153.stdout:6/772: fdatasync dc/d1a/f1c 0 2024-04-16T02:35:44.165 INFO:tasks.workunit.client.0.smithi151.stdout:0/257: mkdir dc/d20/d2c/d51 0 2024-04-16T02:35:44.166 INFO:tasks.workunit.client.1.smithi153.stdout:9/817: dwrite d4/d6/f9d [0,4194304] 0 2024-04-16T02:35:44.166 INFO:tasks.workunit.client.1.smithi153.stdout:6/773: dread - dc/d47/d57/d77/d7a/d97/db7/ffd zero size 2024-04-16T02:35:44.167 INFO:tasks.workunit.client.1.smithi153.stdout:9/818: readlink d4/d6/def/lf6 0 2024-04-16T02:35:44.167 INFO:tasks.workunit.client.0.smithi151.stdout:0/258: mknod dc/d15/d34/d3b/c52 0 2024-04-16T02:35:44.168 INFO:tasks.workunit.client.0.smithi151.stdout:0/259: dread - dc/d15/f29 zero size 2024-04-16T02:35:44.169 INFO:tasks.workunit.client.0.smithi151.stdout:0/260: fsync dc/d15/f2b 0 2024-04-16T02:35:44.171 INFO:tasks.workunit.client.0.smithi151.stdout:0/261: read dc/d20/f42 [98650,91362] 0 2024-04-16T02:35:44.173 INFO:tasks.workunit.client.1.smithi153.stdout:7/738: sync 2024-04-16T02:35:44.174 INFO:tasks.workunit.client.1.smithi153.stdout:7/739: dread - d0/d4/db/d34/d4b/daa/fc9 zero size 2024-04-16T02:35:44.181 INFO:tasks.workunit.client.0.smithi151.stdout:4/293: truncate d0/f15 3163097 0 2024-04-16T02:35:44.182 INFO:tasks.workunit.client.0.smithi151.stdout:4/294: mkdir d0/d11/d19/d31/d34/d3a/d61 0 2024-04-16T02:35:44.183 INFO:tasks.workunit.client.1.smithi153.stdout:6/774: dread dc/d1a/d2d/f44 [0,4194304] 0 2024-04-16T02:35:44.184 INFO:tasks.workunit.client.0.smithi151.stdout:4/295: mknod d0/d11/c62 0 2024-04-16T02:35:44.186 INFO:tasks.workunit.client.0.smithi151.stdout:4/296: symlink d0/d11/d19/d31/d34/l63 0 2024-04-16T02:35:44.187 INFO:tasks.workunit.client.0.smithi151.stdout:4/297: fsync d0/d11/d19/f44 0 2024-04-16T02:35:44.197 INFO:tasks.workunit.client.1.smithi153.stdout:1/784: chown d4/dd/d13/c47 1398124 1 2024-04-16T02:35:44.198 INFO:tasks.workunit.client.1.smithi153.stdout:3/744: mknod d6/d10/d3b/d59/d7c/de1/df0/cfe 0 2024-04-16T02:35:44.200 INFO:tasks.workunit.client.1.smithi153.stdout:2/742: mkdir d9/d2b/df9 0 2024-04-16T02:35:44.201 INFO:tasks.workunit.client.0.smithi151.stdout:4/298: dwrite d0/f15 [0,4194304] 0 2024-04-16T02:35:44.202 INFO:tasks.workunit.client.1.smithi153.stdout:5/818: dread d8/d41/dbd/d39/d6d/f71 [0,4194304] 0 2024-04-16T02:35:44.205 INFO:tasks.workunit.client.1.smithi153.stdout:7/740: creat d0/de8/ffe x:0 0 0 2024-04-16T02:35:44.205 INFO:tasks.workunit.client.1.smithi153.stdout:6/775: mkdir dc/d59/d104 0 2024-04-16T02:35:44.207 INFO:tasks.workunit.client.1.smithi153.stdout:0/872: dwrite dd/d32/d57/f61 [0,4194304] 0 2024-04-16T02:35:44.207 INFO:tasks.workunit.client.0.smithi151.stdout:4/299: dwrite d0/d11/d19/d46/f58 [0,4194304] 0 2024-04-16T02:35:44.208 INFO:tasks.workunit.client.1.smithi153.stdout:8/752: mkdir d2/d92/deb/d47/d77/ddf/dfc 0 2024-04-16T02:35:44.211 INFO:tasks.workunit.client.0.smithi151.stdout:4/300: mkdir d0/d64 0 2024-04-16T02:35:44.211 INFO:tasks.workunit.client.1.smithi153.stdout:8/753: chown d2/d1c/f26 100 1 2024-04-16T02:35:44.212 INFO:tasks.workunit.client.1.smithi153.stdout:1/785: creat d4/dd/d13/d23/d5f/f10c x:0 0 0 2024-04-16T02:35:44.214 INFO:tasks.workunit.client.0.smithi151.stdout:4/301: chown d0/d11/d19/d31/d34/d3a/c56 249734925 1 2024-04-16T02:35:44.214 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:43 smithi153 ceph-mon[36747]: pgmap v19: 65 pgs: 65 active+clean; 2.0 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 65 MiB/s rd, 293 MiB/s wr, 560 op/s 2024-04-16T02:35:44.216 INFO:tasks.workunit.client.1.smithi153.stdout:4/874: creat d3/d9/db/de/d2c/d52/d85/f133 x:0 0 0 2024-04-16T02:35:44.217 INFO:tasks.workunit.client.1.smithi153.stdout:2/743: dwrite d9/d2b/d18/fe4 [0,4194304] 0 2024-04-16T02:35:44.217 INFO:tasks.workunit.client.0.smithi151.stdout:4/302: mkdir d0/d11/d19/d31/d34/d3a/d65 0 2024-04-16T02:35:44.218 INFO:tasks.workunit.client.0.smithi151.stdout:4/303: truncate d0/f8 4742968 0 2024-04-16T02:35:44.219 INFO:tasks.workunit.client.1.smithi153.stdout:5/819: fdatasync d8/df/d2f/d4e/d51/f5c 0 2024-04-16T02:35:44.220 INFO:tasks.workunit.client.1.smithi153.stdout:9/819: truncate d4/d6/f56 1264113 0 2024-04-16T02:35:44.221 INFO:tasks.workunit.client.1.smithi153.stdout:9/820: chown d4/d6/dc/d24/d60 310392 1 2024-04-16T02:35:44.222 INFO:tasks.workunit.client.1.smithi153.stdout:7/741: write d0/d4/db/d34/d4b/d4d/ff4 [366473,75740] 0 2024-04-16T02:35:44.223 INFO:tasks.workunit.client.1.smithi153.stdout:9/821: write d4/d6/ddb/d4a/f80 [273664,4406] 0 2024-04-16T02:35:44.224 INFO:tasks.workunit.client.1.smithi153.stdout:9/822: chown d4/d6/d1b/d6e 93672 1 2024-04-16T02:35:44.234 INFO:tasks.workunit.client.1.smithi153.stdout:5/820: readlink d8/df/d2e/dd1/lcf 0 2024-04-16T02:35:44.238 INFO:tasks.workunit.client.1.smithi153.stdout:7/742: write d0/d4/db/d47/f94 [2879421,126326] 0 2024-04-16T02:35:44.239 INFO:tasks.workunit.client.1.smithi153.stdout:3/745: link d6/d10/d3b/f6d d6/d10/d3b/d59/d7c/fff 0 2024-04-16T02:35:44.243 INFO:tasks.workunit.client.1.smithi153.stdout:5/821: dwrite d8/df/d28/f8a [4194304,4194304] 0 2024-04-16T02:35:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:43 smithi151 ceph-mon[29003]: pgmap v19: 65 pgs: 65 active+clean; 2.0 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 65 MiB/s rd, 293 MiB/s wr, 560 op/s 2024-04-16T02:35:44.246 INFO:tasks.workunit.client.1.smithi153.stdout:0/873: rename dd/d32/d57/lf2 to dd/d14/d19/l118 0 2024-04-16T02:35:44.248 INFO:tasks.workunit.client.1.smithi153.stdout:7/743: creat d0/d4/db/d2b/d8c/d9f/fff x:0 0 0 2024-04-16T02:35:44.249 INFO:tasks.workunit.client.0.smithi151.stdout:4/304: dwrite d0/d11/f23 [0,4194304] 0 2024-04-16T02:35:44.252 INFO:tasks.workunit.client.1.smithi153.stdout:8/754: link d2/d92/deb/d47/d60/f7b d2/d13/d2c/ffd 0 2024-04-16T02:35:44.253 INFO:tasks.workunit.client.1.smithi153.stdout:4/875: dwrite d3/d9/db/d45/f7f [0,4194304] 0 2024-04-16T02:35:44.253 INFO:tasks.workunit.client.1.smithi153.stdout:0/874: write dd/d32/f85 [399657,86811] 0 2024-04-16T02:35:44.254 INFO:tasks.workunit.client.1.smithi153.stdout:1/786: creat d4/dd/d13/d26/f10d x:0 0 0 2024-04-16T02:35:44.254 INFO:tasks.workunit.client.1.smithi153.stdout:6/776: getdents dc/d47/d80 0 2024-04-16T02:35:44.256 INFO:tasks.workunit.client.1.smithi153.stdout:9/823: rmdir d4/d33/dae/dc7/db0 0 2024-04-16T02:35:44.257 INFO:tasks.workunit.client.1.smithi153.stdout:2/744: rename d9/d2b/d1c/l21 to d9/d24/d27/d2c/d5d/lfa 0 2024-04-16T02:35:44.258 INFO:tasks.workunit.client.1.smithi153.stdout:5/822: chown d8/f4d 108 1 2024-04-16T02:35:44.259 INFO:tasks.workunit.client.1.smithi153.stdout:2/745: chown d9/d24/d27/d9a/fb9 21 1 2024-04-16T02:35:44.260 INFO:tasks.workunit.client.1.smithi153.stdout:4/876: unlink d3/d9/db/de/f3b 0 2024-04-16T02:35:44.264 INFO:tasks.workunit.client.1.smithi153.stdout:2/746: stat d9/d24/d34/fce 0 2024-04-16T02:35:44.265 INFO:tasks.workunit.client.1.smithi153.stdout:0/875: write dd/d14/d51/f59 [890878,116560] 0 2024-04-16T02:35:44.266 INFO:tasks.workunit.client.1.smithi153.stdout:5/823: readlink d8/df/d28/d5a/d5e/l95 0 2024-04-16T02:35:44.267 INFO:tasks.workunit.client.1.smithi153.stdout:9/824: chown d4/d6/dc/d2d/d64/db9/dc1 475246 1 2024-04-16T02:35:44.268 INFO:tasks.workunit.client.1.smithi153.stdout:0/876: stat dd/d28/f3c 0 2024-04-16T02:35:44.269 INFO:tasks.workunit.client.1.smithi153.stdout:4/877: truncate d3/d9/d24/d75/d9e/fce 701863 0 2024-04-16T02:35:44.269 INFO:tasks.workunit.client.1.smithi153.stdout:7/744: dwrite d0/d4/db/d2b/d53/d59/f67 [0,4194304] 0 2024-04-16T02:35:44.270 INFO:tasks.workunit.client.0.smithi151.stdout:4/305: dwrite d0/d11/f43 [0,4194304] 0 2024-04-16T02:35:44.270 INFO:tasks.workunit.client.1.smithi153.stdout:5/824: write d8/fde [361282,103213] 0 2024-04-16T02:35:44.272 INFO:tasks.workunit.client.1.smithi153.stdout:1/787: getdents d4/dd/d18/d6c 0 2024-04-16T02:35:44.272 INFO:tasks.workunit.client.1.smithi153.stdout:9/825: chown d4/d6/d1b/d2a/l7a 3580 1 2024-04-16T02:35:44.275 INFO:tasks.workunit.client.0.smithi151.stdout:4/306: creat d0/d20/f66 x:0 0 0 2024-04-16T02:35:44.276 INFO:tasks.workunit.client.0.smithi151.stdout:4/307: write d0/f55 [317807,13965] 0 2024-04-16T02:35:44.281 INFO:tasks.workunit.client.1.smithi153.stdout:0/877: creat dd/d14/da6/da4/db1/f119 x:0 0 0 2024-04-16T02:35:44.283 INFO:tasks.workunit.client.1.smithi153.stdout:9/826: mkdir d4/d6/ddb/d4a/d101 0 2024-04-16T02:35:44.283 INFO:tasks.workunit.client.1.smithi153.stdout:7/745: unlink d0/d4/db/d34/d4b/ffc 0 2024-04-16T02:35:44.284 INFO:tasks.workunit.client.1.smithi153.stdout:6/777: write dc/d47/f7e [274002,30828] 0 2024-04-16T02:35:44.284 INFO:tasks.workunit.client.1.smithi153.stdout:9/827: stat d4/d6/ddb/d4a 0 2024-04-16T02:35:44.286 INFO:tasks.workunit.client.1.smithi153.stdout:4/878: truncate d3/d9/f8d 333696 0 2024-04-16T02:35:44.286 INFO:tasks.workunit.client.1.smithi153.stdout:3/746: dwrite d6/d10/d22/d2e/f80 [0,4194304] 0 2024-04-16T02:35:44.287 INFO:tasks.workunit.client.1.smithi153.stdout:3/747: chown d6 110615800 1 2024-04-16T02:35:44.289 INFO:tasks.workunit.client.1.smithi153.stdout:5/825: rename d8/df/d2e/lf5 to d8/d41/d90/ded/l104 0 2024-04-16T02:35:44.290 INFO:tasks.workunit.client.1.smithi153.stdout:2/747: getdents d9/d2b/d1c/d20 0 2024-04-16T02:35:44.293 INFO:tasks.workunit.client.1.smithi153.stdout:6/778: dread - dc/d1a/f3a zero size 2024-04-16T02:35:44.295 INFO:tasks.workunit.client.1.smithi153.stdout:3/748: creat d6/d10/d3b/d59/d7c/de1/df0/f100 x:0 0 0 2024-04-16T02:35:44.295 INFO:tasks.workunit.client.0.smithi151.stdout:4/308: dwrite d0/d17/f1d [0,4194304] 0 2024-04-16T02:35:44.297 INFO:tasks.workunit.client.1.smithi153.stdout:3/749: dread - d6/d10/d22/d2e/d8a/ff7 zero size 2024-04-16T02:35:44.298 INFO:tasks.workunit.client.1.smithi153.stdout:5/826: unlink d8/df/d2e/f79 0 2024-04-16T02:35:44.298 INFO:tasks.workunit.client.0.smithi151.stdout:4/309: creat d0/d11/d19/d31/d34/d3a/d50/f67 x:0 0 0 2024-04-16T02:35:44.299 INFO:tasks.workunit.client.1.smithi153.stdout:7/746: fsync d0/d4/db/d2b/d53/d59/dc3/fe6 0 2024-04-16T02:35:44.299 INFO:tasks.workunit.client.0.smithi151.stdout:4/310: mkdir d0/d11/d19/d31/d68 0 2024-04-16T02:35:44.301 INFO:tasks.workunit.client.1.smithi153.stdout:2/748: unlink d9/d24/d27/f1f 0 2024-04-16T02:35:44.301 INFO:tasks.workunit.client.1.smithi153.stdout:2/749: readlink d9/d71/la1 0 2024-04-16T02:35:44.302 INFO:tasks.workunit.client.0.smithi151.stdout:4/311: symlink d0/d11/d19/d31/d34/d3a/d61/l69 0 2024-04-16T02:35:44.303 INFO:tasks.workunit.client.0.smithi151.stdout:4/312: mknod d0/d20/c6a 0 2024-04-16T02:35:44.304 INFO:tasks.workunit.client.0.smithi151.stdout:4/313: chown d0/d11/d19/f1b 1013562720 1 2024-04-16T02:35:44.305 INFO:tasks.workunit.client.1.smithi153.stdout:4/879: mkdir d3/d9/db/d17/d28/dc8/de4/d134 0 2024-04-16T02:35:44.305 INFO:tasks.workunit.client.0.smithi151.stdout:4/314: write d0/d11/f43 [5181399,32413] 0 2024-04-16T02:35:44.308 INFO:tasks.workunit.client.0.smithi151.stdout:4/315: truncate d0/f16 374386 0 2024-04-16T02:35:44.309 INFO:tasks.workunit.client.1.smithi153.stdout:0/878: mkdir dd/d14/d19/d77/d87/dda/d11a 0 2024-04-16T02:35:44.309 INFO:tasks.workunit.client.0.smithi151.stdout:4/316: dread - d0/d11/f3d zero size 2024-04-16T02:35:44.309 INFO:tasks.workunit.client.1.smithi153.stdout:5/827: dwrite d8/d41/dbd/f40 [0,4194304] 0 2024-04-16T02:35:44.310 INFO:tasks.workunit.client.1.smithi153.stdout:4/880: write d3/d9/db/fff [947211,108720] 0 2024-04-16T02:35:44.311 INFO:tasks.workunit.client.0.smithi151.stdout:4/317: truncate d0/d11/d19/d31/d34/d3a/d50/f67 734557 0 2024-04-16T02:35:44.314 INFO:tasks.workunit.client.1.smithi153.stdout:1/788: dwrite d4/da2/d65/d82/fb0 [0,4194304] 0 2024-04-16T02:35:44.315 INFO:tasks.workunit.client.1.smithi153.stdout:2/750: creat d9/d24/d34/d36/ffb x:0 0 0 2024-04-16T02:35:44.318 INFO:tasks.workunit.client.1.smithi153.stdout:0/879: mknod dd/d14/d19/d77/d87/c11b 0 2024-04-16T02:35:44.319 INFO:tasks.workunit.client.1.smithi153.stdout:9/828: truncate d4/d6/dc/d2d/d79/f8f 1939596 0 2024-04-16T02:35:44.321 INFO:tasks.workunit.client.1.smithi153.stdout:3/750: creat d6/d10/d22/d2e/d8a/f101 x:0 0 0 2024-04-16T02:35:44.323 INFO:tasks.workunit.client.1.smithi153.stdout:3/751: chown d6/d10/d3b/d59/d7c/cde 83 1 2024-04-16T02:35:44.323 INFO:tasks.workunit.client.1.smithi153.stdout:1/789: truncate d4/dd/d13/d26/f78 431963 0 2024-04-16T02:35:44.324 INFO:tasks.workunit.client.1.smithi153.stdout:0/880: rmdir dd/d14/d51/d54/d65 39 2024-04-16T02:35:44.331 INFO:tasks.workunit.client.1.smithi153.stdout:1/790: dwrite d4/da2/d65/d84/fbd [0,4194304] 0 2024-04-16T02:35:44.332 INFO:tasks.workunit.client.1.smithi153.stdout:1/791: stat d4/c37 0 2024-04-16T02:35:44.332 INFO:tasks.workunit.client.1.smithi153.stdout:0/881: dwrite dd/d32/d3f/f101 [0,4194304] 0 2024-04-16T02:35:44.336 INFO:tasks.workunit.client.1.smithi153.stdout:3/752: unlink d6/d10/d22/d2e/fad 0 2024-04-16T02:35:44.336 INFO:tasks.workunit.client.1.smithi153.stdout:0/882: write dd/d14/d51/f60 [9126480,95915] 0 2024-04-16T02:35:44.336 INFO:tasks.workunit.client.1.smithi153.stdout:7/747: getdents d0/d4/db/d34/d4b 0 2024-04-16T02:35:44.337 INFO:tasks.workunit.client.1.smithi153.stdout:5/828: creat d8/df/d28/d5a/d72/f105 x:0 0 0 2024-04-16T02:35:44.339 INFO:tasks.workunit.client.1.smithi153.stdout:0/883: read dd/d28/f83 [3417949,65228] 0 2024-04-16T02:35:44.345 INFO:tasks.workunit.client.1.smithi153.stdout:2/751: link d9/d2b/d1c/f90 d9/d24/ffc 0 2024-04-16T02:35:44.349 INFO:tasks.workunit.client.0.smithi151.stdout:4/318: dread d0/d11/d19/d31/f47 [0,4194304] 0 2024-04-16T02:35:44.351 INFO:tasks.workunit.client.1.smithi153.stdout:4/881: getdents d3/d9/d24 0 2024-04-16T02:35:44.354 INFO:tasks.workunit.client.0.smithi151.stdout:4/319: rename d0/d11/f43 to d0/d11/d19/d31/d34/d3a/d50/f6b 0 2024-04-16T02:35:44.354 INFO:tasks.workunit.client.1.smithi153.stdout:2/752: dwrite d9/d71/d8d/fd9 [0,4194304] 0 2024-04-16T02:35:44.354 INFO:tasks.workunit.client.1.smithi153.stdout:4/882: fsync d3/d9/db/d17/d3f/f109 0 2024-04-16T02:35:44.355 INFO:tasks.workunit.client.1.smithi153.stdout:9/829: truncate d4/d33/dae/dc7/d66/dbd/fc5 367188 0 2024-04-16T02:35:44.356 INFO:tasks.workunit.client.0.smithi151.stdout:4/320: symlink d0/d20/l6c 0 2024-04-16T02:35:44.357 INFO:tasks.workunit.client.0.smithi151.stdout:4/321: readlink d0/d11/l14 0 2024-04-16T02:35:44.360 INFO:tasks.workunit.client.1.smithi153.stdout:3/753: symlink d6/d10/d22/l102 0 2024-04-16T02:35:44.364 INFO:tasks.workunit.client.1.smithi153.stdout:0/884: mknod dd/d14/d51/d54/c11c 0 2024-04-16T02:35:44.367 INFO:tasks.workunit.client.1.smithi153.stdout:9/830: creat d4/d6/d1b/ddf/f102 x:0 0 0 2024-04-16T02:35:44.368 INFO:tasks.workunit.client.1.smithi153.stdout:4/883: dwrite d3/d9/f1a [0,4194304] 0 2024-04-16T02:35:44.371 INFO:tasks.workunit.client.1.smithi153.stdout:3/754: creat d6/d10/d3b/d59/d7c/db7/f103 x:0 0 0 2024-04-16T02:35:44.372 INFO:tasks.workunit.client.0.smithi151.stdout:4/322: truncate d0/f3 169823 0 2024-04-16T02:35:44.374 INFO:tasks.workunit.client.1.smithi153.stdout:9/831: rmdir d4/d6/dc/d46 39 2024-04-16T02:35:44.374 INFO:tasks.workunit.client.0.smithi151.stdout:4/323: symlink d0/d11/d19/d31/d4a/d60/l6d 0 2024-04-16T02:35:44.375 INFO:tasks.workunit.client.1.smithi153.stdout:2/753: dwrite d9/d24/d27/d2c/d5d/f63 [0,4194304] 0 2024-04-16T02:35:44.376 INFO:tasks.workunit.client.1.smithi153.stdout:7/748: getdents d0/d4/db/d2b/d53/d59/d76/dc8 0 2024-04-16T02:35:44.377 INFO:tasks.workunit.client.1.smithi153.stdout:5/829: dwrite d8/df/d2f/d4e/d51/f5c [4194304,4194304] 0 2024-04-16T02:35:44.379 INFO:tasks.workunit.client.1.smithi153.stdout:3/755: write d6/d10/d3b/d59/d7c/de1/df0/fc6 [219736,44269] 0 2024-04-16T02:35:44.380 INFO:tasks.workunit.client.1.smithi153.stdout:5/830: write d8/df/f1e [997961,24811] 0 2024-04-16T02:35:44.383 INFO:tasks.workunit.client.1.smithi153.stdout:9/832: read d4/d6/dc/d2d/fa8 [782655,9316] 0 2024-04-16T02:35:44.384 INFO:tasks.workunit.client.1.smithi153.stdout:5/831: chown d8/df/d23/d70/ccc 6379 1 2024-04-16T02:35:44.387 INFO:tasks.workunit.client.1.smithi153.stdout:5/832: write d8/df/d2f/f60 [99856,16686] 0 2024-04-16T02:35:44.389 INFO:tasks.workunit.client.1.smithi153.stdout:5/833: write d8/d41/dbd/d83/fc7 [3476346,114784] 0 2024-04-16T02:35:44.392 INFO:tasks.workunit.client.1.smithi153.stdout:1/792: dread d4/dd/d13/d26/f2b [0,4194304] 0 2024-04-16T02:35:44.397 INFO:tasks.workunit.client.0.smithi151.stdout:4/324: truncate d0/d11/d19/d31/d34/d3a/d50/f6b 588944 0 2024-04-16T02:35:44.398 INFO:tasks.workunit.client.1.smithi153.stdout:5/834: dwrite d8/d41/dbd/d39/d6d/fea [4194304,4194304] 0 2024-04-16T02:35:44.400 INFO:tasks.workunit.client.0.smithi151.stdout:4/325: mkdir d0/d11/d19/d31/d34/d3a/d6e 0 2024-04-16T02:35:44.404 INFO:tasks.workunit.client.0.smithi151.stdout:4/326: creat d0/d11/d19/d46/f6f x:0 0 0 2024-04-16T02:35:44.404 INFO:tasks.workunit.client.0.smithi151.stdout:4/327: chown d0/d11/l14 82619429 1 2024-04-16T02:35:44.406 INFO:tasks.workunit.client.0.smithi151.stdout:4/328: truncate d0/d20/f2c 1821235 0 2024-04-16T02:35:44.410 INFO:tasks.workunit.client.0.smithi151.stdout:4/329: creat d0/d11/d19/d31/d4a/d60/f70 x:0 0 0 2024-04-16T02:35:44.416 INFO:tasks.workunit.client.1.smithi153.stdout:4/884: dread d3/d9/f3d [4194304,4194304] 0 2024-04-16T02:35:44.417 INFO:tasks.workunit.client.1.smithi153.stdout:4/885: read - d3/d9/db/de/d2c/d52/d85/f11e zero size 2024-04-16T02:35:44.418 INFO:tasks.workunit.client.1.smithi153.stdout:4/886: write d3/f1c [5015912,115273] 0 2024-04-16T02:35:44.428 INFO:tasks.workunit.client.0.smithi151.stdout:4/330: write d0/d11/f26 [440983,81410] 0 2024-04-16T02:35:44.445 INFO:tasks.workunit.client.1.smithi153.stdout:7/749: rename d0/d4/db/d2b/d53/c2e to d0/d4/db/d87/dc0/c100 0 2024-04-16T02:35:44.446 INFO:tasks.workunit.client.1.smithi153.stdout:9/833: mkdir d4/d33/dae/d103 0 2024-04-16T02:35:44.446 INFO:tasks.workunit.client.1.smithi153.stdout:7/750: readlink d0/d4/db/d2b/l11 0 2024-04-16T02:35:44.450 INFO:tasks.workunit.client.1.smithi153.stdout:1/793: creat d4/dd/d13/d8d/f10e x:0 0 0 2024-04-16T02:35:44.458 INFO:tasks.workunit.client.1.smithi153.stdout:1/794: dwrite d4/dd/d18/d29/f4a [0,4194304] 0 2024-04-16T02:35:44.460 INFO:tasks.workunit.client.1.smithi153.stdout:4/887: creat d3/d9/db/d17/d28/dc8/de4/dfd/d120/f135 x:0 0 0 2024-04-16T02:35:44.461 INFO:tasks.workunit.client.1.smithi153.stdout:8/755: sync 2024-04-16T02:35:44.461 INFO:tasks.workunit.client.1.smithi153.stdout:6/779: sync 2024-04-16T02:35:44.463 INFO:tasks.workunit.client.1.smithi153.stdout:4/888: dread - d3/d9/db/de/d2c/d52/f115 zero size 2024-04-16T02:35:44.463 INFO:tasks.workunit.client.1.smithi153.stdout:9/834: mkdir d4/d6/dc/d24/d104 0 2024-04-16T02:35:44.467 INFO:tasks.workunit.client.1.smithi153.stdout:8/756: dread d2/d93/d97/d9c/d27/fc3 [0,4194304] 0 2024-04-16T02:35:44.471 INFO:tasks.workunit.client.1.smithi153.stdout:9/835: rmdir d4/d6/dc/d24/d60 39 2024-04-16T02:35:44.473 INFO:tasks.workunit.client.1.smithi153.stdout:1/795: symlink d4/dd/d18/l10f 0 2024-04-16T02:35:44.476 INFO:tasks.workunit.client.1.smithi153.stdout:1/796: creat d4/dd/d18/d6c/dd8/f110 x:0 0 0 2024-04-16T02:35:44.477 INFO:tasks.workunit.client.1.smithi153.stdout:9/836: dread d4/d6/d1b/f90 [0,4194304] 0 2024-04-16T02:35:44.479 INFO:tasks.workunit.client.1.smithi153.stdout:9/837: chown d4/d6/ddb/lec 220812957 1 2024-04-16T02:35:44.479 INFO:tasks.workunit.client.1.smithi153.stdout:1/797: mkdir d4/dd/d13/d7e/de6/d111 0 2024-04-16T02:35:44.480 INFO:tasks.workunit.client.1.smithi153.stdout:6/780: symlink dc/d47/d57/d77/d7a/l105 0 2024-04-16T02:35:44.482 INFO:tasks.workunit.client.1.smithi153.stdout:9/838: symlink d4/d33/l105 0 2024-04-16T02:35:44.490 INFO:tasks.workunit.client.1.smithi153.stdout:9/839: dwrite d4/d6/dc/d2d/fa8 [0,4194304] 0 2024-04-16T02:35:44.493 INFO:tasks.workunit.client.1.smithi153.stdout:9/840: mknod d4/d6/d8c/c106 0 2024-04-16T02:35:44.495 INFO:tasks.workunit.client.1.smithi153.stdout:9/841: write d4/d6/dc/d2d/d64/ffd [257611,49353] 0 2024-04-16T02:35:44.500 INFO:tasks.workunit.client.0.smithi151.stdout:2/329: sync 2024-04-16T02:35:44.501 INFO:tasks.workunit.client.0.smithi151.stdout:2/330: chown de/d11/d41/d4b/d60 1 1 2024-04-16T02:35:44.503 INFO:tasks.workunit.client.0.smithi151.stdout:2/331: symlink de/d11/d41/d35/l66 0 2024-04-16T02:35:44.505 INFO:tasks.workunit.client.0.smithi151.stdout:2/332: creat de/d15/d1e/f67 x:0 0 0 2024-04-16T02:35:44.509 INFO:tasks.workunit.client.1.smithi153.stdout:5/835: mknod d8/c106 0 2024-04-16T02:35:44.513 INFO:tasks.workunit.client.1.smithi153.stdout:5/836: creat d8/d1f/f107 x:0 0 0 2024-04-16T02:35:44.514 INFO:tasks.workunit.client.1.smithi153.stdout:9/842: dread d4/d6/dc/d24/d4e/f9c [0,4194304] 0 2024-04-16T02:35:44.516 INFO:tasks.workunit.client.1.smithi153.stdout:9/843: read d4/d6/dc/d2d/f53 [3354841,44488] 0 2024-04-16T02:35:44.521 INFO:tasks.workunit.client.0.smithi151.stdout:2/333: dwrite de/d11/d41/f37 [0,4194304] 0 2024-04-16T02:35:44.521 INFO:tasks.workunit.client.1.smithi153.stdout:7/751: truncate d0/d4/db/d2b/d8c/f9b 724078 0 2024-04-16T02:35:44.524 INFO:tasks.workunit.client.1.smithi153.stdout:5/837: dread d8/f6c [0,4194304] 0 2024-04-16T02:35:44.525 INFO:tasks.workunit.client.0.smithi151.stdout:2/334: mkdir de/d11/d42/d68 0 2024-04-16T02:35:44.527 INFO:tasks.workunit.client.1.smithi153.stdout:5/838: creat d8/df/d28/d5a/d5e/f108 x:0 0 0 2024-04-16T02:35:44.528 INFO:tasks.workunit.client.0.smithi151.stdout:2/335: symlink de/d11/d41/d35/l69 0 2024-04-16T02:35:44.528 INFO:tasks.workunit.client.1.smithi153.stdout:5/839: chown d8/df/d28/c42 38767897 1 2024-04-16T02:35:44.529 INFO:tasks.workunit.client.1.smithi153.stdout:7/752: rmdir d0/d4/db/d87/dc0/dea 0 2024-04-16T02:35:44.532 INFO:tasks.workunit.client.1.smithi153.stdout:7/753: stat d0/d4/db/d2b/d53/d59/d76/dc8/ca1 0 2024-04-16T02:35:44.533 INFO:tasks.workunit.client.1.smithi153.stdout:5/840: truncate d8/df/d28/f82 1622250 0 2024-04-16T02:35:44.534 INFO:tasks.workunit.client.1.smithi153.stdout:7/754: read d0/d4/db/f64 [4257828,66409] 0 2024-04-16T02:35:44.535 INFO:tasks.workunit.client.0.smithi151.stdout:2/336: dwrite de/d15/d1e/f3f [0,4194304] 0 2024-04-16T02:35:44.542 INFO:tasks.workunit.client.1.smithi153.stdout:5/841: dwrite d8/d41/d90/ded/ffa [0,4194304] 0 2024-04-16T02:35:44.554 INFO:tasks.workunit.client.1.smithi153.stdout:3/756: rename d6/d10/d3b/d59/d7c/db7/lf1 to d6/l104 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:3/757: stat d6/d10/d22/l102 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:5/842: mknod d8/df/d28/d5a/d5e/c109 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:5/843: rmdir d8 39 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:3/758: read d6/d10/d22/d2c/f58 [737903,12622] 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:7/755: unlink d0/d4/c10 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:7/756: write d0/d4/db/d2b/d43/d6d/d68/ff7 [951662,97326] 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:4/889: rename d3/ff to d3/d9/db/d17/d28/dc8/de4/f136 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:4/890: fsync d3/d9/d24/d75/d9e/f128 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:0/885: sync 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:0/886: dread - dd/d32/fc8 zero size 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:8/757: mknod d2/cfe 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:7/757: mkdir d0/d4/db/d2b/d53/d59/d76/dc8/d101 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:4/891: creat d3/d9/db/d17/d28/dc8/f137 x:0 0 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:7/758: truncate d0/d4/db/d2b/d43/d6d/d68/fd4 270826 0 2024-04-16T02:35:44.577 INFO:tasks.workunit.client.1.smithi153.stdout:0/887: creat dd/d14/d19/d77/d87/dda/f11d x:0 0 0 2024-04-16T02:35:44.578 INFO:tasks.workunit.client.1.smithi153.stdout:8/758: rmdir d2/d93/d97/d9c/d27/d5e/d63/dc6 39 2024-04-16T02:35:44.578 INFO:tasks.workunit.client.1.smithi153.stdout:0/888: dread - dd/d32/d57/d6c/dd1/f103 zero size 2024-04-16T02:35:44.578 INFO:tasks.workunit.client.1.smithi153.stdout:8/759: write d2/d93/d97/d9c/f3c [743245,58232] 0 2024-04-16T02:35:44.578 INFO:tasks.workunit.client.1.smithi153.stdout:5/844: creat d8/df/d2f/f10a x:0 0 0 2024-04-16T02:35:44.578 INFO:tasks.workunit.client.1.smithi153.stdout:8/760: write d2/d93/d97/d9c/d27/d5e/d63/fb1 [3109860,104956] 0 2024-04-16T02:35:44.578 INFO:tasks.workunit.client.1.smithi153.stdout:8/761: chown d2/d92/deb/d47/d60/d67/fa2 587630 1 2024-04-16T02:35:44.579 INFO:tasks.workunit.client.1.smithi153.stdout:1/798: rename d4/dd/d18/f51 to d4/da2/f112 0 2024-04-16T02:35:44.580 INFO:tasks.workunit.client.1.smithi153.stdout:9/844: write d4/d6/d1b/f71 [303633,39461] 0 2024-04-16T02:35:44.580 INFO:tasks.workunit.client.1.smithi153.stdout:3/759: getdents d6/d10/d3b/dd7 0 2024-04-16T02:35:44.583 INFO:tasks.workunit.client.1.smithi153.stdout:5/845: symlink d8/d41/d90/ded/l10b 0 2024-04-16T02:35:44.583 INFO:tasks.workunit.client.1.smithi153.stdout:5/846: readlink d8/df/d28/l46 0 2024-04-16T02:35:44.585 INFO:tasks.workunit.client.1.smithi153.stdout:4/892: mkdir d3/d9/db/d17/d28/dc8/de4/dec/d138 0 2024-04-16T02:35:44.588 INFO:tasks.workunit.client.1.smithi153.stdout:6/781: rename dc/d47/d57/d77/d15/f86 to dc/d6b/f106 0 2024-04-16T02:35:44.588 INFO:tasks.workunit.client.1.smithi153.stdout:0/889: fdatasync dd/d14/d19/fae 0 2024-04-16T02:35:44.590 INFO:tasks.workunit.client.1.smithi153.stdout:1/799: unlink d4/dd/d18/d6c/fba 0 2024-04-16T02:35:44.590 INFO:tasks.workunit.client.1.smithi153.stdout:1/800: stat d4/da2/d65/f102 0 2024-04-16T02:35:44.593 INFO:tasks.workunit.client.1.smithi153.stdout:5/847: unlink d8/df/fba 0 2024-04-16T02:35:44.594 INFO:tasks.workunit.client.1.smithi153.stdout:8/762: mknod d2/d13/de8/cff 0 2024-04-16T02:35:44.594 INFO:tasks.workunit.client.1.smithi153.stdout:5/848: fsync d8/d41/dbd/d39/d6d/fa0 0 2024-04-16T02:35:44.602 INFO:tasks.workunit.client.1.smithi153.stdout:3/760: mknod d6/d10/d22/c105 0 2024-04-16T02:35:44.603 INFO:tasks.workunit.client.1.smithi153.stdout:7/759: write d0/d4/db/d34/d51/fb3 [127892,101516] 0 2024-04-16T02:35:44.604 INFO:tasks.workunit.client.1.smithi153.stdout:3/761: write d6/d10/d3b/d59/d7c/de1/fe3 [50185,19057] 0 2024-04-16T02:35:44.606 INFO:tasks.workunit.client.1.smithi153.stdout:4/893: fdatasync d3/d9/db/d45/df3/fa6 0 2024-04-16T02:35:44.662 INFO:tasks.workunit.client.1.smithi153.stdout:6/782: creat dc/d1a/dd7/f107 x:0 0 0 2024-04-16T02:35:44.663 INFO:tasks.workunit.client.1.smithi153.stdout:9/845: creat d4/d6/dc/d2d/d64/db9/f107 x:0 0 0 2024-04-16T02:35:44.663 INFO:tasks.workunit.client.1.smithi153.stdout:4/894: write d3/d9/db/d17/d28/dc8/f11f [693443,46614] 0 2024-04-16T02:35:44.663 INFO:tasks.workunit.client.1.smithi153.stdout:3/762: dread d6/d10/d3b/d59/fa9 [0,4194304] 0 2024-04-16T02:35:44.663 INFO:tasks.workunit.client.1.smithi153.stdout:7/760: unlink d0/d4/db/d2b/d53/ld0 0 2024-04-16T02:35:44.665 INFO:tasks.workunit.client.1.smithi153.stdout:6/783: dread - dc/d47/d57/d77/dc4/ff7 zero size 2024-04-16T02:35:44.666 INFO:tasks.workunit.client.1.smithi153.stdout:1/801: rmdir d4/da2/d65/d82/de9 0 2024-04-16T02:35:44.667 INFO:tasks.workunit.client.1.smithi153.stdout:9/846: chown d4/d6/ddb/d4a/d101 19663356 1 2024-04-16T02:35:44.668 INFO:tasks.workunit.client.1.smithi153.stdout:4/895: chown d3/d9/db/d45/df3/ffc 691 1 2024-04-16T02:35:44.668 INFO:tasks.workunit.client.1.smithi153.stdout:7/761: symlink d0/d4/db/d87/l102 0 2024-04-16T02:35:44.668 INFO:tasks.workunit.client.1.smithi153.stdout:1/802: chown d4/dd/d18/c3a 49862343 1 2024-04-16T02:35:44.670 INFO:tasks.workunit.client.1.smithi153.stdout:6/784: truncate dc/d1a/fbe 102624 0 2024-04-16T02:35:44.670 INFO:tasks.workunit.client.1.smithi153.stdout:0/890: dread dd/d14/f4b [0,4194304] 0 2024-04-16T02:35:44.670 INFO:tasks.workunit.client.1.smithi153.stdout:5/849: read - d8/df/d28/d5a/d5e/f108 zero size 2024-04-16T02:35:44.672 INFO:tasks.workunit.client.1.smithi153.stdout:7/762: write d0/d4/db/d2b/d53/d59/f67 [4091418,46814] 0 2024-04-16T02:35:44.673 INFO:tasks.workunit.client.1.smithi153.stdout:1/803: read d4/dd/d13/d23/d5f/f61 [1508450,121570] 0 2024-04-16T02:35:44.676 INFO:tasks.workunit.client.0.smithi151.stdout:1/274: sync 2024-04-16T02:35:44.676 INFO:tasks.workunit.client.0.smithi151.stdout:3/177: sync 2024-04-16T02:35:44.676 INFO:tasks.workunit.client.0.smithi151.stdout:5/316: sync 2024-04-16T02:35:44.676 INFO:tasks.workunit.client.0.smithi151.stdout:1/275: dread - d3/d58/f54 zero size 2024-04-16T02:35:44.678 INFO:tasks.workunit.client.1.smithi153.stdout:7/763: dwrite d0/d4/db/d34/d51/fda [0,4194304] 0 2024-04-16T02:35:44.679 INFO:tasks.workunit.client.0.smithi151.stdout:1/276: creat d3/d58/d1b/d2a/f60 x:0 0 0 2024-04-16T02:35:44.680 INFO:tasks.workunit.client.0.smithi151.stdout:5/317: truncate d2/f12 1315490 0 2024-04-16T02:35:44.683 INFO:tasks.workunit.client.0.smithi151.stdout:5/318: write d2/d6/d19/d1a/f48 [85884,69570] 0 2024-04-16T02:35:44.683 INFO:tasks.workunit.client.0.smithi151.stdout:3/178: dwrite d1/d6/d22/f2c [0,4194304] 0 2024-04-16T02:35:44.693 INFO:tasks.workunit.client.0.smithi151.stdout:1/277: dwrite d3/d58/d1b/d1c/f50 [0,4194304] 0 2024-04-16T02:35:44.694 INFO:tasks.workunit.client.0.smithi151.stdout:3/179: rename d1/d6/d22/d23/f2d to d1/f3e 0 2024-04-16T02:35:44.699 INFO:tasks.workunit.client.0.smithi151.stdout:3/180: mkdir d1/d6/d20/d2a/d3f 0 2024-04-16T02:35:44.701 INFO:tasks.workunit.client.0.smithi151.stdout:3/181: write d1/d6/f33 [1023045,126347] 0 2024-04-16T02:35:44.704 INFO:tasks.workunit.client.0.smithi151.stdout:1/278: dwrite d3/f2f [0,4194304] 0 2024-04-16T02:35:44.705 INFO:tasks.workunit.client.1.smithi153.stdout:3/763: mknod d6/d10/d3b/d59/c106 0 2024-04-16T02:35:44.706 INFO:tasks.workunit.client.1.smithi153.stdout:2/754: sync 2024-04-16T02:35:44.708 INFO:tasks.workunit.client.0.smithi151.stdout:5/319: dwrite d2/d6/f25 [0,4194304] 0 2024-04-16T02:35:44.709 INFO:tasks.workunit.client.0.smithi151.stdout:1/279: mkdir d3/d58/d1b/d61 0 2024-04-16T02:35:44.711 INFO:tasks.workunit.client.0.smithi151.stdout:3/182: symlink d1/d6/d20/d2a/d3f/l40 0 2024-04-16T02:35:44.711 INFO:tasks.workunit.client.0.smithi151.stdout:1/280: write d3/d58/d1b/d2a/f30 [1964301,91698] 0 2024-04-16T02:35:44.712 INFO:tasks.workunit.client.0.smithi151.stdout:1/281: readlink d3/d33/l17 0 2024-04-16T02:35:44.716 INFO:tasks.workunit.client.0.smithi151.stdout:3/183: unlink d1/d6/d20/d2a/c19 0 2024-04-16T02:35:44.717 INFO:tasks.workunit.client.0.smithi151.stdout:3/184: fdatasync d1/d6/d22/f2b 0 2024-04-16T02:35:44.718 INFO:tasks.workunit.client.0.smithi151.stdout:3/185: readlink d1/d37/l3a 0 2024-04-16T02:35:44.718 INFO:tasks.workunit.client.0.smithi151.stdout:1/282: dread d3/d1f/d48/f51 [0,4194304] 0 2024-04-16T02:35:44.719 INFO:tasks.workunit.client.0.smithi151.stdout:3/186: chown d1/d6/d22/f2b 49319 1 2024-04-16T02:35:44.719 INFO:tasks.workunit.client.0.smithi151.stdout:1/283: chown d3/d58/d1b/d1c/f50 4710 1 2024-04-16T02:35:44.721 INFO:tasks.workunit.client.0.smithi151.stdout:1/284: creat d3/d58/f62 x:0 0 0 2024-04-16T02:35:44.722 INFO:tasks.workunit.client.0.smithi151.stdout:3/187: mkdir d1/d6/d22/d23/d41 0 2024-04-16T02:35:44.722 INFO:tasks.workunit.client.0.smithi151.stdout:3/188: chown d1/f17 5457134 1 2024-04-16T02:35:44.723 INFO:tasks.workunit.client.0.smithi151.stdout:3/189: stat d1/d6/d20/d2a/f35 0 2024-04-16T02:35:44.732 INFO:tasks.workunit.client.0.smithi151.stdout:1/285: write d3/d58/df/d38/f5a [1689174,62796] 0 2024-04-16T02:35:44.732 INFO:tasks.workunit.client.0.smithi151.stdout:3/190: write d1/d6/d20/d2a/f1a [1092882,4234] 0 2024-04-16T02:35:44.733 INFO:tasks.workunit.client.1.smithi153.stdout:8/763: sync 2024-04-16T02:35:44.733 INFO:tasks.workunit.client.0.smithi151.stdout:1/286: stat d3/d58/d1b/d1c/f50 0 2024-04-16T02:35:44.736 INFO:tasks.workunit.client.0.smithi151.stdout:1/287: write d3/d1f/d48/f51 [96355,65184] 0 2024-04-16T02:35:44.739 INFO:tasks.workunit.client.0.smithi151.stdout:3/191: truncate d1/d6/d20/d2a/f15 6113519 0 2024-04-16T02:35:44.741 INFO:tasks.workunit.client.0.smithi151.stdout:3/192: fsync d1/fa 0 2024-04-16T02:35:44.741 INFO:tasks.workunit.client.0.smithi151.stdout:5/320: dread d2/f1f [0,4194304] 0 2024-04-16T02:35:44.742 INFO:tasks.workunit.client.0.smithi151.stdout:3/193: dread - d1/d6/d20/d2a/f35 zero size 2024-04-16T02:35:44.742 INFO:tasks.workunit.client.0.smithi151.stdout:5/321: chown d2/d6/d19/d30/d49 1125981 1 2024-04-16T02:35:44.747 INFO:tasks.workunit.client.0.smithi151.stdout:5/322: creat d2/d6/f64 x:0 0 0 2024-04-16T02:35:44.750 INFO:tasks.workunit.client.0.smithi151.stdout:3/194: getdents d1/d6/d20/d2a 0 2024-04-16T02:35:44.753 INFO:tasks.workunit.client.0.smithi151.stdout:3/195: chown d1/d6/d20/d2a/l25 22 1 2024-04-16T02:35:44.754 INFO:tasks.workunit.client.0.smithi151.stdout:5/323: dwrite d2/d6/f25 [0,4194304] 0 2024-04-16T02:35:44.754 INFO:tasks.workunit.client.0.smithi151.stdout:3/196: creat d1/d37/f42 x:0 0 0 2024-04-16T02:35:44.756 INFO:tasks.workunit.client.0.smithi151.stdout:5/324: rmdir d2/d6/d19/d30/d36 39 2024-04-16T02:35:44.758 INFO:tasks.workunit.client.0.smithi151.stdout:3/197: mkdir d1/d6/d20/d2a/d3f/d43 0 2024-04-16T02:35:44.760 INFO:tasks.workunit.client.0.smithi151.stdout:3/198: dread - d1/d6/f21 zero size 2024-04-16T02:35:44.762 INFO:tasks.workunit.client.0.smithi151.stdout:5/325: fsync d2/d6/d19/d30/d36/f3e 0 2024-04-16T02:35:44.762 INFO:tasks.workunit.client.0.smithi151.stdout:3/199: chown d1/d6/f24 855 1 2024-04-16T02:35:44.770 INFO:tasks.workunit.client.0.smithi151.stdout:3/200: getdents d1/d6/d20/d2a/d3f 0 2024-04-16T02:35:44.770 INFO:tasks.workunit.client.0.smithi151.stdout:3/201: stat d1/d6/d20/d2a/f1a 0 2024-04-16T02:35:44.770 INFO:tasks.workunit.client.0.smithi151.stdout:5/326: write d2/d6/d19/f1d [1658003,31817] 0 2024-04-16T02:35:44.771 INFO:tasks.workunit.client.0.smithi151.stdout:3/202: mkdir d1/d6/d22/d23/d30/d44 0 2024-04-16T02:35:44.773 INFO:tasks.workunit.client.0.smithi151.stdout:5/327: rename d2/d6/f13 to d2/d6/d19/d30/d49/f65 0 2024-04-16T02:35:44.774 INFO:tasks.workunit.client.0.smithi151.stdout:3/203: fdatasync d1/f26 0 2024-04-16T02:35:44.777 INFO:tasks.workunit.client.0.smithi151.stdout:3/204: creat d1/d6/d22/d23/d30/f45 x:0 0 0 2024-04-16T02:35:44.779 INFO:tasks.workunit.client.0.smithi151.stdout:3/205: write d1/d6/d20/d2a/f35 [616605,55261] 0 2024-04-16T02:35:44.781 INFO:tasks.workunit.client.0.smithi151.stdout:5/328: getdents d2/d6/d19/d1a 0 2024-04-16T02:35:44.783 INFO:tasks.workunit.client.1.smithi153.stdout:4/896: mkdir d3/d9/db/d17/d28/dc8/d139 0 2024-04-16T02:35:44.783 INFO:tasks.workunit.client.0.smithi151.stdout:5/329: chown d2/d6/d19/d30/d36/d42/f58 47489503 1 2024-04-16T02:35:44.784 INFO:tasks.workunit.client.1.smithi153.stdout:4/897: chown d3/d9/db/de/d2c/d52/d85/f90 522221 1 2024-04-16T02:35:44.784 INFO:tasks.workunit.client.0.smithi151.stdout:3/206: mkdir d1/d6/d22/d23/d30/d44/d46 0 2024-04-16T02:35:44.785 INFO:tasks.workunit.client.1.smithi153.stdout:4/898: write d3/fdf [1002406,71651] 0 2024-04-16T02:35:44.786 INFO:tasks.workunit.client.0.smithi151.stdout:5/330: fsync d2/d6/d19/d30/d36/f3e 0 2024-04-16T02:35:44.787 INFO:tasks.workunit.client.1.smithi153.stdout:9/847: rename d4/d6/lc4 to d4/d6/d1b/d2a/l108 0 2024-04-16T02:35:44.788 INFO:tasks.workunit.client.1.smithi153.stdout:9/848: dread - d4/d6/dc/d24/d4e/df4/ff7 zero size 2024-04-16T02:35:44.789 INFO:tasks.workunit.client.1.smithi153.stdout:0/891: mknod dd/d32/c11e 0 2024-04-16T02:35:44.789 INFO:tasks.workunit.client.0.smithi151.stdout:3/207: unlink d1/d6/c8 0 2024-04-16T02:35:44.790 INFO:tasks.workunit.client.1.smithi153.stdout:1/804: mkdir d4/dd/d18/d29/d6e/d113 0 2024-04-16T02:35:44.791 INFO:tasks.workunit.client.0.smithi151.stdout:5/331: symlink d2/d6/l66 0 2024-04-16T02:35:44.792 INFO:tasks.workunit.client.0.smithi151.stdout:5/332: fdatasync d2/d6/d19/d30/d49/f5c 0 2024-04-16T02:35:44.793 INFO:tasks.workunit.client.0.smithi151.stdout:3/208: rename d1/d6/d20/d2a/d3f/l40 to d1/d6/d22/d23/l47 0 2024-04-16T02:35:44.795 INFO:tasks.workunit.client.0.smithi151.stdout:5/333: creat d2/d6/d43/f67 x:0 0 0 2024-04-16T02:35:44.801 INFO:tasks.workunit.client.0.smithi151.stdout:3/209: dwrite d1/d6/d22/f2c [0,4194304] 0 2024-04-16T02:35:44.803 INFO:tasks.workunit.client.0.smithi151.stdout:5/334: dwrite d2/d6/d19/f1d [0,4194304] 0 2024-04-16T02:35:44.810 INFO:tasks.workunit.client.0.smithi151.stdout:5/335: rename d2/d6/f9 to d2/f68 0 2024-04-16T02:35:44.814 INFO:tasks.workunit.client.0.smithi151.stdout:3/210: dwrite d1/d6/d22/f2c [0,4194304] 0 2024-04-16T02:35:44.821 INFO:tasks.workunit.client.0.smithi151.stdout:5/336: dwrite d2/d6/d19/f5f [0,4194304] 0 2024-04-16T02:35:44.829 INFO:tasks.workunit.client.1.smithi153.stdout:6/785: creat dc/d47/d57/d77/d15/dac/dae/de7/f108 x:0 0 0 2024-04-16T02:35:44.831 INFO:tasks.workunit.client.1.smithi153.stdout:2/755: mkdir d9/d24/dfd 0 2024-04-16T02:35:44.833 INFO:tasks.workunit.client.1.smithi153.stdout:8/764: symlink d2/d13/d3a/dd6/l100 0 2024-04-16T02:35:44.834 INFO:tasks.workunit.client.1.smithi153.stdout:5/850: symlink d8/df/d2f/d4e/l10c 0 2024-04-16T02:35:44.835 INFO:tasks.workunit.client.1.smithi153.stdout:4/899: mkdir d3/d9/db/de/d2c/d13a 0 2024-04-16T02:35:44.837 INFO:tasks.workunit.client.1.smithi153.stdout:5/851: fsync d8/df/d28/d5a/d72/fff 0 2024-04-16T02:35:44.837 INFO:tasks.workunit.client.1.smithi153.stdout:7/764: rename d0/d4/db/d2b/d53/dbd to d0/d4/db/d2b/d53/d59/d76/d103 0 2024-04-16T02:35:44.838 INFO:tasks.workunit.client.1.smithi153.stdout:6/786: dwrite dc/d47/d57/d101/f66 [0,4194304] 0 2024-04-16T02:35:44.841 INFO:tasks.workunit.client.1.smithi153.stdout:5/852: fsync d8/df/d2f/f10a 0 2024-04-16T02:35:44.842 INFO:tasks.workunit.client.1.smithi153.stdout:2/756: symlink d9/d2b/d1c/d20/d88/lfe 0 2024-04-16T02:35:44.844 INFO:tasks.workunit.client.1.smithi153.stdout:1/805: rename d4/dd/d13/d26/d58/f91 to d4/da2/d65/d82/f114 0 2024-04-16T02:35:44.846 INFO:tasks.workunit.client.1.smithi153.stdout:4/900: truncate d3/d9/db/d45/df3/fc4 4985101 0 2024-04-16T02:35:44.847 INFO:tasks.workunit.client.1.smithi153.stdout:3/764: getdents d6/d10/d3b/d59/d7c/db7 0 2024-04-16T02:35:44.848 INFO:tasks.workunit.client.1.smithi153.stdout:5/853: readlink d8/d1f/l27 0 2024-04-16T02:35:44.849 INFO:tasks.workunit.client.1.smithi153.stdout:7/765: dread d0/d4/db/d2b/d8c/fae [0,4194304] 0 2024-04-16T02:35:44.851 INFO:tasks.workunit.client.1.smithi153.stdout:2/757: mknod d9/d24/d27/d2c/d5d/da3/dc0/cff 0 2024-04-16T02:35:44.851 INFO:tasks.workunit.client.1.smithi153.stdout:0/892: link dd/d14/d51/d54/ld6 dd/d28/l11f 0 2024-04-16T02:35:44.852 INFO:tasks.workunit.client.1.smithi153.stdout:8/765: dwrite d2/de/f5f [0,4194304] 0 2024-04-16T02:35:44.855 INFO:tasks.workunit.client.1.smithi153.stdout:2/758: chown d9/d2b/d1c/d20/d45/faf 21 1 2024-04-16T02:35:44.855 INFO:tasks.workunit.client.1.smithi153.stdout:3/765: rmdir d6/d10/d5b 39 2024-04-16T02:35:44.856 INFO:tasks.workunit.client.1.smithi153.stdout:5/854: creat d8/df/d2f/d4e/d51/f10d x:0 0 0 2024-04-16T02:35:44.856 INFO:tasks.workunit.client.1.smithi153.stdout:3/766: fdatasync d6/d10/d3b/fc5 0 2024-04-16T02:35:44.857 INFO:tasks.workunit.client.1.smithi153.stdout:1/806: mkdir d4/dd/d13/d8d/d95/d115 0 2024-04-16T02:35:44.858 INFO:tasks.workunit.client.1.smithi153.stdout:7/766: mknod d0/d4/db/d2b/d43/d6d/d68/d6c/c104 0 2024-04-16T02:35:44.860 INFO:tasks.workunit.client.1.smithi153.stdout:7/767: readlink d0/d4/db/d2b/la8 0 2024-04-16T02:35:44.862 INFO:tasks.workunit.client.1.smithi153.stdout:0/893: fdatasync dd/d14/d19/d77/d87/f91 0 2024-04-16T02:35:44.865 INFO:tasks.workunit.client.1.smithi153.stdout:2/759: creat d9/d24/d34/d60/ddc/f100 x:0 0 0 2024-04-16T02:35:44.865 INFO:tasks.workunit.client.0.smithi151.stdout:8/203: sync 2024-04-16T02:35:44.867 INFO:tasks.workunit.client.0.smithi151.stdout:8/204: write d11/d22/f30 [908551,66966] 0 2024-04-16T02:35:44.868 INFO:tasks.workunit.client.0.smithi151.stdout:8/205: write d11/f43 [204784,61289] 0 2024-04-16T02:35:44.869 INFO:tasks.workunit.client.1.smithi153.stdout:7/768: unlink d0/d4/db/d34/d4b/f75 0 2024-04-16T02:35:44.870 INFO:tasks.workunit.client.1.smithi153.stdout:5/855: mkdir d8/df/d23/d10e 0 2024-04-16T02:35:44.872 INFO:tasks.workunit.client.0.smithi151.stdout:8/206: mknod d11/c44 0 2024-04-16T02:35:44.872 INFO:tasks.workunit.client.1.smithi153.stdout:2/760: dwrite d9/d71/d8d/fd9 [0,4194304] 0 2024-04-16T02:35:44.874 INFO:tasks.workunit.client.0.smithi151.stdout:3/211: write d1/d6/f21 [51232,65659] 0 2024-04-16T02:35:44.875 INFO:tasks.workunit.client.0.smithi151.stdout:8/207: creat d11/d2a/d33/f45 x:0 0 0 2024-04-16T02:35:44.875 INFO:tasks.workunit.client.1.smithi153.stdout:2/761: write d9/d24/d27/d2c/d5d/da3/dc0/fd1 [1335216,119170] 0 2024-04-16T02:35:44.876 INFO:tasks.workunit.client.1.smithi153.stdout:2/762: readlink d9/d24/d34/d60/lda 0 2024-04-16T02:35:44.877 INFO:tasks.workunit.client.0.smithi151.stdout:3/212: creat d1/d6/d20/d2a/d3f/f48 x:0 0 0 2024-04-16T02:35:44.878 INFO:tasks.workunit.client.1.smithi153.stdout:2/763: write d9/d2b/d5c/da2/fe1 [774773,15490] 0 2024-04-16T02:35:44.878 INFO:tasks.workunit.client.1.smithi153.stdout:0/894: mkdir dd/d14/d19/d20/d4a/d120 0 2024-04-16T02:35:44.879 INFO:tasks.workunit.client.0.smithi151.stdout:8/208: creat d11/d22/d2e/f46 x:0 0 0 2024-04-16T02:35:44.879 INFO:tasks.workunit.client.1.smithi153.stdout:5/856: creat d8/d41/f10f x:0 0 0 2024-04-16T02:35:44.881 INFO:tasks.workunit.client.1.smithi153.stdout:7/769: mknod d0/d4/db/d2b/d43/d6d/d68/c105 0 2024-04-16T02:35:44.882 INFO:tasks.workunit.client.0.smithi151.stdout:8/209: read d11/f43 [118616,109381] 0 2024-04-16T02:35:44.884 INFO:tasks.workunit.client.0.smithi151.stdout:3/213: mknod d1/d6/d22/d23/d41/c49 0 2024-04-16T02:35:44.885 INFO:tasks.workunit.client.1.smithi153.stdout:3/767: truncate d6/d10/d3b/f85 578978 0 2024-04-16T02:35:44.885 INFO:tasks.workunit.client.0.smithi151.stdout:3/214: write d1/d37/f3c [771939,37275] 0 2024-04-16T02:35:44.888 INFO:tasks.workunit.client.1.smithi153.stdout:2/764: dwrite d9/d24/d27/d2c/d41/f4c [0,4194304] 0 2024-04-16T02:35:44.889 INFO:tasks.workunit.client.0.smithi151.stdout:3/215: unlink d1/d6/d22/d23/d30/f39 0 2024-04-16T02:35:44.890 INFO:tasks.workunit.client.1.smithi153.stdout:2/765: write d9/fba [1919341,121563] 0 2024-04-16T02:35:44.891 INFO:tasks.workunit.client.0.smithi151.stdout:5/337: mkdir d2/d6/d19/d69 0 2024-04-16T02:35:44.892 INFO:tasks.workunit.client.1.smithi153.stdout:2/766: readlink d9/d2b/d18/lc3 0 2024-04-16T02:35:44.892 INFO:tasks.workunit.client.1.smithi153.stdout:7/770: dread d0/d4/db/d2b/f22 [0,4194304] 0 2024-04-16T02:35:44.894 INFO:tasks.workunit.client.1.smithi153.stdout:7/771: readlink d0/d4/db/d16/l7f 0 2024-04-16T02:35:44.895 INFO:tasks.workunit.client.1.smithi153.stdout:9/849: truncate d4/d33/f4f 2148743 0 2024-04-16T02:35:44.895 INFO:tasks.workunit.client.0.smithi151.stdout:3/216: mkdir d1/d6/d4a 0 2024-04-16T02:35:44.896 INFO:tasks.workunit.client.1.smithi153.stdout:3/768: dread d6/d10/d22/d4d/f6e [4194304,4194304] 0 2024-04-16T02:35:44.900 INFO:tasks.workunit.client.1.smithi153.stdout:9/850: fsync d4/d6/d1b/d6e/f74 0 2024-04-16T02:35:44.901 INFO:tasks.workunit.client.0.smithi151.stdout:5/338: dwrite d2/f16 [0,4194304] 0 2024-04-16T02:35:44.902 INFO:tasks.workunit.client.1.smithi153.stdout:3/769: mkdir d6/d10/dec/ded/d107 0 2024-04-16T02:35:44.903 INFO:tasks.workunit.client.0.smithi151.stdout:3/217: dread d1/d6/f33 [0,4194304] 0 2024-04-16T02:35:44.907 INFO:tasks.workunit.client.0.smithi151.stdout:5/339: symlink d2/d6/d19/d1a/l6a 0 2024-04-16T02:35:44.907 INFO:tasks.workunit.client.1.smithi153.stdout:9/851: mknod d4/d6/def/c109 0 2024-04-16T02:35:44.914 INFO:tasks.workunit.client.0.smithi151.stdout:5/340: dwrite d2/d6/fa [0,4194304] 0 2024-04-16T02:35:44.915 INFO:tasks.workunit.client.1.smithi153.stdout:1/807: truncate d4/dd/fbe 620373 0 2024-04-16T02:35:44.919 INFO:tasks.workunit.client.1.smithi153.stdout:8/766: dwrite d2/d13/f94 [0,4194304] 0 2024-04-16T02:35:44.920 INFO:tasks.workunit.client.0.smithi151.stdout:8/210: dread d11/f1c [0,4194304] 0 2024-04-16T02:35:44.922 INFO:tasks.workunit.client.1.smithi153.stdout:1/808: dread d4/dd/d13/d26/f78 [0,4194304] 0 2024-04-16T02:35:44.922 INFO:tasks.workunit.client.1.smithi153.stdout:0/895: dread dd/d14/d19/f27 [0,4194304] 0 2024-04-16T02:35:44.926 INFO:tasks.workunit.client.0.smithi151.stdout:5/341: dwrite d2/d6/d43/f67 [0,4194304] 0 2024-04-16T02:35:44.927 INFO:tasks.workunit.client.0.smithi151.stdout:8/211: mknod d11/d22/c47 0 2024-04-16T02:35:44.928 INFO:tasks.workunit.client.1.smithi153.stdout:8/767: creat d2/d13/d2c/f101 x:0 0 0 2024-04-16T02:35:44.929 INFO:tasks.workunit.client.1.smithi153.stdout:2/767: dread d9/d2b/d1c/d20/fc2 [0,4194304] 0 2024-04-16T02:35:44.934 INFO:tasks.workunit.client.0.smithi151.stdout:8/212: rename d11/f1f to d11/d2a/d33/f48 0 2024-04-16T02:35:44.935 INFO:tasks.workunit.client.0.smithi151.stdout:5/342: creat d2/d6/d19/d69/f6b x:0 0 0 2024-04-16T02:35:44.935 INFO:tasks.workunit.client.1.smithi153.stdout:1/809: mkdir d4/dd/d18/d29/d6e/d116 0 2024-04-16T02:35:44.936 INFO:tasks.workunit.client.0.smithi151.stdout:8/213: fdatasync d11/d2a/d33/f45 0 2024-04-16T02:35:44.937 INFO:tasks.workunit.client.1.smithi153.stdout:9/852: write d4/d6/dc/d24/dc6/f85 [2885616,73033] 0 2024-04-16T02:35:44.937 INFO:tasks.workunit.client.1.smithi153.stdout:0/896: rename dd/d26/l38 to dd/d14/d19/d77/d87/dda/d11a/l121 0 2024-04-16T02:35:44.938 INFO:tasks.workunit.client.1.smithi153.stdout:1/810: fsync d4/da2/f96 0 2024-04-16T02:35:44.938 INFO:tasks.workunit.client.0.smithi151.stdout:8/214: creat d11/d2a/d33/f49 x:0 0 0 2024-04-16T02:35:44.938 INFO:tasks.workunit.client.1.smithi153.stdout:7/772: dwrite d0/d4/db/d2b/d43/d6d/f63 [0,4194304] 0 2024-04-16T02:35:44.939 INFO:tasks.workunit.client.1.smithi153.stdout:2/768: creat d9/d2b/d1c/d20/f101 x:0 0 0 2024-04-16T02:35:44.940 INFO:tasks.workunit.client.0.smithi151.stdout:5/343: creat d2/d6/f6c x:0 0 0 2024-04-16T02:35:44.940 INFO:tasks.workunit.client.1.smithi153.stdout:7/773: fdatasync d0/de8/ffe 0 2024-04-16T02:35:44.944 INFO:tasks.workunit.client.1.smithi153.stdout:3/770: dread d6/d10/d22/d2c/fba [0,4194304] 0 2024-04-16T02:35:44.946 INFO:tasks.workunit.client.1.smithi153.stdout:1/811: symlink d4/dd/d13/d7e/l117 0 2024-04-16T02:35:44.947 INFO:tasks.workunit.client.0.smithi151.stdout:5/344: rename d2/d6/d43/f4a to d2/d6/d43/f6d 0 2024-04-16T02:35:44.948 INFO:tasks.workunit.client.1.smithi153.stdout:1/812: write d4/dd/d13/d26/db4/ffb [830818,112424] 0 2024-04-16T02:35:44.948 INFO:tasks.workunit.client.0.smithi151.stdout:8/215: dwrite d11/f1b [0,4194304] 0 2024-04-16T02:35:44.948 INFO:tasks.workunit.client.1.smithi153.stdout:7/774: mkdir d0/de8/d106 0 2024-04-16T02:35:44.949 INFO:tasks.workunit.client.1.smithi153.stdout:2/769: creat d9/d71/df2/f102 x:0 0 0 2024-04-16T02:35:44.950 INFO:tasks.workunit.client.1.smithi153.stdout:1/813: write d4/f1f [2258003,71295] 0 2024-04-16T02:35:44.951 INFO:tasks.workunit.client.1.smithi153.stdout:7/775: chown d0/d4/db/d47/d8d/d93/fdd 7260768 1 2024-04-16T02:35:44.951 INFO:tasks.workunit.client.1.smithi153.stdout:0/897: creat dd/d26/d92/f122 x:0 0 0 2024-04-16T02:35:44.952 INFO:tasks.workunit.client.1.smithi153.stdout:0/898: fsync dd/d28/f4c 0 2024-04-16T02:35:44.953 INFO:tasks.workunit.client.0.smithi151.stdout:8/216: write d11/d22/f30 [306023,7892] 0 2024-04-16T02:35:44.954 INFO:tasks.workunit.client.0.smithi151.stdout:8/217: chown d11/d2a/c32 66904010 1 2024-04-16T02:35:44.955 INFO:tasks.workunit.client.1.smithi153.stdout:3/771: creat d6/d10/d3b/d59/d7c/de1/f108 x:0 0 0 2024-04-16T02:35:44.961 INFO:tasks.workunit.client.1.smithi153.stdout:1/814: mknod d4/dd/d18/d6c/dd8/c118 0 2024-04-16T02:35:44.965 INFO:tasks.workunit.client.1.smithi153.stdout:7/776: symlink d0/de8/d106/l107 0 2024-04-16T02:35:44.970 INFO:tasks.workunit.client.1.smithi153.stdout:7/777: chown d0/d4/db/d34/d4b/f69 236849008 1 2024-04-16T02:35:44.970 INFO:tasks.workunit.client.1.smithi153.stdout:0/899: dwrite dd/d14/d19/f27 [0,4194304] 0 2024-04-16T02:35:44.972 INFO:tasks.workunit.client.1.smithi153.stdout:3/772: dread d6/d10/d22/d4d/fbf [0,4194304] 0 2024-04-16T02:35:44.973 INFO:tasks.workunit.client.1.smithi153.stdout:7/778: chown d0/d4/db/d2b/d43/d6d/d68/lef 1 1 2024-04-16T02:35:44.973 INFO:tasks.workunit.client.1.smithi153.stdout:0/900: mknod dd/d14/d19/d20/d4a/c123 0 2024-04-16T02:35:44.978 INFO:tasks.workunit.client.1.smithi153.stdout:0/901: mknod dd/d14/c124 0 2024-04-16T02:35:44.979 INFO:tasks.workunit.client.1.smithi153.stdout:3/773: dread d6/d10/d22/f2a [0,4194304] 0 2024-04-16T02:35:44.981 INFO:tasks.workunit.client.1.smithi153.stdout:9/853: dread d4/d6/d1b/f59 [0,4194304] 0 2024-04-16T02:35:44.981 INFO:tasks.workunit.client.1.smithi153.stdout:3/774: stat d6/d10/d22/d2c/f60 0 2024-04-16T02:35:44.982 INFO:tasks.workunit.client.1.smithi153.stdout:0/902: chown dd/d14/d51/d54/ld6 5 1 2024-04-16T02:35:44.983 INFO:tasks.workunit.client.1.smithi153.stdout:7/779: write d0/d4/db/d2b/d8c/fa6 [540342,73817] 0 2024-04-16T02:35:44.983 INFO:tasks.workunit.client.0.smithi151.stdout:8/218: dread d11/d2a/f42 [0,4194304] 0 2024-04-16T02:35:44.984 INFO:tasks.workunit.client.1.smithi153.stdout:9/854: unlink d4/d6/dc/d24/d4e/f9c 0 2024-04-16T02:35:44.988 INFO:tasks.workunit.client.0.smithi151.stdout:8/219: creat d11/f4a x:0 0 0 2024-04-16T02:35:44.988 INFO:tasks.workunit.client.0.smithi151.stdout:5/345: dread d2/f5 [0,4194304] 0 2024-04-16T02:35:44.989 INFO:tasks.workunit.client.1.smithi153.stdout:3/775: link d6/d10/d3b/d76/fd1 d6/d10/dec/f109 0 2024-04-16T02:35:44.989 INFO:tasks.workunit.client.1.smithi153.stdout:7/780: rmdir d0/d4/db/d2b/d53/d59/d76/d103 39 2024-04-16T02:35:44.989 INFO:tasks.workunit.client.1.smithi153.stdout:3/776: chown d6/d10/d22/d2e/f80 679926 1 2024-04-16T02:35:44.989 INFO:tasks.workunit.client.1.smithi153.stdout:7/781: fdatasync d0/d4/db/d34/d51/fda 0 2024-04-16T02:35:44.990 INFO:tasks.workunit.client.1.smithi153.stdout:9/855: mkdir d4/d33/dae/dc7/d66/d10a 0 2024-04-16T02:35:44.991 INFO:tasks.workunit.client.0.smithi151.stdout:8/220: creat d11/d2a/d33/f4b x:0 0 0 2024-04-16T02:35:44.992 INFO:tasks.workunit.client.0.smithi151.stdout:5/346: rmdir d2/d6/d19/d30/d36/d42 39 2024-04-16T02:35:44.992 INFO:tasks.workunit.client.1.smithi153.stdout:7/782: unlink d0/d4/c49 0 2024-04-16T02:35:44.992 INFO:tasks.workunit.client.0.smithi151.stdout:5/347: truncate d2/d6/f64 299568 0 2024-04-16T02:35:44.994 INFO:tasks.workunit.client.1.smithi153.stdout:3/777: mknod d6/d10/d22/d2e/d8a/c10a 0 2024-04-16T02:35:44.994 INFO:tasks.workunit.client.1.smithi153.stdout:0/903: getdents dd/d14/d51/d7d 0 2024-04-16T02:35:44.994 INFO:tasks.workunit.client.0.smithi151.stdout:5/348: fdatasync d2/d6/d19/f4b 0 2024-04-16T02:35:44.995 INFO:tasks.workunit.client.1.smithi153.stdout:2/770: dread d9/d2b/d1c/d20/f67 [0,4194304] 0 2024-04-16T02:35:44.997 INFO:tasks.workunit.client.1.smithi153.stdout:0/904: readlink dd/d14/ldc 0 2024-04-16T02:35:44.998 INFO:tasks.workunit.client.0.smithi151.stdout:5/349: chown d2/d6/d19/d1a/c4d 12 1 2024-04-16T02:35:44.999 INFO:tasks.workunit.client.0.smithi151.stdout:8/221: dwrite d11/f4a [0,4194304] 0 2024-04-16T02:35:44.999 INFO:tasks.workunit.client.1.smithi153.stdout:3/778: fsync d6/d10/d5b/f75 0 2024-04-16T02:35:45.000 INFO:tasks.workunit.client.0.smithi151.stdout:5/350: write d2/d6/f29 [1307964,9768] 0 2024-04-16T02:35:45.001 INFO:tasks.workunit.client.0.smithi151.stdout:5/351: chown d2/d6/d19/d69 2 1 2024-04-16T02:35:45.002 INFO:tasks.workunit.client.1.smithi153.stdout:2/771: symlink d9/d24/d27/d2c/d41/d7a/l103 0 2024-04-16T02:35:45.002 INFO:tasks.workunit.client.0.smithi151.stdout:8/222: symlink d11/d2a/l4c 0 2024-04-16T02:35:45.003 INFO:tasks.workunit.client.0.smithi151.stdout:5/352: write d2/f68 [3396177,29846] 0 2024-04-16T02:35:45.005 INFO:tasks.workunit.client.0.smithi151.stdout:5/353: write d2/d6/d19/d1a/f2e [218688,40009] 0 2024-04-16T02:35:45.006 INFO:tasks.workunit.client.1.smithi153.stdout:0/905: dread dd/d14/da6/f98 [0,4194304] 0 2024-04-16T02:35:45.009 INFO:tasks.workunit.client.0.smithi151.stdout:5/354: creat d2/d6/d19/d30/d49/f6e x:0 0 0 2024-04-16T02:35:45.010 INFO:tasks.workunit.client.1.smithi153.stdout:3/779: dwrite d6/d10/d3b/d76/fc2 [0,4194304] 0 2024-04-16T02:35:45.013 INFO:tasks.workunit.client.0.smithi151.stdout:5/355: mkdir d2/d6/d19/d30/d36/d42/d6f 0 2024-04-16T02:35:45.017 INFO:tasks.workunit.client.1.smithi153.stdout:3/780: unlink d6/d10/d3b/d59/d7c/db7/f103 0 2024-04-16T02:35:45.017 INFO:tasks.workunit.client.1.smithi153.stdout:2/772: dwrite d9/d2b/d1c/d20/f101 [0,4194304] 0 2024-04-16T02:35:45.018 INFO:tasks.workunit.client.1.smithi153.stdout:3/781: fdatasync d6/d10/d22/d2c/f3d 0 2024-04-16T02:35:45.028 INFO:tasks.workunit.client.1.smithi153.stdout:3/782: dwrite d6/d10/d22/fdd [4194304,4194304] 0 2024-04-16T02:35:45.042 INFO:tasks.workunit.client.1.smithi153.stdout:6/787: sync 2024-04-16T02:35:45.044 INFO:tasks.workunit.client.1.smithi153.stdout:6/788: chown f6 0 1 2024-04-16T02:35:45.051 INFO:tasks.workunit.client.1.smithi153.stdout:6/789: dwrite dc/d47/d93/fa4 [0,4194304] 0 2024-04-16T02:35:45.056 INFO:tasks.workunit.client.1.smithi153.stdout:0/906: dread dd/d14/d51/d54/d65/f72 [0,4194304] 0 2024-04-16T02:35:45.056 INFO:tasks.workunit.client.1.smithi153.stdout:3/783: truncate d6/d10/d3b/d76/f7d 3132332 0 2024-04-16T02:35:45.058 INFO:tasks.workunit.client.1.smithi153.stdout:0/907: write dd/d14/d19/d77/d87/dda/ff9 [207470,58409] 0 2024-04-16T02:35:45.062 INFO:tasks.workunit.client.1.smithi153.stdout:6/790: unlink dc/d47/d57/d77/d15/l50 0 2024-04-16T02:35:45.065 INFO:tasks.workunit.client.1.smithi153.stdout:2/773: dread d9/d2b/d18/d3f/f49 [0,4194304] 0 2024-04-16T02:35:45.066 INFO:tasks.workunit.client.1.smithi153.stdout:3/784: dwrite d6/d10/d3b/d59/d7c/de1/df0/def/ff9 [0,4194304] 0 2024-04-16T02:35:45.067 INFO:tasks.workunit.client.1.smithi153.stdout:6/791: mkdir dc/d47/d57/d77/d15/d109 0 2024-04-16T02:35:45.070 INFO:tasks.workunit.client.1.smithi153.stdout:0/908: dread dd/d14/da6/da4/fb5 [0,4194304] 0 2024-04-16T02:35:45.075 INFO:tasks.workunit.client.1.smithi153.stdout:0/909: creat dd/d32/d80/f125 x:0 0 0 2024-04-16T02:35:45.080 INFO:tasks.workunit.client.1.smithi153.stdout:0/910: fsync dd/d32/fde 0 2024-04-16T02:35:45.082 INFO:tasks.workunit.client.1.smithi153.stdout:3/785: truncate d6/d10/d22/d2c/f33 5009329 0 2024-04-16T02:35:45.082 INFO:tasks.workunit.client.1.smithi153.stdout:2/774: write d9/d24/d27/d2c/d41/fb5 [507866,128186] 0 2024-04-16T02:35:45.082 INFO:tasks.workunit.client.1.smithi153.stdout:6/792: truncate dc/d10/d62/fce 103108 0 2024-04-16T02:35:45.084 INFO:tasks.workunit.client.1.smithi153.stdout:0/911: truncate dd/d14/d19/d77/db6/fec 495978 0 2024-04-16T02:35:45.084 INFO:tasks.workunit.client.0.smithi151.stdout:7/264: sync 2024-04-16T02:35:45.084 INFO:tasks.workunit.client.1.smithi153.stdout:3/786: chown d6/d10/d22/d43/c5a 1880242 1 2024-04-16T02:35:45.085 INFO:tasks.workunit.client.1.smithi153.stdout:2/775: mknod d9/d2b/d1c/c104 0 2024-04-16T02:35:45.086 INFO:tasks.workunit.client.0.smithi151.stdout:7/265: symlink d0/d35/l54 0 2024-04-16T02:35:45.087 INFO:tasks.workunit.client.1.smithi153.stdout:6/793: creat dc/d59/d5b/f10a x:0 0 0 2024-04-16T02:35:45.089 INFO:tasks.workunit.client.1.smithi153.stdout:0/912: symlink dd/d32/d3f/l126 0 2024-04-16T02:35:45.089 INFO:tasks.workunit.client.1.smithi153.stdout:2/776: dread d9/d2b/d1c/d20/d45/f54 [0,4194304] 0 2024-04-16T02:35:45.090 INFO:tasks.workunit.client.1.smithi153.stdout:2/777: write d9/f8b [4714417,123499] 0 2024-04-16T02:35:45.091 INFO:tasks.workunit.client.1.smithi153.stdout:2/778: chown d9/d2b/d18 4796 1 2024-04-16T02:35:45.091 INFO:tasks.workunit.client.1.smithi153.stdout:6/794: rmdir dc/d1a/d2d 39 2024-04-16T02:35:45.092 INFO:tasks.workunit.client.0.smithi151.stdout:7/266: getdents d0/d1f/d46/d4f 0 2024-04-16T02:35:45.093 INFO:tasks.workunit.client.1.smithi153.stdout:0/913: chown dd/d14/f107 0 1 2024-04-16T02:35:45.094 INFO:tasks.workunit.client.0.smithi151.stdout:7/267: mknod d0/c55 0 2024-04-16T02:35:45.095 INFO:tasks.workunit.client.1.smithi153.stdout:2/779: unlink d9/d24/d27/d2c/d5d/da3/fa7 0 2024-04-16T02:35:45.095 INFO:tasks.workunit.client.1.smithi153.stdout:2/780: fdatasync d9/d2b/d5c/fd2 0 2024-04-16T02:35:45.097 INFO:tasks.workunit.client.1.smithi153.stdout:0/914: getdents dd/d14/d51/ddd 0 2024-04-16T02:35:45.101 INFO:tasks.workunit.client.1.smithi153.stdout:2/781: getdents d9/d24/d34/d60/d7c 0 2024-04-16T02:35:45.103 INFO:tasks.workunit.client.0.smithi151.stdout:7/268: mkdir d0/d1f/d33/d56 0 2024-04-16T02:35:45.104 INFO:tasks.workunit.client.1.smithi153.stdout:6/795: dwrite dc/d47/f4a [0,4194304] 0 2024-04-16T02:35:45.105 INFO:tasks.workunit.client.1.smithi153.stdout:0/915: dread dd/d14/da6/d73/f8f [0,4194304] 0 2024-04-16T02:35:45.109 INFO:tasks.workunit.client.0.smithi151.stdout:7/269: write d0/f1d [780869,57526] 0 2024-04-16T02:35:45.111 INFO:tasks.workunit.client.1.smithi153.stdout:2/782: dread d9/d2b/d18/fe4 [0,4194304] 0 2024-04-16T02:35:45.112 INFO:tasks.workunit.client.0.smithi151.stdout:7/270: fdatasync d0/d2c/f3b 0 2024-04-16T02:35:45.113 INFO:tasks.workunit.client.1.smithi153.stdout:3/787: dwrite d6/f39 [0,4194304] 0 2024-04-16T02:35:45.114 INFO:tasks.workunit.client.1.smithi153.stdout:6/796: mknod dc/d47/d57/d77/c10b 0 2024-04-16T02:35:45.114 INFO:tasks.workunit.client.0.smithi151.stdout:7/271: write d0/f7 [8288343,84085] 0 2024-04-16T02:35:45.116 INFO:tasks.workunit.client.1.smithi153.stdout:2/783: symlink d9/d2b/d5c/d5f/l105 0 2024-04-16T02:35:45.116 INFO:tasks.workunit.client.0.smithi151.stdout:7/272: fsync d0/f5 0 2024-04-16T02:35:45.118 INFO:tasks.workunit.client.0.smithi151.stdout:7/273: read d0/d1f/d33/f3a [133799,6076] 0 2024-04-16T02:35:45.121 INFO:tasks.workunit.client.1.smithi153.stdout:0/916: creat dd/d28/f127 x:0 0 0 2024-04-16T02:35:45.121 INFO:tasks.workunit.client.1.smithi153.stdout:0/917: stat dd/d14/da6/d73 0 2024-04-16T02:35:45.124 INFO:tasks.workunit.client.1.smithi153.stdout:6/797: creat dc/d10/f10c x:0 0 0 2024-04-16T02:35:45.125 INFO:tasks.workunit.client.1.smithi153.stdout:2/784: symlink d9/d24/d27/d2c/d41/l106 0 2024-04-16T02:35:45.126 INFO:tasks.workunit.client.1.smithi153.stdout:3/788: unlink d6/d10/d22/d2c/l4f 0 2024-04-16T02:35:45.127 INFO:tasks.workunit.client.1.smithi153.stdout:3/789: chown d6/d10/dec/df8 1256941962 1 2024-04-16T02:35:45.128 INFO:tasks.workunit.client.1.smithi153.stdout:0/918: creat dd/d32/d57/d6c/dd1/f128 x:0 0 0 2024-04-16T02:35:45.129 INFO:tasks.workunit.client.1.smithi153.stdout:6/798: mknod dc/d47/d57/d77/d7a/c10d 0 2024-04-16T02:35:45.132 INFO:tasks.workunit.client.1.smithi153.stdout:0/919: fsync dd/d14/d51/d7d/fce 0 2024-04-16T02:35:45.132 INFO:tasks.workunit.client.0.smithi151.stdout:7/274: truncate d0/d1f/f2a 435235 0 2024-04-16T02:35:45.133 INFO:tasks.workunit.client.1.smithi153.stdout:6/799: fdatasync dc/d47/d7d/fdd 0 2024-04-16T02:35:45.133 INFO:tasks.workunit.client.1.smithi153.stdout:0/920: fsync dd/d14/d19/d77/d87/dda/ff9 0 2024-04-16T02:35:45.134 INFO:tasks.workunit.client.0.smithi151.stdout:7/275: rmdir d0/d1f/d31/d42 39 2024-04-16T02:35:45.135 INFO:tasks.workunit.client.1.smithi153.stdout:6/800: chown dc/d47/d57/d101/cb2 1 1 2024-04-16T02:35:45.135 INFO:tasks.workunit.client.1.smithi153.stdout:6/801: chown dc/l28 19596459 1 2024-04-16T02:35:45.137 INFO:tasks.workunit.client.1.smithi153.stdout:2/785: rename d9/d24/d27/d2c/d5d/c95 to d9/c107 0 2024-04-16T02:35:45.137 INFO:tasks.workunit.client.1.smithi153.stdout:2/786: chown d9/d71/la1 127 1 2024-04-16T02:35:45.137 INFO:tasks.workunit.client.0.smithi151.stdout:7/276: creat d0/d1f/d46/d4f/f57 x:0 0 0 2024-04-16T02:35:45.138 INFO:tasks.workunit.client.0.smithi151.stdout:7/277: write d0/d1f/f52 [894689,156] 0 2024-04-16T02:35:45.139 INFO:tasks.workunit.client.0.smithi151.stdout:7/278: fsync d0/d2c/d38/f3d 0 2024-04-16T02:35:45.141 INFO:tasks.workunit.client.1.smithi153.stdout:6/802: creat dc/d47/d57/d77/dc4/dc9/ded/f10e x:0 0 0 2024-04-16T02:35:45.141 INFO:tasks.workunit.client.1.smithi153.stdout:0/921: creat dd/d14/d51/ddd/f129 x:0 0 0 2024-04-16T02:35:45.141 INFO:tasks.workunit.client.1.smithi153.stdout:6/803: fsync dc/d10/f88 0 2024-04-16T02:35:45.142 INFO:tasks.workunit.client.0.smithi151.stdout:7/279: symlink d0/d1f/d31/l58 0 2024-04-16T02:35:45.142 INFO:tasks.workunit.client.1.smithi153.stdout:0/922: chown dd/d32/d3f 33250930 1 2024-04-16T02:35:45.142 INFO:tasks.workunit.client.1.smithi153.stdout:2/787: dread - d9/d71/d8d/fbd zero size 2024-04-16T02:35:45.143 INFO:tasks.workunit.client.1.smithi153.stdout:0/923: fsync dd/d32/f85 0 2024-04-16T02:35:45.144 INFO:tasks.workunit.client.1.smithi153.stdout:6/804: chown dc/d47/d57/d77/c8d 1716142 1 2024-04-16T02:35:45.144 INFO:tasks.workunit.client.1.smithi153.stdout:0/924: dread - dd/d14/da6/da4/db1/f119 zero size 2024-04-16T02:35:45.147 INFO:tasks.workunit.client.1.smithi153.stdout:6/805: readlink dc/d47/d57/d77/d15/l3e 0 2024-04-16T02:35:45.149 INFO:tasks.workunit.client.0.smithi151.stdout:7/280: dwrite d0/d1f/f52 [0,4194304] 0 2024-04-16T02:35:45.149 INFO:tasks.workunit.client.1.smithi153.stdout:2/788: mknod d9/d24/d27/d2c/d5d/de5/c108 0 2024-04-16T02:35:45.149 INFO:tasks.workunit.client.1.smithi153.stdout:3/790: write d6/d10/d22/d43/f57 [6425117,42460] 0 2024-04-16T02:35:45.151 INFO:tasks.workunit.client.1.smithi153.stdout:0/925: dwrite dd/d14/da6/da4/fd3 [0,4194304] 0 2024-04-16T02:35:45.152 INFO:tasks.workunit.client.0.smithi151.stdout:7/281: write d0/d1f/d33/f3a [1550642,116451] 0 2024-04-16T02:35:45.152 INFO:tasks.workunit.client.1.smithi153.stdout:2/789: dread - d9/d2b/d1c/fe2 zero size 2024-04-16T02:35:45.156 INFO:tasks.workunit.client.0.smithi151.stdout:7/282: creat d0/f59 x:0 0 0 2024-04-16T02:35:45.157 INFO:tasks.workunit.client.1.smithi153.stdout:0/926: creat dd/d14/d51/ddd/f12a x:0 0 0 2024-04-16T02:35:45.158 INFO:tasks.workunit.client.0.smithi151.stdout:7/283: dread - d0/d1f/f29 zero size 2024-04-16T02:35:45.158 INFO:tasks.workunit.client.1.smithi153.stdout:0/927: chown dd/d14/da6/d73/lbd 17 1 2024-04-16T02:35:45.159 INFO:tasks.workunit.client.1.smithi153.stdout:3/791: mkdir d6/d10/d3b/d76/d10b 0 2024-04-16T02:35:45.161 INFO:tasks.workunit.client.1.smithi153.stdout:6/806: rename dc/d1a/f68 to dc/d47/d57/f10f 0 2024-04-16T02:35:45.163 INFO:tasks.workunit.client.0.smithi151.stdout:7/284: creat d0/d1f/d46/f5a x:0 0 0 2024-04-16T02:35:45.164 INFO:tasks.workunit.client.0.smithi151.stdout:7/285: read d0/d1f/f28 [179510,115847] 0 2024-04-16T02:35:45.167 INFO:tasks.workunit.client.1.smithi153.stdout:4/901: sync 2024-04-16T02:35:45.167 INFO:tasks.workunit.client.1.smithi153.stdout:5/857: sync 2024-04-16T02:35:45.167 INFO:tasks.workunit.client.1.smithi153.stdout:0/928: creat dd/d14/dca/f12b x:0 0 0 2024-04-16T02:35:45.167 INFO:tasks.workunit.client.1.smithi153.stdout:3/792: symlink d6/daf/l10c 0 2024-04-16T02:35:45.169 INFO:tasks.workunit.client.0.smithi151.stdout:7/286: symlink d0/d2c/d38/d3e/l5b 0 2024-04-16T02:35:45.171 INFO:tasks.workunit.client.1.smithi153.stdout:5/858: rmdir d8/d41/dbd 39 2024-04-16T02:35:45.171 INFO:tasks.workunit.client.1.smithi153.stdout:4/902: truncate d3/d9/db/de/d2c/d52/d85/ffe 4655661 0 2024-04-16T02:35:45.173 INFO:tasks.workunit.client.1.smithi153.stdout:5/859: readlink d8/df/lc2 0 2024-04-16T02:35:45.177 INFO:tasks.workunit.client.0.smithi151.stdout:7/287: dwrite d0/d27/f2b [0,4194304] 0 2024-04-16T02:35:45.179 INFO:tasks.workunit.client.0.smithi151.stdout:7/288: mknod d0/d2c/d38/d43/c5c 0 2024-04-16T02:35:45.183 INFO:tasks.workunit.client.0.smithi151.stdout:7/289: creat d0/d1f/d46/f5d x:0 0 0 2024-04-16T02:35:45.193 INFO:tasks.workunit.client.0.smithi151.stdout:7/290: dwrite d0/d27/f2b [4194304,4194304] 0 2024-04-16T02:35:45.213 INFO:tasks.workunit.client.0.smithi151.stdout:7/291: dread d0/f22 [4194304,4194304] 0 2024-04-16T02:35:45.220 INFO:tasks.workunit.client.0.smithi151.stdout:7/292: dwrite d0/d1f/f28 [0,4194304] 0 2024-04-16T02:35:45.226 INFO:tasks.workunit.client.0.smithi151.stdout:7/293: dwrite d0/d2c/d38/f3d [0,4194304] 0 2024-04-16T02:35:45.227 INFO:tasks.workunit.client.0.smithi151.stdout:7/294: fdatasync d0/d1f/f40 0 2024-04-16T02:35:45.229 INFO:tasks.workunit.client.0.smithi151.stdout:7/295: mkdir d0/d2c/d38/d5e 0 2024-04-16T02:35:45.231 INFO:tasks.workunit.client.0.smithi151.stdout:7/296: rmdir d0/d27 39 2024-04-16T02:35:45.234 INFO:tasks.workunit.client.0.smithi151.stdout:7/297: dread d0/d1f/f52 [0,4194304] 0 2024-04-16T02:35:45.239 INFO:tasks.workunit.client.0.smithi151.stdout:7/298: getdents d0/d1f 0 2024-04-16T02:35:45.240 INFO:tasks.workunit.client.0.smithi151.stdout:7/299: creat d0/d2c/f5f x:0 0 0 2024-04-16T02:35:45.246 INFO:tasks.workunit.client.0.smithi151.stdout:7/300: dwrite d0/f7 [4194304,4194304] 0 2024-04-16T02:35:45.247 INFO:tasks.workunit.client.0.smithi151.stdout:7/301: fdatasync d0/f59 0 2024-04-16T02:35:45.251 INFO:tasks.workunit.client.0.smithi151.stdout:7/302: mkdir d0/d1f/d60 0 2024-04-16T02:35:45.252 INFO:tasks.workunit.client.0.smithi151.stdout:7/303: dread - d0/d1f/d46/f5a zero size 2024-04-16T02:35:45.253 INFO:tasks.workunit.client.1.smithi153.stdout:3/793: dwrite d6/d10/d3b/d59/d7c/de1/df0/fc6 [0,4194304] 0 2024-04-16T02:35:45.256 INFO:tasks.workunit.client.1.smithi153.stdout:4/903: creat d3/d9/db/d17/d28/dc8/de4/dfd/d120/f13b x:0 0 0 2024-04-16T02:35:45.260 INFO:tasks.workunit.client.1.smithi153.stdout:2/790: dwrite d9/d24/d27/d2c/d41/f44 [0,4194304] 0 2024-04-16T02:35:45.263 INFO:tasks.workunit.client.1.smithi153.stdout:0/929: dwrite dd/f22 [4194304,4194304] 0 2024-04-16T02:35:45.263 INFO:tasks.workunit.client.1.smithi153.stdout:5/860: creat d8/df/f110 x:0 0 0 2024-04-16T02:35:45.265 INFO:tasks.workunit.client.1.smithi153.stdout:4/904: dread - d3/da5/fc6 zero size 2024-04-16T02:35:45.269 INFO:tasks.workunit.client.1.smithi153.stdout:2/791: fsync d9/d24/d27/f92 0 2024-04-16T02:35:45.270 INFO:tasks.workunit.client.1.smithi153.stdout:0/930: creat dd/d14/da6/de0/f12c x:0 0 0 2024-04-16T02:35:45.270 INFO:tasks.workunit.client.1.smithi153.stdout:4/905: write d3/d9/db/d17/d28/dc8/ff1 [158479,17641] 0 2024-04-16T02:35:45.271 INFO:tasks.workunit.client.1.smithi153.stdout:0/931: chown dd/d32/fc8 4760023 1 2024-04-16T02:35:45.271 INFO:tasks.workunit.client.1.smithi153.stdout:5/861: symlink d8/df/d28/d5a/d66/l111 0 2024-04-16T02:35:45.274 INFO:tasks.workunit.client.1.smithi153.stdout:2/792: dread - d9/d24/d34/d60/ff6 zero size 2024-04-16T02:35:45.275 INFO:tasks.workunit.client.1.smithi153.stdout:5/862: write d8/df/d28/f8a [6098561,100424] 0 2024-04-16T02:35:45.275 INFO:tasks.workunit.client.1.smithi153.stdout:4/906: dread - d3/da5/dd2/f127 zero size 2024-04-16T02:35:45.275 INFO:tasks.workunit.client.1.smithi153.stdout:2/793: chown d9/d24/d34/f7d 470044 1 2024-04-16T02:35:45.276 INFO:tasks.workunit.client.1.smithi153.stdout:0/932: mknod dd/d32/d57/d6c/c12d 0 2024-04-16T02:35:45.278 INFO:tasks.workunit.client.1.smithi153.stdout:5/863: dread - d8/d1f/f107 zero size 2024-04-16T02:35:45.278 INFO:tasks.workunit.client.1.smithi153.stdout:3/794: dread d6/d10/d22/d2c/f33 [4194304,4194304] 0 2024-04-16T02:35:45.279 INFO:tasks.workunit.client.0.smithi151.stdout:7/304: truncate d0/d2c/d38/f3d 3084609 0 2024-04-16T02:35:45.279 INFO:tasks.workunit.client.1.smithi153.stdout:3/795: chown d6/d10/d3b/d76 27 1 2024-04-16T02:35:45.281 INFO:tasks.workunit.client.1.smithi153.stdout:4/907: creat d3/d9/d24/d75/f13c x:0 0 0 2024-04-16T02:35:45.282 INFO:tasks.workunit.client.1.smithi153.stdout:2/794: truncate d9/d24/d27/f68 2557585 0 2024-04-16T02:35:45.284 INFO:tasks.workunit.client.1.smithi153.stdout:5/864: write d8/df/d28/d5a/fb8 [744029,25546] 0 2024-04-16T02:35:45.284 INFO:tasks.workunit.client.0.smithi151.stdout:7/305: link d0/d1f/d31/l48 d0/d27/l61 0 2024-04-16T02:35:45.287 INFO:tasks.workunit.client.1.smithi153.stdout:2/795: write d9/d24/d34/d60/ddc/ff1 [723559,47190] 0 2024-04-16T02:35:45.288 INFO:tasks.workunit.client.1.smithi153.stdout:3/796: rename d6/d10/d22/d2e to d6/d10/d3b/d10d 0 2024-04-16T02:35:45.288 INFO:tasks.workunit.client.0.smithi151.stdout:7/306: rename d0/f59 to d0/d1f/d46/d4f/f62 0 2024-04-16T02:35:45.288 INFO:tasks.workunit.client.1.smithi153.stdout:0/933: dwrite dd/d14/d51/f59 [0,4194304] 0 2024-04-16T02:35:45.289 INFO:tasks.workunit.client.1.smithi153.stdout:3/797: stat d6/d10/d22/d43 0 2024-04-16T02:35:45.291 INFO:tasks.workunit.client.0.smithi151.stdout:7/307: mknod d0/d1f/d46/d4f/c63 0 2024-04-16T02:35:45.291 INFO:tasks.workunit.client.1.smithi153.stdout:5/865: unlink d8/df/d23/d70/ccc 0 2024-04-16T02:35:45.291 INFO:tasks.workunit.client.1.smithi153.stdout:3/798: chown d6/d10/d22/d4d 12945339 1 2024-04-16T02:35:45.294 INFO:tasks.workunit.client.1.smithi153.stdout:0/934: write dd/d26/d44/dd2/f111 [78413,43221] 0 2024-04-16T02:35:45.294 INFO:tasks.workunit.client.0.smithi151.stdout:7/308: rename d0/d27/c41 to d0/d2c/d38/d5e/c64 0 2024-04-16T02:35:45.296 INFO:tasks.workunit.client.1.smithi153.stdout:2/796: chown d9/d2b/d5c/da2/lb4 541 1 2024-04-16T02:35:45.298 INFO:tasks.workunit.client.1.smithi153.stdout:3/799: truncate d6/d10/d22/d43/f69 2290094 0 2024-04-16T02:35:45.298 INFO:tasks.workunit.client.0.smithi151.stdout:7/309: symlink d0/d1f/d31/d42/l65 0 2024-04-16T02:35:45.299 INFO:tasks.workunit.client.1.smithi153.stdout:3/800: chown d6/d10/d22/f2a 244729700 1 2024-04-16T02:35:45.300 INFO:tasks.workunit.client.1.smithi153.stdout:5/866: unlink d8/d41/dbd/d83/fc7 0 2024-04-16T02:35:45.301 INFO:tasks.workunit.client.1.smithi153.stdout:4/908: getdents d3/d9/db/d17/d3f 0 2024-04-16T02:35:45.302 INFO:tasks.workunit.client.1.smithi153.stdout:0/935: mkdir dd/d32/d12e 0 2024-04-16T02:35:45.302 INFO:tasks.workunit.client.1.smithi153.stdout:5/867: fdatasync d8/df/d2f/d4e/f94 0 2024-04-16T02:35:45.306 INFO:tasks.workunit.client.1.smithi153.stdout:5/868: fsync d8/df/d23/f32 0 2024-04-16T02:35:45.307 INFO:tasks.workunit.client.1.smithi153.stdout:4/909: creat d3/d9/db/d17/d3f/d6b/f13d x:0 0 0 2024-04-16T02:35:45.307 INFO:tasks.workunit.client.1.smithi153.stdout:4/910: chown d3/f1c 0 1 2024-04-16T02:35:45.307 INFO:tasks.workunit.client.0.smithi151.stdout:9/245: sync 2024-04-16T02:35:45.307 INFO:tasks.workunit.client.0.smithi151.stdout:3/218: sync 2024-04-16T02:35:45.308 INFO:tasks.workunit.client.0.smithi151.stdout:6/425: sync 2024-04-16T02:35:45.308 INFO:tasks.workunit.client.0.smithi151.stdout:2/337: sync 2024-04-16T02:35:45.308 INFO:tasks.workunit.client.0.smithi151.stdout:0/262: sync 2024-04-16T02:35:45.308 INFO:tasks.workunit.client.0.smithi151.stdout:1/288: sync 2024-04-16T02:35:45.308 INFO:tasks.workunit.client.0.smithi151.stdout:4/331: sync 2024-04-16T02:35:45.308 INFO:tasks.workunit.client.0.smithi151.stdout:3/219: write d1/d6/f21 [159204,94311] 0 2024-04-16T02:35:45.309 INFO:tasks.workunit.client.1.smithi153.stdout:5/869: mkdir d8/d41/dbd/d83/d112 0 2024-04-16T02:35:45.309 INFO:tasks.workunit.client.0.smithi151.stdout:9/246: truncate d1/d3/d6/f13 4474448 0 2024-04-16T02:35:45.309 INFO:tasks.workunit.client.0.smithi151.stdout:3/220: read - d1/d37/f38 zero size 2024-04-16T02:35:45.310 INFO:tasks.workunit.client.1.smithi153.stdout:0/936: dwrite dd/d14/d19/d77/d87/dda/f11d [0,4194304] 0 2024-04-16T02:35:45.311 INFO:tasks.workunit.client.0.smithi151.stdout:2/338: chown de/d15/l23 48312 1 2024-04-16T02:35:45.312 INFO:tasks.workunit.client.0.smithi151.stdout:0/263: mknod dc/d15/d34/c53 0 2024-04-16T02:35:45.316 INFO:tasks.workunit.client.1.smithi153.stdout:8/768: sync 2024-04-16T02:35:45.316 INFO:tasks.workunit.client.1.smithi153.stdout:9/856: sync 2024-04-16T02:35:45.316 INFO:tasks.workunit.client.1.smithi153.stdout:7/783: sync 2024-04-16T02:35:45.316 INFO:tasks.workunit.client.1.smithi153.stdout:1/815: sync 2024-04-16T02:35:45.316 INFO:tasks.workunit.client.1.smithi153.stdout:4/911: creat d3/d9/d24/d75/dc5/f13e x:0 0 0 2024-04-16T02:35:45.317 INFO:tasks.workunit.client.1.smithi153.stdout:2/797: link d9/d2b/c23 d9/d2b/df9/c109 0 2024-04-16T02:35:45.318 INFO:tasks.workunit.client.1.smithi153.stdout:8/769: dread - d2/dba/fc9 zero size 2024-04-16T02:35:45.321 INFO:tasks.workunit.client.1.smithi153.stdout:5/870: unlink d8/df/d23/f32 0 2024-04-16T02:35:45.321 INFO:tasks.workunit.client.1.smithi153.stdout:8/770: dread - d2/d93/d97/d9c/d27/ff9 zero size 2024-04-16T02:35:45.321 INFO:tasks.workunit.client.0.smithi151.stdout:1/289: dwrite d3/d58/d1b/d2a/f30 [0,4194304] 0 2024-04-16T02:35:45.321 INFO:tasks.workunit.client.0.smithi151.stdout:7/310: write d0/d1f/f2a [579675,84935] 0 2024-04-16T02:35:45.322 INFO:tasks.workunit.client.0.smithi151.stdout:4/332: dwrite d0/d17/f2b [0,4194304] 0 2024-04-16T02:35:45.322 INFO:tasks.workunit.client.1.smithi153.stdout:0/937: write dd/d14/f4b [943260,71725] 0 2024-04-16T02:35:45.324 INFO:tasks.workunit.client.1.smithi153.stdout:5/871: read - d8/df/d23/d70/fd9 zero size 2024-04-16T02:35:45.324 INFO:tasks.workunit.client.0.smithi151.stdout:7/311: write d0/d1f/d33/f36 [8970718,4335] 0 2024-04-16T02:35:45.325 INFO:tasks.workunit.client.1.smithi153.stdout:1/816: symlink d4/dd/d13/d26/db4/l119 0 2024-04-16T02:35:45.326 INFO:tasks.workunit.client.1.smithi153.stdout:9/857: truncate d4/d6/dc/d24/f83 1282161 0 2024-04-16T02:35:45.329 INFO:tasks.workunit.client.0.smithi151.stdout:1/290: dread - d3/d58/f5e zero size 2024-04-16T02:35:45.331 INFO:tasks.workunit.client.0.smithi151.stdout:0/264: creat dc/d15/d34/f54 x:0 0 0 2024-04-16T02:35:45.331 INFO:tasks.workunit.client.1.smithi153.stdout:8/771: fsync d2/de/f3e 0 2024-04-16T02:35:45.331 INFO:tasks.workunit.client.1.smithi153.stdout:0/938: mknod dd/d14/d51/d7f/d9d/c12f 0 2024-04-16T02:35:45.332 INFO:tasks.workunit.client.0.smithi151.stdout:2/339: getdents de/d3d 0 2024-04-16T02:35:45.333 INFO:tasks.workunit.client.0.smithi151.stdout:2/340: chown de/c22 1761 1 2024-04-16T02:35:45.333 INFO:tasks.workunit.client.0.smithi151.stdout:0/265: dread - dc/f43 zero size 2024-04-16T02:35:45.334 INFO:tasks.workunit.client.1.smithi153.stdout:9/858: write d4/d6/d1b/d6e/f74 [1375608,59749] 0 2024-04-16T02:35:45.334 INFO:tasks.workunit.client.1.smithi153.stdout:1/817: truncate d4/dd/d13/d23/fd3 4942606 0 2024-04-16T02:35:45.335 INFO:tasks.workunit.client.0.smithi151.stdout:3/221: link d1/d6/d20/l2f d1/d6/d20/l4b 0 2024-04-16T02:35:45.337 INFO:tasks.workunit.client.1.smithi153.stdout:7/784: dwrite d0/d4/db/d2b/d53/f46 [0,4194304] 0 2024-04-16T02:35:45.337 INFO:tasks.workunit.client.0.smithi151.stdout:2/341: dread - de/d11/d42/f51 zero size 2024-04-16T02:35:45.337 INFO:tasks.workunit.client.0.smithi151.stdout:3/222: dread - d1/f26 zero size 2024-04-16T02:35:45.338 INFO:tasks.workunit.client.1.smithi153.stdout:2/798: dwrite d9/d24/d34/d60/fa9 [0,4194304] 0 2024-04-16T02:35:45.338 INFO:tasks.workunit.client.0.smithi151.stdout:4/333: dwrite d0/d11/d19/d31/d4a/d60/f70 [0,4194304] 0 2024-04-16T02:35:45.342 INFO:tasks.workunit.client.0.smithi151.stdout:2/342: read - de/f5c zero size 2024-04-16T02:35:45.342 INFO:tasks.workunit.client.1.smithi153.stdout:2/799: chown d9/d24/d34/d60/ddc/f100 770 1 2024-04-16T02:35:45.344 INFO:tasks.workunit.client.1.smithi153.stdout:9/859: truncate d4/d33/ff5 328361 0 2024-04-16T02:35:45.347 INFO:tasks.workunit.client.1.smithi153.stdout:8/772: mkdir d2/d92/d102 0 2024-04-16T02:35:45.352 INFO:tasks.workunit.client.1.smithi153.stdout:8/773: write d2/d92/deb/d47/d60/d67/d86/f89 [574690,82652] 0 2024-04-16T02:35:45.352 INFO:tasks.workunit.client.0.smithi151.stdout:7/312: getdents d0/d1f/d31/d42 0 2024-04-16T02:35:45.353 INFO:tasks.workunit.client.0.smithi151.stdout:0/266: creat dc/d15/d34/d4e/f55 x:0 0 0 2024-04-16T02:35:45.355 INFO:tasks.workunit.client.1.smithi153.stdout:0/939: dwrite dd/d14/da6/f98 [0,4194304] 0 2024-04-16T02:35:45.356 INFO:tasks.workunit.client.1.smithi153.stdout:8/774: read - d2/d13/d2c/f101 zero size 2024-04-16T02:35:45.356 INFO:tasks.workunit.client.1.smithi153.stdout:7/785: symlink d0/d4/db/d2b/d43/d6d/l108 0 2024-04-16T02:35:45.357 INFO:tasks.workunit.client.1.smithi153.stdout:2/800: rename d9/d24/d27/d9a/dae to d9/d24/d27/d2c/d5d/d10a 0 2024-04-16T02:35:45.358 INFO:tasks.workunit.client.0.smithi151.stdout:4/334: dwrite d0/d17/f1d [0,4194304] 0 2024-04-16T02:35:45.359 INFO:tasks.workunit.client.0.smithi151.stdout:2/343: dwrite de/d11/d41/f37 [0,4194304] 0 2024-04-16T02:35:45.360 INFO:tasks.workunit.client.0.smithi151.stdout:0/267: truncate dc/d20/d2c/f3e 690946 0 2024-04-16T02:35:45.360 INFO:tasks.workunit.client.0.smithi151.stdout:7/313: symlink d0/d2c/d38/l66 0 2024-04-16T02:35:45.361 INFO:tasks.workunit.client.1.smithi153.stdout:9/860: dwrite d4/d6/dc/d24/d4e/f81 [4194304,4194304] 0 2024-04-16T02:35:45.361 INFO:tasks.workunit.client.0.smithi151.stdout:3/223: getdents d1 0 2024-04-16T02:35:45.361 INFO:tasks.workunit.client.0.smithi151.stdout:0/268: stat dc/d15/d34/d4e 0 2024-04-16T02:35:45.363 INFO:tasks.workunit.client.0.smithi151.stdout:4/335: write d0/d11/d19/d31/d34/d3a/d50/f67 [670227,79299] 0 2024-04-16T02:35:45.364 INFO:tasks.workunit.client.0.smithi151.stdout:0/269: unlink dc/f11 0 2024-04-16T02:35:45.366 INFO:tasks.workunit.client.1.smithi153.stdout:8/775: write d2/d93/d97/d9c/fdc [893669,51116] 0 2024-04-16T02:35:45.366 INFO:tasks.workunit.client.0.smithi151.stdout:3/224: symlink d1/d6/d20/d2a/d3f/l4c 0 2024-04-16T02:35:45.367 INFO:tasks.workunit.client.0.smithi151.stdout:7/314: rename d0/d1f/d31/l58 to d0/d2c/d38/d43/l67 0 2024-04-16T02:35:45.371 INFO:tasks.workunit.client.0.smithi151.stdout:3/225: creat d1/d6/d22/f4d x:0 0 0 2024-04-16T02:35:45.372 INFO:tasks.workunit.client.0.smithi151.stdout:3/226: dread - d1/f2e zero size 2024-04-16T02:35:45.372 INFO:tasks.workunit.client.1.smithi153.stdout:2/801: fsync d9/d2b/f16 0 2024-04-16T02:35:45.373 INFO:tasks.workunit.client.0.smithi151.stdout:3/227: write d1/f2e [77689,105363] 0 2024-04-16T02:35:45.374 INFO:tasks.workunit.client.0.smithi151.stdout:7/315: rename d0/d1f/d31/d42/c4a to d0/d2c/d38/d3e/c68 0 2024-04-16T02:35:45.374 INFO:tasks.workunit.client.0.smithi151.stdout:9/247: dread d1/d3/d6/f2a [0,4194304] 0 2024-04-16T02:35:45.375 INFO:tasks.workunit.client.0.smithi151.stdout:3/228: chown d1/d6/d22/d23/d30/d44/d46 1194813 1 2024-04-16T02:35:45.376 INFO:tasks.workunit.client.0.smithi151.stdout:9/248: read d1/fc [2317805,38203] 0 2024-04-16T02:35:45.376 INFO:tasks.workunit.client.1.smithi153.stdout:7/786: link d0/d4/db/d2b/d53/d59/f67 d0/d4/db/f109 0 2024-04-16T02:35:45.377 INFO:tasks.workunit.client.1.smithi153.stdout:0/940: fdatasync dd/d14/d19/fa0 0 2024-04-16T02:35:45.378 INFO:tasks.workunit.client.0.smithi151.stdout:3/229: write d1/d37/f38 [713867,46679] 0 2024-04-16T02:35:45.380 INFO:tasks.workunit.client.0.smithi151.stdout:9/249: truncate d1/d3/d6/d1d/d22/d36/f3d 31771 0 2024-04-16T02:35:45.381 INFO:tasks.workunit.client.1.smithi153.stdout:4/912: truncate d3/d9/db/d17/d28/dc8/f11f 468552 0 2024-04-16T02:35:45.384 INFO:tasks.workunit.client.0.smithi151.stdout:7/316: dwrite d0/d1f/d33/f3a [0,4194304] 0 2024-04-16T02:35:45.385 INFO:tasks.workunit.client.1.smithi153.stdout:0/941: getdents dd/d32/d12e 0 2024-04-16T02:35:45.386 INFO:tasks.workunit.client.1.smithi153.stdout:0/942: chown dd/d14/ldc 46134 1 2024-04-16T02:35:45.389 INFO:tasks.workunit.client.1.smithi153.stdout:7/787: rename d0/d4/db/d2b/d43/d6d/de7 to d0/d4/db/d10a 0 2024-04-16T02:35:45.392 INFO:tasks.workunit.client.1.smithi153.stdout:1/818: dread d4/f7 [4194304,4194304] 0 2024-04-16T02:35:45.392 INFO:tasks.workunit.client.1.smithi153.stdout:0/943: unlink dd/d32/d80/l94 0 2024-04-16T02:35:45.393 INFO:tasks.workunit.client.0.smithi151.stdout:7/317: symlink d0/l69 0 2024-04-16T02:35:45.393 INFO:tasks.workunit.client.0.smithi151.stdout:9/250: dwrite d1/d3/fa [0,4194304] 0 2024-04-16T02:35:45.394 INFO:tasks.workunit.client.1.smithi153.stdout:1/819: write d4/fc4 [5199805,125431] 0 2024-04-16T02:35:45.396 INFO:tasks.workunit.client.0.smithi151.stdout:7/318: chown d0/l8 7 1 2024-04-16T02:35:45.399 INFO:tasks.workunit.client.0.smithi151.stdout:9/251: mkdir d1/d3/d6/d4f 0 2024-04-16T02:35:45.399 INFO:tasks.workunit.client.1.smithi153.stdout:4/913: rename d3/d9/db/de/d2c/d38/caa to d3/d9/d24/d3e/c13f 0 2024-04-16T02:35:45.400 INFO:tasks.workunit.client.0.smithi151.stdout:9/252: write d1/d3/f1a [578775,22150] 0 2024-04-16T02:35:45.400 INFO:tasks.workunit.client.1.smithi153.stdout:1/820: stat d4/dd/d13/d8d/d95/dec 0 2024-04-16T02:35:45.400 INFO:tasks.workunit.client.0.smithi151.stdout:7/319: symlink d0/d2c/d38/l6a 0 2024-04-16T02:35:45.401 INFO:tasks.workunit.client.1.smithi153.stdout:0/944: mkdir dd/d14/d19/d130 0 2024-04-16T02:35:45.401 INFO:tasks.workunit.client.0.smithi151.stdout:9/253: truncate d1/d3/d6/f1b 441458 0 2024-04-16T02:35:45.403 INFO:tasks.workunit.client.0.smithi151.stdout:7/320: symlink d0/d1f/d46/l6b 0 2024-04-16T02:35:45.404 INFO:tasks.workunit.client.0.smithi151.stdout:9/254: mknod d1/d3/d6/d17/d40/c50 0 2024-04-16T02:35:45.405 INFO:tasks.workunit.client.1.smithi153.stdout:7/788: getdents d0/d4/db/d2b/d43/d6d/d68/d6c/db8/ddf 0 2024-04-16T02:35:45.405 INFO:tasks.workunit.client.0.smithi151.stdout:9/255: truncate d1/d3/d10/f4d 354532 0 2024-04-16T02:35:45.406 INFO:tasks.workunit.client.0.smithi151.stdout:7/321: creat d0/d1f/d31/d42/f6c x:0 0 0 2024-04-16T02:35:45.408 INFO:tasks.workunit.client.1.smithi153.stdout:4/914: mkdir d3/d9/db/d17/d28/dc8/de4/d140 0 2024-04-16T02:35:45.408 INFO:tasks.workunit.client.0.smithi151.stdout:9/256: dread d1/d3/d10/f4d [0,4194304] 0 2024-04-16T02:35:45.409 INFO:tasks.workunit.client.0.smithi151.stdout:7/322: mknod d0/d2c/d38/c6d 0 2024-04-16T02:35:45.412 INFO:tasks.workunit.client.0.smithi151.stdout:7/323: chown d0/f1d 221661690 1 2024-04-16T02:35:45.412 INFO:tasks.workunit.client.1.smithi153.stdout:7/789: rename d0/d4/db/d2b/d53 to d0/d4/db/d2b/d53/d59/d76/dc8/d10b 22 2024-04-16T02:35:45.414 INFO:tasks.workunit.client.0.smithi151.stdout:9/257: creat d1/dd/f51 x:0 0 0 2024-04-16T02:35:45.414 INFO:tasks.workunit.client.1.smithi153.stdout:1/821: dwrite d4/dd/d13/d26/ffe [0,4194304] 0 2024-04-16T02:35:45.416 INFO:tasks.workunit.client.0.smithi151.stdout:9/258: mknod d1/dd/c52 0 2024-04-16T02:35:45.416 INFO:tasks.workunit.client.0.smithi151.stdout:9/259: stat d1/d3/f1a 0 2024-04-16T02:35:45.418 INFO:tasks.workunit.client.1.smithi153.stdout:1/822: fdatasync d4/dd/d18/d29/f50 0 2024-04-16T02:35:45.419 INFO:tasks.workunit.client.1.smithi153.stdout:0/945: dwrite dd/d28/f3c [0,4194304] 0 2024-04-16T02:35:45.421 INFO:tasks.workunit.client.1.smithi153.stdout:7/790: creat d0/d4/db/d47/f10c x:0 0 0 2024-04-16T02:35:45.424 INFO:tasks.workunit.client.0.smithi151.stdout:1/291: dwrite d3/d33/d10/f5d [0,4194304] 0 2024-04-16T02:35:45.425 INFO:tasks.workunit.client.1.smithi153.stdout:1/823: mknod d4/dd/d13/d23/db6/c11a 0 2024-04-16T02:35:45.426 INFO:tasks.workunit.client.1.smithi153.stdout:0/946: symlink dd/d14/d51/l131 0 2024-04-16T02:35:45.426 INFO:tasks.workunit.client.1.smithi153.stdout:4/915: dwrite d3/d9/db/d17/d28/dc8/ff1 [0,4194304] 0 2024-04-16T02:35:45.428 INFO:tasks.workunit.client.0.smithi151.stdout:1/292: mknod d3/d33/c63 0 2024-04-16T02:35:45.431 INFO:tasks.workunit.client.1.smithi153.stdout:1/824: chown d4/dd/d13/d23/d31/fea 9 1 2024-04-16T02:35:45.431 INFO:tasks.workunit.client.1.smithi153.stdout:7/791: symlink d0/d4/db/d34/d4b/l10d 0 2024-04-16T02:35:45.432 INFO:tasks.workunit.client.1.smithi153.stdout:0/947: stat dd/d14/d19/d20/d4a/l6a 0 2024-04-16T02:35:45.433 INFO:tasks.workunit.client.1.smithi153.stdout:4/916: write d3/d9/db/d17/f55 [1797735,42376] 0 2024-04-16T02:35:45.436 INFO:tasks.workunit.client.1.smithi153.stdout:6/807: sync 2024-04-16T02:35:45.437 INFO:tasks.workunit.client.1.smithi153.stdout:5/872: dwrite d8/df/d2e/f6e [0,4194304] 0 2024-04-16T02:35:45.438 INFO:tasks.workunit.client.1.smithi153.stdout:0/948: fdatasync dd/d32/fc8 0 2024-04-16T02:35:45.440 INFO:tasks.workunit.client.1.smithi153.stdout:1/825: getdents d4/dd/d18/df7 0 2024-04-16T02:35:45.441 INFO:tasks.workunit.client.1.smithi153.stdout:7/792: rmdir d0/d4/db/d2b/d8c 39 2024-04-16T02:35:45.441 INFO:tasks.workunit.client.1.smithi153.stdout:5/873: fsync d8/d41/f10f 0 2024-04-16T02:35:45.441 INFO:tasks.workunit.client.0.smithi151.stdout:1/293: read d3/d58/d1b/d1c/f2c [1589909,9310] 0 2024-04-16T02:35:45.443 INFO:tasks.workunit.client.1.smithi153.stdout:4/917: read d3/d9/db/d17/d3f/f94 [2742792,113817] 0 2024-04-16T02:35:45.444 INFO:tasks.workunit.client.1.smithi153.stdout:5/874: write d8/df/d2f/f60 [1499325,109271] 0 2024-04-16T02:35:45.445 INFO:tasks.workunit.client.1.smithi153.stdout:6/808: unlink dc/d47/d57/f91 0 2024-04-16T02:35:45.445 INFO:tasks.workunit.client.1.smithi153.stdout:7/793: write d0/d4/db/d2b/d43/d6d/d68/fd4 [391469,15468] 0 2024-04-16T02:35:45.445 INFO:tasks.workunit.client.1.smithi153.stdout:0/949: dread dd/d14/da6/da4/fb5 [0,4194304] 0 2024-04-16T02:35:45.446 INFO:tasks.workunit.client.1.smithi153.stdout:5/875: chown d8/df/d28 389795 1 2024-04-16T02:35:45.452 INFO:tasks.workunit.client.1.smithi153.stdout:1/826: dwrite d4/da2/d65/d82/fb0 [0,4194304] 0 2024-04-16T02:35:45.453 INFO:tasks.workunit.client.0.smithi151.stdout:1/294: dwrite d3/d58/d1b/d1c/f2c [0,4194304] 0 2024-04-16T02:35:45.459 INFO:tasks.workunit.client.1.smithi153.stdout:0/950: dwrite dd/d14/d19/d77/db6/fec [0,4194304] 0 2024-04-16T02:35:45.461 INFO:tasks.workunit.client.0.smithi151.stdout:1/295: rename d3/d58/d1b to d3/d4/d64 0 2024-04-16T02:35:45.462 INFO:tasks.workunit.client.1.smithi153.stdout:4/918: truncate d3/d9/db/d17/d28/dc8/de4/dfd/de3/df0/f124 43152 0 2024-04-16T02:35:45.462 INFO:tasks.workunit.client.1.smithi153.stdout:8/776: sync 2024-04-16T02:35:45.462 INFO:tasks.workunit.client.1.smithi153.stdout:3/801: sync 2024-04-16T02:35:45.468 INFO:tasks.workunit.client.1.smithi153.stdout:2/802: write d9/d2b/d1c/d20/f79 [5394208,14779] 0 2024-04-16T02:35:45.470 INFO:tasks.workunit.client.1.smithi153.stdout:1/827: mknod d4/dd/d13/c11b 0 2024-04-16T02:35:45.471 INFO:tasks.workunit.client.1.smithi153.stdout:0/951: write dd/d14/dca/dcd/ffc [9050446,27260] 0 2024-04-16T02:35:45.474 INFO:tasks.workunit.client.1.smithi153.stdout:4/919: symlink d3/d9/d24/d75/d9e/l141 0 2024-04-16T02:35:45.476 INFO:tasks.workunit.client.1.smithi153.stdout:8/777: fdatasync d2/d93/d97/fa9 0 2024-04-16T02:35:45.477 INFO:tasks.workunit.client.1.smithi153.stdout:2/803: read d9/d2b/d1c/d20/d45/faf [780421,73742] 0 2024-04-16T02:35:45.478 INFO:tasks.workunit.client.1.smithi153.stdout:0/952: chown dd/d14/d19/d55/c9b 21944 1 2024-04-16T02:35:45.480 INFO:tasks.workunit.client.1.smithi153.stdout:4/920: read d3/f1c [919432,60297] 0 2024-04-16T02:35:45.480 INFO:tasks.workunit.client.1.smithi153.stdout:1/828: mknod d4/dd/d18/d29/d6e/d113/c11c 0 2024-04-16T02:35:45.481 INFO:tasks.workunit.client.1.smithi153.stdout:8/778: symlink d2/d13/d96/dc7/l103 0 2024-04-16T02:35:45.482 INFO:tasks.workunit.client.1.smithi153.stdout:9/861: dread d4/d6/dc/d2d/fa2 [0,4194304] 0 2024-04-16T02:35:45.483 INFO:tasks.workunit.client.1.smithi153.stdout:1/829: stat d4/da2/d65/d84/fa6 0 2024-04-16T02:35:45.484 INFO:tasks.workunit.client.1.smithi153.stdout:8/779: write d2/d92/deb/d47/fd1 [1228576,35544] 0 2024-04-16T02:35:45.486 INFO:tasks.workunit.client.1.smithi153.stdout:7/794: link d0/d13/f32 d0/d4/db/d2b/d43/d6d/d68/d6c/f10e 0 2024-04-16T02:35:45.488 INFO:tasks.workunit.client.1.smithi153.stdout:3/802: dwrite d6/d10/d3b/d59/f87 [0,4194304] 0 2024-04-16T02:35:45.490 INFO:tasks.workunit.client.1.smithi153.stdout:0/953: getdents dd/d14/dca/dcd 0 2024-04-16T02:35:45.490 INFO:tasks.workunit.client.1.smithi153.stdout:8/780: fdatasync d2/d13/d3a/d5b/f9d 0 2024-04-16T02:35:45.491 INFO:tasks.workunit.client.1.smithi153.stdout:1/830: chown d4/da2/d65/d84/fa6 265572 1 2024-04-16T02:35:45.492 INFO:tasks.workunit.client.1.smithi153.stdout:4/921: dread d3/d9/d24/d75/d9e/fce [0,4194304] 0 2024-04-16T02:35:45.496 INFO:tasks.workunit.client.0.smithi151.stdout:0/270: dread dc/f39 [0,4194304] 0 2024-04-16T02:35:45.498 INFO:tasks.workunit.client.1.smithi153.stdout:9/862: dwrite d4/d6/dc/fd7 [0,4194304] 0 2024-04-16T02:35:45.499 INFO:tasks.workunit.client.1.smithi153.stdout:8/781: read - d2/d93/d97/d9c/d27/ff9 zero size 2024-04-16T02:35:45.501 INFO:tasks.workunit.client.0.smithi151.stdout:0/271: rename dc/d15/f29 to dc/d15/f56 0 2024-04-16T02:35:45.502 INFO:tasks.workunit.client.1.smithi153.stdout:0/954: symlink dd/d14/da6/de0/l132 0 2024-04-16T02:35:45.503 INFO:tasks.workunit.client.0.smithi151.stdout:1/296: creat d3/d4/d64/d2a/f65 x:0 0 0 2024-04-16T02:35:45.503 INFO:tasks.workunit.client.1.smithi153.stdout:8/782: unlink d2/d13/d3a/d5b/l70 0 2024-04-16T02:35:45.503 INFO:tasks.workunit.client.0.smithi151.stdout:0/272: chown dc/c1e 1321128562 1 2024-04-16T02:35:45.506 INFO:tasks.workunit.client.1.smithi153.stdout:4/922: symlink d3/d9/db/de/d2c/de2/d110/l142 0 2024-04-16T02:35:45.507 INFO:tasks.workunit.client.1.smithi153.stdout:7/795: dwrite d0/d4/db/d47/f94 [0,4194304] 0 2024-04-16T02:35:45.508 INFO:tasks.workunit.client.0.smithi151.stdout:0/273: truncate dc/d15/f25 82076 0 2024-04-16T02:35:45.513 INFO:tasks.workunit.client.0.smithi151.stdout:1/297: dwrite d3/d58/df/d32/f37 [0,4194304] 0 2024-04-16T02:35:45.515 INFO:tasks.workunit.client.1.smithi153.stdout:4/923: creat d3/d9/db/d17/d28/f143 x:0 0 0 2024-04-16T02:35:45.515 INFO:tasks.workunit.client.1.smithi153.stdout:8/783: mknod d2/d92/deb/c104 0 2024-04-16T02:35:45.518 INFO:tasks.workunit.client.1.smithi153.stdout:5/876: dread d8/df/d23/f56 [0,4194304] 0 2024-04-16T02:35:45.518 INFO:tasks.workunit.client.1.smithi153.stdout:6/809: dread dc/d47/d57/d77/d15/f19 [0,4194304] 0 2024-04-16T02:35:45.518 INFO:tasks.workunit.client.0.smithi151.stdout:1/298: creat d3/d58/df/d32/d3a/f66 x:0 0 0 2024-04-16T02:35:45.519 INFO:tasks.workunit.client.0.smithi151.stdout:1/299: chown d3/d4/d64/d2a/c31 62828 1 2024-04-16T02:35:45.523 INFO:tasks.workunit.client.1.smithi153.stdout:4/924: symlink d3/da5/l144 0 2024-04-16T02:35:45.526 INFO:tasks.workunit.client.1.smithi153.stdout:4/925: dread - d3/d9/db/d17/d28/f5e zero size 2024-04-16T02:35:45.527 INFO:tasks.workunit.client.1.smithi153.stdout:8/784: dwrite d2/d92/deb/d47/fd1 [0,4194304] 0 2024-04-16T02:35:45.528 INFO:tasks.workunit.client.0.smithi151.stdout:1/300: dwrite d3/d33/f36 [0,4194304] 0 2024-04-16T02:35:45.528 INFO:tasks.workunit.client.1.smithi153.stdout:5/877: creat d8/df/d2e/f113 x:0 0 0 2024-04-16T02:35:45.531 INFO:tasks.workunit.client.1.smithi153.stdout:3/803: write d6/d10/f19 [2325407,67294] 0 2024-04-16T02:35:45.531 INFO:tasks.workunit.client.0.smithi151.stdout:8/223: sync 2024-04-16T02:35:45.538 INFO:tasks.workunit.client.1.smithi153.stdout:6/810: link dc/d47/d57/f10f dc/d10/f110 0 2024-04-16T02:35:45.538 INFO:tasks.workunit.client.1.smithi153.stdout:0/955: dread dd/d26/d44/f50 [0,4194304] 0 2024-04-16T02:35:45.539 INFO:tasks.workunit.client.1.smithi153.stdout:8/785: dread d2/d13/d2c/f8d [0,4194304] 0 2024-04-16T02:35:45.541 INFO:tasks.workunit.client.1.smithi153.stdout:0/956: readlink dd/d14/d51/d7d/ldb 0 2024-04-16T02:35:45.542 INFO:tasks.workunit.client.1.smithi153.stdout:3/804: creat d6/d10/d3b/d76/d10b/f10e x:0 0 0 2024-04-16T02:35:45.543 INFO:tasks.workunit.client.1.smithi153.stdout:4/926: dwrite d3/d9/f12c [0,4194304] 0 2024-04-16T02:35:45.545 INFO:tasks.workunit.client.0.smithi151.stdout:1/301: dwrite d3/d4/d64/d2a/f60 [0,4194304] 0 2024-04-16T02:35:45.546 INFO:tasks.workunit.client.1.smithi153.stdout:0/957: write dd/d14/d19/d77/d87/f91 [586304,56511] 0 2024-04-16T02:35:45.546 INFO:tasks.workunit.client.1.smithi153.stdout:3/805: read - d6/d10/d3b/d10d/ffa zero size 2024-04-16T02:35:45.547 INFO:tasks.workunit.client.1.smithi153.stdout:9/863: truncate d4/d6/dc/d24/d4e/f81 5490485 0 2024-04-16T02:35:45.549 INFO:tasks.workunit.client.1.smithi153.stdout:4/927: write d3/d9/db/d17/d28/f6f [867034,130033] 0 2024-04-16T02:35:45.555 INFO:tasks.workunit.client.1.smithi153.stdout:9/864: symlink d4/d6/dc/d2d/l10b 0 2024-04-16T02:35:45.556 INFO:tasks.workunit.client.1.smithi153.stdout:6/811: dread dc/d47/f87 [0,4194304] 0 2024-04-16T02:35:45.558 INFO:tasks.workunit.client.0.smithi151.stdout:1/302: dwrite d3/d33/d10/f4e [0,4194304] 0 2024-04-16T02:35:45.560 INFO:tasks.workunit.client.0.smithi151.stdout:1/303: chown d3/l53 41513 1 2024-04-16T02:35:45.562 INFO:tasks.workunit.client.0.smithi151.stdout:1/304: chown d3/d4/d64/d2a/f65 759078979 1 2024-04-16T02:35:45.563 INFO:tasks.workunit.client.1.smithi153.stdout:3/806: creat d6/d10/d22/f10f x:0 0 0 2024-04-16T02:35:45.563 INFO:tasks.workunit.client.1.smithi153.stdout:7/796: dwrite d0/d4/db/d2b/d43/d6d/d68/f86 [0,4194304] 0 2024-04-16T02:35:45.563 INFO:tasks.workunit.client.0.smithi151.stdout:1/305: stat d3/d58/df/d32/d3a 0 2024-04-16T02:35:45.566 INFO:tasks.workunit.client.0.smithi151.stdout:8/224: dwrite d11/d2a/f2f [0,4194304] 0 2024-04-16T02:35:45.569 INFO:tasks.workunit.client.0.smithi151.stdout:1/306: unlink d3/d33/l17 0 2024-04-16T02:35:45.571 INFO:tasks.workunit.client.1.smithi153.stdout:3/807: readlink d6/d10/d22/d43/lb3 0 2024-04-16T02:35:45.571 INFO:tasks.workunit.client.1.smithi153.stdout:8/786: write d2/d13/d2c/ffd [1037204,74079] 0 2024-04-16T02:35:45.572 INFO:tasks.workunit.client.1.smithi153.stdout:0/958: dwrite dd/d14/da6/da4/fb5 [0,4194304] 0 2024-04-16T02:35:45.574 INFO:tasks.workunit.client.1.smithi153.stdout:3/808: readlink d6/d10/l1b 0 2024-04-16T02:35:45.574 INFO:tasks.workunit.client.0.smithi151.stdout:8/225: chown d11/l31 982 1 2024-04-16T02:35:45.577 INFO:tasks.workunit.client.1.smithi153.stdout:6/812: creat dc/d47/d57/d77/d15/dac/dae/de7/f111 x:0 0 0 2024-04-16T02:35:45.579 INFO:tasks.workunit.client.1.smithi153.stdout:6/813: stat dc/d47/d93 0 2024-04-16T02:35:45.581 INFO:tasks.workunit.client.1.smithi153.stdout:8/787: creat d2/d93/d97/f105 x:0 0 0 2024-04-16T02:35:45.581 INFO:tasks.workunit.client.0.smithi151.stdout:8/226: fdatasync fb 0 2024-04-16T02:35:45.581 INFO:tasks.workunit.client.1.smithi153.stdout:6/814: chown dc/d1a/l9b 2 1 2024-04-16T02:35:45.583 INFO:tasks.workunit.client.1.smithi153.stdout:9/865: getdents d4/d6/dc/d24/d4e/df4 0 2024-04-16T02:35:45.584 INFO:tasks.workunit.client.1.smithi153.stdout:7/797: creat d0/d4/db/d34/f10f x:0 0 0 2024-04-16T02:35:45.585 INFO:tasks.workunit.client.0.smithi151.stdout:1/307: dwrite d3/d58/df/d32/d3a/f66 [0,4194304] 0 2024-04-16T02:35:45.585 INFO:tasks.workunit.client.1.smithi153.stdout:0/959: creat dd/d14/da6/da4/f133 x:0 0 0 2024-04-16T02:35:45.590 INFO:tasks.workunit.client.1.smithi153.stdout:8/788: symlink d2/d93/dee/l106 0 2024-04-16T02:35:45.590 INFO:tasks.workunit.client.1.smithi153.stdout:6/815: chown dc/d1a/d2d/c3b 6 1 2024-04-16T02:35:45.591 INFO:tasks.workunit.client.0.smithi151.stdout:8/227: dread d11/d22/d2e/f3d [0,4194304] 0 2024-04-16T02:35:45.591 INFO:tasks.workunit.client.1.smithi153.stdout:8/789: fdatasync d2/d1c/f36 0 2024-04-16T02:35:45.591 INFO:tasks.workunit.client.1.smithi153.stdout:9/866: creat d4/d6/d1b/ddf/f10c x:0 0 0 2024-04-16T02:35:45.592 INFO:tasks.workunit.client.1.smithi153.stdout:6/816: write dc/fe [1835680,103276] 0 2024-04-16T02:35:45.594 INFO:tasks.workunit.client.0.smithi151.stdout:8/228: truncate d11/f1a 519829 0 2024-04-16T02:35:45.595 INFO:tasks.workunit.client.1.smithi153.stdout:7/798: rename d0/d4/db/d34/d4b to d0/de8/d110 0 2024-04-16T02:35:45.595 INFO:tasks.workunit.client.0.smithi151.stdout:1/308: dwrite d3/d1f/d48/f51 [0,4194304] 0 2024-04-16T02:35:45.596 INFO:tasks.workunit.client.0.smithi151.stdout:8/229: write d11/d2a/d33/f4b [491543,56709] 0 2024-04-16T02:35:45.597 INFO:tasks.workunit.client.1.smithi153.stdout:0/960: chown dd/d14/d19/d77/d87/dda/d11a/l121 238623558 1 2024-04-16T02:35:45.599 INFO:tasks.workunit.client.1.smithi153.stdout:4/928: dwrite d3/fdd [0,4194304] 0 2024-04-16T02:35:45.600 INFO:tasks.workunit.client.0.smithi151.stdout:8/230: fdatasync d11/d22/f26 0 2024-04-16T02:35:45.604 INFO:tasks.workunit.client.0.smithi151.stdout:8/231: symlink d11/d22/d2e/l4d 0 2024-04-16T02:35:45.604 INFO:tasks.workunit.client.1.smithi153.stdout:8/790: creat d2/d13/de8/f107 x:0 0 0 2024-04-16T02:35:45.607 INFO:tasks.workunit.client.1.smithi153.stdout:9/867: rename d4/d6/f56 to d4/d33/dae/dc7/d66/dbd/f10d 0 2024-04-16T02:35:45.608 INFO:tasks.workunit.client.1.smithi153.stdout:9/868: chown d4/f13 190372658 1 2024-04-16T02:35:45.610 INFO:tasks.workunit.client.0.smithi151.stdout:8/232: mkdir d11/d2a/d33/d4e 0 2024-04-16T02:35:45.610 INFO:tasks.workunit.client.0.smithi151.stdout:1/309: dwrite d3/d33/f39 [4194304,4194304] 0 2024-04-16T02:35:45.611 INFO:tasks.workunit.client.1.smithi153.stdout:7/799: mkdir d0/d4/db/d2b/d8c/db5/d111 0 2024-04-16T02:35:45.611 INFO:tasks.workunit.client.1.smithi153.stdout:4/929: creat d3/d9/d24/d3e/d103/f145 x:0 0 0 2024-04-16T02:35:45.612 INFO:tasks.workunit.client.1.smithi153.stdout:8/791: mknod d2/d93/dee/c108 0 2024-04-16T02:35:45.613 INFO:tasks.workunit.client.0.smithi151.stdout:8/233: mknod d11/d2a/c4f 0 2024-04-16T02:35:45.613 INFO:tasks.workunit.client.1.smithi153.stdout:8/792: chown d2/d93/d97/d9c/d27/fc3 15588719 1 2024-04-16T02:35:45.614 INFO:tasks.workunit.client.1.smithi153.stdout:4/930: dread d3/d9/d24/d3e/f117 [0,4194304] 0 2024-04-16T02:35:45.615 INFO:tasks.workunit.client.0.smithi151.stdout:8/234: stat d11/c12 0 2024-04-16T02:35:45.616 INFO:tasks.workunit.client.0.smithi151.stdout:1/310: mkdir d3/d58/df/d32/d3a/d67 0 2024-04-16T02:35:45.617 INFO:tasks.workunit.client.0.smithi151.stdout:1/311: dread - d3/d4/d64/d1c/f44 zero size 2024-04-16T02:35:45.617 INFO:tasks.workunit.client.1.smithi153.stdout:9/869: creat d4/d33/dae/dc7/d66/dcb/f10e x:0 0 0 2024-04-16T02:35:45.618 INFO:tasks.workunit.client.1.smithi153.stdout:7/800: rmdir d0/de8/d110/daa 39 2024-04-16T02:35:45.619 INFO:tasks.workunit.client.0.smithi151.stdout:1/312: read d3/d33/d10/f5d [2141367,99845] 0 2024-04-16T02:35:45.620 INFO:tasks.workunit.client.1.smithi153.stdout:0/961: read dd/d14/f3b [463361,107414] 0 2024-04-16T02:35:45.620 INFO:tasks.workunit.client.1.smithi153.stdout:7/801: write d0/d4/db/d2b/d43/f9d [4952117,128452] 0 2024-04-16T02:35:45.621 INFO:tasks.workunit.client.1.smithi153.stdout:8/793: rename d2/d92/deb to d2/d13/d96/dc7/d109 0 2024-04-16T02:35:45.625 INFO:tasks.workunit.client.0.smithi151.stdout:1/313: mkdir d3/d58/df/d68 0 2024-04-16T02:35:45.625 INFO:tasks.workunit.client.0.smithi151.stdout:8/235: dwrite d11/d2a/d33/f49 [0,4194304] 0 2024-04-16T02:35:45.625 INFO:tasks.workunit.client.1.smithi153.stdout:9/870: symlink d4/d6/dc/d24/d60/l10f 0 2024-04-16T02:35:45.626 INFO:tasks.workunit.client.1.smithi153.stdout:4/931: dwrite d3/d9/db/d17/d28/dc8/de4/dfd/d120/f135 [0,4194304] 0 2024-04-16T02:35:45.627 INFO:tasks.workunit.client.1.smithi153.stdout:9/871: chown d4/c62 173438286 1 2024-04-16T02:35:45.627 INFO:tasks.workunit.client.0.smithi151.stdout:1/314: fdatasync d3/d58/f62 0 2024-04-16T02:35:45.631 INFO:tasks.workunit.client.0.smithi151.stdout:8/236: rename d11/c44 to d11/c50 0 2024-04-16T02:35:45.633 INFO:tasks.workunit.client.1.smithi153.stdout:2/804: sync 2024-04-16T02:35:45.633 INFO:tasks.workunit.client.1.smithi153.stdout:1/831: sync 2024-04-16T02:35:45.635 INFO:tasks.workunit.client.1.smithi153.stdout:0/962: link dd/d14/da6/de0/fe6 dd/d14/da6/da4/f134 0 2024-04-16T02:35:45.638 INFO:tasks.workunit.client.0.smithi151.stdout:1/315: dwrite d3/d4/d64/d2a/f60 [0,4194304] 0 2024-04-16T02:35:45.639 INFO:tasks.workunit.client.1.smithi153.stdout:9/872: dwrite d4/f48 [8388608,4194304] 0 2024-04-16T02:35:45.640 INFO:tasks.workunit.client.1.smithi153.stdout:2/805: unlink d9/d71/d8d/da8/fbe 0 2024-04-16T02:35:45.642 INFO:tasks.workunit.client.1.smithi153.stdout:4/932: mkdir d3/d9/db/de/d2c/d13a/d146 0 2024-04-16T02:35:45.642 INFO:tasks.workunit.client.1.smithi153.stdout:8/794: getdents d2/d13/d3a/dc2 0 2024-04-16T02:35:45.645 INFO:tasks.workunit.client.0.smithi151.stdout:1/316: rename d3/f2f to d3/d58/df/d32/d3a/d67/f69 0 2024-04-16T02:35:45.647 INFO:tasks.workunit.client.0.smithi151.stdout:1/317: write d3/d4/d64/d2a/f60 [4404274,112186] 0 2024-04-16T02:35:45.649 INFO:tasks.workunit.client.0.smithi151.stdout:1/318: write d3/d4/d64/d1c/f44 [426027,1631] 0 2024-04-16T02:35:45.653 INFO:tasks.workunit.client.1.smithi153.stdout:5/878: sync 2024-04-16T02:35:45.654 INFO:tasks.workunit.client.1.smithi153.stdout:1/832: creat d4/f11d x:0 0 0 2024-04-16T02:35:45.657 INFO:tasks.workunit.client.1.smithi153.stdout:3/809: dwrite d6/d10/d3b/f9d [0,4194304] 0 2024-04-16T02:35:45.659 INFO:tasks.workunit.client.1.smithi153.stdout:1/833: write d4/dd/d13/d26/ffe [2348460,115835] 0 2024-04-16T02:35:45.660 INFO:tasks.workunit.client.1.smithi153.stdout:3/810: truncate d6/d10/d3b/d10d/f4e 485508 0 2024-04-16T02:35:45.660 INFO:tasks.workunit.client.1.smithi153.stdout:9/873: rmdir d4/df9 0 2024-04-16T02:35:45.660 INFO:tasks.workunit.client.1.smithi153.stdout:9/874: chown d4/d6/d1b/d2a 439631092 1 2024-04-16T02:35:45.662 INFO:tasks.workunit.client.0.smithi151.stdout:8/237: dread d11/f15 [0,4194304] 0 2024-04-16T02:35:45.662 INFO:tasks.workunit.client.0.smithi151.stdout:8/238: chown d11/d2a/d35 200 1 2024-04-16T02:35:45.663 INFO:tasks.workunit.client.1.smithi153.stdout:3/811: dread - d6/d10/d3b/d59/d7c/de1/f108 zero size 2024-04-16T02:35:45.664 INFO:tasks.workunit.client.0.smithi151.stdout:8/239: write d11/f15 [2018942,36975] 0 2024-04-16T02:35:45.664 INFO:tasks.workunit.client.1.smithi153.stdout:5/879: chown d8/df/f97 1616774089 1 2024-04-16T02:35:45.665 INFO:tasks.workunit.client.1.smithi153.stdout:0/963: getdents dd/d14/d51/d7f/d9d 0 2024-04-16T02:35:45.669 INFO:tasks.workunit.client.0.smithi151.stdout:8/240: truncate d11/d22/f26 2704741 0 2024-04-16T02:35:45.669 INFO:tasks.workunit.client.0.smithi151.stdout:1/319: dread d3/d4/d64/d1c/f44 [0,4194304] 0 2024-04-16T02:35:45.675 INFO:tasks.workunit.client.1.smithi153.stdout:1/834: symlink d4/da2/d65/l11e 0 2024-04-16T02:35:45.675 INFO:tasks.workunit.client.1.smithi153.stdout:9/875: dwrite d4/d6/dc/d24/f30 [0,4194304] 0 2024-04-16T02:35:45.676 INFO:tasks.workunit.client.0.smithi151.stdout:1/320: link d3/d33/f39 d3/d4/d64/d2a/d52/f6a 0 2024-04-16T02:35:45.678 INFO:tasks.workunit.client.0.smithi151.stdout:8/241: dwrite d11/f1c [0,4194304] 0 2024-04-16T02:35:45.680 INFO:tasks.workunit.client.0.smithi151.stdout:1/321: creat d3/d33/d10/f6b x:0 0 0 2024-04-16T02:35:45.680 INFO:tasks.workunit.client.0.smithi151.stdout:1/322: stat d3/d1f/d48 0 2024-04-16T02:35:45.681 INFO:tasks.workunit.client.1.smithi153.stdout:7/802: write d0/de8/d110/daa/fc9 [319317,62469] 0 2024-04-16T02:35:45.682 INFO:tasks.workunit.client.0.smithi151.stdout:8/242: dread d11/f43 [0,4194304] 0 2024-04-16T02:35:45.685 INFO:tasks.workunit.client.1.smithi153.stdout:2/806: write d9/d2b/d1c/d20/d45/f54 [1175498,99769] 0 2024-04-16T02:35:45.685 INFO:tasks.workunit.client.0.smithi151.stdout:1/323: creat d3/d58/df/f6c x:0 0 0 2024-04-16T02:35:45.685 INFO:tasks.workunit.client.1.smithi153.stdout:9/876: rename d4/c26 to d4/d6/dc/d24/d4e/df4/c110 0 2024-04-16T02:35:45.687 INFO:tasks.workunit.client.1.smithi153.stdout:5/880: dwrite d8/df/d28/d5a/fb8 [0,4194304] 0 2024-04-16T02:35:45.690 INFO:tasks.workunit.client.0.smithi151.stdout:1/324: mkdir d3/d58/df/d32/d3a/d6d 0 2024-04-16T02:35:45.692 INFO:tasks.workunit.client.1.smithi153.stdout:4/933: write d3/d9/db/d45/df3/fc4 [5159992,86445] 0 2024-04-16T02:35:45.693 INFO:tasks.workunit.client.1.smithi153.stdout:9/877: fsync d4/d6/db1/fb4 0 2024-04-16T02:35:45.695 INFO:tasks.workunit.client.1.smithi153.stdout:1/835: write d4/dd/d13/d26/db4/ffb [1704082,55871] 0 2024-04-16T02:35:45.695 INFO:tasks.workunit.client.0.smithi151.stdout:8/243: dread d11/d2a/f2c [0,4194304] 0 2024-04-16T02:35:45.696 INFO:tasks.workunit.client.0.smithi151.stdout:1/325: link d3/d58/l57 d3/d4/d64/d2a/l6e 0 2024-04-16T02:35:45.699 INFO:tasks.workunit.client.0.smithi151.stdout:8/244: unlink d11/d2a/d33/l37 0 2024-04-16T02:35:45.700 INFO:tasks.workunit.client.0.smithi151.stdout:1/326: creat d3/d4/d64/d61/f6f x:0 0 0 2024-04-16T02:35:45.701 INFO:tasks.workunit.client.1.smithi153.stdout:2/807: stat d9/d2b/d1c/d20/f101 0 2024-04-16T02:35:45.703 INFO:tasks.workunit.client.0.smithi151.stdout:1/327: symlink d3/l70 0 2024-04-16T02:35:45.704 INFO:tasks.workunit.client.1.smithi153.stdout:4/934: fdatasync d3/fdd 0 2024-04-16T02:35:45.705 INFO:tasks.workunit.client.0.smithi151.stdout:1/328: write d3/d33/d10/f55 [1799512,104710] 0 2024-04-16T02:35:45.705 INFO:tasks.workunit.client.1.smithi153.stdout:0/964: dwrite dd/d32/d57/f61 [0,4194304] 0 2024-04-16T02:35:45.707 INFO:tasks.workunit.client.0.smithi151.stdout:1/329: write d3/d4/d64/d61/f6f [893812,31403] 0 2024-04-16T02:35:45.708 INFO:tasks.workunit.client.1.smithi153.stdout:6/817: dread dc/d47/d57/d77/d15/f22 [0,4194304] 0 2024-04-16T02:35:45.709 INFO:tasks.workunit.client.0.smithi151.stdout:8/245: dwrite d11/d22/f27 [0,4194304] 0 2024-04-16T02:35:45.710 INFO:tasks.workunit.client.1.smithi153.stdout:1/836: truncate d4/dd/d13/d26/db4/ffb 2759161 0 2024-04-16T02:35:45.711 INFO:tasks.workunit.client.0.smithi151.stdout:1/330: write d3/d4/d64/d1c/f2c [1777988,64349] 0 2024-04-16T02:35:45.712 INFO:tasks.workunit.client.1.smithi153.stdout:1/837: chown d4/dd/d18/d6c/c94 31100855 1 2024-04-16T02:35:45.715 INFO:tasks.workunit.client.0.smithi151.stdout:1/331: write d3/d58/df/d32/f37 [2073649,19722] 0 2024-04-16T02:35:45.715 INFO:tasks.workunit.client.1.smithi153.stdout:0/965: dwrite dd/d26/d44/fc6 [0,4194304] 0 2024-04-16T02:35:45.716 INFO:tasks.workunit.client.0.smithi151.stdout:8/246: unlink d11/d2a/d35/f3f 0 2024-04-16T02:35:45.716 INFO:tasks.workunit.client.1.smithi153.stdout:1/838: readlink d4/da2/d65/d82/la5 0 2024-04-16T02:35:45.721 INFO:tasks.workunit.client.1.smithi153.stdout:3/812: truncate d6/d10/d22/f28 2144463 0 2024-04-16T02:35:45.724 INFO:tasks.workunit.client.0.smithi151.stdout:8/247: dwrite d11/d2a/d35/f40 [0,4194304] 0 2024-04-16T02:35:45.724 INFO:tasks.workunit.client.0.smithi151.stdout:8/248: readlink l8 0 2024-04-16T02:35:45.728 INFO:tasks.workunit.client.0.smithi151.stdout:1/332: rename d3/d4/d64/d61 to d3/d58/df/d32/d3a/d71 0 2024-04-16T02:35:45.731 INFO:tasks.workunit.client.0.smithi151.stdout:8/249: symlink d11/d2a/l51 0 2024-04-16T02:35:45.734 INFO:tasks.workunit.client.0.smithi151.stdout:8/250: mkdir d11/d2a/d35/d52 0 2024-04-16T02:35:45.735 INFO:tasks.workunit.client.1.smithi153.stdout:8/795: dread d2/d13/d96/dc7/d109/f81 [0,4194304] 0 2024-04-16T02:35:45.736 INFO:tasks.workunit.client.0.smithi151.stdout:1/333: creat d3/d4/d64/d2a/d52/f72 x:0 0 0 2024-04-16T02:35:45.738 INFO:tasks.workunit.client.0.smithi151.stdout:8/251: creat d11/d2a/d33/f53 x:0 0 0 2024-04-16T02:35:45.739 INFO:tasks.workunit.client.0.smithi151.stdout:1/334: mkdir d3/d4/d64/d2a/d73 0 2024-04-16T02:35:45.739 INFO:tasks.workunit.client.0.smithi151.stdout:8/252: chown d11/d2a/d33/f53 4 1 2024-04-16T02:35:45.740 INFO:tasks.workunit.client.1.smithi153.stdout:0/966: dread dd/d14/d19/fae [0,4194304] 0 2024-04-16T02:35:45.742 INFO:tasks.workunit.client.1.smithi153.stdout:2/808: unlink d9/c12 0 2024-04-16T02:35:45.742 INFO:tasks.workunit.client.0.smithi151.stdout:1/335: creat d3/d58/df/d38/f74 x:0 0 0 2024-04-16T02:35:45.743 INFO:tasks.workunit.client.1.smithi153.stdout:9/878: creat d4/d6/dc/d24/d4e/df4/f111 x:0 0 0 2024-04-16T02:35:45.744 INFO:tasks.workunit.client.1.smithi153.stdout:2/809: readlink d9/d71/la1 0 2024-04-16T02:35:45.744 INFO:tasks.workunit.client.1.smithi153.stdout:7/803: truncate d0/d4/db/d2b/d53/d59/d76/f7b 251220 0 2024-04-16T02:35:45.745 INFO:tasks.workunit.client.1.smithi153.stdout:5/881: write d8/d1f/ff2 [2396044,115239] 0 2024-04-16T02:35:45.747 INFO:tasks.workunit.client.0.smithi151.stdout:8/253: dwrite d11/d2a/d33/f49 [0,4194304] 0 2024-04-16T02:35:45.748 INFO:tasks.workunit.client.1.smithi153.stdout:4/935: creat d3/d9/db/de/d2c/de2/d110/f147 x:0 0 0 2024-04-16T02:35:45.748 INFO:tasks.workunit.client.1.smithi153.stdout:1/839: rmdir d4/dd/d13/d23/d31 39 2024-04-16T02:35:45.749 INFO:tasks.workunit.client.0.smithi151.stdout:1/336: rename d3/d58/f5e to d3/d58/df/d32/d3a/d71/f75 0 2024-04-16T02:35:45.749 INFO:tasks.workunit.client.0.smithi151.stdout:8/254: readlink d11/l13 0 2024-04-16T02:35:45.753 INFO:tasks.workunit.client.1.smithi153.stdout:8/796: symlink d2/d93/d97/d9c/d27/d5e/d63/dec/l10a 0 2024-04-16T02:35:45.753 INFO:tasks.workunit.client.0.smithi151.stdout:1/337: link d3/d33/l3d d3/d33/d10/l76 0 2024-04-16T02:35:45.754 INFO:tasks.workunit.client.0.smithi151.stdout:8/255: dread d11/d2a/f2f [0,4194304] 0 2024-04-16T02:35:45.756 INFO:tasks.workunit.client.0.smithi151.stdout:1/338: symlink d3/d58/df/d32/d3a/d6d/l77 0 2024-04-16T02:35:45.758 INFO:tasks.workunit.client.0.smithi151.stdout:5/356: sync 2024-04-16T02:35:45.762 INFO:tasks.workunit.client.1.smithi153.stdout:2/810: unlink d9/d24/d34/d60/d7c/ca5 0 2024-04-16T02:35:45.762 INFO:tasks.workunit.client.0.smithi151.stdout:8/256: dwrite d11/d22/f30 [0,4194304] 0 2024-04-16T02:35:45.764 INFO:tasks.workunit.client.1.smithi153.stdout:0/967: dwrite dd/d14/dca/f12b [0,4194304] 0 2024-04-16T02:35:45.766 INFO:tasks.workunit.client.1.smithi153.stdout:7/804: mknod d0/d4/db/d2b/d43/d6d/d68/d6c/c112 0 2024-04-16T02:35:45.766 INFO:tasks.workunit.client.0.smithi151.stdout:5/357: creat d2/d6/d19/d30/f70 x:0 0 0 2024-04-16T02:35:45.769 INFO:tasks.workunit.client.0.smithi151.stdout:1/339: dwrite d3/d58/df/d38/f5a [0,4194304] 0 2024-04-16T02:35:45.770 INFO:tasks.workunit.client.1.smithi153.stdout:6/818: dwrite dc/d47/f87 [0,4194304] 0 2024-04-16T02:35:45.772 INFO:tasks.workunit.client.0.smithi151.stdout:1/340: read - d3/d4/d64/d2a/d52/f72 zero size 2024-04-16T02:35:45.774 INFO:tasks.workunit.client.0.smithi151.stdout:1/341: fdatasync d3/d33/d10/f4e 0 2024-04-16T02:35:45.778 INFO:tasks.workunit.client.1.smithi153.stdout:8/797: creat d2/d93/d97/d9c/d27/d5e/d63/dc6/f10b x:0 0 0 2024-04-16T02:35:45.779 INFO:tasks.workunit.client.1.smithi153.stdout:2/811: write d9/d2b/d1c/d20/d45/faf [11338,60392] 0 2024-04-16T02:35:45.784 INFO:tasks.workunit.client.1.smithi153.stdout:7/805: creat d0/d4/db/d2b/d53/d59/dc3/f113 x:0 0 0 2024-04-16T02:35:45.784 INFO:tasks.workunit.client.1.smithi153.stdout:1/840: link d4/dd/d13/d23/db6/c11a d4/c11f 0 2024-04-16T02:35:45.785 INFO:tasks.workunit.client.1.smithi153.stdout:4/936: dwrite d3/d9/db/d17/d28/dc8/de4/dfd/de3/df0/f124 [0,4194304] 0 2024-04-16T02:35:45.785 INFO:tasks.workunit.client.1.smithi153.stdout:6/819: link dc/fe dc/d47/d57/d77/d7a/f112 0 2024-04-16T02:35:45.788 INFO:tasks.workunit.client.1.smithi153.stdout:2/812: rmdir d9/d24/d34/d60 39 2024-04-16T02:35:45.791 INFO:tasks.workunit.client.1.smithi153.stdout:6/820: truncate dc/d47/d57/d77/dc4/dc9/ded/f10e 436564 0 2024-04-16T02:35:45.794 INFO:tasks.workunit.client.1.smithi153.stdout:6/821: write dc/d10/fe9 [584118,104973] 0 2024-04-16T02:35:45.794 INFO:tasks.workunit.client.1.smithi153.stdout:0/968: mkdir dd/d14/d19/d135 0 2024-04-16T02:35:45.796 INFO:tasks.workunit.client.1.smithi153.stdout:1/841: creat d4/dd/d18/d29/d6e/d116/f120 x:0 0 0 2024-04-16T02:35:45.797 INFO:tasks.workunit.client.0.smithi151.stdout:8/257: dread d11/f17 [0,4194304] 0 2024-04-16T02:35:45.798 INFO:tasks.workunit.client.1.smithi153.stdout:4/937: mknod d3/d9/db/de/d2c/d52/c148 0 2024-04-16T02:35:45.800 INFO:tasks.workunit.client.1.smithi153.stdout:7/806: creat d0/d4/db/d2b/d8c/db5/d111/f114 x:0 0 0 2024-04-16T02:35:45.800 INFO:tasks.workunit.client.1.smithi153.stdout:0/969: symlink dd/d14/da6/da4/db1/l136 0 2024-04-16T02:35:45.802 INFO:tasks.workunit.client.1.smithi153.stdout:1/842: chown d4/dd/d13/d23/d31/f106 12093875 1 2024-04-16T02:35:45.804 INFO:tasks.workunit.client.1.smithi153.stdout:4/938: truncate d3/d9/d24/f4d 3419051 0 2024-04-16T02:35:45.804 INFO:tasks.workunit.client.1.smithi153.stdout:1/843: chown d4/dd/d13/d7e/d80/cbf 5377032 1 2024-04-16T02:35:45.805 INFO:tasks.workunit.client.1.smithi153.stdout:0/970: creat dd/d14/d51/d54/d65/d68/f137 x:0 0 0 2024-04-16T02:35:45.808 INFO:tasks.workunit.client.1.smithi153.stdout:9/879: write d4/d6/dc/d24/f83 [505048,74530] 0 2024-04-16T02:35:45.809 INFO:tasks.workunit.client.1.smithi153.stdout:1/844: creat d4/dd/d13/d8d/f121 x:0 0 0 2024-04-16T02:35:45.811 INFO:tasks.workunit.client.1.smithi153.stdout:7/807: rename d0/d4/db/d2b/d53/d59/dc3/fdc to d0/d4/db/f115 0 2024-04-16T02:35:45.812 INFO:tasks.workunit.client.1.smithi153.stdout:5/882: dread d8/d41/dbd/f40 [0,4194304] 0 2024-04-16T02:35:45.814 INFO:tasks.workunit.client.1.smithi153.stdout:1/845: readlink d4/dd/d13/d23/d5f/lc1 0 2024-04-16T02:35:45.816 INFO:tasks.workunit.client.1.smithi153.stdout:3/813: dwrite d6/d10/d5b/f82 [0,4194304] 0 2024-04-16T02:35:45.817 INFO:tasks.workunit.client.1.smithi153.stdout:5/883: fdatasync d8/d41/dbd/d83/fc1 0 2024-04-16T02:35:45.818 INFO:tasks.workunit.client.1.smithi153.stdout:1/846: chown d4/dd/d13/d23/d31/l107 15 1 2024-04-16T02:35:45.821 INFO:tasks.workunit.client.1.smithi153.stdout:7/808: truncate d0/d4/db/d2b/fc1 387263 0 2024-04-16T02:35:45.824 INFO:tasks.workunit.client.1.smithi153.stdout:4/939: getdents d3/d9/db/de/d2c/d52 0 2024-04-16T02:35:45.824 INFO:tasks.workunit.client.1.smithi153.stdout:9/880: dread d4/f48 [8388608,4194304] 0 2024-04-16T02:35:45.825 INFO:tasks.workunit.client.1.smithi153.stdout:5/884: write d8/df/d28/d5a/d72/f105 [374639,56247] 0 2024-04-16T02:35:45.827 INFO:tasks.workunit.client.1.smithi153.stdout:2/813: truncate d9/d2b/d1c/fd5 922685 0 2024-04-16T02:35:45.828 INFO:tasks.workunit.client.1.smithi153.stdout:3/814: symlink d6/d10/d3b/d59/d7c/de1/df0/l110 0 2024-04-16T02:35:45.833 INFO:tasks.workunit.client.1.smithi153.stdout:4/940: read d3/d9/db/de/fa3 [3464714,40679] 0 2024-04-16T02:35:45.833 INFO:tasks.workunit.client.1.smithi153.stdout:8/798: dwrite d2/d93/d97/d9c/d27/fb9 [4194304,4194304] 0 2024-04-16T02:35:45.833 INFO:tasks.workunit.client.1.smithi153.stdout:2/814: stat d9/d24/d27/d2c/d5d/f63 0 2024-04-16T02:35:45.834 INFO:tasks.workunit.client.1.smithi153.stdout:9/881: mkdir d4/d6/dc/d24/dc6/d112 0 2024-04-16T02:35:45.839 INFO:tasks.workunit.client.1.smithi153.stdout:1/847: dwrite d4/dd/d18/d29/fde [0,4194304] 0 2024-04-16T02:35:45.839 INFO:tasks.workunit.client.1.smithi153.stdout:5/885: creat d8/df/d28/d5a/d72/dc0/dcb/f114 x:0 0 0 2024-04-16T02:35:45.842 INFO:tasks.workunit.client.1.smithi153.stdout:7/809: mknod d0/d4/db/d87/dc0/de0/dee/c116 0 2024-04-16T02:35:45.846 INFO:tasks.workunit.client.1.smithi153.stdout:0/971: dread dd/d14/da6/f4d [0,4194304] 0 2024-04-16T02:35:45.847 INFO:tasks.workunit.client.1.smithi153.stdout:8/799: creat d2/de/d3f/f10c x:0 0 0 2024-04-16T02:35:45.848 INFO:tasks.workunit.client.1.smithi153.stdout:4/941: rename d3/d9/db/de/d2c/d98 to d3/d9/db/d17/d28/d4c/d149 0 2024-04-16T02:35:45.849 INFO:tasks.workunit.client.1.smithi153.stdout:2/815: dread - d9/d24/d34/d60/ddc/f100 zero size 2024-04-16T02:35:45.849 INFO:tasks.workunit.client.1.smithi153.stdout:7/810: write d0/d4/db/d34/f10f [918477,2767] 0 2024-04-16T02:35:45.850 INFO:tasks.workunit.client.1.smithi153.stdout:9/882: mkdir d4/d6/dc/d24/d104/d113 0 2024-04-16T02:35:45.850 INFO:tasks.workunit.client.1.smithi153.stdout:5/886: symlink d8/df/d2f/d4e/l115 0 2024-04-16T02:35:45.851 INFO:tasks.workunit.client.1.smithi153.stdout:1/848: mkdir d4/dd/d13/d8d/d95/d122 0 2024-04-16T02:35:45.853 INFO:tasks.workunit.client.1.smithi153.stdout:3/815: dwrite d6/d10/d3b/d59/d7c/de1/df0/fc6 [0,4194304] 0 2024-04-16T02:35:45.853 INFO:tasks.workunit.client.1.smithi153.stdout:4/942: mknod d3/d9/d24/d3e/c14a 0 2024-04-16T02:35:45.853 INFO:tasks.workunit.client.1.smithi153.stdout:2/816: rmdir d9/d24/d27/d2c/d5d/d10a 39 2024-04-16T02:35:45.856 INFO:tasks.workunit.client.1.smithi153.stdout:7/811: mknod d0/d4/db/d2b/d8c/d9f/c117 0 2024-04-16T02:35:45.857 INFO:tasks.workunit.client.1.smithi153.stdout:7/812: readlink d0/lb2 0 2024-04-16T02:35:45.857 INFO:tasks.workunit.client.1.smithi153.stdout:2/817: chown d9/d24/d27/d2c/d5d/da3/dc0/fd1 6140 1 2024-04-16T02:35:45.863 INFO:tasks.workunit.client.1.smithi153.stdout:7/813: dread - d0/d4/db/d2b/d43/d6d/d68/d6c/db8/ddf/ff0 zero size 2024-04-16T02:35:45.863 INFO:tasks.workunit.client.1.smithi153.stdout:4/943: truncate d3/d9/db/d17/d3f/f109 899053 0 2024-04-16T02:35:45.863 INFO:tasks.workunit.client.1.smithi153.stdout:9/883: rename d4/d6/lac to d4/d6/ddb/l114 0 2024-04-16T02:35:45.863 INFO:tasks.workunit.client.1.smithi153.stdout:7/814: readlink d0/d4/db/d2b/d8c/db5/le3 0 2024-04-16T02:35:45.864 INFO:tasks.workunit.client.1.smithi153.stdout:5/887: symlink d8/df/d28/d5a/d72/l116 0 2024-04-16T02:35:45.864 INFO:tasks.workunit.client.1.smithi153.stdout:9/884: write d4/d33/dae/ffa [197273,58842] 0 2024-04-16T02:35:45.865 INFO:tasks.workunit.client.1.smithi153.stdout:2/818: mknod d9/d2b/d5c/da2/c10b 0 2024-04-16T02:35:45.867 INFO:tasks.workunit.client.1.smithi153.stdout:8/800: truncate d2/d93/d97/d9c/d27/d5e/de4/df3/fbb 1687246 0 2024-04-16T02:35:45.871 INFO:tasks.workunit.client.1.smithi153.stdout:9/885: creat d4/d6/d1b/d2a/d99/f115 x:0 0 0 2024-04-16T02:35:45.872 INFO:tasks.workunit.client.1.smithi153.stdout:8/801: mkdir d2/d13/d3a/dc2/d10d 0 2024-04-16T02:35:45.873 INFO:tasks.workunit.client.1.smithi153.stdout:2/819: unlink d9/d2b/d5c/f75 0 2024-04-16T02:35:45.874 INFO:tasks.workunit.client.1.smithi153.stdout:9/886: dread - d4/d6/dc/fe2 zero size 2024-04-16T02:35:45.874 INFO:tasks.workunit.client.1.smithi153.stdout:5/888: rmdir d8/df/d2f 39 2024-04-16T02:35:45.876 INFO:tasks.workunit.client.1.smithi153.stdout:7/815: dwrite d0/d4/db/d2b/d43/d6d/f6f [0,4194304] 0 2024-04-16T02:35:45.876 INFO:tasks.workunit.client.1.smithi153.stdout:8/802: mkdir d2/d13/d2c/d10e 0 2024-04-16T02:35:45.876 INFO:tasks.workunit.client.1.smithi153.stdout:0/972: write dd/d14/da6/da4/f134 [429868,9909] 0 2024-04-16T02:35:45.880 INFO:tasks.workunit.client.1.smithi153.stdout:5/889: stat d8/df/c1b 0 2024-04-16T02:35:45.880 INFO:tasks.workunit.client.1.smithi153.stdout:2/820: creat d9/d2b/d1c/d20/d88/f10c x:0 0 0 2024-04-16T02:35:45.884 INFO:tasks.workunit.client.1.smithi153.stdout:9/887: mknod d4/d6/ddb/c116 0 2024-04-16T02:35:45.887 INFO:tasks.workunit.client.1.smithi153.stdout:3/816: dread d6/d10/d22/d2c/f63 [0,4194304] 0 2024-04-16T02:35:45.889 INFO:tasks.workunit.client.1.smithi153.stdout:8/803: symlink d2/d93/d97/d9c/d27/l10f 0 2024-04-16T02:35:45.890 INFO:tasks.workunit.client.1.smithi153.stdout:3/817: readlink d6/l15 0 2024-04-16T02:35:45.890 INFO:tasks.workunit.client.1.smithi153.stdout:9/888: fdatasync d4/d6/d1b/ddf/f10c 0 2024-04-16T02:35:45.897 INFO:tasks.workunit.client.1.smithi153.stdout:3/818: chown d6/d10/d22/fb1 10014 1 2024-04-16T02:35:45.897 INFO:tasks.workunit.client.1.smithi153.stdout:4/944: dread d3/d9/db/d45/df3/fa6 [0,4194304] 0 2024-04-16T02:35:45.898 INFO:tasks.workunit.client.1.smithi153.stdout:1/849: dwrite d4/dd/d13/f2a [4194304,4194304] 0 2024-04-16T02:35:45.902 INFO:tasks.workunit.client.1.smithi153.stdout:0/973: dwrite dd/d32/d80/f125 [0,4194304] 0 2024-04-16T02:35:45.902 INFO:tasks.workunit.client.1.smithi153.stdout:7/816: link d0/d4/db/d2b/d8c/d9f/c117 d0/de8/c118 0 2024-04-16T02:35:45.903 INFO:tasks.workunit.client.1.smithi153.stdout:5/890: creat d8/df/d28/d5a/f117 x:0 0 0 2024-04-16T02:35:45.904 INFO:tasks.workunit.client.1.smithi153.stdout:3/819: rmdir d6/d10/d3b 39 2024-04-16T02:35:45.905 INFO:tasks.workunit.client.1.smithi153.stdout:2/821: dwrite d9/d24/d34/d60/fa9 [0,4194304] 0 2024-04-16T02:35:45.907 INFO:tasks.workunit.client.1.smithi153.stdout:8/804: dread d2/d93/d97/d9c/f3c [0,4194304] 0 2024-04-16T02:35:45.914 INFO:tasks.workunit.client.1.smithi153.stdout:9/889: rmdir d4/d6/def/df3 0 2024-04-16T02:35:45.914 INFO:tasks.workunit.client.1.smithi153.stdout:2/822: write d9/d2b/d5c/fd2 [1697458,114029] 0 2024-04-16T02:35:45.914 INFO:tasks.workunit.client.1.smithi153.stdout:1/850: mkdir d4/dd/d13/d7e/de6/d111/d123 0 2024-04-16T02:35:45.916 INFO:tasks.workunit.client.1.smithi153.stdout:5/891: mknod d8/d41/dbd/dd3/c118 0 2024-04-16T02:35:45.918 INFO:tasks.workunit.client.0.smithi151.stdout:5/358: creat d2/d6/d19/d30/d36/d42/f71 x:0 0 0 2024-04-16T02:35:45.919 INFO:tasks.workunit.client.1.smithi153.stdout:5/892: fsync d8/d41/f10f 0 2024-04-16T02:35:45.919 INFO:tasks.workunit.client.0.smithi151.stdout:5/359: chown d2/l17 430650934 1 2024-04-16T02:35:45.920 INFO:tasks.workunit.client.1.smithi153.stdout:0/974: dread dd/d14/d19/d77/db6/fec [0,4194304] 0 2024-04-16T02:35:45.921 INFO:tasks.workunit.client.0.smithi151.stdout:5/360: stat d2/d6/d19/d30/d36/d42/f5a 0 2024-04-16T02:35:45.921 INFO:tasks.workunit.client.1.smithi153.stdout:7/817: creat d0/d4/db/d2b/d8c/f119 x:0 0 0 2024-04-16T02:35:45.922 INFO:tasks.workunit.client.1.smithi153.stdout:2/823: symlink d9/d71/d8d/da8/l10d 0 2024-04-16T02:35:45.923 INFO:tasks.workunit.client.1.smithi153.stdout:5/893: read - d8/df/d28/d5a/fe1 zero size 2024-04-16T02:35:45.924 INFO:tasks.workunit.client.1.smithi153.stdout:4/945: dwrite d3/d9/db/de/d2c/d52/d85/ffe [0,4194304] 0 2024-04-16T02:35:45.925 INFO:tasks.workunit.client.1.smithi153.stdout:8/805: creat d2/f110 x:0 0 0 2024-04-16T02:35:45.925 INFO:tasks.workunit.client.0.smithi151.stdout:5/361: truncate d2/f12 585765 0 2024-04-16T02:35:45.926 INFO:tasks.workunit.client.1.smithi153.stdout:3/820: dwrite d6/d10/dec/ded/fee [4194304,4194304] 0 2024-04-16T02:35:45.926 INFO:tasks.workunit.client.1.smithi153.stdout:1/851: truncate d4/da2/d65/fbb 620092 0 2024-04-16T02:35:45.927 INFO:tasks.workunit.client.0.smithi151.stdout:5/362: stat d2/d6/d19/d1a/f40 0 2024-04-16T02:35:45.928 INFO:tasks.workunit.client.1.smithi153.stdout:0/975: chown dd/d14/c109 126212808 1 2024-04-16T02:35:45.929 INFO:tasks.workunit.client.1.smithi153.stdout:2/824: mkdir d9/d2b/d18/d3f/d10e 0 2024-04-16T02:35:45.929 INFO:tasks.workunit.client.0.smithi151.stdout:5/363: creat d2/d5e/f72 x:0 0 0 2024-04-16T02:35:45.930 INFO:tasks.workunit.client.1.smithi153.stdout:7/818: chown d0/d4/db/d2b/d43/d6d/d68/d6c/cbb 3230654 1 2024-04-16T02:35:45.930 INFO:tasks.workunit.client.1.smithi153.stdout:5/894: dread - d8/d41/dbd/f81 zero size 2024-04-16T02:35:45.931 INFO:tasks.workunit.client.1.smithi153.stdout:3/821: chown d6/d10/d22/cd2 543214927 1 2024-04-16T02:35:45.931 INFO:tasks.workunit.client.1.smithi153.stdout:1/852: fdatasync d4/dd/d18/d29/fd2 0 2024-04-16T02:35:45.932 INFO:tasks.workunit.client.1.smithi153.stdout:4/946: rename d3/d9/db/cf9 to d3/dbc/d10d/c14b 0 2024-04-16T02:35:45.932 INFO:tasks.workunit.client.0.smithi151.stdout:5/364: write d2/f5 [4010015,22074] 0 2024-04-16T02:35:45.932 INFO:tasks.workunit.client.1.smithi153.stdout:0/976: chown dd/d14/da6/da4/f133 0 1 2024-04-16T02:35:45.933 INFO:tasks.workunit.client.1.smithi153.stdout:2/825: write d9/fba [2749980,60775] 0 2024-04-16T02:35:45.934 INFO:tasks.workunit.client.1.smithi153.stdout:2/826: readlink d9/d71/la1 0 2024-04-16T02:35:45.938 INFO:tasks.workunit.client.0.smithi151.stdout:8/258: write d11/d22/d2e/f3d [371711,98616] 0 2024-04-16T02:35:45.938 INFO:tasks.workunit.client.1.smithi153.stdout:5/895: mkdir d8/df/d2e/d119 0 2024-04-16T02:35:45.943 INFO:tasks.workunit.client.1.smithi153.stdout:1/853: dwrite d4/dd/d13/d23/fab [0,4194304] 0 2024-04-16T02:35:45.945 INFO:tasks.workunit.client.0.smithi151.stdout:5/365: link d2/c21 d2/d6/d19/d1a/d4c/c73 0 2024-04-16T02:35:45.946 INFO:tasks.workunit.client.1.smithi153.stdout:4/947: fdatasync d3/d9/db/de/f5c 0 2024-04-16T02:35:45.946 INFO:tasks.workunit.client.1.smithi153.stdout:7/819: mknod d0/d4/db/d47/d8d/d93/d8b/c11a 0 2024-04-16T02:35:45.947 INFO:tasks.workunit.client.1.smithi153.stdout:1/854: chown d4/da2/f96 128245996 1 2024-04-16T02:35:45.947 INFO:tasks.workunit.client.0.smithi151.stdout:5/366: mkdir d2/d6/d19/d1a/d4c/d74 0 2024-04-16T02:35:45.955 INFO:tasks.workunit.client.1.smithi153.stdout:3/822: rename d6/d10/d3b/d10d/c38 to d6/d10/d3b/d59/d7c/c111 0 2024-04-16T02:35:45.955 INFO:tasks.workunit.client.0.smithi151.stdout:5/367: dwrite d2/d6/f25 [4194304,4194304] 0 2024-04-16T02:35:45.955 INFO:tasks.workunit.client.1.smithi153.stdout:8/806: dread d2/f23 [0,4194304] 0 2024-04-16T02:35:45.959 INFO:tasks.workunit.client.1.smithi153.stdout:0/977: dread dd/d14/d19/d20/f88 [0,4194304] 0 2024-04-16T02:35:45.960 INFO:tasks.workunit.client.1.smithi153.stdout:4/948: mknod d3/d9/db/de/d2c/de2/d110/c14c 0 2024-04-16T02:35:45.961 INFO:tasks.workunit.client.1.smithi153.stdout:7/820: dwrite d0/d4/db/d34/f10f [0,4194304] 0 2024-04-16T02:35:45.961 INFO:tasks.workunit.client.1.smithi153.stdout:3/823: write d6/d10/d3b/d10d/d8a/fb6 [739466,52391] 0 2024-04-16T02:35:45.962 INFO:tasks.workunit.client.0.smithi151.stdout:8/259: truncate ff 924717 0 2024-04-16T02:35:45.963 INFO:tasks.workunit.client.0.smithi151.stdout:8/260: write f2 [2755415,74391] 0 2024-04-16T02:35:45.964 INFO:tasks.workunit.client.1.smithi153.stdout:5/896: rename d8/d1f/feb to d8/df/d84/f11a 0 2024-04-16T02:35:45.964 INFO:tasks.workunit.client.1.smithi153.stdout:4/949: write d3/da5/dd2/f127 [770453,126139] 0 2024-04-16T02:35:45.967 INFO:tasks.workunit.client.1.smithi153.stdout:5/897: chown d8/d41/d90/ded/f101 1 1 2024-04-16T02:35:45.967 INFO:tasks.workunit.client.0.smithi151.stdout:5/368: link d2/d6/d19/d1a/l6a d2/d6/d19/d1a/d4c/d74/l75 0 2024-04-16T02:35:45.968 INFO:tasks.workunit.client.0.smithi151.stdout:8/261: symlink d11/l54 0 2024-04-16T02:35:45.968 INFO:tasks.workunit.client.1.smithi153.stdout:9/890: dwrite d4/d6/ddb/d4a/f68 [0,4194304] 0 2024-04-16T02:35:45.970 INFO:tasks.workunit.client.1.smithi153.stdout:6/822: sync 2024-04-16T02:35:45.970 INFO:tasks.workunit.client.0.smithi151.stdout:5/369: mknod d2/d6/d19/d69/c76 0 2024-04-16T02:35:45.971 INFO:tasks.workunit.client.0.smithi151.stdout:8/262: chown d11/d22/c47 419 1 2024-04-16T02:35:45.972 INFO:tasks.workunit.client.1.smithi153.stdout:8/807: creat d2/d13/d96/dc7/d109/d47/d77/ddf/f111 x:0 0 0 2024-04-16T02:35:45.972 INFO:tasks.workunit.client.1.smithi153.stdout:3/824: creat d6/d10/d3b/dd7/f112 x:0 0 0 2024-04-16T02:35:45.975 INFO:tasks.workunit.client.1.smithi153.stdout:4/950: mkdir d3/dbc/d10d/d14d 0 2024-04-16T02:35:45.976 INFO:tasks.workunit.client.1.smithi153.stdout:7/821: symlink d0/d4/db/d2b/d43/d6d/l11b 0 2024-04-16T02:35:45.978 INFO:tasks.workunit.client.1.smithi153.stdout:9/891: mkdir d4/d6/db1/d117 0 2024-04-16T02:35:45.979 INFO:tasks.workunit.client.1.smithi153.stdout:0/978: dread dd/d14/da6/f48 [0,4194304] 0 2024-04-16T02:35:45.980 INFO:tasks.workunit.client.0.smithi151.stdout:5/370: read d2/d6/f8 [2987809,94725] 0 2024-04-16T02:35:45.980 INFO:tasks.workunit.client.0.smithi151.stdout:8/263: dwrite d11/f16 [0,4194304] 0 2024-04-16T02:35:45.981 INFO:tasks.workunit.client.1.smithi153.stdout:6/823: dwrite dc/d6b/f73 [8388608,4194304] 0 2024-04-16T02:35:45.981 INFO:tasks.workunit.client.1.smithi153.stdout:0/979: chown dd/d14/d51/d54/fc7 155 1 2024-04-16T02:35:45.982 INFO:tasks.workunit.client.0.smithi151.stdout:8/264: chown d11/d2a/c2d 93 1 2024-04-16T02:35:45.982 INFO:tasks.workunit.client.1.smithi153.stdout:2/827: write d9/d2b/d5c/d5f/f61 [268434,96984] 0 2024-04-16T02:35:45.984 INFO:tasks.workunit.client.1.smithi153.stdout:5/898: creat d8/d41/dbd/dd3/f11b x:0 0 0 2024-04-16T02:35:45.986 INFO:tasks.workunit.client.1.smithi153.stdout:8/808: stat d2/d93/ce3 0 2024-04-16T02:35:45.986 INFO:tasks.workunit.client.1.smithi153.stdout:5/899: write d8/df/d28/d5a/fe1 [319807,24502] 0 2024-04-16T02:35:45.992 INFO:tasks.workunit.client.1.smithi153.stdout:7/822: unlink d0/d4/db/d47/d8d/d93/l80 0 2024-04-16T02:35:45.992 INFO:tasks.workunit.client.1.smithi153.stdout:9/892: mknod d4/d6/d1b/d2a/c118 0 2024-04-16T02:35:45.993 INFO:tasks.workunit.client.1.smithi153.stdout:1/855: dread d4/fc4 [0,4194304] 0 2024-04-16T02:35:45.993 INFO:tasks.workunit.client.0.smithi151.stdout:5/371: creat d2/d6/d19/d30/f77 x:0 0 0 2024-04-16T02:35:45.993 INFO:tasks.workunit.client.0.smithi151.stdout:8/265: dwrite d11/d2a/f42 [0,4194304] 0 2024-04-16T02:35:45.994 INFO:tasks.workunit.client.1.smithi153.stdout:2/828: dwrite d9/d24/d27/f68 [0,4194304] 0 2024-04-16T02:35:45.995 INFO:tasks.workunit.client.1.smithi153.stdout:7/823: chown d0/d4/db/fd3 1 1 2024-04-16T02:35:45.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:45 smithi151 ceph-mon[29003]: pgmap v20: 65 pgs: 65 active+clean; 2.0 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 62 MiB/s rd, 221 MiB/s wr, 410 op/s 2024-04-16T02:35:45.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:45 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:45.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:45 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:45.996 INFO:tasks.workunit.client.1.smithi153.stdout:6/824: rmdir dc/d47/d57/d77/dc4/dc9 39 2024-04-16T02:35:45.998 INFO:tasks.workunit.client.1.smithi153.stdout:2/829: read d9/d2b/d1c/d20/f101 [3380053,37005] 0 2024-04-16T02:35:46.000 INFO:tasks.workunit.client.0.smithi151.stdout:6/426: sync 2024-04-16T02:35:46.001 INFO:tasks.workunit.client.0.smithi151.stdout:5/372: link d2/l17 d2/d6/d19/d1a/d4c/d74/l78 0 2024-04-16T02:35:46.003 INFO:tasks.workunit.client.1.smithi153.stdout:3/825: creat d6/d10/d22/d4d/da0/deb/f113 x:0 0 0 2024-04-16T02:35:46.004 INFO:tasks.workunit.client.1.smithi153.stdout:4/951: unlink d3/d9/db/de/l10 0 2024-04-16T02:35:46.005 INFO:tasks.workunit.client.0.smithi151.stdout:8/266: dwrite d11/f28 [0,4194304] 0 2024-04-16T02:35:46.007 INFO:tasks.workunit.client.1.smithi153.stdout:8/809: creat d2/d13/d3a/dd6/f112 x:0 0 0 2024-04-16T02:35:46.008 INFO:tasks.workunit.client.1.smithi153.stdout:5/900: unlink d8/d41/dbd/d8d/lfb 0 2024-04-16T02:35:46.009 INFO:tasks.workunit.client.1.smithi153.stdout:8/810: readlink d2/d13/d3a/dc2/le6 0 2024-04-16T02:35:46.009 INFO:tasks.workunit.client.0.smithi151.stdout:6/427: rename d2/d13/d70/f57 to d2/d5/f8f 0 2024-04-16T02:35:46.010 INFO:tasks.workunit.client.1.smithi153.stdout:0/980: dwrite dd/d14/d19/fc4 [0,4194304] 0 2024-04-16T02:35:46.012 INFO:tasks.workunit.client.1.smithi153.stdout:1/856: mknod d4/dd/d13/d7e/c124 0 2024-04-16T02:35:46.012 INFO:tasks.workunit.client.1.smithi153.stdout:0/981: write dd/d14/da6/da4/f133 [70002,96457] 0 2024-04-16T02:35:46.015 INFO:tasks.workunit.client.0.smithi151.stdout:8/267: dwrite d11/f38 [0,4194304] 0 2024-04-16T02:35:46.016 INFO:tasks.workunit.client.1.smithi153.stdout:8/811: read d2/d13/d96/dc7/d109/d47/d60/d67/d86/fde [459032,84247] 0 2024-04-16T02:35:46.016 INFO:tasks.workunit.client.1.smithi153.stdout:9/893: mkdir d4/d6/dc/d2d/d64/db9/dc1/d119 0 2024-04-16T02:35:46.017 INFO:tasks.workunit.client.0.smithi151.stdout:6/428: link d2/l3 d2/d5/df/d23/d82/d8d/d22/d40/d3f/l90 0 2024-04-16T02:35:46.018 INFO:tasks.workunit.client.1.smithi153.stdout:8/812: chown d2/d93/d97/d9c/f7c 36721 1 2024-04-16T02:35:46.021 INFO:tasks.workunit.client.1.smithi153.stdout:7/824: mknod d0/d4/db/d2b/d53/d59/c11c 0 2024-04-16T02:35:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:45 smithi153 ceph-mon[36747]: pgmap v20: 65 pgs: 65 active+clean; 2.0 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 62 MiB/s rd, 221 MiB/s wr, 410 op/s 2024-04-16T02:35:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:45 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:45 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:46.025 INFO:tasks.workunit.client.1.smithi153.stdout:2/830: mkdir d9/d24/d27/d2c/d5d/da3/dc0/d10f 0 2024-04-16T02:35:46.028 INFO:tasks.workunit.client.0.smithi151.stdout:6/429: dwrite d2/d5/df/d20/d4f/f5e [0,4194304] 0 2024-04-16T02:35:46.030 INFO:tasks.workunit.client.1.smithi153.stdout:7/825: dwrite d0/d4/db/d2b/d43/d6d/f63 [4194304,4194304] 0 2024-04-16T02:35:46.031 INFO:tasks.workunit.client.0.smithi151.stdout:6/430: fsync d2/d5/df/d23/d82/d8d/d22/d40/d3f/f66 0 2024-04-16T02:35:46.031 INFO:tasks.workunit.client.1.smithi153.stdout:7/826: write d0/d4/db/d2b/f89 [2159178,123982] 0 2024-04-16T02:35:46.033 INFO:tasks.workunit.client.1.smithi153.stdout:5/901: creat d8/df/d28/d5a/d72/dc0/f11c x:0 0 0 2024-04-16T02:35:46.036 INFO:tasks.workunit.client.0.smithi151.stdout:6/431: getdents d2/d13/d46 0 2024-04-16T02:35:46.038 INFO:tasks.workunit.client.1.smithi153.stdout:9/894: readlink d4/d6/ddb/l114 0 2024-04-16T02:35:46.039 INFO:tasks.workunit.client.0.smithi151.stdout:6/432: rmdir d2/d5/df/d23/d82/d8d/d22/d40/d3f 39 2024-04-16T02:35:46.040 INFO:tasks.workunit.client.1.smithi153.stdout:8/813: creat d2/d93/f113 x:0 0 0 2024-04-16T02:35:46.043 INFO:tasks.workunit.client.1.smithi153.stdout:2/831: creat d9/d24/d34/d60/f110 x:0 0 0 2024-04-16T02:35:46.045 INFO:tasks.workunit.client.1.smithi153.stdout:2/832: dread d9/d71/fca [0,4194304] 0 2024-04-16T02:35:46.047 INFO:tasks.workunit.client.1.smithi153.stdout:7/827: creat d0/d4/db/d2b/d53/d59/dc3/f11d x:0 0 0 2024-04-16T02:35:46.049 INFO:tasks.workunit.client.1.smithi153.stdout:5/902: mknod d8/d41/dbd/c11d 0 2024-04-16T02:35:46.049 INFO:tasks.workunit.client.0.smithi151.stdout:6/433: dwrite d2/d5/f5a [0,4194304] 0 2024-04-16T02:35:46.051 INFO:tasks.workunit.client.0.smithi151.stdout:5/373: truncate d2/d6/f50 3950294 0 2024-04-16T02:35:46.052 INFO:tasks.workunit.client.0.smithi151.stdout:6/434: rename d2/d5 to d2/d5/df/d23/d56/d91 22 2024-04-16T02:35:46.054 INFO:tasks.workunit.client.1.smithi153.stdout:1/857: rename d4/dd/d13/d23/c32 to d4/dd/d13/d23/db6/c125 0 2024-04-16T02:35:46.054 INFO:tasks.workunit.client.0.smithi151.stdout:5/374: rmdir d2/d6/d19/d30/d49 39 2024-04-16T02:35:46.056 INFO:tasks.workunit.client.1.smithi153.stdout:6/825: creat dc/f113 x:0 0 0 2024-04-16T02:35:46.057 INFO:tasks.workunit.client.0.smithi151.stdout:6/435: mknod d2/d5/df/d23/d31/c92 0 2024-04-16T02:35:46.062 INFO:tasks.workunit.client.0.smithi151.stdout:6/436: creat d2/d5/df/d20/d4f/f93 x:0 0 0 2024-04-16T02:35:46.063 INFO:tasks.workunit.client.1.smithi153.stdout:0/982: dread dd/d14/dca/dcd/ffc [0,4194304] 0 2024-04-16T02:35:46.065 INFO:tasks.workunit.client.0.smithi151.stdout:5/375: dwrite d2/d6/d19/d30/f3f [0,4194304] 0 2024-04-16T02:35:46.065 INFO:tasks.workunit.client.0.smithi151.stdout:6/437: dread - d2/d5/d2e/f74 zero size 2024-04-16T02:35:46.067 INFO:tasks.workunit.client.1.smithi153.stdout:9/895: unlink d4/d6/dc/d46/f49 0 2024-04-16T02:35:46.068 INFO:tasks.workunit.client.0.smithi151.stdout:5/376: write d2/d6/d19/d1a/f2e [745963,20542] 0 2024-04-16T02:35:46.068 INFO:tasks.workunit.client.1.smithi153.stdout:9/896: dread - d4/d6/dc/d24/d60/fed zero size 2024-04-16T02:35:46.070 INFO:tasks.workunit.client.0.smithi151.stdout:5/377: write d2/d6/fa [2243635,117072] 0 2024-04-16T02:35:46.070 INFO:tasks.workunit.client.1.smithi153.stdout:1/858: read - d4/dd/d13/d23/d31/f8b zero size 2024-04-16T02:35:46.076 INFO:tasks.workunit.client.0.smithi151.stdout:6/438: creat d2/d5/df/d23/d82/d8d/d22/d40/d3f/d65/f94 x:0 0 0 2024-04-16T02:35:46.084 INFO:tasks.workunit.client.1.smithi153.stdout:7/828: dread d0/d4/db/d16/fb1 [0,4194304] 0 2024-04-16T02:35:46.088 INFO:tasks.workunit.client.1.smithi153.stdout:6/826: creat dc/d47/d57/d77/d15/d109/f114 x:0 0 0 2024-04-16T02:35:46.089 INFO:tasks.workunit.client.0.smithi151.stdout:6/439: write d2/d5/df/f48 [153396,69313] 0 2024-04-16T02:35:46.089 INFO:tasks.workunit.client.1.smithi153.stdout:8/814: rmdir d2/d13/d2c/d10e 0 2024-04-16T02:35:46.091 INFO:tasks.workunit.client.0.smithi151.stdout:5/378: dwrite d2/d6/f50 [0,4194304] 0 2024-04-16T02:35:46.092 INFO:tasks.workunit.client.1.smithi153.stdout:3/826: truncate d6/d10/d3b/d10d/f80 4168248 0 2024-04-16T02:35:46.093 INFO:tasks.workunit.client.0.smithi151.stdout:6/440: readlink d2/d5/df/d23/d56/d72/l78 0 2024-04-16T02:35:46.096 INFO:tasks.workunit.client.1.smithi153.stdout:5/903: getdents d8/df/d23/d70 0 2024-04-16T02:35:46.096 INFO:tasks.workunit.client.0.smithi151.stdout:5/379: link d2/d6/d19/d30/d36/f38 d2/d6/d19/d1a/f79 0 2024-04-16T02:35:46.098 INFO:tasks.workunit.client.1.smithi153.stdout:8/815: truncate d2/f7 3234420 0 2024-04-16T02:35:46.099 INFO:tasks.workunit.client.0.smithi151.stdout:5/380: symlink d2/d6/d19/d69/l7a 0 2024-04-16T02:35:46.100 INFO:tasks.workunit.client.0.smithi151.stdout:6/441: creat d2/d5/df/d23/d82/d8d/d22/d40/f95 x:0 0 0 2024-04-16T02:35:46.100 INFO:tasks.workunit.client.0.smithi151.stdout:5/381: fdatasync d2/d6/d19/f1d 0 2024-04-16T02:35:46.101 INFO:tasks.workunit.client.1.smithi153.stdout:2/833: dread d9/d24/d34/d36/f3b [0,4194304] 0 2024-04-16T02:35:46.101 INFO:tasks.workunit.client.1.smithi153.stdout:3/827: chown d6/l21 246103 1 2024-04-16T02:35:46.102 INFO:tasks.workunit.client.1.smithi153.stdout:7/829: mkdir d0/de8/d110/d11e 0 2024-04-16T02:35:46.103 INFO:tasks.workunit.client.0.smithi151.stdout:6/442: write d2/d5/df/d23/d56/f61 [301988,13987] 0 2024-04-16T02:35:46.106 INFO:tasks.workunit.client.1.smithi153.stdout:2/834: read d9/d24/d27/d2c/d41/d7a/fb2 [2302252,56608] 0 2024-04-16T02:35:46.106 INFO:tasks.workunit.client.1.smithi153.stdout:2/835: dread - d9/d24/d34/d36/ffb zero size 2024-04-16T02:35:46.107 INFO:tasks.workunit.client.0.smithi151.stdout:6/443: mkdir d2/d5/d96 0 2024-04-16T02:35:46.107 INFO:tasks.workunit.client.1.smithi153.stdout:8/816: chown d2/d13/d96/fb3 5841437 1 2024-04-16T02:35:46.110 INFO:tasks.workunit.client.0.smithi151.stdout:5/382: dwrite d2/d6/f6c [0,4194304] 0 2024-04-16T02:35:46.111 INFO:tasks.workunit.client.1.smithi153.stdout:3/828: unlink d6/d10/d22/l37 0 2024-04-16T02:35:46.112 INFO:tasks.workunit.client.1.smithi153.stdout:3/829: write d6/fe4 [650243,29628] 0 2024-04-16T02:35:46.114 INFO:tasks.workunit.client.0.smithi151.stdout:6/444: mkdir d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/d97 0 2024-04-16T02:35:46.117 INFO:tasks.workunit.client.1.smithi153.stdout:6/827: creat dc/f115 x:0 0 0 2024-04-16T02:35:46.118 INFO:tasks.workunit.client.1.smithi153.stdout:2/836: mknod d9/d24/ddd/c111 0 2024-04-16T02:35:46.118 INFO:tasks.workunit.client.1.smithi153.stdout:6/828: write dc/d10/f6c [4755577,1245] 0 2024-04-16T02:35:46.119 INFO:tasks.workunit.client.0.smithi151.stdout:6/445: chown d2/d13/f18 850791 1 2024-04-16T02:35:46.119 INFO:tasks.workunit.client.0.smithi151.stdout:5/383: chown d2/f24 0 1 2024-04-16T02:35:46.119 INFO:tasks.workunit.client.0.smithi151.stdout:5/384: readlink d2/d6/l15 0 2024-04-16T02:35:46.120 INFO:tasks.workunit.client.1.smithi153.stdout:8/817: symlink d2/de/d34/l114 0 2024-04-16T02:35:46.121 INFO:tasks.workunit.client.1.smithi153.stdout:9/897: dread d4/d33/dae/dc7/d66/fb2 [0,4194304] 0 2024-04-16T02:35:46.124 INFO:tasks.workunit.client.0.smithi151.stdout:6/446: getdents d2/d13 0 2024-04-16T02:35:46.126 INFO:tasks.workunit.client.1.smithi153.stdout:7/830: dread d0/d4/db/d2b/d8c/d9f/fd8 [0,4194304] 0 2024-04-16T02:35:46.126 INFO:tasks.workunit.client.0.smithi151.stdout:5/385: write d2/d6/f8 [2984684,85393] 0 2024-04-16T02:35:46.127 INFO:tasks.workunit.client.0.smithi151.stdout:6/447: truncate d2/d5/df/d23/d56/f8a 730540 0 2024-04-16T02:35:46.128 INFO:tasks.workunit.client.0.smithi151.stdout:6/448: chown d2/d5/df/d87 195 1 2024-04-16T02:35:46.129 INFO:tasks.workunit.client.1.smithi153.stdout:2/837: dwrite d9/d24/d34/d60/ff6 [0,4194304] 0 2024-04-16T02:35:46.129 INFO:tasks.workunit.client.1.smithi153.stdout:9/898: rename d4/d33/dae/dc7/d66/dcb/f10e to d4/d33/dae/dc7/d66/d10a/f11a 0 2024-04-16T02:35:46.130 INFO:tasks.workunit.client.0.smithi151.stdout:5/386: creat d2/d6/d19/d62/f7b x:0 0 0 2024-04-16T02:35:46.131 INFO:tasks.workunit.client.1.smithi153.stdout:3/830: dwrite d6/d10/d3b/d59/fa9 [0,4194304] 0 2024-04-16T02:35:46.131 INFO:tasks.workunit.client.1.smithi153.stdout:6/829: mknod dc/d47/d57/d77/dc4/dc9/ded/c116 0 2024-04-16T02:35:46.136 INFO:tasks.workunit.client.1.smithi153.stdout:7/831: rmdir d0/d4/db/d2b/d43/d6d 39 2024-04-16T02:35:46.136 INFO:tasks.workunit.client.0.smithi151.stdout:5/387: mknod d2/d6/d19/d30/d36/d63/c7c 0 2024-04-16T02:35:46.138 INFO:tasks.workunit.client.1.smithi153.stdout:3/831: truncate d6/d10/f18 2312652 0 2024-04-16T02:35:46.140 INFO:tasks.workunit.client.0.smithi151.stdout:6/449: truncate d2/f1a 3142224 0 2024-04-16T02:35:46.140 INFO:tasks.workunit.client.0.smithi151.stdout:5/388: truncate d2/d6/f1c 655194 0 2024-04-16T02:35:46.141 INFO:tasks.workunit.client.1.smithi153.stdout:9/899: dread d4/d6/dc/d2d/d64/db9/fcf [0,4194304] 0 2024-04-16T02:35:46.141 INFO:tasks.workunit.client.0.smithi151.stdout:5/389: fsync d2/d6/f64 0 2024-04-16T02:35:46.141 INFO:tasks.workunit.client.1.smithi153.stdout:6/830: dwrite dc/d47/d57/d77/d15/dac/dae/ffc [0,4194304] 0 2024-04-16T02:35:46.146 INFO:tasks.workunit.client.0.smithi151.stdout:5/390: symlink d2/d6/d19/d30/d36/d42/d6f/l7d 0 2024-04-16T02:35:46.148 INFO:tasks.workunit.client.1.smithi153.stdout:3/832: dwrite d6/d10/d3b/d10d/ffa [0,4194304] 0 2024-04-16T02:35:46.149 INFO:tasks.workunit.client.1.smithi153.stdout:3/833: dread - d6/d10/d3b/d59/fe9 zero size 2024-04-16T02:35:46.150 INFO:tasks.workunit.client.0.smithi151.stdout:5/391: creat d2/d5e/f7e x:0 0 0 2024-04-16T02:35:46.153 INFO:tasks.workunit.client.1.smithi153.stdout:9/900: symlink d4/d6/dc/d24/l11b 0 2024-04-16T02:35:46.155 INFO:tasks.workunit.client.1.smithi153.stdout:9/901: dread d4/d6/dc/d2d/d64/db9/fcf [0,4194304] 0 2024-04-16T02:35:46.156 INFO:tasks.workunit.client.1.smithi153.stdout:7/832: rename d0/de8/d110/fb9 to d0/d4/db/d2b/d53/d59/d76/d103/f11f 0 2024-04-16T02:35:46.157 INFO:tasks.workunit.client.1.smithi153.stdout:7/833: dread - d0/d4/db/d47/d8d/d93/fcf zero size 2024-04-16T02:35:46.159 INFO:tasks.workunit.client.1.smithi153.stdout:6/831: rmdir dc/d47/d57 39 2024-04-16T02:35:46.159 INFO:tasks.workunit.client.0.smithi151.stdout:5/392: mkdir d2/d6/d19/d30/d49/d7f 0 2024-04-16T02:35:46.164 INFO:tasks.workunit.client.1.smithi153.stdout:2/838: truncate d9/d2b/f19 858081 0 2024-04-16T02:35:46.166 INFO:tasks.workunit.client.1.smithi153.stdout:2/839: chown d9/d24/d34/cd4 1702 1 2024-04-16T02:35:46.168 INFO:tasks.workunit.client.1.smithi153.stdout:7/834: getdents d0/d4/db/d2b/d43/d6d/d68/d6c/db8 0 2024-04-16T02:35:46.169 INFO:tasks.workunit.client.1.smithi153.stdout:2/840: truncate d9/d2b/d18/f29 86928 0 2024-04-16T02:35:46.171 INFO:tasks.workunit.client.1.smithi153.stdout:2/841: creat d9/d24/d34/d99/f112 x:0 0 0 2024-04-16T02:35:46.174 INFO:tasks.workunit.client.0.smithi151.stdout:6/450: dwrite d2/d5/df/d23/d82/d8d/d22/f4e [0,4194304] 0 2024-04-16T02:35:46.177 INFO:tasks.workunit.client.0.smithi151.stdout:5/393: rmdir d2/d6 39 2024-04-16T02:35:46.177 INFO:tasks.workunit.client.0.smithi151.stdout:5/394: dread - d2/d5e/f7e zero size 2024-04-16T02:35:46.177 INFO:tasks.workunit.client.1.smithi153.stdout:7/835: getdents d0/d4/db/d2b 0 2024-04-16T02:35:46.179 INFO:tasks.workunit.client.0.smithi151.stdout:6/451: creat d2/d5/df/d23/d82/d8d/f98 x:0 0 0 2024-04-16T02:35:46.180 INFO:tasks.workunit.client.1.smithi153.stdout:3/834: write d6/d10/d3b/d10d/fcf [362371,23822] 0 2024-04-16T02:35:46.180 INFO:tasks.workunit.client.1.smithi153.stdout:9/902: dread d4/d33/dae/dc7/d66/f98 [0,4194304] 0 2024-04-16T02:35:46.182 INFO:tasks.workunit.client.1.smithi153.stdout:4/952: sync 2024-04-16T02:35:46.185 INFO:tasks.workunit.client.0.smithi151.stdout:5/395: dread - d2/d6/d19/d69/f6b zero size 2024-04-16T02:35:46.185 INFO:tasks.workunit.client.0.smithi151.stdout:6/452: rename d2/l4d to d2/d3d/l99 0 2024-04-16T02:35:46.186 INFO:tasks.workunit.client.0.smithi151.stdout:5/396: fsync d2/d6/f64 0 2024-04-16T02:35:46.187 INFO:tasks.workunit.client.1.smithi153.stdout:9/903: mkdir d4/d6/d1b/d2a/d99/d11c 0 2024-04-16T02:35:46.187 INFO:tasks.workunit.client.1.smithi153.stdout:4/953: read d3/d9/db/de/f5c [1530285,12114] 0 2024-04-16T02:35:46.192 INFO:tasks.workunit.client.0.smithi151.stdout:5/397: mknod d2/d6/d19/d1a/c80 0 2024-04-16T02:35:46.193 INFO:tasks.workunit.client.0.smithi151.stdout:6/453: creat d2/d5/df/d23/d45/d62/f9a x:0 0 0 2024-04-16T02:35:46.196 INFO:tasks.workunit.client.0.smithi151.stdout:6/454: chown d2/d5/df/d23/d82/d8d/d22/f4e 18217 1 2024-04-16T02:35:46.197 INFO:tasks.workunit.client.1.smithi153.stdout:4/954: dread d3/d9/db/de/d2c/d52/d85/fc0 [0,4194304] 0 2024-04-16T02:35:46.202 INFO:tasks.workunit.client.1.smithi153.stdout:3/835: write d6/d10/d3b/f6d [516095,1373] 0 2024-04-16T02:35:46.203 INFO:tasks.workunit.client.0.smithi151.stdout:6/455: link d2/d5/df/d23/d82/d8d/d22/d40/d3f/f7f d2/d5/df/d23/d82/d8d/d22/f9b 0 2024-04-16T02:35:46.204 INFO:tasks.workunit.client.1.smithi153.stdout:2/842: dread d9/d2b/d1c/d20/fde [0,4194304] 0 2024-04-16T02:35:46.204 INFO:tasks.workunit.client.0.smithi151.stdout:6/456: chown d2/d5/df/d23/d31/c92 627969910 1 2024-04-16T02:35:46.208 INFO:tasks.workunit.client.0.smithi151.stdout:6/457: creat d2/d5/df/d23/d82/f9c x:0 0 0 2024-04-16T02:35:46.209 INFO:tasks.workunit.client.0.smithi151.stdout:6/458: creat d2/d3d/f9d x:0 0 0 2024-04-16T02:35:46.214 INFO:tasks.workunit.client.0.smithi151.stdout:6/459: getdents d2 0 2024-04-16T02:35:46.214 INFO:tasks.workunit.client.1.smithi153.stdout:3/836: dwrite d6/d10/d3b/d59/d7c/fa8 [0,4194304] 0 2024-04-16T02:35:46.216 INFO:tasks.workunit.client.0.smithi151.stdout:5/398: dwrite d2/f12 [0,4194304] 0 2024-04-16T02:35:46.217 INFO:tasks.workunit.client.1.smithi153.stdout:4/955: dread d3/d9/f68 [0,4194304] 0 2024-04-16T02:35:46.219 INFO:tasks.workunit.client.1.smithi153.stdout:3/837: stat d6/d10/d22/f28 0 2024-04-16T02:35:46.221 INFO:tasks.workunit.client.1.smithi153.stdout:4/956: chown d3/c27 0 1 2024-04-16T02:35:46.223 INFO:tasks.workunit.client.0.smithi151.stdout:5/399: dread d2/f16 [0,4194304] 0 2024-04-16T02:35:46.223 INFO:tasks.workunit.client.1.smithi153.stdout:2/843: dread d9/d24/d27/d2c/d41/f4e [0,4194304] 0 2024-04-16T02:35:46.225 INFO:tasks.workunit.client.0.smithi151.stdout:5/400: write d2/d6/d19/f4b [4707353,12721] 0 2024-04-16T02:35:46.226 INFO:tasks.workunit.client.0.smithi151.stdout:6/460: dwrite d2/f9 [0,4194304] 0 2024-04-16T02:35:46.228 INFO:tasks.workunit.client.0.smithi151.stdout:5/401: chown d2/d6/f6c 311636095 1 2024-04-16T02:35:46.229 INFO:tasks.workunit.client.1.smithi153.stdout:3/838: dwrite d6/d10/d3b/d59/d7c/de1/fe3 [0,4194304] 0 2024-04-16T02:35:46.231 INFO:tasks.workunit.client.1.smithi153.stdout:3/839: write d6/d10/d3b/d76/d10b/f10e [537392,97082] 0 2024-04-16T02:35:46.231 INFO:tasks.workunit.client.1.smithi153.stdout:2/844: mkdir d9/d24/d27/d113 0 2024-04-16T02:35:46.232 INFO:tasks.workunit.client.0.smithi151.stdout:5/402: mknod d2/d6/d19/d30/c81 0 2024-04-16T02:35:46.232 INFO:tasks.workunit.client.1.smithi153.stdout:4/957: mknod d3/d9/db/d17/d28/dc8/de4/dfd/d114/c14e 0 2024-04-16T02:35:46.233 INFO:tasks.workunit.client.0.smithi151.stdout:5/403: chown d2/d6/d19/d1a 375691 1 2024-04-16T02:35:46.233 INFO:tasks.workunit.client.1.smithi153.stdout:3/840: write d6/f39 [3334747,122604] 0 2024-04-16T02:35:46.235 INFO:tasks.workunit.client.0.smithi151.stdout:6/461: getdents d2/d5/df/d23/d82/d8d 0 2024-04-16T02:35:46.238 INFO:tasks.workunit.client.1.smithi153.stdout:4/958: fsync d3/d9/f1a 0 2024-04-16T02:35:46.238 INFO:tasks.workunit.client.1.smithi153.stdout:4/959: fdatasync d3/d9/db/de/d2c/fc9 0 2024-04-16T02:35:46.238 INFO:tasks.workunit.client.0.smithi151.stdout:5/404: rename d2/d6/d19/d1a/c80 to d2/d6/d19/d30/d36/d63/c82 0 2024-04-16T02:35:46.239 INFO:tasks.workunit.client.0.smithi151.stdout:6/462: rmdir d2/d5/df/d23/d31/d52 39 2024-04-16T02:35:46.240 INFO:tasks.workunit.client.1.smithi153.stdout:4/960: readlink d3/d9/db/de/d2c/de2/d119/l9f 0 2024-04-16T02:35:46.241 INFO:tasks.workunit.client.0.smithi151.stdout:5/405: dread d2/d6/f64 [0,4194304] 0 2024-04-16T02:35:46.243 INFO:tasks.workunit.client.0.smithi151.stdout:6/463: creat d2/d5/df/d20/d4f/f9e x:0 0 0 2024-04-16T02:35:46.243 INFO:tasks.workunit.client.0.smithi151.stdout:5/406: write d2/d5e/f72 [31422,61095] 0 2024-04-16T02:35:46.244 INFO:tasks.workunit.client.1.smithi153.stdout:4/961: fsync d3/d9/db/d17/d3f/f94 0 2024-04-16T02:35:46.246 INFO:tasks.workunit.client.0.smithi151.stdout:5/407: write d2/d6/f8 [665477,15725] 0 2024-04-16T02:35:46.247 INFO:tasks.workunit.client.0.smithi151.stdout:5/408: fsync d2/d6/f29 0 2024-04-16T02:35:46.248 INFO:tasks.workunit.client.0.smithi151.stdout:5/409: readlink d2/d6/d19/d30/d36/d42/d6f/l7d 0 2024-04-16T02:35:46.251 INFO:tasks.workunit.client.0.smithi151.stdout:5/410: mkdir d2/d6/d19/d30/d36/d42/d6f/d83 0 2024-04-16T02:35:46.253 INFO:tasks.workunit.client.0.smithi151.stdout:5/411: creat d2/d6/d43/f84 x:0 0 0 2024-04-16T02:35:46.255 INFO:tasks.workunit.client.0.smithi151.stdout:5/412: mkdir d2/d6/d19/d30/d49/d85 0 2024-04-16T02:35:46.256 INFO:tasks.workunit.client.1.smithi153.stdout:3/841: dread d6/d10/f18 [0,4194304] 0 2024-04-16T02:35:46.258 INFO:tasks.workunit.client.0.smithi151.stdout:6/464: write d2/d5/df/d20/f47 [539220,123516] 0 2024-04-16T02:35:46.259 INFO:tasks.workunit.client.1.smithi153.stdout:3/842: creat d6/d10/d3b/d10d/d8a/d8c/f114 x:0 0 0 2024-04-16T02:35:46.260 INFO:tasks.workunit.client.0.smithi151.stdout:6/465: creat d2/f9f x:0 0 0 2024-04-16T02:35:46.261 INFO:tasks.workunit.client.0.smithi151.stdout:5/413: getdents d2/d6/d19/d1a/d4c/d74 0 2024-04-16T02:35:46.261 INFO:tasks.workunit.client.1.smithi153.stdout:3/843: mkdir d6/d10/d5b/d115 0 2024-04-16T02:35:46.262 INFO:tasks.workunit.client.0.smithi151.stdout:6/466: creat d2/d5/df/d23/d56/d72/fa0 x:0 0 0 2024-04-16T02:35:46.264 INFO:tasks.workunit.client.1.smithi153.stdout:3/844: rmdir d6/d10/d3b/d59/d7c/de1/df0 39 2024-04-16T02:35:46.268 INFO:tasks.workunit.client.1.smithi153.stdout:2/845: dread d9/d2b/d1c/d20/f79 [0,4194304] 0 2024-04-16T02:35:46.270 INFO:tasks.workunit.client.1.smithi153.stdout:2/846: write d9/d24/d27/d9a/f9f [235460,994] 0 2024-04-16T02:35:46.273 INFO:tasks.workunit.client.1.smithi153.stdout:3/845: dwrite d6/d10/d3b/d76/fc2 [0,4194304] 0 2024-04-16T02:35:46.273 INFO:tasks.workunit.client.1.smithi153.stdout:4/962: dread d3/d9/db/de/d2c/d38/f9d [0,4194304] 0 2024-04-16T02:35:46.273 INFO:tasks.workunit.client.0.smithi151.stdout:5/414: dwrite d2/d6/d43/f6d [0,4194304] 0 2024-04-16T02:35:46.274 INFO:tasks.workunit.client.1.smithi153.stdout:3/846: fsync d6/d10/d3b/d59/fa9 0 2024-04-16T02:35:46.275 INFO:tasks.workunit.client.1.smithi153.stdout:2/847: truncate d9/d2b/d1c/d20/fc2 2788841 0 2024-04-16T02:35:46.282 INFO:tasks.workunit.client.1.smithi153.stdout:2/848: symlink d9/d24/d27/l114 0 2024-04-16T02:35:46.287 INFO:tasks.workunit.client.1.smithi153.stdout:3/847: dwrite d6/d10/d22/d2c/f60 [0,4194304] 0 2024-04-16T02:35:46.287 INFO:tasks.workunit.client.0.smithi151.stdout:6/467: dread d2/d5/df/d23/d82/d8d/f29 [4194304,4194304] 0 2024-04-16T02:35:46.288 INFO:tasks.workunit.client.1.smithi153.stdout:2/849: mknod d9/d24/c115 0 2024-04-16T02:35:46.296 INFO:tasks.workunit.client.0.smithi151.stdout:6/468: rename d2/d5/df/d23/d31/f7d to d2/d3d/fa1 0 2024-04-16T02:35:46.296 INFO:tasks.workunit.client.1.smithi153.stdout:2/850: rename d9/d2b/d5c/da2 to d9/d24/d27/d2c/d41/d7a/d116 0 2024-04-16T02:35:46.299 INFO:tasks.workunit.client.1.smithi153.stdout:2/851: write d9/d2b/d18/fe4 [5129549,117758] 0 2024-04-16T02:35:46.299 INFO:tasks.workunit.client.0.smithi151.stdout:6/469: creat d2/d13/d63/fa2 x:0 0 0 2024-04-16T02:35:46.301 INFO:tasks.workunit.client.0.smithi151.stdout:6/470: chown d2/d5/df/d23/d56/c5d 120 1 2024-04-16T02:35:46.301 INFO:tasks.workunit.client.1.smithi153.stdout:2/852: truncate d9/d24/d34/d99/f112 926732 0 2024-04-16T02:35:46.302 INFO:tasks.workunit.client.1.smithi153.stdout:4/963: dread d3/d9/d24/f2b [0,4194304] 0 2024-04-16T02:35:46.303 INFO:tasks.workunit.client.1.smithi153.stdout:2/853: chown d9/d24/d27/d9a/fe8 30501970 1 2024-04-16T02:35:46.304 INFO:tasks.workunit.client.0.smithi151.stdout:6/471: truncate d2/d13/f18 2416769 0 2024-04-16T02:35:46.305 INFO:tasks.workunit.client.1.smithi153.stdout:3/848: write d6/d10/d22/f28 [1450156,97117] 0 2024-04-16T02:35:46.305 INFO:tasks.workunit.client.1.smithi153.stdout:2/854: write d9/d2b/d18/d3f/f6e [603917,111518] 0 2024-04-16T02:35:46.305 INFO:tasks.workunit.client.1.smithi153.stdout:4/964: getdents d3/d9/db/d17/d28/dc8/de4/d134 0 2024-04-16T02:35:46.306 INFO:tasks.workunit.client.0.smithi151.stdout:6/472: mkdir d2/d13/d6f/da3 0 2024-04-16T02:35:46.308 INFO:tasks.workunit.client.1.smithi153.stdout:4/965: truncate d3/d9/d24/d75/fb4 963781 0 2024-04-16T02:35:46.311 INFO:tasks.workunit.client.0.smithi151.stdout:6/473: rename d2/d5/df/d87/c8e to d2/d5/df/d23/d82/d8d/d22/d6d/ca4 0 2024-04-16T02:35:46.312 INFO:tasks.workunit.client.1.smithi153.stdout:4/966: creat d3/dbc/d10d/f14f x:0 0 0 2024-04-16T02:35:46.313 INFO:tasks.workunit.client.1.smithi153.stdout:0/983: sync 2024-04-16T02:35:46.313 INFO:tasks.workunit.client.0.smithi151.stdout:6/474: mknod d2/d5/df/d23/d56/ca5 0 2024-04-16T02:35:46.315 INFO:tasks.workunit.client.1.smithi153.stdout:4/967: fdatasync d3/d9/db/de/d2c/d52/d85/f133 0 2024-04-16T02:35:46.315 INFO:tasks.workunit.client.1.smithi153.stdout:2/855: dwrite d9/d71/df2/f102 [0,4194304] 0 2024-04-16T02:35:46.316 INFO:tasks.workunit.client.0.smithi151.stdout:6/475: mkdir d2/d13/d3a/da6 0 2024-04-16T02:35:46.316 INFO:tasks.workunit.client.0.smithi151.stdout:6/476: readlink d2/l4 0 2024-04-16T02:35:46.316 INFO:tasks.workunit.client.0.smithi151.stdout:6/477: stat d2/d13/d46 0 2024-04-16T02:35:46.321 INFO:tasks.workunit.client.1.smithi153.stdout:0/984: creat dd/d14/da6/f138 x:0 0 0 2024-04-16T02:35:46.323 INFO:tasks.workunit.client.0.smithi151.stdout:5/415: dread d2/d6/d19/d30/f44 [0,4194304] 0 2024-04-16T02:35:46.326 INFO:tasks.workunit.client.1.smithi153.stdout:3/849: dwrite d6/d10/d22/d2c/f74 [0,4194304] 0 2024-04-16T02:35:46.326 INFO:tasks.workunit.client.1.smithi153.stdout:0/985: write dd/d14/d19/d77/ff1 [1496200,53968] 0 2024-04-16T02:35:46.327 INFO:tasks.workunit.client.1.smithi153.stdout:2/856: dread d9/d2b/d5c/fd2 [0,4194304] 0 2024-04-16T02:35:46.327 INFO:tasks.workunit.client.0.smithi151.stdout:5/416: creat d2/d5e/f86 x:0 0 0 2024-04-16T02:35:46.328 INFO:tasks.workunit.client.1.smithi153.stdout:4/968: truncate d3/d9/d24/d75/dc5/fdb 23579 0 2024-04-16T02:35:46.329 INFO:tasks.workunit.client.1.smithi153.stdout:2/857: read d9/d24/d27/d2c/d41/f4e [196529,127085] 0 2024-04-16T02:35:46.331 INFO:tasks.workunit.client.1.smithi153.stdout:3/850: unlink d6/d10/d22/d2c/f63 0 2024-04-16T02:35:46.334 INFO:tasks.workunit.client.0.smithi151.stdout:5/417: creat d2/d6/d19/f87 x:0 0 0 2024-04-16T02:35:46.336 INFO:tasks.workunit.client.1.smithi153.stdout:4/969: creat d3/d9/db/d84/f150 x:0 0 0 2024-04-16T02:35:46.336 INFO:tasks.workunit.client.1.smithi153.stdout:2/858: dwrite d9/d24/d27/d2c/d5d/f63 [0,4194304] 0 2024-04-16T02:35:46.337 INFO:tasks.workunit.client.0.smithi151.stdout:5/418: dread - d2/d6/d19/d30/f77 zero size 2024-04-16T02:35:46.337 INFO:tasks.workunit.client.1.smithi153.stdout:3/851: creat d6/d10/d22/d4d/f116 x:0 0 0 2024-04-16T02:35:46.340 INFO:tasks.workunit.client.0.smithi151.stdout:6/478: dwrite d2/f33 [0,4194304] 0 2024-04-16T02:35:46.342 INFO:tasks.workunit.client.0.smithi151.stdout:5/419: getdents d2 0 2024-04-16T02:35:46.348 INFO:tasks.workunit.client.1.smithi153.stdout:0/986: getdents dd/d14/da6 0 2024-04-16T02:35:46.348 INFO:tasks.workunit.client.0.smithi151.stdout:5/420: unlink d2/d6/d19/d30/d49/f65 0 2024-04-16T02:35:46.349 INFO:tasks.workunit.client.1.smithi153.stdout:0/987: stat dd/d26 0 2024-04-16T02:35:46.349 INFO:tasks.workunit.client.1.smithi153.stdout:3/852: dwrite d6/d10/d22/d2c/f60 [0,4194304] 0 2024-04-16T02:35:46.351 INFO:tasks.workunit.client.1.smithi153.stdout:3/853: chown d6/d10/d22 1 1 2024-04-16T02:35:46.352 INFO:tasks.workunit.client.0.smithi151.stdout:6/479: link d2/d5/df/d23/d82/d8d/d22/d40/d3f/f6c d2/d5/df/d20/d4f/fa7 0 2024-04-16T02:35:46.353 INFO:tasks.workunit.client.0.smithi151.stdout:6/480: readlink d2/d13/d63/l77 0 2024-04-16T02:35:46.353 INFO:tasks.workunit.client.1.smithi153.stdout:2/859: dwrite d9/f9e [0,4194304] 0 2024-04-16T02:35:46.356 INFO:tasks.workunit.client.1.smithi153.stdout:3/854: unlink d6/daf/lc8 0 2024-04-16T02:35:46.359 INFO:tasks.workunit.client.0.smithi151.stdout:6/481: rename d2/d5/df/c14 to d2/d13/ca8 0 2024-04-16T02:35:46.359 INFO:tasks.workunit.client.0.smithi151.stdout:5/421: dwrite d2/d6/f6c [0,4194304] 0 2024-04-16T02:35:46.364 INFO:tasks.workunit.client.0.smithi151.stdout:2/344: sync 2024-04-16T02:35:46.365 INFO:tasks.workunit.client.0.smithi151.stdout:2/345: chown de/d15/d1e/l5f 19 1 2024-04-16T02:35:46.365 INFO:tasks.workunit.client.1.smithi153.stdout:0/988: creat dd/d14/d19/f139 x:0 0 0 2024-04-16T02:35:46.365 INFO:tasks.workunit.client.1.smithi153.stdout:3/855: rmdir d6/d10/d3b/d76 39 2024-04-16T02:35:46.366 INFO:tasks.workunit.client.0.smithi151.stdout:2/346: fdatasync de/d15/d1e/f67 0 2024-04-16T02:35:46.369 INFO:tasks.workunit.client.0.smithi151.stdout:6/482: dwrite d2/d5/df/d20/d4f/f5e [0,4194304] 0 2024-04-16T02:35:46.369 INFO:tasks.workunit.client.0.smithi151.stdout:2/347: truncate de/d11/f2b 8592002 0 2024-04-16T02:35:46.370 INFO:tasks.workunit.client.1.smithi153.stdout:0/989: mkdir dd/d32/d57/d13a 0 2024-04-16T02:35:46.370 INFO:tasks.workunit.client.0.smithi151.stdout:5/422: read d2/d6/d19/d30/d36/f38 [3849,47410] 0 2024-04-16T02:35:46.372 INFO:tasks.workunit.client.0.smithi151.stdout:5/423: fdatasync d2/d6/d43/f6d 0 2024-04-16T02:35:46.374 INFO:tasks.workunit.client.1.smithi153.stdout:0/990: creat dd/d14/d19/f13b x:0 0 0 2024-04-16T02:35:46.377 INFO:tasks.workunit.client.0.smithi151.stdout:6/483: creat d2/d5/df/d23/d45/fa9 x:0 0 0 2024-04-16T02:35:46.379 INFO:tasks.workunit.client.0.smithi151.stdout:2/348: dwrite de/d11/d41/f37 [0,4194304] 0 2024-04-16T02:35:46.379 INFO:tasks.workunit.client.0.smithi151.stdout:5/424: mkdir d2/d6/d19/d30/d88 0 2024-04-16T02:35:46.384 INFO:tasks.workunit.client.0.smithi151.stdout:6/484: mknod d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/caa 0 2024-04-16T02:35:46.388 INFO:tasks.workunit.client.0.smithi151.stdout:2/349: dwrite de/d15/f34 [4194304,4194304] 0 2024-04-16T02:35:46.392 INFO:tasks.workunit.client.0.smithi151.stdout:2/350: mkdir de/d3d/d6a 0 2024-04-16T02:35:46.400 INFO:tasks.workunit.client.0.smithi151.stdout:2/351: dwrite de/d11/d41/f33 [0,4194304] 0 2024-04-16T02:35:46.400 INFO:tasks.workunit.client.1.smithi153.stdout:4/970: dread d3/d9/db/d45/df3/fd3 [0,4194304] 0 2024-04-16T02:35:46.401 INFO:tasks.workunit.client.1.smithi153.stdout:4/971: chown d3/d9/db/d17/d28/dc8/de4/dfd/de3 2 1 2024-04-16T02:35:46.402 INFO:tasks.workunit.client.1.smithi153.stdout:4/972: write d3/d9/db/d17/d28/dc8/f137 [223285,109205] 0 2024-04-16T02:35:46.405 INFO:tasks.workunit.client.1.smithi153.stdout:4/973: creat d3/d9/d24/d75/dc5/f151 x:0 0 0 2024-04-16T02:35:46.406 INFO:tasks.workunit.client.1.smithi153.stdout:4/974: chown d3/da5/fc6 23 1 2024-04-16T02:35:46.408 INFO:tasks.workunit.client.1.smithi153.stdout:4/975: chown d3/da5/dd2/f12a 398 1 2024-04-16T02:35:46.408 INFO:tasks.workunit.client.0.smithi151.stdout:2/352: dwrite de/d11/d41/f37 [0,4194304] 0 2024-04-16T02:35:46.409 INFO:tasks.workunit.client.0.smithi151.stdout:2/353: readlink de/d15/d1e/l2f 0 2024-04-16T02:35:46.412 INFO:tasks.workunit.client.0.smithi151.stdout:2/354: mkdir de/d11/d42/d6b 0 2024-04-16T02:35:46.417 INFO:tasks.workunit.client.0.smithi151.stdout:2/355: mkdir de/d11/d42/d5e/d6c 0 2024-04-16T02:35:46.419 INFO:tasks.workunit.client.1.smithi153.stdout:2/860: dwrite d9/d71/d8d/fbd [0,4194304] 0 2024-04-16T02:35:46.420 INFO:tasks.workunit.client.0.smithi151.stdout:5/425: truncate d2/f1f 344503 0 2024-04-16T02:35:46.422 INFO:tasks.workunit.client.1.smithi153.stdout:0/991: write dd/d28/f41 [3524816,58059] 0 2024-04-16T02:35:46.423 INFO:tasks.workunit.client.0.smithi151.stdout:6/485: write d2/d5/df/d23/d82/d8d/d22/d40/f3b [3124613,57434] 0 2024-04-16T02:35:46.423 INFO:tasks.workunit.client.0.smithi151.stdout:5/426: chown d2/d6/d19/d30/d36/d42/d6f/d83 454 1 2024-04-16T02:35:46.427 INFO:tasks.workunit.client.1.smithi153.stdout:2/861: link d9/d2b/d18/d3f/l76 d9/d24/d34/d60/l117 0 2024-04-16T02:35:46.428 INFO:tasks.workunit.client.1.smithi153.stdout:0/992: symlink dd/d14/d51/l13c 0 2024-04-16T02:35:46.430 INFO:tasks.workunit.client.1.smithi153.stdout:2/862: write d9/d24/d27/d2c/d41/d7a/d116/fe1 [1761333,106765] 0 2024-04-16T02:35:46.431 INFO:tasks.workunit.client.0.smithi151.stdout:6/486: fdatasync d2/d5/df/d23/d82/d8d/d22/d40/d3f/f6c 0 2024-04-16T02:35:46.431 INFO:tasks.workunit.client.1.smithi153.stdout:0/993: chown dd/d14/d51/d7f/d9d/cbf 3068 1 2024-04-16T02:35:46.432 INFO:tasks.workunit.client.1.smithi153.stdout:4/976: dwrite d3/d9/f8d [0,4194304] 0 2024-04-16T02:35:46.434 INFO:tasks.workunit.client.1.smithi153.stdout:4/977: write d3/d9/db/d17/d3f/f109 [571326,14543] 0 2024-04-16T02:35:46.435 INFO:tasks.workunit.client.0.smithi151.stdout:6/487: creat d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/d97/fab x:0 0 0 2024-04-16T02:35:46.436 INFO:tasks.workunit.client.1.smithi153.stdout:0/994: symlink dd/d14/d19/d20/l13d 0 2024-04-16T02:35:46.438 INFO:tasks.workunit.client.1.smithi153.stdout:0/995: write dd/d32/d57/d6c/fbb [152423,130604] 0 2024-04-16T02:35:46.438 INFO:tasks.workunit.client.0.smithi151.stdout:6/488: readlink d2/d5/df/d23/d82/d8d/d22/d40/d3f/l90 0 2024-04-16T02:35:46.439 INFO:tasks.workunit.client.0.smithi151.stdout:6/489: chown d2/d5/df/d20/c21 108858924 1 2024-04-16T02:35:46.441 INFO:tasks.workunit.client.1.smithi153.stdout:2/863: getdents d9/d24/d27/d2c/d5d/da3/dc0 0 2024-04-16T02:35:46.443 INFO:tasks.workunit.client.1.smithi153.stdout:0/996: rename dd/d32/d57/f42 to dd/d14/d51/d7d/f13e 0 2024-04-16T02:35:46.444 INFO:tasks.workunit.client.1.smithi153.stdout:0/997: fdatasync dd/d14/d19/d20/f70 0 2024-04-16T02:35:46.444 INFO:tasks.workunit.client.1.smithi153.stdout:4/978: creat d3/d9/f152 x:0 0 0 2024-04-16T02:35:46.446 INFO:tasks.workunit.client.1.smithi153.stdout:0/998: write dd/d14/da6/da4/f134 [443876,110791] 0 2024-04-16T02:35:46.448 INFO:tasks.workunit.client.1.smithi153.stdout:2/864: rename d9/fba to d9/d24/d27/d2c/d41/d7a/d116/f118 0 2024-04-16T02:35:46.452 INFO:tasks.workunit.client.1.smithi153.stdout:4/979: dwrite d3/d9/d24/d3e/ff8 [0,4194304] 0 2024-04-16T02:35:46.452 INFO:tasks.workunit.client.0.smithi151.stdout:6/490: dwrite d2/d13/d46/f53 [0,4194304] 0 2024-04-16T02:35:46.454 INFO:tasks.workunit.client.1.smithi153.stdout:0/999: dread dd/d14/d19/fc4 [0,4194304] 0 2024-04-16T02:35:46.455 INFO:tasks.workunit.client.0.smithi151.stdout:5/427: dread d2/d6/f8 [0,4194304] 0 2024-04-16T02:35:46.456 INFO:tasks.workunit.client.1.smithi153.stdout:4/980: creat d3/d9/db/d17/d28/f153 x:0 0 0 2024-04-16T02:35:46.459 INFO:tasks.workunit.client.1.smithi153.stdout:2/865: dwrite d9/d24/d27/d2c/d41/d7a/d116/fe1 [0,4194304] 0 2024-04-16T02:35:46.462 INFO:tasks.workunit.client.0.smithi151.stdout:5/428: dread d2/d6/f8 [0,4194304] 0 2024-04-16T02:35:46.466 INFO:tasks.workunit.client.0.smithi151.stdout:5/429: creat d2/d6/d19/d30/d36/d63/f89 x:0 0 0 2024-04-16T02:35:46.470 INFO:tasks.workunit.client.0.smithi151.stdout:6/491: write d2/d5/f8f [647010,127622] 0 2024-04-16T02:35:46.472 INFO:tasks.workunit.client.0.smithi151.stdout:2/356: dread de/d11/f18 [4194304,4194304] 0 2024-04-16T02:35:46.473 INFO:tasks.workunit.client.0.smithi151.stdout:5/430: creat d2/d6/d19/d30/d49/d7f/f8a x:0 0 0 2024-04-16T02:35:46.474 INFO:tasks.workunit.client.0.smithi151.stdout:2/357: fsync de/d15/f46 0 2024-04-16T02:35:46.475 INFO:tasks.workunit.client.0.smithi151.stdout:2/358: write de/d11/f18 [769375,24519] 0 2024-04-16T02:35:46.477 INFO:tasks.workunit.client.0.smithi151.stdout:6/492: dread d2/d5/df/d23/d82/d8d/d22/d40/d3f/f66 [0,4194304] 0 2024-04-16T02:35:46.478 INFO:tasks.workunit.client.0.smithi151.stdout:2/359: write de/d11/f2b [4149848,39875] 0 2024-04-16T02:35:46.478 INFO:tasks.workunit.client.0.smithi151.stdout:5/431: dread d2/f16 [0,4194304] 0 2024-04-16T02:35:46.480 INFO:tasks.workunit.client.0.smithi151.stdout:2/360: unlink de/f5c 0 2024-04-16T02:35:46.481 INFO:tasks.workunit.client.0.smithi151.stdout:6/493: chown d2/d5/df/d23/d82/d8d/d22/d6d/ca4 13 1 2024-04-16T02:35:46.484 INFO:tasks.workunit.client.0.smithi151.stdout:6/494: creat d2/d5/df/d23/d56/d72/fac x:0 0 0 2024-04-16T02:35:46.485 INFO:tasks.workunit.client.1.smithi153.stdout:4/981: write d3/d9/db/de/d2c/d38/f9d [996913,77891] 0 2024-04-16T02:35:46.486 INFO:tasks.workunit.client.1.smithi153.stdout:1/859: sync 2024-04-16T02:35:46.486 INFO:tasks.workunit.client.1.smithi153.stdout:8/818: sync 2024-04-16T02:35:46.486 INFO:tasks.workunit.client.0.smithi151.stdout:6/495: dread d2/d13/f67 [0,4194304] 0 2024-04-16T02:35:46.488 INFO:tasks.workunit.client.1.smithi153.stdout:4/982: creat d3/d9/db/d45/df3/f154 x:0 0 0 2024-04-16T02:35:46.490 INFO:tasks.workunit.client.1.smithi153.stdout:2/866: dwrite d9/fb [0,4194304] 0 2024-04-16T02:35:46.492 INFO:tasks.workunit.client.1.smithi153.stdout:2/867: chown d9/d24/d27/d2c/d5d/da3/fd3 60 1 2024-04-16T02:35:46.492 INFO:tasks.workunit.client.1.smithi153.stdout:4/983: rmdir d3/d9/db/d17/d3f 39 2024-04-16T02:35:46.494 INFO:tasks.workunit.client.1.smithi153.stdout:1/860: mknod d4/dd/d13/d7e/de6/d111/d123/c126 0 2024-04-16T02:35:46.494 INFO:tasks.workunit.client.1.smithi153.stdout:2/868: fdatasync d9/d24/d27/d2c/d41/f56 0 2024-04-16T02:35:46.496 INFO:tasks.workunit.client.1.smithi153.stdout:4/984: write d3/d9/db/d17/d28/dc8/de4/dfd/d120/f135 [2962116,57904] 0 2024-04-16T02:35:46.499 INFO:tasks.workunit.client.1.smithi153.stdout:2/869: mknod d9/d24/d34/c119 0 2024-04-16T02:35:46.499 INFO:tasks.workunit.client.1.smithi153.stdout:4/985: write d3/d9/db/d45/f7f [2051586,31727] 0 2024-04-16T02:35:46.501 INFO:tasks.workunit.client.0.smithi151.stdout:2/361: dread de/d15/d1e/f24 [0,4194304] 0 2024-04-16T02:35:46.501 INFO:tasks.workunit.client.1.smithi153.stdout:2/870: write d9/d2b/d1c/fe2 [758712,102529] 0 2024-04-16T02:35:46.501 INFO:tasks.workunit.client.1.smithi153.stdout:8/819: dwrite d2/d93/d97/d9c/f2e [0,4194304] 0 2024-04-16T02:35:46.501 INFO:tasks.workunit.client.1.smithi153.stdout:4/986: chown d3/d9/db/de/d2c/fc9 46268 1 2024-04-16T02:35:46.504 INFO:tasks.workunit.client.1.smithi153.stdout:9/904: sync 2024-04-16T02:35:46.504 INFO:tasks.workunit.client.1.smithi153.stdout:3/856: sync 2024-04-16T02:35:46.504 INFO:tasks.workunit.client.1.smithi153.stdout:7/836: sync 2024-04-16T02:35:46.504 INFO:tasks.workunit.client.1.smithi153.stdout:6/832: sync 2024-04-16T02:35:46.504 INFO:tasks.workunit.client.1.smithi153.stdout:5/904: sync 2024-04-16T02:35:46.508 INFO:tasks.workunit.client.1.smithi153.stdout:9/905: write d4/f48 [13059452,8695] 0 2024-04-16T02:35:46.509 INFO:tasks.workunit.client.1.smithi153.stdout:5/905: rmdir d8/df/d2f/d4e 39 2024-04-16T02:35:46.513 INFO:tasks.workunit.client.1.smithi153.stdout:7/837: creat d0/d4/db/d2b/d53/f120 x:0 0 0 2024-04-16T02:35:46.516 INFO:tasks.workunit.client.0.smithi151.stdout:4/336: sync 2024-04-16T02:35:46.516 INFO:tasks.workunit.client.0.smithi151.stdout:3/230: sync 2024-04-16T02:35:46.519 INFO:tasks.workunit.client.1.smithi153.stdout:2/871: dwrite d9/d24/d34/ff7 [0,4194304] 0 2024-04-16T02:35:46.520 INFO:tasks.workunit.client.1.smithi153.stdout:9/906: truncate d4/d6/d1b/f90 1322711 0 2024-04-16T02:35:46.521 INFO:tasks.workunit.client.1.smithi153.stdout:6/833: creat dc/d47/d57/d77/dc4/f117 x:0 0 0 2024-04-16T02:35:46.522 INFO:tasks.workunit.client.1.smithi153.stdout:4/987: dwrite d3/d9/f1a [4194304,4194304] 0 2024-04-16T02:35:46.522 INFO:tasks.workunit.client.1.smithi153.stdout:8/820: dwrite d2/de/d3f/f10c [0,4194304] 0 2024-04-16T02:35:46.523 INFO:tasks.workunit.client.1.smithi153.stdout:5/906: rename d8/df/c1b to d8/df/d23/c11e 0 2024-04-16T02:35:46.523 INFO:tasks.workunit.client.1.smithi153.stdout:7/838: stat d0/de8/d110/d4d/fce 0 2024-04-16T02:35:46.526 INFO:tasks.workunit.client.1.smithi153.stdout:3/857: dwrite d6/d10/d22/fb1 [0,4194304] 0 2024-04-16T02:35:46.527 INFO:tasks.workunit.client.1.smithi153.stdout:1/861: dread d4/dd/dfd/fa7 [4194304,4194304] 0 2024-04-16T02:35:46.530 INFO:tasks.workunit.client.1.smithi153.stdout:2/872: creat d9/d2b/d1c/d20/d45/f11a x:0 0 0 2024-04-16T02:35:46.536 INFO:tasks.workunit.client.1.smithi153.stdout:4/988: unlink d3/f1c 0 2024-04-16T02:35:46.537 INFO:tasks.workunit.client.1.smithi153.stdout:4/989: dread - d3/d9/d24/d3e/d103/f145 zero size 2024-04-16T02:35:46.542 INFO:tasks.workunit.client.1.smithi153.stdout:6/834: mknod dc/d1a/c118 0 2024-04-16T02:35:46.542 INFO:tasks.workunit.client.0.smithi151.stdout:4/337: dwrite d0/d20/f21 [0,4194304] 0 2024-04-16T02:35:46.544 INFO:tasks.workunit.client.1.smithi153.stdout:6/835: fsync dc/d47/d57/d77/d15/dac/dae/de7/f111 0 2024-04-16T02:35:46.545 INFO:tasks.workunit.client.1.smithi153.stdout:6/836: write dc/d1a/d2d/fc7 [2220964,45413] 0 2024-04-16T02:35:46.545 INFO:tasks.workunit.client.1.smithi153.stdout:5/907: unlink d8/df/d2f/f10a 0 2024-04-16T02:35:46.547 INFO:tasks.workunit.client.1.smithi153.stdout:9/907: symlink d4/d6/dc/d2d/d79/dbb/l11d 0 2024-04-16T02:35:46.551 INFO:tasks.workunit.client.1.smithi153.stdout:2/873: mkdir d9/d24/d34/d36/db1/d11b 0 2024-04-16T02:35:46.552 INFO:tasks.workunit.client.1.smithi153.stdout:4/990: rmdir d3/da5 39 2024-04-16T02:35:46.554 INFO:tasks.workunit.client.1.smithi153.stdout:7/839: symlink d0/d4/l121 0 2024-04-16T02:35:46.557 INFO:tasks.workunit.client.1.smithi153.stdout:9/908: fdatasync d4/d6/d1b/d6e/f100 0 2024-04-16T02:35:46.557 INFO:tasks.workunit.client.1.smithi153.stdout:1/862: symlink d4/dd/d13/d23/l127 0 2024-04-16T02:35:46.559 INFO:tasks.workunit.client.1.smithi153.stdout:6/837: dwrite dc/f115 [0,4194304] 0 2024-04-16T02:35:46.561 INFO:tasks.workunit.client.1.smithi153.stdout:2/874: mknod d9/d24/d34/d36/db1/d11b/c11c 0 2024-04-16T02:35:46.562 INFO:tasks.workunit.client.1.smithi153.stdout:7/840: creat d0/d4/db/d2b/d43/d6d/d68/d6c/db8/f122 x:0 0 0 2024-04-16T02:35:46.567 INFO:tasks.workunit.client.1.smithi153.stdout:4/991: getdents d3/d9/db/de/d2c/d38 0 2024-04-16T02:35:46.570 INFO:tasks.workunit.client.1.smithi153.stdout:5/908: dwrite d8/f6c [0,4194304] 0 2024-04-16T02:35:46.570 INFO:tasks.workunit.client.1.smithi153.stdout:7/841: fsync d0/d4/db/d2b/d8c/d9f/fd8 0 2024-04-16T02:35:46.576 INFO:tasks.workunit.client.1.smithi153.stdout:5/909: write d8/df/f110 [748819,113077] 0 2024-04-16T02:35:46.576 INFO:tasks.workunit.client.1.smithi153.stdout:9/909: dwrite d4/d33/fd1 [0,4194304] 0 2024-04-16T02:35:46.576 INFO:tasks.workunit.client.0.smithi151.stdout:6/496: stat d2/d5/f8 0 2024-04-16T02:35:46.581 INFO:tasks.workunit.client.1.smithi153.stdout:6/838: dwrite dc/d47/d57/d77/dc4/ff7 [0,4194304] 0 2024-04-16T02:35:46.582 INFO:tasks.workunit.client.1.smithi153.stdout:7/842: mknod d0/d4/db/d2b/d43/d6d/d68/d6c/c123 0 2024-04-16T02:35:46.588 INFO:tasks.workunit.client.1.smithi153.stdout:2/875: link d9/d24/d34/d99/lc4 d9/d2b/d5c/l11d 0 2024-04-16T02:35:46.590 INFO:tasks.workunit.client.1.smithi153.stdout:2/876: write d9/d24/d34/f7d [1530841,94359] 0 2024-04-16T02:35:46.593 INFO:tasks.workunit.client.1.smithi153.stdout:5/910: mkdir d8/d41/dbd/dd3/d11f 0 2024-04-16T02:35:46.594 INFO:tasks.workunit.client.0.smithi151.stdout:4/338: rmdir d0/d11/d19/d46 39 2024-04-16T02:35:46.596 INFO:tasks.workunit.client.0.smithi151.stdout:6/497: stat d2/l1d 0 2024-04-16T02:35:46.597 INFO:tasks.workunit.client.0.smithi151.stdout:2/362: mkdir de/d11/d42/d5e/d6c/d6d 0 2024-04-16T02:35:46.598 INFO:tasks.workunit.client.0.smithi151.stdout:3/231: mkdir d1/d6/d4a/d4e 0 2024-04-16T02:35:46.598 INFO:tasks.workunit.client.1.smithi153.stdout:7/843: readlink d0/d4/db/d2b/d43/d6d/d68/lbe 0 2024-04-16T02:35:46.599 INFO:tasks.workunit.client.1.smithi153.stdout:3/858: write d6/d10/d3b/d10d/d8a/f91 [1211551,43642] 0 2024-04-16T02:35:46.599 INFO:tasks.workunit.client.1.smithi153.stdout:8/821: dwrite d2/d93/d97/d9c/d27/f2a [0,4194304] 0 2024-04-16T02:35:46.599 INFO:tasks.workunit.client.1.smithi153.stdout:4/992: link d3/d9/d24/d75/d9e/l141 d3/d9/d24/l155 0 2024-04-16T02:35:46.601 INFO:tasks.workunit.client.1.smithi153.stdout:3/859: write d6/fe4 [930098,5807] 0 2024-04-16T02:35:46.601 INFO:tasks.workunit.client.0.smithi151.stdout:4/339: rename d0/f10 to d0/d64/f71 0 2024-04-16T02:35:46.602 INFO:tasks.workunit.client.0.smithi151.stdout:6/498: symlink d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/lad 0 2024-04-16T02:35:46.602 INFO:tasks.workunit.client.0.smithi151.stdout:6/499: readlink d2/d5/l1e 0 2024-04-16T02:35:46.604 INFO:tasks.workunit.client.0.smithi151.stdout:3/232: chown d1/d6/l1f 4 1 2024-04-16T02:35:46.604 INFO:tasks.workunit.client.1.smithi153.stdout:2/877: mkdir d9/d71/d8d/d11e 0 2024-04-16T02:35:46.605 INFO:tasks.workunit.client.0.smithi151.stdout:4/340: creat d0/d17/f72 x:0 0 0 2024-04-16T02:35:46.606 INFO:tasks.workunit.client.0.smithi151.stdout:4/341: stat d0/d11/d19/d31/d34/d3a/d6e 0 2024-04-16T02:35:46.607 INFO:tasks.workunit.client.0.smithi151.stdout:4/342: stat d0/d11/c62 0 2024-04-16T02:35:46.607 INFO:tasks.workunit.client.0.smithi151.stdout:2/363: creat de/d11/d42/d6b/f6e x:0 0 0 2024-04-16T02:35:46.608 INFO:tasks.workunit.client.0.smithi151.stdout:6/500: fsync d2/d5/df/d23/d82/d8d/d22/d40/d3f/f7f 0 2024-04-16T02:35:46.609 INFO:tasks.workunit.client.1.smithi153.stdout:9/910: fdatasync d4/d6/d1b/d2a/d99/fce 0 2024-04-16T02:35:46.610 INFO:tasks.workunit.client.0.smithi151.stdout:3/233: creat d1/d6/d22/f4f x:0 0 0 2024-04-16T02:35:46.610 INFO:tasks.workunit.client.0.smithi151.stdout:6/501: write d2/d5/df/d23/f8c [846761,18739] 0 2024-04-16T02:35:46.612 INFO:tasks.workunit.client.1.smithi153.stdout:4/993: dwrite d3/d9/db/de/d2c/fc9 [0,4194304] 0 2024-04-16T02:35:46.613 INFO:tasks.workunit.client.0.smithi151.stdout:3/234: write d1/d6/d22/d23/d30/f45 [977598,48517] 0 2024-04-16T02:35:46.614 INFO:tasks.workunit.client.1.smithi153.stdout:3/860: dwrite d6/d10/d3b/d10d/d8a/d8c/f114 [0,4194304] 0 2024-04-16T02:35:46.615 INFO:tasks.workunit.client.1.smithi153.stdout:8/822: truncate d2/d13/f9f 1066018 0 2024-04-16T02:35:46.616 INFO:tasks.workunit.client.1.smithi153.stdout:1/863: write d4/dd/d18/d29/d6e/fb2 [678301,106108] 0 2024-04-16T02:35:46.617 INFO:tasks.workunit.client.0.smithi151.stdout:4/343: mkdir d0/d73 0 2024-04-16T02:35:46.618 INFO:tasks.workunit.client.1.smithi153.stdout:2/878: mkdir d9/d24/d27/d2c/d5d/da3/dc0/d11f 0 2024-04-16T02:35:46.619 INFO:tasks.workunit.client.0.smithi151.stdout:4/344: chown d0/fb 232477441 1 2024-04-16T02:35:46.619 INFO:tasks.workunit.client.0.smithi151.stdout:4/345: chown d0/d11/f3d 778408040 1 2024-04-16T02:35:46.620 INFO:tasks.workunit.client.1.smithi153.stdout:4/994: write d3/d9/db/d17/d28/dc8/de4/dfd/de3/f11c [945755,92658] 0 2024-04-16T02:35:46.621 INFO:tasks.workunit.client.0.smithi151.stdout:4/346: chown d0/d17/f52 1 1 2024-04-16T02:35:46.623 INFO:tasks.workunit.client.0.smithi151.stdout:3/235: creat d1/d6/f50 x:0 0 0 2024-04-16T02:35:46.624 INFO:tasks.workunit.client.0.smithi151.stdout:3/236: dread - d1/f3e zero size 2024-04-16T02:35:46.625 INFO:tasks.workunit.client.1.smithi153.stdout:9/911: mkdir d4/d6/dc/d24/d60/d11e 0 2024-04-16T02:35:46.625 INFO:tasks.workunit.client.0.smithi151.stdout:3/237: dread - d1/d6/f31 zero size 2024-04-16T02:35:46.626 INFO:tasks.workunit.client.1.smithi153.stdout:2/879: dwrite d9/d24/d27/d9a/fe8 [0,4194304] 0 2024-04-16T02:35:46.629 INFO:tasks.workunit.client.0.smithi151.stdout:3/238: write d1/d6/d20/d2a/f1a [1733481,47695] 0 2024-04-16T02:35:46.631 INFO:tasks.workunit.client.1.smithi153.stdout:5/911: write d8/df/d2e/f3d [2043920,91061] 0 2024-04-16T02:35:46.632 INFO:tasks.workunit.client.1.smithi153.stdout:5/912: readlink d8/df/d28/d5a/l103 0 2024-04-16T02:35:46.632 INFO:tasks.workunit.client.0.smithi151.stdout:4/347: fsync d0/f16 0 2024-04-16T02:35:46.634 INFO:tasks.workunit.client.1.smithi153.stdout:6/839: write dc/d10/d62/fce [1088155,72526] 0 2024-04-16T02:35:46.638 INFO:tasks.workunit.client.1.smithi153.stdout:4/995: creat d3/d9/db/d17/d28/d4c/f156 x:0 0 0 2024-04-16T02:35:46.641 INFO:tasks.workunit.client.0.smithi151.stdout:6/502: dread d2/d5/df/d20/d4f/f6e [0,4194304] 0 2024-04-16T02:35:46.642 INFO:tasks.workunit.client.1.smithi153.stdout:5/913: dwrite d8/d1f/f107 [0,4194304] 0 2024-04-16T02:35:46.642 INFO:tasks.workunit.client.0.smithi151.stdout:4/348: link d0/d11/d19/d31/d34/f3f d0/d11/d19/d31/d34/d3a/f74 0 2024-04-16T02:35:46.643 INFO:tasks.workunit.client.1.smithi153.stdout:8/823: dwrite d2/d93/d97/d9c/f3c [0,4194304] 0 2024-04-16T02:35:46.647 INFO:tasks.workunit.client.1.smithi153.stdout:9/912: mknod d4/d33/dae/d103/c11f 0 2024-04-16T02:35:46.647 INFO:tasks.workunit.client.1.smithi153.stdout:4/996: chown d3/l1d 5 1 2024-04-16T02:35:46.648 INFO:tasks.workunit.client.1.smithi153.stdout:4/997: chown d3/dbc/ld8 116 1 2024-04-16T02:35:46.648 INFO:tasks.workunit.client.0.smithi151.stdout:6/503: dread d2/d5/df/d23/f8c [0,4194304] 0 2024-04-16T02:35:46.652 INFO:tasks.workunit.client.1.smithi153.stdout:9/913: chown d4/d6/d1b/d2a/d99/d11c 1964 1 2024-04-16T02:35:46.654 INFO:tasks.workunit.client.0.smithi151.stdout:2/364: dwrite de/f3a [0,4194304] 0 2024-04-16T02:35:46.655 INFO:tasks.workunit.client.1.smithi153.stdout:4/998: symlink d3/d9/d24/d3e/de1/l157 0 2024-04-16T02:35:46.656 INFO:tasks.workunit.client.1.smithi153.stdout:6/840: dread dc/d1a/d2d/f44 [0,4194304] 0 2024-04-16T02:35:46.656 INFO:tasks.workunit.client.0.smithi151.stdout:6/504: creat d2/d5/df/d23/d82/d8d/fae x:0 0 0 2024-04-16T02:35:46.656 INFO:tasks.workunit.client.0.smithi151.stdout:4/349: dwrite d0/f4c [4194304,4194304] 0 2024-04-16T02:35:46.658 INFO:tasks.workunit.client.1.smithi153.stdout:5/914: dwrite d8/d41/dbd/f40 [0,4194304] 0 2024-04-16T02:35:46.660 INFO:tasks.workunit.client.1.smithi153.stdout:6/841: mknod dc/d47/d57/d77/d7a/c119 0 2024-04-16T02:35:46.661 INFO:tasks.workunit.client.1.smithi153.stdout:5/915: readlink d8/df/d2e/dd1/lcf 0 2024-04-16T02:35:46.665 INFO:tasks.workunit.client.0.smithi151.stdout:6/505: mknod d2/d5/df/d23/d45/d62/caf 0 2024-04-16T02:35:46.667 INFO:tasks.workunit.client.0.smithi151.stdout:4/350: rename d0/d20/f49 to d0/d11/d19/d31/d34/d3a/f75 0 2024-04-16T02:35:46.669 INFO:tasks.workunit.client.1.smithi153.stdout:1/864: write d4/da2/f112 [392021,37976] 0 2024-04-16T02:35:46.671 INFO:tasks.workunit.client.0.smithi151.stdout:6/506: symlink d2/d13/d46/lb0 0 2024-04-16T02:35:46.676 INFO:tasks.workunit.client.0.smithi151.stdout:4/351: creat d0/d64/f76 x:0 0 0 2024-04-16T02:35:46.677 INFO:tasks.workunit.client.0.smithi151.stdout:6/507: dread d2/d13/d46/f53 [0,4194304] 0 2024-04-16T02:35:46.679 INFO:tasks.workunit.client.1.smithi153.stdout:7/844: dwrite d0/d4/db/d2b/d8c/f9b [0,4194304] 0 2024-04-16T02:35:46.679 INFO:tasks.workunit.client.1.smithi153.stdout:6/842: dwrite dc/d47/d57/d77/dc4/dc9/ff8 [0,4194304] 0 2024-04-16T02:35:46.679 INFO:tasks.workunit.client.0.smithi151.stdout:6/508: chown d2/d5/df/d20/d4f/f6e 7498184 1 2024-04-16T02:35:46.683 INFO:tasks.workunit.client.1.smithi153.stdout:3/861: dread d6/d10/dec/ded/fee [0,4194304] 0 2024-04-16T02:35:46.688 INFO:tasks.workunit.client.0.smithi151.stdout:6/509: mkdir d2/d5/df/d23/d82/d8d/d22/d40/d3f/db1 0 2024-04-16T02:35:46.688 INFO:tasks.workunit.client.0.smithi151.stdout:4/352: dwrite d0/d11/f3d [0,4194304] 0 2024-04-16T02:35:46.689 INFO:tasks.workunit.client.1.smithi153.stdout:1/865: link d4/dd/d18/c3a d4/da2/d65/c128 0 2024-04-16T02:35:46.696 INFO:tasks.workunit.client.0.smithi151.stdout:4/353: creat d0/d11/d19/d46/f77 x:0 0 0 2024-04-16T02:35:46.696 INFO:tasks.workunit.client.0.smithi151.stdout:4/354: chown d0/c5b 22207102 1 2024-04-16T02:35:46.697 INFO:tasks.workunit.client.1.smithi153.stdout:6/843: getdents dc/d10/d62 0 2024-04-16T02:35:46.697 INFO:tasks.workunit.client.0.smithi151.stdout:6/510: creat d2/d5/df/d20/d4f/d81/fb2 x:0 0 0 2024-04-16T02:35:46.701 INFO:tasks.workunit.client.1.smithi153.stdout:4/999: write d3/d9/db/d17/f8c [1951989,42771] 0 2024-04-16T02:35:46.702 INFO:tasks.workunit.client.0.smithi151.stdout:4/355: mknod d0/d11/d19/d31/d4a/c78 0 2024-04-16T02:35:46.704 INFO:tasks.workunit.client.0.smithi151.stdout:2/365: dread de/d11/d41/f32 [0,4194304] 0 2024-04-16T02:35:46.707 INFO:tasks.workunit.client.0.smithi151.stdout:6/511: getdents d2/d5/df/d23/d82/d8d/d22/d40/d3f/d85 0 2024-04-16T02:35:46.710 INFO:tasks.workunit.client.1.smithi153.stdout:1/866: creat d4/dd/d18/d29/f129 x:0 0 0 2024-04-16T02:35:46.711 INFO:tasks.workunit.client.0.smithi151.stdout:2/366: creat de/d65/f6f x:0 0 0 2024-04-16T02:35:46.711 INFO:tasks.workunit.client.1.smithi153.stdout:5/916: dread d8/df/d28/f82 [0,4194304] 0 2024-04-16T02:35:46.711 INFO:tasks.workunit.client.1.smithi153.stdout:3/862: link d6/d10/c16 d6/c117 0 2024-04-16T02:35:46.711 INFO:tasks.workunit.client.1.smithi153.stdout:6/844: write dc/d47/d93/fd1 [857974,29907] 0 2024-04-16T02:35:46.712 INFO:tasks.workunit.client.0.smithi151.stdout:6/512: creat d2/d13/d3a/da6/fb3 x:0 0 0 2024-04-16T02:35:46.713 INFO:tasks.workunit.client.1.smithi153.stdout:7/845: truncate d0/d4/db/d2b/d43/f9d 4642776 0 2024-04-16T02:35:46.714 INFO:tasks.workunit.client.0.smithi151.stdout:2/367: mknod de/d11/d42/d5e/d6c/c70 0 2024-04-16T02:35:46.714 INFO:tasks.workunit.client.1.smithi153.stdout:8/824: dwrite d2/d13/d96/dc7/d109/d47/d60/d67/fbf [0,4194304] 0 2024-04-16T02:35:46.715 INFO:tasks.workunit.client.0.smithi151.stdout:4/356: dwrite d0/d11/d19/f1b [0,4194304] 0 2024-04-16T02:35:46.715 INFO:tasks.workunit.client.1.smithi153.stdout:6/845: fsync dc/d47/d57/d77/dc4/dc9/ff8 0 2024-04-16T02:35:46.716 INFO:tasks.workunit.client.1.smithi153.stdout:5/917: write d8/df/d28/d5a/f117 [250540,73055] 0 2024-04-16T02:35:46.720 INFO:tasks.workunit.client.1.smithi153.stdout:9/914: dread d4/d6/dc/d2d/f53 [0,4194304] 0 2024-04-16T02:35:46.720 INFO:tasks.workunit.client.1.smithi153.stdout:5/918: read - d8/df/d28/d5a/d72/dc0/f11c zero size 2024-04-16T02:35:46.720 INFO:tasks.workunit.client.1.smithi153.stdout:6/846: fdatasync dc/d47/d57/d77/d15/dac/dae/de7/f108 0 2024-04-16T02:35:46.727 INFO:tasks.workunit.client.0.smithi151.stdout:4/357: rmdir d0/d11/d19/d31/d34/d3a/d6e 0 2024-04-16T02:35:46.728 INFO:tasks.workunit.client.0.smithi151.stdout:6/513: rmdir d2/d13/d3a 39 2024-04-16T02:35:46.730 INFO:tasks.workunit.client.0.smithi151.stdout:6/514: write d2/d5/df/d23/d82/d8d/fae [742878,97355] 0 2024-04-16T02:35:46.731 INFO:tasks.workunit.client.0.smithi151.stdout:4/358: creat d0/d11/d19/d31/d68/f79 x:0 0 0 2024-04-16T02:35:46.735 INFO:tasks.workunit.client.0.smithi151.stdout:2/368: dwrite de/d15/d1e/f24 [0,4194304] 0 2024-04-16T02:35:46.737 INFO:tasks.workunit.client.1.smithi153.stdout:2/880: sync 2024-04-16T02:35:46.738 INFO:tasks.workunit.client.0.smithi151.stdout:4/359: creat d0/d11/d19/d31/d68/f7a x:0 0 0 2024-04-16T02:35:46.738 INFO:tasks.workunit.client.0.smithi151.stdout:2/369: mkdir de/d65/d71 0 2024-04-16T02:35:46.740 INFO:tasks.workunit.client.0.smithi151.stdout:2/370: write de/d11/d41/f37 [3958137,34534] 0 2024-04-16T02:35:46.743 INFO:tasks.workunit.client.0.smithi151.stdout:4/360: mkdir d0/d11/d19/d31/d34/d7b 0 2024-04-16T02:35:46.744 INFO:tasks.workunit.client.0.smithi151.stdout:0/274: sync 2024-04-16T02:35:46.745 INFO:tasks.workunit.client.0.smithi151.stdout:4/361: dread - d0/d11/d19/d31/d68/f79 zero size 2024-04-16T02:35:46.745 INFO:tasks.workunit.client.0.smithi151.stdout:8/268: sync 2024-04-16T02:35:46.745 INFO:tasks.workunit.client.0.smithi151.stdout:7/324: sync 2024-04-16T02:35:46.745 INFO:tasks.workunit.client.0.smithi151.stdout:1/342: sync 2024-04-16T02:35:46.745 INFO:tasks.workunit.client.0.smithi151.stdout:9/260: sync 2024-04-16T02:35:46.745 INFO:tasks.workunit.client.1.smithi153.stdout:2/881: dwrite d9/d24/d27/f68 [4194304,4194304] 0 2024-04-16T02:35:46.747 INFO:tasks.workunit.client.0.smithi151.stdout:2/371: read de/d11/f2b [9051,100788] 0 2024-04-16T02:35:46.751 INFO:tasks.workunit.client.1.smithi153.stdout:3/863: stat d6/d10/dec/ded/lfc 0 2024-04-16T02:35:46.751 INFO:tasks.workunit.client.1.smithi153.stdout:7/846: chown d0/l54 474218876 1 2024-04-16T02:35:46.754 INFO:tasks.workunit.client.0.smithi151.stdout:8/269: mkdir d11/d2a/d35/d55 0 2024-04-16T02:35:46.754 INFO:tasks.workunit.client.1.smithi153.stdout:5/919: write d8/df/d2f/d4e/f94 [4909154,37264] 0 2024-04-16T02:35:46.755 INFO:tasks.workunit.client.1.smithi153.stdout:6/847: read - dc/d47/d57/d77/d15/dac/fb9 zero size 2024-04-16T02:35:46.757 INFO:tasks.workunit.client.1.smithi153.stdout:2/882: truncate f6 2338788 0 2024-04-16T02:35:46.760 INFO:tasks.workunit.client.0.smithi151.stdout:4/362: symlink d0/d11/d19/d31/d68/l7c 0 2024-04-16T02:35:46.760 INFO:tasks.workunit.client.0.smithi151.stdout:7/325: creat d0/d1f/d31/d42/f6e x:0 0 0 2024-04-16T02:35:46.761 INFO:tasks.workunit.client.0.smithi151.stdout:0/275: rmdir dc/d20/d2c/d4b 0 2024-04-16T02:35:46.762 INFO:tasks.workunit.client.1.smithi153.stdout:3/864: creat d6/d10/d22/d4d/da0/f118 x:0 0 0 2024-04-16T02:35:46.762 INFO:tasks.workunit.client.1.smithi153.stdout:7/847: symlink d0/d4/db/d2b/d8c/d9f/l124 0 2024-04-16T02:35:46.763 INFO:tasks.workunit.client.0.smithi151.stdout:2/372: creat de/d65/f72 x:0 0 0 2024-04-16T02:35:46.763 INFO:tasks.workunit.client.1.smithi153.stdout:7/848: write d0/d4/db/d34/d51/fda [2314211,51629] 0 2024-04-16T02:35:46.768 INFO:tasks.workunit.client.1.smithi153.stdout:5/920: write d8/df/f1e [5072705,116185] 0 2024-04-16T02:35:46.768 INFO:tasks.workunit.client.0.smithi151.stdout:6/515: dwrite d2/f1a [0,4194304] 0 2024-04-16T02:35:46.768 INFO:tasks.workunit.client.0.smithi151.stdout:7/326: write d0/d1f/d31/d42/f6e [2715,3825] 0 2024-04-16T02:35:46.769 INFO:tasks.workunit.client.1.smithi153.stdout:5/921: write d8/df/d2e/f3d [406347,31069] 0 2024-04-16T02:35:46.771 INFO:tasks.workunit.client.0.smithi151.stdout:8/270: dwrite d11/f15 [0,4194304] 0 2024-04-16T02:35:46.771 INFO:tasks.workunit.client.0.smithi151.stdout:2/373: write de/d15/f46 [578626,117715] 0 2024-04-16T02:35:46.772 INFO:tasks.workunit.client.1.smithi153.stdout:7/849: dwrite d0/d4/db/d2b/d43/d6d/d68/f86 [0,4194304] 0 2024-04-16T02:35:46.772 INFO:tasks.workunit.client.0.smithi151.stdout:6/516: fsync d2/d5/df/d23/d82/d8d/d22/f4e 0 2024-04-16T02:35:46.778 INFO:tasks.workunit.client.1.smithi153.stdout:2/883: creat d9/d71/d8d/f120 x:0 0 0 2024-04-16T02:35:46.778 INFO:tasks.workunit.client.0.smithi151.stdout:0/276: creat dc/d15/d34/f57 x:0 0 0 2024-04-16T02:35:46.780 INFO:tasks.workunit.client.1.smithi153.stdout:2/884: fsync d9/d71/d8d/fd9 0 2024-04-16T02:35:46.781 INFO:tasks.workunit.client.1.smithi153.stdout:7/850: dwrite d0/d4/db/d47/f94 [0,4194304] 0 2024-04-16T02:35:46.786 INFO:tasks.workunit.client.0.smithi151.stdout:0/277: truncate dc/d15/d34/f54 65807 0 2024-04-16T02:35:46.786 INFO:tasks.workunit.client.1.smithi153.stdout:3/865: write d6/d10/d3b/d76/fc2 [4819292,84960] 0 2024-04-16T02:35:46.788 INFO:tasks.workunit.client.0.smithi151.stdout:0/278: chown dc/d20/f49 27429758 1 2024-04-16T02:35:46.788 INFO:tasks.workunit.client.0.smithi151.stdout:7/327: dwrite d0/d1f/f2a [0,4194304] 0 2024-04-16T02:35:46.789 INFO:tasks.workunit.client.1.smithi153.stdout:3/866: truncate d6/f97 820329 0 2024-04-16T02:35:46.792 INFO:tasks.workunit.client.0.smithi151.stdout:4/363: creat d0/d11/d19/d31/d34/d3a/d65/f7d x:0 0 0 2024-04-16T02:35:46.793 INFO:tasks.workunit.client.0.smithi151.stdout:8/271: mknod d11/d2a/d35/d55/c56 0 2024-04-16T02:35:46.795 INFO:tasks.workunit.client.0.smithi151.stdout:9/261: dread d1/d3/d6/f13 [0,4194304] 0 2024-04-16T02:35:46.795 INFO:tasks.workunit.client.0.smithi151.stdout:7/328: mknod d0/d1f/d33/c6f 0 2024-04-16T02:35:46.795 INFO:tasks.workunit.client.1.smithi153.stdout:8/825: link d2/d13/d3a/l76 d2/d13/d96/dc7/d109/d47/d60/d67/d98/l115 0 2024-04-16T02:35:46.797 INFO:tasks.workunit.client.0.smithi151.stdout:6/517: symlink d2/d13/d3a/lb4 0 2024-04-16T02:35:46.798 INFO:tasks.workunit.client.0.smithi151.stdout:8/272: symlink d11/d2a/d33/l57 0 2024-04-16T02:35:46.801 INFO:tasks.workunit.client.0.smithi151.stdout:1/343: dread d3/d4/d64/d1c/f2c [0,4194304] 0 2024-04-16T02:35:46.802 INFO:tasks.workunit.client.0.smithi151.stdout:7/329: creat d0/d2c/d38/d3e/f70 x:0 0 0 2024-04-16T02:35:46.804 INFO:tasks.workunit.client.0.smithi151.stdout:6/518: fsync d2/d5/df/d23/d82/d8d/d22/d40/f95 0 2024-04-16T02:35:46.805 INFO:tasks.workunit.client.0.smithi151.stdout:1/344: truncate d3/d4/d64/d1c/f44 1046966 0 2024-04-16T02:35:46.809 INFO:tasks.workunit.client.1.smithi153.stdout:7/851: symlink d0/d4/db/d2b/d53/d59/d76/d103/l125 0 2024-04-16T02:35:46.810 INFO:tasks.workunit.client.1.smithi153.stdout:1/867: truncate d4/da2/f112 872015 0 2024-04-16T02:35:46.814 INFO:tasks.workunit.client.1.smithi153.stdout:8/826: creat d2/d93/d97/d9c/d27/d5e/f116 x:0 0 0 2024-04-16T02:35:46.814 INFO:tasks.workunit.client.0.smithi151.stdout:1/345: creat d3/d33/d10/f78 x:0 0 0 2024-04-16T02:35:46.816 INFO:tasks.workunit.client.1.smithi153.stdout:6/848: write f4 [179975,12341] 0 2024-04-16T02:35:46.817 INFO:tasks.workunit.client.1.smithi153.stdout:7/852: dread d0/de8/d110/daa/fc9 [0,4194304] 0 2024-04-16T02:35:46.817 INFO:tasks.workunit.client.1.smithi153.stdout:9/915: link d4/d6/dc/d2d/d64/c9a d4/d6/dc/c120 0 2024-04-16T02:35:46.818 INFO:tasks.workunit.client.1.smithi153.stdout:9/916: chown d4/d33/l105 9001 1 2024-04-16T02:35:46.818 INFO:tasks.workunit.client.0.smithi151.stdout:4/364: getdents d0/d11/d19/d31/d4a 0 2024-04-16T02:35:46.819 INFO:tasks.workunit.client.0.smithi151.stdout:4/365: fsync d0/d20/f42 0 2024-04-16T02:35:46.820 INFO:tasks.workunit.client.0.smithi151.stdout:4/366: chown d0/d11/d19/f44 0 1 2024-04-16T02:35:46.821 INFO:tasks.workunit.client.0.smithi151.stdout:1/346: creat d3/d58/df/d32/d3a/f79 x:0 0 0 2024-04-16T02:35:46.822 INFO:tasks.workunit.client.1.smithi153.stdout:8/827: rename d2/d93/d97/d9c/d27/f6e to d2/dba/f117 0 2024-04-16T02:35:46.823 INFO:tasks.workunit.client.1.smithi153.stdout:8/828: write d2/d93/d97/d9c/fdc [701215,59876] 0 2024-04-16T02:35:46.826 INFO:tasks.workunit.client.0.smithi151.stdout:2/374: truncate de/d15/d1e/f3f 1982049 0 2024-04-16T02:35:46.828 INFO:tasks.workunit.client.0.smithi151.stdout:6/519: dread d2/d5/df/f48 [0,4194304] 0 2024-04-16T02:35:46.828 INFO:tasks.workunit.client.0.smithi151.stdout:2/375: write de/d65/f72 [754873,7005] 0 2024-04-16T02:35:46.828 INFO:tasks.workunit.client.1.smithi153.stdout:6/849: rmdir dc/d1a 39 2024-04-16T02:35:46.829 INFO:tasks.workunit.client.1.smithi153.stdout:6/850: chown dc/d47/d57/d77/dc4/dc9/ded 88524460 1 2024-04-16T02:35:46.830 INFO:tasks.workunit.client.0.smithi151.stdout:2/376: readlink de/d11/l62 0 2024-04-16T02:35:46.831 INFO:tasks.workunit.client.0.smithi151.stdout:4/367: creat d0/d11/d19/f7e x:0 0 0 2024-04-16T02:35:46.832 INFO:tasks.workunit.client.0.smithi151.stdout:6/520: write d2/d5/df/d23/d82/d8d/f29 [6222552,52591] 0 2024-04-16T02:35:46.833 INFO:tasks.workunit.client.1.smithi153.stdout:2/885: write d9/d2b/d18/d3f/f58 [586141,30627] 0 2024-04-16T02:35:46.833 INFO:tasks.workunit.client.0.smithi151.stdout:1/347: dwrite d3/d33/d10/f4e [4194304,4194304] 0 2024-04-16T02:35:46.835 INFO:tasks.workunit.client.1.smithi153.stdout:2/886: write d9/d2b/d5c/fcf [1461078,87472] 0 2024-04-16T02:35:46.836 INFO:tasks.workunit.client.1.smithi153.stdout:7/853: rename d0/d4/db/d34/d51 to d0/d4/db/d16/d126 0 2024-04-16T02:35:46.837 INFO:tasks.workunit.client.0.smithi151.stdout:9/262: truncate d1/d3/f41 116443 0 2024-04-16T02:35:46.837 INFO:tasks.workunit.client.1.smithi153.stdout:2/887: stat d9/d24/d27/d2c/d5d/de5/c108 0 2024-04-16T02:35:46.837 INFO:tasks.workunit.client.1.smithi153.stdout:7/854: dread - d0/d4/db/d2b/d8c/f119 zero size 2024-04-16T02:35:46.839 INFO:tasks.workunit.client.1.smithi153.stdout:8/829: dread - d2/d13/d96/dc7/d109/d47/fd3 zero size 2024-04-16T02:35:46.841 INFO:tasks.workunit.client.0.smithi151.stdout:4/368: write d0/d11/d19/d31/d34/d3a/d50/f51 [813623,125667] 0 2024-04-16T02:35:46.843 INFO:tasks.workunit.client.1.smithi153.stdout:6/851: rmdir dc/d47/d57 39 2024-04-16T02:35:46.845 INFO:tasks.workunit.client.0.smithi151.stdout:9/263: chown d1/d3/d6/f1b 0 1 2024-04-16T02:35:46.845 INFO:tasks.workunit.client.1.smithi153.stdout:5/922: truncate d8/df/d2e/f6e 3007441 0 2024-04-16T02:35:46.846 INFO:tasks.workunit.client.0.smithi151.stdout:9/264: write d1/fc [1637287,20405] 0 2024-04-16T02:35:46.847 INFO:tasks.workunit.client.0.smithi151.stdout:2/377: link de/d15/d1e/f67 de/d11/d41/d4b/f73 0 2024-04-16T02:35:46.847 INFO:tasks.workunit.client.0.smithi151.stdout:4/369: write d0/d64/f71 [7267723,72075] 0 2024-04-16T02:35:46.848 INFO:tasks.workunit.client.0.smithi151.stdout:8/273: dwrite d11/d22/f30 [4194304,4194304] 0 2024-04-16T02:35:46.848 INFO:tasks.workunit.client.0.smithi151.stdout:7/330: dread d0/f3 [8388608,4194304] 0 2024-04-16T02:35:46.849 INFO:tasks.workunit.client.1.smithi153.stdout:7/855: creat d0/d4/db/d2b/d53/d59/f127 x:0 0 0 2024-04-16T02:35:46.851 INFO:tasks.workunit.client.0.smithi151.stdout:7/331: write d0/d2c/f5f [314902,82017] 0 2024-04-16T02:35:46.852 INFO:tasks.workunit.client.0.smithi151.stdout:1/348: getdents d3/d58/df/d32 0 2024-04-16T02:35:46.852 INFO:tasks.workunit.client.1.smithi153.stdout:8/830: creat d2/d93/d97/d9c/d27/d5e/de4/df3/f118 x:0 0 0 2024-04-16T02:35:46.852 INFO:tasks.workunit.client.1.smithi153.stdout:3/867: dwrite d6/d10/d22/d2c/f33 [0,4194304] 0 2024-04-16T02:35:46.852 INFO:tasks.workunit.client.1.smithi153.stdout:8/831: readlink d2/l56 0 2024-04-16T02:35:46.854 INFO:tasks.workunit.client.0.smithi151.stdout:9/265: fsync d1/d3/d6/f13 0 2024-04-16T02:35:46.854 INFO:tasks.workunit.client.1.smithi153.stdout:3/868: chown d6/d10/d3b/d59/fa9 6184 1 2024-04-16T02:35:46.855 INFO:tasks.workunit.client.0.smithi151.stdout:2/378: creat de/d11/d41/d35/f74 x:0 0 0 2024-04-16T02:35:46.856 INFO:tasks.workunit.client.0.smithi151.stdout:7/332: symlink d0/d1f/d31/d42/l71 0 2024-04-16T02:35:46.857 INFO:tasks.workunit.client.0.smithi151.stdout:1/349: mkdir d3/d4/d64/d2a/d7a 0 2024-04-16T02:35:46.858 INFO:tasks.workunit.client.0.smithi151.stdout:1/350: fdatasync d3/d33/f36 0 2024-04-16T02:35:46.858 INFO:tasks.workunit.client.0.smithi151.stdout:4/370: creat d0/d20/d59/f7f x:0 0 0 2024-04-16T02:35:46.861 INFO:tasks.workunit.client.1.smithi153.stdout:8/832: read d2/ff [1289439,105770] 0 2024-04-16T02:35:46.864 INFO:tasks.workunit.client.1.smithi153.stdout:1/868: dread d4/dd/d13/d23/d5f/f7f [0,4194304] 0 2024-04-16T02:35:46.865 INFO:tasks.workunit.client.0.smithi151.stdout:9/266: dwrite d1/d3/d6/d17/f27 [0,4194304] 0 2024-04-16T02:35:46.868 INFO:tasks.workunit.client.1.smithi153.stdout:1/869: write d4/dd/d20/f55 [1338821,39747] 0 2024-04-16T02:35:46.871 INFO:tasks.workunit.client.1.smithi153.stdout:1/870: write d4/dd/d18/d29/d6e/fb2 [1031343,41154] 0 2024-04-16T02:35:46.874 INFO:tasks.workunit.client.1.smithi153.stdout:6/852: dwrite dc/d47/d57/d77/d15/dac/dae/de7/f108 [0,4194304] 0 2024-04-16T02:35:46.875 INFO:tasks.workunit.client.1.smithi153.stdout:1/871: read d4/dd/d18/d29/d71/f7b [39379,125764] 0 2024-04-16T02:35:46.878 INFO:tasks.workunit.client.1.smithi153.stdout:1/872: write d4/dd/d13/d8d/f121 [900592,48025] 0 2024-04-16T02:35:46.897 INFO:tasks.workunit.client.0.smithi151.stdout:8/274: dread d11/f4a [0,4194304] 0 2024-04-16T02:35:46.908 INFO:tasks.workunit.client.1.smithi153.stdout:1/873: dread d4/f7 [0,4194304] 0 2024-04-16T02:35:46.909 INFO:tasks.workunit.client.1.smithi153.stdout:1/874: stat d4/da2/d65/d84 0 2024-04-16T02:35:46.910 INFO:tasks.workunit.client.1.smithi153.stdout:1/875: write d4/dd/d13/d7e/fb1 [1111597,7922] 0 2024-04-16T02:35:46.912 INFO:tasks.workunit.client.0.smithi151.stdout:5/432: sync 2024-04-16T02:35:46.970 INFO:tasks.workunit.client.1.smithi153.stdout:9/917: link d4/d6/dc/d24/d4e/df4/c110 d4/d6/def/c121 0 2024-04-16T02:35:46.970 INFO:tasks.workunit.client.0.smithi151.stdout:7/333: mknod d0/d1f/d33/c72 0 2024-04-16T02:35:46.971 INFO:tasks.workunit.client.1.smithi153.stdout:3/869: creat d6/d10/d22/d4d/da0/f119 x:0 0 0 2024-04-16T02:35:46.973 INFO:tasks.workunit.client.0.smithi151.stdout:4/371: fsync d0/f16 0 2024-04-16T02:35:46.975 INFO:tasks.workunit.client.1.smithi153.stdout:8/833: symlink d2/d1c/l119 0 2024-04-16T02:35:46.980 INFO:tasks.workunit.client.1.smithi153.stdout:2/888: truncate d9/d24/d27/d2c/d41/fb5 125277 0 2024-04-16T02:35:46.981 INFO:tasks.workunit.client.0.smithi151.stdout:4/372: dwrite d0/d64/f76 [0,4194304] 0 2024-04-16T02:35:46.982 INFO:tasks.workunit.client.1.smithi153.stdout:6/853: creat dc/d47/d93/f11a x:0 0 0 2024-04-16T02:35:46.985 INFO:tasks.workunit.client.1.smithi153.stdout:1/876: mkdir d4/dd/d13/d7e/de6/d111/d12a 0 2024-04-16T02:35:46.986 INFO:tasks.workunit.client.1.smithi153.stdout:1/877: fsync d4/dd/d18/d29/fde 0 2024-04-16T02:35:46.989 INFO:tasks.workunit.client.1.smithi153.stdout:5/923: rmdir d8/d41/d90/d9d/dec 0 2024-04-16T02:35:46.992 INFO:tasks.workunit.client.1.smithi153.stdout:7/856: rename d0/d4/db/d2b/d43/d6d/d68 to d0/d4/db/d2b/d43/d128 0 2024-04-16T02:35:46.995 INFO:tasks.workunit.client.1.smithi153.stdout:7/857: dread - d0/de8/ffe zero size 2024-04-16T02:35:46.995 INFO:tasks.workunit.client.0.smithi151.stdout:1/351: creat d3/d58/df/d32/d3a/d6d/f7b x:0 0 0 2024-04-16T02:35:46.996 INFO:tasks.workunit.client.0.smithi151.stdout:2/379: truncate de/f40 6654932 0 2024-04-16T02:35:46.997 INFO:tasks.workunit.client.1.smithi153.stdout:1/878: dwrite d4/dd/d13/d26/ffe [0,4194304] 0 2024-04-16T02:35:47.003 INFO:tasks.workunit.client.0.smithi151.stdout:5/433: unlink d2/d6/d19/d30/d36/d42/f58 0 2024-04-16T02:35:47.004 INFO:tasks.workunit.client.1.smithi153.stdout:5/924: dwrite d8/d41/f10f [0,4194304] 0 2024-04-16T02:35:47.005 INFO:tasks.workunit.client.0.smithi151.stdout:5/434: dread d2/d6/f64 [0,4194304] 0 2024-04-16T02:35:47.006 INFO:tasks.workunit.client.0.smithi151.stdout:7/334: mkdir d0/d27/d73 0 2024-04-16T02:35:47.006 INFO:tasks.workunit.client.1.smithi153.stdout:2/889: unlink d9/d2b/d1c/d20/fc2 0 2024-04-16T02:35:47.007 INFO:tasks.workunit.client.0.smithi151.stdout:5/435: chown d2/d6/f29 751 1 2024-04-16T02:35:47.007 INFO:tasks.workunit.client.1.smithi153.stdout:6/854: mknod dc/d47/c11b 0 2024-04-16T02:35:47.008 INFO:tasks.workunit.client.0.smithi151.stdout:5/436: dread - d2/d6/d19/d30/f70 zero size 2024-04-16T02:35:47.008 INFO:tasks.workunit.client.1.smithi153.stdout:8/834: truncate d2/f7 944454 0 2024-04-16T02:35:47.009 INFO:tasks.workunit.client.1.smithi153.stdout:9/918: link d4/d6/dc/d2d/d79/dbb/cfe d4/d6/db1/d117/c122 0 2024-04-16T02:35:47.014 INFO:tasks.workunit.client.0.smithi151.stdout:4/373: mkdir d0/d11/d19/d31/d80 0 2024-04-16T02:35:47.015 INFO:tasks.workunit.client.1.smithi153.stdout:2/890: dwrite d9/d24/d27/d2c/d5d/f63 [0,4194304] 0 2024-04-16T02:35:47.020 INFO:tasks.workunit.client.0.smithi151.stdout:8/275: link d11/d2a/d33/f45 d11/d2a/d35/d55/f58 0 2024-04-16T02:35:47.026 INFO:tasks.workunit.client.1.smithi153.stdout:8/835: unlink d2/d1c/ca6 0 2024-04-16T02:35:47.026 INFO:tasks.workunit.client.0.smithi151.stdout:4/374: dwrite d0/d11/d19/d31/d34/f5a [0,4194304] 0 2024-04-16T02:35:47.032 INFO:tasks.workunit.client.1.smithi153.stdout:3/870: dread d6/d10/d3b/d59/d7c/de1/df0/def/ff9 [0,4194304] 0 2024-04-16T02:35:47.033 INFO:tasks.workunit.client.1.smithi153.stdout:3/871: chown d6/d10/d22/d43/f57 2191491 1 2024-04-16T02:35:47.036 INFO:tasks.workunit.client.0.smithi151.stdout:5/437: mkdir d2/d6/d19/d30/d36/d42/d6f/d83/d8b 0 2024-04-16T02:35:47.040 INFO:tasks.workunit.client.1.smithi153.stdout:1/879: dread d4/da2/d65/d84/f8c [0,4194304] 0 2024-04-16T02:35:47.041 INFO:tasks.workunit.client.1.smithi153.stdout:1/880: chown d4/dd/d13/d8d/d95/d115 52141377 1 2024-04-16T02:35:47.042 INFO:tasks.workunit.client.1.smithi153.stdout:1/881: write d4/dd/d13/d23/d5f/f10c [327929,52855] 0 2024-04-16T02:35:47.043 INFO:tasks.workunit.client.1.smithi153.stdout:3/872: creat d6/d10/d3b/d10d/d8a/d8c/f11a x:0 0 0 2024-04-16T02:35:47.045 INFO:tasks.workunit.client.0.smithi151.stdout:7/335: rename d0/l3c to d0/d35/l74 0 2024-04-16T02:35:47.046 INFO:tasks.workunit.client.1.smithi153.stdout:9/919: creat d4/d6/d1b/f123 x:0 0 0 2024-04-16T02:35:47.047 INFO:tasks.workunit.client.0.smithi151.stdout:9/267: dwrite d1/d3/d10/f24 [0,4194304] 0 2024-04-16T02:35:47.050 INFO:tasks.workunit.client.1.smithi153.stdout:3/873: dwrite d6/d10/d22/d4d/da0/f118 [0,4194304] 0 2024-04-16T02:35:47.051 INFO:tasks.workunit.client.0.smithi151.stdout:2/380: write de/d15/d1e/f67 [2955,111984] 0 2024-04-16T02:35:47.052 INFO:tasks.workunit.client.0.smithi151.stdout:4/375: link d0/d17/l5c d0/d11/d19/d31/d80/l81 0 2024-04-16T02:35:47.053 INFO:tasks.workunit.client.1.smithi153.stdout:7/858: truncate d0/d4/db/d47/d8d/f98 1514362 0 2024-04-16T02:35:47.055 INFO:tasks.workunit.client.0.smithi151.stdout:8/276: getdents d11 0 2024-04-16T02:35:47.058 INFO:tasks.workunit.client.1.smithi153.stdout:6/855: truncate dc/d47/d57/d77/dc4/fd0 1645911 0 2024-04-16T02:35:47.058 INFO:tasks.workunit.client.0.smithi151.stdout:7/336: rename d0/c1a to d0/d1f/d31/d42/c75 0 2024-04-16T02:35:47.060 INFO:tasks.workunit.client.1.smithi153.stdout:9/920: symlink d4/d33/dae/dc7/d66/dcb/l124 0 2024-04-16T02:35:47.060 INFO:tasks.workunit.client.0.smithi151.stdout:1/352: truncate d3/d58/df/d38/f5a 3164540 0 2024-04-16T02:35:47.061 INFO:tasks.workunit.client.0.smithi151.stdout:2/381: fdatasync de/d11/f2b 0 2024-04-16T02:35:47.062 INFO:tasks.workunit.client.0.smithi151.stdout:1/353: dread - d3/d33/d10/f6b zero size 2024-04-16T02:35:47.063 INFO:tasks.workunit.client.0.smithi151.stdout:4/376: dwrite d0/f15 [0,4194304] 0 2024-04-16T02:35:47.063 INFO:tasks.workunit.client.1.smithi153.stdout:1/882: getdents d4/dd/d13/ded/dfc 0 2024-04-16T02:35:47.065 INFO:tasks.workunit.client.1.smithi153.stdout:1/883: chown d4/da2/d65/d82/fb0 83684 1 2024-04-16T02:35:47.065 INFO:tasks.workunit.client.0.smithi151.stdout:9/268: creat d1/dd/d4a/f53 x:0 0 0 2024-04-16T02:35:47.066 INFO:tasks.workunit.client.1.smithi153.stdout:9/921: dwrite d4/d6/d1b/f71 [0,4194304] 0 2024-04-16T02:35:47.069 INFO:tasks.workunit.client.0.smithi151.stdout:4/377: stat d0/d11/d19/d31/d34/d3a/d61/l69 0 2024-04-16T02:35:47.070 INFO:tasks.workunit.client.1.smithi153.stdout:8/836: getdents d2/d93/d97/d9c 0 2024-04-16T02:35:47.070 INFO:tasks.workunit.client.0.smithi151.stdout:4/378: write d0/d11/d19/f44 [985701,69930] 0 2024-04-16T02:35:47.071 INFO:tasks.workunit.client.0.smithi151.stdout:7/337: creat d0/d1f/d31/d42/f76 x:0 0 0 2024-04-16T02:35:47.073 INFO:tasks.workunit.client.0.smithi151.stdout:5/438: dread d2/f1f [0,4194304] 0 2024-04-16T02:35:47.075 INFO:tasks.workunit.client.0.smithi151.stdout:5/439: chown d2/d6/d19/d69/l7a 321 1 2024-04-16T02:35:47.076 INFO:tasks.workunit.client.0.smithi151.stdout:2/382: rename de/d11/d42/f51 to de/d11/d42/f75 0 2024-04-16T02:35:47.077 INFO:tasks.workunit.client.0.smithi151.stdout:8/277: write d11/d2a/f2f [2250873,122796] 0 2024-04-16T02:35:47.080 INFO:tasks.workunit.client.0.smithi151.stdout:2/383: write de/d11/d41/d35/f74 [464979,45300] 0 2024-04-16T02:35:47.081 INFO:tasks.workunit.client.0.smithi151.stdout:7/338: dread d0/f3 [8388608,4194304] 0 2024-04-16T02:35:47.082 INFO:tasks.workunit.client.1.smithi153.stdout:2/891: dread d9/d24/d27/d2c/d41/fb5 [0,4194304] 0 2024-04-16T02:35:47.084 INFO:tasks.workunit.client.0.smithi151.stdout:4/379: unlink d0/d11/d19/d31/d34/l54 0 2024-04-16T02:35:47.085 INFO:tasks.workunit.client.0.smithi151.stdout:8/278: symlink d11/d2a/d35/l59 0 2024-04-16T02:35:47.086 INFO:tasks.workunit.client.0.smithi151.stdout:9/269: getdents d1/d3/d6/d1d/d22/d36 0 2024-04-16T02:35:47.088 INFO:tasks.workunit.client.1.smithi153.stdout:1/884: unlink d4/dd/d13/d8d/f101 0 2024-04-16T02:35:47.090 INFO:tasks.workunit.client.1.smithi153.stdout:9/922: symlink d4/d6/d1b/d6e/l125 0 2024-04-16T02:35:47.090 INFO:tasks.workunit.client.0.smithi151.stdout:7/339: symlink d0/d1f/d46/d4f/l77 0 2024-04-16T02:35:47.091 INFO:tasks.workunit.client.0.smithi151.stdout:1/354: getdents d3/d58/df/d68 0 2024-04-16T02:35:47.091 INFO:tasks.workunit.client.0.smithi151.stdout:5/440: mknod d2/d6/d19/d30/d36/d42/d6f/d83/d8b/c8c 0 2024-04-16T02:35:47.091 INFO:tasks.workunit.client.1.smithi153.stdout:8/837: creat d2/de/d3f/f11a x:0 0 0 2024-04-16T02:35:47.097 INFO:tasks.workunit.client.1.smithi153.stdout:2/892: creat d9/d24/d27/d2c/f121 x:0 0 0 2024-04-16T02:35:47.097 INFO:tasks.workunit.client.0.smithi151.stdout:5/441: write d2/d6/d19/d30/d49/d7f/f8a [620207,98160] 0 2024-04-16T02:35:47.099 INFO:tasks.workunit.client.0.smithi151.stdout:2/384: mknod de/c76 0 2024-04-16T02:35:47.099 INFO:tasks.workunit.client.0.smithi151.stdout:9/270: mkdir d1/d3/d6/d17/d54 0 2024-04-16T02:35:47.100 INFO:tasks.workunit.client.1.smithi153.stdout:3/874: dread d6/d10/d22/d2c/fd3 [0,4194304] 0 2024-04-16T02:35:47.101 INFO:tasks.workunit.client.0.smithi151.stdout:8/279: mknod d11/d22/c5a 0 2024-04-16T02:35:47.103 INFO:tasks.workunit.client.1.smithi153.stdout:1/885: dread d4/f49 [0,4194304] 0 2024-04-16T02:35:47.103 INFO:tasks.workunit.client.0.smithi151.stdout:5/442: readlink d2/d6/d19/d30/d36/d42/d6f/l7d 0 2024-04-16T02:35:47.103 INFO:tasks.workunit.client.0.smithi151.stdout:7/340: dwrite d0/f1d [0,4194304] 0 2024-04-16T02:35:47.105 INFO:tasks.workunit.client.1.smithi153.stdout:9/923: symlink d4/d6/dc/d24/d60/d11e/l126 0 2024-04-16T02:35:47.106 INFO:tasks.workunit.client.1.smithi153.stdout:8/838: dwrite d2/d93/d97/d9c/d27/d5e/d63/dc6/f10b [0,4194304] 0 2024-04-16T02:35:47.106 INFO:tasks.workunit.client.0.smithi151.stdout:9/271: write d1/d3/d6/d17/f27 [4256362,97935] 0 2024-04-16T02:35:47.107 INFO:tasks.workunit.client.0.smithi151.stdout:2/385: chown de/d11/d41/f33 85802 1 2024-04-16T02:35:47.108 INFO:tasks.workunit.client.1.smithi153.stdout:6/856: creat dc/d1a/f11c x:0 0 0 2024-04-16T02:35:47.108 INFO:tasks.workunit.client.0.smithi151.stdout:9/272: chown d1/d3/d6/d17/d44 6 1 2024-04-16T02:35:47.109 INFO:tasks.workunit.client.0.smithi151.stdout:9/273: chown d1/f2e 5067 1 2024-04-16T02:35:47.113 INFO:tasks.workunit.client.0.smithi151.stdout:8/280: link d11/d2a/d33/f4b d11/f5b 0 2024-04-16T02:35:47.114 INFO:tasks.workunit.client.1.smithi153.stdout:3/875: read d6/d10/d3b/d10d/d8a/fc1 [1812142,58703] 0 2024-04-16T02:35:47.115 INFO:tasks.workunit.client.1.smithi153.stdout:2/893: dwrite d9/d24/d27/f92 [0,4194304] 0 2024-04-16T02:35:47.118 INFO:tasks.workunit.client.1.smithi153.stdout:7/859: dwrite d0/d4/db/d2b/fac [0,4194304] 0 2024-04-16T02:35:47.119 INFO:tasks.workunit.client.0.smithi151.stdout:5/443: mknod d2/d6/d19/d62/c8d 0 2024-04-16T02:35:47.119 INFO:tasks.workunit.client.0.smithi151.stdout:4/380: dwrite d0/d11/d19/d31/d34/f48 [0,4194304] 0 2024-04-16T02:35:47.122 INFO:tasks.workunit.client.0.smithi151.stdout:4/381: read - d0/d20/f66 zero size 2024-04-16T02:35:47.123 INFO:tasks.workunit.client.0.smithi151.stdout:5/444: creat d2/f8e x:0 0 0 2024-04-16T02:35:47.124 INFO:tasks.workunit.client.1.smithi153.stdout:9/924: symlink d4/d6/d1b/ddf/l127 0 2024-04-16T02:35:47.124 INFO:tasks.workunit.client.1.smithi153.stdout:1/886: fdatasync d4/dd/d13/d26/f92 0 2024-04-16T02:35:47.124 INFO:tasks.workunit.client.0.smithi151.stdout:4/382: stat d0/d11 0 2024-04-16T02:35:47.128 INFO:tasks.workunit.client.0.smithi151.stdout:5/445: truncate d2/d5e/f72 604651 0 2024-04-16T02:35:47.128 INFO:tasks.workunit.client.0.smithi151.stdout:2/386: rename de/f40 to de/d11/d42/d5e/d6c/f77 0 2024-04-16T02:35:47.129 INFO:tasks.workunit.client.0.smithi151.stdout:9/274: dread d1/d3/d10/f24 [0,4194304] 0 2024-04-16T02:35:47.129 INFO:tasks.workunit.client.1.smithi153.stdout:2/894: mkdir d9/d24/d34/d99/d122 0 2024-04-16T02:35:47.130 INFO:tasks.workunit.client.1.smithi153.stdout:7/860: dread d0/d4/db/d2b/f4e [0,4194304] 0 2024-04-16T02:35:47.130 INFO:tasks.workunit.client.1.smithi153.stdout:8/839: unlink d2/d93/d97/d9c/d27/d5e/d63/fb1 0 2024-04-16T02:35:47.131 INFO:tasks.workunit.client.1.smithi153.stdout:1/887: readlink d4/da2/d65/le5 0 2024-04-16T02:35:47.132 INFO:tasks.workunit.client.0.smithi151.stdout:7/341: dwrite d0/d1f/d31/d42/f45 [4194304,4194304] 0 2024-04-16T02:35:47.132 INFO:tasks.workunit.client.0.smithi151.stdout:2/387: chown de/d11/d41/l58 11229 1 2024-04-16T02:35:47.133 INFO:tasks.workunit.client.1.smithi153.stdout:3/876: creat d6/d10/d22/d4d/da0/deb/f11b x:0 0 0 2024-04-16T02:35:47.134 INFO:tasks.workunit.client.0.smithi151.stdout:2/388: dread - de/d15/f5b zero size 2024-04-16T02:35:47.134 INFO:tasks.workunit.client.0.smithi151.stdout:7/342: read - d0/d1f/d46/f5a zero size 2024-04-16T02:35:47.136 INFO:tasks.workunit.client.1.smithi153.stdout:6/857: dwrite dc/d47/d7d/fdd [0,4194304] 0 2024-04-16T02:35:47.137 INFO:tasks.workunit.client.1.smithi153.stdout:9/925: creat d4/d6/dc/d24/d104/d113/f128 x:0 0 0 2024-04-16T02:35:47.138 INFO:tasks.workunit.client.0.smithi151.stdout:4/383: rename d0/d11/d19/d31/d34/f3f to d0/f82 0 2024-04-16T02:35:47.138 INFO:tasks.workunit.client.0.smithi151.stdout:3/239: sync 2024-04-16T02:35:47.139 INFO:tasks.workunit.client.1.smithi153.stdout:1/888: rename d4/dd/d13/ded/dfc to d4/dd/d13/ded/dfc/d12b 22 2024-04-16T02:35:47.139 INFO:tasks.workunit.client.0.smithi151.stdout:7/343: chown d0/f18 6 1 2024-04-16T02:35:47.140 INFO:tasks.workunit.client.0.smithi151.stdout:2/389: mkdir de/d15/d78 0 2024-04-16T02:35:47.143 INFO:tasks.workunit.client.1.smithi153.stdout:9/926: chown d4/d6/d1b/d2a/d99/fce 39957496 1 2024-04-16T02:35:47.143 INFO:tasks.workunit.client.1.smithi153.stdout:1/889: readlink d4/dd/d18/d6c/lca 0 2024-04-16T02:35:47.147 INFO:tasks.workunit.client.1.smithi153.stdout:7/861: creat d0/d4/db/d47/d8d/d93/f129 x:0 0 0 2024-04-16T02:35:47.147 INFO:tasks.workunit.client.0.smithi151.stdout:9/275: creat d1/f55 x:0 0 0 2024-04-16T02:35:47.148 INFO:tasks.workunit.client.0.smithi151.stdout:9/276: dread - d1/dd/f51 zero size 2024-04-16T02:35:47.149 INFO:tasks.workunit.client.1.smithi153.stdout:2/895: mknod d9/d71/c123 0 2024-04-16T02:35:47.150 INFO:tasks.workunit.client.1.smithi153.stdout:3/877: dread d6/d10/d22/d2c/f44 [0,4194304] 0 2024-04-16T02:35:47.151 INFO:tasks.workunit.client.0.smithi151.stdout:9/277: chown d1/d3/f33 372166 1 2024-04-16T02:35:47.153 INFO:tasks.workunit.client.0.smithi151.stdout:7/344: mkdir d0/d2c/d38/d43/d78 0 2024-04-16T02:35:47.153 INFO:tasks.workunit.client.1.smithi153.stdout:8/840: creat d2/d13/d96/dc7/d109/d47/d60/d67/db7/f11b x:0 0 0 2024-04-16T02:35:47.154 INFO:tasks.workunit.client.1.smithi153.stdout:9/927: rmdir d4 39 2024-04-16T02:35:47.155 INFO:tasks.workunit.client.1.smithi153.stdout:8/841: write d2/d93/d97/d9c/f3c [4985067,41451] 0 2024-04-16T02:35:47.155 INFO:tasks.workunit.client.0.smithi151.stdout:2/390: unlink de/d15/f39 0 2024-04-16T02:35:47.156 INFO:tasks.workunit.client.1.smithi153.stdout:7/862: symlink d0/d4/db/d2b/d43/dbf/l12a 0 2024-04-16T02:35:47.157 INFO:tasks.workunit.client.1.smithi153.stdout:2/896: truncate d9/d2b/d1c/f28 2988941 0 2024-04-16T02:35:47.158 INFO:tasks.workunit.client.0.smithi151.stdout:9/278: mkdir d1/d3/d6/d17/d40/d56 0 2024-04-16T02:35:47.162 INFO:tasks.workunit.client.1.smithi153.stdout:9/928: fsync d4/d6/dc/fd7 0 2024-04-16T02:35:47.163 INFO:tasks.workunit.client.0.smithi151.stdout:2/391: mkdir de/d11/d42/d6b/d79 0 2024-04-16T02:35:47.165 INFO:tasks.workunit.client.1.smithi153.stdout:9/929: stat d4/d6/ddb/f78 0 2024-04-16T02:35:47.166 INFO:tasks.workunit.client.1.smithi153.stdout:5/925: sync 2024-04-16T02:35:47.166 INFO:tasks.workunit.client.0.smithi151.stdout:9/279: dwrite d1/f2e [0,4194304] 0 2024-04-16T02:35:47.167 INFO:tasks.workunit.client.1.smithi153.stdout:9/930: chown d4/d6/dc/d24/d104/d113/f128 1132 1 2024-04-16T02:35:47.168 INFO:tasks.workunit.client.1.smithi153.stdout:9/931: fsync d4/d6/d1b/d6e/f100 0 2024-04-16T02:35:47.169 INFO:tasks.workunit.client.1.smithi153.stdout:9/932: write d4/d6/dc/d2d/f53 [395457,62907] 0 2024-04-16T02:35:47.169 INFO:tasks.workunit.client.0.smithi151.stdout:7/345: creat d0/d1f/d31/f79 x:0 0 0 2024-04-16T02:35:47.170 INFO:tasks.workunit.client.1.smithi153.stdout:3/878: creat d6/d10/d3b/d59/d7c/de1/f11c x:0 0 0 2024-04-16T02:35:47.171 INFO:tasks.workunit.client.0.smithi151.stdout:8/281: dread d11/f15 [0,4194304] 0 2024-04-16T02:35:47.172 INFO:tasks.workunit.client.0.smithi151.stdout:9/280: mkdir d1/d3/d6/d4f/d57 0 2024-04-16T02:35:47.174 INFO:tasks.workunit.client.1.smithi153.stdout:5/926: symlink d8/d41/d90/d9d/l120 0 2024-04-16T02:35:47.176 INFO:tasks.workunit.client.1.smithi153.stdout:8/842: link d2/d93/d97/d9c/d27/d5e/d63/lb4 d2/d93/d97/d9c/l11c 0 2024-04-16T02:35:47.176 INFO:tasks.workunit.client.0.smithi151.stdout:2/392: dread de/d15/f17 [0,4194304] 0 2024-04-16T02:35:47.177 INFO:tasks.workunit.client.0.smithi151.stdout:7/346: truncate d0/f11 2554305 0 2024-04-16T02:35:47.177 INFO:tasks.workunit.client.1.smithi153.stdout:7/863: getdents d0/d4/db/d2b/d43/d128/d6c/db8 0 2024-04-16T02:35:47.179 INFO:tasks.workunit.client.1.smithi153.stdout:3/879: creat d6/d10/dec/df8/f11d x:0 0 0 2024-04-16T02:35:47.179 INFO:tasks.workunit.client.0.smithi151.stdout:8/282: fdatasync d11/f4a 0 2024-04-16T02:35:47.180 INFO:tasks.workunit.client.1.smithi153.stdout:3/880: write d6/d10/dec/df8/f11d [773510,59748] 0 2024-04-16T02:35:47.181 INFO:tasks.workunit.client.1.smithi153.stdout:3/881: fdatasync d6/d10/f19 0 2024-04-16T02:35:47.183 INFO:tasks.workunit.client.0.smithi151.stdout:7/347: readlink d0/l20 0 2024-04-16T02:35:47.183 INFO:tasks.workunit.client.1.smithi153.stdout:8/843: mkdir d2/d13/d96/d11d 0 2024-04-16T02:35:47.184 INFO:tasks.workunit.client.0.smithi151.stdout:2/393: mkdir de/d11/d42/d7a 0 2024-04-16T02:35:47.184 INFO:tasks.workunit.client.0.smithi151.stdout:3/240: write d1/d6/f1b [3066727,86872] 0 2024-04-16T02:35:47.185 INFO:tasks.workunit.client.1.smithi153.stdout:5/927: symlink d8/df/d28/d5a/d72/l121 0 2024-04-16T02:35:47.185 INFO:tasks.workunit.client.1.smithi153.stdout:8/844: truncate d2/f14 4669768 0 2024-04-16T02:35:47.186 INFO:tasks.workunit.client.1.smithi153.stdout:6/858: dread f9 [0,4194304] 0 2024-04-16T02:35:47.186 INFO:tasks.workunit.client.0.smithi151.stdout:4/384: write d0/d11/d19/f29 [1260579,110428] 0 2024-04-16T02:35:47.187 INFO:tasks.workunit.client.1.smithi153.stdout:5/928: chown d8/df/d2f/d4e/f89 95367870 1 2024-04-16T02:35:47.188 INFO:tasks.workunit.client.1.smithi153.stdout:5/929: chown d8/d41/dbd/d39/f4c 20 1 2024-04-16T02:35:47.188 INFO:tasks.workunit.client.0.smithi151.stdout:4/385: write d0/f55 [4606251,25745] 0 2024-04-16T02:35:47.189 INFO:tasks.workunit.client.1.smithi153.stdout:5/930: fdatasync d8/df/d2e/f3d 0 2024-04-16T02:35:47.190 INFO:tasks.workunit.client.1.smithi153.stdout:9/933: rename d4/d6/dc/d7f/fe9 to d4/d33/dae/dc7/f129 0 2024-04-16T02:35:47.191 INFO:tasks.workunit.client.0.smithi151.stdout:7/348: creat d0/d27/f7a x:0 0 0 2024-04-16T02:35:47.195 INFO:tasks.workunit.client.1.smithi153.stdout:2/897: dread d9/d71/d8d/fd9 [0,4194304] 0 2024-04-16T02:35:47.195 INFO:tasks.workunit.client.0.smithi151.stdout:4/386: mknod d0/d11/d19/d31/d68/c83 0 2024-04-16T02:35:47.196 INFO:tasks.workunit.client.0.smithi151.stdout:5/446: dread d2/d6/f2a [0,4194304] 0 2024-04-16T02:35:47.196 INFO:tasks.workunit.client.1.smithi153.stdout:5/931: dread d8/f6c [0,4194304] 0 2024-04-16T02:35:47.197 INFO:tasks.workunit.client.1.smithi153.stdout:6/859: mknod dc/d47/d7d/c11d 0 2024-04-16T02:35:47.197 INFO:tasks.workunit.client.0.smithi151.stdout:4/387: write d0/d20/f4f [103375,46020] 0 2024-04-16T02:35:47.197 INFO:tasks.workunit.client.0.smithi151.stdout:9/281: getdents d1 0 2024-04-16T02:35:47.197 INFO:tasks.workunit.client.0.smithi151.stdout:3/241: symlink d1/d6/d20/d3d/l51 0 2024-04-16T02:35:47.199 INFO:tasks.workunit.client.1.smithi153.stdout:1/890: truncate d4/dd/d13/f2a 7215306 0 2024-04-16T02:35:47.202 INFO:tasks.workunit.client.1.smithi153.stdout:6/860: dread dc/d47/d7d/fdd [0,4194304] 0 2024-04-16T02:35:47.203 INFO:tasks.workunit.client.1.smithi153.stdout:9/934: rename d4/d6/d1b/f76 to d4/d33/dae/f12a 0 2024-04-16T02:35:47.207 INFO:tasks.workunit.client.1.smithi153.stdout:7/864: creat d0/d13/f12b x:0 0 0 2024-04-16T02:35:47.207 INFO:tasks.workunit.client.1.smithi153.stdout:8/845: dread d2/d13/d2c/f48 [0,4194304] 0 2024-04-16T02:35:47.207 INFO:tasks.workunit.client.0.smithi151.stdout:9/282: dwrite d1/d3/d6/d17/f31 [0,4194304] 0 2024-04-16T02:35:47.208 INFO:tasks.workunit.client.0.smithi151.stdout:9/283: chown d1/dd/d38/l3c 60 1 2024-04-16T02:35:47.209 INFO:tasks.workunit.client.1.smithi153.stdout:2/898: mkdir d9/d24/d27/d2c/d5d/de5/d124 0 2024-04-16T02:35:47.209 INFO:tasks.workunit.client.0.smithi151.stdout:4/388: rename d0/l1 to d0/d11/d19/d31/d4a/l84 0 2024-04-16T02:35:47.209 INFO:tasks.workunit.client.0.smithi151.stdout:9/284: stat d1/dd/l45 0 2024-04-16T02:35:47.211 INFO:tasks.workunit.client.0.smithi151.stdout:5/447: creat d2/d6/d19/d30/d36/d42/d6f/d83/d8b/f8f x:0 0 0 2024-04-16T02:35:47.214 INFO:tasks.workunit.client.0.smithi151.stdout:7/349: link d0/c10 d0/d27/c7b 0 2024-04-16T02:35:47.220 INFO:tasks.workunit.client.0.smithi151.stdout:9/285: dwrite d1/dd/f51 [0,4194304] 0 2024-04-16T02:35:47.220 INFO:tasks.workunit.client.0.smithi151.stdout:5/448: unlink d2/d6/f8 0 2024-04-16T02:35:47.221 INFO:tasks.workunit.client.0.smithi151.stdout:3/242: link d1/d6/d20/d2a/f16 d1/d6/d22/f52 0 2024-04-16T02:35:47.221 INFO:tasks.workunit.client.1.smithi153.stdout:6/861: write dc/d47/d57/d77/d15/fc2 [1342687,23838] 0 2024-04-16T02:35:47.222 INFO:tasks.workunit.client.0.smithi151.stdout:5/449: chown d2/f16 163 1 2024-04-16T02:35:47.223 INFO:tasks.workunit.client.1.smithi153.stdout:6/862: dread - dc/d47/d57/d77/d15/d109/f114 zero size 2024-04-16T02:35:47.224 INFO:tasks.workunit.client.1.smithi153.stdout:3/882: dread d6/d10/d22/d2c/f74 [0,4194304] 0 2024-04-16T02:35:47.225 INFO:tasks.workunit.client.1.smithi153.stdout:9/935: creat d4/d6/d1b/d2a/f12b x:0 0 0 2024-04-16T02:35:47.225 INFO:tasks.workunit.client.0.smithi151.stdout:7/350: symlink d0/d27/l7c 0 2024-04-16T02:35:47.227 INFO:tasks.workunit.client.0.smithi151.stdout:5/450: dread - d2/d6/d19/d1a/f3b zero size 2024-04-16T02:35:47.228 INFO:tasks.workunit.client.1.smithi153.stdout:8/846: mknod d2/d13/d96/c11e 0 2024-04-16T02:35:47.231 INFO:tasks.workunit.client.0.smithi151.stdout:9/286: mkdir d1/d3/d6/d17/d54/d58 0 2024-04-16T02:35:47.232 INFO:tasks.workunit.client.0.smithi151.stdout:7/351: symlink d0/d27/l7d 0 2024-04-16T02:35:47.233 INFO:tasks.workunit.client.1.smithi153.stdout:5/932: link d8/d41/dbd/d39/d6d/fe4 d8/d41/dbd/d83/f122 0 2024-04-16T02:35:47.234 INFO:tasks.workunit.client.0.smithi151.stdout:7/352: write d0/d1f/d46/f5d [772163,85504] 0 2024-04-16T02:35:47.235 INFO:tasks.workunit.client.0.smithi151.stdout:2/394: truncate de/f27 3216259 0 2024-04-16T02:35:47.236 INFO:tasks.workunit.client.1.smithi153.stdout:1/891: link d4/dd/d18/d6c/dd8/ff1 d4/dd/d13/d8d/d95/dc7/f12c 0 2024-04-16T02:35:47.239 INFO:tasks.workunit.client.1.smithi153.stdout:3/883: fdatasync d6/d10/d22/d4d/dbb/fc0 0 2024-04-16T02:35:47.240 INFO:tasks.workunit.client.1.smithi153.stdout:3/884: write d6/d10/d3b/d10d/d8a/f101 [427401,10365] 0 2024-04-16T02:35:47.241 INFO:tasks.workunit.client.0.smithi151.stdout:9/287: stat d1/d3/d6/c1c 0 2024-04-16T02:35:47.241 INFO:tasks.workunit.client.1.smithi153.stdout:9/936: mknod d4/d33/dae/dc7/d66/dbd/c12c 0 2024-04-16T02:35:47.242 INFO:tasks.workunit.client.0.smithi151.stdout:3/243: dwrite d1/d6/f1b [4194304,4194304] 0 2024-04-16T02:35:47.242 INFO:tasks.workunit.client.1.smithi153.stdout:3/885: readlink d6/d10/d22/d2c/l9a 0 2024-04-16T02:35:47.245 INFO:tasks.workunit.client.1.smithi153.stdout:8/847: rename d2/d13/d96/dc7/d109/d47/d60/d67/d86/f89 to d2/d13/d3a/d5b/f11f 0 2024-04-16T02:35:47.245 INFO:tasks.workunit.client.0.smithi151.stdout:7/353: mknod d0/d1f/d46/d4f/c7e 0 2024-04-16T02:35:47.246 INFO:tasks.workunit.client.0.smithi151.stdout:5/451: stat d2/d6/d19/d30/d36/d42/d6f/l7d 0 2024-04-16T02:35:47.246 INFO:tasks.workunit.client.0.smithi151.stdout:8/283: dwrite d11/f43 [0,4194304] 0 2024-04-16T02:35:47.246 INFO:tasks.workunit.client.0.smithi151.stdout:9/288: symlink d1/d3/d6/d28/l59 0 2024-04-16T02:35:47.247 INFO:tasks.workunit.client.1.smithi153.stdout:9/937: dread d4/d6/dc/d2d/fa2 [0,4194304] 0 2024-04-16T02:35:47.248 INFO:tasks.workunit.client.0.smithi151.stdout:3/244: write d1/d6/f50 [427997,45172] 0 2024-04-16T02:35:47.250 INFO:tasks.workunit.client.0.smithi151.stdout:3/245: truncate d1/d6/d22/f2c 4633114 0 2024-04-16T02:35:47.251 INFO:tasks.workunit.client.1.smithi153.stdout:7/865: creat d0/d13/f12c x:0 0 0 2024-04-16T02:35:47.251 INFO:tasks.workunit.client.1.smithi153.stdout:3/886: fdatasync d6/d10/dec/f109 0 2024-04-16T02:35:47.252 INFO:tasks.workunit.client.0.smithi151.stdout:2/395: truncate de/d15/f36 871249 0 2024-04-16T02:35:47.252 INFO:tasks.workunit.client.1.smithi153.stdout:2/899: link d9/d24/d27/l114 d9/d2b/d18/d3f/d10e/l125 0 2024-04-16T02:35:47.253 INFO:tasks.workunit.client.1.smithi153.stdout:5/933: mkdir d8/df/d23/d10e/d123 0 2024-04-16T02:35:47.259 INFO:tasks.workunit.client.1.smithi153.stdout:7/866: write d0/d4/db/f115 [4327458,63232] 0 2024-04-16T02:35:47.260 INFO:tasks.workunit.client.1.smithi153.stdout:8/848: getdents d2/d93/d97/d9c/d27/dda 0 2024-04-16T02:35:47.260 INFO:tasks.workunit.client.1.smithi153.stdout:8/849: fdatasync d2/d13/f94 0 2024-04-16T02:35:47.261 INFO:tasks.workunit.client.0.smithi151.stdout:3/246: fdatasync d1/d6/d20/d2a/f28 0 2024-04-16T02:35:47.263 INFO:tasks.workunit.client.1.smithi153.stdout:8/850: chown d2/d13/de8 910313 1 2024-04-16T02:35:47.263 INFO:tasks.workunit.client.1.smithi153.stdout:5/934: unlink d8/fde 0 2024-04-16T02:35:47.263 INFO:tasks.workunit.client.1.smithi153.stdout:8/851: stat d2/d13/lcb 0 2024-04-16T02:35:47.266 INFO:tasks.workunit.client.1.smithi153.stdout:9/938: rename d4/d6/dc/c120 to d4/d6/ddb/d4a/d101/c12d 0 2024-04-16T02:35:47.266 INFO:tasks.workunit.client.0.smithi151.stdout:9/289: mkdir d1/d3/d5a 0 2024-04-16T02:35:47.266 INFO:tasks.workunit.client.0.smithi151.stdout:4/389: dread d0/fb [4194304,4194304] 0 2024-04-16T02:35:47.266 INFO:tasks.workunit.client.0.smithi151.stdout:9/290: readlink d1/d3/l37 0 2024-04-16T02:35:47.268 INFO:tasks.workunit.client.0.smithi151.stdout:4/390: write d0/d11/f26 [1260989,80157] 0 2024-04-16T02:35:47.268 INFO:tasks.workunit.client.1.smithi153.stdout:9/939: fsync d4/d6/d1b/f90 0 2024-04-16T02:35:47.268 INFO:tasks.workunit.client.1.smithi153.stdout:2/900: dwrite d9/d2b/d1c/d20/f67 [0,4194304] 0 2024-04-16T02:35:47.272 INFO:tasks.workunit.client.1.smithi153.stdout:2/901: chown d9/d24/d27/d2c/d41/d7a/d116/fc1 36651442 1 2024-04-16T02:35:47.273 INFO:tasks.workunit.client.0.smithi151.stdout:2/396: fdatasync de/d15/f36 0 2024-04-16T02:35:47.273 INFO:tasks.workunit.client.1.smithi153.stdout:5/935: dwrite d8/df/d28/d5a/fb8 [0,4194304] 0 2024-04-16T02:35:47.274 INFO:tasks.workunit.client.1.smithi153.stdout:2/902: dread - d9/d2b/d1c/d20/d45/f11a zero size 2024-04-16T02:35:47.276 INFO:tasks.workunit.client.1.smithi153.stdout:2/903: stat d9/d2b/d5c/d5f/l105 0 2024-04-16T02:35:47.276 INFO:tasks.workunit.client.1.smithi153.stdout:8/852: rename d2/d13/d96 to d2/d13/d96/dc7/d109/d120 22 2024-04-16T02:35:47.277 INFO:tasks.workunit.client.0.smithi151.stdout:8/284: write d11/f5b [393393,3995] 0 2024-04-16T02:35:47.277 INFO:tasks.workunit.client.0.smithi151.stdout:3/247: mknod d1/d6/d20/d2a/c53 0 2024-04-16T02:35:47.277 INFO:tasks.workunit.client.0.smithi151.stdout:8/285: readlink ld 0 2024-04-16T02:35:47.278 INFO:tasks.workunit.client.0.smithi151.stdout:9/291: mkdir d1/d3/d6/d17/d54/d5b 0 2024-04-16T02:35:47.278 INFO:tasks.workunit.client.0.smithi151.stdout:3/248: truncate d1/f3e 562201 0 2024-04-16T02:35:47.278 INFO:tasks.workunit.client.0.smithi151.stdout:9/292: read - d1/f55 zero size 2024-04-16T02:35:47.280 INFO:tasks.workunit.client.0.smithi151.stdout:0/279: sync 2024-04-16T02:35:47.281 INFO:tasks.workunit.client.1.smithi153.stdout:1/892: write d4/dd/fbe [954714,80025] 0 2024-04-16T02:35:47.281 INFO:tasks.workunit.client.0.smithi151.stdout:9/293: fsync d1/d3/d6/f1b 0 2024-04-16T02:35:47.282 INFO:tasks.workunit.client.0.smithi151.stdout:9/294: dread - d1/f55 zero size 2024-04-16T02:35:47.282 INFO:tasks.workunit.client.1.smithi153.stdout:9/940: unlink d4/d6/dc/d2d/da0/d69/f6c 0 2024-04-16T02:35:47.283 INFO:tasks.workunit.client.0.smithi151.stdout:9/295: chown d1 491069887 1 2024-04-16T02:35:47.284 INFO:tasks.workunit.client.1.smithi153.stdout:5/936: truncate d8/d1f/f21 612346 0 2024-04-16T02:35:47.286 INFO:tasks.workunit.client.1.smithi153.stdout:2/904: read d9/d2b/fb6 [30055,97294] 0 2024-04-16T02:35:47.286 INFO:tasks.workunit.client.0.smithi151.stdout:8/286: dwrite fb [4194304,4194304] 0 2024-04-16T02:35:47.288 INFO:tasks.workunit.client.1.smithi153.stdout:7/867: dread d0/d4/db/d2b/f89 [0,4194304] 0 2024-04-16T02:35:47.289 INFO:tasks.workunit.client.0.smithi151.stdout:5/452: getdents d2/d6/d19/d1a/d4c 0 2024-04-16T02:35:47.289 INFO:tasks.workunit.client.0.smithi151.stdout:2/397: creat de/d11/d42/d5e/f7b x:0 0 0 2024-04-16T02:35:47.289 INFO:tasks.workunit.client.0.smithi151.stdout:3/249: mkdir d1/d6/d20/d54 0 2024-04-16T02:35:47.290 INFO:tasks.workunit.client.1.smithi153.stdout:2/905: fdatasync d9/d2b/d18/d3f/d40/f70 0 2024-04-16T02:35:47.292 INFO:tasks.workunit.client.0.smithi151.stdout:5/453: stat d2/d6/d19/d1a/f2e 0 2024-04-16T02:35:47.292 INFO:tasks.workunit.client.1.smithi153.stdout:9/941: creat d4/d6/d1b/ddf/f12e x:0 0 0 2024-04-16T02:35:47.293 INFO:tasks.workunit.client.1.smithi153.stdout:3/887: dwrite d6/d10/f1c [0,4194304] 0 2024-04-16T02:35:47.293 INFO:tasks.workunit.client.1.smithi153.stdout:5/937: rename d8/d41/dbd/dd3/cdc to d8/d41/dbd/dd3/c124 0 2024-04-16T02:35:47.294 INFO:tasks.workunit.client.0.smithi151.stdout:7/354: truncate d0/d1f/f2a 1874179 0 2024-04-16T02:35:47.295 INFO:tasks.workunit.client.1.smithi153.stdout:3/888: readlink d6/d10/d3b/d59/l6a 0 2024-04-16T02:35:47.300 INFO:tasks.workunit.client.1.smithi153.stdout:5/938: truncate d8/d41/dbd/d83/ffe 852925 0 2024-04-16T02:35:47.301 INFO:tasks.workunit.client.0.smithi151.stdout:2/398: unlink de/d15/f46 0 2024-04-16T02:35:47.305 INFO:tasks.workunit.client.1.smithi153.stdout:9/942: write d4/d33/dae/dc7/d66/fb2 [1696882,105334] 0 2024-04-16T02:35:47.306 INFO:tasks.workunit.client.0.smithi151.stdout:4/391: dread d0/d11/f23 [0,4194304] 0 2024-04-16T02:35:47.306 INFO:tasks.workunit.client.1.smithi153.stdout:1/893: creat d4/da2/f12d x:0 0 0 2024-04-16T02:35:47.306 INFO:tasks.workunit.client.1.smithi153.stdout:7/868: symlink d0/d4/db/d2b/d53/d59/d76/l12d 0 2024-04-16T02:35:47.308 INFO:tasks.workunit.client.0.smithi151.stdout:2/399: unlink de/d11/f18 0 2024-04-16T02:35:47.310 INFO:tasks.workunit.client.1.smithi153.stdout:8/853: dread d2/d13/f94 [0,4194304] 0 2024-04-16T02:35:47.311 INFO:tasks.workunit.client.1.smithi153.stdout:7/869: rmdir d0/d4/db/d16/d126 39 2024-04-16T02:35:47.313 INFO:tasks.workunit.client.0.smithi151.stdout:0/280: write dc/d15/d34/f35 [842027,100850] 0 2024-04-16T02:35:47.314 INFO:tasks.workunit.client.0.smithi151.stdout:2/400: mkdir de/d65/d71/d7c 0 2024-04-16T02:35:47.315 INFO:tasks.workunit.client.0.smithi151.stdout:0/281: chown fa 789654 1 2024-04-16T02:35:47.315 INFO:tasks.workunit.client.0.smithi151.stdout:2/401: stat de/d15/f5b 0 2024-04-16T02:35:47.315 INFO:tasks.workunit.client.0.smithi151.stdout:0/282: dread - dc/f43 zero size 2024-04-16T02:35:47.317 INFO:tasks.workunit.client.1.smithi153.stdout:5/939: dwrite d8/df/d28/d5a/fe1 [0,4194304] 0 2024-04-16T02:35:47.319 INFO:tasks.workunit.client.0.smithi151.stdout:2/402: creat de/d11/f7d x:0 0 0 2024-04-16T02:35:47.320 INFO:tasks.workunit.client.1.smithi153.stdout:3/889: dwrite d6/d10/d22/f2a [0,4194304] 0 2024-04-16T02:35:47.322 INFO:tasks.workunit.client.0.smithi151.stdout:0/283: dread dc/d15/d34/d3b/f44 [0,4194304] 0 2024-04-16T02:35:47.323 INFO:tasks.workunit.client.0.smithi151.stdout:3/250: dwrite d1/d6/f24 [0,4194304] 0 2024-04-16T02:35:47.323 INFO:tasks.workunit.client.1.smithi153.stdout:5/940: mkdir d8/df/d28/d5a/d5e/d125 0 2024-04-16T02:35:47.323 INFO:tasks.workunit.client.1.smithi153.stdout:2/906: dwrite d9/d2b/d1c/d20/f101 [0,4194304] 0 2024-04-16T02:35:47.327 INFO:tasks.workunit.client.1.smithi153.stdout:9/943: dread d4/d6/dc/d24/f89 [0,4194304] 0 2024-04-16T02:35:47.333 INFO:tasks.workunit.client.0.smithi151.stdout:2/403: rename de/d11/d41/d35/l53 to de/d11/d41/d35/l7e 0 2024-04-16T02:35:47.334 INFO:tasks.workunit.client.1.smithi153.stdout:7/870: dwrite d0/d4/db/d2b/d43/d128/d6c/ff3 [0,4194304] 0 2024-04-16T02:35:47.334 INFO:tasks.workunit.client.1.smithi153.stdout:3/890: fsync d6/d10/d3b/d76/f7f 0 2024-04-16T02:35:47.336 INFO:tasks.workunit.client.0.smithi151.stdout:0/284: read - dc/d20/f3d zero size 2024-04-16T02:35:47.337 INFO:tasks.workunit.client.0.smithi151.stdout:9/296: dwrite d1/d3/f41 [0,4194304] 0 2024-04-16T02:35:47.338 INFO:tasks.workunit.client.1.smithi153.stdout:9/944: mkdir d4/d6/def/d12f 0 2024-04-16T02:35:47.338 INFO:tasks.workunit.client.1.smithi153.stdout:7/871: write d0/d4/db/d2b/d8c/fa6 [1325024,3475] 0 2024-04-16T02:35:47.338 INFO:tasks.workunit.client.0.smithi151.stdout:4/392: write d0/fb [6012952,12383] 0 2024-04-16T02:35:47.339 INFO:tasks.workunit.client.1.smithi153.stdout:9/945: chown d4/d6/d8c 13996293 1 2024-04-16T02:35:47.339 INFO:tasks.workunit.client.1.smithi153.stdout:5/941: creat d8/d41/dbd/f126 x:0 0 0 2024-04-16T02:35:47.340 INFO:tasks.workunit.client.1.smithi153.stdout:9/946: chown d4/d6/dc/d24/d60/d11e 27 1 2024-04-16T02:35:47.341 INFO:tasks.workunit.client.1.smithi153.stdout:5/942: chown d8/df/d28/d5a/fa1 1 1 2024-04-16T02:35:47.347 INFO:tasks.workunit.client.1.smithi153.stdout:5/943: dread d8/d41/f10f [0,4194304] 0 2024-04-16T02:35:47.348 INFO:tasks.workunit.client.1.smithi153.stdout:5/944: stat d8/df/d2e/f113 0 2024-04-16T02:35:47.349 INFO:tasks.workunit.client.0.smithi151.stdout:3/251: creat d1/d6/d20/d2a/d3f/d43/f55 x:0 0 0 2024-04-16T02:35:47.351 INFO:tasks.workunit.client.1.smithi153.stdout:3/891: mknod d6/d10/d5b/c11e 0 2024-04-16T02:35:47.353 INFO:tasks.workunit.client.1.smithi153.stdout:3/892: stat d6/d10/d3b/d59/d7c/db7 0 2024-04-16T02:35:47.353 INFO:tasks.workunit.client.1.smithi153.stdout:7/872: fdatasync d0/d4/f20 0 2024-04-16T02:35:47.356 INFO:tasks.workunit.client.1.smithi153.stdout:9/947: mknod d4/d6/def/d12f/c130 0 2024-04-16T02:35:47.356 INFO:tasks.workunit.client.0.smithi151.stdout:0/285: dwrite dc/d20/f49 [0,4194304] 0 2024-04-16T02:35:47.357 INFO:tasks.workunit.client.1.smithi153.stdout:9/948: chown d4/d6/dc/d2d/d79/dbb 21534 1 2024-04-16T02:35:47.359 INFO:tasks.workunit.client.0.smithi151.stdout:0/286: fdatasync dc/d15/f40 0 2024-04-16T02:35:47.359 INFO:tasks.workunit.client.0.smithi151.stdout:3/252: symlink d1/d6/d22/d23/d41/l56 0 2024-04-16T02:35:47.362 INFO:tasks.workunit.client.0.smithi151.stdout:3/253: chown d1/d6/f50 377839 1 2024-04-16T02:35:47.363 INFO:tasks.workunit.client.0.smithi151.stdout:9/297: dwrite d1/d3/d6/f13 [0,4194304] 0 2024-04-16T02:35:47.364 INFO:tasks.workunit.client.0.smithi151.stdout:0/287: creat dc/d15/f58 x:0 0 0 2024-04-16T02:35:47.365 INFO:tasks.workunit.client.0.smithi151.stdout:9/298: write d1/dd/d4a/f53 [777306,99] 0 2024-04-16T02:35:47.366 INFO:tasks.workunit.client.0.smithi151.stdout:0/288: write dc/d15/d34/f35 [5241216,18495] 0 2024-04-16T02:35:47.372 INFO:tasks.workunit.client.0.smithi151.stdout:3/254: fsync d1/d6/d20/d2a/f16 0 2024-04-16T02:35:47.373 INFO:tasks.workunit.client.0.smithi151.stdout:0/289: mkdir dc/d15/d34/d3b/d59 0 2024-04-16T02:35:47.374 INFO:tasks.workunit.client.0.smithi151.stdout:7/355: dread d0/f3 [4194304,4194304] 0 2024-04-16T02:35:47.376 INFO:tasks.workunit.client.1.smithi153.stdout:8/854: dread d2/d13/f62 [0,4194304] 0 2024-04-16T02:35:47.378 INFO:tasks.workunit.client.0.smithi151.stdout:2/404: getdents de/d11/d41/d35 0 2024-04-16T02:35:47.378 INFO:tasks.workunit.client.0.smithi151.stdout:9/299: dwrite d1/d3/d6/d1d/d22/d2c/f2d [0,4194304] 0 2024-04-16T02:35:47.380 INFO:tasks.workunit.client.0.smithi151.stdout:7/356: write d0/d1f/d46/f5a [611595,67121] 0 2024-04-16T02:35:47.380 INFO:tasks.workunit.client.0.smithi151.stdout:3/255: creat d1/d6/d20/d2a/d3f/d43/f57 x:0 0 0 2024-04-16T02:35:47.382 INFO:tasks.workunit.client.0.smithi151.stdout:0/290: mknod dc/d20/d2c/d51/c5a 0 2024-04-16T02:35:47.385 INFO:tasks.workunit.client.0.smithi151.stdout:0/291: dread - dc/d15/d34/f57 zero size 2024-04-16T02:35:47.385 INFO:tasks.workunit.client.1.smithi153.stdout:8/855: rename d2/de/l41 to d2/d93/d97/de5/l121 0 2024-04-16T02:35:47.386 INFO:tasks.workunit.client.0.smithi151.stdout:2/405: creat de/d65/d71/f7f x:0 0 0 2024-04-16T02:35:47.388 INFO:tasks.workunit.client.0.smithi151.stdout:0/292: read - dc/d20/d2c/f30 zero size 2024-04-16T02:35:47.389 INFO:tasks.workunit.client.0.smithi151.stdout:2/406: symlink de/d15/l80 0 2024-04-16T02:35:47.390 INFO:tasks.workunit.client.1.smithi153.stdout:2/907: dread d9/d24/d34/d60/fa9 [0,4194304] 0 2024-04-16T02:35:47.391 INFO:tasks.workunit.client.0.smithi151.stdout:7/357: rename d0/d1f/d31/d42/c75 to d0/d27/c7f 0 2024-04-16T02:35:47.392 INFO:tasks.workunit.client.1.smithi153.stdout:3/893: truncate d6/d10/d22/d43/f57 6396437 0 2024-04-16T02:35:47.393 INFO:tasks.workunit.client.0.smithi151.stdout:2/407: unlink de/d11/d42/f75 0 2024-04-16T02:35:47.399 INFO:tasks.workunit.client.1.smithi153.stdout:3/894: rename d6/d10/d3b/dd7 to d6/d10/d11f 0 2024-04-16T02:35:47.400 INFO:tasks.workunit.client.0.smithi151.stdout:0/293: rmdir dc/d15/d34/d3b/d59 0 2024-04-16T02:35:47.400 INFO:tasks.workunit.client.1.smithi153.stdout:9/949: dwrite d4/d33/faf [0,4194304] 0 2024-04-16T02:35:47.401 INFO:tasks.workunit.client.1.smithi153.stdout:2/908: dwrite d9/f9e [0,4194304] 0 2024-04-16T02:35:47.403 INFO:tasks.workunit.client.0.smithi151.stdout:7/358: getdents d0/d2c/d38/d3e 0 2024-04-16T02:35:47.406 INFO:tasks.workunit.client.1.smithi153.stdout:9/950: fdatasync d4/f13 0 2024-04-16T02:35:47.407 INFO:tasks.workunit.client.0.smithi151.stdout:0/294: unlink dc/d15/c38 0 2024-04-16T02:35:47.407 INFO:tasks.workunit.client.0.smithi151.stdout:7/359: mknod d0/d2c/d38/d43/c80 0 2024-04-16T02:35:47.409 INFO:tasks.workunit.client.1.smithi153.stdout:7/873: dread d0/d4/f20 [0,4194304] 0 2024-04-16T02:35:47.409 INFO:tasks.workunit.client.1.smithi153.stdout:2/909: fdatasync d9/d2b/d5c/f80 0 2024-04-16T02:35:47.409 INFO:tasks.workunit.client.1.smithi153.stdout:3/895: creat d6/d10/d5b/d115/f120 x:0 0 0 2024-04-16T02:35:47.410 INFO:tasks.workunit.client.1.smithi153.stdout:9/951: chown d4/d6/dc/d2d/d64/db9 32917 1 2024-04-16T02:35:47.411 INFO:tasks.workunit.client.0.smithi151.stdout:9/300: dread d1/d3/f33 [0,4194304] 0 2024-04-16T02:35:47.413 INFO:tasks.workunit.client.1.smithi153.stdout:9/952: symlink d4/d6/d1b/d2a/d99/l131 0 2024-04-16T02:35:47.414 INFO:tasks.workunit.client.0.smithi151.stdout:0/295: symlink dc/l5b 0 2024-04-16T02:35:47.418 INFO:tasks.workunit.client.1.smithi153.stdout:2/910: rename d9/d24/d34/d60/d7c to d9/d24/d34/d99/d126 0 2024-04-16T02:35:47.420 INFO:tasks.workunit.client.1.smithi153.stdout:7/874: dwrite d0/d4/db/d2b/d53/d59/d76/fd1 [0,4194304] 0 2024-04-16T02:35:47.422 INFO:tasks.workunit.client.1.smithi153.stdout:2/911: write d9/d24/d27/d9a/f9f [4173444,23206] 0 2024-04-16T02:35:47.428 INFO:tasks.workunit.client.1.smithi153.stdout:9/953: dwrite d4/d6/d1b/d6e/f74 [4194304,4194304] 0 2024-04-16T02:35:47.430 INFO:tasks.workunit.client.1.smithi153.stdout:2/912: write d9/d24/d27/d2c/d41/d7a/d116/fe1 [2927550,15042] 0 2024-04-16T02:35:47.431 INFO:tasks.workunit.client.0.smithi151.stdout:2/408: dwrite de/d11/d42/d5e/d6c/f77 [0,4194304] 0 2024-04-16T02:35:47.431 INFO:tasks.workunit.client.1.smithi153.stdout:2/913: chown d9/d24/d34/d60/lda 576926 1 2024-04-16T02:35:47.434 INFO:tasks.workunit.client.1.smithi153.stdout:2/914: stat d9/d24/d27/l85 0 2024-04-16T02:35:47.437 INFO:tasks.workunit.client.1.smithi153.stdout:8/856: dread d2/de/f24 [0,4194304] 0 2024-04-16T02:35:47.440 INFO:tasks.workunit.client.1.smithi153.stdout:8/857: readlink d2/d13/d96/dc7/d109/d47/d60/l7e 0 2024-04-16T02:35:47.441 INFO:tasks.workunit.client.1.smithi153.stdout:8/858: read d2/de/f5f [2902435,124061] 0 2024-04-16T02:35:47.486 INFO:tasks.workunit.client.0.smithi151.stdout:4/393: dread d0/d17/f2b [0,4194304] 0 2024-04-16T02:35:47.486 INFO:tasks.workunit.client.0.smithi151.stdout:9/301: truncate d1/d3/d6/d1d/d22/d2c/f2d 1476352 0 2024-04-16T02:35:47.487 INFO:tasks.workunit.client.0.smithi151.stdout:2/409: readlink de/d11/d41/d35/l66 0 2024-04-16T02:35:47.491 INFO:tasks.workunit.client.0.smithi151.stdout:4/394: creat d0/d20/d59/f85 x:0 0 0 2024-04-16T02:35:47.491 INFO:tasks.workunit.client.1.smithi153.stdout:7/875: unlink d0/d4/db/d2b/d8c/d9f/cd5 0 2024-04-16T02:35:47.492 INFO:tasks.workunit.client.0.smithi151.stdout:9/302: symlink d1/d3/d6/d17/d54/l5c 0 2024-04-16T02:35:47.496 INFO:tasks.workunit.client.1.smithi153.stdout:8/859: mkdir d2/d13/de8/d122 0 2024-04-16T02:35:47.497 INFO:tasks.workunit.client.1.smithi153.stdout:8/860: fdatasync d2/d92/f69 0 2024-04-16T02:35:47.500 INFO:tasks.workunit.client.1.smithi153.stdout:7/876: symlink d0/d4/db/d2b/d53/d59/d76/dc8/d101/l12e 0 2024-04-16T02:35:47.502 INFO:tasks.workunit.client.1.smithi153.stdout:2/915: link d9/d24/d34/d36/c6d d9/d71/d8d/da8/c127 0 2024-04-16T02:35:47.504 INFO:tasks.workunit.client.1.smithi153.stdout:3/896: dwrite d6/d10/d3b/d76/f7d [0,4194304] 0 2024-04-16T02:35:47.504 INFO:tasks.workunit.client.0.smithi151.stdout:6/521: sync 2024-04-16T02:35:47.504 INFO:tasks.workunit.client.0.smithi151.stdout:1/355: sync 2024-04-16T02:35:47.508 INFO:tasks.workunit.client.1.smithi153.stdout:3/897: truncate d6/d10/d22/d4d/f116 349936 0 2024-04-16T02:35:47.509 INFO:tasks.workunit.client.1.smithi153.stdout:2/916: truncate d9/d24/d27/d2c/d41/d7a/fb2 4526677 0 2024-04-16T02:35:47.512 INFO:tasks.workunit.client.1.smithi153.stdout:9/954: dwrite d4/d6/ddb/d4a/f91 [0,4194304] 0 2024-04-16T02:35:47.512 INFO:tasks.workunit.client.1.smithi153.stdout:3/898: unlink d6/d10/l1f 0 2024-04-16T02:35:47.513 INFO:tasks.workunit.client.1.smithi153.stdout:9/955: readlink d4/d6/d1b/l36 0 2024-04-16T02:35:47.514 INFO:tasks.workunit.client.1.smithi153.stdout:3/899: fdatasync d6/ff5 0 2024-04-16T02:35:47.516 INFO:tasks.workunit.client.1.smithi153.stdout:7/877: dread d0/d4/db/d2b/d53/d59/dc3/fe6 [0,4194304] 0 2024-04-16T02:35:47.517 INFO:tasks.workunit.client.1.smithi153.stdout:3/900: creat d6/d10/d11f/f121 x:0 0 0 2024-04-16T02:35:47.518 INFO:tasks.workunit.client.1.smithi153.stdout:7/878: rmdir d0/d4/db 39 2024-04-16T02:35:47.519 INFO:tasks.workunit.client.1.smithi153.stdout:3/901: chown d6/d10/lf4 98729 1 2024-04-16T02:35:47.520 INFO:tasks.workunit.client.1.smithi153.stdout:6/863: sync 2024-04-16T02:35:47.522 INFO:tasks.workunit.client.1.smithi153.stdout:9/956: dread d4/d6/dc/d2d/d64/db9/fcf [0,4194304] 0 2024-04-16T02:35:47.522 INFO:tasks.workunit.client.0.smithi151.stdout:2/410: dread de/d11/d41/d35/f74 [0,4194304] 0 2024-04-16T02:35:47.524 INFO:tasks.workunit.client.1.smithi153.stdout:3/902: dread d6/d10/d3b/d10d/d8a/f101 [0,4194304] 0 2024-04-16T02:35:47.524 INFO:tasks.workunit.client.1.smithi153.stdout:6/864: symlink dc/d47/d80/l11e 0 2024-04-16T02:35:47.525 INFO:tasks.workunit.client.0.smithi151.stdout:7/360: dwrite d0/d1f/f2a [0,4194304] 0 2024-04-16T02:35:47.525 INFO:tasks.workunit.client.0.smithi151.stdout:9/303: chown d1/d3/f33 4477 1 2024-04-16T02:35:47.525 INFO:tasks.workunit.client.1.smithi153.stdout:3/903: stat d6/d10/c68 0 2024-04-16T02:35:47.526 INFO:tasks.workunit.client.1.smithi153.stdout:7/879: mknod d0/d4/db/d10a/c12f 0 2024-04-16T02:35:47.526 INFO:tasks.workunit.client.1.smithi153.stdout:6/865: truncate f3 737223 0 2024-04-16T02:35:47.526 INFO:tasks.workunit.client.0.smithi151.stdout:0/296: dwrite dc/f39 [0,4194304] 0 2024-04-16T02:35:47.529 INFO:tasks.workunit.client.0.smithi151.stdout:4/395: write d0/d11/d19/d31/d4a/d60/f70 [3254169,119818] 0 2024-04-16T02:35:47.529 INFO:tasks.workunit.client.0.smithi151.stdout:9/304: chown d1/d3/c8 3 1 2024-04-16T02:35:47.531 INFO:tasks.workunit.client.0.smithi151.stdout:0/297: stat dc/d3c/f3f 0 2024-04-16T02:35:47.531 INFO:tasks.workunit.client.1.smithi153.stdout:6/866: fsync dc/d1a/d2d/f44 0 2024-04-16T02:35:47.532 INFO:tasks.workunit.client.0.smithi151.stdout:6/522: dread - d2/d5/df/d20/d4f/f9e zero size 2024-04-16T02:35:47.532 INFO:tasks.workunit.client.0.smithi151.stdout:7/361: chown d0/l8 2202 1 2024-04-16T02:35:47.532 INFO:tasks.workunit.client.1.smithi153.stdout:6/867: fsync dc/d47/d57/d77/d7a/d97/db7/ffd 0 2024-04-16T02:35:47.535 INFO:tasks.workunit.client.1.smithi153.stdout:9/957: rename d4/d6/d1b/l35 to d4/d6/dc/d24/l132 0 2024-04-16T02:35:47.536 INFO:tasks.workunit.client.1.smithi153.stdout:7/880: fsync d0/d4/db/d2b/d53/d59/f67 0 2024-04-16T02:35:47.539 INFO:tasks.workunit.client.0.smithi151.stdout:2/411: symlink de/d65/d71/d7c/l81 0 2024-04-16T02:35:47.539 INFO:tasks.workunit.client.0.smithi151.stdout:4/396: rmdir d0/d17 39 2024-04-16T02:35:47.539 INFO:tasks.workunit.client.1.smithi153.stdout:9/958: truncate d4/d6/d1b/d2a/d99/fa4 4655987 0 2024-04-16T02:35:47.541 INFO:tasks.workunit.client.1.smithi153.stdout:6/868: creat dc/d47/d57/d77/dc4/dc9/ded/f11f x:0 0 0 2024-04-16T02:35:47.543 INFO:tasks.workunit.client.1.smithi153.stdout:6/869: write dc/d47/d57/d77/d15/dac/dae/ffc [1920850,96707] 0 2024-04-16T02:35:47.543 INFO:tasks.workunit.client.0.smithi151.stdout:9/305: rename d1/d3/d6/d17/d54/d5b to d1/d3/d6/d17/d54/d5b/d5d 22 2024-04-16T02:35:47.546 INFO:tasks.workunit.client.0.smithi151.stdout:7/362: dread d0/d2c/f5f [0,4194304] 0 2024-04-16T02:35:47.546 INFO:tasks.workunit.client.0.smithi151.stdout:2/412: mkdir de/d11/d42/d82 0 2024-04-16T02:35:47.547 INFO:tasks.workunit.client.1.smithi153.stdout:7/881: creat d0/d4/db/d2b/d53/d59/d76/dc8/f130 x:0 0 0 2024-04-16T02:35:47.551 INFO:tasks.workunit.client.0.smithi151.stdout:0/298: dwrite dc/d20/f4a [0,4194304] 0 2024-04-16T02:35:47.551 INFO:tasks.workunit.client.1.smithi153.stdout:7/882: write d0/d4/db/d2b/fac [4655521,81904] 0 2024-04-16T02:35:47.552 INFO:tasks.workunit.client.1.smithi153.stdout:6/870: dwrite dc/d10/f6c [0,4194304] 0 2024-04-16T02:35:47.552 INFO:tasks.workunit.client.0.smithi151.stdout:9/306: stat d1/d3/d6/d17/c25 0 2024-04-16T02:35:47.553 INFO:tasks.workunit.client.0.smithi151.stdout:0/299: dread - dc/d15/f58 zero size 2024-04-16T02:35:47.553 INFO:tasks.workunit.client.0.smithi151.stdout:9/307: chown d1/f2e 97 1 2024-04-16T02:35:47.553 INFO:tasks.workunit.client.1.smithi153.stdout:2/917: dread d9/d2b/d1c/d20/f79 [0,4194304] 0 2024-04-16T02:35:47.554 INFO:tasks.workunit.client.0.smithi151.stdout:2/413: symlink de/l83 0 2024-04-16T02:35:47.554 INFO:tasks.workunit.client.0.smithi151.stdout:6/523: creat d2/d5/df/d23/d56/d72/fb5 x:0 0 0 2024-04-16T02:35:47.555 INFO:tasks.workunit.client.1.smithi153.stdout:3/904: dwrite d6/d10/d5b/f75 [0,4194304] 0 2024-04-16T02:35:47.558 INFO:tasks.workunit.client.0.smithi151.stdout:2/414: symlink de/d65/d71/l84 0 2024-04-16T02:35:47.561 INFO:tasks.workunit.client.0.smithi151.stdout:2/415: chown de/d11/d42/d6b 1 1 2024-04-16T02:35:47.562 INFO:tasks.workunit.client.0.smithi151.stdout:4/397: dread d0/d11/f26 [0,4194304] 0 2024-04-16T02:35:47.562 INFO:tasks.workunit.client.0.smithi151.stdout:6/524: mkdir d2/d5/d2e/db6 0 2024-04-16T02:35:47.563 INFO:tasks.workunit.client.1.smithi153.stdout:2/918: dread d9/d24/d27/d2c/d5d/f63 [0,4194304] 0 2024-04-16T02:35:47.563 INFO:tasks.workunit.client.1.smithi153.stdout:6/871: mknod dc/d6b/c120 0 2024-04-16T02:35:47.564 INFO:tasks.workunit.client.1.smithi153.stdout:6/872: dread - dc/f113 zero size 2024-04-16T02:35:47.566 INFO:tasks.workunit.client.1.smithi153.stdout:2/919: write d9/d71/d8d/fbd [5189917,8922] 0 2024-04-16T02:35:47.567 INFO:tasks.workunit.client.1.smithi153.stdout:2/920: fdatasync d9/fb 0 2024-04-16T02:35:47.570 INFO:tasks.workunit.client.0.smithi151.stdout:9/308: link d1/d3/d10/f4d d1/d3/f5e 0 2024-04-16T02:35:47.570 INFO:tasks.workunit.client.0.smithi151.stdout:0/300: dwrite dc/d20/f42 [0,4194304] 0 2024-04-16T02:35:47.573 INFO:tasks.workunit.client.0.smithi151.stdout:6/525: creat d2/d13/d6f/fb7 x:0 0 0 2024-04-16T02:35:47.573 INFO:tasks.workunit.client.1.smithi153.stdout:3/905: creat d6/d10/d22/d4d/f122 x:0 0 0 2024-04-16T02:35:47.575 INFO:tasks.workunit.client.1.smithi153.stdout:6/873: dread - dc/d47/d57/d77/d7a/d97/fcc zero size 2024-04-16T02:35:47.577 INFO:tasks.workunit.client.0.smithi151.stdout:4/398: mkdir d0/d20/d59/d86 0 2024-04-16T02:35:47.578 INFO:tasks.workunit.client.1.smithi153.stdout:3/906: truncate d6/d10/d3b/d10d/f4e 1144711 0 2024-04-16T02:35:47.578 INFO:tasks.workunit.client.0.smithi151.stdout:6/526: write d2/d5/df/d23/d56/d72/fa0 [28611,13277] 0 2024-04-16T02:35:47.579 INFO:tasks.workunit.client.0.smithi151.stdout:2/416: dwrite de/f14 [0,4194304] 0 2024-04-16T02:35:47.579 INFO:tasks.workunit.client.1.smithi153.stdout:2/921: unlink d9/d71/l9c 0 2024-04-16T02:35:47.581 INFO:tasks.workunit.client.0.smithi151.stdout:9/309: mkdir d1/d3/d5f 0 2024-04-16T02:35:47.581 INFO:tasks.workunit.client.0.smithi151.stdout:0/301: write f3 [1930396,71237] 0 2024-04-16T02:35:47.584 INFO:tasks.workunit.client.0.smithi151.stdout:6/527: write d2/f9f [264245,40177] 0 2024-04-16T02:35:47.585 INFO:tasks.workunit.client.1.smithi153.stdout:6/874: rename dc/d59/d5b/fe0 to dc/d47/d57/d77/d15/d109/f121 0 2024-04-16T02:35:47.585 INFO:tasks.workunit.client.0.smithi151.stdout:9/310: symlink d1/dd/d4a/l60 0 2024-04-16T02:35:47.588 INFO:tasks.workunit.client.1.smithi153.stdout:2/922: rename d9/d2b/d5c/lf8 to d9/d2b/d5c/d5f/l128 0 2024-04-16T02:35:47.590 INFO:tasks.workunit.client.0.smithi151.stdout:9/311: write d1/d3/d6/d1d/d22/d36/f3d [463352,111612] 0 2024-04-16T02:35:47.596 INFO:tasks.workunit.client.0.smithi151.stdout:0/302: dwrite dc/d15/d34/f54 [0,4194304] 0 2024-04-16T02:35:47.596 INFO:tasks.workunit.client.1.smithi153.stdout:2/923: dwrite d9/d71/d8d/f120 [0,4194304] 0 2024-04-16T02:35:47.598 INFO:tasks.workunit.client.0.smithi151.stdout:4/399: rename d0/d11/d19/d31/d34 to d0/d11/d19/d87 0 2024-04-16T02:35:47.600 INFO:tasks.workunit.client.1.smithi153.stdout:2/924: symlink d9/d24/d27/d2c/d41/l129 0 2024-04-16T02:35:47.600 INFO:tasks.workunit.client.0.smithi151.stdout:7/363: dread d0/f3 [0,4194304] 0 2024-04-16T02:35:47.602 INFO:tasks.workunit.client.1.smithi153.stdout:2/925: chown d9/d24/d34/d36/c5b 26271 1 2024-04-16T02:35:47.606 INFO:tasks.workunit.client.0.smithi151.stdout:9/312: creat d1/d3/d6/d17/d40/d56/f61 x:0 0 0 2024-04-16T02:35:47.606 INFO:tasks.workunit.client.0.smithi151.stdout:0/303: dread dc/d3c/f48 [0,4194304] 0 2024-04-16T02:35:47.607 INFO:tasks.workunit.client.1.smithi153.stdout:7/883: write d0/d4/db/f64 [4480601,15885] 0 2024-04-16T02:35:47.608 INFO:tasks.workunit.client.0.smithi151.stdout:4/400: read d0/d11/d19/d87/d3a/d50/f67 [47388,80809] 0 2024-04-16T02:35:47.610 INFO:tasks.workunit.client.1.smithi153.stdout:7/884: fdatasync d0/de8/d110/fc2 0 2024-04-16T02:35:47.611 INFO:tasks.workunit.client.1.smithi153.stdout:2/926: mknod d9/d2b/c12a 0 2024-04-16T02:35:47.612 INFO:tasks.workunit.client.1.smithi153.stdout:2/927: readlink d9/d2b/d18/l4d 0 2024-04-16T02:35:47.612 INFO:tasks.workunit.client.0.smithi151.stdout:7/364: creat d0/d1f/d4c/f81 x:0 0 0 2024-04-16T02:35:47.613 INFO:tasks.workunit.client.1.smithi153.stdout:2/928: write d9/d24/d27/d2c/d41/f78 [1265432,90774] 0 2024-04-16T02:35:47.615 INFO:tasks.workunit.client.1.smithi153.stdout:7/885: mknod d0/d4/db/d2b/d8c/c131 0 2024-04-16T02:35:47.616 INFO:tasks.workunit.client.0.smithi151.stdout:0/304: rmdir dc/d15/d34 39 2024-04-16T02:35:47.617 INFO:tasks.workunit.client.0.smithi151.stdout:2/417: truncate de/d11/d41/d35/f74 233936 0 2024-04-16T02:35:47.617 INFO:tasks.workunit.client.1.smithi153.stdout:2/929: mknod d9/d24/d27/d2c/d5d/de5/d124/c12b 0 2024-04-16T02:35:47.621 INFO:tasks.workunit.client.0.smithi151.stdout:2/418: dread de/d15/f17 [0,4194304] 0 2024-04-16T02:35:47.621 INFO:tasks.workunit.client.0.smithi151.stdout:6/528: write d2/d5/df/d20/d4f/fa7 [2050743,56747] 0 2024-04-16T02:35:47.622 INFO:tasks.workunit.client.0.smithi151.stdout:7/365: dwrite d0/d1f/d33/f3a [0,4194304] 0 2024-04-16T02:35:47.622 INFO:tasks.workunit.client.1.smithi153.stdout:7/886: getdents d0/d4/db/d34 0 2024-04-16T02:35:47.624 INFO:tasks.workunit.client.0.smithi151.stdout:9/313: mknod d1/d3/c62 0 2024-04-16T02:35:47.625 INFO:tasks.workunit.client.1.smithi153.stdout:2/930: creat d9/d24/d27/d2c/d5d/da3/dc0/d11f/f12c x:0 0 0 2024-04-16T02:35:47.625 INFO:tasks.workunit.client.0.smithi151.stdout:9/314: truncate d1/f55 385554 0 2024-04-16T02:35:47.626 INFO:tasks.workunit.client.1.smithi153.stdout:3/907: dread d6/d10/d3b/d59/d7c/de1/fe3 [0,4194304] 0 2024-04-16T02:35:47.626 INFO:tasks.workunit.client.1.smithi153.stdout:2/931: fdatasync d9/d24/d27/d2c/d5d/da3/fd3 0 2024-04-16T02:35:47.628 INFO:tasks.workunit.client.0.smithi151.stdout:0/305: mkdir dc/d15/d5c 0 2024-04-16T02:35:47.628 INFO:tasks.workunit.client.1.smithi153.stdout:7/887: mknod d0/d4/db/d47/d8d/d93/c132 0 2024-04-16T02:35:47.630 INFO:tasks.workunit.client.1.smithi153.stdout:2/932: creat d9/d2b/d1c/d20/d88/f12d x:0 0 0 2024-04-16T02:35:47.633 INFO:tasks.workunit.client.0.smithi151.stdout:7/366: mkdir d0/d1f/d4c/d82 0 2024-04-16T02:35:47.633 INFO:tasks.workunit.client.1.smithi153.stdout:3/908: truncate d6/d10/d22/d2c/f5f 4274320 0 2024-04-16T02:35:47.635 INFO:tasks.workunit.client.1.smithi153.stdout:3/909: chown d6/d10/d3b/d10d/d8a/f91 570 1 2024-04-16T02:35:47.635 INFO:tasks.workunit.client.1.smithi153.stdout:2/933: truncate d9/d24/d27/d2c/d41/fb5 934368 0 2024-04-16T02:35:47.635 INFO:tasks.workunit.client.0.smithi151.stdout:2/419: dread de/d11/d41/f32 [0,4194304] 0 2024-04-16T02:35:47.636 INFO:tasks.workunit.client.0.smithi151.stdout:6/529: unlink d2/d5/df/d23/d82/d8d/f26 0 2024-04-16T02:35:47.636 INFO:tasks.workunit.client.1.smithi153.stdout:3/910: write d6/d10/d3b/d76/f7d [871002,29734] 0 2024-04-16T02:35:47.637 INFO:tasks.workunit.client.1.smithi153.stdout:3/911: chown d6/f11 173044112 1 2024-04-16T02:35:47.638 INFO:tasks.workunit.client.0.smithi151.stdout:6/530: dread - d2/d5/df/d23/d82/f9c zero size 2024-04-16T02:35:47.639 INFO:tasks.workunit.client.0.smithi151.stdout:6/531: chown d2/d13/d3a 9421370 1 2024-04-16T02:35:47.640 INFO:tasks.workunit.client.1.smithi153.stdout:3/912: symlink d6/de6/l123 0 2024-04-16T02:35:47.641 INFO:tasks.workunit.client.0.smithi151.stdout:7/367: creat d0/d27/f83 x:0 0 0 2024-04-16T02:35:47.641 INFO:tasks.workunit.client.0.smithi151.stdout:6/532: fdatasync d2/d5/df/d23/d56/f84 0 2024-04-16T02:35:47.641 INFO:tasks.workunit.client.1.smithi153.stdout:2/934: getdents d9/d24/d34/d99/d126 0 2024-04-16T02:35:47.643 INFO:tasks.workunit.client.1.smithi153.stdout:3/913: mknod d6/d10/d3b/d10d/c124 0 2024-04-16T02:35:47.643 INFO:tasks.workunit.client.1.smithi153.stdout:1/894: sync 2024-04-16T02:35:47.643 INFO:tasks.workunit.client.1.smithi153.stdout:5/945: sync 2024-04-16T02:35:47.643 INFO:tasks.workunit.client.1.smithi153.stdout:2/935: write d9/d24/d27/f92 [1834705,60888] 0 2024-04-16T02:35:47.643 INFO:tasks.workunit.client.0.smithi151.stdout:4/401: getdents d0/d20 0 2024-04-16T02:35:47.643 INFO:tasks.workunit.client.0.smithi151.stdout:2/420: rename de/d11/d42/d6b/f6e to de/d11/d42/d7a/f85 0 2024-04-16T02:35:47.644 INFO:tasks.workunit.client.1.smithi153.stdout:1/895: read d4/f7 [1060491,2221] 0 2024-04-16T02:35:47.646 INFO:tasks.workunit.client.0.smithi151.stdout:9/315: mknod d1/dd/c63 0 2024-04-16T02:35:47.649 INFO:tasks.workunit.client.0.smithi151.stdout:2/421: fdatasync de/d11/d42/d7a/f85 0 2024-04-16T02:35:47.650 INFO:tasks.workunit.client.1.smithi153.stdout:2/936: rename d9/d24/caa to d9/d24/d27/d2c/d41/d7a/d116/c12e 0 2024-04-16T02:35:47.651 INFO:tasks.workunit.client.0.smithi151.stdout:4/402: mknod d0/d11/d19/d87/d3a/d65/c88 0 2024-04-16T02:35:47.652 INFO:tasks.workunit.client.0.smithi151.stdout:7/368: dwrite d0/d2c/f3b [0,4194304] 0 2024-04-16T02:35:47.655 INFO:tasks.workunit.client.0.smithi151.stdout:6/533: rename d2/f9f to d2/d5/df/d23/d82/d8d/d22/d6d/fb8 0 2024-04-16T02:35:47.656 INFO:tasks.workunit.client.0.smithi151.stdout:0/306: creat dc/d15/d34/f5d x:0 0 0 2024-04-16T02:35:47.658 INFO:tasks.workunit.client.1.smithi153.stdout:5/946: dwrite d8/df/d28/d5a/d72/dd0/fe0 [0,4194304] 0 2024-04-16T02:35:47.660 INFO:tasks.workunit.client.0.smithi151.stdout:0/307: creat dc/d20/d2c/f5e x:0 0 0 2024-04-16T02:35:47.662 INFO:tasks.workunit.client.0.smithi151.stdout:6/534: mknod d2/d5/df/d23/d82/d8d/d22/d40/d3f/d65/cb9 0 2024-04-16T02:35:47.662 INFO:tasks.workunit.client.0.smithi151.stdout:0/308: write dc/d20/d2c/f3e [295018,79648] 0 2024-04-16T02:35:47.663 INFO:tasks.workunit.client.1.smithi153.stdout:7/888: dread d0/d4/db/d16/d126/fda [0,4194304] 0 2024-04-16T02:35:47.665 INFO:tasks.workunit.client.1.smithi153.stdout:3/914: link d6/d10/d3b/d10d/d8a/fc3 d6/d10/d22/d2c/f125 0 2024-04-16T02:35:47.668 INFO:tasks.workunit.client.1.smithi153.stdout:7/889: write d0/d4/db/d47/d8d/d93/f129 [54055,67940] 0 2024-04-16T02:35:47.668 INFO:tasks.workunit.client.1.smithi153.stdout:5/947: mknod d8/d41/dbd/d39/d6d/c127 0 2024-04-16T02:35:47.668 INFO:tasks.workunit.client.1.smithi153.stdout:7/890: write d0/d13/f12b [979933,84877] 0 2024-04-16T02:35:47.668 INFO:tasks.workunit.client.0.smithi151.stdout:7/369: creat d0/d1f/d33/d56/f84 x:0 0 0 2024-04-16T02:35:47.669 INFO:tasks.workunit.client.1.smithi153.stdout:5/948: write d8/d41/dbd/d39/f4c [16223,21860] 0 2024-04-16T02:35:47.670 INFO:tasks.workunit.client.1.smithi153.stdout:3/915: unlink d6/d10/d11f/cfd 0 2024-04-16T02:35:47.671 INFO:tasks.workunit.client.1.smithi153.stdout:3/916: write d6/f2d [4265101,103379] 0 2024-04-16T02:35:47.674 INFO:tasks.workunit.client.0.smithi151.stdout:4/403: rename d0/d17 to d0/d11/d19/d31/d89 0 2024-04-16T02:35:47.675 INFO:tasks.workunit.client.1.smithi153.stdout:2/937: dread d9/d2b/d18/d3f/f6e [0,4194304] 0 2024-04-16T02:35:47.675 INFO:tasks.workunit.client.0.smithi151.stdout:6/535: mkdir d2/d5/d2e/dba 0 2024-04-16T02:35:47.676 INFO:tasks.workunit.client.1.smithi153.stdout:3/917: mknod d6/d10/d3b/d59/d7c/de1/df0/def/c126 0 2024-04-16T02:35:47.678 INFO:tasks.workunit.client.0.smithi151.stdout:2/422: write de/d15/d1e/f3f [2810740,118564] 0 2024-04-16T02:35:47.678 INFO:tasks.workunit.client.0.smithi151.stdout:0/309: write dc/d3c/f48 [701911,111524] 0 2024-04-16T02:35:47.679 INFO:tasks.workunit.client.1.smithi153.stdout:7/891: dwrite d0/d4/db/d2b/d53/d59/d76/dc8/fb4 [0,4194304] 0 2024-04-16T02:35:47.680 INFO:tasks.workunit.client.1.smithi153.stdout:3/918: write d6/d10/d3b/d10d/d8a/ff7 [886429,119345] 0 2024-04-16T02:35:47.681 INFO:tasks.workunit.client.0.smithi151.stdout:9/316: dwrite d1/d3/d6/d1d/d22/d2c/f2d [0,4194304] 0 2024-04-16T02:35:47.682 INFO:tasks.workunit.client.1.smithi153.stdout:1/896: truncate d4/dd/d18/d29/d6e/fb2 245743 0 2024-04-16T02:35:47.684 INFO:tasks.workunit.client.0.smithi151.stdout:2/423: creat de/d11/d42/d5e/f86 x:0 0 0 2024-04-16T02:35:47.686 INFO:tasks.workunit.client.0.smithi151.stdout:2/424: stat de/d15/c1f 0 2024-04-16T02:35:47.686 INFO:tasks.workunit.client.1.smithi153.stdout:3/919: mkdir d6/d10/d5b/d127 0 2024-04-16T02:35:47.687 INFO:tasks.workunit.client.0.smithi151.stdout:4/404: truncate d0/d11/d19/d31/d68/f79 83187 0 2024-04-16T02:35:47.687 INFO:tasks.workunit.client.1.smithi153.stdout:1/897: chown d4/dd/d13/d23/fd9 178549321 1 2024-04-16T02:35:47.688 INFO:tasks.workunit.client.1.smithi153.stdout:3/920: chown d6/d10/d3b/d59/fbe 38503 1 2024-04-16T02:35:47.688 INFO:tasks.workunit.client.0.smithi151.stdout:7/370: dwrite d0/d1f/d33/d56/f84 [0,4194304] 0 2024-04-16T02:35:47.691 INFO:tasks.workunit.client.1.smithi153.stdout:7/892: truncate d0/d4/db/d2b/d43/f9d 5322087 0 2024-04-16T02:35:47.691 INFO:tasks.workunit.client.0.smithi151.stdout:6/536: unlink d2/d5/df/d23/d82/d8d/f29 0 2024-04-16T02:35:47.691 INFO:tasks.workunit.client.0.smithi151.stdout:0/310: creat dc/d20/d2c/f5f x:0 0 0 2024-04-16T02:35:47.692 INFO:tasks.workunit.client.1.smithi153.stdout:3/921: write d6/d10/d3b/d10d/d8a/fb6 [688596,8552] 0 2024-04-16T02:35:47.692 INFO:tasks.workunit.client.0.smithi151.stdout:0/311: chown dc/d4c 5 1 2024-04-16T02:35:47.692 INFO:tasks.workunit.client.0.smithi151.stdout:9/317: mkdir d1/dd/d4a/d64 0 2024-04-16T02:35:47.693 INFO:tasks.workunit.client.0.smithi151.stdout:0/312: write dc/d15/d34/f5d [742563,17293] 0 2024-04-16T02:35:47.695 INFO:tasks.workunit.client.1.smithi153.stdout:1/898: creat d4/f12e x:0 0 0 2024-04-16T02:35:47.696 INFO:tasks.workunit.client.1.smithi153.stdout:1/899: readlink d4/dd/d13/d26/l8a 0 2024-04-16T02:35:47.696 INFO:tasks.workunit.client.0.smithi151.stdout:9/318: write d1/d3/d6/d17/d40/d56/f61 [348424,58186] 0 2024-04-16T02:35:47.698 INFO:tasks.workunit.client.1.smithi153.stdout:3/922: mknod d6/d10/d3b/d59/d7c/db7/c128 0 2024-04-16T02:35:47.699 INFO:tasks.workunit.client.0.smithi151.stdout:9/319: write d1/d3/d6/d28/d39/f48 [874050,19371] 0 2024-04-16T02:35:47.700 INFO:tasks.workunit.client.0.smithi151.stdout:2/425: rename de/d11/d41/c54 to de/d15/c87 0 2024-04-16T02:35:47.700 INFO:tasks.workunit.client.0.smithi151.stdout:4/405: mknod d0/d11/d19/d87/c8a 0 2024-04-16T02:35:47.702 INFO:tasks.workunit.client.0.smithi151.stdout:6/537: symlink d2/d5/df/d20/lbb 0 2024-04-16T02:35:47.703 INFO:tasks.workunit.client.0.smithi151.stdout:7/371: dwrite d0/f3 [8388608,4194304] 0 2024-04-16T02:35:47.708 INFO:tasks.workunit.client.1.smithi153.stdout:7/893: dwrite d0/d13/f4f [8388608,4194304] 0 2024-04-16T02:35:47.710 INFO:tasks.workunit.client.0.smithi151.stdout:2/426: mkdir de/d65/d88 0 2024-04-16T02:35:47.711 INFO:tasks.workunit.client.0.smithi151.stdout:6/538: symlink d2/d5/df/d23/d82/d8d/d22/lbc 0 2024-04-16T02:35:47.711 INFO:tasks.workunit.client.0.smithi151.stdout:0/313: truncate dc/d15/f17 1184487 0 2024-04-16T02:35:47.718 INFO:tasks.workunit.client.1.smithi153.stdout:3/923: dwrite d6/d10/d3b/d10d/d8a/d8c/f11a [0,4194304] 0 2024-04-16T02:35:47.718 INFO:tasks.workunit.client.0.smithi151.stdout:7/372: dwrite d0/d27/f7a [0,4194304] 0 2024-04-16T02:35:47.718 INFO:tasks.workunit.client.1.smithi153.stdout:7/894: dread - d0/d4/db/d47/f10c zero size 2024-04-16T02:35:47.721 INFO:tasks.workunit.client.1.smithi153.stdout:7/895: dread - d0/d4/db/d2b/d43/d128/d6c/db8/fe5 zero size 2024-04-16T02:35:47.722 INFO:tasks.workunit.client.1.smithi153.stdout:7/896: write d0/d4/db/d2b/d53/d59/f127 [437983,122906] 0 2024-04-16T02:35:47.723 INFO:tasks.workunit.client.0.smithi151.stdout:9/320: link d1/d3/d6/d17/d40/c50 d1/d3/d10/d20/c65 0 2024-04-16T02:35:47.725 INFO:tasks.workunit.client.0.smithi151.stdout:4/406: getdents d0/d11/d19/d31/d80 0 2024-04-16T02:35:47.732 INFO:tasks.workunit.client.1.smithi153.stdout:2/938: dread d9/d24/d27/f1d [4194304,4194304] 0 2024-04-16T02:35:47.736 INFO:tasks.workunit.client.1.smithi153.stdout:1/900: dwrite d4/dd/d13/f2a [4194304,4194304] 0 2024-04-16T02:35:47.737 INFO:tasks.workunit.client.0.smithi151.stdout:7/373: getdents d0/d2c/d38/d5e 0 2024-04-16T02:35:47.737 INFO:tasks.workunit.client.0.smithi151.stdout:4/407: creat d0/d11/d19/f8b x:0 0 0 2024-04-16T02:35:47.743 INFO:tasks.workunit.client.0.smithi151.stdout:7/374: rename d0/d35 to d0/d2c/d38/d85 0 2024-04-16T02:35:47.745 INFO:tasks.workunit.client.0.smithi151.stdout:7/375: write d0/d1f/d46/d4f/f57 [262626,32264] 0 2024-04-16T02:35:47.745 INFO:tasks.workunit.client.0.smithi151.stdout:4/408: mknod d0/d11/d19/d87/d3a/d50/c8c 0 2024-04-16T02:35:47.749 INFO:tasks.workunit.client.0.smithi151.stdout:2/427: truncate de/f14 7329667 0 2024-04-16T02:35:47.749 INFO:tasks.workunit.client.0.smithi151.stdout:8/287: sync 2024-04-16T02:35:47.753 INFO:tasks.workunit.client.0.smithi151.stdout:4/409: rename d0/d11/d19/f29 to d0/d11/f8d 0 2024-04-16T02:35:47.754 INFO:tasks.workunit.client.0.smithi151.stdout:6/539: truncate d2/d5/df/d20/d4f/f6e 4081439 0 2024-04-16T02:35:47.780 INFO:tasks.workunit.client.1.smithi153.stdout:8/861: sync 2024-04-16T02:35:47.781 INFO:tasks.workunit.client.1.smithi153.stdout:8/862: fsync d2/d13/d2c/ffd 0 2024-04-16T02:35:47.782 INFO:tasks.workunit.client.1.smithi153.stdout:8/863: chown d2/d13/d96/dc7/d109/d47/d77/ddf/f111 118 1 2024-04-16T02:35:47.817 INFO:tasks.workunit.client.1.smithi153.stdout:6/875: sync 2024-04-16T02:35:47.817 INFO:tasks.workunit.client.1.smithi153.stdout:9/959: sync 2024-04-16T02:35:47.819 INFO:tasks.workunit.client.1.smithi153.stdout:6/876: read dc/d47/d57/d77/dc4/dc9/ded/f10e [95982,86364] 0 2024-04-16T02:35:47.820 INFO:tasks.workunit.client.1.smithi153.stdout:6/877: read f3 [224579,20543] 0 2024-04-16T02:35:47.843 INFO:tasks.workunit.client.0.smithi151.stdout:2/428: dread de/d11/d41/d35/f74 [0,4194304] 0 2024-04-16T02:35:47.844 INFO:tasks.workunit.client.0.smithi151.stdout:0/314: dwrite dc/d15/f56 [0,4194304] 0 2024-04-16T02:35:47.845 INFO:tasks.workunit.client.0.smithi151.stdout:6/540: mknod d2/d13/d3a/cbd 0 2024-04-16T02:35:47.853 INFO:tasks.workunit.client.0.smithi151.stdout:9/321: dwrite d1/dd/f16 [0,4194304] 0 2024-04-16T02:35:47.859 INFO:tasks.workunit.client.0.smithi151.stdout:6/541: truncate d2/d5/f8 342720 0 2024-04-16T02:35:47.862 INFO:tasks.workunit.client.0.smithi151.stdout:0/315: mknod dc/d15/c60 0 2024-04-16T02:35:47.862 INFO:tasks.workunit.client.0.smithi151.stdout:9/322: read d1/d3/d6/d1d/d22/f23 [1030405,45910] 0 2024-04-16T02:35:47.863 INFO:tasks.workunit.client.0.smithi151.stdout:6/542: write d2/d5/df/d23/d56/d72/fac [562321,39486] 0 2024-04-16T02:35:47.863 INFO:tasks.workunit.client.0.smithi151.stdout:4/410: write d0/d11/d19/d87/d3a/d50/f6b [227282,9480] 0 2024-04-16T02:35:47.865 INFO:tasks.workunit.client.0.smithi151.stdout:4/411: chown d0/d11/d19/d87/d3a 83145176 1 2024-04-16T02:35:47.866 INFO:tasks.workunit.client.1.smithi153.stdout:7/897: mknod d0/d4/db/d2b/d53/d59/d76/dc8/d101/c133 0 2024-04-16T02:35:47.866 INFO:tasks.workunit.client.1.smithi153.stdout:2/939: symlink d9/d24/d27/d2c/d5d/de5/d124/l12f 0 2024-04-16T02:35:47.868 INFO:tasks.workunit.client.0.smithi151.stdout:9/323: creat d1/d3/d6/d28/d39/d43/f66 x:0 0 0 2024-04-16T02:35:47.869 INFO:tasks.workunit.client.0.smithi151.stdout:4/412: write d0/d11/d19/d87/d3a/f75 [437255,73334] 0 2024-04-16T02:35:47.873 INFO:tasks.workunit.client.0.smithi151.stdout:0/316: symlink dc/d15/d34/d3b/l61 0 2024-04-16T02:35:47.873 INFO:tasks.workunit.client.0.smithi151.stdout:4/413: symlink d0/d20/l8e 0 2024-04-16T02:35:47.874 INFO:tasks.workunit.client.0.smithi151.stdout:9/324: creat d1/dd/d4a/d64/f67 x:0 0 0 2024-04-16T02:35:47.875 INFO:tasks.workunit.client.1.smithi153.stdout:6/878: symlink dc/d47/d93/l122 0 2024-04-16T02:35:47.875 INFO:tasks.workunit.client.0.smithi151.stdout:4/414: readlink d0/d11/l14 0 2024-04-16T02:35:47.876 INFO:tasks.workunit.client.0.smithi151.stdout:0/317: creat dc/d20/d2c/d51/f62 x:0 0 0 2024-04-16T02:35:47.877 INFO:tasks.workunit.client.0.smithi151.stdout:9/325: stat d1/d3/d6/d28/d39/d43 0 2024-04-16T02:35:47.879 INFO:tasks.workunit.client.1.smithi153.stdout:7/898: truncate d0/d4/db/d2b/f40 1539959 0 2024-04-16T02:35:47.880 INFO:tasks.workunit.client.0.smithi151.stdout:2/429: dwrite f7 [0,4194304] 0 2024-04-16T02:35:47.880 INFO:tasks.workunit.client.1.smithi153.stdout:7/899: chown d0/de8/d110/d4d/ff9 2759 1 2024-04-16T02:35:47.882 INFO:tasks.workunit.client.1.smithi153.stdout:2/940: unlink d9/d24/d27/d2c/d5d/da3/dc0/fd1 0 2024-04-16T02:35:47.882 INFO:tasks.workunit.client.0.smithi151.stdout:9/326: creat d1/d3/d5a/f68 x:0 0 0 2024-04-16T02:35:47.883 INFO:tasks.workunit.client.1.smithi153.stdout:2/941: write d9/d71/d8d/f120 [1970220,59572] 0 2024-04-16T02:35:47.883 INFO:tasks.workunit.client.0.smithi151.stdout:2/430: chown de/d15/d1e/f3c 4949540 1 2024-04-16T02:35:47.885 INFO:tasks.workunit.client.1.smithi153.stdout:9/960: symlink d4/d6/dc/d2d/d64/db9/dc1/d119/l133 0 2024-04-16T02:35:47.886 INFO:tasks.workunit.client.0.smithi151.stdout:0/318: link dc/d20/d2c/f30 dc/d15/d34/d4e/f63 0 2024-04-16T02:35:47.888 INFO:tasks.workunit.client.0.smithi151.stdout:0/319: fsync dc/d20/f49 0 2024-04-16T02:35:47.888 INFO:tasks.workunit.client.0.smithi151.stdout:4/415: dwrite d0/d11/d19/f8b [0,4194304] 0 2024-04-16T02:35:47.890 INFO:tasks.workunit.client.1.smithi153.stdout:3/924: rename d6/d10/d22/d2c/f98 to d6/d10/d3b/d59/d7c/de1/f129 0 2024-04-16T02:35:47.890 INFO:tasks.workunit.client.0.smithi151.stdout:9/327: truncate d1/d3/d6/d28/d39/d43/f66 570166 0 2024-04-16T02:35:47.891 INFO:tasks.workunit.client.0.smithi151.stdout:4/416: write d0/d11/d19/d31/d68/f79 [441655,74556] 0 2024-04-16T02:35:47.892 INFO:tasks.workunit.client.0.smithi151.stdout:9/328: chown d1/dd/d4a/f53 36 1 2024-04-16T02:35:47.893 INFO:tasks.workunit.client.0.smithi151.stdout:9/329: fsync d1/d3/d6/d28/d39/f48 0 2024-04-16T02:35:47.897 INFO:tasks.workunit.client.0.smithi151.stdout:2/431: getdents de/d11/d42 0 2024-04-16T02:35:47.901 INFO:tasks.workunit.client.0.smithi151.stdout:9/330: creat d1/d3/d5a/f69 x:0 0 0 2024-04-16T02:35:47.902 INFO:tasks.workunit.client.0.smithi151.stdout:9/331: stat d1 0 2024-04-16T02:35:47.904 INFO:tasks.workunit.client.0.smithi151.stdout:4/417: dwrite d0/f15 [4194304,4194304] 0 2024-04-16T02:35:47.905 INFO:tasks.workunit.client.0.smithi151.stdout:9/332: chown d1/d3/d10/f4b 27672 1 2024-04-16T02:35:47.906 INFO:tasks.workunit.client.0.smithi151.stdout:0/320: dwrite dc/d15/f58 [0,4194304] 0 2024-04-16T02:35:47.908 INFO:tasks.workunit.client.1.smithi153.stdout:2/942: creat d9/d24/d27/d113/f130 x:0 0 0 2024-04-16T02:35:47.908 INFO:tasks.workunit.client.0.smithi151.stdout:4/418: write d0/d20/f21 [2779978,54218] 0 2024-04-16T02:35:47.912 INFO:tasks.workunit.client.0.smithi151.stdout:0/321: dread - dc/d20/d2c/f5f zero size 2024-04-16T02:35:47.913 INFO:tasks.workunit.client.0.smithi151.stdout:2/432: dwrite de/d15/d1e/f67 [0,4194304] 0 2024-04-16T02:35:47.915 INFO:tasks.workunit.client.1.smithi153.stdout:5/949: sync 2024-04-16T02:35:47.916 INFO:tasks.workunit.client.0.smithi151.stdout:4/419: unlink d0/fe 0 2024-04-16T02:35:47.917 INFO:tasks.workunit.client.0.smithi151.stdout:2/433: rmdir de/d65/d71/d7c 39 2024-04-16T02:35:47.918 INFO:tasks.workunit.client.0.smithi151.stdout:2/434: readlink de/d15/d1e/l61 0 2024-04-16T02:35:47.919 INFO:tasks.workunit.client.0.smithi151.stdout:0/322: dread dc/d3c/f48 [0,4194304] 0 2024-04-16T02:35:47.920 INFO:tasks.workunit.client.1.smithi153.stdout:3/925: dread d6/d10/d5b/f82 [0,4194304] 0 2024-04-16T02:35:47.921 INFO:tasks.workunit.client.1.smithi153.stdout:9/961: link d4/l31 d4/d6/dc/d2d/d79/dbb/l134 0 2024-04-16T02:35:47.923 INFO:tasks.workunit.client.1.smithi153.stdout:9/962: fdatasync d4/d6/d1b/ddf/f102 0 2024-04-16T02:35:47.923 INFO:tasks.workunit.client.0.smithi151.stdout:0/323: mknod dc/d20/c64 0 2024-04-16T02:35:47.924 INFO:tasks.workunit.client.1.smithi153.stdout:9/963: chown d4/d33/dae/dc7/d66/fb2 54 1 2024-04-16T02:35:47.924 INFO:tasks.workunit.client.0.smithi151.stdout:0/324: chown dc/d15/f40 10977920 1 2024-04-16T02:35:47.926 INFO:tasks.workunit.client.0.smithi151.stdout:2/435: link de/d15/d1e/f67 de/d15/f89 0 2024-04-16T02:35:47.926 INFO:tasks.workunit.client.0.smithi151.stdout:0/325: fsync dc/d20/f4a 0 2024-04-16T02:35:47.926 INFO:tasks.workunit.client.1.smithi153.stdout:3/926: unlink d6/d10/d3b/d59/d7c/l8f 0 2024-04-16T02:35:47.928 INFO:tasks.workunit.client.0.smithi151.stdout:4/420: dwrite d0/d64/f71 [0,4194304] 0 2024-04-16T02:35:47.929 INFO:tasks.workunit.client.0.smithi151.stdout:2/436: stat de/d65/d71/d7c 0 2024-04-16T02:35:47.930 INFO:tasks.workunit.client.1.smithi153.stdout:8/864: write d2/f14 [35578,41971] 0 2024-04-16T02:35:47.932 INFO:tasks.workunit.client.0.smithi151.stdout:0/326: symlink dc/l65 0 2024-04-16T02:35:47.932 INFO:tasks.workunit.client.1.smithi153.stdout:5/950: link d8/df/d2f/c44 d8/df/d23/d10e/c128 0 2024-04-16T02:35:47.933 INFO:tasks.workunit.client.1.smithi153.stdout:5/951: dread - d8/df/d28/d5a/d72/dc0/ff8 zero size 2024-04-16T02:35:47.933 INFO:tasks.workunit.client.0.smithi151.stdout:2/437: mknod de/c8a 0 2024-04-16T02:35:47.933 INFO:tasks.workunit.client.1.smithi153.stdout:8/865: creat d2/d13/d96/dc7/f123 x:0 0 0 2024-04-16T02:35:47.935 INFO:tasks.workunit.client.1.smithi153.stdout:1/901: truncate d4/dd/d13/d23/d5f/f7f 235446 0 2024-04-16T02:35:47.936 INFO:tasks.workunit.client.0.smithi151.stdout:0/327: symlink dc/d15/d5c/l66 0 2024-04-16T02:35:47.937 INFO:tasks.workunit.client.0.smithi151.stdout:4/421: getdents d0/d11/d19/d87/d3a/d50 0 2024-04-16T02:35:47.939 INFO:tasks.workunit.client.1.smithi153.stdout:5/952: unlink d8/d41/dbd/d39/d6d/c127 0 2024-04-16T02:35:47.939 INFO:tasks.workunit.client.0.smithi151.stdout:0/328: write dc/d20/d2c/f5e [399544,99674] 0 2024-04-16T02:35:47.939 INFO:tasks.workunit.client.0.smithi151.stdout:2/438: rename de/d11/d41/d35 to de/d11/d42/d7a/d8b 0 2024-04-16T02:35:47.942 INFO:tasks.workunit.client.0.smithi151.stdout:4/422: mknod d0/d11/d19/d31/d4a/d60/c8f 0 2024-04-16T02:35:47.943 INFO:tasks.workunit.client.1.smithi153.stdout:8/866: unlink d2/d13/d3a/dc2/le6 0 2024-04-16T02:35:47.945 INFO:tasks.workunit.client.0.smithi151.stdout:2/439: write de/d15/f89 [4279749,17318] 0 2024-04-16T02:35:47.948 INFO:tasks.workunit.client.1.smithi153.stdout:8/867: read d2/d1c/f26 [1570751,114486] 0 2024-04-16T02:35:47.952 INFO:tasks.workunit.client.1.smithi153.stdout:2/943: write d9/d2b/d1c/d20/d88/f9b [299370,97681] 0 2024-04-16T02:35:47.952 INFO:tasks.workunit.client.0.smithi151.stdout:2/440: chown de/l1b 2175 1 2024-04-16T02:35:47.953 INFO:tasks.workunit.client.0.smithi151.stdout:2/441: chown de/d11/d42/l4d 816307 1 2024-04-16T02:35:47.955 INFO:tasks.workunit.client.0.smithi151.stdout:0/329: rename dc/d20/d2c to dc/d15/d34/d67 0 2024-04-16T02:35:47.957 INFO:tasks.workunit.client.1.smithi153.stdout:3/927: write d6/d10/d3b/fab [1715595,29471] 0 2024-04-16T02:35:47.957 INFO:tasks.workunit.client.1.smithi153.stdout:9/964: write d4/d6/dc/d2d/d79/f8f [211495,87562] 0 2024-04-16T02:35:47.957 INFO:tasks.workunit.client.0.smithi151.stdout:0/330: write dc/d15/d34/d67/f5f [667331,79421] 0 2024-04-16T02:35:47.960 INFO:tasks.workunit.client.1.smithi153.stdout:5/953: dwrite d8/df/d23/f56 [0,4194304] 0 2024-04-16T02:35:47.960 INFO:tasks.workunit.client.1.smithi153.stdout:1/902: write d4/dd/d13/d23/d31/f8b [617859,48976] 0 2024-04-16T02:35:47.963 INFO:tasks.workunit.client.0.smithi151.stdout:0/331: symlink dc/l68 0 2024-04-16T02:35:47.963 INFO:tasks.workunit.client.1.smithi153.stdout:6/879: dwrite dc/d47/d57/d77/f85 [4194304,4194304] 0 2024-04-16T02:35:47.965 INFO:tasks.workunit.client.1.smithi153.stdout:2/944: creat d9/d24/d34/d99/d126/f131 x:0 0 0 2024-04-16T02:35:47.966 INFO:tasks.workunit.client.0.smithi151.stdout:0/332: mkdir dc/d15/d34/d67/d69 0 2024-04-16T02:35:47.967 INFO:tasks.workunit.client.0.smithi151.stdout:0/333: fsync dc/f47 0 2024-04-16T02:35:47.970 INFO:tasks.workunit.client.1.smithi153.stdout:7/900: dwrite d0/d13/f32 [0,4194304] 0 2024-04-16T02:35:47.971 INFO:tasks.workunit.client.1.smithi153.stdout:9/965: creat d4/d6/ddb/d4a/d101/f135 x:0 0 0 2024-04-16T02:35:47.971 INFO:tasks.workunit.client.0.smithi151.stdout:0/334: creat dc/d15/d5c/f6a x:0 0 0 2024-04-16T02:35:47.971 INFO:tasks.workunit.client.0.smithi151.stdout:0/335: stat dc/f39 0 2024-04-16T02:35:47.973 INFO:tasks.workunit.client.0.smithi151.stdout:2/442: dwrite de/d15/d1e/f3c [0,4194304] 0 2024-04-16T02:35:47.973 INFO:tasks.workunit.client.0.smithi151.stdout:0/336: truncate f3 4294970 0 2024-04-16T02:35:47.974 INFO:tasks.workunit.client.1.smithi153.stdout:3/928: dread - d6/d10/d22/d4d/fda zero size 2024-04-16T02:35:47.981 INFO:tasks.workunit.client.1.smithi153.stdout:8/868: dwrite d2/d93/d97/d9c/d27/d5e/fcd [0,4194304] 0 2024-04-16T02:35:47.982 INFO:tasks.workunit.client.0.smithi151.stdout:2/443: dwrite de/f3a [0,4194304] 0 2024-04-16T02:35:47.984 INFO:tasks.workunit.client.1.smithi153.stdout:8/869: write d2/d13/d96/dc7/ffa [146729,25474] 0 2024-04-16T02:35:47.988 INFO:tasks.workunit.client.1.smithi153.stdout:8/870: write d2/d93/d97/d9c/f2e [4935742,43720] 0 2024-04-16T02:35:47.990 INFO:tasks.workunit.client.1.smithi153.stdout:7/901: mknod d0/d13/c134 0 2024-04-16T02:35:47.990 INFO:tasks.workunit.client.1.smithi153.stdout:5/954: mkdir d8/d41/dbd/d129 0 2024-04-16T02:35:47.993 INFO:tasks.workunit.client.0.smithi151.stdout:2/444: getdents de/d11/d42/d7a 0 2024-04-16T02:35:47.995 INFO:tasks.workunit.client.1.smithi153.stdout:7/902: write d0/d4/db/d2b/d8c/f9b [3080861,93709] 0 2024-04-16T02:35:47.995 INFO:tasks.workunit.client.1.smithi153.stdout:5/955: stat d8/d41/dbd/dd3/lf0 0 2024-04-16T02:35:47.995 INFO:tasks.workunit.client.1.smithi153.stdout:2/945: getdents d9/d71/d8d/d11e 0 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: pgmap v21: 65 pgs: 65 active+clean; 2.1 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 71 MiB/s rd, 247 MiB/s wr, 448 op/s 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: Adjusting osd_memory_target on smithi151 to 1216M 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:35:47.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:35:47.998 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: Updating smithi151:/etc/ceph/ceph.conf 2024-04-16T02:35:47.998 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:47 smithi151 ceph-mon[29003]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-16T02:35:47.998 INFO:tasks.workunit.client.1.smithi153.stdout:5/956: stat d8/df 0 2024-04-16T02:35:47.999 INFO:tasks.workunit.client.0.smithi151.stdout:2/445: creat de/d11/d42/d7a/f8c x:0 0 0 2024-04-16T02:35:47.999 INFO:tasks.workunit.client.0.smithi151.stdout:2/446: readlink de/d11/d42/l4d 0 2024-04-16T02:35:47.999 INFO:tasks.workunit.client.1.smithi153.stdout:9/966: rename d4/d6/dc/d2d/c7e to d4/d6/d1b/c136 0 2024-04-16T02:35:48.001 INFO:tasks.workunit.client.1.smithi153.stdout:2/946: dread - d9/d24/d27/d2c/d5d/da3/dc0/d11f/f12c zero size 2024-04-16T02:35:48.002 INFO:tasks.workunit.client.0.smithi151.stdout:4/423: dread d0/d11/d19/d46/f58 [0,4194304] 0 2024-04-16T02:35:48.003 INFO:tasks.workunit.client.0.smithi151.stdout:4/424: stat d0/d20/d59/f7f 0 2024-04-16T02:35:48.003 INFO:tasks.workunit.client.1.smithi153.stdout:2/947: fdatasync d9/d24/d27/d2c/f121 0 2024-04-16T02:35:48.005 INFO:tasks.workunit.client.1.smithi153.stdout:3/929: dwrite d6/d10/d22/d2c/fd3 [8388608,4194304] 0 2024-04-16T02:35:48.010 INFO:tasks.workunit.client.1.smithi153.stdout:9/967: dwrite d4/d33/faf [0,4194304] 0 2024-04-16T02:35:48.010 INFO:tasks.workunit.client.1.smithi153.stdout:3/930: creat d6/d10/d3b/d59/f12a x:0 0 0 2024-04-16T02:35:48.014 INFO:tasks.workunit.client.1.smithi153.stdout:2/948: creat d9/d2b/d5c/f132 x:0 0 0 2024-04-16T02:35:48.014 INFO:tasks.workunit.client.1.smithi153.stdout:9/968: dread - d4/d6/ddb/d4a/d101/f135 zero size 2024-04-16T02:35:48.015 INFO:tasks.workunit.client.1.smithi153.stdout:3/931: write d6/d10/d22/d2c/fba [3797063,69719] 0 2024-04-16T02:35:48.019 INFO:tasks.workunit.client.1.smithi153.stdout:6/880: dwrite dc/d47/d57/f70 [0,4194304] 0 2024-04-16T02:35:48.019 INFO:tasks.workunit.client.1.smithi153.stdout:1/903: write d4/dd/d13/d23/d31/f88 [241961,68502] 0 2024-04-16T02:35:48.021 INFO:tasks.workunit.client.1.smithi153.stdout:6/881: fsync dc/d1a/f1c 0 2024-04-16T02:35:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: pgmap v21: 65 pgs: 65 active+clean; 2.1 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 71 MiB/s rd, 247 MiB/s wr, 448 op/s 2024-04-16T02:35:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T02:35:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: Adjusting osd_memory_target on smithi151 to 1216M 2024-04-16T02:35:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:35:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:35:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: Updating smithi151:/etc/ceph/ceph.conf 2024-04-16T02:35:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:47 smithi153 ceph-mon[36747]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-16T02:35:48.027 INFO:tasks.workunit.client.1.smithi153.stdout:6/882: rmdir dc 39 2024-04-16T02:35:48.029 INFO:tasks.workunit.client.1.smithi153.stdout:9/969: truncate d4/d6/d1b/d2a/fc9 132538 0 2024-04-16T02:35:48.031 INFO:tasks.workunit.client.1.smithi153.stdout:2/949: getdents d9/d71/df2 0 2024-04-16T02:35:48.032 INFO:tasks.workunit.client.1.smithi153.stdout:5/957: getdents d8/d41/dbd 0 2024-04-16T02:35:48.038 INFO:tasks.workunit.client.1.smithi153.stdout:2/950: dwrite d9/d24/d34/fce [0,4194304] 0 2024-04-16T02:35:48.040 INFO:tasks.workunit.client.1.smithi153.stdout:7/903: dread d0/d4/db/f64 [0,4194304] 0 2024-04-16T02:35:48.040 INFO:tasks.workunit.client.1.smithi153.stdout:2/951: fdatasync d9/d24/d27/d2c/d41/f78 0 2024-04-16T02:35:48.046 INFO:tasks.workunit.client.1.smithi153.stdout:2/952: dwrite d9/d71/d8d/da8/ff5 [0,4194304] 0 2024-04-16T02:35:48.066 INFO:tasks.workunit.client.1.smithi153.stdout:5/958: mknod d8/d41/dbd/dd3/c12a 0 2024-04-16T02:35:48.068 INFO:tasks.workunit.client.1.smithi153.stdout:8/871: dwrite d2/f17 [4194304,4194304] 0 2024-04-16T02:35:48.069 INFO:tasks.workunit.client.1.smithi153.stdout:8/872: stat d2/d13/d2c/f101 0 2024-04-16T02:35:48.071 INFO:tasks.workunit.client.1.smithi153.stdout:9/970: fdatasync d4/d6/dc/d24/d4e/f81 0 2024-04-16T02:35:48.073 INFO:tasks.workunit.client.1.smithi153.stdout:5/959: rmdir d8/df/d28/d5a/d66 39 2024-04-16T02:35:48.074 INFO:tasks.workunit.client.1.smithi153.stdout:9/971: dread d4/d6/dc/d2d/d64/db9/fcf [0,4194304] 0 2024-04-16T02:35:48.075 INFO:tasks.workunit.client.0.smithi151.stdout:5/454: sync 2024-04-16T02:35:48.076 INFO:tasks.workunit.client.0.smithi151.stdout:5/455: chown d2/d6/d19/d1a/c4e 5263 1 2024-04-16T02:35:48.077 INFO:tasks.workunit.client.0.smithi151.stdout:1/356: sync 2024-04-16T02:35:48.077 INFO:tasks.workunit.client.0.smithi151.stdout:3/256: sync 2024-04-16T02:35:48.077 INFO:tasks.workunit.client.0.smithi151.stdout:3/257: fsync d1/f1d 0 2024-04-16T02:35:48.078 INFO:tasks.workunit.client.1.smithi153.stdout:8/873: mkdir d2/de/d3f/d124 0 2024-04-16T02:35:48.082 INFO:tasks.workunit.client.1.smithi153.stdout:3/932: truncate d6/f2d 3800898 0 2024-04-16T02:35:48.083 INFO:tasks.workunit.client.1.smithi153.stdout:1/904: truncate d4/dd/d13/d23/fab 3091357 0 2024-04-16T02:35:48.085 INFO:tasks.workunit.client.1.smithi153.stdout:8/874: mkdir d2/d13/d96/dc7/d109/d47/d77/d125 0 2024-04-16T02:35:48.088 INFO:tasks.workunit.client.1.smithi153.stdout:9/972: link d4/d33/dae/dc7/d66/f98 d4/d6/d8c/f137 0 2024-04-16T02:35:48.095 INFO:tasks.workunit.client.1.smithi153.stdout:6/883: write dc/d47/d57/d77/d15/de8/fbd [1243114,64309] 0 2024-04-16T02:35:48.096 INFO:tasks.workunit.client.1.smithi153.stdout:3/933: symlink d6/d10/d22/d2c/l12b 0 2024-04-16T02:35:48.098 INFO:tasks.workunit.client.1.smithi153.stdout:7/904: getdents d0/d4/db/d2b 0 2024-04-16T02:35:48.100 INFO:tasks.workunit.client.1.smithi153.stdout:3/934: stat d6/d10/d22/d4d 0 2024-04-16T02:35:48.101 INFO:tasks.workunit.client.1.smithi153.stdout:8/875: getdents d2/de/d3f 0 2024-04-16T02:35:48.105 INFO:tasks.workunit.client.1.smithi153.stdout:2/953: dwrite d9/d2b/f6b [0,4194304] 0 2024-04-16T02:35:48.106 INFO:tasks.workunit.client.0.smithi151.stdout:2/447: creat de/d11/d42/d5e/f8d x:0 0 0 2024-04-16T02:35:48.106 INFO:tasks.workunit.client.1.smithi153.stdout:1/905: dwrite d4/fc4 [0,4194304] 0 2024-04-16T02:35:48.106 INFO:tasks.workunit.client.0.smithi151.stdout:2/448: dread - de/d65/f6f zero size 2024-04-16T02:35:48.107 INFO:tasks.workunit.client.0.smithi151.stdout:2/449: fsync de/d11/d41/f38 0 2024-04-16T02:35:48.108 INFO:tasks.workunit.client.0.smithi151.stdout:4/425: rename d0/l3e to d0/d11/d19/d31/l90 0 2024-04-16T02:35:48.109 INFO:tasks.workunit.client.1.smithi153.stdout:8/876: chown d2/de/l22 399811 1 2024-04-16T02:35:48.110 INFO:tasks.workunit.client.0.smithi151.stdout:0/337: rmdir dc 39 2024-04-16T02:35:48.112 INFO:tasks.workunit.client.1.smithi153.stdout:9/973: dwrite d4/d6/d1b/f71 [0,4194304] 0 2024-04-16T02:35:48.115 INFO:tasks.workunit.client.0.smithi151.stdout:1/357: creat d3/d58/df/d68/f7c x:0 0 0 2024-04-16T02:35:48.115 INFO:tasks.workunit.client.1.smithi153.stdout:5/960: dread d8/d41/dbd/d39/f6b [0,4194304] 0 2024-04-16T02:35:48.117 INFO:tasks.workunit.client.0.smithi151.stdout:3/258: creat d1/d6/d20/d54/f58 x:0 0 0 2024-04-16T02:35:48.118 INFO:tasks.workunit.client.1.smithi153.stdout:9/974: dread - d4/d6/dc/d24/d104/d113/f128 zero size 2024-04-16T02:35:48.118 INFO:tasks.workunit.client.0.smithi151.stdout:4/426: symlink d0/d73/l91 0 2024-04-16T02:35:48.121 INFO:tasks.workunit.client.0.smithi151.stdout:1/358: mknod d3/d58/df/d32/d3a/d67/c7d 0 2024-04-16T02:35:48.123 INFO:tasks.workunit.client.1.smithi153.stdout:6/884: dwrite dc/d47/d93/fa4 [0,4194304] 0 2024-04-16T02:35:48.123 INFO:tasks.workunit.client.0.smithi151.stdout:4/427: creat d0/d11/d19/d31/d45/f92 x:0 0 0 2024-04-16T02:35:48.124 INFO:tasks.workunit.client.1.smithi153.stdout:6/885: chown dc/d47/d57/d77/d7a/c119 112 1 2024-04-16T02:35:48.124 INFO:tasks.workunit.client.0.smithi151.stdout:0/338: truncate dc/d15/d34/d4e/f63 24236 0 2024-04-16T02:35:48.125 INFO:tasks.workunit.client.1.smithi153.stdout:6/886: fsync dc/d47/d57/d77/dc4/f117 0 2024-04-16T02:35:48.125 INFO:tasks.workunit.client.0.smithi151.stdout:4/428: write d0/d11/d19/d87/d3a/d65/f7d [13651,97045] 0 2024-04-16T02:35:48.126 INFO:tasks.workunit.client.1.smithi153.stdout:6/887: truncate dc/d1a/d2d/fc7 4360657 0 2024-04-16T02:35:48.127 INFO:tasks.workunit.client.0.smithi151.stdout:1/359: mkdir d3/d4/d64/d2a/d7e 0 2024-04-16T02:35:48.129 INFO:tasks.workunit.client.0.smithi151.stdout:2/450: write de/d15/f1a [1891003,43133] 0 2024-04-16T02:35:48.131 INFO:tasks.workunit.client.0.smithi151.stdout:5/456: truncate d2/d6/d19/f4b 419562 0 2024-04-16T02:35:48.133 INFO:tasks.workunit.client.0.smithi151.stdout:2/451: unlink de/d11/d42/l48 0 2024-04-16T02:35:48.133 INFO:tasks.workunit.client.0.smithi151.stdout:0/339: link dc/d15/d34/c53 dc/d15/d34/d4e/c6b 0 2024-04-16T02:35:48.135 INFO:tasks.workunit.client.0.smithi151.stdout:1/360: creat d3/d4/d64/d2a/d73/f7f x:0 0 0 2024-04-16T02:35:48.135 INFO:tasks.workunit.client.1.smithi153.stdout:6/888: dwrite dc/d47/d57/d77/d15/dac/dae/de7/f111 [0,4194304] 0 2024-04-16T02:35:48.136 INFO:tasks.workunit.client.0.smithi151.stdout:4/429: rename d0/f2d to d0/f93 0 2024-04-16T02:35:48.136 INFO:tasks.workunit.client.0.smithi151.stdout:1/361: chown d3/d58/df/d68/f7c 839380 1 2024-04-16T02:35:48.138 INFO:tasks.workunit.client.0.smithi151.stdout:2/452: unlink de/d15/f36 0 2024-04-16T02:35:48.146 INFO:tasks.workunit.client.0.smithi151.stdout:4/430: unlink d0/d11/d19/d31/d89/l5c 0 2024-04-16T02:35:48.147 INFO:tasks.workunit.client.0.smithi151.stdout:4/431: readlink d0/l38 0 2024-04-16T02:35:48.148 INFO:tasks.workunit.client.0.smithi151.stdout:5/457: write d2/d6/d19/d30/d36/f5b [4383498,114407] 0 2024-04-16T02:35:48.149 INFO:tasks.workunit.client.0.smithi151.stdout:5/458: stat d2/d5e/f72 0 2024-04-16T02:35:48.150 INFO:tasks.workunit.client.0.smithi151.stdout:4/432: mknod d0/c94 0 2024-04-16T02:35:48.153 INFO:tasks.workunit.client.0.smithi151.stdout:4/433: creat d0/d11/d19/d31/f95 x:0 0 0 2024-04-16T02:35:48.154 INFO:tasks.workunit.client.0.smithi151.stdout:4/434: stat d0/d20/l6c 0 2024-04-16T02:35:48.155 INFO:tasks.workunit.client.0.smithi151.stdout:4/435: readlink d0/l4 0 2024-04-16T02:35:48.161 INFO:tasks.workunit.client.0.smithi151.stdout:5/459: dwrite d2/d6/d19/d1a/f41 [0,4194304] 0 2024-04-16T02:35:48.161 INFO:tasks.workunit.client.0.smithi151.stdout:1/362: dread d3/d58/df/d32/f37 [0,4194304] 0 2024-04-16T02:35:48.161 INFO:tasks.workunit.client.0.smithi151.stdout:0/340: getdents dc/d15/d34/d4e 0 2024-04-16T02:35:48.162 INFO:tasks.workunit.client.0.smithi151.stdout:4/436: read - d0/d11/d19/d31/d68/f7a zero size 2024-04-16T02:35:48.165 INFO:tasks.workunit.client.0.smithi151.stdout:0/341: stat dc/d15/d34/d3b/c52 0 2024-04-16T02:35:48.171 INFO:tasks.workunit.client.0.smithi151.stdout:0/342: symlink dc/d15/d34/d67/l6c 0 2024-04-16T02:35:48.173 INFO:tasks.workunit.client.0.smithi151.stdout:2/453: dread de/d11/d41/d4b/f55 [0,4194304] 0 2024-04-16T02:35:48.175 INFO:tasks.workunit.client.0.smithi151.stdout:5/460: rename d2/d6/f35 to d2/d6/d19/d30/d49/d7f/f90 0 2024-04-16T02:35:48.177 INFO:tasks.workunit.client.0.smithi151.stdout:5/461: fdatasync d2/d6/d19/d30/d36/f5b 0 2024-04-16T02:35:48.178 INFO:tasks.workunit.client.0.smithi151.stdout:0/343: dread dc/d15/f2b [0,4194304] 0 2024-04-16T02:35:48.181 INFO:tasks.workunit.client.0.smithi151.stdout:2/454: dwrite de/d15/f89 [0,4194304] 0 2024-04-16T02:35:48.183 INFO:tasks.workunit.client.0.smithi151.stdout:4/437: link d0/d11/d19/d87/d3a/c3b d0/d11/d19/d87/d3a/d50/c96 0 2024-04-16T02:35:48.184 INFO:tasks.workunit.client.0.smithi151.stdout:4/438: write d0/d11/f3d [2503102,49190] 0 2024-04-16T02:35:48.185 INFO:tasks.workunit.client.0.smithi151.stdout:5/462: creat d2/d6/d19/d62/f91 x:0 0 0 2024-04-16T02:35:48.185 INFO:tasks.workunit.client.0.smithi151.stdout:1/363: write d3/d4/d64/d2a/d52/f6a [3978616,1407] 0 2024-04-16T02:35:48.188 INFO:tasks.workunit.client.0.smithi151.stdout:0/344: creat dc/d15/d34/d4e/f6d x:0 0 0 2024-04-16T02:35:48.190 INFO:tasks.workunit.client.0.smithi151.stdout:5/463: read - d2/d6/d19/d30/d36/d42/d6f/d83/d8b/f8f zero size 2024-04-16T02:35:48.190 INFO:tasks.workunit.client.0.smithi151.stdout:1/364: write d3/d33/d10/f78 [787538,75836] 0 2024-04-16T02:35:48.191 INFO:tasks.workunit.client.0.smithi151.stdout:2/455: dwrite de/d11/f2b [4194304,4194304] 0 2024-04-16T02:35:48.193 INFO:tasks.workunit.client.0.smithi151.stdout:4/439: write d0/d11/f26 [505529,12571] 0 2024-04-16T02:35:48.196 INFO:tasks.workunit.client.0.smithi151.stdout:4/440: chown d0/d11/d19/d31/d89/f1d 82 1 2024-04-16T02:35:48.199 INFO:tasks.workunit.client.0.smithi151.stdout:5/464: creat d2/d6/d19/d1a/d4c/f92 x:0 0 0 2024-04-16T02:35:48.202 INFO:tasks.workunit.client.0.smithi151.stdout:4/441: symlink d0/d11/d19/d31/d4a/l97 0 2024-04-16T02:35:48.205 INFO:tasks.workunit.client.0.smithi151.stdout:2/456: dwrite de/d15/f17 [0,4194304] 0 2024-04-16T02:35:48.207 INFO:tasks.workunit.client.0.smithi151.stdout:2/457: chown de/d15/l80 9878 1 2024-04-16T02:35:48.216 INFO:tasks.workunit.client.0.smithi151.stdout:4/442: dwrite d0/d11/d19/d87/f5a [0,4194304] 0 2024-04-16T02:35:48.217 INFO:tasks.workunit.client.0.smithi151.stdout:4/443: stat d0/d11/d19/f8b 0 2024-04-16T02:35:48.218 INFO:tasks.workunit.client.0.smithi151.stdout:4/444: write d0/d20/f66 [372885,49220] 0 2024-04-16T02:35:48.219 INFO:tasks.workunit.client.0.smithi151.stdout:2/458: dread de/d65/f72 [0,4194304] 0 2024-04-16T02:35:48.220 INFO:tasks.workunit.client.0.smithi151.stdout:2/459: dread - de/d11/d42/d5e/f86 zero size 2024-04-16T02:35:48.221 INFO:tasks.workunit.client.0.smithi151.stdout:5/465: rename d2/d6/d19/d1a/d4c/d74 to d2/d6/d19/d93 0 2024-04-16T02:35:48.222 INFO:tasks.workunit.client.0.smithi151.stdout:4/445: mkdir d0/d11/d19/d87/d3a/d65/d98 0 2024-04-16T02:35:48.224 INFO:tasks.workunit.client.0.smithi151.stdout:2/460: creat de/d11/d42/d7a/d8b/f8e x:0 0 0 2024-04-16T02:35:48.227 INFO:tasks.workunit.client.1.smithi153.stdout:3/935: mkdir d6/d10/d22/d2c/d12c 0 2024-04-16T02:35:48.229 INFO:tasks.workunit.client.0.smithi151.stdout:5/466: rename d2/d6/d19/d30/d36/d42/f5a to d2/d6/d19/d30/d49/f94 0 2024-04-16T02:35:48.229 INFO:tasks.workunit.client.1.smithi153.stdout:3/936: read d6/d10/d22/d2c/f58 [4333086,34340] 0 2024-04-16T02:35:48.231 INFO:tasks.workunit.client.1.smithi153.stdout:1/906: mknod d4/dd/d13/ded/c12f 0 2024-04-16T02:35:48.231 INFO:tasks.workunit.client.1.smithi153.stdout:7/905: dwrite d0/d4/db/d34/f79 [0,4194304] 0 2024-04-16T02:35:48.232 INFO:tasks.workunit.client.0.smithi151.stdout:5/467: chown d2/d6/d19/d1a/d4c/l4f 93 1 2024-04-16T02:35:48.234 INFO:tasks.workunit.client.1.smithi153.stdout:9/975: creat d4/d33/dae/dc7/d66/dcb/f138 x:0 0 0 2024-04-16T02:35:48.234 INFO:tasks.workunit.client.1.smithi153.stdout:6/889: rmdir dc/d47/d57/d77/d7a/d97 39 2024-04-16T02:35:48.235 INFO:tasks.workunit.client.0.smithi151.stdout:2/461: dread f7 [0,4194304] 0 2024-04-16T02:35:48.239 INFO:tasks.workunit.client.1.smithi153.stdout:8/877: dwrite d2/de/d34/f6f [0,4194304] 0 2024-04-16T02:35:48.239 INFO:tasks.workunit.client.1.smithi153.stdout:7/906: dread d0/d4/db/d2b/d43/d128/fd4 [0,4194304] 0 2024-04-16T02:35:48.239 INFO:tasks.workunit.client.0.smithi151.stdout:4/446: dwrite d0/d11/f53 [0,4194304] 0 2024-04-16T02:35:48.239 INFO:tasks.workunit.client.0.smithi151.stdout:1/365: dread d3/d4/d64/d2a/f30 [0,4194304] 0 2024-04-16T02:35:48.242 INFO:tasks.workunit.client.1.smithi153.stdout:1/907: rename d4/dd/d13/d23/d31 to d4/da2/dd5/d130 0 2024-04-16T02:35:48.242 INFO:tasks.workunit.client.0.smithi151.stdout:2/462: mkdir de/d65/d88/d8f 0 2024-04-16T02:35:48.243 INFO:tasks.workunit.client.0.smithi151.stdout:4/447: fdatasync d0/f3c 0 2024-04-16T02:35:48.244 INFO:tasks.workunit.client.1.smithi153.stdout:9/976: dread - d4/d6/dc/fe3 zero size 2024-04-16T02:35:48.246 INFO:tasks.workunit.client.0.smithi151.stdout:0/345: dwrite dc/d15/d34/d4e/f63 [0,4194304] 0 2024-04-16T02:35:48.247 INFO:tasks.workunit.client.0.smithi151.stdout:2/463: mkdir de/d15/d1e/d90 0 2024-04-16T02:35:48.248 INFO:tasks.workunit.client.1.smithi153.stdout:6/890: unlink dc/d47/d93/fab 0 2024-04-16T02:35:48.252 INFO:tasks.workunit.client.1.smithi153.stdout:3/937: mknod d6/d10/d5b/d127/c12d 0 2024-04-16T02:35:48.253 INFO:tasks.workunit.client.1.smithi153.stdout:2/954: link d9/d2b/f19 d9/d24/d27/d2c/d5d/da3/dc0/f133 0 2024-04-16T02:35:48.254 INFO:tasks.workunit.client.0.smithi151.stdout:2/464: symlink de/d3d/d6a/l91 0 2024-04-16T02:35:48.256 INFO:tasks.workunit.client.0.smithi151.stdout:0/346: getdents dc/d15/d5c 0 2024-04-16T02:35:48.260 INFO:tasks.workunit.client.0.smithi151.stdout:4/448: dread d0/d11/d19/d31/d68/f79 [0,4194304] 0 2024-04-16T02:35:48.260 INFO:tasks.workunit.client.0.smithi151.stdout:0/347: stat dc/d15/l2a 0 2024-04-16T02:35:48.262 INFO:tasks.workunit.client.1.smithi153.stdout:5/961: link d8/df/d2f/cb0 d8/df/d28/d5a/c12b 0 2024-04-16T02:35:48.263 INFO:tasks.workunit.client.0.smithi151.stdout:2/465: creat de/d11/d42/d5e/d6c/d6d/f92 x:0 0 0 2024-04-16T02:35:48.265 INFO:tasks.workunit.client.1.smithi153.stdout:2/955: mkdir d9/d24/d27/d2c/d5d/de5/d124/d134 0 2024-04-16T02:35:48.267 INFO:tasks.workunit.client.0.smithi151.stdout:4/449: unlink d0/d11/d19/d87/d3a/f74 0 2024-04-16T02:35:48.268 INFO:tasks.workunit.client.1.smithi153.stdout:8/878: unlink d2/d93/d97/d9c/d27/d5e/de4/df3/fbb 0 2024-04-16T02:35:48.272 INFO:tasks.workunit.client.1.smithi153.stdout:3/938: rename d6/d10/d22/d4d/da0/deb to d6/d10/d22/d4d/da0/d12e 0 2024-04-16T02:35:48.272 INFO:tasks.workunit.client.1.smithi153.stdout:3/939: fdatasync d6/d10/d3b/d59/fe9 0 2024-04-16T02:35:48.276 INFO:tasks.workunit.client.1.smithi153.stdout:3/940: dread d6/d10/d22/f2a [0,4194304] 0 2024-04-16T02:35:48.279 INFO:tasks.workunit.client.0.smithi151.stdout:5/468: dread d2/d6/f6c [0,4194304] 0 2024-04-16T02:35:48.280 INFO:tasks.workunit.client.0.smithi151.stdout:2/466: dwrite de/d11/d42/d7a/d8b/f59 [0,4194304] 0 2024-04-16T02:35:48.281 INFO:tasks.workunit.client.0.smithi151.stdout:5/469: stat d2/d6/d43/f6d 0 2024-04-16T02:35:48.282 INFO:tasks.workunit.client.0.smithi151.stdout:2/467: chown de/d11/d41/d4b 414555753 1 2024-04-16T02:35:48.282 INFO:tasks.workunit.client.1.smithi153.stdout:8/879: creat d2/d1c/f126 x:0 0 0 2024-04-16T02:35:48.283 INFO:tasks.workunit.client.0.smithi151.stdout:5/470: chown d2/d6/d19/d30/f77 43431 1 2024-04-16T02:35:48.285 INFO:tasks.workunit.client.0.smithi151.stdout:2/468: symlink de/d15/l93 0 2024-04-16T02:35:48.286 INFO:tasks.workunit.client.0.smithi151.stdout:5/471: truncate d2/d6/d19/d1a/f1b 4436764 0 2024-04-16T02:35:48.287 INFO:tasks.workunit.client.0.smithi151.stdout:1/366: dread d3/d1f/d48/f51 [0,4194304] 0 2024-04-16T02:35:48.289 INFO:tasks.workunit.client.1.smithi153.stdout:7/907: dwrite d0/d4/db/d2b/f4e [0,4194304] 0 2024-04-16T02:35:48.291 INFO:tasks.workunit.client.1.smithi153.stdout:1/908: dwrite d4/dd/d13/fb3 [0,4194304] 0 2024-04-16T02:35:48.291 INFO:tasks.workunit.client.0.smithi151.stdout:1/367: write d3/d58/df/d32/d3a/f79 [1032086,101616] 0 2024-04-16T02:35:48.292 INFO:tasks.workunit.client.0.smithi151.stdout:5/472: read - d2/d6/d19/d30/d36/f3e zero size 2024-04-16T02:35:48.293 INFO:tasks.workunit.client.0.smithi151.stdout:2/469: rmdir de/d11/d42/d68 0 2024-04-16T02:35:48.294 INFO:tasks.workunit.client.0.smithi151.stdout:2/470: stat de/d11/d41/f38 0 2024-04-16T02:35:48.295 INFO:tasks.workunit.client.1.smithi153.stdout:9/977: getdents d4/d33/dae/dc7 0 2024-04-16T02:35:48.296 INFO:tasks.workunit.client.1.smithi153.stdout:2/956: rename d9/d24/d27/d2c/d41/d7a/d116/fcc to d9/d24/d27/d2c/d5d/da3/f135 0 2024-04-16T02:35:48.298 INFO:tasks.workunit.client.1.smithi153.stdout:6/891: dwrite dc/d47/d57/d77/d7a/f112 [0,4194304] 0 2024-04-16T02:35:48.301 INFO:tasks.workunit.client.0.smithi151.stdout:0/348: dwrite dc/d15/f19 [0,4194304] 0 2024-04-16T02:35:48.303 INFO:tasks.workunit.client.1.smithi153.stdout:2/957: symlink d9/d24/d27/d2c/l136 0 2024-04-16T02:35:48.303 INFO:tasks.workunit.client.0.smithi151.stdout:2/471: dread de/d11/d42/d7a/d8b/f74 [0,4194304] 0 2024-04-16T02:35:48.305 INFO:tasks.workunit.client.1.smithi153.stdout:7/908: rename d0/d4/l121 to d0/d4/db/d87/l135 0 2024-04-16T02:35:48.306 INFO:tasks.workunit.client.0.smithi151.stdout:1/368: write d3/d4/d64/d2a/f30 [2742901,80428] 0 2024-04-16T02:35:48.306 INFO:tasks.workunit.client.1.smithi153.stdout:5/962: link d8/df/f16 d8/df/d28/d5a/d72/f12c 0 2024-04-16T02:35:48.309 INFO:tasks.workunit.client.1.smithi153.stdout:6/892: creat dc/d1a/d2d/f123 x:0 0 0 2024-04-16T02:35:48.309 INFO:tasks.workunit.client.1.smithi153.stdout:1/909: symlink d4/dd/d18/d29/d6e/d113/l131 0 2024-04-16T02:35:48.312 INFO:tasks.workunit.client.0.smithi151.stdout:0/349: chown dc/d15/d34/d67/f3e 56923 1 2024-04-16T02:35:48.312 INFO:tasks.workunit.client.0.smithi151.stdout:1/369: write d3/d4/d64/d2a/f65 [700172,24041] 0 2024-04-16T02:35:48.312 INFO:tasks.workunit.client.0.smithi151.stdout:5/473: dwrite d2/d5e/f7e [0,4194304] 0 2024-04-16T02:35:48.313 INFO:tasks.workunit.client.1.smithi153.stdout:1/910: chown d4/dd/d18/d29/l48 2973589 1 2024-04-16T02:35:48.313 INFO:tasks.workunit.client.1.smithi153.stdout:9/978: dwrite d4/f48 [8388608,4194304] 0 2024-04-16T02:35:48.314 INFO:tasks.workunit.client.1.smithi153.stdout:7/909: symlink d0/d4/l136 0 2024-04-16T02:35:48.314 INFO:tasks.workunit.client.1.smithi153.stdout:6/893: mknod dc/d47/d57/d77/d15/d109/c124 0 2024-04-16T02:35:48.315 INFO:tasks.workunit.client.0.smithi151.stdout:2/472: link l9 de/d11/d41/d4b/d60/l94 0 2024-04-16T02:35:48.316 INFO:tasks.workunit.client.1.smithi153.stdout:2/958: dwrite d9/d71/d8d/fbd [0,4194304] 0 2024-04-16T02:35:48.317 INFO:tasks.workunit.client.0.smithi151.stdout:2/473: write de/d11/d42/d5e/f86 [1750,22206] 0 2024-04-16T02:35:48.319 INFO:tasks.workunit.client.0.smithi151.stdout:2/474: write de/f3a [1960432,15168] 0 2024-04-16T02:35:48.320 INFO:tasks.workunit.client.0.smithi151.stdout:2/475: chown de/d11/d41/f32 0 1 2024-04-16T02:35:48.323 INFO:tasks.workunit.client.1.smithi153.stdout:6/894: creat dc/d59/d5b/f125 x:0 0 0 2024-04-16T02:35:48.326 INFO:tasks.workunit.client.1.smithi153.stdout:5/963: dread d8/df/d28/d5a/d72/dd0/fe0 [4194304,4194304] 0 2024-04-16T02:35:48.330 INFO:tasks.workunit.client.0.smithi151.stdout:0/350: write dc/d15/d34/d67/d51/f62 [962548,60709] 0 2024-04-16T02:35:48.331 INFO:tasks.workunit.client.0.smithi151.stdout:5/474: symlink d2/d6/l95 0 2024-04-16T02:35:48.332 INFO:tasks.workunit.client.1.smithi153.stdout:6/895: symlink dc/d10/d62/l126 0 2024-04-16T02:35:48.332 INFO:tasks.workunit.client.0.smithi151.stdout:1/370: creat d3/f80 x:0 0 0 2024-04-16T02:35:48.333 INFO:tasks.workunit.client.0.smithi151.stdout:4/450: dread d0/d20/f21 [0,4194304] 0 2024-04-16T02:35:48.335 INFO:tasks.workunit.client.0.smithi151.stdout:5/475: dread - d2/d6/d19/d30/d36/f3e zero size 2024-04-16T02:35:48.335 INFO:tasks.workunit.client.0.smithi151.stdout:0/351: symlink dc/d15/d34/d4e/l6e 0 2024-04-16T02:35:48.337 INFO:tasks.workunit.client.0.smithi151.stdout:5/476: write d2/d6/f25 [6755804,2022] 0 2024-04-16T02:35:48.338 INFO:tasks.workunit.client.1.smithi153.stdout:6/896: dwrite dc/d47/d57/d77/d15/fc2 [0,4194304] 0 2024-04-16T02:35:48.341 INFO:tasks.workunit.client.0.smithi151.stdout:1/371: creat d3/d4/d64/d1c/f81 x:0 0 0 2024-04-16T02:35:48.341 INFO:tasks.workunit.client.0.smithi151.stdout:0/352: mkdir dc/d15/d34/d67/d69/d6f 0 2024-04-16T02:35:48.342 INFO:tasks.workunit.client.1.smithi153.stdout:7/910: link d0/d4/db/d87/c9a d0/d4/db/d47/d8d/d93/c137 0 2024-04-16T02:35:48.342 INFO:tasks.workunit.client.1.smithi153.stdout:6/897: write dc/d10/d62/f6a [7977517,130221] 0 2024-04-16T02:35:48.344 INFO:tasks.workunit.client.1.smithi153.stdout:6/898: write dc/d47/d57/d77/dc4/ff7 [5037566,114338] 0 2024-04-16T02:35:48.344 INFO:tasks.workunit.client.0.smithi151.stdout:0/353: dread dc/d15/d34/d3b/f44 [0,4194304] 0 2024-04-16T02:35:48.344 INFO:tasks.workunit.client.0.smithi151.stdout:1/372: symlink d3/d58/df/d32/d3a/d6d/l82 0 2024-04-16T02:35:48.346 INFO:tasks.workunit.client.1.smithi153.stdout:6/899: write dc/d47/d57/d101/ffb [2844465,22490] 0 2024-04-16T02:35:48.348 INFO:tasks.workunit.client.0.smithi151.stdout:5/477: fdatasync d2/d6/d19/d30/d49/d7f/f90 0 2024-04-16T02:35:48.350 INFO:tasks.workunit.client.0.smithi151.stdout:1/373: chown d3/d33/l14 109553 1 2024-04-16T02:35:48.350 INFO:tasks.workunit.client.0.smithi151.stdout:0/354: read dc/d15/f45 [1942574,66420] 0 2024-04-16T02:35:48.352 INFO:tasks.workunit.client.1.smithi153.stdout:7/911: dwrite d0/d4/db/d2b/d53/d59/d76/dc8/f130 [0,4194304] 0 2024-04-16T02:35:48.354 INFO:tasks.workunit.client.0.smithi151.stdout:1/374: creat d3/d58/df/d32/d3a/d71/f83 x:0 0 0 2024-04-16T02:35:48.355 INFO:tasks.workunit.client.1.smithi153.stdout:7/912: rmdir d0/de8 39 2024-04-16T02:35:48.356 INFO:tasks.workunit.client.0.smithi151.stdout:0/355: unlink dc/d15/d34/d3b/l61 0 2024-04-16T02:35:48.357 INFO:tasks.workunit.client.1.smithi153.stdout:7/913: fdatasync d0/d4/db/f109 0 2024-04-16T02:35:48.357 INFO:tasks.workunit.client.0.smithi151.stdout:5/478: creat d2/d6/f96 x:0 0 0 2024-04-16T02:35:48.359 INFO:tasks.workunit.client.0.smithi151.stdout:0/356: creat dc/d15/d34/d3b/f70 x:0 0 0 2024-04-16T02:35:48.361 INFO:tasks.workunit.client.0.smithi151.stdout:5/479: unlink d2/d5e/f86 0 2024-04-16T02:35:48.362 INFO:tasks.workunit.client.0.smithi151.stdout:0/357: fsync dc/d15/d34/f5d 0 2024-04-16T02:35:48.365 INFO:tasks.workunit.client.0.smithi151.stdout:2/476: rename de/d11/d41 to de/d11/d42/d82/d95 0 2024-04-16T02:35:48.366 INFO:tasks.workunit.client.1.smithi153.stdout:7/914: dwrite d0/d4/db/d2b/d8c/fae [0,4194304] 0 2024-04-16T02:35:48.367 INFO:tasks.workunit.client.1.smithi153.stdout:8/880: dwrite d2/d13/d96/dc7/d109/d47/d60/f71 [0,4194304] 0 2024-04-16T02:35:48.372 INFO:tasks.workunit.client.0.smithi151.stdout:0/358: mkdir dc/d15/d34/d3b/d71 0 2024-04-16T02:35:48.373 INFO:tasks.workunit.client.1.smithi153.stdout:5/964: write d8/d41/dbd/f81 [997954,64833] 0 2024-04-16T02:35:48.376 INFO:tasks.workunit.client.0.smithi151.stdout:1/375: link d3/d4/l3c d3/d4/d64/d2a/d52/l84 0 2024-04-16T02:35:48.377 INFO:tasks.workunit.client.0.smithi151.stdout:2/477: dwrite de/d11/d42/d82/d95/f37 [0,4194304] 0 2024-04-16T02:35:48.378 INFO:tasks.workunit.client.1.smithi153.stdout:2/959: dwrite d9/d2b/f17 [4194304,4194304] 0 2024-04-16T02:35:48.379 INFO:tasks.workunit.client.1.smithi153.stdout:6/900: write dc/d47/d57/d77/d15/f23 [1379851,128485] 0 2024-04-16T02:35:48.379 INFO:tasks.workunit.client.1.smithi153.stdout:9/979: dwrite d4/d6/dc/d24/dc6/fd0 [0,4194304] 0 2024-04-16T02:35:48.380 INFO:tasks.workunit.client.1.smithi153.stdout:5/965: stat d8/df/d28/d5a/d66/ffc 0 2024-04-16T02:35:48.382 INFO:tasks.workunit.client.0.smithi151.stdout:4/451: truncate d0/d20/f66 272041 0 2024-04-16T02:35:48.383 INFO:tasks.workunit.client.1.smithi153.stdout:9/980: write d4/d6/dc/d24/d4e/fd8 [985380,49903] 0 2024-04-16T02:35:48.385 INFO:tasks.workunit.client.0.smithi151.stdout:1/376: creat d3/d4/d64/d2a/d73/f85 x:0 0 0 2024-04-16T02:35:48.386 INFO:tasks.workunit.client.1.smithi153.stdout:9/981: stat d4/d6/ddb/d4a/d101 0 2024-04-16T02:35:48.388 INFO:tasks.workunit.client.1.smithi153.stdout:8/881: write d2/de/f15 [5237726,93471] 0 2024-04-16T02:35:48.391 INFO:tasks.workunit.client.1.smithi153.stdout:5/966: rename d8/df/d23/l3e to d8/df/d23/d70/l12d 0 2024-04-16T02:35:48.393 INFO:tasks.workunit.client.0.smithi151.stdout:2/478: creat de/d15/d1e/f96 x:0 0 0 2024-04-16T02:35:48.393 INFO:tasks.workunit.client.0.smithi151.stdout:1/377: read d3/d4/d64/d2a/f60 [85570,83898] 0 2024-04-16T02:35:48.394 INFO:tasks.workunit.client.1.smithi153.stdout:8/882: creat d2/d13/d96/dc7/d109/d47/d77/f127 x:0 0 0 2024-04-16T02:35:48.395 INFO:tasks.workunit.client.0.smithi151.stdout:0/359: mkdir dc/d4c/d72 0 2024-04-16T02:35:48.396 INFO:tasks.workunit.client.1.smithi153.stdout:5/967: chown d8/df/d28/d5a/f68 0 1 2024-04-16T02:35:48.396 INFO:tasks.workunit.client.0.smithi151.stdout:4/452: chown d0/d11/f4b 61888 1 2024-04-16T02:35:48.399 INFO:tasks.workunit.client.0.smithi151.stdout:4/453: chown d0/d11/d19/d31/d4a/d60/c8f 132 1 2024-04-16T02:35:48.400 INFO:tasks.workunit.client.0.smithi151.stdout:4/454: chown d0/d11/d19/d31/d4a/c78 4 1 2024-04-16T02:35:48.400 INFO:tasks.workunit.client.0.smithi151.stdout:4/455: chown d0/d11/d19/d31 535562828 1 2024-04-16T02:35:48.401 INFO:tasks.workunit.client.0.smithi151.stdout:2/479: mkdir de/d11/d42/d82/d95/d4b/d97 0 2024-04-16T02:35:48.402 INFO:tasks.workunit.client.0.smithi151.stdout:1/378: getdents d3/d4/d64/d2a/d7e 0 2024-04-16T02:35:48.402 INFO:tasks.workunit.client.1.smithi153.stdout:9/982: getdents d4/d6/db1/d117 0 2024-04-16T02:35:48.404 INFO:tasks.workunit.client.0.smithi151.stdout:0/360: mknod dc/d15/d34/c73 0 2024-04-16T02:35:48.404 INFO:tasks.workunit.client.0.smithi151.stdout:1/379: truncate d3/d58/df/d32/d3a/d71/f83 336382 0 2024-04-16T02:35:48.406 INFO:tasks.workunit.client.0.smithi151.stdout:0/361: write dc/d15/d34/f4d [3786193,129668] 0 2024-04-16T02:35:48.406 INFO:tasks.workunit.client.1.smithi153.stdout:2/960: dread d9/d2b/d1c/d20/d88/f9b [0,4194304] 0 2024-04-16T02:35:48.407 INFO:tasks.workunit.client.0.smithi151.stdout:2/480: mkdir de/d65/d71/d98 0 2024-04-16T02:35:48.408 INFO:tasks.workunit.client.0.smithi151.stdout:0/362: write dc/d15/d34/f5d [236416,95320] 0 2024-04-16T02:35:48.409 INFO:tasks.workunit.client.0.smithi151.stdout:2/481: read de/d11/d42/d5e/d6c/f77 [6513433,5681] 0 2024-04-16T02:35:48.410 INFO:tasks.workunit.client.1.smithi153.stdout:5/968: creat d8/d41/dbd/d8d/de9/f12e x:0 0 0 2024-04-16T02:35:48.410 INFO:tasks.workunit.client.0.smithi151.stdout:0/363: fsync dc/d20/f49 0 2024-04-16T02:35:48.410 INFO:tasks.workunit.client.0.smithi151.stdout:2/482: chown de 238 1 2024-04-16T02:35:48.411 INFO:tasks.workunit.client.1.smithi153.stdout:9/983: rmdir d4/d33/dae/dc7/dd5 39 2024-04-16T02:35:48.411 INFO:tasks.workunit.client.1.smithi153.stdout:5/969: chown d8/df/d28/f82 6573 1 2024-04-16T02:35:48.413 INFO:tasks.workunit.client.0.smithi151.stdout:1/380: creat d3/d4/d64/d2a/d7a/f86 x:0 0 0 2024-04-16T02:35:48.413 INFO:tasks.workunit.client.1.smithi153.stdout:2/961: creat d9/d24/d34/d99/f137 x:0 0 0 2024-04-16T02:35:48.414 INFO:tasks.workunit.client.1.smithi153.stdout:8/883: getdents d2/d13/d3a/dd6 0 2024-04-16T02:35:48.414 INFO:tasks.workunit.client.0.smithi151.stdout:1/381: stat d3/d4/d64/d2a/c31 0 2024-04-16T02:35:48.414 INFO:tasks.workunit.client.0.smithi151.stdout:5/480: dread d2/d6/f27 [0,4194304] 0 2024-04-16T02:35:48.415 INFO:tasks.workunit.client.0.smithi151.stdout:2/483: creat de/d11/d42/d82/f99 x:0 0 0 2024-04-16T02:35:48.415 INFO:tasks.workunit.client.0.smithi151.stdout:1/382: chown d3/d58/f54 50192 1 2024-04-16T02:35:48.416 INFO:tasks.workunit.client.1.smithi153.stdout:9/984: creat d4/d6/def/d12f/f139 x:0 0 0 2024-04-16T02:35:48.416 INFO:tasks.workunit.client.1.smithi153.stdout:5/970: read d8/d41/dbd/f40 [4039207,71167] 0 2024-04-16T02:35:48.416 INFO:tasks.workunit.client.1.smithi153.stdout:2/962: chown d9/d2b/d18/l4d 1508929 1 2024-04-16T02:35:48.416 INFO:tasks.workunit.client.1.smithi153.stdout:7/915: dwrite d0/de8/d110/d4d/fce [0,4194304] 0 2024-04-16T02:35:48.418 INFO:tasks.workunit.client.0.smithi151.stdout:2/484: dread de/d65/f72 [0,4194304] 0 2024-04-16T02:35:48.418 INFO:tasks.workunit.client.0.smithi151.stdout:5/481: fdatasync d2/d6/f32 0 2024-04-16T02:35:48.419 INFO:tasks.workunit.client.1.smithi153.stdout:6/901: write dc/d1a/d2d/f43 [3340830,98279] 0 2024-04-16T02:35:48.419 INFO:tasks.workunit.client.1.smithi153.stdout:5/971: fsync d8/df/d28/d5a/f117 0 2024-04-16T02:35:48.420 INFO:tasks.workunit.client.0.smithi151.stdout:0/364: creat dc/d15/d34/d3b/d71/f74 x:0 0 0 2024-04-16T02:35:48.421 INFO:tasks.workunit.client.0.smithi151.stdout:1/383: creat d3/d58/df/d32/d3a/f87 x:0 0 0 2024-04-16T02:35:48.421 INFO:tasks.workunit.client.1.smithi153.stdout:8/884: fsync d2/d13/d96/fb3 0 2024-04-16T02:35:48.422 INFO:tasks.workunit.client.1.smithi153.stdout:5/972: chown d8/d1f/f107 1002352 1 2024-04-16T02:35:48.422 INFO:tasks.workunit.client.0.smithi151.stdout:2/485: creat de/d15/f9a x:0 0 0 2024-04-16T02:35:48.424 INFO:tasks.workunit.client.0.smithi151.stdout:5/482: mknod d2/d6/d19/d1a/d4c/c97 0 2024-04-16T02:35:48.425 INFO:tasks.workunit.client.1.smithi153.stdout:5/973: stat d8/d41/dbd/f126 0 2024-04-16T02:35:48.426 INFO:tasks.workunit.client.0.smithi151.stdout:1/384: creat d3/d58/df/d32/f88 x:0 0 0 2024-04-16T02:35:48.426 INFO:tasks.workunit.client.0.smithi151.stdout:5/483: dread - d2/d6/d19/d30/d36/d42/f5d zero size 2024-04-16T02:35:48.428 INFO:tasks.workunit.client.1.smithi153.stdout:7/916: write d0/d4/db/d87/dc0/ffb [1151177,71996] 0 2024-04-16T02:35:48.428 INFO:tasks.workunit.client.0.smithi151.stdout:0/365: dread dc/d20/f49 [0,4194304] 0 2024-04-16T02:35:48.429 INFO:tasks.workunit.client.0.smithi151.stdout:5/484: write d2/d6/d19/d62/f91 [97345,28575] 0 2024-04-16T02:35:48.429 INFO:tasks.workunit.client.0.smithi151.stdout:1/385: read - d3/d4/d64/d2a/d52/f72 zero size 2024-04-16T02:35:48.430 INFO:tasks.workunit.client.1.smithi153.stdout:9/985: creat d4/d6/dc/d2d/d64/db9/dc1/d119/f13a x:0 0 0 2024-04-16T02:35:48.431 INFO:tasks.workunit.client.1.smithi153.stdout:2/963: dwrite d9/d2b/d1c/d20/fde [0,4194304] 0 2024-04-16T02:35:48.432 INFO:tasks.workunit.client.1.smithi153.stdout:9/986: stat d4/d6/dc/d2d/d64/db9/dc1/d119/f13a 0 2024-04-16T02:35:48.433 INFO:tasks.workunit.client.0.smithi151.stdout:0/366: chown dc/d15/d34/f5d 6186 1 2024-04-16T02:35:48.434 INFO:tasks.workunit.client.1.smithi153.stdout:8/885: creat d2/d13/d96/dc7/d109/d47/d60/f128 x:0 0 0 2024-04-16T02:35:48.434 INFO:tasks.workunit.client.1.smithi153.stdout:5/974: dread - d8/df/d2f/d4e/f89 zero size 2024-04-16T02:35:48.435 INFO:tasks.workunit.client.0.smithi151.stdout:0/367: stat dc/d20/l26 0 2024-04-16T02:35:48.437 INFO:tasks.workunit.client.1.smithi153.stdout:2/964: unlink d9/d24/c97 0 2024-04-16T02:35:48.438 INFO:tasks.workunit.client.0.smithi151.stdout:2/486: dwrite f7 [0,4194304] 0 2024-04-16T02:35:48.438 INFO:tasks.workunit.client.0.smithi151.stdout:4/456: rename d0/d20 to d0/d99 0 2024-04-16T02:35:48.438 INFO:tasks.workunit.client.1.smithi153.stdout:7/917: unlink d0/d4/db/d2b/d43/f9d 0 2024-04-16T02:35:48.441 INFO:tasks.workunit.client.0.smithi151.stdout:2/487: chown de/d11/d42/d5e/d6c 114072 1 2024-04-16T02:35:48.442 INFO:tasks.workunit.client.0.smithi151.stdout:5/485: write d2/d6/d19/d30/f44 [722223,33202] 0 2024-04-16T02:35:48.442 INFO:tasks.workunit.client.1.smithi153.stdout:6/902: rename dc/l28 to dc/d47/l127 0 2024-04-16T02:35:48.445 INFO:tasks.workunit.client.0.smithi151.stdout:0/368: creat dc/d3c/f75 x:0 0 0 2024-04-16T02:35:48.446 INFO:tasks.workunit.client.0.smithi151.stdout:2/488: symlink de/d65/d71/d7c/l9b 0 2024-04-16T02:35:48.448 INFO:tasks.workunit.client.0.smithi151.stdout:5/486: read d2/d6/f27 [1229499,104074] 0 2024-04-16T02:35:48.448 INFO:tasks.workunit.client.0.smithi151.stdout:4/457: creat d0/d11/d19/d31/d45/f9a x:0 0 0 2024-04-16T02:35:48.449 INFO:tasks.workunit.client.1.smithi153.stdout:9/987: rename d4/d6/dc/d24/d60/fed to d4/d6/d1b/ddf/f13b 0 2024-04-16T02:35:48.453 INFO:tasks.workunit.client.1.smithi153.stdout:8/886: link d2/d13/d96/dc7/ffa d2/d13/d96/dc7/d109/f129 0 2024-04-16T02:35:48.454 INFO:tasks.workunit.client.0.smithi151.stdout:1/386: dwrite d3/d58/f54 [0,4194304] 0 2024-04-16T02:35:48.457 INFO:tasks.workunit.client.1.smithi153.stdout:2/965: getdents d9/d24/d27/d2c/d41/d7a 0 2024-04-16T02:35:48.457 INFO:tasks.workunit.client.0.smithi151.stdout:5/487: symlink d2/d6/d19/d30/d36/d42/d6f/d83/d8b/l98 0 2024-04-16T02:35:48.457 INFO:tasks.workunit.client.0.smithi151.stdout:2/489: mknod de/d65/d71/c9c 0 2024-04-16T02:35:48.459 INFO:tasks.workunit.client.0.smithi151.stdout:0/369: mkdir dc/d4c/d72/d76 0 2024-04-16T02:35:48.460 INFO:tasks.workunit.client.0.smithi151.stdout:1/387: mknod d3/d58/df/d32/d3a/d71/c89 0 2024-04-16T02:35:48.460 INFO:tasks.workunit.client.1.smithi153.stdout:9/988: rmdir d4/d6/d1b/d2a/d99/d11c 0 2024-04-16T02:35:48.461 INFO:tasks.workunit.client.1.smithi153.stdout:2/966: rmdir d9/d24/d27/d2c/d41/d7a 39 2024-04-16T02:35:48.461 INFO:tasks.workunit.client.0.smithi151.stdout:1/388: chown d3/d4/d64/d1c/f2c 14 1 2024-04-16T02:35:48.461 INFO:tasks.workunit.client.0.smithi151.stdout:4/458: mkdir d0/d9b 0 2024-04-16T02:35:48.462 INFO:tasks.workunit.client.1.smithi153.stdout:9/989: readlink d4/d6/dc/l1c 0 2024-04-16T02:35:48.462 INFO:tasks.workunit.client.0.smithi151.stdout:4/459: chown d0/d11/d19/d31/d45/f92 40661702 1 2024-04-16T02:35:48.463 INFO:tasks.workunit.client.1.smithi153.stdout:9/990: readlink d4/d6/ddb/lec 0 2024-04-16T02:35:48.465 INFO:tasks.workunit.client.1.smithi153.stdout:9/991: write d4/d6/d1b/d2a/d99/ff2 [380635,46333] 0 2024-04-16T02:35:48.465 INFO:tasks.workunit.client.0.smithi151.stdout:0/370: creat dc/d4c/d72/f77 x:0 0 0 2024-04-16T02:35:48.465 INFO:tasks.workunit.client.1.smithi153.stdout:9/992: readlink d4/d33/l45 0 2024-04-16T02:35:48.466 INFO:tasks.workunit.client.1.smithi153.stdout:5/975: write d8/d1f/fcd [516038,111489] 0 2024-04-16T02:35:48.482 INFO:tasks.workunit.client.1.smithi153.stdout:2/967: dread d9/d24/d27/d2c/d41/d7a/d116/fc1 [0,4194304] 0 2024-04-16T02:35:48.484 INFO:tasks.workunit.client.1.smithi153.stdout:8/887: dread d2/f23 [4194304,4194304] 0 2024-04-16T02:35:48.484 INFO:tasks.workunit.client.1.smithi153.stdout:8/888: chown d2/d13/d96/dc7/d109/d47/d77/ddf/dfc 1 1 2024-04-16T02:35:48.489 INFO:tasks.workunit.client.1.smithi153.stdout:8/889: dwrite d2/d93/d97/d9c/d27/d5e/f116 [0,4194304] 0 2024-04-16T02:35:48.490 INFO:tasks.workunit.client.1.smithi153.stdout:8/890: chown d2/d93/d97/d9c/d27/d5e/de4 151119 1 2024-04-16T02:35:48.500 INFO:tasks.workunit.client.0.smithi151.stdout:7/376: sync 2024-04-16T02:35:48.501 INFO:tasks.workunit.client.0.smithi151.stdout:7/377: stat d0/d1f/d31/d42/l71 0 2024-04-16T02:35:48.503 INFO:tasks.workunit.client.1.smithi153.stdout:9/993: symlink d4/d6/dc/d2d/d79/dbb/l13c 0 2024-04-16T02:35:48.503 INFO:tasks.workunit.client.1.smithi153.stdout:7/918: write d0/d4/db/d2b/d53/f27 [1586463,20529] 0 2024-04-16T02:35:48.504 INFO:tasks.workunit.client.0.smithi151.stdout:4/460: write d0/d11/d19/d31/d68/f79 [1036069,45250] 0 2024-04-16T02:35:48.506 INFO:tasks.workunit.client.1.smithi153.stdout:2/968: read d9/d71/d8d/f91 [2579872,59685] 0 2024-04-16T02:35:48.509 INFO:tasks.workunit.client.1.smithi153.stdout:8/891: creat d2/d13/d96/dc7/d109/d47/d60/d67/d98/f12a x:0 0 0 2024-04-16T02:35:48.509 INFO:tasks.workunit.client.1.smithi153.stdout:6/903: truncate dc/d47/d93/fb4 2079335 0 2024-04-16T02:35:48.511 INFO:tasks.workunit.client.1.smithi153.stdout:7/919: dwrite d0/d4/db/d2b/d53/d59/dc3/f113 [0,4194304] 0 2024-04-16T02:35:48.514 INFO:tasks.workunit.client.0.smithi151.stdout:1/389: rename d3/d4/d64/d2a/f30 to d3/f8a 0 2024-04-16T02:35:48.517 INFO:tasks.workunit.client.0.smithi151.stdout:4/461: dread - d0/d11/d19/d46/f77 zero size 2024-04-16T02:35:48.519 INFO:tasks.workunit.client.0.smithi151.stdout:4/462: fsync d0/d11/d19/d87/d3a/f75 0 2024-04-16T02:35:48.519 INFO:tasks.workunit.client.1.smithi153.stdout:8/892: dwrite d2/d13/d96/dc7/d109/d47/d60/f7b [0,4194304] 0 2024-04-16T02:35:48.521 INFO:tasks.workunit.client.1.smithi153.stdout:8/893: write d2/d13/d3a/d5b/f9d [1778081,38405] 0 2024-04-16T02:35:48.521 INFO:tasks.workunit.client.1.smithi153.stdout:7/920: creat d0/d4/db/d34/f138 x:0 0 0 2024-04-16T02:35:48.524 INFO:tasks.workunit.client.0.smithi151.stdout:1/390: mkdir d3/d58/df/d38/d8b 0 2024-04-16T02:35:48.526 INFO:tasks.workunit.client.1.smithi153.stdout:5/976: truncate d8/d41/d90/ded/ffa 672296 0 2024-04-16T02:35:48.526 INFO:tasks.workunit.client.0.smithi151.stdout:0/371: getdents dc/d4c 0 2024-04-16T02:35:48.526 INFO:tasks.workunit.client.0.smithi151.stdout:7/378: creat d0/d1f/d31/f86 x:0 0 0 2024-04-16T02:35:48.530 INFO:tasks.workunit.client.1.smithi153.stdout:2/969: dwrite d9/d24/d34/d36/db1/fd8 [0,4194304] 0 2024-04-16T02:35:48.532 INFO:tasks.workunit.client.1.smithi153.stdout:5/977: mknod d8/d41/dbd/dd3/c12f 0 2024-04-16T02:35:48.533 INFO:tasks.workunit.client.0.smithi151.stdout:1/391: rename d3/d4/d64/d2a/d73/f7f to d3/d1f/d48/f8c 0 2024-04-16T02:35:48.533 INFO:tasks.workunit.client.0.smithi151.stdout:0/372: dread dc/d15/f2b [0,4194304] 0 2024-04-16T02:35:48.535 INFO:tasks.workunit.client.0.smithi151.stdout:4/463: rmdir d0/d99/d59/d86 0 2024-04-16T02:35:48.536 INFO:tasks.workunit.client.0.smithi151.stdout:5/488: dwrite d2/d6/d19/d30/d49/d7f/f90 [0,4194304] 0 2024-04-16T02:35:48.536 INFO:tasks.workunit.client.1.smithi153.stdout:6/904: creat dc/d47/d57/d77/d15/f128 x:0 0 0 2024-04-16T02:35:48.540 INFO:tasks.workunit.client.1.smithi153.stdout:2/970: dread d9/d71/d8d/da8/ff5 [0,4194304] 0 2024-04-16T02:35:48.540 INFO:tasks.workunit.client.1.smithi153.stdout:3/941: sync 2024-04-16T02:35:48.541 INFO:tasks.workunit.client.1.smithi153.stdout:8/894: dwrite d2/d1c/fef [0,4194304] 0 2024-04-16T02:35:48.542 INFO:tasks.workunit.client.1.smithi153.stdout:2/971: readlink d9/d24/d27/d2c/d41/d7a/l103 0 2024-04-16T02:35:48.544 INFO:tasks.workunit.client.0.smithi151.stdout:1/392: creat d3/d4/d64/d2a/f8d x:0 0 0 2024-04-16T02:35:48.545 INFO:tasks.workunit.client.0.smithi151.stdout:7/379: chown d0/c10 2954 1 2024-04-16T02:35:48.545 INFO:tasks.workunit.client.1.smithi153.stdout:8/895: fdatasync d2/d13/d96/dc7/d109/d47/d77/ddf/f111 0 2024-04-16T02:35:48.548 INFO:tasks.workunit.client.1.smithi153.stdout:7/921: write d0/d13/fbc [344138,123098] 0 2024-04-16T02:35:48.550 INFO:tasks.workunit.client.1.smithi153.stdout:6/905: mkdir dc/d1a/d2d/d129 0 2024-04-16T02:35:48.553 INFO:tasks.workunit.client.0.smithi151.stdout:7/380: mkdir d0/d2c/d38/d5e/d87 0 2024-04-16T02:35:48.553 INFO:tasks.workunit.client.1.smithi153.stdout:9/994: link d4/d33/dae/dc7/d66/f98 d4/d33/dae/dc7/dd5/f13d 0 2024-04-16T02:35:48.555 INFO:tasks.workunit.client.0.smithi151.stdout:1/393: rmdir d3/d58/df/d32/d3a/d6d 39 2024-04-16T02:35:48.555 INFO:tasks.workunit.client.1.smithi153.stdout:7/922: symlink d0/d4/db/d2b/l139 0 2024-04-16T02:35:48.556 INFO:tasks.workunit.client.0.smithi151.stdout:1/394: dread - d3/d1f/d48/f8c zero size 2024-04-16T02:35:48.558 INFO:tasks.workunit.client.0.smithi151.stdout:5/489: truncate d2/f24 3143927 0 2024-04-16T02:35:48.559 INFO:tasks.workunit.client.1.smithi153.stdout:2/972: dwrite d9/d71/d8d/f120 [4194304,4194304] 0 2024-04-16T02:35:48.559 INFO:tasks.workunit.client.0.smithi151.stdout:5/490: chown d2/d6/d19/d30/f44 6411989 1 2024-04-16T02:35:48.560 INFO:tasks.workunit.client.1.smithi153.stdout:8/896: dwrite d2/d1c/fef [0,4194304] 0 2024-04-16T02:35:48.568 INFO:tasks.workunit.client.0.smithi151.stdout:5/491: rename d2/d6/f25 to d2/f99 0 2024-04-16T02:35:48.570 INFO:tasks.workunit.client.1.smithi153.stdout:2/973: link d9/d71/df2/f102 d9/d71/f138 0 2024-04-16T02:35:48.570 INFO:tasks.workunit.client.0.smithi151.stdout:7/381: getdents d0/d1f 0 2024-04-16T02:35:48.573 INFO:tasks.workunit.client.0.smithi151.stdout:5/492: truncate d2/d6/d19/d1a/f2e 1763806 0 2024-04-16T02:35:48.575 INFO:tasks.workunit.client.1.smithi153.stdout:2/974: write d9/d24/d27/d2c/d41/f56 [69505,7006] 0 2024-04-16T02:35:48.575 INFO:tasks.workunit.client.1.smithi153.stdout:5/978: write d8/d41/dbd/d39/f6b [2353837,106245] 0 2024-04-16T02:35:48.575 INFO:tasks.workunit.client.0.smithi151.stdout:7/382: rmdir d0/d1f/d31/d42 39 2024-04-16T02:35:48.576 INFO:tasks.workunit.client.1.smithi153.stdout:6/906: write f3 [923746,121273] 0 2024-04-16T02:35:48.578 INFO:tasks.workunit.client.0.smithi151.stdout:7/383: dread - d0/d1f/d31/d42/f76 zero size 2024-04-16T02:35:48.579 INFO:tasks.workunit.client.0.smithi151.stdout:7/384: dread - d0/d1f/f29 zero size 2024-04-16T02:35:48.581 INFO:tasks.workunit.client.0.smithi151.stdout:7/385: dread - d0/d1f/d31/d42/f6c zero size 2024-04-16T02:35:48.582 INFO:tasks.workunit.client.1.smithi153.stdout:3/942: dwrite d6/d10/d22/d4d/dbb/fcb [0,4194304] 0 2024-04-16T02:35:48.583 INFO:tasks.workunit.client.0.smithi151.stdout:5/493: dwrite d2/d6/f64 [0,4194304] 0 2024-04-16T02:35:48.583 INFO:tasks.workunit.client.0.smithi151.stdout:1/395: dread d3/d33/d10/f55 [0,4194304] 0 2024-04-16T02:35:48.584 INFO:tasks.workunit.client.1.smithi153.stdout:2/975: dwrite d9/d24/d34/d36/db1/fd8 [0,4194304] 0 2024-04-16T02:35:48.584 INFO:tasks.workunit.client.0.smithi151.stdout:5/494: fsync d2/d6/d19/d62/f7b 0 2024-04-16T02:35:48.585 INFO:tasks.workunit.client.1.smithi153.stdout:3/943: stat d6/d10/d3b/d59/d7c/db7/c128 0 2024-04-16T02:35:48.586 INFO:tasks.workunit.client.0.smithi151.stdout:5/495: dread - d2/d6/d19/d30/d36/d42/d6f/d83/d8b/f8f zero size 2024-04-16T02:35:48.587 INFO:tasks.workunit.client.0.smithi151.stdout:5/496: readlink d2/d6/l15 0 2024-04-16T02:35:48.588 INFO:tasks.workunit.client.1.smithi153.stdout:9/995: dread d4/d6/dc/d2d/fa2 [0,4194304] 0 2024-04-16T02:35:48.598 INFO:tasks.workunit.client.0.smithi151.stdout:4/464: dread d0/d11/d19/d87/f48 [0,4194304] 0 2024-04-16T02:35:48.599 INFO:tasks.workunit.client.1.smithi153.stdout:9/996: dwrite d4/d6/dc/d24/d104/d113/f128 [0,4194304] 0 2024-04-16T02:35:48.635 INFO:tasks.workunit.client.0.smithi151.stdout:5/497: dread d2/f12 [0,4194304] 0 2024-04-16T02:35:48.641 INFO:tasks.workunit.client.0.smithi151.stdout:5/498: dwrite d2/d6/f96 [0,4194304] 0 2024-04-16T02:35:48.642 INFO:tasks.workunit.client.1.smithi153.stdout:1/911: sync 2024-04-16T02:35:48.642 INFO:tasks.workunit.client.0.smithi151.stdout:5/499: dread - d2/d6/d19/d30/d49/f94 zero size 2024-04-16T02:35:48.644 INFO:tasks.workunit.client.0.smithi151.stdout:8/288: sync 2024-04-16T02:35:48.645 INFO:tasks.workunit.client.0.smithi151.stdout:3/259: sync 2024-04-16T02:35:48.645 INFO:tasks.workunit.client.0.smithi151.stdout:6/543: sync 2024-04-16T02:35:48.645 INFO:tasks.workunit.client.0.smithi151.stdout:9/333: sync 2024-04-16T02:35:48.653 INFO:tasks.workunit.client.0.smithi151.stdout:6/544: dwrite d2/d5/df/d23/d56/f84 [0,4194304] 0 2024-04-16T02:35:48.653 INFO:tasks.workunit.client.0.smithi151.stdout:9/334: dwrite d1/d3/fa [4194304,4194304] 0 2024-04-16T02:35:48.654 INFO:tasks.workunit.client.0.smithi151.stdout:6/545: chown d2/d3d/f55 386421 1 2024-04-16T02:35:48.713 INFO:tasks.workunit.client.1.smithi153.stdout:7/923: write d0/d4/db/f50 [1390232,120586] 0 2024-04-16T02:35:48.713 INFO:tasks.workunit.client.1.smithi153.stdout:8/897: truncate d2/d13/d3a/d5b/f9d 1732 0 2024-04-16T02:35:48.721 INFO:tasks.workunit.client.1.smithi153.stdout:9/997: rmdir d4/d6/d1b/d2a 39 2024-04-16T02:35:48.727 INFO:tasks.workunit.client.1.smithi153.stdout:2/976: dwrite d9/d24/d73/fc8 [4194304,4194304] 0 2024-04-16T02:35:48.731 INFO:tasks.workunit.client.1.smithi153.stdout:3/944: truncate d6/d10/d3b/f9d 2944369 0 2024-04-16T02:35:48.732 INFO:tasks.workunit.client.1.smithi153.stdout:9/998: dwrite d4/d6/dc/d2d/d64/db9/dc1/d119/f13a [0,4194304] 0 2024-04-16T02:35:48.735 INFO:tasks.workunit.client.0.smithi151.stdout:7/386: mknod d0/d2c/d38/d3e/c88 0 2024-04-16T02:35:48.737 INFO:tasks.workunit.client.0.smithi151.stdout:1/396: unlink d3/d4/d64/d2a/l41 0 2024-04-16T02:35:48.742 INFO:tasks.workunit.client.0.smithi151.stdout:7/387: dread d0/d1f/d33/f3a [0,4194304] 0 2024-04-16T02:35:48.742 INFO:tasks.workunit.client.0.smithi151.stdout:4/465: unlink d0/d11/d19/d31/d68/l7c 0 2024-04-16T02:35:48.742 INFO:tasks.workunit.client.1.smithi153.stdout:7/924: dread d0/d4/f58 [0,4194304] 0 2024-04-16T02:35:48.743 INFO:tasks.workunit.client.0.smithi151.stdout:4/466: fdatasync d0/d11/d19/d87/f5a 0 2024-04-16T02:35:48.745 INFO:tasks.workunit.client.0.smithi151.stdout:3/260: write d1/d6/f33 [1979436,69335] 0 2024-04-16T02:35:48.748 INFO:tasks.workunit.client.0.smithi151.stdout:6/546: creat d2/d13/d46/fbe x:0 0 0 2024-04-16T02:35:48.750 INFO:tasks.workunit.client.0.smithi151.stdout:1/397: mknod d3/d33/d10/c8e 0 2024-04-16T02:35:48.753 INFO:tasks.workunit.client.0.smithi151.stdout:7/388: mkdir d0/d1f/d33/d89 0 2024-04-16T02:35:48.753 INFO:tasks.workunit.client.0.smithi151.stdout:7/389: stat d0/f18 0 2024-04-16T02:35:48.760 INFO:tasks.workunit.client.0.smithi151.stdout:9/335: creat d1/d3/d6/d17/d44/f6a x:0 0 0 2024-04-16T02:35:48.763 INFO:tasks.workunit.client.0.smithi151.stdout:6/547: dread d2/f1a [0,4194304] 0 2024-04-16T02:35:48.764 INFO:tasks.workunit.client.0.smithi151.stdout:6/548: stat d2/d5/df/d20/l42 0 2024-04-16T02:35:48.765 INFO:tasks.workunit.client.0.smithi151.stdout:1/398: mknod d3/d58/df/d32/d3a/d67/c8f 0 2024-04-16T02:35:48.766 INFO:tasks.workunit.client.0.smithi151.stdout:7/390: mknod d0/d2c/d38/c8a 0 2024-04-16T02:35:48.767 INFO:tasks.workunit.client.0.smithi151.stdout:5/500: creat d2/d6/d19/f9a x:0 0 0 2024-04-16T02:35:48.769 INFO:tasks.workunit.client.1.smithi153.stdout:5/979: getdents d8/d41/d90/d9d 0 2024-04-16T02:35:48.770 INFO:tasks.workunit.client.0.smithi151.stdout:9/336: mkdir d1/d3/d6/d4f/d6b 0 2024-04-16T02:35:48.771 INFO:tasks.workunit.client.1.smithi153.stdout:1/912: symlink d4/da2/l132 0 2024-04-16T02:35:48.771 INFO:tasks.workunit.client.1.smithi153.stdout:5/980: read d8/df/d2f/f60 [601938,46237] 0 2024-04-16T02:35:48.771 INFO:tasks.workunit.client.1.smithi153.stdout:1/913: read - d4/da2/f12d zero size 2024-04-16T02:35:48.772 INFO:tasks.workunit.client.0.smithi151.stdout:6/549: rmdir d2/d5/df/d20/d4f/d81 39 2024-04-16T02:35:48.772 INFO:tasks.workunit.client.1.smithi153.stdout:6/907: creat dc/d47/d57/d77/d15/dac/f12a x:0 0 0 2024-04-16T02:35:48.773 INFO:tasks.workunit.client.1.smithi153.stdout:1/914: write d4/dd/d13/d23/d5f/f10c [223307,36276] 0 2024-04-16T02:35:48.773 INFO:tasks.workunit.client.1.smithi153.stdout:6/908: chown dc/d10/d62/c64 1361011 1 2024-04-16T02:35:48.774 INFO:tasks.workunit.client.1.smithi153.stdout:6/909: stat dc/d47/d57/d77/d15/dac 0 2024-04-16T02:35:48.774 INFO:tasks.workunit.client.0.smithi151.stdout:7/391: dwrite d0/d2c/f3b [0,4194304] 0 2024-04-16T02:35:48.775 INFO:tasks.workunit.client.1.smithi153.stdout:2/977: stat d9/d2b/d18/d3f/d10e/l125 0 2024-04-16T02:35:48.775 INFO:tasks.workunit.client.1.smithi153.stdout:6/910: stat dc/d47/d57/d77/dc4 0 2024-04-16T02:35:48.776 INFO:tasks.workunit.client.0.smithi151.stdout:6/550: readlink d2/d5/df/d23/d82/d8d/d22/lbc 0 2024-04-16T02:35:48.776 INFO:tasks.workunit.client.1.smithi153.stdout:2/978: fsync d9/d24/d34/fce 0 2024-04-16T02:35:48.777 INFO:tasks.workunit.client.1.smithi153.stdout:3/945: mkdir d6/d10/d5b/d127/d12f 0 2024-04-16T02:35:48.777 INFO:tasks.workunit.client.1.smithi153.stdout:9/999: mkdir d4/d6/d1b/ddf/d13e 0 2024-04-16T02:35:48.778 INFO:tasks.workunit.client.0.smithi151.stdout:1/399: creat d3/d4/d64/d1c/f90 x:0 0 0 2024-04-16T02:35:48.779 INFO:tasks.workunit.client.1.smithi153.stdout:5/981: write d8/d1f/f3f [1562391,78256] 0 2024-04-16T02:35:48.780 INFO:tasks.workunit.client.0.smithi151.stdout:6/551: write d2/d5/df/d23/d82/d8d/d22/d40/d3f/f6c [4801278,49038] 0 2024-04-16T02:35:48.781 INFO:tasks.workunit.client.0.smithi151.stdout:9/337: mknod d1/dd/d4a/c6c 0 2024-04-16T02:35:48.781 INFO:tasks.workunit.client.0.smithi151.stdout:1/400: dread - d3/d4/d64/d1c/f90 zero size 2024-04-16T02:35:48.781 INFO:tasks.workunit.client.0.smithi151.stdout:4/467: dwrite d0/d11/d19/f25 [0,4194304] 0 2024-04-16T02:35:48.785 INFO:tasks.workunit.client.0.smithi151.stdout:9/338: truncate d1/d3/d5a/f68 957500 0 2024-04-16T02:35:48.786 INFO:tasks.workunit.client.1.smithi153.stdout:6/911: creat dc/d59/d5b/f12b x:0 0 0 2024-04-16T02:35:48.786 INFO:tasks.workunit.client.0.smithi151.stdout:9/339: fsync d1/dd/f16 0 2024-04-16T02:35:48.787 INFO:tasks.workunit.client.1.smithi153.stdout:2/979: mknod d9/d2b/d1c/d20/c139 0 2024-04-16T02:35:48.787 INFO:tasks.workunit.client.0.smithi151.stdout:3/261: getdents d1/d6/d22 0 2024-04-16T02:35:48.788 INFO:tasks.workunit.client.1.smithi153.stdout:1/915: dread d4/dd/d18/d29/d6e/fb2 [0,4194304] 0 2024-04-16T02:35:48.789 INFO:tasks.workunit.client.0.smithi151.stdout:3/262: write d1/d37/f3c [648491,13254] 0 2024-04-16T02:35:48.792 INFO:tasks.workunit.client.1.smithi153.stdout:7/925: symlink d0/d4/db/d2b/d43/l13a 0 2024-04-16T02:35:48.794 INFO:tasks.workunit.client.0.smithi151.stdout:5/501: write d2/d6/f32 [967624,77429] 0 2024-04-16T02:35:48.795 INFO:tasks.workunit.client.1.smithi153.stdout:8/898: link d2/d13/d96/dc7/d109/d47/d60/d67/d86/da4/cf8 d2/d13/c12b 0 2024-04-16T02:35:48.795 INFO:tasks.workunit.client.0.smithi151.stdout:1/401: write d3/d1f/d48/f8c [686844,28414] 0 2024-04-16T02:35:48.796 INFO:tasks.workunit.client.1.smithi153.stdout:8/899: read d2/d1c/fef [3543024,100700] 0 2024-04-16T02:35:48.797 INFO:tasks.workunit.client.0.smithi151.stdout:7/392: mknod d0/d2c/d38/d5e/d87/c8b 0 2024-04-16T02:35:48.798 INFO:tasks.workunit.client.1.smithi153.stdout:6/912: read - dc/d47/d57/d77/d15/d109/f121 zero size 2024-04-16T02:35:48.802 INFO:tasks.workunit.client.0.smithi151.stdout:7/393: write d0/d1f/d46/f5a [907367,33573] 0 2024-04-16T02:35:48.802 INFO:tasks.workunit.client.0.smithi151.stdout:7/394: stat d0/l20 0 2024-04-16T02:35:48.802 INFO:tasks.workunit.client.1.smithi153.stdout:2/980: rmdir d9/d2b/df9 39 2024-04-16T02:35:48.803 INFO:tasks.workunit.client.0.smithi151.stdout:9/340: creat d1/d3/d10/d20/f6d x:0 0 0 2024-04-16T02:35:48.803 INFO:tasks.workunit.client.0.smithi151.stdout:1/402: dread d3/d33/d10/f55 [0,4194304] 0 2024-04-16T02:35:48.804 INFO:tasks.workunit.client.1.smithi153.stdout:6/913: dwrite dc/d1a/f11c [0,4194304] 0 2024-04-16T02:35:48.807 INFO:tasks.workunit.client.1.smithi153.stdout:5/982: fsync d8/df/fbf 0 2024-04-16T02:35:48.807 INFO:tasks.workunit.client.0.smithi151.stdout:4/468: creat d0/d11/d19/d46/f9c x:0 0 0 2024-04-16T02:35:48.809 INFO:tasks.workunit.client.0.smithi151.stdout:9/341: fdatasync d1/d3/f33 0 2024-04-16T02:35:48.810 INFO:tasks.workunit.client.1.smithi153.stdout:1/916: getdents d4/dd/d13/d7e/de6/d111/d12a 0 2024-04-16T02:35:48.810 INFO:tasks.workunit.client.0.smithi151.stdout:1/403: symlink d3/d4/d64/d40/l91 0 2024-04-16T02:35:48.810 INFO:tasks.workunit.client.0.smithi151.stdout:3/263: link d1/fa d1/f59 0 2024-04-16T02:35:48.812 INFO:tasks.workunit.client.1.smithi153.stdout:2/981: creat d9/d24/d34/d60/ddc/f13a x:0 0 0 2024-04-16T02:35:48.812 INFO:tasks.workunit.client.0.smithi151.stdout:5/502: rename d2/d6/d19/c23 to d2/d6/d19/c9b 0 2024-04-16T02:35:48.815 INFO:tasks.workunit.client.0.smithi151.stdout:7/395: symlink d0/d1f/d4c/d82/l8c 0 2024-04-16T02:35:48.816 INFO:tasks.workunit.client.1.smithi153.stdout:3/946: link d6/d10/d3b/d10d/d8a/d8c/f114 d6/d10/d3b/d59/d7c/de1/f130 0 2024-04-16T02:35:48.817 INFO:tasks.workunit.client.1.smithi153.stdout:3/947: chown d6/d10/d3b/d10d/d8a/fb6 32283 1 2024-04-16T02:35:48.817 INFO:tasks.workunit.client.0.smithi151.stdout:1/404: creat d3/d58/df/d32/f92 x:0 0 0 2024-04-16T02:35:48.817 INFO:tasks.workunit.client.1.smithi153.stdout:7/926: symlink d0/d4/db/d2b/d53/l13b 0 2024-04-16T02:35:48.818 INFO:tasks.workunit.client.0.smithi151.stdout:6/552: dwrite d2/d5/df/d23/d82/d8d/d22/f68 [0,4194304] 0 2024-04-16T02:35:48.819 INFO:tasks.workunit.client.1.smithi153.stdout:6/914: rmdir dc/d47/d57/d101 39 2024-04-16T02:35:48.819 INFO:tasks.workunit.client.0.smithi151.stdout:9/342: rename d1/fc to d1/d3/d6/d17/d44/f6e 0 2024-04-16T02:35:48.820 INFO:tasks.workunit.client.1.smithi153.stdout:6/915: chown dc/d47/d57/d77/d15/dac/fb9 169696033 1 2024-04-16T02:35:48.821 INFO:tasks.workunit.client.0.smithi151.stdout:1/405: fsync d3/d4/d64/d2a/d7a/f86 0 2024-04-16T02:35:48.826 INFO:tasks.workunit.client.0.smithi151.stdout:7/396: creat d0/d1f/d46/f8d x:0 0 0 2024-04-16T02:35:48.827 INFO:tasks.workunit.client.0.smithi151.stdout:7/397: write d0/d1f/d31/f79 [49678,56608] 0 2024-04-16T02:35:48.829 INFO:tasks.workunit.client.1.smithi153.stdout:5/983: truncate d8/d41/dbd/d83/f122 21280 0 2024-04-16T02:35:48.829 INFO:tasks.workunit.client.0.smithi151.stdout:6/553: creat d2/d13/d46/fbf x:0 0 0 2024-04-16T02:35:48.831 INFO:tasks.workunit.client.1.smithi153.stdout:1/917: fdatasync d4/dd/d13/d26/f34 0 2024-04-16T02:35:48.831 INFO:tasks.workunit.client.0.smithi151.stdout:6/554: dread - d2/d5/df/d23/d56/f7e zero size 2024-04-16T02:35:48.833 INFO:tasks.workunit.client.1.smithi153.stdout:3/948: dread d6/d10/d3b/d59/d7c/fa8 [0,4194304] 0 2024-04-16T02:35:48.833 INFO:tasks.workunit.client.0.smithi151.stdout:1/406: rename d3/d58/df/d32/d3a/d67/f69 to d3/d58/df/d32/d3a/d67/f93 0 2024-04-16T02:35:48.834 INFO:tasks.workunit.client.1.smithi153.stdout:3/949: chown d6/d10/d3b/d59/cf3 7091129 1 2024-04-16T02:35:48.835 INFO:tasks.workunit.client.0.smithi151.stdout:7/398: mkdir d0/d2c/d8e 0 2024-04-16T02:35:48.836 INFO:tasks.workunit.client.1.smithi153.stdout:3/950: chown d6/d10/d22/d2c/f60 180031146 1 2024-04-16T02:35:48.836 INFO:tasks.workunit.client.1.smithi153.stdout:7/927: creat d0/d4/db/d2b/d53/d59/d76/dc8/d101/f13c x:0 0 0 2024-04-16T02:35:48.837 INFO:tasks.workunit.client.0.smithi151.stdout:7/399: truncate d0/d1f/f29 917027 0 2024-04-16T02:35:48.838 INFO:tasks.workunit.client.0.smithi151.stdout:7/400: fdatasync d0/d1f/d46/f5d 0 2024-04-16T02:35:48.838 INFO:tasks.workunit.client.1.smithi153.stdout:1/918: dwrite d4/f1f [4194304,4194304] 0 2024-04-16T02:35:48.839 INFO:tasks.workunit.client.1.smithi153.stdout:6/916: mknod dc/d59/c12c 0 2024-04-16T02:35:48.839 INFO:tasks.workunit.client.0.smithi151.stdout:4/469: write d0/f3 [1092518,82409] 0 2024-04-16T02:35:48.841 INFO:tasks.workunit.client.0.smithi151.stdout:4/470: fsync d0/d99/d59/f85 0 2024-04-16T02:35:48.844 INFO:tasks.workunit.client.0.smithi151.stdout:5/503: dread d2/d6/d19/d30/d36/f5b [0,4194304] 0 2024-04-16T02:35:48.848 INFO:tasks.workunit.client.0.smithi151.stdout:1/407: link d3/d33/d10/f6b d3/d1f/f94 0 2024-04-16T02:35:48.850 INFO:tasks.workunit.client.1.smithi153.stdout:3/951: creat d6/d10/dec/df8/f131 x:0 0 0 2024-04-16T02:35:48.851 INFO:tasks.workunit.client.0.smithi151.stdout:7/401: link d0/d1f/d46/f5d d0/d2c/d38/d43/f8f 0 2024-04-16T02:35:48.852 INFO:tasks.workunit.client.0.smithi151.stdout:9/343: unlink d1/d3/d10/d20/c65 0 2024-04-16T02:35:48.852 INFO:tasks.workunit.client.0.smithi151.stdout:4/471: dwrite d0/d11/d19/f25 [0,4194304] 0 2024-04-16T02:35:48.855 INFO:tasks.workunit.client.0.smithi151.stdout:5/504: mkdir d2/d6/d19/d30/d36/d63/d9c 0 2024-04-16T02:35:48.855 INFO:tasks.workunit.client.0.smithi151.stdout:9/344: readlink d1/d3/l15 0 2024-04-16T02:35:48.855 INFO:tasks.workunit.client.0.smithi151.stdout:1/408: symlink d3/d1f/d48/l95 0 2024-04-16T02:35:48.855 INFO:tasks.workunit.client.0.smithi151.stdout:5/505: readlink d2/d6/d19/l22 0 2024-04-16T02:35:48.856 INFO:tasks.workunit.client.1.smithi153.stdout:3/952: symlink d6/d10/d3b/d59/d7c/l132 0 2024-04-16T02:35:48.858 INFO:tasks.workunit.client.0.smithi151.stdout:6/555: getdents d2/d5/df/d23/d82/d8d/d22/d40/d3f 0 2024-04-16T02:35:48.858 INFO:tasks.workunit.client.0.smithi151.stdout:9/345: readlink d1/l4 0 2024-04-16T02:35:48.860 INFO:tasks.workunit.client.0.smithi151.stdout:5/506: mkdir d2/d6/d43/d9d 0 2024-04-16T02:35:48.861 INFO:tasks.workunit.client.0.smithi151.stdout:5/507: chown d2/d6/l66 60794570 1 2024-04-16T02:35:48.865 INFO:tasks.workunit.client.0.smithi151.stdout:6/556: creat d2/d5/d2e/db6/fc0 x:0 0 0 2024-04-16T02:35:48.866 INFO:tasks.workunit.client.1.smithi153.stdout:3/953: read d6/d10/d3b/d10d/f80 [3495512,123371] 0 2024-04-16T02:35:48.866 INFO:tasks.workunit.client.1.smithi153.stdout:1/919: rename d4/dd/c41 to d4/dd/d13/d23/c133 0 2024-04-16T02:35:48.867 INFO:tasks.workunit.client.1.smithi153.stdout:1/920: chown d4/lb 235852121 1 2024-04-16T02:35:48.867 INFO:tasks.workunit.client.0.smithi151.stdout:3/264: dread d1/f59 [0,4194304] 0 2024-04-16T02:35:48.868 INFO:tasks.workunit.client.0.smithi151.stdout:4/472: dwrite d0/d99/f21 [0,4194304] 0 2024-04-16T02:35:48.869 INFO:tasks.workunit.client.0.smithi151.stdout:1/409: rename d3/d58/f54 to d3/d4/f96 0 2024-04-16T02:35:48.873 INFO:tasks.workunit.client.0.smithi151.stdout:6/557: fdatasync d2/d5/df/d23/d82/d8d/d22/d40/d3f/d65/f94 0 2024-04-16T02:35:48.873 INFO:tasks.workunit.client.0.smithi151.stdout:9/346: dread d1/dd/f16 [0,4194304] 0 2024-04-16T02:35:48.873 INFO:tasks.workunit.client.0.smithi151.stdout:1/410: read d3/d33/f36 [316874,90821] 0 2024-04-16T02:35:48.879 INFO:tasks.workunit.client.1.smithi153.stdout:6/917: rename dc/d1a/l6e to dc/d59/d104/l12d 0 2024-04-16T02:35:48.883 INFO:tasks.workunit.client.1.smithi153.stdout:8/900: truncate d2/d13/d96/dc7/d109/d47/fd1 1677306 0 2024-04-16T02:35:48.883 INFO:tasks.workunit.client.0.smithi151.stdout:3/265: creat d1/d6/d22/d23/f5a x:0 0 0 2024-04-16T02:35:48.885 INFO:tasks.workunit.client.1.smithi153.stdout:2/982: write d9/d2b/d1c/d20/d45/f96 [974982,126065] 0 2024-04-16T02:35:48.887 INFO:tasks.workunit.client.1.smithi153.stdout:2/983: fdatasync d9/d24/d27/d9a/fe8 0 2024-04-16T02:35:48.888 INFO:tasks.workunit.client.1.smithi153.stdout:8/901: creat d2/d13/d96/dc7/d109/d47/d60/d67/db7/df1/f12c x:0 0 0 2024-04-16T02:35:48.889 INFO:tasks.workunit.client.0.smithi151.stdout:6/558: write d2/d13/f67 [1399415,115105] 0 2024-04-16T02:35:48.892 INFO:tasks.workunit.client.0.smithi151.stdout:1/411: mkdir d3/d4/d64/d2a/d52/d97 0 2024-04-16T02:35:48.892 INFO:tasks.workunit.client.0.smithi151.stdout:7/402: dwrite d0/d2c/d38/f3d [0,4194304] 0 2024-04-16T02:35:48.894 INFO:tasks.workunit.client.1.smithi153.stdout:5/984: dwrite d8/df/d2e/fdf [0,4194304] 0 2024-04-16T02:35:48.894 INFO:tasks.workunit.client.0.smithi151.stdout:3/266: creat d1/d6/d22/d23/d41/f5b x:0 0 0 2024-04-16T02:35:48.896 INFO:tasks.workunit.client.0.smithi151.stdout:9/347: rmdir d1/d3/d6/d17/d54/d5b 0 2024-04-16T02:35:48.897 INFO:tasks.workunit.client.0.smithi151.stdout:7/403: write d0/d2c/d38/d3e/f70 [288733,19124] 0 2024-04-16T02:35:48.897 INFO:tasks.workunit.client.0.smithi151.stdout:6/559: dread - d2/d5/df/d20/f24 zero size 2024-04-16T02:35:48.898 INFO:tasks.workunit.client.0.smithi151.stdout:1/412: mknod d3/d4/d64/d2a/d73/c98 0 2024-04-16T02:35:48.899 INFO:tasks.workunit.client.1.smithi153.stdout:8/902: creat d2/d93/d97/d9c/d27/d5e/d63/f12d x:0 0 0 2024-04-16T02:35:48.900 INFO:tasks.workunit.client.0.smithi151.stdout:4/473: creat d0/d11/d19/d31/d80/f9d x:0 0 0 2024-04-16T02:35:48.901 INFO:tasks.workunit.client.1.smithi153.stdout:2/984: getdents d9/d24/d34/d60/ddc 0 2024-04-16T02:35:48.902 INFO:tasks.workunit.client.1.smithi153.stdout:6/918: dread dc/d47/d57/d77/d15/f22 [0,4194304] 0 2024-04-16T02:35:48.903 INFO:tasks.workunit.client.0.smithi151.stdout:7/404: unlink d0/d1f/d31/d42/l71 0 2024-04-16T02:35:48.904 INFO:tasks.workunit.client.0.smithi151.stdout:1/413: fsync d3/d4/d64/d2a/f60 0 2024-04-16T02:35:48.904 INFO:tasks.workunit.client.1.smithi153.stdout:3/954: write d6/d10/d3b/d10d/f35 [3749627,93046] 0 2024-04-16T02:35:48.908 INFO:tasks.workunit.client.1.smithi153.stdout:5/985: rename d8/c49 to d8/d41/dbd/d8d/c130 0 2024-04-16T02:35:48.909 INFO:tasks.workunit.client.1.smithi153.stdout:2/985: write d9/d24/d27/d2c/d41/f56 [2614550,62709] 0 2024-04-16T02:35:48.910 INFO:tasks.workunit.client.0.smithi151.stdout:4/474: symlink d0/d11/d19/d31/d4a/d60/l9e 0 2024-04-16T02:35:48.911 INFO:tasks.workunit.client.0.smithi151.stdout:6/560: dread d2/d13/f67 [0,4194304] 0 2024-04-16T02:35:48.911 INFO:tasks.workunit.client.1.smithi153.stdout:7/928: dwrite d0/de8/d110/f69 [0,4194304] 0 2024-04-16T02:35:48.911 INFO:tasks.workunit.client.0.smithi151.stdout:3/267: creat d1/f5c x:0 0 0 2024-04-16T02:35:48.912 INFO:tasks.workunit.client.0.smithi151.stdout:7/405: creat d0/d1f/d33/d89/f90 x:0 0 0 2024-04-16T02:35:48.913 INFO:tasks.workunit.client.1.smithi153.stdout:6/919: dread - dc/d47/d57/f10f zero size 2024-04-16T02:35:48.913 INFO:tasks.workunit.client.1.smithi153.stdout:2/986: fsync d9/d2b/d1c/d20/fde 0 2024-04-16T02:35:48.914 INFO:tasks.workunit.client.1.smithi153.stdout:1/921: dwrite d4/dd/d13/d23/f6d [0,4194304] 0 2024-04-16T02:35:48.918 INFO:tasks.workunit.client.0.smithi151.stdout:1/414: rename d3/d58/df/d32/d3a/d71/c89 to d3/d33/c99 0 2024-04-16T02:35:48.919 INFO:tasks.workunit.client.0.smithi151.stdout:1/415: fdatasync d3/d58/df/d32/f88 0 2024-04-16T02:35:48.920 INFO:tasks.workunit.client.0.smithi151.stdout:7/406: creat d0/d1f/d60/f91 x:0 0 0 2024-04-16T02:35:48.920 INFO:tasks.workunit.client.0.smithi151.stdout:4/475: creat d0/d11/d19/f9f x:0 0 0 2024-04-16T02:35:48.921 INFO:tasks.workunit.client.1.smithi153.stdout:2/987: creat d9/d24/d34/d60/f13b x:0 0 0 2024-04-16T02:35:48.924 INFO:tasks.workunit.client.1.smithi153.stdout:8/903: creat d2/d93/d97/d9c/d27/d5e/d63/f12e x:0 0 0 2024-04-16T02:35:48.926 INFO:tasks.workunit.client.0.smithi151.stdout:1/416: symlink d3/d4/d64/d1c/l9a 0 2024-04-16T02:35:48.927 INFO:tasks.workunit.client.0.smithi151.stdout:7/407: creat d0/d2c/d38/d5e/f92 x:0 0 0 2024-04-16T02:35:48.928 INFO:tasks.workunit.client.1.smithi153.stdout:1/922: rename d4/dd/d13/d23/db6/c125 to d4/da2/dd5/d130/c134 0 2024-04-16T02:35:48.929 INFO:tasks.workunit.client.1.smithi153.stdout:7/929: rename d0/d13/f4f to d0/d13/f13d 0 2024-04-16T02:35:48.930 INFO:tasks.workunit.client.0.smithi151.stdout:1/417: fsync d3/d1f/d48/f8c 0 2024-04-16T02:35:48.931 INFO:tasks.workunit.client.0.smithi151.stdout:7/408: write d0/d27/f2b [2475145,4739] 0 2024-04-16T02:35:48.934 INFO:tasks.workunit.client.1.smithi153.stdout:6/920: creat dc/d47/d57/d77/dc4/f12e x:0 0 0 2024-04-16T02:35:48.935 INFO:tasks.workunit.client.1.smithi153.stdout:2/988: mkdir d9/d2b/d18/d13c 0 2024-04-16T02:35:48.936 INFO:tasks.workunit.client.0.smithi151.stdout:1/418: dread d3/d4/d64/d1c/f44 [0,4194304] 0 2024-04-16T02:35:48.937 INFO:tasks.workunit.client.1.smithi153.stdout:2/989: chown d9/d2b/d18/d3f/d40 250233841 1 2024-04-16T02:35:48.937 INFO:tasks.workunit.client.1.smithi153.stdout:1/923: fdatasync d4/da2/d65/d84/f8c 0 2024-04-16T02:35:48.938 INFO:tasks.workunit.client.0.smithi151.stdout:7/409: rename d0/d27/d73 to d0/d2c/d38/d93 0 2024-04-16T02:35:48.939 INFO:tasks.workunit.client.1.smithi153.stdout:2/990: chown d9/d24/c115 7184848 1 2024-04-16T02:35:48.941 INFO:tasks.workunit.client.1.smithi153.stdout:1/924: write d4/dd/d18/f4b [2822734,50078] 0 2024-04-16T02:35:48.941 INFO:tasks.workunit.client.1.smithi153.stdout:5/986: dwrite d8/df/f97 [0,4194304] 0 2024-04-16T02:35:48.942 INFO:tasks.workunit.client.0.smithi151.stdout:9/348: dwrite d1/d3/f5e [0,4194304] 0 2024-04-16T02:35:48.942 INFO:tasks.workunit.client.0.smithi151.stdout:1/419: creat d3/d33/d10/d4f/f9b x:0 0 0 2024-04-16T02:35:48.943 INFO:tasks.workunit.client.1.smithi153.stdout:7/930: read d0/d4/db/d2b/f40 [543083,111108] 0 2024-04-16T02:35:48.943 INFO:tasks.workunit.client.1.smithi153.stdout:6/921: fdatasync dc/d47/d57/d77/d7a/d97/db7/ffd 0 2024-04-16T02:35:48.943 INFO:tasks.workunit.client.0.smithi151.stdout:9/349: read d1/d3/fa [6037992,128448] 0 2024-04-16T02:35:48.945 INFO:tasks.workunit.client.0.smithi151.stdout:6/561: dread d2/d13/f18 [0,4194304] 0 2024-04-16T02:35:48.947 INFO:tasks.workunit.client.1.smithi153.stdout:3/955: dread d6/d10/d3b/d59/d7c/de1/fe3 [0,4194304] 0 2024-04-16T02:35:48.950 INFO:tasks.workunit.client.1.smithi153.stdout:2/991: mkdir d9/d24/d27/d2c/d5d/d10a/d13d 0 2024-04-16T02:35:48.952 INFO:tasks.workunit.client.0.smithi151.stdout:1/420: symlink d3/d4/d64/d2a/d52/d97/l9c 0 2024-04-16T02:35:48.953 INFO:tasks.workunit.client.1.smithi153.stdout:5/987: dwrite d8/df/d28/d5a/d5e/f108 [0,4194304] 0 2024-04-16T02:35:48.953 INFO:tasks.workunit.client.0.smithi151.stdout:4/476: truncate d0/d11/f53 1700013 0 2024-04-16T02:35:48.953 INFO:tasks.workunit.client.1.smithi153.stdout:3/956: chown d6/c23 24844 1 2024-04-16T02:35:48.954 INFO:tasks.workunit.client.1.smithi153.stdout:5/988: write d8/df/d28/d5a/d66/ffc [75378,77082] 0 2024-04-16T02:35:48.959 INFO:tasks.workunit.client.1.smithi153.stdout:2/992: readlink d9/d2b/l65 0 2024-04-16T02:35:48.960 INFO:tasks.workunit.client.1.smithi153.stdout:2/993: chown d9/d2b/d18/l4d 2776 1 2024-04-16T02:35:48.961 INFO:tasks.workunit.client.1.smithi153.stdout:5/989: unlink d8/df/d23/c2d 0 2024-04-16T02:35:48.962 INFO:tasks.workunit.client.1.smithi153.stdout:1/925: getdents d4/dd/d13/d23/db6 0 2024-04-16T02:35:48.962 INFO:tasks.workunit.client.0.smithi151.stdout:1/421: dwrite d3/d4/d64/d2a/f8d [0,4194304] 0 2024-04-16T02:35:48.962 INFO:tasks.workunit.client.1.smithi153.stdout:5/990: write d8/d1f/fcd [3916361,123207] 0 2024-04-16T02:35:48.963 INFO:tasks.workunit.client.0.smithi151.stdout:1/422: fsync d3/d1f/d48/f8c 0 2024-04-16T02:35:48.964 INFO:tasks.workunit.client.1.smithi153.stdout:5/991: chown d8/df/d23/f93 3259579 1 2024-04-16T02:35:48.964 INFO:tasks.workunit.client.1.smithi153.stdout:2/994: creat d9/d24/d27/d9a/f13e x:0 0 0 2024-04-16T02:35:48.971 INFO:tasks.workunit.client.1.smithi153.stdout:3/957: getdents d6/d10/d3b/d76/d10b 0 2024-04-16T02:35:48.973 INFO:tasks.workunit.client.1.smithi153.stdout:1/926: mkdir d4/dd/d13/ded/dfc/d135 0 2024-04-16T02:35:48.974 INFO:tasks.workunit.client.1.smithi153.stdout:6/922: dwrite dc/d1a/f3a [0,4194304] 0 2024-04-16T02:35:48.977 INFO:tasks.workunit.client.0.smithi151.stdout:9/350: dread d1/d3/d6/d1d/d22/f23 [0,4194304] 0 2024-04-16T02:35:48.977 INFO:tasks.workunit.client.0.smithi151.stdout:6/562: write d2/d13/f18 [1885550,61289] 0 2024-04-16T02:35:48.979 INFO:tasks.workunit.client.0.smithi151.stdout:6/563: write d2/d13/d63/fa2 [904494,36950] 0 2024-04-16T02:35:48.979 INFO:tasks.workunit.client.1.smithi153.stdout:5/992: link d8/d41/dbd/d83/fae d8/df/d2e/d119/f131 0 2024-04-16T02:35:48.980 INFO:tasks.workunit.client.0.smithi151.stdout:6/564: write d2/d13/d46/fbe [755420,65890] 0 2024-04-16T02:35:48.982 INFO:tasks.workunit.client.1.smithi153.stdout:1/927: rename d4/dd/d13/d23/d5f/c10a to d4/dd/d13/d7e/d80/c136 0 2024-04-16T02:35:48.983 INFO:tasks.workunit.client.1.smithi153.stdout:3/958: mkdir d6/d10/d22/d2c/d133 0 2024-04-16T02:35:48.984 INFO:tasks.workunit.client.0.smithi151.stdout:6/565: mknod d2/d13/d63/cc1 0 2024-04-16T02:35:48.984 INFO:tasks.workunit.client.0.smithi151.stdout:9/351: dread d1/d3/d10/f4d [0,4194304] 0 2024-04-16T02:35:48.985 INFO:tasks.workunit.client.1.smithi153.stdout:5/993: fsync d8/df/fbf 0 2024-04-16T02:35:48.987 INFO:tasks.workunit.client.1.smithi153.stdout:2/995: truncate d9/f9e 2444584 0 2024-04-16T02:35:48.987 INFO:tasks.workunit.client.0.smithi151.stdout:9/352: mknod d1/dd/d4a/c6f 0 2024-04-16T02:35:48.988 INFO:tasks.workunit.client.1.smithi153.stdout:1/928: creat d4/da2/dd5/f137 x:0 0 0 2024-04-16T02:35:48.989 INFO:tasks.workunit.client.1.smithi153.stdout:1/929: chown d4/dd/d13/d7e/c124 1327994 1 2024-04-16T02:35:48.991 INFO:tasks.workunit.client.1.smithi153.stdout:3/959: mkdir d6/d10/d3b/d59/d7c/de1/df0/d134 0 2024-04-16T02:35:48.991 INFO:tasks.workunit.client.1.smithi153.stdout:1/930: read d4/da2/d65/fe2 [2941092,14226] 0 2024-04-16T02:35:48.991 INFO:tasks.workunit.client.0.smithi151.stdout:4/477: dread d0/d11/d19/f1b [4194304,4194304] 0 2024-04-16T02:35:48.991 INFO:tasks.workunit.client.0.smithi151.stdout:9/353: mkdir d1/d3/d6/d17/d54/d70 0 2024-04-16T02:35:48.994 INFO:tasks.workunit.client.0.smithi151.stdout:9/354: truncate d1/d3/f1a 4686266 0 2024-04-16T02:35:48.994 INFO:tasks.workunit.client.0.smithi151.stdout:6/566: rename d2/d5/df/d23/d45/d62/f9a to d2/d5/df/d23/d31/d52/fc2 0 2024-04-16T02:35:48.994 INFO:tasks.workunit.client.1.smithi153.stdout:2/996: fdatasync d9/d2b/d1c/d20/f79 0 2024-04-16T02:35:48.995 INFO:tasks.workunit.client.1.smithi153.stdout:3/960: mknod d6/d10/d11f/c135 0 2024-04-16T02:35:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:48 smithi151 ceph-mon[29003]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:35:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:48 smithi151 ceph-mon[29003]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:35:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:48 smithi151 ceph-mon[29003]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:35:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:48 smithi151 ceph-mon[29003]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:35:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:48 smithi151 ceph-mon[29003]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:35:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:48 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:48 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:48.997 INFO:tasks.workunit.client.1.smithi153.stdout:1/931: dread d4/dd/d13/d8d/f121 [0,4194304] 0 2024-04-16T02:35:48.998 INFO:tasks.workunit.client.0.smithi151.stdout:1/423: dread d3/d33/f36 [0,4194304] 0 2024-04-16T02:35:48.998 INFO:tasks.workunit.client.0.smithi151.stdout:9/355: symlink d1/d3/d6/d17/d54/l71 0 2024-04-16T02:35:48.999 INFO:tasks.workunit.client.1.smithi153.stdout:1/932: write d4/dd/d18/f4b [2168205,124819] 0 2024-04-16T02:35:49.000 INFO:tasks.workunit.client.1.smithi153.stdout:3/961: mknod d6/d10/d22/c136 0 2024-04-16T02:35:49.000 INFO:tasks.workunit.client.0.smithi151.stdout:9/356: chown d1/dd/d38/c4c 3288 1 2024-04-16T02:35:49.001 INFO:tasks.workunit.client.0.smithi151.stdout:9/357: readlink d1/l4 0 2024-04-16T02:35:49.001 INFO:tasks.workunit.client.1.smithi153.stdout:2/997: dwrite d9/d24/d27/d2c/f121 [0,4194304] 0 2024-04-16T02:35:49.005 INFO:tasks.workunit.client.0.smithi151.stdout:4/478: mkdir d0/d11/d19/d31/d80/da0 0 2024-04-16T02:35:49.006 INFO:tasks.workunit.client.1.smithi153.stdout:1/933: write d4/da2/d65/fe2 [155019,74299] 0 2024-04-16T02:35:49.008 INFO:tasks.workunit.client.1.smithi153.stdout:2/998: dread - d9/d24/d27/d9a/fb9 zero size 2024-04-16T02:35:49.008 INFO:tasks.workunit.client.0.smithi151.stdout:9/358: mknod d1/dd/d4a/c72 0 2024-04-16T02:35:49.009 INFO:tasks.workunit.client.1.smithi153.stdout:2/999: write d9/d24/d27/d113/f130 [219999,490] 0 2024-04-16T02:35:49.010 INFO:tasks.workunit.client.0.smithi151.stdout:1/424: dwrite d3/d58/df/d32/f92 [0,4194304] 0 2024-04-16T02:35:49.012 INFO:tasks.workunit.client.0.smithi151.stdout:1/425: dread - d3/d4/d64/d1c/f90 zero size 2024-04-16T02:35:49.013 INFO:tasks.workunit.client.0.smithi151.stdout:6/567: dread d2/d5/df/f3c [0,4194304] 0 2024-04-16T02:35:49.013 INFO:tasks.workunit.client.0.smithi151.stdout:1/426: fdatasync d3/d58/df/f6c 0 2024-04-16T02:35:49.013 INFO:tasks.workunit.client.0.smithi151.stdout:4/479: mknod d0/d11/d19/d87/d3a/d61/ca1 0 2024-04-16T02:35:49.017 INFO:tasks.workunit.client.0.smithi151.stdout:4/480: chown d0/d11/f26 13863086 1 2024-04-16T02:35:49.017 INFO:tasks.workunit.client.1.smithi153.stdout:5/994: dwrite d8/d41/dbd/d39/fb4 [4194304,4194304] 0 2024-04-16T02:35:49.018 INFO:tasks.workunit.client.1.smithi153.stdout:5/995: stat d8/df/d2f/d4e/l115 0 2024-04-16T02:35:49.019 INFO:tasks.workunit.client.0.smithi151.stdout:6/568: rename d2/d5/df/d23/d82/d8d/c51 to d2/d5/d2e/dba/cc3 0 2024-04-16T02:35:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:48 smithi153 ceph-mon[36747]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:35:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:48 smithi153 ceph-mon[36747]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:35:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:48 smithi153 ceph-mon[36747]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:35:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:48 smithi153 ceph-mon[36747]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:35:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:48 smithi153 ceph-mon[36747]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:35:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:48 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:48 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:49.022 INFO:tasks.workunit.client.1.smithi153.stdout:1/934: getdents d4/dd/dd4 0 2024-04-16T02:35:49.022 INFO:tasks.workunit.client.0.smithi151.stdout:4/481: rmdir d0/d11/d19/d31/d68 39 2024-04-16T02:35:49.022 INFO:tasks.workunit.client.0.smithi151.stdout:4/482: chown d0/c1a 442 1 2024-04-16T02:35:49.023 INFO:tasks.workunit.client.0.smithi151.stdout:1/427: creat d3/f9d x:0 0 0 2024-04-16T02:35:49.024 INFO:tasks.workunit.client.0.smithi151.stdout:6/569: chown d2/d5/df/d20/d4f/d81 8 1 2024-04-16T02:35:49.025 INFO:tasks.workunit.client.1.smithi153.stdout:3/962: truncate d6/f39 1961383 0 2024-04-16T02:35:49.027 INFO:tasks.workunit.client.0.smithi151.stdout:6/570: symlink d2/d13/d70/lc4 0 2024-04-16T02:35:49.028 INFO:tasks.workunit.client.1.smithi153.stdout:3/963: symlink d6/d10/d3b/d10d/l137 0 2024-04-16T02:35:49.029 INFO:tasks.workunit.client.0.smithi151.stdout:1/428: creat d3/d58/df/f9e x:0 0 0 2024-04-16T02:35:49.030 INFO:tasks.workunit.client.0.smithi151.stdout:4/483: creat d0/d11/d19/d31/fa2 x:0 0 0 2024-04-16T02:35:49.030 INFO:tasks.workunit.client.1.smithi153.stdout:1/935: dwrite d4/da2/d65/fe2 [0,4194304] 0 2024-04-16T02:35:49.030 INFO:tasks.workunit.client.1.smithi153.stdout:1/936: write d4/fc4 [2270808,45254] 0 2024-04-16T02:35:49.032 INFO:tasks.workunit.client.0.smithi151.stdout:6/571: mknod d2/d5/df/d23/d45/d62/cc5 0 2024-04-16T02:35:49.032 INFO:tasks.workunit.client.0.smithi151.stdout:4/484: chown d0/d11/d19/d87/d3a/d65/f7d 1006290 1 2024-04-16T02:35:49.033 INFO:tasks.workunit.client.1.smithi153.stdout:3/964: symlink d6/d10/d3b/d10d/d8a/d8c/l138 0 2024-04-16T02:35:49.034 INFO:tasks.workunit.client.1.smithi153.stdout:3/965: fsync d6/d10/d22/d4d/f122 0 2024-04-16T02:35:49.035 INFO:tasks.workunit.client.0.smithi151.stdout:1/429: creat d3/d4/d64/d2a/d7a/f9f x:0 0 0 2024-04-16T02:35:49.039 INFO:tasks.workunit.client.1.smithi153.stdout:1/937: rename d4/dd/d13/c79 to d4/dd/d13/d8d/d95/d115/c138 0 2024-04-16T02:35:49.041 INFO:tasks.workunit.client.0.smithi151.stdout:6/572: rename d2/f33 to d2/d5/df/d23/d82/d8d/d22/d40/d3f/d65/fc6 0 2024-04-16T02:35:49.042 INFO:tasks.workunit.client.1.smithi153.stdout:5/996: dread d8/df/f110 [0,4194304] 0 2024-04-16T02:35:49.044 INFO:tasks.workunit.client.0.smithi151.stdout:6/573: creat d2/d3d/fc7 x:0 0 0 2024-04-16T02:35:49.044 INFO:tasks.workunit.client.0.smithi151.stdout:9/359: dread d1/d3/f33 [0,4194304] 0 2024-04-16T02:35:49.046 INFO:tasks.workunit.client.1.smithi153.stdout:1/938: rename d4/da2/dd5/d130/l107 to d4/dd/d18/d29/l139 0 2024-04-16T02:35:49.046 INFO:tasks.workunit.client.0.smithi151.stdout:6/574: read d2/d5/df/d23/d56/d72/fac [146599,46181] 0 2024-04-16T02:35:49.047 INFO:tasks.workunit.client.1.smithi153.stdout:5/997: truncate d8/d41/d90/d9d/fd6 174024 0 2024-04-16T02:35:49.048 INFO:tasks.workunit.client.0.smithi151.stdout:9/360: mknod d1/d3/c73 0 2024-04-16T02:35:49.049 INFO:tasks.workunit.client.1.smithi153.stdout:5/998: write d8/df/d28/d5a/d72/f105 [1282999,81370] 0 2024-04-16T02:35:49.051 INFO:tasks.workunit.client.0.smithi151.stdout:6/575: rmdir d2/d13/d70 39 2024-04-16T02:35:49.054 INFO:tasks.workunit.client.0.smithi151.stdout:9/361: dread d1/d3/f1a [4194304,4194304] 0 2024-04-16T02:35:49.055 INFO:tasks.workunit.client.1.smithi153.stdout:3/966: write d6/d10/d3b/f7b [1248128,84879] 0 2024-04-16T02:35:49.057 INFO:tasks.workunit.client.1.smithi153.stdout:5/999: dread d8/df/d28/d5a/fa1 [0,4194304] 0 2024-04-16T02:35:49.058 INFO:tasks.workunit.client.0.smithi151.stdout:1/430: truncate d3/d33/d10/f55 1980405 0 2024-04-16T02:35:49.059 INFO:tasks.workunit.client.0.smithi151.stdout:8/289: sync 2024-04-16T02:35:49.059 INFO:tasks.workunit.client.0.smithi151.stdout:2/490: sync 2024-04-16T02:35:49.059 INFO:tasks.workunit.client.0.smithi151.stdout:0/373: sync 2024-04-16T02:35:49.060 INFO:tasks.workunit.client.1.smithi153.stdout:3/967: unlink d6/d10/d3b/d59/d7c/de1/df0/f100 0 2024-04-16T02:35:49.060 INFO:tasks.workunit.client.0.smithi151.stdout:6/576: dwrite d2/d13/d46/f79 [0,4194304] 0 2024-04-16T02:35:49.061 INFO:tasks.workunit.client.0.smithi151.stdout:6/577: stat d2/d5/l39 0 2024-04-16T02:35:49.061 INFO:tasks.workunit.client.0.smithi151.stdout:1/431: write d3/d4/d64/d2a/d7a/f86 [70869,46212] 0 2024-04-16T02:35:49.063 INFO:tasks.workunit.client.0.smithi151.stdout:9/362: creat d1/d3/d6/f74 x:0 0 0 2024-04-16T02:35:49.064 INFO:tasks.workunit.client.0.smithi151.stdout:0/374: dread - dc/d15/d5c/f6a zero size 2024-04-16T02:35:49.064 INFO:tasks.workunit.client.0.smithi151.stdout:2/491: chown de/d11/d42/d82/f99 46 1 2024-04-16T02:35:49.066 INFO:tasks.workunit.client.0.smithi151.stdout:6/578: readlink d2/d5/df/d23/d31/l35 0 2024-04-16T02:35:49.066 INFO:tasks.workunit.client.0.smithi151.stdout:1/432: dread - d3/d4/d64/d1c/f81 zero size 2024-04-16T02:35:49.067 INFO:tasks.workunit.client.1.smithi153.stdout:3/968: dwrite d6/d10/d22/d2c/f33 [0,4194304] 0 2024-04-16T02:35:49.069 INFO:tasks.workunit.client.0.smithi151.stdout:9/363: read d1/d3/d6/f1b [67776,48134] 0 2024-04-16T02:35:49.069 INFO:tasks.workunit.client.1.smithi153.stdout:1/939: truncate d4/dd/d18/f4b 519869 0 2024-04-16T02:35:49.070 INFO:tasks.workunit.client.0.smithi151.stdout:2/492: truncate de/f27 875137 0 2024-04-16T02:35:49.070 INFO:tasks.workunit.client.0.smithi151.stdout:8/290: dwrite f2 [0,4194304] 0 2024-04-16T02:35:49.073 INFO:tasks.workunit.client.0.smithi151.stdout:9/364: chown d1/d3/d6/d1d/d22/d2c/c3b 84 1 2024-04-16T02:35:49.073 INFO:tasks.workunit.client.0.smithi151.stdout:0/375: creat dc/d15/d34/d67/f78 x:0 0 0 2024-04-16T02:35:49.074 INFO:tasks.workunit.client.0.smithi151.stdout:6/579: rmdir d2/d5/df/d23/d56/d72 39 2024-04-16T02:35:49.074 INFO:tasks.workunit.client.0.smithi151.stdout:0/376: write dc/f47 [370579,85264] 0 2024-04-16T02:35:49.080 INFO:tasks.workunit.client.0.smithi151.stdout:9/365: dwrite d1/d3/d6/d17/f27 [0,4194304] 0 2024-04-16T02:35:49.083 INFO:tasks.workunit.client.0.smithi151.stdout:0/377: dwrite dc/d15/d34/f4d [0,4194304] 0 2024-04-16T02:35:49.085 INFO:tasks.workunit.client.0.smithi151.stdout:9/366: read d1/d3/d6/d17/d40/d56/f61 [70588,25349] 0 2024-04-16T02:35:49.112 INFO:tasks.workunit.client.0.smithi151.stdout:1/433: symlink d3/d4/d64/d2a/d7e/la0 0 2024-04-16T02:35:49.114 INFO:tasks.workunit.client.1.smithi153.stdout:3/969: rename d6/d10/d22/d43/c5a to d6/d10/d11f/c139 0 2024-04-16T02:35:49.114 INFO:tasks.workunit.client.0.smithi151.stdout:8/291: chown d11/d22 3875 1 2024-04-16T02:35:49.115 INFO:tasks.workunit.client.1.smithi153.stdout:3/970: write d6/d10/d3b/d59/f87 [1608972,34786] 0 2024-04-16T02:35:49.119 INFO:tasks.workunit.client.1.smithi153.stdout:3/971: creat d6/d10/d3b/d59/d7c/de1/df0/f13a x:0 0 0 2024-04-16T02:35:49.123 INFO:tasks.workunit.client.1.smithi153.stdout:1/940: write d4/dd/d13/d26/f92 [3745330,50330] 0 2024-04-16T02:35:49.125 INFO:tasks.workunit.client.0.smithi151.stdout:1/434: symlink d3/d58/df/d38/la1 0 2024-04-16T02:35:49.125 INFO:tasks.workunit.client.1.smithi153.stdout:1/941: read - d4/dd/d18/d29/f59 zero size 2024-04-16T02:35:49.126 INFO:tasks.workunit.client.0.smithi151.stdout:1/435: dread - d3/d33/d10/d4f/f9b zero size 2024-04-16T02:35:49.126 INFO:tasks.workunit.client.0.smithi151.stdout:8/292: chown d11/d2a/l51 235895469 1 2024-04-16T02:35:49.130 INFO:tasks.workunit.client.0.smithi151.stdout:2/493: link de/d11/d42/d82/f99 de/d65/d71/d7c/f9d 0 2024-04-16T02:35:49.131 INFO:tasks.workunit.client.1.smithi153.stdout:1/942: dwrite d4/f7 [0,4194304] 0 2024-04-16T02:35:49.131 INFO:tasks.workunit.client.0.smithi151.stdout:0/378: symlink dc/l79 0 2024-04-16T02:35:49.132 INFO:tasks.workunit.client.0.smithi151.stdout:2/494: write de/d15/d1e/f67 [3611873,69128] 0 2024-04-16T02:35:49.132 INFO:tasks.workunit.client.0.smithi151.stdout:0/379: readlink dc/l68 0 2024-04-16T02:35:49.135 INFO:tasks.workunit.client.0.smithi151.stdout:2/495: dread - de/d65/d71/f7f zero size 2024-04-16T02:35:49.135 INFO:tasks.workunit.client.0.smithi151.stdout:0/380: chown dc/d15/d34/f57 58431 1 2024-04-16T02:35:49.136 INFO:tasks.workunit.client.1.smithi153.stdout:1/943: rmdir d4/dd/d13/d8d/d95/d122 0 2024-04-16T02:35:49.137 INFO:tasks.workunit.client.0.smithi151.stdout:8/293: dwrite d11/d2a/f2f [0,4194304] 0 2024-04-16T02:35:49.137 INFO:tasks.workunit.client.0.smithi151.stdout:2/496: stat de/d65/d71/d7c/f9d 0 2024-04-16T02:35:49.138 INFO:tasks.workunit.client.1.smithi153.stdout:1/944: write d4/dd/d18/d29/d6e/d116/f120 [781842,95541] 0 2024-04-16T02:35:49.140 INFO:tasks.workunit.client.0.smithi151.stdout:0/381: rmdir dc 39 2024-04-16T02:35:49.144 INFO:tasks.workunit.client.0.smithi151.stdout:9/367: getdents d1/d3/d6/d1d/d22/d2c 0 2024-04-16T02:35:49.145 INFO:tasks.workunit.client.0.smithi151.stdout:8/294: creat d11/d22/d2e/f5c x:0 0 0 2024-04-16T02:35:49.148 INFO:tasks.workunit.client.0.smithi151.stdout:2/497: symlink de/d15/l9e 0 2024-04-16T02:35:49.150 INFO:tasks.workunit.client.1.smithi153.stdout:3/972: dread d6/d10/d22/d43/f57 [0,4194304] 0 2024-04-16T02:35:49.152 INFO:tasks.workunit.client.0.smithi151.stdout:2/498: symlink de/d65/l9f 0 2024-04-16T02:35:49.153 INFO:tasks.workunit.client.0.smithi151.stdout:2/499: chown de/d15/d1e/l61 1055900 1 2024-04-16T02:35:49.154 INFO:tasks.workunit.client.0.smithi151.stdout:0/382: fsync dc/d15/f23 0 2024-04-16T02:35:49.154 INFO:tasks.workunit.client.1.smithi153.stdout:1/945: dread d4/da2/d65/d82/f114 [0,4194304] 0 2024-04-16T02:35:49.155 INFO:tasks.workunit.client.1.smithi153.stdout:3/973: rename d6/d10/dec/ded/d107 to d6/d10/dec/d13b 0 2024-04-16T02:35:49.158 INFO:tasks.workunit.client.0.smithi151.stdout:9/368: dwrite d1/d3/d6/d17/d44/f6e [0,4194304] 0 2024-04-16T02:35:49.159 INFO:tasks.workunit.client.0.smithi151.stdout:9/369: stat d1/d3/l37 0 2024-04-16T02:35:49.159 INFO:tasks.workunit.client.0.smithi151.stdout:9/370: write d1/d3/f41 [1576750,69384] 0 2024-04-16T02:35:49.163 INFO:tasks.workunit.client.0.smithi151.stdout:1/436: write d3/d58/df/d38/f5a [3428061,29047] 0 2024-04-16T02:35:49.166 INFO:tasks.workunit.client.0.smithi151.stdout:0/383: link dc/f47 dc/d3c/f7a 0 2024-04-16T02:35:49.168 INFO:tasks.workunit.client.0.smithi151.stdout:2/500: link de/d11/d42/d7a/d8b/f74 de/d11/d42/d82/fa0 0 2024-04-16T02:35:49.169 INFO:tasks.workunit.client.0.smithi151.stdout:2/501: chown de/d15/l93 238 1 2024-04-16T02:35:49.170 INFO:tasks.workunit.client.0.smithi151.stdout:2/502: chown de/d3d/d6a/l91 27 1 2024-04-16T02:35:49.171 INFO:tasks.workunit.client.1.smithi153.stdout:1/946: dwrite d4/dd/f1d [0,4194304] 0 2024-04-16T02:35:49.172 INFO:tasks.workunit.client.1.smithi153.stdout:3/974: dread d6/d10/d3b/f85 [0,4194304] 0 2024-04-16T02:35:49.172 INFO:tasks.workunit.client.0.smithi151.stdout:8/295: truncate d11/f38 2152008 0 2024-04-16T02:35:49.173 INFO:tasks.workunit.client.1.smithi153.stdout:1/947: chown d4/dd/d18/d29/d6e/d113/c11c 3296413 1 2024-04-16T02:35:49.173 INFO:tasks.workunit.client.0.smithi151.stdout:9/371: mkdir d1/d75 0 2024-04-16T02:35:49.174 INFO:tasks.workunit.client.0.smithi151.stdout:8/296: truncate d11/d2a/d33/f53 264040 0 2024-04-16T02:35:49.174 INFO:tasks.workunit.client.1.smithi153.stdout:1/948: chown d4/dd/d13/d23/d5f/f10c 1298 1 2024-04-16T02:35:49.180 INFO:tasks.workunit.client.0.smithi151.stdout:2/503: rename de/d11/d42/d82/f99 to de/d11/d42/d82/d95/d4b/fa1 0 2024-04-16T02:35:49.181 INFO:tasks.workunit.client.1.smithi153.stdout:1/949: dwrite d4/da2/f96 [4194304,4194304] 0 2024-04-16T02:35:49.183 INFO:tasks.workunit.client.0.smithi151.stdout:8/297: write d11/f1b [231701,16403] 0 2024-04-16T02:35:49.183 INFO:tasks.workunit.client.1.smithi153.stdout:3/975: write d6/d10/d22/d2c/f5f [1562389,62683] 0 2024-04-16T02:35:49.185 INFO:tasks.workunit.client.0.smithi151.stdout:2/504: mkdir de/d11/d42/d5e/d6c/da2 0 2024-04-16T02:35:49.186 INFO:tasks.workunit.client.0.smithi151.stdout:0/384: write dc/d3c/f48 [1692006,53872] 0 2024-04-16T02:35:49.187 INFO:tasks.workunit.client.1.smithi153.stdout:8/904: sync 2024-04-16T02:35:49.187 INFO:tasks.workunit.client.1.smithi153.stdout:7/931: sync 2024-04-16T02:35:49.187 INFO:tasks.workunit.client.1.smithi153.stdout:6/923: sync 2024-04-16T02:35:49.190 INFO:tasks.workunit.client.1.smithi153.stdout:8/905: dread - d2/d13/d2c/f101 zero size 2024-04-16T02:35:49.193 INFO:tasks.workunit.client.1.smithi153.stdout:1/950: dwrite d4/dd/d18/d29/fd2 [0,4194304] 0 2024-04-16T02:35:49.194 INFO:tasks.workunit.client.1.smithi153.stdout:3/976: link d6/d10/d3b/d59/d7c/c7e d6/d10/dec/d13b/c13c 0 2024-04-16T02:35:49.195 INFO:tasks.workunit.client.0.smithi151.stdout:1/437: truncate d3/d4/d64/d1c/f50 2178473 0 2024-04-16T02:35:49.197 INFO:tasks.workunit.client.1.smithi153.stdout:8/906: chown d2/d13/d96/dc7/d109/d47/d60/d67/d98/lf2 129323469 1 2024-04-16T02:35:49.197 INFO:tasks.workunit.client.0.smithi151.stdout:1/438: fsync d3/d58/df/d38/f74 0 2024-04-16T02:35:49.197 INFO:tasks.workunit.client.0.smithi151.stdout:2/505: chown l9 1503 1 2024-04-16T02:35:49.198 INFO:tasks.workunit.client.1.smithi153.stdout:1/951: mkdir d4/dd/d18/d6c/d13a 0 2024-04-16T02:35:49.198 INFO:tasks.workunit.client.1.smithi153.stdout:7/932: symlink d0/d4/db/d34/l13e 0 2024-04-16T02:35:49.199 INFO:tasks.workunit.client.1.smithi153.stdout:1/952: chown d4/da2/d65 3 1 2024-04-16T02:35:49.200 INFO:tasks.workunit.client.1.smithi153.stdout:1/953: stat d4/f12e 0 2024-04-16T02:35:49.200 INFO:tasks.workunit.client.1.smithi153.stdout:6/924: dwrite dc/d10/f88 [0,4194304] 0 2024-04-16T02:35:49.200 INFO:tasks.workunit.client.0.smithi151.stdout:1/439: write d3/d4/d64/d2a/d7a/f86 [298198,107957] 0 2024-04-16T02:35:49.202 INFO:tasks.workunit.client.0.smithi151.stdout:0/385: truncate dc/ff 5116827 0 2024-04-16T02:35:49.202 INFO:tasks.workunit.client.0.smithi151.stdout:9/372: rename d1/d3/d5a/f68 to d1/dd/d4a/d64/f76 0 2024-04-16T02:35:49.203 INFO:tasks.workunit.client.1.smithi153.stdout:3/977: creat d6/d10/d3b/d59/f13d x:0 0 0 2024-04-16T02:35:49.206 INFO:tasks.workunit.client.0.smithi151.stdout:9/373: write d1/d3/fa [4410045,16781] 0 2024-04-16T02:35:49.210 INFO:tasks.workunit.client.0.smithi151.stdout:2/506: dread de/d15/d1e/f67 [0,4194304] 0 2024-04-16T02:35:49.211 INFO:tasks.workunit.client.1.smithi153.stdout:1/954: rmdir d4/dd/d13/ded/dfc 39 2024-04-16T02:35:49.214 INFO:tasks.workunit.client.1.smithi153.stdout:8/907: fdatasync d2/d13/d96/dc7/d109/d47/fd1 0 2024-04-16T02:35:49.218 INFO:tasks.workunit.client.0.smithi151.stdout:1/440: creat d3/d4/d64/d2a/d73/fa2 x:0 0 0 2024-04-16T02:35:49.218 INFO:tasks.workunit.client.1.smithi153.stdout:3/978: rename d6/d10/d3b/d59/d7c/de1/df0/def/ff9 to d6/d10/d5b/d115/f13e 0 2024-04-16T02:35:49.220 INFO:tasks.workunit.client.1.smithi153.stdout:1/955: fdatasync d4/da2/d65/f102 0 2024-04-16T02:35:49.221 INFO:tasks.workunit.client.0.smithi151.stdout:0/386: mkdir dc/d20/d7b 0 2024-04-16T02:35:49.221 INFO:tasks.workunit.client.0.smithi151.stdout:8/298: dread d11/d2a/f42 [0,4194304] 0 2024-04-16T02:35:49.224 INFO:tasks.workunit.client.0.smithi151.stdout:9/374: rename d1/dd/f35 to d1/d3/d6/d4f/d6b/f77 0 2024-04-16T02:35:49.224 INFO:tasks.workunit.client.0.smithi151.stdout:9/375: readlink d1/dd/d4a/l60 0 2024-04-16T02:35:49.226 INFO:tasks.workunit.client.0.smithi151.stdout:2/507: unlink de/d11/d42/d82/d95/f37 0 2024-04-16T02:35:49.227 INFO:tasks.workunit.client.1.smithi153.stdout:1/956: creat d4/dd/d13/ded/f13b x:0 0 0 2024-04-16T02:35:49.227 INFO:tasks.workunit.client.0.smithi151.stdout:2/508: stat de/d15/d1e/d90 0 2024-04-16T02:35:49.233 INFO:tasks.workunit.client.1.smithi153.stdout:7/933: write d0/d4/db/d47/d8d/f98 [793354,123186] 0 2024-04-16T02:35:49.234 INFO:tasks.workunit.client.1.smithi153.stdout:8/908: write d2/d93/d97/d9c/d27/d5e/de4/df3/f95 [506486,93178] 0 2024-04-16T02:35:49.235 INFO:tasks.workunit.client.1.smithi153.stdout:8/909: write d2/dba/fc9 [648517,11960] 0 2024-04-16T02:35:49.236 INFO:tasks.workunit.client.1.smithi153.stdout:3/979: truncate d6/d10/d22/d2c/f5f 1536412 0 2024-04-16T02:35:49.238 INFO:tasks.workunit.client.1.smithi153.stdout:6/925: dread dc/d47/d57/f70 [0,4194304] 0 2024-04-16T02:35:49.238 INFO:tasks.workunit.client.1.smithi153.stdout:7/934: rmdir d0/d4/db/d2b/d8c/db5 39 2024-04-16T02:35:49.239 INFO:tasks.workunit.client.1.smithi153.stdout:7/935: write d0/d13/f32 [4245695,26221] 0 2024-04-16T02:35:49.241 INFO:tasks.workunit.client.1.smithi153.stdout:1/957: truncate d4/da2/d65/d82/fb0 272636 0 2024-04-16T02:35:49.246 INFO:tasks.workunit.client.1.smithi153.stdout:8/910: mkdir d2/d13/d96/d11d/d12f 0 2024-04-16T02:35:49.247 INFO:tasks.workunit.client.1.smithi153.stdout:8/911: readlink d2/d93/d97/d9c/d27/l10f 0 2024-04-16T02:35:49.247 INFO:tasks.workunit.client.1.smithi153.stdout:1/958: dwrite d4/f7 [0,4194304] 0 2024-04-16T02:35:49.250 INFO:tasks.workunit.client.1.smithi153.stdout:6/926: getdents dc/d1a/d2d/d129 0 2024-04-16T02:35:49.251 INFO:tasks.workunit.client.1.smithi153.stdout:7/936: symlink d0/d4/db/d47/l13f 0 2024-04-16T02:35:49.252 INFO:tasks.workunit.client.1.smithi153.stdout:7/937: stat d0/d4/db/f64 0 2024-04-16T02:35:49.253 INFO:tasks.workunit.client.1.smithi153.stdout:7/938: read d0/de8/d110/f69 [16341,97740] 0 2024-04-16T02:35:49.256 INFO:tasks.workunit.client.1.smithi153.stdout:3/980: creat d6/d10/d3b/d59/d7c/de1/f13f x:0 0 0 2024-04-16T02:35:49.259 INFO:tasks.workunit.client.1.smithi153.stdout:6/927: rename dc/d47/d57/d77/dc4/f117 to dc/d47/d57/d77/d15/de8/f12f 0 2024-04-16T02:35:49.260 INFO:tasks.workunit.client.1.smithi153.stdout:1/959: creat d4/dd/d13/d26/db4/df2/f13c x:0 0 0 2024-04-16T02:35:49.262 INFO:tasks.workunit.client.1.smithi153.stdout:6/928: rename dc/d59/lf9 to dc/d10/l130 0 2024-04-16T02:35:49.264 INFO:tasks.workunit.client.1.smithi153.stdout:8/912: dread d2/de/d34/f99 [0,4194304] 0 2024-04-16T02:35:49.265 INFO:tasks.workunit.client.1.smithi153.stdout:3/981: mkdir d6/d10/d3b/d59/d7c/de1/d140 0 2024-04-16T02:35:49.268 INFO:tasks.workunit.client.1.smithi153.stdout:6/929: read - dc/d47/d57/d101/fbf zero size 2024-04-16T02:35:49.268 INFO:tasks.workunit.client.1.smithi153.stdout:1/960: creat d4/dd/d13/ded/dfc/d135/f13d x:0 0 0 2024-04-16T02:35:49.270 INFO:tasks.workunit.client.1.smithi153.stdout:6/930: write dc/d47/d57/d77/d15/de8/f12f [993752,65156] 0 2024-04-16T02:35:49.271 INFO:tasks.workunit.client.1.smithi153.stdout:3/982: mkdir d6/d10/d3b/d59/d7c/de1/df0/d134/d141 0 2024-04-16T02:35:49.273 INFO:tasks.workunit.client.1.smithi153.stdout:1/961: creat d4/da2/dd5/d130/f13e x:0 0 0 2024-04-16T02:35:49.274 INFO:tasks.workunit.client.1.smithi153.stdout:6/931: creat dc/d47/d7d/f131 x:0 0 0 2024-04-16T02:35:49.275 INFO:tasks.workunit.client.1.smithi153.stdout:8/913: dread d2/d93/d97/d9c/d27/d5e/f73 [0,4194304] 0 2024-04-16T02:35:49.275 INFO:tasks.workunit.client.1.smithi153.stdout:7/939: truncate d0/d4/db/d2b/d8c/fa6 1306391 0 2024-04-16T02:35:49.276 INFO:tasks.workunit.client.1.smithi153.stdout:8/914: chown d2/d13/d2c/l42 36 1 2024-04-16T02:35:49.276 INFO:tasks.workunit.client.1.smithi153.stdout:1/962: mkdir d4/dd/d18/d29/d71/d13f 0 2024-04-16T02:35:49.277 INFO:tasks.workunit.client.1.smithi153.stdout:1/963: readlink d4/dd/d18/l10f 0 2024-04-16T02:35:49.280 INFO:tasks.workunit.client.1.smithi153.stdout:3/983: getdents d6/d10 0 2024-04-16T02:35:49.281 INFO:tasks.workunit.client.1.smithi153.stdout:1/964: rmdir d4/dd/d13/d26 39 2024-04-16T02:35:49.283 INFO:tasks.workunit.client.1.smithi153.stdout:6/932: rename dc/d6b/f6f to dc/d47/d57/d77/d7a/f132 0 2024-04-16T02:35:49.283 INFO:tasks.workunit.client.1.smithi153.stdout:8/915: dwrite d2/d13/d96/dc7/d109/d47/d77/f127 [0,4194304] 0 2024-04-16T02:35:49.287 INFO:tasks.workunit.client.1.smithi153.stdout:1/965: fdatasync d4/dd/d13/d8d/d95/dc7/f12c 0 2024-04-16T02:35:49.289 INFO:tasks.workunit.client.1.smithi153.stdout:3/984: dwrite d6/d10/d22/d4d/dbb/fcb [0,4194304] 0 2024-04-16T02:35:49.291 INFO:tasks.workunit.client.1.smithi153.stdout:6/933: rename dc/d47/d93/c9e to dc/d47/d57/d77/db8/c133 0 2024-04-16T02:35:49.291 INFO:tasks.workunit.client.1.smithi153.stdout:7/940: write d0/f36 [3706978,17433] 0 2024-04-16T02:35:49.295 INFO:tasks.workunit.client.1.smithi153.stdout:3/985: mknod d6/d10/d3b/d59/d7c/de1/df0/d134/c142 0 2024-04-16T02:35:49.298 INFO:tasks.workunit.client.1.smithi153.stdout:3/986: chown d6/d10/c68 4 1 2024-04-16T02:35:49.299 INFO:tasks.workunit.client.1.smithi153.stdout:1/966: symlink d4/dd/d13/d26/l140 0 2024-04-16T02:35:49.300 INFO:tasks.workunit.client.1.smithi153.stdout:3/987: chown d6/d10/d11f/c135 3863200 1 2024-04-16T02:35:49.302 INFO:tasks.workunit.client.1.smithi153.stdout:6/934: dwrite dc/d47/d57/d77/dc4/ff7 [4194304,4194304] 0 2024-04-16T02:35:49.303 INFO:tasks.workunit.client.1.smithi153.stdout:3/988: creat d6/d10/d3b/d59/d7c/de1/df0/def/f143 x:0 0 0 2024-04-16T02:35:49.304 INFO:tasks.workunit.client.1.smithi153.stdout:3/989: read d6/d10/d22/d43/f57 [2191822,110] 0 2024-04-16T02:35:49.307 INFO:tasks.workunit.client.1.smithi153.stdout:3/990: mknod d6/d10/d3b/d76/d10b/c144 0 2024-04-16T02:35:49.308 INFO:tasks.workunit.client.1.smithi153.stdout:1/967: read d4/dd/d13/d26/f92 [1155229,16429] 0 2024-04-16T02:35:49.309 INFO:tasks.workunit.client.1.smithi153.stdout:1/968: write d4/dd/d13/d26/f10d [61999,69842] 0 2024-04-16T02:35:49.311 INFO:tasks.workunit.client.1.smithi153.stdout:6/935: chown dc/d47/d57/d77/db8/c133 8491 1 2024-04-16T02:35:49.312 INFO:tasks.workunit.client.1.smithi153.stdout:8/916: dread d2/f23 [0,4194304] 0 2024-04-16T02:35:49.312 INFO:tasks.workunit.client.1.smithi153.stdout:3/991: truncate d6/d10/d3b/d59/d7c/de1/df0/f13a 686674 0 2024-04-16T02:35:49.312 INFO:tasks.workunit.client.1.smithi153.stdout:6/936: write f4 [958886,14158] 0 2024-04-16T02:35:49.312 INFO:tasks.workunit.client.1.smithi153.stdout:1/969: rmdir d4/dd/dfd 39 2024-04-16T02:35:49.314 INFO:tasks.workunit.client.1.smithi153.stdout:7/941: dread d0/d13/f12b [0,4194304] 0 2024-04-16T02:35:49.317 INFO:tasks.workunit.client.1.smithi153.stdout:8/917: rename d2/d13/l19 to d2/d93/d97/l130 0 2024-04-16T02:35:49.318 INFO:tasks.workunit.client.1.smithi153.stdout:8/918: write d2/f14 [4967120,130694] 0 2024-04-16T02:35:49.318 INFO:tasks.workunit.client.1.smithi153.stdout:7/942: dread d0/de8/d110/daa/fc9 [0,4194304] 0 2024-04-16T02:35:49.320 INFO:tasks.workunit.client.1.smithi153.stdout:1/970: rename d4/da2/d65/d82/c100 to d4/c141 0 2024-04-16T02:35:49.323 INFO:tasks.workunit.client.1.smithi153.stdout:1/971: chown d4/da2/dd5/d130/c134 215241 1 2024-04-16T02:35:49.327 INFO:tasks.workunit.client.1.smithi153.stdout:1/972: creat d4/dd/d13/d7e/d80/f142 x:0 0 0 2024-04-16T02:35:49.327 INFO:tasks.workunit.client.1.smithi153.stdout:8/919: link d2/d1c/cb6 d2/d13/d96/c131 0 2024-04-16T02:35:49.328 INFO:tasks.workunit.client.1.smithi153.stdout:8/920: chown d2/d13/f1d 1787445 1 2024-04-16T02:35:49.329 INFO:tasks.workunit.client.1.smithi153.stdout:3/992: rename d6/d10/d22/d4d/f6e to d6/d10/d22/d43/f145 0 2024-04-16T02:35:49.329 INFO:tasks.workunit.client.1.smithi153.stdout:7/943: write d0/d4/f20 [3119656,34974] 0 2024-04-16T02:35:49.329 INFO:tasks.workunit.client.1.smithi153.stdout:6/937: dread dc/d1a/f58 [0,4194304] 0 2024-04-16T02:35:49.330 INFO:tasks.workunit.client.1.smithi153.stdout:1/973: truncate d4/dd/d18/d29/d6e/fb2 1128248 0 2024-04-16T02:35:49.330 INFO:tasks.workunit.client.1.smithi153.stdout:3/993: stat d6/d10/d22/d4d/dbb/le5 0 2024-04-16T02:35:49.330 INFO:tasks.workunit.client.1.smithi153.stdout:7/944: dread - d0/d4/db/d34/f138 zero size 2024-04-16T02:35:49.331 INFO:tasks.workunit.client.1.smithi153.stdout:3/994: fdatasync d6/d10/d3b/fc5 0 2024-04-16T02:35:49.332 INFO:tasks.workunit.client.1.smithi153.stdout:6/938: unlink dc/l45 0 2024-04-16T02:35:49.332 INFO:tasks.workunit.client.1.smithi153.stdout:3/995: write d6/d10/d22/d4d/da0/d12e/f11b [266688,130874] 0 2024-04-16T02:35:49.334 INFO:tasks.workunit.client.1.smithi153.stdout:6/939: chown dc/d47/d57/d77/d15/f22 1 1 2024-04-16T02:35:49.334 INFO:tasks.workunit.client.1.smithi153.stdout:6/940: chown dc/d1a/dd7 632046 1 2024-04-16T02:35:49.335 INFO:tasks.workunit.client.1.smithi153.stdout:3/996: creat d6/de6/f146 x:0 0 0 2024-04-16T02:35:49.336 INFO:tasks.workunit.client.1.smithi153.stdout:6/941: write dc/d59/d5b/f10a [628418,29034] 0 2024-04-16T02:35:49.337 INFO:tasks.workunit.client.1.smithi153.stdout:3/997: write d6/d10/d3b/d59/d7c/de1/f108 [172312,44305] 0 2024-04-16T02:35:49.338 INFO:tasks.workunit.client.1.smithi153.stdout:1/974: link d4/dd/d13/d26/l8a d4/da2/l143 0 2024-04-16T02:35:49.339 INFO:tasks.workunit.client.1.smithi153.stdout:6/942: write dc/d47/d57/d77/d15/dac/dae/ffc [3339152,95901] 0 2024-04-16T02:35:49.340 INFO:tasks.workunit.client.1.smithi153.stdout:6/943: chown dc/d1a/dd7 0 1 2024-04-16T02:35:49.340 INFO:tasks.workunit.client.1.smithi153.stdout:3/998: creat d6/d10/d3b/d59/f147 x:0 0 0 2024-04-16T02:35:49.341 INFO:tasks.workunit.client.1.smithi153.stdout:1/975: dread d4/dd/d13/d8d/f121 [0,4194304] 0 2024-04-16T02:35:49.343 INFO:tasks.workunit.client.1.smithi153.stdout:3/999: creat d6/d10/d3b/d10d/d8a/d8c/f148 x:0 0 0 2024-04-16T02:35:49.344 INFO:tasks.workunit.client.1.smithi153.stdout:1/976: mkdir d4/dd/d13/d8d/d95/dec/d144 0 2024-04-16T02:35:49.347 INFO:tasks.workunit.client.1.smithi153.stdout:6/944: dwrite dc/d47/d57/d77/d7a/f112 [0,4194304] 0 2024-04-16T02:35:49.349 INFO:tasks.workunit.client.1.smithi153.stdout:7/945: dread d0/d4/db/d34/f79 [0,4194304] 0 2024-04-16T02:35:49.351 INFO:tasks.workunit.client.1.smithi153.stdout:6/945: symlink dc/d59/dfe/l134 0 2024-04-16T02:35:49.352 INFO:tasks.workunit.client.1.smithi153.stdout:1/977: dwrite d4/da2/dd5/f137 [0,4194304] 0 2024-04-16T02:35:49.358 INFO:tasks.workunit.client.0.smithi151.stdout:0/387: rmdir dc/d15/d34/d67/d69 39 2024-04-16T02:35:49.360 INFO:tasks.workunit.client.1.smithi153.stdout:7/946: rename d0/d4/db/d2b/d43/d128/d6c to d0/d4/db/d2b/d53/d59/d76/d140 0 2024-04-16T02:35:49.362 INFO:tasks.workunit.client.0.smithi151.stdout:2/509: creat de/d11/d42/d82/fa3 x:0 0 0 2024-04-16T02:35:49.362 INFO:tasks.workunit.client.1.smithi153.stdout:1/978: link d4/dd/d13/d26/db4/df2/f13c d4/dd/dfd/f145 0 2024-04-16T02:35:49.364 INFO:tasks.workunit.client.0.smithi151.stdout:9/376: dwrite d1/d3/d6/f74 [0,4194304] 0 2024-04-16T02:35:49.364 INFO:tasks.workunit.client.1.smithi153.stdout:7/947: creat d0/f141 x:0 0 0 2024-04-16T02:35:49.366 INFO:tasks.workunit.client.0.smithi151.stdout:2/510: chown de/d11/f7d 170669 1 2024-04-16T02:35:49.368 INFO:tasks.workunit.client.1.smithi153.stdout:7/948: rmdir d0/d4/db/d2b/d43/d6d 39 2024-04-16T02:35:49.368 INFO:tasks.workunit.client.1.smithi153.stdout:7/949: stat d0/d4/db/d47/l13f 0 2024-04-16T02:35:49.369 INFO:tasks.workunit.client.0.smithi151.stdout:9/377: creat d1/d3/d6/d1d/f78 x:0 0 0 2024-04-16T02:35:49.370 INFO:tasks.workunit.client.0.smithi151.stdout:8/299: dread fb [4194304,4194304] 0 2024-04-16T02:35:49.371 INFO:tasks.workunit.client.0.smithi151.stdout:1/441: dwrite d3/d33/d10/f55 [0,4194304] 0 2024-04-16T02:35:49.372 INFO:tasks.workunit.client.0.smithi151.stdout:9/378: fsync d1/d3/d6/f13 0 2024-04-16T02:35:49.374 INFO:tasks.workunit.client.0.smithi151.stdout:9/379: chown d1/dd/d4a/f53 24 1 2024-04-16T02:35:49.377 INFO:tasks.workunit.client.0.smithi151.stdout:8/300: rmdir d11/d22 39 2024-04-16T02:35:49.377 INFO:tasks.workunit.client.0.smithi151.stdout:1/442: write d3/f8a [4099993,24695] 0 2024-04-16T02:35:49.378 INFO:tasks.workunit.client.0.smithi151.stdout:9/380: write d1/d3/d6/d28/d39/f48 [1259520,97191] 0 2024-04-16T02:35:49.384 INFO:tasks.workunit.client.0.smithi151.stdout:8/301: chown d11/d2a/d33/f45 20142 1 2024-04-16T02:35:49.384 INFO:tasks.workunit.client.0.smithi151.stdout:1/443: mkdir d3/d4/d64/d40/da3 0 2024-04-16T02:35:49.385 INFO:tasks.workunit.client.0.smithi151.stdout:8/302: write d11/d2a/d33/f48 [696446,91587] 0 2024-04-16T02:35:49.389 INFO:tasks.workunit.client.0.smithi151.stdout:1/444: write d3/d4/d64/d2a/d52/f6a [1489425,60805] 0 2024-04-16T02:35:49.389 INFO:tasks.workunit.client.0.smithi151.stdout:9/381: rename d1/d3/d6/f2a to d1/d3/d6/d17/d54/f79 0 2024-04-16T02:35:49.389 INFO:tasks.workunit.client.0.smithi151.stdout:8/303: dread - d11/d2a/d33/f45 zero size 2024-04-16T02:35:49.396 INFO:tasks.workunit.client.0.smithi151.stdout:8/304: dread fb [4194304,4194304] 0 2024-04-16T02:35:49.397 INFO:tasks.workunit.client.0.smithi151.stdout:9/382: rename d1/d3/d10/d20 to d1/d3/d6/d17/d7a 0 2024-04-16T02:35:49.400 INFO:tasks.workunit.client.0.smithi151.stdout:1/445: dread d3/d58/df/d32/d3a/f79 [0,4194304] 0 2024-04-16T02:35:49.402 INFO:tasks.workunit.client.0.smithi151.stdout:1/446: chown d3/d58 2497188 1 2024-04-16T02:35:49.402 INFO:tasks.workunit.client.1.smithi153.stdout:7/950: dread d0/d4/db/d2b/d53/d59/f67 [0,4194304] 0 2024-04-16T02:35:49.403 INFO:tasks.workunit.client.0.smithi151.stdout:8/305: mknod d11/d2a/d33/c5d 0 2024-04-16T02:35:49.403 INFO:tasks.workunit.client.0.smithi151.stdout:1/447: write d3/d4/d64/d2a/d7a/f86 [143329,118425] 0 2024-04-16T02:35:49.407 INFO:tasks.workunit.client.0.smithi151.stdout:9/383: rename d1/d3/d6/d17/d54/d70 to d1/d3/d10/d7b 0 2024-04-16T02:35:49.410 INFO:tasks.workunit.client.0.smithi151.stdout:8/306: mknod d11/d2a/d33/c5e 0 2024-04-16T02:35:49.411 INFO:tasks.workunit.client.0.smithi151.stdout:2/511: dread de/d11/d42/d82/d95/d4b/f55 [0,4194304] 0 2024-04-16T02:35:49.411 INFO:tasks.workunit.client.0.smithi151.stdout:8/307: stat d11/d2a/d33/l39 0 2024-04-16T02:35:49.416 INFO:tasks.workunit.client.0.smithi151.stdout:1/448: dwrite d3/d58/df/d32/d3a/d71/f83 [0,4194304] 0 2024-04-16T02:35:49.418 INFO:tasks.workunit.client.0.smithi151.stdout:1/449: chown d3/d33/d10/c8e 1 1 2024-04-16T02:35:49.420 INFO:tasks.workunit.client.0.smithi151.stdout:1/450: chown d3/d1f 5602823 1 2024-04-16T02:35:49.424 INFO:tasks.workunit.client.0.smithi151.stdout:9/384: dwrite d1/d3/d10/f4d [4194304,4194304] 0 2024-04-16T02:35:49.425 INFO:tasks.workunit.client.0.smithi151.stdout:1/451: creat d3/d4/d64/fa4 x:0 0 0 2024-04-16T02:35:49.432 INFO:tasks.workunit.client.0.smithi151.stdout:9/385: link d1/d3/d6/f13 d1/d3/d6/d17/d40/f7c 0 2024-04-16T02:35:49.437 INFO:tasks.workunit.client.0.smithi151.stdout:1/452: getdents d3/d58/df/d32/d3a/d67 0 2024-04-16T02:35:49.439 INFO:tasks.workunit.client.0.smithi151.stdout:1/453: chown d3/d58/df/c43 10311 1 2024-04-16T02:35:49.440 INFO:tasks.workunit.client.0.smithi151.stdout:9/386: rmdir d1/d3/d6/d4f/d57 0 2024-04-16T02:35:49.442 INFO:tasks.workunit.client.0.smithi151.stdout:1/454: chown d3/d33/c99 55382 1 2024-04-16T02:35:49.443 INFO:tasks.workunit.client.0.smithi151.stdout:1/455: write d3/d4/d64/fa4 [37665,56641] 0 2024-04-16T02:35:49.445 INFO:tasks.workunit.client.0.smithi151.stdout:9/387: mkdir d1/d3/d6/d17/d40/d7d 0 2024-04-16T02:35:49.447 INFO:tasks.workunit.client.0.smithi151.stdout:1/456: write d3/d4/d64/d1c/f44 [758833,67389] 0 2024-04-16T02:35:49.448 INFO:tasks.workunit.client.0.smithi151.stdout:1/457: dread - d3/d58/df/d32/f88 zero size 2024-04-16T02:35:49.449 INFO:tasks.workunit.client.0.smithi151.stdout:9/388: rename d1/dd/d38/l3c to d1/d3/d6/d17/d40/l7e 0 2024-04-16T02:35:49.466 INFO:tasks.workunit.client.1.smithi153.stdout:7/951: rename d0/de8/d110/f69 to d0/de8/f142 0 2024-04-16T02:35:49.466 INFO:tasks.workunit.client.1.smithi153.stdout:7/952: stat d0/d4/db 0 2024-04-16T02:35:49.468 INFO:tasks.workunit.client.1.smithi153.stdout:1/979: truncate d4/da2/d65/d84/fbd 3280799 0 2024-04-16T02:35:49.468 INFO:tasks.workunit.client.1.smithi153.stdout:1/980: chown d4/dd/d13/d23/fd3 194289 1 2024-04-16T02:35:49.469 INFO:tasks.workunit.client.1.smithi153.stdout:8/921: sync 2024-04-16T02:35:49.473 INFO:tasks.workunit.client.1.smithi153.stdout:7/953: dwrite d0/de8/d110/d4d/ff4 [0,4194304] 0 2024-04-16T02:35:49.474 INFO:tasks.workunit.client.1.smithi153.stdout:7/954: stat d0/d4/db/d2b/f22 0 2024-04-16T02:35:49.476 INFO:tasks.workunit.client.1.smithi153.stdout:8/922: unlink d2/d1c/l119 0 2024-04-16T02:35:49.482 INFO:tasks.workunit.client.1.smithi153.stdout:7/955: rename d0/d13/f32 to d0/d4/db/d2b/d53/d59/d76/d103/f143 0 2024-04-16T02:35:49.496 INFO:tasks.workunit.client.1.smithi153.stdout:8/923: symlink d2/d93/d97/d9c/d27/dda/l132 0 2024-04-16T02:35:49.498 INFO:tasks.workunit.client.1.smithi153.stdout:7/956: mkdir d0/d4/db/d34/d144 0 2024-04-16T02:35:49.501 INFO:tasks.workunit.client.1.smithi153.stdout:7/957: truncate d0/d4/db/d47/d8d/d93/fdd 289125 0 2024-04-16T02:35:49.504 INFO:tasks.workunit.client.1.smithi153.stdout:7/958: mknod d0/d4/db/d2b/d8c/c145 0 2024-04-16T02:35:49.504 INFO:tasks.workunit.client.1.smithi153.stdout:7/959: stat d0/d4/db/d2b/d43/d128/le4 0 2024-04-16T02:35:49.514 INFO:tasks.workunit.client.0.smithi151.stdout:1/458: write d3/d58/df/d32/d3a/d71/f75 [117098,113969] 0 2024-04-16T02:35:49.516 INFO:tasks.workunit.client.0.smithi151.stdout:1/459: write d3/d58/df/d32/d3a/f66 [3802729,5098] 0 2024-04-16T02:35:49.521 INFO:tasks.workunit.client.0.smithi151.stdout:1/460: chown d3/l4b 0 1 2024-04-16T02:35:49.522 INFO:tasks.workunit.client.0.smithi151.stdout:9/389: dwrite d1/dd/d4a/d64/f76 [0,4194304] 0 2024-04-16T02:35:49.526 INFO:tasks.workunit.client.1.smithi153.stdout:6/946: sync 2024-04-16T02:35:49.526 INFO:tasks.workunit.client.1.smithi153.stdout:6/947: fdatasync f4 0 2024-04-16T02:35:49.527 INFO:tasks.workunit.client.1.smithi153.stdout:6/948: chown dc/d47/d57/d77/d7a/d97/lc1 32499109 1 2024-04-16T02:35:49.528 INFO:tasks.workunit.client.0.smithi151.stdout:9/390: rename d1/dd/l45 to d1/d3/d6/d28/d39/l7f 0 2024-04-16T02:35:49.530 INFO:tasks.workunit.client.0.smithi151.stdout:2/512: dread de/d15/f17 [0,4194304] 0 2024-04-16T02:35:49.531 INFO:tasks.workunit.client.0.smithi151.stdout:1/461: mkdir d3/d58/df/d32/d3a/da5 0 2024-04-16T02:35:49.532 INFO:tasks.workunit.client.1.smithi153.stdout:6/949: mknod dc/d47/d57/d77/d15/dac/dae/c135 0 2024-04-16T02:35:49.537 INFO:tasks.workunit.client.1.smithi153.stdout:6/950: rename dc/d10/fe9 to dc/d47/d57/d77/d15/dac/dae/f136 0 2024-04-16T02:35:49.538 INFO:tasks.workunit.client.0.smithi151.stdout:9/391: creat d1/dd/d4a/d64/f80 x:0 0 0 2024-04-16T02:35:49.538 INFO:tasks.workunit.client.0.smithi151.stdout:2/513: rmdir de/d11/d42/d82/d95/d4b 39 2024-04-16T02:35:49.539 INFO:tasks.workunit.client.0.smithi151.stdout:1/462: creat d3/d4/d64/d40/da3/fa6 x:0 0 0 2024-04-16T02:35:49.541 INFO:tasks.workunit.client.1.smithi153.stdout:6/951: dread dc/d1a/f3a [0,4194304] 0 2024-04-16T02:35:49.547 INFO:tasks.workunit.client.1.smithi153.stdout:6/952: dwrite dc/d10/f6c [4194304,4194304] 0 2024-04-16T02:35:49.553 INFO:tasks.workunit.client.0.smithi151.stdout:2/514: dwrite de/d11/d42/d82/d95/f38 [0,4194304] 0 2024-04-16T02:35:49.554 INFO:tasks.workunit.client.0.smithi151.stdout:2/515: readlink de/d15/d1e/l61 0 2024-04-16T02:35:49.555 INFO:tasks.workunit.client.1.smithi153.stdout:6/953: rmdir dc/d47/d57/d77/d7a/d97/db7 39 2024-04-16T02:35:49.555 INFO:tasks.workunit.client.1.smithi153.stdout:6/954: chown dc/d1a/l71 4911524 1 2024-04-16T02:35:49.558 INFO:tasks.workunit.client.0.smithi151.stdout:2/516: symlink de/d3d/d6a/la4 0 2024-04-16T02:35:49.559 INFO:tasks.workunit.client.0.smithi151.stdout:9/392: link d1/c11 d1/d3/d5a/c81 0 2024-04-16T02:35:49.565 INFO:tasks.workunit.client.0.smithi151.stdout:9/393: creat d1/d3/d5a/f82 x:0 0 0 2024-04-16T02:35:49.570 INFO:tasks.workunit.client.0.smithi151.stdout:9/394: mknod d1/d3/d6/d17/d7a/c83 0 2024-04-16T02:35:49.583 INFO:tasks.workunit.client.0.smithi151.stdout:2/517: dread de/d11/f2b [4194304,4194304] 0 2024-04-16T02:35:49.587 INFO:tasks.workunit.client.0.smithi151.stdout:2/518: mkdir de/d11/d42/d5e/d6c/d6d/da5 0 2024-04-16T02:35:49.591 INFO:tasks.workunit.client.0.smithi151.stdout:9/395: truncate d1/d3/d10/f4b 320135 0 2024-04-16T02:35:49.596 INFO:tasks.workunit.client.0.smithi151.stdout:9/396: truncate d1/d3/d6/d17/d40/d56/f61 435800 0 2024-04-16T02:35:49.598 INFO:tasks.workunit.client.0.smithi151.stdout:9/397: symlink d1/d75/l84 0 2024-04-16T02:35:49.599 INFO:tasks.workunit.client.1.smithi153.stdout:8/924: mkdir d2/d93/d97/d9c/d133 0 2024-04-16T02:35:49.599 INFO:tasks.workunit.client.0.smithi151.stdout:9/398: fsync d1/d3/d6/d17/f27 0 2024-04-16T02:35:49.600 INFO:tasks.workunit.client.1.smithi153.stdout:8/925: readlink d2/de/d34/l114 0 2024-04-16T02:35:49.600 INFO:tasks.workunit.client.1.smithi153.stdout:8/926: write d2/d13/d2c/ffd [3215336,64237] 0 2024-04-16T02:35:49.603 INFO:tasks.workunit.client.1.smithi153.stdout:8/927: mknod d2/d13/de8/c134 0 2024-04-16T02:35:49.607 INFO:tasks.workunit.client.1.smithi153.stdout:8/928: mkdir d2/d13/d3a/dd6/d135 0 2024-04-16T02:35:49.608 INFO:tasks.workunit.client.0.smithi151.stdout:9/399: dwrite d1/d3/d10/f4d [0,4194304] 0 2024-04-16T02:35:49.608 INFO:tasks.workunit.client.1.smithi153.stdout:1/981: dwrite d4/dd/d18/d29/d6e/fb2 [0,4194304] 0 2024-04-16T02:35:49.609 INFO:tasks.workunit.client.0.smithi151.stdout:9/400: readlink d1/dd/d4a/l60 0 2024-04-16T02:35:49.609 INFO:tasks.workunit.client.1.smithi153.stdout:1/982: stat d4/dd/d13/d23/fdd 0 2024-04-16T02:35:49.609 INFO:tasks.workunit.client.1.smithi153.stdout:1/983: chown d4 159 1 2024-04-16T02:35:49.613 INFO:tasks.workunit.client.0.smithi151.stdout:9/401: write d1/d3/d5a/f82 [158206,93040] 0 2024-04-16T02:35:49.618 INFO:tasks.workunit.client.0.smithi151.stdout:9/402: rename d1/dd/d4a/d64/f67 to d1/d3/d10/d7b/f85 0 2024-04-16T02:35:49.621 INFO:tasks.workunit.client.0.smithi151.stdout:9/403: write d1/d3/d6/d28/d39/f48 [2047357,120841] 0 2024-04-16T02:35:49.623 INFO:tasks.workunit.client.1.smithi153.stdout:7/960: dwrite d0/d4/db/d2b/d53/f5d [0,4194304] 0 2024-04-16T02:35:49.627 INFO:tasks.workunit.client.1.smithi153.stdout:1/984: dread d4/dd/dfd/fa9 [0,4194304] 0 2024-04-16T02:35:49.628 INFO:tasks.workunit.client.1.smithi153.stdout:1/985: chown d4/dd/d18/d29/d71/d13f 5229173 1 2024-04-16T02:35:49.628 INFO:tasks.workunit.client.0.smithi151.stdout:9/404: dread d1/d3/d6/f74 [0,4194304] 0 2024-04-16T02:35:49.629 INFO:tasks.workunit.client.1.smithi153.stdout:1/986: truncate d4/dd/d18/d29/d6e/d116/f120 961001 0 2024-04-16T02:35:49.640 INFO:tasks.workunit.client.0.smithi151.stdout:9/405: dwrite d1/d3/d6/f1b [0,4194304] 0 2024-04-16T02:35:49.642 INFO:tasks.workunit.client.0.smithi151.stdout:9/406: chown d1/d3/f41 98578935 1 2024-04-16T02:35:49.645 INFO:tasks.workunit.client.0.smithi151.stdout:9/407: chown d1/d3/l15 7760027 1 2024-04-16T02:35:49.645 INFO:tasks.workunit.client.0.smithi151.stdout:9/408: fdatasync d1/dd/d4a/f53 0 2024-04-16T02:35:49.671 INFO:tasks.workunit.client.0.smithi151.stdout:9/409: dread d1/d3/d6/d17/f31 [0,4194304] 0 2024-04-16T02:35:49.676 INFO:tasks.workunit.client.1.smithi153.stdout:7/961: creat d0/d13/f146 x:0 0 0 2024-04-16T02:35:49.677 INFO:tasks.workunit.client.1.smithi153.stdout:1/987: getdents d4/dd/d18/d29/d71 0 2024-04-16T02:35:49.678 INFO:tasks.workunit.client.1.smithi153.stdout:1/988: chown d4/dd/d13/d7e/d80/ldf 96518 1 2024-04-16T02:35:49.686 INFO:tasks.workunit.client.1.smithi153.stdout:1/989: dwrite d4/da2/d65/d84/f8c [0,4194304] 0 2024-04-16T02:35:49.688 INFO:tasks.workunit.client.1.smithi153.stdout:6/955: dwrite dc/d6b/f106 [0,4194304] 0 2024-04-16T02:35:49.692 INFO:tasks.workunit.client.1.smithi153.stdout:7/962: dread d0/fa [0,4194304] 0 2024-04-16T02:35:49.694 INFO:tasks.workunit.client.1.smithi153.stdout:8/929: sync 2024-04-16T02:35:49.694 INFO:tasks.workunit.client.1.smithi153.stdout:6/956: symlink dc/d10/l137 0 2024-04-16T02:35:49.698 INFO:tasks.workunit.client.1.smithi153.stdout:1/990: dwrite d4/da2/d65/d82/f114 [4194304,4194304] 0 2024-04-16T02:35:49.700 INFO:tasks.workunit.client.1.smithi153.stdout:1/991: write d4/dd/f1d [3547716,85527] 0 2024-04-16T02:35:49.752 INFO:tasks.workunit.client.0.smithi151.stdout:9/410: getdents d1/d3/d6/d1d/d22/d36 0 2024-04-16T02:35:49.761 INFO:tasks.workunit.client.0.smithi151.stdout:3/268: sync 2024-04-16T02:35:49.761 INFO:tasks.workunit.client.0.smithi151.stdout:7/410: sync 2024-04-16T02:35:49.761 INFO:tasks.workunit.client.0.smithi151.stdout:5/508: sync 2024-04-16T02:35:49.767 INFO:tasks.workunit.client.0.smithi151.stdout:7/411: dwrite d0/d2c/d38/d3e/f70 [0,4194304] 0 2024-04-16T02:35:49.786 INFO:tasks.workunit.client.0.smithi151.stdout:9/411: dwrite d1/d3/d6/d17/d40/d56/f61 [0,4194304] 0 2024-04-16T02:35:49.786 INFO:tasks.workunit.client.0.smithi151.stdout:5/509: getdents d2/d6/d19/d62 0 2024-04-16T02:35:49.790 INFO:tasks.workunit.client.0.smithi151.stdout:7/412: getdents d0/d1f/d4c 0 2024-04-16T02:35:49.795 INFO:tasks.workunit.client.0.smithi151.stdout:9/412: fsync d1/d3/d6/d17/f27 0 2024-04-16T02:35:49.795 INFO:tasks.workunit.client.0.smithi151.stdout:7/413: read d0/d2c/d38/f3d [1385415,85184] 0 2024-04-16T02:35:49.795 INFO:tasks.workunit.client.0.smithi151.stdout:3/269: write d1/fa [4271956,40231] 0 2024-04-16T02:35:49.803 INFO:tasks.workunit.client.0.smithi151.stdout:7/414: mkdir d0/d1f/d94 0 2024-04-16T02:35:49.804 INFO:tasks.workunit.client.0.smithi151.stdout:5/510: dwrite d2/d6/d19/d30/d36/d42/f5d [0,4194304] 0 2024-04-16T02:35:49.807 INFO:tasks.workunit.client.0.smithi151.stdout:3/270: symlink d1/d6/d22/d23/d30/l5d 0 2024-04-16T02:35:49.809 INFO:tasks.workunit.client.0.smithi151.stdout:7/415: read - d0/d1f/d31/d42/f6c zero size 2024-04-16T02:35:49.814 INFO:tasks.workunit.client.0.smithi151.stdout:7/416: creat d0/d27/f95 x:0 0 0 2024-04-16T02:35:49.814 INFO:tasks.workunit.client.0.smithi151.stdout:5/511: chown d2/d6/d19/d30/d49 493690 1 2024-04-16T02:35:49.821 INFO:tasks.workunit.client.0.smithi151.stdout:9/413: dwrite d1/dd/d4a/d64/f76 [4194304,4194304] 0 2024-04-16T02:35:49.822 INFO:tasks.workunit.client.0.smithi151.stdout:3/271: dread d1/d6/d22/d23/d30/f45 [0,4194304] 0 2024-04-16T02:35:49.823 INFO:tasks.workunit.client.0.smithi151.stdout:9/414: chown d1/d3/d5a 105 1 2024-04-16T02:35:49.831 INFO:tasks.workunit.client.0.smithi151.stdout:9/415: dread d1/d3/d6/d1d/d22/d36/f3d [0,4194304] 0 2024-04-16T02:35:49.859 INFO:tasks.workunit.client.0.smithi151.stdout:7/417: creat d0/d2c/d38/d93/f96 x:0 0 0 2024-04-16T02:35:49.859 INFO:tasks.workunit.client.0.smithi151.stdout:7/418: readlink d0/d27/l61 0 2024-04-16T02:35:49.863 INFO:tasks.workunit.client.0.smithi151.stdout:5/512: chown d2/d6/d19/d30/d36/d63/c82 53 1 2024-04-16T02:35:49.864 INFO:tasks.workunit.client.0.smithi151.stdout:5/513: readlink d2/d6/d19/l28 0 2024-04-16T02:35:49.864 INFO:tasks.workunit.client.0.smithi151.stdout:5/514: readlink d2/l2b 0 2024-04-16T02:35:49.867 INFO:tasks.workunit.client.0.smithi151.stdout:9/416: mkdir d1/d3/d6/d1d/d22/d2c/d86 0 2024-04-16T02:35:49.869 INFO:tasks.workunit.client.0.smithi151.stdout:7/419: symlink d0/d1f/d46/d4f/l97 0 2024-04-16T02:35:49.873 INFO:tasks.workunit.client.0.smithi151.stdout:5/515: fsync d2/f16 0 2024-04-16T02:35:49.873 INFO:tasks.workunit.client.0.smithi151.stdout:7/420: dread - d0/d1f/d31/d42/f6c zero size 2024-04-16T02:35:49.874 INFO:tasks.workunit.client.0.smithi151.stdout:5/516: dread - d2/d6/d19/f87 zero size 2024-04-16T02:35:49.878 INFO:tasks.workunit.client.0.smithi151.stdout:7/421: mkdir d0/d1f/d4c/d98 0 2024-04-16T02:35:49.881 INFO:tasks.workunit.client.0.smithi151.stdout:9/417: mkdir d1/d3/d6/d17/d40/d7d/d87 0 2024-04-16T02:35:49.882 INFO:tasks.workunit.client.0.smithi151.stdout:9/418: stat d1/d3/d6/d28/l32 0 2024-04-16T02:35:49.883 INFO:tasks.workunit.client.0.smithi151.stdout:5/517: fdatasync d2/d6/f6c 0 2024-04-16T02:35:49.883 INFO:tasks.workunit.client.0.smithi151.stdout:9/419: write d1/d3/d6/d17/d44/f6e [4579508,42622] 0 2024-04-16T02:35:49.885 INFO:tasks.workunit.client.0.smithi151.stdout:3/272: getdents d1/d6/d22/d23/d41 0 2024-04-16T02:35:49.888 INFO:tasks.workunit.client.0.smithi151.stdout:3/273: write d1/d6/f24 [3440221,38545] 0 2024-04-16T02:35:49.891 INFO:tasks.workunit.client.0.smithi151.stdout:3/274: fsync d1/d6/d22/d23/d41/f5b 0 2024-04-16T02:35:49.892 INFO:tasks.workunit.client.0.smithi151.stdout:3/275: stat d1/d6/f31 0 2024-04-16T02:35:49.894 INFO:tasks.workunit.client.0.smithi151.stdout:9/420: dwrite d1/d3/d5a/f69 [0,4194304] 0 2024-04-16T02:35:49.897 INFO:tasks.workunit.client.0.smithi151.stdout:3/276: dread - d1/d6/d20/d2a/d3f/f48 zero size 2024-04-16T02:35:49.903 INFO:tasks.workunit.client.0.smithi151.stdout:3/277: creat d1/d37/f5e x:0 0 0 2024-04-16T02:35:49.907 INFO:tasks.workunit.client.0.smithi151.stdout:3/278: creat d1/d6/d22/d23/d30/d44/f5f x:0 0 0 2024-04-16T02:35:49.907 INFO:tasks.workunit.client.0.smithi151.stdout:3/279: write d1/f5c [580224,44755] 0 2024-04-16T02:35:49.912 INFO:tasks.workunit.client.0.smithi151.stdout:3/280: symlink d1/d6/d20/d3d/l60 0 2024-04-16T02:35:49.920 INFO:tasks.workunit.client.0.smithi151.stdout:9/421: rename d1/d3/d6/d17/d40/f7c to d1/dd/f88 0 2024-04-16T02:35:49.920 INFO:tasks.workunit.client.0.smithi151.stdout:3/281: dwrite d1/d37/f3c [0,4194304] 0 2024-04-16T02:35:49.922 INFO:tasks.workunit.client.0.smithi151.stdout:5/518: dread d2/d6/d19/d30/f39 [0,4194304] 0 2024-04-16T02:35:49.925 INFO:tasks.workunit.client.0.smithi151.stdout:5/519: rmdir d2/d6/d19/d30/d36 39 2024-04-16T02:35:49.926 INFO:tasks.workunit.client.0.smithi151.stdout:5/520: stat d2/d6/d19/d30/d49/d7f 0 2024-04-16T02:35:49.932 INFO:tasks.workunit.client.0.smithi151.stdout:5/521: truncate d2/d6/f1c 1046517 0 2024-04-16T02:35:49.935 INFO:tasks.workunit.client.0.smithi151.stdout:5/522: mknod d2/d6/d19/d93/c9e 0 2024-04-16T02:35:49.943 INFO:tasks.workunit.client.0.smithi151.stdout:5/523: dwrite d2/f99 [0,4194304] 0 2024-04-16T02:35:49.945 INFO:tasks.workunit.client.0.smithi151.stdout:9/422: write d1/dd/f88 [4487052,45076] 0 2024-04-16T02:35:49.946 INFO:tasks.workunit.client.0.smithi151.stdout:7/422: mkdir d0/d1f/d99 0 2024-04-16T02:35:49.951 INFO:tasks.workunit.client.0.smithi151.stdout:9/423: unlink d1/dd/d4a/c6f 0 2024-04-16T02:35:49.958 INFO:tasks.workunit.client.0.smithi151.stdout:3/282: dwrite d1/d6/ff [4194304,4194304] 0 2024-04-16T02:35:49.962 INFO:tasks.workunit.client.0.smithi151.stdout:5/524: dwrite d2/d6/f6c [0,4194304] 0 2024-04-16T02:35:49.963 INFO:tasks.workunit.client.0.smithi151.stdout:7/423: dwrite d0/d2c/f3b [0,4194304] 0 2024-04-16T02:35:49.963 INFO:tasks.workunit.client.0.smithi151.stdout:3/283: chown d1/f17 177762020 1 2024-04-16T02:35:49.971 INFO:tasks.workunit.client.0.smithi151.stdout:9/424: dread d1/dd/f16 [0,4194304] 0 2024-04-16T02:35:49.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:49 smithi151 ceph-mon[29003]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:35:49.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:49 smithi151 ceph-mon[29003]: pgmap v22: 65 pgs: 65 active+clean; 2.7 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 100 MiB/s rd, 324 MiB/s wr, 618 op/s 2024-04-16T02:35:49.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:49 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:49.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:49 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:49.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:49 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:49.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:49 smithi151 ceph-mon[29003]: Reconfiguring prometheus.smithi151 (dependencies changed)... 2024-04-16T02:35:49.973 INFO:tasks.workunit.client.0.smithi151.stdout:5/525: creat d2/d6/d19/d30/d88/f9f x:0 0 0 2024-04-16T02:35:49.978 INFO:tasks.workunit.client.0.smithi151.stdout:5/526: read - d2/d6/d19/d62/f7b zero size 2024-04-16T02:35:49.979 INFO:tasks.workunit.client.0.smithi151.stdout:7/424: dread d0/d1f/d46/f5d [0,4194304] 0 2024-04-16T02:35:49.980 INFO:tasks.workunit.client.0.smithi151.stdout:3/284: dwrite d1/d37/f5e [0,4194304] 0 2024-04-16T02:35:49.983 INFO:tasks.workunit.client.0.smithi151.stdout:9/425: creat d1/d3/d6/d1d/f89 x:0 0 0 2024-04-16T02:35:49.988 INFO:tasks.workunit.client.0.smithi151.stdout:3/285: mknod d1/d6/d20/d2a/c61 0 2024-04-16T02:35:49.989 INFO:tasks.workunit.client.0.smithi151.stdout:3/286: fsync d1/d6/d22/f2b 0 2024-04-16T02:35:49.992 INFO:tasks.workunit.client.0.smithi151.stdout:5/527: rename d2/d6/d19/d62/f91 to d2/d6/d19/d30/d36/d42/d6f/d83/fa0 0 2024-04-16T02:35:49.994 INFO:tasks.workunit.client.0.smithi151.stdout:3/287: fsync d1/d6/d22/f52 0 2024-04-16T02:35:49.995 INFO:tasks.workunit.client.0.smithi151.stdout:5/528: truncate d2/d6/d19/d30/d36/d42/f71 368842 0 2024-04-16T02:35:49.995 INFO:tasks.workunit.client.0.smithi151.stdout:3/288: write d1/d6/d22/f4f [475710,27857] 0 2024-04-16T02:35:49.996 INFO:tasks.workunit.client.0.smithi151.stdout:9/426: rename d1/dd/f16 to d1/dd/f8a 0 2024-04-16T02:35:49.999 INFO:tasks.workunit.client.0.smithi151.stdout:9/427: truncate d1/d3/d6/d17/d40/d56/f61 4547186 0 2024-04-16T02:35:50.000 INFO:tasks.workunit.client.1.smithi153.stdout:7/963: fdatasync d0/d4/db/d2b/d43/fec 0 2024-04-16T02:35:50.007 INFO:tasks.workunit.client.0.smithi151.stdout:3/289: link d1/d6/d20/d2a/d3f/d43/f57 d1/d6/d22/d23/f62 0 2024-04-16T02:35:50.007 INFO:tasks.workunit.client.0.smithi151.stdout:7/425: write d0/f11 [281903,114276] 0 2024-04-16T02:35:50.009 INFO:tasks.workunit.client.0.smithi151.stdout:3/290: chown d1/d6/f1b 3350 1 2024-04-16T02:35:50.010 INFO:tasks.workunit.client.0.smithi151.stdout:7/426: write d0/d2c/d38/d3e/f70 [2872886,65390] 0 2024-04-16T02:35:50.016 INFO:tasks.workunit.client.0.smithi151.stdout:9/428: getdents d1/dd/d4a 0 2024-04-16T02:35:50.018 INFO:tasks.workunit.client.0.smithi151.stdout:9/429: write d1/d3/f33 [993115,28615] 0 2024-04-16T02:35:50.019 INFO:tasks.workunit.client.1.smithi153.stdout:6/957: dread dc/d47/d57/fe3 [0,4194304] 0 2024-04-16T02:35:50.021 INFO:tasks.workunit.client.0.smithi151.stdout:5/529: dwrite d2/f1f [0,4194304] 0 2024-04-16T02:35:50.021 INFO:tasks.workunit.client.0.smithi151.stdout:9/430: read d1/dd/d4a/f53 [567759,21471] 0 2024-04-16T02:35:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:49 smithi153 ceph-mon[36747]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:35:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:49 smithi153 ceph-mon[36747]: pgmap v22: 65 pgs: 65 active+clean; 2.7 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 100 MiB/s rd, 324 MiB/s wr, 618 op/s 2024-04-16T02:35:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:49 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:49 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:49 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:35:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:49 smithi153 ceph-mon[36747]: Reconfiguring prometheus.smithi151 (dependencies changed)... 2024-04-16T02:35:50.028 INFO:tasks.workunit.client.0.smithi151.stdout:9/431: fdatasync d1/d3/f33 0 2024-04-16T02:35:50.031 INFO:tasks.workunit.client.0.smithi151.stdout:3/291: dwrite d1/f26 [0,4194304] 0 2024-04-16T02:35:50.040 INFO:tasks.workunit.client.0.smithi151.stdout:7/427: dread d0/d1f/d33/f3a [0,4194304] 0 2024-04-16T02:35:50.042 INFO:tasks.workunit.client.0.smithi151.stdout:7/428: write d0/d1f/d46/d4f/f57 [66035,38869] 0 2024-04-16T02:35:50.043 INFO:tasks.workunit.client.0.smithi151.stdout:7/429: fsync d0/d1f/d31/f79 0 2024-04-16T02:35:50.045 INFO:tasks.workunit.client.0.smithi151.stdout:7/430: truncate d0/d1f/d31/d42/f6c 323158 0 2024-04-16T02:35:50.059 INFO:tasks.workunit.client.0.smithi151.stdout:5/530: dread d2/f99 [4194304,4194304] 0 2024-04-16T02:35:50.065 INFO:tasks.workunit.client.0.smithi151.stdout:7/431: dread d0/f22 [0,4194304] 0 2024-04-16T02:35:50.066 INFO:tasks.workunit.client.0.smithi151.stdout:7/432: chown d0/d2c/f3b 660850382 1 2024-04-16T02:35:50.067 INFO:tasks.workunit.client.0.smithi151.stdout:7/433: dread - d0/d2c/d38/d93/f96 zero size 2024-04-16T02:35:50.079 INFO:tasks.workunit.client.0.smithi151.stdout:5/531: dread d2/f5 [0,4194304] 0 2024-04-16T02:35:50.247 INFO:tasks.workunit.client.1.smithi153.stdout:8/930: creat d2/d13/de8/d122/f136 x:0 0 0 2024-04-16T02:35:50.380 INFO:tasks.workunit.client.1.smithi153.stdout:7/964: rename d0/d4/db/d47/d8d/d93/d8b/fb0 to d0/de8/d106/f147 0 2024-04-16T02:35:50.381 INFO:tasks.workunit.client.1.smithi153.stdout:1/992: mknod d4/da2/d65/c146 0 2024-04-16T02:35:50.382 INFO:tasks.workunit.client.0.smithi151.stdout:4/485: sync 2024-04-16T02:35:50.403 INFO:tasks.workunit.client.1.smithi153.stdout:8/931: rename d2/d13/de8 to d2/d93/d97/d9c/d27/d5e/d63/d137 0 2024-04-16T02:35:50.403 INFO:tasks.workunit.client.1.smithi153.stdout:8/932: fsync d2/f23 0 2024-04-16T02:35:50.406 INFO:tasks.workunit.client.1.smithi153.stdout:8/933: creat d2/de/d3f/f138 x:0 0 0 2024-04-16T02:35:50.409 INFO:tasks.workunit.client.1.smithi153.stdout:6/958: dwrite dc/d47/d93/faa [0,4194304] 0 2024-04-16T02:35:50.409 INFO:tasks.workunit.client.1.smithi153.stdout:6/959: write f3 [1944857,55800] 0 2024-04-16T02:35:50.429 INFO:tasks.workunit.client.0.smithi151.stdout:9/432: rename d1/d3/d6/d1d/d22/d2c/f2d to d1/d75/f8b 0 2024-04-16T02:35:50.432 INFO:tasks.workunit.client.0.smithi151.stdout:9/433: truncate d1/d3/d6/d17/d44/f6a 550968 0 2024-04-16T02:35:50.438 INFO:tasks.workunit.client.0.smithi151.stdout:7/434: creat d0/d2c/d38/d43/d78/f9a x:0 0 0 2024-04-16T02:35:50.439 INFO:tasks.workunit.client.0.smithi151.stdout:7/435: stat d0/d2c/d38/d43 0 2024-04-16T02:35:50.439 INFO:tasks.workunit.client.0.smithi151.stdout:4/486: creat d0/d9b/fa3 x:0 0 0 2024-04-16T02:35:50.440 INFO:tasks.workunit.client.0.smithi151.stdout:7/436: fdatasync d0/f11 0 2024-04-16T02:35:50.442 INFO:tasks.workunit.client.0.smithi151.stdout:9/434: mknod d1/d75/c8c 0 2024-04-16T02:35:50.442 INFO:tasks.workunit.client.0.smithi151.stdout:7/437: write d0/d2c/d38/d43/d78/f9a [1008069,43949] 0 2024-04-16T02:35:50.442 INFO:tasks.workunit.client.0.smithi151.stdout:9/435: chown d1/dd 614 1 2024-04-16T02:35:50.443 INFO:tasks.workunit.client.0.smithi151.stdout:9/436: stat d1/d3/d6/d1d/d22 0 2024-04-16T02:35:50.447 INFO:tasks.workunit.client.0.smithi151.stdout:4/487: creat d0/d11/d19/d87/d3a/d61/fa4 x:0 0 0 2024-04-16T02:35:50.449 INFO:tasks.workunit.client.0.smithi151.stdout:5/532: rename d2/d6/d19/d1a/f3b to d2/d6/d19/d30/fa1 0 2024-04-16T02:35:50.451 INFO:tasks.workunit.client.0.smithi151.stdout:9/437: mkdir d1/d3/d6/d17/d54/d8d 0 2024-04-16T02:35:50.453 INFO:tasks.workunit.client.0.smithi151.stdout:3/292: truncate d1/f2e 91243 0 2024-04-16T02:35:50.455 INFO:tasks.workunit.client.0.smithi151.stdout:9/438: mkdir d1/d3/d5a/d8e 0 2024-04-16T02:35:50.457 INFO:tasks.workunit.client.0.smithi151.stdout:7/438: getdents d0/d2c/d38 0 2024-04-16T02:35:50.459 INFO:tasks.workunit.client.0.smithi151.stdout:5/533: dwrite d2/d6/d19/d1a/d4c/f92 [0,4194304] 0 2024-04-16T02:35:50.463 INFO:tasks.workunit.client.0.smithi151.stdout:4/488: creat d0/d11/d19/d31/d80/fa5 x:0 0 0 2024-04-16T02:35:50.464 INFO:tasks.workunit.client.0.smithi151.stdout:7/439: dread d0/d2c/f3b [0,4194304] 0 2024-04-16T02:35:50.465 INFO:tasks.workunit.client.0.smithi151.stdout:5/534: readlink d2/d6/d19/d1a/l6a 0 2024-04-16T02:35:50.466 INFO:tasks.workunit.client.0.smithi151.stdout:7/440: stat d0/d2c/d38/d3e/c68 0 2024-04-16T02:35:50.467 INFO:tasks.workunit.client.0.smithi151.stdout:7/441: stat d0/d2c/d38/d85 0 2024-04-16T02:35:50.474 INFO:tasks.workunit.client.0.smithi151.stdout:4/489: fdatasync d0/d11/d19/d46/f58 0 2024-04-16T02:35:50.476 INFO:tasks.workunit.client.0.smithi151.stdout:5/535: creat d2/d6/d19/d30/d36/d63/d9c/fa2 x:0 0 0 2024-04-16T02:35:50.476 INFO:tasks.workunit.client.0.smithi151.stdout:3/293: truncate d1/d6/d22/f4f 357254 0 2024-04-16T02:35:50.478 INFO:tasks.workunit.client.0.smithi151.stdout:5/536: chown d2/d6/d19/d30/d36/d63/c7c 136 1 2024-04-16T02:35:50.480 INFO:tasks.workunit.client.0.smithi151.stdout:5/537: unlink d2/d6/d19/d1a/d4c/c97 0 2024-04-16T02:35:50.483 INFO:tasks.workunit.client.0.smithi151.stdout:3/294: dread d1/d37/f38 [0,4194304] 0 2024-04-16T02:35:50.485 INFO:tasks.workunit.client.0.smithi151.stdout:9/439: dread d1/dd/f88 [0,4194304] 0 2024-04-16T02:35:50.486 INFO:tasks.workunit.client.0.smithi151.stdout:5/538: rename d2/d6/d19/d30/d36/d42/f5d to d2/d6/d19/d30/d36/d42/d6f/fa3 0 2024-04-16T02:35:50.486 INFO:tasks.workunit.client.0.smithi151.stdout:7/442: dwrite d0/d1f/d33/f3a [4194304,4194304] 0 2024-04-16T02:35:50.488 INFO:tasks.workunit.client.0.smithi151.stdout:3/295: mkdir d1/d6/d22/d23/d30/d44/d63 0 2024-04-16T02:35:50.489 INFO:tasks.workunit.client.0.smithi151.stdout:5/539: read d2/d5e/f7e [3544370,28465] 0 2024-04-16T02:35:50.491 INFO:tasks.workunit.client.0.smithi151.stdout:7/443: truncate d0/d1f/d46/d4f/f50 702472 0 2024-04-16T02:35:50.494 INFO:tasks.workunit.client.0.smithi151.stdout:5/540: symlink d2/d6/d19/d30/d49/la4 0 2024-04-16T02:35:50.497 INFO:tasks.workunit.client.0.smithi151.stdout:3/296: dwrite d1/d6/d22/f4d [0,4194304] 0 2024-04-16T02:35:50.499 INFO:tasks.workunit.client.0.smithi151.stdout:7/444: dread d0/f22 [0,4194304] 0 2024-04-16T02:35:50.501 INFO:tasks.workunit.client.0.smithi151.stdout:7/445: truncate d0/d1f/d60/f91 263472 0 2024-04-16T02:35:50.501 INFO:tasks.workunit.client.0.smithi151.stdout:9/440: link d1/d3/d6/d17/d54/f79 d1/d3/d6/d28/f8f 0 2024-04-16T02:35:50.503 INFO:tasks.workunit.client.0.smithi151.stdout:9/441: chown d1/d3/d10/d7b/f85 75031946 1 2024-04-16T02:35:50.503 INFO:tasks.workunit.client.0.smithi151.stdout:3/297: chown d1/lb 5202799 1 2024-04-16T02:35:50.504 INFO:tasks.workunit.client.0.smithi151.stdout:3/298: stat d1/d6/l18 0 2024-04-16T02:35:50.505 INFO:tasks.workunit.client.0.smithi151.stdout:7/446: truncate d0/f11 3559076 0 2024-04-16T02:35:50.508 INFO:tasks.workunit.client.0.smithi151.stdout:9/442: creat d1/d3/d6/d17/d54/d8d/f90 x:0 0 0 2024-04-16T02:35:50.509 INFO:tasks.workunit.client.0.smithi151.stdout:7/447: dread d0/f22 [0,4194304] 0 2024-04-16T02:35:50.509 INFO:tasks.workunit.client.0.smithi151.stdout:3/299: symlink d1/d6/d22/d23/d30/d44/l64 0 2024-04-16T02:35:50.510 INFO:tasks.workunit.client.0.smithi151.stdout:7/448: chown d0/d27 485950660 1 2024-04-16T02:35:50.511 INFO:tasks.workunit.client.0.smithi151.stdout:9/443: symlink d1/d3/d5a/l91 0 2024-04-16T02:35:50.512 INFO:tasks.workunit.client.0.smithi151.stdout:3/300: creat d1/d6/d20/d54/f65 x:0 0 0 2024-04-16T02:35:50.514 INFO:tasks.workunit.client.0.smithi151.stdout:9/444: creat d1/dd/d4a/f92 x:0 0 0 2024-04-16T02:35:50.516 INFO:tasks.workunit.client.0.smithi151.stdout:3/301: mkdir d1/d6/d22/d23/d66 0 2024-04-16T02:35:50.518 INFO:tasks.workunit.client.0.smithi151.stdout:7/449: getdents d0/d1f/d46/d4f 0 2024-04-16T02:35:50.520 INFO:tasks.workunit.client.0.smithi151.stdout:3/302: mknod d1/d6/d4a/d4e/c67 0 2024-04-16T02:35:50.520 INFO:tasks.workunit.client.0.smithi151.stdout:5/541: dwrite d2/d6/f27 [0,4194304] 0 2024-04-16T02:35:50.524 INFO:tasks.workunit.client.0.smithi151.stdout:3/303: creat d1/d6/d20/f68 x:0 0 0 2024-04-16T02:35:50.524 INFO:tasks.workunit.client.0.smithi151.stdout:5/542: symlink d2/d6/d19/d30/d36/d42/d6f/d83/la5 0 2024-04-16T02:35:50.526 INFO:tasks.workunit.client.0.smithi151.stdout:3/304: fdatasync d1/f3e 0 2024-04-16T02:35:50.527 INFO:tasks.workunit.client.0.smithi151.stdout:3/305: dread - d1/d6/d20/d54/f65 zero size 2024-04-16T02:35:50.528 INFO:tasks.workunit.client.0.smithi151.stdout:5/543: mknod d2/d6/d19/d1a/d4c/ca6 0 2024-04-16T02:35:50.529 INFO:tasks.workunit.client.0.smithi151.stdout:5/544: chown d2/d5e/f7e 22696 1 2024-04-16T02:35:50.532 INFO:tasks.workunit.client.0.smithi151.stdout:5/545: symlink d2/d6/d19/d30/la7 0 2024-04-16T02:35:50.533 INFO:tasks.workunit.client.0.smithi151.stdout:5/546: fsync d2/d6/d19/f5f 0 2024-04-16T02:35:50.535 INFO:tasks.workunit.client.0.smithi151.stdout:5/547: fsync d2/d6/f34 0 2024-04-16T02:35:50.535 INFO:tasks.workunit.client.0.smithi151.stdout:3/306: dwrite d1/d6/d20/d2a/f35 [0,4194304] 0 2024-04-16T02:35:50.539 INFO:tasks.workunit.client.0.smithi151.stdout:5/548: write d2/d6/d19/d30/f3f [4419052,120059] 0 2024-04-16T02:35:50.539 INFO:tasks.workunit.client.0.smithi151.stdout:3/307: write d1/d6/d22/d23/d41/f5b [375913,78108] 0 2024-04-16T02:35:50.541 INFO:tasks.workunit.client.0.smithi151.stdout:3/308: write d1/d6/f31 [429309,93659] 0 2024-04-16T02:35:50.543 INFO:tasks.workunit.client.0.smithi151.stdout:5/549: mkdir d2/d6/d19/d30/d36/d42/d6f/da8 0 2024-04-16T02:35:50.544 INFO:tasks.workunit.client.0.smithi151.stdout:5/550: chown d2/d6/d19/d1a/f79 6143 1 2024-04-16T02:35:50.545 INFO:tasks.workunit.client.0.smithi151.stdout:5/551: write d2/d6/fa [4849383,4489] 0 2024-04-16T02:35:50.546 INFO:tasks.workunit.client.0.smithi151.stdout:3/309: mkdir d1/d69 0 2024-04-16T02:35:50.550 INFO:tasks.workunit.client.0.smithi151.stdout:7/450: dread d0/d1f/f28 [0,4194304] 0 2024-04-16T02:35:50.554 INFO:tasks.workunit.client.0.smithi151.stdout:5/552: dwrite d2/d6/d19/d30/d49/d7f/f8a [0,4194304] 0 2024-04-16T02:35:50.556 INFO:tasks.workunit.client.0.smithi151.stdout:3/310: rmdir d1/d6/d20/d2a/d3f 39 2024-04-16T02:35:50.556 INFO:tasks.workunit.client.0.smithi151.stdout:3/311: stat d1/d6/d20/f68 0 2024-04-16T02:35:50.558 INFO:tasks.workunit.client.0.smithi151.stdout:7/451: creat d0/d2c/d38/f9b x:0 0 0 2024-04-16T02:35:50.559 INFO:tasks.workunit.client.0.smithi151.stdout:5/553: fsync d2/d6/d19/d1a/f40 0 2024-04-16T02:35:50.563 INFO:tasks.workunit.client.0.smithi151.stdout:7/452: truncate d0/d1f/d46/d4f/f50 1207278 0 2024-04-16T02:35:50.564 INFO:tasks.workunit.client.0.smithi151.stdout:5/554: symlink d2/d6/d19/d30/la9 0 2024-04-16T02:35:50.565 INFO:tasks.workunit.client.0.smithi151.stdout:3/312: dwrite d1/d6/d22/d23/d41/f5b [0,4194304] 0 2024-04-16T02:35:50.567 INFO:tasks.workunit.client.0.smithi151.stdout:7/453: creat d0/d2c/d38/d3e/f9c x:0 0 0 2024-04-16T02:35:50.573 INFO:tasks.workunit.client.0.smithi151.stdout:7/454: dread d0/d1f/d31/f79 [0,4194304] 0 2024-04-16T02:35:50.574 INFO:tasks.workunit.client.0.smithi151.stdout:3/313: dwrite d1/d6/f33 [0,4194304] 0 2024-04-16T02:35:50.576 INFO:tasks.workunit.client.0.smithi151.stdout:7/455: fdatasync d0/d1f/f40 0 2024-04-16T02:35:50.577 INFO:tasks.workunit.client.0.smithi151.stdout:5/555: write d2/d6/d43/f6d [3285561,106618] 0 2024-04-16T02:35:50.581 INFO:tasks.workunit.client.0.smithi151.stdout:7/456: write d0/f1d [1990130,54184] 0 2024-04-16T02:35:50.583 INFO:tasks.workunit.client.0.smithi151.stdout:3/314: unlink d1/d6/d20/d2a/l25 0 2024-04-16T02:35:50.584 INFO:tasks.workunit.client.0.smithi151.stdout:7/457: rename d0/d1f/d31/d42/c4d to d0/d1f/d46/d4f/c9d 0 2024-04-16T02:35:50.587 INFO:tasks.workunit.client.0.smithi151.stdout:5/556: getdents d2/d6/d19 0 2024-04-16T02:35:50.590 INFO:tasks.workunit.client.0.smithi151.stdout:7/458: rename d0/f1d to d0/f9e 0 2024-04-16T02:35:50.591 INFO:tasks.workunit.client.0.smithi151.stdout:7/459: readlink d0/d2c/d38/d3e/l5b 0 2024-04-16T02:35:50.593 INFO:tasks.workunit.client.0.smithi151.stdout:5/557: link d2/d6/d19/d30/d36/d63/f89 d2/d6/d19/d69/faa 0 2024-04-16T02:35:50.594 INFO:tasks.workunit.client.0.smithi151.stdout:7/460: creat d0/d27/f9f x:0 0 0 2024-04-16T02:35:50.595 INFO:tasks.workunit.client.0.smithi151.stdout:5/558: write d2/d6/d19/d30/d49/d7f/f90 [1203636,15180] 0 2024-04-16T02:35:50.599 INFO:tasks.workunit.client.0.smithi151.stdout:7/461: rename d0/d2c/d38/d43/f8f to d0/d1f/d4c/fa0 0 2024-04-16T02:35:50.600 INFO:tasks.workunit.client.0.smithi151.stdout:7/462: dread - d0/d1f/d4c/f81 zero size 2024-04-16T02:35:50.602 INFO:tasks.workunit.client.0.smithi151.stdout:3/315: dwrite d1/f2e [0,4194304] 0 2024-04-16T02:35:50.603 INFO:tasks.workunit.client.0.smithi151.stdout:5/559: mknod d2/d6/d19/d30/d88/cab 0 2024-04-16T02:35:50.606 INFO:tasks.workunit.client.0.smithi151.stdout:5/560: dread - d2/d6/d19/d30/d36/d63/f89 zero size 2024-04-16T02:35:50.609 INFO:tasks.workunit.client.0.smithi151.stdout:7/463: dwrite d0/d1f/f2a [0,4194304] 0 2024-04-16T02:35:50.609 INFO:tasks.workunit.client.0.smithi151.stdout:3/316: symlink d1/d6/d22/d23/l6a 0 2024-04-16T02:35:50.610 INFO:tasks.workunit.client.0.smithi151.stdout:7/464: dread - d0/d1f/d4c/f81 zero size 2024-04-16T02:35:50.611 INFO:tasks.workunit.client.0.smithi151.stdout:3/317: fsync d1/d6/d22/d23/d30/d44/f5f 0 2024-04-16T02:35:50.616 INFO:tasks.workunit.client.0.smithi151.stdout:7/465: rename d0/d1f/f1e to d0/d1f/d94/fa1 0 2024-04-16T02:35:50.618 INFO:tasks.workunit.client.0.smithi151.stdout:5/561: symlink d2/d6/d19/d30/d36/d42/d6f/da8/lac 0 2024-04-16T02:35:50.618 INFO:tasks.workunit.client.0.smithi151.stdout:5/562: stat d2/d6/l31 0 2024-04-16T02:35:50.621 INFO:tasks.workunit.client.0.smithi151.stdout:7/466: unlink d0/d1f/d31/f79 0 2024-04-16T02:35:50.624 INFO:tasks.workunit.client.0.smithi151.stdout:5/563: symlink d2/d6/d19/d30/d36/d42/lad 0 2024-04-16T02:35:50.625 INFO:tasks.workunit.client.0.smithi151.stdout:3/318: mkdir d1/d6/d20/d2a/d6b 0 2024-04-16T02:35:50.628 INFO:tasks.workunit.client.0.smithi151.stdout:3/319: fdatasync d1/d6/f21 0 2024-04-16T02:35:50.628 INFO:tasks.workunit.client.0.smithi151.stdout:7/467: dwrite d0/d1f/d33/d56/f84 [0,4194304] 0 2024-04-16T02:35:50.629 INFO:tasks.workunit.client.0.smithi151.stdout:5/564: truncate d2/d6/f1c 1432932 0 2024-04-16T02:35:50.631 INFO:tasks.workunit.client.0.smithi151.stdout:7/468: mknod d0/d1f/d33/d56/ca2 0 2024-04-16T02:35:50.632 INFO:tasks.workunit.client.0.smithi151.stdout:7/469: dread - d0/d1f/d46/f8d zero size 2024-04-16T02:35:50.633 INFO:tasks.workunit.client.0.smithi151.stdout:5/565: fdatasync d2/f2d 0 2024-04-16T02:35:50.634 INFO:tasks.workunit.client.0.smithi151.stdout:5/566: write d2/f8e [807922,13899] 0 2024-04-16T02:35:50.635 INFO:tasks.workunit.client.0.smithi151.stdout:7/470: creat d0/d1f/d4c/d82/fa3 x:0 0 0 2024-04-16T02:35:50.638 INFO:tasks.workunit.client.0.smithi151.stdout:5/567: creat d2/d6/d19/d30/d49/d85/fae x:0 0 0 2024-04-16T02:35:50.639 INFO:tasks.workunit.client.0.smithi151.stdout:5/568: truncate d2/d6/d19/d30/f70 698658 0 2024-04-16T02:35:50.640 INFO:tasks.workunit.client.0.smithi151.stdout:7/471: creat d0/d1f/d4c/d98/fa4 x:0 0 0 2024-04-16T02:35:50.642 INFO:tasks.workunit.client.0.smithi151.stdout:5/569: read d2/d6/d43/f67 [899961,35840] 0 2024-04-16T02:35:50.643 INFO:tasks.workunit.client.0.smithi151.stdout:7/472: write d0/d1f/f28 [2808047,8831] 0 2024-04-16T02:35:50.645 INFO:tasks.workunit.client.0.smithi151.stdout:5/570: unlink d2/d6/d19/d30/c59 0 2024-04-16T02:35:50.649 INFO:tasks.workunit.client.0.smithi151.stdout:7/473: link d0/d2c/d38/d85/l54 d0/la5 0 2024-04-16T02:35:50.650 INFO:tasks.workunit.client.0.smithi151.stdout:7/474: read d0/d1f/f29 [671865,55714] 0 2024-04-16T02:35:50.652 INFO:tasks.workunit.client.0.smithi151.stdout:7/475: mkdir d0/d27/da6 0 2024-04-16T02:35:50.653 INFO:tasks.workunit.client.0.smithi151.stdout:7/476: chown d0/d1f/d31/d42 21 1 2024-04-16T02:35:50.655 INFO:tasks.workunit.client.0.smithi151.stdout:7/477: mknod d0/d1f/d33/ca7 0 2024-04-16T02:35:50.655 INFO:tasks.workunit.client.0.smithi151.stdout:5/571: write d2/d6/d19/d30/d49/f6e [848642,78656] 0 2024-04-16T02:35:50.657 INFO:tasks.workunit.client.0.smithi151.stdout:7/478: mkdir d0/d1f/da8 0 2024-04-16T02:35:50.658 INFO:tasks.workunit.client.0.smithi151.stdout:5/572: mkdir d2/d6/d19/d30/d36/d42/d6f/da8/daf 0 2024-04-16T02:35:50.658 INFO:tasks.workunit.client.0.smithi151.stdout:7/479: write d0/d2c/d38/d93/f96 [580880,85513] 0 2024-04-16T02:35:50.659 INFO:tasks.workunit.client.0.smithi151.stdout:5/573: chown d2/d6/f1e 62056 1 2024-04-16T02:35:50.661 INFO:tasks.workunit.client.0.smithi151.stdout:7/480: unlink d0/d2c/d38/l66 0 2024-04-16T02:35:50.664 INFO:tasks.workunit.client.0.smithi151.stdout:5/574: creat d2/d6/d19/d30/fb0 x:0 0 0 2024-04-16T02:35:50.666 INFO:tasks.workunit.client.0.smithi151.stdout:5/575: mknod d2/d6/d43/cb1 0 2024-04-16T02:35:50.667 INFO:tasks.workunit.client.0.smithi151.stdout:7/481: dwrite d0/f22 [0,4194304] 0 2024-04-16T02:35:50.669 INFO:tasks.workunit.client.0.smithi151.stdout:7/482: creat d0/d1f/d46/fa9 x:0 0 0 2024-04-16T02:35:50.671 INFO:tasks.workunit.client.0.smithi151.stdout:7/483: creat d0/d2c/d38/d5e/faa x:0 0 0 2024-04-16T02:35:50.671 INFO:tasks.workunit.client.0.smithi151.stdout:7/484: chown d0/d2c/f3b 923 1 2024-04-16T02:35:50.674 INFO:tasks.workunit.client.0.smithi151.stdout:7/485: mkdir d0/d2c/d38/dab 0 2024-04-16T02:35:50.674 INFO:tasks.workunit.client.0.smithi151.stdout:5/576: dread d2/d6/d19/f4b [0,4194304] 0 2024-04-16T02:35:50.690 INFO:tasks.workunit.client.0.smithi151.stdout:7/486: write d0/d1f/f40 [94864,124739] 0 2024-04-16T02:35:50.691 INFO:tasks.workunit.client.0.smithi151.stdout:5/577: write d2/f5 [4066239,18131] 0 2024-04-16T02:35:50.691 INFO:tasks.workunit.client.0.smithi151.stdout:7/487: readlink d0/d27/l7c 0 2024-04-16T02:35:50.693 INFO:tasks.workunit.client.0.smithi151.stdout:5/578: unlink d2/d6/l95 0 2024-04-16T02:35:50.695 INFO:tasks.workunit.client.0.smithi151.stdout:7/488: unlink d0/d2c/d38/d93/f96 0 2024-04-16T02:35:50.697 INFO:tasks.workunit.client.0.smithi151.stdout:5/579: mkdir d2/d6/d19/d30/d36/d42/d6f/db2 0 2024-04-16T02:35:50.700 INFO:tasks.workunit.client.0.smithi151.stdout:5/580: mknod d2/d6/d43/d9d/cb3 0 2024-04-16T02:35:50.703 INFO:tasks.workunit.client.0.smithi151.stdout:7/489: dwrite d0/d1f/d4c/d82/fa3 [0,4194304] 0 2024-04-16T02:35:50.706 INFO:tasks.workunit.client.0.smithi151.stdout:7/490: fdatasync d0/d1f/f52 0 2024-04-16T02:35:50.724 INFO:tasks.workunit.client.0.smithi151.stdout:7/491: dread d0/d1f/d31/d42/f45 [4194304,4194304] 0 2024-04-16T02:35:50.725 INFO:tasks.workunit.client.0.smithi151.stdout:5/581: dread d2/d6/f34 [0,4194304] 0 2024-04-16T02:35:50.726 INFO:tasks.workunit.client.0.smithi151.stdout:7/492: unlink d0/d27/l7c 0 2024-04-16T02:35:50.726 INFO:tasks.workunit.client.0.smithi151.stdout:5/582: mknod d2/d6/d43/d9d/cb4 0 2024-04-16T02:35:50.728 INFO:tasks.workunit.client.0.smithi151.stdout:7/493: unlink d0/l8 0 2024-04-16T02:35:50.756 INFO:tasks.workunit.client.0.smithi151.stdout:7/494: rename d0/d1f/d46/d4f/c9d to d0/d27/da6/cac 0 2024-04-16T02:35:50.757 INFO:tasks.workunit.client.0.smithi151.stdout:5/583: dwrite d2/d6/d19/d30/fb0 [0,4194304] 0 2024-04-16T02:35:50.759 INFO:tasks.workunit.client.0.smithi151.stdout:7/495: dread d0/d1f/d60/f91 [0,4194304] 0 2024-04-16T02:35:50.761 INFO:tasks.workunit.client.0.smithi151.stdout:7/496: rmdir d0/d2c 39 2024-04-16T02:35:50.764 INFO:tasks.workunit.client.0.smithi151.stdout:6/580: sync 2024-04-16T02:35:50.765 INFO:tasks.workunit.client.0.smithi151.stdout:6/581: readlink d2/l4 0 2024-04-16T02:35:50.767 INFO:tasks.workunit.client.0.smithi151.stdout:5/584: getdents d2/d6/d19/d93 0 2024-04-16T02:35:50.768 INFO:tasks.workunit.client.0.smithi151.stdout:5/585: chown d2/d6/fa 0 1 2024-04-16T02:35:50.768 INFO:tasks.workunit.client.0.smithi151.stdout:6/582: mkdir d2/d5/df/d23/d82/d8d/dc8 0 2024-04-16T02:35:50.769 INFO:tasks.workunit.client.0.smithi151.stdout:7/497: creat d0/d2c/d38/d5e/d87/fad x:0 0 0 2024-04-16T02:35:50.771 INFO:tasks.workunit.client.0.smithi151.stdout:6/583: creat d2/d13/d3a/fc9 x:0 0 0 2024-04-16T02:35:50.772 INFO:tasks.workunit.client.0.smithi151.stdout:5/586: mknod d2/d6/d19/d30/d36/d42/d6f/da8/cb5 0 2024-04-16T02:35:50.775 INFO:tasks.workunit.client.0.smithi151.stdout:5/587: write d2/d6/d19/d30/d36/d42/d6f/fa3 [4015308,47643] 0 2024-04-16T02:35:50.777 INFO:tasks.workunit.client.0.smithi151.stdout:5/588: mknod d2/d6/d43/cb6 0 2024-04-16T02:35:50.779 INFO:tasks.workunit.client.0.smithi151.stdout:5/589: write d2/d6/d19/d1a/f41 [3655321,35954] 0 2024-04-16T02:35:50.779 INFO:tasks.workunit.client.0.smithi151.stdout:6/584: dread d2/d5/df/d23/d56/d72/fac [0,4194304] 0 2024-04-16T02:35:50.782 INFO:tasks.workunit.client.0.smithi151.stdout:6/585: write d2/d5/df/d23/d82/d8d/d22/f4e [1318968,25335] 0 2024-04-16T02:35:50.787 INFO:tasks.workunit.client.0.smithi151.stdout:6/586: creat d2/d5/d2e/db6/fca x:0 0 0 2024-04-16T02:35:50.787 INFO:tasks.workunit.client.0.smithi151.stdout:5/590: dwrite d2/d6/d19/d30/f44 [0,4194304] 0 2024-04-16T02:35:50.789 INFO:tasks.workunit.client.0.smithi151.stdout:6/587: write d2/d5/df/d23/d82/d8d/f98 [393550,81251] 0 2024-04-16T02:35:50.791 INFO:tasks.workunit.client.0.smithi151.stdout:5/591: fdatasync d2/d6/f1c 0 2024-04-16T02:35:50.791 INFO:tasks.workunit.client.0.smithi151.stdout:6/588: readlink d2/d5/df/d23/d31/l35 0 2024-04-16T02:35:50.796 INFO:tasks.workunit.client.0.smithi151.stdout:7/498: dwrite d0/d1f/f29 [0,4194304] 0 2024-04-16T02:35:50.798 INFO:tasks.workunit.client.0.smithi151.stdout:6/589: rename d2/d3d/fc7 to d2/d13/d6f/da3/fcb 0 2024-04-16T02:35:50.804 INFO:tasks.workunit.client.1.smithi153.stdout:7/965: write d0/d4/db/d47/fa7 [396053,66521] 0 2024-04-16T02:35:50.805 INFO:tasks.workunit.client.1.smithi153.stdout:8/934: creat d2/d13/d3a/dd6/f139 x:0 0 0 2024-04-16T02:35:50.805 INFO:tasks.workunit.client.0.smithi151.stdout:7/499: symlink d0/lae 0 2024-04-16T02:35:50.807 INFO:tasks.workunit.client.1.smithi153.stdout:6/960: mkdir dc/d47/d57/d77/dc4/dc9/d138 0 2024-04-16T02:35:50.807 INFO:tasks.workunit.client.0.smithi151.stdout:7/500: truncate d0/d2c/d38/d5e/faa 917823 0 2024-04-16T02:35:50.810 INFO:tasks.workunit.client.0.smithi151.stdout:7/501: mknod d0/d2c/d38/d5e/caf 0 2024-04-16T02:35:50.810 INFO:tasks.workunit.client.0.smithi151.stdout:7/502: readlink d0/d1f/d46/l6b 0 2024-04-16T02:35:50.812 INFO:tasks.workunit.client.1.smithi153.stdout:6/961: dwrite dc/d47/d57/d77/d15/dac/dae/ffc [0,4194304] 0 2024-04-16T02:35:50.813 INFO:tasks.workunit.client.1.smithi153.stdout:6/962: fdatasync dc/d47/d57/d101/f95 0 2024-04-16T02:35:50.824 INFO:tasks.workunit.client.0.smithi151.stdout:5/592: dread d2/f16 [0,4194304] 0 2024-04-16T02:35:50.825 INFO:tasks.workunit.client.0.smithi151.stdout:5/593: chown d2/d6/d43/f6d 473 1 2024-04-16T02:35:50.846 INFO:tasks.workunit.client.1.smithi153.stdout:7/966: creat d0/d4/db/d2b/f148 x:0 0 0 2024-04-16T02:35:50.847 INFO:tasks.workunit.client.1.smithi153.stdout:7/967: write d0/d4/db/d2b/d53/f27 [980033,47383] 0 2024-04-16T02:35:50.854 INFO:tasks.workunit.client.1.smithi153.stdout:7/968: creat d0/d4/db/d2b/d53/d59/d76/f149 x:0 0 0 2024-04-16T02:35:50.858 INFO:tasks.workunit.client.1.smithi153.stdout:7/969: creat d0/d4/db/d47/d8d/d93/f14a x:0 0 0 2024-04-16T02:35:50.859 INFO:tasks.workunit.client.1.smithi153.stdout:7/970: chown d0/d4/db/d2b/d53/d59/f127 229138 1 2024-04-16T02:35:50.859 INFO:tasks.workunit.client.1.smithi153.stdout:7/971: chown d0/de8/d110/daa 0 1 2024-04-16T02:35:50.864 INFO:tasks.workunit.client.1.smithi153.stdout:7/972: dwrite d0/d4/db/d2b/d53/d59/d76/d140/db8/f122 [0,4194304] 0 2024-04-16T02:35:50.868 INFO:tasks.workunit.client.1.smithi153.stdout:7/973: truncate d0/f5b 564135 0 2024-04-16T02:35:50.874 INFO:tasks.workunit.client.1.smithi153.stdout:7/974: dwrite d0/de8/ffe [0,4194304] 0 2024-04-16T02:35:50.880 INFO:tasks.workunit.client.1.smithi153.stdout:8/935: truncate d2/d13/d96/dc7/d109/d47/d60/f71 4089529 0 2024-04-16T02:35:50.885 INFO:tasks.workunit.client.1.smithi153.stdout:7/975: dwrite d0/f141 [0,4194304] 0 2024-04-16T02:35:50.886 INFO:tasks.workunit.client.1.smithi153.stdout:8/936: creat d2/d92/d102/f13a x:0 0 0 2024-04-16T02:35:50.887 INFO:tasks.workunit.client.1.smithi153.stdout:6/963: dwrite dc/d47/d57/f10f [0,4194304] 0 2024-04-16T02:35:50.888 INFO:tasks.workunit.client.1.smithi153.stdout:8/937: chown d2/d1c/l33 758 1 2024-04-16T02:35:50.889 INFO:tasks.workunit.client.1.smithi153.stdout:7/976: symlink d0/d4/db/d2b/d53/d59/d76/dc8/l14b 0 2024-04-16T02:35:50.891 INFO:tasks.workunit.client.1.smithi153.stdout:8/938: mknod d2/d13/d96/dc7/d109/c13b 0 2024-04-16T02:35:50.896 INFO:tasks.workunit.client.1.smithi153.stdout:6/964: dwrite dc/d47/d57/d77/d15/de8/f12f [0,4194304] 0 2024-04-16T02:35:50.899 INFO:tasks.workunit.client.1.smithi153.stdout:7/977: dwrite d0/d4/db/d2b/f4e [0,4194304] 0 2024-04-16T02:35:50.900 INFO:tasks.workunit.client.1.smithi153.stdout:7/978: write d0/de8/ffe [1144131,96533] 0 2024-04-16T02:35:50.910 INFO:tasks.workunit.client.1.smithi153.stdout:7/979: mknod d0/de8/d110/d11e/c14c 0 2024-04-16T02:35:50.915 INFO:tasks.workunit.client.1.smithi153.stdout:7/980: creat d0/d4/db/d87/dc0/de0/dee/f14d x:0 0 0 2024-04-16T02:35:50.919 INFO:tasks.workunit.client.1.smithi153.stdout:7/981: creat d0/d4/db/d2b/d43/f14e x:0 0 0 2024-04-16T02:35:50.934 INFO:tasks.workunit.client.1.smithi153.stdout:6/965: dread dc/d47/f87 [0,4194304] 0 2024-04-16T02:35:50.934 INFO:tasks.workunit.client.1.smithi153.stdout:6/966: write dc/f113 [161971,90430] 0 2024-04-16T02:35:50.936 INFO:tasks.workunit.client.1.smithi153.stdout:7/982: dread d0/d13/f81 [0,4194304] 0 2024-04-16T02:35:50.937 INFO:tasks.workunit.client.1.smithi153.stdout:6/967: rename dc/d1a/c5c to dc/d6b/c139 0 2024-04-16T02:35:50.938 INFO:tasks.workunit.client.1.smithi153.stdout:7/983: mkdir d0/d4/d14f 0 2024-04-16T02:35:50.939 INFO:tasks.workunit.client.1.smithi153.stdout:7/984: fsync d0/de8/d110/d4d/ff9 0 2024-04-16T02:35:50.941 INFO:tasks.workunit.client.1.smithi153.stdout:7/985: dread d0/d13/f12b [0,4194304] 0 2024-04-16T02:35:50.943 INFO:tasks.workunit.client.1.smithi153.stdout:7/986: mknod d0/d4/db/d2b/d53/d59/c150 0 2024-04-16T02:35:50.995 INFO:tasks.workunit.client.1.smithi153.stdout:7/987: mkdir d0/de8/d110/daa/d151 0 2024-04-16T02:35:50.996 INFO:tasks.workunit.client.1.smithi153.stdout:7/988: creat d0/d4/d14f/f152 x:0 0 0 2024-04-16T02:35:51.000 INFO:tasks.workunit.client.1.smithi153.stdout:7/989: getdents d0/d4/db/d2b/d53/d59/d76/dc8 0 2024-04-16T02:35:51.001 INFO:tasks.workunit.client.1.smithi153.stdout:7/990: truncate d0/d4/db/d47/f94 4635541 0 2024-04-16T02:35:51.002 INFO:tasks.workunit.client.1.smithi153.stdout:7/991: chown d0/d4/db/c88 1 1 2024-04-16T02:35:51.015 INFO:tasks.workunit.client.1.smithi153.stdout:7/992: dread d0/d4/db/d2b/f2f [0,4194304] 0 2024-04-16T02:35:51.081 INFO:tasks.workunit.client.1.smithi153.stdout:7/993: write d0/d4/db/fd3 [370904,63662] 0 2024-04-16T02:35:51.082 INFO:tasks.workunit.client.1.smithi153.stdout:7/994: chown d0/d4/db/d2b/d53/d59/d76/l12d 110971596 1 2024-04-16T02:35:51.083 INFO:tasks.workunit.client.1.smithi153.stdout:7/995: chown d0/d4/db/d34/f79 59127733 1 2024-04-16T02:35:51.096 INFO:tasks.workunit.client.1.smithi153.stdout:7/996: dread d0/d4/db/d16/d126/fda [0,4194304] 0 2024-04-16T02:35:51.097 INFO:tasks.workunit.client.1.smithi153.stdout:7/997: write d0/d4/db/d2b/d53/d59/d76/d140/db8/fe5 [199124,4086] 0 2024-04-16T02:35:51.100 INFO:tasks.workunit.client.1.smithi153.stdout:7/998: mknod d0/d4/db/d2b/d53/d59/dc3/c153 0 2024-04-16T02:35:51.105 INFO:tasks.workunit.client.1.smithi153.stdout:7/999: truncate d0/d4/db/d2b/d53/d59/d76/fd1 4494947 0 2024-04-16T02:35:51.139 INFO:tasks.workunit.client.0.smithi151.stdout:7/503: rename d0/d2c/d38/d93 to d0/d2c/d38/db0 0 2024-04-16T02:35:51.140 INFO:tasks.workunit.client.0.smithi151.stdout:7/504: chown d0/d2c/f5f 1746 1 2024-04-16T02:35:51.141 INFO:tasks.workunit.client.0.smithi151.stdout:5/594: mkdir d2/d6/db7 0 2024-04-16T02:35:51.144 INFO:tasks.workunit.client.0.smithi151.stdout:7/505: read d0/d1f/d31/d42/f45 [1316298,36749] 0 2024-04-16T02:35:51.144 INFO:tasks.workunit.client.0.smithi151.stdout:5/595: dread - d2/d6/d19/d69/faa zero size 2024-04-16T02:35:51.145 INFO:tasks.workunit.client.0.smithi151.stdout:6/590: dwrite d2/d5/fc [0,4194304] 0 2024-04-16T02:35:51.148 INFO:tasks.workunit.client.0.smithi151.stdout:7/506: unlink d0/d1f/d46/fa9 0 2024-04-16T02:35:51.149 INFO:tasks.workunit.client.0.smithi151.stdout:7/507: read d0/d1f/d33/d56/f84 [862627,78115] 0 2024-04-16T02:35:51.151 INFO:tasks.workunit.client.0.smithi151.stdout:5/596: mknod d2/d6/d19/d30/d36/d42/d6f/d83/cb8 0 2024-04-16T02:35:51.158 INFO:tasks.workunit.client.0.smithi151.stdout:7/508: symlink d0/d1f/d94/lb1 0 2024-04-16T02:35:51.160 INFO:tasks.workunit.client.0.smithi151.stdout:5/597: dwrite d2/d6/d19/d30/fb0 [0,4194304] 0 2024-04-16T02:35:51.160 INFO:tasks.workunit.client.0.smithi151.stdout:6/591: write d2/d5/df/d23/d82/d8d/d22/d40/d3f/f6c [4013924,69883] 0 2024-04-16T02:35:51.164 INFO:tasks.workunit.client.0.smithi151.stdout:6/592: readlink d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/l76 0 2024-04-16T02:35:51.165 INFO:tasks.workunit.client.0.smithi151.stdout:6/593: fsync d2/d5/df/d23/d56/d72/fb5 0 2024-04-16T02:35:51.174 INFO:tasks.workunit.client.1.smithi153.stdout:8/939: sync 2024-04-16T02:35:51.176 INFO:tasks.workunit.client.1.smithi153.stdout:8/940: chown d2/d13/d96/dc7/d109/d47/d77/c87 1186 1 2024-04-16T02:35:51.178 INFO:tasks.workunit.client.0.smithi151.stdout:7/509: mkdir d0/d1f/da8/db2 0 2024-04-16T02:35:51.179 INFO:tasks.workunit.client.0.smithi151.stdout:7/510: chown d0/d2c/d38/l6a 11064 1 2024-04-16T02:35:51.180 INFO:tasks.workunit.client.0.smithi151.stdout:5/598: rmdir d2/d6/d19/d30/d36/d42/d6f/da8/daf 0 2024-04-16T02:35:51.181 INFO:tasks.workunit.client.0.smithi151.stdout:5/599: chown d2/d6/c56 97 1 2024-04-16T02:35:51.181 INFO:tasks.workunit.client.0.smithi151.stdout:5/600: stat d2/d6/d19/d30/d49/l54 0 2024-04-16T02:35:51.192 INFO:tasks.workunit.client.1.smithi153.stdout:8/941: dwrite d2/de/d3f/f75 [0,4194304] 0 2024-04-16T02:35:51.193 INFO:tasks.workunit.client.0.smithi151.stdout:7/511: dread d0/f7 [0,4194304] 0 2024-04-16T02:35:51.194 INFO:tasks.workunit.client.0.smithi151.stdout:7/512: chown d0/d2c/d38/d85/c37 55977864 1 2024-04-16T02:35:51.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.191+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3632341800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd48101090 msgr2=0x7efd481014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:51.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.191+0000 7efd4cbd6700 1 --2- 172.21.15.151:0/3632341800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd48101090 0x7efd481014b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efd38009870 tx=0x7efd38009b80 comp rx=0 tx=0).stop 2024-04-16T02:35:51.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.192+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3632341800 shutdown_connections 2024-04-16T02:35:51.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.192+0000 7efd4cbd6700 1 --2- 172.21.15.151:0/3632341800 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd480ffcf0 0x7efd48100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.192+0000 7efd4cbd6700 1 --2- 172.21.15.151:0/3632341800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd48101090 0x7efd481014b0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efd38009870 tx=0x7efd38009b80 comp rx=0 tx=0).stop 2024-04-16T02:35:51.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.192+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3632341800 >> 172.21.15.151:0/3632341800 conn(0x7efd480fa7c0 msgr2=0x7efd480fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:51.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.192+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3632341800 shutdown_connections 2024-04-16T02:35:51.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.192+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3632341800 wait complete. 2024-04-16T02:35:51.195 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:50 smithi151 ceph-mon[29003]: Reconfiguring daemon prometheus.smithi151 on smithi151 2024-04-16T02:35:51.197 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.193+0000 7efd4cbd6700 1 Processor -- start 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.193+0000 7efd4cbd6700 1 -- start start 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd4cbd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd480ffcf0 0x7efd481a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd4cbd6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd481a5950 0x7efd481aa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd4cbd6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd481a5e60 con 0x7efd481a5950 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd4cbd6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd481a5fd0 con 0x7efd480ffcf0 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd45d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd481a5950 0x7efd481aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd45d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd481a5950 0x7efd481aa970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46766/0 (socket says 172.21.15.151:46766) 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd45d9b700 1 -- 172.21.15.151:0/3961400764 learned_addr learned my addr 172.21.15.151:0/3961400764 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd4659c700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd480ffcf0 0x7efd481a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:51.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd45d9b700 1 -- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd480ffcf0 msgr2=0x7efd481a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.194+0000 7efd45d9b700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd480ffcf0 0x7efd481a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.195+0000 7efd45d9b700 1 -- 172.21.15.151:0/3961400764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd38009520 con 0x7efd481a5950 2024-04-16T02:35:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.195+0000 7efd4659c700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd480ffcf0 0x7efd481a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:35:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.195+0000 7efd45d9b700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd481a5950 0x7efd481aa970 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00e970 tx=0x7efd3c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.195+0000 7efd377fe700 1 -- 172.21.15.151:0/3961400764 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3c0041f0 con 0x7efd481a5950 2024-04-16T02:35:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.195+0000 7efd377fe700 1 -- 172.21.15.151:0/3961400764 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efd3c00cc70 con 0x7efd481a5950 2024-04-16T02:35:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.195+0000 7efd377fe700 1 -- 172.21.15.151:0/3961400764 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3c003cb0 con 0x7efd481a5950 2024-04-16T02:35:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.196+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd481aaf10 con 0x7efd481a5950 2024-04-16T02:35:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.196+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd481ab4b0 con 0x7efd481a5950 2024-04-16T02:35:51.200 INFO:tasks.workunit.client.1.smithi153.stdout:8/942: symlink d2/d13/d96/dc7/d109/d47/d77/ddf/l13c 0 2024-04-16T02:35:51.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.197+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd481067b0 con 0x7efd481a5950 2024-04-16T02:35:51.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.198+0000 7efd377fe700 1 -- 172.21.15.151:0/3961400764 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7efd3c00c7c0 con 0x7efd481a5950 2024-04-16T02:35:51.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.199+0000 7efd377fe700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7efd3006d850 0x7efd3006fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:51.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.199+0000 7efd377fe700 1 -- 172.21.15.151:0/3961400764 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7efd3c014070 con 0x7efd481a5950 2024-04-16T02:35:51.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.199+0000 7efd4659c700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7efd3006d850 0x7efd3006fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:51.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.200+0000 7efd4659c700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7efd3006d850 0x7efd3006fd10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efd38009230 tx=0x7efd38019040 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:51.203 INFO:tasks.workunit.client.1.smithi153.stdout:8/943: dwrite d2/d93/d97/d9c/d27/d5e/fcd [0,4194304] 0 2024-04-16T02:35:51.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.205+0000 7efd377fe700 1 -- 172.21.15.151:0/3961400764 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7efd3c056690 con 0x7efd481a5950 2024-04-16T02:35:51.207 INFO:tasks.workunit.client.1.smithi153.stdout:8/944: symlink d2/d1c/l13d 0 2024-04-16T02:35:51.207 INFO:tasks.workunit.client.1.smithi153.stdout:8/945: stat d2/d93/d97/d9c 0 2024-04-16T02:35:51.220 INFO:tasks.workunit.client.1.smithi153.stdout:8/946: fsync d2/d93/d97/d9c/d27/d5e/fcd 0 2024-04-16T02:35:51.221 INFO:tasks.workunit.client.1.smithi153.stdout:8/947: stat d2/d93/dee/c108 0 2024-04-16T02:35:51.225 INFO:tasks.workunit.client.1.smithi153.stdout:8/948: getdents d2/d93/d97/de5 0 2024-04-16T02:35:51.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:50 smithi153 ceph-mon[36747]: Reconfiguring daemon prometheus.smithi151 on smithi151 2024-04-16T02:35:51.272 INFO:tasks.workunit.client.0.smithi151.stdout:7/513: rename d0/d2c/d38/d3e to d0/d1f/d4c/d82/db3 0 2024-04-16T02:35:51.329 INFO:tasks.workunit.client.1.smithi153.stdout:6/968: sync 2024-04-16T02:35:51.335 INFO:tasks.workunit.client.1.smithi153.stdout:6/969: dwrite dc/d47/d57/d77/d15/dac/dae/ffc [0,4194304] 0 2024-04-16T02:35:51.345 INFO:tasks.workunit.client.1.smithi153.stdout:8/949: sync 2024-04-16T02:35:51.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.476+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd48105650 con 0x7efd3006d850 2024-04-16T02:35:51.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.478+0000 7efd377fe700 1 -- 172.21.15.151:0/3961400764 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7efd48105650 con 0x7efd3006d850 2024-04-16T02:35:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.481+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7efd3006d850 msgr2=0x7efd3006fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.481+0000 7efd4cbd6700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7efd3006d850 0x7efd3006fd10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efd38009230 tx=0x7efd38019040 comp rx=0 tx=0).stop 2024-04-16T02:35:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.481+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd481a5950 msgr2=0x7efd481aa970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.481+0000 7efd4cbd6700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd481a5950 0x7efd481aa970 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00e970 tx=0x7efd3c00ec80 comp rx=0 tx=0).stop 2024-04-16T02:35:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.481+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 shutdown_connections 2024-04-16T02:35:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.481+0000 7efd4cbd6700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7efd3006d850 0x7efd3006fd10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.481+0000 7efd4cbd6700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd480ffcf0 0x7efd481a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.481+0000 7efd4cbd6700 1 --2- 172.21.15.151:0/3961400764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd481a5950 0x7efd481aa970 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.481+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 >> 172.21.15.151:0/3961400764 conn(0x7efd480fa7c0 msgr2=0x7efd48103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:51.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.482+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 shutdown_connections 2024-04-16T02:35:51.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.482+0000 7efd4cbd6700 1 -- 172.21.15.151:0/3961400764 wait complete. 2024-04-16T02:35:51.496 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:35:51.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.598+0000 7f7d411f8700 1 -- 172.21.15.151:0/3071021508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d3c0ffd30 msgr2=0x7f7d3c1001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.598+0000 7f7d411f8700 1 --2- 172.21.15.151:0/3071021508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d3c0ffd30 0x7f7d3c1001b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009910 tx=0x7f7d30009c20 comp rx=0 tx=0).stop 2024-04-16T02:35:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.598+0000 7f7d411f8700 1 -- 172.21.15.151:0/3071021508 shutdown_connections 2024-04-16T02:35:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.598+0000 7f7d411f8700 1 --2- 172.21.15.151:0/3071021508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d3c0ffd30 0x7f7d3c1001b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.598+0000 7f7d411f8700 1 --2- 172.21.15.151:0/3071021508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d3c1010d0 0x7f7d3c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.598+0000 7f7d411f8700 1 -- 172.21.15.151:0/3071021508 >> 172.21.15.151:0/3071021508 conn(0x7f7d3c0fa7c0 msgr2=0x7f7d3c0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.598+0000 7f7d411f8700 1 -- 172.21.15.151:0/3071021508 shutdown_connections 2024-04-16T02:35:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.598+0000 7f7d411f8700 1 -- 172.21.15.151:0/3071021508 wait complete. 2024-04-16T02:35:51.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.599+0000 7f7d411f8700 1 Processor -- start 2024-04-16T02:35:51.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d411f8700 1 -- start start 2024-04-16T02:35:51.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d411f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d3c0ffd30 0x7f7d3c0ff780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:51.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d411f8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d3c1010d0 0x7f7d3c19b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:51.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d411f8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c19b6c0 con 0x7f7d3c1010d0 2024-04-16T02:35:51.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d411f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c19b800 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d3a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d3c1010d0 0x7f7d3c19b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d3a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d3c1010d0 0x7f7d3c19b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46796/0 (socket says 172.21.15.151:46796) 2024-04-16T02:35:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d3a59c700 1 -- 172.21.15.151:0/3717250297 learned_addr learned my addr 172.21.15.151:0/3717250297 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d3ad9d700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d3c0ffd30 0x7f7d3c0ff780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d3ad9d700 1 -- 172.21.15.151:0/3717250297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d3c1010d0 msgr2=0x7f7d3c19b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d3ad9d700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d3c1010d0 0x7f7d3c19b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.600+0000 7f7d3ad9d700 1 -- 172.21.15.151:0/3717250297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300095f0 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.601+0000 7f7d3ad9d700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d3c0ffd30 0x7f7d3c0ff780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00e970 tx=0x7f7d2c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.601+0000 7f7d23fff700 1 -- 172.21.15.151:0/3717250297 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2c00c9d0 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.601+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c19bae0 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.601+0000 7f7d23fff700 1 -- 172.21.15.151:0/3717250297 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d2c004030 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.601+0000 7f7d23fff700 1 -- 172.21.15.151:0/3717250297 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2c003e40 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.601+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c19c030 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.602+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c108660 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.603+0000 7f7d23fff700 1 -- 172.21.15.151:0/3717250297 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7f7d2c00cb30 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.604+0000 7f7d23fff700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f7d2406d7e0 0x7f7d2406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:51.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.604+0000 7f7d23fff700 1 -- 172.21.15.151:0/3717250297 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f7d2c014070 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.604+0000 7f7d3a59c700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f7d2406d7e0 0x7f7d2406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:51.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.604+0000 7f7d3a59c700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f7d2406d7e0 0x7f7d2406fca0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0ffd00 tx=0x7f7d30005940 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:51.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.608+0000 7f7d23fff700 1 -- 172.21.15.151:0/3717250297 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f7d2c05a3b0 con 0x7f7d3c0ffd30 2024-04-16T02:35:51.914 INFO:tasks.workunit.client.0.smithi151.stdout:8/308: sync 2024-04-16T02:35:51.914 INFO:tasks.workunit.client.0.smithi151.stdout:0/388: sync 2024-04-16T02:35:51.915 INFO:tasks.workunit.client.0.smithi151.stdout:8/309: stat d11/d2a/d33/f4b 0 2024-04-16T02:35:51.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.964+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d3c105690 con 0x7f7d2406d7e0 2024-04-16T02:35:51.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.966+0000 7f7d23fff700 1 -- 172.21.15.151:0/3717250297 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f7d3c105690 con 0x7f7d2406d7e0 2024-04-16T02:35:51.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f7d2406d7e0 msgr2=0x7f7d2406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:51.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f7d2406d7e0 0x7f7d2406fca0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0ffd00 tx=0x7f7d30005940 comp rx=0 tx=0).stop 2024-04-16T02:35:51.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d3c0ffd30 msgr2=0x7f7d3c0ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:51.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d3c0ffd30 0x7f7d3c0ff780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00e970 tx=0x7f7d2c00ec80 comp rx=0 tx=0).stop 2024-04-16T02:35:51.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 shutdown_connections 2024-04-16T02:35:51.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f7d2406d7e0 0x7f7d2406fca0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d3c0ffd30 0x7f7d3c0ff780 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 --2- 172.21.15.151:0/3717250297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d3c1010d0 0x7f7d3c19b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:51.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 >> 172.21.15.151:0/3717250297 conn(0x7f7d3c0fa7c0 msgr2=0x7f7d3c103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:51.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.969+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 shutdown_connections 2024-04-16T02:35:51.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:51.970+0000 7f7d411f8700 1 -- 172.21.15.151:0/3717250297 wait complete. 2024-04-16T02:35:52.004 INFO:tasks.workunit.client.0.smithi151.stdout:6/594: sync 2024-04-16T02:35:52.005 INFO:tasks.workunit.client.0.smithi151.stdout:3/320: sync 2024-04-16T02:35:52.005 INFO:tasks.workunit.client.0.smithi151.stdout:7/514: sync 2024-04-16T02:35:52.005 INFO:tasks.workunit.client.0.smithi151.stdout:2/519: sync 2024-04-16T02:35:52.005 INFO:tasks.workunit.client.0.smithi151.stdout:9/445: sync 2024-04-16T02:35:52.005 INFO:tasks.workunit.client.0.smithi151.stdout:1/463: sync 2024-04-16T02:35:52.005 INFO:tasks.workunit.client.0.smithi151.stdout:4/490: sync 2024-04-16T02:35:52.008 INFO:tasks.workunit.client.0.smithi151.stdout:3/321: write d1/d6/f24 [795288,30474] 0 2024-04-16T02:35:52.011 INFO:tasks.workunit.client.0.smithi151.stdout:1/464: truncate d3/d4/f96 4276374 0 2024-04-16T02:35:52.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.094+0000 7f02bba13700 1 -- 172.21.15.151:0/4133616874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 msgr2=0x7f02b4103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:52.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.094+0000 7f02bba13700 1 --2- 172.21.15.151:0/4133616874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 0x7f02b4103a30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009910 tx=0x7f02b0009c20 comp rx=0 tx=0).stop 2024-04-16T02:35:52.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.094+0000 7f02bba13700 1 -- 172.21.15.151:0/4133616874 shutdown_connections 2024-04-16T02:35:52.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.094+0000 7f02bba13700 1 --2- 172.21.15.151:0/4133616874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4103f70 0x7f02b4106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.094+0000 7f02bba13700 1 --2- 172.21.15.151:0/4133616874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 0x7f02b4103a30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.094+0000 7f02bba13700 1 -- 172.21.15.151:0/4133616874 >> 172.21.15.151:0/4133616874 conn(0x7f02b40fb030 msgr2=0x7f02b40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:52.097 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:51 smithi151 ceph-mon[29003]: pgmap v23: 65 pgs: 65 active+clean; 2.7 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 84 MiB/s rd, 254 MiB/s wr, 478 op/s 2024-04-16T02:35:52.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.094+0000 7f02bba13700 1 -- 172.21.15.151:0/4133616874 shutdown_connections 2024-04-16T02:35:52.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.094+0000 7f02bba13700 1 -- 172.21.15.151:0/4133616874 wait complete. 2024-04-16T02:35:52.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.095+0000 7f02bba13700 1 Processor -- start 2024-04-16T02:35:52.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.095+0000 7f02bba13700 1 -- start start 2024-04-16T02:35:52.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.095+0000 7f02bba13700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 0x7f02b4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.095+0000 7f02bba13700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4103f70 0x7f02b4198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.096+0000 7f02bba13700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b4199080 con 0x7f02b4101640 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.096+0000 7f02bba13700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b41991c0 con 0x7f02b4103f70 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.096+0000 7f02b97af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 0x7f02b4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.096+0000 7f02b97af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 0x7f02b4198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46810/0 (socket says 172.21.15.151:46810) 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.096+0000 7f02b97af700 1 -- 172.21.15.151:0/2899613678 learned_addr learned my addr 172.21.15.151:0/2899613678 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.096+0000 7f02b97af700 1 -- 172.21.15.151:0/2899613678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4103f70 msgr2=0x7f02b4198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.096+0000 7f02b97af700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4103f70 0x7f02b4198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.096+0000 7f02b97af700 1 -- 172.21.15.151:0/2899613678 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b00095f0 con 0x7f02b4101640 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.097+0000 7f02b97af700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 0x7f02b4198520 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f02b000bd70 tx=0x7f02b000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.097+0000 7f02aa7fc700 1 -- 172.21.15.151:0/2899613678 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02b001d070 con 0x7f02b4101640 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.097+0000 7f02aa7fc700 1 -- 172.21.15.151:0/2899613678 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f02b0022470 con 0x7f02b4101640 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.097+0000 7f02bba13700 1 -- 172.21.15.151:0/2899613678 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b419dc10 con 0x7f02b4101640 2024-04-16T02:35:52.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.097+0000 7f02aa7fc700 1 -- 172.21.15.151:0/2899613678 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02b000f670 con 0x7f02b4101640 2024-04-16T02:35:52.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.097+0000 7f02bba13700 1 -- 172.21.15.151:0/2899613678 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b419e040 con 0x7f02b4101640 2024-04-16T02:35:52.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.099+0000 7f029ffff700 1 -- 172.21.15.151:0/2899613678 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02940052f0 con 0x7f02b4101640 2024-04-16T02:35:52.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.100+0000 7f02aa7fc700 1 -- 172.21.15.151:0/2899613678 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7f02b0022a50 con 0x7f02b4101640 2024-04-16T02:35:52.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.101+0000 7f02aa7fc700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f02a006d850 0x7f02a006fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:52.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.101+0000 7f02b8fae700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f02a006d850 0x7f02a006fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:52.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.101+0000 7f02aa7fc700 1 -- 172.21.15.151:0/2899613678 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f02b008e830 con 0x7f02b4101640 2024-04-16T02:35:52.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.102+0000 7f02b8fae700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f02a006d850 0x7f02a006fd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f02a4005950 tx=0x7f02a40058e0 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:52.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.106+0000 7f02aa7fc700 1 -- 172.21.15.151:0/2899613678 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f02b005b650 con 0x7f02b4101640 2024-04-16T02:35:52.139 INFO:tasks.workunit.client.1.smithi153.stdout:6/970: stat dc/c92 0 2024-04-16T02:35:52.139 INFO:tasks.workunit.client.1.smithi153.stdout:6/971: dread - dc/d59/d5b/f12b zero size 2024-04-16T02:35:52.145 INFO:tasks.workunit.client.1.smithi153.stdout:8/950: write d2/d13/d96/dc7/d109/d47/d60/fcf [1224046,45411] 0 2024-04-16T02:35:52.146 INFO:tasks.workunit.client.1.smithi153.stdout:8/951: dread - d2/d92/d102/f13a zero size 2024-04-16T02:35:52.152 INFO:tasks.workunit.client.1.smithi153.stdout:8/952: dread d2/d13/d96/dc7/d109/d47/fd1 [0,4194304] 0 2024-04-16T02:35:52.153 INFO:tasks.workunit.client.1.smithi153.stdout:8/953: fdatasync d2/d13/d96/dc7/d109/d47/d77/f127 0 2024-04-16T02:35:52.205 INFO:tasks.workunit.client.1.smithi153.stdout:6/972: rename dc/d10/c98 to dc/d6b/dba/c13a 0 2024-04-16T02:35:52.207 INFO:tasks.workunit.client.1.smithi153.stdout:6/973: write dc/d47/d57/d77/f85 [6619025,107510] 0 2024-04-16T02:35:52.207 INFO:tasks.workunit.client.1.smithi153.stdout:8/954: creat d2/d93/d97/f13e x:0 0 0 2024-04-16T02:35:52.208 INFO:tasks.workunit.client.1.smithi153.stdout:8/955: write d2/d1c/f126 [108974,84111] 0 2024-04-16T02:35:52.211 INFO:tasks.workunit.client.1.smithi153.stdout:6/974: fsync dc/d47/d57/d77/d15/dac/fb9 0 2024-04-16T02:35:52.214 INFO:tasks.workunit.client.1.smithi153.stdout:8/956: mkdir d2/de/d3f/d13f 0 2024-04-16T02:35:52.215 INFO:tasks.workunit.client.1.smithi153.stdout:8/957: chown d2/de/l22 10598214 1 2024-04-16T02:35:52.217 INFO:tasks.workunit.client.1.smithi153.stdout:6/975: dwrite dc/d59/d5b/f125 [0,4194304] 0 2024-04-16T02:35:52.221 INFO:tasks.workunit.client.1.smithi153.stdout:8/958: fsync d2/dba/fd9 0 2024-04-16T02:35:52.223 INFO:tasks.workunit.client.1.smithi153.stdout:8/959: truncate d2/d13/d96/dc7/d109/d47/d60/f128 940014 0 2024-04-16T02:35:52.224 INFO:tasks.workunit.client.1.smithi153.stdout:8/960: dread - d2/d13/d96/dc7/d109/d47/d77/ddf/f111 zero size 2024-04-16T02:35:52.231 INFO:tasks.workunit.client.1.smithi153.stdout:8/961: read d2/d13/d2c/fea [866001,3639] 0 2024-04-16T02:35:52.236 INFO:tasks.workunit.client.1.smithi153.stdout:8/962: fdatasync d2/d93/d97/d9c/d27/fb9 0 2024-04-16T02:35:52.243 INFO:tasks.workunit.client.1.smithi153.stdout:8/963: mknod d2/de/c140 0 2024-04-16T02:35:52.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:51 smithi153 ceph-mon[36747]: pgmap v23: 65 pgs: 65 active+clean; 2.7 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 84 MiB/s rd, 254 MiB/s wr, 478 op/s 2024-04-16T02:35:52.286 INFO:tasks.workunit.client.1.smithi153.stdout:8/964: write d2/f12 [1962940,87592] 0 2024-04-16T02:35:52.289 INFO:tasks.workunit.client.1.smithi153.stdout:6/976: dwrite dc/d47/d57/d77/dc4/dc9/fca [0,4194304] 0 2024-04-16T02:35:52.291 INFO:tasks.workunit.client.1.smithi153.stdout:8/965: mknod d2/d13/d96/dc7/d109/d47/d60/d67/d98/c141 0 2024-04-16T02:35:52.294 INFO:tasks.workunit.client.1.smithi153.stdout:8/966: truncate d2/d13/f1d 5054845 0 2024-04-16T02:35:52.299 INFO:tasks.workunit.client.1.smithi153.stdout:8/967: symlink d2/d13/d3a/dc2/d10d/l142 0 2024-04-16T02:35:52.304 INFO:tasks.workunit.client.1.smithi153.stdout:8/968: creat d2/d13/d96/dc7/d109/d47/d60/d67/db7/f143 x:0 0 0 2024-04-16T02:35:52.308 INFO:tasks.workunit.client.1.smithi153.stdout:8/969: mkdir d2/d13/d96/dc7/d109/d47/d77/ddf/dfc/d144 0 2024-04-16T02:35:52.312 INFO:tasks.workunit.client.1.smithi153.stdout:8/970: read d2/d1c/f26 [1798641,72173] 0 2024-04-16T02:35:52.315 INFO:tasks.workunit.client.1.smithi153.stdout:8/971: mkdir d2/d13/d96/dc7/d109/d47/d77/ddf/dfc/d144/d145 0 2024-04-16T02:35:52.315 INFO:tasks.workunit.client.1.smithi153.stdout:8/972: chown d2/d13/laf 452686 1 2024-04-16T02:35:52.317 INFO:tasks.workunit.client.1.smithi153.stdout:6/977: dread dc/d47/d57/d101/f66 [0,4194304] 0 2024-04-16T02:35:52.317 INFO:tasks.workunit.client.1.smithi153.stdout:8/973: write d2/d93/d97/d9c/d27/d5e/d63/dc6/f10b [3195061,98362] 0 2024-04-16T02:35:52.318 INFO:tasks.workunit.client.1.smithi153.stdout:6/978: chown dc/d47/d57/d77/d15/f128 798 1 2024-04-16T02:35:52.324 INFO:tasks.workunit.client.1.smithi153.stdout:6/979: dwrite dc/d47/d57/d101/f95 [4194304,4194304] 0 2024-04-16T02:35:52.325 INFO:tasks.workunit.client.1.smithi153.stdout:8/974: unlink d2/d1c/l13d 0 2024-04-16T02:35:52.329 INFO:tasks.workunit.client.1.smithi153.stdout:8/975: fdatasync d2/d13/d96/fab 0 2024-04-16T02:35:52.330 INFO:tasks.workunit.client.1.smithi153.stdout:8/976: fsync d2/de/f5f 0 2024-04-16T02:35:52.337 INFO:tasks.workunit.client.1.smithi153.stdout:6/980: creat dc/d47/d57/d77/dc4/dc9/d138/f13b x:0 0 0 2024-04-16T02:35:52.341 INFO:tasks.workunit.client.1.smithi153.stdout:8/977: rename d2/d13/d96/dc7/d109/d47/d77/f127 to d2/d93/d97/d9c/d27/d5e/d63/f146 0 2024-04-16T02:35:52.344 INFO:tasks.workunit.client.1.smithi153.stdout:6/981: dwrite dc/d47/d57/d77/f85 [0,4194304] 0 2024-04-16T02:35:52.353 INFO:tasks.workunit.client.1.smithi153.stdout:8/978: dread d2/d93/d97/d9c/f2e [4194304,4194304] 0 2024-04-16T02:35:52.356 INFO:tasks.workunit.client.1.smithi153.stdout:8/979: symlink d2/d13/d96/dc7/d109/l147 0 2024-04-16T02:35:52.357 INFO:tasks.workunit.client.1.smithi153.stdout:8/980: write d2/d93/d97/f13e [700242,124016] 0 2024-04-16T02:35:52.359 INFO:tasks.workunit.client.1.smithi153.stdout:6/982: write dc/f11 [3449901,19982] 0 2024-04-16T02:35:52.362 INFO:tasks.workunit.client.1.smithi153.stdout:6/983: symlink dc/d59/d5b/l13c 0 2024-04-16T02:35:52.366 INFO:tasks.workunit.client.1.smithi153.stdout:8/981: dread d2/d1c/f85 [0,4194304] 0 2024-04-16T02:35:52.369 INFO:tasks.workunit.client.1.smithi153.stdout:6/984: link dc/d47/d57/d77/d7a/d97/fcc dc/d59/d104/f13d 0 2024-04-16T02:35:52.370 INFO:tasks.workunit.client.1.smithi153.stdout:8/982: creat d2/d13/d96/dc7/d109/d47/d77/ddf/dfc/d144/d145/f148 x:0 0 0 2024-04-16T02:35:52.376 INFO:tasks.workunit.client.1.smithi153.stdout:8/983: getdents d2/d13/d2c 0 2024-04-16T02:35:52.379 INFO:tasks.workunit.client.1.smithi153.stdout:6/985: write dc/d47/d57/d77/d7a/d97/fcc [36582,25642] 0 2024-04-16T02:35:52.381 INFO:tasks.workunit.client.1.smithi153.stdout:8/984: creat d2/d13/d96/dc7/d109/d47/d60/d67/db7/df1/f149 x:0 0 0 2024-04-16T02:35:52.385 INFO:tasks.workunit.client.1.smithi153.stdout:6/986: dwrite dc/d47/f87 [0,4194304] 0 2024-04-16T02:35:52.387 INFO:tasks.workunit.client.1.smithi153.stdout:8/985: chown d2/d13/d96/dc7/d109/d47/d60/d67/f80 2042 1 2024-04-16T02:35:52.388 INFO:tasks.workunit.client.1.smithi153.stdout:6/987: symlink dc/d6b/dba/l13e 0 2024-04-16T02:35:52.392 INFO:tasks.workunit.client.1.smithi153.stdout:6/988: readlink dc/d47/d57/ldb 0 2024-04-16T02:35:52.396 INFO:tasks.workunit.client.1.smithi153.stdout:6/989: symlink dc/d47/d57/d77/dc4/dc9/l13f 0 2024-04-16T02:35:52.397 INFO:tasks.workunit.client.1.smithi153.stdout:8/986: dwrite d2/d93/d97/d9c/d27/d5e/d63/f8f [0,4194304] 0 2024-04-16T02:35:52.418 INFO:tasks.workunit.client.1.smithi153.stdout:8/987: dread d2/d1c/f36 [0,4194304] 0 2024-04-16T02:35:52.418 INFO:tasks.workunit.client.1.smithi153.stdout:6/990: dread dc/d47/d57/d77/dc4/dc9/ff8 [0,4194304] 0 2024-04-16T02:35:52.423 INFO:tasks.workunit.client.1.smithi153.stdout:8/988: rename d2/d93/d97/d9c/d27/d5e/d63 to d2/d13/d96/dc7/d109/d4b/d14a 0 2024-04-16T02:35:52.424 INFO:tasks.workunit.client.1.smithi153.stdout:6/991: truncate dc/d47/d93/fb4 1842316 0 2024-04-16T02:35:52.425 INFO:tasks.workunit.client.1.smithi153.stdout:8/989: symlink d2/d13/d96/dc7/d109/d47/l14b 0 2024-04-16T02:35:52.434 INFO:tasks.workunit.client.1.smithi153.stdout:8/990: write d2/d13/d96/dc7/d109/d47/d77/ff6 [819738,29470] 0 2024-04-16T02:35:52.438 INFO:tasks.workunit.client.1.smithi153.stdout:6/992: dwrite dc/d47/d57/d77/d15/f2f [4194304,4194304] 0 2024-04-16T02:35:52.438 INFO:tasks.workunit.client.1.smithi153.stdout:6/993: fsync dc/d1a/f11c 0 2024-04-16T02:35:52.440 INFO:tasks.workunit.client.1.smithi153.stdout:8/991: getdents d2/d13/d96/dc7/d109/d47/d77/ddf/dfc 0 2024-04-16T02:35:52.445 INFO:tasks.workunit.client.1.smithi153.stdout:8/992: mkdir d2/d13/d96/dc7/d109/d47/d60/d67/db7/df1/d14c 0 2024-04-16T02:35:52.448 INFO:tasks.workunit.client.1.smithi153.stdout:6/994: dwrite dc/d47/d57/d77/dc4/dc9/d138/f13b [0,4194304] 0 2024-04-16T02:35:52.448 INFO:tasks.workunit.client.1.smithi153.stdout:8/993: write d2/d13/d96/dc7/d109/d47/d60/fcf [569403,49418] 0 2024-04-16T02:35:52.450 INFO:tasks.workunit.client.1.smithi153.stdout:6/995: write dc/d47/d57/d77/d15/d109/f114 [25869,46414] 0 2024-04-16T02:35:52.451 INFO:tasks.workunit.client.1.smithi153.stdout:6/996: stat dc/d47/d57/d101/lf6 0 2024-04-16T02:35:52.452 INFO:tasks.workunit.client.1.smithi153.stdout:8/994: symlink d2/d13/d96/dc7/d109/d47/d60/l14d 0 2024-04-16T02:35:52.455 INFO:tasks.workunit.client.1.smithi153.stdout:8/995: symlink d2/d13/d96/l14e 0 2024-04-16T02:35:52.456 INFO:tasks.workunit.client.1.smithi153.stdout:8/996: write d2/d13/d96/dc7/d109/d47/d60/d67/db7/f11b [467112,12688] 0 2024-04-16T02:35:52.457 INFO:tasks.workunit.client.1.smithi153.stdout:8/997: dread - d2/d13/d2c/f101 zero size 2024-04-16T02:35:52.459 INFO:tasks.workunit.client.1.smithi153.stdout:6/997: creat dc/d47/d57/f140 x:0 0 0 2024-04-16T02:35:52.462 INFO:tasks.workunit.client.1.smithi153.stdout:6/998: creat dc/d1a/d2d/d129/f141 x:0 0 0 2024-04-16T02:35:52.463 INFO:tasks.workunit.client.1.smithi153.stdout:8/998: dread d2/d13/d96/dc7/d109/f129 [0,4194304] 0 2024-04-16T02:35:52.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.464+0000 7f029ffff700 1 -- 172.21.15.151:0/2899613678 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0294000bc0 con 0x7f02a006d850 2024-04-16T02:35:52.468 INFO:tasks.workunit.client.1.smithi153.stdout:6/999: creat dc/d47/d57/d77/d15/dac/f142 x:0 0 0 2024-04-16T02:35:52.473 INFO:tasks.workunit.client.1.smithi153.stdout:8/999: write d2/d13/d96/dc7/d109/d47/d77/fe2 [933240,84462] 0 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.476+0000 7f02aa7fc700 1 -- 172.21.15.151:0/2899613678 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0294000bc0 con 0x7f02a006d850 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (7m) 7s ago 8m 24.6M - 0.25.0 c8568f914cd2 9f7a3dfb7aa4 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (9m) 7s ago 9m 7750k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (7m) 32s ago 7m 7650k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (8m) 7s ago 8m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (7m) 32s ago 7m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (7m) 7s ago 8m 82.7M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (3m) 7s ago 3m 99.9M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:35:52.478 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (3m) 7s ago 3m 114M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (3m) 32s ago 3m 58.3M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (3m) 32s ago 3m 66.0M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:9283,8765,8443 running (10m) 7s ago 10m 340M - 18.2.1 5be31c24972a 9e98c5bc45da 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (65s) 32s ago 7m 489M - 19.0.0-3084-g87dbfb63 a452a57266c7 cd0d253df44f 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (10m) 7s ago 10m 51.0M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (7m) 32s ago 7m 40.0M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (8m) 7s ago 8m 15.6M - 1.5.0 0da6a335fe13 7748f5130613 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 32s ago 7m 16.3M - 1.5.0 0da6a335fe13 6a289489fe28 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (6m) 7s ago 6m 112M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (6m) 7s ago 6m 123M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (5m) 7s ago 5m 103M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (5m) 32s ago 5m 125M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (5m) 32s ago 5m 121M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (4m) 32s ago 4m 108M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:35:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (7m) 7s ago 8m 61.1M - 2.43.0 a07b618ecd1d 9a1c8cd0261d 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.478+0000 7f029ffff700 1 -- 172.21.15.151:0/2899613678 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f02a006d850 msgr2=0x7f02a006fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.478+0000 7f029ffff700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f02a006d850 0x7f02a006fd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f02a4005950 tx=0x7f02a40058e0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.478+0000 7f029ffff700 1 -- 172.21.15.151:0/2899613678 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 msgr2=0x7f02b4198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.478+0000 7f029ffff700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 0x7f02b4198520 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f02b000bd70 tx=0x7f02b000be50 comp rx=0 tx=0).stop 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.478+0000 7f029ffff700 1 -- 172.21.15.151:0/2899613678 shutdown_connections 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.478+0000 7f029ffff700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f02a006d850 0x7f02a006fd10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.478+0000 7f029ffff700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b4101640 0x7f02b4198520 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.478+0000 7f029ffff700 1 --2- 172.21.15.151:0/2899613678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4103f70 0x7f02b4198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.478+0000 7f029ffff700 1 -- 172.21.15.151:0/2899613678 >> 172.21.15.151:0/2899613678 conn(0x7f02b40fb030 msgr2=0x7f02b4104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.479+0000 7f029ffff700 1 -- 172.21.15.151:0/2899613678 shutdown_connections 2024-04-16T02:35:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.479+0000 7f029ffff700 1 -- 172.21.15.151:0/2899613678 wait complete. 2024-04-16T02:35:52.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.614+0000 7fa2ae970700 1 -- 172.21.15.151:0/602414752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 msgr2=0x7fa2a8100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:52.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.614+0000 7fa2ae970700 1 --2- 172.21.15.151:0/602414752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 0x7fa2a8100ff0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa298009910 tx=0x7fa298009c20 comp rx=0 tx=0).stop 2024-04-16T02:35:52.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.614+0000 7fa2ae970700 1 -- 172.21.15.151:0/602414752 shutdown_connections 2024-04-16T02:35:52.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.614+0000 7fa2ae970700 1 --2- 172.21.15.151:0/602414752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2a8101dd0 0x7fa2a8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.614+0000 7fa2ae970700 1 --2- 172.21.15.151:0/602414752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 0x7fa2a8100ff0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.614+0000 7fa2ae970700 1 -- 172.21.15.151:0/602414752 >> 172.21.15.151:0/602414752 conn(0x7fa2a80fc190 msgr2=0x7fa2a80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:52.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.614+0000 7fa2ae970700 1 -- 172.21.15.151:0/602414752 shutdown_connections 2024-04-16T02:35:52.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.614+0000 7fa2ae970700 1 -- 172.21.15.151:0/602414752 wait complete. 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.615+0000 7fa2ae970700 1 Processor -- start 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.615+0000 7fa2ae970700 1 -- start start 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.615+0000 7fa2ae970700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 0x7fa2a8196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.615+0000 7fa2ae970700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2a8101dd0 0x7fa2a81968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.615+0000 7fa2ae970700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8196ec0 con 0x7fa2a8100bd0 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.615+0000 7fa2ae970700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8197000 con 0x7fa2a8101dd0 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 0x7fa2a8196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 0x7fa2a8196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46826/0 (socket says 172.21.15.151:46826) 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a7fff700 1 -- 172.21.15.151:0/2580258582 learned_addr learned my addr 172.21.15.151:0/2580258582 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a77fe700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2a8101dd0 0x7fa2a81968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a7fff700 1 -- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2a8101dd0 msgr2=0x7fa2a81968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a7fff700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2a8101dd0 0x7fa2a81968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:52.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a7fff700 1 -- 172.21.15.151:0/2580258582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2980095f0 con 0x7fa2a8100bd0 2024-04-16T02:35:52.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a77fe700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2a8101dd0 0x7fa2a81968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:35:52.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a7fff700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 0x7fa2a8196360 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa29800bd70 tx=0x7fa29800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:52.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2a57fa700 1 -- 172.21.15.151:0/2580258582 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa29801d070 con 0x7fa2a8100bd0 2024-04-16T02:35:52.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.616+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a819ba50 con 0x7fa2a8100bd0 2024-04-16T02:35:52.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.617+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a819bfc0 con 0x7fa2a8100bd0 2024-04-16T02:35:52.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.617+0000 7fa2a57fa700 1 -- 172.21.15.151:0/2580258582 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa29800f460 con 0x7fa2a8100bd0 2024-04-16T02:35:52.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.617+0000 7fa2a57fa700 1 -- 172.21.15.151:0/2580258582 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa298021640 con 0x7fa2a8100bd0 2024-04-16T02:35:52.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.618+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa294005320 con 0x7fa2a8100bd0 2024-04-16T02:35:52.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.619+0000 7fa2a57fa700 1 -- 172.21.15.151:0/2580258582 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7fa298003a60 con 0x7fa2a8100bd0 2024-04-16T02:35:52.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.619+0000 7fa2a57fa700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fa29006d8a0 0x7fa29006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:52.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.620+0000 7fa2a57fa700 1 -- 172.21.15.151:0/2580258582 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fa29808ebc0 con 0x7fa2a8100bd0 2024-04-16T02:35:52.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.620+0000 7fa2a77fe700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fa29006d8a0 0x7fa29006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:52.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.621+0000 7fa2a77fe700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fa29006d8a0 0x7fa29006fd60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa29c005950 tx=0x7fa29c0058e0 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:52.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:52.627+0000 7fa2a57fa700 1 -- 172.21.15.151:0/2580258582 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fa29805ba30 con 0x7fa2a8100bd0 2024-04-16T02:35:52.646 INFO:tasks.workunit.client.0.smithi151.stdout:5/601: write d2/d5e/f7e [3286844,51136] 0 2024-04-16T02:35:52.861 INFO:tasks.workunit.client.1.smithi153.stdout:1/993: rmdir d4/dd/d13 39 2024-04-16T02:35:52.863 INFO:tasks.workunit.client.1.smithi153.stdout:1/994: read - d4/dd/d18/f4f zero size 2024-04-16T02:35:52.864 INFO:tasks.workunit.client.0.smithi151.stdout:8/310: unlink cc 0 2024-04-16T02:35:52.868 INFO:tasks.workunit.client.1.smithi153.stdout:1/995: dread - d4/dd/d13/d8d/fe8 zero size 2024-04-16T02:35:52.870 INFO:tasks.workunit.client.0.smithi151.stdout:3/322: write d1/d6/d20/d54/f65 [812855,23113] 0 2024-04-16T02:35:52.871 INFO:tasks.workunit.client.0.smithi151.stdout:6/595: rename d2/d5/df/d23/d56/d72/l78 to d2/d5/df/d23/d82/d8d/d22/d40/d3f/db1/lcc 0 2024-04-16T02:35:52.872 INFO:tasks.workunit.client.0.smithi151.stdout:1/465: mkdir d3/d4/d64/da7 0 2024-04-16T02:35:52.873 INFO:tasks.workunit.client.1.smithi153.stdout:1/996: truncate d4/dd/d13/d8d/fe8 688154 0 2024-04-16T02:35:52.877 INFO:tasks.workunit.client.1.smithi153.stdout:1/997: rmdir d4/dd/d13/d7e/de6/d111/d123 39 2024-04-16T02:35:52.879 INFO:tasks.workunit.client.1.smithi153.stdout:1/998: mkdir d4/dd/d13/d7e/d147 0 2024-04-16T02:35:52.888 INFO:tasks.workunit.client.1.smithi153.stdout:1/999: unlink d4/da2/d65/d82/fb0 0 2024-04-16T02:35:52.895 INFO:tasks.workunit.client.1.smithi153.stderr:+ rm -rf -- ./tmp.wcURSL0jp3 2024-04-16T02:35:52.896 INFO:tasks.workunit.client.0.smithi151.stdout:0/389: dwrite dc/d20/f3d [0,4194304] 0 2024-04-16T02:35:52.912 INFO:tasks.workunit.client.0.smithi151.stdout:0/390: dread f3 [0,4194304] 0 2024-04-16T02:35:52.913 INFO:tasks.workunit.client.0.smithi151.stdout:0/391: readlink dc/d15/d34/d67/l6c 0 2024-04-16T02:35:52.923 INFO:tasks.workunit.client.0.smithi151.stdout:2/520: mknod de/d65/d71/d98/ca6 0 2024-04-16T02:35:52.924 INFO:tasks.workunit.client.0.smithi151.stdout:2/521: chown de/d11/d42 1586948756 1 2024-04-16T02:35:52.926 INFO:tasks.workunit.client.0.smithi151.stdout:7/515: symlink d0/d1f/da8/db2/lb4 0 2024-04-16T02:35:52.930 INFO:tasks.workunit.client.0.smithi151.stdout:3/323: mkdir d1/d6/d22/d6c 0 2024-04-16T02:35:52.932 INFO:tasks.workunit.client.0.smithi151.stdout:3/324: chown d1/d6/d4a/d4e/c67 14577 1 2024-04-16T02:35:52.934 INFO:tasks.workunit.client.0.smithi151.stdout:4/491: rename d0/d99/f2c to d0/d11/d19/d87/d3a/d61/fa6 0 2024-04-16T02:35:52.934 INFO:tasks.workunit.client.0.smithi151.stdout:7/516: dwrite d0/d27/f95 [0,4194304] 0 2024-04-16T02:35:52.936 INFO:tasks.workunit.client.0.smithi151.stdout:6/596: mknod d2/d5/df/d23/d82/d8d/d22/ccd 0 2024-04-16T02:35:52.938 INFO:tasks.workunit.client.0.smithi151.stdout:1/466: creat d3/d4/d64/fa8 x:0 0 0 2024-04-16T02:35:52.953 INFO:tasks.workunit.client.0.smithi151.stdout:4/492: rmdir d0/d11/d19/d87/d3a 39 2024-04-16T02:35:52.956 INFO:tasks.workunit.client.0.smithi151.stdout:7/517: creat d0/d1f/d4c/d98/fb5 x:0 0 0 2024-04-16T02:35:52.957 INFO:tasks.workunit.client.0.smithi151.stdout:1/467: dread d3/d1f/d48/f51 [0,4194304] 0 2024-04-16T02:35:52.957 INFO:tasks.workunit.client.0.smithi151.stdout:6/597: creat d2/d5/df/d23/d82/fce x:0 0 0 2024-04-16T02:35:52.964 INFO:tasks.workunit.client.0.smithi151.stdout:3/325: mkdir d1/d6/d20/d2a/d3f/d43/d6d 0 2024-04-16T02:35:52.967 INFO:tasks.workunit.client.0.smithi151.stdout:7/518: mkdir d0/d1f/d4c/db6 0 2024-04-16T02:35:52.970 INFO:tasks.workunit.client.0.smithi151.stdout:8/311: getdents d11/d2a/d35 0 2024-04-16T02:35:52.972 INFO:tasks.workunit.client.0.smithi151.stdout:2/522: creat de/d11/d42/d82/d95/d4b/d60/fa7 x:0 0 0 2024-04-16T02:35:52.978 INFO:tasks.workunit.client.0.smithi151.stdout:6/598: rename d2/d5/df/d23/d82/d8d/d22/d40/d3f/db1/lcc to d2/d5/df/d23/d82/d8d/lcf 0 2024-04-16T02:35:52.980 INFO:tasks.workunit.client.0.smithi151.stdout:6/599: chown d2/d5/df/d23/d82/d8d/fae 895201 1 2024-04-16T02:35:52.982 INFO:tasks.workunit.client.0.smithi151.stdout:2/523: creat de/d15/fa8 x:0 0 0 2024-04-16T02:35:52.984 INFO:tasks.workunit.client.0.smithi151.stdout:3/326: symlink d1/l6e 0 2024-04-16T02:35:52.987 INFO:tasks.workunit.client.0.smithi151.stdout:4/493: creat d0/d11/d19/fa7 x:0 0 0 2024-04-16T02:35:52.989 INFO:tasks.workunit.client.0.smithi151.stdout:8/312: truncate d11/d22/f27 5186068 0 2024-04-16T02:35:52.991 INFO:tasks.workunit.client.0.smithi151.stdout:6/600: dread - d2/d5/df/d23/d45/fa9 zero size 2024-04-16T02:35:52.992 INFO:tasks.workunit.client.0.smithi151.stdout:2/524: dwrite de/d11/d42/d7a/d8b/f8e [0,4194304] 0 2024-04-16T02:35:52.993 INFO:tasks.workunit.client.0.smithi151.stdout:4/494: creat d0/d99/fa8 x:0 0 0 2024-04-16T02:35:52.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:52 smithi151 ceph-mon[29003]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:52.996 INFO:tasks.workunit.client.0.smithi151.stdout:3/327: dread d1/d6/f33 [0,4194304] 0 2024-04-16T02:35:52.997 INFO:tasks.workunit.client.0.smithi151.stdout:3/328: chown d1/d6/d22/d23/d30/d44 363 1 2024-04-16T02:35:53.000 INFO:tasks.workunit.client.0.smithi151.stdout:4/495: creat d0/d64/fa9 x:0 0 0 2024-04-16T02:35:53.000 INFO:tasks.workunit.client.0.smithi151.stdout:2/525: read - de/d15/f50 zero size 2024-04-16T02:35:53.003 INFO:tasks.workunit.client.0.smithi151.stdout:4/496: fdatasync d0/d11/d19/d31/f95 0 2024-04-16T02:35:53.003 INFO:tasks.workunit.client.0.smithi151.stdout:8/313: symlink d11/d2a/d35/d52/l5f 0 2024-04-16T02:35:53.007 INFO:tasks.workunit.client.0.smithi151.stdout:9/446: write d1/d3/d6/d17/d54/f79 [309101,25344] 0 2024-04-16T02:35:53.013 INFO:tasks.workunit.client.0.smithi151.stdout:2/526: dwrite de/d65/f6f [0,4194304] 0 2024-04-16T02:35:53.014 INFO:tasks.workunit.client.0.smithi151.stdout:8/314: read d11/f17 [498742,17709] 0 2024-04-16T02:35:53.015 INFO:tasks.workunit.client.0.smithi151.stdout:9/447: creat d1/d3/d6/d1d/d22/d2c/f93 x:0 0 0 2024-04-16T02:35:53.020 INFO:tasks.workunit.client.0.smithi151.stdout:2/527: read - de/d11/f7d zero size 2024-04-16T02:35:53.024 INFO:tasks.workunit.client.0.smithi151.stdout:2/528: fdatasync de/d15/f5b 0 2024-04-16T02:35:53.024 INFO:tasks.workunit.client.0.smithi151.stdout:5/602: dwrite d2/d6/d19/d1a/d4c/f53 [0,4194304] 0 2024-04-16T02:35:53.027 INFO:tasks.workunit.client.0.smithi151.stdout:0/392: dwrite dc/d3c/f3f [0,4194304] 0 2024-04-16T02:35:53.030 INFO:tasks.workunit.client.0.smithi151.stdout:9/448: link d1/dd/d4a/d64/f76 d1/dd/d4a/f94 0 2024-04-16T02:35:53.031 INFO:tasks.workunit.client.0.smithi151.stdout:7/519: write d0/d1f/d33/d56/f84 [5167869,85298] 0 2024-04-16T02:35:53.032 INFO:tasks.workunit.client.0.smithi151.stdout:9/449: stat d1/d3/d6/d17/d54 0 2024-04-16T02:35:53.035 INFO:tasks.workunit.client.0.smithi151.stdout:9/450: fsync d1/d3/d6/d17/d54/f79 0 2024-04-16T02:35:53.039 INFO:tasks.workunit.client.0.smithi151.stdout:9/451: rename d1 to d1/d3/d6/d17/d40/d56/d95 22 2024-04-16T02:35:53.040 INFO:tasks.workunit.client.0.smithi151.stdout:7/520: mkdir d0/d2c/d38/d85/db7 0 2024-04-16T02:35:53.044 INFO:tasks.workunit.client.0.smithi151.stdout:2/529: creat de/d11/d42/d6b/d79/fa9 x:0 0 0 2024-04-16T02:35:53.046 INFO:tasks.workunit.client.0.smithi151.stdout:9/452: fsync d1/d75/f8b 0 2024-04-16T02:35:53.047 INFO:tasks.workunit.client.0.smithi151.stdout:6/601: write d2/d5/df/d23/d56/f58 [561958,35708] 0 2024-04-16T02:35:53.049 INFO:tasks.workunit.client.0.smithi151.stdout:2/530: write de/d11/d42/d7a/d8b/f8e [3540255,100618] 0 2024-04-16T02:35:53.051 INFO:tasks.workunit.client.0.smithi151.stdout:4/497: write d0/f16 [862654,41136] 0 2024-04-16T02:35:53.051 INFO:tasks.workunit.client.0.smithi151.stdout:2/531: readlink de/d15/l57 0 2024-04-16T02:35:53.054 INFO:tasks.workunit.client.0.smithi151.stdout:2/532: chown de/d15/f9a 100343 1 2024-04-16T02:35:53.055 INFO:tasks.workunit.client.0.smithi151.stdout:9/453: truncate d1/d3/d6/d1d/d22/d2c/f93 997685 0 2024-04-16T02:35:53.055 INFO:tasks.workunit.client.0.smithi151.stdout:7/521: rename d0/f5 to d0/d1f/d94/fb8 0 2024-04-16T02:35:53.061 INFO:tasks.workunit.client.0.smithi151.stdout:7/522: fsync d0/f7 0 2024-04-16T02:35:53.061 INFO:tasks.workunit.client.0.smithi151.stdout:9/454: write d1/d3/d6/d28/f8f [2966300,128615] 0 2024-04-16T02:35:53.065 INFO:tasks.workunit.client.0.smithi151.stdout:6/602: dread d2/d5/df/d23/d82/d8d/d22/f9b [0,4194304] 0 2024-04-16T02:35:53.067 INFO:tasks.workunit.client.0.smithi151.stdout:4/498: rmdir d0/d11/d19/d87/d3a/d65 39 2024-04-16T02:35:53.074 INFO:tasks.workunit.client.0.smithi151.stdout:7/523: dread d0/d2c/d38/d43/d78/f9a [0,4194304] 0 2024-04-16T02:35:53.075 INFO:tasks.workunit.client.0.smithi151.stdout:4/499: rmdir d0/d11/d19/d31/d4a 39 2024-04-16T02:35:53.079 INFO:tasks.workunit.client.0.smithi151.stdout:0/393: dread dc/d15/d34/d67/f30 [0,4194304] 0 2024-04-16T02:35:53.080 INFO:tasks.workunit.client.0.smithi151.stdout:8/315: dwrite d11/f38 [0,4194304] 0 2024-04-16T02:35:53.081 INFO:tasks.workunit.client.0.smithi151.stdout:9/455: dread d1/d3/d6/d17/f31 [0,4194304] 0 2024-04-16T02:35:53.082 INFO:tasks.workunit.client.0.smithi151.stdout:7/524: symlink d0/d1f/d4c/d82/lb9 0 2024-04-16T02:35:53.083 INFO:tasks.workunit.client.0.smithi151.stdout:9/456: chown d1/d3/d6/d28/l59 45269185 1 2024-04-16T02:35:53.084 INFO:tasks.workunit.client.0.smithi151.stdout:2/533: dread de/d11/d42/d82/d95/d4b/f73 [0,4194304] 0 2024-04-16T02:35:53.087 INFO:tasks.workunit.client.0.smithi151.stdout:6/603: link d2/d5/df/d23/d56/f7e d2/d5/df/fd0 0 2024-04-16T02:35:53.090 INFO:tasks.workunit.client.0.smithi151.stdout:0/394: dread - dc/d15/d34/d4e/f55 zero size 2024-04-16T02:35:53.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.088+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa294005cc0 con 0x7fa2a8100bd0 2024-04-16T02:35:53.090 INFO:tasks.workunit.client.0.smithi151.stdout:6/604: write d2/d5/df/d23/d56/f84 [953844,104399] 0 2024-04-16T02:35:53.091 INFO:tasks.workunit.client.0.smithi151.stdout:2/534: dread de/d15/d1e/f67 [0,4194304] 0 2024-04-16T02:35:53.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.090+0000 7fa2a57fa700 1 -- 172.21.15.151:0/2580258582 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7fa29805b5c0 con 0x7fa2a8100bd0 2024-04-16T02:35:53.093 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:35:53.093 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:35:53.093 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:35:53.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 1 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 1 2024-04-16T02:35:53.094 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:35:53.095 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:35:53.095 INFO:tasks.workunit.client.0.smithi151.stdout:4/500: mkdir d0/daa 0 2024-04-16T02:35:53.095 INFO:tasks.workunit.client.0.smithi151.stdout:4/501: chown d0/d73/l91 621 1 2024-04-16T02:35:53.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fa29006d8a0 msgr2=0x7fa29006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:53.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fa29006d8a0 0x7fa29006fd60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa29c005950 tx=0x7fa29c0058e0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 msgr2=0x7fa2a8196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 0x7fa2a8196360 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa29800bd70 tx=0x7fa29800be50 comp rx=0 tx=0).stop 2024-04-16T02:35:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 shutdown_connections 2024-04-16T02:35:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fa29006d8a0 0x7fa29006fd60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2a8100bd0 0x7fa2a8196360 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 --2- 172.21.15.151:0/2580258582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2a8101dd0 0x7fa2a81968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 >> 172.21.15.151:0/2580258582 conn(0x7fa2a80fc190 msgr2=0x7fa2a8105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 shutdown_connections 2024-04-16T02:35:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.094+0000 7fa2ae970700 1 -- 172.21.15.151:0/2580258582 wait complete. 2024-04-16T02:35:53.097 INFO:tasks.workunit.client.0.smithi151.stdout:2/535: mkdir de/d65/daa 0 2024-04-16T02:35:53.098 INFO:tasks.workunit.client.0.smithi151.stdout:6/605: rmdir d2/d5/df/d23/d56/d72 39 2024-04-16T02:35:53.099 INFO:tasks.workunit.client.0.smithi151.stdout:9/457: getdents d1/d3/d5f 0 2024-04-16T02:35:53.102 INFO:tasks.workunit.client.0.smithi151.stdout:4/502: rmdir d0/d64 39 2024-04-16T02:35:53.104 INFO:tasks.workunit.client.0.smithi151.stdout:2/536: truncate de/d15/f5b 182399 0 2024-04-16T02:35:53.106 INFO:tasks.workunit.client.0.smithi151.stdout:6/606: creat d2/d5/df/d23/d45/fd1 x:0 0 0 2024-04-16T02:35:53.106 INFO:tasks.workunit.client.0.smithi151.stdout:0/395: write dc/f39 [1505088,20465] 0 2024-04-16T02:35:53.107 INFO:tasks.workunit.client.0.smithi151.stdout:8/316: rename d11/c50 to d11/d22/c60 0 2024-04-16T02:35:53.109 INFO:tasks.workunit.client.0.smithi151.stdout:7/525: getdents d0/d1f/d33/d56 0 2024-04-16T02:35:53.111 INFO:tasks.workunit.client.0.smithi151.stdout:7/526: write d0/d1f/d4c/d82/fa3 [3081883,33334] 0 2024-04-16T02:35:53.113 INFO:tasks.workunit.client.0.smithi151.stdout:0/396: mknod dc/d20/c7c 0 2024-04-16T02:35:53.116 INFO:tasks.workunit.client.0.smithi151.stdout:6/607: read d2/d5/df/d23/f8c [292230,106355] 0 2024-04-16T02:35:53.119 INFO:tasks.workunit.client.0.smithi151.stdout:2/537: write de/d11/d42/d82/d95/d4b/f55 [4030302,15073] 0 2024-04-16T02:35:53.122 INFO:tasks.workunit.client.0.smithi151.stdout:4/503: symlink d0/d11/d19/d31/d4a/d60/lab 0 2024-04-16T02:35:53.124 INFO:tasks.workunit.client.0.smithi151.stdout:7/527: creat d0/d2c/d38/fba x:0 0 0 2024-04-16T02:35:53.126 INFO:tasks.workunit.client.0.smithi151.stdout:0/397: creat dc/d20/f7d x:0 0 0 2024-04-16T02:35:53.130 INFO:tasks.workunit.client.0.smithi151.stdout:6/608: creat d2/d5/df/d23/d82/d8d/d22/d40/d3f/fd2 x:0 0 0 2024-04-16T02:35:53.132 INFO:tasks.workunit.client.0.smithi151.stdout:8/317: dread d11/f43 [0,4194304] 0 2024-04-16T02:35:53.132 INFO:tasks.workunit.client.0.smithi151.stdout:9/458: getdents d1/d3/d6/d17/d7a 0 2024-04-16T02:35:53.135 INFO:tasks.workunit.client.0.smithi151.stdout:0/398: creat dc/d4c/d72/f7e x:0 0 0 2024-04-16T02:35:53.136 INFO:tasks.workunit.client.0.smithi151.stdout:9/459: write d1/d3/d10/f4d [7771498,115882] 0 2024-04-16T02:35:53.143 INFO:tasks.workunit.client.0.smithi151.stdout:4/504: write d0/d99/d59/f7f [630310,100714] 0 2024-04-16T02:35:53.146 INFO:tasks.workunit.client.0.smithi151.stdout:7/528: dwrite d0/d2c/d38/d43/d78/f9a [0,4194304] 0 2024-04-16T02:35:53.147 INFO:tasks.workunit.client.0.smithi151.stdout:2/538: creat de/fab x:0 0 0 2024-04-16T02:35:53.155 INFO:tasks.workunit.client.0.smithi151.stdout:7/529: write d0/f9e [2323161,70975] 0 2024-04-16T02:35:53.157 INFO:tasks.workunit.client.0.smithi151.stdout:7/530: readlink d0/d27/l7d 0 2024-04-16T02:35:53.158 INFO:tasks.workunit.client.0.smithi151.stdout:9/460: dwrite d1/d3/d6/d17/d7a/f6d [0,4194304] 0 2024-04-16T02:35:53.159 INFO:tasks.workunit.client.0.smithi151.stdout:6/609: dwrite d2/d3d/f9d [0,4194304] 0 2024-04-16T02:35:53.162 INFO:tasks.workunit.client.0.smithi151.stdout:2/539: creat de/d11/d42/d5e/d6c/d6d/da5/fac x:0 0 0 2024-04-16T02:35:53.165 INFO:tasks.workunit.client.0.smithi151.stdout:0/399: dread dc/d15/f45 [0,4194304] 0 2024-04-16T02:35:53.166 INFO:tasks.workunit.client.0.smithi151.stdout:4/505: rename d0/d11/f53 to d0/d11/d19/d31/d80/da0/fac 0 2024-04-16T02:35:53.168 INFO:tasks.workunit.client.0.smithi151.stdout:8/318: getdents d11/d2a/d35/d52 0 2024-04-16T02:35:53.169 INFO:tasks.workunit.client.0.smithi151.stdout:6/610: write d2/d13/d6f/da3/fcb [487753,33680] 0 2024-04-16T02:35:53.170 INFO:tasks.workunit.client.0.smithi151.stdout:2/540: symlink de/d11/d42/d82/d95/d4b/d60/lad 0 2024-04-16T02:35:53.171 INFO:tasks.workunit.client.0.smithi151.stdout:7/531: truncate d0/d1f/d46/f5d 205525 0 2024-04-16T02:35:53.174 INFO:tasks.workunit.client.0.smithi151.stdout:9/461: link d1/d3/d6/d4f/d6b/f77 d1/dd/d38/f96 0 2024-04-16T02:35:53.176 INFO:tasks.workunit.client.0.smithi151.stdout:4/506: fdatasync d0/d11/d19/f9f 0 2024-04-16T02:35:53.176 INFO:tasks.workunit.client.0.smithi151.stdout:7/532: unlink d0/d1f/d31/d42/f76 0 2024-04-16T02:35:53.187 INFO:tasks.workunit.client.0.smithi151.stdout:0/400: rename dc/d15/f19 to dc/d15/f7f 0 2024-04-16T02:35:53.189 INFO:tasks.workunit.client.0.smithi151.stdout:0/401: chown dc/d15/f45 61438531 1 2024-04-16T02:35:53.189 INFO:tasks.workunit.client.0.smithi151.stdout:9/462: dwrite d1/d3/f41 [0,4194304] 0 2024-04-16T02:35:53.192 INFO:tasks.workunit.client.0.smithi151.stdout:9/463: stat d1/d3/d10 0 2024-04-16T02:35:53.193 INFO:tasks.workunit.client.0.smithi151.stdout:6/611: symlink d2/d13/ld3 0 2024-04-16T02:35:53.193 INFO:tasks.workunit.client.0.smithi151.stdout:4/507: chown d0/d11/d19/d87/d3a/d50/c96 996960825 1 2024-04-16T02:35:53.199 INFO:tasks.workunit.client.0.smithi151.stdout:2/541: truncate de/d15/f17 1201343 0 2024-04-16T02:35:53.201 INFO:tasks.workunit.client.0.smithi151.stdout:0/402: creat dc/d20/f80 x:0 0 0 2024-04-16T02:35:53.204 INFO:tasks.workunit.client.0.smithi151.stdout:7/533: write d0/d1f/d60/f91 [1147690,124937] 0 2024-04-16T02:35:53.206 INFO:tasks.workunit.client.0.smithi151.stdout:4/508: truncate d0/d11/d19/f7e 706046 0 2024-04-16T02:35:53.213 INFO:tasks.workunit.client.0.smithi151.stdout:0/403: rename dc/d15/d34/c73 to dc/d20/c81 0 2024-04-16T02:35:53.214 INFO:tasks.workunit.client.0.smithi151.stdout:7/534: dwrite d0/f9e [0,4194304] 0 2024-04-16T02:35:53.222 INFO:tasks.workunit.client.0.smithi151.stdout:7/535: rename d0/d1f/d33/d56/ca2 to d0/d2c/d38/d85/cbb 0 2024-04-16T02:35:53.222 INFO:tasks.workunit.client.0.smithi151.stdout:6/612: dread d2/d5/df/d23/d82/d8d/d22/f4e [0,4194304] 0 2024-04-16T02:35:53.223 INFO:tasks.workunit.client.0.smithi151.stdout:9/464: dread d1/d3/d5a/f69 [0,4194304] 0 2024-04-16T02:35:53.224 INFO:tasks.workunit.client.0.smithi151.stdout:4/509: mkdir d0/d11/d19/d31/dad 0 2024-04-16T02:35:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:52 smithi153 ceph-mon[36747]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:53.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.269+0000 7fc5eb469700 1 -- 172.21.15.151:0/2825943503 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 msgr2=0x7fc5e4101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:53.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.269+0000 7fc5eb469700 1 --2- 172.21.15.151:0/2825943503 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 0x7fc5e4101020 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009910 tx=0x7fc5e0009c20 comp rx=0 tx=0).stop 2024-04-16T02:35:53.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.269+0000 7fc5eb469700 1 -- 172.21.15.151:0/2825943503 shutdown_connections 2024-04-16T02:35:53.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.269+0000 7fc5eb469700 1 --2- 172.21.15.151:0/2825943503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5e4101e00 0x7fc5e4102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.269+0000 7fc5eb469700 1 --2- 172.21.15.151:0/2825943503 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 0x7fc5e4101020 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.269+0000 7fc5eb469700 1 -- 172.21.15.151:0/2825943503 >> 172.21.15.151:0/2825943503 conn(0x7fc5e40fc180 msgr2=0x7fc5e40fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:53.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.269+0000 7fc5eb469700 1 -- 172.21.15.151:0/2825943503 shutdown_connections 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.269+0000 7fc5eb469700 1 -- 172.21.15.151:0/2825943503 wait complete. 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5eb469700 1 Processor -- start 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5eb469700 1 -- start start 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5eb469700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 0x7fc5e41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5eb469700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5e4101e00 0x7fc5e4198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5eb469700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e4199100 con 0x7fc5e4100c00 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5eb469700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e4199240 con 0x7fc5e4101e00 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5e9205700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 0x7fc5e41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5e9205700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 0x7fc5e41985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46834/0 (socket says 172.21.15.151:46834) 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5e9205700 1 -- 172.21.15.151:0/3817404097 learned_addr learned my addr 172.21.15.151:0/3817404097 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5e8a04700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5e4101e00 0x7fc5e4198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5e9205700 1 -- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5e4101e00 msgr2=0x7fc5e4198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5e9205700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5e4101e00 0x7fc5e4198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.270+0000 7fc5e9205700 1 -- 172.21.15.151:0/3817404097 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e00095f0 con 0x7fc5e4100c00 2024-04-16T02:35:53.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.271+0000 7fc5e9205700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 0x7fc5e41985a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000bda0 tx=0x7fc5e000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:53.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.271+0000 7fc5d67fc700 1 -- 172.21.15.151:0/3817404097 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e001d070 con 0x7fc5e4100c00 2024-04-16T02:35:53.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.271+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e419dc90 con 0x7fc5e4100c00 2024-04-16T02:35:53.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.271+0000 7fc5d67fc700 1 -- 172.21.15.151:0/3817404097 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc5e000f460 con 0x7fc5e4100c00 2024-04-16T02:35:53.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.271+0000 7fc5d67fc700 1 -- 172.21.15.151:0/3817404097 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e0021640 con 0x7fc5e4100c00 2024-04-16T02:35:53.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.271+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e419e180 con 0x7fc5e4100c00 2024-04-16T02:35:53.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.272+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e4105ee0 con 0x7fc5e4100c00 2024-04-16T02:35:53.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.273+0000 7fc5d67fc700 1 -- 172.21.15.151:0/3817404097 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7fc5e000fa00 con 0x7fc5e4100c00 2024-04-16T02:35:53.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.273+0000 7fc5d67fc700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc5d006d850 0x7fc5d006fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:35:53.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.273+0000 7fc5d67fc700 1 -- 172.21.15.151:0/3817404097 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fc5e008e620 con 0x7fc5e4100c00 2024-04-16T02:35:53.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.274+0000 7fc5e8a04700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc5d006d850 0x7fc5d006fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:35:53.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.274+0000 7fc5e8a04700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc5d006d850 0x7fc5d006fd10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8005950 tx=0x7fc5d80058e0 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:35:53.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.278+0000 7fc5d67fc700 1 -- 172.21.15.151:0/3817404097 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fc5e005b310 con 0x7fc5e4100c00 2024-04-16T02:35:53.316 INFO:tasks.workunit.client.0.smithi151.stdout:0/404: symlink dc/d15/d34/d67/l82 0 2024-04-16T02:35:53.317 INFO:tasks.workunit.client.0.smithi151.stdout:9/465: creat d1/d3/d6/d17/d40/d56/f97 x:0 0 0 2024-04-16T02:35:53.319 INFO:tasks.workunit.client.0.smithi151.stdout:4/510: stat d0/d11/d19/d87/d3a/d65 0 2024-04-16T02:35:53.320 INFO:tasks.workunit.client.0.smithi151.stdout:2/542: dread de/d11/f2b [0,4194304] 0 2024-04-16T02:35:53.322 INFO:tasks.workunit.client.0.smithi151.stdout:7/536: rename d0/d1f/f2a to d0/fbc 0 2024-04-16T02:35:53.324 INFO:tasks.workunit.client.0.smithi151.stdout:9/466: symlink d1/d3/d6/d17/d40/d7d/l98 0 2024-04-16T02:35:53.331 INFO:tasks.workunit.client.0.smithi151.stdout:7/537: creat d0/d1f/d60/fbd x:0 0 0 2024-04-16T02:35:53.334 INFO:tasks.workunit.client.0.smithi151.stdout:4/511: mknod d0/d11/d19/d87/d3a/d65/d98/cae 0 2024-04-16T02:35:53.335 INFO:tasks.workunit.client.0.smithi151.stdout:9/467: symlink d1/d3/d5f/l99 0 2024-04-16T02:35:53.337 INFO:tasks.workunit.client.0.smithi151.stdout:7/538: creat d0/d1f/d33/d89/fbe x:0 0 0 2024-04-16T02:35:53.338 INFO:tasks.workunit.client.0.smithi151.stdout:0/405: getdents dc/d15/d34/d3b/d71 0 2024-04-16T02:35:53.343 INFO:tasks.workunit.client.0.smithi151.stdout:7/539: mknod d0/d2c/cbf 0 2024-04-16T02:35:53.344 INFO:tasks.workunit.client.0.smithi151.stdout:6/613: truncate d2/d13/d46/fbe 724114 0 2024-04-16T02:35:53.347 INFO:tasks.workunit.client.0.smithi151.stdout:2/543: dread de/f3e [0,4194304] 0 2024-04-16T02:35:53.349 INFO:tasks.workunit.client.0.smithi151.stdout:7/540: read d0/f18 [348821,113274] 0 2024-04-16T02:35:53.352 INFO:tasks.workunit.client.0.smithi151.stdout:0/406: dread dc/f1f [0,4194304] 0 2024-04-16T02:35:53.356 INFO:tasks.workunit.client.0.smithi151.stdout:2/544: creat de/d15/d78/fae x:0 0 0 2024-04-16T02:35:53.358 INFO:tasks.workunit.client.0.smithi151.stdout:7/541: mkdir d0/d1f/d99/dc0 0 2024-04-16T02:35:53.358 INFO:tasks.workunit.client.0.smithi151.stdout:4/512: write d0/d64/f76 [638911,117986] 0 2024-04-16T02:35:53.359 INFO:tasks.workunit.client.0.smithi151.stdout:0/407: creat dc/d15/d34/d4e/f83 x:0 0 0 2024-04-16T02:35:53.360 INFO:tasks.workunit.client.0.smithi151.stdout:0/408: readlink dc/l5b 0 2024-04-16T02:35:53.362 INFO:tasks.workunit.client.0.smithi151.stdout:9/468: dread d1/f2e [0,4194304] 0 2024-04-16T02:35:53.362 INFO:tasks.workunit.client.0.smithi151.stdout:4/513: readlink d0/d11/d19/d87/d3a/d50/l57 0 2024-04-16T02:35:53.367 INFO:tasks.workunit.client.0.smithi151.stdout:4/514: creat d0/d9b/faf x:0 0 0 2024-04-16T02:35:53.372 INFO:tasks.workunit.client.0.smithi151.stdout:6/614: dwrite d2/d13/d46/f53 [0,4194304] 0 2024-04-16T02:35:53.372 INFO:tasks.workunit.client.0.smithi151.stdout:7/542: rename d0/c47 to d0/d2c/d38/dab/cc1 0 2024-04-16T02:35:53.375 INFO:tasks.workunit.client.0.smithi151.stdout:0/409: truncate dc/d3c/f48 845101 0 2024-04-16T02:35:53.376 INFO:tasks.workunit.client.0.smithi151.stdout:7/543: chown d0/d1f/d94/fa1 0 1 2024-04-16T02:35:53.377 INFO:tasks.workunit.client.0.smithi151.stdout:0/410: chown dc/d3c/l41 154 1 2024-04-16T02:35:53.378 INFO:tasks.workunit.client.0.smithi151.stdout:6/615: creat d2/d5/df/d20/d4f/d81/fd4 x:0 0 0 2024-04-16T02:35:53.383 INFO:tasks.workunit.client.0.smithi151.stdout:2/545: dwrite de/d11/d42/d7a/d8b/f74 [0,4194304] 0 2024-04-16T02:35:53.384 INFO:tasks.workunit.client.0.smithi151.stdout:6/616: chown d2/d5/df/d23/d56/f58 7 1 2024-04-16T02:35:53.386 INFO:tasks.workunit.client.0.smithi151.stdout:2/546: stat de/d11/d42/d82/d95/d4b/f73 0 2024-04-16T02:35:53.386 INFO:tasks.workunit.client.0.smithi151.stdout:7/544: truncate d0/d1f/d33/d89/f90 775280 0 2024-04-16T02:35:53.390 INFO:tasks.workunit.client.0.smithi151.stdout:6/617: write d2/d5/df/d23/d56/d72/fa0 [191332,34660] 0 2024-04-16T02:35:53.394 INFO:tasks.workunit.client.0.smithi151.stdout:7/545: fdatasync d0/d1f/d46/d4f/f50 0 2024-04-16T02:35:53.395 INFO:tasks.workunit.client.0.smithi151.stdout:0/411: getdents dc/d15/d34/d67 0 2024-04-16T02:35:53.397 INFO:tasks.workunit.client.0.smithi151.stdout:9/469: dread d1/d3/d6/f13 [0,4194304] 0 2024-04-16T02:35:53.404 INFO:tasks.workunit.client.0.smithi151.stdout:0/412: dread dc/d15/d34/d3b/f44 [0,4194304] 0 2024-04-16T02:35:53.404 INFO:tasks.workunit.client.0.smithi151.stdout:2/547: fsync de/d11/d42/d7a/d8b/f74 0 2024-04-16T02:35:53.406 INFO:tasks.workunit.client.0.smithi151.stdout:6/618: link d2/d5/df/d20/d4f/f9e d2/d5/df/d23/d82/d8d/d22/d40/d3f/fd5 0 2024-04-16T02:35:53.407 INFO:tasks.workunit.client.0.smithi151.stdout:4/515: dread d0/f55 [0,4194304] 0 2024-04-16T02:35:53.408 INFO:tasks.workunit.client.0.smithi151.stdout:9/470: dwrite d1/d3/d6/f1b [0,4194304] 0 2024-04-16T02:35:53.408 INFO:tasks.workunit.client.0.smithi151.stdout:2/548: stat de/d11/d42/d5e 0 2024-04-16T02:35:53.410 INFO:tasks.workunit.client.0.smithi151.stdout:7/546: dread d0/d27/f95 [0,4194304] 0 2024-04-16T02:35:53.412 INFO:tasks.workunit.client.0.smithi151.stdout:0/413: rename dc/f1f to dc/d3c/f84 0 2024-04-16T02:35:53.413 INFO:tasks.workunit.client.0.smithi151.stdout:6/619: dread - d2/d5/df/d23/d82/d8d/d22/d40/d3f/fd2 zero size 2024-04-16T02:35:53.416 INFO:tasks.workunit.client.0.smithi151.stdout:9/471: write d1/d3/d6/d4f/d6b/f77 [1935480,10960] 0 2024-04-16T02:35:53.416 INFO:tasks.workunit.client.0.smithi151.stdout:4/516: creat d0/d11/d19/d87/d3a/d50/fb0 x:0 0 0 2024-04-16T02:35:53.417 INFO:tasks.workunit.client.0.smithi151.stdout:7/547: mkdir d0/d1f/d4c/db6/dc2 0 2024-04-16T02:35:53.424 INFO:tasks.workunit.client.0.smithi151.stdout:6/620: rename d2/d5/la to d2/d5/d2e/dba/ld6 0 2024-04-16T02:35:53.425 INFO:tasks.workunit.client.0.smithi151.stdout:9/472: creat d1/d3/d5a/d8e/f9a x:0 0 0 2024-04-16T02:35:53.426 INFO:tasks.workunit.client.0.smithi151.stdout:7/548: dwrite d0/d1f/d46/f5a [0,4194304] 0 2024-04-16T02:35:53.427 INFO:tasks.workunit.client.0.smithi151.stdout:6/621: chown d2/d13/l19 1720849 1 2024-04-16T02:35:53.428 INFO:tasks.workunit.client.0.smithi151.stdout:9/473: write d1/d3/d6/d1d/f78 [1012662,85484] 0 2024-04-16T02:35:53.428 INFO:tasks.workunit.client.0.smithi151.stdout:0/414: creat dc/d15/d34/d67/d69/d6f/f85 x:0 0 0 2024-04-16T02:35:53.434 INFO:tasks.workunit.client.0.smithi151.stdout:7/549: mkdir d0/dc3 0 2024-04-16T02:35:53.436 INFO:tasks.workunit.client.0.smithi151.stdout:0/415: truncate dc/d15/f45 154119 0 2024-04-16T02:35:53.436 INFO:tasks.workunit.client.0.smithi151.stdout:0/416: chown dc/d3c/f75 0 1 2024-04-16T02:35:53.437 INFO:tasks.workunit.client.0.smithi151.stdout:7/550: truncate d0/d27/f9f 847088 0 2024-04-16T02:35:53.438 INFO:tasks.workunit.client.0.smithi151.stdout:7/551: fdatasync d0/d1f/d46/f8d 0 2024-04-16T02:35:53.440 INFO:tasks.workunit.client.0.smithi151.stdout:9/474: rename d1/d3/d6/d1d to d1/d3/d6/d17/d54/d9b 0 2024-04-16T02:35:53.443 INFO:tasks.workunit.client.0.smithi151.stdout:2/549: dread de/f49 [0,4194304] 0 2024-04-16T02:35:53.444 INFO:tasks.workunit.client.0.smithi151.stdout:2/550: chown de/d15/fa8 11 1 2024-04-16T02:35:53.447 INFO:tasks.workunit.client.0.smithi151.stdout:7/552: unlink d0/d2c/d38/d5e/caf 0 2024-04-16T02:35:53.448 INFO:tasks.workunit.client.0.smithi151.stdout:7/553: chown d0/d2c/d38/d5e/d87/fad 189898384 1 2024-04-16T02:35:53.450 INFO:tasks.workunit.client.0.smithi151.stdout:0/417: fsync dc/d15/f25 0 2024-04-16T02:35:53.455 INFO:tasks.workunit.client.0.smithi151.stdout:0/418: truncate dc/d15/d34/d67/f3e 1732637 0 2024-04-16T02:35:53.457 INFO:tasks.workunit.client.0.smithi151.stdout:6/622: dwrite d2/d5/f8f [0,4194304] 0 2024-04-16T02:35:53.458 INFO:tasks.workunit.client.0.smithi151.stdout:6/623: truncate d2/d13/d46/fbf 163529 0 2024-04-16T02:35:53.459 INFO:tasks.workunit.client.0.smithi151.stdout:6/624: dread - d2/d5/df/d23/d31/d52/fc2 zero size 2024-04-16T02:35:53.464 INFO:tasks.workunit.client.0.smithi151.stdout:0/419: symlink dc/d4c/l86 0 2024-04-16T02:35:53.464 INFO:tasks.workunit.client.0.smithi151.stdout:9/475: creat d1/d3/d6/f9c x:0 0 0 2024-04-16T02:35:53.465 INFO:tasks.workunit.client.0.smithi151.stdout:7/554: symlink d0/d2c/d38/d85/db7/lc4 0 2024-04-16T02:35:53.466 INFO:tasks.workunit.client.0.smithi151.stdout:0/420: readlink dc/d15/d34/d67/l82 0 2024-04-16T02:35:53.468 INFO:tasks.workunit.client.0.smithi151.stdout:2/551: dwrite de/d15/f34 [4194304,4194304] 0 2024-04-16T02:35:53.472 INFO:tasks.workunit.client.0.smithi151.stdout:7/555: creat d0/d2c/d8e/fc5 x:0 0 0 2024-04-16T02:35:53.473 INFO:tasks.workunit.client.0.smithi151.stdout:7/556: chown d0/fbc 116 1 2024-04-16T02:35:53.475 INFO:tasks.workunit.client.0.smithi151.stdout:0/421: creat dc/d15/f87 x:0 0 0 2024-04-16T02:35:53.475 INFO:tasks.workunit.client.0.smithi151.stdout:9/476: mkdir d1/d3/d6/d17/d40/d7d/d87/d9d 0 2024-04-16T02:35:53.477 INFO:tasks.workunit.client.0.smithi151.stdout:9/477: write d1/d3/d6/d17/d40/d56/f61 [2209720,32549] 0 2024-04-16T02:35:53.478 INFO:tasks.workunit.client.0.smithi151.stdout:7/557: rename d0/d1f/d94/lb1 to d0/d1f/d33/lc6 0 2024-04-16T02:35:53.478 INFO:tasks.workunit.client.0.smithi151.stdout:7/558: stat d0/d27/l7d 0 2024-04-16T02:35:53.479 INFO:tasks.workunit.client.0.smithi151.stdout:7/559: chown d0/d1f/da8 1 1 2024-04-16T02:35:53.481 INFO:tasks.workunit.client.0.smithi151.stdout:7/560: fdatasync d0/d27/f83 0 2024-04-16T02:35:53.483 INFO:tasks.workunit.client.0.smithi151.stdout:2/552: getdents de 0 2024-04-16T02:35:53.485 INFO:tasks.workunit.client.0.smithi151.stdout:2/553: chown de/d11/d42/d7a/d8b/l66 107 1 2024-04-16T02:35:53.488 INFO:tasks.workunit.client.0.smithi151.stdout:9/478: link d1/dd/d4a/d64/f80 d1/d3/d6/d17/d54/d9b/d22/d36/f9e 0 2024-04-16T02:35:53.494 INFO:tasks.workunit.client.0.smithi151.stdout:7/561: dwrite d0/d27/f95 [0,4194304] 0 2024-04-16T02:35:53.495 INFO:tasks.workunit.client.0.smithi151.stdout:9/479: dread d1/dd/d4a/f53 [0,4194304] 0 2024-04-16T02:35:53.496 INFO:tasks.workunit.client.0.smithi151.stdout:2/554: truncate de/d15/f17 734458 0 2024-04-16T02:35:53.498 INFO:tasks.workunit.client.0.smithi151.stdout:2/555: chown de/d65/d71/d7c/l81 10 1 2024-04-16T02:35:53.499 INFO:tasks.workunit.client.0.smithi151.stdout:9/480: readlink d1/d3/d6/d17/d54/d9b/l4e 0 2024-04-16T02:35:53.500 INFO:tasks.workunit.client.0.smithi151.stdout:2/556: readlink de/l83 0 2024-04-16T02:35:53.502 INFO:tasks.workunit.client.0.smithi151.stdout:9/481: fdatasync d1/d3/d6/d17/d40/d56/f97 0 2024-04-16T02:35:53.502 INFO:tasks.workunit.client.0.smithi151.stdout:2/557: stat l9 0 2024-04-16T02:35:53.505 INFO:tasks.workunit.client.0.smithi151.stdout:0/422: dwrite dc/ff [4194304,4194304] 0 2024-04-16T02:35:53.519 INFO:tasks.workunit.client.0.smithi151.stdout:2/558: link de/d11/c31 de/d65/d71/d7c/caf 0 2024-04-16T02:35:53.520 INFO:tasks.workunit.client.0.smithi151.stdout:7/562: dread d0/f11 [0,4194304] 0 2024-04-16T02:35:53.520 INFO:tasks.workunit.client.0.smithi151.stdout:0/423: dwrite dc/d15/d34/f4d [0,4194304] 0 2024-04-16T02:35:53.522 INFO:tasks.workunit.client.0.smithi151.stdout:9/482: link d1/d3/c62 d1/d3/d10/c9f 0 2024-04-16T02:35:53.524 INFO:tasks.workunit.client.0.smithi151.stdout:9/483: fsync d1/d3/f41 0 2024-04-16T02:35:53.527 INFO:tasks.workunit.client.0.smithi151.stdout:0/424: creat dc/d4c/f88 x:0 0 0 2024-04-16T02:35:53.528 INFO:tasks.workunit.client.0.smithi151.stdout:9/484: mkdir d1/d3/d6/d17/d40/d7d/da0 0 2024-04-16T02:35:53.531 INFO:tasks.workunit.client.0.smithi151.stdout:9/485: chown d1/d3/d6/f13 4548 1 2024-04-16T02:35:53.531 INFO:tasks.workunit.client.0.smithi151.stdout:0/425: rename dc/d15/d34 to dc/d15/d34/d89 22 2024-04-16T02:35:53.533 INFO:tasks.workunit.client.0.smithi151.stdout:9/486: fdatasync d1/d3/d5a/f82 0 2024-04-16T02:35:53.534 INFO:tasks.workunit.client.0.smithi151.stdout:7/563: dwrite d0/fbc [0,4194304] 0 2024-04-16T02:35:53.536 INFO:tasks.workunit.client.0.smithi151.stdout:9/487: rmdir d1 39 2024-04-16T02:35:53.538 INFO:tasks.workunit.client.0.smithi151.stdout:0/426: mkdir dc/d15/d34/d67/d69/d8a 0 2024-04-16T02:35:53.539 INFO:tasks.workunit.client.0.smithi151.stdout:7/564: rmdir d0/d1f/d33/d89 39 2024-04-16T02:35:53.540 INFO:tasks.workunit.client.0.smithi151.stdout:0/427: write dc/d20/f4a [1257022,97907] 0 2024-04-16T02:35:53.542 INFO:tasks.workunit.client.0.smithi151.stdout:7/565: write d0/d1f/d4c/d82/db3/f70 [5081045,99464] 0 2024-04-16T02:35:53.552 INFO:tasks.workunit.client.0.smithi151.stdout:9/488: creat d1/d3/d6/d28/d39/d43/fa1 x:0 0 0 2024-04-16T02:35:53.561 INFO:tasks.workunit.client.0.smithi151.stdout:0/428: dread dc/d3c/f84 [0,4194304] 0 2024-04-16T02:35:53.562 INFO:tasks.workunit.client.0.smithi151.stdout:9/489: mknod d1/dd/d38/ca2 0 2024-04-16T02:35:53.562 INFO:tasks.workunit.client.0.smithi151.stdout:0/429: stat dc/c24 0 2024-04-16T02:35:53.565 INFO:tasks.workunit.client.0.smithi151.stdout:9/490: write d1/dd/d38/f96 [1171013,74159] 0 2024-04-16T02:35:53.567 INFO:tasks.workunit.client.0.smithi151.stdout:2/559: dread de/f3a [0,4194304] 0 2024-04-16T02:35:53.592 INFO:tasks.workunit.client.0.smithi151.stdout:0/430: creat dc/d15/d34/d67/d69/f8b x:0 0 0 2024-04-16T02:35:53.593 INFO:tasks.workunit.client.0.smithi151.stdout:9/491: creat d1/d3/d6/d28/d39/d43/fa3 x:0 0 0 2024-04-16T02:35:53.596 INFO:tasks.workunit.client.0.smithi151.stdout:7/566: getdents d0/d1f/d4c/d82/db3 0 2024-04-16T02:35:53.598 INFO:tasks.workunit.client.0.smithi151.stdout:7/567: fdatasync d0/d1f/d33/d56/f84 0 2024-04-16T02:35:53.599 INFO:tasks.workunit.client.0.smithi151.stdout:0/431: mknod dc/d15/d34/d67/d69/d8a/c8c 0 2024-04-16T02:35:53.600 INFO:tasks.workunit.client.0.smithi151.stdout:7/568: chown d0/d2c/d38/d5e/d87/fad 0 1 2024-04-16T02:35:53.603 INFO:tasks.workunit.client.0.smithi151.stdout:0/432: mknod dc/d15/d34/d3b/c8d 0 2024-04-16T02:35:53.605 INFO:tasks.workunit.client.0.smithi151.stdout:7/569: unlink d0/d1f/d4c/d82/db3/l5b 0 2024-04-16T02:35:53.613 INFO:tasks.workunit.client.0.smithi151.stdout:0/433: getdents dc 0 2024-04-16T02:35:53.614 INFO:tasks.workunit.client.0.smithi151.stdout:7/570: getdents d0/d1f/d4c/d98 0 2024-04-16T02:35:53.615 INFO:tasks.workunit.client.0.smithi151.stdout:7/571: truncate d0/d2c/d38/fba 280664 0 2024-04-16T02:35:53.620 INFO:tasks.workunit.client.0.smithi151.stdout:0/434: dread dc/d15/d34/f5d [0,4194304] 0 2024-04-16T02:35:53.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.656+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5e4061b90 con 0x7fc5d006d850 2024-04-16T02:35:53.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.658+0000 7fc5d67fc700 1 -- 172.21.15.151:0/3817404097 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fc5e4061b90 con 0x7fc5d006d850 2024-04-16T02:35:53.660 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:35:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:35:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:35:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-16T02:35:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [], 2024-04-16T02:35:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "1/2 daemons upgraded", 2024-04-16T02:35:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "", 2024-04-16T02:35:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:35:53.660 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:35:53.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc5d006d850 msgr2=0x7fc5d006fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:53.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc5d006d850 0x7fc5d006fd10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8005950 tx=0x7fc5d80058e0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 msgr2=0x7fc5e41985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:35:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 0x7fc5e41985a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000bda0 tx=0x7fc5e000be80 comp rx=0 tx=0).stop 2024-04-16T02:35:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 shutdown_connections 2024-04-16T02:35:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc5d006d850 0x7fc5d006fd10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5e4100c00 0x7fc5e41985a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 --2- 172.21.15.151:0/3817404097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5e4101e00 0x7fc5e4198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:35:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 >> 172.21.15.151:0/3817404097 conn(0x7fc5e40fc180 msgr2=0x7fc5e4103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:35:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 shutdown_connections 2024-04-16T02:35:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:35:53.660+0000 7fc5eb469700 1 -- 172.21.15.151:0/3817404097 wait complete. 2024-04-16T02:35:53.864 INFO:tasks.workunit.client.0.smithi151.stdout:0/435: dread dc/d15/d34/f5d [0,4194304] 0 2024-04-16T02:35:53.864 INFO:tasks.workunit.client.0.smithi151.stdout:7/572: link d0/d2c/l2f d0/d1f/d33/d89/lc7 0 2024-04-16T02:35:53.869 INFO:tasks.workunit.client.0.smithi151.stdout:7/573: symlink d0/d2c/d38/dab/lc8 0 2024-04-16T02:35:53.869 INFO:tasks.workunit.client.0.smithi151.stdout:2/560: dwrite de/f3e [0,4194304] 0 2024-04-16T02:35:53.869 INFO:tasks.workunit.client.0.smithi151.stdout:9/492: dwrite d1/d3/f1e [0,4194304] 0 2024-04-16T02:35:53.870 INFO:tasks.workunit.client.0.smithi151.stdout:0/436: getdents dc/d4c/d72/d76 0 2024-04-16T02:35:53.871 INFO:tasks.workunit.client.0.smithi151.stdout:2/561: stat de/d65/d88/d8f 0 2024-04-16T02:35:53.872 INFO:tasks.workunit.client.0.smithi151.stdout:0/437: chown dc/l68 16115 1 2024-04-16T02:35:53.873 INFO:tasks.workunit.client.0.smithi151.stdout:0/438: truncate dc/d15/f87 893941 0 2024-04-16T02:35:53.875 INFO:tasks.workunit.client.0.smithi151.stdout:7/574: mkdir d0/d2c/d38/d5e/d87/dc9 0 2024-04-16T02:35:53.878 INFO:tasks.workunit.client.0.smithi151.stdout:0/439: rename dc/d15/d34/f35 to dc/d4c/f8e 0 2024-04-16T02:35:53.880 INFO:tasks.workunit.client.0.smithi151.stdout:9/493: creat d1/d3/fa4 x:0 0 0 2024-04-16T02:35:53.881 INFO:tasks.workunit.client.0.smithi151.stdout:7/575: creat d0/d1f/d4c/d82/fca x:0 0 0 2024-04-16T02:35:53.882 INFO:tasks.workunit.client.0.smithi151.stdout:0/440: truncate dc/d15/f23 3715141 0 2024-04-16T02:35:53.884 INFO:tasks.workunit.client.0.smithi151.stdout:9/494: creat d1/d3/d6/d17/d54/fa5 x:0 0 0 2024-04-16T02:35:53.886 INFO:tasks.workunit.client.0.smithi151.stdout:7/576: mknod d0/d2c/d38/ccb 0 2024-04-16T02:35:53.887 INFO:tasks.workunit.client.0.smithi151.stdout:0/441: fsync dc/d15/f58 0 2024-04-16T02:35:53.888 INFO:tasks.workunit.client.0.smithi151.stdout:7/577: write d0/d1f/d46/d4f/f57 [412791,85270] 0 2024-04-16T02:35:53.889 INFO:tasks.workunit.client.0.smithi151.stdout:9/495: mkdir d1/d3/d10/da6 0 2024-04-16T02:35:53.891 INFO:tasks.workunit.client.0.smithi151.stdout:9/496: dread - d1/d3/d10/d7b/f85 zero size 2024-04-16T02:35:53.891 INFO:tasks.workunit.client.0.smithi151.stdout:2/562: dread de/f63 [0,4194304] 0 2024-04-16T02:35:53.893 INFO:tasks.workunit.client.0.smithi151.stdout:0/442: mknod dc/d15/d5c/c8f 0 2024-04-16T02:35:53.894 INFO:tasks.workunit.client.0.smithi151.stdout:7/578: creat d0/d1f/d46/fcc x:0 0 0 2024-04-16T02:35:53.895 INFO:tasks.workunit.client.0.smithi151.stdout:7/579: stat d0/d1f/d99/dc0 0 2024-04-16T02:35:53.896 INFO:tasks.workunit.client.0.smithi151.stdout:9/497: creat d1/d3/d6/d17/d54/d58/fa7 x:0 0 0 2024-04-16T02:35:53.896 INFO:tasks.workunit.client.0.smithi151.stdout:2/563: mknod de/d65/d88/d8f/cb0 0 2024-04-16T02:35:53.899 INFO:tasks.workunit.client.0.smithi151.stdout:7/580: mkdir d0/d27/da6/dcd 0 2024-04-16T02:35:53.899 INFO:tasks.workunit.client.0.smithi151.stdout:9/498: fdatasync d1/d3/d6/f74 0 2024-04-16T02:35:53.902 INFO:tasks.workunit.client.0.smithi151.stdout:2/564: rename de/d15/d1e to de/d11/d42/d7a/d8b/db1 0 2024-04-16T02:35:53.905 INFO:tasks.workunit.client.0.smithi151.stdout:2/565: symlink de/d65/d88/d8f/lb2 0 2024-04-16T02:35:53.906 INFO:tasks.workunit.client.0.smithi151.stdout:2/566: chown de/d11/d42/d7a 127186 1 2024-04-16T02:35:53.907 INFO:tasks.workunit.client.0.smithi151.stdout:2/567: mknod de/d11/cb3 0 2024-04-16T02:35:53.910 INFO:tasks.workunit.client.0.smithi151.stdout:2/568: symlink de/lb4 0 2024-04-16T02:35:53.919 INFO:tasks.workunit.client.0.smithi151.stdout:0/443: write dc/d15/d34/d3b/d71/f74 [4184,124107] 0 2024-04-16T02:35:53.923 INFO:tasks.workunit.client.0.smithi151.stdout:0/444: unlink dc/d15/d5c/c8f 0 2024-04-16T02:35:53.925 INFO:tasks.workunit.client.0.smithi151.stdout:9/499: truncate d1/d3/d6/d17/f31 1359487 0 2024-04-16T02:35:53.926 INFO:tasks.workunit.client.0.smithi151.stdout:0/445: write dc/d15/d34/d4e/f63 [5154934,34898] 0 2024-04-16T02:35:53.927 INFO:tasks.workunit.client.0.smithi151.stdout:9/500: read d1/d3/d6/f13 [817979,110678] 0 2024-04-16T02:35:53.928 INFO:tasks.workunit.client.0.smithi151.stdout:0/446: dread - dc/d3c/f75 zero size 2024-04-16T02:35:53.930 INFO:tasks.workunit.client.0.smithi151.stdout:2/569: mknod de/d11/d42/d6b/cb5 0 2024-04-16T02:35:53.931 INFO:tasks.workunit.client.0.smithi151.stdout:9/501: symlink d1/d3/d6/d17/d54/d9b/d22/la8 0 2024-04-16T02:35:53.935 INFO:tasks.workunit.client.0.smithi151.stdout:0/447: link dc/d15/d5c/f6a dc/d15/f90 0 2024-04-16T02:35:53.938 INFO:tasks.workunit.client.0.smithi151.stdout:0/448: fdatasync dc/d15/f90 0 2024-04-16T02:35:53.944 INFO:tasks.workunit.client.0.smithi151.stdout:2/570: write de/d15/f5b [593427,85153] 0 2024-04-16T02:35:53.946 INFO:tasks.workunit.client.0.smithi151.stdout:9/502: truncate d1/d3/d6/d4f/d6b/f77 3222544 0 2024-04-16T02:35:53.947 INFO:tasks.workunit.client.0.smithi151.stdout:2/571: creat de/d11/fb6 x:0 0 0 2024-04-16T02:35:53.950 INFO:tasks.workunit.client.0.smithi151.stdout:2/572: truncate de/d11/d42/d82/d95/d4b/fa1 726000 0 2024-04-16T02:35:53.951 INFO:tasks.workunit.client.0.smithi151.stdout:9/503: getdents d1/d3/d6/d28 0 2024-04-16T02:35:53.955 INFO:tasks.workunit.client.0.smithi151.stdout:2/573: dread de/d11/d42/d7a/d8b/f74 [0,4194304] 0 2024-04-16T02:35:53.958 INFO:tasks.workunit.client.0.smithi151.stdout:2/574: symlink de/d15/lb7 0 2024-04-16T02:35:53.959 INFO:tasks.workunit.client.0.smithi151.stdout:9/504: dwrite d1/dd/f88 [0,4194304] 0 2024-04-16T02:35:53.963 INFO:tasks.workunit.client.0.smithi151.stdout:9/505: unlink d1/d3/d6/d28/f8f 0 2024-04-16T02:35:53.964 INFO:tasks.workunit.client.0.smithi151.stdout:9/506: fdatasync d1/d3/d6/d17/d44/f6a 0 2024-04-16T02:35:53.966 INFO:tasks.workunit.client.0.smithi151.stdout:9/507: symlink d1/d3/d6/d17/d40/la9 0 2024-04-16T02:35:53.968 INFO:tasks.workunit.client.0.smithi151.stdout:9/508: dread - d1/d3/d6/d28/d39/d43/fa1 zero size 2024-04-16T02:35:53.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:53 smithi151 ceph-mon[29003]: from='client.24411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:53.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:53 smithi151 ceph-mon[29003]: pgmap v24: 65 pgs: 65 active+clean; 3.2 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 123 MiB/s rd, 345 MiB/s wr, 676 op/s 2024-04-16T02:35:53.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:53 smithi151 ceph-mon[29003]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:53.972 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:53 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2580258582' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:35:53.975 INFO:tasks.workunit.client.0.smithi151.stdout:9/509: rename d1/d3/d6/d17/d40/d7d/d87 to d1/d3/d6/d17/d54/d8d/daa 0 2024-04-16T02:35:53.976 INFO:tasks.workunit.client.0.smithi151.stdout:2/575: dwrite de/d65/f72 [0,4194304] 0 2024-04-16T02:35:53.977 INFO:tasks.workunit.client.0.smithi151.stdout:9/510: write d1/dd/f88 [3737557,30298] 0 2024-04-16T02:35:53.984 INFO:tasks.workunit.client.0.smithi151.stdout:2/576: dread de/f3e [0,4194304] 0 2024-04-16T02:35:53.986 INFO:tasks.workunit.client.0.smithi151.stdout:9/511: rename d1/d3/d6/d28/d39/d43 to d1/d3/d6/d17/d40/d56/dab 0 2024-04-16T02:35:53.987 INFO:tasks.workunit.client.0.smithi151.stdout:9/512: dread - d1/d3/d6/d17/d54/d8d/f90 zero size 2024-04-16T02:35:53.988 INFO:tasks.workunit.client.0.smithi151.stdout:2/577: mkdir de/d11/d42/d5e/d6c/db8 0 2024-04-16T02:35:53.991 INFO:tasks.workunit.client.0.smithi151.stdout:2/578: symlink de/d65/d88/d8f/lb9 0 2024-04-16T02:35:53.993 INFO:tasks.workunit.client.0.smithi151.stdout:2/579: mkdir de/d65/daa/dba 0 2024-04-16T02:35:53.995 INFO:tasks.workunit.client.0.smithi151.stdout:9/513: dwrite d1/d3/d6/d17/d40/d56/f61 [0,4194304] 0 2024-04-16T02:35:53.997 INFO:tasks.workunit.client.0.smithi151.stdout:9/514: fsync d1/d3/d6/d17/d40/d56/dab/fa3 0 2024-04-16T02:35:53.997 INFO:tasks.workunit.client.0.smithi151.stdout:2/580: creat de/d11/d42/d7a/fbb x:0 0 0 2024-04-16T02:35:54.001 INFO:tasks.workunit.client.0.smithi151.stdout:9/515: mknod d1/d3/d10/d7b/cac 0 2024-04-16T02:35:54.005 INFO:tasks.workunit.client.0.smithi151.stdout:2/581: dwrite de/d65/d71/f7f [0,4194304] 0 2024-04-16T02:35:54.008 INFO:tasks.workunit.client.0.smithi151.stdout:2/582: symlink de/d11/d42/d82/d95/d4b/d60/lbc 0 2024-04-16T02:35:54.012 INFO:tasks.workunit.client.0.smithi151.stdout:2/583: creat de/d11/d42/d82/d95/d4b/d97/fbd x:0 0 0 2024-04-16T02:35:54.014 INFO:tasks.workunit.client.0.smithi151.stdout:2/584: readlink de/d3d/d6a/l91 0 2024-04-16T02:35:54.018 INFO:tasks.workunit.client.0.smithi151.stdout:2/585: dread de/d11/d42/d82/d95/d4b/f73 [0,4194304] 0 2024-04-16T02:35:54.020 INFO:tasks.workunit.client.0.smithi151.stdout:2/586: creat de/d65/fbe x:0 0 0 2024-04-16T02:35:54.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:53 smithi153 ceph-mon[36747]: from='client.24411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:54.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:53 smithi153 ceph-mon[36747]: pgmap v24: 65 pgs: 65 active+clean; 3.2 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 123 MiB/s rd, 345 MiB/s wr, 676 op/s 2024-04-16T02:35:54.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:53 smithi153 ceph-mon[36747]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:54.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:53 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2580258582' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:35:54.026 INFO:tasks.workunit.client.0.smithi151.stdout:2/587: dwrite de/d15/f34 [4194304,4194304] 0 2024-04-16T02:35:54.029 INFO:tasks.workunit.client.0.smithi151.stdout:2/588: symlink de/d65/d71/lbf 0 2024-04-16T02:35:54.041 INFO:tasks.workunit.client.0.smithi151.stdout:2/589: dwrite de/d11/d42/d7a/f85 [0,4194304] 0 2024-04-16T02:35:54.043 INFO:tasks.workunit.client.0.smithi151.stdout:2/590: write de/d65/f6f [1747723,85553] 0 2024-04-16T02:35:54.044 INFO:tasks.workunit.client.0.smithi151.stdout:2/591: readlink de/d11/d42/d7a/d8b/l4f 0 2024-04-16T02:35:54.047 INFO:tasks.workunit.client.0.smithi151.stdout:2/592: mkdir de/d65/dc0 0 2024-04-16T02:35:54.053 INFO:tasks.workunit.client.0.smithi151.stdout:2/593: write de/f14 [3087628,52508] 0 2024-04-16T02:35:54.057 INFO:tasks.workunit.client.0.smithi151.stdout:2/594: link de/d11/d42/d5e/d6c/f77 de/d15/d78/fc1 0 2024-04-16T02:35:54.062 INFO:tasks.workunit.client.0.smithi151.stdout:2/595: write de/d11/d42/d7a/d8b/db1/f96 [157930,45671] 0 2024-04-16T02:35:54.064 INFO:tasks.workunit.client.0.smithi151.stdout:2/596: rmdir de/d11/d42/d82/d95/d4b 39 2024-04-16T02:35:54.067 INFO:tasks.workunit.client.0.smithi151.stdout:2/597: rename de/d65/d88/d8f/lb2 to de/d11/d42/d5e/d6c/d6d/da5/lc2 0 2024-04-16T02:35:54.068 INFO:tasks.workunit.client.0.smithi151.stdout:2/598: fdatasync de/d15/f5b 0 2024-04-16T02:35:54.071 INFO:tasks.workunit.client.0.smithi151.stdout:2/599: unlink de/d65/d71/c9c 0 2024-04-16T02:35:54.073 INFO:tasks.workunit.client.0.smithi151.stdout:2/600: symlink de/d11/d42/d5e/d6c/d6d/lc3 0 2024-04-16T02:35:54.074 INFO:tasks.workunit.client.0.smithi151.stdout:2/601: mkdir de/d65/d71/d7c/dc4 0 2024-04-16T02:35:54.077 INFO:tasks.workunit.client.0.smithi151.stdout:2/602: rename de/d15/lb7 to de/d11/d42/d5e/d6c/db8/lc5 0 2024-04-16T02:35:54.085 INFO:tasks.workunit.client.0.smithi151.stdout:2/603: truncate de/d65/d71/d7c/f9d 319518 0 2024-04-16T02:35:54.087 INFO:tasks.workunit.client.0.smithi151.stdout:2/604: creat de/d11/d42/d7a/d8b/db1/fc6 x:0 0 0 2024-04-16T02:35:54.089 INFO:tasks.workunit.client.0.smithi151.stdout:2/605: truncate de/d15/f9a 204731 0 2024-04-16T02:35:54.096 INFO:tasks.workunit.client.0.smithi151.stdout:2/606: getdents de/d3d 0 2024-04-16T02:35:54.098 INFO:tasks.workunit.client.0.smithi151.stdout:2/607: mkdir de/d65/d71/d7c/dc7 0 2024-04-16T02:35:54.100 INFO:tasks.workunit.client.0.smithi151.stdout:2/608: fdatasync de/d15/f50 0 2024-04-16T02:35:54.101 INFO:tasks.workunit.client.0.smithi151.stdout:2/609: chown de/d11/d42/d82/d95/f33 449927977 1 2024-04-16T02:35:54.103 INFO:tasks.workunit.client.0.smithi151.stdout:2/610: mknod de/d65/d71/cc8 0 2024-04-16T02:35:54.105 INFO:tasks.workunit.client.0.smithi151.stdout:2/611: dread - de/d11/d42/d5e/f8d zero size 2024-04-16T02:35:54.112 INFO:tasks.workunit.client.0.smithi151.stdout:2/612: dwrite de/d15/d78/fae [0,4194304] 0 2024-04-16T02:35:54.113 INFO:tasks.workunit.client.0.smithi151.stdout:2/613: chown de/d15/l9e 16760459 1 2024-04-16T02:35:54.117 INFO:tasks.workunit.client.0.smithi151.stdout:2/614: creat de/d65/d71/d7c/fc9 x:0 0 0 2024-04-16T02:35:54.120 INFO:tasks.workunit.client.0.smithi151.stdout:2/615: fdatasync de/d11/d42/d82/d95/d4b/f73 0 2024-04-16T02:35:54.122 INFO:tasks.workunit.client.0.smithi151.stdout:2/616: symlink de/d15/lca 0 2024-04-16T02:35:54.133 INFO:tasks.workunit.client.0.smithi151.stdout:2/617: dwrite de/d11/d42/d7a/d8b/db1/f24 [0,4194304] 0 2024-04-16T02:35:54.139 INFO:tasks.workunit.client.0.smithi151.stdout:2/618: link de/d3d/c64 de/d65/daa/dba/ccb 0 2024-04-16T02:35:54.144 INFO:tasks.workunit.client.0.smithi151.stdout:2/619: write de/d11/d42/d5e/f7b [891607,20553] 0 2024-04-16T02:35:54.146 INFO:tasks.workunit.client.0.smithi151.stdout:2/620: unlink de/fab 0 2024-04-16T02:35:54.151 INFO:tasks.workunit.client.0.smithi151.stdout:2/621: rename de/d11/d42/d82/d95/d4b/d97 to de/d11/d42/d5e/d6c/db8/dcc 0 2024-04-16T02:35:54.153 INFO:tasks.workunit.client.0.smithi151.stdout:2/622: readlink de/d15/lca 0 2024-04-16T02:35:54.154 INFO:tasks.workunit.client.0.smithi151.stdout:2/623: mkdir de/d15/dcd 0 2024-04-16T02:35:54.162 INFO:tasks.workunit.client.0.smithi151.stdout:2/624: dwrite de/d11/d42/d7a/d8b/f59 [0,4194304] 0 2024-04-16T02:35:54.166 INFO:tasks.workunit.client.0.smithi151.stdout:2/625: mkdir de/d65/d88/d8f/dce 0 2024-04-16T02:35:54.170 INFO:tasks.workunit.client.0.smithi151.stdout:2/626: link de/d11/d42/d82/d95/d4b/f55 de/d11/d42/d82/d95/d4b/d60/fcf 0 2024-04-16T02:35:54.172 INFO:tasks.workunit.client.0.smithi151.stdout:2/627: mkdir de/d15/dcd/dd0 0 2024-04-16T02:35:54.175 INFO:tasks.workunit.client.0.smithi151.stdout:2/628: symlink de/d11/d42/d7a/d8b/db1/d90/ld1 0 2024-04-16T02:35:54.175 INFO:tasks.workunit.client.0.smithi151.stdout:2/629: dread - de/d15/f50 zero size 2024-04-16T02:35:54.177 INFO:tasks.workunit.client.0.smithi151.stdout:2/630: creat de/d65/daa/fd2 x:0 0 0 2024-04-16T02:35:54.183 INFO:tasks.workunit.client.0.smithi151.stdout:2/631: dwrite de/d15/d78/fae [0,4194304] 0 2024-04-16T02:35:54.185 INFO:tasks.workunit.client.0.smithi151.stdout:2/632: write de/d11/d42/d5e/d6c/d6d/da5/fac [532078,35189] 0 2024-04-16T02:35:54.188 INFO:tasks.workunit.client.0.smithi151.stdout:2/633: creat de/d3d/d6a/fd3 x:0 0 0 2024-04-16T02:35:54.188 INFO:tasks.workunit.client.0.smithi151.stdout:2/634: readlink de/d65/d88/d8f/lb9 0 2024-04-16T02:35:54.190 INFO:tasks.workunit.client.0.smithi151.stdout:2/635: chown de/d11/d42/d7a/d8b/l7e 4941602 1 2024-04-16T02:35:54.190 INFO:tasks.workunit.client.0.smithi151.stdout:2/636: read - de/d11/d42/d7a/fbb zero size 2024-04-16T02:35:54.195 INFO:tasks.workunit.client.0.smithi151.stdout:2/637: read de/d11/d42/d7a/d8b/db1/f3f [1860024,91706] 0 2024-04-16T02:35:54.198 INFO:tasks.workunit.client.0.smithi151.stdout:2/638: creat de/d11/d42/d5e/d6c/fd4 x:0 0 0 2024-04-16T02:35:54.203 INFO:tasks.workunit.client.0.smithi151.stdout:2/639: dwrite de/d65/f72 [0,4194304] 0 2024-04-16T02:35:54.207 INFO:tasks.workunit.client.0.smithi151.stdout:2/640: dread de/d15/f5b [0,4194304] 0 2024-04-16T02:35:54.209 INFO:tasks.workunit.client.0.smithi151.stdout:2/641: write de/d65/daa/fd2 [471179,117424] 0 2024-04-16T02:35:54.212 INFO:tasks.workunit.client.0.smithi151.stdout:2/642: mknod de/d11/d42/d82/d95/d4b/cd5 0 2024-04-16T02:35:54.214 INFO:tasks.workunit.client.0.smithi151.stdout:2/643: symlink de/d11/d42/d5e/d6c/da2/ld6 0 2024-04-16T02:35:54.214 INFO:tasks.workunit.client.0.smithi151.stdout:2/644: stat de/d15/f50 0 2024-04-16T02:35:54.214 INFO:tasks.workunit.client.0.smithi151.stdout:2/645: chown de/d11/cb3 189546943 1 2024-04-16T02:35:54.217 INFO:tasks.workunit.client.0.smithi151.stdout:2/646: fdatasync de/d15/f17 0 2024-04-16T02:35:54.221 INFO:tasks.workunit.client.0.smithi151.stdout:2/647: getdents de/d11/d42/d7a 0 2024-04-16T02:35:54.262 INFO:tasks.workunit.client.0.smithi151.stdout:2/648: dwrite de/d11/d42/d7a/d8b/f59 [0,4194304] 0 2024-04-16T02:35:54.263 INFO:tasks.workunit.client.0.smithi151.stdout:2/649: stat de/d65/d71/d7c/fc9 0 2024-04-16T02:35:54.266 INFO:tasks.workunit.client.0.smithi151.stdout:2/650: creat de/d11/d42/fd7 x:0 0 0 2024-04-16T02:35:54.269 INFO:tasks.workunit.client.0.smithi151.stdout:2/651: dread de/f3e [0,4194304] 0 2024-04-16T02:35:54.271 INFO:tasks.workunit.client.0.smithi151.stdout:2/652: creat de/d11/d42/d82/fd8 x:0 0 0 2024-04-16T02:35:54.273 INFO:tasks.workunit.client.0.smithi151.stdout:2/653: mknod de/d11/d42/d6b/cd9 0 2024-04-16T02:35:54.526 INFO:tasks.workunit.client.0.smithi151.stdout:2/654: truncate de/d65/d71/f7f 894393 0 2024-04-16T02:35:54.527 INFO:tasks.workunit.client.0.smithi151.stdout:2/655: chown de/d11/d42/d5e/d6c/db8 6517441 1 2024-04-16T02:35:54.528 INFO:tasks.workunit.client.0.smithi151.stdout:2/656: mknod de/d65/d88/cda 0 2024-04-16T02:35:54.529 INFO:tasks.workunit.client.0.smithi151.stdout:2/657: write de/d15/f34 [1025939,47297] 0 2024-04-16T02:35:54.530 INFO:tasks.workunit.client.0.smithi151.stdout:2/658: dread - de/d15/fa8 zero size 2024-04-16T02:35:54.532 INFO:tasks.workunit.client.0.smithi151.stdout:2/659: symlink de/d11/d42/d82/d95/d4b/ldb 0 2024-04-16T02:35:54.535 INFO:tasks.workunit.client.0.smithi151.stdout:2/660: unlink de/d11/d42/d7a/d8b/db1/f96 0 2024-04-16T02:35:54.537 INFO:tasks.workunit.client.0.smithi151.stdout:2/661: creat de/d15/d78/fdc x:0 0 0 2024-04-16T02:35:54.540 INFO:tasks.workunit.client.0.smithi151.stdout:2/662: link de/d11/d42/d7a/d8b/db1/l5f de/d11/d42/d5e/ldd 0 2024-04-16T02:35:54.545 INFO:tasks.workunit.client.0.smithi151.stdout:2/663: dwrite de/d11/d42/d82/fa0 [0,4194304] 0 2024-04-16T02:35:54.547 INFO:tasks.workunit.client.0.smithi151.stdout:2/664: creat de/d11/d42/d7a/fde x:0 0 0 2024-04-16T02:35:54.549 INFO:tasks.workunit.client.0.smithi151.stdout:2/665: dread - de/d11/d42/d7a/d8b/db1/fc6 zero size 2024-04-16T02:35:54.551 INFO:tasks.workunit.client.0.smithi151.stdout:2/666: chown de/d11/d42/d7a/d8b/db1/f24 10 1 2024-04-16T02:35:54.553 INFO:tasks.workunit.client.0.smithi151.stdout:2/667: mknod de/d11/d42/d5e/d6c/db8/cdf 0 2024-04-16T02:35:54.555 INFO:tasks.workunit.client.0.smithi151.stdout:2/668: symlink de/d11/d42/d7a/le0 0 2024-04-16T02:35:54.558 INFO:tasks.workunit.client.0.smithi151.stdout:2/669: stat de/d11/d42/d5e/d6c/db8/lc5 0 2024-04-16T02:35:54.559 INFO:tasks.workunit.client.0.smithi151.stdout:2/670: rmdir de/d3d 39 2024-04-16T02:35:54.568 INFO:tasks.workunit.client.0.smithi151.stdout:2/671: dread de/d11/d42/d7a/d8b/db1/f67 [0,4194304] 0 2024-04-16T02:35:54.600 INFO:tasks.workunit.client.0.smithi151.stdout:2/672: creat de/d11/d42/d5e/fe1 x:0 0 0 2024-04-16T02:35:54.601 INFO:tasks.workunit.client.0.smithi151.stdout:2/673: readlink de/d11/d42/d7a/d8b/db1/l61 0 2024-04-16T02:35:54.602 INFO:tasks.workunit.client.0.smithi151.stdout:2/674: chown de/d65/d71/lbf 55260 1 2024-04-16T02:35:54.606 INFO:tasks.workunit.client.0.smithi151.stdout:2/675: link de/d65/d71/d7c/l81 de/d11/d42/d5e/d6c/da2/le2 0 2024-04-16T02:35:54.606 INFO:tasks.workunit.client.0.smithi151.stdout:2/676: stat de/d11/d42/d6b/cb5 0 2024-04-16T02:35:54.686 INFO:tasks.workunit.client.0.smithi151.stdout:2/677: dwrite de/d65/d71/f7f [0,4194304] 0 2024-04-16T02:35:54.689 INFO:tasks.workunit.client.0.smithi151.stdout:2/678: symlink de/d65/le3 0 2024-04-16T02:35:54.693 INFO:tasks.workunit.client.0.smithi151.stdout:2/679: rename de/d11/d42/c5d to de/d11/d42/d5e/d6c/da2/ce4 0 2024-04-16T02:35:54.696 INFO:tasks.workunit.client.0.smithi151.stdout:2/680: creat de/d65/dc0/fe5 x:0 0 0 2024-04-16T02:35:54.696 INFO:tasks.workunit.client.0.smithi151.stdout:2/681: dread - de/d15/fa8 zero size 2024-04-16T02:35:54.697 INFO:tasks.workunit.client.0.smithi151.stdout:2/682: write de/d15/fa8 [683597,72057] 0 2024-04-16T02:35:54.698 INFO:tasks.workunit.client.0.smithi151.stdout:2/683: fdatasync de/d11/d42/d7a/d8b/f74 0 2024-04-16T02:35:54.702 INFO:tasks.workunit.client.0.smithi151.stdout:2/684: readlink de/d3d/d6a/l91 0 2024-04-16T02:35:54.702 INFO:tasks.workunit.client.0.smithi151.stdout:2/685: truncate de/d11/d42/d82/fa0 5089668 0 2024-04-16T02:35:54.704 INFO:tasks.workunit.client.0.smithi151.stdout:2/686: rename de/d11/d42/d5e/d6c/db8/cdf to de/d11/d42/d5e/d6c/da2/ce6 0 2024-04-16T02:35:54.705 INFO:tasks.workunit.client.0.smithi151.stdout:2/687: chown de/d11/d42/d6b/cd9 14368 1 2024-04-16T02:35:54.707 INFO:tasks.workunit.client.0.smithi151.stdout:2/688: read de/d15/d78/fc1 [3094544,26667] 0 2024-04-16T02:35:54.713 INFO:tasks.workunit.client.0.smithi151.stdout:2/689: dwrite de/d11/d42/d7a/fbb [0,4194304] 0 2024-04-16T02:35:54.715 INFO:tasks.workunit.client.0.smithi151.stdout:2/690: chown de/c76 1 1 2024-04-16T02:35:54.720 INFO:tasks.workunit.client.0.smithi151.stdout:2/691: read de/d11/d42/d7a/d8b/db1/f3c [4087979,77971] 0 2024-04-16T02:35:54.722 INFO:tasks.workunit.client.0.smithi151.stdout:2/692: symlink de/d15/d78/le7 0 2024-04-16T02:35:54.723 INFO:tasks.workunit.client.0.smithi151.stdout:2/693: truncate de/d65/daa/fd2 1486770 0 2024-04-16T02:35:54.724 INFO:tasks.workunit.client.0.smithi151.stdout:2/694: fsync de/d11/d42/d7a/d8b/db1/fc6 0 2024-04-16T02:35:54.726 INFO:tasks.workunit.client.0.smithi151.stdout:2/695: dread de/d15/f5b [0,4194304] 0 2024-04-16T02:35:54.727 INFO:tasks.workunit.client.0.smithi151.stdout:2/696: write de/d11/d42/d5e/d6c/d6d/da5/fac [1512334,43548] 0 2024-04-16T02:35:54.729 INFO:tasks.workunit.client.0.smithi151.stdout:2/697: mkdir de/d3d/de8 0 2024-04-16T02:35:54.732 INFO:tasks.workunit.client.0.smithi151.stdout:2/698: dread de/d11/d42/d7a/d8b/db1/f67 [0,4194304] 0 2024-04-16T02:35:54.733 INFO:tasks.workunit.client.0.smithi151.stdout:2/699: readlink de/d11/d42/d7a/d8b/l69 0 2024-04-16T02:35:54.734 INFO:tasks.workunit.client.0.smithi151.stdout:2/700: write de/d11/d42/d5e/fe1 [277859,107311] 0 2024-04-16T02:35:54.736 INFO:tasks.workunit.client.0.smithi151.stdout:2/701: mknod de/d3d/de8/ce9 0 2024-04-16T02:35:54.748 INFO:tasks.workunit.client.0.smithi151.stdout:2/702: dwrite de/d11/d42/d5e/d6c/d6d/f92 [0,4194304] 0 2024-04-16T02:35:54.750 INFO:tasks.workunit.client.0.smithi151.stdout:2/703: read de/d15/f17 [375831,32984] 0 2024-04-16T02:35:54.758 INFO:tasks.workunit.client.0.smithi151.stdout:2/704: dwrite de/d11/d42/d7a/d8b/db1/fc6 [0,4194304] 0 2024-04-16T02:35:54.769 INFO:tasks.workunit.client.0.smithi151.stdout:2/705: link de/f27 de/d65/fea 0 2024-04-16T02:35:54.772 INFO:tasks.workunit.client.0.smithi151.stdout:2/706: rmdir de/d11/d42/d5e/d6c/db8 39 2024-04-16T02:35:54.779 INFO:tasks.workunit.client.0.smithi151.stdout:2/707: dwrite de/d11/d42/d5e/fe1 [0,4194304] 0 2024-04-16T02:35:54.781 INFO:tasks.workunit.client.0.smithi151.stdout:2/708: fdatasync de/d11/d42/d82/fa3 0 2024-04-16T02:35:54.783 INFO:tasks.workunit.client.0.smithi151.stdout:2/709: symlink de/d11/d42/d82/d95/d4b/leb 0 2024-04-16T02:35:54.784 INFO:tasks.workunit.client.0.smithi151.stdout:2/710: stat de/d11/d42/d82/d95/d4b/f73 0 2024-04-16T02:35:54.793 INFO:tasks.workunit.client.0.smithi151.stdout:2/711: dwrite de/d65/dc0/fe5 [0,4194304] 0 2024-04-16T02:35:54.794 INFO:tasks.workunit.client.0.smithi151.stdout:2/712: chown de/d11/d42/d82/d95/d4b/cd5 621837 1 2024-04-16T02:35:54.806 INFO:tasks.workunit.client.0.smithi151.stdout:2/713: dwrite de/d11/d42/d7a/d8b/db1/f3c [0,4194304] 0 2024-04-16T02:35:54.810 INFO:tasks.workunit.client.0.smithi151.stdout:2/714: creat de/d11/d42/d5e/d6c/d6d/fec x:0 0 0 2024-04-16T02:35:54.858 INFO:tasks.workunit.client.0.smithi151.stdout:2/715: write de/d11/d42/d82/d95/f32 [806517,112960] 0 2024-04-16T02:35:54.858 INFO:tasks.workunit.client.0.smithi151.stdout:2/716: chown de/c76 126924 1 2024-04-16T02:35:54.884 INFO:tasks.workunit.client.0.smithi151.stdout:2/717: mknod de/d15/ced 0 2024-04-16T02:35:54.885 INFO:tasks.workunit.client.0.smithi151.stdout:2/718: chown de/d11/d42/d7a/f85 64610906 1 2024-04-16T02:35:54.889 INFO:tasks.workunit.client.0.smithi151.stdout:2/719: getdents de/d11/d42/d7a 0 2024-04-16T02:35:54.893 INFO:tasks.workunit.client.0.smithi151.stdout:2/720: dread de/d11/d42/d7a/f85 [0,4194304] 0 2024-04-16T02:35:54.894 INFO:tasks.workunit.client.0.smithi151.stdout:2/721: fdatasync de/d11/fb6 0 2024-04-16T02:35:54.897 INFO:tasks.workunit.client.0.smithi151.stdout:2/722: rename de/d11/c30 to de/d11/d42/d7a/d8b/db1/d90/cee 0 2024-04-16T02:35:54.908 INFO:tasks.workunit.client.0.smithi151.stdout:2/723: dwrite de/d11/d42/d82/d95/f33 [4194304,4194304] 0 2024-04-16T02:35:54.909 INFO:tasks.workunit.client.0.smithi151.stdout:2/724: stat de/d11/l62 0 2024-04-16T02:35:54.910 INFO:tasks.workunit.client.0.smithi151.stdout:2/725: readlink de/d65/d88/d8f/lb9 0 2024-04-16T02:35:54.912 INFO:tasks.workunit.client.0.smithi151.stdout:2/726: creat de/d65/dc0/fef x:0 0 0 2024-04-16T02:35:54.919 INFO:tasks.workunit.client.0.smithi151.stdout:2/727: dwrite de/d11/d42/d7a/f85 [0,4194304] 0 2024-04-16T02:35:54.922 INFO:tasks.workunit.client.0.smithi151.stdout:2/728: mknod de/d65/dc0/cf0 0 2024-04-16T02:35:54.947 INFO:tasks.workunit.client.0.smithi151.stdout:2/729: dread de/d11/d42/d82/d95/f33 [0,4194304] 0 2024-04-16T02:35:54.951 INFO:tasks.workunit.client.0.smithi151.stdout:2/730: write de/d11/d42/d82/fa3 [391616,115775] 0 2024-04-16T02:35:54.954 INFO:tasks.workunit.client.0.smithi151.stdout:2/731: mkdir de/d65/daa/dba/df1 0 2024-04-16T02:35:54.954 INFO:tasks.workunit.client.0.smithi151.stdout:2/732: chown de/d11/c44 2531 1 2024-04-16T02:35:54.956 INFO:tasks.workunit.client.0.smithi151.stdout:2/733: write de/d11/d42/d82/d95/f32 [1088098,25183] 0 2024-04-16T02:35:54.958 INFO:tasks.workunit.client.0.smithi151.stdout:2/734: fdatasync de/d11/f7d 0 2024-04-16T02:35:54.960 INFO:tasks.workunit.client.0.smithi151.stdout:2/735: write de/d11/d42/d7a/d8b/db1/f24 [2543319,31293] 0 2024-04-16T02:35:54.966 INFO:tasks.workunit.client.0.smithi151.stdout:2/736: dwrite de/d15/d78/fdc [0,4194304] 0 2024-04-16T02:35:54.973 INFO:tasks.workunit.client.0.smithi151.stdout:2/737: dwrite de/d11/d42/d7a/d8b/f74 [0,4194304] 0 2024-04-16T02:35:54.977 INFO:tasks.workunit.client.0.smithi151.stdout:2/738: mkdir de/df2 0 2024-04-16T02:35:54.979 INFO:tasks.workunit.client.0.smithi151.stdout:2/739: mknod de/d11/d42/d6b/cf3 0 2024-04-16T02:35:54.981 INFO:tasks.workunit.client.0.smithi151.stdout:2/740: readlink de/d11/d42/d82/d95/d4b/d60/lad 0 2024-04-16T02:35:54.985 INFO:tasks.workunit.client.0.smithi151.stdout:2/741: dread de/d11/d42/d7a/d8b/db1/f67 [0,4194304] 0 2024-04-16T02:35:54.989 INFO:tasks.workunit.client.0.smithi151.stdout:2/742: rename de/d15/ced to de/d65/daa/dba/df1/cf4 0 2024-04-16T02:35:54.991 INFO:tasks.workunit.client.0.smithi151.stdout:2/743: symlink de/d11/d42/d82/d95/d4b/lf5 0 2024-04-16T02:35:54.994 INFO:tasks.workunit.client.0.smithi151.stdout:2/744: creat de/ff6 x:0 0 0 2024-04-16T02:35:54.996 INFO:tasks.workunit.client.0.smithi151.stdout:2/745: creat de/d11/d42/d82/ff7 x:0 0 0 2024-04-16T02:35:55.002 INFO:tasks.workunit.client.0.smithi151.stdout:2/746: dwrite de/d65/d71/d7c/fc9 [0,4194304] 0 2024-04-16T02:35:55.010 INFO:tasks.workunit.client.0.smithi151.stdout:2/747: dwrite de/ff6 [0,4194304] 0 2024-04-16T02:35:55.016 INFO:tasks.workunit.client.0.smithi151.stdout:2/748: dwrite de/ff6 [0,4194304] 0 2024-04-16T02:35:55.021 INFO:tasks.workunit.client.0.smithi151.stdout:2/749: getdents de/d15/d78 0 2024-04-16T02:35:55.024 INFO:tasks.workunit.client.0.smithi151.stdout:2/750: mknod de/d15/dcd/dd0/cf8 0 2024-04-16T02:35:55.027 INFO:tasks.workunit.client.0.smithi151.stdout:2/751: unlink de/d11/d42/d7a/le0 0 2024-04-16T02:35:55.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:54 smithi151 ceph-mon[29003]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:54 smithi153 ceph-mon[36747]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:35:56.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:55 smithi151 ceph-mon[29003]: pgmap v25: 65 pgs: 65 active+clean; 3.2 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 83 MiB/s rd, 216 MiB/s wr, 434 op/s 2024-04-16T02:35:56.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:55 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:35:56.251 INFO:tasks.workunit.client.0.smithi151.stdout:1/468: sync 2024-04-16T02:35:56.254 INFO:tasks.workunit.client.0.smithi151.stdout:1/469: getdents d3/d4/d64 0 2024-04-16T02:35:56.261 INFO:tasks.workunit.client.0.smithi151.stdout:1/470: link d3/d58/df/d32/d3a/d6d/f7b d3/d4/d64/d2a/d52/fa9 0 2024-04-16T02:35:56.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:55 smithi153 ceph-mon[36747]: pgmap v25: 65 pgs: 65 active+clean; 3.2 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 83 MiB/s rd, 216 MiB/s wr, 434 op/s 2024-04-16T02:35:56.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:55 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:35:56.277 INFO:tasks.workunit.client.0.smithi151.stdout:1/471: write d3/d58/df/d32/d3a/d71/f75 [997804,78579] 0 2024-04-16T02:35:56.280 INFO:tasks.workunit.client.0.smithi151.stdout:1/472: write d3/d4/d64/d2a/f8d [1806905,7422] 0 2024-04-16T02:35:56.282 INFO:tasks.workunit.client.0.smithi151.stdout:1/473: truncate d3/d33/d10/f55 2703200 0 2024-04-16T02:35:56.285 INFO:tasks.workunit.client.0.smithi151.stdout:1/474: truncate d3/d4/d64/d2a/d7a/f86 97042 0 2024-04-16T02:35:56.288 INFO:tasks.workunit.client.0.smithi151.stdout:1/475: creat d3/d4/faa x:0 0 0 2024-04-16T02:35:56.289 INFO:tasks.workunit.client.0.smithi151.stdout:1/476: chown d3/d1f/d48/l95 58731171 1 2024-04-16T02:35:56.302 INFO:tasks.workunit.client.0.smithi151.stdout:1/477: dwrite d3/d4/d64/d2a/d52/fa9 [0,4194304] 0 2024-04-16T02:35:56.305 INFO:tasks.workunit.client.0.smithi151.stdout:1/478: dread - d3/f9d zero size 2024-04-16T02:35:56.307 INFO:tasks.workunit.client.0.smithi151.stdout:1/479: fsync d3/d58/df/d32/d3a/f79 0 2024-04-16T02:35:56.311 INFO:tasks.workunit.client.0.smithi151.stdout:5/603: sync 2024-04-16T02:35:56.312 INFO:tasks.workunit.client.0.smithi151.stdout:3/329: sync 2024-04-16T02:35:56.314 INFO:tasks.workunit.client.0.smithi151.stdout:5/604: mkdir d2/d6/d19/d30/d36/d63/d9c/db9 0 2024-04-16T02:35:56.321 INFO:tasks.workunit.client.0.smithi151.stdout:5/605: dwrite d2/d6/d19/d1a/d4c/f53 [0,4194304] 0 2024-04-16T02:35:56.326 INFO:tasks.workunit.client.0.smithi151.stdout:5/606: write d2/d6/fa [2308567,98077] 0 2024-04-16T02:35:56.328 INFO:tasks.workunit.client.0.smithi151.stdout:5/607: chown d2/d6/d19/d1a/f47 884652 1 2024-04-16T02:35:56.331 INFO:tasks.workunit.client.0.smithi151.stdout:5/608: dread - d2/d6/d19/d30/d49/f5c zero size 2024-04-16T02:35:56.332 INFO:tasks.workunit.client.0.smithi151.stdout:5/609: chown d2/d6/d19/d30/f44 93 1 2024-04-16T02:35:56.333 INFO:tasks.workunit.client.0.smithi151.stdout:5/610: stat d2/d6/d19/d30/f77 0 2024-04-16T02:35:56.336 INFO:tasks.workunit.client.0.smithi151.stdout:5/611: creat d2/d6/d19/d1a/d4c/fba x:0 0 0 2024-04-16T02:35:56.346 INFO:tasks.workunit.client.0.smithi151.stdout:1/480: dread d3/d33/d10/f4e [4194304,4194304] 0 2024-04-16T02:35:56.372 INFO:tasks.workunit.client.0.smithi151.stdout:5/612: dread d2/d6/d19/d30/d36/d42/f71 [0,4194304] 0 2024-04-16T02:35:56.374 INFO:tasks.workunit.client.0.smithi151.stdout:1/481: truncate d3/d4/d64/d2a/f60 155472 0 2024-04-16T02:35:56.379 INFO:tasks.workunit.client.0.smithi151.stdout:5/613: write d2/d6/d19/f5f [868620,118908] 0 2024-04-16T02:35:56.381 INFO:tasks.workunit.client.0.smithi151.stdout:1/482: write d3/d4/d64/d2a/f60 [28472,111746] 0 2024-04-16T02:35:56.384 INFO:tasks.workunit.client.0.smithi151.stdout:5/614: write d2/d6/f50 [2884153,15081] 0 2024-04-16T02:35:56.387 INFO:tasks.workunit.client.0.smithi151.stdout:5/615: read d2/d6/d19/d1a/f41 [2359979,65794] 0 2024-04-16T02:35:56.391 INFO:tasks.workunit.client.0.smithi151.stdout:1/483: dwrite d3/d4/d64/d2a/f65 [0,4194304] 0 2024-04-16T02:35:56.396 INFO:tasks.workunit.client.0.smithi151.stdout:5/616: dwrite d2/d6/d43/f6d [0,4194304] 0 2024-04-16T02:35:56.403 INFO:tasks.workunit.client.0.smithi151.stdout:1/484: dwrite d3/d33/d10/f78 [0,4194304] 0 2024-04-16T02:35:56.405 INFO:tasks.workunit.client.0.smithi151.stdout:1/485: stat d3/d4/d64/d2a/d52/f72 0 2024-04-16T02:35:56.406 INFO:tasks.workunit.client.0.smithi151.stdout:1/486: chown d3/d33/f39 8663 1 2024-04-16T02:35:56.408 INFO:tasks.workunit.client.0.smithi151.stdout:5/617: dread d2/f8e [0,4194304] 0 2024-04-16T02:35:56.412 INFO:tasks.workunit.client.0.smithi151.stdout:1/487: rename d3/d33/l3d to d3/d4/d64/d2a/d52/lab 0 2024-04-16T02:35:56.414 INFO:tasks.workunit.client.0.smithi151.stdout:1/488: fsync d3/d58/df/d32/d3a/d71/f75 0 2024-04-16T02:35:56.415 INFO:tasks.workunit.client.0.smithi151.stdout:5/618: rename d2/d6/d19/d30/f3a to d2/fbb 0 2024-04-16T02:35:56.417 INFO:tasks.workunit.client.0.smithi151.stdout:1/489: symlink d3/d4/d64/d2a/d52/d97/lac 0 2024-04-16T02:35:56.417 INFO:tasks.workunit.client.0.smithi151.stdout:1/490: dread - d3/d1f/f94 zero size 2024-04-16T02:35:56.418 INFO:tasks.workunit.client.0.smithi151.stdout:5/619: creat d2/d6/d19/d62/fbc x:0 0 0 2024-04-16T02:35:56.419 INFO:tasks.workunit.client.0.smithi151.stdout:5/620: chown d2/d6/d43/d9d/cb3 2594 1 2024-04-16T02:35:56.420 INFO:tasks.workunit.client.0.smithi151.stdout:1/491: rmdir d3/d58/df/d32/d3a 39 2024-04-16T02:35:56.420 INFO:tasks.workunit.client.0.smithi151.stdout:5/621: chown d2/d6/c46 867911 1 2024-04-16T02:35:56.422 INFO:tasks.workunit.client.0.smithi151.stdout:5/622: rmdir d2/d6/d19/d30/d36/d42 39 2024-04-16T02:35:56.424 INFO:tasks.workunit.client.0.smithi151.stdout:1/492: truncate d3/d58/df/d32/f37 830115 0 2024-04-16T02:35:56.426 INFO:tasks.workunit.client.0.smithi151.stdout:5/623: mkdir d2/d6/d19/d62/dbd 0 2024-04-16T02:35:56.430 INFO:tasks.workunit.client.0.smithi151.stdout:5/624: dread d2/d6/d19/d30/d36/d42/d6f/d83/fa0 [0,4194304] 0 2024-04-16T02:35:56.433 INFO:tasks.workunit.client.0.smithi151.stdout:5/625: creat d2/d6/d19/d93/fbe x:0 0 0 2024-04-16T02:35:56.436 INFO:tasks.workunit.client.0.smithi151.stdout:1/493: dwrite d3/d4/d64/d2a/d52/f6a [4194304,4194304] 0 2024-04-16T02:35:56.437 INFO:tasks.workunit.client.0.smithi151.stdout:5/626: unlink d2/d6/d19/d30/d36/c60 0 2024-04-16T02:35:56.441 INFO:tasks.workunit.client.0.smithi151.stdout:5/627: mknod d2/d6/d19/d1a/cbf 0 2024-04-16T02:35:56.446 INFO:tasks.workunit.client.0.smithi151.stdout:5/628: dread d2/d5e/f72 [0,4194304] 0 2024-04-16T02:35:56.449 INFO:tasks.workunit.client.0.smithi151.stdout:1/494: dwrite d3/f8a [4194304,4194304] 0 2024-04-16T02:35:56.451 INFO:tasks.workunit.client.0.smithi151.stdout:1/495: symlink d3/d58/lad 0 2024-04-16T02:35:56.455 INFO:tasks.workunit.client.0.smithi151.stdout:1/496: getdents d3/d4/d64/d2a/d73 0 2024-04-16T02:35:56.457 INFO:tasks.workunit.client.0.smithi151.stdout:1/497: truncate d3/d33/f36 4582212 0 2024-04-16T02:35:56.459 INFO:tasks.workunit.client.0.smithi151.stdout:1/498: symlink d3/d1f/lae 0 2024-04-16T02:35:56.463 INFO:tasks.workunit.client.0.smithi151.stdout:1/499: mknod d3/d58/df/d32/d3a/d71/caf 0 2024-04-16T02:35:56.463 INFO:tasks.workunit.client.0.smithi151.stdout:1/500: readlink d3/d4/d64/d1c/l45 0 2024-04-16T02:35:56.465 INFO:tasks.workunit.client.0.smithi151.stdout:1/501: chown d3/d4/d64/d2a/d52/l84 425 1 2024-04-16T02:35:56.472 INFO:tasks.workunit.client.0.smithi151.stdout:1/502: dwrite d3/d4/d64/d1c/f2c [0,4194304] 0 2024-04-16T02:35:56.473 INFO:tasks.workunit.client.0.smithi151.stdout:5/629: dread d2/f5 [0,4194304] 0 2024-04-16T02:35:56.476 INFO:tasks.workunit.client.0.smithi151.stdout:2/752: sync 2024-04-16T02:35:56.476 INFO:tasks.workunit.client.0.smithi151.stdout:8/319: sync 2024-04-16T02:35:56.476 INFO:tasks.workunit.client.0.smithi151.stdout:6/625: sync 2024-04-16T02:35:56.476 INFO:tasks.workunit.client.0.smithi151.stdout:7/581: sync 2024-04-16T02:35:56.476 INFO:tasks.workunit.client.0.smithi151.stdout:0/449: sync 2024-04-16T02:35:56.477 INFO:tasks.workunit.client.0.smithi151.stdout:4/517: sync 2024-04-16T02:35:56.477 INFO:tasks.workunit.client.0.smithi151.stdout:9/516: sync 2024-04-16T02:35:56.477 INFO:tasks.workunit.client.0.smithi151.stdout:0/450: readlink dc/l5b 0 2024-04-16T02:35:56.480 INFO:tasks.workunit.client.0.smithi151.stdout:0/451: readlink dc/d15/d34/d4e/l6e 0 2024-04-16T02:35:56.481 INFO:tasks.workunit.client.0.smithi151.stdout:2/753: chown de/d11/d42/d7a/d8b/db1/l2f 488309 1 2024-04-16T02:35:56.482 INFO:tasks.workunit.client.0.smithi151.stdout:9/517: chown d1/d3/d6/d17/d44/f6a 61024 1 2024-04-16T02:35:56.486 INFO:tasks.workunit.client.0.smithi151.stdout:5/630: write d2/f1f [72013,124630] 0 2024-04-16T02:35:56.489 INFO:tasks.workunit.client.0.smithi151.stdout:4/518: dread d0/d11/d19/d87/d3a/d65/f7d [0,4194304] 0 2024-04-16T02:35:56.489 INFO:tasks.workunit.client.0.smithi151.stdout:0/452: creat dc/d15/d34/d67/d51/f91 x:0 0 0 2024-04-16T02:35:56.490 INFO:tasks.workunit.client.0.smithi151.stdout:1/503: truncate d3/d33/f36 307334 0 2024-04-16T02:35:56.491 INFO:tasks.workunit.client.0.smithi151.stdout:7/582: dwrite d0/d1f/d4c/d82/fa3 [0,4194304] 0 2024-04-16T02:35:56.491 INFO:tasks.workunit.client.0.smithi151.stdout:9/518: mkdir d1/d3/d6/d17/d40/d56/dad 0 2024-04-16T02:35:56.495 INFO:tasks.workunit.client.0.smithi151.stdout:5/631: mknod d2/d6/d19/d30/d36/d63/d9c/cc0 0 2024-04-16T02:35:56.495 INFO:tasks.workunit.client.0.smithi151.stdout:2/754: symlink de/d65/d71/lf9 0 2024-04-16T02:35:56.497 INFO:tasks.workunit.client.0.smithi151.stdout:0/453: read - dc/d15/d34/d67/f78 zero size 2024-04-16T02:35:56.497 INFO:tasks.workunit.client.0.smithi151.stdout:1/504: creat d3/d33/d10/d4f/fb0 x:0 0 0 2024-04-16T02:35:56.499 INFO:tasks.workunit.client.0.smithi151.stdout:7/583: creat d0/d1f/da8/db2/fce x:0 0 0 2024-04-16T02:35:56.501 INFO:tasks.workunit.client.0.smithi151.stdout:5/632: symlink d2/d6/d19/lc1 0 2024-04-16T02:35:56.503 INFO:tasks.workunit.client.0.smithi151.stdout:8/320: dwrite d11/d22/d2e/f3d [0,4194304] 0 2024-04-16T02:35:56.506 INFO:tasks.workunit.client.0.smithi151.stdout:2/755: creat de/d11/ffa x:0 0 0 2024-04-16T02:35:56.506 INFO:tasks.workunit.client.0.smithi151.stdout:9/519: rename d1/d3/d6/d4f to d1/d3/d6/d17/d54/d9b/dae 0 2024-04-16T02:35:56.512 INFO:tasks.workunit.client.0.smithi151.stdout:5/633: rmdir d2/d6/d19/d30/d36/d42/d6f/d83 39 2024-04-16T02:35:56.513 INFO:tasks.workunit.client.0.smithi151.stdout:5/634: chown d2/d6/f64 7927 1 2024-04-16T02:35:56.514 INFO:tasks.workunit.client.0.smithi151.stdout:4/519: link d0/d64/f71 d0/d11/d19/d31/d45/fb1 0 2024-04-16T02:35:56.516 INFO:tasks.workunit.client.0.smithi151.stdout:1/505: mknod d3/d58/df/d32/d3a/cb1 0 2024-04-16T02:35:56.517 INFO:tasks.workunit.client.0.smithi151.stdout:8/321: rmdir d11/d2a/d35 39 2024-04-16T02:35:56.518 INFO:tasks.workunit.client.0.smithi151.stdout:6/626: dread d2/d5/f5a [0,4194304] 0 2024-04-16T02:35:56.519 INFO:tasks.workunit.client.0.smithi151.stdout:0/454: dwrite dc/d15/f50 [0,4194304] 0 2024-04-16T02:35:56.520 INFO:tasks.workunit.client.0.smithi151.stdout:9/520: mkdir d1/d3/d6/d17/d40/d56/dad/daf 0 2024-04-16T02:35:56.528 INFO:tasks.workunit.client.0.smithi151.stdout:3/330: sync 2024-04-16T02:35:56.548 INFO:tasks.workunit.client.0.smithi151.stdout:6/627: mkdir d2/d13/d5f/dd7 0 2024-04-16T02:35:56.548 INFO:tasks.workunit.client.0.smithi151.stdout:0/455: stat dc/d15/f23 0 2024-04-16T02:35:56.551 INFO:tasks.workunit.client.0.smithi151.stdout:9/521: chown d1/d3/d6/d17/d40/l49 2075279 1 2024-04-16T02:35:56.553 INFO:tasks.workunit.client.0.smithi151.stdout:1/506: mkdir d3/d4/db2 0 2024-04-16T02:35:56.555 INFO:tasks.workunit.client.0.smithi151.stdout:3/331: creat d1/d6/d4a/f6f x:0 0 0 2024-04-16T02:35:56.555 INFO:tasks.workunit.client.0.smithi151.stdout:2/756: getdents de/d11/d42/d82/d95/d4b/d60 0 2024-04-16T02:35:56.561 INFO:tasks.workunit.client.0.smithi151.stdout:0/456: write dc/d15/d34/f57 [862573,80554] 0 2024-04-16T02:35:56.563 INFO:tasks.workunit.client.0.smithi151.stdout:6/628: creat d2/d5/df/d23/d31/fd8 x:0 0 0 2024-04-16T02:35:56.567 INFO:tasks.workunit.client.0.smithi151.stdout:7/584: dwrite d0/d1f/d31/d42/f6c [0,4194304] 0 2024-04-16T02:35:56.569 INFO:tasks.workunit.client.0.smithi151.stdout:9/522: mknod d1/d3/d6/d17/d54/d9b/d22/d36/cb0 0 2024-04-16T02:35:56.569 INFO:tasks.workunit.client.0.smithi151.stdout:8/322: dwrite d11/d2a/d33/f53 [0,4194304] 0 2024-04-16T02:35:56.570 INFO:tasks.workunit.client.0.smithi151.stdout:6/629: chown d2/d5/df/d23/d56/d72/fac 193578782 1 2024-04-16T02:35:56.578 INFO:tasks.workunit.client.0.smithi151.stdout:4/520: dwrite d0/d11/d19/d46/f6f [0,4194304] 0 2024-04-16T02:35:56.584 INFO:tasks.workunit.client.0.smithi151.stdout:4/521: write d0/d11/d19/d46/f6f [1315083,26993] 0 2024-04-16T02:35:56.586 INFO:tasks.workunit.client.0.smithi151.stdout:3/332: dwrite d1/d6/d22/f2b [0,4194304] 0 2024-04-16T02:35:56.587 INFO:tasks.workunit.client.0.smithi151.stdout:3/333: readlink d1/d6/d22/d23/l6a 0 2024-04-16T02:35:56.594 INFO:tasks.workunit.client.0.smithi151.stdout:1/507: truncate d3/d4/d64/d2a/d73/f85 670838 0 2024-04-16T02:35:56.597 INFO:tasks.workunit.client.0.smithi151.stdout:2/757: rename de/lb4 to de/d11/d42/d7a/d8b/db1/d90/lfb 0 2024-04-16T02:35:56.598 INFO:tasks.workunit.client.0.smithi151.stdout:8/323: symlink d11/d22/d2e/l61 0 2024-04-16T02:35:56.599 INFO:tasks.workunit.client.0.smithi151.stdout:7/585: symlink d0/d1f/d4c/db6/lcf 0 2024-04-16T02:35:56.600 INFO:tasks.workunit.client.0.smithi151.stdout:9/523: rmdir d1/d3/d6/d17/d54/d9b/d22/d2c 39 2024-04-16T02:35:56.602 INFO:tasks.workunit.client.0.smithi151.stdout:9/524: chown d1/d3/d6 9896041 1 2024-04-16T02:35:56.603 INFO:tasks.workunit.client.0.smithi151.stdout:4/522: fdatasync d0/d11/f3d 0 2024-04-16T02:35:56.604 INFO:tasks.workunit.client.0.smithi151.stdout:4/523: readlink d0/l4 0 2024-04-16T02:35:56.606 INFO:tasks.workunit.client.0.smithi151.stdout:1/508: chown d3/d4/d64/d1c/f50 28 1 2024-04-16T02:35:56.606 INFO:tasks.workunit.client.0.smithi151.stdout:0/457: dwrite dc/d15/d34/d3b/f44 [0,4194304] 0 2024-04-16T02:35:56.606 INFO:tasks.workunit.client.0.smithi151.stdout:4/524: fdatasync d0/d64/f76 0 2024-04-16T02:35:56.609 INFO:tasks.workunit.client.0.smithi151.stdout:4/525: chown d0/d11/d19/d31/d4a/d60/lab 889 1 2024-04-16T02:35:56.610 INFO:tasks.workunit.client.0.smithi151.stdout:0/458: readlink dc/d15/d5c/l66 0 2024-04-16T02:35:56.610 INFO:tasks.workunit.client.0.smithi151.stdout:2/758: rename de/d15/lca to de/d15/dcd/dd0/lfc 0 2024-04-16T02:35:56.611 INFO:tasks.workunit.client.0.smithi151.stdout:0/459: fdatasync dc/d15/f87 0 2024-04-16T02:35:56.613 INFO:tasks.workunit.client.0.smithi151.stdout:7/586: rmdir d0/d2c/d38 39 2024-04-16T02:35:56.619 INFO:tasks.workunit.client.0.smithi151.stdout:8/324: mkdir d11/d2a/d33/d4e/d62 0 2024-04-16T02:35:56.621 INFO:tasks.workunit.client.0.smithi151.stdout:1/509: truncate d3/d33/d10/f6b 979224 0 2024-04-16T02:35:56.623 INFO:tasks.workunit.client.0.smithi151.stdout:2/759: mkdir de/d65/d88/dfd 0 2024-04-16T02:35:56.624 INFO:tasks.workunit.client.0.smithi151.stdout:7/587: dread - d0/d1f/d31/f86 zero size 2024-04-16T02:35:56.627 INFO:tasks.workunit.client.0.smithi151.stdout:9/525: dread d1/d3/d6/d28/d39/f48 [0,4194304] 0 2024-04-16T02:35:56.630 INFO:tasks.workunit.client.0.smithi151.stdout:7/588: write d0/d1f/d46/f8d [629361,62339] 0 2024-04-16T02:35:56.632 INFO:tasks.workunit.client.0.smithi151.stdout:1/510: dwrite d3/d58/df/d32/d3a/d71/f75 [0,4194304] 0 2024-04-16T02:35:56.633 INFO:tasks.workunit.client.0.smithi151.stdout:2/760: dread de/d15/f89 [0,4194304] 0 2024-04-16T02:35:56.637 INFO:tasks.workunit.client.0.smithi151.stdout:1/511: symlink d3/d33/d10/d4f/lb3 0 2024-04-16T02:35:56.637 INFO:tasks.workunit.client.0.smithi151.stdout:9/526: chown d1/d3/d6/d17/d54/d9b/d22/d2c/c3b 24142530 1 2024-04-16T02:35:56.640 INFO:tasks.workunit.client.0.smithi151.stdout:3/334: write d1/d6/f1b [4100891,116908] 0 2024-04-16T02:35:56.641 INFO:tasks.workunit.client.0.smithi151.stdout:2/761: creat de/d11/d42/d7a/d8b/db1/d90/ffe x:0 0 0 2024-04-16T02:35:56.645 INFO:tasks.workunit.client.0.smithi151.stdout:3/335: readlink d1/d6/l1f 0 2024-04-16T02:35:56.647 INFO:tasks.workunit.client.0.smithi151.stdout:4/526: write d0/d9b/fa3 [94443,7271] 0 2024-04-16T02:35:56.650 INFO:tasks.workunit.client.0.smithi151.stdout:2/762: mkdir de/d65/d71/d98/dff 0 2024-04-16T02:35:56.650 INFO:tasks.workunit.client.0.smithi151.stdout:7/589: dwrite d0/d1f/d33/d56/f84 [4194304,4194304] 0 2024-04-16T02:35:56.653 INFO:tasks.workunit.client.0.smithi151.stdout:6/630: dwrite d2/d5/df/d23/d82/d8d/fae [0,4194304] 0 2024-04-16T02:35:56.655 INFO:tasks.workunit.client.0.smithi151.stdout:9/527: rename d1/d3/d6/d17/d40/d56/dab to d1/d3/d6/d17/d7a/db1 0 2024-04-16T02:35:56.656 INFO:tasks.workunit.client.0.smithi151.stdout:4/527: fdatasync d0/d11/d19/f8b 0 2024-04-16T02:35:56.656 INFO:tasks.workunit.client.0.smithi151.stdout:8/325: dread d11/d22/f26 [0,4194304] 0 2024-04-16T02:35:56.657 INFO:tasks.workunit.client.0.smithi151.stdout:1/512: truncate d3/d33/f39 606055 0 2024-04-16T02:35:56.659 INFO:tasks.workunit.client.0.smithi151.stdout:7/590: fdatasync d0/f18 0 2024-04-16T02:35:56.661 INFO:tasks.workunit.client.0.smithi151.stdout:6/631: fsync d2/d3d/f55 0 2024-04-16T02:35:56.664 INFO:tasks.workunit.client.0.smithi151.stdout:1/513: fsync d3/d33/d10/f4e 0 2024-04-16T02:35:56.666 INFO:tasks.workunit.client.0.smithi151.stdout:1/514: stat d3/d4/d64/d2a/f65 0 2024-04-16T02:35:56.666 INFO:tasks.workunit.client.0.smithi151.stdout:3/336: write d1/d37/f38 [1473723,111497] 0 2024-04-16T02:35:56.668 INFO:tasks.workunit.client.0.smithi151.stdout:7/591: symlink d0/d1f/d99/ld0 0 2024-04-16T02:35:56.670 INFO:tasks.workunit.client.0.smithi151.stdout:2/763: mknod de/d65/d71/d7c/dc7/c100 0 2024-04-16T02:35:56.671 INFO:tasks.workunit.client.0.smithi151.stdout:4/528: read - d0/d11/d19/d31/d80/fa5 zero size 2024-04-16T02:35:56.672 INFO:tasks.workunit.client.0.smithi151.stdout:4/529: write d0/f16 [292991,40398] 0 2024-04-16T02:35:56.681 INFO:tasks.workunit.client.0.smithi151.stdout:8/326: write d11/f17 [1191006,12317] 0 2024-04-16T02:35:56.683 INFO:tasks.workunit.client.0.smithi151.stdout:8/327: write d11/d2a/d33/f53 [1022058,16888] 0 2024-04-16T02:35:56.685 INFO:tasks.workunit.client.0.smithi151.stdout:7/592: truncate d0/d1f/d4c/d82/db3/f9c 268302 0 2024-04-16T02:35:56.685 INFO:tasks.workunit.client.0.smithi151.stdout:2/764: dread de/d15/f17 [0,4194304] 0 2024-04-16T02:35:56.688 INFO:tasks.workunit.client.0.smithi151.stdout:9/528: dread d1/d3/d6/d17/d44/f6e [4194304,4194304] 0 2024-04-16T02:35:56.688 INFO:tasks.workunit.client.0.smithi151.stdout:2/765: chown de/d11/d42/d7a/d8b/f59 172 1 2024-04-16T02:35:56.690 INFO:tasks.workunit.client.0.smithi151.stdout:1/515: dwrite d3/d4/f96 [0,4194304] 0 2024-04-16T02:35:56.695 INFO:tasks.workunit.client.0.smithi151.stdout:3/337: write d1/f26 [993694,9889] 0 2024-04-16T02:35:56.697 INFO:tasks.workunit.client.0.smithi151.stdout:1/516: dread - d3/d4/d64/d1c/f90 zero size 2024-04-16T02:35:56.697 INFO:tasks.workunit.client.0.smithi151.stdout:2/766: write de/d11/d42/d5e/d6c/db8/dcc/fbd [70603,51002] 0 2024-04-16T02:35:56.698 INFO:tasks.workunit.client.0.smithi151.stdout:3/338: chown d1/d6/d22/f2b 2083011 1 2024-04-16T02:35:56.699 INFO:tasks.workunit.client.0.smithi151.stdout:8/328: write d11/d22/f26 [446374,55290] 0 2024-04-16T02:35:56.700 INFO:tasks.workunit.client.0.smithi151.stdout:6/632: dwrite d2/d5/df/d23/d82/d8d/f80 [0,4194304] 0 2024-04-16T02:35:56.700 INFO:tasks.workunit.client.0.smithi151.stdout:2/767: dread - de/d11/d42/d82/fd8 zero size 2024-04-16T02:35:56.701 INFO:tasks.workunit.client.0.smithi151.stdout:7/593: mkdir d0/d2c/d38/d5e/dd1 0 2024-04-16T02:35:56.706 INFO:tasks.workunit.client.0.smithi151.stdout:4/530: getdents d0/d11/d19/d87/d3a 0 2024-04-16T02:35:56.709 INFO:tasks.workunit.client.0.smithi151.stdout:7/594: rename d0/d1f/d46/d4f/l77 to d0/d2c/d38/dab/ld2 0 2024-04-16T02:35:56.718 INFO:tasks.workunit.client.0.smithi151.stdout:1/517: write d3/d1f/f94 [1642477,92788] 0 2024-04-16T02:35:56.719 INFO:tasks.workunit.client.0.smithi151.stdout:6/633: link d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/l76 d2/d5/df/d23/d45/d62/ld9 0 2024-04-16T02:35:56.721 INFO:tasks.workunit.client.0.smithi151.stdout:2/768: dwrite de/f3e [0,4194304] 0 2024-04-16T02:35:56.721 INFO:tasks.workunit.client.0.smithi151.stdout:8/329: getdents d11 0 2024-04-16T02:35:56.723 INFO:tasks.workunit.client.0.smithi151.stdout:7/595: creat d0/d1f/d99/dc0/fd3 x:0 0 0 2024-04-16T02:35:56.725 INFO:tasks.workunit.client.0.smithi151.stdout:9/529: dwrite d1/f55 [0,4194304] 0 2024-04-16T02:35:56.730 INFO:tasks.workunit.client.0.smithi151.stdout:2/769: creat de/d11/d42/d5e/d6c/d6d/f101 x:0 0 0 2024-04-16T02:35:56.732 INFO:tasks.workunit.client.0.smithi151.stdout:9/530: symlink d1/d3/d6/d17/d54/d9b/lb2 0 2024-04-16T02:35:56.733 INFO:tasks.workunit.client.0.smithi151.stdout:6/634: write d2/d13/f67 [2005016,9240] 0 2024-04-16T02:35:56.737 INFO:tasks.workunit.client.0.smithi151.stdout:8/330: mknod d11/d2a/d35/d55/c63 0 2024-04-16T02:35:56.741 INFO:tasks.workunit.client.0.smithi151.stdout:3/339: dread d1/f2e [0,4194304] 0 2024-04-16T02:35:56.742 INFO:tasks.workunit.client.0.smithi151.stdout:4/531: dread d0/d11/d19/d87/d3a/d50/f6b [0,4194304] 0 2024-04-16T02:35:56.744 INFO:tasks.workunit.client.0.smithi151.stdout:3/340: dread - d1/d6/d22/d23/f62 zero size 2024-04-16T02:35:56.744 INFO:tasks.workunit.client.0.smithi151.stdout:1/518: dwrite d3/d4/d64/fa4 [0,4194304] 0 2024-04-16T02:35:56.752 INFO:tasks.workunit.client.0.smithi151.stdout:8/331: rename d11/d2a/d35/d55 to d11/d2a/d35/d64 0 2024-04-16T02:35:56.754 INFO:tasks.workunit.client.0.smithi151.stdout:6/635: write d2/d13/f18 [1188295,85673] 0 2024-04-16T02:35:56.755 INFO:tasks.workunit.client.0.smithi151.stdout:7/596: truncate d0/d2c/d38/d43/d78/f9a 3318755 0 2024-04-16T02:35:56.757 INFO:tasks.workunit.client.0.smithi151.stdout:1/519: mkdir d3/d33/db4 0 2024-04-16T02:35:56.757 INFO:tasks.workunit.client.0.smithi151.stdout:2/770: write de/f45 [3788292,13524] 0 2024-04-16T02:35:56.758 INFO:tasks.workunit.client.0.smithi151.stdout:7/597: write d0/d1f/da8/db2/fce [523177,77064] 0 2024-04-16T02:35:56.763 INFO:tasks.workunit.client.0.smithi151.stdout:1/520: readlink d3/d4/l13 0 2024-04-16T02:35:56.766 INFO:tasks.workunit.client.0.smithi151.stdout:2/771: chown de/c13 799 1 2024-04-16T02:35:56.768 INFO:tasks.workunit.client.0.smithi151.stdout:6/636: mknod d2/d13/d5f/dd7/cda 0 2024-04-16T02:35:56.769 INFO:tasks.workunit.client.0.smithi151.stdout:8/332: write d11/f28 [7850814,37608] 0 2024-04-16T02:35:56.770 INFO:tasks.workunit.client.0.smithi151.stdout:1/521: chown d3/d4/d64/d2a/d52/l84 94605332 1 2024-04-16T02:35:56.771 INFO:tasks.workunit.client.0.smithi151.stdout:2/772: mkdir de/d65/dc0/d102 0 2024-04-16T02:35:56.771 INFO:tasks.workunit.client.0.smithi151.stdout:9/531: dread d1/d75/f8b [0,4194304] 0 2024-04-16T02:35:56.776 INFO:tasks.workunit.client.0.smithi151.stdout:7/598: dwrite d0/d1f/d46/f8d [0,4194304] 0 2024-04-16T02:35:56.782 INFO:tasks.workunit.client.0.smithi151.stdout:3/341: dwrite d1/d6/d22/f52 [0,4194304] 0 2024-04-16T02:35:56.782 INFO:tasks.workunit.client.0.smithi151.stdout:1/522: truncate d3/d1f/f94 2011425 0 2024-04-16T02:35:56.784 INFO:tasks.workunit.client.0.smithi151.stdout:9/532: creat d1/d3/d6/d28/fb3 x:0 0 0 2024-04-16T02:35:56.785 INFO:tasks.workunit.client.0.smithi151.stdout:4/532: dwrite d0/d11/d19/d31/d45/fb1 [0,4194304] 0 2024-04-16T02:35:56.787 INFO:tasks.workunit.client.0.smithi151.stdout:7/599: symlink d0/d1f/d4c/db6/ld4 0 2024-04-16T02:35:56.788 INFO:tasks.workunit.client.0.smithi151.stdout:3/342: creat d1/d6/d22/d23/f70 x:0 0 0 2024-04-16T02:35:56.793 INFO:tasks.workunit.client.0.smithi151.stdout:4/533: mknod d0/d11/cb2 0 2024-04-16T02:35:56.793 INFO:tasks.workunit.client.0.smithi151.stdout:7/600: mknod d0/d1f/da8/cd5 0 2024-04-16T02:35:56.794 INFO:tasks.workunit.client.0.smithi151.stdout:6/637: write d2/d5/df/d23/d82/d8d/d22/d40/f3b [746877,6354] 0 2024-04-16T02:35:56.797 INFO:tasks.workunit.client.0.smithi151.stdout:2/773: dwrite de/d11/d42/d7a/f85 [0,4194304] 0 2024-04-16T02:35:56.799 INFO:tasks.workunit.client.0.smithi151.stdout:2/774: dread - de/d11/d42/d6b/d79/fa9 zero size 2024-04-16T02:35:56.801 INFO:tasks.workunit.client.0.smithi151.stdout:1/523: rename d3/d58/df/d32/d3a/d67/c8f to d3/d58/df/d38/d8b/cb5 0 2024-04-16T02:35:56.803 INFO:tasks.workunit.client.0.smithi151.stdout:7/601: creat d0/d27/da6/fd6 x:0 0 0 2024-04-16T02:35:56.803 INFO:tasks.workunit.client.0.smithi151.stdout:4/534: mknod d0/d11/d19/cb3 0 2024-04-16T02:35:56.807 INFO:tasks.workunit.client.0.smithi151.stdout:3/343: link d1/d6/d20/d2a/c53 d1/d6/d22/d23/d30/d44/d46/c71 0 2024-04-16T02:35:56.809 INFO:tasks.workunit.client.0.smithi151.stdout:9/533: dwrite d1/d3/d6/d17/d54/d9b/d22/f23 [0,4194304] 0 2024-04-16T02:35:56.810 INFO:tasks.workunit.client.0.smithi151.stdout:6/638: link d2/d5/df/d23/d82/fce d2/d5/df/d23/d45/fdb 0 2024-04-16T02:35:56.812 INFO:tasks.workunit.client.0.smithi151.stdout:4/535: dread d0/d11/f23 [4194304,4194304] 0 2024-04-16T02:35:56.814 INFO:tasks.workunit.client.0.smithi151.stdout:4/536: chown d0/d11/d19/d31/d4a/d60/l6d 7405257 1 2024-04-16T02:35:56.815 INFO:tasks.workunit.client.0.smithi151.stdout:7/602: fdatasync d0/d2c/d38/d5e/d87/fad 0 2024-04-16T02:35:56.817 INFO:tasks.workunit.client.0.smithi151.stdout:3/344: creat d1/d6/d20/d2a/d3f/d43/d6d/f72 x:0 0 0 2024-04-16T02:35:56.819 INFO:tasks.workunit.client.0.smithi151.stdout:1/524: read d3/d58/df/d32/d3a/f66 [3808422,108967] 0 2024-04-16T02:35:56.820 INFO:tasks.workunit.client.0.smithi151.stdout:6/639: rmdir d2/d5/df/d20/d4f/d81 39 2024-04-16T02:35:56.822 INFO:tasks.workunit.client.0.smithi151.stdout:7/603: mkdir d0/d1f/d4c/db6/dd7 0 2024-04-16T02:35:56.824 INFO:tasks.workunit.client.0.smithi151.stdout:3/345: truncate d1/d6/d22/f4d 4796582 0 2024-04-16T02:35:56.825 INFO:tasks.workunit.client.0.smithi151.stdout:8/333: dread d11/d2a/d35/f40 [0,4194304] 0 2024-04-16T02:35:56.826 INFO:tasks.workunit.client.0.smithi151.stdout:1/525: symlink d3/d33/d10/lb6 0 2024-04-16T02:35:56.828 INFO:tasks.workunit.client.0.smithi151.stdout:4/537: link d0/l4 d0/d11/d19/d31/d45/lb4 0 2024-04-16T02:35:56.830 INFO:tasks.workunit.client.0.smithi151.stdout:6/640: dwrite d2/d5/df/d23/d56/f58 [4194304,4194304] 0 2024-04-16T02:35:56.833 INFO:tasks.workunit.client.0.smithi151.stdout:8/334: stat ff 0 2024-04-16T02:35:56.835 INFO:tasks.workunit.client.0.smithi151.stdout:1/526: creat d3/d1f/d48/fb7 x:0 0 0 2024-04-16T02:35:56.836 INFO:tasks.workunit.client.0.smithi151.stdout:2/775: truncate de/d11/d42/d7a/d8b/f74 1200039 0 2024-04-16T02:35:56.844 INFO:tasks.workunit.client.0.smithi151.stdout:2/776: dwrite de/d15/fa8 [0,4194304] 0 2024-04-16T02:35:56.846 INFO:tasks.workunit.client.0.smithi151.stdout:2/777: dread - de/d11/d42/d7a/d8b/db1/d90/ffe zero size 2024-04-16T02:35:56.846 INFO:tasks.workunit.client.0.smithi151.stdout:1/527: unlink d3/d58/df/d32/d3a/d71/f6f 0 2024-04-16T02:35:56.847 INFO:tasks.workunit.client.0.smithi151.stdout:7/604: rmdir d0/d1f/d4c/db6/dc2 0 2024-04-16T02:35:56.870 INFO:tasks.workunit.client.0.smithi151.stdout:8/335: dread d11/d2a/f42 [4194304,4194304] 0 2024-04-16T02:35:56.911 INFO:tasks.workunit.client.0.smithi151.stdout:7/605: dread d0/f9e [0,4194304] 0 2024-04-16T02:35:56.911 INFO:tasks.workunit.client.0.smithi151.stdout:6/641: rename d2/d5/df/d23/d82/fce to d2/d5/df/d23/d45/d62/fdc 0 2024-04-16T02:35:56.913 INFO:tasks.workunit.client.0.smithi151.stdout:3/346: rmdir d1/d6/d20/d2a/d6b 0 2024-04-16T02:35:56.915 INFO:tasks.workunit.client.0.smithi151.stdout:1/528: read - d3/d58/df/f47 zero size 2024-04-16T02:35:56.919 INFO:tasks.workunit.client.0.smithi151.stdout:4/538: dwrite d0/d11/d19/d31/fa2 [0,4194304] 0 2024-04-16T02:35:56.922 INFO:tasks.workunit.client.0.smithi151.stdout:7/606: dread d0/d1f/d4c/d82/db3/f9c [0,4194304] 0 2024-04-16T02:35:56.923 INFO:tasks.workunit.client.0.smithi151.stdout:2/778: rename de/d15/l93 to de/d65/l103 0 2024-04-16T02:35:56.927 INFO:tasks.workunit.client.0.smithi151.stdout:8/336: write d11/d2a/f2f [4573887,121489] 0 2024-04-16T02:35:56.928 INFO:tasks.workunit.client.0.smithi151.stdout:1/529: truncate d3/d4/d64/d2a/d7a/f9f 1035151 0 2024-04-16T02:35:56.929 INFO:tasks.workunit.client.0.smithi151.stdout:1/530: chown d3/d33/c99 60506 1 2024-04-16T02:35:56.931 INFO:tasks.workunit.client.0.smithi151.stdout:7/607: symlink d0/d27/da6/ld8 0 2024-04-16T02:35:56.932 INFO:tasks.workunit.client.0.smithi151.stdout:2/779: mknod de/d11/d42/d82/c104 0 2024-04-16T02:35:56.933 INFO:tasks.workunit.client.0.smithi151.stdout:6/642: truncate d2/d5/df/d20/d4f/f9e 374198 0 2024-04-16T02:35:56.934 INFO:tasks.workunit.client.0.smithi151.stdout:2/780: truncate de/d11/ffa 842084 0 2024-04-16T02:35:56.935 INFO:tasks.workunit.client.0.smithi151.stdout:3/347: creat d1/d6/d20/d3d/f73 x:0 0 0 2024-04-16T02:35:56.935 INFO:tasks.workunit.client.0.smithi151.stdout:6/643: fsync d2/d5/df/d23/d82/d8d/d22/d40/d3f/fd2 0 2024-04-16T02:35:56.936 INFO:tasks.workunit.client.0.smithi151.stdout:8/337: rename d11/d22/f30 to d11/d22/d2e/f65 0 2024-04-16T02:35:56.937 INFO:tasks.workunit.client.0.smithi151.stdout:6/644: write d2/d5/f27 [3307846,97492] 0 2024-04-16T02:35:56.939 INFO:tasks.workunit.client.0.smithi151.stdout:8/338: write d11/d22/d2e/f3d [3734178,66412] 0 2024-04-16T02:35:56.941 INFO:tasks.workunit.client.0.smithi151.stdout:7/608: creat d0/d1f/d46/d4f/fd9 x:0 0 0 2024-04-16T02:35:56.942 INFO:tasks.workunit.client.0.smithi151.stdout:2/781: creat de/d11/d42/d7a/d8b/f105 x:0 0 0 2024-04-16T02:35:56.945 INFO:tasks.workunit.client.0.smithi151.stdout:8/339: truncate d11/d2a/d33/f4b 1087653 0 2024-04-16T02:35:56.948 INFO:tasks.workunit.client.0.smithi151.stdout:6/645: truncate d2/d5/df/d23/d31/f43 808681 0 2024-04-16T02:35:56.948 INFO:tasks.workunit.client.0.smithi151.stdout:2/782: chown de/l1b 13438 1 2024-04-16T02:35:56.948 INFO:tasks.workunit.client.0.smithi151.stdout:8/340: chown fb 0 1 2024-04-16T02:35:56.951 INFO:tasks.workunit.client.0.smithi151.stdout:4/539: dwrite d0/d11/d19/d31/d89/f1d [0,4194304] 0 2024-04-16T02:35:56.952 INFO:tasks.workunit.client.0.smithi151.stdout:2/783: chown de/d11/d42/d82/d95/f38 0 1 2024-04-16T02:35:56.954 INFO:tasks.workunit.client.0.smithi151.stdout:7/609: dwrite d0/d1f/d33/d89/fbe [0,4194304] 0 2024-04-16T02:35:56.954 INFO:tasks.workunit.client.0.smithi151.stdout:1/531: link d3/d4/d64/d2a/d7a/f86 d3/d4/db2/fb8 0 2024-04-16T02:35:56.956 INFO:tasks.workunit.client.0.smithi151.stdout:3/348: getdents d1/d6/d20/d54 0 2024-04-16T02:35:56.958 INFO:tasks.workunit.client.0.smithi151.stdout:8/341: creat d11/d2a/d33/f66 x:0 0 0 2024-04-16T02:35:56.964 INFO:tasks.workunit.client.0.smithi151.stdout:4/540: mknod d0/d99/cb5 0 2024-04-16T02:35:56.967 INFO:tasks.workunit.client.0.smithi151.stdout:1/532: fdatasync d3/d33/d10/d4f/f9b 0 2024-04-16T02:35:56.968 INFO:tasks.workunit.client.0.smithi151.stdout:2/784: dwrite de/d11/d42/d7a/d8b/db1/f67 [0,4194304] 0 2024-04-16T02:35:56.969 INFO:tasks.workunit.client.0.smithi151.stdout:7/610: symlink d0/d1f/d31/d42/lda 0 2024-04-16T02:35:56.971 INFO:tasks.workunit.client.0.smithi151.stdout:7/611: readlink d0/d27/da6/ld8 0 2024-04-16T02:35:56.971 INFO:tasks.workunit.client.0.smithi151.stdout:2/785: dread - de/d65/dc0/fef zero size 2024-04-16T02:35:56.972 INFO:tasks.workunit.client.0.smithi151.stdout:8/342: creat d11/d2a/d35/f67 x:0 0 0 2024-04-16T02:35:56.973 INFO:tasks.workunit.client.0.smithi151.stdout:6/646: dwrite d2/d5/df/d23/d82/d8d/d22/f4e [0,4194304] 0 2024-04-16T02:35:56.975 INFO:tasks.workunit.client.0.smithi151.stdout:1/533: mkdir d3/d4/d64/d1c/db9 0 2024-04-16T02:35:56.978 INFO:tasks.workunit.client.0.smithi151.stdout:1/534: fdatasync d3/d33/d10/d4f/fb0 0 2024-04-16T02:35:56.978 INFO:tasks.workunit.client.0.smithi151.stdout:2/786: rename de/d11/d42/d82/fd8 to de/d65/d88/dfd/f106 0 2024-04-16T02:35:56.979 INFO:tasks.workunit.client.0.smithi151.stdout:8/343: dread - d11/d22/d2e/f5c zero size 2024-04-16T02:35:56.981 INFO:tasks.workunit.client.0.smithi151.stdout:8/344: readlink d11/l31 0 2024-04-16T02:35:56.982 INFO:tasks.workunit.client.0.smithi151.stdout:8/345: chown d11/d2a/d33/d4e/d62 2894 1 2024-04-16T02:35:56.982 INFO:tasks.workunit.client.0.smithi151.stdout:8/346: stat f2 0 2024-04-16T02:35:56.983 INFO:tasks.workunit.client.0.smithi151.stdout:6/647: dread d2/d5/df/d23/d82/d8d/d22/f4e [0,4194304] 0 2024-04-16T02:35:56.989 INFO:tasks.workunit.client.0.smithi151.stdout:7/612: write d0/d1f/d4c/d98/fa4 [207988,69799] 0 2024-04-16T02:35:56.990 INFO:tasks.workunit.client.0.smithi151.stdout:8/347: symlink d11/d2a/d35/d64/l68 0 2024-04-16T02:35:56.991 INFO:tasks.workunit.client.0.smithi151.stdout:6/648: creat d2/d5/df/d23/d82/d8d/d22/d40/d3f/d65/fdd x:0 0 0 2024-04-16T02:35:56.992 INFO:tasks.workunit.client.0.smithi151.stdout:3/349: dwrite d1/d6/d22/f2c [0,4194304] 0 2024-04-16T02:35:56.992 INFO:tasks.workunit.client.0.smithi151.stdout:2/787: truncate de/f49 3144634 0 2024-04-16T02:35:56.998 INFO:tasks.workunit.client.0.smithi151.stdout:8/348: rename d11/d22/c5a to d11/d2a/d35/c69 0 2024-04-16T02:35:57.009 INFO:tasks.workunit.client.0.smithi151.stdout:2/788: dwrite de/d11/d42/d7a/d8b/db1/d90/ffe [0,4194304] 0 2024-04-16T02:35:57.010 INFO:tasks.workunit.client.0.smithi151.stdout:1/535: dwrite d3/d4/d64/d1c/f81 [0,4194304] 0 2024-04-16T02:35:57.013 INFO:tasks.workunit.client.0.smithi151.stdout:1/536: dread - d3/d4/d64/d2a/d73/fa2 zero size 2024-04-16T02:35:57.017 INFO:tasks.workunit.client.0.smithi151.stdout:6/649: truncate d2/d5/df/d23/d82/d8d/d22/f9b 1205064 0 2024-04-16T02:35:57.018 INFO:tasks.workunit.client.0.smithi151.stdout:2/789: read de/d11/d42/d82/d95/f33 [2068731,56393] 0 2024-04-16T02:35:57.020 INFO:tasks.workunit.client.0.smithi151.stdout:6/650: readlink d2/d5/l1e 0 2024-04-16T02:35:57.026 INFO:tasks.workunit.client.0.smithi151.stdout:4/541: dread d0/d99/f21 [0,4194304] 0 2024-04-16T02:35:57.028 INFO:tasks.workunit.client.0.smithi151.stdout:8/349: dwrite d11/f1b [4194304,4194304] 0 2024-04-16T02:35:57.028 INFO:tasks.workunit.client.0.smithi151.stdout:3/350: dwrite d1/d6/d20/d2a/d3f/d43/f55 [0,4194304] 0 2024-04-16T02:35:57.030 INFO:tasks.workunit.client.0.smithi151.stdout:2/790: rename de/d65/fea to de/d15/dcd/f107 0 2024-04-16T02:35:57.032 INFO:tasks.workunit.client.0.smithi151.stdout:6/651: mknod d2/d13/d70/cde 0 2024-04-16T02:35:57.035 INFO:tasks.workunit.client.0.smithi151.stdout:2/791: rename de/c76 to de/d65/dc0/d102/c108 0 2024-04-16T02:35:57.036 INFO:tasks.workunit.client.0.smithi151.stdout:3/351: creat d1/d6/d22/d23/d66/f74 x:0 0 0 2024-04-16T02:35:57.038 INFO:tasks.workunit.client.0.smithi151.stdout:4/542: creat d0/fb6 x:0 0 0 2024-04-16T02:35:57.039 INFO:tasks.workunit.client.0.smithi151.stdout:2/792: rmdir de/d65/d71/d7c 39 2024-04-16T02:35:57.042 INFO:tasks.workunit.client.0.smithi151.stdout:7/613: dread d0/d1f/d33/d56/f84 [0,4194304] 0 2024-04-16T02:35:57.045 INFO:tasks.workunit.client.0.smithi151.stdout:2/793: mknod de/d11/d42/d5e/d6c/d6d/da5/c109 0 2024-04-16T02:35:57.046 INFO:tasks.workunit.client.0.smithi151.stdout:2/794: chown de/d11/c21 127958 1 2024-04-16T02:35:57.047 INFO:tasks.workunit.client.0.smithi151.stdout:7/614: symlink d0/d2c/d38/d85/db7/ldb 0 2024-04-16T02:35:57.050 INFO:tasks.workunit.client.0.smithi151.stdout:2/795: unlink de/d11/d42/d82/d95/f38 0 2024-04-16T02:35:57.053 INFO:tasks.workunit.client.0.smithi151.stdout:1/537: dwrite d3/d58/f62 [0,4194304] 0 2024-04-16T02:35:57.054 INFO:tasks.workunit.client.0.smithi151.stdout:8/350: write d11/f43 [3256053,63390] 0 2024-04-16T02:35:57.056 INFO:tasks.workunit.client.0.smithi151.stdout:2/796: rmdir de/d11/d42/d82/d95/d4b 39 2024-04-16T02:35:57.058 INFO:tasks.workunit.client.0.smithi151.stdout:1/538: write d3/d4/d64/d1c/f2c [5061440,110736] 0 2024-04-16T02:35:57.061 INFO:tasks.workunit.client.0.smithi151.stdout:4/543: dread d0/f55 [0,4194304] 0 2024-04-16T02:35:57.061 INFO:tasks.workunit.client.0.smithi151.stdout:8/351: stat d11/d22/d2e/f65 0 2024-04-16T02:35:57.070 INFO:tasks.workunit.client.0.smithi151.stdout:1/539: write d3/d4/d64/d2a/f8d [3356734,26296] 0 2024-04-16T02:35:57.072 INFO:tasks.workunit.client.0.smithi151.stdout:6/652: dread d2/d5/df/d20/f2b [0,4194304] 0 2024-04-16T02:35:57.072 INFO:tasks.workunit.client.0.smithi151.stdout:8/352: mknod d11/d2a/c6a 0 2024-04-16T02:35:57.072 INFO:tasks.workunit.client.0.smithi151.stdout:3/352: dwrite d1/d6/d22/d23/f62 [0,4194304] 0 2024-04-16T02:35:57.076 INFO:tasks.workunit.client.0.smithi151.stdout:1/540: rmdir d3/d58/df/d68 39 2024-04-16T02:35:57.077 INFO:tasks.workunit.client.0.smithi151.stdout:7/615: dread d0/d27/f7a [0,4194304] 0 2024-04-16T02:35:57.078 INFO:tasks.workunit.client.0.smithi151.stdout:7/616: chown d0/d1f/d94 15 1 2024-04-16T02:35:57.078 INFO:tasks.workunit.client.0.smithi151.stdout:3/353: chown d1/d6/f1e 115022168 1 2024-04-16T02:35:57.081 INFO:tasks.workunit.client.0.smithi151.stdout:4/544: link d0/d11/d19/d87/d3a/d50/l57 d0/d99/lb7 0 2024-04-16T02:35:57.087 INFO:tasks.workunit.client.0.smithi151.stdout:3/354: chown d1/d6/d20/l32 25110987 1 2024-04-16T02:35:57.087 INFO:tasks.workunit.client.0.smithi151.stdout:7/617: dwrite d0/d1f/d46/f8d [0,4194304] 0 2024-04-16T02:35:57.087 INFO:tasks.workunit.client.0.smithi151.stdout:8/353: mkdir d11/d2a/d33/d4e/d62/d6b 0 2024-04-16T02:35:57.091 INFO:tasks.workunit.client.0.smithi151.stdout:4/545: mknod d0/d11/d19/d87/d3a/d65/d98/cb8 0 2024-04-16T02:35:57.093 INFO:tasks.workunit.client.0.smithi151.stdout:1/541: write d3/d4/d64/d1c/f90 [518956,37231] 0 2024-04-16T02:35:57.095 INFO:tasks.workunit.client.0.smithi151.stdout:6/653: creat d2/d5/df/d23/d31/fdf x:0 0 0 2024-04-16T02:35:57.096 INFO:tasks.workunit.client.0.smithi151.stdout:3/355: mknod d1/d6/d22/d23/d30/d44/c75 0 2024-04-16T02:35:57.096 INFO:tasks.workunit.client.0.smithi151.stdout:8/354: dread - d11/d22/d2e/f46 zero size 2024-04-16T02:35:57.103 INFO:tasks.workunit.client.0.smithi151.stdout:2/797: dread de/d11/d42/d82/d95/f33 [4194304,4194304] 0 2024-04-16T02:35:57.105 INFO:tasks.workunit.client.0.smithi151.stdout:6/654: mknod d2/d5/df/d23/d82/d8d/d22/d40/ce0 0 2024-04-16T02:35:57.105 INFO:tasks.workunit.client.0.smithi151.stdout:4/546: write d0/d11/d19/d46/f77 [232931,88415] 0 2024-04-16T02:35:57.105 INFO:tasks.workunit.client.0.smithi151.stdout:7/618: dread d0/d1f/d46/d4f/f57 [0,4194304] 0 2024-04-16T02:35:57.105 INFO:tasks.workunit.client.0.smithi151.stdout:6/655: stat d2/l4 0 2024-04-16T02:35:57.106 INFO:tasks.workunit.client.0.smithi151.stdout:4/547: readlink d0/l38 0 2024-04-16T02:35:57.106 INFO:tasks.workunit.client.0.smithi151.stdout:8/355: mkdir d11/d22/d6c 0 2024-04-16T02:35:57.107 INFO:tasks.workunit.client.0.smithi151.stdout:7/619: stat d0/d2c/d38/dab 0 2024-04-16T02:35:57.108 INFO:tasks.workunit.client.0.smithi151.stdout:1/542: rmdir d3/d33/db4 0 2024-04-16T02:35:57.109 INFO:tasks.workunit.client.0.smithi151.stdout:2/798: chown de/d11/d42/d82/d95/d4b/lf5 2 1 2024-04-16T02:35:57.110 INFO:tasks.workunit.client.0.smithi151.stdout:2/799: fdatasync de/d65/fbe 0 2024-04-16T02:35:57.110 INFO:tasks.workunit.client.0.smithi151.stdout:6/656: rmdir d2/d5/d2e/db6 39 2024-04-16T02:35:57.112 INFO:tasks.workunit.client.0.smithi151.stdout:8/356: symlink d11/l6d 0 2024-04-16T02:35:57.113 INFO:tasks.workunit.client.0.smithi151.stdout:4/548: mknod d0/d11/d19/d31/d80/da0/cb9 0 2024-04-16T02:35:57.114 INFO:tasks.workunit.client.0.smithi151.stdout:3/356: getdents d1/d6/d4a/d4e 0 2024-04-16T02:35:57.115 INFO:tasks.workunit.client.0.smithi151.stdout:3/357: write d1/d6/d22/f52 [3413033,129565] 0 2024-04-16T02:35:57.116 INFO:tasks.workunit.client.0.smithi151.stdout:1/543: dread - d3/d4/d64/d40/da3/fa6 zero size 2024-04-16T02:35:57.117 INFO:tasks.workunit.client.0.smithi151.stdout:7/620: write d0/f11 [4482293,16943] 0 2024-04-16T02:35:57.118 INFO:tasks.workunit.client.0.smithi151.stdout:2/800: chown de/d11/c31 430433 1 2024-04-16T02:35:57.118 INFO:tasks.workunit.client.0.smithi151.stdout:6/657: truncate d2/d5/f6 784219 0 2024-04-16T02:35:57.120 INFO:tasks.workunit.client.0.smithi151.stdout:6/658: fsync d2/d13/d46/fbf 0 2024-04-16T02:35:57.121 INFO:tasks.workunit.client.0.smithi151.stdout:4/549: mkdir d0/d11/d19/d31/d4a/dba 0 2024-04-16T02:35:57.121 INFO:tasks.workunit.client.0.smithi151.stdout:7/621: write d0/d1f/d46/f8d [3843934,27200] 0 2024-04-16T02:35:57.122 INFO:tasks.workunit.client.0.smithi151.stdout:4/550: dread - d0/f3c zero size 2024-04-16T02:35:57.123 INFO:tasks.workunit.client.0.smithi151.stdout:3/358: creat d1/d6/d22/d23/d41/f76 x:0 0 0 2024-04-16T02:35:57.127 INFO:tasks.workunit.client.0.smithi151.stdout:7/622: dread d0/d1f/da8/db2/fce [0,4194304] 0 2024-04-16T02:35:57.128 INFO:tasks.workunit.client.0.smithi151.stdout:2/801: mkdir de/d11/d42/d5e/d6c/db8/dcc/d10a 0 2024-04-16T02:35:57.129 INFO:tasks.workunit.client.0.smithi151.stdout:2/802: stat de/d11/d42/d5e/d6c/d6d/lc3 0 2024-04-16T02:35:57.130 INFO:tasks.workunit.client.0.smithi151.stdout:1/544: mknod d3/d33/cba 0 2024-04-16T02:35:57.133 INFO:tasks.workunit.client.0.smithi151.stdout:6/659: read d2/d5/df/d23/d82/d8d/d22/d40/d3f/fd5 [315872,45268] 0 2024-04-16T02:35:57.134 INFO:tasks.workunit.client.0.smithi151.stdout:6/660: stat d2/d5/df/d23/d31 0 2024-04-16T02:35:57.134 INFO:tasks.workunit.client.0.smithi151.stdout:8/357: mkdir d11/d2a/d35/d6e 0 2024-04-16T02:35:57.135 INFO:tasks.workunit.client.0.smithi151.stdout:3/359: truncate d1/d6/f31 1101926 0 2024-04-16T02:35:57.136 INFO:tasks.workunit.client.0.smithi151.stdout:7/623: rename d0/l69 to d0/d1f/d46/ldc 0 2024-04-16T02:35:57.138 INFO:tasks.workunit.client.0.smithi151.stdout:2/803: read de/d11/d42/d82/fa0 [277693,118132] 0 2024-04-16T02:35:57.140 INFO:tasks.workunit.client.0.smithi151.stdout:6/661: creat d2/fe1 x:0 0 0 2024-04-16T02:35:57.142 INFO:tasks.workunit.client.0.smithi151.stdout:3/360: truncate d1/d6/d20/d2a/d3f/f48 165010 0 2024-04-16T02:35:57.142 INFO:tasks.workunit.client.0.smithi151.stdout:7/624: rmdir d0/d1f/d4c 39 2024-04-16T02:35:57.143 INFO:tasks.workunit.client.0.smithi151.stdout:3/361: read - d1/d6/d22/d23/f70 zero size 2024-04-16T02:35:57.143 INFO:tasks.workunit.client.0.smithi151.stdout:7/625: fsync d0/d2c/d8e/fc5 0 2024-04-16T02:35:57.145 INFO:tasks.workunit.client.0.smithi151.stdout:4/551: creat d0/fbb x:0 0 0 2024-04-16T02:35:57.146 INFO:tasks.workunit.client.0.smithi151.stdout:2/804: unlink de/d11/d42/d7a/d8b/f59 0 2024-04-16T02:35:57.149 INFO:tasks.workunit.client.0.smithi151.stdout:6/662: symlink d2/d5/df/d23/d82/le2 0 2024-04-16T02:35:57.151 INFO:tasks.workunit.client.0.smithi151.stdout:3/362: readlink d1/d6/d22/d23/l47 0 2024-04-16T02:35:57.153 INFO:tasks.workunit.client.0.smithi151.stdout:7/626: mkdir d0/d2c/d38/d43/ddd 0 2024-04-16T02:35:57.155 INFO:tasks.workunit.client.0.smithi151.stdout:7/627: chown d0/d1f/f52 44260 1 2024-04-16T02:35:57.157 INFO:tasks.workunit.client.0.smithi151.stdout:8/358: mkdir d11/d22/d6f 0 2024-04-16T02:35:57.158 INFO:tasks.workunit.client.0.smithi151.stdout:3/363: dwrite d1/d6/d22/d23/f70 [0,4194304] 0 2024-04-16T02:35:57.160 INFO:tasks.workunit.client.0.smithi151.stdout:3/364: fdatasync d1/d6/f24 0 2024-04-16T02:35:57.160 INFO:tasks.workunit.client.0.smithi151.stdout:2/805: rename de/d65/d71/f7f to de/d11/d42/d7a/d8b/f10b 0 2024-04-16T02:35:57.162 INFO:tasks.workunit.client.0.smithi151.stdout:1/545: write d3/d33/f36 [1157503,16016] 0 2024-04-16T02:35:57.165 INFO:tasks.workunit.client.0.smithi151.stdout:5/635: sync 2024-04-16T02:35:57.166 INFO:tasks.workunit.client.0.smithi151.stdout:6/663: symlink d2/d5/df/d23/d82/d8d/d22/d40/d3f/d85/le3 0 2024-04-16T02:35:57.170 INFO:tasks.workunit.client.0.smithi151.stdout:2/806: creat de/d11/d42/d7a/d8b/f10c x:0 0 0 2024-04-16T02:35:57.201 INFO:tasks.workunit.client.0.smithi151.stdout:4/552: write d0/d11/d19/d31/d45/f9a [589451,8798] 0 2024-04-16T02:35:57.203 INFO:tasks.workunit.client.0.smithi151.stdout:1/546: dread d3/d33/f36 [0,4194304] 0 2024-04-16T02:35:57.205 INFO:tasks.workunit.client.0.smithi151.stdout:7/628: dwrite d0/d1f/d4c/d82/fca [0,4194304] 0 2024-04-16T02:35:57.205 INFO:tasks.workunit.client.0.smithi151.stdout:4/553: chown d0/d99/f42 28299 1 2024-04-16T02:35:57.206 INFO:tasks.workunit.client.0.smithi151.stdout:8/359: dwrite d11/f1b [4194304,4194304] 0 2024-04-16T02:35:57.211 INFO:tasks.workunit.client.0.smithi151.stdout:5/636: fsync d2/d6/d19/d30/d36/f38 0 2024-04-16T02:35:57.212 INFO:tasks.workunit.client.0.smithi151.stdout:2/807: write de/f3a [858402,22009] 0 2024-04-16T02:35:57.212 INFO:tasks.workunit.client.0.smithi151.stdout:4/554: write d0/d11/d19/d87/d3a/d50/fb0 [161948,79047] 0 2024-04-16T02:35:57.214 INFO:tasks.workunit.client.0.smithi151.stdout:3/365: write d1/d37/f42 [370202,13761] 0 2024-04-16T02:35:57.218 INFO:tasks.workunit.client.0.smithi151.stdout:6/664: rename d2/d5/df/d23/d82/d8d/d22/d40/d3f/d85/le3 to d2/d5/d2e/db6/le4 0 2024-04-16T02:35:57.219 INFO:tasks.workunit.client.0.smithi151.stdout:3/366: write d1/d37/f38 [1536311,41973] 0 2024-04-16T02:35:57.221 INFO:tasks.workunit.client.0.smithi151.stdout:1/547: read d3/d58/df/d32/f37 [296293,49357] 0 2024-04-16T02:35:57.222 INFO:tasks.workunit.client.0.smithi151.stdout:4/555: dread d0/d11/d19/d87/d3a/d50/f67 [0,4194304] 0 2024-04-16T02:35:57.224 INFO:tasks.workunit.client.0.smithi151.stdout:2/808: mkdir de/d3d/d10d 0 2024-04-16T02:35:57.225 INFO:tasks.workunit.client.0.smithi151.stdout:5/637: rename d2/d6/d19/f4b to d2/d6/d19/fc2 0 2024-04-16T02:35:57.228 INFO:tasks.workunit.client.0.smithi151.stdout:3/367: mkdir d1/d6/d4a/d4e/d77 0 2024-04-16T02:35:57.230 INFO:tasks.workunit.client.0.smithi151.stdout:3/368: read d1/d6/f1b [8042267,27935] 0 2024-04-16T02:35:57.230 INFO:tasks.workunit.client.0.smithi151.stdout:1/548: creat d3/d4/d64/d40/fbb x:0 0 0 2024-04-16T02:35:57.231 INFO:tasks.workunit.client.0.smithi151.stdout:4/556: creat d0/d11/d19/d87/d3a/d65/d98/fbc x:0 0 0 2024-04-16T02:35:57.232 INFO:tasks.workunit.client.0.smithi151.stdout:3/369: chown d1/d6/d20/d2a/f35 0 1 2024-04-16T02:35:57.233 INFO:tasks.workunit.client.0.smithi151.stdout:2/809: creat de/d11/d42/d7a/d8b/db1/d90/f10e x:0 0 0 2024-04-16T02:35:57.234 INFO:tasks.workunit.client.0.smithi151.stdout:2/810: dread - de/d65/dc0/fef zero size 2024-04-16T02:35:57.235 INFO:tasks.workunit.client.0.smithi151.stdout:8/360: getdents d11/d2a/d35/d64 0 2024-04-16T02:35:57.236 INFO:tasks.workunit.client.0.smithi151.stdout:8/361: chown d11/d2a/c6a 11974048 1 2024-04-16T02:35:57.239 INFO:tasks.workunit.client.0.smithi151.stdout:4/557: unlink d0/d11/d19/f44 0 2024-04-16T02:35:57.240 INFO:tasks.workunit.client.0.smithi151.stdout:5/638: mknod d2/d6/cc3 0 2024-04-16T02:35:57.242 INFO:tasks.workunit.client.0.smithi151.stdout:1/549: dread d3/d4/d64/d1c/f50 [0,4194304] 0 2024-04-16T02:35:57.242 INFO:tasks.workunit.client.0.smithi151.stdout:8/362: mkdir d11/d2a/d35/d70 0 2024-04-16T02:35:57.243 INFO:tasks.workunit.client.0.smithi151.stdout:7/629: dwrite d0/d2c/d38/d43/d78/f9a [0,4194304] 0 2024-04-16T02:35:57.246 INFO:tasks.workunit.client.0.smithi151.stdout:7/630: readlink d0/d1f/d4c/d82/l8c 0 2024-04-16T02:35:57.246 INFO:tasks.workunit.client.0.smithi151.stdout:5/639: unlink d2/d6/d19/d30/f77 0 2024-04-16T02:35:57.248 INFO:tasks.workunit.client.0.smithi151.stdout:7/631: chown d0/d1f/d60/fbd 334522591 1 2024-04-16T02:35:57.249 INFO:tasks.workunit.client.0.smithi151.stdout:8/363: mknod d11/d2a/d35/d64/c71 0 2024-04-16T02:35:57.250 INFO:tasks.workunit.client.0.smithi151.stdout:8/364: dread - d11/d22/d2e/f5c zero size 2024-04-16T02:35:57.252 INFO:tasks.workunit.client.0.smithi151.stdout:5/640: read - d2/d6/d19/d62/f7b zero size 2024-04-16T02:35:57.252 INFO:tasks.workunit.client.0.smithi151.stdout:2/811: write de/d11/f2b [5985292,42552] 0 2024-04-16T02:35:57.253 INFO:tasks.workunit.client.0.smithi151.stdout:1/550: mkdir d3/d58/df/dbc 0 2024-04-16T02:35:57.255 INFO:tasks.workunit.client.0.smithi151.stdout:8/365: creat d11/d2a/d33/d4e/f72 x:0 0 0 2024-04-16T02:35:57.258 INFO:tasks.workunit.client.0.smithi151.stdout:5/641: mkdir d2/d6/d19/d30/d88/dc4 0 2024-04-16T02:35:57.260 INFO:tasks.workunit.client.0.smithi151.stdout:4/558: truncate d0/d64/f71 5305637 0 2024-04-16T02:35:57.264 INFO:tasks.workunit.client.0.smithi151.stdout:5/642: unlink d2/d5e/f7e 0 2024-04-16T02:35:57.266 INFO:tasks.workunit.client.0.smithi151.stdout:4/559: fsync d0/d11/d19/d46/f58 0 2024-04-16T02:35:57.268 INFO:tasks.workunit.client.0.smithi151.stdout:7/632: getdents d0/d1f/da8 0 2024-04-16T02:35:57.268 INFO:tasks.workunit.client.0.smithi151.stdout:2/812: dwrite de/d11/d42/d7a/d8b/db1/fc6 [0,4194304] 0 2024-04-16T02:35:57.274 INFO:tasks.workunit.client.0.smithi151.stdout:5/643: unlink d2/d6/d19/d93/fbe 0 2024-04-16T02:35:57.281 INFO:tasks.workunit.client.0.smithi151.stdout:2/813: mkdir de/d11/d42/d5e/d6c/db8/dcc/d10f 0 2024-04-16T02:35:57.282 INFO:tasks.workunit.client.0.smithi151.stdout:4/560: dwrite d0/d11/d19/d31/d45/f9a [0,4194304] 0 2024-04-16T02:35:57.285 INFO:tasks.workunit.client.0.smithi151.stdout:1/551: dread d3/d33/d10/f55 [0,4194304] 0 2024-04-16T02:35:57.289 INFO:tasks.workunit.client.0.smithi151.stdout:8/366: dwrite d11/d2a/d35/d64/f58 [0,4194304] 0 2024-04-16T02:35:57.289 INFO:tasks.workunit.client.0.smithi151.stdout:1/552: readlink d3/d58/df/d32/d3a/d6d/l82 0 2024-04-16T02:35:57.293 INFO:tasks.workunit.client.0.smithi151.stdout:1/553: chown d3/d58/df/d38 1926562738 1 2024-04-16T02:35:57.295 INFO:tasks.workunit.client.0.smithi151.stdout:4/561: creat d0/d99/fbd x:0 0 0 2024-04-16T02:35:57.297 INFO:tasks.workunit.client.0.smithi151.stdout:8/367: fdatasync d11/d2a/f2c 0 2024-04-16T02:35:57.300 INFO:tasks.workunit.client.0.smithi151.stdout:5/644: link d2/f68 d2/d6/d19/d30/d88/dc4/fc5 0 2024-04-16T02:35:57.302 INFO:tasks.workunit.client.0.smithi151.stdout:7/633: dwrite d0/f18 [0,4194304] 0 2024-04-16T02:35:57.307 INFO:tasks.workunit.client.0.smithi151.stdout:1/554: rmdir d3/d58/df/d68 39 2024-04-16T02:35:57.313 INFO:tasks.workunit.client.0.smithi151.stdout:8/368: mknod d11/c73 0 2024-04-16T02:35:57.315 INFO:tasks.workunit.client.0.smithi151.stdout:7/634: mknod d0/d2c/d38/dab/cde 0 2024-04-16T02:35:57.318 INFO:tasks.workunit.client.0.smithi151.stdout:7/635: write d0/d1f/d4c/d98/fb5 [379123,105625] 0 2024-04-16T02:35:57.319 INFO:tasks.workunit.client.0.smithi151.stdout:1/555: dread d3/d58/df/d32/d3a/d71/f75 [0,4194304] 0 2024-04-16T02:35:57.320 INFO:tasks.workunit.client.0.smithi151.stdout:8/369: readlink d11/l21 0 2024-04-16T02:35:57.321 INFO:tasks.workunit.client.0.smithi151.stdout:2/814: link de/d11/d42/d5e/d6c/da2/ce4 de/d65/d71/d7c/c110 0 2024-04-16T02:35:57.326 INFO:tasks.workunit.client.0.smithi151.stdout:8/370: write d11/d2a/d33/f66 [761442,93506] 0 2024-04-16T02:35:57.326 INFO:tasks.workunit.client.0.smithi151.stdout:4/562: dwrite d0/f1f [0,4194304] 0 2024-04-16T02:35:57.328 INFO:tasks.workunit.client.0.smithi151.stdout:8/371: readlink d11/d22/d2e/l61 0 2024-04-16T02:35:57.328 INFO:tasks.workunit.client.0.smithi151.stdout:7/636: mkdir d0/ddf 0 2024-04-16T02:35:57.329 INFO:tasks.workunit.client.0.smithi151.stdout:2/815: creat de/d15/d78/f111 x:0 0 0 2024-04-16T02:35:57.331 INFO:tasks.workunit.client.0.smithi151.stdout:1/556: write d3/f80 [23734,14354] 0 2024-04-16T02:35:57.334 INFO:tasks.workunit.client.0.smithi151.stdout:8/372: creat d11/d2a/d35/d64/f74 x:0 0 0 2024-04-16T02:35:57.335 INFO:tasks.workunit.client.0.smithi151.stdout:4/563: mknod d0/d11/cbe 0 2024-04-16T02:35:57.336 INFO:tasks.workunit.client.0.smithi151.stdout:2/816: write de/d15/dcd/f107 [1224869,16545] 0 2024-04-16T02:35:57.337 INFO:tasks.workunit.client.0.smithi151.stdout:1/557: rmdir d3/d4/d64/d40/da3 39 2024-04-16T02:35:57.540 INFO:tasks.workunit.client.0.smithi151.stdout:7/637: rename d0/d1f/d33/d89/f90 to d0/fe0 0 2024-04-16T02:35:57.542 INFO:tasks.workunit.client.0.smithi151.stdout:5/645: dread d2/d6/f27 [0,4194304] 0 2024-04-16T02:35:57.542 INFO:tasks.workunit.client.0.smithi151.stdout:8/373: rmdir d11/d2a/d35/d64 39 2024-04-16T02:35:57.546 INFO:tasks.workunit.client.0.smithi151.stdout:4/564: chown d0/d11/d19/f7e 982642 1 2024-04-16T02:35:57.549 INFO:tasks.workunit.client.0.smithi151.stdout:2/817: rename de/f3e to de/d65/f112 0 2024-04-16T02:35:57.550 INFO:tasks.workunit.client.0.smithi151.stdout:2/818: write de/d11/d42/d7a/fbb [1037374,43593] 0 2024-04-16T02:35:57.551 INFO:tasks.workunit.client.0.smithi151.stdout:7/638: symlink d0/d1f/d4c/d82/le1 0 2024-04-16T02:35:57.552 INFO:tasks.workunit.client.0.smithi151.stdout:5/646: rmdir d2/d6/d19/d30/d36/d42 39 2024-04-16T02:35:57.553 INFO:tasks.workunit.client.0.smithi151.stdout:8/374: truncate ff 1094844 0 2024-04-16T02:35:57.555 INFO:tasks.workunit.client.0.smithi151.stdout:1/558: unlink d3/d4/d64/d2a/c4d 0 2024-04-16T02:35:57.556 INFO:tasks.workunit.client.0.smithi151.stdout:1/559: chown d3/d58/df/dbc 3283 1 2024-04-16T02:35:57.558 INFO:tasks.workunit.client.0.smithi151.stdout:7/639: creat d0/d2c/d38/db0/fe2 x:0 0 0 2024-04-16T02:35:57.568 INFO:tasks.workunit.client.0.smithi151.stdout:2/819: rename de/d65/l103 to de/d11/l113 0 2024-04-16T02:35:57.569 INFO:tasks.workunit.client.0.smithi151.stdout:5/647: symlink d2/d6/d19/d30/d88/dc4/lc6 0 2024-04-16T02:35:57.571 INFO:tasks.workunit.client.0.smithi151.stdout:4/565: creat d0/fbf x:0 0 0 2024-04-16T02:35:57.573 INFO:tasks.workunit.client.0.smithi151.stdout:7/640: dread d0/d1f/d94/fb8 [4194304,4194304] 0 2024-04-16T02:35:57.578 INFO:tasks.workunit.client.0.smithi151.stdout:1/560: mkdir d3/d58/df/d32/dbd 0 2024-04-16T02:35:57.579 INFO:tasks.workunit.client.0.smithi151.stdout:2/820: creat de/d15/f114 x:0 0 0 2024-04-16T02:35:57.579 INFO:tasks.workunit.client.0.smithi151.stdout:5/648: mknod d2/cc7 0 2024-04-16T02:35:57.580 INFO:tasks.workunit.client.0.smithi151.stdout:7/641: write d0/d2c/d38/fba [954499,57270] 0 2024-04-16T02:35:57.580 INFO:tasks.workunit.client.0.smithi151.stdout:5/649: stat d2/d6/l15 0 2024-04-16T02:35:57.581 INFO:tasks.workunit.client.0.smithi151.stdout:2/821: read de/d11/f2b [4429684,38434] 0 2024-04-16T02:35:57.582 INFO:tasks.workunit.client.0.smithi151.stdout:5/650: read - d2/d6/d19/d1a/d4c/fba zero size 2024-04-16T02:35:57.583 INFO:tasks.workunit.client.0.smithi151.stdout:8/375: link d11/d2a/d35/c69 d11/d22/d6c/c75 0 2024-04-16T02:35:57.583 INFO:tasks.workunit.client.0.smithi151.stdout:5/651: dread - d2/d6/d19/d62/fbc zero size 2024-04-16T02:35:57.586 INFO:tasks.workunit.client.0.smithi151.stdout:7/642: creat d0/d1f/d4c/d82/fe3 x:0 0 0 2024-04-16T02:35:57.587 INFO:tasks.workunit.client.0.smithi151.stdout:1/561: symlink d3/d4/lbe 0 2024-04-16T02:35:57.588 INFO:tasks.workunit.client.0.smithi151.stdout:7/643: fdatasync d0/d2c/d8e/fc5 0 2024-04-16T02:35:57.589 INFO:tasks.workunit.client.0.smithi151.stdout:2/822: rmdir de/d11/d42/d82/d95 39 2024-04-16T02:35:57.590 INFO:tasks.workunit.client.0.smithi151.stdout:8/376: dwrite d11/d22/f26 [0,4194304] 0 2024-04-16T02:35:57.592 INFO:tasks.workunit.client.0.smithi151.stdout:8/377: readlink d11/d2a/d33/l39 0 2024-04-16T02:35:57.596 INFO:tasks.workunit.client.0.smithi151.stdout:1/562: dwrite d3/d4/d64/d2a/f8d [0,4194304] 0 2024-04-16T02:35:57.599 INFO:tasks.workunit.client.0.smithi151.stdout:2/823: dwrite de/f3a [0,4194304] 0 2024-04-16T02:35:57.600 INFO:tasks.workunit.client.0.smithi151.stdout:4/566: rename d0/d11/d19/fa7 to d0/d11/d19/d31/d80/fc0 0 2024-04-16T02:35:57.605 INFO:tasks.workunit.client.0.smithi151.stdout:7/644: fdatasync d0/d2c/d38/d5e/faa 0 2024-04-16T02:35:57.606 INFO:tasks.workunit.client.0.smithi151.stdout:7/645: dread - d0/d1f/d60/fbd zero size 2024-04-16T02:35:57.607 INFO:tasks.workunit.client.0.smithi151.stdout:7/646: stat d0/d2c/d38/dab 0 2024-04-16T02:35:57.608 INFO:tasks.workunit.client.0.smithi151.stdout:7/647: truncate d0/fe0 1535714 0 2024-04-16T02:35:57.614 INFO:tasks.workunit.client.0.smithi151.stdout:2/824: creat de/d65/daa/dba/f115 x:0 0 0 2024-04-16T02:35:57.619 INFO:tasks.workunit.client.0.smithi151.stdout:5/652: write d2/d6/d19/f1d [2017620,53416] 0 2024-04-16T02:35:57.620 INFO:tasks.workunit.client.0.smithi151.stdout:1/563: getdents d3/d58/df/dbc 0 2024-04-16T02:35:57.624 INFO:tasks.workunit.client.0.smithi151.stdout:4/567: mknod d0/d11/d19/d87/d7b/cc1 0 2024-04-16T02:35:57.625 INFO:tasks.workunit.client.0.smithi151.stdout:7/648: symlink d0/d2c/d38/d5e/dd1/le4 0 2024-04-16T02:35:57.626 INFO:tasks.workunit.client.0.smithi151.stdout:8/378: creat d11/d22/f76 x:0 0 0 2024-04-16T02:35:57.628 INFO:tasks.workunit.client.0.smithi151.stdout:1/564: creat d3/d58/df/d32/d3a/d71/fbf x:0 0 0 2024-04-16T02:35:57.629 INFO:tasks.workunit.client.0.smithi151.stdout:7/649: truncate d0/d27/da6/fd6 552132 0 2024-04-16T02:35:57.630 INFO:tasks.workunit.client.0.smithi151.stdout:8/379: creat d11/d2a/d33/d4e/f77 x:0 0 0 2024-04-16T02:35:57.630 INFO:tasks.workunit.client.0.smithi151.stdout:1/565: fsync d3/d58/f62 0 2024-04-16T02:35:57.631 INFO:tasks.workunit.client.0.smithi151.stdout:5/653: mknod d2/d6/d19/d30/d36/d42/d6f/cc8 0 2024-04-16T02:35:57.634 INFO:tasks.workunit.client.0.smithi151.stdout:4/568: dwrite d0/d11/d19/d31/f95 [0,4194304] 0 2024-04-16T02:35:57.635 INFO:tasks.workunit.client.0.smithi151.stdout:7/650: write d0/d2c/d38/d5e/faa [1759998,84078] 0 2024-04-16T02:35:57.638 INFO:tasks.workunit.client.0.smithi151.stdout:8/380: dread d11/d2a/f2c [0,4194304] 0 2024-04-16T02:35:57.638 INFO:tasks.workunit.client.0.smithi151.stdout:1/566: unlink d3/d58/lad 0 2024-04-16T02:35:57.639 INFO:tasks.workunit.client.0.smithi151.stdout:5/654: read - d2/d6/d19/d30/d49/f94 zero size 2024-04-16T02:35:57.644 INFO:tasks.workunit.client.0.smithi151.stdout:7/651: unlink d0/d1f/d33/l44 0 2024-04-16T02:35:57.644 INFO:tasks.workunit.client.0.smithi151.stdout:2/825: dread de/ff6 [0,4194304] 0 2024-04-16T02:35:57.647 INFO:tasks.workunit.client.0.smithi151.stdout:8/381: chown d11/f28 92304827 1 2024-04-16T02:35:57.648 INFO:tasks.workunit.client.0.smithi151.stdout:1/567: creat d3/d33/d10/d4f/fc0 x:0 0 0 2024-04-16T02:35:57.650 INFO:tasks.workunit.client.0.smithi151.stdout:5/655: creat d2/d6/d19/d1a/d4c/fc9 x:0 0 0 2024-04-16T02:35:57.652 INFO:tasks.workunit.client.0.smithi151.stdout:7/652: dread d0/d1f/d60/f91 [0,4194304] 0 2024-04-16T02:35:57.654 INFO:tasks.workunit.client.0.smithi151.stdout:7/653: chown d0/d1f/d4c/d98/fb5 1 1 2024-04-16T02:35:57.655 INFO:tasks.workunit.client.0.smithi151.stdout:5/656: dread d2/d6/f1c [0,4194304] 0 2024-04-16T02:35:57.660 INFO:tasks.workunit.client.0.smithi151.stdout:7/654: creat d0/d1f/d94/fe5 x:0 0 0 2024-04-16T02:35:57.661 INFO:tasks.workunit.client.0.smithi151.stdout:5/657: read - d2/d6/d19/d69/f6b zero size 2024-04-16T02:35:57.662 INFO:tasks.workunit.client.0.smithi151.stdout:8/382: mkdir d11/d2a/d35/d70/d78 0 2024-04-16T02:35:57.663 INFO:tasks.workunit.client.0.smithi151.stdout:1/568: mknod d3/d58/df/d32/d3a/cc1 0 2024-04-16T02:35:57.663 INFO:tasks.workunit.client.0.smithi151.stdout:4/569: dwrite d0/f3c [0,4194304] 0 2024-04-16T02:35:57.670 INFO:tasks.workunit.client.0.smithi151.stdout:4/570: creat d0/d73/fc2 x:0 0 0 2024-04-16T02:35:57.679 INFO:tasks.workunit.client.0.smithi151.stdout:5/658: mknod d2/d6/d19/d30/cca 0 2024-04-16T02:35:57.680 INFO:tasks.workunit.client.0.smithi151.stdout:5/659: chown d2/d6/d43/d9d/cb3 1877796625 1 2024-04-16T02:35:57.684 INFO:tasks.workunit.client.0.smithi151.stdout:7/655: creat d0/d1f/d33/fe6 x:0 0 0 2024-04-16T02:35:57.686 INFO:tasks.workunit.client.0.smithi151.stdout:5/660: dwrite d2/d6/f50 [0,4194304] 0 2024-04-16T02:35:57.689 INFO:tasks.workunit.client.0.smithi151.stdout:8/383: rename d11/d2a/d35/d70 to d11/d22/d79 0 2024-04-16T02:35:57.691 INFO:tasks.workunit.client.0.smithi151.stdout:1/569: dwrite d3/d33/d10/f55 [0,4194304] 0 2024-04-16T02:35:57.693 INFO:tasks.workunit.client.0.smithi151.stdout:1/570: write d3/d4/d64/d1c/f90 [772612,26154] 0 2024-04-16T02:35:57.694 INFO:tasks.workunit.client.0.smithi151.stdout:4/571: write d0/d11/d19/f7e [1126837,18142] 0 2024-04-16T02:35:57.698 INFO:tasks.workunit.client.0.smithi151.stdout:4/572: dread - d0/d11/d19/d31/d45/f92 zero size 2024-04-16T02:35:57.700 INFO:tasks.workunit.client.0.smithi151.stdout:1/571: creat d3/d58/df/d32/dbd/fc2 x:0 0 0 2024-04-16T02:35:57.705 INFO:tasks.workunit.client.0.smithi151.stdout:4/573: rename d0/d99/l6c to d0/d11/d19/d31/d80/da0/lc3 0 2024-04-16T02:35:57.705 INFO:tasks.workunit.client.0.smithi151.stdout:8/384: write d11/d22/d2e/f5c [778363,20171] 0 2024-04-16T02:35:57.707 INFO:tasks.workunit.client.0.smithi151.stdout:8/385: chown d11/d2a/f2c 1941 1 2024-04-16T02:35:57.708 INFO:tasks.workunit.client.0.smithi151.stdout:1/572: dwrite d3/d4/d64/d2a/f60 [0,4194304] 0 2024-04-16T02:35:57.710 INFO:tasks.workunit.client.0.smithi151.stdout:1/573: readlink d3/d1f/lae 0 2024-04-16T02:35:57.711 INFO:tasks.workunit.client.0.smithi151.stdout:4/574: mknod d0/d11/d19/d31/cc4 0 2024-04-16T02:35:57.713 INFO:tasks.workunit.client.0.smithi151.stdout:1/574: creat d3/d4/db2/fc3 x:0 0 0 2024-04-16T02:35:57.720 INFO:tasks.workunit.client.0.smithi151.stdout:4/575: link d0/d11/d19/d87/d3a/d61/ca1 d0/d11/d19/d31/d80/da0/cc5 0 2024-04-16T02:35:57.720 INFO:tasks.workunit.client.0.smithi151.stdout:5/661: dread d2/f68 [0,4194304] 0 2024-04-16T02:35:57.720 INFO:tasks.workunit.client.0.smithi151.stdout:1/575: dread d3/d4/d64/d2a/d7a/f86 [0,4194304] 0 2024-04-16T02:35:57.723 INFO:tasks.workunit.client.0.smithi151.stdout:4/576: fdatasync d0/d11/d19/d87/d3a/d61/fa4 0 2024-04-16T02:35:57.726 INFO:tasks.workunit.client.0.smithi151.stdout:1/576: unlink d3/d33/d10/d4f/f9b 0 2024-04-16T02:35:57.730 INFO:tasks.workunit.client.0.smithi151.stdout:4/577: chown d0/f93 88479899 1 2024-04-16T02:35:57.731 INFO:tasks.workunit.client.0.smithi151.stdout:4/578: chown d0/d11/d19/d87/c8a 94156 1 2024-04-16T02:35:57.734 INFO:tasks.workunit.client.0.smithi151.stdout:8/386: dread fb [0,4194304] 0 2024-04-16T02:35:57.737 INFO:tasks.workunit.client.0.smithi151.stdout:4/579: dwrite d0/f1f [4194304,4194304] 0 2024-04-16T02:35:57.737 INFO:tasks.workunit.client.0.smithi151.stdout:1/577: symlink d3/d4/lc4 0 2024-04-16T02:35:57.739 INFO:tasks.workunit.client.0.smithi151.stdout:8/387: mknod d11/d2a/d35/d52/c7a 0 2024-04-16T02:35:57.739 INFO:tasks.workunit.client.0.smithi151.stdout:4/580: readlink d0/d11/d19/d31/d4a/d60/l6d 0 2024-04-16T02:35:57.742 INFO:tasks.workunit.client.0.smithi151.stdout:1/578: creat d3/d58/df/d32/d3a/d6d/fc5 x:0 0 0 2024-04-16T02:35:57.744 INFO:tasks.workunit.client.0.smithi151.stdout:4/581: unlink d0/f55 0 2024-04-16T02:35:57.745 INFO:tasks.workunit.client.0.smithi151.stdout:8/388: dread - d11/d2a/d35/d64/f74 zero size 2024-04-16T02:35:57.746 INFO:tasks.workunit.client.0.smithi151.stdout:5/662: dwrite d2/d6/d19/d1a/d4c/f92 [0,4194304] 0 2024-04-16T02:35:57.749 INFO:tasks.workunit.client.0.smithi151.stdout:4/582: read d0/d11/d19/d87/d3a/d50/fb0 [141034,107428] 0 2024-04-16T02:35:57.749 INFO:tasks.workunit.client.0.smithi151.stdout:8/389: rmdir d11/d22 39 2024-04-16T02:35:57.751 INFO:tasks.workunit.client.0.smithi151.stdout:5/663: fsync d2/fb 0 2024-04-16T02:35:57.751 INFO:tasks.workunit.client.0.smithi151.stdout:8/390: readlink d11/d2a/d35/d64/l68 0 2024-04-16T02:35:57.758 INFO:tasks.workunit.client.0.smithi151.stdout:5/664: rmdir d2/d6/d19/d93 39 2024-04-16T02:35:57.760 INFO:tasks.workunit.client.0.smithi151.stdout:4/583: rename d0/c1a to d0/d11/d19/d31/d4a/cc6 0 2024-04-16T02:35:57.761 INFO:tasks.workunit.client.0.smithi151.stdout:4/584: chown d0/d11/d19/d31/d89/f1d 4 1 2024-04-16T02:35:57.764 INFO:tasks.workunit.client.0.smithi151.stdout:5/665: mknod d2/d6/d19/d30/d36/d63/ccb 0 2024-04-16T02:35:57.765 INFO:tasks.workunit.client.0.smithi151.stdout:4/585: mkdir d0/d11/d19/d87/d3a/d50/dc7 0 2024-04-16T02:35:57.766 INFO:tasks.workunit.client.0.smithi151.stdout:8/391: creat d11/d22/d79/d78/f7b x:0 0 0 2024-04-16T02:35:57.767 INFO:tasks.workunit.client.0.smithi151.stdout:8/392: fsync d11/f28 0 2024-04-16T02:35:57.768 INFO:tasks.workunit.client.0.smithi151.stdout:8/393: write d11/d22/d2e/f3d [2259462,78028] 0 2024-04-16T02:35:57.772 INFO:tasks.workunit.client.0.smithi151.stdout:5/666: mknod d2/d6/d19/ccc 0 2024-04-16T02:35:57.773 INFO:tasks.workunit.client.0.smithi151.stdout:5/667: chown d2/d6/d19 54896203 1 2024-04-16T02:35:57.774 INFO:tasks.workunit.client.0.smithi151.stdout:4/586: mkdir d0/d11/d19/d31/dad/dc8 0 2024-04-16T02:35:57.776 INFO:tasks.workunit.client.0.smithi151.stdout:8/394: rename d11/d2a/d33/f53 to d11/d2a/d33/d4e/f7c 0 2024-04-16T02:35:57.786 INFO:tasks.workunit.client.0.smithi151.stdout:5/668: dread d2/f1f [0,4194304] 0 2024-04-16T02:35:57.820 INFO:tasks.workunit.client.0.smithi151.stdout:8/395: mkdir d11/d7d 0 2024-04-16T02:35:57.825 INFO:tasks.workunit.client.0.smithi151.stdout:5/669: getdents d2/d6/d19/d30/d36/d42 0 2024-04-16T02:35:57.827 INFO:tasks.workunit.client.0.smithi151.stdout:5/670: mknod d2/d6/d19/d30/d49/ccd 0 2024-04-16T02:35:57.830 INFO:tasks.workunit.client.0.smithi151.stdout:5/671: creat d2/d6/d19/d30/d49/fce x:0 0 0 2024-04-16T02:35:57.833 INFO:tasks.workunit.client.0.smithi151.stdout:5/672: truncate d2/d6/d19/d30/d49/f94 481264 0 2024-04-16T02:35:57.833 INFO:tasks.workunit.client.0.smithi151.stdout:8/396: write d11/d22/d2e/f65 [2413451,72295] 0 2024-04-16T02:35:57.837 INFO:tasks.workunit.client.0.smithi151.stdout:4/587: dwrite d0/d11/d19/d87/d3a/f75 [0,4194304] 0 2024-04-16T02:35:57.842 INFO:tasks.workunit.client.0.smithi151.stdout:8/397: dread d11/d2a/d35/d64/f58 [0,4194304] 0 2024-04-16T02:35:57.845 INFO:tasks.workunit.client.0.smithi151.stdout:8/398: dread fb [4194304,4194304] 0 2024-04-16T02:35:57.847 INFO:tasks.workunit.client.0.smithi151.stdout:5/673: creat d2/d6/d19/d30/d36/d42/d6f/d83/d8b/fcf x:0 0 0 2024-04-16T02:35:57.848 INFO:tasks.workunit.client.0.smithi151.stdout:4/588: symlink d0/d11/d19/d31/dad/dc8/lc9 0 2024-04-16T02:35:57.849 INFO:tasks.workunit.client.0.smithi151.stdout:4/589: chown d0/d11/d19/d31/d4a/c78 29053 1 2024-04-16T02:35:57.850 INFO:tasks.workunit.client.0.smithi151.stdout:8/399: mkdir d11/d22/d7e 0 2024-04-16T02:35:57.851 INFO:tasks.workunit.client.0.smithi151.stdout:8/400: write d11/d2a/d33/f48 [4877978,30418] 0 2024-04-16T02:35:57.863 INFO:tasks.workunit.client.0.smithi151.stdout:0/460: sync 2024-04-16T02:35:57.863 INFO:tasks.workunit.client.0.smithi151.stdout:0/461: chown dc/d15/l31 147925 1 2024-04-16T02:35:57.868 INFO:tasks.workunit.client.0.smithi151.stdout:5/674: rename d2/d6/d19/d30/d36/d63 to d2/d6/d19/d62/dd0 0 2024-04-16T02:35:57.871 INFO:tasks.workunit.client.0.smithi151.stdout:5/675: dread d2/f68 [0,4194304] 0 2024-04-16T02:35:57.871 INFO:tasks.workunit.client.0.smithi151.stdout:5/676: readlink d2/l2b 0 2024-04-16T02:35:57.874 INFO:tasks.workunit.client.0.smithi151.stdout:8/401: creat d11/d2a/d35/d64/f7f x:0 0 0 2024-04-16T02:35:57.880 INFO:tasks.workunit.client.0.smithi151.stdout:0/462: mknod dc/d4c/d72/d76/c92 0 2024-04-16T02:35:57.882 INFO:tasks.workunit.client.0.smithi151.stdout:5/677: write d2/d6/d19/d69/f6b [762314,1736] 0 2024-04-16T02:35:57.882 INFO:tasks.workunit.client.0.smithi151.stdout:4/590: dwrite d0/d11/d19/d87/d3a/d61/fa4 [0,4194304] 0 2024-04-16T02:35:57.885 INFO:tasks.workunit.client.0.smithi151.stdout:0/463: truncate dc/d4c/d72/f77 390614 0 2024-04-16T02:35:57.887 INFO:tasks.workunit.client.0.smithi151.stdout:5/678: dread - d2/d6/d19/d30/d88/f9f zero size 2024-04-16T02:35:57.890 INFO:tasks.workunit.client.0.smithi151.stdout:8/402: dread d11/d2a/d33/f48 [0,4194304] 0 2024-04-16T02:35:57.892 INFO:tasks.workunit.client.0.smithi151.stdout:0/464: rename dc/d15/d5c/l66 to dc/d15/d34/d67/d69/d6f/l93 0 2024-04-16T02:35:57.894 INFO:tasks.workunit.client.0.smithi151.stdout:5/679: creat d2/d6/d19/d30/d49/d85/fd1 x:0 0 0 2024-04-16T02:35:57.894 INFO:tasks.workunit.client.0.smithi151.stdout:4/591: write d0/d11/d19/d31/d80/f9d [419890,102798] 0 2024-04-16T02:35:57.899 INFO:tasks.workunit.client.0.smithi151.stdout:0/465: creat dc/d20/f94 x:0 0 0 2024-04-16T02:35:57.900 INFO:tasks.workunit.client.0.smithi151.stdout:0/466: dread - dc/d15/f90 zero size 2024-04-16T02:35:57.900 INFO:tasks.workunit.client.0.smithi151.stdout:5/680: creat d2/d6/d19/d62/dd0/fd2 x:0 0 0 2024-04-16T02:35:57.904 INFO:tasks.workunit.client.0.smithi151.stdout:4/592: rename d0/fb6 to d0/d11/d19/d87/d3a/d50/fca 0 2024-04-16T02:35:57.908 INFO:tasks.workunit.client.0.smithi151.stdout:0/467: unlink dc/d15/d34/d67/l82 0 2024-04-16T02:35:57.910 INFO:tasks.workunit.client.0.smithi151.stdout:4/593: rename d0/d11/c62 to d0/d11/d19/d87/d3a/d65/ccb 0 2024-04-16T02:35:57.918 INFO:tasks.workunit.client.0.smithi151.stdout:4/594: dwrite d0/d9b/fa3 [0,4194304] 0 2024-04-16T02:35:57.918 INFO:tasks.workunit.client.0.smithi151.stdout:0/468: getdents dc/d15/d34/d4e 0 2024-04-16T02:35:57.922 INFO:tasks.workunit.client.0.smithi151.stdout:4/595: stat d0/d11/d19/d31/d80/da0/cb9 0 2024-04-16T02:35:57.928 INFO:tasks.workunit.client.0.smithi151.stdout:8/403: dwrite d11/f5b [0,4194304] 0 2024-04-16T02:35:57.928 INFO:tasks.workunit.client.0.smithi151.stdout:5/681: dwrite d2/fb [0,4194304] 0 2024-04-16T02:35:57.934 INFO:tasks.workunit.client.0.smithi151.stdout:4/596: mkdir d0/daa/dcc 0 2024-04-16T02:35:57.938 INFO:tasks.workunit.client.0.smithi151.stdout:0/469: rename dc/d20/f7d to dc/d15/d34/d67/d69/d8a/f95 0 2024-04-16T02:35:57.940 INFO:tasks.workunit.client.0.smithi151.stdout:5/682: mkdir d2/d6/d19/d30/d36/d42/d6f/da8/dd3 0 2024-04-16T02:35:57.942 INFO:tasks.workunit.client.0.smithi151.stdout:5/683: write d2/d6/d19/d1a/f79 [1571508,91259] 0 2024-04-16T02:35:57.944 INFO:tasks.workunit.client.0.smithi151.stdout:8/404: creat d11/d2a/d33/d4e/d62/d6b/f80 x:0 0 0 2024-04-16T02:35:57.947 INFO:tasks.workunit.client.0.smithi151.stdout:0/470: rename dc/d4c/d72/f7e to dc/d4c/f96 0 2024-04-16T02:35:57.950 INFO:tasks.workunit.client.0.smithi151.stdout:4/597: mkdir d0/d11/d19/d46/dcd 0 2024-04-16T02:35:57.954 INFO:tasks.workunit.client.0.smithi151.stdout:0/471: symlink dc/d4c/l97 0 2024-04-16T02:35:57.955 INFO:tasks.workunit.client.0.smithi151.stdout:4/598: mknod d0/d11/d19/d87/d3a/cce 0 2024-04-16T02:35:57.956 INFO:tasks.workunit.client.0.smithi151.stdout:4/599: write d0/d64/f76 [637280,4461] 0 2024-04-16T02:35:57.957 INFO:tasks.workunit.client.0.smithi151.stdout:8/405: creat d11/d22/d7e/f81 x:0 0 0 2024-04-16T02:35:57.957 INFO:tasks.workunit.client.0.smithi151.stdout:4/600: chown d0/d11/d19/d31/d89/c28 640782 1 2024-04-16T02:35:57.963 INFO:tasks.workunit.client.0.smithi151.stdout:8/406: mknod d11/d2a/c82 0 2024-04-16T02:35:57.968 INFO:tasks.workunit.client.0.smithi151.stdout:8/407: unlink d11/d2a/d33/d4e/d62/d6b/f80 0 2024-04-16T02:35:57.968 INFO:tasks.workunit.client.0.smithi151.stdout:0/472: write f3 [3741829,118023] 0 2024-04-16T02:35:57.969 INFO:tasks.workunit.client.0.smithi151.stdout:5/684: dwrite d2/d6/d19/d1a/f48 [0,4194304] 0 2024-04-16T02:35:57.969 INFO:tasks.workunit.client.0.smithi151.stdout:0/473: readlink dc/d4c/l97 0 2024-04-16T02:35:57.975 INFO:tasks.workunit.client.0.smithi151.stdout:0/474: fdatasync dc/d20/f42 0 2024-04-16T02:35:57.975 INFO:tasks.workunit.client.0.smithi151.stdout:4/601: dwrite d0/d11/d19/d87/f5a [0,4194304] 0 2024-04-16T02:35:57.978 INFO:tasks.workunit.client.0.smithi151.stdout:8/408: mkdir d11/d2a/d83 0 2024-04-16T02:35:57.980 INFO:tasks.workunit.client.0.smithi151.stdout:5/685: mknod d2/d6/d19/d69/cd4 0 2024-04-16T02:35:57.989 INFO:tasks.workunit.client.0.smithi151.stdout:8/409: mkdir d11/d22/d6c/d84 0 2024-04-16T02:35:57.990 INFO:tasks.workunit.client.0.smithi151.stdout:5/686: symlink d2/d6/d19/d62/dbd/ld5 0 2024-04-16T02:35:57.994 INFO:tasks.workunit.client.0.smithi151.stdout:5/687: creat d2/d6/d19/d1a/d4c/fd6 x:0 0 0 2024-04-16T02:35:57.998 INFO:tasks.workunit.client.0.smithi151.stdout:4/602: write d0/d11/f26 [1375635,124721] 0 2024-04-16T02:35:57.999 INFO:tasks.workunit.client.0.smithi151.stdout:0/475: dread dc/d20/f3d [0,4194304] 0 2024-04-16T02:35:58.001 INFO:tasks.workunit.client.0.smithi151.stdout:4/603: symlink d0/d64/lcf 0 2024-04-16T02:35:58.001 INFO:tasks.workunit.client.0.smithi151.stdout:8/410: dread d11/f1a [0,4194304] 0 2024-04-16T02:35:58.002 INFO:tasks.workunit.client.0.smithi151.stdout:8/411: read d11/f1a [821976,13203] 0 2024-04-16T02:35:58.003 INFO:tasks.workunit.client.0.smithi151.stdout:8/412: dread - d11/d22/d79/d78/f7b zero size 2024-04-16T02:35:58.007 INFO:tasks.workunit.client.0.smithi151.stdout:8/413: mkdir d11/d2a/d85 0 2024-04-16T02:35:58.007 INFO:tasks.workunit.client.0.smithi151.stdout:8/414: chown d11/d2a/d35/d64/f7f 1393751 1 2024-04-16T02:35:58.009 INFO:tasks.workunit.client.0.smithi151.stdout:0/476: write dc/d15/d34/d67/f78 [552657,103051] 0 2024-04-16T02:35:58.009 INFO:tasks.workunit.client.0.smithi151.stdout:0/477: dread - dc/d20/f94 zero size 2024-04-16T02:35:58.011 INFO:tasks.workunit.client.0.smithi151.stdout:0/478: mknod dc/d15/d34/d67/d69/d8a/c98 0 2024-04-16T02:35:58.012 INFO:tasks.workunit.client.0.smithi151.stdout:0/479: dread - dc/d15/d34/d3b/f70 zero size 2024-04-16T02:35:58.013 INFO:tasks.workunit.client.0.smithi151.stdout:0/480: dread - dc/d3c/f75 zero size 2024-04-16T02:35:58.020 INFO:tasks.workunit.client.0.smithi151.stdout:0/481: write dc/d15/f17 [336467,92906] 0 2024-04-16T02:35:58.023 INFO:tasks.workunit.client.0.smithi151.stdout:0/482: creat dc/d15/d34/d67/d69/d6f/f99 x:0 0 0 2024-04-16T02:35:58.023 INFO:tasks.workunit.client.0.smithi151.stdout:0/483: chown cb 179 1 2024-04-16T02:35:58.095 INFO:tasks.workunit.client.0.smithi151.stdout:9/534: sync 2024-04-16T02:35:58.096 INFO:tasks.workunit.client.0.smithi151.stdout:9/535: chown d1/d3/f1e 0 1 2024-04-16T02:35:58.097 INFO:tasks.workunit.client.0.smithi151.stdout:9/536: fdatasync d1/d3/d6/d17/d54/d9b/d22/d2c/f93 0 2024-04-16T02:35:58.102 INFO:tasks.workunit.client.0.smithi151.stdout:9/537: dwrite d1/d3/d6/d17/d54/d9b/d22/f23 [0,4194304] 0 2024-04-16T02:35:58.104 INFO:tasks.workunit.client.0.smithi151.stdout:9/538: chown d1/d3/d6/d17/d54/d9b/d22/d36/f9e 7152357 1 2024-04-16T02:35:58.111 INFO:tasks.workunit.client.0.smithi151.stdout:9/539: getdents d1/d3/d5a/d8e 0 2024-04-16T02:35:58.117 INFO:tasks.workunit.client.0.smithi151.stdout:9/540: dwrite d1/d3/d5a/d8e/f9a [0,4194304] 0 2024-04-16T02:35:58.118 INFO:tasks.workunit.client.0.smithi151.stdout:9/541: chown d1/d3/d5f 41595 1 2024-04-16T02:35:58.119 INFO:tasks.workunit.client.0.smithi151.stdout:9/542: write d1/d3/d6/d17/d54/d9b/d22/d2c/f93 [752851,74915] 0 2024-04-16T02:35:58.123 INFO:tasks.workunit.client.0.smithi151.stdout:9/543: mkdir d1/d3/d6/d17/db4 0 2024-04-16T02:35:58.128 INFO:tasks.workunit.client.0.smithi151.stdout:9/544: truncate d1/d3/d6/d17/d54/d9b/dae/d6b/f77 748612 0 2024-04-16T02:35:58.128 INFO:tasks.workunit.client.0.smithi151.stdout:9/545: chown d1/d75 11483 1 2024-04-16T02:35:58.129 INFO:tasks.workunit.client.0.smithi151.stdout:9/546: chown d1/d3/d6/d17/d54/d9b/d22/d36/f3d 951212 1 2024-04-16T02:35:58.139 INFO:tasks.workunit.client.0.smithi151.stdout:9/547: write d1/dd/d4a/d64/f80 [1030524,127520] 0 2024-04-16T02:35:58.143 INFO:tasks.workunit.client.0.smithi151.stdout:9/548: read d1/d3/d6/d17/d7a/db1/f66 [152200,94495] 0 2024-04-16T02:35:58.143 INFO:tasks.workunit.client.0.smithi151.stdout:9/549: read - d1/d3/d6/d17/d7a/db1/fa3 zero size 2024-04-16T02:35:58.146 INFO:tasks.workunit.client.0.smithi151.stdout:9/550: creat d1/d3/d6/d17/d40/d56/dad/fb5 x:0 0 0 2024-04-16T02:35:58.150 INFO:tasks.workunit.client.0.smithi151.stdout:9/551: dwrite d1/d3/f1e [0,4194304] 0 2024-04-16T02:35:58.153 INFO:tasks.workunit.client.0.smithi151.stdout:9/552: creat d1/d3/d6/d17/d54/d9b/d22/d2c/fb6 x:0 0 0 2024-04-16T02:35:58.168 INFO:tasks.workunit.client.0.smithi151.stdout:9/553: write d1/d3/d6/d17/d54/f79 [4082735,128007] 0 2024-04-16T02:35:58.169 INFO:tasks.workunit.client.0.smithi151.stdout:9/554: fdatasync d1/d3/f1e 0 2024-04-16T02:35:58.173 INFO:tasks.workunit.client.0.smithi151.stdout:9/555: rename d1/d3/d6/d28/l59 to d1/d3/d5f/lb7 0 2024-04-16T02:35:58.174 INFO:tasks.workunit.client.0.smithi151.stdout:9/556: mknod d1/dd/d4a/d64/cb8 0 2024-04-16T02:35:58.181 INFO:tasks.workunit.client.0.smithi151.stdout:9/557: dwrite d1/d3/d6/d17/d54/d9b/d22/d36/f9e [0,4194304] 0 2024-04-16T02:35:58.186 INFO:tasks.workunit.client.0.smithi151.stdout:9/558: getdents d1/d3/d10/d7b 0 2024-04-16T02:35:58.193 INFO:tasks.workunit.client.0.smithi151.stdout:9/559: dwrite d1/d3/d6/d17/d54/d9b/d22/d2c/fb6 [0,4194304] 0 2024-04-16T02:35:58.198 INFO:tasks.workunit.client.0.smithi151.stdout:9/560: mkdir d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9 0 2024-04-16T02:35:58.202 INFO:tasks.workunit.client.0.smithi151.stdout:9/561: mknod d1/d3/d6/d17/d54/d9b/dae/d6b/cba 0 2024-04-16T02:35:58.203 INFO:tasks.workunit.client.0.smithi151.stdout:9/562: write d1/d3/d5a/f82 [998464,80182] 0 2024-04-16T02:35:58.204 INFO:tasks.workunit.client.0.smithi151.stdout:9/563: chown d1/f55 1339 1 2024-04-16T02:35:58.208 INFO:tasks.workunit.client.0.smithi151.stdout:9/564: unlink d1/d3/d6/d17/d54/d9b/d22/d2c/fb6 0 2024-04-16T02:35:58.221 INFO:tasks.workunit.client.0.smithi151.stdout:9/565: dwrite d1/dd/d4a/f94 [4194304,4194304] 0 2024-04-16T02:35:58.223 INFO:tasks.workunit.client.0.smithi151.stdout:9/566: write d1/d3/d6/d17/d7a/db1/fa1 [569872,121350] 0 2024-04-16T02:35:58.224 INFO:tasks.workunit.client.0.smithi151.stdout:9/567: stat d1/d3/l37 0 2024-04-16T02:35:58.226 INFO:tasks.workunit.client.0.smithi151.stdout:9/568: mkdir d1/dd/d4a/dbb 0 2024-04-16T02:35:58.229 INFO:tasks.workunit.client.0.smithi151.stdout:9/569: fsync d1/dd/d4a/f92 0 2024-04-16T02:35:58.231 INFO:tasks.workunit.client.0.smithi151.stdout:9/570: creat d1/d3/d6/d17/d54/d9b/dae/d6b/fbc x:0 0 0 2024-04-16T02:35:58.239 INFO:tasks.workunit.client.0.smithi151.stdout:9/571: dwrite d1/d3/d10/d7b/f85 [0,4194304] 0 2024-04-16T02:35:58.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:57 smithi151 ceph-mon[29003]: pgmap v26: 65 pgs: 65 active+clean; 3.2 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 85 MiB/s rd, 221 MiB/s wr, 470 op/s 2024-04-16T02:35:58.251 INFO:tasks.workunit.client.0.smithi151.stdout:9/572: truncate d1/d3/d6/d17/d40/d56/f61 1774563 0 2024-04-16T02:35:58.252 INFO:tasks.workunit.client.0.smithi151.stdout:9/573: fdatasync d1/d3/d6/d17/d40/d56/dad/fb5 0 2024-04-16T02:35:58.259 INFO:tasks.workunit.client.0.smithi151.stdout:9/574: dwrite d1/d3/d6/d17/d44/f6a [0,4194304] 0 2024-04-16T02:35:58.261 INFO:tasks.workunit.client.0.smithi151.stdout:9/575: truncate d1/dd/f8a 5160384 0 2024-04-16T02:35:58.269 INFO:tasks.workunit.client.0.smithi151.stdout:9/576: dwrite d1/d3/d6/d17/d54/f79 [0,4194304] 0 2024-04-16T02:35:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:57 smithi153 ceph-mon[36747]: pgmap v26: 65 pgs: 65 active+clean; 3.2 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 85 MiB/s rd, 221 MiB/s wr, 470 op/s 2024-04-16T02:35:58.274 INFO:tasks.workunit.client.0.smithi151.stdout:9/577: mknod d1/d3/d10/da6/cbd 0 2024-04-16T02:35:58.275 INFO:tasks.workunit.client.0.smithi151.stdout:9/578: truncate d1/d75/f8b 4378028 0 2024-04-16T02:35:58.278 INFO:tasks.workunit.client.0.smithi151.stdout:9/579: read d1/d3/d6/d17/d54/d9b/d22/d36/f3d [154996,52875] 0 2024-04-16T02:35:58.279 INFO:tasks.workunit.client.0.smithi151.stdout:6/665: sync 2024-04-16T02:35:58.284 INFO:tasks.workunit.client.0.smithi151.stdout:9/580: rmdir d1/d3/d6/d17/d54 39 2024-04-16T02:35:58.287 INFO:tasks.workunit.client.0.smithi151.stdout:9/581: fsync d1/d3/d6/d17/d54/d9b/f78 0 2024-04-16T02:35:58.288 INFO:tasks.workunit.client.0.smithi151.stdout:9/582: chown d1/d3/d6/d17/d54/d9b/d22/d2c 3887098 1 2024-04-16T02:35:58.296 INFO:tasks.workunit.client.0.smithi151.stdout:6/666: getdents d2/d5/df/d23/d82/d8d/d22/d40/d3f/db1 0 2024-04-16T02:35:58.299 INFO:tasks.workunit.client.0.smithi151.stdout:9/583: creat d1/d3/d6/fbe x:0 0 0 2024-04-16T02:35:58.302 INFO:tasks.workunit.client.0.smithi151.stdout:6/667: fsync d2/d5/f3e 0 2024-04-16T02:35:58.309 INFO:tasks.workunit.client.0.smithi151.stdout:9/584: dwrite d1/d3/d10/f4d [4194304,4194304] 0 2024-04-16T02:35:58.314 INFO:tasks.workunit.client.0.smithi151.stdout:6/668: dwrite d2/d5/df/d23/d82/d8d/d22/d40/d3f/d65/fdd [0,4194304] 0 2024-04-16T02:35:58.325 INFO:tasks.workunit.client.0.smithi151.stdout:6/669: dread d2/d13/d63/fa2 [0,4194304] 0 2024-04-16T02:35:58.331 INFO:tasks.workunit.client.0.smithi151.stdout:6/670: fdatasync d2/d13/d3a/f54 0 2024-04-16T02:35:58.335 INFO:tasks.workunit.client.0.smithi151.stdout:9/585: write d1/d3/d6/d28/d39/f48 [424152,2186] 0 2024-04-16T02:35:58.337 INFO:tasks.workunit.client.0.smithi151.stdout:9/586: write d1/d3/d6/d28/d39/f48 [2560991,68323] 0 2024-04-16T02:35:58.339 INFO:tasks.workunit.client.0.smithi151.stdout:6/671: dread - d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/d97/fab zero size 2024-04-16T02:35:58.340 INFO:tasks.workunit.client.0.smithi151.stdout:6/672: readlink d2/d5/df/d20/l42 0 2024-04-16T02:35:58.343 INFO:tasks.workunit.client.0.smithi151.stdout:9/587: creat d1/d75/fbf x:0 0 0 2024-04-16T02:35:58.352 INFO:tasks.workunit.client.0.smithi151.stdout:6/673: mkdir d2/d5/df/d20/d4f/d81/de5 0 2024-04-16T02:35:58.376 INFO:tasks.workunit.client.0.smithi151.stdout:6/674: dwrite d2/d5/df/fd0 [0,4194304] 0 2024-04-16T02:35:58.384 INFO:tasks.workunit.client.0.smithi151.stdout:6/675: dwrite d2/d5/df/d23/d56/f7e [0,4194304] 0 2024-04-16T02:35:58.399 INFO:tasks.workunit.client.0.smithi151.stdout:6/676: write d2/d5/df/d23/d82/d8d/d22/f68 [529881,93383] 0 2024-04-16T02:35:58.407 INFO:tasks.workunit.client.0.smithi151.stdout:6/677: dwrite d2/d5/df/d23/d56/f84 [4194304,4194304] 0 2024-04-16T02:35:58.420 INFO:tasks.workunit.client.0.smithi151.stdout:6/678: write d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/d97/fab [485601,94406] 0 2024-04-16T02:35:58.423 INFO:tasks.workunit.client.0.smithi151.stdout:6/679: mkdir d2/d5/df/d20/de6 0 2024-04-16T02:35:58.426 INFO:tasks.workunit.client.0.smithi151.stdout:6/680: mknod d2/d13/d46/ce7 0 2024-04-16T02:35:58.427 INFO:tasks.workunit.client.0.smithi151.stdout:6/681: write d2/d5/f27 [2094757,67690] 0 2024-04-16T02:35:58.432 INFO:tasks.workunit.client.0.smithi151.stdout:6/682: creat d2/d5/df/d23/d82/d8d/d22/d40/d3f/d85/fe8 x:0 0 0 2024-04-16T02:35:58.436 INFO:tasks.workunit.client.0.smithi151.stdout:6/683: rmdir d2/d5/d96 0 2024-04-16T02:35:58.437 INFO:tasks.workunit.client.0.smithi151.stdout:6/684: chown d2/d5/df/d23 0 1 2024-04-16T02:35:58.453 INFO:tasks.workunit.client.0.smithi151.stdout:6/685: truncate d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/d97/fab 365260 0 2024-04-16T02:35:58.460 INFO:tasks.workunit.client.0.smithi151.stdout:6/686: dwrite d2/d5/df/d23/d56/f7e [0,4194304] 0 2024-04-16T02:35:58.478 INFO:tasks.workunit.client.0.smithi151.stdout:6/687: truncate d2/d5/df/d23/d56/f84 2765182 0 2024-04-16T02:35:58.485 INFO:tasks.workunit.client.0.smithi151.stdout:6/688: dread d2/d5/df/d20/f47 [0,4194304] 0 2024-04-16T02:35:58.488 INFO:tasks.workunit.client.0.smithi151.stdout:6/689: creat d2/d5/df/d20/de6/fe9 x:0 0 0 2024-04-16T02:35:58.491 INFO:tasks.workunit.client.0.smithi151.stdout:6/690: dread - d2/d13/d6f/fb7 zero size 2024-04-16T02:35:58.495 INFO:tasks.workunit.client.0.smithi151.stdout:6/691: unlink d2/d5/df/d20/c21 0 2024-04-16T02:35:58.627 INFO:tasks.workunit.client.0.smithi151.stdout:6/692: dwrite d2/d5/df/d20/f24 [0,4194304] 0 2024-04-16T02:35:58.677 INFO:tasks.workunit.client.0.smithi151.stdout:6/693: dwrite d2/d5/df/d20/d4f/f93 [0,4194304] 0 2024-04-16T02:35:58.682 INFO:tasks.workunit.client.0.smithi151.stdout:6/694: read d2/d5/df/d20/f2b [4122396,43179] 0 2024-04-16T02:35:58.684 INFO:tasks.workunit.client.0.smithi151.stdout:6/695: creat d2/d5/df/d20/de6/fea x:0 0 0 2024-04-16T02:35:58.714 INFO:tasks.workunit.client.0.smithi151.stdout:6/696: dwrite d2/d5/df/d23/d45/d62/fdc [0,4194304] 0 2024-04-16T02:35:58.723 INFO:tasks.workunit.client.0.smithi151.stdout:6/697: link d2/l1d d2/d5/df/d20/de6/leb 0 2024-04-16T02:35:58.739 INFO:tasks.workunit.client.0.smithi151.stdout:6/698: dwrite d2/d5/df/d23/f8c [0,4194304] 0 2024-04-16T02:35:58.741 INFO:tasks.workunit.client.0.smithi151.stdout:6/699: creat d2/d5/df/d20/de6/fec x:0 0 0 2024-04-16T02:35:58.742 INFO:tasks.workunit.client.0.smithi151.stdout:6/700: stat d2/d5/df/d23/d45 0 2024-04-16T02:35:58.758 INFO:tasks.workunit.client.0.smithi151.stdout:6/701: dwrite d2/d5/df/d23/d56/d72/fa0 [0,4194304] 0 2024-04-16T02:35:58.764 INFO:tasks.workunit.client.0.smithi151.stdout:6/702: write d2/d13/d6f/fb7 [1032624,41804] 0 2024-04-16T02:35:58.767 INFO:tasks.workunit.client.0.smithi151.stdout:6/703: truncate d2/d5/df/d23/d45/fdb 5209421 0 2024-04-16T02:35:58.769 INFO:tasks.workunit.client.0.smithi151.stdout:6/704: creat d2/d5/df/d23/d82/d8d/d22/d40/d3f/d73/fed x:0 0 0 2024-04-16T02:35:58.771 INFO:tasks.workunit.client.0.smithi151.stdout:6/705: chown d2/d13/ca8 126406076 1 2024-04-16T02:35:58.774 INFO:tasks.workunit.client.0.smithi151.stdout:6/706: creat d2/d5/df/d20/d4f/d81/de5/fee x:0 0 0 2024-04-16T02:35:58.785 INFO:tasks.workunit.client.0.smithi151.stdout:6/707: dwrite d2/d3d/f9d [0,4194304] 0 2024-04-16T02:35:58.791 INFO:tasks.workunit.client.0.smithi151.stdout:6/708: rename d2/d5 to d2/d13/d63/def 0 2024-04-16T02:35:58.796 INFO:tasks.workunit.client.0.smithi151.stdout:6/709: truncate d2/d13/d63/def/f6 710144 0 2024-04-16T02:35:58.799 INFO:tasks.workunit.client.0.smithi151.stdout:6/710: truncate d2/d13/d63/def/df/d20/d4f/f9e 1399219 0 2024-04-16T02:35:58.800 INFO:tasks.workunit.client.0.smithi151.stdout:6/711: creat d2/d13/d5f/ff0 x:0 0 0 2024-04-16T02:35:58.805 INFO:tasks.workunit.client.0.smithi151.stdout:6/712: dread d2/d13/d63/def/df/d23/f8c [0,4194304] 0 2024-04-16T02:35:58.808 INFO:tasks.workunit.client.0.smithi151.stdout:6/713: read - d2/d13/d63/def/df/d23/d56/d72/fb5 zero size 2024-04-16T02:35:58.813 INFO:tasks.workunit.client.0.smithi151.stdout:6/714: unlink d2/d13/d63/def/df/d23/d82/d8d/lcf 0 2024-04-16T02:35:58.814 INFO:tasks.workunit.client.0.smithi151.stdout:6/715: rmdir d2/d13/d6f/da3 39 2024-04-16T02:35:58.817 INFO:tasks.workunit.client.0.smithi151.stdout:6/716: unlink d2/d13/d63/def/df/d20/d4f/d81/fb2 0 2024-04-16T02:35:58.820 INFO:tasks.workunit.client.0.smithi151.stdout:6/717: creat d2/d13/d63/def/df/d20/de6/ff1 x:0 0 0 2024-04-16T02:35:58.821 INFO:tasks.workunit.client.0.smithi151.stdout:6/718: write d2/d13/d63/def/df/d23/d82/d8d/fae [1368008,75905] 0 2024-04-16T02:35:58.824 INFO:tasks.workunit.client.0.smithi151.stdout:6/719: creat d2/d13/d3a/da6/ff2 x:0 0 0 2024-04-16T02:35:58.826 INFO:tasks.workunit.client.0.smithi151.stdout:6/720: write d2/d13/d63/def/df/d20/de6/fea [615554,65050] 0 2024-04-16T02:35:58.827 INFO:tasks.workunit.client.0.smithi151.stdout:6/721: chown d2/d13/d5f/dd7/cda 26927 1 2024-04-16T02:35:58.835 INFO:tasks.workunit.client.0.smithi151.stdout:6/722: link d2/d13/d63/def/df/d20/l42 d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d73/lf3 0 2024-04-16T02:35:58.838 INFO:tasks.workunit.client.0.smithi151.stdout:6/723: rename d2/f71 to d2/d13/d5f/ff4 0 2024-04-16T02:35:58.844 INFO:tasks.workunit.client.0.smithi151.stdout:6/724: dwrite d2/d13/d63/def/df/d23/d82/d8d/fae [0,4194304] 0 2024-04-16T02:35:58.849 INFO:tasks.workunit.client.0.smithi151.stdout:6/725: creat d2/d13/d63/def/df/d23/d82/ff5 x:0 0 0 2024-04-16T02:35:58.851 INFO:tasks.workunit.client.0.smithi151.stdout:6/726: write d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/fb8 [999202,2645] 0 2024-04-16T02:35:58.853 INFO:tasks.workunit.client.0.smithi151.stdout:6/727: fsync d2/d13/d63/def/df/f48 0 2024-04-16T02:35:58.855 INFO:tasks.workunit.client.0.smithi151.stdout:6/728: chown d2/d13/d63/def/df/d23/d45/d62/caf 159 1 2024-04-16T02:35:58.856 INFO:tasks.workunit.client.0.smithi151.stdout:6/729: symlink d2/d13/d46/lf6 0 2024-04-16T02:35:58.958 INFO:tasks.workunit.client.0.smithi151.stdout:3/370: sync 2024-04-16T02:35:58.961 INFO:tasks.workunit.client.0.smithi151.stdout:3/371: mkdir d1/d6/d20/d78 0 2024-04-16T02:35:58.966 INFO:tasks.workunit.client.0.smithi151.stdout:3/372: dwrite d1/d6/d22/d23/f62 [4194304,4194304] 0 2024-04-16T02:35:58.996 INFO:tasks.workunit.client.0.smithi151.stdout:3/373: dread d1/d6/d22/d23/f70 [0,4194304] 0 2024-04-16T02:35:58.998 INFO:tasks.workunit.client.0.smithi151.stdout:3/374: chown d1/d6/d20/f68 10161 1 2024-04-16T02:35:58.999 INFO:tasks.workunit.client.0.smithi151.stdout:3/375: write d1/d6/d22/f2b [833821,16468] 0 2024-04-16T02:35:58.999 INFO:tasks.workunit.client.0.smithi151.stdout:3/376: readlink d1/d6/l18 0 2024-04-16T02:35:59.000 INFO:tasks.workunit.client.0.smithi151.stdout:3/377: chown d1/d6/d4a/d4e 404 1 2024-04-16T02:35:59.007 INFO:tasks.workunit.client.0.smithi151.stdout:3/378: dwrite d1/d6/d20/d3d/f73 [0,4194304] 0 2024-04-16T02:35:59.011 INFO:tasks.workunit.client.0.smithi151.stdout:3/379: rename d1/d6/d20/l32 to d1/d6/d20/d3d/l79 0 2024-04-16T02:35:59.013 INFO:tasks.workunit.client.0.smithi151.stdout:3/380: chown d1/d6/d20/d2a/c53 170497457 1 2024-04-16T02:35:59.015 INFO:tasks.workunit.client.0.smithi151.stdout:3/381: unlink d1/f17 0 2024-04-16T02:35:59.019 INFO:tasks.workunit.client.0.smithi151.stdout:3/382: unlink d1/f59 0 2024-04-16T02:35:59.021 INFO:tasks.workunit.client.0.smithi151.stdout:3/383: mknod d1/d6/d20/d54/c7a 0 2024-04-16T02:35:59.023 INFO:tasks.workunit.client.0.smithi151.stdout:3/384: dread - d1/f1d zero size 2024-04-16T02:35:59.024 INFO:tasks.workunit.client.0.smithi151.stdout:3/385: truncate d1/d37/f38 2039671 0 2024-04-16T02:35:59.025 INFO:tasks.workunit.client.0.smithi151.stdout:3/386: creat d1/d6/d22/d23/d66/f7b x:0 0 0 2024-04-16T02:35:59.026 INFO:tasks.workunit.client.0.smithi151.stdout:3/387: chown d1/d6/d22/d23/d30/d44/d46 6043 1 2024-04-16T02:35:59.029 INFO:tasks.workunit.client.0.smithi151.stdout:3/388: creat d1/d6/d22/d23/d30/d44/d46/f7c x:0 0 0 2024-04-16T02:35:59.034 INFO:tasks.workunit.client.0.smithi151.stdout:3/389: creat d1/d6/d20/f7d x:0 0 0 2024-04-16T02:35:59.037 INFO:tasks.workunit.client.0.smithi151.stdout:3/390: rmdir d1/d6/d20/d78 0 2024-04-16T02:35:59.039 INFO:tasks.workunit.client.0.smithi151.stdout:3/391: creat d1/d69/f7e x:0 0 0 2024-04-16T02:35:59.041 INFO:tasks.workunit.client.0.smithi151.stdout:3/392: mkdir d1/d69/d7f 0 2024-04-16T02:35:59.043 INFO:tasks.workunit.client.0.smithi151.stdout:3/393: creat d1/d6/d20/d54/f80 x:0 0 0 2024-04-16T02:35:59.045 INFO:tasks.workunit.client.0.smithi151.stdout:3/394: creat d1/d6/d20/d2a/d3f/d43/d6d/f81 x:0 0 0 2024-04-16T02:35:59.055 INFO:tasks.workunit.client.0.smithi151.stdout:2/826: sync 2024-04-16T02:35:59.055 INFO:tasks.workunit.client.0.smithi151.stdout:7/656: sync 2024-04-16T02:35:59.057 INFO:tasks.workunit.client.0.smithi151.stdout:2/827: read de/f45 [1667439,110262] 0 2024-04-16T02:35:59.058 INFO:tasks.workunit.client.0.smithi151.stdout:2/828: readlink de/d15/d78/le7 0 2024-04-16T02:35:59.059 INFO:tasks.workunit.client.0.smithi151.stdout:7/657: mkdir d0/d2c/d38/de7 0 2024-04-16T02:35:59.062 INFO:tasks.workunit.client.0.smithi151.stdout:3/395: dwrite d1/d6/ff [0,4194304] 0 2024-04-16T02:35:59.064 INFO:tasks.workunit.client.0.smithi151.stdout:3/396: fsync d1/d69/f7e 0 2024-04-16T02:35:59.066 INFO:tasks.workunit.client.0.smithi151.stdout:7/658: unlink d0/d2c/l2f 0 2024-04-16T02:35:59.066 INFO:tasks.workunit.client.0.smithi151.stdout:7/659: chown d0/d1f/d99/ld0 0 1 2024-04-16T02:35:59.068 INFO:tasks.workunit.client.0.smithi151.stdout:2/829: getdents de/d65/dc0 0 2024-04-16T02:35:59.071 INFO:tasks.workunit.client.0.smithi151.stdout:3/397: symlink d1/d6/l82 0 2024-04-16T02:35:59.077 INFO:tasks.workunit.client.0.smithi151.stdout:7/660: creat d0/d1f/fe8 x:0 0 0 2024-04-16T02:35:59.078 INFO:tasks.workunit.client.0.smithi151.stdout:3/398: dwrite d1/d6/d22/d23/d66/f74 [0,4194304] 0 2024-04-16T02:35:59.079 INFO:tasks.workunit.client.0.smithi151.stdout:3/399: chown d1/d6/d4a/d4e 1576428 1 2024-04-16T02:35:59.087 INFO:tasks.workunit.client.0.smithi151.stdout:3/400: mknod d1/d37/c83 0 2024-04-16T02:35:59.089 INFO:tasks.workunit.client.0.smithi151.stdout:2/830: link de/d15/f17 de/d11/d42/f116 0 2024-04-16T02:35:59.095 INFO:tasks.workunit.client.0.smithi151.stdout:3/401: mknod d1/d6/d20/d3d/c84 0 2024-04-16T02:35:59.096 INFO:tasks.workunit.client.0.smithi151.stdout:2/831: unlink de/d65/d71/d7c/caf 0 2024-04-16T02:35:59.098 INFO:tasks.workunit.client.0.smithi151.stdout:3/402: creat d1/d6/d22/d23/d30/d44/f85 x:0 0 0 2024-04-16T02:35:59.098 INFO:tasks.workunit.client.0.smithi151.stdout:3/403: chown d1 962374228 1 2024-04-16T02:35:59.102 INFO:tasks.workunit.client.0.smithi151.stdout:7/661: write d0/d1f/d31/d42/f45 [2510922,111207] 0 2024-04-16T02:35:59.105 INFO:tasks.workunit.client.0.smithi151.stdout:2/832: dwrite de/d11/d42/d82/d95/f32 [0,4194304] 0 2024-04-16T02:35:59.109 INFO:tasks.workunit.client.0.smithi151.stdout:7/662: creat d0/d2c/d38/de7/fe9 x:0 0 0 2024-04-16T02:35:59.110 INFO:tasks.workunit.client.0.smithi151.stdout:3/404: truncate d1/d6/d22/f52 3264140 0 2024-04-16T02:35:59.113 INFO:tasks.workunit.client.0.smithi151.stdout:7/663: creat d0/d1f/d4c/d82/db3/fea x:0 0 0 2024-04-16T02:35:59.114 INFO:tasks.workunit.client.0.smithi151.stdout:2/833: symlink de/d65/d71/d98/dff/l117 0 2024-04-16T02:35:59.116 INFO:tasks.workunit.client.0.smithi151.stdout:3/405: rename d1/d6/d20/d2a/c61 to d1/d6/d22/c86 0 2024-04-16T02:35:59.119 INFO:tasks.workunit.client.0.smithi151.stdout:2/834: readlink de/d11/l113 0 2024-04-16T02:35:59.120 INFO:tasks.workunit.client.0.smithi151.stdout:7/664: dwrite d0/f18 [0,4194304] 0 2024-04-16T02:35:59.125 INFO:tasks.workunit.client.0.smithi151.stdout:7/665: rename d0/dc3 to d0/d2c/d38/d5e/deb 0 2024-04-16T02:35:59.126 INFO:tasks.workunit.client.0.smithi151.stdout:7/666: truncate d0/d1f/fe8 927385 0 2024-04-16T02:35:59.133 INFO:tasks.workunit.client.0.smithi151.stdout:2/835: dwrite de/f63 [0,4194304] 0 2024-04-16T02:35:59.135 INFO:tasks.workunit.client.0.smithi151.stdout:2/836: truncate de/d11/d42/d5e/d6c/d6d/f101 457301 0 2024-04-16T02:35:59.141 INFO:tasks.workunit.client.0.smithi151.stdout:2/837: rmdir de/d3d/d10d 0 2024-04-16T02:35:59.143 INFO:tasks.workunit.client.0.smithi151.stdout:1/579: sync 2024-04-16T02:35:59.143 INFO:tasks.workunit.client.0.smithi151.stdout:6/730: sync 2024-04-16T02:35:59.143 INFO:tasks.workunit.client.0.smithi151.stdout:5/688: sync 2024-04-16T02:35:59.143 INFO:tasks.workunit.client.0.smithi151.stdout:0/484: sync 2024-04-16T02:35:59.143 INFO:tasks.workunit.client.0.smithi151.stdout:4/604: sync 2024-04-16T02:35:59.143 INFO:tasks.workunit.client.0.smithi151.stdout:8/415: sync 2024-04-16T02:35:59.144 INFO:tasks.workunit.client.0.smithi151.stdout:9/588: sync 2024-04-16T02:35:59.146 INFO:tasks.workunit.client.0.smithi151.stdout:1/580: chown d3/d58/df/f6c 937 1 2024-04-16T02:35:59.146 INFO:tasks.workunit.client.0.smithi151.stdout:4/605: truncate d0/d64/f76 4987192 0 2024-04-16T02:35:59.147 INFO:tasks.workunit.client.0.smithi151.stdout:5/689: dread - d2/d6/d19/d62/f7b zero size 2024-04-16T02:35:59.147 INFO:tasks.workunit.client.0.smithi151.stdout:0/485: chown dc/d15/d34/f5d 0 1 2024-04-16T02:35:59.149 INFO:tasks.workunit.client.0.smithi151.stdout:4/606: read d0/d11/f3d [1613849,10720] 0 2024-04-16T02:35:59.150 INFO:tasks.workunit.client.0.smithi151.stdout:6/731: fsync d2/d13/d63/def/df/d23/d45/fd1 0 2024-04-16T02:35:59.152 INFO:tasks.workunit.client.0.smithi151.stdout:8/416: truncate d11/d2a/d33/f49 1857156 0 2024-04-16T02:35:59.153 INFO:tasks.workunit.client.0.smithi151.stdout:6/732: stat d2/d13/d63/def/df/d23/d82 0 2024-04-16T02:35:59.157 INFO:tasks.workunit.client.0.smithi151.stdout:0/486: mkdir dc/d15/d34/d67/d69/d9a 0 2024-04-16T02:35:59.158 INFO:tasks.workunit.client.0.smithi151.stdout:0/487: write dc/d15/d34/d67/d69/d6f/f99 [813879,102929] 0 2024-04-16T02:35:59.160 INFO:tasks.workunit.client.0.smithi151.stdout:2/838: dread de/d11/d42/d7a/d8b/db1/f24 [0,4194304] 0 2024-04-16T02:35:59.162 INFO:tasks.workunit.client.0.smithi151.stdout:9/589: dread d1/d3/d5a/f69 [0,4194304] 0 2024-04-16T02:35:59.163 INFO:tasks.workunit.client.0.smithi151.stdout:9/590: truncate d1/d3/d6/f74 5078026 0 2024-04-16T02:35:59.164 INFO:tasks.workunit.client.0.smithi151.stdout:9/591: write d1/d3/f5e [8461291,61019] 0 2024-04-16T02:35:59.165 INFO:tasks.workunit.client.0.smithi151.stdout:1/581: symlink d3/d58/df/d32/lc6 0 2024-04-16T02:35:59.167 INFO:tasks.workunit.client.0.smithi151.stdout:8/417: creat d11/d22/d79/f86 x:0 0 0 2024-04-16T02:35:59.169 INFO:tasks.workunit.client.0.smithi151.stdout:6/733: mkdir d2/d13/d63/def/df/d23/d31/d52/df7 0 2024-04-16T02:35:59.169 INFO:tasks.workunit.client.0.smithi151.stdout:6/734: stat d2/d13/d63/def/df/d23/d45/fd1 0 2024-04-16T02:35:59.170 INFO:tasks.workunit.client.0.smithi151.stdout:4/607: fdatasync d0/f82 0 2024-04-16T02:35:59.173 INFO:tasks.workunit.client.0.smithi151.stdout:2/839: mkdir de/d11/d42/d7a/d118 0 2024-04-16T02:35:59.174 INFO:tasks.workunit.client.0.smithi151.stdout:2/840: stat de/d11/d42/d5e/d6c/db8 0 2024-04-16T02:35:59.179 INFO:tasks.workunit.client.0.smithi151.stdout:5/690: truncate d2/d6/d19/d1a/f48 2643293 0 2024-04-16T02:35:59.180 INFO:tasks.workunit.client.0.smithi151.stdout:1/582: mkdir d3/d4/d64/d2a/d7a/dc7 0 2024-04-16T02:35:59.181 INFO:tasks.workunit.client.0.smithi151.stdout:1/583: chown d3/d58/df/c43 4 1 2024-04-16T02:35:59.182 INFO:tasks.workunit.client.0.smithi151.stdout:9/592: symlink d1/d3/d6/d28/d39/lc0 0 2024-04-16T02:35:59.183 INFO:tasks.workunit.client.0.smithi151.stdout:8/418: write d11/f1a [1370491,7979] 0 2024-04-16T02:35:59.194 INFO:tasks.workunit.client.0.smithi151.stdout:5/691: mkdir d2/d6/d19/d62/dd0/d9c/dd7 0 2024-04-16T02:35:59.197 INFO:tasks.workunit.client.0.smithi151.stdout:1/584: dread d3/d4/f96 [0,4194304] 0 2024-04-16T02:35:59.198 INFO:tasks.workunit.client.0.smithi151.stdout:1/585: stat d3/d4/d64/d2a/d52/fa9 0 2024-04-16T02:35:59.200 INFO:tasks.workunit.client.0.smithi151.stdout:0/488: write dc/d4c/f8e [3913715,110445] 0 2024-04-16T02:35:59.200 INFO:tasks.workunit.client.0.smithi151.stdout:1/586: chown d3/d4/d64/d2a/d7a/f86 237 1 2024-04-16T02:35:59.203 INFO:tasks.workunit.client.0.smithi151.stdout:4/608: dwrite d0/d99/d59/f85 [0,4194304] 0 2024-04-16T02:35:59.204 INFO:tasks.workunit.client.0.smithi151.stdout:4/609: chown d0/d11/d19/d31/d4a/d60 287 1 2024-04-16T02:35:59.209 INFO:tasks.workunit.client.0.smithi151.stdout:2/841: rename de/d3d/c64 to de/d11/d42/d5e/d6c/db8/dcc/d10f/c119 0 2024-04-16T02:35:59.210 INFO:tasks.workunit.client.0.smithi151.stdout:1/587: dwrite d3/d4/d64/fa4 [0,4194304] 0 2024-04-16T02:35:59.218 INFO:tasks.workunit.client.0.smithi151.stdout:6/735: link d2/d13/d63/def/df/d23/d45/d62/cc5 d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/cf8 0 2024-04-16T02:35:59.220 INFO:tasks.workunit.client.0.smithi151.stdout:2/842: dwrite de/d15/d78/fae [0,4194304] 0 2024-04-16T02:35:59.227 INFO:tasks.workunit.client.0.smithi151.stdout:9/593: rename d1/dd/f88 to d1/d3/d6/d17/d7a/fc1 0 2024-04-16T02:35:59.228 INFO:tasks.workunit.client.0.smithi151.stdout:9/594: chown d1/d3/d10/d7b/cac 7 1 2024-04-16T02:35:59.229 INFO:tasks.workunit.client.0.smithi151.stdout:9/595: chown d1/d3/d6/d28 2565 1 2024-04-16T02:35:59.241 INFO:tasks.workunit.client.0.smithi151.stdout:8/419: dwrite d11/f15 [0,4194304] 0 2024-04-16T02:35:59.241 INFO:tasks.workunit.client.0.smithi151.stdout:1/588: dread d3/d33/f39 [0,4194304] 0 2024-04-16T02:35:59.242 INFO:tasks.workunit.client.0.smithi151.stdout:0/489: rename dc/c24 to dc/d15/d34/d67/d69/d9a/c9b 0 2024-04-16T02:35:59.244 INFO:tasks.workunit.client.0.smithi151.stdout:8/420: read d11/d2a/f2f [2347072,60339] 0 2024-04-16T02:35:59.246 INFO:tasks.workunit.client.0.smithi151.stdout:4/610: link d0/d99/f42 d0/d9b/fd0 0 2024-04-16T02:35:59.250 INFO:tasks.workunit.client.0.smithi151.stdout:2/843: write de/d11/d42/d5e/d6c/f77 [7441263,16900] 0 2024-04-16T02:35:59.255 INFO:tasks.workunit.client.0.smithi151.stdout:6/736: dread d2/d13/d63/def/df/d20/d4f/f5e [0,4194304] 0 2024-04-16T02:35:59.255 INFO:tasks.workunit.client.0.smithi151.stdout:5/692: rename d2/d6/d19/d93/c9e to d2/d6/d19/d30/d49/d7f/cd8 0 2024-04-16T02:35:59.256 INFO:tasks.workunit.client.0.smithi151.stdout:1/589: symlink d3/d4/d64/d2a/d52/d97/lc8 0 2024-04-16T02:35:59.259 INFO:tasks.workunit.client.0.smithi151.stdout:5/693: dread d2/fb [0,4194304] 0 2024-04-16T02:35:59.260 INFO:tasks.workunit.client.0.smithi151.stdout:9/596: mkdir d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2 0 2024-04-16T02:35:59.261 INFO:tasks.workunit.client.0.smithi151.stdout:9/597: truncate d1/d75/fbf 449992 0 2024-04-16T02:35:59.261 INFO:tasks.workunit.client.0.smithi151.stdout:8/421: write d11/d2a/d35/d64/f58 [1315635,31529] 0 2024-04-16T02:35:59.264 INFO:tasks.workunit.client.0.smithi151.stdout:8/422: truncate d11/d22/d79/d78/f7b 850227 0 2024-04-16T02:35:59.264 INFO:tasks.workunit.client.0.smithi151.stdout:2/844: creat de/d11/d42/d6b/d79/f11a x:0 0 0 2024-04-16T02:35:59.265 INFO:tasks.workunit.client.0.smithi151.stdout:8/423: fdatasync d11/d2a/d33/f45 0 2024-04-16T02:35:59.265 INFO:tasks.workunit.client.0.smithi151.stdout:2/845: dread - de/d11/f7d zero size 2024-04-16T02:35:59.267 INFO:tasks.workunit.client.0.smithi151.stdout:6/737: dread - d2/d13/d63/def/d2e/db6/fc0 zero size 2024-04-16T02:35:59.268 INFO:tasks.workunit.client.0.smithi151.stdout:4/611: dread d0/d11/d19/d31/d89/f1d [0,4194304] 0 2024-04-16T02:35:59.273 INFO:tasks.workunit.client.0.smithi151.stdout:5/694: creat d2/d6/d19/d30/d88/fd9 x:0 0 0 2024-04-16T02:35:59.274 INFO:tasks.workunit.client.0.smithi151.stdout:9/598: fdatasync d1/d3/f33 0 2024-04-16T02:35:59.278 INFO:tasks.workunit.client.0.smithi151.stdout:9/599: truncate d1/d3/d6/f74 5982939 0 2024-04-16T02:35:59.282 INFO:tasks.workunit.client.0.smithi151.stdout:0/490: dwrite dc/d20/f94 [0,4194304] 0 2024-04-16T02:35:59.284 INFO:tasks.workunit.client.0.smithi151.stdout:1/590: dwrite d3/f9d [0,4194304] 0 2024-04-16T02:35:59.286 INFO:tasks.workunit.client.0.smithi151.stdout:2/846: rename de/d15/d78/f111 to de/d11/d42/d7a/d8b/f11b 0 2024-04-16T02:35:59.288 INFO:tasks.workunit.client.0.smithi151.stdout:2/847: chown de/d3d/de8/ce9 30731363 1 2024-04-16T02:35:59.291 INFO:tasks.workunit.client.0.smithi151.stdout:4/612: unlink d0/d11/d19/d87/c35 0 2024-04-16T02:35:59.294 INFO:tasks.workunit.client.0.smithi151.stdout:1/591: dread d3/d4/f96 [0,4194304] 0 2024-04-16T02:35:59.302 INFO:tasks.workunit.client.0.smithi151.stdout:6/738: dwrite d2/d13/f5c [0,4194304] 0 2024-04-16T02:35:59.308 INFO:tasks.workunit.client.0.smithi151.stdout:2/848: rename de/d65/d71/d98/dff to de/d11/d42/d6b/d11c 0 2024-04-16T02:35:59.311 INFO:tasks.workunit.client.0.smithi151.stdout:0/491: dwrite dc/d15/d34/d67/d69/f8b [0,4194304] 0 2024-04-16T02:35:59.314 INFO:tasks.workunit.client.0.smithi151.stdout:8/424: write d11/f38 [2800324,38705] 0 2024-04-16T02:35:59.319 INFO:tasks.workunit.client.0.smithi151.stdout:6/739: unlink d2/d13/d63/def/df/d23/d56/d72/fac 0 2024-04-16T02:35:59.322 INFO:tasks.workunit.client.0.smithi151.stdout:5/695: write d2/d6/d19/fc2 [6828,19133] 0 2024-04-16T02:35:59.325 INFO:tasks.workunit.client.0.smithi151.stdout:2/849: creat de/d11/d42/f11d x:0 0 0 2024-04-16T02:35:59.326 INFO:tasks.workunit.client.0.smithi151.stdout:2/850: write de/f27 [179957,42288] 0 2024-04-16T02:35:59.334 INFO:tasks.workunit.client.0.smithi151.stdout:9/600: dwrite d1/d3/d6/d17/d40/d56/f61 [0,4194304] 0 2024-04-16T02:35:59.334 INFO:tasks.workunit.client.0.smithi151.stdout:5/696: unlink d2/d6/f2a 0 2024-04-16T02:35:59.340 INFO:tasks.workunit.client.0.smithi151.stdout:1/592: creat d3/d58/df/fc9 x:0 0 0 2024-04-16T02:35:59.342 INFO:tasks.workunit.client.0.smithi151.stdout:2/851: rmdir de/d3d/de8 39 2024-04-16T02:35:59.344 INFO:tasks.workunit.client.0.smithi151.stdout:2/852: write de/d11/d42/d7a/fde [57716,85038] 0 2024-04-16T02:35:59.344 INFO:tasks.workunit.client.0.smithi151.stdout:0/492: mkdir dc/d20/d7b/d9c 0 2024-04-16T02:35:59.344 INFO:tasks.workunit.client.0.smithi151.stdout:2/853: chown de/d65/d88 0 1 2024-04-16T02:35:59.347 INFO:tasks.workunit.client.0.smithi151.stdout:4/613: rename d0/c94 to d0/d11/d19/d46/cd1 0 2024-04-16T02:35:59.349 INFO:tasks.workunit.client.0.smithi151.stdout:9/601: mkdir d1/d3/d10/da6/dc3 0 2024-04-16T02:35:59.351 INFO:tasks.workunit.client.0.smithi151.stdout:1/593: creat d3/d58/fca x:0 0 0 2024-04-16T02:35:59.352 INFO:tasks.workunit.client.0.smithi151.stdout:5/697: dread d2/d6/d19/d30/f39 [0,4194304] 0 2024-04-16T02:35:59.352 INFO:tasks.workunit.client.0.smithi151.stdout:6/740: creat d2/d13/d63/def/d2e/ff9 x:0 0 0 2024-04-16T02:35:59.353 INFO:tasks.workunit.client.0.smithi151.stdout:9/602: dread d1/d3/d6/f74 [4194304,4194304] 0 2024-04-16T02:35:59.355 INFO:tasks.workunit.client.0.smithi151.stdout:4/614: mkdir d0/d11/d19/d31/d45/dd2 0 2024-04-16T02:35:59.355 INFO:tasks.workunit.client.0.smithi151.stdout:5/698: dread - d2/d6/d19/d62/dd0/f89 zero size 2024-04-16T02:35:59.357 INFO:tasks.workunit.client.0.smithi151.stdout:6/741: chown d2/d13/d63/def/df/d23/d31/c92 341120 1 2024-04-16T02:35:59.357 INFO:tasks.workunit.client.0.smithi151.stdout:0/493: mkdir dc/d4c/d9d 0 2024-04-16T02:35:59.357 INFO:tasks.workunit.client.0.smithi151.stdout:9/603: read d1/d3/d6/d17/d7a/db1/fa1 [2686,49092] 0 2024-04-16T02:35:59.359 INFO:tasks.workunit.client.0.smithi151.stdout:0/494: write dc/d15/d34/f57 [621712,108915] 0 2024-04-16T02:35:59.360 INFO:tasks.workunit.client.0.smithi151.stdout:8/425: rename d11/f43 to d11/d7d/f87 0 2024-04-16T02:35:59.362 INFO:tasks.workunit.client.0.smithi151.stdout:1/594: truncate d3/d4/d64/d1c/f44 1220675 0 2024-04-16T02:35:59.373 INFO:tasks.workunit.client.0.smithi151.stdout:0/495: mkdir dc/d15/d34/d67/d69/d6f/d9e 0 2024-04-16T02:35:59.375 INFO:tasks.workunit.client.0.smithi151.stdout:8/426: creat d11/d2a/d35/d64/f88 x:0 0 0 2024-04-16T02:35:59.377 INFO:tasks.workunit.client.0.smithi151.stdout:1/595: unlink d3/d4/d64/d1c/l5c 0 2024-04-16T02:35:59.378 INFO:tasks.workunit.client.0.smithi151.stdout:1/596: write d3/f9d [1564731,81555] 0 2024-04-16T02:35:59.382 INFO:tasks.workunit.client.0.smithi151.stdout:9/604: mknod d1/dd/d4a/dbb/cc4 0 2024-04-16T02:35:59.382 INFO:tasks.workunit.client.0.smithi151.stdout:6/742: truncate d2/d13/d63/def/df/d20/d4f/f6e 546883 0 2024-04-16T02:35:59.382 INFO:tasks.workunit.client.0.smithi151.stdout:1/597: chown d3/d4/d64/d2a/d52/d97/lc8 98622 1 2024-04-16T02:35:59.384 INFO:tasks.workunit.client.0.smithi151.stdout:5/699: dwrite d2/d6/d19/d62/dd0/d9c/fa2 [0,4194304] 0 2024-04-16T02:35:59.397 INFO:tasks.workunit.client.0.smithi151.stdout:4/615: truncate d0/f3c 741637 0 2024-04-16T02:35:59.401 INFO:tasks.workunit.client.0.smithi151.stdout:8/427: mknod d11/d2a/d35/c89 0 2024-04-16T02:35:59.402 INFO:tasks.workunit.client.0.smithi151.stdout:8/428: write d11/d22/d2e/f65 [3271552,71757] 0 2024-04-16T02:35:59.403 INFO:tasks.workunit.client.0.smithi151.stdout:9/605: mknod d1/d3/d6/d17/d54/d8d/cc5 0 2024-04-16T02:35:59.405 INFO:tasks.workunit.client.0.smithi151.stdout:6/743: rename d2/d13/d70/cde to d2/d13/d5f/dd7/cfa 0 2024-04-16T02:35:59.406 INFO:tasks.workunit.client.0.smithi151.stdout:0/496: write dc/d15/d34/d67/f5f [1003044,10187] 0 2024-04-16T02:35:59.408 INFO:tasks.workunit.client.0.smithi151.stdout:5/700: mknod d2/d6/d19/d30/d36/cda 0 2024-04-16T02:35:59.408 INFO:tasks.workunit.client.0.smithi151.stdout:6/744: stat d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/fb8 0 2024-04-16T02:35:59.412 INFO:tasks.workunit.client.0.smithi151.stdout:6/745: chown d2/d13/d63/def/df/d23/d31/l35 9350263 1 2024-04-16T02:35:59.414 INFO:tasks.workunit.client.0.smithi151.stdout:9/606: chown d1/d3/f41 334 1 2024-04-16T02:35:59.416 INFO:tasks.workunit.client.0.smithi151.stdout:1/598: dwrite d3/d58/df/d32/f88 [0,4194304] 0 2024-04-16T02:35:59.418 INFO:tasks.workunit.client.0.smithi151.stdout:0/497: write dc/d15/d34/d67/d69/d8a/f95 [86018,22404] 0 2024-04-16T02:35:59.422 INFO:tasks.workunit.client.0.smithi151.stdout:5/701: dwrite d2/d6/d19/d30/d88/fd9 [0,4194304] 0 2024-04-16T02:35:59.429 INFO:tasks.workunit.client.0.smithi151.stdout:4/616: getdents d0/daa/dcc 0 2024-04-16T02:35:59.429 INFO:tasks.workunit.client.0.smithi151.stdout:6/746: creat d2/ffb x:0 0 0 2024-04-16T02:35:59.429 INFO:tasks.workunit.client.0.smithi151.stdout:6/747: chown d2/d13/d46 113778694 1 2024-04-16T02:35:59.432 INFO:tasks.workunit.client.0.smithi151.stdout:8/429: rename d11/c12 to d11/d2a/d83/c8a 0 2024-04-16T02:35:59.435 INFO:tasks.workunit.client.0.smithi151.stdout:0/498: stat dc/d15/d34/d67/l6c 0 2024-04-16T02:35:59.436 INFO:tasks.workunit.client.0.smithi151.stdout:0/499: fdatasync f3 0 2024-04-16T02:35:59.438 INFO:tasks.workunit.client.0.smithi151.stdout:6/748: dwrite d2/d13/d63/def/df/d20/de6/ff1 [0,4194304] 0 2024-04-16T02:35:59.457 INFO:tasks.workunit.client.0.smithi151.stdout:9/607: creat d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/fc6 x:0 0 0 2024-04-16T02:35:59.461 INFO:tasks.workunit.client.0.smithi151.stdout:5/702: dwrite d2/d6/d19/d30/d88/dc4/fc5 [0,4194304] 0 2024-04-16T02:35:59.461 INFO:tasks.workunit.client.0.smithi151.stdout:1/599: readlink d3/d58/df/d38/la1 0 2024-04-16T02:35:59.469 INFO:tasks.workunit.client.0.smithi151.stdout:0/500: mkdir dc/d3c/d9f 0 2024-04-16T02:35:59.471 INFO:tasks.workunit.client.0.smithi151.stdout:6/749: dwrite d2/d13/d3a/da6/fb3 [0,4194304] 0 2024-04-16T02:35:59.474 INFO:tasks.workunit.client.0.smithi151.stdout:9/608: creat d1/dd/d4a/dbb/fc7 x:0 0 0 2024-04-16T02:35:59.478 INFO:tasks.workunit.client.0.smithi151.stdout:5/703: chown d2/d6/d19/d1a/d4c/c73 371888904 1 2024-04-16T02:35:59.481 INFO:tasks.workunit.client.0.smithi151.stdout:4/617: rename d0/d11/d19/d31 to d0/d11/d19/dd3 0 2024-04-16T02:35:59.482 INFO:tasks.workunit.client.0.smithi151.stdout:6/750: dwrite d2/d13/d63/def/df/d23/d45/d62/fdc [0,4194304] 0 2024-04-16T02:35:59.484 INFO:tasks.workunit.client.0.smithi151.stdout:0/501: fdatasync dc/d15/d34/d3b/f70 0 2024-04-16T02:35:59.487 INFO:tasks.workunit.client.0.smithi151.stdout:8/430: dwrite d11/d2a/d33/f49 [0,4194304] 0 2024-04-16T02:35:59.487 INFO:tasks.workunit.client.0.smithi151.stdout:0/502: stat dc/d15/f2b 0 2024-04-16T02:35:59.493 INFO:tasks.workunit.client.0.smithi151.stdout:9/609: rename d1/d3/d5a/l91 to d1/d3/d6/d17/d40/d7d/lc8 0 2024-04-16T02:35:59.494 INFO:tasks.workunit.client.0.smithi151.stdout:1/600: write d3/d33/d10/f78 [1405489,81173] 0 2024-04-16T02:35:59.498 INFO:tasks.workunit.client.0.smithi151.stdout:8/431: dwrite d11/d2a/d35/d64/f7f [0,4194304] 0 2024-04-16T02:35:59.502 INFO:tasks.workunit.client.0.smithi151.stdout:0/503: symlink dc/d15/d34/d67/d69/d9a/la0 0 2024-04-16T02:35:59.508 INFO:tasks.workunit.client.0.smithi151.stdout:5/704: rename d2/d6/d19/d30/d36/d42/d6f/db2 to d2/d6/d19/d30/d36/d42/d6f/da8/dd3/ddb 0 2024-04-16T02:35:59.511 INFO:tasks.workunit.client.0.smithi151.stdout:6/751: unlink d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/l7b 0 2024-04-16T02:35:59.514 INFO:tasks.workunit.client.0.smithi151.stdout:0/504: fsync dc/d15/d34/d67/d69/d6f/f85 0 2024-04-16T02:35:59.519 INFO:tasks.workunit.client.0.smithi151.stdout:4/618: dwrite d0/f4c [4194304,4194304] 0 2024-04-16T02:35:59.526 INFO:tasks.workunit.client.0.smithi151.stdout:4/619: dwrite d0/d11/d19/dd3/f95 [0,4194304] 0 2024-04-16T02:35:59.528 INFO:tasks.workunit.client.0.smithi151.stdout:6/752: rmdir d2/d13/d63/def/df/d23/d45 39 2024-04-16T02:35:59.528 INFO:tasks.workunit.client.0.smithi151.stdout:5/705: write d2/d6/d19/d30/f3f [5230573,28930] 0 2024-04-16T02:35:59.530 INFO:tasks.workunit.client.0.smithi151.stdout:8/432: creat d11/d2a/d85/f8b x:0 0 0 2024-04-16T02:35:59.535 INFO:tasks.workunit.client.0.smithi151.stdout:0/505: unlink dc/d15/f23 0 2024-04-16T02:35:59.541 INFO:tasks.workunit.client.0.smithi151.stdout:6/753: rename d2/d13/ld3 to d2/d13/d63/def/df/d23/d82/d8d/dc8/lfc 0 2024-04-16T02:35:59.541 INFO:tasks.workunit.client.0.smithi151.stdout:7/667: sync 2024-04-16T02:35:59.541 INFO:tasks.workunit.client.0.smithi151.stdout:3/406: sync 2024-04-16T02:35:59.541 INFO:tasks.workunit.client.0.smithi151.stdout:2/854: sync 2024-04-16T02:35:59.543 INFO:tasks.workunit.client.0.smithi151.stdout:2/855: write de/d15/fa8 [2224286,48717] 0 2024-04-16T02:35:59.543 INFO:tasks.workunit.client.0.smithi151.stdout:3/407: chown d1/d6/d22/d23/f62 98067 1 2024-04-16T02:35:59.544 INFO:tasks.workunit.client.0.smithi151.stdout:7/668: dread d0/d1f/fe8 [0,4194304] 0 2024-04-16T02:35:59.546 INFO:tasks.workunit.client.0.smithi151.stdout:9/610: getdents d1/d3 0 2024-04-16T02:35:59.547 INFO:tasks.workunit.client.0.smithi151.stdout:0/506: creat dc/d20/d7b/fa1 x:0 0 0 2024-04-16T02:35:59.547 INFO:tasks.workunit.client.0.smithi151.stdout:2/856: truncate de/d11/d42/d5e/d6c/d6d/fec 851046 0 2024-04-16T02:35:59.550 INFO:tasks.workunit.client.0.smithi151.stdout:1/601: truncate d3/d4/d64/d2a/f65 3629271 0 2024-04-16T02:35:59.551 INFO:tasks.workunit.client.0.smithi151.stdout:1/602: dread - d3/d4/db2/fc3 zero size 2024-04-16T02:35:59.551 INFO:tasks.workunit.client.0.smithi151.stdout:4/620: dread d0/d11/d19/d87/d3a/d65/f7d [0,4194304] 0 2024-04-16T02:35:59.552 INFO:tasks.workunit.client.0.smithi151.stdout:1/603: dread - d3/d58/df/d32/dbd/fc2 zero size 2024-04-16T02:35:59.553 INFO:tasks.workunit.client.0.smithi151.stdout:5/706: rename d2/d6/d19/d30/d36/d42/d6f to d2/d6/d19/d30/d88/ddc 0 2024-04-16T02:35:59.561 INFO:tasks.workunit.client.0.smithi151.stdout:3/408: mkdir d1/d37/d87 0 2024-04-16T02:35:59.564 INFO:tasks.workunit.client.0.smithi151.stdout:0/507: mknod dc/d15/d34/d67/d69/d9a/ca2 0 2024-04-16T02:35:59.564 INFO:tasks.workunit.client.0.smithi151.stdout:0/508: dread - dc/d3c/f75 zero size 2024-04-16T02:35:59.567 INFO:tasks.workunit.client.0.smithi151.stdout:1/604: mkdir d3/d58/df/d32/d3a/d67/dcb 0 2024-04-16T02:35:59.569 INFO:tasks.workunit.client.0.smithi151.stdout:7/669: rename d0/d1f/d4c/d82/fa3 to d0/d1f/d60/fec 0 2024-04-16T02:35:59.570 INFO:tasks.workunit.client.0.smithi151.stdout:6/754: creat d2/d13/d6f/da3/ffd x:0 0 0 2024-04-16T02:35:59.571 INFO:tasks.workunit.client.0.smithi151.stdout:5/707: creat d2/d6/d19/d30/d36/d42/fdd x:0 0 0 2024-04-16T02:35:59.573 INFO:tasks.workunit.client.0.smithi151.stdout:8/433: creat d11/d2a/f8c x:0 0 0 2024-04-16T02:35:59.575 INFO:tasks.workunit.client.0.smithi151.stdout:3/409: dread d1/d6/f24 [0,4194304] 0 2024-04-16T02:35:59.580 INFO:tasks.workunit.client.0.smithi151.stdout:1/605: fsync d3/d58/df/d32/d3a/d71/f83 0 2024-04-16T02:35:59.585 INFO:tasks.workunit.client.0.smithi151.stdout:9/611: rename d1/d3/d5a/f69 to d1/d75/fc9 0 2024-04-16T02:35:59.585 INFO:tasks.workunit.client.0.smithi151.stdout:6/755: symlink d2/d13/d46/lfe 0 2024-04-16T02:35:59.586 INFO:tasks.workunit.client.0.smithi151.stdout:1/606: dread - d3/d4/d64/d2a/d73/fa2 zero size 2024-04-16T02:35:59.586 INFO:tasks.workunit.client.0.smithi151.stdout:9/612: fsync d1/d3/d6/d17/d7a/db1/fa3 0 2024-04-16T02:35:59.588 INFO:tasks.workunit.client.0.smithi151.stdout:2/857: creat de/d11/d42/d82/d95/f11e x:0 0 0 2024-04-16T02:35:59.590 INFO:tasks.workunit.client.0.smithi151.stdout:8/434: mknod d11/d2a/d33/d4e/d62/d6b/c8d 0 2024-04-16T02:35:59.592 INFO:tasks.workunit.client.0.smithi151.stdout:0/509: write dc/d15/d34/d4e/f55 [177973,6846] 0 2024-04-16T02:35:59.602 INFO:tasks.workunit.client.0.smithi151.stdout:7/670: dwrite d0/d1f/f29 [0,4194304] 0 2024-04-16T02:35:59.617 INFO:tasks.workunit.client.0.smithi151.stdout:6/756: mkdir d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/dff 0 2024-04-16T02:35:59.623 INFO:tasks.workunit.client.0.smithi151.stdout:4/621: getdents d0/d11/d19/dd3/d45 0 2024-04-16T02:35:59.626 INFO:tasks.workunit.client.0.smithi151.stdout:7/671: fsync d0/d1f/d60/fec 0 2024-04-16T02:35:59.626 INFO:tasks.workunit.client.0.smithi151.stdout:5/708: link d2/d6/d19/d62/c8d d2/d6/d19/d1a/d4c/cde 0 2024-04-16T02:35:59.631 INFO:tasks.workunit.client.0.smithi151.stdout:0/510: dwrite dc/d20/f3d [0,4194304] 0 2024-04-16T02:35:59.632 INFO:tasks.workunit.client.0.smithi151.stdout:0/511: stat dc/d15/d34/d4e/l6e 0 2024-04-16T02:35:59.635 INFO:tasks.workunit.client.0.smithi151.stdout:4/622: creat d0/d73/fd4 x:0 0 0 2024-04-16T02:35:59.637 INFO:tasks.workunit.client.0.smithi151.stdout:5/709: mknod d2/d6/d19/d30/d88/ddc/d83/d8b/cdf 0 2024-04-16T02:35:59.639 INFO:tasks.workunit.client.0.smithi151.stdout:6/757: symlink d2/d13/d63/def/df/d23/l100 0 2024-04-16T02:35:59.642 INFO:tasks.workunit.client.0.smithi151.stdout:9/613: write d1/d3/d6/d17/d54/d9b/f89 [615323,85665] 0 2024-04-16T02:35:59.643 INFO:tasks.workunit.client.0.smithi151.stdout:7/672: symlink d0/d2c/led 0 2024-04-16T02:35:59.644 INFO:tasks.workunit.client.0.smithi151.stdout:3/410: write d1/d6/d20/d2a/f16 [3542695,53021] 0 2024-04-16T02:35:59.646 INFO:tasks.workunit.client.0.smithi151.stdout:4/623: symlink d0/d11/d19/d87/d3a/d65/ld5 0 2024-04-16T02:35:59.649 INFO:tasks.workunit.client.0.smithi151.stdout:1/607: rename d3/d1f/d48/f51 to d3/d58/df/d32/d3a/fcc 0 2024-04-16T02:35:59.651 INFO:tasks.workunit.client.0.smithi151.stdout:2/858: dwrite de/d15/f50 [0,4194304] 0 2024-04-16T02:35:59.653 INFO:tasks.workunit.client.0.smithi151.stdout:8/435: link d11/c1d d11/d2a/d33/d4e/c8e 0 2024-04-16T02:35:59.657 INFO:tasks.workunit.client.0.smithi151.stdout:0/512: symlink dc/d4c/d9d/la3 0 2024-04-16T02:35:59.660 INFO:tasks.workunit.client.0.smithi151.stdout:4/624: dwrite d0/f4c [0,4194304] 0 2024-04-16T02:35:59.663 INFO:tasks.workunit.client.0.smithi151.stdout:8/436: dwrite d11/d22/d2e/f65 [4194304,4194304] 0 2024-04-16T02:35:59.669 INFO:tasks.workunit.client.0.smithi151.stdout:6/758: write d2/d13/d63/def/df/d23/d82/f9c [898513,13437] 0 2024-04-16T02:35:59.671 INFO:tasks.workunit.client.0.smithi151.stdout:8/437: dwrite d11/d2a/d33/f4b [0,4194304] 0 2024-04-16T02:35:59.674 INFO:tasks.workunit.client.0.smithi151.stdout:7/673: rename d0/d2c/d38/d5e/d87 to d0/d2c/d38/d5e/dd1/dee 0 2024-04-16T02:35:59.677 INFO:tasks.workunit.client.0.smithi151.stdout:1/608: symlink d3/d33/d10/lcd 0 2024-04-16T02:35:59.686 INFO:tasks.workunit.client.0.smithi151.stdout:5/710: truncate d2/d6/d43/f6d 1481510 0 2024-04-16T02:35:59.689 INFO:tasks.workunit.client.0.smithi151.stdout:4/625: creat d0/d64/fd6 x:0 0 0 2024-04-16T02:35:59.690 INFO:tasks.workunit.client.0.smithi151.stdout:3/411: symlink d1/d6/d4a/d4e/d77/l88 0 2024-04-16T02:35:59.691 INFO:tasks.workunit.client.0.smithi151.stdout:9/614: dread d1/d3/d6/d17/d7a/f6d [0,4194304] 0 2024-04-16T02:35:59.692 INFO:tasks.workunit.client.0.smithi151.stdout:9/615: write d1/d3/d6/d28/d39/f48 [3650876,2298] 0 2024-04-16T02:35:59.693 INFO:tasks.workunit.client.0.smithi151.stdout:8/438: mknod d11/d2a/d85/c8f 0 2024-04-16T02:35:59.695 INFO:tasks.workunit.client.0.smithi151.stdout:9/616: read d1/d3/d6/d17/d54/d9b/d22/d36/f3d [335905,42297] 0 2024-04-16T02:35:59.696 INFO:tasks.workunit.client.0.smithi151.stdout:6/759: rename d2/d13/d63/def/df/d20/d4f/d81 to d2/d13/d63/def/df/d20/d101 0 2024-04-16T02:35:59.721 INFO:tasks.workunit.client.0.smithi151.stdout:1/609: symlink d3/d4/d64/d2a/d52/d97/lce 0 2024-04-16T02:35:59.721 INFO:tasks.workunit.client.0.smithi151.stdout:2/859: creat de/df2/f11f x:0 0 0 2024-04-16T02:35:59.725 INFO:tasks.workunit.client.0.smithi151.stdout:7/674: write d0/d2c/f3b [3904432,102984] 0 2024-04-16T02:35:59.726 INFO:tasks.workunit.client.0.smithi151.stdout:0/513: write dc/d15/d34/d67/f3e [618365,10819] 0 2024-04-16T02:35:59.726 INFO:tasks.workunit.client.0.smithi151.stdout:9/617: mknod d1/d3/d6/d17/d54/d9b/d22/d2c/d86/cca 0 2024-04-16T02:35:59.726 INFO:tasks.workunit.client.0.smithi151.stdout:8/439: dwrite d11/d22/d79/d78/f7b [0,4194304] 0 2024-04-16T02:35:59.727 INFO:tasks.workunit.client.0.smithi151.stdout:1/610: fsync d3/d58/df/f9e 0 2024-04-16T02:35:59.729 INFO:tasks.workunit.client.0.smithi151.stdout:1/611: readlink d3/d4/lbe 0 2024-04-16T02:35:59.729 INFO:tasks.workunit.client.0.smithi151.stdout:4/626: mkdir d0/d11/d19/dd7 0 2024-04-16T02:35:59.734 INFO:tasks.workunit.client.0.smithi151.stdout:3/412: mkdir d1/d89 0 2024-04-16T02:35:59.742 INFO:tasks.workunit.client.0.smithi151.stdout:0/514: mknod dc/d15/d34/d3b/ca4 0 2024-04-16T02:35:59.743 INFO:tasks.workunit.client.0.smithi151.stdout:8/440: creat d11/d2a/d85/f90 x:0 0 0 2024-04-16T02:35:59.744 INFO:tasks.workunit.client.0.smithi151.stdout:7/675: truncate d0/d27/f95 3527560 0 2024-04-16T02:35:59.746 INFO:tasks.workunit.client.0.smithi151.stdout:9/618: mknod d1/d3/d6/d17/d54/d58/ccb 0 2024-04-16T02:35:59.747 INFO:tasks.workunit.client.0.smithi151.stdout:4/627: creat d0/d9b/fd8 x:0 0 0 2024-04-16T02:35:59.749 INFO:tasks.workunit.client.0.smithi151.stdout:6/760: creat d2/d13/d6f/f102 x:0 0 0 2024-04-16T02:35:59.752 INFO:tasks.workunit.client.0.smithi151.stdout:7/676: creat d0/d2c/d38/db0/fef x:0 0 0 2024-04-16T02:35:59.753 INFO:tasks.workunit.client.0.smithi151.stdout:9/619: mknod d1/d75/ccc 0 2024-04-16T02:35:59.758 INFO:tasks.workunit.client.0.smithi151.stdout:0/515: unlink dc/f43 0 2024-04-16T02:35:59.761 INFO:tasks.workunit.client.0.smithi151.stdout:7/677: read - d0/d2c/d38/d5e/f92 zero size 2024-04-16T02:35:59.762 INFO:tasks.workunit.client.0.smithi151.stdout:7/678: readlink d0/d1f/d4c/d82/le1 0 2024-04-16T02:35:59.762 INFO:tasks.workunit.client.0.smithi151.stdout:9/620: fsync d1/d3/d10/f24 0 2024-04-16T02:35:59.765 INFO:tasks.workunit.client.0.smithi151.stdout:4/628: mknod d0/daa/dcc/cd9 0 2024-04-16T02:35:59.766 INFO:tasks.workunit.client.0.smithi151.stdout:7/679: read d0/d1f/d46/f8d [3965765,87242] 0 2024-04-16T02:35:59.767 INFO:tasks.workunit.client.0.smithi151.stdout:7/680: readlink d0/d1f/d46/l6b 0 2024-04-16T02:35:59.768 INFO:tasks.workunit.client.0.smithi151.stdout:7/681: chown d0/d1f/d94/fa1 6428015 1 2024-04-16T02:35:59.770 INFO:tasks.workunit.client.0.smithi151.stdout:7/682: readlink d0/d1f/d46/l6b 0 2024-04-16T02:35:59.776 INFO:tasks.workunit.client.0.smithi151.stdout:0/516: write dc/d4c/f96 [671024,10469] 0 2024-04-16T02:35:59.777 INFO:tasks.workunit.client.0.smithi151.stdout:6/761: creat d2/d13/d63/def/df/d23/d82/d8d/d22/d40/f103 x:0 0 0 2024-04-16T02:35:59.777 INFO:tasks.workunit.client.0.smithi151.stdout:7/683: mkdir d0/d27/da6/df0 0 2024-04-16T02:35:59.779 INFO:tasks.workunit.client.0.smithi151.stdout:4/629: dwrite d0/fbf [0,4194304] 0 2024-04-16T02:35:59.781 INFO:tasks.workunit.client.0.smithi151.stdout:7/684: write d0/d1f/d31/d42/f6c [1361712,7770] 0 2024-04-16T02:35:59.781 INFO:tasks.workunit.client.0.smithi151.stdout:4/630: fdatasync d0/f4c 0 2024-04-16T02:35:59.789 INFO:tasks.workunit.client.0.smithi151.stdout:7/685: creat d0/d2c/d38/d5e/dd1/ff1 x:0 0 0 2024-04-16T02:35:59.792 INFO:tasks.workunit.client.0.smithi151.stdout:6/762: dread d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/fd5 [0,4194304] 0 2024-04-16T02:35:59.795 INFO:tasks.workunit.client.0.smithi151.stdout:0/517: dwrite dc/d15/d34/d67/f5f [0,4194304] 0 2024-04-16T02:35:59.795 INFO:tasks.workunit.client.0.smithi151.stdout:7/686: mkdir d0/d2c/d38/db0/df2 0 2024-04-16T02:35:59.797 INFO:tasks.workunit.client.0.smithi151.stdout:2/860: write de/d11/d42/d82/d95/d4b/d60/fa7 [798955,76094] 0 2024-04-16T02:35:59.800 INFO:tasks.workunit.client.0.smithi151.stdout:4/631: getdents d0/d11/d19/d87/d3a/d65/d98 0 2024-04-16T02:35:59.805 INFO:tasks.workunit.client.0.smithi151.stdout:0/518: creat dc/d15/d34/d67/d69/d6f/fa5 x:0 0 0 2024-04-16T02:35:59.807 INFO:tasks.workunit.client.0.smithi151.stdout:0/519: write dc/d15/d34/f57 [786136,126101] 0 2024-04-16T02:35:59.809 INFO:tasks.workunit.client.0.smithi151.stdout:4/632: creat d0/d11/d19/dd3/d4a/d60/fda x:0 0 0 2024-04-16T02:35:59.816 INFO:tasks.workunit.client.0.smithi151.stdout:6/763: dread d2/d13/d63/def/f5a [0,4194304] 0 2024-04-16T02:35:59.818 INFO:tasks.workunit.client.0.smithi151.stdout:6/764: readlink d2/d13/d63/def/df/d23/l100 0 2024-04-16T02:35:59.820 INFO:tasks.workunit.client.0.smithi151.stdout:6/765: read - d2/d13/d63/def/d2e/db6/fc0 zero size 2024-04-16T02:35:59.849 INFO:tasks.workunit.client.0.smithi151.stdout:4/633: creat d0/d11/d19/d87/d3a/d50/fdb x:0 0 0 2024-04-16T02:35:59.852 INFO:tasks.workunit.client.0.smithi151.stdout:0/520: symlink dc/d3c/d9f/la6 0 2024-04-16T02:35:59.853 INFO:tasks.workunit.client.0.smithi151.stdout:0/521: readlink dc/d15/l31 0 2024-04-16T02:35:59.854 INFO:tasks.workunit.client.0.smithi151.stdout:4/634: creat d0/d11/d19/d87/d3a/d50/fdc x:0 0 0 2024-04-16T02:35:59.858 INFO:tasks.workunit.client.0.smithi151.stdout:6/766: creat d2/d13/d63/def/df/f104 x:0 0 0 2024-04-16T02:35:59.864 INFO:tasks.workunit.client.0.smithi151.stdout:3/413: truncate d1/d6/d22/f2b 677644 0 2024-04-16T02:35:59.868 INFO:tasks.workunit.client.0.smithi151.stdout:0/522: mknod dc/d20/d7b/d9c/ca7 0 2024-04-16T02:35:59.868 INFO:tasks.workunit.client.0.smithi151.stdout:6/767: dread d2/d13/d63/def/f6 [0,4194304] 0 2024-04-16T02:35:59.869 INFO:tasks.workunit.client.0.smithi151.stdout:8/441: write d11/f4a [4587949,10646] 0 2024-04-16T02:35:59.870 INFO:tasks.workunit.client.0.smithi151.stdout:8/442: write d11/d2a/d33/f66 [1333929,117688] 0 2024-04-16T02:35:59.875 INFO:tasks.workunit.client.0.smithi151.stdout:6/768: creat d2/d13/d63/def/d2e/db6/f105 x:0 0 0 2024-04-16T02:35:59.884 INFO:tasks.workunit.client.0.smithi151.stdout:8/443: dwrite d11/d7d/f87 [4194304,4194304] 0 2024-04-16T02:35:59.885 INFO:tasks.workunit.client.0.smithi151.stdout:8/444: chown d11/d2a/d33/d4e/d62/d6b/c8d 11577 1 2024-04-16T02:35:59.894 INFO:tasks.workunit.client.0.smithi151.stdout:2/861: write de/d65/f6f [1841297,68426] 0 2024-04-16T02:35:59.896 INFO:tasks.workunit.client.0.smithi151.stdout:7/687: dwrite d0/d1f/d31/f86 [0,4194304] 0 2024-04-16T02:35:59.905 INFO:tasks.workunit.client.0.smithi151.stdout:2/862: dwrite de/d11/d42/d6b/d79/fa9 [0,4194304] 0 2024-04-16T02:35:59.910 INFO:tasks.workunit.client.0.smithi151.stdout:7/688: dwrite d0/fe0 [0,4194304] 0 2024-04-16T02:35:59.913 INFO:tasks.workunit.client.0.smithi151.stdout:2/863: mknod de/d15/dcd/dd0/c120 0 2024-04-16T02:35:59.918 INFO:tasks.workunit.client.0.smithi151.stdout:7/689: mkdir d0/d27/da6/dcd/df3 0 2024-04-16T02:35:59.923 INFO:tasks.workunit.client.0.smithi151.stdout:2/864: creat de/d11/d42/d5e/d6c/f121 x:0 0 0 2024-04-16T02:35:59.928 INFO:tasks.workunit.client.0.smithi151.stdout:2/865: creat de/d65/d88/f122 x:0 0 0 2024-04-16T02:35:59.961 INFO:tasks.workunit.client.0.smithi151.stdout:0/523: link dc/d15/d34/d67/c33 dc/d15/d34/d67/d69/ca8 0 2024-04-16T02:35:59.962 INFO:tasks.workunit.client.0.smithi151.stdout:0/524: stat dc/d3c/f75 0 2024-04-16T02:35:59.963 INFO:tasks.workunit.client.0.smithi151.stdout:8/445: dwrite d11/f16 [0,4194304] 0 2024-04-16T02:35:59.964 INFO:tasks.workunit.client.0.smithi151.stdout:8/446: readlink d11/l21 0 2024-04-16T02:35:59.965 INFO:tasks.workunit.client.0.smithi151.stdout:8/447: fdatasync d11/f38 0 2024-04-16T02:35:59.969 INFO:tasks.workunit.client.0.smithi151.stdout:8/448: mknod d11/d22/d2e/c91 0 2024-04-16T02:35:59.971 INFO:tasks.workunit.client.0.smithi151.stdout:8/449: chown d11/d22/d79/d78/f7b 3015 1 2024-04-16T02:35:59.972 INFO:tasks.workunit.client.0.smithi151.stdout:0/525: link dc/d15/d34/d67/f30 dc/d15/d34/d67/d69/d6f/fa9 0 2024-04-16T02:35:59.973 INFO:tasks.workunit.client.0.smithi151.stdout:0/526: fsync dc/d15/d34/d67/f3e 0 2024-04-16T02:35:59.974 INFO:tasks.workunit.client.0.smithi151.stdout:0/527: write dc/d15/f17 [71512,64221] 0 2024-04-16T02:35:59.977 INFO:tasks.workunit.client.0.smithi151.stdout:0/528: truncate dc/d15/d34/d67/d51/f62 1242452 0 2024-04-16T02:35:59.978 INFO:tasks.workunit.client.0.smithi151.stdout:4/635: rename d0/d11/d19/dd3/c5e to d0/d11/d19/d87/d3a/cdd 0 2024-04-16T02:35:59.979 INFO:tasks.workunit.client.0.smithi151.stdout:0/529: readlink dc/l79 0 2024-04-16T02:35:59.981 INFO:tasks.workunit.client.0.smithi151.stdout:0/530: unlink dc/d20/f49 0 2024-04-16T02:35:59.983 INFO:tasks.workunit.client.0.smithi151.stdout:0/531: readlink dc/l5b 0 2024-04-16T02:35:59.984 INFO:tasks.workunit.client.0.smithi151.stdout:0/532: mknod dc/d20/d7b/d9c/caa 0 2024-04-16T02:35:59.985 INFO:tasks.workunit.client.0.smithi151.stdout:0/533: chown dc/d15/l22 0 1 2024-04-16T02:35:59.985 INFO:tasks.workunit.client.0.smithi151.stdout:3/414: rename d1/d6/d20/d2a to d1/d6/d22/d23/d30/d44/d46/d8a 0 2024-04-16T02:35:59.988 INFO:tasks.workunit.client.0.smithi151.stdout:3/415: write d1/d6/d22/d23/d66/f7b [178049,39346] 0 2024-04-16T02:35:59.988 INFO:tasks.workunit.client.0.smithi151.stdout:6/769: rename d2/d13/d63/def/f5a to d2/d13/d63/def/df/d23/d31/d52/df7/f106 0 2024-04-16T02:35:59.992 INFO:tasks.workunit.client.0.smithi151.stdout:2/866: dwrite de/d11/d42/d5e/f8d [0,4194304] 0 2024-04-16T02:35:59.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:35:59 smithi151 ceph-mon[29003]: pgmap v27: 65 pgs: 65 active+clean; 3.2 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 212 MiB/s wr, 483 op/s 2024-04-16T02:35:59.998 INFO:tasks.workunit.client.0.smithi151.stdout:7/690: rename d0/d1f/d33/f36 to d0/d2c/d38/dab/ff4 0 2024-04-16T02:36:00.006 INFO:tasks.workunit.client.0.smithi151.stdout:2/867: mkdir de/d11/d123 0 2024-04-16T02:36:00.007 INFO:tasks.workunit.client.0.smithi151.stdout:2/868: chown de/d11/d42/d7a/d8b/db1/fc6 6274 1 2024-04-16T02:36:00.010 INFO:tasks.workunit.client.0.smithi151.stdout:8/450: rename d11/d22/d79 to d11/d22/d6f/d92 0 2024-04-16T02:36:00.010 INFO:tasks.workunit.client.0.smithi151.stdout:7/691: fdatasync d0/d1f/d46/f8d 0 2024-04-16T02:36:00.013 INFO:tasks.workunit.client.0.smithi151.stdout:7/692: dread d0/d1f/d94/fa1 [0,4194304] 0 2024-04-16T02:36:00.013 INFO:tasks.workunit.client.0.smithi151.stdout:3/416: link d1/d6/d22/f2c d1/d37/d87/f8b 0 2024-04-16T02:36:00.014 INFO:tasks.workunit.client.0.smithi151.stdout:3/417: truncate d1/f26 5226219 0 2024-04-16T02:36:00.015 INFO:tasks.workunit.client.0.smithi151.stdout:2/869: rmdir de/d11/d42/d82/d95 39 2024-04-16T02:36:00.018 INFO:tasks.workunit.client.0.smithi151.stdout:0/534: creat dc/d15/d34/fab x:0 0 0 2024-04-16T02:36:00.025 INFO:tasks.workunit.client.0.smithi151.stdout:0/535: mknod dc/d15/d34/d4e/cac 0 2024-04-16T02:36:00.026 INFO:tasks.workunit.client.0.smithi151.stdout:3/418: mknod d1/d6/d22/d6c/c8c 0 2024-04-16T02:36:00.028 INFO:tasks.workunit.client.0.smithi151.stdout:3/419: truncate d1/d6/d22/d23/d66/f74 4262600 0 2024-04-16T02:36:00.029 INFO:tasks.workunit.client.0.smithi151.stdout:3/420: chown d1/d6/ff 244 1 2024-04-16T02:36:00.029 INFO:tasks.workunit.client.0.smithi151.stdout:4/636: rename d0/d11/d19/dd3 to d0/dde 0 2024-04-16T02:36:00.036 INFO:tasks.workunit.client.0.smithi151.stdout:0/536: dread dc/d15/d34/d3b/d71/f74 [0,4194304] 0 2024-04-16T02:36:00.036 INFO:tasks.workunit.client.0.smithi151.stdout:3/421: dread d1/f3e [0,4194304] 0 2024-04-16T02:36:00.036 INFO:tasks.workunit.client.0.smithi151.stdout:4/637: symlink d0/d11/d19/d87/d3a/d65/d98/ldf 0 2024-04-16T02:36:00.038 INFO:tasks.workunit.client.0.smithi151.stdout:7/693: link d0/d2c/d38/d5e/dd1/ff1 d0/d1f/d33/d89/ff5 0 2024-04-16T02:36:00.044 INFO:tasks.workunit.client.0.smithi151.stdout:3/422: creat d1/d6/d22/d23/d41/f8d x:0 0 0 2024-04-16T02:36:00.044 INFO:tasks.workunit.client.0.smithi151.stdout:6/770: dwrite d2/d13/d63/def/df/d23/d82/d8d/f4b [0,4194304] 0 2024-04-16T02:36:00.048 INFO:tasks.workunit.client.0.smithi151.stdout:0/537: rename dc/l65 to dc/d3c/d9f/lad 0 2024-04-16T02:36:00.049 INFO:tasks.workunit.client.0.smithi151.stdout:8/451: write d11/d22/f27 [6083968,82628] 0 2024-04-16T02:36:00.049 INFO:tasks.workunit.client.0.smithi151.stdout:7/694: read d0/d1f/d46/f5d [167911,109948] 0 2024-04-16T02:36:00.051 INFO:tasks.workunit.client.0.smithi151.stdout:8/452: readlink d11/d2a/d35/d64/l68 0 2024-04-16T02:36:00.052 INFO:tasks.workunit.client.0.smithi151.stdout:4/638: mknod d0/d11/d19/ce0 0 2024-04-16T02:36:00.057 INFO:tasks.workunit.client.0.smithi151.stdout:2/870: truncate de/d65/d71/d7c/fc9 2596529 0 2024-04-16T02:36:00.062 INFO:tasks.workunit.client.0.smithi151.stdout:0/538: symlink dc/d15/d34/d67/d69/d9a/lae 0 2024-04-16T02:36:00.064 INFO:tasks.workunit.client.0.smithi151.stdout:7/695: dwrite d0/d2c/f3b [4194304,4194304] 0 2024-04-16T02:36:00.066 INFO:tasks.workunit.client.0.smithi151.stdout:8/453: dwrite d11/d22/d6f/d92/f86 [0,4194304] 0 2024-04-16T02:36:00.071 INFO:tasks.workunit.client.0.smithi151.stdout:2/871: truncate de/d11/d42/d7a/d8b/f105 55963 0 2024-04-16T02:36:00.074 INFO:tasks.workunit.client.0.smithi151.stdout:6/771: dwrite d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d65/fc6 [0,4194304] 0 2024-04-16T02:36:00.075 INFO:tasks.workunit.client.0.smithi151.stdout:7/696: mknod d0/d2c/d38/db0/cf6 0 2024-04-16T02:36:00.075 INFO:tasks.workunit.client.0.smithi151.stdout:0/539: read dc/d15/d34/d67/f5e [127710,107021] 0 2024-04-16T02:36:00.076 INFO:tasks.workunit.client.0.smithi151.stdout:3/423: getdents d1/d6/d20 0 2024-04-16T02:36:00.078 INFO:tasks.workunit.client.0.smithi151.stdout:4/639: rename d0/d11/d19/d87/d3a/d65/ld5 to d0/d11/d19/d87/d3a/le1 0 2024-04-16T02:36:00.084 INFO:tasks.workunit.client.0.smithi151.stdout:4/640: write d0/d11/d19/d87/d3a/d65/d98/fbc [499856,66538] 0 2024-04-16T02:36:00.084 INFO:tasks.workunit.client.0.smithi151.stdout:8/454: dwrite d11/d22/f76 [0,4194304] 0 2024-04-16T02:36:00.088 INFO:tasks.workunit.client.0.smithi151.stdout:6/772: creat d2/d13/d3a/f107 x:0 0 0 2024-04-16T02:36:00.092 INFO:tasks.workunit.client.0.smithi151.stdout:0/540: creat dc/d15/d5c/faf x:0 0 0 2024-04-16T02:36:00.093 INFO:tasks.workunit.client.0.smithi151.stdout:0/541: chown dc 12419 1 2024-04-16T02:36:00.095 INFO:tasks.workunit.client.0.smithi151.stdout:2/872: dwrite de/d65/d88/dfd/f106 [0,4194304] 0 2024-04-16T02:36:00.096 INFO:tasks.workunit.client.0.smithi151.stdout:3/424: read d1/d6/d20/d3d/f73 [1603657,3266] 0 2024-04-16T02:36:00.097 INFO:tasks.workunit.client.0.smithi151.stdout:7/697: unlink d0/d1f/d4c/d82/db3/f9c 0 2024-04-16T02:36:00.099 INFO:tasks.workunit.client.0.smithi151.stdout:4/641: creat d0/d64/fe2 x:0 0 0 2024-04-16T02:36:00.099 INFO:tasks.workunit.client.0.smithi151.stdout:8/455: creat d11/d22/d6f/d92/f93 x:0 0 0 2024-04-16T02:36:00.101 INFO:tasks.workunit.client.0.smithi151.stdout:8/456: dread - d11/d2a/d35/d64/f88 zero size 2024-04-16T02:36:00.101 INFO:tasks.workunit.client.0.smithi151.stdout:2/873: creat de/d11/d42/d6b/d11c/f124 x:0 0 0 2024-04-16T02:36:00.102 INFO:tasks.workunit.client.0.smithi151.stdout:7/698: write d0/d2c/f3b [1356440,101070] 0 2024-04-16T02:36:00.105 INFO:tasks.workunit.client.0.smithi151.stdout:3/425: creat d1/d89/f8e x:0 0 0 2024-04-16T02:36:00.105 INFO:tasks.workunit.client.0.smithi151.stdout:8/457: rmdir d11/d2a 39 2024-04-16T02:36:00.108 INFO:tasks.workunit.client.0.smithi151.stdout:4/642: symlink d0/d11/d19/d87/d3a/d50/dc7/le3 0 2024-04-16T02:36:00.108 INFO:tasks.workunit.client.0.smithi151.stdout:4/643: stat d0/f4c 0 2024-04-16T02:36:00.110 INFO:tasks.workunit.client.0.smithi151.stdout:3/426: creat d1/d6/f8f x:0 0 0 2024-04-16T02:36:00.111 INFO:tasks.workunit.client.0.smithi151.stdout:4/644: unlink d0/d11/d19/d87/l63 0 2024-04-16T02:36:00.112 INFO:tasks.workunit.client.0.smithi151.stdout:4/645: stat d0/dde/d45/f92 0 2024-04-16T02:36:00.113 INFO:tasks.workunit.client.0.smithi151.stdout:8/458: mkdir d11/d2a/d33/d4e/d62/d94 0 2024-04-16T02:36:00.113 INFO:tasks.workunit.client.0.smithi151.stdout:2/874: dread de/d11/d42/d5e/d6c/db8/dcc/fbd [0,4194304] 0 2024-04-16T02:36:00.113 INFO:tasks.workunit.client.0.smithi151.stdout:4/646: chown d0/l12 9 1 2024-04-16T02:36:00.114 INFO:tasks.workunit.client.0.smithi151.stdout:3/427: unlink d1/d6/d20/d3d/f73 0 2024-04-16T02:36:00.115 INFO:tasks.workunit.client.0.smithi151.stdout:2/875: dread - de/d11/d42/d6b/d79/f11a zero size 2024-04-16T02:36:00.117 INFO:tasks.workunit.client.0.smithi151.stdout:4/647: symlink d0/dde/d4a/d60/le4 0 2024-04-16T02:36:00.117 INFO:tasks.workunit.client.0.smithi151.stdout:3/428: mknod d1/d69/c90 0 2024-04-16T02:36:00.118 INFO:tasks.workunit.client.0.smithi151.stdout:3/429: chown d1/d69/c90 627094 1 2024-04-16T02:36:00.121 INFO:tasks.workunit.client.0.smithi151.stdout:2/876: chown de/d65/daa/dba/ccb 1709 1 2024-04-16T02:36:00.129 INFO:tasks.workunit.client.0.smithi151.stdout:4/648: dread d0/d99/f42 [0,4194304] 0 2024-04-16T02:36:00.129 INFO:tasks.workunit.client.0.smithi151.stdout:2/877: chown de/f49 8 1 2024-04-16T02:36:00.130 INFO:tasks.workunit.client.0.smithi151.stdout:6/773: write d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/fd5 [932750,14000] 0 2024-04-16T02:36:00.132 INFO:tasks.workunit.client.0.smithi151.stdout:7/699: dread d0/d1f/d33/f3a [4194304,4194304] 0 2024-04-16T02:36:00.132 INFO:tasks.workunit.client.0.smithi151.stdout:3/430: dread d1/d6/d22/d23/d30/d44/d46/d8a/f16 [0,4194304] 0 2024-04-16T02:36:00.133 INFO:tasks.workunit.client.0.smithi151.stdout:0/542: write dc/ff [143645,112554] 0 2024-04-16T02:36:00.136 INFO:tasks.workunit.client.0.smithi151.stdout:4/649: symlink d0/dde/d4a/dba/le5 0 2024-04-16T02:36:00.138 INFO:tasks.workunit.client.0.smithi151.stdout:7/700: fsync d0/f3 0 2024-04-16T02:36:00.141 INFO:tasks.workunit.client.0.smithi151.stdout:0/543: chown dc/c1e 64164038 1 2024-04-16T02:36:00.141 INFO:tasks.workunit.client.0.smithi151.stdout:8/459: write fb [8862991,124481] 0 2024-04-16T02:36:00.142 INFO:tasks.workunit.client.0.smithi151.stdout:0/544: chown dc/d15/l2a 7657845 1 2024-04-16T02:36:00.142 INFO:tasks.workunit.client.0.smithi151.stdout:8/460: dread - d11/d2a/d85/f90 zero size 2024-04-16T02:36:00.143 INFO:tasks.workunit.client.0.smithi151.stdout:4/650: rmdir d0/dde/d80 39 2024-04-16T02:36:00.145 INFO:tasks.workunit.client.0.smithi151.stdout:7/701: creat d0/d1f/d99/dc0/ff7 x:0 0 0 2024-04-16T02:36:00.146 INFO:tasks.workunit.client.0.smithi151.stdout:2/878: write de/d11/d42/d5e/f7b [558761,24547] 0 2024-04-16T02:36:00.147 INFO:tasks.workunit.client.0.smithi151.stdout:3/431: mknod d1/d6/c91 0 2024-04-16T02:36:00.149 INFO:tasks.workunit.client.0.smithi151.stdout:0/545: mkdir dc/d4c/d72/d76/db0 0 2024-04-16T02:36:00.151 INFO:tasks.workunit.client.0.smithi151.stdout:8/461: rename d11/d2a/l51 to d11/d2a/d83/l95 0 2024-04-16T02:36:00.154 INFO:tasks.workunit.client.0.smithi151.stdout:3/432: dwrite d1/d6/d22/d23/d30/d44/f85 [0,4194304] 0 2024-04-16T02:36:00.156 INFO:tasks.workunit.client.0.smithi151.stdout:2/879: symlink de/d11/d42/d7a/d8b/db1/d90/l125 0 2024-04-16T02:36:00.158 INFO:tasks.workunit.client.0.smithi151.stdout:4/651: write d0/f3 [138032,29817] 0 2024-04-16T02:36:00.158 INFO:tasks.workunit.client.0.smithi151.stdout:6/774: getdents d2/d13/d5f 0 2024-04-16T02:36:00.159 INFO:tasks.workunit.client.0.smithi151.stdout:0/546: creat dc/d3c/fb1 x:0 0 0 2024-04-16T02:36:00.160 INFO:tasks.workunit.client.0.smithi151.stdout:0/547: fsync dc/d15/d34/d67/f78 0 2024-04-16T02:36:00.166 INFO:tasks.workunit.client.0.smithi151.stdout:2/880: rename de/d11/d42/f11d to de/d11/d42/d5e/d6c/db8/f126 0 2024-04-16T02:36:00.167 INFO:tasks.workunit.client.0.smithi151.stdout:4/652: fdatasync d0/d99/f4f 0 2024-04-16T02:36:00.168 INFO:tasks.workunit.client.0.smithi151.stdout:4/653: stat d0/f4c 0 2024-04-16T02:36:00.170 INFO:tasks.workunit.client.0.smithi151.stdout:6/775: rmdir d2/d13/d63/def/df 39 2024-04-16T02:36:00.170 INFO:tasks.workunit.client.0.smithi151.stdout:7/702: dread d0/d27/f2b [0,4194304] 0 2024-04-16T02:36:00.174 INFO:tasks.workunit.client.0.smithi151.stdout:8/462: write f2 [2869287,6286] 0 2024-04-16T02:36:00.183 INFO:tasks.workunit.client.0.smithi151.stdout:3/433: dwrite d1/d37/d87/f8b [0,4194304] 0 2024-04-16T02:36:00.184 INFO:tasks.workunit.client.0.smithi151.stdout:3/434: fdatasync d1/d37/f38 0 2024-04-16T02:36:00.187 INFO:tasks.workunit.client.0.smithi151.stdout:0/548: rename dc/c21 to dc/d15/d34/d67/cb2 0 2024-04-16T02:36:00.188 INFO:tasks.workunit.client.0.smithi151.stdout:0/549: chown dc/d15/d34/d3b/c52 614 1 2024-04-16T02:36:00.190 INFO:tasks.workunit.client.0.smithi151.stdout:2/881: truncate de/d11/d42/d7a/d8b/db1/fc6 1336156 0 2024-04-16T02:36:00.191 INFO:tasks.workunit.client.0.smithi151.stdout:3/435: symlink d1/d6/d22/d23/d30/d44/d46/d8a/d3f/l92 0 2024-04-16T02:36:00.195 INFO:tasks.workunit.client.0.smithi151.stdout:4/654: rename d0/dde/d89 to d0/dde/d4a/d60/de6 0 2024-04-16T02:36:00.196 INFO:tasks.workunit.client.0.smithi151.stdout:8/463: getdents d11/d2a/d33/d4e/d62/d6b 0 2024-04-16T02:36:00.206 INFO:tasks.workunit.client.0.smithi151.stdout:2/882: write de/d11/d42/d5e/f86 [573019,37360] 0 2024-04-16T02:36:00.209 INFO:tasks.workunit.client.0.smithi151.stdout:6/776: rename d2/d13/d63/def/df/d23/d45/d62/cc5 to d2/d13/d63/def/d2e/dba/c108 0 2024-04-16T02:36:00.210 INFO:tasks.workunit.client.0.smithi151.stdout:4/655: creat d0/d11/d19/dd7/fe7 x:0 0 0 2024-04-16T02:36:00.219 INFO:tasks.workunit.client.0.smithi151.stdout:6/777: mknod d2/d13/d3a/c109 0 2024-04-16T02:36:00.221 INFO:tasks.workunit.client.0.smithi151.stdout:8/464: truncate d11/d22/d6f/d92/d78/f7b 3536810 0 2024-04-16T02:36:00.223 INFO:tasks.workunit.client.0.smithi151.stdout:2/883: link de/d11/ffa de/d15/dcd/f127 0 2024-04-16T02:36:00.226 INFO:tasks.workunit.client.0.smithi151.stdout:3/436: getdents d1/d6/d22/d23/d30 0 2024-04-16T02:36:00.229 INFO:tasks.workunit.client.0.smithi151.stdout:0/550: dwrite dc/d15/d34/d67/d69/d6f/fa9 [4194304,4194304] 0 2024-04-16T02:36:00.230 INFO:tasks.workunit.client.0.smithi151.stdout:0/551: readlink dc/l79 0 2024-04-16T02:36:00.232 INFO:tasks.workunit.client.0.smithi151.stdout:2/884: mkdir de/d65/daa/dba/d128 0 2024-04-16T02:36:00.233 INFO:tasks.workunit.client.0.smithi151.stdout:4/656: creat d0/d11/d19/d87/d3a/fe8 x:0 0 0 2024-04-16T02:36:00.235 INFO:tasks.workunit.client.0.smithi151.stdout:3/437: rmdir d1/d6/d4a 39 2024-04-16T02:36:00.236 INFO:tasks.workunit.client.0.smithi151.stdout:3/438: chown d1/d6/d20 249488457 1 2024-04-16T02:36:00.237 INFO:tasks.workunit.client.0.smithi151.stdout:6/778: symlink d2/d13/d63/def/df/d20/d4f/l10a 0 2024-04-16T02:36:00.240 INFO:tasks.workunit.client.0.smithi151.stdout:0/552: creat dc/d15/d34/d3b/d71/fb3 x:0 0 0 2024-04-16T02:36:00.242 INFO:tasks.workunit.client.0.smithi151.stdout:6/779: dread d2/d13/d63/def/df/d23/d45/d62/fdc [0,4194304] 0 2024-04-16T02:36:00.245 INFO:tasks.workunit.client.0.smithi151.stdout:8/465: rename d11/f5b to d11/d22/f96 0 2024-04-16T02:36:00.245 INFO:tasks.workunit.client.0.smithi151.stdout:3/439: rename d1/d6/d22/d23 to d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/d93 22 2024-04-16T02:36:00.247 INFO:tasks.workunit.client.0.smithi151.stdout:4/657: mknod d0/dde/d45/dd2/ce9 0 2024-04-16T02:36:00.252 INFO:tasks.workunit.client.0.smithi151.stdout:2/885: link de/d15/f89 de/f129 0 2024-04-16T02:36:00.253 INFO:tasks.workunit.client.0.smithi151.stdout:0/553: creat dc/d15/d34/d67/d51/fb4 x:0 0 0 2024-04-16T02:36:00.254 INFO:tasks.workunit.client.0.smithi151.stdout:4/658: dwrite d0/d11/d19/d87/f5a [0,4194304] 0 2024-04-16T02:36:00.257 INFO:tasks.workunit.client.0.smithi151.stdout:8/466: symlink d11/d2a/d33/d4e/d62/d94/l97 0 2024-04-16T02:36:00.259 INFO:tasks.workunit.client.0.smithi151.stdout:2/886: creat de/d11/d42/d5e/d6c/d6d/da5/f12a x:0 0 0 2024-04-16T02:36:00.266 INFO:tasks.workunit.client.0.smithi151.stdout:0/554: mkdir dc/d4c/d72/d76/db0/db5 0 2024-04-16T02:36:00.268 INFO:tasks.workunit.client.0.smithi151.stdout:2/887: mknod de/d11/d42/d5e/d6c/da2/c12b 0 2024-04-16T02:36:00.270 INFO:tasks.workunit.client.0.smithi151.stdout:8/467: getdents d11/d22/d6c/d84 0 2024-04-16T02:36:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:35:59 smithi153 ceph-mon[36747]: pgmap v27: 65 pgs: 65 active+clean; 3.2 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 212 MiB/s wr, 483 op/s 2024-04-16T02:36:00.272 INFO:tasks.workunit.client.0.smithi151.stdout:0/555: symlink dc/d20/d7b/d9c/lb6 0 2024-04-16T02:36:00.274 INFO:tasks.workunit.client.0.smithi151.stdout:0/556: chown dc/d15/d34/d3b/f70 235968655 1 2024-04-16T02:36:00.276 INFO:tasks.workunit.client.0.smithi151.stdout:2/888: mkdir de/d12c 0 2024-04-16T02:36:00.278 INFO:tasks.workunit.client.0.smithi151.stdout:4/659: getdents d0/d11/d19/d87/d3a 0 2024-04-16T02:36:00.279 INFO:tasks.workunit.client.0.smithi151.stdout:3/440: dwrite d1/fa [4194304,4194304] 0 2024-04-16T02:36:00.283 INFO:tasks.workunit.client.0.smithi151.stdout:8/468: link d11/d2a/d85/f8b d11/d2a/d35/d52/f98 0 2024-04-16T02:36:00.287 INFO:tasks.workunit.client.0.smithi151.stdout:0/557: truncate dc/d15/d34/d67/f78 1514341 0 2024-04-16T02:36:00.288 INFO:tasks.workunit.client.0.smithi151.stdout:2/889: fdatasync de/d11/d42/d82/d95/d4b/d60/fa7 0 2024-04-16T02:36:00.289 INFO:tasks.workunit.client.0.smithi151.stdout:2/890: dread - de/d11/d42/d82/ff7 zero size 2024-04-16T02:36:00.290 INFO:tasks.workunit.client.0.smithi151.stdout:8/469: creat d11/d2a/d85/f99 x:0 0 0 2024-04-16T02:36:00.291 INFO:tasks.workunit.client.0.smithi151.stdout:2/891: fsync de/d11/d42/d5e/d6c/f121 0 2024-04-16T02:36:00.293 INFO:tasks.workunit.client.0.smithi151.stdout:0/558: fdatasync dc/d3c/f7a 0 2024-04-16T02:36:00.296 INFO:tasks.workunit.client.0.smithi151.stdout:3/441: dread d1/d6/f50 [0,4194304] 0 2024-04-16T02:36:00.296 INFO:tasks.workunit.client.0.smithi151.stdout:4/660: unlink d0/d11/d19/d87/d3a/d61/fa6 0 2024-04-16T02:36:00.296 INFO:tasks.workunit.client.0.smithi151.stdout:4/661: chown d0/f4c 10521 1 2024-04-16T02:36:00.297 INFO:tasks.workunit.client.0.smithi151.stdout:2/892: creat de/d11/d42/d5e/d6c/d6d/da5/f12d x:0 0 0 2024-04-16T02:36:00.297 INFO:tasks.workunit.client.0.smithi151.stdout:4/662: stat d0/dde/d45/l5f 0 2024-04-16T02:36:00.298 INFO:tasks.workunit.client.0.smithi151.stdout:4/663: readlink d0/dde/d4a/l97 0 2024-04-16T02:36:00.299 INFO:tasks.workunit.client.0.smithi151.stdout:8/470: rename d11/d22/d2e/l4d to d11/d2a/d35/d64/l9a 0 2024-04-16T02:36:00.300 INFO:tasks.workunit.client.0.smithi151.stdout:8/471: write d11/f38 [87623,119188] 0 2024-04-16T02:36:00.304 INFO:tasks.workunit.client.0.smithi151.stdout:3/442: truncate d1/f2e 2927213 0 2024-04-16T02:36:00.305 INFO:tasks.workunit.client.0.smithi151.stdout:3/443: truncate d1/d89/f8e 187231 0 2024-04-16T02:36:00.306 INFO:tasks.workunit.client.0.smithi151.stdout:2/893: symlink de/d15/dcd/l12e 0 2024-04-16T02:36:00.306 INFO:tasks.workunit.client.0.smithi151.stdout:3/444: write d1/d6/d22/d23/d30/d44/d46/d8a/f15 [339442,109926] 0 2024-04-16T02:36:00.312 INFO:tasks.workunit.client.0.smithi151.stdout:3/445: symlink d1/d6/d22/d23/d30/l94 0 2024-04-16T02:36:00.313 INFO:tasks.workunit.client.0.smithi151.stdout:8/472: mknod d11/c9b 0 2024-04-16T02:36:00.314 INFO:tasks.workunit.client.0.smithi151.stdout:0/559: link dc/d15/f7f dc/d4c/d72/d76/db0/fb7 0 2024-04-16T02:36:00.315 INFO:tasks.workunit.client.0.smithi151.stdout:2/894: fsync de/d15/f17 0 2024-04-16T02:36:00.316 INFO:tasks.workunit.client.0.smithi151.stdout:2/895: write de/d11/d42/d5e/d6c/d6d/f92 [3347322,89997] 0 2024-04-16T02:36:00.317 INFO:tasks.workunit.client.0.smithi151.stdout:3/446: rename d1/d6/d22/d23/l34 to d1/d6/d22/d23/d30/d44/l95 0 2024-04-16T02:36:00.319 INFO:tasks.workunit.client.0.smithi151.stdout:3/447: chown d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/f72 0 1 2024-04-16T02:36:00.321 INFO:tasks.workunit.client.0.smithi151.stdout:4/664: getdents d0/dde/dad/dc8 0 2024-04-16T02:36:00.323 INFO:tasks.workunit.client.0.smithi151.stdout:0/560: dread dc/d15/d34/d67/f30 [4194304,4194304] 0 2024-04-16T02:36:00.323 INFO:tasks.workunit.client.0.smithi151.stdout:0/561: stat f3 0 2024-04-16T02:36:00.326 INFO:tasks.workunit.client.0.smithi151.stdout:2/896: symlink de/d11/d42/l12f 0 2024-04-16T02:36:00.328 INFO:tasks.workunit.client.0.smithi151.stdout:2/897: truncate de/d65/daa/dba/f115 738248 0 2024-04-16T02:36:00.328 INFO:tasks.workunit.client.0.smithi151.stdout:8/473: rename d11/d7d/f87 to d11/d2a/d83/f9c 0 2024-04-16T02:36:00.331 INFO:tasks.workunit.client.0.smithi151.stdout:3/448: mknod d1/d6/d20/d54/c96 0 2024-04-16T02:36:00.335 INFO:tasks.workunit.client.0.smithi151.stdout:0/562: read dc/d15/d34/f4d [2730203,108638] 0 2024-04-16T02:36:00.336 INFO:tasks.workunit.client.0.smithi151.stdout:0/563: stat dc/d20/c64 0 2024-04-16T02:36:00.337 INFO:tasks.workunit.client.0.smithi151.stdout:2/898: mkdir de/d11/d42/d5e/d6c/db8/dcc/d10f/d130 0 2024-04-16T02:36:00.479 INFO:tasks.workunit.client.0.smithi151.stdout:3/449: creat d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/f97 x:0 0 0 2024-04-16T02:36:00.479 INFO:tasks.workunit.client.0.smithi151.stdout:4/665: creat d0/d11/d19/d46/dcd/fea x:0 0 0 2024-04-16T02:36:00.481 INFO:tasks.workunit.client.0.smithi151.stdout:2/899: mkdir de/d65/daa/dba/d131 0 2024-04-16T02:36:00.484 INFO:tasks.workunit.client.0.smithi151.stdout:8/474: rename d11/d22/c29 to d11/d22/d7e/c9d 0 2024-04-16T02:36:00.486 INFO:tasks.workunit.client.0.smithi151.stdout:4/666: mknod d0/daa/ceb 0 2024-04-16T02:36:00.489 INFO:tasks.workunit.client.0.smithi151.stdout:8/475: creat d11/d2a/d33/d4e/f9e x:0 0 0 2024-04-16T02:36:00.489 INFO:tasks.workunit.client.0.smithi151.stdout:8/476: chown d11/d2a/d35/d52/f98 8234085 1 2024-04-16T02:36:00.490 INFO:tasks.workunit.client.0.smithi151.stdout:4/667: symlink d0/d99/lec 0 2024-04-16T02:36:00.491 INFO:tasks.workunit.client.0.smithi151.stdout:2/900: truncate de/d11/d42/d7a/d8b/db1/fc6 95874 0 2024-04-16T02:36:00.494 INFO:tasks.workunit.client.0.smithi151.stdout:4/668: creat d0/dde/d4a/fed x:0 0 0 2024-04-16T02:36:00.495 INFO:tasks.workunit.client.0.smithi151.stdout:8/477: rename d11/d2a/d35/f67 to d11/d2a/d85/f9f 0 2024-04-16T02:36:00.497 INFO:tasks.workunit.client.0.smithi151.stdout:3/450: getdents d1/d6/d20 0 2024-04-16T02:36:00.498 INFO:tasks.workunit.client.0.smithi151.stdout:5/711: sync 2024-04-16T02:36:00.500 INFO:tasks.workunit.client.0.smithi151.stdout:3/451: mkdir d1/d6/d22/d23/d41/d98 0 2024-04-16T02:36:00.501 INFO:tasks.workunit.client.0.smithi151.stdout:3/452: write d1/d6/f24 [3501284,123824] 0 2024-04-16T02:36:00.503 INFO:tasks.workunit.client.0.smithi151.stdout:8/478: dread d11/f1a [0,4194304] 0 2024-04-16T02:36:00.507 INFO:tasks.workunit.client.0.smithi151.stdout:4/669: getdents d0/d99/d59 0 2024-04-16T02:36:00.508 INFO:tasks.workunit.client.0.smithi151.stdout:3/453: mknod d1/d69/d7f/c99 0 2024-04-16T02:36:00.508 INFO:tasks.workunit.client.0.smithi151.stdout:4/670: write d0/d64/fe2 [791870,13976] 0 2024-04-16T02:36:00.511 INFO:tasks.workunit.client.0.smithi151.stdout:4/671: creat d0/dde/d45/dd2/fee x:0 0 0 2024-04-16T02:36:00.511 INFO:tasks.workunit.client.0.smithi151.stdout:2/901: dread de/d65/f112 [0,4194304] 0 2024-04-16T02:36:00.512 INFO:tasks.workunit.client.0.smithi151.stdout:4/672: dread - d0/d99/fbd zero size 2024-04-16T02:36:00.512 INFO:tasks.workunit.client.0.smithi151.stdout:4/673: fdatasync d0/dde/fa2 0 2024-04-16T02:36:00.513 INFO:tasks.workunit.client.0.smithi151.stdout:8/479: mknod d11/d2a/d35/ca0 0 2024-04-16T02:36:00.514 INFO:tasks.workunit.client.0.smithi151.stdout:3/454: mknod d1/d6/d22/d23/d30/d44/d63/c9a 0 2024-04-16T02:36:00.519 INFO:tasks.workunit.client.0.smithi151.stdout:4/674: mkdir d0/d11/d19/d46/dcd/def 0 2024-04-16T02:36:00.522 INFO:tasks.workunit.client.0.smithi151.stdout:4/675: write d0/d11/f8d [362777,67587] 0 2024-04-16T02:36:00.525 INFO:tasks.workunit.client.0.smithi151.stdout:8/480: creat d11/d22/fa1 x:0 0 0 2024-04-16T02:36:00.526 INFO:tasks.workunit.client.0.smithi151.stdout:3/455: link d1/d6/d22/d23/d30/d44/d46/d8a/f28 d1/d69/f9b 0 2024-04-16T02:36:00.530 INFO:tasks.workunit.client.0.smithi151.stdout:3/456: creat d1/d6/d22/d6c/f9c x:0 0 0 2024-04-16T02:36:00.533 INFO:tasks.workunit.client.0.smithi151.stdout:8/481: rename d11/d2a/d35/l59 to d11/d2a/la2 0 2024-04-16T02:36:00.534 INFO:tasks.workunit.client.0.smithi151.stdout:3/457: mkdir d1/d6/d22/d23/d41/d9d 0 2024-04-16T02:36:00.536 INFO:tasks.workunit.client.0.smithi151.stdout:8/482: creat d11/d2a/d33/d4e/d62/d94/fa3 x:0 0 0 2024-04-16T02:36:00.539 INFO:tasks.workunit.client.0.smithi151.stdout:5/712: truncate d2/d6/d19/d62/dd0/d9c/fa2 541896 0 2024-04-16T02:36:00.541 INFO:tasks.workunit.client.0.smithi151.stdout:8/483: mkdir d11/d2a/d33/d4e/d62/d94/da4 0 2024-04-16T02:36:00.547 INFO:tasks.workunit.client.0.smithi151.stdout:2/902: write de/f45 [2063344,66781] 0 2024-04-16T02:36:00.547 INFO:tasks.workunit.client.0.smithi151.stdout:3/458: getdents d1/d89 0 2024-04-16T02:36:00.548 INFO:tasks.workunit.client.0.smithi151.stdout:3/459: fdatasync d1/d6/f8f 0 2024-04-16T02:36:00.550 INFO:tasks.workunit.client.0.smithi151.stdout:5/713: mknod d2/d6/d19/d62/dd0/d9c/dd7/ce0 0 2024-04-16T02:36:00.554 INFO:tasks.workunit.client.0.smithi151.stdout:2/903: mknod de/d65/daa/dba/c132 0 2024-04-16T02:36:00.555 INFO:tasks.workunit.client.0.smithi151.stdout:8/484: dread d11/f15 [0,4194304] 0 2024-04-16T02:36:00.556 INFO:tasks.workunit.client.0.smithi151.stdout:3/460: fdatasync d1/d6/f50 0 2024-04-16T02:36:00.560 INFO:tasks.workunit.client.0.smithi151.stdout:5/714: rename d2/d6/d19/d30/f70 to d2/d6/d19/d30/d88/ddc/d83/fe1 0 2024-04-16T02:36:00.565 INFO:tasks.workunit.client.0.smithi151.stdout:2/904: creat de/d65/dc0/d102/f133 x:0 0 0 2024-04-16T02:36:00.570 INFO:tasks.workunit.client.0.smithi151.stdout:8/485: unlink d11/d2a/d35/d64/c63 0 2024-04-16T02:36:00.570 INFO:tasks.workunit.client.0.smithi151.stdout:8/486: stat d11/d2a/c2d 0 2024-04-16T02:36:00.574 INFO:tasks.workunit.client.0.smithi151.stdout:3/461: unlink d1/d6/d20/f7d 0 2024-04-16T02:36:00.576 INFO:tasks.workunit.client.0.smithi151.stdout:5/715: fdatasync d2/d6/d19/d62/f7b 0 2024-04-16T02:36:00.578 INFO:tasks.workunit.client.0.smithi151.stdout:2/905: mkdir de/d65/daa/dba/df1/d134 0 2024-04-16T02:36:00.580 INFO:tasks.workunit.client.0.smithi151.stdout:8/487: mknod d11/d22/d6f/ca5 0 2024-04-16T02:36:00.589 INFO:tasks.workunit.client.0.smithi151.stdout:5/716: mknod d2/d6/d19/d69/ce2 0 2024-04-16T02:36:00.592 INFO:tasks.workunit.client.0.smithi151.stdout:8/488: chown d11/l25 937921971 1 2024-04-16T02:36:00.593 INFO:tasks.workunit.client.0.smithi151.stdout:8/489: dread - d11/d2a/d33/d4e/d62/d94/fa3 zero size 2024-04-16T02:36:00.595 INFO:tasks.workunit.client.0.smithi151.stdout:3/462: dread d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/f55 [0,4194304] 0 2024-04-16T02:36:00.602 INFO:tasks.workunit.client.0.smithi151.stdout:8/490: write d11/d2a/d33/f4b [1853044,73117] 0 2024-04-16T02:36:00.603 INFO:tasks.workunit.client.0.smithi151.stdout:3/463: stat d1/d6/d22/d23/d30/d44/l95 0 2024-04-16T02:36:00.605 INFO:tasks.workunit.client.0.smithi151.stdout:5/717: truncate d2/d6/d43/f6d 1762284 0 2024-04-16T02:36:00.608 INFO:tasks.workunit.client.0.smithi151.stdout:3/464: truncate d1/d6/d4a/f6f 236933 0 2024-04-16T02:36:00.609 INFO:tasks.workunit.client.0.smithi151.stdout:5/718: mknod d2/d6/d19/d30/d49/ce3 0 2024-04-16T02:36:00.609 INFO:tasks.workunit.client.0.smithi151.stdout:3/465: write d1/d6/d22/d23/d30/d44/f85 [3091773,107360] 0 2024-04-16T02:36:00.610 INFO:tasks.workunit.client.0.smithi151.stdout:8/491: mknod d11/d2a/d35/ca6 0 2024-04-16T02:36:00.613 INFO:tasks.workunit.client.0.smithi151.stdout:5/719: symlink d2/le4 0 2024-04-16T02:36:00.613 INFO:tasks.workunit.client.0.smithi151.stdout:2/906: write de/d65/d71/d7c/fc9 [2516194,119746] 0 2024-04-16T02:36:00.615 INFO:tasks.workunit.client.0.smithi151.stdout:8/492: symlink d11/d22/d6f/d92/la7 0 2024-04-16T02:36:00.615 INFO:tasks.workunit.client.0.smithi151.stdout:2/907: read de/d11/d42/d5e/d6c/f77 [3390503,18662] 0 2024-04-16T02:36:00.618 INFO:tasks.workunit.client.0.smithi151.stdout:5/720: mknod d2/d6/d19/d30/d49/ce5 0 2024-04-16T02:36:00.620 INFO:tasks.workunit.client.0.smithi151.stdout:3/466: creat d1/d6/d22/d23/d41/d9d/f9e x:0 0 0 2024-04-16T02:36:00.621 INFO:tasks.workunit.client.0.smithi151.stdout:3/467: chown d1/d6/d22/d23/d66/f7b 39 1 2024-04-16T02:36:00.622 INFO:tasks.workunit.client.0.smithi151.stdout:2/908: rename de/d65/d71/cc8 to de/d65/d71/d98/c135 0 2024-04-16T02:36:00.623 INFO:tasks.workunit.client.0.smithi151.stdout:2/909: chown de/d15/d78/fae 1586655555 1 2024-04-16T02:36:00.625 INFO:tasks.workunit.client.0.smithi151.stdout:8/493: unlink d11/c73 0 2024-04-16T02:36:00.626 INFO:tasks.workunit.client.0.smithi151.stdout:5/721: truncate d2/d6/d19/d30/d88/ddc/d83/d8b/f8f 506041 0 2024-04-16T02:36:00.627 INFO:tasks.workunit.client.0.smithi151.stdout:3/468: dwrite d1/d37/d87/f8b [0,4194304] 0 2024-04-16T02:36:00.629 INFO:tasks.workunit.client.0.smithi151.stdout:5/722: write d2/d6/d19/d30/d49/d85/fd1 [256328,81256] 0 2024-04-16T02:36:00.630 INFO:tasks.workunit.client.0.smithi151.stdout:3/469: fsync d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/f57 0 2024-04-16T02:36:00.630 INFO:tasks.workunit.client.0.smithi151.stdout:5/723: readlink d2/d6/d19/d30/d49/l54 0 2024-04-16T02:36:00.632 INFO:tasks.workunit.client.0.smithi151.stdout:3/470: truncate d1/d6/d22/f2c 4962091 0 2024-04-16T02:36:00.632 INFO:tasks.workunit.client.0.smithi151.stdout:2/910: unlink de/d11/d42/l4d 0 2024-04-16T02:36:00.633 INFO:tasks.workunit.client.0.smithi151.stdout:8/494: creat d11/d22/d6f/d92/d78/fa8 x:0 0 0 2024-04-16T02:36:00.635 INFO:tasks.workunit.client.0.smithi151.stdout:5/724: symlink d2/le6 0 2024-04-16T02:36:00.636 INFO:tasks.workunit.client.0.smithi151.stdout:2/911: mkdir de/d65/d71/d98/d136 0 2024-04-16T02:36:00.637 INFO:tasks.workunit.client.0.smithi151.stdout:2/912: chown de/d65/d71/l84 6782 1 2024-04-16T02:36:00.639 INFO:tasks.workunit.client.0.smithi151.stdout:8/495: mknod d11/d22/d6f/d92/ca9 0 2024-04-16T02:36:00.642 INFO:tasks.workunit.client.0.smithi151.stdout:8/496: write d11/d2a/d33/f66 [2152660,112046] 0 2024-04-16T02:36:00.642 INFO:tasks.workunit.client.0.smithi151.stdout:3/471: dwrite d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/f97 [0,4194304] 0 2024-04-16T02:36:00.645 INFO:tasks.workunit.client.0.smithi151.stdout:3/472: chown d1/d6/f1b 25159848 1 2024-04-16T02:36:00.650 INFO:tasks.workunit.client.0.smithi151.stdout:8/497: dwrite d11/d22/d6f/d92/d78/fa8 [0,4194304] 0 2024-04-16T02:36:00.653 INFO:tasks.workunit.client.0.smithi151.stdout:2/913: truncate de/d11/d42/d82/d95/d4b/f55 4114242 0 2024-04-16T02:36:00.660 INFO:tasks.workunit.client.0.smithi151.stdout:2/914: creat de/d11/d42/f137 x:0 0 0 2024-04-16T02:36:00.661 INFO:tasks.workunit.client.0.smithi151.stdout:8/498: rename d11/l54 to d11/d2a/d33/d4e/d62/d94/laa 0 2024-04-16T02:36:00.662 INFO:tasks.workunit.client.0.smithi151.stdout:5/725: write d2/d6/d19/d30/d49/f6e [1571381,82473] 0 2024-04-16T02:36:00.665 INFO:tasks.workunit.client.0.smithi151.stdout:5/726: write d2/d6/d19/d30/d88/ddc/d83/fa0 [460371,69990] 0 2024-04-16T02:36:00.668 INFO:tasks.workunit.client.0.smithi151.stdout:5/727: mkdir d2/d6/d19/d30/d49/de7 0 2024-04-16T02:36:00.670 INFO:tasks.workunit.client.0.smithi151.stdout:1/612: sync 2024-04-16T02:36:00.671 INFO:tasks.workunit.client.0.smithi151.stdout:5/728: creat d2/d6/d43/fe8 x:0 0 0 2024-04-16T02:36:00.672 INFO:tasks.workunit.client.0.smithi151.stdout:5/729: stat d2/d6/d19/d62/dbd 0 2024-04-16T02:36:00.675 INFO:tasks.workunit.client.0.smithi151.stdout:5/730: read d2/d6/d19/f1d [3282510,76360] 0 2024-04-16T02:36:00.676 INFO:tasks.workunit.client.0.smithi151.stdout:3/473: dread d1/d6/d22/d23/d30/d44/d46/d8a/f35 [0,4194304] 0 2024-04-16T02:36:00.676 INFO:tasks.workunit.client.0.smithi151.stdout:1/613: unlink d3/c7 0 2024-04-16T02:36:00.677 INFO:tasks.workunit.client.0.smithi151.stdout:2/915: write de/d11/d42/d7a/d8b/f8e [4495691,21893] 0 2024-04-16T02:36:00.678 INFO:tasks.workunit.client.0.smithi151.stdout:8/499: truncate d11/f16 2358310 0 2024-04-16T02:36:00.679 INFO:tasks.workunit.client.0.smithi151.stdout:2/916: chown de/d11/d42/d5e/d6c/db8/dcc/d10f/c119 4 1 2024-04-16T02:36:00.680 INFO:tasks.workunit.client.0.smithi151.stdout:2/917: chown de/d65/daa/dba/df1 27 1 2024-04-16T02:36:00.681 INFO:tasks.workunit.client.0.smithi151.stdout:5/731: symlink d2/d6/d19/d62/dd0/le9 0 2024-04-16T02:36:00.692 INFO:tasks.workunit.client.0.smithi151.stdout:2/918: dread - de/d11/d42/fd7 zero size 2024-04-16T02:36:00.693 INFO:tasks.workunit.client.0.smithi151.stdout:2/919: chown de/d65/f72 1738623 1 2024-04-16T02:36:00.696 INFO:tasks.workunit.client.0.smithi151.stdout:5/732: dread d2/d6/d43/f67 [0,4194304] 0 2024-04-16T02:36:00.698 INFO:tasks.workunit.client.0.smithi151.stdout:3/474: mknod d1/d6/c9f 0 2024-04-16T02:36:00.702 INFO:tasks.workunit.client.0.smithi151.stdout:2/920: symlink de/d11/l138 0 2024-04-16T02:36:00.703 INFO:tasks.workunit.client.0.smithi151.stdout:8/500: dwrite d11/d2a/d85/f8b [0,4194304] 0 2024-04-16T02:36:00.707 INFO:tasks.workunit.client.0.smithi151.stdout:5/733: creat d2/d6/d19/d30/d88/ddc/d83/d8b/fea x:0 0 0 2024-04-16T02:36:00.708 INFO:tasks.workunit.client.0.smithi151.stdout:5/734: readlink d2/d6/d19/d1a/l6a 0 2024-04-16T02:36:00.710 INFO:tasks.workunit.client.0.smithi151.stdout:1/614: creat d3/d4/d64/d40/fcf x:0 0 0 2024-04-16T02:36:00.714 INFO:tasks.workunit.client.0.smithi151.stdout:8/501: unlink d11/d22/f26 0 2024-04-16T02:36:00.715 INFO:tasks.workunit.client.0.smithi151.stdout:2/921: chown de/d11/d42/d5e/d6c/da2/le2 51 1 2024-04-16T02:36:00.717 INFO:tasks.workunit.client.0.smithi151.stdout:3/475: unlink d1/d6/l13 0 2024-04-16T02:36:00.720 INFO:tasks.workunit.client.0.smithi151.stdout:2/922: fdatasync de/d11/f7d 0 2024-04-16T02:36:00.721 INFO:tasks.workunit.client.0.smithi151.stdout:2/923: read - de/d11/d42/d5e/d6c/d6d/da5/f12a zero size 2024-04-16T02:36:00.722 INFO:tasks.workunit.client.0.smithi151.stdout:3/476: mknod d1/d37/ca0 0 2024-04-16T02:36:00.723 INFO:tasks.workunit.client.0.smithi151.stdout:2/924: fdatasync de/d11/d42/d6b/d79/f11a 0 2024-04-16T02:36:00.723 INFO:tasks.workunit.client.0.smithi151.stdout:2/925: dread - de/d11/d42/d5e/d6c/db8/f126 zero size 2024-04-16T02:36:00.726 INFO:tasks.workunit.client.0.smithi151.stdout:3/477: symlink d1/d6/d20/d3d/la1 0 2024-04-16T02:36:00.731 INFO:tasks.workunit.client.0.smithi151.stdout:5/735: dread d2/d6/d19/d1a/f41 [0,4194304] 0 2024-04-16T02:36:00.732 INFO:tasks.workunit.client.0.smithi151.stdout:5/736: readlink d2/d6/d19/d93/l75 0 2024-04-16T02:36:00.735 INFO:tasks.workunit.client.0.smithi151.stdout:3/478: dread d1/f5c [0,4194304] 0 2024-04-16T02:36:00.739 INFO:tasks.workunit.client.0.smithi151.stdout:5/737: mkdir d2/d6/d19/d30/d88/ddc/da8/dd3/deb 0 2024-04-16T02:36:00.740 INFO:tasks.workunit.client.0.smithi151.stdout:1/615: dwrite d3/d33/d10/f6b [0,4194304] 0 2024-04-16T02:36:00.744 INFO:tasks.workunit.client.0.smithi151.stdout:2/926: dread de/d65/dc0/fe5 [0,4194304] 0 2024-04-16T02:36:00.746 INFO:tasks.workunit.client.0.smithi151.stdout:5/738: rename d2/d6/f50 to d2/d6/d19/d30/d49/d7f/fec 0 2024-04-16T02:36:00.750 INFO:tasks.workunit.client.0.smithi151.stdout:2/927: dread de/d15/f17 [0,4194304] 0 2024-04-16T02:36:00.751 INFO:tasks.workunit.client.0.smithi151.stdout:1/616: creat d3/d4/d64/d40/da3/fd0 x:0 0 0 2024-04-16T02:36:00.755 INFO:tasks.workunit.client.0.smithi151.stdout:5/739: creat d2/d6/d43/fed x:0 0 0 2024-04-16T02:36:00.755 INFO:tasks.workunit.client.0.smithi151.stdout:5/740: fsync d2/d6/d19/fc2 0 2024-04-16T02:36:00.756 INFO:tasks.workunit.client.0.smithi151.stdout:3/479: link d1/d6/d22/d23/d30/d44/d46/d8a/d3f/l92 d1/d6/d22/d23/d30/d44/la2 0 2024-04-16T02:36:00.759 INFO:tasks.workunit.client.0.smithi151.stdout:2/928: rename de/d11/d42/d5e/d6c/d6d to de/d12c/d139 0 2024-04-16T02:36:00.760 INFO:tasks.workunit.client.0.smithi151.stdout:2/929: write de/d12c/d139/da5/f12d [1037985,125651] 0 2024-04-16T02:36:00.764 INFO:tasks.workunit.client.0.smithi151.stdout:5/741: mknod d2/d6/d19/d69/cee 0 2024-04-16T02:36:00.767 INFO:tasks.workunit.client.0.smithi151.stdout:1/617: dwrite d3/d4/f96 [0,4194304] 0 2024-04-16T02:36:00.768 INFO:tasks.workunit.client.0.smithi151.stdout:2/930: dwrite de/d11/d42/d7a/f85 [0,4194304] 0 2024-04-16T02:36:00.778 INFO:tasks.workunit.client.0.smithi151.stdout:2/931: creat de/d11/d42/d7a/d118/f13a x:0 0 0 2024-04-16T02:36:00.780 INFO:tasks.workunit.client.0.smithi151.stdout:3/480: link d1/d6/d22/d23/d30/d44/d46/d8a/f1a d1/d6/d22/d23/d30/d44/d46/d8a/fa3 0 2024-04-16T02:36:00.781 INFO:tasks.workunit.client.0.smithi151.stdout:3/481: write d1/d6/d22/d23/d41/d9d/f9e [316958,220] 0 2024-04-16T02:36:00.782 INFO:tasks.workunit.client.0.smithi151.stdout:2/932: fdatasync de/d11/f7d 0 2024-04-16T02:36:00.785 INFO:tasks.workunit.client.0.smithi151.stdout:2/933: mkdir de/d65/d88/d13b 0 2024-04-16T02:36:00.790 INFO:tasks.workunit.client.0.smithi151.stdout:3/482: dwrite d1/d6/d22/d23/d66/f74 [0,4194304] 0 2024-04-16T02:36:00.796 INFO:tasks.workunit.client.0.smithi151.stdout:2/934: read de/d15/f34 [5521736,103741] 0 2024-04-16T02:36:00.805 INFO:tasks.workunit.client.0.smithi151.stdout:2/935: dwrite de/f27 [0,4194304] 0 2024-04-16T02:36:00.811 INFO:tasks.workunit.client.0.smithi151.stdout:2/936: creat de/d11/d42/d5e/d6c/db8/dcc/d10f/f13c x:0 0 0 2024-04-16T02:36:00.814 INFO:tasks.workunit.client.0.smithi151.stdout:2/937: mkdir de/d15/d13d 0 2024-04-16T02:36:00.819 INFO:tasks.workunit.client.0.smithi151.stdout:1/618: write d3/d4/d64/d2a/d7a/f86 [130427,35925] 0 2024-04-16T02:36:00.822 INFO:tasks.workunit.client.0.smithi151.stdout:5/742: dwrite d2/d6/f27 [0,4194304] 0 2024-04-16T02:36:00.825 INFO:tasks.workunit.client.0.smithi151.stdout:2/938: rename de/d65/d71/d7c/c110 to de/c13e 0 2024-04-16T02:36:00.830 INFO:tasks.workunit.client.0.smithi151.stdout:1/619: dwrite d3/d1f/f94 [0,4194304] 0 2024-04-16T02:36:00.831 INFO:tasks.workunit.client.0.smithi151.stdout:5/743: chown d2/d6/d19/d30/d49/d7f/cd8 4067056 1 2024-04-16T02:36:00.831 INFO:tasks.workunit.client.0.smithi151.stdout:2/939: read de/d12c/d139/da5/fac [1184584,10713] 0 2024-04-16T02:36:00.835 INFO:tasks.workunit.client.0.smithi151.stdout:2/940: write de/d12c/d139/f92 [949258,31337] 0 2024-04-16T02:36:00.841 INFO:tasks.workunit.client.0.smithi151.stdout:5/744: symlink d2/d6/db7/lef 0 2024-04-16T02:36:00.842 INFO:tasks.workunit.client.0.smithi151.stdout:5/745: fsync d2/d6/d19/d30/d36/d42/fdd 0 2024-04-16T02:36:00.845 INFO:tasks.workunit.client.0.smithi151.stdout:2/941: fdatasync de/d11/d42/d7a/d8b/f11b 0 2024-04-16T02:36:00.845 INFO:tasks.workunit.client.0.smithi151.stdout:1/620: truncate d3/d33/f39 370352 0 2024-04-16T02:36:00.847 INFO:tasks.workunit.client.0.smithi151.stdout:1/621: write d3/d4/d64/d40/fbb [504061,103075] 0 2024-04-16T02:36:00.848 INFO:tasks.workunit.client.0.smithi151.stdout:5/746: rmdir d2/d6/d19/d30/d49/d7f 39 2024-04-16T02:36:00.850 INFO:tasks.workunit.client.0.smithi151.stdout:5/747: fsync d2/d6/d19/d30/d49/f6e 0 2024-04-16T02:36:00.853 INFO:tasks.workunit.client.0.smithi151.stdout:1/622: rename d3/d58/df/d38/d8b to d3/d33/d10/d4f/dd1 0 2024-04-16T02:36:00.853 INFO:tasks.workunit.client.0.smithi151.stdout:2/942: read de/d11/d42/d7a/fde [134350,74523] 0 2024-04-16T02:36:00.860 INFO:tasks.workunit.client.0.smithi151.stdout:1/623: readlink d3/l70 0 2024-04-16T02:36:00.864 INFO:tasks.workunit.client.0.smithi151.stdout:2/943: rmdir de/d11/d42/d82 39 2024-04-16T02:36:00.873 INFO:tasks.workunit.client.0.smithi151.stdout:5/748: creat d2/d6/d19/ff0 x:0 0 0 2024-04-16T02:36:00.873 INFO:tasks.workunit.client.0.smithi151.stdout:5/749: stat d2/f99 0 2024-04-16T02:36:00.875 INFO:tasks.workunit.client.0.smithi151.stdout:1/624: rename d3/d33/cba to d3/d4/d64/d2a/d7e/cd2 0 2024-04-16T02:36:00.875 INFO:tasks.workunit.client.0.smithi151.stdout:2/944: truncate de/d11/d42/d82/fa0 1533558 0 2024-04-16T02:36:00.878 INFO:tasks.workunit.client.0.smithi151.stdout:1/625: write d3/f9d [3105547,38067] 0 2024-04-16T02:36:00.878 INFO:tasks.workunit.client.0.smithi151.stdout:2/945: symlink de/d65/dc0/l13f 0 2024-04-16T02:36:00.880 INFO:tasks.workunit.client.0.smithi151.stdout:5/750: mkdir d2/d6/df1 0 2024-04-16T02:36:00.884 INFO:tasks.workunit.client.0.smithi151.stdout:5/751: creat d2/d6/d19/d30/d88/ddc/da8/dd3/ff2 x:0 0 0 2024-04-16T02:36:00.888 INFO:tasks.workunit.client.0.smithi151.stdout:5/752: mknod d2/d6/d19/d69/cf3 0 2024-04-16T02:36:00.889 INFO:tasks.workunit.client.0.smithi151.stdout:1/626: creat d3/d4/d64/d40/fd3 x:0 0 0 2024-04-16T02:36:00.890 INFO:tasks.workunit.client.0.smithi151.stdout:1/627: stat d3/d4/d64/d2a/l6e 0 2024-04-16T02:36:00.891 INFO:tasks.workunit.client.0.smithi151.stdout:5/753: rmdir d2 39 2024-04-16T02:36:00.893 INFO:tasks.workunit.client.0.smithi151.stdout:5/754: chown d2/d6/d19/d30/d88/ddc/da8/dd3/ff2 60 1 2024-04-16T02:36:00.898 INFO:tasks.workunit.client.0.smithi151.stdout:5/755: truncate d2/d6/d19/d30/fa1 531442 0 2024-04-16T02:36:00.898 INFO:tasks.workunit.client.0.smithi151.stdout:2/946: dwrite de/f49 [0,4194304] 0 2024-04-16T02:36:00.902 INFO:tasks.workunit.client.0.smithi151.stdout:2/947: rename de/d15/d13d to de/d11/d42/d5e/d6c/db8/dcc/d10f/d140 0 2024-04-16T02:36:00.921 INFO:tasks.workunit.client.0.smithi151.stdout:5/756: dread d2/d6/d19/d30/d88/ddc/fa3 [0,4194304] 0 2024-04-16T02:36:00.923 INFO:tasks.workunit.client.0.smithi151.stdout:5/757: fsync d2/d6/d19/d30/d88/ddc/d83/d8b/fea 0 2024-04-16T02:36:00.949 INFO:tasks.workunit.client.0.smithi151.stdout:9/621: sync 2024-04-16T02:36:00.951 INFO:tasks.workunit.client.0.smithi151.stdout:9/622: write d1/d3/d6/d17/d40/d56/f61 [1455909,2972] 0 2024-04-16T02:36:00.952 INFO:tasks.workunit.client.0.smithi151.stdout:5/758: creat d2/d6/d19/d30/d88/ddc/d83/d8b/ff4 x:0 0 0 2024-04-16T02:36:00.959 INFO:tasks.workunit.client.0.smithi151.stdout:9/623: truncate d1/dd/d4a/f53 1357951 0 2024-04-16T02:36:00.964 INFO:tasks.workunit.client.0.smithi151.stdout:2/948: dwrite de/d15/f5b [0,4194304] 0 2024-04-16T02:36:00.971 INFO:tasks.workunit.client.0.smithi151.stdout:5/759: dwrite d2/f16 [4194304,4194304] 0 2024-04-16T02:36:00.973 INFO:tasks.workunit.client.0.smithi151.stdout:9/624: dwrite d1/d3/d10/d7b/f85 [0,4194304] 0 2024-04-16T02:36:00.973 INFO:tasks.workunit.client.0.smithi151.stdout:2/949: mkdir de/d11/d42/d82/d95/d4b/d141 0 2024-04-16T02:36:00.976 INFO:tasks.workunit.client.0.smithi151.stdout:2/950: fdatasync de/d11/d42/d5e/d6c/f121 0 2024-04-16T02:36:00.977 INFO:tasks.workunit.client.0.smithi151.stdout:2/951: chown de/d12c/d139/da5/c109 92 1 2024-04-16T02:36:00.984 INFO:tasks.workunit.client.0.smithi151.stdout:2/952: rename de/d15/f9a to de/d65/d88/d8f/dce/f142 0 2024-04-16T02:36:00.987 INFO:tasks.workunit.client.0.smithi151.stdout:2/953: mkdir de/d65/daa/dba/df1/d143 0 2024-04-16T02:36:00.991 INFO:tasks.workunit.client.0.smithi151.stdout:2/954: creat de/d11/d123/f144 x:0 0 0 2024-04-16T02:36:00.991 INFO:tasks.workunit.client.0.smithi151.stdout:5/760: dwrite d2/d6/d19/d30/d49/d85/fae [0,4194304] 0 2024-04-16T02:36:00.997 INFO:tasks.workunit.client.0.smithi151.stdout:2/955: link de/d11/l113 de/d11/d42/d6b/d79/l145 0 2024-04-16T02:36:00.999 INFO:tasks.workunit.client.0.smithi151.stdout:2/956: mkdir de/d11/d42/d7a/d8b/db1/d146 0 2024-04-16T02:36:01.001 INFO:tasks.workunit.client.0.smithi151.stdout:2/957: read de/d11/d42/d7a/fbb [3415994,32030] 0 2024-04-16T02:36:01.003 INFO:tasks.workunit.client.0.smithi151.stdout:5/761: write d2/d6/d43/f6d [2773588,43876] 0 2024-04-16T02:36:01.003 INFO:tasks.workunit.client.0.smithi151.stdout:5/762: stat d2/l10 0 2024-04-16T02:36:01.004 INFO:tasks.workunit.client.0.smithi151.stdout:2/958: creat de/d12c/d139/f147 x:0 0 0 2024-04-16T02:36:01.006 INFO:tasks.workunit.client.0.smithi151.stdout:2/959: creat de/d65/daa/dba/d131/f148 x:0 0 0 2024-04-16T02:36:01.008 INFO:tasks.workunit.client.0.smithi151.stdout:2/960: rmdir de/d11/d42/d82/d95/d4b/d60 39 2024-04-16T02:36:01.009 INFO:tasks.workunit.client.0.smithi151.stdout:2/961: creat de/d15/dcd/f149 x:0 0 0 2024-04-16T02:36:01.012 INFO:tasks.workunit.client.0.smithi151.stdout:2/962: dread de/d65/d88/d8f/dce/f142 [0,4194304] 0 2024-04-16T02:36:01.014 INFO:tasks.workunit.client.0.smithi151.stdout:5/763: write d2/d6/d19/d1a/f1b [690828,8551] 0 2024-04-16T02:36:01.015 INFO:tasks.workunit.client.0.smithi151.stdout:2/963: mkdir de/d11/d42/d6b/d14a 0 2024-04-16T02:36:01.016 INFO:tasks.workunit.client.0.smithi151.stdout:2/964: stat de/d15/c1f 0 2024-04-16T02:36:01.018 INFO:tasks.workunit.client.0.smithi151.stdout:2/965: write de/d11/d42/d7a/d8b/f10b [152103,13151] 0 2024-04-16T02:36:01.021 INFO:tasks.workunit.client.0.smithi151.stdout:5/764: link d2/d6/d19/d1a/f79 d2/d6/d19/d1a/d4c/ff5 0 2024-04-16T02:36:01.026 INFO:tasks.workunit.client.0.smithi151.stdout:9/625: dread d1/d3/d6/d17/d44/f6e [0,4194304] 0 2024-04-16T02:36:01.026 INFO:tasks.workunit.client.0.smithi151.stdout:2/966: dwrite de/d11/d42/d6b/d11c/f124 [0,4194304] 0 2024-04-16T02:36:01.027 INFO:tasks.workunit.client.0.smithi151.stdout:9/626: stat d1/d3/d6/d17/d54/l5c 0 2024-04-16T02:36:01.029 INFO:tasks.workunit.client.0.smithi151.stdout:9/627: chown d1/d3/d6/d28/d39/lc0 73 1 2024-04-16T02:36:01.029 INFO:tasks.workunit.client.0.smithi151.stdout:9/628: fdatasync d1/d75/fbf 0 2024-04-16T02:36:01.034 INFO:tasks.workunit.client.0.smithi151.stdout:2/967: mkdir de/d65/d88/dfd/d14b 0 2024-04-16T02:36:01.034 INFO:tasks.workunit.client.0.smithi151.stdout:9/629: write d1/d3/d6/f13 [2113599,24877] 0 2024-04-16T02:36:01.037 INFO:tasks.workunit.client.0.smithi151.stdout:9/630: write d1/d3/d6/d17/d54/d9b/d22/d36/f9e [3299657,117370] 0 2024-04-16T02:36:01.041 INFO:tasks.workunit.client.0.smithi151.stdout:2/968: dwrite de/d11/d42/d7a/d8b/db1/d90/f10e [0,4194304] 0 2024-04-16T02:36:01.046 INFO:tasks.workunit.client.0.smithi151.stdout:2/969: link de/d11/c21 de/d65/d71/d98/d136/c14c 0 2024-04-16T02:36:01.054 INFO:tasks.workunit.client.0.smithi151.stdout:2/970: dwrite de/d11/d42/d5e/f8d [0,4194304] 0 2024-04-16T02:36:01.055 INFO:tasks.workunit.client.0.smithi151.stdout:2/971: dread - de/d11/d42/d82/ff7 zero size 2024-04-16T02:36:01.058 INFO:tasks.workunit.client.0.smithi151.stdout:2/972: mknod de/d11/d42/d6b/d79/c14d 0 2024-04-16T02:36:01.060 INFO:tasks.workunit.client.0.smithi151.stdout:2/973: fsync de/d12c/d139/da5/fac 0 2024-04-16T02:36:01.062 INFO:tasks.workunit.client.0.smithi151.stdout:2/974: rmdir de/d15/dcd/dd0 39 2024-04-16T02:36:01.065 INFO:tasks.workunit.client.0.smithi151.stdout:9/631: dread d1/d3/f33 [0,4194304] 0 2024-04-16T02:36:01.071 INFO:tasks.workunit.client.0.smithi151.stdout:2/975: write de/d11/d42/d7a/d8b/db1/f67 [5343424,19089] 0 2024-04-16T02:36:01.074 INFO:tasks.workunit.client.0.smithi151.stdout:2/976: creat de/d11/d42/d7a/d8b/db1/f14e x:0 0 0 2024-04-16T02:36:01.078 INFO:tasks.workunit.client.0.smithi151.stdout:2/977: rename de/f14 to de/d3d/de8/f14f 0 2024-04-16T02:36:01.079 INFO:tasks.workunit.client.0.smithi151.stdout:2/978: rmdir de/d65/dc0 39 2024-04-16T02:36:01.080 INFO:tasks.workunit.client.0.smithi151.stdout:2/979: write de/d15/d78/fae [3454473,977] 0 2024-04-16T02:36:01.082 INFO:tasks.workunit.client.0.smithi151.stdout:2/980: rmdir de/d11/d42/d82/d95 39 2024-04-16T02:36:01.083 INFO:tasks.workunit.client.0.smithi151.stdout:2/981: fdatasync de/d15/f50 0 2024-04-16T02:36:01.085 INFO:tasks.workunit.client.0.smithi151.stdout:2/982: mkdir de/d11/d42/d82/d150 0 2024-04-16T02:36:01.086 INFO:tasks.workunit.client.0.smithi151.stdout:2/983: truncate de/d65/d88/f122 1013573 0 2024-04-16T02:36:01.088 INFO:tasks.workunit.client.0.smithi151.stdout:2/984: readlink de/d11/d42/d82/d95/d4b/lf5 0 2024-04-16T02:36:01.090 INFO:tasks.workunit.client.0.smithi151.stdout:2/985: write de/d11/d42/d7a/fde [205160,59344] 0 2024-04-16T02:36:01.091 INFO:tasks.workunit.client.0.smithi151.stdout:9/632: dread d1/f2e [0,4194304] 0 2024-04-16T02:36:01.092 INFO:tasks.workunit.client.0.smithi151.stdout:2/986: fdatasync f7 0 2024-04-16T02:36:01.097 INFO:tasks.workunit.client.0.smithi151.stdout:2/987: rename de/d65/d88/d8f/dce/f142 to de/d65/d71/d7c/dc4/f151 0 2024-04-16T02:36:01.097 INFO:tasks.workunit.client.0.smithi151.stdout:2/988: chown de/d15/d78 13139756 1 2024-04-16T02:36:01.098 INFO:tasks.workunit.client.0.smithi151.stdout:9/633: truncate d1/d3/d5a/f82 908864 0 2024-04-16T02:36:01.099 INFO:tasks.workunit.client.0.smithi151.stdout:2/989: fdatasync de/d11/d42/d7a/d8b/db1/f24 0 2024-04-16T02:36:01.101 INFO:tasks.workunit.client.0.smithi151.stdout:2/990: creat de/d3d/d6a/f152 x:0 0 0 2024-04-16T02:36:01.103 INFO:tasks.workunit.client.0.smithi151.stdout:9/634: dread d1/d3/d10/d7b/f85 [0,4194304] 0 2024-04-16T02:36:01.110 INFO:tasks.workunit.client.0.smithi151.stdout:2/991: truncate de/d12c/d139/f92 1493980 0 2024-04-16T02:36:01.113 INFO:tasks.workunit.client.0.smithi151.stdout:2/992: creat de/d11/d42/d6b/d79/f153 x:0 0 0 2024-04-16T02:36:01.123 INFO:tasks.workunit.client.0.smithi151.stdout:2/993: dwrite de/d65/dc0/fe5 [0,4194304] 0 2024-04-16T02:36:01.129 INFO:tasks.workunit.client.0.smithi151.stdout:2/994: write de/d15/f34 [1487784,50478] 0 2024-04-16T02:36:01.132 INFO:tasks.workunit.client.0.smithi151.stdout:2/995: unlink de/d65/daa/fd2 0 2024-04-16T02:36:01.134 INFO:tasks.workunit.client.0.smithi151.stdout:2/996: mknod de/d65/daa/dba/df1/d143/c154 0 2024-04-16T02:36:01.138 INFO:tasks.workunit.client.0.smithi151.stdout:2/997: dread de/d65/d71/d7c/f9d [0,4194304] 0 2024-04-16T02:36:01.158 INFO:tasks.workunit.client.0.smithi151.stdout:7/703: sync 2024-04-16T02:36:01.159 INFO:tasks.workunit.client.0.smithi151.stdout:2/998: mkdir de/d12c/d155 0 2024-04-16T02:36:01.160 INFO:tasks.workunit.client.0.smithi151.stdout:7/704: readlink d0/d2c/d38/d85/l74 0 2024-04-16T02:36:01.164 INFO:tasks.workunit.client.0.smithi151.stdout:2/999: dread de/d11/d42/d7a/fde [0,4194304] 0 2024-04-16T02:36:01.172 INFO:tasks.workunit.client.0.smithi151.stdout:7/705: dwrite d0/d1f/d4c/fa0 [0,4194304] 0 2024-04-16T02:36:01.182 INFO:tasks.workunit.client.0.smithi151.stdout:7/706: dwrite d0/d27/f7a [4194304,4194304] 0 2024-04-16T02:36:01.183 INFO:tasks.workunit.client.0.smithi151.stdout:7/707: chown d0/d1f/d46/d4f/f62 6 1 2024-04-16T02:36:01.186 INFO:tasks.workunit.client.0.smithi151.stdout:7/708: creat d0/ddf/ff8 x:0 0 0 2024-04-16T02:36:01.188 INFO:tasks.workunit.client.0.smithi151.stdout:7/709: rmdir d0/d2c/d38/d5e 39 2024-04-16T02:36:01.195 INFO:tasks.workunit.client.0.smithi151.stdout:7/710: dwrite d0/d2c/d38/db0/fe2 [0,4194304] 0 2024-04-16T02:36:01.197 INFO:tasks.workunit.client.0.smithi151.stdout:7/711: mknod d0/d1f/d4c/cf9 0 2024-04-16T02:36:01.211 INFO:tasks.workunit.client.0.smithi151.stdout:7/712: truncate d0/d1f/d46/f5d 2529724 0 2024-04-16T02:36:01.216 INFO:tasks.workunit.client.0.smithi151.stdout:7/713: dread d0/d1f/da8/db2/fce [0,4194304] 0 2024-04-16T02:36:01.225 INFO:tasks.workunit.client.0.smithi151.stdout:7/714: dwrite d0/d2c/d38/f3d [4194304,4194304] 0 2024-04-16T02:36:01.226 INFO:tasks.workunit.client.0.smithi151.stdout:7/715: write d0/f3 [11070186,19823] 0 2024-04-16T02:36:01.229 INFO:tasks.workunit.client.0.smithi151.stdout:7/716: mknod d0/d2c/d38/d43/d78/cfa 0 2024-04-16T02:36:01.233 INFO:tasks.workunit.client.0.smithi151.stdout:7/717: dread d0/d1f/f40 [0,4194304] 0 2024-04-16T02:36:01.238 INFO:tasks.workunit.client.0.smithi151.stdout:7/718: link d0/c34 d0/d1f/d33/cfb 0 2024-04-16T02:36:01.240 INFO:tasks.workunit.client.0.smithi151.stdout:7/719: symlink d0/lfc 0 2024-04-16T02:36:01.247 INFO:tasks.workunit.client.0.smithi151.stdout:7/720: dwrite d0/d1f/d33/fe6 [0,4194304] 0 2024-04-16T02:36:01.253 INFO:tasks.workunit.client.0.smithi151.stdout:7/721: link d0/d1f/da8/cd5 d0/d2c/d38/d85/cfd 0 2024-04-16T02:36:01.255 INFO:tasks.workunit.client.0.smithi151.stdout:7/722: mkdir d0/d2c/d38/de7/dfe 0 2024-04-16T02:36:01.257 INFO:tasks.workunit.client.0.smithi151.stdout:7/723: mkdir d0/d1f/d46/dff 0 2024-04-16T02:36:01.257 INFO:tasks.workunit.client.0.smithi151.stdout:7/724: dread - d0/ddf/ff8 zero size 2024-04-16T02:36:01.258 INFO:tasks.workunit.client.0.smithi151.stdout:7/725: rmdir d0/d1f/d31 39 2024-04-16T02:36:01.261 INFO:tasks.workunit.client.0.smithi151.stdout:7/726: getdents d0/d1f/d60 0 2024-04-16T02:36:01.270 INFO:tasks.workunit.client.0.smithi151.stdout:7/727: dwrite d0/d1f/da8/db2/fce [0,4194304] 0 2024-04-16T02:36:01.271 INFO:tasks.workunit.client.0.smithi151.stdout:7/728: dread - d0/d1f/d99/dc0/ff7 zero size 2024-04-16T02:36:01.273 INFO:tasks.workunit.client.0.smithi151.stdout:7/729: write d0/d1f/d4c/d98/fa4 [427893,18981] 0 2024-04-16T02:36:01.277 INFO:tasks.workunit.client.0.smithi151.stdout:7/730: creat d0/d2c/d38/d5e/f100 x:0 0 0 2024-04-16T02:36:01.358 INFO:tasks.workunit.client.0.smithi151.stdout:6/780: sync 2024-04-16T02:36:01.358 INFO:tasks.workunit.client.0.smithi151.stdout:0/564: sync 2024-04-16T02:36:01.359 INFO:tasks.workunit.client.0.smithi151.stdout:6/781: dread - d2/d13/d3a/f107 zero size 2024-04-16T02:36:01.361 INFO:tasks.workunit.client.0.smithi151.stdout:0/565: creat dc/d15/d34/d67/d69/d6f/d9e/fb8 x:0 0 0 2024-04-16T02:36:01.365 INFO:tasks.workunit.client.0.smithi151.stdout:0/566: dread dc/d4c/f96 [0,4194304] 0 2024-04-16T02:36:01.365 INFO:tasks.workunit.client.0.smithi151.stdout:0/567: chown dc/l79 4034 1 2024-04-16T02:36:01.403 INFO:tasks.workunit.client.0.smithi151.stdout:0/568: read dc/d3c/f48 [179154,123128] 0 2024-04-16T02:36:01.405 INFO:tasks.workunit.client.0.smithi151.stdout:6/782: write d2/d13/d63/def/df/d23/d31/d52/df7/f106 [1040531,65479] 0 2024-04-16T02:36:01.409 INFO:tasks.workunit.client.0.smithi151.stdout:6/783: fsync d2/d13/d63/def/df/d23/d82/d8d/d22/f9b 0 2024-04-16T02:36:01.411 INFO:tasks.workunit.client.0.smithi151.stdout:6/784: mkdir d2/d13/d46/d10b 0 2024-04-16T02:36:01.413 INFO:tasks.workunit.client.0.smithi151.stdout:6/785: write d2/d13/d63/def/df/d23/d82/d8d/f80 [3099623,127313] 0 2024-04-16T02:36:01.414 INFO:tasks.workunit.client.0.smithi151.stdout:6/786: chown d2/d13/f5c 6 1 2024-04-16T02:36:01.415 INFO:tasks.workunit.client.0.smithi151.stdout:6/787: stat d2/d13/d63/def/df/d23/f8c 0 2024-04-16T02:36:01.420 INFO:tasks.workunit.client.0.smithi151.stdout:6/788: dwrite d2/d13/d63/def/df/d23/d82/d8d/fae [0,4194304] 0 2024-04-16T02:36:01.421 INFO:tasks.workunit.client.0.smithi151.stdout:6/789: fsync d2/d13/d6f/fb7 0 2024-04-16T02:36:01.422 INFO:tasks.workunit.client.0.smithi151.stdout:6/790: stat d2/d13/d63/l77 0 2024-04-16T02:36:01.425 INFO:tasks.workunit.client.0.smithi151.stdout:6/791: fsync d2/d13/d63/def/df/d23/d56/f8b 0 2024-04-16T02:36:01.428 INFO:tasks.workunit.client.0.smithi151.stdout:6/792: chown d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/cf8 0 1 2024-04-16T02:36:01.457 INFO:tasks.workunit.client.0.smithi151.stdout:6/793: dread d2/d13/d63/def/df/d20/d4f/fa7 [0,4194304] 0 2024-04-16T02:36:01.466 INFO:tasks.workunit.client.0.smithi151.stdout:6/794: truncate d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d65/fc6 3108323 0 2024-04-16T02:36:01.469 INFO:tasks.workunit.client.0.smithi151.stdout:6/795: rename d2/d13/d63/def/f8 to d2/f10c 0 2024-04-16T02:36:01.471 INFO:tasks.workunit.client.0.smithi151.stdout:6/796: mkdir d2/d13/d63/def/d2e/dba/d10d 0 2024-04-16T02:36:01.772 INFO:tasks.workunit.client.0.smithi151.stdout:4/676: sync 2024-04-16T02:36:01.772 INFO:tasks.workunit.client.0.smithi151.stdout:8/502: sync 2024-04-16T02:36:01.773 INFO:tasks.workunit.client.0.smithi151.stdout:4/677: readlink d0/d99/lb7 0 2024-04-16T02:36:01.777 INFO:tasks.workunit.client.0.smithi151.stdout:8/503: rename d11/l13 to d11/d7d/lab 0 2024-04-16T02:36:01.778 INFO:tasks.workunit.client.0.smithi151.stdout:4/678: write d0/d11/d19/d87/d3a/d50/fb0 [820635,92916] 0 2024-04-16T02:36:01.781 INFO:tasks.workunit.client.0.smithi151.stdout:8/504: dread d11/f17 [0,4194304] 0 2024-04-16T02:36:01.783 INFO:tasks.workunit.client.0.smithi151.stdout:8/505: chown d11/d2a/d85/f99 1604376 1 2024-04-16T02:36:01.785 INFO:tasks.workunit.client.0.smithi151.stdout:4/679: dwrite d0/d11/d19/d46/dcd/fea [0,4194304] 0 2024-04-16T02:36:01.786 INFO:tasks.workunit.client.0.smithi151.stdout:8/506: fsync ff 0 2024-04-16T02:36:01.788 INFO:tasks.workunit.client.0.smithi151.stdout:8/507: truncate d11/d2a/d85/f99 711210 0 2024-04-16T02:36:01.790 INFO:tasks.workunit.client.0.smithi151.stdout:8/508: read d11/d2a/d33/f49 [1528544,121830] 0 2024-04-16T02:36:01.791 INFO:tasks.workunit.client.0.smithi151.stdout:8/509: truncate d11/d2a/f8c 745461 0 2024-04-16T02:36:01.793 INFO:tasks.workunit.client.0.smithi151.stdout:8/510: chown d11/d2a/d35/d64/f7f 560 1 2024-04-16T02:36:01.808 INFO:tasks.workunit.client.0.smithi151.stdout:8/511: truncate d11/d22/f76 462873 0 2024-04-16T02:36:01.810 INFO:tasks.workunit.client.0.smithi151.stdout:8/512: mknod d11/d2a/d33/d4e/d62/cac 0 2024-04-16T02:36:01.812 INFO:tasks.workunit.client.0.smithi151.stdout:4/680: dwrite d0/dde/d68/f7a [0,4194304] 0 2024-04-16T02:36:01.814 INFO:tasks.workunit.client.0.smithi151.stdout:8/513: creat d11/d22/d6f/d92/d78/fad x:0 0 0 2024-04-16T02:36:01.814 INFO:tasks.workunit.client.0.smithi151.stdout:4/681: rmdir d0/dde/d4a 39 2024-04-16T02:36:01.815 INFO:tasks.workunit.client.0.smithi151.stdout:8/514: chown d11/d22/d6c 294298 1 2024-04-16T02:36:01.817 INFO:tasks.workunit.client.0.smithi151.stdout:8/515: mkdir d11/d2a/d83/dae 0 2024-04-16T02:36:01.821 INFO:tasks.workunit.client.0.smithi151.stdout:8/516: symlink d11/d2a/d33/d4e/d62/d6b/laf 0 2024-04-16T02:36:01.822 INFO:tasks.workunit.client.0.smithi151.stdout:8/517: dread - d11/d2a/d33/d4e/f72 zero size 2024-04-16T02:36:01.822 INFO:tasks.workunit.client.0.smithi151.stdout:8/518: chown d11/l6d 20498 1 2024-04-16T02:36:01.828 INFO:tasks.workunit.client.0.smithi151.stdout:4/682: write d0/d99/f21 [1205460,90098] 0 2024-04-16T02:36:01.828 INFO:tasks.workunit.client.0.smithi151.stdout:8/519: link d11/d2a/f2f d11/d2a/d33/fb0 0 2024-04-16T02:36:01.828 INFO:tasks.workunit.client.0.smithi151.stdout:4/683: chown d0/dde/d45/f9a 2 1 2024-04-16T02:36:01.830 INFO:tasks.workunit.client.0.smithi151.stdout:8/520: mkdir d11/d2a/db1 0 2024-04-16T02:36:01.831 INFO:tasks.workunit.client.0.smithi151.stdout:8/521: fdatasync d11/d22/d2e/f5c 0 2024-04-16T02:36:01.833 INFO:tasks.workunit.client.0.smithi151.stdout:4/684: mknod d0/dde/d4a/d60/de6/cf0 0 2024-04-16T02:36:01.835 INFO:tasks.workunit.client.0.smithi151.stdout:8/522: symlink d11/lb2 0 2024-04-16T02:36:01.836 INFO:tasks.workunit.client.0.smithi151.stdout:8/523: dread - d11/d2a/d35/d64/f88 zero size 2024-04-16T02:36:01.836 INFO:tasks.workunit.client.0.smithi151.stdout:4/685: symlink d0/dde/d80/lf1 0 2024-04-16T02:36:01.838 INFO:tasks.workunit.client.0.smithi151.stdout:4/686: rmdir d0/d11/d19/d87/d3a/d65/d98 39 2024-04-16T02:36:01.838 INFO:tasks.workunit.client.0.smithi151.stdout:4/687: readlink d0/d11/l14 0 2024-04-16T02:36:01.839 INFO:tasks.workunit.client.0.smithi151.stdout:5/765: sync 2024-04-16T02:36:01.839 INFO:tasks.workunit.client.0.smithi151.stdout:1/628: sync 2024-04-16T02:36:01.839 INFO:tasks.workunit.client.0.smithi151.stdout:0/569: sync 2024-04-16T02:36:01.840 INFO:tasks.workunit.client.0.smithi151.stdout:7/731: sync 2024-04-16T02:36:01.840 INFO:tasks.workunit.client.0.smithi151.stdout:3/483: sync 2024-04-16T02:36:01.840 INFO:tasks.workunit.client.0.smithi151.stdout:6/797: sync 2024-04-16T02:36:01.840 INFO:tasks.workunit.client.0.smithi151.stdout:9/635: sync 2024-04-16T02:36:01.840 INFO:tasks.workunit.client.0.smithi151.stdout:4/688: rmdir d0/dde/d68 39 2024-04-16T02:36:01.841 INFO:tasks.workunit.client.0.smithi151.stdout:8/524: getdents d11/d22/d7e 0 2024-04-16T02:36:01.843 INFO:tasks.workunit.client.0.smithi151.stdout:9/636: write d1/d3/d6/d28/fb3 [421411,86031] 0 2024-04-16T02:36:01.846 INFO:tasks.workunit.client.0.smithi151.stdout:7/732: symlink d0/d2c/d38/d43/d78/l101 0 2024-04-16T02:36:01.849 INFO:tasks.workunit.client.0.smithi151.stdout:1/629: chown d3/d58/df/d32/d3a/d67/c7d 250034 1 2024-04-16T02:36:01.850 INFO:tasks.workunit.client.0.smithi151.stdout:0/570: unlink dc/d20/f4a 0 2024-04-16T02:36:01.850 INFO:tasks.workunit.client.0.smithi151.stdout:4/689: mknod d0/d11/d19/d46/dcd/cf2 0 2024-04-16T02:36:01.853 INFO:tasks.workunit.client.0.smithi151.stdout:5/766: dread d2/d6/d43/f6d [0,4194304] 0 2024-04-16T02:36:01.855 INFO:tasks.workunit.client.0.smithi151.stdout:8/525: mkdir d11/d2a/d35/d52/db3 0 2024-04-16T02:36:01.855 INFO:tasks.workunit.client.0.smithi151.stdout:5/767: chown d2/d6/d19/d1a/f2e 42848 1 2024-04-16T02:36:01.861 INFO:tasks.workunit.client.0.smithi151.stdout:1/630: rename d3/d4/d64/d40/fd3 to d3/d33/d10/d4f/dd1/fd4 0 2024-04-16T02:36:01.861 INFO:tasks.workunit.client.0.smithi151.stdout:3/484: dread d1/d6/d22/d23/d30/d44/d46/d8a/fa3 [0,4194304] 0 2024-04-16T02:36:01.862 INFO:tasks.workunit.client.0.smithi151.stdout:8/526: dwrite d11/d2a/d35/d64/f7f [4194304,4194304] 0 2024-04-16T02:36:01.862 INFO:tasks.workunit.client.0.smithi151.stdout:0/571: creat dc/d15/d34/d67/d51/fb9 x:0 0 0 2024-04-16T02:36:01.866 INFO:tasks.workunit.client.0.smithi151.stdout:7/733: write d0/d1f/d33/f3a [3022860,49521] 0 2024-04-16T02:36:01.868 INFO:tasks.workunit.client.0.smithi151.stdout:5/768: mkdir d2/d6/d19/d62/dd0/df6 0 2024-04-16T02:36:01.870 INFO:tasks.workunit.client.0.smithi151.stdout:3/485: mkdir d1/d69/da4 0 2024-04-16T02:36:01.870 INFO:tasks.workunit.client.0.smithi151.stdout:6/798: dwrite d2/d13/d63/def/df/d23/d45/fd1 [0,4194304] 0 2024-04-16T02:36:01.876 INFO:tasks.workunit.client.0.smithi151.stdout:8/527: creat d11/d22/d6f/d92/d78/fb4 x:0 0 0 2024-04-16T02:36:01.878 INFO:tasks.workunit.client.0.smithi151.stdout:9/637: truncate d1/dd/f8a 4149892 0 2024-04-16T02:36:01.881 INFO:tasks.workunit.client.0.smithi151.stdout:0/572: rename dc/d20/d7b/d9c/ca7 to dc/d15/d34/d67/d69/cba 0 2024-04-16T02:36:01.882 INFO:tasks.workunit.client.0.smithi151.stdout:1/631: mknod d3/d58/df/d32/d3a/d67/dcb/cd5 0 2024-04-16T02:36:01.883 INFO:tasks.workunit.client.0.smithi151.stdout:5/769: rmdir d2/d6/d19/d30/d36/d42 39 2024-04-16T02:36:01.885 INFO:tasks.workunit.client.0.smithi151.stdout:3/486: dread - d1/d6/d22/d23/d30/d44/d46/d8a/f28 zero size 2024-04-16T02:36:01.886 INFO:tasks.workunit.client.0.smithi151.stdout:3/487: read - d1/d69/f9b zero size 2024-04-16T02:36:01.886 INFO:tasks.workunit.client.0.smithi151.stdout:6/799: mkdir d2/d13/d63/def/df/d20/d101/de5/d10e 0 2024-04-16T02:36:01.887 INFO:tasks.workunit.client.0.smithi151.stdout:8/528: symlink d11/d2a/d33/d4e/d62/lb5 0 2024-04-16T02:36:01.889 INFO:tasks.workunit.client.0.smithi151.stdout:9/638: creat d1/d3/d6/d28/fcd x:0 0 0 2024-04-16T02:36:01.897 INFO:tasks.workunit.client.0.smithi151.stdout:3/488: fsync d1/f5c 0 2024-04-16T02:36:01.897 INFO:tasks.workunit.client.0.smithi151.stdout:6/800: fdatasync d2/d3d/f55 0 2024-04-16T02:36:01.901 INFO:tasks.workunit.client.0.smithi151.stdout:7/734: dread d0/f9e [0,4194304] 0 2024-04-16T02:36:01.902 INFO:tasks.workunit.client.0.smithi151.stdout:9/639: truncate d1/d3/fa 9057624 0 2024-04-16T02:36:01.904 INFO:tasks.workunit.client.0.smithi151.stdout:4/690: dwrite d0/dde/d68/f79 [0,4194304] 0 2024-04-16T02:36:01.904 INFO:tasks.workunit.client.0.smithi151.stdout:1/632: dwrite d3/d33/d10/d4f/dd1/fd4 [0,4194304] 0 2024-04-16T02:36:01.909 INFO:tasks.workunit.client.0.smithi151.stdout:0/573: write fa [848038,73339] 0 2024-04-16T02:36:01.909 INFO:tasks.workunit.client.0.smithi151.stdout:8/529: creat d11/d2a/d35/d52/db3/fb6 x:0 0 0 2024-04-16T02:36:01.913 INFO:tasks.workunit.client.0.smithi151.stdout:6/801: write d2/d3d/f55 [584377,91053] 0 2024-04-16T02:36:01.914 INFO:tasks.workunit.client.0.smithi151.stdout:3/489: symlink d1/d37/la5 0 2024-04-16T02:36:01.918 INFO:tasks.workunit.client.0.smithi151.stdout:9/640: rename d1/d3/d6/d28/l32 to d1/d3/d10/da6/lce 0 2024-04-16T02:36:01.920 INFO:tasks.workunit.client.0.smithi151.stdout:4/691: mknod d0/d11/d19/dd7/cf3 0 2024-04-16T02:36:01.921 INFO:tasks.workunit.client.0.smithi151.stdout:5/770: dwrite d2/d6/f1c [0,4194304] 0 2024-04-16T02:36:01.924 INFO:tasks.workunit.client.0.smithi151.stdout:0/574: creat dc/d20/fbb x:0 0 0 2024-04-16T02:36:01.925 INFO:tasks.workunit.client.0.smithi151.stdout:3/490: dread d1/f5c [0,4194304] 0 2024-04-16T02:36:01.927 INFO:tasks.workunit.client.0.smithi151.stdout:3/491: chown d1/d6/d22/d23/d41/f5b 5966 1 2024-04-16T02:36:01.928 INFO:tasks.workunit.client.0.smithi151.stdout:9/641: stat d1/d3/d6/d17/d54/d9b/d22/c26 0 2024-04-16T02:36:01.928 INFO:tasks.workunit.client.0.smithi151.stdout:8/530: dread d11/d22/d2e/f5c [0,4194304] 0 2024-04-16T02:36:01.932 INFO:tasks.workunit.client.0.smithi151.stdout:7/735: dread d0/d1f/d4c/d98/fa4 [0,4194304] 0 2024-04-16T02:36:01.932 INFO:tasks.workunit.client.0.smithi151.stdout:5/771: symlink d2/d6/d19/d30/d88/ddc/da8/lf7 0 2024-04-16T02:36:01.934 INFO:tasks.workunit.client.0.smithi151.stdout:7/736: write d0/d1f/d60/fbd [549249,117683] 0 2024-04-16T02:36:01.934 INFO:tasks.workunit.client.0.smithi151.stdout:3/492: fdatasync d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/f97 0 2024-04-16T02:36:01.936 INFO:tasks.workunit.client.0.smithi151.stdout:6/802: link d2/d13/f67 d2/d13/d6f/da3/f10f 0 2024-04-16T02:36:01.937 INFO:tasks.workunit.client.0.smithi151.stdout:1/633: rmdir d3/d58/df/d32/d3a/da5 0 2024-04-16T02:36:01.940 INFO:tasks.workunit.client.0.smithi151.stdout:0/575: creat dc/d20/fbc x:0 0 0 2024-04-16T02:36:01.941 INFO:tasks.workunit.client.0.smithi151.stdout:8/531: mkdir d11/d2a/d33/d4e/d62/db7 0 2024-04-16T02:36:01.941 INFO:tasks.workunit.client.0.smithi151.stdout:9/642: symlink d1/d75/lcf 0 2024-04-16T02:36:01.943 INFO:tasks.workunit.client.0.smithi151.stdout:5/772: fsync d2/d6/f6c 0 2024-04-16T02:36:01.947 INFO:tasks.workunit.client.0.smithi151.stdout:6/803: dwrite d2/d13/d63/def/df/d20/de6/ff1 [0,4194304] 0 2024-04-16T02:36:01.948 INFO:tasks.workunit.client.0.smithi151.stdout:3/493: unlink d1/d6/ff 0 2024-04-16T02:36:01.951 INFO:tasks.workunit.client.0.smithi151.stdout:1/634: dread - d3/d58/df/f6c zero size 2024-04-16T02:36:01.954 INFO:tasks.workunit.client.0.smithi151.stdout:9/643: dwrite d1/d3/d6/d17/d40/d56/dad/fb5 [0,4194304] 0 2024-04-16T02:36:01.954 INFO:tasks.workunit.client.0.smithi151.stdout:4/692: rename d0/d64/fe2 to d0/d11/d19/ff4 0 2024-04-16T02:36:01.954 INFO:tasks.workunit.client.0.smithi151.stdout:1/635: chown d3/d4/db2/fc3 0 1 2024-04-16T02:36:01.958 INFO:tasks.workunit.client.0.smithi151.stdout:0/576: rmdir dc/d15/d34/d67/d69/d6f/d9e 39 2024-04-16T02:36:01.965 INFO:tasks.workunit.client.0.smithi151.stdout:6/804: dwrite d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d85/fe8 [0,4194304] 0 2024-04-16T02:36:01.965 INFO:tasks.workunit.client.0.smithi151.stdout:7/737: mkdir d0/d1f/d33/d102 0 2024-04-16T02:36:01.967 INFO:tasks.workunit.client.0.smithi151.stdout:8/532: chown d11/d22/d6f/d92/d78/f7b 773989007 1 2024-04-16T02:36:01.967 INFO:tasks.workunit.client.0.smithi151.stdout:5/773: write d2/d6/d43/f6d [3608932,31098] 0 2024-04-16T02:36:01.972 INFO:tasks.workunit.client.0.smithi151.stdout:1/636: dwrite d3/d58/df/d32/d3a/d71/fbf [0,4194304] 0 2024-04-16T02:36:01.980 INFO:tasks.workunit.client.0.smithi151.stdout:0/577: mknod dc/d15/d5c/cbd 0 2024-04-16T02:36:01.985 INFO:tasks.workunit.client.0.smithi151.stdout:5/774: truncate d2/f99 6472519 0 2024-04-16T02:36:01.986 INFO:tasks.workunit.client.0.smithi151.stdout:8/533: mknod d11/d2a/d85/cb8 0 2024-04-16T02:36:01.991 INFO:tasks.workunit.client.0.smithi151.stdout:0/578: truncate dc/d15/d34/d4e/f6d 439686 0 2024-04-16T02:36:01.993 INFO:tasks.workunit.client.0.smithi151.stdout:7/738: fdatasync d0/d1f/d4c/fa0 0 2024-04-16T02:36:01.995 INFO:tasks.workunit.client.0.smithi151.stdout:6/805: symlink d2/d13/d6f/l110 0 2024-04-16T02:36:01.997 INFO:tasks.workunit.client.0.smithi151.stdout:8/534: stat d11/l34 0 2024-04-16T02:36:02.000 INFO:tasks.workunit.client.0.smithi151.stdout:4/693: link d0/dde/d4a/d60/de6/f2b d0/dde/dad/ff5 0 2024-04-16T02:36:02.000 INFO:tasks.workunit.client.0.smithi151.stdout:9/644: dread d1/d3/d6/d17/d40/d56/f61 [0,4194304] 0 2024-04-16T02:36:02.003 INFO:tasks.workunit.client.0.smithi151.stdout:9/645: write d1/d3/d5a/d8e/f9a [4456648,96673] 0 2024-04-16T02:36:02.004 INFO:tasks.workunit.client.0.smithi151.stdout:3/494: dwrite d1/d6/d22/d23/d30/d44/f5f [0,4194304] 0 2024-04-16T02:36:02.005 INFO:tasks.workunit.client.0.smithi151.stdout:9/646: readlink d1/d3/l37 0 2024-04-16T02:36:02.006 INFO:tasks.workunit.client.0.smithi151.stdout:3/495: write d1/d6/d22/f2c [2857099,91794] 0 2024-04-16T02:36:02.007 INFO:tasks.workunit.client.0.smithi151.stdout:7/739: symlink d0/d1f/d4c/d82/l103 0 2024-04-16T02:36:02.008 INFO:tasks.workunit.client.0.smithi151.stdout:3/496: chown d1/d6/d20/d3d 6973 1 2024-04-16T02:36:02.009 INFO:tasks.workunit.client.0.smithi151.stdout:7/740: dread - d0/d2c/d38/d5e/f92 zero size 2024-04-16T02:36:02.014 INFO:tasks.workunit.client.0.smithi151.stdout:1/637: creat d3/fd6 x:0 0 0 2024-04-16T02:36:02.014 INFO:tasks.workunit.client.0.smithi151.stdout:9/647: dwrite d1/d3/d6/d17/d40/d56/dad/fb5 [0,4194304] 0 2024-04-16T02:36:02.017 INFO:tasks.workunit.client.0.smithi151.stdout:9/648: chown d1/d3/d6/d17/d54/d9b/d22/d36 92377 1 2024-04-16T02:36:02.018 INFO:tasks.workunit.client.0.smithi151.stdout:8/535: rmdir d11/d2a/d33 39 2024-04-16T02:36:02.019 INFO:tasks.workunit.client.0.smithi151.stdout:9/649: readlink d1/d3/d6/d17/d54/l71 0 2024-04-16T02:36:02.021 INFO:tasks.workunit.client.0.smithi151.stdout:5/775: rename d2/d6/d19/d30/d36/cda to d2/d6/d19/d30/d49/d7f/cf8 0 2024-04-16T02:36:02.028 INFO:tasks.workunit.client.0.smithi151.stdout:6/806: dwrite d2/d13/d6f/da3/f10f [0,4194304] 0 2024-04-16T02:36:02.030 INFO:tasks.workunit.client.0.smithi151.stdout:4/694: chown d0/d99/l2a 599334 1 2024-04-16T02:36:02.033 INFO:tasks.workunit.client.0.smithi151.stdout:1/638: unlink d3/d4/db2/fb8 0 2024-04-16T02:36:02.038 INFO:tasks.workunit.client.0.smithi151.stdout:9/650: rename d1/d3/d6/fbe to d1/d3/d6/d17/d54/d9b/dae/fd0 0 2024-04-16T02:36:02.040 INFO:tasks.workunit.client.0.smithi151.stdout:0/579: write dc/f39 [2675352,130277] 0 2024-04-16T02:36:02.041 INFO:tasks.workunit.client.0.smithi151.stdout:5/776: fdatasync d2/d6/d19/d30/f39 0 2024-04-16T02:36:02.042 INFO:tasks.workunit.client.0.smithi151.stdout:0/580: chown dc/d15/d34/d3b/d71/fb3 197850 1 2024-04-16T02:36:02.044 INFO:tasks.workunit.client.0.smithi151.stdout:0/581: dread - dc/d15/d34/d3b/f70 zero size 2024-04-16T02:36:02.048 INFO:tasks.workunit.client.0.smithi151.stdout:1/639: mkdir d3/d1f/dd7 0 2024-04-16T02:36:02.048 INFO:tasks.workunit.client.0.smithi151.stdout:8/536: mkdir d11/d2a/d33/d4e/d62/d6b/db9 0 2024-04-16T02:36:02.051 INFO:tasks.workunit.client.0.smithi151.stdout:9/651: rename d1/d3/l37 to d1/d3/d5a/d8e/ld1 0 2024-04-16T02:36:02.054 INFO:tasks.workunit.client.0.smithi151.stdout:6/807: creat d2/d13/d63/def/df/d20/d101/de5/d10e/f111 x:0 0 0 2024-04-16T02:36:02.055 INFO:tasks.workunit.client.0.smithi151.stdout:7/741: dread d0/d2c/f3b [0,4194304] 0 2024-04-16T02:36:02.055 INFO:tasks.workunit.client.0.smithi151.stdout:3/497: stat d1/d6/d22/d23/d30/d44/d46/d8a/d3f/f48 0 2024-04-16T02:36:02.057 INFO:tasks.workunit.client.0.smithi151.stdout:3/498: chown d1/d6/d22/d23/l6a 434223 1 2024-04-16T02:36:02.058 INFO:tasks.workunit.client.0.smithi151.stdout:9/652: creat d1/d75/fd2 x:0 0 0 2024-04-16T02:36:02.058 INFO:tasks.workunit.client.0.smithi151.stdout:5/777: symlink d2/d6/d19/lf9 0 2024-04-16T02:36:02.059 INFO:tasks.workunit.client.0.smithi151.stdout:5/778: write d2/d6/f27 [987740,21678] 0 2024-04-16T02:36:02.061 INFO:tasks.workunit.client.0.smithi151.stdout:6/808: mkdir d2/d13/d63/d112 0 2024-04-16T02:36:02.065 INFO:tasks.workunit.client.0.smithi151.stdout:5/779: truncate d2/d6/d19/ff0 176527 0 2024-04-16T02:36:02.065 INFO:tasks.workunit.client.0.smithi151.stdout:0/582: link dc/d4c/d72/d76/db0/fb7 dc/d15/d34/d67/d51/fbe 0 2024-04-16T02:36:02.066 INFO:tasks.workunit.client.0.smithi151.stdout:7/742: getdents d0/d1f/d33/d102 0 2024-04-16T02:36:02.068 INFO:tasks.workunit.client.0.smithi151.stdout:6/809: mknod d2/d13/d3a/c113 0 2024-04-16T02:36:02.069 INFO:tasks.workunit.client.0.smithi151.stdout:9/653: dwrite d1/dd/d4a/dbb/fc7 [0,4194304] 0 2024-04-16T02:36:02.072 INFO:tasks.workunit.client.0.smithi151.stdout:4/695: write d0/d11/d19/f1b [8351781,31843] 0 2024-04-16T02:36:02.077 INFO:tasks.workunit.client.0.smithi151.stdout:7/743: creat d0/d1f/d4c/f104 x:0 0 0 2024-04-16T02:36:02.080 INFO:tasks.workunit.client.0.smithi151.stdout:1/640: write d3/d4/d64/fa8 [114932,52250] 0 2024-04-16T02:36:02.082 INFO:tasks.workunit.client.0.smithi151.stdout:4/696: mknod d0/daa/dcc/cf6 0 2024-04-16T02:36:02.082 INFO:tasks.workunit.client.0.smithi151.stdout:8/537: dread d11/d2a/d33/f4b [0,4194304] 0 2024-04-16T02:36:02.088 INFO:tasks.workunit.client.0.smithi151.stdout:0/583: creat dc/d15/d34/fbf x:0 0 0 2024-04-16T02:36:02.093 INFO:tasks.workunit.client.0.smithi151.stdout:4/697: creat d0/d9b/ff7 x:0 0 0 2024-04-16T02:36:02.096 INFO:tasks.workunit.client.0.smithi151.stdout:6/810: link d2/d13/d6f/da3/f10f d2/d13/d6f/f114 0 2024-04-16T02:36:02.096 INFO:tasks.workunit.client.0.smithi151.stdout:7/744: dwrite d0/d1f/d33/d56/f84 [0,4194304] 0 2024-04-16T02:36:02.097 INFO:tasks.workunit.client.0.smithi151.stdout:5/780: getdents d2/d6/d19/d30/d88/ddc/d83 0 2024-04-16T02:36:02.097 INFO:tasks.workunit.client.0.smithi151.stdout:8/538: dwrite d11/d2a/f42 [4194304,4194304] 0 2024-04-16T02:36:02.099 INFO:tasks.workunit.client.0.smithi151.stdout:9/654: creat d1/d3/d6/d17/d54/d9b/d22/d2c/d86/fd3 x:0 0 0 2024-04-16T02:36:02.100 INFO:tasks.workunit.client.0.smithi151.stdout:5/781: readlink d2/d6/d19/l22 0 2024-04-16T02:36:02.101 INFO:tasks.workunit.client.0.smithi151.stdout:3/499: dwrite d1/f1d [0,4194304] 0 2024-04-16T02:36:02.105 INFO:tasks.workunit.client.0.smithi151.stdout:6/811: fsync d2/d13/d63/def/df/d23/d56/f58 0 2024-04-16T02:36:02.106 INFO:tasks.workunit.client.0.smithi151.stdout:0/584: getdents dc/d15/d34/d67/d69/d6f 0 2024-04-16T02:36:02.106 INFO:tasks.workunit.client.0.smithi151.stdout:4/698: truncate d0/dde/d4a/d60/fda 586506 0 2024-04-16T02:36:02.109 INFO:tasks.workunit.client.0.smithi151.stdout:7/745: readlink d0/d1f/d4c/d82/le1 0 2024-04-16T02:36:02.111 INFO:tasks.workunit.client.0.smithi151.stdout:7/746: chown d0/d2c/d38/f9b 90598415 1 2024-04-16T02:36:02.112 INFO:tasks.workunit.client.0.smithi151.stdout:5/782: read d2/d6/f32 [503530,121184] 0 2024-04-16T02:36:02.114 INFO:tasks.workunit.client.0.smithi151.stdout:1/641: dwrite d3/d4/d64/d2a/d52/fa9 [0,4194304] 0 2024-04-16T02:36:02.119 INFO:tasks.workunit.client.0.smithi151.stdout:0/585: creat dc/d4c/d72/d76/db0/db5/fc0 x:0 0 0 2024-04-16T02:36:02.119 INFO:tasks.workunit.client.0.smithi151.stdout:5/783: chown d2/f1f 317 1 2024-04-16T02:36:02.120 INFO:tasks.workunit.client.0.smithi151.stdout:1/642: chown d3/d58/df/d68 14512 1 2024-04-16T02:36:02.121 INFO:tasks.workunit.client.0.smithi151.stdout:0/586: chown dc/d15/d34/d67/d69/d9a/lae 35081 1 2024-04-16T02:36:02.123 INFO:tasks.workunit.client.0.smithi151.stdout:4/699: mkdir d0/d11/d19/d87/d3a/d65/d98/df8 0 2024-04-16T02:36:02.123 INFO:tasks.workunit.client.0.smithi151.stdout:6/812: symlink d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d65/l115 0 2024-04-16T02:36:02.125 INFO:tasks.workunit.client.0.smithi151.stdout:7/747: dwrite d0/d1f/d33/d56/f84 [4194304,4194304] 0 2024-04-16T02:36:02.127 INFO:tasks.workunit.client.0.smithi151.stdout:4/700: creat d0/dde/d45/dd2/ff9 x:0 0 0 2024-04-16T02:36:02.128 INFO:tasks.workunit.client.0.smithi151.stdout:1/643: fdatasync d3/d33/f39 0 2024-04-16T02:36:02.128 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:01 smithi151 ceph-mon[29003]: pgmap v28: 65 pgs: 65 active+clean; 3.2 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 49 MiB/s rd, 113 MiB/s wr, 284 op/s 2024-04-16T02:36:02.131 INFO:tasks.workunit.client.0.smithi151.stdout:0/587: symlink dc/d15/d34/d67/lc1 0 2024-04-16T02:36:02.138 INFO:tasks.workunit.client.0.smithi151.stdout:4/701: write d0/dde/d80/fc0 [505381,111606] 0 2024-04-16T02:36:02.141 INFO:tasks.workunit.client.0.smithi151.stdout:1/644: creat d3/d33/d10/d4f/fd8 x:0 0 0 2024-04-16T02:36:02.142 INFO:tasks.workunit.client.0.smithi151.stdout:0/588: readlink dc/l3a 0 2024-04-16T02:36:02.142 INFO:tasks.workunit.client.0.smithi151.stdout:3/500: dread d1/d37/f38 [0,4194304] 0 2024-04-16T02:36:02.142 INFO:tasks.workunit.client.0.smithi151.stdout:1/645: read - d3/d1f/d48/fb7 zero size 2024-04-16T02:36:02.145 INFO:tasks.workunit.client.0.smithi151.stdout:3/501: truncate d1/d6/d22/d6c/f9c 641818 0 2024-04-16T02:36:02.145 INFO:tasks.workunit.client.0.smithi151.stdout:7/748: unlink d0/d27/da6/cac 0 2024-04-16T02:36:02.146 INFO:tasks.workunit.client.0.smithi151.stdout:4/702: truncate d0/dde/f47 1883472 0 2024-04-16T02:36:02.147 INFO:tasks.workunit.client.0.smithi151.stdout:1/646: creat d3/d33/d10/fd9 x:0 0 0 2024-04-16T02:36:02.150 INFO:tasks.workunit.client.0.smithi151.stdout:7/749: mkdir d0/d2c/d38/d85/db7/d105 0 2024-04-16T02:36:02.151 INFO:tasks.workunit.client.0.smithi151.stdout:0/589: fsync dc/d15/d34/d4e/f6d 0 2024-04-16T02:36:02.155 INFO:tasks.workunit.client.0.smithi151.stdout:8/539: truncate d11/d2a/d33/d4e/f7c 1097403 0 2024-04-16T02:36:02.158 INFO:tasks.workunit.client.0.smithi151.stdout:3/502: dwrite d1/d6/d22/f52 [0,4194304] 0 2024-04-16T02:36:02.158 INFO:tasks.workunit.client.0.smithi151.stdout:1/647: rename d3/d4/d64/d1c/db9 to d3/d1f/dd7/dda 0 2024-04-16T02:36:02.162 INFO:tasks.workunit.client.0.smithi151.stdout:0/590: dwrite fa [0,4194304] 0 2024-04-16T02:36:02.163 INFO:tasks.workunit.client.0.smithi151.stdout:5/784: write d2/d6/d19/d30/d88/ddc/d83/fe1 [85221,111668] 0 2024-04-16T02:36:02.164 INFO:tasks.workunit.client.0.smithi151.stdout:5/785: stat d2/d6/d19/d62/dd0/c82 0 2024-04-16T02:36:02.166 INFO:tasks.workunit.client.0.smithi151.stdout:5/786: chown d2/d6/d19/d62/dd0/c7c 2 1 2024-04-16T02:36:02.167 INFO:tasks.workunit.client.0.smithi151.stdout:9/655: dread d1/d75/f8b [0,4194304] 0 2024-04-16T02:36:02.168 INFO:tasks.workunit.client.0.smithi151.stdout:6/813: truncate d2/d3d/f55 963914 0 2024-04-16T02:36:02.170 INFO:tasks.workunit.client.0.smithi151.stdout:3/503: mkdir d1/d6/d20/d3d/da6 0 2024-04-16T02:36:02.173 INFO:tasks.workunit.client.0.smithi151.stdout:0/591: mkdir dc/d3c/dc2 0 2024-04-16T02:36:02.175 INFO:tasks.workunit.client.0.smithi151.stdout:8/540: mknod d11/d2a/d83/dae/cba 0 2024-04-16T02:36:02.177 INFO:tasks.workunit.client.0.smithi151.stdout:5/787: symlink d2/d6/db7/lfa 0 2024-04-16T02:36:02.181 INFO:tasks.workunit.client.0.smithi151.stdout:6/814: unlink d2/fe1 0 2024-04-16T02:36:02.181 INFO:tasks.workunit.client.0.smithi151.stdout:7/750: write d0/d27/f2b [7751788,121599] 0 2024-04-16T02:36:02.181 INFO:tasks.workunit.client.0.smithi151.stdout:4/703: dwrite d0/d9b/faf [0,4194304] 0 2024-04-16T02:36:02.184 INFO:tasks.workunit.client.0.smithi151.stdout:9/656: mkdir d1/dd4 0 2024-04-16T02:36:02.184 INFO:tasks.workunit.client.0.smithi151.stdout:6/815: chown d2/d13/d63/def/d2e/dba/c108 50755 1 2024-04-16T02:36:02.185 INFO:tasks.workunit.client.0.smithi151.stdout:0/592: chown dc/f47 34493 1 2024-04-16T02:36:02.186 INFO:tasks.workunit.client.0.smithi151.stdout:0/593: dread - dc/d20/fbb zero size 2024-04-16T02:36:02.189 INFO:tasks.workunit.client.0.smithi151.stdout:8/541: symlink d11/d22/d6f/d92/d78/lbb 0 2024-04-16T02:36:02.192 INFO:tasks.workunit.client.0.smithi151.stdout:5/788: mknod d2/d6/d19/d30/d49/d85/cfb 0 2024-04-16T02:36:02.195 INFO:tasks.workunit.client.0.smithi151.stdout:7/751: mkdir d0/d1f/d4c/d82/d106 0 2024-04-16T02:36:02.196 INFO:tasks.workunit.client.0.smithi151.stdout:7/752: write d0/d1f/d33/fe6 [2857648,6502] 0 2024-04-16T02:36:02.197 INFO:tasks.workunit.client.0.smithi151.stdout:4/704: dread d0/f3 [0,4194304] 0 2024-04-16T02:36:02.199 INFO:tasks.workunit.client.0.smithi151.stdout:4/705: write d0/d11/d19/f1b [3614254,26397] 0 2024-04-16T02:36:02.200 INFO:tasks.workunit.client.0.smithi151.stdout:9/657: write d1/d3/d5a/d8e/f9a [4025039,60054] 0 2024-04-16T02:36:02.202 INFO:tasks.workunit.client.0.smithi151.stdout:6/816: truncate d2/d13/d63/def/df/d20/f47 1094024 0 2024-04-16T02:36:02.202 INFO:tasks.workunit.client.0.smithi151.stdout:9/658: chown d1/dd/c52 2077482 1 2024-04-16T02:36:02.206 INFO:tasks.workunit.client.0.smithi151.stdout:3/504: link d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/f97 d1/d6/d22/d23/d41/d9d/fa7 0 2024-04-16T02:36:02.208 INFO:tasks.workunit.client.0.smithi151.stdout:1/648: truncate d3/d4/d64/d1c/f81 816136 0 2024-04-16T02:36:02.209 INFO:tasks.workunit.client.0.smithi151.stdout:8/542: mkdir d11/d22/d6f/dbc 0 2024-04-16T02:36:02.215 INFO:tasks.workunit.client.0.smithi151.stdout:7/753: dwrite d0/d1f/d4c/d98/fb5 [0,4194304] 0 2024-04-16T02:36:02.219 INFO:tasks.workunit.client.0.smithi151.stdout:0/594: dwrite dc/d15/d34/d67/f5f [0,4194304] 0 2024-04-16T02:36:02.222 INFO:tasks.workunit.client.0.smithi151.stdout:4/706: dwrite d0/d64/fd6 [0,4194304] 0 2024-04-16T02:36:02.225 INFO:tasks.workunit.client.0.smithi151.stdout:4/707: chown d0/d11/d19/d87/d3a/d50/l57 2455 1 2024-04-16T02:36:02.227 INFO:tasks.workunit.client.0.smithi151.stdout:5/789: mkdir d2/d6/dfc 0 2024-04-16T02:36:02.234 INFO:tasks.workunit.client.0.smithi151.stdout:8/543: rmdir d11/d2a/d33/d4e/d62/d6b 39 2024-04-16T02:36:02.236 INFO:tasks.workunit.client.0.smithi151.stdout:8/544: write d11/d2a/d33/d4e/f9e [174844,92757] 0 2024-04-16T02:36:02.237 INFO:tasks.workunit.client.0.smithi151.stdout:3/505: rename d1/d69/f9b to d1/d6/d22/d23/d30/fa8 0 2024-04-16T02:36:02.238 INFO:tasks.workunit.client.0.smithi151.stdout:0/595: chown dc/d15/d34/d67/f3e 60 1 2024-04-16T02:36:02.242 INFO:tasks.workunit.client.0.smithi151.stdout:4/708: mkdir d0/d11/d19/d87/d7b/dfa 0 2024-04-16T02:36:02.243 INFO:tasks.workunit.client.0.smithi151.stdout:9/659: dread d1/d3/d6/f13 [0,4194304] 0 2024-04-16T02:36:02.245 INFO:tasks.workunit.client.0.smithi151.stdout:8/545: dwrite d11/d2a/d35/d64/f7f [4194304,4194304] 0 2024-04-16T02:36:02.249 INFO:tasks.workunit.client.0.smithi151.stdout:3/506: rmdir d1/d6/d4a 39 2024-04-16T02:36:02.249 INFO:tasks.workunit.client.0.smithi151.stdout:9/660: fdatasync d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/fc6 0 2024-04-16T02:36:02.252 INFO:tasks.workunit.client.0.smithi151.stdout:0/596: mkdir dc/d15/d34/d3b/d71/dc3 0 2024-04-16T02:36:02.255 INFO:tasks.workunit.client.0.smithi151.stdout:0/597: stat dc/d20/f94 0 2024-04-16T02:36:02.255 INFO:tasks.workunit.client.0.smithi151.stdout:1/649: dwrite d3/d58/df/d38/f74 [0,4194304] 0 2024-04-16T02:36:02.257 INFO:tasks.workunit.client.0.smithi151.stdout:1/650: chown d3/d4/d64/d2a/d52/d97/lc8 1731589 1 2024-04-16T02:36:02.261 INFO:tasks.workunit.client.0.smithi151.stdout:6/817: getdents d2/d13/d63/def/df/d23/d82 0 2024-04-16T02:36:02.262 INFO:tasks.workunit.client.0.smithi151.stdout:9/661: fdatasync d1/d3/d6/f1b 0 2024-04-16T02:36:02.266 INFO:tasks.workunit.client.0.smithi151.stdout:0/598: fsync dc/d3c/f84 0 2024-04-16T02:36:02.268 INFO:tasks.workunit.client.0.smithi151.stdout:7/754: link d0/d2c/d38/d43/l67 d0/d2c/d8e/l107 0 2024-04-16T02:36:02.268 INFO:tasks.workunit.client.0.smithi151.stdout:0/599: stat dc/d15/d34/d3b/d71/dc3 0 2024-04-16T02:36:02.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:01 smithi153 ceph-mon[36747]: pgmap v28: 65 pgs: 65 active+clean; 3.2 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 49 MiB/s rd, 113 MiB/s wr, 284 op/s 2024-04-16T02:36:02.402 INFO:tasks.workunit.client.0.smithi151.stdout:1/651: mkdir d3/d1f/dd7/ddb 0 2024-04-16T02:36:02.404 INFO:tasks.workunit.client.0.smithi151.stdout:3/507: fsync d1/d6/d22/f2b 0 2024-04-16T02:36:02.407 INFO:tasks.workunit.client.0.smithi151.stdout:9/662: mkdir d1/d3/d6/d17/d54/dd5 0 2024-04-16T02:36:02.408 INFO:tasks.workunit.client.0.smithi151.stdout:6/818: creat d2/d13/d63/def/df/d23/d31/d52/df7/f116 x:0 0 0 2024-04-16T02:36:02.408 INFO:tasks.workunit.client.0.smithi151.stdout:5/790: link d2/d6/d19/l2c d2/d6/d19/d30/d36/d42/lfd 0 2024-04-16T02:36:02.410 INFO:tasks.workunit.client.0.smithi151.stdout:5/791: chown d2/d6/d19/d1a/f47 6744778 1 2024-04-16T02:36:02.412 INFO:tasks.workunit.client.0.smithi151.stdout:0/600: creat dc/d15/d34/d4e/fc4 x:0 0 0 2024-04-16T02:36:02.414 INFO:tasks.workunit.client.0.smithi151.stdout:1/652: symlink d3/d1f/d48/ldc 0 2024-04-16T02:36:02.414 INFO:tasks.workunit.client.0.smithi151.stdout:6/819: truncate d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d65/fdd 5068276 0 2024-04-16T02:36:02.416 INFO:tasks.workunit.client.0.smithi151.stdout:8/546: rmdir d11/d2a/d33/d4e/d62/d94/da4 0 2024-04-16T02:36:02.418 INFO:tasks.workunit.client.0.smithi151.stdout:5/792: truncate d2/d6/d19/d1a/f40 1172503 0 2024-04-16T02:36:02.419 INFO:tasks.workunit.client.0.smithi151.stdout:0/601: fsync dc/d15/d34/d67/d51/f91 0 2024-04-16T02:36:02.420 INFO:tasks.workunit.client.0.smithi151.stdout:8/547: dread d11/d2a/d33/d4e/f9e [0,4194304] 0 2024-04-16T02:36:02.421 INFO:tasks.workunit.client.0.smithi151.stdout:8/548: stat d11/d2a/d33/l57 0 2024-04-16T02:36:02.422 INFO:tasks.workunit.client.0.smithi151.stdout:3/508: symlink d1/d6/la9 0 2024-04-16T02:36:02.423 INFO:tasks.workunit.client.0.smithi151.stdout:6/820: mknod d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d73/d97/c117 0 2024-04-16T02:36:02.425 INFO:tasks.workunit.client.0.smithi151.stdout:6/821: chown d2/d13/d63/def/df/d23/d82/d8d/d22/f9b 63506545 1 2024-04-16T02:36:02.425 INFO:tasks.workunit.client.0.smithi151.stdout:7/755: rmdir d0/d27/da6/dcd/df3 0 2024-04-16T02:36:02.426 INFO:tasks.workunit.client.0.smithi151.stdout:7/756: fdatasync d0/d2c/d38/d5e/f100 0 2024-04-16T02:36:02.427 INFO:tasks.workunit.client.0.smithi151.stdout:7/757: chown d0/d1f/d4c/d82/db3 555669 1 2024-04-16T02:36:02.430 INFO:tasks.workunit.client.0.smithi151.stdout:0/602: mknod dc/d15/d34/d67/d69/d6f/d9e/cc5 0 2024-04-16T02:36:02.432 INFO:tasks.workunit.client.0.smithi151.stdout:3/509: mkdir d1/d89/daa 0 2024-04-16T02:36:02.433 INFO:tasks.workunit.client.0.smithi151.stdout:6/822: mknod d2/d13/d5f/dd7/c118 0 2024-04-16T02:36:02.435 INFO:tasks.workunit.client.0.smithi151.stdout:1/653: write d3/d58/df/d32/f37 [1724270,61819] 0 2024-04-16T02:36:02.436 INFO:tasks.workunit.client.0.smithi151.stdout:5/793: symlink d2/d6/d19/d30/d88/ddc/lfe 0 2024-04-16T02:36:02.437 INFO:tasks.workunit.client.0.smithi151.stdout:8/549: mknod d11/d2a/db1/cbd 0 2024-04-16T02:36:02.438 INFO:tasks.workunit.client.0.smithi151.stdout:0/603: truncate dc/d3c/f48 1627317 0 2024-04-16T02:36:02.439 INFO:tasks.workunit.client.0.smithi151.stdout:3/510: creat d1/d6/d22/fab x:0 0 0 2024-04-16T02:36:02.440 INFO:tasks.workunit.client.0.smithi151.stdout:6/823: creat d2/d13/d63/def/df/d23/d56/d72/f119 x:0 0 0 2024-04-16T02:36:02.442 INFO:tasks.workunit.client.0.smithi151.stdout:7/758: unlink d0/c16 0 2024-04-16T02:36:02.443 INFO:tasks.workunit.client.0.smithi151.stdout:3/511: dread d1/d6/f50 [0,4194304] 0 2024-04-16T02:36:02.444 INFO:tasks.workunit.client.0.smithi151.stdout:8/550: creat d11/d22/d6f/d92/fbe x:0 0 0 2024-04-16T02:36:02.444 INFO:tasks.workunit.client.0.smithi151.stdout:7/759: write d0/d1f/d33/d56/f84 [6603726,60840] 0 2024-04-16T02:36:02.450 INFO:tasks.workunit.client.0.smithi151.stdout:7/760: fdatasync d0/f7 0 2024-04-16T02:36:02.450 INFO:tasks.workunit.client.0.smithi151.stdout:3/512: truncate d1/d6/d22/d23/f5a 390287 0 2024-04-16T02:36:02.453 INFO:tasks.workunit.client.0.smithi151.stdout:3/513: fdatasync d1/d6/d22/d23/d66/f74 0 2024-04-16T02:36:02.454 INFO:tasks.workunit.client.0.smithi151.stdout:6/824: link d2/d13/d63/def/df/d20/d4f/l10a d2/d13/d63/def/df/d20/de6/l11a 0 2024-04-16T02:36:02.455 INFO:tasks.workunit.client.0.smithi151.stdout:8/551: dwrite d11/d2a/d33/d4e/f72 [0,4194304] 0 2024-04-16T02:36:02.455 INFO:tasks.workunit.client.0.smithi151.stdout:3/514: chown d1/d6/c91 10192222 1 2024-04-16T02:36:02.458 INFO:tasks.workunit.client.0.smithi151.stdout:6/825: chown d2/d13/d63/def/df/d23/d45/d62/caf 2 1 2024-04-16T02:36:02.459 INFO:tasks.workunit.client.0.smithi151.stdout:6/826: stat d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/ca4 0 2024-04-16T02:36:02.461 INFO:tasks.workunit.client.0.smithi151.stdout:3/515: mkdir d1/d69/d7f/dac 0 2024-04-16T02:36:02.462 INFO:tasks.workunit.client.0.smithi151.stdout:8/552: read f2 [2569704,64211] 0 2024-04-16T02:36:02.465 INFO:tasks.workunit.client.0.smithi151.stdout:3/516: dread d1/d37/f38 [0,4194304] 0 2024-04-16T02:36:02.465 INFO:tasks.workunit.client.0.smithi151.stdout:6/827: creat d2/d13/d63/d112/f11b x:0 0 0 2024-04-16T02:36:02.468 INFO:tasks.workunit.client.0.smithi151.stdout:6/828: write d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/fd5 [1584295,64414] 0 2024-04-16T02:36:02.472 INFO:tasks.workunit.client.0.smithi151.stdout:3/517: link d1/d6/d22/d23/d66/f74 d1/d69/fad 0 2024-04-16T02:36:02.472 INFO:tasks.workunit.client.0.smithi151.stdout:0/604: write dc/d3c/f75 [366583,97332] 0 2024-04-16T02:36:02.481 INFO:tasks.workunit.client.0.smithi151.stdout:5/794: dwrite d2/d6/d19/d30/d88/ddc/d83/d8b/f8f [0,4194304] 0 2024-04-16T02:36:02.481 INFO:tasks.workunit.client.0.smithi151.stdout:3/518: truncate d1/d6/f31 1872757 0 2024-04-16T02:36:02.484 INFO:tasks.workunit.client.0.smithi151.stdout:0/605: dread dc/d15/d34/d67/f5f [0,4194304] 0 2024-04-16T02:36:02.486 INFO:tasks.workunit.client.0.smithi151.stdout:6/829: link d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d65/f94 d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d85/f11c 0 2024-04-16T02:36:02.487 INFO:tasks.workunit.client.0.smithi151.stdout:8/553: dread d11/d22/d6f/d92/d78/f7b [0,4194304] 0 2024-04-16T02:36:02.488 INFO:tasks.workunit.client.0.smithi151.stdout:5/795: unlink d2/d6/d19/d30/d88/cab 0 2024-04-16T02:36:02.488 INFO:tasks.workunit.client.0.smithi151.stdout:6/830: readlink d2/d13/d63/def/df/d23/d82/le2 0 2024-04-16T02:36:02.489 INFO:tasks.workunit.client.0.smithi151.stdout:3/519: mknod d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/cae 0 2024-04-16T02:36:02.490 INFO:tasks.workunit.client.0.smithi151.stdout:5/796: fsync d2/d6/f6c 0 2024-04-16T02:36:02.492 INFO:tasks.workunit.client.0.smithi151.stdout:8/554: rename d11/d2a/d33/d4e/d62 to d11/d2a/d33/d4e/d62/d6b/dbf 22 2024-04-16T02:36:02.496 INFO:tasks.workunit.client.0.smithi151.stdout:6/831: creat d2/d13/d63/def/df/d23/d82/d8d/d22/f11d x:0 0 0 2024-04-16T02:36:02.496 INFO:tasks.workunit.client.0.smithi151.stdout:3/520: dread d1/d6/d22/f4d [4194304,4194304] 0 2024-04-16T02:36:02.499 INFO:tasks.workunit.client.0.smithi151.stdout:5/797: write d2/d6/d19/d1a/f41 [2359249,42055] 0 2024-04-16T02:36:02.501 INFO:tasks.workunit.client.0.smithi151.stdout:8/555: dwrite d11/d2a/d35/d64/f88 [0,4194304] 0 2024-04-16T02:36:02.504 INFO:tasks.workunit.client.0.smithi151.stdout:5/798: truncate d2/d6/d19/d62/fbc 645481 0 2024-04-16T02:36:02.504 INFO:tasks.workunit.client.0.smithi151.stdout:8/556: chown d11/d22/d6f/d92/ca9 15173 1 2024-04-16T02:36:02.508 INFO:tasks.workunit.client.0.smithi151.stdout:5/799: symlink d2/d6/d43/lff 0 2024-04-16T02:36:02.512 INFO:tasks.workunit.client.0.smithi151.stdout:8/557: rename d11/d22/d6f to d11/d22/d2e/dc0 0 2024-04-16T02:36:02.516 INFO:tasks.workunit.client.0.smithi151.stdout:8/558: creat d11/d22/d7e/fc1 x:0 0 0 2024-04-16T02:36:02.519 INFO:tasks.workunit.client.0.smithi151.stdout:8/559: creat d11/d22/d2e/dc0/d92/fc2 x:0 0 0 2024-04-16T02:36:02.520 INFO:tasks.workunit.client.0.smithi151.stdout:8/560: chown d11/d2a/d85 127 1 2024-04-16T02:36:02.522 INFO:tasks.workunit.client.0.smithi151.stdout:3/521: dwrite d1/d6/d22/d23/d30/fa8 [0,4194304] 0 2024-04-16T02:36:02.522 INFO:tasks.workunit.client.0.smithi151.stdout:8/561: stat d11/d2a/d35 0 2024-04-16T02:36:02.522 INFO:tasks.workunit.client.0.smithi151.stdout:8/562: chown d11/d7d 3 1 2024-04-16T02:36:02.523 INFO:tasks.workunit.client.0.smithi151.stdout:5/800: write d2/d6/d19/d30/f39 [2743529,59629] 0 2024-04-16T02:36:02.524 INFO:tasks.workunit.client.0.smithi151.stdout:8/563: write d11/d2a/d35/d64/f74 [138770,43448] 0 2024-04-16T02:36:02.530 INFO:tasks.workunit.client.0.smithi151.stdout:8/564: read d11/d2a/d33/f48 [418304,117977] 0 2024-04-16T02:36:02.530 INFO:tasks.workunit.client.0.smithi151.stdout:5/801: write d2/f8e [736569,95946] 0 2024-04-16T02:36:02.538 INFO:tasks.workunit.client.0.smithi151.stdout:3/522: dwrite d1/d6/d22/d23/d30/f45 [0,4194304] 0 2024-04-16T02:36:02.542 INFO:tasks.workunit.client.0.smithi151.stdout:5/802: link d2/d5e/f72 d2/d6/d19/d30/d88/ddc/da8/dd3/f100 0 2024-04-16T02:36:02.542 INFO:tasks.workunit.client.0.smithi151.stdout:8/565: dwrite d11/d22/d7e/fc1 [0,4194304] 0 2024-04-16T02:36:02.544 INFO:tasks.workunit.client.0.smithi151.stdout:3/523: write d1/d6/d22/d23/d41/d9d/f9e [992057,20039] 0 2024-04-16T02:36:02.547 INFO:tasks.workunit.client.0.smithi151.stdout:8/566: symlink d11/d22/d6c/lc3 0 2024-04-16T02:36:02.633 INFO:tasks.workunit.client.0.smithi151.stdout:3/524: creat d1/d6/d22/d23/d30/d44/faf x:0 0 0 2024-04-16T02:36:02.635 INFO:tasks.workunit.client.0.smithi151.stdout:5/803: dread d2/f24 [0,4194304] 0 2024-04-16T02:36:02.825 INFO:tasks.workunit.client.0.smithi151.stdout:8/567: creat d11/d22/d6c/d84/fc4 x:0 0 0 2024-04-16T02:36:02.828 INFO:tasks.workunit.client.0.smithi151.stdout:5/804: getdents d2/d6/d19/d93 0 2024-04-16T02:36:02.829 INFO:tasks.workunit.client.0.smithi151.stdout:5/805: stat d2/d6/d19/d30/d36/d42 0 2024-04-16T02:36:02.829 INFO:tasks.workunit.client.0.smithi151.stdout:8/568: getdents d11/d7d 0 2024-04-16T02:36:02.832 INFO:tasks.workunit.client.0.smithi151.stdout:8/569: dread d11/f17 [0,4194304] 0 2024-04-16T02:36:02.836 INFO:tasks.workunit.client.0.smithi151.stdout:8/570: symlink d11/d2a/d83/lc5 0 2024-04-16T02:36:02.836 INFO:tasks.workunit.client.0.smithi151.stdout:5/806: dwrite d2/f12 [0,4194304] 0 2024-04-16T02:36:02.840 INFO:tasks.workunit.client.0.smithi151.stdout:8/571: rename l8 to d11/d2a/d83/lc6 0 2024-04-16T02:36:02.841 INFO:tasks.workunit.client.0.smithi151.stdout:8/572: write d11/d2a/f8c [131256,43227] 0 2024-04-16T02:36:02.845 INFO:tasks.workunit.client.0.smithi151.stdout:8/573: unlink d11/c3e 0 2024-04-16T02:36:02.847 INFO:tasks.workunit.client.0.smithi151.stdout:8/574: rmdir d11/d2a/d33 39 2024-04-16T02:36:02.850 INFO:tasks.workunit.client.0.smithi151.stdout:8/575: fsync d11/d2a/f2f 0 2024-04-16T02:36:02.855 INFO:tasks.workunit.client.0.smithi151.stdout:8/576: rename d11/d2a/d85/f8b to d11/d22/d6c/d84/fc7 0 2024-04-16T02:36:02.860 INFO:tasks.workunit.client.0.smithi151.stdout:8/577: mkdir d11/d2a/d33/d4e/d62/dc8 0 2024-04-16T02:36:02.860 INFO:tasks.workunit.client.0.smithi151.stdout:8/578: chown d11/d22 229815 1 2024-04-16T02:36:02.865 INFO:tasks.workunit.client.0.smithi151.stdout:8/579: rename d11/d2a/d83/dae to d11/d22/d6c/d84/dc9 0 2024-04-16T02:36:02.866 INFO:tasks.workunit.client.0.smithi151.stdout:8/580: chown d11/d22/d6c/d84/fc7 469850 1 2024-04-16T02:36:02.868 INFO:tasks.workunit.client.0.smithi151.stdout:5/807: write d2/d6/d19/d30/d36/f38 [3720530,14289] 0 2024-04-16T02:36:02.872 INFO:tasks.workunit.client.0.smithi151.stdout:3/525: dwrite d1/d6/d22/f4f [0,4194304] 0 2024-04-16T02:36:02.872 INFO:tasks.workunit.client.0.smithi151.stdout:8/581: rename d11/d2a/c4f to d11/d2a/d33/d4e/d62/dc8/cca 0 2024-04-16T02:36:02.878 INFO:tasks.workunit.client.0.smithi151.stdout:8/582: creat d11/d2a/d35/d52/db3/fcb x:0 0 0 2024-04-16T02:36:02.881 INFO:tasks.workunit.client.0.smithi151.stdout:5/808: getdents d2/d6/d19/d1a 0 2024-04-16T02:36:02.882 INFO:tasks.workunit.client.0.smithi151.stdout:3/526: dwrite d1/d37/f38 [0,4194304] 0 2024-04-16T02:36:02.884 INFO:tasks.workunit.client.0.smithi151.stdout:5/809: symlink d2/d6/d19/d62/l101 0 2024-04-16T02:36:02.885 INFO:tasks.workunit.client.0.smithi151.stdout:5/810: readlink d2/d6/d19/d1a/d4c/l4f 0 2024-04-16T02:36:02.887 INFO:tasks.workunit.client.0.smithi151.stdout:3/527: stat d1/d6/d22/d23/d30/d44/d46/d8a/f15 0 2024-04-16T02:36:02.888 INFO:tasks.workunit.client.0.smithi151.stdout:5/811: dread d2/d6/d19/d62/fbc [0,4194304] 0 2024-04-16T02:36:02.892 INFO:tasks.workunit.client.0.smithi151.stdout:3/528: creat d1/d6/d22/d23/d30/d44/d46/d8a/d3f/fb0 x:0 0 0 2024-04-16T02:36:02.893 INFO:tasks.workunit.client.0.smithi151.stdout:3/529: readlink d1/ld 0 2024-04-16T02:36:02.895 INFO:tasks.workunit.client.0.smithi151.stdout:8/583: dread d11/d2a/f2c [0,4194304] 0 2024-04-16T02:36:02.895 INFO:tasks.workunit.client.0.smithi151.stdout:5/812: rename d2/d6/c46 to d2/d6/d19/d30/c102 0 2024-04-16T02:36:02.897 INFO:tasks.workunit.client.0.smithi151.stdout:8/584: write d11/d2a/d35/d52/db3/fb6 [717299,125620] 0 2024-04-16T02:36:02.899 INFO:tasks.workunit.client.0.smithi151.stdout:3/530: rename d1/d6/d22/d23/d66/f74 to d1/d6/d4a/fb1 0 2024-04-16T02:36:02.901 INFO:tasks.workunit.client.0.smithi151.stdout:3/531: write d1/d69/f7e [743772,16791] 0 2024-04-16T02:36:02.901 INFO:tasks.workunit.client.0.smithi151.stdout:3/532: chown d1 2245716 1 2024-04-16T02:36:02.903 INFO:tasks.workunit.client.0.smithi151.stdout:3/533: chown d1/d37/ca0 35 1 2024-04-16T02:36:02.906 INFO:tasks.workunit.client.0.smithi151.stdout:5/813: rename d2/d5e/f72 to d2/d6/d19/d30/f103 0 2024-04-16T02:36:02.906 INFO:tasks.workunit.client.0.smithi151.stdout:8/585: dwrite d11/d2a/d35/d64/f74 [0,4194304] 0 2024-04-16T02:36:02.908 INFO:tasks.workunit.client.0.smithi151.stdout:3/534: dread d1/d6/d22/d23/d66/f7b [0,4194304] 0 2024-04-16T02:36:02.908 INFO:tasks.workunit.client.0.smithi151.stdout:5/814: chown d2/d6/d19/d30/d49/fce 5621728 1 2024-04-16T02:36:02.910 INFO:tasks.workunit.client.0.smithi151.stdout:8/586: mkdir d11/d2a/d35/d52/db3/dcc 0 2024-04-16T02:36:02.914 INFO:tasks.workunit.client.0.smithi151.stdout:8/587: fsync d11/d22/d6c/d84/fc7 0 2024-04-16T02:36:02.918 INFO:tasks.workunit.client.0.smithi151.stdout:8/588: getdents d11/d2a/d85 0 2024-04-16T02:36:02.921 INFO:tasks.workunit.client.0.smithi151.stdout:8/589: truncate d11/d2a/d33/d4e/f9e 1227478 0 2024-04-16T02:36:02.927 INFO:tasks.workunit.client.0.smithi151.stdout:5/815: truncate d2/d6/d19/d1a/d4c/f92 3638488 0 2024-04-16T02:36:02.928 INFO:tasks.workunit.client.0.smithi151.stdout:3/535: dread d1/d6/d4a/fb1 [0,4194304] 0 2024-04-16T02:36:02.930 INFO:tasks.workunit.client.0.smithi151.stdout:3/536: creat d1/d6/d22/d23/d66/fb2 x:0 0 0 2024-04-16T02:36:02.936 INFO:tasks.workunit.client.0.smithi151.stdout:5/816: dwrite d2/d6/d19/d30/d49/d85/fd1 [0,4194304] 0 2024-04-16T02:36:02.938 INFO:tasks.workunit.client.0.smithi151.stdout:3/537: write d1/d6/f50 [1222544,90284] 0 2024-04-16T02:36:02.943 INFO:tasks.workunit.client.0.smithi151.stdout:5/817: rmdir d2/d6/d19/d30/d88 39 2024-04-16T02:36:02.945 INFO:tasks.workunit.client.0.smithi151.stdout:5/818: mknod d2/d6/d19/d30/d49/d85/c104 0 2024-04-16T02:36:02.947 INFO:tasks.workunit.client.0.smithi151.stdout:3/538: link d1/d37/f42 d1/d6/d22/d23/fb3 0 2024-04-16T02:36:02.948 INFO:tasks.workunit.client.0.smithi151.stdout:3/539: dread d1/f5c [0,4194304] 0 2024-04-16T02:36:02.949 INFO:tasks.workunit.client.0.smithi151.stdout:5/819: rename d2/d6/d19/d1a/f48 to d2/d6/d19/d1a/f105 0 2024-04-16T02:36:02.955 INFO:tasks.workunit.client.0.smithi151.stdout:5/820: mknod d2/d6/dfc/c106 0 2024-04-16T02:36:02.957 INFO:tasks.workunit.client.0.smithi151.stdout:3/540: rmdir d1/d6/d22 39 2024-04-16T02:36:02.958 INFO:tasks.workunit.client.0.smithi151.stdout:5/821: write d2/d6/d19/d1a/d4c/ff5 [957477,78791] 0 2024-04-16T02:36:02.962 INFO:tasks.workunit.client.0.smithi151.stdout:3/541: truncate d1/d6/d20/f68 863901 0 2024-04-16T02:36:02.963 INFO:tasks.workunit.client.0.smithi151.stdout:3/542: readlink d1/l6e 0 2024-04-16T02:36:02.971 INFO:tasks.workunit.client.0.smithi151.stdout:5/822: dwrite d2/d6/f32 [0,4194304] 0 2024-04-16T02:36:02.972 INFO:tasks.workunit.client.0.smithi151.stdout:3/543: dwrite d1/d69/fad [4194304,4194304] 0 2024-04-16T02:36:02.978 INFO:tasks.workunit.client.0.smithi151.stdout:5/823: creat d2/f107 x:0 0 0 2024-04-16T02:36:02.980 INFO:tasks.workunit.client.0.smithi151.stdout:3/544: read - d1/d6/d22/d23/d66/fb2 zero size 2024-04-16T02:36:02.982 INFO:tasks.workunit.client.0.smithi151.stdout:5/824: creat d2/d6/d19/f108 x:0 0 0 2024-04-16T02:36:02.985 INFO:tasks.workunit.client.0.smithi151.stdout:5/825: mknod d2/d6/d19/d62/dd0/d9c/dd7/c109 0 2024-04-16T02:36:03.001 INFO:tasks.workunit.client.0.smithi151.stdout:5/826: rename d2/d6/d19/d62/dd0/d9c/dd7 to d2/d6/d19/d93/d10a 0 2024-04-16T02:36:03.008 INFO:tasks.workunit.client.0.smithi151.stdout:3/545: dread d1/d37/d87/f8b [0,4194304] 0 2024-04-16T02:36:03.009 INFO:tasks.workunit.client.0.smithi151.stdout:5/827: write d2/d6/d19/d30/d49/f94 [1460033,22874] 0 2024-04-16T02:36:03.010 INFO:tasks.workunit.client.0.smithi151.stdout:3/546: read d1/d6/f1b [6565048,107197] 0 2024-04-16T02:36:03.013 INFO:tasks.workunit.client.0.smithi151.stdout:3/547: getdents d1/d89/daa 0 2024-04-16T02:36:03.014 INFO:tasks.workunit.client.0.smithi151.stdout:5/828: dread d2/f12 [0,4194304] 0 2024-04-16T02:36:03.017 INFO:tasks.workunit.client.0.smithi151.stdout:5/829: unlink d2/d6/d43/l55 0 2024-04-16T02:36:03.018 INFO:tasks.workunit.client.0.smithi151.stdout:3/548: rename d1/d6/d22/d23/d30/d44/c75 to d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/cb4 0 2024-04-16T02:36:03.020 INFO:tasks.workunit.client.0.smithi151.stdout:3/549: dread d1/d6/d4a/f6f [0,4194304] 0 2024-04-16T02:36:03.021 INFO:tasks.workunit.client.0.smithi151.stdout:5/830: rename d2/d6/d19/fc2 to d2/d6/d19/d1a/d4c/f10b 0 2024-04-16T02:36:03.026 INFO:tasks.workunit.client.0.smithi151.stdout:3/550: truncate d1/d6/d22/d23/f62 6941153 0 2024-04-16T02:36:03.029 INFO:tasks.workunit.client.0.smithi151.stdout:3/551: chown d1/d6/d22/d23/d30/d44/d46/d8a/c53 3832770 1 2024-04-16T02:36:03.031 INFO:tasks.workunit.client.0.smithi151.stdout:3/552: truncate d1/d37/f5e 3022626 0 2024-04-16T02:36:03.032 INFO:tasks.workunit.client.0.smithi151.stdout:5/831: write d2/f99 [4115527,36498] 0 2024-04-16T02:36:03.034 INFO:tasks.workunit.client.0.smithi151.stdout:3/553: read d1/d6/d22/d23/d30/d44/d46/d8a/fa3 [1521112,87095] 0 2024-04-16T02:36:03.036 INFO:tasks.workunit.client.0.smithi151.stdout:3/554: write d1/fa [6520432,26467] 0 2024-04-16T02:36:03.037 INFO:tasks.workunit.client.0.smithi151.stdout:5/832: getdents d2/d6 0 2024-04-16T02:36:03.043 INFO:tasks.workunit.client.0.smithi151.stdout:3/555: creat d1/d69/da4/fb5 x:0 0 0 2024-04-16T02:36:03.044 INFO:tasks.workunit.client.0.smithi151.stdout:5/833: rmdir d2/d6/d19/d30/d88/dc4 39 2024-04-16T02:36:03.046 INFO:tasks.workunit.client.0.smithi151.stdout:3/556: truncate d1/d6/d22/d23/d30/d44/d46/d8a/d3f/fb0 1038045 0 2024-04-16T02:36:03.048 INFO:tasks.workunit.client.0.smithi151.stdout:3/557: chown d1/d6/d20/d54/f80 2005798238 1 2024-04-16T02:36:03.053 INFO:tasks.workunit.client.0.smithi151.stdout:3/558: truncate d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/f72 198983 0 2024-04-16T02:36:03.054 INFO:tasks.workunit.client.0.smithi151.stdout:5/834: dwrite d2/d6/d19/f1d [4194304,4194304] 0 2024-04-16T02:36:03.057 INFO:tasks.workunit.client.0.smithi151.stdout:3/559: mknod d1/d6/d22/d23/cb6 0 2024-04-16T02:36:03.059 INFO:tasks.workunit.client.0.smithi151.stdout:3/560: creat d1/d37/fb7 x:0 0 0 2024-04-16T02:36:03.061 INFO:tasks.workunit.client.0.smithi151.stdout:3/561: rmdir d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d 39 2024-04-16T02:36:03.062 INFO:tasks.workunit.client.0.smithi151.stdout:3/562: write d1/d6/d20/d54/f80 [895092,115880] 0 2024-04-16T02:36:03.065 INFO:tasks.workunit.client.0.smithi151.stdout:3/563: creat d1/d6/d20/d3d/fb8 x:0 0 0 2024-04-16T02:36:03.067 INFO:tasks.workunit.client.0.smithi151.stdout:3/564: creat d1/d6/d22/d23/fb9 x:0 0 0 2024-04-16T02:36:03.069 INFO:tasks.workunit.client.0.smithi151.stdout:3/565: rmdir d1/d6/d22/d23/d30/d44/d46 39 2024-04-16T02:36:03.070 INFO:tasks.workunit.client.0.smithi151.stdout:3/566: chown d1/d37/f5e 6 1 2024-04-16T02:36:03.072 INFO:tasks.workunit.client.0.smithi151.stdout:3/567: mknod d1/d6/d22/cba 0 2024-04-16T02:36:03.079 INFO:tasks.workunit.client.0.smithi151.stdout:3/568: write d1/d6/d22/d23/f62 [1503818,67564] 0 2024-04-16T02:36:03.081 INFO:tasks.workunit.client.0.smithi151.stdout:3/569: creat d1/d6/d4a/fbb x:0 0 0 2024-04-16T02:36:03.083 INFO:tasks.workunit.client.0.smithi151.stdout:3/570: mknod d1/d6/d22/d23/d30/cbc 0 2024-04-16T02:36:03.095 INFO:tasks.workunit.client.0.smithi151.stdout:3/571: dread d1/d6/d22/f4d [0,4194304] 0 2024-04-16T02:36:03.098 INFO:tasks.workunit.client.0.smithi151.stdout:3/572: fsync d1/d6/d22/d23/d30/d44/d46/d8a/f16 0 2024-04-16T02:36:03.107 INFO:tasks.workunit.client.0.smithi151.stdout:3/573: dwrite d1/f2e [0,4194304] 0 2024-04-16T02:36:03.111 INFO:tasks.workunit.client.0.smithi151.stdout:3/574: dread d1/d6/d20/f68 [0,4194304] 0 2024-04-16T02:36:03.117 INFO:tasks.workunit.client.0.smithi151.stdout:3/575: dwrite d1/d6/d22/d23/d30/d44/f5f [0,4194304] 0 2024-04-16T02:36:03.118 INFO:tasks.workunit.client.0.smithi151.stdout:3/576: stat d1/d6/d22/d23/d30/l5d 0 2024-04-16T02:36:03.120 INFO:tasks.workunit.client.0.smithi151.stdout:3/577: write d1/d69/da4/fb5 [751417,31801] 0 2024-04-16T02:36:03.122 INFO:tasks.workunit.client.0.smithi151.stdout:3/578: mknod d1/d69/d7f/dac/cbd 0 2024-04-16T02:36:03.127 INFO:tasks.workunit.client.0.smithi151.stdout:3/579: dread d1/d6/d22/d23/d30/d44/d46/d8a/d3f/fb0 [0,4194304] 0 2024-04-16T02:36:03.130 INFO:tasks.workunit.client.0.smithi151.stdout:3/580: dread d1/d6/f50 [0,4194304] 0 2024-04-16T02:36:03.138 INFO:tasks.workunit.client.0.smithi151.stdout:3/581: truncate d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/f55 1944100 0 2024-04-16T02:36:03.141 INFO:tasks.workunit.client.0.smithi151.stdout:3/582: read d1/d6/d22/d23/d30/d44/d46/d8a/f1a [461498,112884] 0 2024-04-16T02:36:03.223 INFO:tasks.workunit.client.0.smithi151.stdout:3/583: rename d1/d6/d22/f2b to d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/fbe 0 2024-04-16T02:36:03.224 INFO:tasks.workunit.client.0.smithi151.stdout:3/584: creat d1/d6/d22/fbf x:0 0 0 2024-04-16T02:36:03.225 INFO:tasks.workunit.client.0.smithi151.stdout:3/585: readlink d1/d6/l82 0 2024-04-16T02:36:03.907 INFO:tasks.workunit.client.0.smithi151.stdout:1/654: sync 2024-04-16T02:36:03.907 INFO:tasks.workunit.client.0.smithi151.stdout:9/663: sync 2024-04-16T02:36:03.907 INFO:tasks.workunit.client.0.smithi151.stdout:4/709: sync 2024-04-16T02:36:03.907 INFO:tasks.workunit.client.0.smithi151.stdout:1/655: stat d3/d4/d64/fa4 0 2024-04-16T02:36:03.908 INFO:tasks.workunit.client.0.smithi151.stdout:4/710: write d0/d11/f8d [2199464,107205] 0 2024-04-16T02:36:03.913 INFO:tasks.workunit.client.0.smithi151.stdout:9/664: rename d1/d3/d6/d17/d40/d56/f61 to d1/d3/d6/d17/d54/d9b/fd6 0 2024-04-16T02:36:03.913 INFO:tasks.workunit.client.0.smithi151.stdout:9/665: chown d1/dd/d38/c4c 566399885 1 2024-04-16T02:36:03.914 INFO:tasks.workunit.client.0.smithi151.stdout:4/711: creat d0/dde/d80/da0/ffb x:0 0 0 2024-04-16T02:36:03.920 INFO:tasks.workunit.client.0.smithi151.stdout:9/666: creat d1/d3/d6/d17/d54/d9b/d22/d2c/fd7 x:0 0 0 2024-04-16T02:36:03.923 INFO:tasks.workunit.client.0.smithi151.stdout:9/667: creat d1/d3/d6/d17/d54/dd5/fd8 x:0 0 0 2024-04-16T02:36:03.941 INFO:tasks.workunit.client.0.smithi151.stdout:1/656: write d3/d33/f39 [978143,100430] 0 2024-04-16T02:36:03.945 INFO:tasks.workunit.client.0.smithi151.stdout:1/657: rename d3/d4/l3c to d3/d4/db2/ldd 0 2024-04-16T02:36:03.948 INFO:tasks.workunit.client.0.smithi151.stdout:9/668: write d1/d3/f33 [3641881,10734] 0 2024-04-16T02:36:03.951 INFO:tasks.workunit.client.0.smithi151.stdout:4/712: dwrite d0/d11/f3d [0,4194304] 0 2024-04-16T02:36:03.953 INFO:tasks.workunit.client.0.smithi151.stdout:9/669: creat d1/d3/d6/d17/d54/dd5/fd9 x:0 0 0 2024-04-16T02:36:03.956 INFO:tasks.workunit.client.0.smithi151.stdout:1/658: write d3/d33/f36 [1752789,94343] 0 2024-04-16T02:36:03.957 INFO:tasks.workunit.client.0.smithi151.stdout:1/659: write d3/d58/df/d32/f88 [2170526,89022] 0 2024-04-16T02:36:03.957 INFO:tasks.workunit.client.0.smithi151.stdout:4/713: rename d0/dde/d80/l81 to d0/d11/d19/dd7/lfc 0 2024-04-16T02:36:03.959 INFO:tasks.workunit.client.0.smithi151.stdout:9/670: mknod d1/d3/d10/d7b/cda 0 2024-04-16T02:36:03.962 INFO:tasks.workunit.client.0.smithi151.stdout:9/671: mknod d1/d3/d6/d17/d54/dd5/cdb 0 2024-04-16T02:36:03.966 INFO:tasks.workunit.client.0.smithi151.stdout:9/672: symlink d1/d3/d6/d17/d44/ldc 0 2024-04-16T02:36:03.967 INFO:tasks.workunit.client.0.smithi151.stdout:4/714: link d0/d11/d19/cb3 d0/cfd 0 2024-04-16T02:36:03.968 INFO:tasks.workunit.client.0.smithi151.stdout:1/660: write d3/d33/d10/d4f/fb0 [363342,50659] 0 2024-04-16T02:36:03.970 INFO:tasks.workunit.client.0.smithi151.stdout:9/673: unlink d1/dd/d38/f96 0 2024-04-16T02:36:03.971 INFO:tasks.workunit.client.0.smithi151.stdout:1/661: creat d3/d58/df/d32/d3a/d71/fde x:0 0 0 2024-04-16T02:36:03.972 INFO:tasks.workunit.client.0.smithi151.stdout:4/715: mkdir d0/d11/d19/d46/dcd/def/dfe 0 2024-04-16T02:36:03.973 INFO:tasks.workunit.client.0.smithi151.stdout:1/662: symlink d3/d58/df/d32/d3a/d71/ldf 0 2024-04-16T02:36:03.974 INFO:tasks.workunit.client.0.smithi151.stdout:9/674: creat d1/d3/d6/d17/d40/d7d/da0/fdd x:0 0 0 2024-04-16T02:36:03.976 INFO:tasks.workunit.client.0.smithi151.stdout:4/716: mknod d0/d64/cff 0 2024-04-16T02:36:03.976 INFO:tasks.workunit.client.0.smithi151.stdout:9/675: write d1/d3/d6/d28/fb3 [1410762,125772] 0 2024-04-16T02:36:03.979 INFO:tasks.workunit.client.0.smithi151.stdout:4/717: mkdir d0/dde/dad/d100 0 2024-04-16T02:36:03.979 INFO:tasks.workunit.client.0.smithi151.stdout:4/718: fsync d0/fbf 0 2024-04-16T02:36:03.980 INFO:tasks.workunit.client.0.smithi151.stdout:9/676: mkdir d1/d3/d6/d17/d54/d58/dde 0 2024-04-16T02:36:03.985 INFO:tasks.workunit.client.0.smithi151.stdout:1/663: link d3/d4/l23 d3/d58/df/d32/d3a/d6d/le0 0 2024-04-16T02:36:03.990 INFO:tasks.workunit.client.0.smithi151.stdout:9/677: creat d1/d3/d6/d17/d54/d9b/d22/fdf x:0 0 0 2024-04-16T02:36:03.990 INFO:tasks.workunit.client.0.smithi151.stdout:9/678: chown d1/dd 48858011 1 2024-04-16T02:36:03.992 INFO:tasks.workunit.client.0.smithi151.stdout:1/664: creat d3/fe1 x:0 0 0 2024-04-16T02:36:03.994 INFO:tasks.workunit.client.0.smithi151.stdout:9/679: creat d1/d3/d10/da6/dc3/fe0 x:0 0 0 2024-04-16T02:36:03.997 INFO:tasks.workunit.client.0.smithi151.stdout:6/832: sync 2024-04-16T02:36:03.997 INFO:tasks.workunit.client.0.smithi151.stdout:5/835: sync 2024-04-16T02:36:03.997 INFO:tasks.workunit.client.0.smithi151.stdout:7/761: sync 2024-04-16T02:36:03.997 INFO:tasks.workunit.client.0.smithi151.stdout:8/590: sync 2024-04-16T02:36:03.998 INFO:tasks.workunit.client.0.smithi151.stdout:3/586: sync 2024-04-16T02:36:03.998 INFO:tasks.workunit.client.0.smithi151.stdout:0/606: sync 2024-04-16T02:36:04.004 INFO:tasks.workunit.client.0.smithi151.stdout:9/680: dread d1/dd/d4a/dbb/fc7 [0,4194304] 0 2024-04-16T02:36:04.005 INFO:tasks.workunit.client.0.smithi151.stdout:6/833: chown d2/d13/d63/def/df/d23/d31/d52/l86 34832 1 2024-04-16T02:36:04.006 INFO:tasks.workunit.client.0.smithi151.stdout:8/591: mkdir d11/d2a/d35/d52/db3/dcd 0 2024-04-16T02:36:04.009 INFO:tasks.workunit.client.0.smithi151.stdout:7/762: unlink d0/d1f/d33/d89/ff5 0 2024-04-16T02:36:04.011 INFO:tasks.workunit.client.0.smithi151.stdout:3/587: truncate d1/d6/d22/d23/d41/f76 810718 0 2024-04-16T02:36:04.013 INFO:tasks.workunit.client.0.smithi151.stdout:8/592: fsync d11/d2a/d33/d4e/d62/d94/fa3 0 2024-04-16T02:36:04.014 INFO:tasks.workunit.client.0.smithi151.stdout:0/607: dwrite dc/d20/f3d [0,4194304] 0 2024-04-16T02:36:04.017 INFO:tasks.workunit.client.0.smithi151.stdout:4/719: dwrite d0/d11/d19/d87/d3a/d65/f7d [0,4194304] 0 2024-04-16T02:36:04.017 INFO:tasks.workunit.client.0.smithi151.stdout:6/834: creat d2/d13/d63/def/df/d23/d56/d72/f11e x:0 0 0 2024-04-16T02:36:04.023 INFO:tasks.workunit.client.0.smithi151.stdout:5/836: write d2/d6/d19/d1a/f52 [1827057,24923] 0 2024-04-16T02:36:04.025 INFO:tasks.workunit.client.0.smithi151.stdout:9/681: chown d1/d3/d6/d17/d54/d9b/dae/d6b/f77 14608 1 2024-04-16T02:36:04.029 INFO:tasks.workunit.client.0.smithi151.stdout:1/665: dwrite d3/d4/d64/d40/da3/fa6 [0,4194304] 0 2024-04-16T02:36:04.034 INFO:tasks.workunit.client.0.smithi151.stdout:6/835: chown d2/d13/d63/def/df/d23/d31/fd8 11934698 1 2024-04-16T02:36:04.036 INFO:tasks.workunit.client.0.smithi151.stdout:6/836: dread - d2/d13/d3a/f54 zero size 2024-04-16T02:36:04.040 INFO:tasks.workunit.client.0.smithi151.stdout:0/608: symlink dc/d15/d34/d67/d69/lc6 0 2024-04-16T02:36:04.044 INFO:tasks.workunit.client.0.smithi151.stdout:5/837: write d2/d6/d19/d30/d88/dc4/fc5 [4135926,96135] 0 2024-04-16T02:36:04.048 INFO:tasks.workunit.client.0.smithi151.stdout:3/588: dwrite d1/d37/d87/f8b [0,4194304] 0 2024-04-16T02:36:04.050 INFO:tasks.workunit.client.0.smithi151.stdout:0/609: chown dc/d20/d7b/d9c/caa 16916 1 2024-04-16T02:36:04.050 INFO:tasks.workunit.client.0.smithi151.stdout:7/763: dread d0/d1f/d33/f3a [4194304,4194304] 0 2024-04-16T02:36:04.050 INFO:tasks.workunit.client.0.smithi151.stdout:6/837: fdatasync d2/d13/d63/def/df/d20/d101/de5/d10e/f111 0 2024-04-16T02:36:04.051 INFO:tasks.workunit.client.0.smithi151.stdout:5/838: write d2/d6/d19/d30/d88/ddc/d83/d8b/ff4 [754563,4267] 0 2024-04-16T02:36:04.053 INFO:tasks.workunit.client.0.smithi151.stdout:7/764: write d0/d27/f7a [4854138,12129] 0 2024-04-16T02:36:04.054 INFO:tasks.workunit.client.0.smithi151.stdout:1/666: link d3/d58/df/f9e d3/d33/d10/d4f/fe2 0 2024-04-16T02:36:04.055 INFO:tasks.workunit.client.0.smithi151.stdout:6/838: chown d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d73/caa 0 1 2024-04-16T02:36:04.057 INFO:tasks.workunit.client.0.smithi151.stdout:7/765: write d0/d1f/d4c/d82/db3/fea [369460,34818] 0 2024-04-16T02:36:04.057 INFO:tasks.workunit.client.0.smithi151.stdout:3/589: creat d1/d89/fc0 x:0 0 0 2024-04-16T02:36:04.059 INFO:tasks.workunit.client.0.smithi151.stdout:3/590: chown d1/d37/c83 1489 1 2024-04-16T02:36:04.066 INFO:tasks.workunit.client.0.smithi151.stdout:9/682: rename d1/d3/d6/d17/d54/l5c to d1/d3/d6/d17/le1 0 2024-04-16T02:36:04.067 INFO:tasks.workunit.client.0.smithi151.stdout:7/766: truncate d0/d1f/d60/fbd 827604 0 2024-04-16T02:36:04.067 INFO:tasks.workunit.client.0.smithi151.stdout:5/839: unlink d2/d6/d19/d30/d36/f38 0 2024-04-16T02:36:04.069 INFO:tasks.workunit.client.0.smithi151.stdout:4/720: dwrite d0/f16 [0,4194304] 0 2024-04-16T02:36:04.071 INFO:tasks.workunit.client.0.smithi151.stdout:6/839: mknod d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/c11f 0 2024-04-16T02:36:04.074 INFO:tasks.workunit.client.0.smithi151.stdout:5/840: chown d2/f12 2457396 1 2024-04-16T02:36:04.074 INFO:tasks.workunit.client.0.smithi151.stdout:1/667: dwrite d3/d58/df/d32/f37 [0,4194304] 0 2024-04-16T02:36:04.076 INFO:tasks.workunit.client.0.smithi151.stdout:4/721: write d0/dde/d45/dd2/ff9 [421451,87924] 0 2024-04-16T02:36:04.080 INFO:tasks.workunit.client.0.smithi151.stdout:5/841: chown d2/d6/d19/d30/d88/ddc/da8/cb5 481746192 1 2024-04-16T02:36:04.082 INFO:tasks.workunit.client.0.smithi151.stdout:5/842: readlink d2/d6/d19/d30/d88/ddc/l7d 0 2024-04-16T02:36:04.087 INFO:tasks.workunit.client.0.smithi151.stdout:9/683: mkdir d1/d3/d5f/de2 0 2024-04-16T02:36:04.089 INFO:tasks.workunit.client.0.smithi151.stdout:8/593: dwrite d11/d2a/d33/d4e/f9e [0,4194304] 0 2024-04-16T02:36:04.094 INFO:tasks.workunit.client.0.smithi151.stdout:5/843: creat d2/d6/db7/f10c x:0 0 0 2024-04-16T02:36:04.096 INFO:tasks.workunit.client.0.smithi151.stdout:7/767: dwrite d0/d1f/d4c/d82/db3/fea [0,4194304] 0 2024-04-16T02:36:04.101 INFO:tasks.workunit.client.0.smithi151.stdout:1/668: symlink d3/d1f/dd7/ddb/le3 0 2024-04-16T02:36:04.102 INFO:tasks.workunit.client.0.smithi151.stdout:7/768: creat d0/d27/da6/dcd/f108 x:0 0 0 2024-04-16T02:36:04.103 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:03 smithi151 ceph-mon[29003]: pgmap v29: 65 pgs: 65 active+clean; 2.9 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 68 MiB/s rd, 170 MiB/s wr, 440 op/s 2024-04-16T02:36:04.104 INFO:tasks.workunit.client.0.smithi151.stdout:9/684: read d1/d3/d6/d17/d54/d9b/d22/f23 [527400,62469] 0 2024-04-16T02:36:04.105 INFO:tasks.workunit.client.0.smithi151.stdout:9/685: fsync d1/d3/d6/d17/d54/dd5/fd9 0 2024-04-16T02:36:04.106 INFO:tasks.workunit.client.0.smithi151.stdout:1/669: fsync d3/d58/df/f6c 0 2024-04-16T02:36:04.108 INFO:tasks.workunit.client.0.smithi151.stdout:5/844: mknod d2/d6/d19/d30/d88/ddc/da8/dd3/deb/c10d 0 2024-04-16T02:36:04.109 INFO:tasks.workunit.client.0.smithi151.stdout:3/591: write d1/d6/f1b [4170558,108834] 0 2024-04-16T02:36:04.110 INFO:tasks.workunit.client.0.smithi151.stdout:0/610: dread dc/d3c/f3f [0,4194304] 0 2024-04-16T02:36:04.124 INFO:tasks.workunit.client.0.smithi151.stdout:4/722: dwrite d0/d11/f26 [0,4194304] 0 2024-04-16T02:36:04.126 INFO:tasks.workunit.client.0.smithi151.stdout:3/592: dwrite d1/d69/da4/fb5 [0,4194304] 0 2024-04-16T02:36:04.128 INFO:tasks.workunit.client.0.smithi151.stdout:8/594: getdents d11/d2a/d83 0 2024-04-16T02:36:04.134 INFO:tasks.workunit.client.0.smithi151.stdout:7/769: dwrite d0/d1f/f52 [0,4194304] 0 2024-04-16T02:36:04.136 INFO:tasks.workunit.client.0.smithi151.stdout:3/593: creat d1/d6/d20/d3d/fc1 x:0 0 0 2024-04-16T02:36:04.137 INFO:tasks.workunit.client.0.smithi151.stdout:8/595: mkdir d11/d2a/d83/dce 0 2024-04-16T02:36:04.137 INFO:tasks.workunit.client.0.smithi151.stdout:0/611: dread dc/d15/d34/d4e/f63 [0,4194304] 0 2024-04-16T02:36:04.138 INFO:tasks.workunit.client.0.smithi151.stdout:1/670: rename d3/d1f/d48/f8c to d3/d4/d64/d40/fe4 0 2024-04-16T02:36:04.146 INFO:tasks.workunit.client.0.smithi151.stdout:9/686: rename d1/d3/d10/da6 to d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3 0 2024-04-16T02:36:04.148 INFO:tasks.workunit.client.0.smithi151.stdout:1/671: symlink d3/d33/d10/d4f/dd1/le5 0 2024-04-16T02:36:04.149 INFO:tasks.workunit.client.0.smithi151.stdout:0/612: creat dc/d20/d7b/fc7 x:0 0 0 2024-04-16T02:36:04.152 INFO:tasks.workunit.client.0.smithi151.stdout:9/687: fsync d1/d3/d6/d17/d54/d9b/d22/d36/f9e 0 2024-04-16T02:36:04.152 INFO:tasks.workunit.client.0.smithi151.stdout:8/596: dwrite d11/d2a/d35/d52/f98 [0,4194304] 0 2024-04-16T02:36:04.155 INFO:tasks.workunit.client.0.smithi151.stdout:9/688: fsync d1/d3/d6/d17/d54/dd5/fd9 0 2024-04-16T02:36:04.157 INFO:tasks.workunit.client.0.smithi151.stdout:1/672: creat d3/d1f/dd7/dda/fe6 x:0 0 0 2024-04-16T02:36:04.159 INFO:tasks.workunit.client.0.smithi151.stdout:4/723: write d0/d9b/fa3 [958839,107606] 0 2024-04-16T02:36:04.160 INFO:tasks.workunit.client.0.smithi151.stdout:8/597: symlink d11/d2a/d85/lcf 0 2024-04-16T02:36:04.166 INFO:tasks.workunit.client.0.smithi151.stdout:3/594: dread d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/f55 [0,4194304] 0 2024-04-16T02:36:04.169 INFO:tasks.workunit.client.0.smithi151.stdout:8/598: creat d11/d2a/db1/fd0 x:0 0 0 2024-04-16T02:36:04.171 INFO:tasks.workunit.client.0.smithi151.stdout:4/724: creat d0/dde/dad/d100/f101 x:0 0 0 2024-04-16T02:36:04.174 INFO:tasks.workunit.client.0.smithi151.stdout:7/770: dwrite d0/d1f/d46/f8d [0,4194304] 0 2024-04-16T02:36:04.176 INFO:tasks.workunit.client.0.smithi151.stdout:3/595: dread d1/d6/d22/d6c/f9c [0,4194304] 0 2024-04-16T02:36:04.179 INFO:tasks.workunit.client.0.smithi151.stdout:4/725: mknod d0/daa/dcc/c102 0 2024-04-16T02:36:04.179 INFO:tasks.workunit.client.0.smithi151.stdout:5/845: dwrite d2/d6/d19/d1a/d4c/fc9 [0,4194304] 0 2024-04-16T02:36:04.186 INFO:tasks.workunit.client.0.smithi151.stdout:3/596: write d1/d6/d22/f4f [3409072,14898] 0 2024-04-16T02:36:04.188 INFO:tasks.workunit.client.0.smithi151.stdout:1/673: write d3/d33/d10/d4f/fe2 [390776,50860] 0 2024-04-16T02:36:04.189 INFO:tasks.workunit.client.0.smithi151.stdout:7/771: getdents d0/d1f/d46/dff 0 2024-04-16T02:36:04.191 INFO:tasks.workunit.client.0.smithi151.stdout:5/846: creat d2/d6/d19/d62/dd0/d9c/db9/f10e x:0 0 0 2024-04-16T02:36:04.193 INFO:tasks.workunit.client.0.smithi151.stdout:4/726: rename d0/dde/d4a/d60/de6/f1d to d0/d11/d19/d46/dcd/def/dfe/f103 0 2024-04-16T02:36:04.197 INFO:tasks.workunit.client.0.smithi151.stdout:7/772: fsync d0/d1f/d46/f5a 0 2024-04-16T02:36:04.197 INFO:tasks.workunit.client.0.smithi151.stdout:1/674: fsync d3/d4/d64/d2a/f60 0 2024-04-16T02:36:04.198 INFO:tasks.workunit.client.0.smithi151.stdout:0/613: dwrite dc/d4c/d72/f77 [0,4194304] 0 2024-04-16T02:36:04.198 INFO:tasks.workunit.client.0.smithi151.stdout:5/847: mkdir d2/d10f 0 2024-04-16T02:36:04.206 INFO:tasks.workunit.client.0.smithi151.stdout:3/597: dwrite d1/d6/d22/d23/d30/f45 [0,4194304] 0 2024-04-16T02:36:04.208 INFO:tasks.workunit.client.0.smithi151.stdout:8/599: truncate d11/d2a/d33/f4b 3027665 0 2024-04-16T02:36:04.213 INFO:tasks.workunit.client.0.smithi151.stdout:3/598: read d1/d6/d22/d23/d30/d44/d46/d8a/d3f/f48 [133835,129510] 0 2024-04-16T02:36:04.215 INFO:tasks.workunit.client.0.smithi151.stdout:0/614: dread dc/d15/d34/d67/f30 [0,4194304] 0 2024-04-16T02:36:04.221 INFO:tasks.workunit.client.0.smithi151.stdout:1/675: link d3/d33/d10/d4f/fc0 d3/d1f/dd7/ddb/fe7 0 2024-04-16T02:36:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:03 smithi153 ceph-mon[36747]: pgmap v29: 65 pgs: 65 active+clean; 2.9 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 68 MiB/s rd, 170 MiB/s wr, 440 op/s 2024-04-16T02:36:04.337 INFO:tasks.workunit.client.0.smithi151.stdout:7/773: getdents d0/d27/da6/dcd 0 2024-04-16T02:36:04.338 INFO:tasks.workunit.client.0.smithi151.stdout:3/599: rename d1/d6/l1c to d1/d6/d22/d23/d30/d44/d46/d8a/d3f/lc2 0 2024-04-16T02:36:04.342 INFO:tasks.workunit.client.0.smithi151.stdout:0/615: dwrite dc/d4c/d72/d76/db0/db5/fc0 [0,4194304] 0 2024-04-16T02:36:04.342 INFO:tasks.workunit.client.0.smithi151.stdout:8/600: getdents d11/d22/d2e/dc0/d92 0 2024-04-16T02:36:04.342 INFO:tasks.workunit.client.0.smithi151.stdout:5/848: dread d2/d6/d19/d30/d88/ddc/fa3 [0,4194304] 0 2024-04-16T02:36:04.400 INFO:tasks.workunit.client.0.smithi151.stdout:1/676: rename d3/d58/df/d32/f37 to d3/d1f/d48/fe8 0 2024-04-16T02:36:04.401 INFO:tasks.workunit.client.0.smithi151.stdout:3/600: chown d1/d6/d22/d6c/c8c 7529522 1 2024-04-16T02:36:04.401 INFO:tasks.workunit.client.0.smithi151.stdout:4/727: dwrite d0/fb [4194304,4194304] 0 2024-04-16T02:36:04.403 INFO:tasks.workunit.client.0.smithi151.stdout:1/677: chown d3/d1f/dd7/ddb 986848 1 2024-04-16T02:36:04.403 INFO:tasks.workunit.client.0.smithi151.stdout:0/616: mknod dc/d4c/d9d/cc8 0 2024-04-16T02:36:04.406 INFO:tasks.workunit.client.0.smithi151.stdout:7/774: rename d0/d1f/d46/d4f/l97 to d0/d27/da6/df0/l109 0 2024-04-16T02:36:04.407 INFO:tasks.workunit.client.0.smithi151.stdout:8/601: mknod d11/d2a/d33/d4e/d62/dc8/cd1 0 2024-04-16T02:36:04.409 INFO:tasks.workunit.client.0.smithi151.stdout:8/602: readlink d11/d2a/d85/lcf 0 2024-04-16T02:36:04.409 INFO:tasks.workunit.client.0.smithi151.stdout:5/849: creat d2/d6/dfc/f110 x:0 0 0 2024-04-16T02:36:04.412 INFO:tasks.workunit.client.0.smithi151.stdout:0/617: mknod dc/d3c/d9f/cc9 0 2024-04-16T02:36:04.413 INFO:tasks.workunit.client.0.smithi151.stdout:1/678: unlink d3/d4/d64/d1c/f2c 0 2024-04-16T02:36:04.413 INFO:tasks.workunit.client.0.smithi151.stdout:3/601: dwrite d1/d37/d87/f8b [0,4194304] 0 2024-04-16T02:36:04.414 INFO:tasks.workunit.client.0.smithi151.stdout:4/728: rmdir d0/d11/d19/d46/dcd/def/dfe 39 2024-04-16T02:36:04.418 INFO:tasks.workunit.client.0.smithi151.stdout:0/618: rmdir dc/d4c/d72/d76/db0 39 2024-04-16T02:36:04.421 INFO:tasks.workunit.client.0.smithi151.stdout:4/729: unlink d0/dde/d45/f92 0 2024-04-16T02:36:04.422 INFO:tasks.workunit.client.0.smithi151.stdout:5/850: symlink d2/d6/d19/d30/l111 0 2024-04-16T02:36:04.423 INFO:tasks.workunit.client.0.smithi151.stdout:8/603: mkdir d11/d2a/d35/d52/db3/dcd/dd2 0 2024-04-16T02:36:04.425 INFO:tasks.workunit.client.0.smithi151.stdout:3/602: rename d1/d6/d22/d23/d30/d44/l64 to d1/d6/d22/d23/d30/d44/d46/d8a/d3f/lc3 0 2024-04-16T02:36:04.426 INFO:tasks.workunit.client.0.smithi151.stdout:3/603: chown d1/d6/d22 4202037 1 2024-04-16T02:36:04.427 INFO:tasks.workunit.client.0.smithi151.stdout:7/775: rmdir d0/d1f/d4c/d82/d106 0 2024-04-16T02:36:04.433 INFO:tasks.workunit.client.0.smithi151.stdout:1/679: getdents d3/d33/d10/d4f/dd1 0 2024-04-16T02:36:04.435 INFO:tasks.workunit.client.0.smithi151.stdout:7/776: getdents d0/d1f/d4c/db6/dd7 0 2024-04-16T02:36:04.437 INFO:tasks.workunit.client.0.smithi151.stdout:8/604: rename d11/d22/d2e/f5c to d11/d2a/d33/d4e/d62/fd3 0 2024-04-16T02:36:04.446 INFO:tasks.workunit.client.0.smithi151.stdout:8/605: rename f2 to d11/d2a/d35/d52/db3/dcd/fd4 0 2024-04-16T02:36:04.447 INFO:tasks.workunit.client.0.smithi151.stdout:8/606: fdatasync d11/d22/d2e/f65 0 2024-04-16T02:36:04.452 INFO:tasks.workunit.client.0.smithi151.stdout:1/680: write d3/d58/df/f6c [159901,31812] 0 2024-04-16T02:36:04.452 INFO:tasks.workunit.client.0.smithi151.stdout:7/777: link d0/d2c/d38/dab/lc8 d0/d2c/d38/d85/db7/l10a 0 2024-04-16T02:36:04.454 INFO:tasks.workunit.client.0.smithi151.stdout:5/851: dwrite d2/d6/d19/d30/d49/d7f/f90 [0,4194304] 0 2024-04-16T02:36:04.456 INFO:tasks.workunit.client.0.smithi151.stdout:1/681: readlink d3/d58/df/l35 0 2024-04-16T02:36:04.457 INFO:tasks.workunit.client.0.smithi151.stdout:8/607: creat d11/d2a/fd5 x:0 0 0 2024-04-16T02:36:04.460 INFO:tasks.workunit.client.0.smithi151.stdout:0/619: dwrite dc/d15/d34/d67/f5e [0,4194304] 0 2024-04-16T02:36:04.460 INFO:tasks.workunit.client.0.smithi151.stdout:7/778: mkdir d0/d1f/d4c/d98/d10b 0 2024-04-16T02:36:04.462 INFO:tasks.workunit.client.0.smithi151.stdout:5/852: mknod d2/d6/d43/c112 0 2024-04-16T02:36:04.463 INFO:tasks.workunit.client.0.smithi151.stdout:1/682: creat d3/d33/d10/fe9 x:0 0 0 2024-04-16T02:36:04.464 INFO:tasks.workunit.client.0.smithi151.stdout:0/620: chown dc/d15/d34/d67/c33 62330421 1 2024-04-16T02:36:04.467 INFO:tasks.workunit.client.0.smithi151.stdout:7/779: symlink d0/d1f/d4c/d82/db3/l10c 0 2024-04-16T02:36:04.475 INFO:tasks.workunit.client.0.smithi151.stdout:5/853: dread d2/d6/d19/d30/d88/ddc/d83/fe1 [0,4194304] 0 2024-04-16T02:36:04.476 INFO:tasks.workunit.client.0.smithi151.stdout:1/683: mknod d3/d33/d10/cea 0 2024-04-16T02:36:04.477 INFO:tasks.workunit.client.0.smithi151.stdout:7/780: creat d0/d1f/d99/f10d x:0 0 0 2024-04-16T02:36:04.479 INFO:tasks.workunit.client.0.smithi151.stdout:1/684: read d3/d4/d64/fa4 [2207136,64518] 0 2024-04-16T02:36:04.479 INFO:tasks.workunit.client.0.smithi151.stdout:1/685: chown d3/d58/df/d68 59774449 1 2024-04-16T02:36:04.480 INFO:tasks.workunit.client.0.smithi151.stdout:0/621: fdatasync dc/d15/d34/d67/d51/fbe 0 2024-04-16T02:36:04.481 INFO:tasks.workunit.client.0.smithi151.stdout:0/622: stat dc/d4c/d9d/cc8 0 2024-04-16T02:36:04.486 INFO:tasks.workunit.client.0.smithi151.stdout:1/686: symlink d3/d1f/dd7/leb 0 2024-04-16T02:36:04.487 INFO:tasks.workunit.client.0.smithi151.stdout:0/623: mknod dc/d4c/d72/cca 0 2024-04-16T02:36:04.488 INFO:tasks.workunit.client.0.smithi151.stdout:0/624: fsync dc/d20/fbc 0 2024-04-16T02:36:04.489 INFO:tasks.workunit.client.0.smithi151.stdout:7/781: truncate d0/d27/f95 1599210 0 2024-04-16T02:36:04.489 INFO:tasks.workunit.client.0.smithi151.stdout:0/625: fsync dc/d15/d34/d67/f78 0 2024-04-16T02:36:04.494 INFO:tasks.workunit.client.0.smithi151.stdout:1/687: mkdir d3/d4/d64/d2a/d52/dec 0 2024-04-16T02:36:04.497 INFO:tasks.workunit.client.0.smithi151.stdout:8/608: truncate d11/d22/d2e/dc0/d92/f86 353721 0 2024-04-16T02:36:04.498 INFO:tasks.workunit.client.0.smithi151.stdout:7/782: dread d0/d27/f9f [0,4194304] 0 2024-04-16T02:36:04.501 INFO:tasks.workunit.client.0.smithi151.stdout:1/688: rename d3/d4/d64/da7 to d3/d58/df/d32/d3a/d71/ded 0 2024-04-16T02:36:04.502 INFO:tasks.workunit.client.0.smithi151.stdout:1/689: dread - d3/fe1 zero size 2024-04-16T02:36:04.505 INFO:tasks.workunit.client.0.smithi151.stdout:7/783: creat d0/d2c/d38/d85/db7/f10e x:0 0 0 2024-04-16T02:36:04.510 INFO:tasks.workunit.client.0.smithi151.stdout:7/784: creat d0/d1f/d4c/db6/f10f x:0 0 0 2024-04-16T02:36:04.512 INFO:tasks.workunit.client.0.smithi151.stdout:8/609: rename ld to d11/d2a/d33/ld6 0 2024-04-16T02:36:04.513 INFO:tasks.workunit.client.0.smithi151.stdout:7/785: write d0/d1f/d4c/db6/f10f [116972,76232] 0 2024-04-16T02:36:04.516 INFO:tasks.workunit.client.0.smithi151.stdout:7/786: dread - d0/d1f/d4c/f81 zero size 2024-04-16T02:36:04.518 INFO:tasks.workunit.client.0.smithi151.stdout:5/854: dwrite d2/d6/d19/d62/dd0/d9c/fa2 [0,4194304] 0 2024-04-16T02:36:04.518 INFO:tasks.workunit.client.0.smithi151.stdout:8/610: mknod d11/d2a/d83/dce/cd7 0 2024-04-16T02:36:04.519 INFO:tasks.workunit.client.0.smithi151.stdout:5/855: readlink d2/d6/l15 0 2024-04-16T02:36:04.520 INFO:tasks.workunit.client.0.smithi151.stdout:8/611: read d11/f1a [1363600,31100] 0 2024-04-16T02:36:04.523 INFO:tasks.workunit.client.0.smithi151.stdout:8/612: truncate d11/d22/f27 6415766 0 2024-04-16T02:36:04.524 INFO:tasks.workunit.client.0.smithi151.stdout:7/787: mkdir d0/d2c/d38/d85/db7/d110 0 2024-04-16T02:36:04.529 INFO:tasks.workunit.client.0.smithi151.stdout:5/856: mkdir d2/d6/d19/d30/d113 0 2024-04-16T02:36:04.530 INFO:tasks.workunit.client.0.smithi151.stdout:8/613: mknod d11/d2a/d33/d4e/d62/d6b/cd8 0 2024-04-16T02:36:04.535 INFO:tasks.workunit.client.0.smithi151.stdout:1/690: dread d3/d33/d10/f6b [0,4194304] 0 2024-04-16T02:36:04.539 INFO:tasks.workunit.client.0.smithi151.stdout:7/788: link d0/d1f/d4c/d82/db3/c68 d0/d2c/d38/de7/c111 0 2024-04-16T02:36:04.541 INFO:tasks.workunit.client.0.smithi151.stdout:8/614: dwrite d11/d22/d2e/dc0/d92/d78/fb4 [0,4194304] 0 2024-04-16T02:36:04.544 INFO:tasks.workunit.client.0.smithi151.stdout:7/789: truncate d0/f22 1140902 0 2024-04-16T02:36:04.548 INFO:tasks.workunit.client.0.smithi151.stdout:8/615: mknod d11/d2a/d33/d4e/d62/db7/cd9 0 2024-04-16T02:36:04.554 INFO:tasks.workunit.client.0.smithi151.stdout:8/616: symlink d11/d2a/d33/lda 0 2024-04-16T02:36:04.557 INFO:tasks.workunit.client.0.smithi151.stdout:8/617: dread - d11/d22/d2e/dc0/d92/f93 zero size 2024-04-16T02:36:04.559 INFO:tasks.workunit.client.0.smithi151.stdout:7/790: dwrite d0/d1f/d31/d42/f6c [0,4194304] 0 2024-04-16T02:36:04.559 INFO:tasks.workunit.client.0.smithi151.stdout:5/857: write d2/d6/d19/d30/d88/ddc/da8/dd3/f100 [1002935,116034] 0 2024-04-16T02:36:04.563 INFO:tasks.workunit.client.0.smithi151.stdout:8/618: creat d11/d22/d6c/d84/dc9/fdb x:0 0 0 2024-04-16T02:36:04.567 INFO:tasks.workunit.client.0.smithi151.stdout:1/691: truncate d3/d1f/d48/fe8 2641675 0 2024-04-16T02:36:04.572 INFO:tasks.workunit.client.0.smithi151.stdout:5/858: stat d2/d6/d19/d1a/f52 0 2024-04-16T02:36:04.574 INFO:tasks.workunit.client.0.smithi151.stdout:1/692: truncate d3/d58/df/d32/f92 4532821 0 2024-04-16T02:36:04.578 INFO:tasks.workunit.client.0.smithi151.stdout:1/693: unlink d3/d33/d10/l76 0 2024-04-16T02:36:04.580 INFO:tasks.workunit.client.0.smithi151.stdout:5/859: rename d2/d6/d19/d30/d36/f5b to d2/d6/d19/d30/d36/f114 0 2024-04-16T02:36:04.585 INFO:tasks.workunit.client.0.smithi151.stdout:1/694: creat d3/d4/d64/fee x:0 0 0 2024-04-16T02:36:04.586 INFO:tasks.workunit.client.0.smithi151.stdout:8/619: truncate d11/d2a/d33/f48 1123365 0 2024-04-16T02:36:04.588 INFO:tasks.workunit.client.0.smithi151.stdout:5/860: rename d2/d6/d19/d1a/d4c/c73 to d2/d6/d19/d30/d88/ddc/da8/dd3/ddb/c115 0 2024-04-16T02:36:04.589 INFO:tasks.workunit.client.0.smithi151.stdout:7/791: dwrite d0/d2c/d8e/fc5 [0,4194304] 0 2024-04-16T02:36:04.590 INFO:tasks.workunit.client.0.smithi151.stdout:7/792: fsync d0/d1f/d60/fbd 0 2024-04-16T02:36:04.591 INFO:tasks.workunit.client.0.smithi151.stdout:8/620: creat d11/d22/d2e/fdc x:0 0 0 2024-04-16T02:36:04.595 INFO:tasks.workunit.client.0.smithi151.stdout:1/695: rename d3/d4/d64/d2a/c46 to d3/d1f/dd7/ddb/cef 0 2024-04-16T02:36:04.601 INFO:tasks.workunit.client.0.smithi151.stdout:1/696: link d3/d58/df/d32/d3a/d6d/l77 d3/d58/df/d32/d3a/d6d/lf0 0 2024-04-16T02:36:04.603 INFO:tasks.workunit.client.0.smithi151.stdout:1/697: fsync d3/d58/df/d38/f5a 0 2024-04-16T02:36:04.605 INFO:tasks.workunit.client.0.smithi151.stdout:1/698: fsync d3/d4/d64/d40/da3/fa6 0 2024-04-16T02:36:04.613 INFO:tasks.workunit.client.0.smithi151.stdout:8/621: dread d11/d2a/d33/d4e/f7c [0,4194304] 0 2024-04-16T02:36:04.646 INFO:tasks.workunit.client.0.smithi151.stdout:7/793: dread d0/d1f/d46/d4f/f57 [0,4194304] 0 2024-04-16T02:36:04.648 INFO:tasks.workunit.client.0.smithi151.stdout:5/861: write d2/d6/d19/d62/f7b [942935,58824] 0 2024-04-16T02:36:04.654 INFO:tasks.workunit.client.0.smithi151.stdout:5/862: dread d2/d6/d19/d62/f7b [0,4194304] 0 2024-04-16T02:36:04.655 INFO:tasks.workunit.client.0.smithi151.stdout:5/863: readlink d2/d6/d19/d30/l111 0 2024-04-16T02:36:04.655 INFO:tasks.workunit.client.0.smithi151.stdout:5/864: chown d2/d6/d19/d93 30742141 1 2024-04-16T02:36:04.678 INFO:tasks.workunit.client.0.smithi151.stdout:1/699: mkdir d3/d58/df/d32/d3a/df1 0 2024-04-16T02:36:04.678 INFO:tasks.workunit.client.0.smithi151.stdout:7/794: creat d0/d2c/d38/db0/f112 x:0 0 0 2024-04-16T02:36:04.681 INFO:tasks.workunit.client.0.smithi151.stdout:8/622: creat d11/fdd x:0 0 0 2024-04-16T02:36:04.683 INFO:tasks.workunit.client.0.smithi151.stdout:7/795: truncate d0/f11 5527380 0 2024-04-16T02:36:04.684 INFO:tasks.workunit.client.0.smithi151.stdout:5/865: getdents d2/d6/d19/d30/d49/de7 0 2024-04-16T02:36:04.689 INFO:tasks.workunit.client.0.smithi151.stdout:7/796: link d0/d2c/f5f d0/d1f/d4c/d98/f113 0 2024-04-16T02:36:04.690 INFO:tasks.workunit.client.0.smithi151.stdout:7/797: fsync d0/d27/f9f 0 2024-04-16T02:36:04.693 INFO:tasks.workunit.client.0.smithi151.stdout:7/798: chown d0/f22 2 1 2024-04-16T02:36:04.699 INFO:tasks.workunit.client.0.smithi151.stdout:7/799: dread d0/d1f/d4c/d98/f113 [0,4194304] 0 2024-04-16T02:36:04.715 INFO:tasks.workunit.client.0.smithi151.stdout:1/700: dwrite d3/d33/d10/f5d [4194304,4194304] 0 2024-04-16T02:36:04.717 INFO:tasks.workunit.client.0.smithi151.stdout:8/623: dwrite d11/d2a/d33/d4e/f7c [0,4194304] 0 2024-04-16T02:36:04.720 INFO:tasks.workunit.client.0.smithi151.stdout:7/800: dwrite d0/d1f/d4c/d98/f113 [0,4194304] 0 2024-04-16T02:36:04.725 INFO:tasks.workunit.client.0.smithi151.stdout:8/624: chown c9 22 1 2024-04-16T02:36:04.726 INFO:tasks.workunit.client.0.smithi151.stdout:1/701: mknod d3/d4/db2/cf2 0 2024-04-16T02:36:04.727 INFO:tasks.workunit.client.0.smithi151.stdout:1/702: stat d3/d4/d64/d2a/d52/dec 0 2024-04-16T02:36:04.731 INFO:tasks.workunit.client.0.smithi151.stdout:8/625: mknod d11/d2a/cde 0 2024-04-16T02:36:04.734 INFO:tasks.workunit.client.0.smithi151.stdout:8/626: creat d11/d2a/d83/fdf x:0 0 0 2024-04-16T02:36:04.736 INFO:tasks.workunit.client.0.smithi151.stdout:8/627: write d11/d22/d2e/fdc [677430,28491] 0 2024-04-16T02:36:04.737 INFO:tasks.workunit.client.0.smithi151.stdout:8/628: stat d11/d22/d6c/d84/dc9/cba 0 2024-04-16T02:36:04.737 INFO:tasks.workunit.client.0.smithi151.stdout:8/629: stat ff 0 2024-04-16T02:36:04.750 INFO:tasks.workunit.client.0.smithi151.stdout:8/630: dread d11/d22/d2e/fdc [0,4194304] 0 2024-04-16T02:36:04.754 INFO:tasks.workunit.client.0.smithi151.stdout:8/631: creat d11/d22/d2e/fe0 x:0 0 0 2024-04-16T02:36:04.755 INFO:tasks.workunit.client.0.smithi151.stdout:1/703: dread d3/d4/d64/d2a/f8d [0,4194304] 0 2024-04-16T02:36:04.755 INFO:tasks.workunit.client.0.smithi151.stdout:7/801: dwrite d0/d1f/fe8 [0,4194304] 0 2024-04-16T02:36:04.757 INFO:tasks.workunit.client.0.smithi151.stdout:8/632: readlink d11/d22/d2e/dc0/d92/la7 0 2024-04-16T02:36:04.760 INFO:tasks.workunit.client.0.smithi151.stdout:7/802: creat d0/d27/da6/dcd/f114 x:0 0 0 2024-04-16T02:36:04.762 INFO:tasks.workunit.client.0.smithi151.stdout:8/633: fdatasync d11/f1b 0 2024-04-16T02:36:04.766 INFO:tasks.workunit.client.0.smithi151.stdout:7/803: link d0/d1f/d46/d4f/f57 d0/d27/f115 0 2024-04-16T02:36:04.769 INFO:tasks.workunit.client.0.smithi151.stdout:1/704: rename d3/d4/d64/d2a to d3/d58/df/d32/d3a/df3 0 2024-04-16T02:36:04.770 INFO:tasks.workunit.client.0.smithi151.stdout:7/804: creat d0/d1f/d99/dc0/f116 x:0 0 0 2024-04-16T02:36:04.772 INFO:tasks.workunit.client.0.smithi151.stdout:8/634: rename d11/lb2 to d11/d22/d2e/dc0/d92/le1 0 2024-04-16T02:36:04.774 INFO:tasks.workunit.client.0.smithi151.stdout:1/705: symlink d3/d58/df/d32/dbd/lf4 0 2024-04-16T02:36:04.775 INFO:tasks.workunit.client.0.smithi151.stdout:1/706: stat d3/d33/d10/f4e 0 2024-04-16T02:36:04.776 INFO:tasks.workunit.client.0.smithi151.stdout:1/707: stat d3/d58/df/d32/d3a/df3 0 2024-04-16T02:36:04.777 INFO:tasks.workunit.client.0.smithi151.stdout:8/635: rename d11/d2a/cde to d11/d2a/d35/d52/db3/dcc/ce2 0 2024-04-16T02:36:04.780 INFO:tasks.workunit.client.0.smithi151.stdout:8/636: chown d11/d2a/d33/d4e/d62/d94/laa 572223067 1 2024-04-16T02:36:04.786 INFO:tasks.workunit.client.0.smithi151.stdout:7/805: dread d0/f7 [0,4194304] 0 2024-04-16T02:36:04.789 INFO:tasks.workunit.client.0.smithi151.stdout:7/806: write d0/d1f/d46/d4f/f62 [61296,109979] 0 2024-04-16T02:36:04.797 INFO:tasks.workunit.client.0.smithi151.stdout:7/807: dwrite d0/fe0 [0,4194304] 0 2024-04-16T02:36:04.797 INFO:tasks.workunit.client.0.smithi151.stdout:1/708: dread d3/d4/d64/fa4 [0,4194304] 0 2024-04-16T02:36:04.803 INFO:tasks.workunit.client.0.smithi151.stdout:7/808: unlink d0/d1f/d33/d56/f84 0 2024-04-16T02:36:04.803 INFO:tasks.workunit.client.0.smithi151.stdout:8/637: dwrite d11/d22/f76 [0,4194304] 0 2024-04-16T02:36:04.804 INFO:tasks.workunit.client.0.smithi151.stdout:1/709: write d3/d58/df/d32/d3a/d67/f93 [3751782,54706] 0 2024-04-16T02:36:04.806 INFO:tasks.workunit.client.0.smithi151.stdout:7/809: write d0/d2c/d38/d85/db7/f10e [897219,49857] 0 2024-04-16T02:36:04.807 INFO:tasks.workunit.client.0.smithi151.stdout:1/710: truncate d3/d33/d10/fe9 92222 0 2024-04-16T02:36:04.809 INFO:tasks.workunit.client.0.smithi151.stdout:8/638: mknod d11/d22/d6c/d84/dc9/ce3 0 2024-04-16T02:36:04.812 INFO:tasks.workunit.client.0.smithi151.stdout:8/639: fdatasync d11/d2a/d85/f9f 0 2024-04-16T02:36:04.814 INFO:tasks.workunit.client.0.smithi151.stdout:8/640: write d11/d22/f27 [2100318,10987] 0 2024-04-16T02:36:04.883 INFO:tasks.workunit.client.0.smithi151.stdout:1/711: getdents d3/d33/d10/d4f 0 2024-04-16T02:36:04.884 INFO:tasks.workunit.client.0.smithi151.stdout:7/810: write d0/d1f/d46/d4f/f57 [1215903,67607] 0 2024-04-16T02:36:04.892 INFO:tasks.workunit.client.0.smithi151.stdout:7/811: truncate d0/d2c/d38/d5e/faa 1596614 0 2024-04-16T02:36:04.896 INFO:tasks.workunit.client.0.smithi151.stdout:7/812: dread d0/d1f/f52 [0,4194304] 0 2024-04-16T02:36:04.899 INFO:tasks.workunit.client.0.smithi151.stdout:7/813: mkdir d0/d2c/d38/d85/d117 0 2024-04-16T02:36:04.901 INFO:tasks.workunit.client.0.smithi151.stdout:7/814: creat d0/d2c/d38/d5e/deb/f118 x:0 0 0 2024-04-16T02:36:04.902 INFO:tasks.workunit.client.0.smithi151.stdout:7/815: chown d0/d2c/cbf 3704 1 2024-04-16T02:36:04.909 INFO:tasks.workunit.client.0.smithi151.stdout:7/816: dread d0/f22 [0,4194304] 0 2024-04-16T02:36:04.926 INFO:tasks.workunit.client.0.smithi151.stdout:7/817: creat d0/d1f/d33/d56/f119 x:0 0 0 2024-04-16T02:36:04.929 INFO:tasks.workunit.client.0.smithi151.stdout:7/818: creat d0/d27/f11a x:0 0 0 2024-04-16T02:36:04.930 INFO:tasks.workunit.client.0.smithi151.stdout:7/819: write d0/f3 [13339220,89427] 0 2024-04-16T02:36:04.941 INFO:tasks.workunit.client.0.smithi151.stdout:8/641: symlink d11/d2a/d33/d4e/d62/le4 0 2024-04-16T02:36:04.944 INFO:tasks.workunit.client.0.smithi151.stdout:8/642: creat d11/d2a/d33/d4e/d62/dc8/fe5 x:0 0 0 2024-04-16T02:36:04.945 INFO:tasks.workunit.client.0.smithi151.stdout:8/643: dread - d11/fdd zero size 2024-04-16T02:36:04.947 INFO:tasks.workunit.client.0.smithi151.stdout:7/820: dread d0/d2c/d38/f3d [0,4194304] 0 2024-04-16T02:36:04.948 INFO:tasks.workunit.client.0.smithi151.stdout:8/644: read d11/d22/d2e/dc0/d92/d78/fa8 [1658099,32203] 0 2024-04-16T02:36:04.949 INFO:tasks.workunit.client.0.smithi151.stdout:8/645: read - d11/d22/d2e/dc0/d92/d78/fad zero size 2024-04-16T02:36:04.955 INFO:tasks.workunit.client.0.smithi151.stdout:8/646: creat d11/d22/d2e/dc0/fe6 x:0 0 0 2024-04-16T02:36:04.961 INFO:tasks.workunit.client.0.smithi151.stdout:7/821: link d0/d2c/d38/d85/db7/lc4 d0/d1f/d33/d89/l11b 0 2024-04-16T02:36:04.962 INFO:tasks.workunit.client.0.smithi151.stdout:7/822: chown d0/d2c/d38/d5e/dd1/le4 369779 1 2024-04-16T02:36:04.968 INFO:tasks.workunit.client.0.smithi151.stdout:7/823: dread d0/d1f/d60/f91 [0,4194304] 0 2024-04-16T02:36:04.973 INFO:tasks.workunit.client.0.smithi151.stdout:7/824: dread d0/d1f/d31/d42/f6c [0,4194304] 0 2024-04-16T02:36:04.974 INFO:tasks.workunit.client.0.smithi151.stdout:7/825: readlink d0/d1f/d4c/d82/l103 0 2024-04-16T02:36:04.977 INFO:tasks.workunit.client.0.smithi151.stdout:7/826: truncate d0/d1f/d4c/d82/fe3 734504 0 2024-04-16T02:36:04.979 INFO:tasks.workunit.client.0.smithi151.stdout:7/827: truncate d0/d1f/d46/d4f/f50 974662 0 2024-04-16T02:36:04.982 INFO:tasks.workunit.client.0.smithi151.stdout:7/828: mkdir d0/d1f/d4c/db6/d11c 0 2024-04-16T02:36:04.983 INFO:tasks.workunit.client.0.smithi151.stdout:1/712: write d3/d58/df/d32/d3a/df3/d52/f72 [849942,87052] 0 2024-04-16T02:36:05.002 INFO:tasks.workunit.client.0.smithi151.stdout:7/829: write d0/d1f/d46/d4f/f50 [673773,104012] 0 2024-04-16T02:36:05.223 INFO:tasks.workunit.client.0.smithi151.stdout:6/840: sync 2024-04-16T02:36:05.226 INFO:tasks.workunit.client.0.smithi151.stdout:6/841: creat d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/f120 x:0 0 0 2024-04-16T02:36:05.227 INFO:tasks.workunit.client.0.smithi151.stdout:6/842: dread - d2/d13/d63/def/d2e/f74 zero size 2024-04-16T02:36:05.231 INFO:tasks.workunit.client.0.smithi151.stdout:6/843: write d2/d13/d63/def/df/d23/d45/fdb [3012129,89003] 0 2024-04-16T02:36:05.233 INFO:tasks.workunit.client.0.smithi151.stdout:6/844: fsync d2/d13/d63/def/df/d23/d82/d8d/f80 0 2024-04-16T02:36:05.237 INFO:tasks.workunit.client.0.smithi151.stdout:6/845: rename d2/d13/d63/def/df/d23/d82/d8d/f4b to d2/d13/d63/f121 0 2024-04-16T02:36:05.239 INFO:tasks.workunit.client.0.smithi151.stdout:6/846: fsync d2/d13/d63/def/df/d23/d45/fdb 0 2024-04-16T02:36:05.242 INFO:tasks.workunit.client.0.smithi151.stdout:6/847: creat d2/d13/d63/def/df/d23/d45/f122 x:0 0 0 2024-04-16T02:36:05.258 INFO:tasks.workunit.client.0.smithi151.stdout:6/848: dread d2/d13/d63/def/df/d23/d82/d8d/d22/f68 [0,4194304] 0 2024-04-16T02:36:05.260 INFO:tasks.workunit.client.0.smithi151.stdout:6/849: creat d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d73/d97/f123 x:0 0 0 2024-04-16T02:36:05.261 INFO:tasks.workunit.client.0.smithi151.stdout:6/850: chown d2/d13/d70/lc4 0 1 2024-04-16T02:36:05.263 INFO:tasks.workunit.client.0.smithi151.stdout:6/851: mknod d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d85/c124 0 2024-04-16T02:36:05.265 INFO:tasks.workunit.client.0.smithi151.stdout:6/852: symlink d2/d13/l125 0 2024-04-16T02:36:05.267 INFO:tasks.workunit.client.0.smithi151.stdout:6/853: mkdir d2/d13/d63/def/df/d20/d101/d126 0 2024-04-16T02:36:05.269 INFO:tasks.workunit.client.0.smithi151.stdout:6/854: read d2/d13/d46/f79 [260439,90275] 0 2024-04-16T02:36:05.278 INFO:tasks.workunit.client.0.smithi151.stdout:6/855: rename d2/d13/d63/def/df/d20/d101 to d2/d13/d63/def/df/d20/d4f/d127 0 2024-04-16T02:36:05.285 INFO:tasks.workunit.client.0.smithi151.stdout:6/856: dread d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/f7f [0,4194304] 0 2024-04-16T02:36:05.285 INFO:tasks.workunit.client.0.smithi151.stdout:6/857: readlink d2/d13/d46/lfe 0 2024-04-16T02:36:05.295 INFO:tasks.workunit.client.0.smithi151.stdout:6/858: dwrite d2/d13/d63/def/df/d23/d56/d72/fa0 [4194304,4194304] 0 2024-04-16T02:36:05.299 INFO:tasks.workunit.client.0.smithi151.stdout:6/859: getdents d2/d13/d63/def/df/d23/d82 0 2024-04-16T02:36:05.300 INFO:tasks.workunit.client.0.smithi151.stdout:6/860: chown d2/d13/d3a/da6/ff2 19 1 2024-04-16T02:36:05.301 INFO:tasks.workunit.client.0.smithi151.stdout:6/861: write d2/d13/d6f/f102 [82632,60451] 0 2024-04-16T02:36:05.306 INFO:tasks.workunit.client.0.smithi151.stdout:6/862: creat d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/f128 x:0 0 0 2024-04-16T02:36:05.308 INFO:tasks.workunit.client.0.smithi151.stdout:6/863: unlink d2/d13/d63/def/df/d23/d82/d8d/d22/d40/f3b 0 2024-04-16T02:36:05.311 INFO:tasks.workunit.client.0.smithi151.stdout:6/864: creat d2/d13/d63/def/df/d23/d31/d52/f129 x:0 0 0 2024-04-16T02:36:05.334 INFO:tasks.workunit.client.0.smithi151.stdout:6/865: dread d2/d13/d63/f121 [0,4194304] 0 2024-04-16T02:36:05.336 INFO:tasks.workunit.client.0.smithi151.stdout:6/866: truncate d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/f120 522715 0 2024-04-16T02:36:05.340 INFO:tasks.workunit.client.0.smithi151.stdout:6/867: rename d2/d13/d63/def/df/d23/d56/f7e to d2/d13/d63/def/df/d23/d31/f12a 0 2024-04-16T02:36:05.343 INFO:tasks.workunit.client.0.smithi151.stdout:6/868: symlink d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f/d85/l12b 0 2024-04-16T02:36:05.347 INFO:tasks.workunit.client.0.smithi151.stdout:6/869: rename d2/d13/d3a/f54 to d2/d13/d63/def/d2e/f12c 0 2024-04-16T02:36:05.348 INFO:tasks.workunit.client.0.smithi151.stdout:6/870: creat d2/d13/d46/f12d x:0 0 0 2024-04-16T02:36:05.350 INFO:tasks.workunit.client.0.smithi151.stdout:6/871: write d2/d13/d63/def/df/d23/d82/d8d/f80 [596861,43534] 0 2024-04-16T02:36:05.363 INFO:tasks.workunit.client.0.smithi151.stdout:6/872: dwrite d2/d13/d63/def/df/d20/de6/fe9 [0,4194304] 0 2024-04-16T02:36:05.367 INFO:tasks.workunit.client.0.smithi151.stdout:6/873: mknod d2/d13/d63/def/df/d23/d56/c12e 0 2024-04-16T02:36:05.369 INFO:tasks.workunit.client.0.smithi151.stdout:6/874: symlink d2/d13/d63/def/df/d23/d45/l12f 0 2024-04-16T02:36:05.376 INFO:tasks.workunit.client.0.smithi151.stdout:6/875: rename d2/d13/d63/def/df/d23/d82/d8d/d22/d40/d3f to d2/d13/d3a/da6/d130 0 2024-04-16T02:36:05.378 INFO:tasks.workunit.client.0.smithi151.stdout:6/876: mknod d2/d13/d63/def/df/d23/d82/d8d/c131 0 2024-04-16T02:36:05.379 INFO:tasks.workunit.client.0.smithi151.stdout:6/877: chown d2/d13/d3a 19 1 2024-04-16T02:36:05.383 INFO:tasks.workunit.client.0.smithi151.stdout:6/878: link d2/d13/d63/def/f6 d2/d13/d63/def/df/d23/d82/d8d/d22/f132 0 2024-04-16T02:36:05.385 INFO:tasks.workunit.client.0.smithi151.stdout:6/879: write d2/d13/d63/f121 [516452,118182] 0 2024-04-16T02:36:05.393 INFO:tasks.workunit.client.0.smithi151.stdout:6/880: write d2/d13/d63/def/df/d23/d82/d8d/f59 [3278579,114336] 0 2024-04-16T02:36:05.395 INFO:tasks.workunit.client.0.smithi151.stdout:6/881: creat d2/d13/d63/d112/f133 x:0 0 0 2024-04-16T02:36:05.397 INFO:tasks.workunit.client.0.smithi151.stdout:6/882: truncate d2/d13/d63/def/df/d23/d82/f9c 1015518 0 2024-04-16T02:36:05.398 INFO:tasks.workunit.client.0.smithi151.stdout:6/883: dread - d2/d13/d3a/da6/d130/d73/fed zero size 2024-04-16T02:36:05.414 INFO:tasks.workunit.client.0.smithi151.stdout:6/884: dread d2/d13/d63/def/df/d23/d45/fd1 [0,4194304] 0 2024-04-16T02:36:05.416 INFO:tasks.workunit.client.0.smithi151.stdout:6/885: chown d2/d13/d63/def/df/d23/d82/d8d/d22/c7c 124380 1 2024-04-16T02:36:05.422 INFO:tasks.workunit.client.0.smithi151.stdout:6/886: dwrite d2/d13/d63/def/df/d20/de6/fe9 [0,4194304] 0 2024-04-16T02:36:05.425 INFO:tasks.workunit.client.0.smithi151.stdout:6/887: mknod d2/d13/d63/def/df/d23/d45/c134 0 2024-04-16T02:36:05.428 INFO:tasks.workunit.client.0.smithi151.stdout:6/888: creat d2/d13/d3a/da6/d130/d65/f135 x:0 0 0 2024-04-16T02:36:05.434 INFO:tasks.workunit.client.0.smithi151.stdout:6/889: rename d2/d13/d63/def/d2e/db6 to d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/d136 0 2024-04-16T02:36:05.436 INFO:tasks.workunit.client.0.smithi151.stdout:6/890: chown d2/d13/d63/def/df/d20/d4f/d127/de5/d10e/f111 32 1 2024-04-16T02:36:05.461 INFO:tasks.workunit.client.0.smithi151.stdout:6/891: dread d2/d13/d63/def/df/f3c [0,4194304] 0 2024-04-16T02:36:05.464 INFO:tasks.workunit.client.0.smithi151.stdout:6/892: fdatasync d2/d13/d3a/da6/d130/f66 0 2024-04-16T02:36:05.467 INFO:tasks.workunit.client.0.smithi151.stdout:6/893: unlink d2/d13/d6f/f114 0 2024-04-16T02:36:05.475 INFO:tasks.workunit.client.0.smithi151.stdout:6/894: truncate d2/d13/d63/def/df/d23/d31/f12a 709493 0 2024-04-16T02:36:05.489 INFO:tasks.workunit.client.0.smithi151.stdout:6/895: dwrite d2/d13/d6f/da3/fcb [0,4194304] 0 2024-04-16T02:36:05.490 INFO:tasks.workunit.client.0.smithi151.stdout:6/896: chown d2/d13/d63/def/df/d23/c2f 2826461 1 2024-04-16T02:36:05.501 INFO:tasks.workunit.client.0.smithi151.stdout:6/897: link d2/d13/d3a/c113 d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/dff/c137 0 2024-04-16T02:36:05.506 INFO:tasks.workunit.client.0.smithi151.stdout:6/898: read d2/d13/d63/def/df/d20/f24 [802625,65423] 0 2024-04-16T02:36:05.527 INFO:tasks.workunit.client.0.smithi151.stdout:6/899: dwrite d2/d13/d63/def/df/d23/d31/f12a [0,4194304] 0 2024-04-16T02:36:05.533 INFO:tasks.workunit.client.0.smithi151.stdout:6/900: getdents d2/d13/d63/def/df/d20/d4f/d127/de5 0 2024-04-16T02:36:05.541 INFO:tasks.workunit.client.0.smithi151.stdout:6/901: write d2/d13/d63/def/df/d23/d56/f61 [1728181,78472] 0 2024-04-16T02:36:05.544 INFO:tasks.workunit.client.0.smithi151.stdout:6/902: creat d2/d13/f138 x:0 0 0 2024-04-16T02:36:05.547 INFO:tasks.workunit.client.0.smithi151.stdout:6/903: creat d2/d13/d3a/da6/f139 x:0 0 0 2024-04-16T02:36:05.550 INFO:tasks.workunit.client.0.smithi151.stdout:6/904: mknod d2/d13/d63/def/df/d23/d56/c13a 0 2024-04-16T02:36:05.556 INFO:tasks.workunit.client.0.smithi151.stdout:6/905: dwrite d2/d13/d46/f79 [0,4194304] 0 2024-04-16T02:36:05.561 INFO:tasks.workunit.client.0.smithi151.stdout:6/906: dread d2/d13/d63/f121 [0,4194304] 0 2024-04-16T02:36:05.563 INFO:tasks.workunit.client.0.smithi151.stdout:6/907: write d2/d13/d3a/da6/d130/d65/fdd [3899504,64043] 0 2024-04-16T02:36:05.565 INFO:tasks.workunit.client.0.smithi151.stdout:6/908: write d2/d13/d63/def/df/d23/d56/d72/f119 [1010393,38233] 0 2024-04-16T02:36:05.567 INFO:tasks.workunit.client.0.smithi151.stdout:6/909: rmdir d2/d13/d3a/da6/d130/d73 39 2024-04-16T02:36:05.569 INFO:tasks.workunit.client.0.smithi151.stdout:6/910: mknod d2/d13/d63/def/df/d23/d31/d52/c13b 0 2024-04-16T02:36:05.571 INFO:tasks.workunit.client.0.smithi151.stdout:6/911: dread - d2/d13/d63/def/df/d23/d56/d72/fb5 zero size 2024-04-16T02:36:05.637 INFO:tasks.workunit.client.0.smithi151.stdout:6/912: write d2/f9 [50567,40472] 0 2024-04-16T02:36:05.644 INFO:tasks.workunit.client.0.smithi151.stdout:6/913: write d2/d13/d63/def/df/d23/d82/d8d/d22/f9b [558454,13346] 0 2024-04-16T02:36:05.657 INFO:tasks.workunit.client.0.smithi151.stdout:6/914: dwrite d2/d13/d63/def/df/fd0 [0,4194304] 0 2024-04-16T02:36:05.689 INFO:tasks.workunit.client.0.smithi151.stdout:6/915: dread d2/d13/d63/def/df/d23/d56/f58 [4194304,4194304] 0 2024-04-16T02:36:05.691 INFO:tasks.workunit.client.0.smithi151.stdout:6/916: read - d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/d136/fc0 zero size 2024-04-16T02:36:05.695 INFO:tasks.workunit.client.0.smithi151.stdout:6/917: link d2/d13/d63/def/df/d23/d56/f58 d2/d13/d5f/dd7/f13c 0 2024-04-16T02:36:05.697 INFO:tasks.workunit.client.0.smithi151.stdout:6/918: unlink d2/d13/d63/def/df/d23/d82/ff5 0 2024-04-16T02:36:05.710 INFO:tasks.workunit.client.0.smithi151.stdout:6/919: dread d2/d13/d63/def/df/d23/d45/fdb [0,4194304] 0 2024-04-16T02:36:05.713 INFO:tasks.workunit.client.0.smithi151.stdout:6/920: truncate d2/d13/d3a/da6/d130/f66 1478439 0 2024-04-16T02:36:05.714 INFO:tasks.workunit.client.0.smithi151.stdout:6/921: creat d2/d13/d46/f13d x:0 0 0 2024-04-16T02:36:05.715 INFO:tasks.workunit.client.0.smithi151.stdout:6/922: mkdir d2/d13/d3a/da6/d13e 0 2024-04-16T02:36:05.720 INFO:tasks.workunit.client.0.smithi151.stdout:6/923: dread d2/d13/d6f/da3/fcb [0,4194304] 0 2024-04-16T02:36:05.720 INFO:tasks.workunit.client.0.smithi151.stdout:6/924: chown d2/d3d 880379699 1 2024-04-16T02:36:05.727 INFO:tasks.workunit.client.0.smithi151.stdout:6/925: dwrite d2/d13/d63/def/df/d20/de6/fe9 [4194304,4194304] 0 2024-04-16T02:36:05.731 INFO:tasks.workunit.client.0.smithi151.stdout:6/926: write d2/d13/d63/def/df/d23/d82/d8d/d22/f4e [4568313,118985] 0 2024-04-16T02:36:05.732 INFO:tasks.workunit.client.0.smithi151.stdout:6/927: chown d2/d13/d63/def/df/d23/d31/d52/f129 3761417 1 2024-04-16T02:36:05.735 INFO:tasks.workunit.client.0.smithi151.stdout:6/928: creat d2/d13/d63/def/df/d23/d45/f13f x:0 0 0 2024-04-16T02:36:05.736 INFO:tasks.workunit.client.0.smithi151.stdout:6/929: readlink d2/l83 0 2024-04-16T02:36:05.737 INFO:tasks.workunit.client.0.smithi151.stdout:6/930: stat d2/d13/d63/def/df/d23/d82/d8d/c50 0 2024-04-16T02:36:05.744 INFO:tasks.workunit.client.0.smithi151.stdout:6/931: symlink d2/d13/d63/def/d2e/dba/d10d/l140 0 2024-04-16T02:36:05.755 INFO:tasks.workunit.client.0.smithi151.stdout:6/932: dwrite d2/d13/d63/def/df/d23/d82/d8d/d22/f68 [0,4194304] 0 2024-04-16T02:36:05.758 INFO:tasks.workunit.client.0.smithi151.stdout:6/933: truncate d2/d13/d6f/da3/ffd 184289 0 2024-04-16T02:36:05.767 INFO:tasks.workunit.client.0.smithi151.stdout:6/934: dwrite d2/d13/d5f/ff0 [0,4194304] 0 2024-04-16T02:36:05.772 INFO:tasks.workunit.client.0.smithi151.stdout:6/935: getdents d2/d13/d63/def/df/d23/d82/d8d/d22/d6d 0 2024-04-16T02:36:05.773 INFO:tasks.workunit.client.0.smithi151.stdout:6/936: dread - d2/d13/d63/def/df/d23/d31/d52/df7/f116 zero size 2024-04-16T02:36:05.776 INFO:tasks.workunit.client.0.smithi151.stdout:6/937: rmdir d2/d13/d63/def/df/d23/d82/d8d/d22 39 2024-04-16T02:36:05.787 INFO:tasks.workunit.client.0.smithi151.stdout:6/938: dwrite d2/d13/d6f/da3/ffd [0,4194304] 0 2024-04-16T02:36:05.788 INFO:tasks.workunit.client.0.smithi151.stdout:6/939: write d2/d13/d63/def/df/d23/d31/d52/f129 [223321,41206] 0 2024-04-16T02:36:05.800 INFO:tasks.workunit.client.0.smithi151.stdout:6/940: dwrite d2/d13/d3a/da6/d130/d85/f11c [0,4194304] 0 2024-04-16T02:36:05.803 INFO:tasks.workunit.client.0.smithi151.stdout:6/941: readlink d2/l4 0 2024-04-16T02:36:05.805 INFO:tasks.workunit.client.0.smithi151.stdout:6/942: fsync d2/d13/d46/fbf 0 2024-04-16T02:36:05.807 INFO:tasks.workunit.client.0.smithi151.stdout:6/943: stat d2/d13/d63/def/df/d23/d31/d52/f129 0 2024-04-16T02:36:05.810 INFO:tasks.workunit.client.0.smithi151.stdout:6/944: fsync d2/d13/d63/def/df/d23/d82/d8d/d22/d40/f6a 0 2024-04-16T02:36:05.822 INFO:tasks.workunit.client.0.smithi151.stdout:6/945: dread d2/d3d/f55 [0,4194304] 0 2024-04-16T02:36:05.825 INFO:tasks.workunit.client.0.smithi151.stdout:6/946: rmdir d2/d13/d46/d10b 0 2024-04-16T02:36:05.829 INFO:tasks.workunit.client.0.smithi151.stdout:6/947: symlink d2/d13/d63/def/df/d23/l141 0 2024-04-16T02:36:05.831 INFO:tasks.workunit.client.0.smithi151.stdout:6/948: fsync d2/d13/d5f/ff4 0 2024-04-16T02:36:05.835 INFO:tasks.workunit.client.0.smithi151.stdout:6/949: mkdir d2/d13/d63/def/df/d20/d4f/d127/de5/d142 0 2024-04-16T02:36:05.837 INFO:tasks.workunit.client.0.smithi151.stdout:6/950: readlink d2/d13/d3a/da6/d130/d73/lad 0 2024-04-16T02:36:05.840 INFO:tasks.workunit.client.0.smithi151.stdout:6/951: getdents d2/d13/d3a 0 2024-04-16T02:36:05.847 INFO:tasks.workunit.client.0.smithi151.stdout:6/952: dwrite d2/d13/f138 [0,4194304] 0 2024-04-16T02:36:05.854 INFO:tasks.workunit.client.0.smithi151.stdout:6/953: write d2/d13/d63/def/df/d20/de6/fea [839886,22151] 0 2024-04-16T02:36:05.856 INFO:tasks.workunit.client.0.smithi151.stdout:6/954: mkdir d2/d13/d63/def/d2e/dba/d143 0 2024-04-16T02:36:05.875 INFO:tasks.workunit.client.0.smithi151.stdout:6/955: dwrite d2/d13/d3a/da6/d130/d73/fed [0,4194304] 0 2024-04-16T02:36:05.881 INFO:tasks.workunit.client.0.smithi151.stdout:6/956: creat d2/d13/d63/def/df/d23/d31/f144 x:0 0 0 2024-04-16T02:36:05.884 INFO:tasks.workunit.client.0.smithi151.stdout:6/957: mknod d2/d13/d3a/da6/d130/d73/d97/c145 0 2024-04-16T02:36:05.902 INFO:tasks.workunit.client.0.smithi151.stdout:6/958: dread d2/d13/d63/def/df/d23/f8c [0,4194304] 0 2024-04-16T02:36:05.904 INFO:tasks.workunit.client.0.smithi151.stdout:6/959: creat d2/d13/d63/def/df/d20/d4f/d127/f146 x:0 0 0 2024-04-16T02:36:05.910 INFO:tasks.workunit.client.0.smithi151.stdout:6/960: rename d2/d13/d63/def/df/d23/d31 to d2/d13/d3a/da6/d147 0 2024-04-16T02:36:05.912 INFO:tasks.workunit.client.0.smithi151.stdout:6/961: symlink d2/d13/d70/l148 0 2024-04-16T02:36:05.914 INFO:tasks.workunit.client.0.smithi151.stdout:6/962: unlink d2/l83 0 2024-04-16T02:36:05.915 INFO:tasks.workunit.client.0.smithi151.stdout:6/963: dread - d2/d13/d63/def/df/d20/de6/fec zero size 2024-04-16T02:36:05.917 INFO:tasks.workunit.client.0.smithi151.stdout:6/964: rmdir d2/d13/d63/def/df/d23/d56/d72 39 2024-04-16T02:36:05.918 INFO:tasks.workunit.client.0.smithi151.stdout:6/965: chown d2/d13/d3a/da6/d147/f12a 1 1 2024-04-16T02:36:05.926 INFO:tasks.workunit.client.0.smithi151.stdout:3/604: sync 2024-04-16T02:36:05.926 INFO:tasks.workunit.client.0.smithi151.stdout:9/689: sync 2024-04-16T02:36:05.926 INFO:tasks.workunit.client.0.smithi151.stdout:4/730: sync 2024-04-16T02:36:05.927 INFO:tasks.workunit.client.0.smithi151.stdout:3/605: stat d1 0 2024-04-16T02:36:05.927 INFO:tasks.workunit.client.0.smithi151.stdout:6/966: dwrite d2/d13/d3a/da6/d147/d52/fc2 [0,4194304] 0 2024-04-16T02:36:05.932 INFO:tasks.workunit.client.0.smithi151.stdout:9/690: creat d1/d3/d6/d17/d7a/fe4 x:0 0 0 2024-04-16T02:36:05.933 INFO:tasks.workunit.client.0.smithi151.stdout:9/691: chown d1/d3/d6/d17/d7a/db1 1 1 2024-04-16T02:36:05.934 INFO:tasks.workunit.client.0.smithi151.stdout:9/692: stat d1/c11 0 2024-04-16T02:36:05.934 INFO:tasks.workunit.client.0.smithi151.stdout:3/606: rename d1/d6/d22/d23/l47 to d1/d6/d4a/lc4 0 2024-04-16T02:36:05.941 INFO:tasks.workunit.client.0.smithi151.stdout:4/731: dwrite d0/d64/f76 [0,4194304] 0 2024-04-16T02:36:05.942 INFO:tasks.workunit.client.0.smithi151.stdout:6/967: link d2/d13/d63/def/df/d23/d45/c134 d2/d13/d3a/da6/d130/c149 0 2024-04-16T02:36:05.943 INFO:tasks.workunit.client.0.smithi151.stdout:4/732: readlink d0/dde/d4a/d60/lab 0 2024-04-16T02:36:05.950 INFO:tasks.workunit.client.0.smithi151.stdout:4/733: getdents d0/dde/d4a/d60 0 2024-04-16T02:36:05.958 INFO:tasks.workunit.client.0.smithi151.stdout:6/968: dwrite d2/d13/d3a/da6/d147/d52/fc2 [0,4194304] 0 2024-04-16T02:36:05.959 INFO:tasks.workunit.client.0.smithi151.stdout:9/693: dwrite d1/d3/d6/d17/d7a/db1/fa1 [0,4194304] 0 2024-04-16T02:36:05.963 INFO:tasks.workunit.client.0.smithi151.stdout:4/734: dwrite d0/d99/f21 [0,4194304] 0 2024-04-16T02:36:05.963 INFO:tasks.workunit.client.0.smithi151.stdout:9/694: chown d1/d3/d6/d17/d54/d9b/f89 56 1 2024-04-16T02:36:05.963 INFO:tasks.workunit.client.0.smithi151.stdout:6/969: fsync d2/d13/d3a/da6/f139 0 2024-04-16T02:36:05.967 INFO:tasks.workunit.client.0.smithi151.stdout:6/970: chown d2/d13/d6f/da3/fcb 170 1 2024-04-16T02:36:05.969 INFO:tasks.workunit.client.0.smithi151.stdout:9/695: symlink d1/d75/le5 0 2024-04-16T02:36:05.971 INFO:tasks.workunit.client.0.smithi151.stdout:0/626: sync 2024-04-16T02:36:05.971 INFO:tasks.workunit.client.0.smithi151.stdout:8/647: sync 2024-04-16T02:36:05.971 INFO:tasks.workunit.client.0.smithi151.stdout:5/866: sync 2024-04-16T02:36:05.971 INFO:tasks.workunit.client.0.smithi151.stdout:4/735: fsync d0/f82 0 2024-04-16T02:36:05.976 INFO:tasks.workunit.client.0.smithi151.stdout:3/607: dread d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/f57 [0,4194304] 0 2024-04-16T02:36:05.976 INFO:tasks.workunit.client.0.smithi151.stdout:8/648: dread - d11/d22/d6c/d84/dc9/fdb zero size 2024-04-16T02:36:05.979 INFO:tasks.workunit.client.0.smithi151.stdout:4/736: dread - d0/d11/f37 zero size 2024-04-16T02:36:05.980 INFO:tasks.workunit.client.0.smithi151.stdout:5/867: creat d2/d6/f116 x:0 0 0 2024-04-16T02:36:05.983 INFO:tasks.workunit.client.0.smithi151.stdout:9/696: unlink d1/d3/d6/d17/d40/d56/f97 0 2024-04-16T02:36:05.990 INFO:tasks.workunit.client.0.smithi151.stdout:3/608: link d1/d6/d22/d23/d30/d44/f5f d1/d6/d20/fc5 0 2024-04-16T02:36:05.991 INFO:tasks.workunit.client.0.smithi151.stdout:4/737: creat d0/d11/d19/d87/d3a/d65/d98/df8/f104 x:0 0 0 2024-04-16T02:36:05.995 INFO:tasks.workunit.client.0.smithi151.stdout:3/609: link d1/d6/d22/d23/d41/c49 d1/d6/d22/d6c/cc6 0 2024-04-16T02:36:05.997 INFO:tasks.workunit.client.0.smithi151.stdout:3/610: fsync d1/d6/d4a/f6f 0 2024-04-16T02:36:06.001 INFO:tasks.workunit.client.0.smithi151.stdout:3/611: fsync d1/d6/d20/d54/f65 0 2024-04-16T02:36:06.004 INFO:tasks.workunit.client.0.smithi151.stdout:3/612: write d1/d6/f1b [8525343,81813] 0 2024-04-16T02:36:06.007 INFO:tasks.workunit.client.0.smithi151.stdout:5/868: stat d2/d6/d19/d30/d88/ddc/da8 0 2024-04-16T02:36:06.009 INFO:tasks.workunit.client.0.smithi151.stdout:9/697: dread d1/d3/d6/d28/d39/f48 [0,4194304] 0 2024-04-16T02:36:06.009 INFO:tasks.workunit.client.0.smithi151.stdout:6/971: dwrite d2/d13/f18 [0,4194304] 0 2024-04-16T02:36:06.011 INFO:tasks.workunit.client.0.smithi151.stdout:5/869: stat d2/d6/d19/d62/dbd 0 2024-04-16T02:36:06.013 INFO:tasks.workunit.client.0.smithi151.stdout:8/649: dread d11/d22/f96 [0,4194304] 0 2024-04-16T02:36:06.014 INFO:tasks.workunit.client.0.smithi151.stdout:0/627: dwrite dc/d15/f17 [0,4194304] 0 2024-04-16T02:36:06.017 INFO:tasks.workunit.client.0.smithi151.stdout:9/698: rmdir d1 39 2024-04-16T02:36:06.017 INFO:tasks.workunit.client.0.smithi151.stdout:5/870: write d2/fb [989809,481] 0 2024-04-16T02:36:06.018 INFO:tasks.workunit.client.0.smithi151.stdout:3/613: symlink d1/d89/daa/lc7 0 2024-04-16T02:36:06.019 INFO:tasks.workunit.client.0.smithi151.stdout:5/871: fdatasync d2/d6/d19/d62/dd0/d9c/db9/f10e 0 2024-04-16T02:36:06.020 INFO:tasks.workunit.client.0.smithi151.stdout:5/872: chown d2/f8e 239309162 1 2024-04-16T02:36:06.020 INFO:tasks.workunit.client.0.smithi151.stdout:8/650: creat d11/d22/d6c/d84/fe7 x:0 0 0 2024-04-16T02:36:06.021 INFO:tasks.workunit.client.0.smithi151.stdout:6/972: fsync d2/f10c 0 2024-04-16T02:36:06.022 INFO:tasks.workunit.client.0.smithi151.stdout:0/628: chown dc/d15/f45 23 1 2024-04-16T02:36:06.023 INFO:tasks.workunit.client.0.smithi151.stdout:0/629: readlink dc/d3c/l41 0 2024-04-16T02:36:06.024 INFO:tasks.workunit.client.0.smithi151.stdout:3/614: creat d1/d6/d22/d23/d30/d44/d46/fc8 x:0 0 0 2024-04-16T02:36:06.026 INFO:tasks.workunit.client.0.smithi151.stdout:9/699: fdatasync d1/d3/d6/d17/d54/d9b/d22/d36/f9e 0 2024-04-16T02:36:06.027 INFO:tasks.workunit.client.0.smithi151.stdout:5/873: creat d2/f117 x:0 0 0 2024-04-16T02:36:06.027 INFO:tasks.workunit.client.0.smithi151.stdout:3/615: chown d1/d6/d22/d23/d30/d44 630999 1 2024-04-16T02:36:06.032 INFO:tasks.workunit.client.0.smithi151.stdout:6/973: creat d2/d13/d63/def/df/d20/f14a x:0 0 0 2024-04-16T02:36:06.034 INFO:tasks.workunit.client.0.smithi151.stdout:5/874: creat d2/d6/d19/d62/dd0/d9c/f118 x:0 0 0 2024-04-16T02:36:06.036 INFO:tasks.workunit.client.0.smithi151.stdout:5/875: write d2/d6/d19/d30/d49/f6e [2468834,109052] 0 2024-04-16T02:36:06.038 INFO:tasks.workunit.client.0.smithi151.stdout:5/876: chown d2/d6/d19/d30/d49/ce5 0 1 2024-04-16T02:36:06.042 INFO:tasks.workunit.client.0.smithi151.stdout:8/651: write d11/d22/d7e/f81 [982879,106529] 0 2024-04-16T02:36:06.043 INFO:tasks.workunit.client.0.smithi151.stdout:5/877: creat d2/d6/df1/f119 x:0 0 0 2024-04-16T02:36:06.043 INFO:tasks.workunit.client.0.smithi151.stdout:0/630: write dc/d15/f45 [50717,125525] 0 2024-04-16T02:36:06.045 INFO:tasks.workunit.client.0.smithi151.stdout:6/974: write d2/d13/d63/def/d2e/f74 [33948,66642] 0 2024-04-16T02:36:06.048 INFO:tasks.workunit.client.0.smithi151.stdout:5/878: mkdir d2/d6/df1/d11a 0 2024-04-16T02:36:06.050 INFO:tasks.workunit.client.0.smithi151.stdout:6/975: mknod d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/c14b 0 2024-04-16T02:36:06.050 INFO:tasks.workunit.client.0.smithi151.stdout:8/652: dread d11/d2a/d35/d52/db3/fb6 [0,4194304] 0 2024-04-16T02:36:06.051 INFO:tasks.workunit.client.0.smithi151.stdout:5/879: readlink d2/d6/d43/lff 0 2024-04-16T02:36:06.056 INFO:tasks.workunit.client.0.smithi151.stdout:6/976: creat d2/d13/d63/def/df/d23/d82/d8d/d22/d6d/f14c x:0 0 0 2024-04-16T02:36:06.056 INFO:tasks.workunit.client.0.smithi151.stdout:9/700: dread d1/d3/d6/d17/d54/f79 [0,4194304] 0 2024-04-16T02:36:06.059 INFO:tasks.workunit.client.0.smithi151.stdout:3/616: dread d1/d6/d22/f4f [0,4194304] 0 2024-04-16T02:36:06.061 INFO:tasks.workunit.client.0.smithi151.stdout:5/880: rename d2/d6/d19/d1a/f41 to d2/d6/d19/d30/f11b 0 2024-04-16T02:36:06.064 INFO:tasks.workunit.client.0.smithi151.stdout:9/701: stat d1/d3/d5a/d8e/ld1 0 2024-04-16T02:36:06.065 INFO:tasks.workunit.client.0.smithi151.stdout:8/653: dwrite d11/d2a/d33/d4e/d62/fd3 [0,4194304] 0 2024-04-16T02:36:06.065 INFO:tasks.workunit.client.0.smithi151.stdout:5/881: fdatasync d2/d6/d19/ff0 0 2024-04-16T02:36:06.066 INFO:tasks.workunit.client.0.smithi151.stdout:3/617: truncate d1/d6/d20/f68 674274 0 2024-04-16T02:36:06.072 INFO:tasks.workunit.client.0.smithi151.stdout:6/977: dwrite d2/d13/d6f/da3/ffd [4194304,4194304] 0 2024-04-16T02:36:06.073 INFO:tasks.workunit.client.0.smithi151.stdout:9/702: write d1/d3/d6/d17/d40/d56/dad/fb5 [3314573,64245] 0 2024-04-16T02:36:06.073 INFO:tasks.workunit.client.0.smithi151.stdout:8/654: write d11/d2a/d35/d52/db3/fcb [127850,124502] 0 2024-04-16T02:36:06.077 INFO:tasks.workunit.client.0.smithi151.stdout:3/618: symlink d1/d37/d87/lc9 0 2024-04-16T02:36:06.079 INFO:tasks.workunit.client.0.smithi151.stdout:9/703: creat d1/d3/d5f/fe6 x:0 0 0 2024-04-16T02:36:06.082 INFO:tasks.workunit.client.0.smithi151.stdout:6/978: creat d2/d13/d3a/da6/d147/d52/f14d x:0 0 0 2024-04-16T02:36:06.082 INFO:tasks.workunit.client.0.smithi151.stdout:3/619: rmdir d1/d6/d4a/d4e 39 2024-04-16T02:36:06.085 INFO:tasks.workunit.client.0.smithi151.stdout:5/882: creat d2/d6/d19/d30/d88/ddc/f11c x:0 0 0 2024-04-16T02:36:06.088 INFO:tasks.workunit.client.0.smithi151.stdout:5/883: truncate d2/d6/d19/d30/d88/ddc/f11c 407618 0 2024-04-16T02:36:06.089 INFO:tasks.workunit.client.0.smithi151.stdout:9/704: mkdir d1/d3/d6/d17/d54/d9b/d22/d2c/d86/de7 0 2024-04-16T02:36:06.092 INFO:tasks.workunit.client.0.smithi151.stdout:3/620: creat d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d/fca x:0 0 0 2024-04-16T02:36:06.099 INFO:tasks.workunit.client.0.smithi151.stdout:9/705: mknod d1/d3/d6/d17/d40/d56/ce8 0 2024-04-16T02:36:06.100 INFO:tasks.workunit.client.0.smithi151.stdout:5/884: truncate d2/d6/d19/d1a/d4c/f53 3705024 0 2024-04-16T02:36:06.100 INFO:tasks.workunit.client.0.smithi151.stdout:8/655: dread d11/f1c [0,4194304] 0 2024-04-16T02:36:06.100 INFO:tasks.workunit.client.0.smithi151.stdout:6/979: link d2/d13/d63/def/f8f d2/d13/d63/def/df/d23/d82/f14e 0 2024-04-16T02:36:06.101 INFO:tasks.workunit.client.0.smithi151.stdout:5/885: symlink d2/d5e/l11d 0 2024-04-16T02:36:06.103 INFO:tasks.workunit.client.0.smithi151.stdout:3/621: creat d1/fcb x:0 0 0 2024-04-16T02:36:06.104 INFO:tasks.workunit.client.0.smithi151.stdout:6/980: fdatasync d2/ffb 0 2024-04-16T02:36:06.105 INFO:tasks.workunit.client.0.smithi151.stdout:9/706: rename d1/d3/d6/d17/d7a/c83 to d1/d3/d6/d28/ce9 0 2024-04-16T02:36:06.108 INFO:tasks.workunit.client.0.smithi151.stdout:6/981: write d2/d13/d3a/da6/d130/d65/fdd [18189,55909] 0 2024-04-16T02:36:06.110 INFO:tasks.workunit.client.0.smithi151.stdout:3/622: write d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/f57 [6116106,26549] 0 2024-04-16T02:36:06.112 INFO:tasks.workunit.client.0.smithi151.stdout:3/623: readlink d1/d6/d22/d23/d30/l94 0 2024-04-16T02:36:06.113 INFO:tasks.workunit.client.0.smithi151.stdout:8/656: rename d11/d2a/d85/cb8 to d11/d22/d2e/dc0/d92/d78/ce8 0 2024-04-16T02:36:06.114 INFO:tasks.workunit.client.0.smithi151.stdout:8/657: chown d11/d22/c47 222777752 1 2024-04-16T02:36:06.116 INFO:tasks.workunit.client.0.smithi151.stdout:9/707: readlink d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/lce 0 2024-04-16T02:36:06.120 INFO:tasks.workunit.client.0.smithi151.stdout:8/658: creat d11/d2a/d35/d64/fe9 x:0 0 0 2024-04-16T02:36:06.121 INFO:tasks.workunit.client.0.smithi151.stdout:8/659: fdatasync d11/d2a/f8c 0 2024-04-16T02:36:06.123 INFO:tasks.workunit.client.0.smithi151.stdout:5/886: write d2/f2d [922605,109478] 0 2024-04-16T02:36:06.127 INFO:tasks.workunit.client.0.smithi151.stdout:8/660: dread d11/d22/d2e/dc0/d92/d78/fb4 [0,4194304] 0 2024-04-16T02:36:06.134 INFO:tasks.workunit.client.0.smithi151.stdout:8/661: symlink d11/d2a/d33/d4e/d62/db7/lea 0 2024-04-16T02:36:06.135 INFO:tasks.workunit.client.0.smithi151.stdout:6/982: dwrite d2/d13/d63/def/df/d20/d4f/d127/de5/fee [0,4194304] 0 2024-04-16T02:36:06.136 INFO:tasks.workunit.client.0.smithi151.stdout:8/662: chown d11/d2a/d83 13303 1 2024-04-16T02:36:06.137 INFO:tasks.workunit.client.0.smithi151.stdout:3/624: link d1/d6/d22/d23/d30/d44/d46/d8a/d3f/lc3 d1/d6/d22/d23/d30/d44/lcc 0 2024-04-16T02:36:06.142 INFO:tasks.workunit.client.0.smithi151.stdout:6/983: chown d2/d3d/f55 1 1 2024-04-16T02:36:06.142 INFO:tasks.workunit.client.0.smithi151.stdout:3/625: chown d1/d6/d22/d23/d66 0 1 2024-04-16T02:36:06.142 INFO:tasks.workunit.client.0.smithi151.stdout:9/708: dwrite d1/d3/d6/d17/d44/f6a [0,4194304] 0 2024-04-16T02:36:06.144 INFO:tasks.workunit.client.0.smithi151.stdout:6/984: fsync d2/d13/d63/def/df/d23/d82/d8d/d22/f4e 0 2024-04-16T02:36:06.145 INFO:tasks.workunit.client.0.smithi151.stdout:9/709: unlink d1/dd/d38/ca2 0 2024-04-16T02:36:06.148 INFO:tasks.workunit.client.0.smithi151.stdout:3/626: creat d1/fcd x:0 0 0 2024-04-16T02:36:06.150 INFO:tasks.workunit.client.0.smithi151.stdout:3/627: write d1/d6/d22/d23/fb9 [91008,21815] 0 2024-04-16T02:36:06.154 INFO:tasks.workunit.client.0.smithi151.stdout:5/887: write d2/d6/d19/d30/fa1 [343322,86030] 0 2024-04-16T02:36:06.155 INFO:tasks.workunit.client.0.smithi151.stdout:6/985: getdents d2/d13/d63/def/df/d20 0 2024-04-16T02:36:06.160 INFO:tasks.workunit.client.0.smithi151.stdout:6/986: rename d2/d13/d63/def/df/d20/d4f/d127/de5/d10e/f111 to d2/d13/d63/def/d2e/dba/d10d/f14f 0 2024-04-16T02:36:06.163 INFO:tasks.workunit.client.0.smithi151.stdout:5/888: dread d2/d6/d19/d30/fa1 [0,4194304] 0 2024-04-16T02:36:06.165 INFO:tasks.workunit.client.0.smithi151.stdout:9/710: write d1/d3/d6/d17/d54/d9b/f89 [1710062,80990] 0 2024-04-16T02:36:06.167 INFO:tasks.workunit.client.0.smithi151.stdout:9/711: chown d1/d3/d6/d17/d54 4623727 1 2024-04-16T02:36:06.167 INFO:tasks.workunit.client.0.smithi151.stdout:3/628: truncate d1/d6/d22/d23/d30/d44/d46/d8a/f28 3819048 0 2024-04-16T02:36:06.168 INFO:tasks.workunit.client.0.smithi151.stdout:8/663: dread d11/d2a/d35/d64/f7f [4194304,4194304] 0 2024-04-16T02:36:06.168 INFO:tasks.workunit.client.0.smithi151.stdout:3/629: chown d1/d6/f8f 1036 1 2024-04-16T02:36:06.169 INFO:tasks.workunit.client.0.smithi151.stdout:9/712: unlink d1/d75/le5 0 2024-04-16T02:36:06.171 INFO:tasks.workunit.client.0.smithi151.stdout:5/889: rename d2/d6/d19/d30/d88/ddc/d83/la5 to d2/d6/d19/l11e 0 2024-04-16T02:36:06.171 INFO:tasks.workunit.client.0.smithi151.stdout:8/664: truncate d11/d2a/d33/f49 2630170 0 2024-04-16T02:36:06.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:05 smithi151 ceph-mon[29003]: pgmap v30: 65 pgs: 65 active+clean; 2.9 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 29 MiB/s rd, 79 MiB/s wr, 242 op/s 2024-04-16T02:36:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:05 smithi151 ceph-mon[29003]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:36:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:05 smithi153 ceph-mon[36747]: pgmap v30: 65 pgs: 65 active+clean; 2.9 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 29 MiB/s rd, 79 MiB/s wr, 242 op/s 2024-04-16T02:36:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:05 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:36:06.275 INFO:tasks.workunit.client.0.smithi151.stdout:5/890: mkdir d2/d6/d19/d30/d88/ddc/d83/d11f 0 2024-04-16T02:36:06.276 INFO:tasks.workunit.client.0.smithi151.stdout:7/830: sync 2024-04-16T02:36:06.276 INFO:tasks.workunit.client.0.smithi151.stdout:1/713: sync 2024-04-16T02:36:06.278 INFO:tasks.workunit.client.0.smithi151.stdout:8/665: write d11/d2a/d83/f9c [8475593,33267] 0 2024-04-16T02:36:06.279 INFO:tasks.workunit.client.0.smithi151.stdout:9/713: link d1/d3/d10/d7b/cac d1/d3/d6/d17/d54/d9b/dae/cea 0 2024-04-16T02:36:06.281 INFO:tasks.workunit.client.0.smithi151.stdout:5/891: read d2/fbb [1356163,89898] 0 2024-04-16T02:36:06.281 INFO:tasks.workunit.client.0.smithi151.stdout:3/630: dwrite d1/d6/d22/d23/d30/d44/d46/d8a/f15 [0,4194304] 0 2024-04-16T02:36:06.282 INFO:tasks.workunit.client.0.smithi151.stdout:1/714: unlink d3/d58/df/d32/d3a/d71/ldf 0 2024-04-16T02:36:06.283 INFO:tasks.workunit.client.0.smithi151.stdout:3/631: chown d1/d6/c9f 29397931 1 2024-04-16T02:36:06.283 INFO:tasks.workunit.client.0.smithi151.stdout:8/666: readlink d11/d2a/la2 0 2024-04-16T02:36:06.284 INFO:tasks.workunit.client.0.smithi151.stdout:1/715: read - d3/d58/df/d32/d3a/df3/d73/fa2 zero size 2024-04-16T02:36:06.284 INFO:tasks.workunit.client.0.smithi151.stdout:3/632: stat d1/d6/d22/d23/d41 0 2024-04-16T02:36:06.285 INFO:tasks.workunit.client.0.smithi151.stdout:1/716: chown d3/d1f/d48 1629 1 2024-04-16T02:36:06.290 INFO:tasks.workunit.client.0.smithi151.stdout:7/831: rename d0/d2c/d38/d5e/dd1/ff1 to d0/d1f/d46/dff/f11d 0 2024-04-16T02:36:06.292 INFO:tasks.workunit.client.0.smithi151.stdout:1/717: creat d3/d58/df/d32/d3a/df3/d52/d97/ff5 x:0 0 0 2024-04-16T02:36:06.293 INFO:tasks.workunit.client.0.smithi151.stdout:1/718: readlink d3/d4/l13 0 2024-04-16T02:36:06.297 INFO:tasks.workunit.client.0.smithi151.stdout:7/832: unlink d0/d2c/d38/d5e/deb/f118 0 2024-04-16T02:36:06.297 INFO:tasks.workunit.client.0.smithi151.stdout:8/667: link d11/d2a/d83/lc6 d11/d2a/d35/d52/db3/leb 0 2024-04-16T02:36:06.299 INFO:tasks.workunit.client.0.smithi151.stdout:3/633: link d1/d6/c91 d1/d37/d87/cce 0 2024-04-16T02:36:06.301 INFO:tasks.workunit.client.0.smithi151.stdout:7/833: creat d0/d1f/d33/d56/f11e x:0 0 0 2024-04-16T02:36:06.302 INFO:tasks.workunit.client.0.smithi151.stdout:8/668: fsync d11/f38 0 2024-04-16T02:36:06.306 INFO:tasks.workunit.client.0.smithi151.stdout:9/714: rename d1/d3/d6/d17/d7a/fc1 to d1/feb 0 2024-04-16T02:36:06.307 INFO:tasks.workunit.client.0.smithi151.stdout:9/715: dread - d1/d3/d6/d17/d7a/fe4 zero size 2024-04-16T02:36:06.308 INFO:tasks.workunit.client.0.smithi151.stdout:3/634: rmdir d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/d6d 39 2024-04-16T02:36:06.316 INFO:tasks.workunit.client.0.smithi151.stdout:1/719: rename d3/d1f/d48 to d3/d33/d10/df6 0 2024-04-16T02:36:06.319 INFO:tasks.workunit.client.0.smithi151.stdout:9/716: symlink d1/dd/d4a/dbb/lec 0 2024-04-16T02:36:06.323 INFO:tasks.workunit.client.0.smithi151.stdout:1/720: dwrite d3/d58/df/d32/d3a/df3/d52/fa9 [0,4194304] 0 2024-04-16T02:36:06.324 INFO:tasks.workunit.client.0.smithi151.stdout:1/721: readlink d3/l70 0 2024-04-16T02:36:06.325 INFO:tasks.workunit.client.0.smithi151.stdout:1/722: fsync d3/d58/df/f9e 0 2024-04-16T02:36:06.326 INFO:tasks.workunit.client.0.smithi151.stdout:5/892: truncate d2/d6/f32 1640745 0 2024-04-16T02:36:06.327 INFO:tasks.workunit.client.0.smithi151.stdout:9/717: fdatasync d1/d3/d6/f1b 0 2024-04-16T02:36:06.333 INFO:tasks.workunit.client.0.smithi151.stdout:5/893: chown d2/d6/dfc/c106 120 1 2024-04-16T02:36:06.335 INFO:tasks.workunit.client.0.smithi151.stdout:7/834: dwrite d0/d1f/d31/d42/f45 [0,4194304] 0 2024-04-16T02:36:06.336 INFO:tasks.workunit.client.0.smithi151.stdout:9/718: chown d1/d3/d6/d17/f31 67177 1 2024-04-16T02:36:06.337 INFO:tasks.workunit.client.0.smithi151.stdout:8/669: getdents d11/d2a/d35/d64 0 2024-04-16T02:36:06.341 INFO:tasks.workunit.client.0.smithi151.stdout:1/723: mknod d3/cf7 0 2024-04-16T02:36:06.345 INFO:tasks.workunit.client.0.smithi151.stdout:3/635: getdents d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43 0 2024-04-16T02:36:06.345 INFO:tasks.workunit.client.0.smithi151.stdout:7/835: unlink d0/f18 0 2024-04-16T02:36:06.348 INFO:tasks.workunit.client.0.smithi151.stdout:5/894: getdents d2/d6/d19/d30/d88/ddc/d83/d11f 0 2024-04-16T02:36:06.349 INFO:tasks.workunit.client.0.smithi151.stdout:3/636: symlink d1/d6/d22/d23/d41/d9d/lcf 0 2024-04-16T02:36:06.354 INFO:tasks.workunit.client.0.smithi151.stdout:1/724: dwrite d3/d58/df/d32/d3a/df3/d52/fa9 [4194304,4194304] 0 2024-04-16T02:36:06.354 INFO:tasks.workunit.client.0.smithi151.stdout:3/637: read d1/d69/fad [754008,11970] 0 2024-04-16T02:36:06.356 INFO:tasks.workunit.client.0.smithi151.stdout:9/719: dwrite d1/d3/d10/d7b/f85 [0,4194304] 0 2024-04-16T02:36:06.356 INFO:tasks.workunit.client.0.smithi151.stdout:7/836: rename d0/d1f to d0/d2c/d38/de7/d11f 0 2024-04-16T02:36:06.356 INFO:tasks.workunit.client.0.smithi151.stdout:3/638: stat d1/d6/d22/d23/d41/f5b 0 2024-04-16T02:36:06.358 INFO:tasks.workunit.client.0.smithi151.stdout:3/639: fsync d1/d6/d22/f52 0 2024-04-16T02:36:06.359 INFO:tasks.workunit.client.0.smithi151.stdout:1/725: mknod d3/d33/d10/cf8 0 2024-04-16T02:36:06.363 INFO:tasks.workunit.client.0.smithi151.stdout:8/670: write d11/d2a/d35/d64/f58 [4622312,67048] 0 2024-04-16T02:36:06.368 INFO:tasks.workunit.client.0.smithi151.stdout:3/640: mkdir d1/d6/d22/d23/d30/d44/d46/d8a/d3f/dd0 0 2024-04-16T02:36:06.370 INFO:tasks.workunit.client.0.smithi151.stdout:5/895: rename d2/d6/d19/d62/c8d to d2/c120 0 2024-04-16T02:36:06.372 INFO:tasks.workunit.client.0.smithi151.stdout:1/726: mkdir d3/d58/df/d32/d3a/df3/d7a/df9 0 2024-04-16T02:36:06.374 INFO:tasks.workunit.client.0.smithi151.stdout:8/671: creat d11/d2a/d33/d4e/d62/db7/fec x:0 0 0 2024-04-16T02:36:06.375 INFO:tasks.workunit.client.0.smithi151.stdout:7/837: write d0/d2c/d38/f9b [267687,106074] 0 2024-04-16T02:36:06.376 INFO:tasks.workunit.client.0.smithi151.stdout:9/720: mkdir d1/d3/d6/d17/d54/d8d/daa/d9d/ded 0 2024-04-16T02:36:06.379 INFO:tasks.workunit.client.0.smithi151.stdout:1/727: creat d3/d58/ffa x:0 0 0 2024-04-16T02:36:06.381 INFO:tasks.workunit.client.0.smithi151.stdout:5/896: dwrite d2/d6/d19/d1a/d4c/fc9 [0,4194304] 0 2024-04-16T02:36:06.381 INFO:tasks.workunit.client.0.smithi151.stdout:9/721: creat d1/d3/d6/d17/d54/d9b/fee x:0 0 0 2024-04-16T02:36:06.382 INFO:tasks.workunit.client.0.smithi151.stdout:7/838: symlink d0/d2c/d38/de7/d11f/d31/l120 0 2024-04-16T02:36:06.383 INFO:tasks.workunit.client.0.smithi151.stdout:9/722: stat d1/dd/d4a/d64 0 2024-04-16T02:36:06.384 INFO:tasks.workunit.client.0.smithi151.stdout:1/728: mknod d3/d58/df/d32/d3a/df3/d52/cfb 0 2024-04-16T02:36:06.387 INFO:tasks.workunit.client.0.smithi151.stdout:7/839: readlink d0/d2c/d38/dab/lc8 0 2024-04-16T02:36:06.388 INFO:tasks.workunit.client.0.smithi151.stdout:7/840: stat d0/d2c/d38/db0/fef 0 2024-04-16T02:36:06.392 INFO:tasks.workunit.client.0.smithi151.stdout:7/841: creat d0/d2c/d38/d5e/f121 x:0 0 0 2024-04-16T02:36:06.393 INFO:tasks.workunit.client.0.smithi151.stdout:9/723: mknod d1/d3/d6/d17/d54/d9b/d22/cef 0 2024-04-16T02:36:06.395 INFO:tasks.workunit.client.0.smithi151.stdout:9/724: read - d1/d3/d6/d17/d7a/fe4 zero size 2024-04-16T02:36:06.395 INFO:tasks.workunit.client.0.smithi151.stdout:7/842: mkdir d0/d2c/d38/d43/d122 0 2024-04-16T02:36:06.396 INFO:tasks.workunit.client.0.smithi151.stdout:1/729: dread d3/d4/d64/d40/fe4 [0,4194304] 0 2024-04-16T02:36:06.397 INFO:tasks.workunit.client.0.smithi151.stdout:9/725: chown d1/dd/d4a/dbb/fc7 1251 1 2024-04-16T02:36:06.397 INFO:tasks.workunit.client.0.smithi151.stdout:8/672: write d11/f28 [9404953,73451] 0 2024-04-16T02:36:06.398 INFO:tasks.workunit.client.0.smithi151.stdout:7/843: dread d0/f22 [0,4194304] 0 2024-04-16T02:36:06.400 INFO:tasks.workunit.client.0.smithi151.stdout:1/730: rmdir d3/d58/df/d32/d3a/df3 39 2024-04-16T02:36:06.401 INFO:tasks.workunit.client.0.smithi151.stdout:7/844: chown d0/d2c/d38/de7/d11f/d31/d42/f6e 11724850 1 2024-04-16T02:36:06.402 INFO:tasks.workunit.client.0.smithi151.stdout:1/731: chown d3/d58/df/d38/f74 3626 1 2024-04-16T02:36:06.402 INFO:tasks.workunit.client.0.smithi151.stdout:9/726: unlink d1/d3/d6/l3e 0 2024-04-16T02:36:06.404 INFO:tasks.workunit.client.0.smithi151.stdout:7/845: mkdir d0/d2c/d123 0 2024-04-16T02:36:06.406 INFO:tasks.workunit.client.0.smithi151.stdout:1/732: mkdir d3/d33/d10/d4f/dfc 0 2024-04-16T02:36:06.408 INFO:tasks.workunit.client.0.smithi151.stdout:9/727: unlink d1/d3/d6/d17/d44/f6a 0 2024-04-16T02:36:06.408 INFO:tasks.workunit.client.0.smithi151.stdout:8/673: dwrite d11/d2a/d35/d52/db3/dcd/fd4 [0,4194304] 0 2024-04-16T02:36:06.412 INFO:tasks.workunit.client.0.smithi151.stdout:7/846: symlink d0/ddf/l124 0 2024-04-16T02:36:06.413 INFO:tasks.workunit.client.0.smithi151.stdout:8/674: creat d11/d22/d2e/dc0/d92/d78/fed x:0 0 0 2024-04-16T02:36:06.413 INFO:tasks.workunit.client.0.smithi151.stdout:9/728: dread - d1/d3/d6/d17/d54/d8d/f90 zero size 2024-04-16T02:36:06.415 INFO:tasks.workunit.client.0.smithi151.stdout:5/897: dread d2/f1f [0,4194304] 0 2024-04-16T02:36:06.415 INFO:tasks.workunit.client.0.smithi151.stdout:9/729: stat d1/d75/fc9 0 2024-04-16T02:36:06.416 INFO:tasks.workunit.client.0.smithi151.stdout:1/733: mkdir d3/d58/df/d32/d3a/df3/d7a/dc7/dfd 0 2024-04-16T02:36:06.417 INFO:tasks.workunit.client.0.smithi151.stdout:5/898: write d2/d6/d19/d30/d49/f6e [172015,7701] 0 2024-04-16T02:36:06.418 INFO:tasks.workunit.client.0.smithi151.stdout:7/847: creat d0/d2c/d38/de7/d11f/da8/f125 x:0 0 0 2024-04-16T02:36:06.419 INFO:tasks.workunit.client.0.smithi151.stdout:8/675: symlink d11/d2a/d35/d52/db3/dcc/lee 0 2024-04-16T02:36:06.419 INFO:tasks.workunit.client.0.smithi151.stdout:7/848: stat d0/fbc 0 2024-04-16T02:36:06.421 INFO:tasks.workunit.client.0.smithi151.stdout:5/899: chown d2/d6/l15 30 1 2024-04-16T02:36:06.423 INFO:tasks.workunit.client.0.smithi151.stdout:9/730: creat d1/d3/d6/d17/d54/d8d/daa/ff0 x:0 0 0 2024-04-16T02:36:06.427 INFO:tasks.workunit.client.0.smithi151.stdout:5/900: rmdir d2/d6/d19/d30/d88/ddc/d83/d8b 39 2024-04-16T02:36:06.429 INFO:tasks.workunit.client.0.smithi151.stdout:9/731: creat d1/d3/d6/d17/d44/ff1 x:0 0 0 2024-04-16T02:36:06.430 INFO:tasks.workunit.client.0.smithi151.stdout:8/676: dwrite d11/d2a/fd5 [0,4194304] 0 2024-04-16T02:36:06.433 INFO:tasks.workunit.client.0.smithi151.stdout:9/732: symlink d1/d3/d5a/lf2 0 2024-04-16T02:36:06.435 INFO:tasks.workunit.client.0.smithi151.stdout:8/677: rmdir d11/d22/d6c/d84/dc9 39 2024-04-16T02:36:06.438 INFO:tasks.workunit.client.0.smithi151.stdout:9/733: symlink d1/d3/d10/d7b/lf3 0 2024-04-16T02:36:06.444 INFO:tasks.workunit.client.0.smithi151.stdout:8/678: getdents d11/d2a/d85 0 2024-04-16T02:36:06.450 INFO:tasks.workunit.client.0.smithi151.stdout:9/734: dwrite d1/d3/d6/d17/d54/dd5/fd8 [0,4194304] 0 2024-04-16T02:36:06.455 INFO:tasks.workunit.client.0.smithi151.stdout:1/734: dwrite d3/d58/df/d32/d3a/df3/d7a/f86 [0,4194304] 0 2024-04-16T02:36:06.456 INFO:tasks.workunit.client.0.smithi151.stdout:9/735: symlink d1/dd/lf4 0 2024-04-16T02:36:06.456 INFO:tasks.workunit.client.0.smithi151.stdout:5/901: dread d2/d6/d19/d30/d49/d7f/f8a [0,4194304] 0 2024-04-16T02:36:06.457 INFO:tasks.workunit.client.0.smithi151.stdout:5/902: chown d2/f1f 38664574 1 2024-04-16T02:36:06.462 INFO:tasks.workunit.client.0.smithi151.stdout:9/736: mknod d1/d3/d6/d17/cf5 0 2024-04-16T02:36:06.465 INFO:tasks.workunit.client.0.smithi151.stdout:1/735: dwrite d3/d33/d10/fe9 [0,4194304] 0 2024-04-16T02:36:06.489 INFO:tasks.workunit.client.0.smithi151.stdout:5/903: creat d2/d6/d19/d62/f121 x:0 0 0 2024-04-16T02:36:06.493 INFO:tasks.workunit.client.0.smithi151.stdout:9/737: mknod d1/dd/d4a/dbb/cf6 0 2024-04-16T02:36:06.494 INFO:tasks.workunit.client.0.smithi151.stdout:5/904: unlink d2/le4 0 2024-04-16T02:36:06.498 INFO:tasks.workunit.client.0.smithi151.stdout:5/905: unlink d2/d6/d19/d30/d88/ddc/cc8 0 2024-04-16T02:36:06.500 INFO:tasks.workunit.client.0.smithi151.stdout:1/736: getdents d3/d58/df/d32/d3a/df3/d52/d97 0 2024-04-16T02:36:06.500 INFO:tasks.workunit.client.0.smithi151.stdout:5/906: truncate d2/d6/d43/fed 216490 0 2024-04-16T02:36:06.505 INFO:tasks.workunit.client.0.smithi151.stdout:5/907: symlink d2/d6/d19/d30/d88/ddc/d83/d8b/l122 0 2024-04-16T02:36:06.509 INFO:tasks.workunit.client.0.smithi151.stdout:5/908: write d2/d6/d19/d30/d49/d7f/fec [820525,46542] 0 2024-04-16T02:36:06.514 INFO:tasks.workunit.client.0.smithi151.stdout:5/909: symlink d2/d6/d19/d62/dd0/df6/l123 0 2024-04-16T02:36:06.515 INFO:tasks.workunit.client.0.smithi151.stdout:9/738: truncate d1/d3/d6/d17/d40/d56/dad/fb5 2510021 0 2024-04-16T02:36:06.523 INFO:tasks.workunit.client.0.smithi151.stdout:5/910: dwrite d2/d6/d19/d62/dd0/d9c/f118 [0,4194304] 0 2024-04-16T02:36:06.532 INFO:tasks.workunit.client.0.smithi151.stdout:5/911: dwrite d2/f24 [4194304,4194304] 0 2024-04-16T02:36:06.560 INFO:tasks.workunit.client.0.smithi151.stdout:5/912: dwrite d2/d6/d19/d1a/d4c/ff5 [0,4194304] 0 2024-04-16T02:36:06.562 INFO:tasks.workunit.client.0.smithi151.stdout:5/913: chown d2/d6/d19/d30/d49/d7f/fec 26432976 1 2024-04-16T02:36:06.566 INFO:tasks.workunit.client.0.smithi151.stdout:5/914: mknod d2/d6/d19/d30/d113/c124 0 2024-04-16T02:36:06.569 INFO:tasks.workunit.client.0.smithi151.stdout:5/915: symlink d2/d6/df1/d11a/l125 0 2024-04-16T02:36:06.570 INFO:tasks.workunit.client.0.smithi151.stdout:5/916: chown d2/d6/d19/d62/dd0/df6 92 1 2024-04-16T02:36:06.573 INFO:tasks.workunit.client.0.smithi151.stdout:5/917: truncate d2/fbb 4512119 0 2024-04-16T02:36:06.586 INFO:tasks.workunit.client.0.smithi151.stdout:5/918: truncate d2/d6/f1c 765366 0 2024-04-16T02:36:06.589 INFO:tasks.workunit.client.0.smithi151.stdout:5/919: mknod d2/d6/dfc/c126 0 2024-04-16T02:36:06.591 INFO:tasks.workunit.client.0.smithi151.stdout:5/920: symlink d2/d6/d19/d30/l127 0 2024-04-16T02:36:06.594 INFO:tasks.workunit.client.0.smithi151.stdout:5/921: symlink d2/d6/d19/d30/d88/ddc/da8/dd3/l128 0 2024-04-16T02:36:06.595 INFO:tasks.workunit.client.0.smithi151.stdout:5/922: readlink d2/d6/d19/d30/d88/ddc/d83/d8b/l98 0 2024-04-16T02:36:06.607 INFO:tasks.workunit.client.0.smithi151.stdout:5/923: write d2/d6/d19/d62/dd0/fd2 [56226,22888] 0 2024-04-16T02:36:06.612 INFO:tasks.workunit.client.0.smithi151.stdout:5/924: dread d2/d6/d19/d1a/f79 [0,4194304] 0 2024-04-16T02:36:06.617 INFO:tasks.workunit.client.0.smithi151.stdout:5/925: dread d2/d6/d19/d62/dd0/d9c/f118 [0,4194304] 0 2024-04-16T02:36:06.617 INFO:tasks.workunit.client.0.smithi151.stdout:5/926: chown d2/d6/dfc 1 1 2024-04-16T02:36:06.620 INFO:tasks.workunit.client.0.smithi151.stdout:5/927: mknod d2/d6/d19/d1a/c129 0 2024-04-16T02:36:07.097 INFO:tasks.workunit.client.0.smithi151.stdout:4/738: sync 2024-04-16T02:36:07.103 INFO:tasks.workunit.client.0.smithi151.stdout:4/739: dread d0/d99/d59/f7f [0,4194304] 0 2024-04-16T02:36:07.104 INFO:tasks.workunit.client.0.smithi151.stdout:4/740: stat d0/dde/f47 0 2024-04-16T02:36:07.106 INFO:tasks.workunit.client.0.smithi151.stdout:4/741: symlink d0/d11/l105 0 2024-04-16T02:36:07.107 INFO:tasks.workunit.client.0.smithi151.stdout:4/742: read - d0/d11/d19/f9f zero size 2024-04-16T02:36:07.109 INFO:tasks.workunit.client.0.smithi151.stdout:4/743: rmdir d0/d11/d19/d87/d3a/d65/d98 39 2024-04-16T02:36:07.123 INFO:tasks.workunit.client.0.smithi151.stdout:4/744: dread d0/f4c [0,4194304] 0 2024-04-16T02:36:07.146 INFO:tasks.workunit.client.0.smithi151.stdout:4/745: dwrite d0/d99/f4f [0,4194304] 0 2024-04-16T02:36:07.149 INFO:tasks.workunit.client.0.smithi151.stdout:6/987: sync 2024-04-16T02:36:07.149 INFO:tasks.workunit.client.0.smithi151.stdout:3/641: sync 2024-04-16T02:36:07.149 INFO:tasks.workunit.client.0.smithi151.stdout:7/849: sync 2024-04-16T02:36:07.149 INFO:tasks.workunit.client.0.smithi151.stdout:0/631: sync 2024-04-16T02:36:07.150 INFO:tasks.workunit.client.0.smithi151.stdout:0/632: readlink dc/l79 0 2024-04-16T02:36:07.152 INFO:tasks.workunit.client.0.smithi151.stdout:4/746: rename d0/dde/d45 to d0/dde/dad/d106 0 2024-04-16T02:36:07.155 INFO:tasks.workunit.client.0.smithi151.stdout:7/850: creat d0/d2c/d38/dab/f126 x:0 0 0 2024-04-16T02:36:07.161 INFO:tasks.workunit.client.0.smithi151.stdout:6/988: dwrite d2/d13/d63/def/df/d23/d82/d8d/d22/f68 [4194304,4194304] 0 2024-04-16T02:36:07.161 INFO:tasks.workunit.client.0.smithi151.stdout:7/851: creat d0/d2c/d38/de7/d11f/d60/f127 x:0 0 0 2024-04-16T02:36:07.163 INFO:tasks.workunit.client.0.smithi151.stdout:4/747: mkdir d0/dde/d80/d107 0 2024-04-16T02:36:07.166 INFO:tasks.workunit.client.0.smithi151.stdout:4/748: creat d0/d99/d59/f108 x:0 0 0 2024-04-16T02:36:07.169 INFO:tasks.workunit.client.0.smithi151.stdout:7/852: getdents d0/ddf 0 2024-04-16T02:36:07.170 INFO:tasks.workunit.client.0.smithi151.stdout:7/853: dread - d0/d2c/d38/db0/f112 zero size 2024-04-16T02:36:07.172 INFO:tasks.workunit.client.0.smithi151.stdout:7/854: read d0/d27/f9f [603091,32885] 0 2024-04-16T02:36:07.174 INFO:tasks.workunit.client.0.smithi151.stdout:6/989: rmdir d2/d13/d63/def/df/d20/d4f/d127/de5/d10e 0 2024-04-16T02:36:07.176 INFO:tasks.workunit.client.0.smithi151.stdout:4/749: symlink d0/l109 0 2024-04-16T02:36:07.178 INFO:tasks.workunit.client.0.smithi151.stdout:7/855: truncate d0/d2c/d38/db0/fef 979372 0 2024-04-16T02:36:07.179 INFO:tasks.workunit.client.0.smithi151.stdout:4/750: fdatasync d0/dde/dad/d106/f9a 0 2024-04-16T02:36:07.184 INFO:tasks.workunit.client.0.smithi151.stdout:3/642: dwrite d1/d6/d22/d23/d30/d44/d46/d8a/d3f/fb0 [0,4194304] 0 2024-04-16T02:36:07.188 INFO:tasks.workunit.client.0.smithi151.stdout:0/633: dread dc/d15/d34/d67/d69/f8b [0,4194304] 0 2024-04-16T02:36:07.189 INFO:tasks.workunit.client.0.smithi151.stdout:3/643: rmdir d1/d6/d22/d23 39 2024-04-16T02:36:07.196 INFO:tasks.workunit.client.0.smithi151.stdout:0/634: mknod dc/d15/d34/ccb 0 2024-04-16T02:36:07.198 INFO:tasks.workunit.client.0.smithi151.stdout:3/644: dwrite d1/d6/f1b [8388608,4194304] 0 2024-04-16T02:36:07.198 INFO:tasks.workunit.client.0.smithi151.stdout:6/990: truncate d2/d13/d3a/da6/d130/d73/fed 153431 0 2024-04-16T02:36:07.199 INFO:tasks.workunit.client.0.smithi151.stdout:3/645: readlink d1/lb 0 2024-04-16T02:36:07.199 INFO:tasks.workunit.client.0.smithi151.stdout:7/856: write d0/d2c/d38/d5e/faa [1984493,34733] 0 2024-04-16T02:36:07.200 INFO:tasks.workunit.client.0.smithi151.stdout:7/857: readlink d0/l20 0 2024-04-16T02:36:07.203 INFO:tasks.workunit.client.0.smithi151.stdout:7/858: dread - d0/d2c/d38/de7/d11f/d4c/f104 zero size 2024-04-16T02:36:07.204 INFO:tasks.workunit.client.0.smithi151.stdout:6/991: creat d2/d13/d3a/da6/d147/d52/f150 x:0 0 0 2024-04-16T02:36:07.205 INFO:tasks.workunit.client.0.smithi151.stdout:7/859: write d0/f3 [5454698,85786] 0 2024-04-16T02:36:07.206 INFO:tasks.workunit.client.0.smithi151.stdout:7/860: read - d0/d2c/d38/dab/f126 zero size 2024-04-16T02:36:07.207 INFO:tasks.workunit.client.0.smithi151.stdout:3/646: read d1/d6/d22/d23/d30/d44/d46/d8a/f16 [3498904,87662] 0 2024-04-16T02:36:07.208 INFO:tasks.workunit.client.0.smithi151.stdout:7/861: write d0/fe0 [934209,120703] 0 2024-04-16T02:36:07.208 INFO:tasks.workunit.client.0.smithi151.stdout:3/647: chown d1/d6/d22/d23/d30/d44/d46 6085 1 2024-04-16T02:36:07.210 INFO:tasks.workunit.client.0.smithi151.stdout:4/751: dread d0/d11/d19/f25 [0,4194304] 0 2024-04-16T02:36:07.211 INFO:tasks.workunit.client.0.smithi151.stdout:7/862: mknod d0/d2c/d123/c128 0 2024-04-16T02:36:07.212 INFO:tasks.workunit.client.0.smithi151.stdout:3/648: mkdir d1/d6/d22/d23/d30/d44/d46/d8a/d3f/d43/dd1 0 2024-04-16T02:36:07.214 INFO:tasks.workunit.client.0.smithi151.stdout:4/752: chown d0/d11/d19/d87/d3a/d65/ccb 17749 1 2024-04-16T02:36:07.214 INFO:tasks.workunit.client.0.smithi151.stdout:7/863: symlink d0/d2c/d38/de7/d11f/d99/dc0/l129 0 2024-04-16T02:36:07.217 INFO:tasks.workunit.client.0.smithi151.stdout:7/864: rename d0/d2c/d38/de7/d11f/d46/dff to d0/d2c/d38/de7/d11f/d46/dff/d12a 22 2024-04-16T02:36:07.220 INFO:tasks.workunit.client.0.smithi151.stdout:4/753: getdents d0/dde/d80/da0 0 2024-04-16T02:36:07.223 INFO:tasks.workunit.client.0.smithi151.stdout:4/754: chown d0/d11/d19/d87/d3a/d61/ca1 21 1 2024-04-16T02:36:07.227 INFO:tasks.workunit.client.0.smithi151.stdout:4/755: creat d0/d11/f10a x:0 0 0 2024-04-16T02:36:07.229 INFO:tasks.workunit.client.0.smithi151.stdout:4/756: mknod d0/dde/dad/dc8/c10b 0 2024-04-16T02:36:07.232 INFO:tasks.workunit.client.0.smithi151.stdout:4/757: rmdir d0/dde/d4a/dba 39 2024-04-16T02:36:07.232 INFO:tasks.workunit.client.0.smithi151.stdout:3/649: dwrite d1/d6/f21 [0,4194304] 0 2024-04-16T02:36:07.234 INFO:tasks.workunit.client.0.smithi151.stdout:4/758: truncate d0/dde/dad/d100/f101 1048410 0 2024-04-16T02:36:07.235 INFO:tasks.workunit.client.0.smithi151.stdout:3/650: rmdir d1/d37 39 2024-04-16T02:36:07.237 INFO:tasks.workunit.client.0.smithi151.stdout:3/651: mknod d1/d69/cd2 0 2024-04-16T02:36:07.240 INFO:tasks.workunit.client.0.smithi151.stdout:3/652: mkdir d1/d6/d20/d3d/da6/dd3 0 2024-04-16T02:36:07.251 INFO:tasks.workunit.client.0.smithi151.stdout:3/653: write d1/f26 [2331352,14862] 0 2024-04-16T02:36:07.253 INFO:tasks.workunit.client.0.smithi151.stdout:3/654: dread - d1/d6/d22/d23/d41/f8d zero size 2024-04-16T02:36:07.255 INFO:tasks.workunit.client.0.smithi151.stdout:3/655: mkdir d1/d6/d20/d3d/dd4 0 2024-04-16T02:36:07.259 INFO:tasks.workunit.client.0.smithi151.stdout:4/759: dread d0/dde/dad/d106/f9a [0,4194304] 0 2024-04-16T02:36:07.262 INFO:tasks.workunit.client.0.smithi151.stdout:4/760: rename d0/d99/d59/f7f to d0/dde/d80/d107/f10c 0 2024-04-16T02:36:07.263 INFO:tasks.workunit.client.0.smithi151.stdout:3/656: write d1/d6/d22/d23/d30/d44/d46/d8a/f28 [2424446,22480] 0 2024-04-16T02:36:07.265 INFO:tasks.workunit.client.0.smithi151.stdout:3/657: mknod d1/d6/d20/cd5 0 2024-04-16T02:36:07.267 INFO:tasks.workunit.client.0.smithi151.stdout:4/761: creat d0/dde/d80/f10d x:0 0 0 2024-04-16T02:36:07.269 INFO:tasks.workunit.client.0.smithi151.stdout:4/762: symlink d0/d11/d19/d87/d3a/l10e 0 2024-04-16T02:36:07.270 INFO:tasks.workunit.client.0.smithi151.stdout:4/763: write d0/dde/dad/d106/dd2/ff9 [592484,69709] 0 2024-04-16T02:36:07.273 INFO:tasks.workunit.client.0.smithi151.stdout:4/764: unlink d0/dde/d80/f9d 0 2024-04-16T02:36:07.280 INFO:tasks.workunit.client.0.smithi151.stdout:4/765: write d0/f3c [1535933,87052] 0 2024-04-16T02:36:07.285 INFO:tasks.workunit.client.0.smithi151.stdout:4/766: symlink d0/d11/d19/d87/d3a/d65/d98/df8/l10f 0 2024-04-16T02:36:07.288 INFO:tasks.workunit.client.0.smithi151.stdout:4/767: rmdir d0/dde/d80 39 2024-04-16T02:36:07.288 INFO:tasks.workunit.client.0.smithi151.stdout:4/768: stat d0/dde/dad 0 2024-04-16T02:36:07.292 INFO:tasks.workunit.client.0.smithi151.stdout:4/769: write d0/f82 [4315051,4848] 0 2024-04-16T02:36:07.308 INFO:tasks.workunit.client.0.smithi151.stdout:4/770: dread d0/f15 [0,4194304] 0 2024-04-16T02:36:07.309 INFO:tasks.workunit.client.0.smithi151.stdout:4/771: read d0/d99/f21 [3411223,34246] 0 2024-04-16T02:36:07.312 INFO:tasks.workunit.client.0.smithi151.stdout:4/772: write d0/d11/d19/d87/d3a/d50/fb0 [258306,58997] 0 2024-04-16T02:36:07.314 INFO:tasks.workunit.client.0.smithi151.stdout:4/773: symlink d0/d64/l110 0 2024-04-16T02:36:07.315 INFO:tasks.workunit.client.0.smithi151.stdout:4/774: write d0/fb [9003371,40777] 0 2024-04-16T02:36:07.316 INFO:tasks.workunit.client.0.smithi151.stdout:4/775: chown d0/d99 6529 1 2024-04-16T02:36:07.333 INFO:tasks.workunit.client.0.smithi151.stdout:4/776: dread d0/d11/d19/d46/dcd/def/dfe/f103 [0,4194304] 0 2024-04-16T02:36:07.363 INFO:tasks.workunit.client.0.smithi151.stdout:9/739: sync 2024-04-16T02:36:07.364 INFO:tasks.workunit.client.0.smithi151.stdout:5/928: sync 2024-04-16T02:36:07.364 INFO:tasks.workunit.client.0.smithi151.stdout:1/737: sync 2024-04-16T02:36:07.364 INFO:tasks.workunit.client.0.smithi151.stdout:8/679: sync 2024-04-16T02:36:07.366 INFO:tasks.workunit.client.0.smithi151.stdout:9/740: symlink d1/d3/d6/d17/d40/d56/lf7 0 2024-04-16T02:36:07.367 INFO:tasks.workunit.client.0.smithi151.stdout:8/680: creat d11/d2a/d35/d64/fef x:0 0 0 2024-04-16T02:36:07.368 INFO:tasks.workunit.client.0.smithi151.stdout:1/738: truncate d3/d33/d10/f55 3188631 0 2024-04-16T02:36:07.373 INFO:tasks.workunit.client.0.smithi151.stdout:9/741: write d1/d3/d6/d17/d40/d7d/da0/fdd [707320,61424] 0 2024-04-16T02:36:07.373 INFO:tasks.workunit.client.0.smithi151.stdout:8/681: chown d11/d2a/d33/d4e/d62/d6b 4070 1 2024-04-16T02:36:07.376 INFO:tasks.workunit.client.0.smithi151.stdout:5/929: dwrite d2/d6/d19/f1d [0,4194304] 0 2024-04-16T02:36:07.377 INFO:tasks.workunit.client.0.smithi151.stdout:9/742: dread - d1/d3/d6/d17/d54/d8d/f90 zero size 2024-04-16T02:36:07.385 INFO:tasks.workunit.client.0.smithi151.stdout:1/739: dwrite d3/d33/d10/d4f/fb0 [0,4194304] 0 2024-04-16T02:36:07.388 INFO:tasks.workunit.client.0.smithi151.stdout:8/682: write d11/f15 [2974166,81271] 0 2024-04-16T02:36:07.390 INFO:tasks.workunit.client.0.smithi151.stdout:5/930: dwrite d2/d6/d19/ff0 [0,4194304] 0 2024-04-16T02:36:07.394 INFO:tasks.workunit.client.0.smithi151.stdout:1/740: symlink d3/d58/df/lfe 0 2024-04-16T02:36:07.402 INFO:tasks.workunit.client.0.smithi151.stdout:1/741: rename d3/d33/d10/df6/fe8 to d3/d1f/dd7/fff 0 2024-04-16T02:36:07.406 INFO:tasks.workunit.client.0.smithi151.stdout:1/742: creat d3/d4/d64/d40/f100 x:0 0 0 2024-04-16T02:36:07.408 INFO:tasks.workunit.client.0.smithi151.stdout:1/743: stat d3/d58/df/d32/d3a/df3/d52/lab 0 2024-04-16T02:36:07.411 INFO:tasks.workunit.client.0.smithi151.stdout:9/743: dread d1/dd/f8a [0,4194304] 0 2024-04-16T02:36:07.414 INFO:tasks.workunit.client.0.smithi151.stdout:5/931: write d2/d6/d19/d30/fb0 [1486269,33391] 0 2024-04-16T02:36:07.416 INFO:tasks.workunit.client.0.smithi151.stdout:5/932: read d2/f2d [450149,122771] 0 2024-04-16T02:36:07.417 INFO:tasks.workunit.client.0.smithi151.stdout:9/744: rename d1/d3/d6/d17/d54/d8d/daa/d9d/ded to d1/dd4/df8 0 2024-04-16T02:36:07.417 INFO:tasks.workunit.client.0.smithi151.stdout:8/683: write d11/f16 [2969161,24393] 0 2024-04-16T02:36:07.417 INFO:tasks.workunit.client.0.smithi151.stdout:1/744: link d3/d4/faa d3/d58/df/d32/d3a/df3/d7a/df9/f101 0 2024-04-16T02:36:07.420 INFO:tasks.workunit.client.0.smithi151.stdout:5/933: symlink d2/d6/d19/d30/d88/dc4/l12a 0 2024-04-16T02:36:07.425 INFO:tasks.workunit.client.0.smithi151.stdout:9/745: creat d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/dc3/ff9 x:0 0 0 2024-04-16T02:36:07.426 INFO:tasks.workunit.client.0.smithi151.stdout:9/746: dread - d1/d3/d6/d17/d54/d8d/f90 zero size 2024-04-16T02:36:07.429 INFO:tasks.workunit.client.0.smithi151.stdout:5/934: mkdir d2/d6/d19/d30/d88/ddc/d83/d12b 0 2024-04-16T02:36:07.431 INFO:tasks.workunit.client.0.smithi151.stdout:1/745: link d3/d58/df/d32/dbd/lf4 d3/d58/df/d32/d3a/d67/l102 0 2024-04-16T02:36:07.432 INFO:tasks.workunit.client.0.smithi151.stdout:1/746: chown d3/d58/df/d68/f7c 4475 1 2024-04-16T02:36:07.434 INFO:tasks.workunit.client.0.smithi151.stdout:1/747: read d3/d33/d10/d4f/fe2 [243985,46470] 0 2024-04-16T02:36:07.434 INFO:tasks.workunit.client.0.smithi151.stdout:1/748: dread - d3/fd6 zero size 2024-04-16T02:36:07.442 INFO:tasks.workunit.client.0.smithi151.stdout:1/749: dwrite d3/d58/ffa [0,4194304] 0 2024-04-16T02:36:07.445 INFO:tasks.workunit.client.0.smithi151.stdout:8/684: write d11/d22/d2e/f3d [1091425,24127] 0 2024-04-16T02:36:07.447 INFO:tasks.workunit.client.0.smithi151.stdout:5/935: truncate d2/fb 3614801 0 2024-04-16T02:36:07.450 INFO:tasks.workunit.client.0.smithi151.stdout:8/685: creat d11/d2a/d33/ff0 x:0 0 0 2024-04-16T02:36:07.456 INFO:tasks.workunit.client.0.smithi151.stdout:1/750: dwrite d3/d58/df/d32/d3a/df3/d52/f72 [0,4194304] 0 2024-04-16T02:36:07.457 INFO:tasks.workunit.client.0.smithi151.stdout:9/747: dwrite d1/d3/d5a/f82 [0,4194304] 0 2024-04-16T02:36:07.458 INFO:tasks.workunit.client.0.smithi151.stdout:1/751: write d3/d33/d10/fe9 [2482389,112820] 0 2024-04-16T02:36:07.459 INFO:tasks.workunit.client.0.smithi151.stdout:8/686: unlink d11/d2a/d33/c5d 0 2024-04-16T02:36:07.461 INFO:tasks.workunit.client.0.smithi151.stdout:8/687: chown d11/d2a/d33/d4e/d62/d6b/c8d 3333688 1 2024-04-16T02:36:07.466 INFO:tasks.workunit.client.0.smithi151.stdout:9/748: mkdir d1/d3/d5f/dfa 0 2024-04-16T02:36:07.466 INFO:tasks.workunit.client.0.smithi151.stdout:1/752: creat d3/d58/df/d32/d3a/d71/ded/f103 x:0 0 0 2024-04-16T02:36:07.467 INFO:tasks.workunit.client.0.smithi151.stdout:8/688: dread - d11/d22/fa1 zero size 2024-04-16T02:36:07.471 INFO:tasks.workunit.client.0.smithi151.stdout:8/689: symlink d11/d2a/d33/d4e/d62/dc8/lf1 0 2024-04-16T02:36:07.472 INFO:tasks.workunit.client.0.smithi151.stdout:9/749: mkdir d1/d3/d6/d17/d40/d7d/dfb 0 2024-04-16T02:36:07.476 INFO:tasks.workunit.client.0.smithi151.stdout:8/690: rename d11/d2a/d85/f99 to d11/d22/d2e/dc0/ff2 0 2024-04-16T02:36:07.477 INFO:tasks.workunit.client.0.smithi151.stdout:8/691: readlink d11/l25 0 2024-04-16T02:36:07.478 INFO:tasks.workunit.client.0.smithi151.stdout:8/692: chown d11/l21 12 1 2024-04-16T02:36:07.478 INFO:tasks.workunit.client.0.smithi151.stdout:9/750: truncate d1/d3/d6/d17/d54/d58/fa7 152880 0 2024-04-16T02:36:07.479 INFO:tasks.workunit.client.0.smithi151.stdout:8/693: dread - d11/d22/d6c/d84/fc4 zero size 2024-04-16T02:36:07.480 INFO:tasks.workunit.client.0.smithi151.stdout:8/694: write d11/d22/d2e/dc0/fe6 [592341,11292] 0 2024-04-16T02:36:07.485 INFO:tasks.workunit.client.0.smithi151.stdout:8/695: rename d11/f1c to d11/d2a/d35/d6e/ff3 0 2024-04-16T02:36:07.490 INFO:tasks.workunit.client.0.smithi151.stdout:8/696: chown d11/d22/d6c/d84/dc9/ce3 7087047 1 2024-04-16T02:36:07.491 INFO:tasks.workunit.client.0.smithi151.stdout:5/936: dread d2/d6/d19/d30/d88/dc4/fc5 [0,4194304] 0 2024-04-16T02:36:07.493 INFO:tasks.workunit.client.0.smithi151.stdout:9/751: getdents d1/d3/d5a/d8e 0 2024-04-16T02:36:07.501 INFO:tasks.workunit.client.0.smithi151.stdout:8/697: rmdir d11/d2a/d33/d4e/d62/db7 39 2024-04-16T02:36:07.502 INFO:tasks.workunit.client.0.smithi151.stdout:1/753: dwrite d3/d58/df/fc9 [0,4194304] 0 2024-04-16T02:36:07.506 INFO:tasks.workunit.client.0.smithi151.stdout:9/752: creat d1/d3/d6/d17/d40/d56/ffc x:0 0 0 2024-04-16T02:36:07.506 INFO:tasks.workunit.client.0.smithi151.stdout:9/753: chown d1/d3/d6/d17/d54/d9b/d22 365363 1 2024-04-16T02:36:07.522 INFO:tasks.workunit.client.0.smithi151.stdout:8/698: rename d11/d22/d2e/dc0/d92 to d11/d2a/df4 0 2024-04-16T02:36:07.522 INFO:tasks.workunit.client.0.smithi151.stdout:9/754: creat d1/d3/d6/d17/d54/d9b/d22/ffd x:0 0 0 2024-04-16T02:36:07.522 INFO:tasks.workunit.client.0.smithi151.stdout:9/755: chown d1/d3/d5a 3 1 2024-04-16T02:36:07.523 INFO:tasks.workunit.client.0.smithi151.stdout:8/699: fsync d11/d22/d6c/d84/fe7 0 2024-04-16T02:36:07.523 INFO:tasks.workunit.client.0.smithi151.stdout:1/754: getdents d3/d4/d64 0 2024-04-16T02:36:07.526 INFO:tasks.workunit.client.0.smithi151.stdout:8/700: chown d11/d2a/d35/d52/c7a 2740102 1 2024-04-16T02:36:07.529 INFO:tasks.workunit.client.0.smithi151.stdout:9/756: read d1/d3/d6/d17/d7a/db1/f66 [29840,44677] 0 2024-04-16T02:36:07.531 INFO:tasks.workunit.client.0.smithi151.stdout:8/701: creat d11/d2a/d33/d4e/d62/db7/ff5 x:0 0 0 2024-04-16T02:36:07.534 INFO:tasks.workunit.client.0.smithi151.stdout:8/702: creat d11/d2a/d33/d4e/ff6 x:0 0 0 2024-04-16T02:36:07.596 INFO:tasks.workunit.client.0.smithi151.stdout:9/757: dwrite d1/d75/f8b [0,4194304] 0 2024-04-16T02:36:07.596 INFO:tasks.workunit.client.0.smithi151.stdout:5/937: dread d2/d6/f27 [0,4194304] 0 2024-04-16T02:36:07.596 INFO:tasks.workunit.client.0.smithi151.stdout:9/758: chown d1/dd/d4a/d64 6817 1 2024-04-16T02:36:07.598 INFO:tasks.workunit.client.0.smithi151.stdout:0/635: sync 2024-04-16T02:36:07.598 INFO:tasks.workunit.client.0.smithi151.stdout:9/759: stat d1/dd/d4a/dbb 0 2024-04-16T02:36:07.598 INFO:tasks.workunit.client.0.smithi151.stdout:8/703: link d11/d2a/f8c d11/d2a/d33/d4e/ff7 0 2024-04-16T02:36:07.599 INFO:tasks.workunit.client.0.smithi151.stdout:5/938: write d2/d6/d19/d1a/d4c/f10b [128264,96996] 0 2024-04-16T02:36:07.600 INFO:tasks.workunit.client.0.smithi151.stdout:9/760: creat d1/d3/d10/ffe x:0 0 0 2024-04-16T02:36:07.602 INFO:tasks.workunit.client.0.smithi151.stdout:9/761: write d1/d3/d6/d17/d54/d8d/daa/ff0 [736361,79223] 0 2024-04-16T02:36:07.602 INFO:tasks.workunit.client.0.smithi151.stdout:5/939: symlink d2/d6/d19/d93/l12c 0 2024-04-16T02:36:07.666 INFO:tasks.workunit.client.0.smithi151.stdout:8/704: getdents d11/d2a/d35/d52/db3/dcd/dd2 0 2024-04-16T02:36:07.677 INFO:tasks.workunit.client.0.smithi151.stdout:9/762: dwrite d1/d3/d6/d17/d40/d56/ffc [0,4194304] 0 2024-04-16T02:36:07.678 INFO:tasks.workunit.client.0.smithi151.stdout:9/763: write d1/d75/f8b [3519647,52292] 0 2024-04-16T02:36:07.738 INFO:tasks.workunit.client.0.smithi151.stdout:0/636: dwrite dc/d15/d34/d3b/d71/f74 [0,4194304] 0 2024-04-16T02:36:07.769 INFO:tasks.workunit.client.0.smithi151.stdout:0/637: fdatasync dc/d15/d34/d3b/f44 0 2024-04-16T02:36:07.770 INFO:tasks.workunit.client.0.smithi151.stdout:5/940: fsync d2/d6/d19/d1a/f40 0 2024-04-16T02:36:07.773 INFO:tasks.workunit.client.0.smithi151.stdout:8/705: getdents d11/d7d 0 2024-04-16T02:36:07.773 INFO:tasks.workunit.client.0.smithi151.stdout:5/941: write d2/d6/d43/f6d [4740666,84314] 0 2024-04-16T02:36:07.774 INFO:tasks.workunit.client.0.smithi151.stdout:9/764: unlink d1/d3/l14 0 2024-04-16T02:36:07.775 INFO:tasks.workunit.client.0.smithi151.stdout:5/942: write d2/d6/d19/d30/fb0 [3488751,90791] 0 2024-04-16T02:36:07.782 INFO:tasks.workunit.client.0.smithi151.stdout:8/706: truncate d11/d2a/d85/f90 368722 0 2024-04-16T02:36:07.786 INFO:tasks.workunit.client.0.smithi151.stdout:8/707: read d11/d22/d2e/dc0/fe6 [42620,42776] 0 2024-04-16T02:36:07.786 INFO:tasks.workunit.client.0.smithi151.stdout:9/765: dwrite d1/feb [0,4194304] 0 2024-04-16T02:36:07.788 INFO:tasks.workunit.client.0.smithi151.stdout:8/708: read - d11/d2a/d83/fdf zero size 2024-04-16T02:36:07.788 INFO:tasks.workunit.client.0.smithi151.stdout:0/638: dread dc/d20/f3d [0,4194304] 0 2024-04-16T02:36:07.789 INFO:tasks.workunit.client.0.smithi151.stdout:0/639: stat dc/d20/f3d 0 2024-04-16T02:36:07.789 INFO:tasks.workunit.client.0.smithi151.stdout:8/709: read - d11/d22/d6c/d84/fe7 zero size 2024-04-16T02:36:07.790 INFO:tasks.workunit.client.0.smithi151.stdout:5/943: rename d2/d6/d19/d30/d88 to d2/d6/db7/d12d 0 2024-04-16T02:36:07.790 INFO:tasks.workunit.client.0.smithi151.stdout:8/710: chown d11/d22/d2e/c3b 11896 1 2024-04-16T02:36:07.793 INFO:tasks.workunit.client.0.smithi151.stdout:5/944: creat d2/d6/df1/d11a/f12e x:0 0 0 2024-04-16T02:36:07.793 INFO:tasks.workunit.client.0.smithi151.stdout:8/711: dread - d11/d2a/df4/d78/fed zero size 2024-04-16T02:36:07.797 INFO:tasks.workunit.client.0.smithi151.stdout:5/945: truncate d2/fb 226972 0 2024-04-16T02:36:07.819 INFO:tasks.workunit.client.0.smithi151.stdout:5/946: truncate d2/d6/db7/d12d/ddc/da8/dd3/f100 583360 0 2024-04-16T02:36:07.820 INFO:tasks.workunit.client.0.smithi151.stdout:0/640: dwrite dc/d20/f94 [0,4194304] 0 2024-04-16T02:36:07.821 INFO:tasks.workunit.client.0.smithi151.stdout:9/766: dread d1/dd/d4a/dbb/fc7 [0,4194304] 0 2024-04-16T02:36:07.828 INFO:tasks.workunit.client.0.smithi151.stdout:0/641: creat dc/d4c/d72/fcc x:0 0 0 2024-04-16T02:36:07.834 INFO:tasks.workunit.client.0.smithi151.stdout:0/642: creat dc/d15/d34/d67/d69/d9a/fcd x:0 0 0 2024-04-16T02:36:07.836 INFO:tasks.workunit.client.0.smithi151.stdout:5/947: link d2/d6/db7/d12d/ddc/lfe d2/d6/d19/d30/d49/d7f/l12f 0 2024-04-16T02:36:07.846 INFO:tasks.workunit.client.0.smithi151.stdout:9/767: dwrite d1/d3/d6/d28/d39/f48 [0,4194304] 0 2024-04-16T02:36:07.849 INFO:tasks.workunit.client.0.smithi151.stdout:0/643: dwrite dc/d15/d34/d4e/f6d [0,4194304] 0 2024-04-16T02:36:07.854 INFO:tasks.workunit.client.0.smithi151.stdout:9/768: fdatasync d1/d75/fc9 0 2024-04-16T02:36:07.856 INFO:tasks.workunit.client.0.smithi151.stdout:5/948: dread d2/d6/d19/f1d [4194304,4194304] 0 2024-04-16T02:36:07.858 INFO:tasks.workunit.client.0.smithi151.stdout:5/949: stat d2/d6/d19/d30/d49/ccd 0 2024-04-16T02:36:07.858 INFO:tasks.workunit.client.0.smithi151.stdout:5/950: dread - d2/d6/db7/f10c zero size 2024-04-16T02:36:07.877 INFO:tasks.workunit.client.0.smithi151.stdout:5/951: write d2/d6/d19/d1a/f105 [2067511,126916] 0 2024-04-16T02:36:07.881 INFO:tasks.workunit.client.0.smithi151.stdout:5/952: creat d2/d6/db7/d12d/ddc/d83/d12b/f130 x:0 0 0 2024-04-16T02:36:07.882 INFO:tasks.workunit.client.0.smithi151.stdout:0/644: dwrite dc/d15/d34/d67/d69/d6f/f99 [0,4194304] 0 2024-04-16T02:36:07.884 INFO:tasks.workunit.client.0.smithi151.stdout:5/953: chown d2/d6/d19/f9a 15 1 2024-04-16T02:36:07.886 INFO:tasks.workunit.client.0.smithi151.stdout:0/645: symlink dc/d15/d34/d3b/lce 0 2024-04-16T02:36:07.886 INFO:tasks.workunit.client.0.smithi151.stdout:9/769: dread d1/d3/d6/d17/d54/f79 [0,4194304] 0 2024-04-16T02:36:07.890 INFO:tasks.workunit.client.0.smithi151.stdout:0/646: creat dc/d15/d34/d67/d69/fcf x:0 0 0 2024-04-16T02:36:07.890 INFO:tasks.workunit.client.0.smithi151.stdout:9/770: mknod d1/d3/d6/cff 0 2024-04-16T02:36:07.891 INFO:tasks.workunit.client.0.smithi151.stdout:5/954: truncate d2/d6/d19/d1a/f2e 1463208 0 2024-04-16T02:36:07.892 INFO:tasks.workunit.client.0.smithi151.stdout:9/771: chown d1/d3/d6/d17/d54/d9b/d22/d2c 4722 1 2024-04-16T02:36:07.897 INFO:tasks.workunit.client.0.smithi151.stdout:5/955: rename d2/d6/d19/d62/dbd to d2/d6/d19/d30/d36/d42/d131 0 2024-04-16T02:36:07.901 INFO:tasks.workunit.client.0.smithi151.stdout:9/772: dwrite d1/d3/d6/d17/d40/d56/ffc [0,4194304] 0 2024-04-16T02:36:07.903 INFO:tasks.workunit.client.0.smithi151.stdout:0/647: write dc/f39 [3711478,91889] 0 2024-04-16T02:36:07.905 INFO:tasks.workunit.client.0.smithi151.stdout:0/648: fdatasync dc/d20/d7b/fc7 0 2024-04-16T02:36:07.906 INFO:tasks.workunit.client.0.smithi151.stdout:5/956: creat d2/d6/db7/d12d/ddc/d83/d11f/f132 x:0 0 0 2024-04-16T02:36:07.907 INFO:tasks.workunit.client.0.smithi151.stdout:9/773: fsync d1/d3/d6/d17/d54/d9b/f78 0 2024-04-16T02:36:07.910 INFO:tasks.workunit.client.0.smithi151.stdout:5/957: chown d2/d6/d19/d1a/d4c/f53 1 1 2024-04-16T02:36:07.911 INFO:tasks.workunit.client.0.smithi151.stdout:9/774: mknod d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/dc3/c100 0 2024-04-16T02:36:07.913 INFO:tasks.workunit.client.0.smithi151.stdout:5/958: creat d2/d6/db7/d12d/ddc/d83/d11f/f133 x:0 0 0 2024-04-16T02:36:07.916 INFO:tasks.workunit.client.0.smithi151.stdout:9/775: creat d1/d3/d6/f101 x:0 0 0 2024-04-16T02:36:07.918 INFO:tasks.workunit.client.0.smithi151.stdout:0/649: dread dc/d15/d34/d67/f3e [0,4194304] 0 2024-04-16T02:36:07.920 INFO:tasks.workunit.client.0.smithi151.stdout:9/776: truncate d1/d3/d6/d17/d54/d58/fa7 582374 0 2024-04-16T02:36:07.925 INFO:tasks.workunit.client.0.smithi151.stdout:9/777: rename d1/d75/fd2 to d1/d3/d6/d17/d54/d9b/d22/d36/f102 0 2024-04-16T02:36:07.930 INFO:tasks.workunit.client.0.smithi151.stdout:9/778: symlink d1/d3/l103 0 2024-04-16T02:36:07.933 INFO:tasks.workunit.client.0.smithi151.stdout:9/779: symlink d1/d3/d10/d7b/l104 0 2024-04-16T02:36:07.935 INFO:tasks.workunit.client.0.smithi151.stdout:0/650: truncate dc/d15/d34/d4e/f6d 2210454 0 2024-04-16T02:36:07.939 INFO:tasks.workunit.client.0.smithi151.stdout:5/959: truncate d2/d6/d19/d1a/f2e 2207403 0 2024-04-16T02:36:07.940 INFO:tasks.workunit.client.0.smithi151.stdout:9/780: write d1/dd/f8a [98022,34193] 0 2024-04-16T02:36:07.941 INFO:tasks.workunit.client.0.smithi151.stdout:5/960: write d2/d6/db7/d12d/ddc/d83/d11f/f132 [975769,111042] 0 2024-04-16T02:36:07.944 INFO:tasks.workunit.client.0.smithi151.stdout:5/961: dread - d2/d6/d19/d62/f121 zero size 2024-04-16T02:36:07.946 INFO:tasks.workunit.client.0.smithi151.stdout:9/781: rename d1/d3/d6/d17/d54/l71 to d1/d3/d6/d17/d54/d9b/dae/d6b/l105 0 2024-04-16T02:36:07.948 INFO:tasks.workunit.client.0.smithi151.stdout:5/962: creat d2/d6/d19/f134 x:0 0 0 2024-04-16T02:36:07.951 INFO:tasks.workunit.client.0.smithi151.stdout:9/782: rmdir d1/d3/d5f/dfa 0 2024-04-16T02:36:07.952 INFO:tasks.workunit.client.0.smithi151.stdout:0/651: dread f3 [0,4194304] 0 2024-04-16T02:36:07.954 INFO:tasks.workunit.client.0.smithi151.stdout:0/652: mknod dc/d4c/d72/cd0 0 2024-04-16T02:36:07.955 INFO:tasks.workunit.client.0.smithi151.stdout:9/783: creat d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/f106 x:0 0 0 2024-04-16T02:36:07.956 INFO:tasks.workunit.client.0.smithi151.stdout:0/653: creat dc/d20/fd1 x:0 0 0 2024-04-16T02:36:07.958 INFO:tasks.workunit.client.0.smithi151.stdout:0/654: mkdir dc/d15/d34/d67/dd2 0 2024-04-16T02:36:08.020 INFO:tasks.workunit.client.0.smithi151.stdout:9/784: getdents d1/d3/d6/d17/d40 0 2024-04-16T02:36:08.020 INFO:tasks.workunit.client.0.smithi151.stdout:0/655: read - dc/d3c/fb1 zero size 2024-04-16T02:36:08.021 INFO:tasks.workunit.client.0.smithi151.stdout:0/656: stat dc/d15/d5c 0 2024-04-16T02:36:08.021 INFO:tasks.workunit.client.0.smithi151.stdout:9/785: write d1/d3/f33 [1981343,40792] 0 2024-04-16T02:36:08.026 INFO:tasks.workunit.client.0.smithi151.stdout:9/786: rename d1/d3/d10/f24 to d1/d3/d6/d17/f107 0 2024-04-16T02:36:08.034 INFO:tasks.workunit.client.0.smithi151.stdout:9/787: dwrite d1/d3/d5f/fe6 [0,4194304] 0 2024-04-16T02:36:08.038 INFO:tasks.workunit.client.0.smithi151.stdout:9/788: readlink d1/l4 0 2024-04-16T02:36:08.040 INFO:tasks.workunit.client.0.smithi151.stdout:0/657: dread dc/d15/f17 [0,4194304] 0 2024-04-16T02:36:08.041 INFO:tasks.workunit.client.0.smithi151.stdout:9/789: mkdir d1/d3/d6/d17/d54/d8d/daa/d108 0 2024-04-16T02:36:08.051 INFO:tasks.workunit.client.0.smithi151.stdout:0/658: dwrite dc/d15/d5c/faf [0,4194304] 0 2024-04-16T02:36:08.063 INFO:tasks.workunit.client.0.smithi151.stdout:0/659: dwrite dc/d4c/d72/d76/db0/fb7 [0,4194304] 0 2024-04-16T02:36:08.065 INFO:tasks.workunit.client.0.smithi151.stdout:9/790: dread d1/d3/d6/f74 [0,4194304] 0 2024-04-16T02:36:08.069 INFO:tasks.workunit.client.0.smithi151.stdout:9/791: truncate d1/d3/d6/f1b 4744486 0 2024-04-16T02:36:08.069 INFO:tasks.workunit.client.0.smithi151.stdout:9/792: dread - d1/d3/d6/d17/d54/d8d/f90 zero size 2024-04-16T02:36:08.070 INFO:tasks.workunit.client.0.smithi151.stdout:9/793: readlink d1/d3/d6/l42 0 2024-04-16T02:36:08.073 INFO:tasks.workunit.client.0.smithi151.stdout:9/794: mkdir d1/d3/d6/d17/d54/d9b/d22/d109 0 2024-04-16T02:36:08.078 INFO:tasks.workunit.client.0.smithi151.stdout:0/660: fdatasync dc/d15/d34/f4d 0 2024-04-16T02:36:08.081 INFO:tasks.workunit.client.0.smithi151.stdout:9/795: write d1/d3/f5e [489706,2402] 0 2024-04-16T02:36:08.086 INFO:tasks.workunit.client.0.smithi151.stdout:9/796: fsync d1/d3/d6/d17/d54/d9b/d22/d36/f102 0 2024-04-16T02:36:08.087 INFO:tasks.workunit.client.0.smithi151.stdout:9/797: readlink d1/d3/d6/l42 0 2024-04-16T02:36:08.090 INFO:tasks.workunit.client.0.smithi151.stdout:0/661: dwrite dc/d15/d34/d67/d51/f62 [0,4194304] 0 2024-04-16T02:36:08.092 INFO:tasks.workunit.client.0.smithi151.stdout:9/798: truncate d1/dd/d4a/d64/f76 7206744 0 2024-04-16T02:36:08.094 INFO:tasks.workunit.client.0.smithi151.stdout:9/799: chown d1/d3/d6/d17/d40/d7d/da0 1 1 2024-04-16T02:36:08.095 INFO:tasks.workunit.client.0.smithi151.stdout:0/662: getdents dc/d4c/d72/d76 0 2024-04-16T02:36:08.095 INFO:tasks.workunit.client.0.smithi151.stdout:9/800: readlink d1/d3/d6/d17/d54/d9b/lb2 0 2024-04-16T02:36:08.098 INFO:tasks.workunit.client.0.smithi151.stdout:9/801: creat d1/d3/d6/d17/d40/d56/f10a x:0 0 0 2024-04-16T02:36:08.099 INFO:tasks.workunit.client.0.smithi151.stdout:9/802: chown d1/d3/d6/d17/d40/d56/ce8 226545 1 2024-04-16T02:36:08.104 INFO:tasks.workunit.client.0.smithi151.stdout:9/803: link d1/d3/d6/d17/d54/dd5/fd9 d1/d3/d5f/f10b 0 2024-04-16T02:36:08.108 INFO:tasks.workunit.client.0.smithi151.stdout:9/804: mknod d1/dd/c10c 0 2024-04-16T02:36:08.111 INFO:tasks.workunit.client.0.smithi151.stdout:0/663: dwrite dc/d15/d34/d67/f3e [0,4194304] 0 2024-04-16T02:36:08.113 INFO:tasks.workunit.client.0.smithi151.stdout:0/664: symlink dc/d20/ld3 0 2024-04-16T02:36:08.115 INFO:tasks.workunit.client.0.smithi151.stdout:9/805: creat d1/d3/d6/d17/d54/d8d/f10d x:0 0 0 2024-04-16T02:36:08.116 INFO:tasks.workunit.client.0.smithi151.stdout:0/665: fdatasync dc/d20/f42 0 2024-04-16T02:36:08.116 INFO:tasks.workunit.client.0.smithi151.stdout:9/806: chown d1/d3/d6/d17/d54/d8d/f10d 1435 1 2024-04-16T02:36:08.118 INFO:tasks.workunit.client.0.smithi151.stdout:0/666: stat dc/d15/d34/d67/l6c 0 2024-04-16T02:36:08.119 INFO:tasks.workunit.client.0.smithi151.stdout:9/807: mknod d1/d3/d6/d17/d54/d8d/daa/d9d/c10e 0 2024-04-16T02:36:08.120 INFO:tasks.workunit.client.0.smithi151.stdout:0/667: dread - dc/d15/d34/d67/d69/fcf zero size 2024-04-16T02:36:08.124 INFO:tasks.workunit.client.0.smithi151.stdout:0/668: creat dc/d15/fd4 x:0 0 0 2024-04-16T02:36:08.128 INFO:tasks.workunit.client.0.smithi151.stdout:0/669: dread dc/d20/f3d [0,4194304] 0 2024-04-16T02:36:08.128 INFO:tasks.workunit.client.0.smithi151.stdout:9/808: write d1/d3/d6/d17/d54/d9b/fd6 [2689065,78333] 0 2024-04-16T02:36:08.130 INFO:tasks.workunit.client.0.smithi151.stdout:0/670: fdatasync dc/d15/d34/d67/f3e 0 2024-04-16T02:36:08.131 INFO:tasks.workunit.client.0.smithi151.stdout:0/671: chown dc/d15/d34/d67/d51 6 1 2024-04-16T02:36:08.132 INFO:tasks.workunit.client.0.smithi151.stdout:0/672: chown dc/d15/d34/d67/d69/d6f/l93 1 1 2024-04-16T02:36:08.134 INFO:tasks.workunit.client.0.smithi151.stdout:9/809: rename d1/d3/d6/d17/d54/d8d/cc5 to d1/c10f 0 2024-04-16T02:36:08.136 INFO:tasks.workunit.client.0.smithi151.stdout:0/673: creat dc/d3c/dc2/fd5 x:0 0 0 2024-04-16T02:36:08.136 INFO:tasks.workunit.client.0.smithi151.stdout:9/810: read d1/f2e [2366373,91479] 0 2024-04-16T02:36:08.140 INFO:tasks.workunit.client.0.smithi151.stdout:0/674: dread dc/d15/d34/d67/d51/fbe [0,4194304] 0 2024-04-16T02:36:08.142 INFO:tasks.workunit.client.0.smithi151.stdout:9/811: truncate d1/d3/f5e 1640838 0 2024-04-16T02:36:08.143 INFO:tasks.workunit.client.0.smithi151.stdout:0/675: truncate dc/d15/d34/d67/d69/fcf 536147 0 2024-04-16T02:36:08.144 INFO:tasks.workunit.client.0.smithi151.stdout:9/812: mknod d1/dd/d38/c110 0 2024-04-16T02:36:08.145 INFO:tasks.workunit.client.0.smithi151.stdout:0/676: rmdir dc/d15/d34/d67/d69/d6f/d9e 39 2024-04-16T02:36:08.146 INFO:tasks.workunit.client.0.smithi151.stdout:9/813: symlink d1/d3/d6/d17/d40/d7d/da0/l111 0 2024-04-16T02:36:08.150 INFO:tasks.workunit.client.0.smithi151.stdout:9/814: mknod d1/d3/d6/d17/d54/d9b/d22/d2c/d86/c112 0 2024-04-16T02:36:08.151 INFO:tasks.workunit.client.0.smithi151.stdout:9/815: write d1/dd/f8a [1517136,118971] 0 2024-04-16T02:36:08.151 INFO:tasks.workunit.client.0.smithi151.stdout:0/677: dwrite dc/d15/d34/f4d [0,4194304] 0 2024-04-16T02:36:08.154 INFO:tasks.workunit.client.0.smithi151.stdout:9/816: symlink d1/dd/d4a/d64/l113 0 2024-04-16T02:36:08.158 INFO:tasks.workunit.client.0.smithi151.stdout:0/678: rename dc/d15/d34/d67/d69/d9a to dc/d20/d7b/d9c/dd6 0 2024-04-16T02:36:08.159 INFO:tasks.workunit.client.0.smithi151.stdout:9/817: creat d1/dd4/f114 x:0 0 0 2024-04-16T02:36:08.165 INFO:tasks.workunit.client.0.smithi151.stdout:9/818: rename d1/d3/fa4 to d1/d3/d6/d17/d54/dd5/f115 0 2024-04-16T02:36:08.166 INFO:tasks.workunit.client.0.smithi151.stdout:9/819: chown d1/d3/d6/d17/l2f 234 1 2024-04-16T02:36:08.179 INFO:tasks.workunit.client.0.smithi151.stdout:9/820: dwrite d1/d3/d6/d17/d54/d9b/dae/d6b/f77 [0,4194304] 0 2024-04-16T02:36:08.182 INFO:tasks.workunit.client.0.smithi151.stdout:9/821: creat d1/dd/d4a/f116 x:0 0 0 2024-04-16T02:36:08.186 INFO:tasks.workunit.client.0.smithi151.stdout:9/822: symlink d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/l117 0 2024-04-16T02:36:08.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:07 smithi151 ceph-mon[29003]: pgmap v31: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 33 MiB/s rd, 89 MiB/s wr, 259 op/s 2024-04-16T02:36:08.255 INFO:tasks.workunit.client.0.smithi151.stdout:9/823: rmdir d1/d3/d6/d17/d54/d9b/d22/d36 39 2024-04-16T02:36:08.257 INFO:tasks.workunit.client.0.smithi151.stdout:9/824: creat d1/d3/d10/f118 x:0 0 0 2024-04-16T02:36:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:07 smithi153 ceph-mon[36747]: pgmap v31: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 33 MiB/s rd, 89 MiB/s wr, 259 op/s 2024-04-16T02:36:08.272 INFO:tasks.workunit.client.0.smithi151.stdout:9/825: dread d1/d3/d10/d7b/f85 [0,4194304] 0 2024-04-16T02:36:08.273 INFO:tasks.workunit.client.0.smithi151.stdout:9/826: stat d1/d3/d6/d17/d54/d9b/dae/fd0 0 2024-04-16T02:36:08.278 INFO:tasks.workunit.client.0.smithi151.stdout:9/827: write d1/d3/d6/d17/d54/dd5/fd9 [625741,15900] 0 2024-04-16T02:36:08.278 INFO:tasks.workunit.client.0.smithi151.stdout:9/828: rmdir d1 39 2024-04-16T02:36:08.328 INFO:tasks.workunit.client.0.smithi151.stdout:9/829: dread - d1/dd/d4a/f92 zero size 2024-04-16T02:36:08.331 INFO:tasks.workunit.client.0.smithi151.stdout:9/830: getdents d1 0 2024-04-16T02:36:08.339 INFO:tasks.workunit.client.0.smithi151.stdout:9/831: dwrite d1/d3/d6/d17/d54/d9b/dae/d6b/fbc [0,4194304] 0 2024-04-16T02:36:08.342 INFO:tasks.workunit.client.0.smithi151.stdout:9/832: creat d1/d3/d6/d28/d39/f119 x:0 0 0 2024-04-16T02:36:08.347 INFO:tasks.workunit.client.0.smithi151.stdout:9/833: write d1/d3/d6/d17/d44/f6e [3836213,129949] 0 2024-04-16T02:36:08.492 INFO:tasks.workunit.client.0.smithi151.stdout:6/992: sync 2024-04-16T02:36:08.494 INFO:tasks.workunit.client.0.smithi151.stdout:6/993: mkdir d2/d13/d3a/da6/d130/d73/d97/d151 0 2024-04-16T02:36:08.500 INFO:tasks.workunit.client.0.smithi151.stdout:6/994: dwrite d2/d13/d46/f13d [0,4194304] 0 2024-04-16T02:36:08.520 INFO:tasks.workunit.client.0.smithi151.stdout:6/995: dread d2/d3d/fa1 [0,4194304] 0 2024-04-16T02:36:08.522 INFO:tasks.workunit.client.0.smithi151.stdout:6/996: dread - d2/d13/d63/def/df/d23/d82/d8d/d22/d40/f6a zero size 2024-04-16T02:36:08.525 INFO:tasks.workunit.client.0.smithi151.stdout:6/997: mkdir d2/d13/d63/def/d2e/d152 0 2024-04-16T02:36:08.529 INFO:tasks.workunit.client.0.smithi151.stdout:6/998: write d2/d13/d3a/da6/fb3 [1298231,14597] 0 2024-04-16T02:36:08.530 INFO:tasks.workunit.client.0.smithi151.stdout:6/999: chown d2/d13/d5f/dd7/c118 1911190414 1 2024-04-16T02:36:08.671 INFO:tasks.workunit.client.0.smithi151.stdout:7/865: sync 2024-04-16T02:36:08.687 INFO:tasks.workunit.client.0.smithi151.stdout:7/866: read d0/d27/f95 [715956,14141] 0 2024-04-16T02:36:08.689 INFO:tasks.workunit.client.0.smithi151.stdout:7/867: fsync d0/d2c/d38/de7/d11f/d33/fe6 0 2024-04-16T02:36:08.691 INFO:tasks.workunit.client.0.smithi151.stdout:7/868: dread - d0/d2c/d38/de7/d11f/d99/f10d zero size 2024-04-16T02:36:08.694 INFO:tasks.workunit.client.0.smithi151.stdout:7/869: mkdir d0/d2c/d38/de7/d11f/d33/d12b 0 2024-04-16T02:36:08.697 INFO:tasks.workunit.client.0.smithi151.stdout:7/870: creat d0/d2c/d38/de7/d11f/d46/f12c x:0 0 0 2024-04-16T02:36:08.698 INFO:tasks.workunit.client.0.smithi151.stdout:7/871: stat d0/d2c/d38/de7/d11f/d60/f127 0 2024-04-16T02:36:08.700 INFO:tasks.workunit.client.0.smithi151.stdout:7/872: symlink d0/d2c/d38/d85/l12d 0 2024-04-16T02:36:08.702 INFO:tasks.workunit.client.0.smithi151.stdout:7/873: symlink d0/d2c/d38/de7/d11f/d4c/db6/l12e 0 2024-04-16T02:36:08.703 INFO:tasks.workunit.client.0.smithi151.stdout:7/874: chown d0/d2c/d38/de7/d11f/d99/dc0/l129 1889452456 1 2024-04-16T02:36:08.704 INFO:tasks.workunit.client.0.smithi151.stdout:7/875: fsync d0/d2c/d38/de7/d11f/d33/f3a 0 2024-04-16T02:36:08.707 INFO:tasks.workunit.client.0.smithi151.stdout:7/876: truncate d0/d2c/d38/de7/d11f/d33/d89/fbe 2823833 0 2024-04-16T02:36:08.709 INFO:tasks.workunit.client.0.smithi151.stdout:7/877: fdatasync d0/d2c/d38/db0/fe2 0 2024-04-16T02:36:08.711 INFO:tasks.workunit.client.0.smithi151.stdout:7/878: truncate d0/d2c/d38/d5e/f92 778563 0 2024-04-16T02:36:08.713 INFO:tasks.workunit.client.0.smithi151.stdout:7/879: symlink d0/d2c/d38/d85/d117/l12f 0 2024-04-16T02:36:08.718 INFO:tasks.workunit.client.0.smithi151.stdout:7/880: write d0/d2c/d38/de7/d11f/f40 [1039900,14955] 0 2024-04-16T02:36:08.720 INFO:tasks.workunit.client.0.smithi151.stdout:7/881: symlink d0/d2c/d38/de7/d11f/d46/dff/l130 0 2024-04-16T02:36:08.721 INFO:tasks.workunit.client.0.smithi151.stdout:7/882: readlink d0/d2c/d38/d85/db7/ldb 0 2024-04-16T02:36:08.722 INFO:tasks.workunit.client.0.smithi151.stdout:7/883: mkdir d0/d2c/d38/de7/d11f/d60/d131 0 2024-04-16T02:36:08.723 INFO:tasks.workunit.client.0.smithi151.stdout:7/884: write d0/d27/da6/dcd/f114 [1012006,15156] 0 2024-04-16T02:36:08.725 INFO:tasks.workunit.client.0.smithi151.stdout:7/885: mknod d0/d2c/d123/c132 0 2024-04-16T02:36:08.726 INFO:tasks.workunit.client.0.smithi151.stdout:7/886: symlink d0/d2c/d8e/l133 0 2024-04-16T02:36:08.728 INFO:tasks.workunit.client.0.smithi151.stdout:7/887: creat d0/d2c/d38/de7/d11f/d99/dc0/f134 x:0 0 0 2024-04-16T02:36:08.734 INFO:tasks.workunit.client.0.smithi151.stdout:7/888: dwrite d0/d2c/d38/de7/d11f/d46/f12c [0,4194304] 0 2024-04-16T02:36:08.735 INFO:tasks.workunit.client.0.smithi151.stdout:7/889: fdatasync d0/d2c/d38/d5e/f121 0 2024-04-16T02:36:08.740 INFO:tasks.workunit.client.0.smithi151.stdout:7/890: creat d0/d2c/d38/d5e/dd1/dee/dc9/f135 x:0 0 0 2024-04-16T02:36:08.744 INFO:tasks.workunit.client.0.smithi151.stdout:7/891: rename d0/d2c/d38/de7/d11f/d31/d42/f6c to d0/d2c/d38/d43/ddd/f136 0 2024-04-16T02:36:08.745 INFO:tasks.workunit.client.0.smithi151.stdout:7/892: creat d0/d27/da6/f137 x:0 0 0 2024-04-16T02:36:08.748 INFO:tasks.workunit.client.0.smithi151.stdout:7/893: creat d0/d2c/d38/de7/d11f/f138 x:0 0 0 2024-04-16T02:36:08.749 INFO:tasks.workunit.client.0.smithi151.stdout:7/894: mkdir d0/d2c/d38/de7/d11f/d94/d139 0 2024-04-16T02:36:08.751 INFO:tasks.workunit.client.0.smithi151.stdout:7/895: mkdir d0/d2c/d38/d43/d122/d13a 0 2024-04-16T02:36:08.756 INFO:tasks.workunit.client.0.smithi151.stdout:7/896: dread d0/d2c/d38/de7/d11f/d60/f91 [0,4194304] 0 2024-04-16T02:36:08.765 INFO:tasks.workunit.client.0.smithi151.stdout:7/897: dwrite d0/d2c/d38/de7/d11f/f28 [0,4194304] 0 2024-04-16T02:36:08.768 INFO:tasks.workunit.client.0.smithi151.stdout:7/898: rename d0/c10 to d0/d2c/d38/c13b 0 2024-04-16T02:36:08.770 INFO:tasks.workunit.client.0.smithi151.stdout:7/899: getdents d0/d2c/d38/de7/d11f/d4c/db6/dd7 0 2024-04-16T02:36:08.778 INFO:tasks.workunit.client.0.smithi151.stdout:7/900: dwrite d0/d27/da6/fd6 [0,4194304] 0 2024-04-16T02:36:08.781 INFO:tasks.workunit.client.0.smithi151.stdout:7/901: mkdir d0/d2c/d38/d85/d13c 0 2024-04-16T02:36:08.801 INFO:tasks.workunit.client.0.smithi151.stdout:7/902: dread d0/d2c/d38/de7/d11f/d4c/d82/fca [0,4194304] 0 2024-04-16T02:36:08.805 INFO:tasks.workunit.client.0.smithi151.stdout:7/903: link d0/d2c/d38/d85/db7/f10e d0/d2c/d38/de7/d11f/d4c/d82/f13d 0 2024-04-16T02:36:08.808 INFO:tasks.workunit.client.0.smithi151.stdout:7/904: rename d0/d2c/d38/de7/d11f/d33/cfb to d0/ddf/c13e 0 2024-04-16T02:36:08.809 INFO:tasks.workunit.client.0.smithi151.stdout:7/905: chown d0/d2c/d38/de7/d11f/d4c/f104 2138058883 1 2024-04-16T02:36:08.825 INFO:tasks.workunit.client.0.smithi151.stdout:7/906: dread d0/fbc [0,4194304] 0 2024-04-16T02:36:08.826 INFO:tasks.workunit.client.0.smithi151.stdout:7/907: stat d0/d2c/d38/de7/d11f/d46/d4f/fd9 0 2024-04-16T02:36:08.827 INFO:tasks.workunit.client.0.smithi151.stdout:7/908: chown d0/d27/da6/dcd/f108 992 1 2024-04-16T02:36:08.828 INFO:tasks.workunit.client.0.smithi151.stdout:7/909: fsync d0/d27/da6/dcd/f108 0 2024-04-16T02:36:08.830 INFO:tasks.workunit.client.0.smithi151.stdout:7/910: symlink d0/d2c/d38/de7/d11f/d4c/d98/d10b/l13f 0 2024-04-16T02:36:08.831 INFO:tasks.workunit.client.0.smithi151.stdout:7/911: chown d0/d2c/d38/d5e/deb 3401 1 2024-04-16T02:36:08.833 INFO:tasks.workunit.client.0.smithi151.stdout:7/912: creat d0/d2c/d38/d5e/dd1/dee/dc9/f140 x:0 0 0 2024-04-16T02:36:08.835 INFO:tasks.workunit.client.0.smithi151.stdout:7/913: chown d0/d2c/d38/de7/d11f/d33/d89/fbe 1 1 2024-04-16T02:36:08.835 INFO:tasks.workunit.client.0.smithi151.stdout:7/914: stat d0/d2c/d38/de7/d11f/d31 0 2024-04-16T02:36:08.836 INFO:tasks.workunit.client.0.smithi151.stdout:7/915: fsync d0/d2c/d38/de7/d11f/da8/f125 0 2024-04-16T02:36:08.837 INFO:tasks.workunit.client.0.smithi151.stdout:7/916: chown d0/d27/da6 30309689 1 2024-04-16T02:36:08.838 INFO:tasks.workunit.client.0.smithi151.stdout:7/917: creat d0/d27/f141 x:0 0 0 2024-04-16T02:36:08.849 INFO:tasks.workunit.client.0.smithi151.stdout:7/918: dwrite d0/d27/f2b [4194304,4194304] 0 2024-04-16T02:36:08.851 INFO:tasks.workunit.client.0.smithi151.stdout:7/919: fsync d0/d2c/d38/de7/d11f/d31/d42/f45 0 2024-04-16T02:36:08.855 INFO:tasks.workunit.client.0.smithi151.stdout:7/920: creat d0/d2c/f142 x:0 0 0 2024-04-16T02:36:08.866 INFO:tasks.workunit.client.0.smithi151.stdout:7/921: write d0/d2c/d38/de7/d11f/d4c/d82/f13d [1554390,108827] 0 2024-04-16T02:36:08.866 INFO:tasks.workunit.client.0.smithi151.stdout:3/658: sync 2024-04-16T02:36:08.868 INFO:tasks.workunit.client.0.smithi151.stdout:7/922: truncate d0/d2c/d38/de7/d11f/d60/f127 790283 0 2024-04-16T02:36:08.869 INFO:tasks.workunit.client.0.smithi151.stdout:7/923: write d0/d27/f11a [411878,124916] 0 2024-04-16T02:36:08.873 INFO:tasks.workunit.client.0.smithi151.stdout:7/924: mkdir d0/d2c/d38/de7/d11f/d4c/d98/d10b/d143 0 2024-04-16T02:36:08.874 INFO:tasks.workunit.client.0.smithi151.stdout:7/925: chown d0/d2c/d38/d5e/deb 274 1 2024-04-16T02:36:08.877 INFO:tasks.workunit.client.0.smithi151.stdout:3/659: dwrite d1/d6/d22/d23/d30/d44/f5f [4194304,4194304] 0 2024-04-16T02:36:08.879 INFO:tasks.workunit.client.0.smithi151.stdout:7/926: rename d0/d2c/d38/de7/d11f/d33/d56/f11e to d0/d2c/d38/de7/d11f/d31/f144 0 2024-04-16T02:36:08.883 INFO:tasks.workunit.client.0.smithi151.stdout:3/660: fsync d1/d6/d20/d54/f80 0 2024-04-16T02:36:08.883 INFO:tasks.workunit.client.0.smithi151.stdout:3/661: chown d1/d69/d7f 32077963 1 2024-04-16T02:36:08.886 INFO:tasks.workunit.client.0.smithi151.stdout:3/662: dread d1/f3e [0,4194304] 0 2024-04-16T02:36:08.891 INFO:tasks.workunit.client.0.smithi151.stdout:7/927: dwrite d0/d2c/d38/de7/d11f/d99/dc0/f116 [0,4194304] 0 2024-04-16T02:36:08.893 INFO:tasks.workunit.client.0.smithi151.stdout:3/663: rename d1/d6/d22/d23/d30 to d1/d6/d20/d3d/dd4/dd6 0 2024-04-16T02:36:08.895 INFO:tasks.workunit.client.0.smithi151.stdout:7/928: rename d0/d2c/d38/d43/ddd to d0/d2c/d38/d43/d122/d13a/d145 0 2024-04-16T02:36:08.898 INFO:tasks.workunit.client.0.smithi151.stdout:3/664: chown d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/cae 1783748 1 2024-04-16T02:36:08.899 INFO:tasks.workunit.client.0.smithi151.stdout:3/665: write d1/d6/d20/fc5 [4549524,103148] 0 2024-04-16T02:36:08.902 INFO:tasks.workunit.client.0.smithi151.stdout:7/929: rename d0/d2c/d38/de7/d11f/d4c/d98 to d0/d146 0 2024-04-16T02:36:08.904 INFO:tasks.workunit.client.0.smithi151.stdout:3/666: getdents d1/d6/d22/d6c 0 2024-04-16T02:36:08.907 INFO:tasks.workunit.client.0.smithi151.stdout:3/667: readlink d1/d37/d87/lc9 0 2024-04-16T02:36:08.910 INFO:tasks.workunit.client.0.smithi151.stdout:3/668: creat d1/d6/d20/d3d/da6/fd7 x:0 0 0 2024-04-16T02:36:08.910 INFO:tasks.workunit.client.0.smithi151.stdout:1/755: sync 2024-04-16T02:36:08.910 INFO:tasks.workunit.client.0.smithi151.stdout:4/777: sync 2024-04-16T02:36:08.911 INFO:tasks.workunit.client.0.smithi151.stdout:3/669: write d1/d6/d20/d3d/da6/fd7 [441712,4433] 0 2024-04-16T02:36:08.914 INFO:tasks.workunit.client.0.smithi151.stdout:1/756: mknod d3/d4/d64/d1c/c104 0 2024-04-16T02:36:08.916 INFO:tasks.workunit.client.0.smithi151.stdout:4/778: chown d0/d11/d19/d87/d3a/c3b 24996 1 2024-04-16T02:36:08.919 INFO:tasks.workunit.client.0.smithi151.stdout:4/779: readlink d0/dde/d4a/d60/l9e 0 2024-04-16T02:36:08.922 INFO:tasks.workunit.client.0.smithi151.stdout:7/930: dwrite d0/d2c/d38/de7/d11f/d4c/f81 [0,4194304] 0 2024-04-16T02:36:08.927 INFO:tasks.workunit.client.0.smithi151.stdout:3/670: dwrite d1/d6/d22/f2c [4194304,4194304] 0 2024-04-16T02:36:08.929 INFO:tasks.workunit.client.0.smithi151.stdout:1/757: dwrite d3/d4/d64/d40/f100 [0,4194304] 0 2024-04-16T02:36:08.937 INFO:tasks.workunit.client.0.smithi151.stdout:4/780: rename d0/d11/d19/d87/d7b/cc1 to d0/d11/d19/d87/d3a/d50/dc7/c111 0 2024-04-16T02:36:08.939 INFO:tasks.workunit.client.0.smithi151.stdout:7/931: dread d0/d2c/d38/de7/d11f/d46/f12c [0,4194304] 0 2024-04-16T02:36:08.940 INFO:tasks.workunit.client.0.smithi151.stdout:1/758: dwrite d3/d58/df/d32/d3a/d67/f93 [0,4194304] 0 2024-04-16T02:36:08.940 INFO:tasks.workunit.client.0.smithi151.stdout:7/932: dread - d0/d2c/f142 zero size 2024-04-16T02:36:08.945 INFO:tasks.workunit.client.0.smithi151.stdout:4/781: rename d0/dde/d80/da0/ffb to d0/d64/f112 0 2024-04-16T02:36:08.945 INFO:tasks.workunit.client.0.smithi151.stdout:4/782: fdatasync d0/dde/d80/f10d 0 2024-04-16T02:36:08.945 INFO:tasks.workunit.client.0.smithi151.stdout:7/933: mkdir d0/d147 0 2024-04-16T02:36:08.947 INFO:tasks.workunit.client.0.smithi151.stdout:4/783: chown d0/dde/d80/da0/cb9 16438 1 2024-04-16T02:36:08.947 INFO:tasks.workunit.client.0.smithi151.stdout:1/759: fsync d3/d33/d10/d4f/fc0 0 2024-04-16T02:36:08.948 INFO:tasks.workunit.client.0.smithi151.stdout:4/784: fsync d0/d11/f8d 0 2024-04-16T02:36:08.948 INFO:tasks.workunit.client.0.smithi151.stdout:7/934: creat d0/d27/da6/f148 x:0 0 0 2024-04-16T02:36:08.950 INFO:tasks.workunit.client.0.smithi151.stdout:1/760: truncate d3/d33/f39 1501332 0 2024-04-16T02:36:08.953 INFO:tasks.workunit.client.0.smithi151.stdout:4/785: mknod d0/d11/d19/d46/c113 0 2024-04-16T02:36:08.958 INFO:tasks.workunit.client.0.smithi151.stdout:7/935: dwrite d0/fe0 [4194304,4194304] 0 2024-04-16T02:36:08.959 INFO:tasks.workunit.client.0.smithi151.stdout:4/786: rename d0/d99/d59/f108 to d0/dde/dad/d100/f114 0 2024-04-16T02:36:08.959 INFO:tasks.workunit.client.0.smithi151.stdout:1/761: link d3/d33/d10/d4f/lb3 d3/d58/df/d32/d3a/d71/ded/l105 0 2024-04-16T02:36:08.961 INFO:tasks.workunit.client.0.smithi151.stdout:4/787: chown d0/dde/d80/da0/lc3 7 1 2024-04-16T02:36:08.961 INFO:tasks.workunit.client.0.smithi151.stdout:7/936: mknod d0/d2c/d38/d5e/deb/c149 0 2024-04-16T02:36:08.963 INFO:tasks.workunit.client.0.smithi151.stdout:7/937: read d0/d2c/d38/de7/d11f/d4c/f81 [512997,119882] 0 2024-04-16T02:36:08.968 INFO:tasks.workunit.client.0.smithi151.stdout:4/788: rename d0/d11/d19/d87/d3a/d65/f7d to d0/dde/d4a/d60/f115 0 2024-04-16T02:36:08.971 INFO:tasks.workunit.client.0.smithi151.stdout:4/789: read d0/d99/f4f [835930,64135] 0 2024-04-16T02:36:08.974 INFO:tasks.workunit.client.0.smithi151.stdout:9/834: sync 2024-04-16T02:36:08.974 INFO:tasks.workunit.client.0.smithi151.stdout:5/963: sync 2024-04-16T02:36:08.975 INFO:tasks.workunit.client.0.smithi151.stdout:8/712: sync 2024-04-16T02:36:08.975 INFO:tasks.workunit.client.0.smithi151.stdout:0/679: sync 2024-04-16T02:36:08.976 INFO:tasks.workunit.client.0.smithi151.stdout:7/938: rename d0/d2c/d38/d5e/c64 to d0/d147/c14a 0 2024-04-16T02:36:08.976 INFO:tasks.workunit.client.0.smithi151.stdout:8/713: dread - d11/d22/d2e/f46 zero size 2024-04-16T02:36:08.977 INFO:tasks.workunit.client.0.smithi151.stdout:0/680: chown dc/d15/d34/d67/d51/fb9 97991 1 2024-04-16T02:36:08.979 INFO:tasks.workunit.client.0.smithi151.stdout:7/939: chown d0/d2c/d38/d5e/dd1/dee/dc9 782856810 1 2024-04-16T02:36:08.980 INFO:tasks.workunit.client.0.smithi151.stdout:1/762: truncate d3/d1f/dd7/ddb/fe7 58242 0 2024-04-16T02:36:08.981 INFO:tasks.workunit.client.0.smithi151.stdout:3/671: dread d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/f1a [0,4194304] 0 2024-04-16T02:36:08.990 INFO:tasks.workunit.client.0.smithi151.stdout:5/964: creat d2/d10f/f135 x:0 0 0 2024-04-16T02:36:08.992 INFO:tasks.workunit.client.0.smithi151.stdout:4/790: dread d0/d64/f76 [0,4194304] 0 2024-04-16T02:36:08.996 INFO:tasks.workunit.client.0.smithi151.stdout:3/672: mkdir d1/d6/d20/d3d/da6/dd8 0 2024-04-16T02:36:08.996 INFO:tasks.workunit.client.0.smithi151.stdout:1/763: dread d3/d58/df/d32/d3a/df3/d7a/f9f [0,4194304] 0 2024-04-16T02:36:08.998 INFO:tasks.workunit.client.0.smithi151.stdout:3/673: write d1/d6/d20/d3d/dd4/dd6/fa8 [1768093,72298] 0 2024-04-16T02:36:08.999 INFO:tasks.workunit.client.0.smithi151.stdout:8/714: creat d11/d2a/d33/d4e/d62/d6b/db9/ff8 x:0 0 0 2024-04-16T02:36:09.000 INFO:tasks.workunit.client.0.smithi151.stdout:5/965: readlink d2/d6/l31 0 2024-04-16T02:36:09.001 INFO:tasks.workunit.client.0.smithi151.stdout:3/674: dread d1/d6/d4a/f6f [0,4194304] 0 2024-04-16T02:36:09.002 INFO:tasks.workunit.client.0.smithi151.stdout:4/791: mknod d0/dde/dad/d106/c116 0 2024-04-16T02:36:09.004 INFO:tasks.workunit.client.0.smithi151.stdout:3/675: readlink d1/d6/l18 0 2024-04-16T02:36:09.004 INFO:tasks.workunit.client.0.smithi151.stdout:0/681: write dc/d15/f56 [4633968,30813] 0 2024-04-16T02:36:09.006 INFO:tasks.workunit.client.0.smithi151.stdout:7/940: write d0/d2c/d38/de7/d11f/d4c/d82/fca [5097546,96405] 0 2024-04-16T02:36:09.007 INFO:tasks.workunit.client.0.smithi151.stdout:9/835: dwrite d1/d3/d6/d17/f27 [0,4194304] 0 2024-04-16T02:36:09.013 INFO:tasks.workunit.client.0.smithi151.stdout:9/836: stat d1/d3/d6/d17/d40/d56/dad/daf 0 2024-04-16T02:36:09.015 INFO:tasks.workunit.client.0.smithi151.stdout:3/676: dread - d1/d6/d22/fab zero size 2024-04-16T02:36:09.017 INFO:tasks.workunit.client.0.smithi151.stdout:0/682: write dc/d4c/f96 [1717257,58651] 0 2024-04-16T02:36:09.022 INFO:tasks.workunit.client.0.smithi151.stdout:5/966: truncate d2/d6/db7/d12d/ddc/fa3 5207397 0 2024-04-16T02:36:09.023 INFO:tasks.workunit.client.0.smithi151.stdout:7/941: creat d0/d2c/d38/d85/db7/d110/f14b x:0 0 0 2024-04-16T02:36:09.024 INFO:tasks.workunit.client.0.smithi151.stdout:9/837: symlink d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/l11a 0 2024-04-16T02:36:09.025 INFO:tasks.workunit.client.0.smithi151.stdout:4/792: dwrite d0/d9b/ff7 [0,4194304] 0 2024-04-16T02:36:09.026 INFO:tasks.workunit.client.0.smithi151.stdout:8/715: link d11/d2a/d83/lc6 d11/lf9 0 2024-04-16T02:36:09.027 INFO:tasks.workunit.client.0.smithi151.stdout:0/683: dread - dc/d15/d34/fab zero size 2024-04-16T02:36:09.029 INFO:tasks.workunit.client.0.smithi151.stdout:1/764: link d3/d58/df/d32/d3a/d67/l102 d3/d58/df/d32/d3a/df3/d7a/dc7/l106 0 2024-04-16T02:36:09.030 INFO:tasks.workunit.client.0.smithi151.stdout:7/942: creat d0/d2c/d38/de7/d11f/da8/f14c x:0 0 0 2024-04-16T02:36:09.030 INFO:tasks.workunit.client.0.smithi151.stdout:9/838: chown d1/d3/d6/d28/d39/c3f 813157 1 2024-04-16T02:36:09.031 INFO:tasks.workunit.client.0.smithi151.stdout:4/793: creat d0/daa/dcc/f117 x:0 0 0 2024-04-16T02:36:09.032 INFO:tasks.workunit.client.0.smithi151.stdout:8/716: rmdir d11/d2a/d33/d4e/d62/dc8 39 2024-04-16T02:36:09.121 INFO:tasks.workunit.client.0.smithi151.stdout:7/943: stat d0/d2c/d38/de7/d11f/d4c/db6/ld4 0 2024-04-16T02:36:09.123 INFO:tasks.workunit.client.0.smithi151.stdout:7/944: chown d0/d2c/d38/d85/db7/d110 57 1 2024-04-16T02:36:09.124 INFO:tasks.workunit.client.0.smithi151.stdout:7/945: stat d0/d2c/d38/de7/d11f/d4c/d82/f13d 0 2024-04-16T02:36:09.125 INFO:tasks.workunit.client.0.smithi151.stdout:3/677: creat d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/fd9 x:0 0 0 2024-04-16T02:36:09.126 INFO:tasks.workunit.client.0.smithi151.stdout:7/946: write d0/d2c/d38/de7/d11f/f138 [663979,107285] 0 2024-04-16T02:36:09.127 INFO:tasks.workunit.client.0.smithi151.stdout:1/765: mkdir d3/d58/df/d32/d3a/df3/d7a/df9/d107 0 2024-04-16T02:36:09.132 INFO:tasks.workunit.client.0.smithi151.stdout:9/839: truncate d1/d3/d6/d17/d7a/f6d 2099982 0 2024-04-16T02:36:09.137 INFO:tasks.workunit.client.0.smithi151.stdout:8/717: dread d11/d2a/df4/f86 [0,4194304] 0 2024-04-16T02:36:09.139 INFO:tasks.workunit.client.0.smithi151.stdout:7/947: creat d0/d2c/d38/de7/d11f/d33/f14d x:0 0 0 2024-04-16T02:36:09.140 INFO:tasks.workunit.client.0.smithi151.stdout:4/794: dwrite d0/d11/d19/d87/d3a/d50/fca [0,4194304] 0 2024-04-16T02:36:09.143 INFO:tasks.workunit.client.0.smithi151.stdout:8/718: chown d11/d2a/d33/d4e/d62/d94 4900989 1 2024-04-16T02:36:09.145 INFO:tasks.workunit.client.0.smithi151.stdout:3/678: write d1/d6/d20/d3d/dd4/dd6/d44/faf [897222,23705] 0 2024-04-16T02:36:09.146 INFO:tasks.workunit.client.0.smithi151.stdout:7/948: stat d0/c17 0 2024-04-16T02:36:09.147 INFO:tasks.workunit.client.0.smithi151.stdout:1/766: dwrite d3/d58/df/d32/d3a/df3/d7a/f86 [4194304,4194304] 0 2024-04-16T02:36:09.151 INFO:tasks.workunit.client.0.smithi151.stdout:9/840: dwrite d1/d3/d5a/f82 [4194304,4194304] 0 2024-04-16T02:36:09.152 INFO:tasks.workunit.client.0.smithi151.stdout:4/795: truncate d0/fbf 2900489 0 2024-04-16T02:36:09.152 INFO:tasks.workunit.client.0.smithi151.stdout:3/679: dread - d1/d6/d22/fab zero size 2024-04-16T02:36:09.158 INFO:tasks.workunit.client.0.smithi151.stdout:3/680: chown d1/d6/d22/d6c/f9c 1774937 1 2024-04-16T02:36:09.159 INFO:tasks.workunit.client.0.smithi151.stdout:4/796: truncate d0/dde/f95 4133469 0 2024-04-16T02:36:09.161 INFO:tasks.workunit.client.0.smithi151.stdout:9/841: truncate d1/d3/d6/d17/d7a/fe4 224165 0 2024-04-16T02:36:09.161 INFO:tasks.workunit.client.0.smithi151.stdout:8/719: dwrite d11/d2a/d33/d4e/d62/d6b/db9/ff8 [0,4194304] 0 2024-04-16T02:36:09.170 INFO:tasks.workunit.client.0.smithi151.stdout:7/949: write d0/d2c/d38/de7/d11f/d31/f86 [924903,39131] 0 2024-04-16T02:36:09.172 INFO:tasks.workunit.client.0.smithi151.stdout:7/950: read d0/d2c/d38/de7/d11f/d46/f12c [3465353,120020] 0 2024-04-16T02:36:09.176 INFO:tasks.workunit.client.0.smithi151.stdout:1/767: rename d3/d58/df/d32/d3a/d67/l102 to d3/d58/df/d32/d3a/df3/d7a/df9/l108 0 2024-04-16T02:36:09.179 INFO:tasks.workunit.client.0.smithi151.stdout:9/842: rename d1/d3/d10/d7b/l104 to d1/d3/l11b 0 2024-04-16T02:36:09.182 INFO:tasks.workunit.client.0.smithi151.stdout:7/951: rename d0/d147 to d0/d146/d14e 0 2024-04-16T02:36:09.183 INFO:tasks.workunit.client.0.smithi151.stdout:7/952: chown d0/d2c/d38/de7/d11f/d4c/d82/db3 74668 1 2024-04-16T02:36:09.185 INFO:tasks.workunit.client.0.smithi151.stdout:9/843: unlink d1/d3/d6/d17/f31 0 2024-04-16T02:36:09.187 INFO:tasks.workunit.client.0.smithi151.stdout:7/953: fsync d0/d2c/d38/de7/d11f/d99/dc0/fd3 0 2024-04-16T02:36:09.190 INFO:tasks.workunit.client.0.smithi151.stdout:4/797: write d0/d99/f42 [1518471,32441] 0 2024-04-16T02:36:09.191 INFO:tasks.workunit.client.0.smithi151.stdout:8/720: write d11/d2a/df4/fbe [445003,93494] 0 2024-04-16T02:36:09.200 INFO:tasks.workunit.client.0.smithi151.stdout:3/681: dwrite d1/d6/d20/d3d/dd4/dd6/d44/d46/f7c [0,4194304] 0 2024-04-16T02:36:09.202 INFO:tasks.workunit.client.0.smithi151.stdout:7/954: link d0/d2c/d38/d5e/dd1/dee/fad d0/d2c/d38/d43/d122/d13a/f14f 0 2024-04-16T02:36:09.204 INFO:tasks.workunit.client.0.smithi151.stdout:9/844: dread d1/dd/d4a/d64/f76 [0,4194304] 0 2024-04-16T02:36:09.206 INFO:tasks.workunit.client.0.smithi151.stdout:4/798: write d0/d11/d19/f25 [5133362,58704] 0 2024-04-16T02:36:09.207 INFO:tasks.workunit.client.0.smithi151.stdout:7/955: dread d0/d2c/d38/de7/d11f/d94/fa1 [0,4194304] 0 2024-04-16T02:36:09.208 INFO:tasks.workunit.client.0.smithi151.stdout:8/721: link d11/d2a/d33/d4e/d62/d6b/c8d d11/d2a/d35/d52/db3/cfa 0 2024-04-16T02:36:09.210 INFO:tasks.workunit.client.0.smithi151.stdout:4/799: dread - d0/d64/fa9 zero size 2024-04-16T02:36:09.211 INFO:tasks.workunit.client.0.smithi151.stdout:4/800: stat d0/fbb 0 2024-04-16T02:36:09.212 INFO:tasks.workunit.client.0.smithi151.stdout:4/801: write d0/d11/f3d [3506529,97344] 0 2024-04-16T02:36:09.214 INFO:tasks.workunit.client.0.smithi151.stdout:3/682: write d1/d6/d22/fab [772045,59873] 0 2024-04-16T02:36:09.215 INFO:tasks.workunit.client.0.smithi151.stdout:8/722: stat d11/d2a/d35/d52/db3/cfa 0 2024-04-16T02:36:09.218 INFO:tasks.workunit.client.0.smithi151.stdout:8/723: truncate d11/f38 486381 0 2024-04-16T02:36:09.225 INFO:tasks.workunit.client.0.smithi151.stdout:3/683: rename d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/f57 to d1/d6/d4a/d4e/d77/fda 0 2024-04-16T02:36:09.226 INFO:tasks.workunit.client.0.smithi151.stdout:4/802: write d0/d11/d19/d87/d3a/d50/f67 [687010,47583] 0 2024-04-16T02:36:09.226 INFO:tasks.workunit.client.0.smithi151.stdout:9/845: dread d1/d3/d10/f4d [0,4194304] 0 2024-04-16T02:36:09.229 INFO:tasks.workunit.client.0.smithi151.stdout:4/803: stat d0/d64/l110 0 2024-04-16T02:36:09.231 INFO:tasks.workunit.client.0.smithi151.stdout:7/956: dwrite d0/ddf/ff8 [0,4194304] 0 2024-04-16T02:36:09.233 INFO:tasks.workunit.client.0.smithi151.stdout:9/846: fsync d1/d3/d6/d17/d54/d9b/fd6 0 2024-04-16T02:36:09.235 INFO:tasks.workunit.client.0.smithi151.stdout:8/724: dwrite d11/d2a/d33/d4e/d62/fd3 [4194304,4194304] 0 2024-04-16T02:36:09.238 INFO:tasks.workunit.client.0.smithi151.stdout:3/684: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/ldb 0 2024-04-16T02:36:09.238 INFO:tasks.workunit.client.0.smithi151.stdout:8/725: write d11/d2a/d33/d4e/ff6 [97739,129183] 0 2024-04-16T02:36:09.240 INFO:tasks.workunit.client.0.smithi151.stdout:3/685: stat d1/d6/d22/d23/d41/d98 0 2024-04-16T02:36:09.240 INFO:tasks.workunit.client.0.smithi151.stdout:7/957: creat d0/d2c/d38/de7/d11f/d4c/d82/db3/f150 x:0 0 0 2024-04-16T02:36:09.243 INFO:tasks.workunit.client.0.smithi151.stdout:8/726: creat d11/d2a/d85/ffb x:0 0 0 2024-04-16T02:36:09.245 INFO:tasks.workunit.client.0.smithi151.stdout:3/686: read - d1/d6/d20/d3d/fb8 zero size 2024-04-16T02:36:09.248 INFO:tasks.workunit.client.0.smithi151.stdout:4/804: dread d0/dde/d4a/d60/fda [0,4194304] 0 2024-04-16T02:36:09.252 INFO:tasks.workunit.client.0.smithi151.stdout:3/687: unlink d1/d89/daa/lc7 0 2024-04-16T02:36:09.252 INFO:tasks.workunit.client.0.smithi151.stdout:8/727: dwrite d11/d22/d2e/f3d [0,4194304] 0 2024-04-16T02:36:09.253 INFO:tasks.workunit.client.0.smithi151.stdout:4/805: creat d0/d9b/f118 x:0 0 0 2024-04-16T02:36:09.258 INFO:tasks.workunit.client.0.smithi151.stdout:8/728: unlink d11/d2a/d33/ff0 0 2024-04-16T02:36:09.261 INFO:tasks.workunit.client.0.smithi151.stdout:8/729: mknod d11/d2a/d83/dce/cfc 0 2024-04-16T02:36:09.262 INFO:tasks.workunit.client.0.smithi151.stdout:8/730: chown d11/d22/d2e/dc0/dbc 227089 1 2024-04-16T02:36:09.263 INFO:tasks.workunit.client.0.smithi151.stdout:4/806: creat d0/d11/d19/d46/dcd/def/f119 x:0 0 0 2024-04-16T02:36:09.264 INFO:tasks.workunit.client.0.smithi151.stdout:8/731: mkdir d11/d2a/d33/d4e/d62/d94/dfd 0 2024-04-16T02:36:09.265 INFO:tasks.workunit.client.0.smithi151.stdout:8/732: readlink d11/d2a/d85/lcf 0 2024-04-16T02:36:09.267 INFO:tasks.workunit.client.0.smithi151.stdout:9/847: write d1/d3/d6/d17/f107 [4884728,37010] 0 2024-04-16T02:36:09.267 INFO:tasks.workunit.client.0.smithi151.stdout:4/807: read d0/dde/d4a/d60/de6/f2b [2757198,44644] 0 2024-04-16T02:36:09.269 INFO:tasks.workunit.client.0.smithi151.stdout:7/958: truncate d0/d2c/d38/de7/d11f/d31/f86 3243781 0 2024-04-16T02:36:09.273 INFO:tasks.workunit.client.0.smithi151.stdout:3/688: write d1/d6/d4a/f6f [87578,17781] 0 2024-04-16T02:36:09.274 INFO:tasks.workunit.client.0.smithi151.stdout:3/689: readlink d1/l6e 0 2024-04-16T02:36:09.275 INFO:tasks.workunit.client.0.smithi151.stdout:8/733: dwrite d11/f17 [0,4194304] 0 2024-04-16T02:36:09.275 INFO:tasks.workunit.client.0.smithi151.stdout:7/959: truncate d0/f7 646433 0 2024-04-16T02:36:09.277 INFO:tasks.workunit.client.0.smithi151.stdout:7/960: stat d0/d2c/d38/de7/d11f/f29 0 2024-04-16T02:36:09.279 INFO:tasks.workunit.client.0.smithi151.stdout:9/848: mknod d1/d3/d6/d17/d54/d9b/d22/d36/c11c 0 2024-04-16T02:36:09.281 INFO:tasks.workunit.client.0.smithi151.stdout:9/849: fsync d1/d3/d6/d17/d40/d7d/da0/fdd 0 2024-04-16T02:36:09.281 INFO:tasks.workunit.client.0.smithi151.stdout:3/690: creat d1/d89/fdc x:0 0 0 2024-04-16T02:36:09.286 INFO:tasks.workunit.client.0.smithi151.stdout:4/808: write d0/d99/fa8 [556549,25567] 0 2024-04-16T02:36:09.288 INFO:tasks.workunit.client.0.smithi151.stdout:3/691: fsync d1/d6/f50 0 2024-04-16T02:36:09.294 INFO:tasks.workunit.client.0.smithi151.stdout:4/809: creat d0/d11/d19/d87/d7b/dfa/f11a x:0 0 0 2024-04-16T02:36:09.296 INFO:tasks.workunit.client.0.smithi151.stdout:8/734: write d11/d2a/df4/d78/fa8 [280650,87818] 0 2024-04-16T02:36:09.299 INFO:tasks.workunit.client.0.smithi151.stdout:9/850: getdents d1/d3/d6/d17/d7a/db1 0 2024-04-16T02:36:09.299 INFO:tasks.workunit.client.0.smithi151.stdout:7/961: write d0/d2c/d38/de7/d11f/d99/dc0/ff7 [113196,106183] 0 2024-04-16T02:36:09.303 INFO:tasks.workunit.client.0.smithi151.stdout:8/735: truncate d11/d2a/d33/d4e/d62/db7/ff5 585564 0 2024-04-16T02:36:09.303 INFO:tasks.workunit.client.0.smithi151.stdout:7/962: dread - d0/d2c/d38/de7/d11f/d33/d56/f119 zero size 2024-04-16T02:36:09.304 INFO:tasks.workunit.client.0.smithi151.stdout:9/851: creat d1/d3/d6/d17/d54/d9b/d22/d2c/f11d x:0 0 0 2024-04-16T02:36:09.308 INFO:tasks.workunit.client.0.smithi151.stdout:9/852: dread - d1/d3/d6/d17/d54/dd5/f115 zero size 2024-04-16T02:36:09.313 INFO:tasks.workunit.client.0.smithi151.stdout:9/853: chown d1/d3/d6/d17/d54/d9b/d22/d2c/d86/fd3 98144 1 2024-04-16T02:36:09.313 INFO:tasks.workunit.client.0.smithi151.stdout:7/963: dwrite d0/d2c/d38/de7/d11f/d31/d42/f45 [0,4194304] 0 2024-04-16T02:36:09.316 INFO:tasks.workunit.client.0.smithi151.stdout:4/810: dwrite d0/d11/d19/d87/d3a/f75 [0,4194304] 0 2024-04-16T02:36:09.316 INFO:tasks.workunit.client.0.smithi151.stdout:3/692: dread d1/d69/fad [4194304,4194304] 0 2024-04-16T02:36:09.320 INFO:tasks.workunit.client.0.smithi151.stdout:7/964: dread d0/d27/f11a [0,4194304] 0 2024-04-16T02:36:09.322 INFO:tasks.workunit.client.0.smithi151.stdout:9/854: mkdir d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/d11e 0 2024-04-16T02:36:09.323 INFO:tasks.workunit.client.0.smithi151.stdout:3/693: chown d1/d6/d20/d3d/c84 28072755 1 2024-04-16T02:36:09.327 INFO:tasks.workunit.client.0.smithi151.stdout:7/965: mkdir d0/d2c/d38/d5e/dd1/dee/dc9/d151 0 2024-04-16T02:36:09.329 INFO:tasks.workunit.client.0.smithi151.stdout:3/694: mknod d1/d6/d20/d3d/da6/dd3/cdd 0 2024-04-16T02:36:09.330 INFO:tasks.workunit.client.0.smithi151.stdout:8/736: dread d11/d22/d7e/fc1 [0,4194304] 0 2024-04-16T02:36:09.331 INFO:tasks.workunit.client.0.smithi151.stdout:7/966: creat d0/d2c/d38/de7/dfe/f152 x:0 0 0 2024-04-16T02:36:09.333 INFO:tasks.workunit.client.0.smithi151.stdout:8/737: fdatasync d11/f4a 0 2024-04-16T02:36:09.335 INFO:tasks.workunit.client.0.smithi151.stdout:8/738: truncate d11/d2a/df4/fbe 1390978 0 2024-04-16T02:36:09.338 INFO:tasks.workunit.client.0.smithi151.stdout:7/967: dwrite d0/d2c/d38/de7/d11f/d4c/d82/fca [4194304,4194304] 0 2024-04-16T02:36:09.339 INFO:tasks.workunit.client.0.smithi151.stdout:8/739: mknod d11/d2a/d33/d4e/d62/dc8/cfe 0 2024-04-16T02:36:09.341 INFO:tasks.workunit.client.0.smithi151.stdout:7/968: mknod d0/d27/da6/dcd/c153 0 2024-04-16T02:36:09.344 INFO:tasks.workunit.client.0.smithi151.stdout:8/740: creat d11/d2a/d33/d4e/d62/fff x:0 0 0 2024-04-16T02:36:09.346 INFO:tasks.workunit.client.0.smithi151.stdout:7/969: creat d0/d2c/d38/d5e/dd1/dee/dc9/d151/f154 x:0 0 0 2024-04-16T02:36:09.349 INFO:tasks.workunit.client.0.smithi151.stdout:8/741: creat d11/d2a/d33/d4e/f100 x:0 0 0 2024-04-16T02:36:09.349 INFO:tasks.workunit.client.0.smithi151.stdout:7/970: stat d0/d2c/d38/de7/d11f/da8/f14c 0 2024-04-16T02:36:09.350 INFO:tasks.workunit.client.0.smithi151.stdout:3/695: write d1/d69/fad [9430453,110367] 0 2024-04-16T02:36:09.354 INFO:tasks.workunit.client.0.smithi151.stdout:9/855: dwrite d1/d3/f5e [0,4194304] 0 2024-04-16T02:36:09.355 INFO:tasks.workunit.client.0.smithi151.stdout:8/742: write d11/d2a/d33/d4e/d62/d94/fa3 [194597,67934] 0 2024-04-16T02:36:09.356 INFO:tasks.workunit.client.0.smithi151.stdout:3/696: creat d1/d37/fde x:0 0 0 2024-04-16T02:36:09.358 INFO:tasks.workunit.client.0.smithi151.stdout:7/971: creat d0/d2c/d38/de7/d11f/da8/f155 x:0 0 0 2024-04-16T02:36:09.358 INFO:tasks.workunit.client.0.smithi151.stdout:8/743: creat d11/d2a/db1/f101 x:0 0 0 2024-04-16T02:36:09.359 INFO:tasks.workunit.client.0.smithi151.stdout:8/744: chown d11/d2a/db1 6 1 2024-04-16T02:36:09.366 INFO:tasks.workunit.client.0.smithi151.stdout:9/856: dwrite d1/d3/d10/d7b/f85 [0,4194304] 0 2024-04-16T02:36:09.416 INFO:tasks.workunit.client.0.smithi151.stdout:7/972: mkdir d0/d2c/d38/de7/d11f/d33/d89/d156 0 2024-04-16T02:36:09.418 INFO:tasks.workunit.client.0.smithi151.stdout:3/697: mknod d1/d6/d20/d3d/da6/cdf 0 2024-04-16T02:36:09.419 INFO:tasks.workunit.client.0.smithi151.stdout:9/857: fdatasync d1/dd/d4a/dbb/fc7 0 2024-04-16T02:36:09.422 INFO:tasks.workunit.client.0.smithi151.stdout:3/698: rename d1/d6/d22/d23/d66/f7b to d1/d37/fe0 0 2024-04-16T02:36:09.423 INFO:tasks.workunit.client.0.smithi151.stdout:7/973: creat d0/d2c/d38/de7/d11f/d33/f157 x:0 0 0 2024-04-16T02:36:09.425 INFO:tasks.workunit.client.0.smithi151.stdout:8/745: dwrite d11/d2a/f42 [4194304,4194304] 0 2024-04-16T02:36:09.502 INFO:tasks.workunit.client.0.smithi151.stdout:3/699: symlink d1/d6/d4a/d4e/le1 0 2024-04-16T02:36:09.505 INFO:tasks.workunit.client.0.smithi151.stdout:8/746: rmdir d11/d2a/d33/d4e/d62 39 2024-04-16T02:36:09.507 INFO:tasks.workunit.client.0.smithi151.stdout:9/858: link d1/d3/d6/cff d1/dd/d4a/dbb/c11f 0 2024-04-16T02:36:09.507 INFO:tasks.workunit.client.0.smithi151.stdout:9/859: chown d1/d3 14980 1 2024-04-16T02:36:09.511 INFO:tasks.workunit.client.0.smithi151.stdout:3/700: truncate d1/d6/d22/d23/d41/f5b 288686 0 2024-04-16T02:36:09.511 INFO:tasks.workunit.client.0.smithi151.stdout:7/974: write d0/d27/f9f [1031485,46694] 0 2024-04-16T02:36:09.513 INFO:tasks.workunit.client.0.smithi151.stdout:9/860: truncate d1/d75/fc9 4578771 0 2024-04-16T02:36:09.514 INFO:tasks.workunit.client.0.smithi151.stdout:3/701: mknod d1/d6/d20/d3d/da6/ce2 0 2024-04-16T02:36:09.515 INFO:tasks.workunit.client.0.smithi151.stdout:8/747: creat d11/d22/f102 x:0 0 0 2024-04-16T02:36:09.518 INFO:tasks.workunit.client.0.smithi151.stdout:7/975: rmdir d0/d2c/d38/d5e/dd1/dee/dc9/d151 39 2024-04-16T02:36:09.519 INFO:tasks.workunit.client.0.smithi151.stdout:7/976: chown d0/d2c/d38/de7/d11f/d46/d4f/f50 0 1 2024-04-16T02:36:09.524 INFO:tasks.workunit.client.0.smithi151.stdout:9/861: write d1/dd/d4a/f94 [456963,28379] 0 2024-04-16T02:36:09.525 INFO:tasks.workunit.client.0.smithi151.stdout:7/977: dread - d0/d2c/d38/d5e/dd1/dee/dc9/d151/f154 zero size 2024-04-16T02:36:09.526 INFO:tasks.workunit.client.0.smithi151.stdout:9/862: rename d1/d3/d6/d17 to d1/d3/d6/d17/d44/d120 22 2024-04-16T02:36:09.527 INFO:tasks.workunit.client.0.smithi151.stdout:8/748: write d11/d2a/d33/d4e/f77 [116991,96363] 0 2024-04-16T02:36:09.528 INFO:tasks.workunit.client.0.smithi151.stdout:7/978: mkdir d0/d2c/d38/d85/db7/d110/d158 0 2024-04-16T02:36:09.533 INFO:tasks.workunit.client.0.smithi151.stdout:7/979: truncate d0/d27/f9f 1820037 0 2024-04-16T02:36:09.534 INFO:tasks.workunit.client.0.smithi151.stdout:3/702: dwrite d1/d6/d20/d54/f65 [0,4194304] 0 2024-04-16T02:36:09.534 INFO:tasks.workunit.client.0.smithi151.stdout:9/863: write d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/f106 [1005489,123573] 0 2024-04-16T02:36:09.535 INFO:tasks.workunit.client.0.smithi151.stdout:8/749: mkdir d11/d2a/d33/d4e/d62/d6b/d103 0 2024-04-16T02:36:09.542 INFO:tasks.workunit.client.0.smithi151.stdout:7/980: symlink d0/d2c/d38/de7/d11f/d31/l159 0 2024-04-16T02:36:09.543 INFO:tasks.workunit.client.0.smithi151.stdout:8/750: symlink d11/d2a/d35/l104 0 2024-04-16T02:36:09.545 INFO:tasks.workunit.client.0.smithi151.stdout:9/864: dwrite d1/dd4/f114 [0,4194304] 0 2024-04-16T02:36:09.545 INFO:tasks.workunit.client.0.smithi151.stdout:7/981: symlink d0/d2c/d8e/l15a 0 2024-04-16T02:36:09.547 INFO:tasks.workunit.client.0.smithi151.stdout:3/703: creat d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/fe3 x:0 0 0 2024-04-16T02:36:09.552 INFO:tasks.workunit.client.0.smithi151.stdout:3/704: chown d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/f97 1450095831 1 2024-04-16T02:36:09.554 INFO:tasks.workunit.client.0.smithi151.stdout:8/751: dread d11/d22/d2e/f3d [0,4194304] 0 2024-04-16T02:36:09.557 INFO:tasks.workunit.client.0.smithi151.stdout:9/865: dwrite d1/d3/d6/d17/d7a/fe4 [0,4194304] 0 2024-04-16T02:36:09.558 INFO:tasks.workunit.client.0.smithi151.stdout:8/752: dread d11/d2a/d33/d4e/f77 [0,4194304] 0 2024-04-16T02:36:09.559 INFO:tasks.workunit.client.0.smithi151.stdout:3/705: creat d1/d6/d22/d23/fe4 x:0 0 0 2024-04-16T02:36:09.560 INFO:tasks.workunit.client.0.smithi151.stdout:8/753: chown d11/d2a/d35/d52/db3/fcb 11222984 1 2024-04-16T02:36:09.561 INFO:tasks.workunit.client.0.smithi151.stdout:9/866: creat d1/d3/d6/d17/d40/d7d/da0/f121 x:0 0 0 2024-04-16T02:36:09.562 INFO:tasks.workunit.client.0.smithi151.stdout:8/754: chown d11/d2a/d33/d4e/d62 56720 1 2024-04-16T02:36:09.563 INFO:tasks.workunit.client.0.smithi151.stdout:9/867: write d1/d3/d6/d17/d54/d9b/fd6 [2196642,29099] 0 2024-04-16T02:36:09.569 INFO:tasks.workunit.client.0.smithi151.stdout:7/982: truncate d0/d27/da6/dcd/f114 323208 0 2024-04-16T02:36:09.570 INFO:tasks.workunit.client.0.smithi151.stdout:9/868: mknod d1/d3/d6/d17/d54/c122 0 2024-04-16T02:36:09.571 INFO:tasks.workunit.client.0.smithi151.stdout:3/706: dwrite d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/fd9 [0,4194304] 0 2024-04-16T02:36:09.576 INFO:tasks.workunit.client.0.smithi151.stdout:8/755: link fb d11/d2a/d35/d52/db3/dcd/dd2/f105 0 2024-04-16T02:36:09.577 INFO:tasks.workunit.client.0.smithi151.stdout:3/707: creat d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/fe5 x:0 0 0 2024-04-16T02:36:09.582 INFO:tasks.workunit.client.0.smithi151.stdout:3/708: creat d1/d69/da4/fe6 x:0 0 0 2024-04-16T02:36:09.584 INFO:tasks.workunit.client.0.smithi151.stdout:8/756: creat d11/d22/d2e/dc0/dbc/f106 x:0 0 0 2024-04-16T02:36:09.585 INFO:tasks.workunit.client.0.smithi151.stdout:3/709: mknod d1/d69/d7f/dac/ce7 0 2024-04-16T02:36:09.592 INFO:tasks.workunit.client.0.smithi151.stdout:8/757: link d11/d22/d6c/d84/fc7 d11/d22/d6c/d84/dc9/f107 0 2024-04-16T02:36:09.596 INFO:tasks.workunit.client.0.smithi151.stdout:8/758: rmdir d11/d2a/d33/d4e/d62/d6b 39 2024-04-16T02:36:09.603 INFO:tasks.workunit.client.0.smithi151.stdout:9/869: dwrite d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/fc6 [0,4194304] 0 2024-04-16T02:36:09.603 INFO:tasks.workunit.client.0.smithi151.stdout:8/759: symlink d11/d2a/d33/d4e/d62/d94/l108 0 2024-04-16T02:36:09.605 INFO:tasks.workunit.client.0.smithi151.stdout:8/760: write d11/d2a/d35/d64/f58 [1138152,77382] 0 2024-04-16T02:36:09.613 INFO:tasks.workunit.client.0.smithi151.stdout:3/710: dread d1/d69/fad [0,4194304] 0 2024-04-16T02:36:09.613 INFO:tasks.workunit.client.0.smithi151.stdout:9/870: mkdir d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/dc3/d123 0 2024-04-16T02:36:09.615 INFO:tasks.workunit.client.0.smithi151.stdout:9/871: write d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/f106 [49337,4696] 0 2024-04-16T02:36:09.619 INFO:tasks.workunit.client.0.smithi151.stdout:8/761: symlink d11/d2a/l109 0 2024-04-16T02:36:09.619 INFO:tasks.workunit.client.0.smithi151.stdout:9/872: symlink d1/d3/d6/d17/d7a/db1/l124 0 2024-04-16T02:36:09.620 INFO:tasks.workunit.client.0.smithi151.stdout:8/762: stat d11/d2a/d33/d4e/ff7 0 2024-04-16T02:36:09.623 INFO:tasks.workunit.client.0.smithi151.stdout:8/763: chown d11/d2a/d33/d4e/d62/d94 7126964 1 2024-04-16T02:36:09.625 INFO:tasks.workunit.client.0.smithi151.stdout:3/711: dwrite d1/d69/fad [0,4194304] 0 2024-04-16T02:36:09.634 INFO:tasks.workunit.client.0.smithi151.stdout:9/873: dwrite d1/d3/d6/d17/d40/d56/ffc [4194304,4194304] 0 2024-04-16T02:36:09.638 INFO:tasks.workunit.client.0.smithi151.stdout:8/764: dwrite d11/fdd [0,4194304] 0 2024-04-16T02:36:09.642 INFO:tasks.workunit.client.0.smithi151.stdout:3/712: truncate d1/d6/d22/fbf 962960 0 2024-04-16T02:36:09.646 INFO:tasks.workunit.client.0.smithi151.stdout:9/874: getdents d1/dd4/df8 0 2024-04-16T02:36:09.647 INFO:tasks.workunit.client.0.smithi151.stdout:3/713: mknod d1/d69/da4/ce8 0 2024-04-16T02:36:09.650 INFO:tasks.workunit.client.0.smithi151.stdout:9/875: chown d1/d3/d6/d17/d54/d9b/d22/d2c/d86/cca 180356461 1 2024-04-16T02:36:09.651 INFO:tasks.workunit.client.0.smithi151.stdout:3/714: mknod d1/d37/d87/ce9 0 2024-04-16T02:36:09.656 INFO:tasks.workunit.client.0.smithi151.stdout:3/715: creat d1/d6/d20/d3d/dd4/dd6/d44/d46/fea x:0 0 0 2024-04-16T02:36:09.658 INFO:tasks.workunit.client.0.smithi151.stdout:9/876: rmdir d1/d3/d6/d17/d54/d9b/d22/d2c/d86/de7 0 2024-04-16T02:36:09.660 INFO:tasks.workunit.client.0.smithi151.stdout:3/716: rename d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/dd0 to d1/d89/deb 0 2024-04-16T02:36:09.661 INFO:tasks.workunit.client.0.smithi151.stdout:9/877: mknod d1/dd4/c125 0 2024-04-16T02:36:09.662 INFO:tasks.workunit.client.0.smithi151.stdout:3/717: symlink d1/d6/d22/d6c/lec 0 2024-04-16T02:36:09.669 INFO:tasks.workunit.client.0.smithi151.stdout:9/878: rename d1/d3/d6/d17/d54/d58/fa7 to d1/d3/d6/d17/db4/f126 0 2024-04-16T02:36:09.683 INFO:tasks.workunit.client.0.smithi151.stdout:8/765: dread d11/d22/f27 [0,4194304] 0 2024-04-16T02:36:09.701 INFO:tasks.workunit.client.0.smithi151.stdout:3/718: dread d1/d6/d20/d3d/dd4/dd6/f45 [0,4194304] 0 2024-04-16T02:36:09.705 INFO:tasks.workunit.client.0.smithi151.stdout:9/879: dwrite d1/d75/fbf [0,4194304] 0 2024-04-16T02:36:09.706 INFO:tasks.workunit.client.0.smithi151.stdout:3/719: fsync d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/fd9 0 2024-04-16T02:36:09.722 INFO:tasks.workunit.client.0.smithi151.stdout:8/766: mkdir d11/d2a/df4/d78/d10a 0 2024-04-16T02:36:09.724 INFO:tasks.workunit.client.0.smithi151.stdout:8/767: write d11/d2a/d35/d64/fe9 [312512,76636] 0 2024-04-16T02:36:09.726 INFO:tasks.workunit.client.0.smithi151.stdout:3/720: fsync d1/d6/d22/d23/d41/d9d/fa7 0 2024-04-16T02:36:09.729 INFO:tasks.workunit.client.0.smithi151.stdout:9/880: rename d1/d3/d6/d17/d54/d9b/d22/d2c/fd7 to d1/d3/d6/d17/d54/d8d/daa/f127 0 2024-04-16T02:36:09.731 INFO:tasks.workunit.client.0.smithi151.stdout:3/721: mknod d1/d6/d22/d23/ced 0 2024-04-16T02:36:09.734 INFO:tasks.workunit.client.0.smithi151.stdout:8/768: rename d11/d2a/d33/d4e/d62/fd3 to d11/d2a/d33/f10b 0 2024-04-16T02:36:09.734 INFO:tasks.workunit.client.0.smithi151.stdout:9/881: dread d1/d3/d10/d7b/f85 [0,4194304] 0 2024-04-16T02:36:09.742 INFO:tasks.workunit.client.0.smithi151.stdout:9/882: dwrite d1/d75/fbf [0,4194304] 0 2024-04-16T02:36:09.747 INFO:tasks.workunit.client.0.smithi151.stdout:8/769: getdents d11/d2a/d83/dce 0 2024-04-16T02:36:09.749 INFO:tasks.workunit.client.0.smithi151.stdout:3/722: dread d1/d6/f1e [0,4194304] 0 2024-04-16T02:36:09.751 INFO:tasks.workunit.client.0.smithi151.stdout:8/770: unlink d11/d22/d2e/fe0 0 2024-04-16T02:36:09.752 INFO:tasks.workunit.client.0.smithi151.stdout:3/723: dread d1/f3e [0,4194304] 0 2024-04-16T02:36:09.757 INFO:tasks.workunit.client.0.smithi151.stdout:3/724: dread d1/d6/d4a/fb1 [4194304,4194304] 0 2024-04-16T02:36:09.758 INFO:tasks.workunit.client.0.smithi151.stdout:8/771: unlink d11/d2a/d35/d64/c56 0 2024-04-16T02:36:09.759 INFO:tasks.workunit.client.0.smithi151.stdout:8/772: dread - d11/d2a/d85/f9f zero size 2024-04-16T02:36:09.760 INFO:tasks.workunit.client.0.smithi151.stdout:9/883: dread d1/d3/d6/d17/d54/d9b/d22/f23 [0,4194304] 0 2024-04-16T02:36:09.761 INFO:tasks.workunit.client.0.smithi151.stdout:3/725: creat d1/d69/d7f/fee x:0 0 0 2024-04-16T02:36:09.762 INFO:tasks.workunit.client.0.smithi151.stdout:9/884: stat d1/d3/d6/d17/f107 0 2024-04-16T02:36:09.763 INFO:tasks.workunit.client.0.smithi151.stdout:8/773: creat d11/d2a/d33/d4e/f10c x:0 0 0 2024-04-16T02:36:09.764 INFO:tasks.workunit.client.0.smithi151.stdout:3/726: mknod d1/d69/d7f/dac/cef 0 2024-04-16T02:36:09.766 INFO:tasks.workunit.client.0.smithi151.stdout:9/885: creat d1/dd/d4a/d64/f128 x:0 0 0 2024-04-16T02:36:09.768 INFO:tasks.workunit.client.0.smithi151.stdout:3/727: chown d1/d6/d22/c86 339275 1 2024-04-16T02:36:09.771 INFO:tasks.workunit.client.0.smithi151.stdout:3/728: dread - d1/d6/f8f zero size 2024-04-16T02:36:09.771 INFO:tasks.workunit.client.0.smithi151.stdout:8/774: link d11/d2a/db1/cbd d11/d2a/df4/d78/c10d 0 2024-04-16T02:36:09.774 INFO:tasks.workunit.client.0.smithi151.stdout:9/886: dread d1/d3/d6/d17/d54/d9b/d22/d2c/f93 [0,4194304] 0 2024-04-16T02:36:09.775 INFO:tasks.workunit.client.0.smithi151.stdout:8/775: creat d11/d2a/d85/f10e x:0 0 0 2024-04-16T02:36:09.776 INFO:tasks.workunit.client.0.smithi151.stdout:5/967: sync 2024-04-16T02:36:09.777 INFO:tasks.workunit.client.0.smithi151.stdout:8/776: write d11/d2a/d33/f45 [2008482,86434] 0 2024-04-16T02:36:09.780 INFO:tasks.workunit.client.0.smithi151.stdout:8/777: write d11/d2a/d33/d4e/f10c [1040344,18049] 0 2024-04-16T02:36:09.783 INFO:tasks.workunit.client.0.smithi151.stdout:3/729: rmdir d1/d6/d22/d23/d41/d98 0 2024-04-16T02:36:09.784 INFO:tasks.workunit.client.0.smithi151.stdout:9/887: dwrite d1/d3/d6/d17/d54/d9b/fd6 [4194304,4194304] 0 2024-04-16T02:36:09.785 INFO:tasks.workunit.client.0.smithi151.stdout:5/968: mknod d2/d6/df1/d11a/c136 0 2024-04-16T02:36:09.786 INFO:tasks.workunit.client.0.smithi151.stdout:3/730: fdatasync d1/d6/d4a/fb1 0 2024-04-16T02:36:09.788 INFO:tasks.workunit.client.0.smithi151.stdout:9/888: creat d1/d3/d6/d17/d54/dd5/f129 x:0 0 0 2024-04-16T02:36:09.790 INFO:tasks.workunit.client.0.smithi151.stdout:5/969: fsync d2/d6/db7/d12d/ddc/d83/d8b/ff4 0 2024-04-16T02:36:09.792 INFO:tasks.workunit.client.0.smithi151.stdout:9/889: dread - d1/d3/d6/d17/d54/d9b/d22/ffd zero size 2024-04-16T02:36:09.793 INFO:tasks.workunit.client.0.smithi151.stdout:3/731: creat d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/dd1/ff0 x:0 0 0 2024-04-16T02:36:09.794 INFO:tasks.workunit.client.0.smithi151.stdout:5/970: fdatasync d2/d6/d19/d30/f44 0 2024-04-16T02:36:09.796 INFO:tasks.workunit.client.0.smithi151.stdout:3/732: chown d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/c53 350 1 2024-04-16T02:36:09.798 INFO:tasks.workunit.client.0.smithi151.stdout:9/890: symlink d1/d3/d5f/de2/l12a 0 2024-04-16T02:36:09.803 INFO:tasks.workunit.client.0.smithi151.stdout:9/891: creat d1/d3/d6/d17/d54/dd5/f12b x:0 0 0 2024-04-16T02:36:09.804 INFO:tasks.workunit.client.0.smithi151.stdout:9/892: chown d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2 337724 1 2024-04-16T02:36:09.805 INFO:tasks.workunit.client.0.smithi151.stdout:9/893: stat d1/d3/d6/d17/cf5 0 2024-04-16T02:36:09.805 INFO:tasks.workunit.client.0.smithi151.stdout:5/971: link d2/d6/d19/d30/d49/ce5 d2/d10f/c137 0 2024-04-16T02:36:09.808 INFO:tasks.workunit.client.0.smithi151.stdout:5/972: mkdir d2/d6/d19/d62/dd0/d9c/db9/d138 0 2024-04-16T02:36:09.812 INFO:tasks.workunit.client.0.smithi151.stdout:9/894: dwrite d1/dd/d4a/dbb/fc7 [0,4194304] 0 2024-04-16T02:36:09.815 INFO:tasks.workunit.client.0.smithi151.stdout:5/973: unlink d2/d6/db7/d12d/dc4/lc6 0 2024-04-16T02:36:09.815 INFO:tasks.workunit.client.0.smithi151.stdout:9/895: fsync d1/d3/d6/d17/d54/d9b/dae/d6b/f77 0 2024-04-16T02:36:09.854 INFO:tasks.workunit.client.0.smithi151.stdout:5/974: fdatasync d2/d6/d19/d1a/f40 0 2024-04-16T02:36:09.863 INFO:tasks.workunit.client.0.smithi151.stdout:5/975: dwrite d2/d6/db7/d12d/ddc/d83/d11f/f133 [0,4194304] 0 2024-04-16T02:36:09.893 INFO:tasks.workunit.client.0.smithi151.stdout:9/896: creat d1/d3/d6/d17/d40/f12c x:0 0 0 2024-04-16T02:36:09.893 INFO:tasks.workunit.client.0.smithi151.stdout:3/733: write d1/d6/d22/d23/d41/d9d/fa7 [3692377,27351] 0 2024-04-16T02:36:09.901 INFO:tasks.workunit.client.0.smithi151.stdout:3/734: symlink d1/d89/daa/lf1 0 2024-04-16T02:36:09.903 INFO:tasks.workunit.client.0.smithi151.stdout:9/897: fdatasync d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/dc3/fe0 0 2024-04-16T02:36:09.905 INFO:tasks.workunit.client.0.smithi151.stdout:5/976: symlink d2/d6/db7/d12d/l139 0 2024-04-16T02:36:09.913 INFO:tasks.workunit.client.0.smithi151.stdout:5/977: creat d2/d6/d43/d9d/f13a x:0 0 0 2024-04-16T02:36:09.920 INFO:tasks.workunit.client.0.smithi151.stdout:5/978: dwrite d2/d6/d43/d9d/f13a [0,4194304] 0 2024-04-16T02:36:09.921 INFO:tasks.workunit.client.0.smithi151.stdout:5/979: chown d2/d6/d19/f87 325 1 2024-04-16T02:36:09.945 INFO:tasks.workunit.client.0.smithi151.stdout:9/898: creat d1/dd/f12d x:0 0 0 2024-04-16T02:36:09.946 INFO:tasks.workunit.client.0.smithi151.stdout:9/899: chown d1/d3/d6/d17/d54/d9b/d22/d36/f9e 11632 1 2024-04-16T02:36:09.954 INFO:tasks.workunit.client.0.smithi151.stdout:9/900: dwrite d1/d3/d6/d17/d54/d9b/d22/ffd [0,4194304] 0 2024-04-16T02:36:09.959 INFO:tasks.workunit.client.0.smithi151.stdout:9/901: mknod d1/d3/d6/d28/d39/c12e 0 2024-04-16T02:36:09.961 INFO:tasks.workunit.client.0.smithi151.stdout:9/902: chown d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/dc3 137620885 1 2024-04-16T02:36:09.962 INFO:tasks.workunit.client.0.smithi151.stdout:9/903: chown d1/d3/d6/d17/d54/dd5/f115 146322 1 2024-04-16T02:36:09.981 INFO:tasks.workunit.client.0.smithi151.stdout:9/904: rmdir d1 39 2024-04-16T02:36:09.987 INFO:tasks.workunit.client.0.smithi151.stdout:3/735: dwrite d1/d6/d20/d3d/fc1 [0,4194304] 0 2024-04-16T02:36:09.992 INFO:tasks.workunit.client.0.smithi151.stdout:9/905: dwrite d1/d3/f5e [0,4194304] 0 2024-04-16T02:36:09.995 INFO:tasks.workunit.client.0.smithi151.stdout:3/736: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/lf2 0 2024-04-16T02:36:09.999 INFO:tasks.workunit.client.0.smithi151.stdout:9/906: creat d1/d3/d6/d17/d54/d9b/dae/d6b/f12f x:0 0 0 2024-04-16T02:36:10.000 INFO:tasks.workunit.client.0.smithi151.stdout:4/811: sync 2024-04-16T02:36:10.000 INFO:tasks.workunit.client.0.smithi151.stdout:0/684: sync 2024-04-16T02:36:10.000 INFO:tasks.workunit.client.0.smithi151.stdout:1/768: sync 2024-04-16T02:36:10.000 INFO:tasks.workunit.client.0.smithi151.stdout:7/983: sync 2024-04-16T02:36:10.001 INFO:tasks.workunit.client.0.smithi151.stdout:1/769: chown d3/d58 907 1 2024-04-16T02:36:10.006 INFO:tasks.workunit.client.0.smithi151.stdout:1/770: creat d3/d33/d10/d4f/f109 x:0 0 0 2024-04-16T02:36:10.008 INFO:tasks.workunit.client.0.smithi151.stdout:1/771: write d3/d58/df/d32/d3a/df3/d52/f72 [712848,57396] 0 2024-04-16T02:36:10.008 INFO:tasks.workunit.client.0.smithi151.stdout:1/772: fdatasync d3/d58/ffa 0 2024-04-16T02:36:10.023 INFO:tasks.workunit.client.0.smithi151.stdout:7/984: truncate d0/d27/da6/dcd/f114 1285050 0 2024-04-16T02:36:10.024 INFO:tasks.workunit.client.0.smithi151.stdout:9/907: dread d1/d3/f1e [0,4194304] 0 2024-04-16T02:36:10.025 INFO:tasks.workunit.client.0.smithi151.stdout:9/908: fdatasync d1/dd/d4a/d64/f128 0 2024-04-16T02:36:10.026 INFO:tasks.workunit.client.0.smithi151.stdout:0/685: symlink dc/d15/d34/ld7 0 2024-04-16T02:36:10.031 INFO:tasks.workunit.client.0.smithi151.stdout:1/773: creat d3/d58/df/d32/d3a/d71/f10a x:0 0 0 2024-04-16T02:36:10.037 INFO:tasks.workunit.client.0.smithi151.stdout:9/909: link d1/dd4/f114 d1/d3/d6/d17/d40/d7d/da0/f130 0 2024-04-16T02:36:10.040 INFO:tasks.workunit.client.0.smithi151.stdout:3/737: dread d1/d6/d4a/d4e/d77/fda [0,4194304] 0 2024-04-16T02:36:10.042 INFO:tasks.workunit.client.0.smithi151.stdout:7/985: getdents d0/d27/da6/dcd 0 2024-04-16T02:36:10.043 INFO:tasks.workunit.client.0.smithi151.stdout:3/738: fdatasync d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/f15 0 2024-04-16T02:36:10.048 INFO:tasks.workunit.client.0.smithi151.stdout:7/986: dread - d0/d27/f83 zero size 2024-04-16T02:36:10.049 INFO:tasks.workunit.client.0.smithi151.stdout:1/774: link d3/d58/df/d32/d3a/d6d/l77 d3/d58/df/d32/d3a/df3/d7a/df9/d107/l10b 0 2024-04-16T02:36:10.051 INFO:tasks.workunit.client.0.smithi151.stdout:3/739: symlink d1/d6/d20/d3d/dd4/dd6/d44/lf3 0 2024-04-16T02:36:10.052 INFO:tasks.workunit.client.0.smithi151.stdout:3/740: readlink d1/lb 0 2024-04-16T02:36:10.054 INFO:tasks.workunit.client.0.smithi151.stdout:9/910: dread d1/d3/d6/d17/d54/d9b/fd6 [0,4194304] 0 2024-04-16T02:36:10.056 INFO:tasks.workunit.client.0.smithi151.stdout:3/741: creat d1/d6/d20/d3d/ff4 x:0 0 0 2024-04-16T02:36:10.059 INFO:tasks.workunit.client.0.smithi151.stdout:1/775: truncate d3/d58/df/d32/d3a/df3/d73/f85 47658 0 2024-04-16T02:36:10.060 INFO:tasks.workunit.client.0.smithi151.stdout:4/812: truncate d0/fb 6089958 0 2024-04-16T02:36:10.061 INFO:tasks.workunit.client.0.smithi151.stdout:7/987: getdents d0/d2c/d38/de7/d11f/d46 0 2024-04-16T02:36:10.063 INFO:tasks.workunit.client.0.smithi151.stdout:7/988: fdatasync d0/d2c/d38/de7/d11f/d99/dc0/f134 0 2024-04-16T02:36:10.065 INFO:tasks.workunit.client.0.smithi151.stdout:1/776: rmdir d3/d33 39 2024-04-16T02:36:10.066 INFO:tasks.workunit.client.0.smithi151.stdout:4/813: mknod d0/d9b/c11b 0 2024-04-16T02:36:10.068 INFO:tasks.workunit.client.0.smithi151.stdout:3/742: creat d1/d6/d20/d3d/da6/dd8/ff5 x:0 0 0 2024-04-16T02:36:10.070 INFO:tasks.workunit.client.0.smithi151.stdout:7/989: unlink d0/d2c/d38/c8a 0 2024-04-16T02:36:10.072 INFO:tasks.workunit.client.0.smithi151.stdout:1/777: mknod d3/d58/df/d32/d3a/df3/d7a/c10c 0 2024-04-16T02:36:10.073 INFO:tasks.workunit.client.0.smithi151.stdout:4/814: fsync d0/d11/d19/d87/d3a/d50/f51 0 2024-04-16T02:36:10.074 INFO:tasks.workunit.client.0.smithi151.stdout:4/815: chown d0/d9b/ff7 0 1 2024-04-16T02:36:10.075 INFO:tasks.workunit.client.0.smithi151.stdout:9/911: link d1/d3/d6/d17/d7a/f6d d1/d3/d6/d17/d7a/db1/f131 0 2024-04-16T02:36:10.077 INFO:tasks.workunit.client.0.smithi151.stdout:7/990: truncate d0/d2c/d38/d43/d78/f9a 2749819 0 2024-04-16T02:36:10.080 INFO:tasks.workunit.client.0.smithi151.stdout:1/778: unlink d3/d58/ffa 0 2024-04-16T02:36:10.080 INFO:tasks.workunit.client.0.smithi151.stdout:9/912: dread d1/d75/fbf [0,4194304] 0 2024-04-16T02:36:10.089 INFO:tasks.workunit.client.0.smithi151.stdout:1/779: fsync d3/d4/d64/d40/fbb 0 2024-04-16T02:36:10.092 INFO:tasks.workunit.client.0.smithi151.stdout:9/913: chown d1/dd/d4a/dbb/c11f 818267 1 2024-04-16T02:36:10.093 INFO:tasks.workunit.client.0.smithi151.stdout:9/914: dread - d1/d3/d6/d17/d54/d8d/f90 zero size 2024-04-16T02:36:10.096 INFO:tasks.workunit.client.0.smithi151.stdout:3/743: write d1/f5c [1667854,90573] 0 2024-04-16T02:36:10.098 INFO:tasks.workunit.client.0.smithi151.stdout:1/780: creat d3/d58/df/d32/d3a/d71/f10d x:0 0 0 2024-04-16T02:36:10.102 INFO:tasks.workunit.client.0.smithi151.stdout:4/816: write d0/d9b/faf [638798,5096] 0 2024-04-16T02:36:10.105 INFO:tasks.workunit.client.0.smithi151.stdout:1/781: chown d3/d33/c63 0 1 2024-04-16T02:36:10.106 INFO:tasks.workunit.client.0.smithi151.stdout:4/817: dread - d0/d11/d19/dd7/fe7 zero size 2024-04-16T02:36:10.107 INFO:tasks.workunit.client.0.smithi151.stdout:9/915: creat d1/dd/f132 x:0 0 0 2024-04-16T02:36:10.109 INFO:tasks.workunit.client.0.smithi151.stdout:7/991: write d0/d2c/d38/de7/d11f/d31/f86 [755702,74641] 0 2024-04-16T02:36:10.113 INFO:tasks.workunit.client.0.smithi151.stdout:9/916: rmdir d1/d3/d6/d17/d54/d8d 39 2024-04-16T02:36:10.114 INFO:tasks.workunit.client.0.smithi151.stdout:4/818: dwrite d0/d11/f10a [0,4194304] 0 2024-04-16T02:36:10.115 INFO:tasks.workunit.client.0.smithi151.stdout:9/917: fdatasync d1/dd/d4a/d64/f80 0 2024-04-16T02:36:10.117 INFO:tasks.workunit.client.0.smithi151.stdout:3/744: getdents d1/d6/d20/d3d/dd4/dd6/d44/d63 0 2024-04-16T02:36:10.118 INFO:tasks.workunit.client.0.smithi151.stdout:1/782: mkdir d3/d58/df/dbc/d10e 0 2024-04-16T02:36:10.119 INFO:tasks.workunit.client.0.smithi151.stdout:7/992: mkdir d0/ddf/d15b 0 2024-04-16T02:36:10.121 INFO:tasks.workunit.client.0.smithi151.stdout:4/819: unlink d0/d11/d19/f1b 0 2024-04-16T02:36:10.124 INFO:tasks.workunit.client.0.smithi151.stdout:9/918: symlink d1/d3/d6/d17/d44/l133 0 2024-04-16T02:36:10.126 INFO:tasks.workunit.client.0.smithi151.stdout:1/783: fsync d3/d58/df/d32/d3a/f66 0 2024-04-16T02:36:10.129 INFO:tasks.workunit.client.0.smithi151.stdout:4/820: stat d0/dde/f95 0 2024-04-16T02:36:10.130 INFO:tasks.workunit.client.0.smithi151.stdout:1/784: chown d3/d58/df/d32/d3a/df3/d7e/la0 90403 1 2024-04-16T02:36:10.130 INFO:tasks.workunit.client.0.smithi151.stdout:9/919: dread d1/d3/d10/f4d [0,4194304] 0 2024-04-16T02:36:10.131 INFO:tasks.workunit.client.0.smithi151.stdout:3/745: symlink d1/d6/lf6 0 2024-04-16T02:36:10.137 INFO:tasks.workunit.client.0.smithi151.stdout:9/920: creat d1/d3/d6/d17/d54/d9b/dae/d6b/f134 x:0 0 0 2024-04-16T02:36:10.137 INFO:tasks.workunit.client.0.smithi151.stdout:7/993: write d0/d27/f95 [4892,19630] 0 2024-04-16T02:36:10.139 INFO:tasks.workunit.client.0.smithi151.stdout:3/746: truncate d1/d37/f3c 747599 0 2024-04-16T02:36:10.140 INFO:tasks.workunit.client.0.smithi151.stdout:1/785: symlink d3/d4/d64/d40/l10f 0 2024-04-16T02:36:10.143 INFO:tasks.workunit.client.0.smithi151.stdout:7/994: truncate d0/d2c/d38/de7/d11f/f29 192344 0 2024-04-16T02:36:10.144 INFO:tasks.workunit.client.0.smithi151.stdout:7/995: fdatasync d0/d27/da6/fd6 0 2024-04-16T02:36:10.145 INFO:tasks.workunit.client.0.smithi151.stdout:1/786: creat d3/d58/df/f110 x:0 0 0 2024-04-16T02:36:10.146 INFO:tasks.workunit.client.0.smithi151.stdout:3/747: getdents d1/d89 0 2024-04-16T02:36:10.148 INFO:tasks.workunit.client.0.smithi151.stdout:7/996: creat d0/d2c/d38/de7/f15c x:0 0 0 2024-04-16T02:36:10.150 INFO:tasks.workunit.client.0.smithi151.stdout:4/821: dread d0/d11/d19/d46/dcd/fea [0,4194304] 0 2024-04-16T02:36:10.155 INFO:tasks.workunit.client.0.smithi151.stdout:9/921: write d1/d75/fbf [2298364,784] 0 2024-04-16T02:36:10.158 INFO:tasks.workunit.client.0.smithi151.stdout:4/822: creat d0/d11/d19/d46/dcd/f11c x:0 0 0 2024-04-16T02:36:10.160 INFO:tasks.workunit.client.0.smithi151.stdout:3/748: link d1/d6/d4a/d4e/d77/fda d1/d89/daa/ff7 0 2024-04-16T02:36:10.161 INFO:tasks.workunit.client.0.smithi151.stdout:3/749: truncate d1/d69/da4/fe6 190294 0 2024-04-16T02:36:10.162 INFO:tasks.workunit.client.0.smithi151.stdout:9/922: mkdir d1/d3/d6/d17/d54/d9b/d22/d36/d135 0 2024-04-16T02:36:10.164 INFO:tasks.workunit.client.0.smithi151.stdout:4/823: symlink d0/dde/dad/d106/l11d 0 2024-04-16T02:36:10.166 INFO:tasks.workunit.client.0.smithi151.stdout:7/997: creat d0/d2c/d38/de7/d11f/d33/d89/f15d x:0 0 0 2024-04-16T02:36:10.166 INFO:tasks.workunit.client.0.smithi151.stdout:7/998: chown d0/d2c/d38/db0/f112 77 1 2024-04-16T02:36:10.167 INFO:tasks.workunit.client.0.smithi151.stdout:3/750: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/lf8 0 2024-04-16T02:36:10.170 INFO:tasks.workunit.client.0.smithi151.stdout:4/824: symlink d0/dde/dad/d106/l11e 0 2024-04-16T02:36:10.172 INFO:tasks.workunit.client.0.smithi151.stdout:7/999: creat d0/d2c/d38/d5e/dd1/f15e x:0 0 0 2024-04-16T02:36:10.173 INFO:tasks.workunit.client.0.smithi151.stdout:3/751: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/dd1/lf9 0 2024-04-16T02:36:10.174 INFO:tasks.workunit.client.0.smithi151.stdout:1/787: dread d3/d58/df/d32/d3a/d6d/f7b [0,4194304] 0 2024-04-16T02:36:10.175 INFO:tasks.workunit.client.0.smithi151.stdout:3/752: read d1/d6/d20/d54/f80 [450478,16680] 0 2024-04-16T02:36:10.176 INFO:tasks.workunit.client.0.smithi151.stdout:1/788: read - d3/d4/db2/fc3 zero size 2024-04-16T02:36:10.177 INFO:tasks.workunit.client.0.smithi151.stdout:3/753: stat d1/d6/d22/d23/d41/f76 0 2024-04-16T02:36:10.178 INFO:tasks.workunit.client.0.smithi151.stdout:1/789: mknod d3/d58/c111 0 2024-04-16T02:36:10.218 INFO:tasks.workunit.client.0.smithi151.stdout:9/923: truncate d1/d3/d6/d17/d7a/db1/fa1 3636980 0 2024-04-16T02:36:10.218 INFO:tasks.workunit.client.0.smithi151.stdout:4/825: write d0/d99/f66 [461726,82765] 0 2024-04-16T02:36:10.220 INFO:tasks.workunit.client.0.smithi151.stdout:1/790: truncate d3/f8a 4137458 0 2024-04-16T02:36:10.221 INFO:tasks.workunit.client.0.smithi151.stdout:9/924: chown d1/d3/d6/d17/d54/d9b/d22/c26 130299 1 2024-04-16T02:36:10.222 INFO:tasks.workunit.client.0.smithi151.stdout:4/826: read d0/d11/d19/d46/f77 [288524,34490] 0 2024-04-16T02:36:10.223 INFO:tasks.workunit.client.0.smithi151.stdout:9/925: write d1/d3/d6/d17/d54/d9b/d22/d36/f9e [598700,50489] 0 2024-04-16T02:36:10.224 INFO:tasks.workunit.client.0.smithi151.stdout:3/754: write d1/d6/d22/fbf [267569,36052] 0 2024-04-16T02:36:10.226 INFO:tasks.workunit.client.0.smithi151.stdout:1/791: creat d3/d58/df/dbc/d10e/f112 x:0 0 0 2024-04-16T02:36:10.229 INFO:tasks.workunit.client.0.smithi151.stdout:4/827: readlink d0/dde/dad/d106/lb4 0 2024-04-16T02:36:10.230 INFO:tasks.workunit.client.0.smithi151.stdout:9/926: mkdir d1/d3/d5f/de2/d136 0 2024-04-16T02:36:10.232 INFO:tasks.workunit.client.0.smithi151.stdout:1/792: mknod d3/d4/db2/c113 0 2024-04-16T02:36:10.233 INFO:tasks.workunit.client.0.smithi151.stdout:3/755: dwrite d1/f5c [0,4194304] 0 2024-04-16T02:36:10.237 INFO:tasks.workunit.client.0.smithi151.stdout:3/756: read d1/d6/d20/d3d/dd4/dd6/d44/f5f [5862793,59348] 0 2024-04-16T02:36:10.240 INFO:tasks.workunit.client.0.smithi151.stdout:9/927: dwrite d1/d3/d6/d17/d44/ff1 [0,4194304] 0 2024-04-16T02:36:10.243 INFO:tasks.workunit.client.0.smithi151.stdout:4/828: write d0/dde/f95 [2938288,5171] 0 2024-04-16T02:36:10.243 INFO:tasks.workunit.client.0.smithi151.stdout:1/793: creat d3/d58/df/f114 x:0 0 0 2024-04-16T02:36:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:09 smithi151 ceph-mon[29003]: pgmap v32: 65 pgs: 65 active+clean; 2.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 43 MiB/s rd, 111 MiB/s wr, 332 op/s 2024-04-16T02:36:10.246 INFO:tasks.workunit.client.0.smithi151.stdout:9/928: creat d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/f137 x:0 0 0 2024-04-16T02:36:10.248 INFO:tasks.workunit.client.0.smithi151.stdout:4/829: mkdir d0/daa/d11f 0 2024-04-16T02:36:10.252 INFO:tasks.workunit.client.0.smithi151.stdout:4/830: dread d0/d64/f76 [4194304,4194304] 0 2024-04-16T02:36:10.254 INFO:tasks.workunit.client.0.smithi151.stdout:9/929: dread d1/d75/fbf [0,4194304] 0 2024-04-16T02:36:10.256 INFO:tasks.workunit.client.0.smithi151.stdout:4/831: creat d0/d11/d19/dd7/f120 x:0 0 0 2024-04-16T02:36:10.258 INFO:tasks.workunit.client.0.smithi151.stdout:9/930: mkdir d1/d3/d6/d17/d40/d7d/dfb/d138 0 2024-04-16T02:36:10.259 INFO:tasks.workunit.client.0.smithi151.stdout:4/832: read - d0/dde/d4a/d60/de6/f52 zero size 2024-04-16T02:36:10.260 INFO:tasks.workunit.client.0.smithi151.stdout:9/931: chown d1/dd/d4a/c6c 62090 1 2024-04-16T02:36:10.263 INFO:tasks.workunit.client.0.smithi151.stdout:9/932: chown d1/d3/d6/d17/d54/d9b/d22/d36/f3d 2 1 2024-04-16T02:36:10.265 INFO:tasks.workunit.client.0.smithi151.stdout:1/794: dwrite d3/d58/df/d32/d3a/df3/f60 [0,4194304] 0 2024-04-16T02:36:10.267 INFO:tasks.workunit.client.0.smithi151.stdout:3/757: dread d1/d6/d20/fc5 [0,4194304] 0 2024-04-16T02:36:10.268 INFO:tasks.workunit.client.0.smithi151.stdout:9/933: creat d1/d3/d6/d17/d54/d9b/d22/d2c/d86/f139 x:0 0 0 2024-04-16T02:36:10.270 INFO:tasks.workunit.client.0.smithi151.stdout:1/795: mknod d3/d58/df/d32/d3a/df3/c115 0 2024-04-16T02:36:10.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:09 smithi153 ceph-mon[36747]: pgmap v32: 65 pgs: 65 active+clean; 2.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 43 MiB/s rd, 111 MiB/s wr, 332 op/s 2024-04-16T02:36:10.273 INFO:tasks.workunit.client.0.smithi151.stdout:9/934: mkdir d1/d3/d6/d17/d40/d7d/dfb/d13a 0 2024-04-16T02:36:10.274 INFO:tasks.workunit.client.0.smithi151.stdout:4/833: write d0/dde/d4a/d60/de6/f52 [563442,49123] 0 2024-04-16T02:36:10.275 INFO:tasks.workunit.client.0.smithi151.stdout:9/935: write d1/d3/d5f/f10b [449203,122954] 0 2024-04-16T02:36:10.277 INFO:tasks.workunit.client.0.smithi151.stdout:1/796: rename d3/d58/df/d32/d3a/df3/d52/fa9 to d3/d4/d64/d1c/f116 0 2024-04-16T02:36:10.279 INFO:tasks.workunit.client.0.smithi151.stdout:3/758: link d1/d6/f8f d1/d6/d20/d3d/dd4/dd6/ffa 0 2024-04-16T02:36:10.280 INFO:tasks.workunit.client.0.smithi151.stdout:4/834: creat d0/d99/f121 x:0 0 0 2024-04-16T02:36:10.281 INFO:tasks.workunit.client.0.smithi151.stdout:3/759: rmdir d1/d6/d22/d6c 39 2024-04-16T02:36:10.283 INFO:tasks.workunit.client.0.smithi151.stdout:1/797: symlink d3/d58/df/d32/d3a/df3/d73/l117 0 2024-04-16T02:36:10.289 INFO:tasks.workunit.client.0.smithi151.stdout:1/798: rename d3/d58/df/d32/d3a/d67/dcb/cd5 to d3/d58/df/d32/d3a/d67/c118 0 2024-04-16T02:36:10.295 INFO:tasks.workunit.client.0.smithi151.stdout:3/760: truncate d1/d37/d87/f8b 7555592 0 2024-04-16T02:36:10.296 INFO:tasks.workunit.client.0.smithi151.stdout:1/799: write d3/d58/df/d32/f88 [4502339,12438] 0 2024-04-16T02:36:10.301 INFO:tasks.workunit.client.0.smithi151.stdout:9/936: dwrite d1/d3/d6/d17/d40/d56/dad/fb5 [0,4194304] 0 2024-04-16T02:36:10.304 INFO:tasks.workunit.client.0.smithi151.stdout:4/835: dread d0/d99/f21 [0,4194304] 0 2024-04-16T02:36:10.306 INFO:tasks.workunit.client.0.smithi151.stdout:1/800: mknod d3/d58/df/d32/d3a/df3/d52/d97/c119 0 2024-04-16T02:36:10.308 INFO:tasks.workunit.client.0.smithi151.stdout:4/836: read d0/dde/dad/ff5 [1725428,73276] 0 2024-04-16T02:36:10.311 INFO:tasks.workunit.client.0.smithi151.stdout:3/761: dwrite d1/d6/f31 [0,4194304] 0 2024-04-16T02:36:10.366 INFO:tasks.workunit.client.0.smithi151.stdout:3/762: link d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/f35 d1/d6/d20/d3d/da6/dd8/ffb 0 2024-04-16T02:36:10.366 INFO:tasks.workunit.client.0.smithi151.stdout:9/937: write d1/d75/fc9 [2927804,66666] 0 2024-04-16T02:36:10.366 INFO:tasks.workunit.client.0.smithi151.stdout:4/837: write d0/d11/d19/d46/dcd/def/dfe/f103 [2891305,12874] 0 2024-04-16T02:36:10.370 INFO:tasks.workunit.client.0.smithi151.stdout:3/763: rename d1/d6/d22/d23/f70 to d1/d69/d7f/ffc 0 2024-04-16T02:36:10.371 INFO:tasks.workunit.client.0.smithi151.stdout:3/764: chown d1/d6/d22/fbf 2337 1 2024-04-16T02:36:10.372 INFO:tasks.workunit.client.0.smithi151.stdout:9/938: mknod d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/c13b 0 2024-04-16T02:36:10.373 INFO:tasks.workunit.client.0.smithi151.stdout:3/765: fdatasync d1/d6/d20/d3d/dd4/dd6/d44/d46/fea 0 2024-04-16T02:36:10.375 INFO:tasks.workunit.client.0.smithi151.stdout:9/939: rmdir d1/d3/d6/d17/d54/dd5 39 2024-04-16T02:36:10.379 INFO:tasks.workunit.client.0.smithi151.stdout:3/766: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/lfd 0 2024-04-16T02:36:10.380 INFO:tasks.workunit.client.0.smithi151.stdout:9/940: symlink d1/d3/d6/d17/d54/d9b/d22/l13c 0 2024-04-16T02:36:10.381 INFO:tasks.workunit.client.0.smithi151.stdout:4/838: dwrite d0/fb [4194304,4194304] 0 2024-04-16T02:36:10.384 INFO:tasks.workunit.client.0.smithi151.stdout:9/941: fdatasync d1/d3/d6/d17/d54/d9b/d22/d2c/d86/fd3 0 2024-04-16T02:36:10.386 INFO:tasks.workunit.client.0.smithi151.stdout:3/767: link d1/d37/f38 d1/d6/d4a/d4e/ffe 0 2024-04-16T02:36:10.387 INFO:tasks.workunit.client.0.smithi151.stdout:9/942: stat d1/d3/d6/d17/d40/d56/dad 0 2024-04-16T02:36:10.388 INFO:tasks.workunit.client.0.smithi151.stdout:3/768: write d1/f26 [5907342,105] 0 2024-04-16T02:36:10.388 INFO:tasks.workunit.client.0.smithi151.stdout:9/943: chown d1/d3/d6/d17/d44 243259184 1 2024-04-16T02:36:10.392 INFO:tasks.workunit.client.0.smithi151.stdout:3/769: mknod d1/d69/d7f/dac/cff 0 2024-04-16T02:36:10.394 INFO:tasks.workunit.client.0.smithi151.stdout:9/944: write d1/dd/f51 [5212411,99298] 0 2024-04-16T02:36:10.395 INFO:tasks.workunit.client.0.smithi151.stdout:3/770: fsync d1/d6/d4a/d4e/d77/fda 0 2024-04-16T02:36:10.396 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-16T02:36:10.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-16T02:36:10.400 INFO:tasks.workunit.client.0.smithi151.stdout:4/839: dwrite d0/d11/d19/d87/d3a/d50/fdb [0,4194304] 0 2024-04-16T02:36:10.400 INFO:tasks.workunit.client.0.smithi151.stdout:3/771: symlink d1/d6/l100 0 2024-04-16T02:36:10.402 INFO:tasks.workunit.client.0.smithi151.stdout:9/945: creat d1/d3/d6/d17/d54/f13d x:0 0 0 2024-04-16T02:36:10.405 INFO:tasks.workunit.client.0.smithi151.stdout:4/840: creat d0/d11/d19/d87/d3a/d65/d98/df8/f122 x:0 0 0 2024-04-16T02:36:10.407 INFO:tasks.workunit.client.0.smithi151.stdout:9/946: rename d1/d3/d6/d17/d7a/db1 to d1/d3/d6/d17/d40/d7d/da0/d13e 0 2024-04-16T02:36:10.409 INFO:tasks.workunit.client.0.smithi151.stdout:3/772: write d1/f3e [1164650,108171] 0 2024-04-16T02:36:10.410 INFO:tasks.workunit.client.0.smithi151.stdout:9/947: symlink d1/d3/d5f/de2/l13f 0 2024-04-16T02:36:10.412 INFO:tasks.workunit.client.0.smithi151.stdout:4/841: link d0/d64/f76 d0/dde/f123 0 2024-04-16T02:36:10.412 INFO:tasks.workunit.client.0.smithi151.stdout:3/773: chown d1/d69/d7f/dac/cbd 2 1 2024-04-16T02:36:10.414 INFO:tasks.workunit.client.0.smithi151.stdout:4/842: mkdir d0/daa/dcc/d124 0 2024-04-16T02:36:10.420 INFO:tasks.workunit.client.0.smithi151.stdout:4/843: dwrite d0/d99/fa8 [0,4194304] 0 2024-04-16T02:36:10.421 INFO:tasks.workunit.client.0.smithi151.stdout:3/774: write d1/d37/f5e [953662,968] 0 2024-04-16T02:36:10.422 INFO:tasks.workunit.client.0.smithi151.stdout:9/948: truncate d1/d3/d6/d17/d40/d56/dad/fb5 2603115 0 2024-04-16T02:36:10.427 INFO:tasks.workunit.client.0.smithi151.stdout:9/949: mknod d1/d3/d6/d17/d54/d8d/daa/d9d/c140 0 2024-04-16T02:36:10.427 INFO:tasks.workunit.client.0.smithi151.stdout:4/844: getdents d0/d11/d19/d87/d3a/d50/dc7 0 2024-04-16T02:36:10.429 INFO:tasks.workunit.client.0.smithi151.stdout:9/950: creat d1/d3/d6/d17/d7a/f141 x:0 0 0 2024-04-16T02:36:10.439 INFO:tasks.workunit.client.0.smithi151.stdout:4/845: write d0/d11/d19/d87/d3a/d50/fdc [463383,116011] 0 2024-04-16T02:36:10.444 INFO:tasks.workunit.client.0.smithi151.stdout:9/951: dwrite d1/d3/d6/d17/d40/d7d/da0/d13e/f131 [0,4194304] 0 2024-04-16T02:36:10.447 INFO:tasks.workunit.client.0.smithi151.stdout:9/952: fsync d1/d3/d6/d17/d54/d9b/dae/d6b/f134 0 2024-04-16T02:36:10.448 INFO:tasks.workunit.client.0.smithi151.stdout:9/953: dread - d1/dd/d4a/f116 zero size 2024-04-16T02:36:10.449 INFO:tasks.workunit.client.0.smithi151.stdout:3/775: dread d1/d6/f21 [0,4194304] 0 2024-04-16T02:36:10.449 INFO:tasks.workunit.client.0.smithi151.stdout:4/846: dwrite d0/d11/d19/d46/f58 [0,4194304] 0 2024-04-16T02:36:10.450 INFO:tasks.workunit.client.0.smithi151.stdout:9/954: write d1/d3/d6/f13 [4262204,106835] 0 2024-04-16T02:36:10.452 INFO:tasks.workunit.client.0.smithi151.stdout:9/955: chown d1/d3/d5f/fe6 232275782 1 2024-04-16T02:36:10.455 INFO:tasks.workunit.client.0.smithi151.stdout:4/847: symlink d0/d11/d19/d46/dcd/def/dfe/l125 0 2024-04-16T02:36:10.459 INFO:tasks.workunit.client.0.smithi151.stdout:9/956: symlink d1/d3/d6/d17/d7a/l142 0 2024-04-16T02:36:10.459 INFO:tasks.workunit.client.0.smithi151.stdout:3/776: dwrite d1/d6/d20/d54/f65 [0,4194304] 0 2024-04-16T02:36:10.460 INFO:tasks.workunit.client.0.smithi151.stdout:4/848: write d0/d11/d19/d87/d3a/d50/fdb [2863244,117778] 0 2024-04-16T02:36:10.464 INFO:tasks.workunit.client.0.smithi151.stdout:3/777: creat d1/d69/f101 x:0 0 0 2024-04-16T02:36:10.466 INFO:tasks.workunit.client.0.smithi151.stdout:9/957: rename d1/l4 to d1/dd/d38/l143 0 2024-04-16T02:36:10.467 INFO:tasks.workunit.client.0.smithi151.stdout:3/778: unlink d1/d6/d22/d23/f62 0 2024-04-16T02:36:10.473 INFO:tasks.workunit.client.0.smithi151.stdout:3/779: dwrite d1/d69/f101 [0,4194304] 0 2024-04-16T02:36:10.475 INFO:tasks.workunit.client.0.smithi151.stdout:4/849: dread d0/d11/d19/d46/f77 [0,4194304] 0 2024-04-16T02:36:10.480 INFO:tasks.workunit.client.0.smithi151.stdout:3/780: dread d1/d6/d20/d54/f80 [0,4194304] 0 2024-04-16T02:36:10.623 INFO:tasks.workunit.client.0.smithi151.stdout:3/781: creat d1/d6/d20/d3d/dd4/dd6/d44/d63/f102 x:0 0 0 2024-04-16T02:36:10.624 INFO:tasks.workunit.client.0.smithi151.stdout:3/782: dread - d1/d89/fdc zero size 2024-04-16T02:36:10.624 INFO:tasks.workunit.client.0.smithi151.stdout:4/850: dwrite d0/dde/dad/ff5 [0,4194304] 0 2024-04-16T02:36:10.626 INFO:tasks.workunit.client.0.smithi151.stdout:3/783: stat d1/d6/d20/d3d/c84 0 2024-04-16T02:36:10.628 INFO:tasks.workunit.client.0.smithi151.stdout:9/958: dread d1/d3/f5e [0,4194304] 0 2024-04-16T02:36:10.630 INFO:tasks.workunit.client.0.smithi151.stdout:3/784: rmdir d1/d69/da4 39 2024-04-16T02:36:10.630 INFO:tasks.workunit.client.0.smithi151.stdout:9/959: write d1/d3/d5f/fe6 [366291,126431] 0 2024-04-16T02:36:10.637 INFO:tasks.workunit.client.0.smithi151.stdout:3/785: creat d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/f103 x:0 0 0 2024-04-16T02:36:10.639 INFO:tasks.workunit.client.0.smithi151.stdout:4/851: dread d0/dde/f123 [0,4194304] 0 2024-04-16T02:36:10.645 INFO:tasks.workunit.client.0.smithi151.stdout:9/960: dwrite d1/d3/d10/f4d [4194304,4194304] 0 2024-04-16T02:36:10.648 INFO:tasks.workunit.client.0.smithi151.stdout:3/786: truncate d1/f5c 3736821 0 2024-04-16T02:36:10.649 INFO:tasks.workunit.client.0.smithi151.stdout:9/961: truncate d1/d3/d6/d17/d40/d7d/da0/f121 228114 0 2024-04-16T02:36:10.655 INFO:tasks.workunit.client.0.smithi151.stdout:3/787: rename d1/d6/d22/f4d to d1/d6/d4a/d4e/f104 0 2024-04-16T02:36:10.657 INFO:tasks.workunit.client.0.smithi151.stdout:3/788: truncate d1/d6/d22/d23/fb9 600225 0 2024-04-16T02:36:10.658 INFO:tasks.workunit.client.0.smithi151.stdout:9/962: dwrite d1/d3/d6/d17/f107 [0,4194304] 0 2024-04-16T02:36:10.661 INFO:tasks.workunit.client.0.smithi151.stdout:4/852: dread d0/f1f [0,4194304] 0 2024-04-16T02:36:10.661 INFO:tasks.workunit.client.0.smithi151.stdout:9/963: stat d1/dd/d4a/l60 0 2024-04-16T02:36:10.662 INFO:tasks.workunit.client.0.smithi151.stdout:3/789: creat d1/d6/d20/d3d/dd4/dd6/f105 x:0 0 0 2024-04-16T02:36:10.669 INFO:tasks.workunit.client.0.smithi151.stdout:9/964: truncate d1/d3/d6/d17/d40/d7d/da0/fdd 327933 0 2024-04-16T02:36:10.672 INFO:tasks.workunit.client.0.smithi151.stdout:3/790: rename d1/d6/d22/d23/d41/f8d to d1/d6/d4a/f106 0 2024-04-16T02:36:10.676 INFO:tasks.workunit.client.0.smithi151.stdout:9/965: rename d1/d3/d10 to d1/d3/d6/d17/d54/d8d/daa/d9d/d144 0 2024-04-16T02:36:10.678 INFO:tasks.workunit.client.0.smithi151.stdout:3/791: truncate d1/d89/fc0 467389 0 2024-04-16T02:36:10.687 INFO:tasks.workunit.client.0.smithi151.stdout:4/853: write d0/d11/f37 [846043,122631] 0 2024-04-16T02:36:10.697 INFO:tasks.workunit.client.0.smithi151.stdout:3/792: dwrite d1/d37/fb7 [0,4194304] 0 2024-04-16T02:36:10.730 INFO:tasks.workunit.client.0.smithi151.stdout:3/793: mkdir d1/d6/d20/d3d/dd4/d107 0 2024-04-16T02:36:10.731 INFO:tasks.workunit.client.0.smithi151.stdout:3/794: fdatasync d1/d6/d20/d3d/dd4/dd6/d44/d46/f7c 0 2024-04-16T02:36:10.732 INFO:tasks.workunit.client.0.smithi151.stdout:8/778: sync 2024-04-16T02:36:10.733 INFO:tasks.workunit.client.0.smithi151.stdout:5/980: sync 2024-04-16T02:36:10.733 INFO:tasks.workunit.client.0.smithi151.stdout:0/686: sync 2024-04-16T02:36:10.733 INFO:tasks.workunit.client.0.smithi151.stdout:3/795: write d1/fcd [608862,104722] 0 2024-04-16T02:36:10.736 INFO:tasks.workunit.client.0.smithi151.stdout:9/966: link d1/d3/d6/d17/le1 d1/d3/d6/d17/d40/d7d/dfb/d138/l145 0 2024-04-16T02:36:10.740 INFO:tasks.workunit.client.0.smithi151.stdout:8/779: creat d11/d2a/d35/d64/f10f x:0 0 0 2024-04-16T02:36:10.743 INFO:tasks.workunit.client.0.smithi151.stdout:0/687: rename dc/d15/d34/d3b/f44 to dc/d20/d7b/d9c/fd8 0 2024-04-16T02:36:10.743 INFO:tasks.workunit.client.0.smithi151.stdout:0/688: chown dc/d15/d34/d3b/lce 770202 1 2024-04-16T02:36:10.749 INFO:tasks.workunit.client.0.smithi151.stdout:5/981: unlink d2/d6/f29 0 2024-04-16T02:36:10.750 INFO:tasks.workunit.client.0.smithi151.stdout:8/780: creat d11/d2a/df4/f110 x:0 0 0 2024-04-16T02:36:10.751 INFO:tasks.workunit.client.0.smithi151.stdout:0/689: mkdir dc/d20/d7b/d9c/dd6/dd9 0 2024-04-16T02:36:10.752 INFO:tasks.workunit.client.0.smithi151.stdout:8/781: read d11/d2a/d33/d4e/d62/db7/ff5 [273522,3775] 0 2024-04-16T02:36:10.754 INFO:tasks.workunit.client.0.smithi151.stdout:9/967: link d1/d3/d6/d17/d54/d8d/daa/d9d/d144/f118 d1/d3/d6/d17/d54/d58/dde/f146 0 2024-04-16T02:36:10.758 INFO:tasks.workunit.client.0.smithi151.stdout:8/782: rename d11/d2a/d33/d4e/d62/db7/fec to d11/d22/d7e/f111 0 2024-04-16T02:36:10.761 INFO:tasks.workunit.client.0.smithi151.stdout:0/690: symlink dc/d15/d34/lda 0 2024-04-16T02:36:10.764 INFO:tasks.workunit.client.0.smithi151.stdout:5/982: rename d2/d6/d19/d30/l111 to d2/d6/d19/d30/d49/de7/l13b 0 2024-04-16T02:36:10.765 INFO:tasks.workunit.client.0.smithi151.stdout:5/983: chown d2/d6/d19/f134 2608365 1 2024-04-16T02:36:10.766 INFO:tasks.workunit.client.0.smithi151.stdout:8/783: mknod d11/d22/d2e/dc0/c112 0 2024-04-16T02:36:10.769 INFO:tasks.workunit.client.0.smithi151.stdout:0/691: rename dc/d15/l31 to dc/d20/d7b/d9c/dd6/ldb 0 2024-04-16T02:36:10.772 INFO:tasks.workunit.client.0.smithi151.stdout:9/968: getdents d1/d3/d6/d17/d40/d7d/dfb 0 2024-04-16T02:36:10.775 INFO:tasks.workunit.client.0.smithi151.stdout:9/969: creat d1/dd/d4a/dbb/f147 x:0 0 0 2024-04-16T02:36:10.776 INFO:tasks.workunit.client.0.smithi151.stdout:5/984: getdents d2/d6/d19/d62/dd0/d9c/db9 0 2024-04-16T02:36:10.780 INFO:tasks.workunit.client.0.smithi151.stdout:5/985: creat d2/d6/db7/f13c x:0 0 0 2024-04-16T02:36:10.781 INFO:tasks.workunit.client.0.smithi151.stdout:9/970: mkdir d1/d3/d6/d17/d54/d9b/d22/d36/d135/d148 0 2024-04-16T02:36:10.782 INFO:tasks.workunit.client.0.smithi151.stdout:5/986: write d2/d6/d19/ff0 [121325,115591] 0 2024-04-16T02:36:10.787 INFO:tasks.workunit.client.0.smithi151.stdout:9/971: chown d1/d3/fa 1 1 2024-04-16T02:36:10.789 INFO:tasks.workunit.client.0.smithi151.stdout:4/854: dwrite d0/d11/f23 [0,4194304] 0 2024-04-16T02:36:10.789 INFO:tasks.workunit.client.0.smithi151.stdout:3/796: write d1/d37/f42 [260634,22990] 0 2024-04-16T02:36:10.793 INFO:tasks.workunit.client.0.smithi151.stdout:3/797: dread - d1/d6/d20/d3d/dd4/dd6/d44/d63/f102 zero size 2024-04-16T02:36:10.799 INFO:tasks.workunit.client.0.smithi151.stdout:4/855: dwrite d0/d11/d19/d87/d3a/d65/d98/df8/f122 [0,4194304] 0 2024-04-16T02:36:10.806 INFO:tasks.workunit.client.0.smithi151.stdout:3/798: dread d1/d6/d22/d23/f5a [0,4194304] 0 2024-04-16T02:36:10.808 INFO:tasks.workunit.client.0.smithi151.stdout:3/799: chown d1/d6/d20/d3d/dd4/dd6/d44/d46/c71 1834 1 2024-04-16T02:36:10.810 INFO:tasks.workunit.client.0.smithi151.stdout:4/856: truncate d0/d11/d19/d87/d3a/fe8 66036 0 2024-04-16T02:36:10.811 INFO:tasks.workunit.client.0.smithi151.stdout:3/800: creat d1/d6/d22/d23/d41/d9d/f108 x:0 0 0 2024-04-16T02:36:10.815 INFO:tasks.workunit.client.0.smithi151.stdout:5/987: write d2/f16 [9384148,109270] 0 2024-04-16T02:36:10.816 INFO:tasks.workunit.client.0.smithi151.stdout:8/784: dwrite d11/d2a/d33/f4b [0,4194304] 0 2024-04-16T02:36:10.817 INFO:tasks.workunit.client.0.smithi151.stdout:5/988: chown d2/d6/d19/c9b 4 1 2024-04-16T02:36:10.823 INFO:tasks.workunit.client.0.smithi151.stdout:3/801: mknod d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/c109 0 2024-04-16T02:36:10.824 INFO:tasks.workunit.client.0.smithi151.stdout:9/972: dwrite d1/d3/d6/d17/d54/d9b/d22/d2c/f93 [0,4194304] 0 2024-04-16T02:36:10.827 INFO:tasks.workunit.client.0.smithi151.stdout:9/973: chown d1/d3/d6/d17/d54/d9b/d22/d36/f102 9 1 2024-04-16T02:36:10.827 INFO:tasks.workunit.client.0.smithi151.stdout:8/785: mkdir d11/d2a/d33/d4e/d62/d6b/d113 0 2024-04-16T02:36:10.833 INFO:tasks.workunit.client.0.smithi151.stdout:8/786: symlink d11/d2a/d33/d4e/d62/db7/l114 0 2024-04-16T02:36:10.834 INFO:tasks.workunit.client.0.smithi151.stdout:8/787: fdatasync d11/f1a 0 2024-04-16T02:36:10.840 INFO:tasks.workunit.client.0.smithi151.stdout:9/974: dwrite d1/d75/f8b [4194304,4194304] 0 2024-04-16T02:36:10.840 INFO:tasks.workunit.client.0.smithi151.stdout:4/857: truncate d0/dde/d4a/d60/f115 2611757 0 2024-04-16T02:36:10.848 INFO:tasks.workunit.client.0.smithi151.stdout:3/802: truncate d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/fbe 1490819 0 2024-04-16T02:36:10.849 INFO:tasks.workunit.client.0.smithi151.stdout:4/858: truncate d0/d11/d19/d87/f48 2493055 0 2024-04-16T02:36:10.850 INFO:tasks.workunit.client.0.smithi151.stdout:4/859: read d0/dde/d4a/d60/de6/f2b [2903396,92372] 0 2024-04-16T02:36:10.853 INFO:tasks.workunit.client.0.smithi151.stdout:4/860: write d0/d11/d19/f25 [3413352,56416] 0 2024-04-16T02:36:10.857 INFO:tasks.workunit.client.0.smithi151.stdout:3/803: dwrite d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/fe3 [0,4194304] 0 2024-04-16T02:36:10.864 INFO:tasks.workunit.client.0.smithi151.stdout:4/861: readlink d0/d11/d19/d87/d3a/le1 0 2024-04-16T02:36:10.868 INFO:tasks.workunit.client.0.smithi151.stdout:4/862: mknod d0/d99/d59/c126 0 2024-04-16T02:36:10.869 INFO:tasks.workunit.client.0.smithi151.stdout:8/788: dwrite d11/d2a/f2c [0,4194304] 0 2024-04-16T02:36:10.871 INFO:tasks.workunit.client.0.smithi151.stdout:3/804: rename d1/d6/d20/d3d/dd4/dd6/l94 to d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/l10a 0 2024-04-16T02:36:10.872 INFO:tasks.workunit.client.0.smithi151.stdout:8/789: mknod d11/d2a/d85/c115 0 2024-04-16T02:36:10.873 INFO:tasks.workunit.client.0.smithi151.stdout:3/805: chown d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/f16 1 1 2024-04-16T02:36:10.875 INFO:tasks.workunit.client.0.smithi151.stdout:3/806: read d1/d6/d20/d3d/fc1 [3388525,42438] 0 2024-04-16T02:36:10.876 INFO:tasks.workunit.client.0.smithi151.stdout:3/807: fdatasync d1/d6/d22/d23/fb3 0 2024-04-16T02:36:10.877 INFO:tasks.workunit.client.0.smithi151.stdout:8/790: rename d11/l25 to d11/d2a/d35/d6e/l116 0 2024-04-16T02:36:10.878 INFO:tasks.workunit.client.0.smithi151.stdout:8/791: read - d11/d22/d7e/f111 zero size 2024-04-16T02:36:10.881 INFO:tasks.workunit.client.0.smithi151.stdout:8/792: dread d11/d22/d2e/dc0/ff2 [0,4194304] 0 2024-04-16T02:36:10.891 INFO:tasks.workunit.client.0.smithi151.stdout:4/863: write d0/d11/d19/d87/d3a/d50/f51 [1090127,42029] 0 2024-04-16T02:36:10.894 INFO:tasks.workunit.client.0.smithi151.stdout:3/808: truncate d1/d6/d22/f2c 6003553 0 2024-04-16T02:36:10.895 INFO:tasks.workunit.client.0.smithi151.stdout:3/809: read - d1/d6/d22/d23/d41/d9d/f108 zero size 2024-04-16T02:36:10.896 INFO:tasks.workunit.client.0.smithi151.stdout:8/793: dread d11/f16 [0,4194304] 0 2024-04-16T02:36:10.897 INFO:tasks.workunit.client.0.smithi151.stdout:8/794: stat d11/d2a/d35/d52/db3/fcb 0 2024-04-16T02:36:10.898 INFO:tasks.workunit.client.0.smithi151.stdout:8/795: chown d11/d2a/db1/f101 3 1 2024-04-16T02:36:10.900 INFO:tasks.workunit.client.0.smithi151.stdout:3/810: dread d1/d6/d4a/fb1 [4194304,4194304] 0 2024-04-16T02:36:10.900 INFO:tasks.workunit.client.0.smithi151.stdout:4/864: link d0/d11/f26 d0/daa/d11f/f127 0 2024-04-16T02:36:10.901 INFO:tasks.workunit.client.0.smithi151.stdout:8/796: symlink d11/d2a/d35/d6e/l117 0 2024-04-16T02:36:10.902 INFO:tasks.workunit.client.0.smithi151.stdout:8/797: chown d11/d2a/d33/l57 0 1 2024-04-16T02:36:10.903 INFO:tasks.workunit.client.0.smithi151.stdout:4/865: rename d0/d64/fd6 to d0/daa/f128 0 2024-04-16T02:36:10.905 INFO:tasks.workunit.client.0.smithi151.stdout:3/811: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/l10b 0 2024-04-16T02:36:10.906 INFO:tasks.workunit.client.0.smithi151.stdout:8/798: creat d11/d2a/d83/dce/f118 x:0 0 0 2024-04-16T02:36:10.909 INFO:tasks.workunit.client.0.smithi151.stdout:3/812: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/l10c 0 2024-04-16T02:36:10.910 INFO:tasks.workunit.client.0.smithi151.stdout:3/813: fsync d1/d89/f8e 0 2024-04-16T02:36:10.913 INFO:tasks.workunit.client.0.smithi151.stdout:4/866: creat d0/d11/d19/d87/f129 x:0 0 0 2024-04-16T02:36:10.916 INFO:tasks.workunit.client.0.smithi151.stdout:3/814: creat d1/d89/deb/f10d x:0 0 0 2024-04-16T02:36:10.922 INFO:tasks.workunit.client.0.smithi151.stdout:3/815: rename d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/fb0 to d1/d6/d20/d3d/da6/f10e 0 2024-04-16T02:36:10.923 INFO:tasks.workunit.client.0.smithi151.stdout:3/816: write d1/d69/d7f/fee [148978,91241] 0 2024-04-16T02:36:10.926 INFO:tasks.workunit.client.0.smithi151.stdout:4/867: getdents d0/dde/d4a 0 2024-04-16T02:36:10.931 INFO:tasks.workunit.client.0.smithi151.stdout:3/817: truncate d1/d6/d4a/fb1 7014789 0 2024-04-16T02:36:10.933 INFO:tasks.workunit.client.0.smithi151.stdout:4/868: creat d0/daa/dcc/d124/f12a x:0 0 0 2024-04-16T02:36:10.934 INFO:tasks.workunit.client.0.smithi151.stdout:4/869: chown d0/dde/dad/d106/fb1 789055 1 2024-04-16T02:36:10.935 INFO:tasks.workunit.client.0.smithi151.stdout:4/870: write d0/d11/d19/d87/d3a/d50/fdb [527874,40558] 0 2024-04-16T02:36:10.943 INFO:tasks.workunit.client.0.smithi151.stdout:4/871: dread d0/f3 [0,4194304] 0 2024-04-16T02:36:10.946 INFO:tasks.workunit.client.0.smithi151.stdout:3/818: dread d1/f26 [0,4194304] 0 2024-04-16T02:36:10.946 INFO:tasks.workunit.client.0.smithi151.stdout:4/872: symlink d0/d11/d19/d87/d3a/d50/dc7/l12b 0 2024-04-16T02:36:10.947 INFO:tasks.workunit.client.0.smithi151.stdout:4/873: chown d0/dde/d68/f7a 50276 1 2024-04-16T02:36:10.948 INFO:tasks.workunit.client.0.smithi151.stdout:4/874: write d0/d9b/faf [2009004,6186] 0 2024-04-16T02:36:10.950 INFO:tasks.workunit.client.0.smithi151.stdout:3/819: truncate d1/d37/f38 4688691 0 2024-04-16T02:36:10.953 INFO:tasks.workunit.client.0.smithi151.stdout:3/820: dread d1/d6/d20/fc5 [0,4194304] 0 2024-04-16T02:36:10.953 INFO:tasks.workunit.client.0.smithi151.stdout:4/875: link d0/d9b/faf d0/dde/d4a/f12c 0 2024-04-16T02:36:10.955 INFO:tasks.workunit.client.0.smithi151.stdout:4/876: mknod d0/d73/c12d 0 2024-04-16T02:36:10.956 INFO:tasks.workunit.client.0.smithi151.stdout:3/821: creat d1/d6/d20/d3d/dd4/dd6/f10f x:0 0 0 2024-04-16T02:36:10.957 INFO:tasks.workunit.client.0.smithi151.stdout:3/822: chown d1/d6/d4a/d4e/le1 39381 1 2024-04-16T02:36:10.960 INFO:tasks.workunit.client.0.smithi151.stdout:4/877: creat d0/d11/d19/d46/f12e x:0 0 0 2024-04-16T02:36:10.964 INFO:tasks.workunit.client.0.smithi151.stdout:3/823: dread d1/d6/f50 [0,4194304] 0 2024-04-16T02:36:10.964 INFO:tasks.workunit.client.0.smithi151.stdout:4/878: symlink d0/d11/d19/d87/d3a/l12f 0 2024-04-16T02:36:10.966 INFO:tasks.workunit.client.0.smithi151.stdout:3/824: mkdir d1/d6/d22/d23/d66/d110 0 2024-04-16T02:36:10.968 INFO:tasks.workunit.client.0.smithi151.stdout:3/825: write d1/d6/d22/d23/d41/d9d/f108 [588023,20786] 0 2024-04-16T02:36:10.975 INFO:tasks.workunit.client.0.smithi151.stdout:3/826: dwrite d1/d6/d22/d23/d41/d9d/fa7 [4194304,4194304] 0 2024-04-16T02:36:10.978 INFO:tasks.workunit.client.0.smithi151.stdout:3/827: chown d1/d6/d20/d3d/da6/f10e 408684 1 2024-04-16T02:36:10.985 INFO:tasks.workunit.client.0.smithi151.stdout:3/828: getdents d1/d89/deb 0 2024-04-16T02:36:10.987 INFO:tasks.workunit.client.0.smithi151.stdout:3/829: write d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/f16 [1093494,29160] 0 2024-04-16T02:36:10.989 INFO:tasks.workunit.client.0.smithi151.stdout:3/830: chown d1/d6/d20/d3d/dd4/dd6/d44/f5f 3 1 2024-04-16T02:36:10.989 INFO:tasks.workunit.client.0.smithi151.stdout:3/831: chown d1 204 1 2024-04-16T02:36:10.990 INFO:tasks.workunit.client.0.smithi151.stdout:1/801: sync 2024-04-16T02:36:10.990 INFO:tasks.workunit.client.0.smithi151.stdout:3/832: truncate d1/d89/fdc 203456 0 2024-04-16T02:36:10.993 INFO:tasks.workunit.client.0.smithi151.stdout:1/802: creat d3/d58/df/d68/f11a x:0 0 0 2024-04-16T02:36:10.994 INFO:tasks.workunit.client.0.smithi151.stdout:4/879: dread d0/dde/d4a/f12c [0,4194304] 0 2024-04-16T02:36:10.994 INFO:tasks.workunit.client.0.smithi151.stdout:3/833: rmdir d1/d6/d20/d3d/da6/dd8 39 2024-04-16T02:36:10.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:10 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:36:11.015 INFO:tasks.workunit.client.0.smithi151.stdout:1/803: symlink d3/d58/df/d32/d3a/df3/d7a/l11b 0 2024-04-16T02:36:11.015 INFO:tasks.workunit.client.0.smithi151.stdout:4/880: readlink d0/d11/d19/dd7/lfc 0 2024-04-16T02:36:11.015 INFO:tasks.workunit.client.0.smithi151.stdout:3/834: dread - d1/d6/d20/d3d/dd4/dd6/f105 zero size 2024-04-16T02:36:11.016 INFO:tasks.workunit.client.0.smithi151.stdout:4/881: stat d0/d11/d19/d87/d3a/d50/l57 0 2024-04-16T02:36:11.018 INFO:tasks.workunit.client.0.smithi151.stdout:1/804: mknod d3/d58/df/d68/c11c 0 2024-04-16T02:36:11.020 INFO:tasks.workunit.client.0.smithi151.stdout:3/835: mkdir d1/d6/d20/d3d/da6/d111 0 2024-04-16T02:36:11.021 INFO:tasks.workunit.client.0.smithi151.stdout:3/836: chown d1/d6/f21 11996381 1 2024-04-16T02:36:11.083 INFO:tasks.workunit.client.0.smithi151.stdout:1/805: creat d3/d58/df/d32/d3a/df3/d7e/f11d x:0 0 0 2024-04-16T02:36:11.089 INFO:tasks.workunit.client.0.smithi151.stdout:3/837: dwrite d1/d37/f5e [0,4194304] 0 2024-04-16T02:36:11.089 INFO:tasks.workunit.client.0.smithi151.stdout:1/806: unlink d3/d58/df/d32/d3a/d71/f75 0 2024-04-16T02:36:11.092 INFO:tasks.workunit.client.0.smithi151.stdout:4/882: dread d0/d11/d19/f25 [0,4194304] 0 2024-04-16T02:36:11.150 INFO:tasks.workunit.client.0.smithi151.stdout:3/838: mkdir d1/d6/d20/d54/d112 0 2024-04-16T02:36:11.156 INFO:tasks.workunit.client.0.smithi151.stdout:3/839: readlink d1/d6/d20/l4b 0 2024-04-16T02:36:11.160 INFO:tasks.workunit.client.0.smithi151.stdout:1/807: dwrite d3/d4/d64/d40/da3/fa6 [0,4194304] 0 2024-04-16T02:36:11.163 INFO:tasks.workunit.client.0.smithi151.stdout:4/883: dwrite d0/fbb [0,4194304] 0 2024-04-16T02:36:11.167 INFO:tasks.workunit.client.0.smithi151.stdout:4/884: dread - d0/d11/d19/d46/dcd/f11c zero size 2024-04-16T02:36:11.169 INFO:tasks.workunit.client.0.smithi151.stdout:3/840: getdents d1/d6/d20/d3d/dd4/dd6/d44/d46 0 2024-04-16T02:36:11.170 INFO:tasks.workunit.client.0.smithi151.stdout:3/841: readlink d1/d6/d20/d3d/l51 0 2024-04-16T02:36:11.170 INFO:tasks.workunit.client.0.smithi151.stdout:1/808: creat d3/d58/df/d32/d3a/d71/f11e x:0 0 0 2024-04-16T02:36:11.171 INFO:tasks.workunit.client.0.smithi151.stdout:3/842: stat d1/d37/fde 0 2024-04-16T02:36:11.176 INFO:tasks.workunit.client.0.smithi151.stdout:1/809: symlink d3/d58/df/d32/d3a/df3/d7a/l11f 0 2024-04-16T02:36:11.180 INFO:tasks.workunit.client.0.smithi151.stdout:3/843: dread d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/f72 [0,4194304] 0 2024-04-16T02:36:11.184 INFO:tasks.workunit.client.0.smithi151.stdout:3/844: symlink d1/d89/daa/l113 0 2024-04-16T02:36:11.184 INFO:tasks.workunit.client.0.smithi151.stdout:4/885: link d0/dde/d4a/fed d0/dde/d4a/d60/de6/f130 0 2024-04-16T02:36:11.185 INFO:tasks.workunit.client.0.smithi151.stdout:3/845: chown d1/d6/d20/d3d/dd4/d107 56484710 1 2024-04-16T02:36:11.186 INFO:tasks.workunit.client.0.smithi151.stdout:4/886: symlink d0/daa/l131 0 2024-04-16T02:36:11.188 INFO:tasks.workunit.client.0.smithi151.stdout:3/846: creat d1/d6/d22/d23/d41/f114 x:0 0 0 2024-04-16T02:36:11.192 INFO:tasks.workunit.client.0.smithi151.stdout:3/847: unlink d1/d89/daa/l113 0 2024-04-16T02:36:11.192 INFO:tasks.workunit.client.0.smithi151.stdout:4/887: dread d0/d11/d19/d87/d3a/d50/f67 [0,4194304] 0 2024-04-16T02:36:11.196 INFO:tasks.workunit.client.0.smithi151.stdout:3/848: dread d1/d89/fc0 [0,4194304] 0 2024-04-16T02:36:11.198 INFO:tasks.workunit.client.0.smithi151.stdout:3/849: dread - d1/d6/d20/d3d/dd4/dd6/d44/d63/f102 zero size 2024-04-16T02:36:11.198 INFO:tasks.workunit.client.0.smithi151.stdout:1/810: truncate d3/d58/df/d32/d3a/df3/f60 1689189 0 2024-04-16T02:36:11.198 INFO:tasks.workunit.client.0.smithi151.stdout:3/850: read d1/d6/d22/f52 [3519892,9412] 0 2024-04-16T02:36:11.203 INFO:tasks.workunit.client.0.smithi151.stdout:3/851: chown d1/d6/d22/d23/d41/f5b 39737 1 2024-04-16T02:36:11.204 INFO:tasks.workunit.client.0.smithi151.stdout:4/888: write d0/dde/dad/ff5 [3774649,3212] 0 2024-04-16T02:36:11.206 INFO:tasks.workunit.client.0.smithi151.stdout:1/811: symlink d3/d4/l120 0 2024-04-16T02:36:11.207 INFO:tasks.workunit.client.0.smithi151.stdout:1/812: stat d3/d4/d64/d40/fcf 0 2024-04-16T02:36:11.208 INFO:tasks.workunit.client.0.smithi151.stdout:3/852: mknod d1/d6/d20/d3d/da6/dd3/c115 0 2024-04-16T02:36:11.214 INFO:tasks.workunit.client.0.smithi151.stdout:4/889: dread d0/dde/f47 [0,4194304] 0 2024-04-16T02:36:11.216 INFO:tasks.workunit.client.0.smithi151.stdout:3/853: creat d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/f116 x:0 0 0 2024-04-16T02:36:11.219 INFO:tasks.workunit.client.0.smithi151.stdout:1/813: write d3/d4/d64/d40/fcf [107230,73559] 0 2024-04-16T02:36:11.222 INFO:tasks.workunit.client.0.smithi151.stdout:1/814: truncate d3/d58/df/f114 781058 0 2024-04-16T02:36:11.223 INFO:tasks.workunit.client.0.smithi151.stdout:3/854: dwrite d1/d37/f42 [0,4194304] 0 2024-04-16T02:36:11.225 INFO:tasks.workunit.client.0.smithi151.stdout:3/855: read d1/d6/d20/d3d/dd4/dd6/fa8 [2053929,35739] 0 2024-04-16T02:36:11.226 INFO:tasks.workunit.client.0.smithi151.stdout:3/856: chown d1/d6/d20/d3d/dd4/dd6/f45 7368739 1 2024-04-16T02:36:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:10 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:36:11.289 INFO:tasks.workunit.client.0.smithi151.stdout:1/815: mkdir d3/d58/df/d32/d3a/d71/ded/d121 0 2024-04-16T02:36:11.291 INFO:tasks.workunit.client.0.smithi151.stdout:3/857: rmdir d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/dd1 39 2024-04-16T02:36:11.297 INFO:tasks.workunit.client.0.smithi151.stdout:1/816: rmdir d3/d58/df/d32/d3a/d6d 39 2024-04-16T02:36:11.299 INFO:tasks.workunit.client.0.smithi151.stdout:3/858: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/l117 0 2024-04-16T02:36:11.304 INFO:tasks.workunit.client.0.smithi151.stdout:3/859: creat d1/d37/d87/f118 x:0 0 0 2024-04-16T02:36:11.305 INFO:tasks.workunit.client.0.smithi151.stdout:3/860: dread - d1/d6/d20/d54/f58 zero size 2024-04-16T02:36:11.309 INFO:tasks.workunit.client.0.smithi151.stdout:3/861: mkdir d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d119 0 2024-04-16T02:36:11.310 INFO:tasks.workunit.client.0.smithi151.stdout:1/817: dwrite d3/d33/d10/d4f/dd1/fd4 [0,4194304] 0 2024-04-16T02:36:11.315 INFO:tasks.workunit.client.0.smithi151.stdout:3/862: creat d1/d6/d20/d3d/f11a x:0 0 0 2024-04-16T02:36:11.319 INFO:tasks.workunit.client.0.smithi151.stdout:3/863: mknod d1/d6/d20/d3d/dd4/c11b 0 2024-04-16T02:36:11.328 INFO:tasks.workunit.client.0.smithi151.stdout:1/818: dwrite d3/d1f/dd7/fff [0,4194304] 0 2024-04-16T02:36:11.328 INFO:tasks.workunit.client.0.smithi151.stdout:1/819: dread - d3/fd6 zero size 2024-04-16T02:36:11.332 INFO:tasks.workunit.client.0.smithi151.stdout:1/820: fdatasync d3/d58/df/d32/d3a/f79 0 2024-04-16T02:36:11.336 INFO:tasks.workunit.client.0.smithi151.stdout:3/864: dwrite d1/d6/d20/d3d/dd4/dd6/fa8 [4194304,4194304] 0 2024-04-16T02:36:11.338 INFO:tasks.workunit.client.0.smithi151.stdout:1/821: fdatasync d3/d58/df/d32/d3a/d71/fbf 0 2024-04-16T02:36:11.338 INFO:tasks.workunit.client.0.smithi151.stdout:1/822: stat d3/d58/df/d32/lc6 0 2024-04-16T02:36:11.348 INFO:tasks.workunit.client.0.smithi151.stdout:1/823: unlink d3/c49 0 2024-04-16T02:36:11.353 INFO:tasks.workunit.client.0.smithi151.stdout:1/824: link d3/d58/df/d38/f74 d3/d1f/dd7/dda/f122 0 2024-04-16T02:36:11.353 INFO:tasks.workunit.client.0.smithi151.stdout:1/825: chown d3/d4/d64 286104086 1 2024-04-16T02:36:11.355 INFO:tasks.workunit.client.0.smithi151.stdout:1/826: rename d3/d58/df/l35 to d3/d1f/dd7/l123 0 2024-04-16T02:36:11.357 INFO:tasks.workunit.client.0.smithi151.stdout:1/827: dread - d3/d33/d10/fd9 zero size 2024-04-16T02:36:11.359 INFO:tasks.workunit.client.0.smithi151.stdout:3/865: write d1/d6/d20/d3d/da6/dd8/ffb [3753326,17633] 0 2024-04-16T02:36:11.361 INFO:tasks.workunit.client.0.smithi151.stdout:0/692: sync 2024-04-16T02:36:11.361 INFO:tasks.workunit.client.0.smithi151.stdout:3/866: symlink d1/d89/l11c 0 2024-04-16T02:36:11.363 INFO:tasks.workunit.client.0.smithi151.stdout:1/828: link d3/d58/df/d32/d3a/df3/d73/c98 d3/d58/df/c124 0 2024-04-16T02:36:11.363 INFO:tasks.workunit.client.0.smithi151.stdout:1/829: stat d3/d4/f96 0 2024-04-16T02:36:11.366 INFO:tasks.workunit.client.0.smithi151.stdout:1/830: mknod d3/d58/df/d32/d3a/df3/d73/c125 0 2024-04-16T02:36:11.367 INFO:tasks.workunit.client.0.smithi151.stdout:1/831: fsync d3/d58/df/d32/f88 0 2024-04-16T02:36:11.368 INFO:tasks.workunit.client.0.smithi151.stdout:3/867: creat d1/d6/d22/d6c/f11d x:0 0 0 2024-04-16T02:36:11.372 INFO:tasks.workunit.client.0.smithi151.stdout:1/832: symlink d3/d58/df/d32/d3a/d67/l126 0 2024-04-16T02:36:11.376 INFO:tasks.workunit.client.0.smithi151.stdout:1/833: symlink d3/d58/df/d32/d3a/df3/d7a/df9/d107/l127 0 2024-04-16T02:36:11.379 INFO:tasks.workunit.client.0.smithi151.stdout:1/834: dread - d3/d58/df/d32/d3a/df3/d7a/df9/f101 zero size 2024-04-16T02:36:11.380 INFO:tasks.workunit.client.0.smithi151.stdout:1/835: chown d3/d58/df/d32/d3a/d71/f10a 0 1 2024-04-16T02:36:11.384 INFO:tasks.workunit.client.0.smithi151.stdout:0/693: truncate dc/d15/d34/d67/d69/d6f/f99 2036049 0 2024-04-16T02:36:11.384 INFO:tasks.workunit.client.0.smithi151.stdout:0/694: readlink dc/d3c/l41 0 2024-04-16T02:36:11.387 INFO:tasks.workunit.client.0.smithi151.stdout:3/868: write d1/d6/d22/d23/f5a [871475,85370] 0 2024-04-16T02:36:11.389 INFO:tasks.workunit.client.0.smithi151.stdout:1/836: dread d3/d4/d64/d1c/f50 [0,4194304] 0 2024-04-16T02:36:11.390 INFO:tasks.workunit.client.0.smithi151.stdout:1/837: read - d3/d58/df/d32/d3a/df3/d52/d97/ff5 zero size 2024-04-16T02:36:11.395 INFO:tasks.workunit.client.0.smithi151.stdout:1/838: fdatasync d3/d33/d10/df6/fb7 0 2024-04-16T02:36:11.397 INFO:tasks.workunit.client.0.smithi151.stdout:3/869: dwrite d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/dd1/ff0 [0,4194304] 0 2024-04-16T02:36:11.401 INFO:tasks.workunit.client.0.smithi151.stdout:0/695: write dc/d20/d7b/d9c/fd8 [3395277,82980] 0 2024-04-16T02:36:11.403 INFO:tasks.workunit.client.0.smithi151.stdout:1/839: chown d3/d58/df/d32/d3a/df3/d73/f85 258239056 1 2024-04-16T02:36:11.406 INFO:tasks.workunit.client.0.smithi151.stdout:3/870: dwrite d1/fcb [0,4194304] 0 2024-04-16T02:36:11.412 INFO:tasks.workunit.client.0.smithi151.stdout:1/840: mknod d3/d58/df/d32/d3a/df3/d52/c128 0 2024-04-16T02:36:11.414 INFO:tasks.workunit.client.0.smithi151.stdout:3/871: write d1/d6/d20/d3d/dd4/dd6/f45 [652740,129401] 0 2024-04-16T02:36:11.416 INFO:tasks.workunit.client.0.smithi151.stdout:1/841: truncate d3/d4/d64/d40/fbb 58344 0 2024-04-16T02:36:11.418 INFO:tasks.workunit.client.0.smithi151.stdout:3/872: unlink d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/lf2 0 2024-04-16T02:36:11.421 INFO:tasks.workunit.client.0.smithi151.stdout:1/842: mkdir d3/d58/df/d32/d3a/df3/d52/d129 0 2024-04-16T02:36:11.426 INFO:tasks.workunit.client.0.smithi151.stdout:3/873: dread d1/d6/d20/fc5 [4194304,4194304] 0 2024-04-16T02:36:11.426 INFO:tasks.workunit.client.0.smithi151.stdout:0/696: dwrite dc/d15/d5c/f6a [0,4194304] 0 2024-04-16T02:36:11.427 INFO:tasks.workunit.client.0.smithi151.stdout:3/874: chown d1/d6/d22/d23/ced 371068 1 2024-04-16T02:36:11.438 INFO:tasks.workunit.client.0.smithi151.stdout:1/843: dread d3/f9d [0,4194304] 0 2024-04-16T02:36:11.439 INFO:tasks.workunit.client.0.smithi151.stdout:1/844: read d3/d1f/dd7/fff [534586,115630] 0 2024-04-16T02:36:11.439 INFO:tasks.workunit.client.0.smithi151.stdout:0/697: rename dc/d15/d34/d67/d69/d6f/l93 to dc/d15/d34/d3b/d71/dc3/ldc 0 2024-04-16T02:36:11.439 INFO:tasks.workunit.client.0.smithi151.stdout:3/875: truncate d1/d6/d4a/d4e/f104 4322480 0 2024-04-16T02:36:11.440 INFO:tasks.workunit.client.0.smithi151.stdout:0/698: stat dc/d15/d34/d67/d69/fcf 0 2024-04-16T02:36:11.441 INFO:tasks.workunit.client.0.smithi151.stdout:0/699: readlink dc/d3c/d9f/la6 0 2024-04-16T02:36:11.442 INFO:tasks.workunit.client.0.smithi151.stdout:0/700: chown dc/d15/d34/d67/d69/ca8 629159 1 2024-04-16T02:36:11.444 INFO:tasks.workunit.client.0.smithi151.stdout:3/876: dread d1/d37/fb7 [0,4194304] 0 2024-04-16T02:36:11.444 INFO:tasks.workunit.client.0.smithi151.stdout:5/989: sync 2024-04-16T02:36:11.444 INFO:tasks.workunit.client.0.smithi151.stdout:9/975: sync 2024-04-16T02:36:11.456 INFO:tasks.workunit.client.0.smithi151.stdout:3/877: symlink d1/d6/d22/d6c/l11e 0 2024-04-16T02:36:11.456 INFO:tasks.workunit.client.0.smithi151.stdout:1/845: dread d3/d58/df/d32/d3a/d6d/f7b [4194304,4194304] 0 2024-04-16T02:36:11.457 INFO:tasks.workunit.client.0.smithi151.stdout:3/878: write d1/d89/fdc [952615,74525] 0 2024-04-16T02:36:11.461 INFO:tasks.workunit.client.0.smithi151.stdout:5/990: rmdir d2/d6/d43 39 2024-04-16T02:36:11.462 INFO:tasks.workunit.client.0.smithi151.stdout:0/701: dwrite dc/d15/d34/f54 [4194304,4194304] 0 2024-04-16T02:36:11.465 INFO:tasks.workunit.client.0.smithi151.stdout:1/846: creat d3/d4/d64/d1c/f12a x:0 0 0 2024-04-16T02:36:11.469 INFO:tasks.workunit.client.0.smithi151.stdout:1/847: read d3/d33/d10/f4e [768161,116462] 0 2024-04-16T02:36:11.471 INFO:tasks.workunit.client.0.smithi151.stdout:0/702: truncate dc/d15/d34/fab 429472 0 2024-04-16T02:36:11.473 INFO:tasks.workunit.client.0.smithi151.stdout:9/976: rmdir d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/dc3/d123 0 2024-04-16T02:36:11.474 INFO:tasks.workunit.client.0.smithi151.stdout:5/991: read - d2/d6/dfc/f110 zero size 2024-04-16T02:36:11.476 INFO:tasks.workunit.client.0.smithi151.stdout:9/977: write d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/fc6 [3004103,19935] 0 2024-04-16T02:36:11.477 INFO:tasks.workunit.client.0.smithi151.stdout:9/978: write d1/dd/f8a [3001951,41961] 0 2024-04-16T02:36:11.477 INFO:tasks.workunit.client.0.smithi151.stdout:1/848: fdatasync d3/d58/df/d32/d3a/d6d/f7b 0 2024-04-16T02:36:11.478 INFO:tasks.workunit.client.0.smithi151.stdout:9/979: dread - d1/dd/d4a/f92 zero size 2024-04-16T02:36:11.479 INFO:tasks.workunit.client.0.smithi151.stdout:3/879: creat d1/d6/d20/d3d/dd4/dd6/d44/f11f x:0 0 0 2024-04-16T02:36:11.481 INFO:tasks.workunit.client.0.smithi151.stdout:0/703: rmdir dc/d15/d34/d4e 39 2024-04-16T02:36:11.482 INFO:tasks.workunit.client.0.smithi151.stdout:0/704: chown dc/d3c/d9f/cc9 11790 1 2024-04-16T02:36:11.484 INFO:tasks.workunit.client.0.smithi151.stdout:5/992: creat d2/d6/d19/d30/d36/d42/f13d x:0 0 0 2024-04-16T02:36:11.487 INFO:tasks.workunit.client.0.smithi151.stdout:1/849: readlink d3/d33/l14 0 2024-04-16T02:36:11.490 INFO:tasks.workunit.client.0.smithi151.stdout:5/993: dread d2/d6/db7/d12d/ddc/d83/d8b/ff4 [0,4194304] 0 2024-04-16T02:36:11.491 INFO:tasks.workunit.client.0.smithi151.stdout:9/980: write d1/d3/d6/d17/d54/dd5/f129 [326233,127697] 0 2024-04-16T02:36:11.492 INFO:tasks.workunit.client.0.smithi151.stdout:3/880: fsync d1/d6/d22/d23/d41/f5b 0 2024-04-16T02:36:11.494 INFO:tasks.workunit.client.0.smithi151.stdout:3/881: chown d1/d6/d22/d23/d66 137247673 1 2024-04-16T02:36:11.498 INFO:tasks.workunit.client.0.smithi151.stdout:0/705: dread dc/d15/f17 [0,4194304] 0 2024-04-16T02:36:11.499 INFO:tasks.workunit.client.0.smithi151.stdout:3/882: creat d1/d37/f120 x:0 0 0 2024-04-16T02:36:11.500 INFO:tasks.workunit.client.0.smithi151.stdout:5/994: truncate d2/d6/d19/d30/fb0 1498800 0 2024-04-16T02:36:11.502 INFO:tasks.workunit.client.0.smithi151.stdout:0/706: symlink dc/ldd 0 2024-04-16T02:36:11.505 INFO:tasks.workunit.client.0.smithi151.stdout:1/850: dread d3/d58/df/d32/d3a/f66 [0,4194304] 0 2024-04-16T02:36:11.505 INFO:tasks.workunit.client.0.smithi151.stdout:3/883: rename d1/d6/d22/d6c to d1/d6/d20/d3d/dd4/d121 0 2024-04-16T02:36:11.510 INFO:tasks.workunit.client.0.smithi151.stdout:3/884: dread d1/d6/d20/d3d/dd4/dd6/fa8 [4194304,4194304] 0 2024-04-16T02:36:11.513 INFO:tasks.workunit.client.0.smithi151.stdout:1/851: unlink d3/d58/df/d32/d3a/df3/d52/d97/c119 0 2024-04-16T02:36:11.518 INFO:tasks.workunit.client.0.smithi151.stdout:5/995: dwrite d2/d6/db7/d12d/fd9 [4194304,4194304] 0 2024-04-16T02:36:11.518 INFO:tasks.workunit.client.0.smithi151.stdout:0/707: mkdir dc/dde 0 2024-04-16T02:36:11.521 INFO:tasks.workunit.client.0.smithi151.stdout:3/885: write d1/f26 [4686565,125833] 0 2024-04-16T02:36:11.525 INFO:tasks.workunit.client.0.smithi151.stdout:1/852: fsync d3/d58/df/d32/d3a/d71/fde 0 2024-04-16T02:36:11.527 INFO:tasks.workunit.client.0.smithi151.stdout:1/853: write d3/d58/df/d32/d3a/df3/d52/d97/ff5 [188658,5755] 0 2024-04-16T02:36:11.528 INFO:tasks.workunit.client.0.smithi151.stdout:5/996: fsync d2/d6/d43/fed 0 2024-04-16T02:36:11.533 INFO:tasks.workunit.client.0.smithi151.stdout:0/708: dread - dc/d15/d34/d4e/f83 zero size 2024-04-16T02:36:11.534 INFO:tasks.workunit.client.0.smithi151.stdout:0/709: stat dc/d15/d34/d67/d69/d6f 0 2024-04-16T02:36:11.534 INFO:tasks.workunit.client.0.smithi151.stdout:3/886: mkdir d1/d6/d20/d3d/da6/d111/d122 0 2024-04-16T02:36:11.535 INFO:tasks.workunit.client.0.smithi151.stdout:0/710: chown dc/d4c/d72/d76/db0/db5/fc0 479851 1 2024-04-16T02:36:11.538 INFO:tasks.workunit.client.0.smithi151.stdout:5/997: mkdir d2/d6/db7/d12d/ddc/d83/d13e 0 2024-04-16T02:36:11.539 INFO:tasks.workunit.client.0.smithi151.stdout:5/998: fdatasync d2/d6/d19/d62/dd0/fd2 0 2024-04-16T02:36:11.541 INFO:tasks.workunit.client.0.smithi151.stdout:0/711: fsync dc/d4c/f88 0 2024-04-16T02:36:11.542 INFO:tasks.workunit.client.0.smithi151.stdout:0/712: chown dc/d15/d34/d4e/l6e 72191554 1 2024-04-16T02:36:11.551 INFO:tasks.workunit.client.0.smithi151.stdout:1/854: dwrite d3/d4/d64/d40/fbb [0,4194304] 0 2024-04-16T02:36:11.554 INFO:tasks.workunit.client.0.smithi151.stdout:3/887: rename d1/d69/f101 to d1/d6/f123 0 2024-04-16T02:36:11.556 INFO:tasks.workunit.client.0.smithi151.stdout:0/713: creat dc/d15/d34/d67/dd2/fdf x:0 0 0 2024-04-16T02:36:11.557 INFO:tasks.workunit.client.0.smithi151.stdout:3/888: truncate d1/d6/d20/d3d/da6/dd8/ff5 686942 0 2024-04-16T02:36:11.559 INFO:tasks.workunit.client.0.smithi151.stdout:1/855: mkdir d3/d58/df/d38/d12b 0 2024-04-16T02:36:11.562 INFO:tasks.workunit.client.0.smithi151.stdout:0/714: read dc/d4c/d72/d76/db0/db5/fc0 [1309611,16333] 0 2024-04-16T02:36:11.563 INFO:tasks.workunit.client.0.smithi151.stdout:3/889: mknod d1/d69/d7f/dac/c124 0 2024-04-16T02:36:11.566 INFO:tasks.workunit.client.0.smithi151.stdout:3/890: chown d1/d6/d20/d3d/dd4/dd6/d44/f11f 126 1 2024-04-16T02:36:11.566 INFO:tasks.workunit.client.0.smithi151.stdout:0/715: mkdir dc/d20/d7b/d9c/de0 0 2024-04-16T02:36:11.567 INFO:tasks.workunit.client.0.smithi151.stdout:3/891: truncate d1/d37/f42 5031994 0 2024-04-16T02:36:11.567 INFO:tasks.workunit.client.0.smithi151.stdout:3/892: chown d1/f26 1175282 1 2024-04-16T02:36:11.594 INFO:tasks.workunit.client.0.smithi151.stdout:3/893: dread d1/f1d [0,4194304] 0 2024-04-16T02:36:11.598 INFO:tasks.workunit.client.0.smithi151.stdout:3/894: getdents d1/d89/daa 0 2024-04-16T02:36:11.680 INFO:tasks.workunit.client.0.smithi151.stdout:8/799: sync 2024-04-16T02:36:11.681 INFO:tasks.workunit.client.0.smithi151.stdout:4/890: sync 2024-04-16T02:36:11.684 INFO:tasks.workunit.client.0.smithi151.stdout:3/895: readlink d1/d6/d20/d3d/dd4/dd6/d44/la2 0 2024-04-16T02:36:11.696 INFO:tasks.workunit.client.0.smithi151.stdout:8/800: read d11/d2a/d33/d4e/f9e [2686712,71542] 0 2024-04-16T02:36:11.699 INFO:tasks.workunit.client.0.smithi151.stdout:4/891: write d0/d11/d19/d87/d3a/d61/fa4 [747940,99315] 0 2024-04-16T02:36:11.718 INFO:tasks.workunit.client.0.smithi151.stdout:8/801: dread d11/d2a/d33/f45 [0,4194304] 0 2024-04-16T02:36:11.736 INFO:tasks.workunit.client.0.smithi151.stdout:3/896: creat d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/dd1/f125 x:0 0 0 2024-04-16T02:36:11.737 INFO:tasks.workunit.client.0.smithi151.stdout:3/897: write d1/d6/d20/d3d/da6/dd8/ff5 [665339,118172] 0 2024-04-16T02:36:11.743 INFO:tasks.workunit.client.0.smithi151.stdout:8/802: mknod d11/d2a/d33/d4e/d62/c119 0 2024-04-16T02:36:11.748 INFO:tasks.workunit.client.0.smithi151.stdout:3/898: getdents d1/d6/d20/d3d/dd4/dd6/d44/d46 0 2024-04-16T02:36:11.751 INFO:tasks.workunit.client.0.smithi151.stdout:8/803: getdents d11/d2a/d35 0 2024-04-16T02:36:11.754 INFO:tasks.workunit.client.0.smithi151.stdout:3/899: creat d1/d6/d20/d54/d112/f126 x:0 0 0 2024-04-16T02:36:11.759 INFO:tasks.workunit.client.0.smithi151.stdout:3/900: mkdir d1/d6/d127 0 2024-04-16T02:36:11.761 INFO:tasks.workunit.client.0.smithi151.stdout:8/804: rename d11/d2a/d83/f9c to d11/d22/f11a 0 2024-04-16T02:36:11.766 INFO:tasks.workunit.client.0.smithi151.stdout:3/901: link d1/d6/d22/fbf d1/d37/f128 0 2024-04-16T02:36:11.766 INFO:tasks.workunit.client.0.smithi151.stdout:8/805: dwrite d11/d2a/df4/f110 [0,4194304] 0 2024-04-16T02:36:11.773 INFO:tasks.workunit.client.0.smithi151.stdout:3/902: symlink d1/d6/d4a/l129 0 2024-04-16T02:36:11.774 INFO:tasks.workunit.client.0.smithi151.stdout:8/806: creat d11/d2a/d33/d4e/d62/d94/dfd/f11b x:0 0 0 2024-04-16T02:36:11.782 INFO:tasks.workunit.client.0.smithi151.stdout:4/892: write d0/dde/d4a/d60/f70 [2483801,43749] 0 2024-04-16T02:36:11.785 INFO:tasks.workunit.client.0.smithi151.stdout:4/893: creat d0/d11/d19/d87/d3a/d50/f132 x:0 0 0 2024-04-16T02:36:11.787 INFO:tasks.workunit.client.0.smithi151.stdout:4/894: mkdir d0/dde/d80/da0/d133 0 2024-04-16T02:36:11.790 INFO:tasks.workunit.client.0.smithi151.stdout:4/895: creat d0/dde/dad/d106/f134 x:0 0 0 2024-04-16T02:36:11.792 INFO:tasks.workunit.client.0.smithi151.stdout:4/896: dread d0/dde/dad/d100/f101 [0,4194304] 0 2024-04-16T02:36:11.797 INFO:tasks.workunit.client.0.smithi151.stdout:3/903: write d1/d6/d4a/d4e/d77/fda [4069572,108819] 0 2024-04-16T02:36:11.801 INFO:tasks.workunit.client.0.smithi151.stdout:8/807: dwrite d11/d2a/d33/d4e/f72 [0,4194304] 0 2024-04-16T02:36:11.803 INFO:tasks.workunit.client.0.smithi151.stdout:3/904: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/dd1/l12a 0 2024-04-16T02:36:11.804 INFO:tasks.workunit.client.0.smithi151.stdout:8/808: mkdir d11/d2a/d33/d4e/d62/dc8/d11c 0 2024-04-16T02:36:11.804 INFO:tasks.workunit.client.0.smithi151.stdout:3/905: write d1/d89/deb/f10d [820722,44423] 0 2024-04-16T02:36:11.806 INFO:tasks.workunit.client.0.smithi151.stdout:8/809: chown d11/d2a/d35/c36 4 1 2024-04-16T02:36:11.808 INFO:tasks.workunit.client.0.smithi151.stdout:4/897: dread d0/dde/f95 [0,4194304] 0 2024-04-16T02:36:11.808 INFO:tasks.workunit.client.0.smithi151.stdout:3/906: mkdir d1/d6/d20/d3d/dd4/dd6/d12b 0 2024-04-16T02:36:11.808 INFO:tasks.workunit.client.0.smithi151.stdout:8/810: truncate d11/d22/d2e/dc0/dbc/f106 211891 0 2024-04-16T02:36:11.810 INFO:tasks.workunit.client.0.smithi151.stdout:4/898: symlink d0/dde/d80/d107/l135 0 2024-04-16T02:36:11.816 INFO:tasks.workunit.client.0.smithi151.stdout:3/907: symlink d1/d6/d20/l12c 0 2024-04-16T02:36:11.817 INFO:tasks.workunit.client.0.smithi151.stdout:8/811: mkdir d11/d2a/d33/d4e/d62/d94/d11d 0 2024-04-16T02:36:11.817 INFO:tasks.workunit.client.0.smithi151.stdout:3/908: chown d1/d6/c9f 5478 1 2024-04-16T02:36:11.817 INFO:tasks.workunit.client.0.smithi151.stdout:8/812: readlink d11/d2a/df4/d78/lbb 0 2024-04-16T02:36:11.819 INFO:tasks.workunit.client.0.smithi151.stdout:4/899: dread d0/d11/d19/d87/d3a/d50/fb0 [0,4194304] 0 2024-04-16T02:36:11.826 INFO:tasks.workunit.client.0.smithi151.stdout:3/909: dwrite d1/d6/d20/fc5 [4194304,4194304] 0 2024-04-16T02:36:11.833 INFO:tasks.workunit.client.0.smithi151.stdout:3/910: truncate d1/d69/da4/fb5 2677562 0 2024-04-16T02:36:11.836 INFO:tasks.workunit.client.0.smithi151.stdout:3/911: dread d1/d6/d20/d54/f80 [0,4194304] 0 2024-04-16T02:36:11.837 INFO:tasks.workunit.client.0.smithi151.stdout:3/912: chown d1/d6/d22/c86 176469655 1 2024-04-16T02:36:11.839 INFO:tasks.workunit.client.0.smithi151.stdout:3/913: dread - d1/d6/d22/d23/d66/fb2 zero size 2024-04-16T02:36:11.840 INFO:tasks.workunit.client.0.smithi151.stdout:8/813: dread d11/d2a/d35/d64/f88 [0,4194304] 0 2024-04-16T02:36:11.843 INFO:tasks.workunit.client.0.smithi151.stdout:8/814: stat d11/d2a/d83/lc6 0 2024-04-16T02:36:11.847 INFO:tasks.workunit.client.0.smithi151.stdout:8/815: symlink d11/d2a/d35/d52/db3/dcd/dd2/l11e 0 2024-04-16T02:36:11.849 INFO:tasks.workunit.client.0.smithi151.stdout:8/816: truncate d11/d2a/d33/d4e/d62/d94/dfd/f11b 160321 0 2024-04-16T02:36:11.850 INFO:tasks.workunit.client.0.smithi151.stdout:8/817: chown d11/d2a 451 1 2024-04-16T02:36:11.852 INFO:tasks.workunit.client.0.smithi151.stdout:8/818: fsync d11/d2a/df4/f110 0 2024-04-16T02:36:11.855 INFO:tasks.workunit.client.0.smithi151.stdout:4/900: dwrite d0/f93 [0,4194304] 0 2024-04-16T02:36:11.862 INFO:tasks.workunit.client.0.smithi151.stdout:4/901: rename d0/d11/d19/d87/d3a/d65/ccb to d0/dde/dad/d106/c136 0 2024-04-16T02:36:11.865 INFO:tasks.workunit.client.0.smithi151.stdout:3/914: dwrite d1/d6/d4a/d4e/f104 [0,4194304] 0 2024-04-16T02:36:11.867 INFO:tasks.workunit.client.0.smithi151.stdout:3/915: read d1/d6/d22/d23/d41/d9d/fa7 [5800927,47391] 0 2024-04-16T02:36:11.868 INFO:tasks.workunit.client.0.smithi151.stdout:3/916: chown d1/d6/d20/l12c 6176 1 2024-04-16T02:36:11.873 INFO:tasks.workunit.client.0.smithi151.stdout:3/917: rename d1/d6/d20/d3d/da6/dd3 to d1/d89/daa/d12d 0 2024-04-16T02:36:11.876 INFO:tasks.workunit.client.0.smithi151.stdout:3/918: write d1/d6/d20/d3d/dd4/dd6/d44/f5f [2583895,125045] 0 2024-04-16T02:36:11.882 INFO:tasks.workunit.client.0.smithi151.stdout:3/919: symlink d1/d6/d20/d3d/da6/dd8/l12e 0 2024-04-16T02:36:11.884 INFO:tasks.workunit.client.0.smithi151.stdout:3/920: truncate d1/d69/d7f/fee 479664 0 2024-04-16T02:36:11.885 INFO:tasks.workunit.client.0.smithi151.stdout:4/902: dread d0/d9b/fd0 [0,4194304] 0 2024-04-16T02:36:11.888 INFO:tasks.workunit.client.0.smithi151.stdout:3/921: rename d1/d37/f128 to d1/d89/daa/d12d/f12f 0 2024-04-16T02:36:11.889 INFO:tasks.workunit.client.0.smithi151.stdout:4/903: creat d0/d11/d19/d87/d7b/dfa/f137 x:0 0 0 2024-04-16T02:36:11.889 INFO:tasks.workunit.client.0.smithi151.stdout:3/922: chown d1/d6/d22/d23/d66/d110 214821 1 2024-04-16T02:36:11.890 INFO:tasks.workunit.client.0.smithi151.stdout:4/904: readlink d0/d99/l2a 0 2024-04-16T02:36:11.890 INFO:tasks.workunit.client.0.smithi151.stdout:3/923: readlink d1/d89/l11c 0 2024-04-16T02:36:11.890 INFO:tasks.workunit.client.0.smithi151.stdout:3/924: chown d1/d6/d22/d23/d41/f114 10789883 1 2024-04-16T02:36:11.898 INFO:tasks.workunit.client.0.smithi151.stdout:3/925: dread d1/d37/f3c [0,4194304] 0 2024-04-16T02:36:11.901 INFO:tasks.workunit.client.0.smithi151.stdout:3/926: fdatasync d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/f97 0 2024-04-16T02:36:11.901 INFO:tasks.workunit.client.0.smithi151.stdout:3/927: fdatasync d1/f3e 0 2024-04-16T02:36:11.910 INFO:tasks.workunit.client.0.smithi151.stdout:3/928: link d1/d6/f24 d1/d6/d4a/d4e/d77/f130 0 2024-04-16T02:36:11.997 INFO:tasks.workunit.client.0.smithi151.stdout:3/929: creat d1/d6/d20/d3d/dd4/dd6/d44/d63/f131 x:0 0 0 2024-04-16T02:36:12.012 INFO:tasks.workunit.client.0.smithi151.stdout:3/930: dread d1/d6/d22/f52 [0,4194304] 0 2024-04-16T02:36:12.013 INFO:tasks.workunit.client.0.smithi151.stdout:3/931: read - d1/d6/d20/d3d/dd4/dd6/d44/d63/f102 zero size 2024-04-16T02:36:12.019 INFO:tasks.workunit.client.0.smithi151.stdout:3/932: write d1/d6/d20/d54/f58 [58099,4744] 0 2024-04-16T02:36:12.021 INFO:tasks.workunit.client.0.smithi151.stdout:3/933: chown d1/d6/d20/d3d/dd4/dd6/d44/d46/fc8 1 1 2024-04-16T02:36:12.023 INFO:tasks.workunit.client.0.smithi151.stdout:3/934: mknod d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/dd1/c132 0 2024-04-16T02:36:12.024 INFO:tasks.workunit.client.0.smithi151.stdout:3/935: stat d1/d6/d20/d3d/dd4/dd6/d44/d63/f102 0 2024-04-16T02:36:12.026 INFO:tasks.workunit.client.0.smithi151.stdout:3/936: dread d1/d6/d22/d23/d41/f76 [0,4194304] 0 2024-04-16T02:36:12.028 INFO:tasks.workunit.client.0.smithi151.stdout:3/937: chown d1/d6/d4a/d4e/ffe 12693734 1 2024-04-16T02:36:12.038 INFO:tasks.workunit.client.0.smithi151.stdout:3/938: dwrite d1/d6/d20/d54/f80 [0,4194304] 0 2024-04-16T02:36:12.043 INFO:tasks.workunit.client.0.smithi151.stdout:3/939: symlink d1/d6/d20/d3d/da6/dd8/l133 0 2024-04-16T02:36:12.056 INFO:tasks.workunit.client.0.smithi151.stdout:3/940: dwrite d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/f81 [0,4194304] 0 2024-04-16T02:36:12.059 INFO:tasks.workunit.client.0.smithi151.stdout:3/941: unlink d1/d6/f1b 0 2024-04-16T02:36:12.061 INFO:tasks.workunit.client.0.smithi151.stdout:3/942: symlink d1/d6/d4a/d4e/d77/l134 0 2024-04-16T02:36:12.062 INFO:tasks.workunit.client.0.smithi151.stdout:3/943: chown d1/d6/d20/d3d/dd4/dd6/d44 1314759 1 2024-04-16T02:36:12.063 INFO:tasks.workunit.client.0.smithi151.stdout:3/944: write d1/d6/d22/d23/fe4 [206293,87171] 0 2024-04-16T02:36:12.065 INFO:tasks.workunit.client.0.smithi151.stdout:3/945: write d1/d6/d20/fc5 [6598339,120436] 0 2024-04-16T02:36:12.069 INFO:tasks.workunit.client.0.smithi151.stdout:3/946: creat d1/d6/d20/d3d/dd4/dd6/d44/f135 x:0 0 0 2024-04-16T02:36:12.074 INFO:tasks.workunit.client.0.smithi151.stdout:3/947: getdents d1/d6/d22/d23/d41/d9d 0 2024-04-16T02:36:12.078 INFO:tasks.workunit.client.0.smithi151.stdout:1/856: sync 2024-04-16T02:36:12.078 INFO:tasks.workunit.client.0.smithi151.stdout:9/981: sync 2024-04-16T02:36:12.078 INFO:tasks.workunit.client.0.smithi151.stdout:5/999: sync 2024-04-16T02:36:12.078 INFO:tasks.workunit.client.0.smithi151.stdout:3/948: symlink d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/l136 0 2024-04-16T02:36:12.082 INFO:tasks.workunit.client.0.smithi151.stdout:3/949: creat d1/d89/daa/f137 x:0 0 0 2024-04-16T02:36:12.083 INFO:tasks.workunit.client.0.smithi151.stdout:1/857: rmdir d3/d58/df/d68 39 2024-04-16T02:36:12.086 INFO:tasks.workunit.client.0.smithi151.stdout:3/950: creat d1/d6/d4a/d4e/d77/f138 x:0 0 0 2024-04-16T02:36:12.088 INFO:tasks.workunit.client.0.smithi151.stdout:3/951: chown d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/fd9 323 1 2024-04-16T02:36:12.089 INFO:tasks.workunit.client.0.smithi151.stdout:3/952: write d1/d6/d22/d23/fb3 [4134553,91223] 0 2024-04-16T02:36:12.090 INFO:tasks.workunit.client.0.smithi151.stdout:3/953: readlink d1/d6/d20/l4b 0 2024-04-16T02:36:12.095 INFO:tasks.workunit.client.0.smithi151.stdout:9/982: truncate d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/f106 991154 0 2024-04-16T02:36:12.098 INFO:tasks.workunit.client.0.smithi151.stdout:3/954: unlink d1/d6/d20/d54/f80 0 2024-04-16T02:36:12.104 INFO:tasks.workunit.client.0.smithi151.stdout:1/858: dread d3/d58/df/d32/d3a/df3/f65 [0,4194304] 0 2024-04-16T02:36:12.104 INFO:tasks.workunit.client.0.smithi151.stdout:3/955: rename d1/d6/f50 to d1/d69/f139 0 2024-04-16T02:36:12.108 INFO:tasks.workunit.client.0.smithi151.stdout:3/956: creat d1/d6/d20/d3d/dd4/dd6/f13a x:0 0 0 2024-04-16T02:36:12.110 INFO:tasks.workunit.client.0.smithi151.stdout:1/859: rmdir d3/d58/df/d32/d3a/df1 0 2024-04-16T02:36:12.112 INFO:tasks.workunit.client.0.smithi151.stdout:1/860: write d3/d33/d10/d4f/f109 [160143,111447] 0 2024-04-16T02:36:12.113 INFO:tasks.workunit.client.0.smithi151.stdout:9/983: dread d1/d3/d6/f1b [0,4194304] 0 2024-04-16T02:36:12.113 INFO:tasks.workunit.client.0.smithi151.stdout:1/861: dread - d3/d33/d10/df6/fb7 zero size 2024-04-16T02:36:12.118 INFO:tasks.workunit.client.0.smithi151.stdout:9/984: creat d1/d3/d6/d17/d54/d9b/d22/d2c/d86/db9/dc2/de3/f149 x:0 0 0 2024-04-16T02:36:12.119 INFO:tasks.workunit.client.0.smithi151.stdout:3/957: dread d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/fa3 [0,4194304] 0 2024-04-16T02:36:12.120 INFO:tasks.workunit.client.0.smithi151.stdout:3/958: stat d1/d6/d20/d3d/dd4/dd6/c36 0 2024-04-16T02:36:12.123 INFO:tasks.workunit.client.0.smithi151.stdout:9/985: mkdir d1/d3/d6/d17/d54/d9b/dae/d14a 0 2024-04-16T02:36:12.124 INFO:tasks.workunit.client.0.smithi151.stdout:3/959: creat d1/d6/d22/d23/d41/d9d/f13b x:0 0 0 2024-04-16T02:36:12.127 INFO:tasks.workunit.client.0.smithi151.stdout:3/960: creat d1/d89/daa/f13c x:0 0 0 2024-04-16T02:36:12.128 INFO:tasks.workunit.client.0.smithi151.stdout:9/986: truncate d1/d3/f1a 1502075 0 2024-04-16T02:36:12.131 INFO:tasks.workunit.client.0.smithi151.stdout:3/961: rename d1/d69/d7f/fee to d1/d6/d20/d3d/da6/dd8/f13d 0 2024-04-16T02:36:12.132 INFO:tasks.workunit.client.0.smithi151.stdout:9/987: unlink d1/d3/d6/d17/d54/d8d/daa/f127 0 2024-04-16T02:36:12.135 INFO:tasks.workunit.client.0.smithi151.stdout:1/862: write d3/d58/df/d32/d3a/f66 [3344515,56243] 0 2024-04-16T02:36:12.135 INFO:tasks.workunit.client.0.smithi151.stdout:3/962: creat d1/d6/d20/d3d/f13e x:0 0 0 2024-04-16T02:36:12.137 INFO:tasks.workunit.client.0.smithi151.stdout:9/988: mkdir d1/d3/d6/d17/d54/d9b/dae/d6b/d14b 0 2024-04-16T02:36:12.138 INFO:tasks.workunit.client.0.smithi151.stdout:9/989: chown d1/d3/d6/d17/d54/d9b/fee 52365208 1 2024-04-16T02:36:12.138 INFO:tasks.workunit.client.0.smithi151.stdout:1/863: mknod d3/d58/df/dbc/c12c 0 2024-04-16T02:36:12.145 INFO:tasks.workunit.client.0.smithi151.stdout:9/990: dwrite d1/d3/d6/d17/d54/dd5/f129 [0,4194304] 0 2024-04-16T02:36:12.149 INFO:tasks.workunit.client.0.smithi151.stdout:1/864: mkdir d3/d58/df/d32/d3a/d71/d12d 0 2024-04-16T02:36:12.150 INFO:tasks.workunit.client.0.smithi151.stdout:3/963: fsync d1/d6/d20/d3d/dd4/dd6/ffa 0 2024-04-16T02:36:12.154 INFO:tasks.workunit.client.0.smithi151.stdout:1/865: rmdir d3/d33 39 2024-04-16T02:36:12.154 INFO:tasks.workunit.client.0.smithi151.stdout:8/819: sync 2024-04-16T02:36:12.154 INFO:tasks.workunit.client.0.smithi151.stdout:0/716: sync 2024-04-16T02:36:12.157 INFO:tasks.workunit.client.0.smithi151.stdout:8/820: write d11/d2a/d33/d4e/d62/d6b/db9/ff8 [2548787,49126] 0 2024-04-16T02:36:12.161 INFO:tasks.workunit.client.0.smithi151.stdout:3/964: read d1/d6/d4a/d4e/d77/f130 [3202453,103239] 0 2024-04-16T02:36:12.161 INFO:tasks.workunit.client.0.smithi151.stdout:1/866: readlink d3/d58/df/d32/d3a/d71/ded/l105 0 2024-04-16T02:36:12.162 INFO:tasks.workunit.client.0.smithi151.stdout:1/867: chown d3/d58/df/d32/dbd 48689 1 2024-04-16T02:36:12.165 INFO:tasks.workunit.client.0.smithi151.stdout:3/965: creat d1/d6/d22/d23/d41/f13f x:0 0 0 2024-04-16T02:36:12.166 INFO:tasks.workunit.client.0.smithi151.stdout:1/868: fdatasync d3/d4/d64/fee 0 2024-04-16T02:36:12.169 INFO:tasks.workunit.client.0.smithi151.stdout:9/991: dread d1/d3/d6/d17/f107 [0,4194304] 0 2024-04-16T02:36:12.170 INFO:tasks.workunit.client.0.smithi151.stdout:8/821: creat d11/d2a/d33/d4e/d62/d6b/d113/f11f x:0 0 0 2024-04-16T02:36:12.172 INFO:tasks.workunit.client.0.smithi151.stdout:9/992: write d1/d3/fa [6416105,85011] 0 2024-04-16T02:36:12.175 INFO:tasks.workunit.client.0.smithi151.stdout:0/717: link dc/d4c/d72/d76/db0/db5/fc0 dc/d15/d34/d67/d69/d8a/fe1 0 2024-04-16T02:36:12.178 INFO:tasks.workunit.client.0.smithi151.stdout:9/993: fdatasync d1/d3/d6/d28/fcd 0 2024-04-16T02:36:12.180 INFO:tasks.workunit.client.0.smithi151.stdout:1/869: creat d3/d58/df/d32/d3a/df3/d7a/dc7/dfd/f12e x:0 0 0 2024-04-16T02:36:12.182 INFO:tasks.workunit.client.0.smithi151.stdout:0/718: creat dc/d15/d34/d3b/fe2 x:0 0 0 2024-04-16T02:36:12.182 INFO:tasks.workunit.client.0.smithi151.stdout:3/966: creat d1/f140 x:0 0 0 2024-04-16T02:36:12.184 INFO:tasks.workunit.client.0.smithi151.stdout:3/967: read - d1/d89/daa/f137 zero size 2024-04-16T02:36:12.193 INFO:tasks.workunit.client.0.smithi151.stdout:9/994: dread d1/d3/d6/d17/d7a/f6d [0,4194304] 0 2024-04-16T02:36:12.194 INFO:tasks.workunit.client.0.smithi151.stdout:3/968: mknod d1/d6/d22/c141 0 2024-04-16T02:36:12.195 INFO:tasks.workunit.client.0.smithi151.stdout:9/995: read d1/d3/f33 [1585385,87561] 0 2024-04-16T02:36:12.199 INFO:tasks.workunit.client.0.smithi151.stdout:1/870: getdents d3/d58/df/d32/d3a/d71 0 2024-04-16T02:36:12.201 INFO:tasks.workunit.client.0.smithi151.stdout:3/969: mkdir d1/d6/d22/d23/d66/d142 0 2024-04-16T02:36:12.202 INFO:tasks.workunit.client.0.smithi151.stdout:9/996: dwrite d1/d3/d6/f13 [0,4194304] 0 2024-04-16T02:36:12.202 INFO:tasks.workunit.client.0.smithi151.stdout:3/970: readlink d1/d6/d20/d3d/la1 0 2024-04-16T02:36:12.204 INFO:tasks.workunit.client.0.smithi151.stdout:0/719: dread dc/d15/d34/d67/f5f [0,4194304] 0 2024-04-16T02:36:12.207 INFO:tasks.workunit.client.0.smithi151.stdout:0/720: mknod dc/d15/d34/d3b/ce3 0 2024-04-16T02:36:12.208 INFO:tasks.workunit.client.0.smithi151.stdout:9/997: dread d1/d3/d6/d17/f107 [0,4194304] 0 2024-04-16T02:36:12.210 INFO:tasks.workunit.client.0.smithi151.stdout:9/998: write d1/d3/d6/d17/d54/d9b/dae/d6b/fbc [4954595,104785] 0 2024-04-16T02:36:12.213 INFO:tasks.workunit.client.0.smithi151.stdout:0/721: rename dc/d3c to dc/d20/d7b/d9c/dd6/dd9/de4 0 2024-04-16T02:36:12.214 INFO:tasks.workunit.client.0.smithi151.stdout:1/871: link d3/d33/d10/c8e d3/d4/d64/d1c/c12f 0 2024-04-16T02:36:12.219 INFO:tasks.workunit.client.0.smithi151.stdout:9/999: chown d1/d3/c8 9 1 2024-04-16T02:36:12.220 INFO:tasks.workunit.client.0.smithi151.stdout:0/722: rename dc/d4c/f88 to dc/d20/fe5 0 2024-04-16T02:36:12.228 INFO:tasks.workunit.client.0.smithi151.stdout:3/971: dread d1/d6/d22/d23/d41/d9d/fa7 [0,4194304] 0 2024-04-16T02:36:12.229 INFO:tasks.workunit.client.0.smithi151.stdout:1/872: write d3/d33/d10/df6/fb7 [794852,5912] 0 2024-04-16T02:36:12.235 INFO:tasks.workunit.client.0.smithi151.stdout:0/723: dwrite dc/d20/fbc [0,4194304] 0 2024-04-16T02:36:12.236 INFO:tasks.workunit.client.0.smithi151.stdout:0/724: write dc/f39 [4820132,124557] 0 2024-04-16T02:36:12.237 INFO:tasks.workunit.client.0.smithi151.stdout:1/873: symlink d3/d58/df/d68/l130 0 2024-04-16T02:36:12.240 INFO:tasks.workunit.client.0.smithi151.stdout:0/725: rename fa to dc/d15/d34/d3b/d71/fe6 0 2024-04-16T02:36:12.244 INFO:tasks.workunit.client.0.smithi151.stdout:3/972: write d1/f5c [3118316,31165] 0 2024-04-16T02:36:12.245 INFO:tasks.workunit.client.0.smithi151.stdout:1/874: mkdir d3/d58/df/d32/d3a/d71/ded/d121/d131 0 2024-04-16T02:36:12.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:11 smithi151 ceph-mon[29003]: pgmap v33: 65 pgs: 65 active+clean; 2.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 35 MiB/s rd, 94 MiB/s wr, 280 op/s 2024-04-16T02:36:12.245 INFO:tasks.workunit.client.0.smithi151.stdout:1/875: chown d3/d58/df 408055 1 2024-04-16T02:36:12.246 INFO:tasks.workunit.client.0.smithi151.stdout:0/726: truncate dc/d15/d34/d4e/f83 896292 0 2024-04-16T02:36:12.249 INFO:tasks.workunit.client.0.smithi151.stdout:1/876: symlink d3/d33/d10/d4f/dd1/l132 0 2024-04-16T02:36:12.250 INFO:tasks.workunit.client.0.smithi151.stdout:3/973: creat d1/d6/d20/d3d/dd4/d107/f143 x:0 0 0 2024-04-16T02:36:12.253 INFO:tasks.workunit.client.0.smithi151.stdout:1/877: mknod d3/d33/c133 0 2024-04-16T02:36:12.260 INFO:tasks.workunit.client.0.smithi151.stdout:3/974: rename d1/d6/d20/d54/f58 to d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/f144 0 2024-04-16T02:36:12.261 INFO:tasks.workunit.client.0.smithi151.stdout:0/727: dread dc/d20/f3d [0,4194304] 0 2024-04-16T02:36:12.263 INFO:tasks.workunit.client.0.smithi151.stdout:3/975: read d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/f15 [1946514,129956] 0 2024-04-16T02:36:12.269 INFO:tasks.workunit.client.0.smithi151.stdout:3/976: symlink d1/d6/d22/d23/d66/l145 0 2024-04-16T02:36:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:11 smithi153 ceph-mon[36747]: pgmap v33: 65 pgs: 65 active+clean; 2.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 35 MiB/s rd, 94 MiB/s wr, 280 op/s 2024-04-16T02:36:12.276 INFO:tasks.workunit.client.0.smithi151.stdout:3/977: dwrite d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/f15 [8388608,4194304] 0 2024-04-16T02:36:12.279 INFO:tasks.workunit.client.0.smithi151.stdout:3/978: write d1/d6/d20/d3d/f13e [576975,122913] 0 2024-04-16T02:36:12.285 INFO:tasks.workunit.client.0.smithi151.stdout:1/878: dwrite d3/d58/df/d32/d3a/df3/d73/f85 [0,4194304] 0 2024-04-16T02:36:12.286 INFO:tasks.workunit.client.0.smithi151.stdout:3/979: fsync d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/fe5 0 2024-04-16T02:36:12.288 INFO:tasks.workunit.client.0.smithi151.stdout:0/728: dwrite dc/d15/f40 [0,4194304] 0 2024-04-16T02:36:12.293 INFO:tasks.workunit.client.0.smithi151.stdout:3/980: truncate d1/d89/f8e 559382 0 2024-04-16T02:36:12.295 INFO:tasks.workunit.client.0.smithi151.stdout:0/729: fsync dc/f47 0 2024-04-16T02:36:12.297 INFO:tasks.workunit.client.0.smithi151.stdout:3/981: truncate d1/d89/fc0 1395897 0 2024-04-16T02:36:12.297 INFO:tasks.workunit.client.0.smithi151.stdout:1/879: dwrite d3/d58/df/d32/d3a/df3/d73/f85 [4194304,4194304] 0 2024-04-16T02:36:12.299 INFO:tasks.workunit.client.0.smithi151.stdout:0/730: stat dc/d20/d7b/d9c/dd6/dd9/de4/l41 0 2024-04-16T02:36:12.301 INFO:tasks.workunit.client.0.smithi151.stdout:1/880: creat d3/d58/df/d38/f134 x:0 0 0 2024-04-16T02:36:12.304 INFO:tasks.workunit.client.0.smithi151.stdout:0/731: rename dc/d4c/d72/cca to dc/d15/d34/d67/d69/ce7 0 2024-04-16T02:36:12.305 INFO:tasks.workunit.client.0.smithi151.stdout:3/982: link d1/d6/d20/d3d/f11a d1/d6/d20/d3d/dd4/dd6/d12b/f146 0 2024-04-16T02:36:12.308 INFO:tasks.workunit.client.0.smithi151.stdout:1/881: rename d3/d33/c63 to d3/d33/d10/d4f/dd1/c135 0 2024-04-16T02:36:12.314 INFO:tasks.workunit.client.0.smithi151.stdout:1/882: dread - d3/d4/d64/d40/da3/fd0 zero size 2024-04-16T02:36:12.316 INFO:tasks.workunit.client.0.smithi151.stdout:1/883: write d3/d33/d10/d4f/dd1/fd4 [3702594,86070] 0 2024-04-16T02:36:12.318 INFO:tasks.workunit.client.0.smithi151.stdout:0/732: creat dc/fe8 x:0 0 0 2024-04-16T02:36:12.323 INFO:tasks.workunit.client.0.smithi151.stdout:1/884: dwrite d3/d58/df/d32/d3a/df3/d73/f85 [4194304,4194304] 0 2024-04-16T02:36:12.325 INFO:tasks.workunit.client.0.smithi151.stdout:1/885: write d3/d33/d10/d4f/f109 [1192817,25845] 0 2024-04-16T02:36:12.325 INFO:tasks.workunit.client.0.smithi151.stdout:3/983: creat d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/f147 x:0 0 0 2024-04-16T02:36:12.333 INFO:tasks.workunit.client.0.smithi151.stdout:4/905: sync 2024-04-16T02:36:12.333 INFO:tasks.workunit.client.0.smithi151.stdout:1/886: read - d3/d33/d10/d4f/fd8 zero size 2024-04-16T02:36:12.337 INFO:tasks.workunit.client.0.smithi151.stdout:0/733: creat dc/d15/d34/d67/fe9 x:0 0 0 2024-04-16T02:36:12.341 INFO:tasks.workunit.client.0.smithi151.stdout:4/906: mknod d0/daa/d11f/c138 0 2024-04-16T02:36:12.342 INFO:tasks.workunit.client.0.smithi151.stdout:4/907: chown d0/d11/d19/dd7/cf3 4637556 1 2024-04-16T02:36:12.343 INFO:tasks.workunit.client.0.smithi151.stdout:0/734: creat dc/d20/d7b/d9c/dd6/dd9/de4/dc2/fea x:0 0 0 2024-04-16T02:36:12.347 INFO:tasks.workunit.client.0.smithi151.stdout:1/887: write d3/d4/d64/d40/fe4 [1656831,37] 0 2024-04-16T02:36:12.349 INFO:tasks.workunit.client.0.smithi151.stdout:4/908: mkdir d0/d11/d19/d46/d139 0 2024-04-16T02:36:12.351 INFO:tasks.workunit.client.0.smithi151.stdout:3/984: dwrite d1/d6/d20/d3d/dd4/dd6/ffa [0,4194304] 0 2024-04-16T02:36:12.355 INFO:tasks.workunit.client.0.smithi151.stdout:8/822: sync 2024-04-16T02:36:12.356 INFO:tasks.workunit.client.0.smithi151.stdout:1/888: unlink d3/d4/l23 0 2024-04-16T02:36:12.361 INFO:tasks.workunit.client.0.smithi151.stdout:4/909: dwrite d0/daa/dcc/f117 [0,4194304] 0 2024-04-16T02:36:12.365 INFO:tasks.workunit.client.0.smithi151.stdout:0/735: dwrite dc/d15/d34/d4e/f55 [0,4194304] 0 2024-04-16T02:36:12.365 INFO:tasks.workunit.client.0.smithi151.stdout:4/910: chown d0/d99/f21 25 1 2024-04-16T02:36:12.367 INFO:tasks.workunit.client.0.smithi151.stdout:8/823: truncate d11/d2a/df4/d78/fad 428464 0 2024-04-16T02:36:12.369 INFO:tasks.workunit.client.0.smithi151.stdout:8/824: write d11/d2a/d33/f4b [923527,105023] 0 2024-04-16T02:36:12.370 INFO:tasks.workunit.client.0.smithi151.stdout:4/911: unlink d0/d99/f21 0 2024-04-16T02:36:12.375 INFO:tasks.workunit.client.0.smithi151.stdout:0/736: creat dc/d20/d7b/d9c/dd6/dd9/feb x:0 0 0 2024-04-16T02:36:12.379 INFO:tasks.workunit.client.0.smithi151.stdout:3/985: write d1/d6/d4a/f106 [146797,53304] 0 2024-04-16T02:36:12.380 INFO:tasks.workunit.client.0.smithi151.stdout:4/912: write d0/d11/f4b [809977,39394] 0 2024-04-16T02:36:12.381 INFO:tasks.workunit.client.0.smithi151.stdout:0/737: dread - dc/d15/d34/d67/d69/d6f/f85 zero size 2024-04-16T02:36:12.382 INFO:tasks.workunit.client.0.smithi151.stdout:8/825: fdatasync d11/d2a/df4/d78/fad 0 2024-04-16T02:36:12.383 INFO:tasks.workunit.client.0.smithi151.stdout:3/986: read d1/d6/d20/d3d/dd4/dd6/d44/f5f [3710994,40735] 0 2024-04-16T02:36:12.386 INFO:tasks.workunit.client.0.smithi151.stdout:1/889: write d3/d33/d10/f4e [5518495,43506] 0 2024-04-16T02:36:12.390 INFO:tasks.workunit.client.0.smithi151.stdout:0/738: dread - dc/d15/d34/fbf zero size 2024-04-16T02:36:12.393 INFO:tasks.workunit.client.0.smithi151.stdout:3/987: dwrite d1/d6/d22/d23/d41/f114 [0,4194304] 0 2024-04-16T02:36:12.395 INFO:tasks.workunit.client.0.smithi151.stdout:8/826: creat d11/d2a/d33/d4e/d62/d94/dfd/f120 x:0 0 0 2024-04-16T02:36:12.398 INFO:tasks.workunit.client.0.smithi151.stdout:3/988: dread - d1/d6/d20/d3d/f11a zero size 2024-04-16T02:36:12.402 INFO:tasks.workunit.client.0.smithi151.stdout:1/890: creat d3/d4/d64/d1c/f136 x:0 0 0 2024-04-16T02:36:12.403 INFO:tasks.workunit.client.0.smithi151.stdout:1/891: dread - d3/d58/df/d32/d3a/df3/d7e/f11d zero size 2024-04-16T02:36:12.403 INFO:tasks.workunit.client.0.smithi151.stdout:0/739: mknod dc/d20/d7b/d9c/dd6/dd9/de4/d9f/cec 0 2024-04-16T02:36:12.405 INFO:tasks.workunit.client.0.smithi151.stdout:1/892: chown d3/d58/df/dbc/d10e 0 1 2024-04-16T02:36:12.405 INFO:tasks.workunit.client.0.smithi151.stdout:4/913: dread d0/d11/d19/f8b [0,4194304] 0 2024-04-16T02:36:12.410 INFO:tasks.workunit.client.0.smithi151.stdout:3/989: rename d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/f15 to d1/d6/d20/f148 0 2024-04-16T02:36:12.412 INFO:tasks.workunit.client.0.smithi151.stdout:8/827: creat d11/d2a/df4/d78/d10a/f121 x:0 0 0 2024-04-16T02:36:12.413 INFO:tasks.workunit.client.0.smithi151.stdout:0/740: mkdir dc/d4c/d9d/ded 0 2024-04-16T02:36:12.413 INFO:tasks.workunit.client.0.smithi151.stdout:0/741: dread - dc/d4c/d72/fcc zero size 2024-04-16T02:36:12.414 INFO:tasks.workunit.client.0.smithi151.stdout:0/742: dread - dc/d20/fd1 zero size 2024-04-16T02:36:12.415 INFO:tasks.workunit.client.0.smithi151.stdout:1/893: mknod d3/d58/df/d32/d3a/df3/d7a/c137 0 2024-04-16T02:36:12.419 INFO:tasks.workunit.client.0.smithi151.stdout:4/914: rename d0/dde/dad/d106/dd2 to d0/d11/d19/d87/d7b/d13a 0 2024-04-16T02:36:12.419 INFO:tasks.workunit.client.0.smithi151.stdout:3/990: symlink d1/d69/d7f/dac/l149 0 2024-04-16T02:36:12.419 INFO:tasks.workunit.client.0.smithi151.stdout:4/915: read - d0/dde/dad/d106/f134 zero size 2024-04-16T02:36:12.421 INFO:tasks.workunit.client.0.smithi151.stdout:4/916: dread d0/d9b/fd0 [0,4194304] 0 2024-04-16T02:36:12.424 INFO:tasks.workunit.client.0.smithi151.stdout:4/917: write d0/d11/d19/d87/d3a/d65/d98/df8/f122 [1315643,54180] 0 2024-04-16T02:36:12.426 INFO:tasks.workunit.client.0.smithi151.stdout:1/894: dread d3/f80 [0,4194304] 0 2024-04-16T02:36:12.427 INFO:tasks.workunit.client.0.smithi151.stdout:0/743: dread - dc/d15/d34/d67/d69/d6f/d9e/fb8 zero size 2024-04-16T02:36:12.429 INFO:tasks.workunit.client.0.smithi151.stdout:3/991: dwrite d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d43/d6d/f81 [0,4194304] 0 2024-04-16T02:36:12.432 INFO:tasks.workunit.client.0.smithi151.stdout:4/918: symlink d0/d9b/l13b 0 2024-04-16T02:36:12.436 INFO:tasks.workunit.client.0.smithi151.stdout:1/895: dwrite d3/d4/d64/d40/fcf [0,4194304] 0 2024-04-16T02:36:12.438 INFO:tasks.workunit.client.0.smithi151.stdout:0/744: symlink dc/d15/d34/d3b/d71/lee 0 2024-04-16T02:36:12.439 INFO:tasks.workunit.client.0.smithi151.stdout:0/745: write dc/d20/f94 [2976022,37552] 0 2024-04-16T02:36:12.439 INFO:tasks.workunit.client.0.smithi151.stdout:8/828: rename d11/d2a/df4/d78/lbb to d11/d2a/d33/d4e/l122 0 2024-04-16T02:36:12.440 INFO:tasks.workunit.client.0.smithi151.stdout:4/919: rename d0/d11/d19 to d0/d11/d19/d87/d3a/d50/d13c 22 2024-04-16T02:36:12.441 INFO:tasks.workunit.client.0.smithi151.stdout:3/992: creat d1/d6/d20/d54/f14a x:0 0 0 2024-04-16T02:36:12.443 INFO:tasks.workunit.client.0.smithi151.stdout:4/920: readlink d0/d11/d19/d87/d3a/l12f 0 2024-04-16T02:36:12.443 INFO:tasks.workunit.client.0.smithi151.stdout:3/993: chown d1/d89/daa/d12d/f12f 2092581 1 2024-04-16T02:36:12.444 INFO:tasks.workunit.client.0.smithi151.stdout:8/829: creat d11/d2a/df4/d78/f123 x:0 0 0 2024-04-16T02:36:12.447 INFO:tasks.workunit.client.0.smithi151.stdout:1/896: mknod d3/d33/d10/d4f/dfc/c138 0 2024-04-16T02:36:12.450 INFO:tasks.workunit.client.0.smithi151.stdout:0/746: rename dc/d20/d7b/d9c to dc/d15/d34/d67/dd2/def 0 2024-04-16T02:36:12.453 INFO:tasks.workunit.client.0.smithi151.stdout:1/897: dwrite d3/d58/df/d32/f88 [0,4194304] 0 2024-04-16T02:36:12.454 INFO:tasks.workunit.client.0.smithi151.stdout:1/898: write d3/d58/df/d32/d3a/d6d/f7b [177060,10363] 0 2024-04-16T02:36:12.458 INFO:tasks.workunit.client.0.smithi151.stdout:3/994: mkdir d1/d6/d127/d14b 0 2024-04-16T02:36:12.464 INFO:tasks.workunit.client.0.smithi151.stdout:1/899: rmdir d3/d4/db2 39 2024-04-16T02:36:12.468 INFO:tasks.workunit.client.0.smithi151.stdout:0/747: fsync dc/d4c/d72/d76/db0/db5/fc0 0 2024-04-16T02:36:12.469 INFO:tasks.workunit.client.0.smithi151.stdout:1/900: dread d3/d58/df/d32/f88 [0,4194304] 0 2024-04-16T02:36:12.473 INFO:tasks.workunit.client.0.smithi151.stdout:8/830: rmdir d11/d2a/d33/d4e/d62/d6b/d103 0 2024-04-16T02:36:12.473 INFO:tasks.workunit.client.0.smithi151.stdout:8/831: chown d11/d22/d2e/f3d 42860137 1 2024-04-16T02:36:12.478 INFO:tasks.workunit.client.0.smithi151.stdout:0/748: read dc/d15/d34/d67/f30 [7081226,3074] 0 2024-04-16T02:36:12.479 INFO:tasks.workunit.client.0.smithi151.stdout:4/921: truncate d0/d11/d19/d87/d3a/d50/fdc 82165 0 2024-04-16T02:36:12.479 INFO:tasks.workunit.client.0.smithi151.stdout:1/901: rmdir d3/d33/d10/df6 39 2024-04-16T02:36:12.484 INFO:tasks.workunit.client.0.smithi151.stdout:0/749: mknod dc/d15/d34/d4e/cf0 0 2024-04-16T02:36:12.485 INFO:tasks.workunit.client.0.smithi151.stdout:4/922: symlink d0/d11/d19/d87/d3a/d65/d98/df8/l13d 0 2024-04-16T02:36:12.486 INFO:tasks.workunit.client.0.smithi151.stdout:8/832: mkdir d11/d2a/d33/d4e/d62/dc8/d11c/d124 0 2024-04-16T02:36:12.486 INFO:tasks.workunit.client.0.smithi151.stdout:3/995: dwrite d1/f1d [0,4194304] 0 2024-04-16T02:36:12.489 INFO:tasks.workunit.client.0.smithi151.stdout:4/923: write d0/d11/d19/d87/d7b/d13a/ff9 [259396,64116] 0 2024-04-16T02:36:12.493 INFO:tasks.workunit.client.0.smithi151.stdout:4/924: chown d0/dde/dad/dc8/lc9 153927504 1 2024-04-16T02:36:12.493 INFO:tasks.workunit.client.0.smithi151.stdout:4/925: dread - d0/dde/d80/f10d zero size 2024-04-16T02:36:12.494 INFO:tasks.workunit.client.0.smithi151.stdout:8/833: truncate d11/d2a/d35/d64/f88 1419094 0 2024-04-16T02:36:12.496 INFO:tasks.workunit.client.0.smithi151.stdout:4/926: mkdir d0/d9b/d13e 0 2024-04-16T02:36:12.498 INFO:tasks.workunit.client.0.smithi151.stdout:3/996: rename d1/d37/fe0 to d1/d6/d22/d23/d66/d110/f14c 0 2024-04-16T02:36:12.500 INFO:tasks.workunit.client.0.smithi151.stdout:8/834: dwrite d11/d2a/d33/f4b [0,4194304] 0 2024-04-16T02:36:12.506 INFO:tasks.workunit.client.0.smithi151.stdout:3/997: mkdir d1/d6/d22/d23/d66/d110/d14d 0 2024-04-16T02:36:12.508 INFO:tasks.workunit.client.0.smithi151.stdout:4/927: mkdir d0/d11/d19/d46/d13f 0 2024-04-16T02:36:12.508 INFO:tasks.workunit.client.0.smithi151.stdout:0/750: write dc/ff [227037,68786] 0 2024-04-16T02:36:12.509 INFO:tasks.workunit.client.0.smithi151.stdout:3/998: creat d1/d6/d127/f14e x:0 0 0 2024-04-16T02:36:12.510 INFO:tasks.workunit.client.0.smithi151.stdout:4/928: readlink d0/d11/d19/d87/d3a/d50/dc7/l12b 0 2024-04-16T02:36:12.513 INFO:tasks.workunit.client.0.smithi151.stdout:0/751: rename dc/d15/d34/d67/dd2/def/dd6/la0 to dc/d15/d34/d4e/lf1 0 2024-04-16T02:36:12.518 INFO:tasks.workunit.client.0.smithi151.stdout:0/752: chown dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f3f 478307 1 2024-04-16T02:36:12.518 INFO:tasks.workunit.client.0.smithi151.stdout:4/929: dwrite d0/d99/f121 [0,4194304] 0 2024-04-16T02:36:12.523 INFO:tasks.workunit.client.0.smithi151.stdout:3/999: mkdir d1/d6/d20/d3d/dd4/dd6/d44/d46/d8a/d3f/d119/d14f 0 2024-04-16T02:36:12.524 INFO:tasks.workunit.client.0.smithi151.stdout:4/930: chown d0/dde/d4a/dba 266769511 1 2024-04-16T02:36:12.530 INFO:tasks.workunit.client.0.smithi151.stdout:4/931: dwrite d0/d11/f4b [0,4194304] 0 2024-04-16T02:36:12.532 INFO:tasks.workunit.client.0.smithi151.stdout:0/753: unlink dc/d15/d34/d67/dd2/def/dd6/c9b 0 2024-04-16T02:36:12.535 INFO:tasks.workunit.client.0.smithi151.stdout:4/932: symlink d0/dde/dad/l140 0 2024-04-16T02:36:12.539 INFO:tasks.workunit.client.0.smithi151.stdout:4/933: dread d0/dde/d4a/d60/fda [0,4194304] 0 2024-04-16T02:36:12.539 INFO:tasks.workunit.client.0.smithi151.stdout:0/754: getdents dc/d20 0 2024-04-16T02:36:12.542 INFO:tasks.workunit.client.0.smithi151.stdout:0/755: truncate dc/d15/f58 4923690 0 2024-04-16T02:36:12.545 INFO:tasks.workunit.client.0.smithi151.stdout:4/934: link d0/dde/d4a/d60/f70 d0/d64/f141 0 2024-04-16T02:36:12.551 INFO:tasks.workunit.client.0.smithi151.stdout:0/756: creat dc/d15/d34/ff2 x:0 0 0 2024-04-16T02:36:12.556 INFO:tasks.workunit.client.0.smithi151.stdout:4/935: write d0/d11/d19/d87/f5a [2626565,74969] 0 2024-04-16T02:36:12.558 INFO:tasks.workunit.client.0.smithi151.stdout:0/757: fdatasync dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f48 0 2024-04-16T02:36:12.566 INFO:tasks.workunit.client.0.smithi151.stdout:0/758: dwrite dc/d15/d34/f4d [0,4194304] 0 2024-04-16T02:36:12.580 INFO:tasks.workunit.client.0.smithi151.stdout:4/936: dread d0/f1f [0,4194304] 0 2024-04-16T02:36:12.588 INFO:tasks.workunit.client.0.smithi151.stdout:4/937: link d0/d11/d19/d46/dcd/fea d0/dde/d80/da0/d133/f142 0 2024-04-16T02:36:12.589 INFO:tasks.workunit.client.0.smithi151.stdout:4/938: chown d0/daa/l131 1903 1 2024-04-16T02:36:12.595 INFO:tasks.workunit.client.0.smithi151.stdout:4/939: creat d0/d11/d19/d46/dcd/f143 x:0 0 0 2024-04-16T02:36:12.616 INFO:tasks.workunit.client.0.smithi151.stdout:4/940: dwrite d0/d11/d19/d87/d3a/d50/f6b [0,4194304] 0 2024-04-16T02:36:12.627 INFO:tasks.workunit.client.0.smithi151.stdout:4/941: write d0/dde/d80/fa5 [280200,96711] 0 2024-04-16T02:36:12.629 INFO:tasks.workunit.client.0.smithi151.stdout:4/942: rmdir d0/d11/d19/d46/dcd/def/dfe 39 2024-04-16T02:36:12.630 INFO:tasks.workunit.client.0.smithi151.stdout:4/943: dread - d0/dde/dad/d106/f134 zero size 2024-04-16T02:36:12.634 INFO:tasks.workunit.client.0.smithi151.stdout:4/944: rename d0/dde/d80/fc0 to d0/d99/d59/f144 0 2024-04-16T02:36:12.635 INFO:tasks.workunit.client.0.smithi151.stdout:4/945: readlink d0/d11/d19/d87/d3a/l10e 0 2024-04-16T02:36:12.638 INFO:tasks.workunit.client.0.smithi151.stdout:4/946: creat d0/d9b/d13e/f145 x:0 0 0 2024-04-16T02:36:12.639 INFO:tasks.workunit.client.0.smithi151.stdout:4/947: stat d0/d9b/c11b 0 2024-04-16T02:36:12.646 INFO:tasks.workunit.client.0.smithi151.stdout:4/948: write d0/daa/f128 [4403869,122169] 0 2024-04-16T02:36:12.648 INFO:tasks.workunit.client.0.smithi151.stdout:4/949: creat d0/dde/d4a/d60/de6/f146 x:0 0 0 2024-04-16T02:36:12.912 INFO:tasks.workunit.client.0.smithi151.stdout:1/902: sync 2024-04-16T02:36:12.920 INFO:tasks.workunit.client.0.smithi151.stdout:1/903: write d3/d1f/dd7/dda/f122 [3918774,85615] 0 2024-04-16T02:36:12.928 INFO:tasks.workunit.client.0.smithi151.stdout:1/904: dread d3/d4/d64/d1c/f81 [0,4194304] 0 2024-04-16T02:36:12.930 INFO:tasks.workunit.client.0.smithi151.stdout:1/905: getdents d3/d58/df/d32/d3a/df3/d52/dec 0 2024-04-16T02:36:12.931 INFO:tasks.workunit.client.0.smithi151.stdout:1/906: rmdir d3 39 2024-04-16T02:36:12.934 INFO:tasks.workunit.client.0.smithi151.stdout:1/907: chown d3/d4/db2/c113 13 1 2024-04-16T02:36:12.936 INFO:tasks.workunit.client.0.smithi151.stdout:1/908: mkdir d3/d58/df/d32/d3a/df3/d52/d97/d139 0 2024-04-16T02:36:12.940 INFO:tasks.workunit.client.0.smithi151.stdout:1/909: getdents d3/d58/df/d32/dbd 0 2024-04-16T02:36:12.941 INFO:tasks.workunit.client.0.smithi151.stdout:1/910: read - d3/d4/d64/d1c/f12a zero size 2024-04-16T02:36:12.942 INFO:tasks.workunit.client.0.smithi151.stdout:1/911: stat d3/d58/df/c25 0 2024-04-16T02:36:12.945 INFO:tasks.workunit.client.0.smithi151.stdout:1/912: fsync d3/d58/df/d32/d3a/df3/d73/fa2 0 2024-04-16T02:36:12.947 INFO:tasks.workunit.client.0.smithi151.stdout:1/913: mkdir d3/d58/df/d32/d3a/d67/d13a 0 2024-04-16T02:36:12.958 INFO:tasks.workunit.client.0.smithi151.stdout:1/914: truncate d3/d58/df/d32/d3a/df3/d52/f72 2245267 0 2024-04-16T02:36:12.974 INFO:tasks.workunit.client.0.smithi151.stdout:1/915: dread d3/d58/df/d38/f74 [0,4194304] 0 2024-04-16T02:36:12.976 INFO:tasks.workunit.client.0.smithi151.stdout:1/916: creat d3/d33/d10/d4f/dfc/f13b x:0 0 0 2024-04-16T02:36:12.978 INFO:tasks.workunit.client.0.smithi151.stdout:1/917: mknod d3/d58/df/d32/d3a/c13c 0 2024-04-16T02:36:12.980 INFO:tasks.workunit.client.0.smithi151.stdout:1/918: creat d3/d58/df/d32/d3a/df3/d7a/dc7/f13d x:0 0 0 2024-04-16T02:36:12.980 INFO:tasks.workunit.client.0.smithi151.stdout:1/919: stat d3/d4/d64 0 2024-04-16T02:36:12.990 INFO:tasks.workunit.client.0.smithi151.stdout:1/920: dread d3/d58/df/d38/f5a [0,4194304] 0 2024-04-16T02:36:12.991 INFO:tasks.workunit.client.0.smithi151.stdout:1/921: chown d3/fd6 3 1 2024-04-16T02:36:12.997 INFO:tasks.workunit.client.0.smithi151.stdout:1/922: dwrite d3/d4/d64/d40/fcf [0,4194304] 0 2024-04-16T02:36:13.000 INFO:tasks.workunit.client.0.smithi151.stdout:1/923: creat d3/d58/df/d32/d3a/d71/f13e x:0 0 0 2024-04-16T02:36:13.014 INFO:tasks.workunit.client.0.smithi151.stdout:8/835: sync 2024-04-16T02:36:13.014 INFO:tasks.workunit.client.0.smithi151.stdout:4/950: sync 2024-04-16T02:36:13.014 INFO:tasks.workunit.client.0.smithi151.stdout:0/759: sync 2024-04-16T02:36:13.016 INFO:tasks.workunit.client.0.smithi151.stdout:8/836: readlink d11/d2a/d35/d6e/l117 0 2024-04-16T02:36:13.016 INFO:tasks.workunit.client.0.smithi151.stdout:1/924: dwrite d3/d58/df/d32/d3a/df3/f8d [0,4194304] 0 2024-04-16T02:36:13.020 INFO:tasks.workunit.client.0.smithi151.stdout:0/760: mkdir dc/d15/d34/d67/d69/d8a/df3 0 2024-04-16T02:36:13.021 INFO:tasks.workunit.client.0.smithi151.stdout:4/951: rename d0/d11/d19/d87/d3a/d61/l69 to d0/d64/l147 0 2024-04-16T02:36:13.024 INFO:tasks.workunit.client.0.smithi151.stdout:8/837: write d11/f15 [5082718,56823] 0 2024-04-16T02:36:13.025 INFO:tasks.workunit.client.0.smithi151.stdout:0/761: mknod dc/d15/d34/d3b/d71/dc3/cf4 0 2024-04-16T02:36:13.026 INFO:tasks.workunit.client.0.smithi151.stdout:4/952: mkdir d0/daa/dcc/d124/d148 0 2024-04-16T02:36:13.027 INFO:tasks.workunit.client.0.smithi151.stdout:1/925: creat d3/f13f x:0 0 0 2024-04-16T02:36:13.033 INFO:tasks.workunit.client.0.smithi151.stdout:4/953: fdatasync d0/d99/d59/f144 0 2024-04-16T02:36:13.038 INFO:tasks.workunit.client.0.smithi151.stdout:1/926: truncate d3/d58/df/d32/d3a/df3/d52/f72 1221607 0 2024-04-16T02:36:13.044 INFO:tasks.workunit.client.0.smithi151.stdout:8/838: getdents d11/d2a/d35/d64 0 2024-04-16T02:36:13.045 INFO:tasks.workunit.client.0.smithi151.stdout:0/762: dwrite dc/d4c/f8e [0,4194304] 0 2024-04-16T02:36:13.045 INFO:tasks.workunit.client.0.smithi151.stdout:1/927: fdatasync d3/d4/d64/fa8 0 2024-04-16T02:36:13.047 INFO:tasks.workunit.client.0.smithi151.stdout:0/763: chown dc/d15/d34/d3b/d71/dc3/cf4 7703 1 2024-04-16T02:36:13.049 INFO:tasks.workunit.client.0.smithi151.stdout:4/954: mkdir d0/dde/d4a/d149 0 2024-04-16T02:36:13.050 INFO:tasks.workunit.client.0.smithi151.stdout:4/955: stat d0/d11/d19/d87/d3a/d50/f51 0 2024-04-16T02:36:13.054 INFO:tasks.workunit.client.0.smithi151.stdout:8/839: mkdir d11/d2a/d35/d64/d125 0 2024-04-16T02:36:13.056 INFO:tasks.workunit.client.0.smithi151.stdout:1/928: read d3/d4/d64/d1c/f44 [382878,34962] 0 2024-04-16T02:36:13.059 INFO:tasks.workunit.client.0.smithi151.stdout:0/764: creat dc/d15/d34/d67/d69/ff5 x:0 0 0 2024-04-16T02:36:13.061 INFO:tasks.workunit.client.0.smithi151.stdout:4/956: write d0/dde/f123 [961375,7690] 0 2024-04-16T02:36:13.063 INFO:tasks.workunit.client.0.smithi151.stdout:8/840: dwrite d11/d22/d2e/dc0/dbc/f106 [0,4194304] 0 2024-04-16T02:36:13.066 INFO:tasks.workunit.client.0.smithi151.stdout:1/929: read d3/d58/df/d32/f92 [3528565,3034] 0 2024-04-16T02:36:13.068 INFO:tasks.workunit.client.0.smithi151.stdout:1/930: write d3/d58/df/d32/d3a/df3/d52/d97/ff5 [1104191,72890] 0 2024-04-16T02:36:13.069 INFO:tasks.workunit.client.0.smithi151.stdout:0/765: dread dc/d4c/f96 [0,4194304] 0 2024-04-16T02:36:13.071 INFO:tasks.workunit.client.0.smithi151.stdout:4/957: rmdir d0/d11 39 2024-04-16T02:36:13.073 INFO:tasks.workunit.client.0.smithi151.stdout:4/958: read d0/fbb [447309,121142] 0 2024-04-16T02:36:13.074 INFO:tasks.workunit.client.0.smithi151.stdout:8/841: rename d11/l34 to d11/d2a/df4/d78/d10a/l126 0 2024-04-16T02:36:13.076 INFO:tasks.workunit.client.0.smithi151.stdout:1/931: mkdir d3/d4/d64/d140 0 2024-04-16T02:36:13.077 INFO:tasks.workunit.client.0.smithi151.stdout:1/932: chown d3/d33/d10/d4f/dfc/f13b 234171 1 2024-04-16T02:36:13.079 INFO:tasks.workunit.client.0.smithi151.stdout:1/933: write d3/d58/df/dbc/d10e/f112 [988018,16344] 0 2024-04-16T02:36:13.081 INFO:tasks.workunit.client.0.smithi151.stdout:8/842: chown d11/d2a/d33/d4e/c8e 107 1 2024-04-16T02:36:13.082 INFO:tasks.workunit.client.0.smithi151.stdout:8/843: chown d11/d2a/d35/d64/c71 114 1 2024-04-16T02:36:13.091 INFO:tasks.workunit.client.0.smithi151.stdout:1/934: rename d3/d58/df/d32/d3a/df3/d52/d97 to d3/d58/df/d32/d141 0 2024-04-16T02:36:13.092 INFO:tasks.workunit.client.0.smithi151.stdout:8/844: link d11/d2a/d33/d4e/f77 d11/d2a/d35/d52/db3/f127 0 2024-04-16T02:36:13.093 INFO:tasks.workunit.client.0.smithi151.stdout:0/766: dwrite dc/d15/f50 [0,4194304] 0 2024-04-16T02:36:13.097 INFO:tasks.workunit.client.0.smithi151.stdout:8/845: mkdir d11/d2a/df4/d78/d128 0 2024-04-16T02:36:13.100 INFO:tasks.workunit.client.0.smithi151.stdout:0/767: unlink dc/d15/d34/d67/dd2/fdf 0 2024-04-16T02:36:13.102 INFO:tasks.workunit.client.0.smithi151.stdout:1/935: symlink d3/d33/d10/l142 0 2024-04-16T02:36:13.104 INFO:tasks.workunit.client.0.smithi151.stdout:8/846: creat d11/d2a/d33/d4e/d62/dc8/d11c/f129 x:0 0 0 2024-04-16T02:36:13.106 INFO:tasks.workunit.client.0.smithi151.stdout:1/936: truncate d3/d33/d10/f78 295511 0 2024-04-16T02:36:13.107 INFO:tasks.workunit.client.0.smithi151.stdout:8/847: mknod d11/d7d/c12a 0 2024-04-16T02:36:13.109 INFO:tasks.workunit.client.0.smithi151.stdout:1/937: symlink d3/d58/df/d68/l143 0 2024-04-16T02:36:13.120 INFO:tasks.workunit.client.0.smithi151.stdout:0/768: link dc/d15/l22 dc/d15/d34/d67/lf6 0 2024-04-16T02:36:13.127 INFO:tasks.workunit.client.0.smithi151.stdout:8/848: dwrite d11/f1b [0,4194304] 0 2024-04-16T02:36:13.134 INFO:tasks.workunit.client.0.smithi151.stdout:1/938: write d3/f8a [1426939,18379] 0 2024-04-16T02:36:13.135 INFO:tasks.workunit.client.0.smithi151.stdout:8/849: dwrite d11/d2a/d35/d64/fe9 [0,4194304] 0 2024-04-16T02:36:13.139 INFO:tasks.workunit.client.0.smithi151.stdout:1/939: symlink d3/d58/df/d32/d3a/df3/d52/l144 0 2024-04-16T02:36:13.145 INFO:tasks.workunit.client.0.smithi151.stdout:1/940: truncate d3/d58/df/fc9 1179997 0 2024-04-16T02:36:13.148 INFO:tasks.workunit.client.0.smithi151.stdout:8/850: dwrite d11/d2a/df4/d78/fed [0,4194304] 0 2024-04-16T02:36:13.148 INFO:tasks.workunit.client.0.smithi151.stdout:0/769: dwrite dc/d15/d34/d4e/f6d [0,4194304] 0 2024-04-16T02:36:13.151 INFO:tasks.workunit.client.0.smithi151.stdout:8/851: dread - d11/d2a/df4/d78/f123 zero size 2024-04-16T02:36:13.159 INFO:tasks.workunit.client.0.smithi151.stdout:1/941: link d3/d58/l57 d3/d58/df/d38/l145 0 2024-04-16T02:36:13.165 INFO:tasks.workunit.client.0.smithi151.stdout:8/852: dwrite d11/d2a/d33/f48 [4194304,4194304] 0 2024-04-16T02:36:13.165 INFO:tasks.workunit.client.0.smithi151.stdout:1/942: read d3/d33/d10/f5d [4298237,104201] 0 2024-04-16T02:36:13.166 INFO:tasks.workunit.client.0.smithi151.stdout:8/853: stat d11/d2a/d35/d64/f88 0 2024-04-16T02:36:13.172 INFO:tasks.workunit.client.0.smithi151.stdout:1/943: dread d3/d58/df/f6c [0,4194304] 0 2024-04-16T02:36:13.175 INFO:tasks.workunit.client.0.smithi151.stdout:8/854: dwrite d11/d2a/df4/d78/d10a/f121 [0,4194304] 0 2024-04-16T02:36:13.184 INFO:tasks.workunit.client.0.smithi151.stdout:0/770: dread dc/d15/d34/d67/d69/d6f/f99 [0,4194304] 0 2024-04-16T02:36:13.187 INFO:tasks.workunit.client.0.smithi151.stdout:1/944: creat d3/d58/df/d32/d3a/df3/d52/d129/f146 x:0 0 0 2024-04-16T02:36:13.189 INFO:tasks.workunit.client.0.smithi151.stdout:8/855: creat d11/d2a/d33/d4e/d62/d94/d11d/f12b x:0 0 0 2024-04-16T02:36:13.190 INFO:tasks.workunit.client.0.smithi151.stdout:1/945: unlink d3/d58/df/c43 0 2024-04-16T02:36:13.192 INFO:tasks.workunit.client.0.smithi151.stdout:8/856: fsync d11/d2a/d85/f90 0 2024-04-16T02:36:13.194 INFO:tasks.workunit.client.0.smithi151.stdout:8/857: truncate d11/d2a/d33/d4e/d62/d94/dfd/f120 678915 0 2024-04-16T02:36:13.200 INFO:tasks.workunit.client.0.smithi151.stdout:8/858: dwrite d11/d2a/df4/d78/fa8 [0,4194304] 0 2024-04-16T02:36:13.201 INFO:tasks.workunit.client.0.smithi151.stdout:8/859: stat d11/d2a/l4c 0 2024-04-16T02:36:13.205 INFO:tasks.workunit.client.0.smithi151.stdout:8/860: symlink d11/d2a/d83/l12c 0 2024-04-16T02:36:13.208 INFO:tasks.workunit.client.0.smithi151.stdout:8/861: creat d11/d2a/d35/f12d x:0 0 0 2024-04-16T02:36:13.210 INFO:tasks.workunit.client.0.smithi151.stdout:8/862: fsync fb 0 2024-04-16T02:36:13.222 INFO:tasks.workunit.client.0.smithi151.stdout:8/863: dread d11/d2a/d33/f66 [0,4194304] 0 2024-04-16T02:36:13.224 INFO:tasks.workunit.client.0.smithi151.stdout:8/864: symlink d11/d2a/d85/l12e 0 2024-04-16T02:36:13.226 INFO:tasks.workunit.client.0.smithi151.stdout:8/865: truncate d11/d2a/d33/d4e/d62/dc8/d11c/f129 211901 0 2024-04-16T02:36:13.226 INFO:tasks.workunit.client.0.smithi151.stdout:8/866: chown d11/d22/d6c/d84/fc7 5 1 2024-04-16T02:36:13.228 INFO:tasks.workunit.client.0.smithi151.stdout:8/867: symlink d11/d2a/d83/l12f 0 2024-04-16T02:36:13.229 INFO:tasks.workunit.client.0.smithi151.stdout:8/868: stat d11/d2a/d33/d4e/l122 0 2024-04-16T02:36:13.232 INFO:tasks.workunit.client.0.smithi151.stdout:8/869: rename ff to d11/d2a/d35/d64/f130 0 2024-04-16T02:36:13.233 INFO:tasks.workunit.client.0.smithi151.stdout:8/870: fsync d11/d2a/df4/d78/f7b 0 2024-04-16T02:36:13.239 INFO:tasks.workunit.client.0.smithi151.stdout:8/871: dwrite d11/d22/f102 [0,4194304] 0 2024-04-16T02:36:13.240 INFO:tasks.workunit.client.0.smithi151.stdout:8/872: fdatasync d11/d22/d2e/dc0/fe6 0 2024-04-16T02:36:13.242 INFO:tasks.workunit.client.0.smithi151.stdout:8/873: dread d11/d2a/df4/d78/fad [0,4194304] 0 2024-04-16T02:36:13.246 INFO:tasks.workunit.client.0.smithi151.stdout:8/874: getdents d11/d2a/d33/d4e 0 2024-04-16T02:36:13.247 INFO:tasks.workunit.client.0.smithi151.stdout:8/875: read fb [2595614,67649] 0 2024-04-16T02:36:13.248 INFO:tasks.workunit.client.0.smithi151.stdout:8/876: mknod d11/d22/d2e/dc0/c131 0 2024-04-16T02:36:13.253 INFO:tasks.workunit.client.0.smithi151.stdout:8/877: dread d11/d2a/df4/fbe [0,4194304] 0 2024-04-16T02:36:13.254 INFO:tasks.workunit.client.0.smithi151.stdout:8/878: write d11/d2a/db1/f101 [99822,18145] 0 2024-04-16T02:36:13.260 INFO:tasks.workunit.client.0.smithi151.stdout:8/879: rmdir d11/d2a/d33 39 2024-04-16T02:36:13.265 INFO:tasks.workunit.client.0.smithi151.stdout:8/880: truncate d11/d2a/d35/f40 3673752 0 2024-04-16T02:36:13.269 INFO:tasks.workunit.client.0.smithi151.stdout:8/881: truncate d11/d2a/d33/f66 2044766 0 2024-04-16T02:36:13.269 INFO:tasks.workunit.client.0.smithi151.stdout:8/882: stat d11/d22/d2e 0 2024-04-16T02:36:13.271 INFO:tasks.workunit.client.0.smithi151.stdout:8/883: write d11/d2a/d33/d4e/d62/d6b/db9/ff8 [2233087,31094] 0 2024-04-16T02:36:13.273 INFO:tasks.workunit.client.0.smithi151.stdout:8/884: dread d11/d2a/df4/d78/fad [0,4194304] 0 2024-04-16T02:36:13.275 INFO:tasks.workunit.client.0.smithi151.stdout:1/946: dread d3/d58/df/d32/d3a/fcc [0,4194304] 0 2024-04-16T02:36:13.276 INFO:tasks.workunit.client.0.smithi151.stdout:8/885: creat d11/d22/d7e/f132 x:0 0 0 2024-04-16T02:36:13.277 INFO:tasks.workunit.client.0.smithi151.stdout:8/886: chown d11/d22/d7e/fc1 6757 1 2024-04-16T02:36:13.277 INFO:tasks.workunit.client.0.smithi151.stdout:1/947: truncate d3/d4/d64/d1c/f12a 368721 0 2024-04-16T02:36:13.280 INFO:tasks.workunit.client.0.smithi151.stdout:1/948: rmdir d3/d58/df/dbc 39 2024-04-16T02:36:13.283 INFO:tasks.workunit.client.0.smithi151.stdout:8/887: dwrite d11/d2a/d33/f48 [4194304,4194304] 0 2024-04-16T02:36:13.286 INFO:tasks.workunit.client.0.smithi151.stdout:1/949: link d3/d58/df/f114 d3/d58/df/d32/d141/f147 0 2024-04-16T02:36:13.294 INFO:tasks.workunit.client.0.smithi151.stdout:8/888: dwrite d11/d2a/d85/ffb [0,4194304] 0 2024-04-16T02:36:13.299 INFO:tasks.workunit.client.0.smithi151.stdout:8/889: unlink d11/d2a/df4/d78/fad 0 2024-04-16T02:36:13.305 INFO:tasks.workunit.client.0.smithi151.stdout:8/890: truncate d11/d2a/d35/d64/fef 362552 0 2024-04-16T02:36:13.307 INFO:tasks.workunit.client.0.smithi151.stdout:1/950: write d3/d4/d64/d1c/f50 [1850144,7387] 0 2024-04-16T02:36:13.310 INFO:tasks.workunit.client.0.smithi151.stdout:1/951: readlink d3/d58/df/d32/d3a/df3/d7a/dc7/l106 0 2024-04-16T02:36:13.312 INFO:tasks.workunit.client.0.smithi151.stdout:8/891: dwrite d11/fdd [0,4194304] 0 2024-04-16T02:36:13.319 INFO:tasks.workunit.client.0.smithi151.stdout:1/952: link d3/d4/d64/fa4 d3/d58/df/d32/d3a/df3/d7e/f148 0 2024-04-16T02:36:13.321 INFO:tasks.workunit.client.0.smithi151.stdout:8/892: link d11/d2a/df4/fbe d11/d2a/d33/d4e/d62/dc8/f133 0 2024-04-16T02:36:13.325 INFO:tasks.workunit.client.0.smithi151.stdout:1/953: fsync d3/d58/df/d32/d3a/df3/d7a/df9/f101 0 2024-04-16T02:36:13.329 INFO:tasks.workunit.client.0.smithi151.stdout:1/954: symlink d3/d33/d10/d4f/l149 0 2024-04-16T02:36:13.342 INFO:tasks.workunit.client.0.smithi151.stdout:8/893: dread d11/f1a [0,4194304] 0 2024-04-16T02:36:13.349 INFO:tasks.workunit.client.0.smithi151.stdout:1/955: dwrite d3/d1f/dd7/ddb/fe7 [0,4194304] 0 2024-04-16T02:36:13.351 INFO:tasks.workunit.client.0.smithi151.stdout:8/894: dwrite d11/d2a/d33/d4e/d62/d94/d11d/f12b [0,4194304] 0 2024-04-16T02:36:13.355 INFO:tasks.workunit.client.0.smithi151.stdout:1/956: chown d3/d58/df/fc9 3086 1 2024-04-16T02:36:13.356 INFO:tasks.workunit.client.0.smithi151.stdout:8/895: mkdir d11/d22/d6c/d134 0 2024-04-16T02:36:13.359 INFO:tasks.workunit.client.0.smithi151.stdout:8/896: mknod d11/c135 0 2024-04-16T02:36:13.362 INFO:tasks.workunit.client.0.smithi151.stdout:1/957: rename d3/d33/d10/df6/fb7 to d3/d58/df/d38/d12b/f14a 0 2024-04-16T02:36:13.363 INFO:tasks.workunit.client.0.smithi151.stdout:8/897: rmdir d11/d2a/d35/d52/db3/dcc 39 2024-04-16T02:36:13.363 INFO:tasks.workunit.client.0.smithi151.stdout:1/958: fdatasync d3/d33/d10/f4e 0 2024-04-16T02:36:13.367 INFO:tasks.workunit.client.0.smithi151.stdout:1/959: fdatasync d3/d4/d64/d1c/f90 0 2024-04-16T02:36:13.372 INFO:tasks.workunit.client.0.smithi151.stdout:1/960: link d3/l4b d3/d58/df/d32/dbd/l14b 0 2024-04-16T02:36:13.375 INFO:tasks.workunit.client.0.smithi151.stdout:1/961: getdents d3/d58/df/d32/d3a/d71/ded 0 2024-04-16T02:36:13.378 INFO:tasks.workunit.client.0.smithi151.stdout:1/962: creat d3/d58/df/d38/f14c x:0 0 0 2024-04-16T02:36:13.383 INFO:tasks.workunit.client.0.smithi151.stdout:8/898: dread d11/d2a/d33/f45 [0,4194304] 0 2024-04-16T02:36:13.385 INFO:tasks.workunit.client.0.smithi151.stdout:1/963: dwrite d3/d4/d64/d40/fe4 [0,4194304] 0 2024-04-16T02:36:13.386 INFO:tasks.workunit.client.0.smithi151.stdout:8/899: creat d11/d2a/d83/dce/f136 x:0 0 0 2024-04-16T02:36:13.389 INFO:tasks.workunit.client.0.smithi151.stdout:1/964: mknod d3/d58/df/d32/d3a/df3/d52/c14d 0 2024-04-16T02:36:13.393 INFO:tasks.workunit.client.0.smithi151.stdout:1/965: symlink d3/d58/df/dbc/l14e 0 2024-04-16T02:36:13.393 INFO:tasks.workunit.client.0.smithi151.stdout:1/966: chown d3/d33/d10/d4f/dd1/cb5 3815227 1 2024-04-16T02:36:13.396 INFO:tasks.workunit.client.0.smithi151.stdout:1/967: creat d3/d1f/dd7/ddb/f14f x:0 0 0 2024-04-16T02:36:13.399 INFO:tasks.workunit.client.0.smithi151.stdout:1/968: dread d3/d4/d64/fa8 [0,4194304] 0 2024-04-16T02:36:13.402 INFO:tasks.workunit.client.0.smithi151.stdout:1/969: fsync d3/d58/df/d32/d3a/fcc 0 2024-04-16T02:36:13.404 INFO:tasks.workunit.client.0.smithi151.stdout:1/970: write d3/d58/df/f110 [531907,22002] 0 2024-04-16T02:36:13.405 INFO:tasks.workunit.client.0.smithi151.stdout:8/900: write d11/d2a/d33/d4e/d62/dc8/fe5 [495032,33360] 0 2024-04-16T02:36:13.407 INFO:tasks.workunit.client.0.smithi151.stdout:8/901: rename d11/d2a/d35/d52/db3 to d11/d2a/d35/d52/db3/dcd/d137 22 2024-04-16T02:36:13.410 INFO:tasks.workunit.client.0.smithi151.stdout:8/902: write d11/d2a/d33/f10b [9038614,68880] 0 2024-04-16T02:36:13.414 INFO:tasks.workunit.client.0.smithi151.stdout:1/971: unlink d3/d58/df/d32/d3a/d6d/lf0 0 2024-04-16T02:36:13.416 INFO:tasks.workunit.client.0.smithi151.stdout:8/903: dwrite d11/d2a/d33/d4e/d62/d94/dfd/f120 [0,4194304] 0 2024-04-16T02:36:13.419 INFO:tasks.workunit.client.0.smithi151.stdout:8/904: fdatasync d11/d22/d2e/f65 0 2024-04-16T02:36:13.419 INFO:tasks.workunit.client.0.smithi151.stdout:1/972: getdents d3/d58/df/d32/d3a/d67/d13a 0 2024-04-16T02:36:13.422 INFO:tasks.workunit.client.0.smithi151.stdout:8/905: chown d11/d2a/d33/d4e/f77 23910 1 2024-04-16T02:36:13.425 INFO:tasks.workunit.client.0.smithi151.stdout:1/973: rename d3/d4/d64/d1c/f50 to d3/d33/d10/df6/f150 0 2024-04-16T02:36:13.429 INFO:tasks.workunit.client.0.smithi151.stdout:8/906: dwrite d11/d2a/df4/d78/d10a/f121 [4194304,4194304] 0 2024-04-16T02:36:13.433 INFO:tasks.workunit.client.0.smithi151.stdout:1/974: creat d3/d58/df/d32/d141/d139/f151 x:0 0 0 2024-04-16T02:36:13.434 INFO:tasks.workunit.client.0.smithi151.stdout:8/907: symlink d11/d2a/d33/d4e/d62/dc8/l138 0 2024-04-16T02:36:13.435 INFO:tasks.workunit.client.0.smithi151.stdout:8/908: stat d11/d2a/d33/d4e/f7c 0 2024-04-16T02:36:13.437 INFO:tasks.workunit.client.0.smithi151.stdout:1/975: mkdir d3/d58/df/d68/d152 0 2024-04-16T02:36:13.439 INFO:tasks.workunit.client.0.smithi151.stdout:8/909: mknod d11/d22/d2e/c139 0 2024-04-16T02:36:13.440 INFO:tasks.workunit.client.0.smithi151.stdout:1/976: creat d3/d1f/dd7/ddb/f153 x:0 0 0 2024-04-16T02:36:13.442 INFO:tasks.workunit.client.0.smithi151.stdout:8/910: truncate d11/f1a 1522900 0 2024-04-16T02:36:13.444 INFO:tasks.workunit.client.0.smithi151.stdout:8/911: creat d11/d2a/d33/d4e/d62/d94/f13a x:0 0 0 2024-04-16T02:36:13.452 INFO:tasks.workunit.client.0.smithi151.stdout:1/977: write d3/d4/d64/fee [1017990,117868] 0 2024-04-16T02:36:13.455 INFO:tasks.workunit.client.0.smithi151.stdout:1/978: readlink d3/d33/l14 0 2024-04-16T02:36:13.457 INFO:tasks.workunit.client.0.smithi151.stdout:8/912: dread d11/d2a/d35/d64/f74 [0,4194304] 0 2024-04-16T02:36:13.458 INFO:tasks.workunit.client.0.smithi151.stdout:1/979: creat d3/d58/df/d32/dbd/f154 x:0 0 0 2024-04-16T02:36:13.458 INFO:tasks.workunit.client.0.smithi151.stdout:8/913: chown d11/d2a/d33/d4e/d62/d6b/d113 118139 1 2024-04-16T02:36:13.460 INFO:tasks.workunit.client.0.smithi151.stdout:1/980: mkdir d3/d4/d64/d1c/d155 0 2024-04-16T02:36:13.467 INFO:tasks.workunit.client.0.smithi151.stdout:1/981: dwrite d3/d58/df/d32/d3a/df3/d7e/f11d [0,4194304] 0 2024-04-16T02:36:13.473 INFO:tasks.workunit.client.0.smithi151.stdout:8/914: write d11/d2a/df4/f93 [599893,7223] 0 2024-04-16T02:36:13.474 INFO:tasks.workunit.client.0.smithi151.stdout:8/915: readlink d11/d2a/d35/d52/l5f 0 2024-04-16T02:36:13.474 INFO:tasks.workunit.client.0.smithi151.stdout:8/916: stat d11/d2a/l109 0 2024-04-16T02:36:13.476 INFO:tasks.workunit.client.0.smithi151.stdout:1/982: link d3/d58/df/d32/d3a/df3/d52/c14d d3/d58/df/d32/d3a/df3/c156 0 2024-04-16T02:36:13.478 INFO:tasks.workunit.client.0.smithi151.stdout:8/917: rename d11/d2a/d35/d6e to d11/d22/d2e/dc0/d13b 0 2024-04-16T02:36:13.479 INFO:tasks.workunit.client.0.smithi151.stdout:1/983: fsync d3/d1f/dd7/dda/f122 0 2024-04-16T02:36:13.483 INFO:tasks.workunit.client.0.smithi151.stdout:1/984: read d3/d33/d10/f78 [48824,25797] 0 2024-04-16T02:36:13.485 INFO:tasks.workunit.client.0.smithi151.stdout:8/918: dwrite d11/d2a/d33/f4b [0,4194304] 0 2024-04-16T02:36:13.489 INFO:tasks.workunit.client.0.smithi151.stdout:1/985: creat d3/d58/df/d32/d3a/df3/d7a/f157 x:0 0 0 2024-04-16T02:36:13.490 INFO:tasks.workunit.client.0.smithi151.stdout:8/919: creat d11/d2a/d33/d4e/d62/d6b/db9/f13c x:0 0 0 2024-04-16T02:36:13.494 INFO:tasks.workunit.client.0.smithi151.stdout:1/986: unlink d3/d58/df/d32/d3a/df3/d7e/cd2 0 2024-04-16T02:36:13.498 INFO:tasks.workunit.client.0.smithi151.stdout:8/920: dwrite d11/d2a/d33/d4e/d62/d94/dfd/f120 [0,4194304] 0 2024-04-16T02:36:13.500 INFO:tasks.workunit.client.0.smithi151.stdout:8/921: write d11/d22/d6c/d84/fe7 [889110,129611] 0 2024-04-16T02:36:13.502 INFO:tasks.workunit.client.0.smithi151.stdout:8/922: write d11/d22/d2e/dc0/dbc/f106 [495839,63523] 0 2024-04-16T02:36:13.506 INFO:tasks.workunit.client.0.smithi151.stdout:8/923: mkdir d11/d22/d2e/d13d 0 2024-04-16T02:36:13.509 INFO:tasks.workunit.client.0.smithi151.stdout:1/987: write d3/d33/d10/f55 [116460,23957] 0 2024-04-16T02:36:13.513 INFO:tasks.workunit.client.0.smithi151.stdout:1/988: mkdir d3/d58/df/dbc/d10e/d158 0 2024-04-16T02:36:13.514 INFO:tasks.workunit.client.0.smithi151.stdout:8/924: dwrite d11/d2a/d33/d4e/d62/dc8/fe5 [0,4194304] 0 2024-04-16T02:36:13.519 INFO:tasks.workunit.client.0.smithi151.stdout:1/989: mkdir d3/d159 0 2024-04-16T02:36:13.526 INFO:tasks.workunit.client.0.smithi151.stdout:1/990: dwrite d3/d58/df/d32/d3a/df3/d73/f85 [4194304,4194304] 0 2024-04-16T02:36:13.535 INFO:tasks.workunit.client.0.smithi151.stdout:8/925: dwrite d11/d22/d7e/f81 [0,4194304] 0 2024-04-16T02:36:13.536 INFO:tasks.workunit.client.0.smithi151.stdout:1/991: rename d3/d58/df/d32/d141 to d3/d4/d64/d40/da3/d15a 0 2024-04-16T02:36:13.538 INFO:tasks.workunit.client.0.smithi151.stdout:8/926: write d11/d2a/d35/d64/f10f [390234,87575] 0 2024-04-16T02:36:13.542 INFO:tasks.workunit.client.0.smithi151.stdout:1/992: creat d3/d1f/dd7/dda/f15b x:0 0 0 2024-04-16T02:36:13.543 INFO:tasks.workunit.client.0.smithi151.stdout:1/993: chown d3/d1f/dd7/ddb/f14f 8164745 1 2024-04-16T02:36:13.551 INFO:tasks.workunit.client.0.smithi151.stdout:8/927: dread d11/d2a/d35/d52/db3/fcb [0,4194304] 0 2024-04-16T02:36:13.552 INFO:tasks.workunit.client.0.smithi151.stdout:8/928: read d11/d2a/d35/d64/f88 [553893,22800] 0 2024-04-16T02:36:13.557 INFO:tasks.workunit.client.0.smithi151.stdout:8/929: rename d11/d2a/d35/d64/f74 to d11/d2a/d33/d4e/d62/d94/d11d/f13e 0 2024-04-16T02:36:13.561 INFO:tasks.workunit.client.0.smithi151.stdout:8/930: rename d11/d2a/d85/c115 to d11/d2a/d33/d4e/d62/d94/dfd/c13f 0 2024-04-16T02:36:13.562 INFO:tasks.workunit.client.0.smithi151.stdout:1/994: dwrite d3/d58/df/fc9 [0,4194304] 0 2024-04-16T02:36:13.562 INFO:tasks.workunit.client.0.smithi151.stdout:8/931: stat d11/d2a/d33/d4e/d62/db7/lea 0 2024-04-16T02:36:13.569 INFO:tasks.workunit.client.0.smithi151.stdout:1/995: rmdir d3/d58 39 2024-04-16T02:36:13.572 INFO:tasks.workunit.client.0.smithi151.stdout:8/932: truncate d11/d2a/d35/d64/f7f 1658128 0 2024-04-16T02:36:13.576 INFO:tasks.workunit.client.0.smithi151.stdout:8/933: creat d11/d22/d2e/dc0/d13b/f140 x:0 0 0 2024-04-16T02:36:13.579 INFO:tasks.workunit.client.0.smithi151.stdout:1/996: mkdir d3/d58/df/d15c 0 2024-04-16T02:36:13.581 INFO:tasks.workunit.client.0.smithi151.stdout:1/997: fsync d3/d58/df/d68/f7c 0 2024-04-16T02:36:13.584 INFO:tasks.workunit.client.0.smithi151.stdout:0/771: sync 2024-04-16T02:36:13.584 INFO:tasks.workunit.client.0.smithi151.stdout:4/959: sync 2024-04-16T02:36:13.588 INFO:tasks.workunit.client.0.smithi151.stdout:1/998: creat d3/d33/d10/d4f/dd1/f15d x:0 0 0 2024-04-16T02:36:13.591 INFO:tasks.workunit.client.0.smithi151.stdout:0/772: rename dc/d4c/d72/d76/db0/db5/fc0 to dc/d15/d34/d67/dd2/def/dd6/dd9/ff7 0 2024-04-16T02:36:13.595 INFO:tasks.workunit.client.0.smithi151.stdout:0/773: read dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f75 [333608,61528] 0 2024-04-16T02:36:13.596 INFO:tasks.workunit.client.0.smithi151.stdout:4/960: truncate d0/d11/d19/d87/d3a/d50/fdc 111069 0 2024-04-16T02:36:13.596 INFO:tasks.workunit.client.0.smithi151.stdout:0/774: chown dc/d15/d34/d67/d51/fb9 1950 1 2024-04-16T02:36:13.600 INFO:tasks.workunit.client.0.smithi151.stdout:0/775: mkdir dc/dde/df8 0 2024-04-16T02:36:13.602 INFO:tasks.workunit.client.0.smithi151.stdout:0/776: rename dc/d15/d34/fbf to dc/d4c/d72/d76/db0/db5/ff9 0 2024-04-16T02:36:13.605 INFO:tasks.workunit.client.0.smithi151.stdout:0/777: creat dc/d15/d34/d67/d69/d8a/ffa x:0 0 0 2024-04-16T02:36:13.606 INFO:tasks.workunit.client.0.smithi151.stdout:0/778: chown dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f3f 105891381 1 2024-04-16T02:36:13.609 INFO:tasks.workunit.client.0.smithi151.stdout:4/961: dwrite d0/dde/d68/f7a [0,4194304] 0 2024-04-16T02:36:13.612 INFO:tasks.workunit.client.0.smithi151.stdout:0/779: fdatasync dc/d15/d34/d67/f78 0 2024-04-16T02:36:13.613 INFO:tasks.workunit.client.0.smithi151.stdout:4/962: symlink d0/d99/l14a 0 2024-04-16T02:36:13.618 INFO:tasks.workunit.client.0.smithi151.stdout:0/780: mknod dc/d4c/cfb 0 2024-04-16T02:36:13.619 INFO:tasks.workunit.client.0.smithi151.stdout:0/781: fdatasync dc/d15/d5c/f6a 0 2024-04-16T02:36:13.621 INFO:tasks.workunit.client.0.smithi151.stdout:0/782: dread - dc/d15/d34/d67/dd2/def/dd6/dd9/de4/dc2/fea zero size 2024-04-16T02:36:14.034 INFO:tasks.workunit.client.0.smithi151.stdout:8/934: sync 2024-04-16T02:36:14.039 INFO:tasks.workunit.client.0.smithi151.stdout:8/935: fsync d11/d2a/d33/d4e/d62/dc8/f133 0 2024-04-16T02:36:14.043 INFO:tasks.workunit.client.0.smithi151.stdout:8/936: fsync d11/d22/d6c/d84/dc9/f107 0 2024-04-16T02:36:14.049 INFO:tasks.workunit.client.0.smithi151.stdout:8/937: rename d11/d22/f76 to d11/d2a/d35/f141 0 2024-04-16T02:36:14.055 INFO:tasks.workunit.client.0.smithi151.stdout:8/938: dwrite d11/d2a/d33/d4e/d62/dc8/fe5 [0,4194304] 0 2024-04-16T02:36:14.066 INFO:tasks.workunit.client.0.smithi151.stdout:8/939: write d11/d2a/d35/d52/db3/fb6 [275867,49600] 0 2024-04-16T02:36:14.067 INFO:tasks.workunit.client.0.smithi151.stdout:8/940: fsync d11/f1b 0 2024-04-16T02:36:14.068 INFO:tasks.workunit.client.0.smithi151.stdout:1/999: sync 2024-04-16T02:36:14.068 INFO:tasks.workunit.client.0.smithi151.stdout:4/963: sync 2024-04-16T02:36:14.068 INFO:tasks.workunit.client.0.smithi151.stdout:0/783: sync 2024-04-16T02:36:14.070 INFO:tasks.workunit.client.0.smithi151.stdout:8/941: creat d11/d7d/f142 x:0 0 0 2024-04-16T02:36:14.071 INFO:tasks.workunit.client.0.smithi151.stdout:8/942: fdatasync d11/d22/d6c/d84/fe7 0 2024-04-16T02:36:14.074 INFO:tasks.workunit.client.0.smithi151.stdout:0/784: unlink dc/l68 0 2024-04-16T02:36:14.075 INFO:tasks.workunit.client.0.smithi151.stdout:0/785: stat dc/d15/d34/d67/d51/f91 0 2024-04-16T02:36:14.075 INFO:tasks.workunit.client.0.smithi151.stdout:8/943: dread d11/d22/f96 [0,4194304] 0 2024-04-16T02:36:14.077 INFO:tasks.workunit.client.0.smithi151.stdout:4/964: mknod d0/dde/d4a/c14b 0 2024-04-16T02:36:14.078 INFO:tasks.workunit.client.0.smithi151.stdout:4/965: fsync d0/d11/d19/d46/f58 0 2024-04-16T02:36:14.079 INFO:tasks.workunit.client.0.smithi151.stdout:0/786: truncate dc/d15/fd4 51919 0 2024-04-16T02:36:14.084 INFO:tasks.workunit.client.0.smithi151.stdout:4/966: creat d0/d73/f14c x:0 0 0 2024-04-16T02:36:14.091 INFO:tasks.workunit.client.0.smithi151.stdout:4/967: mkdir d0/dde/d4a/d149/d14d 0 2024-04-16T02:36:14.094 INFO:tasks.workunit.client.0.smithi151.stdout:0/787: getdents dc/d15/d34/d67/dd2/def/dd6/dd9/de4/dc2 0 2024-04-16T02:36:14.096 INFO:tasks.workunit.client.0.smithi151.stdout:8/944: write d11/d22/fa1 [719334,114415] 0 2024-04-16T02:36:14.098 INFO:tasks.workunit.client.0.smithi151.stdout:8/945: read fb [7715202,41037] 0 2024-04-16T02:36:14.099 INFO:tasks.workunit.client.0.smithi151.stdout:0/788: symlink dc/d15/d34/d67/d69/d6f/d9e/lfc 0 2024-04-16T02:36:14.104 INFO:tasks.workunit.client.0.smithi151.stdout:0/789: symlink dc/d15/d34/d67/d69/lfd 0 2024-04-16T02:36:14.104 INFO:tasks.workunit.client.0.smithi151.stdout:0/790: chown dc/d15/d34/d67/dd2/def/fd8 23989809 1 2024-04-16T02:36:14.108 INFO:tasks.workunit.client.0.smithi151.stdout:0/791: mknod dc/d15/d34/d67/dd2/cfe 0 2024-04-16T02:36:14.108 INFO:tasks.workunit.client.0.smithi151.stdout:4/968: write d0/d64/f71 [5094453,73615] 0 2024-04-16T02:36:14.110 INFO:tasks.workunit.client.0.smithi151.stdout:8/946: mkdir d11/d22/d2e/d13d/d143 0 2024-04-16T02:36:14.117 INFO:tasks.workunit.client.0.smithi151.stdout:8/947: mkdir d11/d2a/d33/d4e/d62/db7/d144 0 2024-04-16T02:36:14.123 INFO:tasks.workunit.client.0.smithi151.stdout:0/792: write dc/d20/f42 [3856774,77678] 0 2024-04-16T02:36:14.126 INFO:tasks.workunit.client.0.smithi151.stdout:0/793: mkdir dc/d15/d34/d67/dd2/dff 0 2024-04-16T02:36:14.128 INFO:tasks.workunit.client.0.smithi151.stdout:4/969: dread d0/d11/d19/d46/f6f [0,4194304] 0 2024-04-16T02:36:14.128 INFO:tasks.workunit.client.0.smithi151.stdout:4/970: readlink d0/daa/l131 0 2024-04-16T02:36:14.130 INFO:tasks.workunit.client.0.smithi151.stdout:0/794: creat dc/d15/d34/d67/d69/d8a/f100 x:0 0 0 2024-04-16T02:36:14.133 INFO:tasks.workunit.client.0.smithi151.stdout:4/971: read d0/d99/f4f [48679,43151] 0 2024-04-16T02:36:14.135 INFO:tasks.workunit.client.0.smithi151.stdout:4/972: chown d0/d99/l14a 0 1 2024-04-16T02:36:14.136 INFO:tasks.workunit.client.0.smithi151.stdout:8/948: dwrite d11/d2a/d35/d52/db3/fcb [0,4194304] 0 2024-04-16T02:36:14.143 INFO:tasks.workunit.client.0.smithi151.stdout:8/949: dwrite d11/f1b [0,4194304] 0 2024-04-16T02:36:14.145 INFO:tasks.workunit.client.0.smithi151.stdout:8/950: write d11/d2a/d83/dce/f118 [136148,58271] 0 2024-04-16T02:36:14.146 INFO:tasks.workunit.client.0.smithi151.stdout:4/973: link d0/d11/d19/d46/dcd/def/f119 d0/d11/d19/d87/d7b/d13a/f14e 0 2024-04-16T02:36:14.154 INFO:tasks.workunit.client.0.smithi151.stdout:8/951: getdents d11/d2a/d33/d4e/d62/db7/d144 0 2024-04-16T02:36:14.155 INFO:tasks.workunit.client.0.smithi151.stdout:4/974: stat d0/dde/d4a/d60/f115 0 2024-04-16T02:36:14.157 INFO:tasks.workunit.client.0.smithi151.stdout:4/975: chown d0/d11/d19/d87/d3a/d65/d98/df8/f122 55981 1 2024-04-16T02:36:14.161 INFO:tasks.workunit.client.0.smithi151.stdout:4/976: unlink d0/dde/d4a/d60/f70 0 2024-04-16T02:36:14.176 INFO:tasks.workunit.client.0.smithi151.stdout:8/952: dwrite fb [4194304,4194304] 0 2024-04-16T02:36:14.184 INFO:tasks.workunit.client.0.smithi151.stdout:4/977: dwrite d0/d11/d19/d87/d3a/d65/d98/df8/f104 [0,4194304] 0 2024-04-16T02:36:14.184 INFO:tasks.workunit.client.0.smithi151.stdout:4/978: readlink d0/l4 0 2024-04-16T02:36:14.195 INFO:tasks.workunit.client.0.smithi151.stdout:4/979: rmdir d0/d11/d19/dd7 39 2024-04-16T02:36:14.200 INFO:tasks.workunit.client.0.smithi151.stdout:4/980: chown d0/d11/d19/d87/d3a/c56 171838914 1 2024-04-16T02:36:14.204 INFO:tasks.workunit.client.0.smithi151.stdout:8/953: rename d11/d22/d6c/c75 to d11/d2a/c145 0 2024-04-16T02:36:14.205 INFO:tasks.workunit.client.0.smithi151.stdout:4/981: dread d0/dde/d80/da0/fac [0,4194304] 0 2024-04-16T02:36:14.209 INFO:tasks.workunit.client.0.smithi151.stdout:8/954: mknod d11/d2a/d35/d52/db3/dcd/c146 0 2024-04-16T02:36:14.215 INFO:tasks.workunit.client.0.smithi151.stdout:4/982: truncate d0/d11/d19/d87/d3a/fe8 41578 0 2024-04-16T02:36:14.220 INFO:tasks.workunit.client.0.smithi151.stdout:4/983: fdatasync d0/d11/d19/d87/d3a/d50/fdc 0 2024-04-16T02:36:14.222 INFO:tasks.workunit.client.0.smithi151.stdout:8/955: write d11/d2a/d33/f66 [207471,30894] 0 2024-04-16T02:36:14.235 INFO:tasks.workunit.client.0.smithi151.stdout:4/984: rename d0/fbf to d0/d11/f14f 0 2024-04-16T02:36:14.241 INFO:tasks.workunit.client.0.smithi151.stdout:4/985: unlink d0/d11/d19/dd7/fe7 0 2024-04-16T02:36:14.241 INFO:tasks.workunit.client.0.smithi151.stdout:8/956: dwrite d11/d2a/d33/d4e/d62/d94/d11d/f13e [0,4194304] 0 2024-04-16T02:36:14.243 INFO:tasks.workunit.client.0.smithi151.stdout:4/986: mkdir d0/dde/d4a/d60/de6/d150 0 2024-04-16T02:36:14.244 INFO:tasks.workunit.client.0.smithi151.stdout:8/957: symlink d11/d22/d6c/d134/l147 0 2024-04-16T02:36:14.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:13 smithi151 ceph-mon[29003]: pgmap v34: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 54 MiB/s rd, 134 MiB/s wr, 418 op/s 2024-04-16T02:36:14.246 INFO:tasks.workunit.client.0.smithi151.stdout:8/958: write d11/d2a/d33/f10b [3877931,119590] 0 2024-04-16T02:36:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:13 smithi153 ceph-mon[36747]: pgmap v34: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 54 MiB/s rd, 134 MiB/s wr, 418 op/s 2024-04-16T02:36:14.275 INFO:tasks.workunit.client.0.smithi151.stdout:4/987: write d0/dde/d4a/d60/de6/f72 [969062,105084] 0 2024-04-16T02:36:14.281 INFO:tasks.workunit.client.0.smithi151.stdout:4/988: dread d0/d11/d19/d87/d3a/d65/d98/df8/f104 [0,4194304] 0 2024-04-16T02:36:14.282 INFO:tasks.workunit.client.0.smithi151.stdout:8/959: dwrite d11/d2a/d35/d52/db3/dcd/fd4 [0,4194304] 0 2024-04-16T02:36:14.288 INFO:tasks.workunit.client.0.smithi151.stdout:8/960: rename d11/d2a/d33/d4e/ff6 to d11/d2a/d33/f148 0 2024-04-16T02:36:14.290 INFO:tasks.workunit.client.0.smithi151.stdout:8/961: chown d11/d2a/d33/d4e/d62/d6b/db9/f13c 0 1 2024-04-16T02:36:14.290 INFO:tasks.workunit.client.0.smithi151.stdout:4/989: dwrite d0/d11/d19/d87/d7b/dfa/f11a [0,4194304] 0 2024-04-16T02:36:14.298 INFO:tasks.workunit.client.0.smithi151.stdout:4/990: creat d0/d9b/d13e/f151 x:0 0 0 2024-04-16T02:36:14.300 INFO:tasks.workunit.client.0.smithi151.stdout:8/962: rename d11/d2a/d33/d4e/d62/dc8/d11c to d11/d22/d2e/dc0/dbc/d149 0 2024-04-16T02:36:14.318 INFO:tasks.workunit.client.0.smithi151.stdout:8/963: truncate d11/d22/d2e/f3d 7620596 0 2024-04-16T02:36:14.320 INFO:tasks.workunit.client.0.smithi151.stdout:8/964: chown d11/d2a/c6a 51615 1 2024-04-16T02:36:14.322 INFO:tasks.workunit.client.0.smithi151.stdout:4/991: dwrite d0/fbb [4194304,4194304] 0 2024-04-16T02:36:14.327 INFO:tasks.workunit.client.0.smithi151.stdout:8/965: rename d11/d2a/d33/d4e/f7c to d11/f14a 0 2024-04-16T02:36:14.332 INFO:tasks.workunit.client.0.smithi151.stdout:8/966: rename d11/d2a/d33/d4e/d62/fff to d11/d2a/d35/d64/f14b 0 2024-04-16T02:36:14.332 INFO:tasks.workunit.client.0.smithi151.stdout:8/967: chown d11/d22/d6c/d84/fc7 4047290 1 2024-04-16T02:36:14.662 INFO:tasks.workunit.client.0.smithi151.stdout:0/795: sync 2024-04-16T02:36:14.672 INFO:tasks.workunit.client.0.smithi151.stdout:0/796: mknod dc/d15/d34/d67/dd2/def/dd6/dd9/de4/c101 0 2024-04-16T02:36:14.673 INFO:tasks.workunit.client.0.smithi151.stdout:0/797: chown dc/d15/d34/d67/dd2/def/lb6 1 1 2024-04-16T02:36:14.676 INFO:tasks.workunit.client.0.smithi151.stdout:0/798: unlink dc/d15/d34/d67/c33 0 2024-04-16T02:36:14.678 INFO:tasks.workunit.client.0.smithi151.stdout:0/799: creat dc/d15/d34/d67/dd2/def/dd6/dd9/f102 x:0 0 0 2024-04-16T02:36:14.680 INFO:tasks.workunit.client.0.smithi151.stdout:0/800: fsync dc/d15/d34/d67/d51/f91 0 2024-04-16T02:36:14.705 INFO:tasks.workunit.client.0.smithi151.stdout:4/992: sync 2024-04-16T02:36:14.705 INFO:tasks.workunit.client.0.smithi151.stdout:8/968: sync 2024-04-16T02:36:14.712 INFO:tasks.workunit.client.0.smithi151.stdout:4/993: dwrite d0/d11/f23 [0,4194304] 0 2024-04-16T02:36:14.718 INFO:tasks.workunit.client.0.smithi151.stdout:0/801: write dc/d15/d34/f5d [1141638,109869] 0 2024-04-16T02:36:14.719 INFO:tasks.workunit.client.0.smithi151.stdout:0/802: chown dc/d20/c81 3070768 1 2024-04-16T02:36:14.782 INFO:tasks.workunit.client.0.smithi151.stdout:4/994: getdents d0/dde/d4a/d60/de6 0 2024-04-16T02:36:14.784 INFO:tasks.workunit.client.0.smithi151.stdout:4/995: mknod d0/d99/c152 0 2024-04-16T02:36:14.785 INFO:tasks.workunit.client.0.smithi151.stdout:4/996: dread - d0/d11/d19/d87/d7b/dfa/f137 zero size 2024-04-16T02:36:14.793 INFO:tasks.workunit.client.0.smithi151.stdout:4/997: getdents d0/dde/d80 0 2024-04-16T02:36:14.798 INFO:tasks.workunit.client.0.smithi151.stdout:4/998: mkdir d0/dde/d80/da0/d153 0 2024-04-16T02:36:14.803 INFO:tasks.workunit.client.0.smithi151.stdout:4/999: mkdir d0/daa/dcc/d124/d148/d154 0 2024-04-16T02:36:14.808 INFO:tasks.workunit.client.0.smithi151.stdout:0/803: write dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f7a [493682,49693] 0 2024-04-16T02:36:14.819 INFO:tasks.workunit.client.0.smithi151.stdout:0/804: dwrite dc/d15/fd4 [0,4194304] 0 2024-04-16T02:36:14.929 INFO:tasks.workunit.client.0.smithi151.stdout:0/805: unlink dc/d4c/l97 0 2024-04-16T02:36:14.929 INFO:tasks.workunit.client.0.smithi151.stdout:0/806: chown dc/d15/d34/d67/lc1 16 1 2024-04-16T02:36:14.934 INFO:tasks.workunit.client.0.smithi151.stdout:0/807: dread dc/d15/d34/d4e/f83 [0,4194304] 0 2024-04-16T02:36:14.936 INFO:tasks.workunit.client.0.smithi151.stdout:0/808: dread dc/d15/f25 [0,4194304] 0 2024-04-16T02:36:14.938 INFO:tasks.workunit.client.0.smithi151.stdout:0/809: chown dc/d15/d34/d67/d69 624 1 2024-04-16T02:36:14.947 INFO:tasks.workunit.client.0.smithi151.stdout:0/810: dwrite dc/d15/d34/d67/d69/d8a/f100 [0,4194304] 0 2024-04-16T02:36:14.952 INFO:tasks.workunit.client.0.smithi151.stdout:0/811: creat dc/d15/d34/d3b/d71/dc3/f103 x:0 0 0 2024-04-16T02:36:14.954 INFO:tasks.workunit.client.0.smithi151.stdout:0/812: fdatasync dc/d4c/f96 0 2024-04-16T02:36:14.959 INFO:tasks.workunit.client.0.smithi151.stdout:0/813: truncate dc/d20/d7b/fc7 977134 0 2024-04-16T02:36:14.987 INFO:tasks.workunit.client.0.smithi151.stdout:0/814: dwrite dc/d15/d5c/faf [4194304,4194304] 0 2024-04-16T02:36:14.998 INFO:tasks.workunit.client.0.smithi151.stdout:0/815: dwrite dc/d15/d34/d67/d69/f8b [0,4194304] 0 2024-04-16T02:36:15.003 INFO:tasks.workunit.client.0.smithi151.stdout:0/816: truncate dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f75 1432098 0 2024-04-16T02:36:15.006 INFO:tasks.workunit.client.0.smithi151.stdout:0/817: truncate dc/d20/d7b/fc7 1212260 0 2024-04-16T02:36:15.017 INFO:tasks.workunit.client.0.smithi151.stdout:0/818: write dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f75 [151741,118400] 0 2024-04-16T02:36:15.019 INFO:tasks.workunit.client.0.smithi151.stdout:0/819: dread - dc/d4c/d72/d76/db0/db5/ff9 zero size 2024-04-16T02:36:15.024 INFO:tasks.workunit.client.0.smithi151.stdout:0/820: link dc/d15/d34/d67/dd2/def/caa dc/d15/d34/d67/dd2/def/c104 0 2024-04-16T02:36:15.026 INFO:tasks.workunit.client.0.smithi151.stdout:0/821: mknod dc/d20/c105 0 2024-04-16T02:36:15.029 INFO:tasks.workunit.client.0.smithi151.stdout:0/822: creat dc/d15/d34/d67/dd2/def/f106 x:0 0 0 2024-04-16T02:36:15.030 INFO:tasks.workunit.client.0.smithi151.stdout:0/823: stat dc/d4c/d72/d76/db0/db5/ff9 0 2024-04-16T02:36:15.032 INFO:tasks.workunit.client.0.smithi151.stdout:0/824: write dc/d15/d34/d67/dd2/def/dd6/dd9/feb [397173,107042] 0 2024-04-16T02:36:15.043 INFO:tasks.workunit.client.0.smithi151.stdout:0/825: dwrite dc/d15/d34/d67/f5e [0,4194304] 0 2024-04-16T02:36:15.050 INFO:tasks.workunit.client.0.smithi151.stdout:0/826: link dc/d20/c81 dc/d15/d34/d67/dd2/def/dd6/dd9/de4/d9f/c107 0 2024-04-16T02:36:15.057 INFO:tasks.workunit.client.0.smithi151.stdout:0/827: write dc/d15/d34/d4e/fc4 [323789,32388] 0 2024-04-16T02:36:15.072 INFO:tasks.workunit.client.0.smithi151.stdout:0/828: truncate dc/d15/d34/d67/d69/d6f/d9e/fb8 45747 0 2024-04-16T02:36:15.076 INFO:tasks.workunit.client.0.smithi151.stdout:0/829: getdents dc/d15/d34/d67 0 2024-04-16T02:36:15.077 INFO:tasks.workunit.client.0.smithi151.stdout:0/830: chown dc/d15/f56 730 1 2024-04-16T02:36:15.079 INFO:tasks.workunit.client.0.smithi151.stdout:0/831: mknod dc/d15/d34/d67/c108 0 2024-04-16T02:36:15.093 INFO:tasks.workunit.client.0.smithi151.stdout:0/832: dwrite dc/d15/d34/f57 [0,4194304] 0 2024-04-16T02:36:15.108 INFO:tasks.workunit.client.0.smithi151.stdout:8/969: mkdir d11/d2a/d35/d64/d14c 0 2024-04-16T02:36:15.228 INFO:tasks.workunit.client.0.smithi151.stdout:8/970: dwrite d11/d2a/df4/d78/f7b [0,4194304] 0 2024-04-16T02:36:15.230 INFO:tasks.workunit.client.0.smithi151.stdout:8/971: truncate d11/d22/d7e/f132 484708 0 2024-04-16T02:36:15.231 INFO:tasks.workunit.client.0.smithi151.stdout:8/972: dread - d11/d2a/d33/d4e/d62/d6b/db9/f13c zero size 2024-04-16T02:36:16.152 INFO:tasks.workunit.client.0.smithi151.stdout:8/973: mknod d11/d2a/d33/d4e/d62/d94/dfd/c14d 0 2024-04-16T02:36:16.157 INFO:tasks.workunit.client.0.smithi151.stdout:8/974: write d11/d2a/d35/d52/db3/f127 [763626,52447] 0 2024-04-16T02:36:16.160 INFO:tasks.workunit.client.0.smithi151.stdout:8/975: write d11/d2a/d33/d4e/d62/d94/d11d/f12b [4527407,124922] 0 2024-04-16T02:36:16.162 INFO:tasks.workunit.client.0.smithi151.stdout:8/976: truncate d11/d2a/d33/d4e/f9e 3333017 0 2024-04-16T02:36:16.163 INFO:tasks.workunit.client.0.smithi151.stdout:8/977: read d11/d2a/db1/f101 [110520,26211] 0 2024-04-16T02:36:16.170 INFO:tasks.workunit.client.0.smithi151.stdout:8/978: dwrite d11/d2a/d35/d52/db3/fb6 [0,4194304] 0 2024-04-16T02:36:16.171 INFO:tasks.workunit.client.0.smithi151.stdout:8/979: stat d11/d22/d2e/dc0/dbc/d149/f129 0 2024-04-16T02:36:16.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:15 smithi151 ceph-mon[29003]: pgmap v35: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 35 MiB/s rd, 77 MiB/s wr, 263 op/s 2024-04-16T02:36:16.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:15 smithi153 ceph-mon[36747]: pgmap v35: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 35 MiB/s rd, 77 MiB/s wr, 263 op/s 2024-04-16T02:36:17.212 INFO:tasks.workunit.client.0.smithi151.stdout:8/980: sync 2024-04-16T02:36:17.212 INFO:tasks.workunit.client.0.smithi151.stdout:0/833: sync 2024-04-16T02:36:17.215 INFO:tasks.workunit.client.0.smithi151.stdout:0/834: creat dc/dde/df8/f109 x:0 0 0 2024-04-16T02:36:17.218 INFO:tasks.workunit.client.0.smithi151.stdout:8/981: write d11/f14a [2405243,16317] 0 2024-04-16T02:36:17.220 INFO:tasks.workunit.client.0.smithi151.stdout:0/835: creat dc/d15/d34/d67/dd2/def/dd6/f10a x:0 0 0 2024-04-16T02:36:17.222 INFO:tasks.workunit.client.0.smithi151.stdout:8/982: creat d11/d2a/d33/d4e/d62/db7/d144/f14e x:0 0 0 2024-04-16T02:36:17.223 INFO:tasks.workunit.client.0.smithi151.stdout:0/836: rmdir dc/d4c/d72/d76/db0/db5 39 2024-04-16T02:36:17.224 INFO:tasks.workunit.client.0.smithi151.stdout:8/983: dread d11/d22/d2e/dc0/dbc/d149/f129 [0,4194304] 0 2024-04-16T02:36:17.225 INFO:tasks.workunit.client.0.smithi151.stdout:0/837: mknod dc/d20/c10b 0 2024-04-16T02:36:17.226 INFO:tasks.workunit.client.0.smithi151.stdout:8/984: rmdir d11/d2a/d35/d52 39 2024-04-16T02:36:17.227 INFO:tasks.workunit.client.0.smithi151.stdout:0/838: rmdir dc/d4c/d72 39 2024-04-16T02:36:17.241 INFO:tasks.workunit.client.0.smithi151.stdout:8/985: dread d11/f28 [4194304,4194304] 0 2024-04-16T02:36:17.245 INFO:tasks.workunit.client.0.smithi151.stdout:8/986: dread d11/d2a/d35/d64/fef [0,4194304] 0 2024-04-16T02:36:17.258 INFO:tasks.workunit.client.0.smithi151.stdout:8/987: unlink d11/d22/c24 0 2024-04-16T02:36:17.265 INFO:tasks.workunit.client.0.smithi151.stdout:0/839: truncate dc/d20/fbc 2959961 0 2024-04-16T02:36:17.265 INFO:tasks.workunit.client.0.smithi151.stdout:8/988: write d11/d2a/db1/fd0 [797949,121997] 0 2024-04-16T02:36:17.266 INFO:tasks.workunit.client.0.smithi151.stdout:0/840: dread - dc/d15/d34/d3b/fe2 zero size 2024-04-16T02:36:17.267 INFO:tasks.workunit.client.0.smithi151.stdout:8/989: chown d11/d2a/d33/d4e/d62/db7/cd9 89965 1 2024-04-16T02:36:17.268 INFO:tasks.workunit.client.0.smithi151.stdout:8/990: readlink d11/d2a/d33/d4e/d62/db7/lea 0 2024-04-16T02:36:17.270 INFO:tasks.workunit.client.0.smithi151.stdout:8/991: write d11/d2a/d35/d64/f10f [1281792,52261] 0 2024-04-16T02:36:17.272 INFO:tasks.workunit.client.0.smithi151.stdout:0/841: dread dc/f39 [4194304,4194304] 0 2024-04-16T02:36:17.277 INFO:tasks.workunit.client.0.smithi151.stdout:0/842: mknod dc/d15/d34/d67/dd2/def/dd6/dd9/de4/c10c 0 2024-04-16T02:36:17.279 INFO:tasks.workunit.client.0.smithi151.stdout:8/992: mknod d11/d2a/d35/d52/db3/dcd/c14f 0 2024-04-16T02:36:17.284 INFO:tasks.workunit.client.0.smithi151.stdout:8/993: fdatasync d11/d22/d2e/f3d 0 2024-04-16T02:36:17.285 INFO:tasks.workunit.client.0.smithi151.stdout:8/994: dread - d11/d2a/d33/d4e/f100 zero size 2024-04-16T02:36:17.292 INFO:tasks.workunit.client.0.smithi151.stdout:8/995: creat d11/d7d/f150 x:0 0 0 2024-04-16T02:36:17.292 INFO:tasks.workunit.client.0.smithi151.stdout:8/996: chown d11/d22/d2e/l61 15116 1 2024-04-16T02:36:17.301 INFO:tasks.workunit.client.0.smithi151.stdout:8/997: rename d11/d2a/d33/d4e/d62/d94 to d11/d22/d2e/dc0/dbc/d149/d124/d151 0 2024-04-16T02:36:17.307 INFO:tasks.workunit.client.0.smithi151.stdout:0/843: write dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f48 [1438882,121930] 0 2024-04-16T02:36:17.312 INFO:tasks.workunit.client.0.smithi151.stdout:0/844: mkdir dc/d20/d10d 0 2024-04-16T02:36:17.316 INFO:tasks.workunit.client.0.smithi151.stdout:0/845: mkdir dc/d4c/d9d/ded/d10e 0 2024-04-16T02:36:17.317 INFO:tasks.workunit.client.0.smithi151.stdout:0/846: chown dc/d4c/l86 1 1 2024-04-16T02:36:17.325 INFO:tasks.workunit.client.0.smithi151.stdout:8/998: dwrite d11/f28 [0,4194304] 0 2024-04-16T02:36:17.331 INFO:tasks.workunit.client.0.smithi151.stdout:8/999: mknod d11/d22/d2e/dc0/dbc/d149/c152 0 2024-04-16T02:36:17.353 INFO:tasks.workunit.client.0.smithi151.stdout:0/847: dread dc/f39 [0,4194304] 0 2024-04-16T02:36:17.354 INFO:tasks.workunit.client.0.smithi151.stdout:0/848: chown dc/d20/c4f 37 1 2024-04-16T02:36:17.357 INFO:tasks.workunit.client.0.smithi151.stdout:0/849: mknod dc/d4c/d9d/c10f 0 2024-04-16T02:36:17.360 INFO:tasks.workunit.client.0.smithi151.stdout:0/850: symlink dc/d15/d34/d67/d51/l110 0 2024-04-16T02:36:17.364 INFO:tasks.workunit.client.0.smithi151.stdout:0/851: fdatasync dc/d15/d34/fab 0 2024-04-16T02:36:17.368 INFO:tasks.workunit.client.0.smithi151.stdout:0/852: dread dc/f47 [0,4194304] 0 2024-04-16T02:36:17.368 INFO:tasks.workunit.client.0.smithi151.stdout:0/853: chown dc/d20 2 1 2024-04-16T02:36:17.373 INFO:tasks.workunit.client.0.smithi151.stdout:0/854: read dc/d15/d34/d67/d69/d6f/d9e/fb8 [35085,87440] 0 2024-04-16T02:36:17.375 INFO:tasks.workunit.client.0.smithi151.stdout:0/855: getdents dc/d15/d34/d67/dd2/def/de0 0 2024-04-16T02:36:17.378 INFO:tasks.workunit.client.0.smithi151.stdout:0/856: mkdir dc/d4c/d72/d76/db0/d111 0 2024-04-16T02:36:17.381 INFO:tasks.workunit.client.0.smithi151.stdout:0/857: mknod dc/d4c/d72/d76/db0/db5/c112 0 2024-04-16T02:36:17.381 INFO:tasks.workunit.client.0.smithi151.stdout:0/858: read - dc/d20/fd1 zero size 2024-04-16T02:36:17.387 INFO:tasks.workunit.client.0.smithi151.stdout:0/859: write dc/d15/f58 [2527485,108341] 0 2024-04-16T02:36:17.390 INFO:tasks.workunit.client.0.smithi151.stdout:0/860: creat dc/d15/d34/d3b/d71/dc3/f113 x:0 0 0 2024-04-16T02:36:17.401 INFO:tasks.workunit.client.0.smithi151.stdout:0/861: dread dc/d15/d34/d67/d51/fbe [0,4194304] 0 2024-04-16T02:36:17.415 INFO:tasks.workunit.client.0.smithi151.stdout:0/862: dread dc/d15/f58 [0,4194304] 0 2024-04-16T02:36:18.059 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:17 smithi151 ceph-mon[29003]: pgmap v36: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 38 MiB/s rd, 86 MiB/s wr, 293 op/s 2024-04-16T02:36:18.081 INFO:tasks.workunit.client.0.smithi151.stdout:0/863: sync 2024-04-16T02:36:18.107 INFO:tasks.workunit.client.0.smithi151.stdout:0/864: dwrite dc/d15/d34/d67/f78 [0,4194304] 0 2024-04-16T02:36:18.108 INFO:tasks.workunit.client.0.smithi151.stdout:0/865: chown dc/d15/f17 20 1 2024-04-16T02:36:18.109 INFO:tasks.workunit.client.0.smithi151.stdout:0/866: write dc/d15/d34/d67/dd2/def/dd6/dd9/de4/dc2/fea [22490,45996] 0 2024-04-16T02:36:18.112 INFO:tasks.workunit.client.0.smithi151.stdout:0/867: read dc/d15/d34/d3b/d71/f74 [1861906,60743] 0 2024-04-16T02:36:18.116 INFO:tasks.workunit.client.0.smithi151.stdout:0/868: dread dc/d15/d34/d4e/fc4 [0,4194304] 0 2024-04-16T02:36:18.119 INFO:tasks.workunit.client.0.smithi151.stdout:0/869: creat dc/d15/d34/d67/f114 x:0 0 0 2024-04-16T02:36:18.121 INFO:tasks.workunit.client.0.smithi151.stdout:0/870: symlink dc/d4c/d9d/l115 0 2024-04-16T02:36:18.126 INFO:tasks.workunit.client.0.smithi151.stdout:0/871: dread dc/d15/fd4 [0,4194304] 0 2024-04-16T02:36:18.130 INFO:tasks.workunit.client.0.smithi151.stdout:0/872: write dc/d15/d34/d67/f3e [192094,63412] 0 2024-04-16T02:36:18.134 INFO:tasks.workunit.client.0.smithi151.stdout:0/873: rename dc/d4c/d72/d76 to dc/d116 0 2024-04-16T02:36:18.136 INFO:tasks.workunit.client.0.smithi151.stdout:0/874: truncate dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f7a 1192046 0 2024-04-16T02:36:18.143 INFO:tasks.workunit.client.0.smithi151.stdout:0/875: dwrite dc/d15/d34/d67/d69/f8b [4194304,4194304] 0 2024-04-16T02:36:18.145 INFO:tasks.workunit.client.0.smithi151.stdout:0/876: stat dc/d15/d34/d67/f30 0 2024-04-16T02:36:18.162 INFO:tasks.workunit.client.0.smithi151.stdout:0/877: dwrite dc/d20/f94 [0,4194304] 0 2024-04-16T02:36:18.166 INFO:tasks.workunit.client.0.smithi151.stdout:0/878: unlink dc/d15/d34/d4e/cac 0 2024-04-16T02:36:18.169 INFO:tasks.workunit.client.0.smithi151.stdout:0/879: rmdir dc/dde/df8 39 2024-04-16T02:36:18.176 INFO:tasks.workunit.client.0.smithi151.stdout:0/880: dwrite dc/f47 [0,4194304] 0 2024-04-16T02:36:18.189 INFO:tasks.workunit.client.0.smithi151.stdout:0/881: write dc/d15/d34/d3b/d71/f74 [3238176,35856] 0 2024-04-16T02:36:18.190 INFO:tasks.workunit.client.0.smithi151.stdout:0/882: rmdir dc/dde/df8 39 2024-04-16T02:36:18.192 INFO:tasks.workunit.client.0.smithi151.stdout:0/883: dread - dc/fe8 zero size 2024-04-16T02:36:18.197 INFO:tasks.workunit.client.0.smithi151.stdout:0/884: read dc/d15/d34/d67/dd2/def/dd6/dd9/ff7 [2047322,126635] 0 2024-04-16T02:36:18.202 INFO:tasks.workunit.client.0.smithi151.stdout:0/885: readlink dc/d15/d34/d3b/d71/dc3/ldc 0 2024-04-16T02:36:18.223 INFO:tasks.workunit.client.0.smithi151.stdout:0/886: dwrite dc/d116/db0/db5/ff9 [0,4194304] 0 2024-04-16T02:36:18.227 INFO:tasks.workunit.client.0.smithi151.stdout:0/887: symlink dc/d116/l117 0 2024-04-16T02:36:18.256 INFO:tasks.workunit.client.0.smithi151.stdout:0/888: dread dc/d15/d5c/f6a [0,4194304] 0 2024-04-16T02:36:18.259 INFO:tasks.workunit.client.0.smithi151.stdout:0/889: truncate dc/d15/d34/f54 4975525 0 2024-04-16T02:36:18.261 INFO:tasks.workunit.client.0.smithi151.stdout:0/890: creat dc/d15/d34/d67/d51/f118 x:0 0 0 2024-04-16T02:36:18.263 INFO:tasks.workunit.client.0.smithi151.stdout:0/891: rmdir dc/d15/d34/d67/d69 39 2024-04-16T02:36:18.265 INFO:tasks.workunit.client.0.smithi151.stdout:0/892: creat dc/d4c/d72/f119 x:0 0 0 2024-04-16T02:36:18.270 INFO:tasks.workunit.client.0.smithi151.stdout:0/893: link dc/d20/d7b/fc7 dc/d15/d34/d67/dd2/dff/f11a 0 2024-04-16T02:36:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:17 smithi153 ceph-mon[36747]: pgmap v36: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 38 MiB/s rd, 86 MiB/s wr, 293 op/s 2024-04-16T02:36:18.276 INFO:tasks.workunit.client.0.smithi151.stdout:0/894: link dc/d116/db0/fb7 dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f11b 0 2024-04-16T02:36:18.279 INFO:tasks.workunit.client.0.smithi151.stdout:0/895: creat dc/d4c/d9d/ded/f11c x:0 0 0 2024-04-16T02:36:18.280 INFO:tasks.workunit.client.0.smithi151.stdout:0/896: write dc/f47 [2869451,49190] 0 2024-04-16T02:36:18.282 INFO:tasks.workunit.client.0.smithi151.stdout:0/897: truncate dc/d116/db0/db5/ff9 4511612 0 2024-04-16T02:36:18.291 INFO:tasks.workunit.client.0.smithi151.stdout:0/898: dwrite dc/d15/d34/d67/d69/d8a/f100 [0,4194304] 0 2024-04-16T02:36:18.806 INFO:tasks.workunit.client.0.smithi151.stdout:0/899: sync 2024-04-16T02:36:18.810 INFO:tasks.workunit.client.0.smithi151.stdout:0/900: dread dc/d15/d34/d67/f78 [0,4194304] 0 2024-04-16T02:36:18.815 INFO:tasks.workunit.client.0.smithi151.stdout:0/901: mknod dc/dde/df8/c11d 0 2024-04-16T02:36:18.815 INFO:tasks.workunit.client.0.smithi151.stdout:0/902: stat dc/d15/d34/d3b/d71 0 2024-04-16T02:36:18.820 INFO:tasks.workunit.client.0.smithi151.stdout:0/903: creat dc/d15/d34/d67/f11e x:0 0 0 2024-04-16T02:36:18.824 INFO:tasks.workunit.client.0.smithi151.stdout:0/904: creat dc/d15/d34/d67/dd2/def/dd6/dd9/de4/d9f/f11f x:0 0 0 2024-04-16T02:36:18.826 INFO:tasks.workunit.client.0.smithi151.stdout:0/905: chown dc/d15/d34/d67/dd2/def/dd6/dd9/de4/d9f/f11f 386 1 2024-04-16T02:36:18.841 INFO:tasks.workunit.client.0.smithi151.stdout:0/906: dread dc/d15/d34/d67/f3e [0,4194304] 0 2024-04-16T02:36:18.844 INFO:tasks.workunit.client.0.smithi151.stdout:0/907: creat dc/dde/df8/f120 x:0 0 0 2024-04-16T02:36:19.245 INFO:tasks.workunit.client.0.smithi151.stdout:0/908: sync 2024-04-16T02:36:19.262 INFO:tasks.workunit.client.0.smithi151.stdout:0/909: dwrite dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f84 [0,4194304] 0 2024-04-16T02:36:19.273 INFO:tasks.workunit.client.0.smithi151.stdout:0/910: write dc/d15/f2b [1700288,122048] 0 2024-04-16T02:36:19.277 INFO:tasks.workunit.client.0.smithi151.stdout:0/911: truncate dc/d15/d34/d4e/f55 4964753 0 2024-04-16T02:36:19.278 INFO:tasks.workunit.client.0.smithi151.stdout:0/912: symlink dc/d4c/d72/l121 0 2024-04-16T02:36:19.279 INFO:tasks.workunit.client.0.smithi151.stdout:0/913: dread - dc/d20/fbb zero size 2024-04-16T02:36:19.282 INFO:tasks.workunit.client.0.smithi151.stdout:0/914: mknod dc/d15/d34/d67/dd2/def/dd6/dd9/c122 0 2024-04-16T02:36:19.292 INFO:tasks.workunit.client.0.smithi151.stdout:0/915: write dc/d15/d34/d67/d69/d8a/ffa [756385,121780] 0 2024-04-16T02:36:19.294 INFO:tasks.workunit.client.0.smithi151.stdout:0/916: creat dc/dde/df8/f123 x:0 0 0 2024-04-16T02:36:19.298 INFO:tasks.workunit.client.0.smithi151.stdout:0/917: symlink dc/d15/d34/d67/d69/d6f/l124 0 2024-04-16T02:36:19.301 INFO:tasks.workunit.client.0.smithi151.stdout:0/918: unlink dc/d15/d34/d67/f78 0 2024-04-16T02:36:19.306 INFO:tasks.workunit.client.0.smithi151.stdout:0/919: rename dc/d15/d34/d67/dd2/def/dd6/dd9/de4/d9f/cc9 to dc/d4c/d72/c125 0 2024-04-16T02:36:19.313 INFO:tasks.workunit.client.0.smithi151.stdout:0/920: dwrite dc/d20/f94 [0,4194304] 0 2024-04-16T02:36:19.315 INFO:tasks.workunit.client.0.smithi151.stdout:0/921: stat dc/d15/d34/d67/dd2/dff 0 2024-04-16T02:36:19.322 INFO:tasks.workunit.client.0.smithi151.stdout:0/922: getdents dc/d15/d34/d67/dd2/def 0 2024-04-16T02:36:19.323 INFO:tasks.workunit.client.0.smithi151.stdout:0/923: chown dc/d15/d34/d67/f30 46645 1 2024-04-16T02:36:19.327 INFO:tasks.workunit.client.0.smithi151.stdout:0/924: creat dc/d15/d34/d3b/d71/dc3/f126 x:0 0 0 2024-04-16T02:36:19.336 INFO:tasks.workunit.client.0.smithi151.stdout:0/925: dwrite dc/d15/d34/d4e/fc4 [0,4194304] 0 2024-04-16T02:36:19.342 INFO:tasks.workunit.client.0.smithi151.stdout:0/926: rmdir dc/d15/d34/d67/dd2/def/dd6 39 2024-04-16T02:36:19.351 INFO:tasks.workunit.client.0.smithi151.stdout:0/927: dwrite dc/d15/d34/d67/dd2/def/dd6/f10a [0,4194304] 0 2024-04-16T02:36:19.356 INFO:tasks.workunit.client.0.smithi151.stdout:0/928: symlink dc/d15/l127 0 2024-04-16T02:36:19.359 INFO:tasks.workunit.client.0.smithi151.stdout:0/929: creat dc/d15/d34/d4e/f128 x:0 0 0 2024-04-16T02:36:19.365 INFO:tasks.workunit.client.0.smithi151.stdout:0/930: link c7 dc/d15/d34/d67/dd2/def/dd6/dd9/de4/c129 0 2024-04-16T02:36:19.380 INFO:tasks.workunit.client.0.smithi151.stdout:0/931: dwrite dc/d15/f56 [0,4194304] 0 2024-04-16T02:36:19.384 INFO:tasks.workunit.client.0.smithi151.stdout:0/932: chown dc/d15/d34/d67/dd2/def/dd6/dd9/de4/d9f/c107 10044489 1 2024-04-16T02:36:19.388 INFO:tasks.workunit.client.0.smithi151.stdout:0/933: mkdir dc/d15/d34/d67/dd2/def/d12a 0 2024-04-16T02:36:19.390 INFO:tasks.workunit.client.0.smithi151.stdout:0/934: write dc/d15/d34/d67/dd2/def/dd6/dd9/de4/d9f/f11f [828220,116490] 0 2024-04-16T02:36:19.396 INFO:tasks.workunit.client.0.smithi151.stdout:0/935: read dc/d15/d34/d67/d69/d6f/fa9 [7043860,94238] 0 2024-04-16T02:36:19.397 INFO:tasks.workunit.client.0.smithi151.stdout:0/936: readlink dc/d15/d34/d67/dd2/def/lb6 0 2024-04-16T02:36:19.403 INFO:tasks.workunit.client.0.smithi151.stdout:0/937: getdents dc/d15/d34/d67/dd2/def 0 2024-04-16T02:36:19.404 INFO:tasks.workunit.client.0.smithi151.stdout:0/938: rmdir dc/d15/d34/d67/dd2/def/dd6/dd9/de4/d9f 39 2024-04-16T02:36:19.413 INFO:tasks.workunit.client.0.smithi151.stdout:0/939: dwrite dc/d15/d34/ff2 [0,4194304] 0 2024-04-16T02:36:19.418 INFO:tasks.workunit.client.0.smithi151.stdout:0/940: creat dc/d15/d5c/f12b x:0 0 0 2024-04-16T02:36:19.418 INFO:tasks.workunit.client.0.smithi151.stdout:0/941: chown dc/d15/d34/d4e/f55 10015 1 2024-04-16T02:36:19.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:19 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:19.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:19 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:19.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:19 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:36:19.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:19 smithi153 ceph-mon[36747]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:36:19.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:19 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:36:19.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:19 smithi153 ceph-mon[36747]: Upgrade: Need to upgrade myself (mgr.smithi153.esbtzf) 2024-04-16T02:36:19.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:19 smithi153 ceph-mon[36747]: Upgrade: Need to upgrade myself (mgr.smithi153.esbtzf) 2024-04-16T02:36:19.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:19 smithi153 ceph-mon[36747]: pgmap v37: 65 pgs: 65 active+clean; 2.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 42 MiB/s rd, 94 MiB/s wr, 358 op/s 2024-04-16T02:36:19.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:19 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:19.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:19 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:19.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:19 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:36:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:19 smithi151 ceph-mon[29003]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:36:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:19 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:36:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:19 smithi151 ceph-mon[29003]: Upgrade: Need to upgrade myself (mgr.smithi153.esbtzf) 2024-04-16T02:36:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:19 smithi151 ceph-mon[29003]: Upgrade: Need to upgrade myself (mgr.smithi153.esbtzf) 2024-04-16T02:36:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:19 smithi151 ceph-mon[29003]: pgmap v37: 65 pgs: 65 active+clean; 2.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 42 MiB/s rd, 94 MiB/s wr, 358 op/s 2024-04-16T02:36:20.084 INFO:tasks.workunit.client.0.smithi151.stdout:0/942: sync 2024-04-16T02:36:20.085 INFO:tasks.workunit.client.0.smithi151.stdout:0/943: stat dc/d15/f50 0 2024-04-16T02:36:20.086 INFO:tasks.workunit.client.0.smithi151.stdout:0/944: mkdir dc/d12c 0 2024-04-16T02:36:20.088 INFO:tasks.workunit.client.0.smithi151.stdout:0/945: write dc/d15/f58 [2905373,45442] 0 2024-04-16T02:36:20.088 INFO:tasks.workunit.client.0.smithi151.stdout:0/946: chown dc/d15/d34/f57 415 1 2024-04-16T02:36:20.091 INFO:tasks.workunit.client.0.smithi151.stdout:0/947: unlink dc/dde/df8/f123 0 2024-04-16T02:36:20.096 INFO:tasks.workunit.client.0.smithi151.stdout:0/948: write dc/d15/d34/d67/d51/fb9 [565074,15643] 0 2024-04-16T02:36:20.098 INFO:tasks.workunit.client.0.smithi151.stdout:0/949: symlink dc/d15/d34/d67/l12d 0 2024-04-16T02:36:20.100 INFO:tasks.workunit.client.0.smithi151.stdout:0/950: read dc/d15/f50 [4158548,92223] 0 2024-04-16T02:36:20.100 INFO:tasks.workunit.client.0.smithi151.stdout:0/951: readlink dc/ldd 0 2024-04-16T02:36:20.103 INFO:tasks.workunit.client.0.smithi151.stdout:0/952: creat dc/d15/d34/d67/d69/d6f/f12e x:0 0 0 2024-04-16T02:36:20.108 INFO:tasks.workunit.client.0.smithi151.stdout:0/953: write dc/d15/d34/d67/d51/fbe [893530,114824] 0 2024-04-16T02:36:20.110 INFO:tasks.workunit.client.0.smithi151.stdout:0/954: mknod dc/d15/d34/d67/dd2/dff/c12f 0 2024-04-16T02:36:20.112 INFO:tasks.workunit.client.0.smithi151.stdout:0/955: truncate f3 3862850 0 2024-04-16T02:36:20.113 INFO:tasks.workunit.client.0.smithi151.stdout:0/956: stat dc/d15/d34/d4e/f63 0 2024-04-16T02:36:20.114 INFO:tasks.workunit.client.0.smithi151.stdout:0/957: truncate dc/ff 494036 0 2024-04-16T02:36:20.128 INFO:tasks.workunit.client.0.smithi151.stdout:0/958: dwrite dc/ff [0,4194304] 0 2024-04-16T02:36:20.130 INFO:tasks.workunit.client.0.smithi151.stdout:0/959: chown dc/d15/d34/d67/l12d 261 1 2024-04-16T02:36:20.134 INFO:tasks.workunit.client.0.smithi151.stdout:0/960: getdents dc/d15/d5c 0 2024-04-16T02:36:20.145 INFO:tasks.workunit.client.0.smithi151.stdout:0/961: write dc/d15/d34/d67/dd2/def/fd8 [1848540,54621] 0 2024-04-16T02:36:20.151 INFO:tasks.workunit.client.0.smithi151.stdout:0/962: write dc/d15/d34/d67/d69/d6f/f99 [2728967,87146] 0 2024-04-16T02:36:20.152 INFO:tasks.workunit.client.0.smithi151.stdout:0/963: readlink dc/l5b 0 2024-04-16T02:36:20.152 INFO:tasks.workunit.client.0.smithi151.stdout:0/964: chown dc/d15/d34/d4e/f55 93128 1 2024-04-16T02:36:20.155 INFO:tasks.workunit.client.0.smithi151.stdout:0/965: creat dc/d15/f130 x:0 0 0 2024-04-16T02:36:20.156 INFO:tasks.workunit.client.0.smithi151.stdout:0/966: chown dc/d15/d34/d67/dd2/def/dd6/lae 6098157 1 2024-04-16T02:36:20.164 INFO:tasks.workunit.client.0.smithi151.stdout:0/967: dwrite dc/d15/f87 [0,4194304] 0 2024-04-16T02:36:20.168 INFO:tasks.workunit.client.0.smithi151.stdout:0/968: mkdir dc/d15/d5c/d131 0 2024-04-16T02:36:20.171 INFO:tasks.workunit.client.0.smithi151.stdout:0/969: rmdir dc/d15/d34/d3b/d71 39 2024-04-16T02:36:20.173 INFO:tasks.workunit.client.0.smithi151.stdout:0/970: chown dc/d15/d34/d3b/d71/dc3/f103 5639 1 2024-04-16T02:36:20.185 INFO:tasks.workunit.client.0.smithi151.stdout:0/971: write dc/d15/f17 [2863446,116577] 0 2024-04-16T02:36:20.200 INFO:tasks.workunit.client.0.smithi151.stdout:0/972: dread dc/d15/f17 [0,4194304] 0 2024-04-16T02:36:20.203 INFO:tasks.workunit.client.0.smithi151.stdout:0/973: mkdir dc/d4c/d9d/ded/d10e/d132 0 2024-04-16T02:36:20.205 INFO:tasks.workunit.client.0.smithi151.stdout:0/974: symlink dc/d20/d7b/l133 0 2024-04-16T02:36:20.206 INFO:tasks.workunit.client.0.smithi151.stdout:0/975: readlink dc/d4c/l86 0 2024-04-16T02:36:20.206 INFO:tasks.workunit.client.0.smithi151.stdout:0/976: chown dc/d15/f7f 6 1 2024-04-16T02:36:20.208 INFO:tasks.workunit.client.0.smithi151.stdout:0/977: symlink dc/d116/db0/l134 0 2024-04-16T02:36:20.211 INFO:tasks.workunit.client.0.smithi151.stdout:0/978: mknod dc/d15/d34/d67/d69/d6f/c135 0 2024-04-16T02:36:20.213 INFO:tasks.workunit.client.0.smithi151.stdout:0/979: mknod dc/d4c/d9d/c136 0 2024-04-16T02:36:20.221 INFO:tasks.workunit.client.0.smithi151.stdout:0/980: dwrite dc/d15/d34/d3b/d71/dc3/f126 [0,4194304] 0 2024-04-16T02:36:20.222 INFO:tasks.workunit.client.0.smithi151.stdout:0/981: readlink dc/d116/db0/l134 0 2024-04-16T02:36:20.232 INFO:tasks.workunit.client.0.smithi151.stdout:0/982: dwrite dc/d15/fd4 [4194304,4194304] 0 2024-04-16T02:36:20.237 INFO:tasks.workunit.client.0.smithi151.stdout:0/983: creat dc/d4c/d9d/ded/d10e/d132/f137 x:0 0 0 2024-04-16T02:36:20.241 INFO:tasks.workunit.client.0.smithi151.stdout:0/984: truncate dc/d15/d34/d67/f5f 2325693 0 2024-04-16T02:36:20.243 INFO:tasks.workunit.client.0.smithi151.stdout:0/985: write dc/d15/d34/d67/dd2/def/fd8 [419463,21400] 0 2024-04-16T02:36:20.246 INFO:tasks.workunit.client.0.smithi151.stdout:0/986: fsync dc/d15/d34/d4e/f63 0 2024-04-16T02:36:20.248 INFO:tasks.workunit.client.0.smithi151.stdout:0/987: chown dc/d4c/d9d/ded/f11c 1731 1 2024-04-16T02:36:20.258 INFO:tasks.workunit.client.0.smithi151.stdout:0/988: link dc/d15/d34/d67/dd2/def/caa dc/d20/d10d/c138 0 2024-04-16T02:36:20.259 INFO:tasks.workunit.client.0.smithi151.stdout:0/989: chown dc/d4c/d72/cd0 26585884 1 2024-04-16T02:36:20.259 INFO:tasks.workunit.client.0.smithi151.stdout:0/990: readlink dc/ldd 0 2024-04-16T02:36:20.260 INFO:tasks.workunit.client.0.smithi151.stdout:0/991: dread - dc/d15/d34/d3b/fe2 zero size 2024-04-16T02:36:20.266 INFO:tasks.workunit.client.0.smithi151.stdout:0/992: creat dc/d15/d34/d67/d69/d6f/f139 x:0 0 0 2024-04-16T02:36:20.273 INFO:tasks.workunit.client.0.smithi151.stdout:0/993: write dc/d15/f50 [3495498,24167] 0 2024-04-16T02:36:20.275 INFO:tasks.workunit.client.0.smithi151.stdout:0/994: write dc/d15/d34/f5d [1405161,64032] 0 2024-04-16T02:36:20.280 INFO:tasks.workunit.client.0.smithi151.stdout:0/995: truncate dc/d15/d34/d67/fe9 219900 0 2024-04-16T02:36:20.292 INFO:tasks.workunit.client.0.smithi151.stdout:0/996: truncate dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f84 3692398 0 2024-04-16T02:36:20.296 INFO:tasks.workunit.client.0.smithi151.stdout:0/997: truncate dc/d15/d34/d67/dd2/def/dd6/dd9/de4/f75 2032051 0 2024-04-16T02:36:20.299 INFO:tasks.workunit.client.0.smithi151.stdout:0/998: unlink dc/d15/d34/d3b/d71/dc3/ldc 0 2024-04-16T02:36:20.303 INFO:tasks.workunit.client.0.smithi151.stdout:0/999: symlink dc/d15/d34/d67/d69/d8a/df3/l13a 0 2024-04-16T02:36:20.308 INFO:tasks.workunit.client.0.smithi151.stderr:+ rm -rf -- ./tmp.04Kqyy33Bl 2024-04-16T02:36:22.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:21 smithi151 ceph-mon[29003]: pgmap v38: 65 pgs: 65 active+clean; 2.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 30 MiB/s rd, 67 MiB/s wr, 249 op/s 2024-04-16T02:36:22.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:21 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:22.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:21 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi151.uqqame", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:36:22.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:21 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi151.uqqame", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:36:22.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:21 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:36:22.053 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:21 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:36:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:21 smithi153 ceph-mon[36747]: pgmap v38: 65 pgs: 65 active+clean; 2.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 30 MiB/s rd, 67 MiB/s wr, 249 op/s 2024-04-16T02:36:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:21 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:21 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi151.uqqame", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:36:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:21 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi151.uqqame", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:36:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:21 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:36:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:21 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:36:23.105 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:22 smithi151 ceph-mon[29003]: Upgrade: Updating mgr.smithi151.uqqame 2024-04-16T02:36:23.105 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:22 smithi151 ceph-mon[29003]: Deploying daemon mgr.smithi151.uqqame on smithi151 2024-04-16T02:36:23.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:22 smithi153 ceph-mon[36747]: Upgrade: Updating mgr.smithi151.uqqame 2024-04-16T02:36:23.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:22 smithi153 ceph-mon[36747]: Deploying daemon mgr.smithi151.uqqame on smithi151 2024-04-16T02:36:23.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.807+0000 7f1558305700 1 -- 172.21.15.151:0/1688566203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550107d90 msgr2=0x7f15501081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:23.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.807+0000 7f1558305700 1 --2- 172.21.15.151:0/1688566203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550107d90 0x7f15501081f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f154000a980 tx=0x7f154000ac90 comp rx=0 tx=0).stop 2024-04-16T02:36:23.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.807+0000 7f1558305700 1 -- 172.21.15.151:0/1688566203 shutdown_connections 2024-04-16T02:36:23.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.807+0000 7f1558305700 1 --2- 172.21.15.151:0/1688566203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550107d90 0x7f15501081f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:23.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.807+0000 7f1558305700 1 --2- 172.21.15.151:0/1688566203 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1550109040 0x7f1550109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:23.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.807+0000 7f1558305700 1 -- 172.21.15.151:0/1688566203 >> 172.21.15.151:0/1688566203 conn(0x7f155006d9f0 msgr2=0x7f155006fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.807+0000 7f1558305700 1 -- 172.21.15.151:0/1688566203 shutdown_connections 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.807+0000 7f1558305700 1 -- 172.21.15.151:0/1688566203 wait complete. 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.808+0000 7f1558305700 1 Processor -- start 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.808+0000 7f1558305700 1 -- start start 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.808+0000 7f1558305700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1550107d90 0x7f155010bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.808+0000 7f1558305700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550109040 0x7f155010a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.808+0000 7f1558305700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155010a6c0 con 0x7f1550107d90 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.808+0000 7f1558305700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155010a800 con 0x7f1550109040 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.809+0000 7f15558a0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550109040 0x7f155010a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.809+0000 7f15558a0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550109040 0x7f155010a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:40034/0 (socket says 172.21.15.151:40034) 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.809+0000 7f15558a0700 1 -- 172.21.15.151:0/2908225952 learned_addr learned my addr 172.21.15.151:0/2908225952 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.809+0000 7f15560a1700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1550107d90 0x7f155010bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.809+0000 7f15558a0700 1 -- 172.21.15.151:0/2908225952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1550107d90 msgr2=0x7f155010bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.809+0000 7f15558a0700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1550107d90 0x7f155010bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:23.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.809+0000 7f15558a0700 1 -- 172.21.15.151:0/2908225952 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0095f0 con 0x7f1550109040 2024-04-16T02:36:23.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.809+0000 7f15558a0700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550109040 0x7f155010a180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f154000c3f0 tx=0x7f154000cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:23.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.810+0000 7f15477fe700 1 -- 172.21.15.151:0/2908225952 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1540004100 con 0x7f1550109040 2024-04-16T02:36:23.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.810+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154000a630 con 0x7f1550109040 2024-04-16T02:36:23.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.810+0000 7f15477fe700 1 -- 172.21.15.151:0/2908225952 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1540004260 con 0x7f1550109040 2024-04-16T02:36:23.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.810+0000 7f15477fe700 1 -- 172.21.15.151:0/2908225952 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f154001f820 con 0x7f1550109040 2024-04-16T02:36:23.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.810+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155010ade0 con 0x7f1550109040 2024-04-16T02:36:23.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.811+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1550198b30 con 0x7f1550109040 2024-04-16T02:36:23.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.812+0000 7f15477fe700 1 -- 172.21.15.151:0/2908225952 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7f154000e450 con 0x7f1550109040 2024-04-16T02:36:23.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.813+0000 7f15477fe700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f153c06d8a0 0x7f153c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:23.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.813+0000 7f15477fe700 1 -- 172.21.15.151:0/2908225952 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f1540012070 con 0x7f1550109040 2024-04-16T02:36:23.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.813+0000 7f15560a1700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f153c06d8a0 0x7f153c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:23.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.813+0000 7f15560a1700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f153c06d8a0 0x7f153c06fd60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1550072cb0 tx=0x7f154c009310 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:23.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:23.817+0000 7f15477fe700 1 -- 172.21.15.151:0/2908225952 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f154005be40 con 0x7f1550109040 2024-04-16T02:36:24.054 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:23 smithi153 ceph-mon[36747]: pgmap v39: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 33 MiB/s rd, 74 MiB/s wr, 369 op/s 2024-04-16T02:36:24.140 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:23 smithi151 ceph-mon[29003]: pgmap v39: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 33 MiB/s rd, 74 MiB/s wr, 369 op/s 2024-04-16T02:36:24.146 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.144+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f15500611d0 con 0x7f153c06d8a0 2024-04-16T02:36:24.147 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.146+0000 7f15477fe700 1 -- 172.21.15.151:0/2908225952 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f15500611d0 con 0x7f153c06d8a0 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.148+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f153c06d8a0 msgr2=0x7f153c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.148+0000 7f1558305700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f153c06d8a0 0x7f153c06fd60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1550072cb0 tx=0x7f154c009310 comp rx=0 tx=0).stop 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.148+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550109040 msgr2=0x7f155010a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.148+0000 7f1558305700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550109040 0x7f155010a180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f154000c3f0 tx=0x7f154000cf40 comp rx=0 tx=0).stop 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.148+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 shutdown_connections 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.148+0000 7f1558305700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7f153c06d8a0 0x7f153c06fd60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.148+0000 7f1558305700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1550107d90 0x7f155010bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.148+0000 7f1558305700 1 --2- 172.21.15.151:0/2908225952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1550109040 0x7f155010a180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.148+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 >> 172.21.15.151:0/2908225952 conn(0x7f155006d9f0 msgr2=0x7f155006e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.149+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 shutdown_connections 2024-04-16T02:36:24.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.149+0000 7f1558305700 1 -- 172.21.15.151:0/2908225952 wait complete. 2024-04-16T02:36:24.165 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:36:24.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.305+0000 7fc68ffff700 1 -- 172.21.15.151:0/2014785341 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc690007bf0 con 0x7fc698072b80 2024-04-16T02:36:24.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.305+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2014785341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 msgr2=0x7fc69810aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:24.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.305+0000 7fc69f6b7700 1 --2- 172.21.15.151:0/2014785341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 0x7fc69810aeb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc69000b0a0 tx=0x7fc69000b3b0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.306+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2014785341 shutdown_connections 2024-04-16T02:36:24.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.306+0000 7fc69f6b7700 1 --2- 172.21.15.151:0/2014785341 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69810b480 0x7fc69810d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.306+0000 7fc69f6b7700 1 --2- 172.21.15.151:0/2014785341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 0x7fc69810aeb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.306+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2014785341 >> 172.21.15.151:0/2014785341 conn(0x7fc69806d9f0 msgr2=0x7fc69806fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:24.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.307+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2014785341 shutdown_connections 2024-04-16T02:36:24.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.307+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2014785341 wait complete. 2024-04-16T02:36:24.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.307+0000 7fc69f6b7700 1 Processor -- start 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.307+0000 7fc69f6b7700 1 -- start start 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.307+0000 7fc69f6b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 0x7fc69819c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.308+0000 7fc69f6b7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69810b480 0x7fc69819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.308+0000 7fc69f6b7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69819d540 con 0x7fc69810b480 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.308+0000 7fc69f6b7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69819d680 con 0x7fc698072b80 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.308+0000 7fc69d453700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 0x7fc69819c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.308+0000 7fc69d453700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 0x7fc69819c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:40040/0 (socket says 172.21.15.151:40040) 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.308+0000 7fc69d453700 1 -- 172.21.15.151:0/2590945374 learned_addr learned my addr 172.21.15.151:0/2590945374 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.308+0000 7fc69d453700 1 -- 172.21.15.151:0/2590945374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69810b480 msgr2=0x7fc69819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.308+0000 7fc69d453700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69810b480 0x7fc69819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.308+0000 7fc69d453700 1 -- 172.21.15.151:0/2590945374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc690009d00 con 0x7fc698072b80 2024-04-16T02:36:24.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.309+0000 7fc69d453700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 0x7fc69819c9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc69000b830 tx=0x7fc690003ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:24.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.309+0000 7fc68e7fc700 1 -- 172.21.15.151:0/2590945374 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc69000e040 con 0x7fc698072b80 2024-04-16T02:36:24.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.309+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6981a20d0 con 0x7fc698072b80 2024-04-16T02:36:24.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.309+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6981a26a0 con 0x7fc698072b80 2024-04-16T02:36:24.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.309+0000 7fc68e7fc700 1 -- 172.21.15.151:0/2590945374 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc6900096a0 con 0x7fc698072b80 2024-04-16T02:36:24.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.309+0000 7fc68e7fc700 1 -- 172.21.15.151:0/2590945374 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc690012900 con 0x7fc698072b80 2024-04-16T02:36:24.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.312+0000 7fc68e7fc700 1 -- 172.21.15.151:0/2590945374 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7fc690019070 con 0x7fc698072b80 2024-04-16T02:36:24.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.312+0000 7fc68e7fc700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc68406d840 0x7fc68406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:24.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.312+0000 7fc68e7fc700 1 -- 172.21.15.151:0/2590945374 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fc69008e880 con 0x7fc698072b80 2024-04-16T02:36:24.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.313+0000 7fc69cc52700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc68406d840 0x7fc68406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:24.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.313+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc67c005320 con 0x7fc698072b80 2024-04-16T02:36:24.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.314+0000 7fc69cc52700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc68406d840 0x7fc68406fd00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc688005950 tx=0x7fc6880058e0 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:24.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.324+0000 7fc68e7fc700 1 -- 172.21.15.151:0/2590945374 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fc69005b5f0 con 0x7fc698072b80 2024-04-16T02:36:24.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.696+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc67c000bf0 con 0x7fc68406d840 2024-04-16T02:36:24.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.697+0000 7fc68e7fc700 1 -- 172.21.15.151:0/2590945374 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fc67c000bf0 con 0x7fc68406d840 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.699+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc68406d840 msgr2=0x7fc68406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.699+0000 7fc69f6b7700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc68406d840 0x7fc68406fd00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc688005950 tx=0x7fc6880058e0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.699+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 msgr2=0x7fc69819c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.699+0000 7fc69f6b7700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 0x7fc69819c9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc69000b830 tx=0x7fc690003ce0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.700+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 shutdown_connections 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.700+0000 7fc69f6b7700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fc68406d840 0x7fc68406fd00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.700+0000 7fc69f6b7700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc698072b80 0x7fc69819c9e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.700+0000 7fc69f6b7700 1 --2- 172.21.15.151:0/2590945374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc69810b480 0x7fc69819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.700+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 >> 172.21.15.151:0/2590945374 conn(0x7fc69806d9f0 msgr2=0x7fc698108620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.700+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 shutdown_connections 2024-04-16T02:36:24.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.700+0000 7fc69f6b7700 1 -- 172.21.15.151:0/2590945374 wait complete. 2024-04-16T02:36:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.862+0000 7fd725605700 1 -- 172.21.15.151:0/1852357181 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd720100cb0 msgr2=0x7fd720101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.862+0000 7fd725605700 1 --2- 172.21.15.151:0/1852357181 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd720100cb0 0x7fd720101110 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd71400ba80 tx=0x7fd71400bd90 comp rx=0 tx=0).stop 2024-04-16T02:36:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.862+0000 7fd725605700 1 -- 172.21.15.151:0/1852357181 shutdown_connections 2024-04-16T02:36:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.862+0000 7fd725605700 1 --2- 172.21.15.151:0/1852357181 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd720100cb0 0x7fd720101110 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.862+0000 7fd725605700 1 --2- 172.21.15.151:0/1852357181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7200ffab0 0x7fd7200ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.862+0000 7fd725605700 1 -- 172.21.15.151:0/1852357181 >> 172.21.15.151:0/1852357181 conn(0x7fd7200fb050 msgr2=0x7fd7200fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.863+0000 7fd725605700 1 -- 172.21.15.151:0/1852357181 shutdown_connections 2024-04-16T02:36:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.863+0000 7fd725605700 1 -- 172.21.15.151:0/1852357181 wait complete. 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.863+0000 7fd725605700 1 Processor -- start 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.864+0000 7fd725605700 1 -- start start 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.864+0000 7fd725605700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7200ffab0 0x7fd720194120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.864+0000 7fd725605700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd720100cb0 0x7fd720194660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.864+0000 7fd725605700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd720194c80 con 0x7fd7200ffab0 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.864+0000 7fd725605700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd720194dc0 con 0x7fd720100cb0 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.864+0000 7fd71effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7200ffab0 0x7fd720194120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.864+0000 7fd71effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7200ffab0 0x7fd720194120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47384/0 (socket says 172.21.15.151:47384) 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.864+0000 7fd71effd700 1 -- 172.21.15.151:0/77645604 learned_addr learned my addr 172.21.15.151:0/77645604 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.864+0000 7fd71e7fc700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd720100cb0 0x7fd720194660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.865+0000 7fd71effd700 1 -- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd720100cb0 msgr2=0x7fd720194660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.865+0000 7fd71effd700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd720100cb0 0x7fd720194660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:24.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.865+0000 7fd71effd700 1 -- 172.21.15.151:0/77645604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71400b730 con 0x7fd7200ffab0 2024-04-16T02:36:24.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.865+0000 7fd71effd700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7200ffab0 0x7fd720194120 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd71000e970 tx=0x7fd71000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:24.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.865+0000 7fd707fff700 1 -- 172.21.15.151:0/77645604 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd71000c9d0 con 0x7fd7200ffab0 2024-04-16T02:36:24.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.866+0000 7fd707fff700 1 -- 172.21.15.151:0/77645604 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd710004020 con 0x7fd7200ffab0 2024-04-16T02:36:24.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.866+0000 7fd707fff700 1 -- 172.21.15.151:0/77645604 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd710003d00 con 0x7fd7200ffab0 2024-04-16T02:36:24.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.866+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd720199870 con 0x7fd7200ffab0 2024-04-16T02:36:24.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.866+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd720199e40 con 0x7fd7200ffab0 2024-04-16T02:36:24.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.867+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd720107500 con 0x7fd7200ffab0 2024-04-16T02:36:24.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.868+0000 7fd707fff700 1 -- 172.21.15.151:0/77645604 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7fd710004190 con 0x7fd7200ffab0 2024-04-16T02:36:24.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.869+0000 7fd707fff700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd70806d890 0x7fd70806fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:24.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.869+0000 7fd707fff700 1 -- 172.21.15.151:0/77645604 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fd710014070 con 0x7fd7200ffab0 2024-04-16T02:36:24.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.869+0000 7fd71e7fc700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd70806d890 0x7fd70806fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:24.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.870+0000 7fd71e7fc700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd70806d890 0x7fd70806fd50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd71400ba50 tx=0x7fd71400d330 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:24.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:24.879+0000 7fd707fff700 1 -- 172.21.15.151:0/77645604 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fd71005a270 con 0x7fd7200ffab0 2024-04-16T02:36:25.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.252+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd720105600 con 0x7fd70806d890 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (7m) 40s ago 9m 24.6M - 0.25.0 c8568f914cd2 9f7a3dfb7aa4 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (9m) 40s ago 9m 7750k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (8m) 64s ago 8m 7650k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (9m) 40s ago 9m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (8m) 64s ago 8m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (7m) 40s ago 9m 82.7M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (3m) 40s ago 3m 99.9M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (3m) 40s ago 3m 114M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (3m) 64s ago 3m 58.3M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:36:25.267 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (3m) 64s ago 3m 66.0M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:9283,8765,8443 running (10m) 40s ago 10m 340M - 18.2.1 5be31c24972a 9e98c5bc45da 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (98s) 64s ago 8m 489M - 19.0.0-3084-g87dbfb63 a452a57266c7 cd0d253df44f 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (10m) 40s ago 10m 51.0M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (8m) 64s ago 8m 40.0M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (9m) 40s ago 9m 15.6M - 1.5.0 0da6a335fe13 7748f5130613 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (8m) 64s ago 8m 16.3M - 1.5.0 0da6a335fe13 6a289489fe28 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (7m) 40s ago 7m 112M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (6m) 40s ago 6m 123M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (6m) 40s ago 6m 103M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (6m) 64s ago 6m 125M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (5m) 64s ago 5m 121M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (5m) 64s ago 5m 108M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 starting - - - - 2024-04-16T02:36:25.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.264+0000 7fd707fff700 1 -- 172.21.15.151:0/77645604 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd720105600 con 0x7fd70806d890 2024-04-16T02:36:25.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd70806d890 msgr2=0x7fd70806fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:25.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd70806d890 0x7fd70806fd50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd71400ba50 tx=0x7fd71400d330 comp rx=0 tx=0).stop 2024-04-16T02:36:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7200ffab0 msgr2=0x7fd720194120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7200ffab0 0x7fd720194120 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd71000e970 tx=0x7fd71000ed30 comp rx=0 tx=0).stop 2024-04-16T02:36:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 shutdown_connections 2024-04-16T02:36:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fd70806d890 0x7fd70806fd50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7200ffab0 0x7fd720194120 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd71000e970 tx=0x7fd71000ed30 comp rx=0 tx=0).stop 2024-04-16T02:36:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 --2- 172.21.15.151:0/77645604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd720100cb0 0x7fd720194660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 >> 172.21.15.151:0/77645604 conn(0x7fd7200fb050 msgr2=0x7fd720103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 shutdown_connections 2024-04-16T02:36:25.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.268+0000 7fd725605700 1 -- 172.21.15.151:0/77645604 wait complete. 2024-04-16T02:36:25.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.449+0000 7fefeb59e700 1 -- 172.21.15.151:0/3236448670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefec100ae0 msgr2=0x7fefec100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:25.452 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.449+0000 7fefeb59e700 1 --2- 172.21.15.151:0/3236448670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefec100ae0 0x7fefec100f40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009910 tx=0x7fefe0009c20 comp rx=0 tx=0).stop 2024-04-16T02:36:25.452 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.449+0000 7fefeb59e700 1 -- 172.21.15.151:0/3236448670 shutdown_connections 2024-04-16T02:36:25.452 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.449+0000 7fefeb59e700 1 --2- 172.21.15.151:0/3236448670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefec100ae0 0x7fefec100f40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:25.452 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.449+0000 7fefeb59e700 1 --2- 172.21.15.151:0/3236448670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefec0ff8e0 0x7fefec0ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:25.452 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.449+0000 7fefeb59e700 1 -- 172.21.15.151:0/3236448670 >> 172.21.15.151:0/3236448670 conn(0x7fefec0fae40 msgr2=0x7fefec0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:25.452 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.450+0000 7fefeb59e700 1 -- 172.21.15.151:0/3236448670 shutdown_connections 2024-04-16T02:36:25.452 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.450+0000 7fefeb59e700 1 -- 172.21.15.151:0/3236448670 wait complete. 2024-04-16T02:36:25.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.451+0000 7fefeb59e700 1 Processor -- start 2024-04-16T02:36:25.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.451+0000 7fefeb59e700 1 -- start start 2024-04-16T02:36:25.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.451+0000 7fefeb59e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefec0ff8e0 0x7fefec19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:25.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.451+0000 7fefeb59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefec100ae0 0x7fefec19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:25.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.451+0000 7fefeb59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec19d310 con 0x7fefec100ae0 2024-04-16T02:36:25.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.451+0000 7fefeb59e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec19d450 con 0x7fefec0ff8e0 2024-04-16T02:36:25.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.451+0000 7fefe9d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefec100ae0 0x7fefec19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:25.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.451+0000 7fefe9d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefec100ae0 0x7fefec19ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47402/0 (socket says 172.21.15.151:47402) 2024-04-16T02:36:25.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.451+0000 7fefe9d9b700 1 -- 172.21.15.151:0/3712906757 learned_addr learned my addr 172.21.15.151:0/3712906757 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:25.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.452+0000 7fefea59c700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefec0ff8e0 0x7fefec19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:25.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.452+0000 7fefe9d9b700 1 -- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefec0ff8e0 msgr2=0x7fefec19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.452+0000 7fefe9d9b700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefec0ff8e0 0x7fefec19c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.452+0000 7fefe9d9b700 1 -- 172.21.15.151:0/3712906757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe00095f0 con 0x7fefec100ae0 2024-04-16T02:36:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.452+0000 7fefe9d9b700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefec100ae0 0x7fefec19ccf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fefe00092e0 tx=0x7fefe000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.453+0000 7fefdb7fe700 1 -- 172.21.15.151:0/3712906757 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe001d070 con 0x7fefec100ae0 2024-04-16T02:36:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.453+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec1a1e50 con 0x7fefec100ae0 2024-04-16T02:36:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.453+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec1a2310 con 0x7fefec100ae0 2024-04-16T02:36:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.453+0000 7fefdb7fe700 1 -- 172.21.15.151:0/3712906757 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fefe0005dc0 con 0x7fefec100ae0 2024-04-16T02:36:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.453+0000 7fefdb7fe700 1 -- 172.21.15.151:0/3712906757 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe0017630 con 0x7fefec100ae0 2024-04-16T02:36:25.457 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.455+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec108440 con 0x7fefec100ae0 2024-04-16T02:36:25.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.456+0000 7fefdb7fe700 1 -- 172.21.15.151:0/3712906757 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7fefe000faf0 con 0x7fefec100ae0 2024-04-16T02:36:25.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.457+0000 7fefdb7fe700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fefd406d880 0x7fefd406fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:25.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.457+0000 7fefdb7fe700 1 -- 172.21.15.151:0/3712906757 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fefe008e5c0 con 0x7fefec100ae0 2024-04-16T02:36:25.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.458+0000 7fefea59c700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fefd406d880 0x7fefd406fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:25.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.459+0000 7fefea59c700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fefd406d880 0x7fefd406fd40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fefec100940 tx=0x7fefdc009260 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:25.468 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.466+0000 7fefdb7fe700 1 -- 172.21.15.151:0/3712906757 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fefe005b3d0 con 0x7fefec100ae0 2024-04-16T02:36:25.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.899+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fefec108660 con 0x7fefec100ae0 2024-04-16T02:36:25.901 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:25 smithi151 ceph-mon[29003]: from='client.24425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:25.901 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:25 smithi151 ceph-mon[29003]: pgmap v40: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 14 MiB/s rd, 34 MiB/s wr, 231 op/s 2024-04-16T02:36:25.901 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:25 smithi151 ceph-mon[29003]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:25.901 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:25 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.899+0000 7fefdb7fe700 1 -- 172.21.15.151:0/3712906757 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7fefe0057030 con 0x7fefec100ae0 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 1 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:36:25.902 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:36:25.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:36:25.903 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:36:25.903 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:36:25.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:36:25.903 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:36:25.903 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-04-16T02:36:25.903 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 1 2024-04-16T02:36:25.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:36:25.903 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:36:25.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fefd406d880 msgr2=0x7fefd406fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:25.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fefd406d880 0x7fefd406fd40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fefec100940 tx=0x7fefdc009260 comp rx=0 tx=0).stop 2024-04-16T02:36:25.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefec100ae0 msgr2=0x7fefec19ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:25.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefec100ae0 0x7fefec19ccf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fefe00092e0 tx=0x7fefe000f690 comp rx=0 tx=0).stop 2024-04-16T02:36:25.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 shutdown_connections 2024-04-16T02:36:25.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fefd406d880 0x7fefd406fd40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:25.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefec0ff8e0 0x7fefec19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:25.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 --2- 172.21.15.151:0/3712906757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefec100ae0 0x7fefec19ccf0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:25.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 >> 172.21.15.151:0/3712906757 conn(0x7fefec0fae40 msgr2=0x7fefec068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:25.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 shutdown_connections 2024-04-16T02:36:25.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:25.903+0000 7fefeb59e700 1 -- 172.21.15.151:0/3712906757 wait complete. 2024-04-16T02:36:26.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.063+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/2557876860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a40ff240 msgr2=0x7fe8a40ff660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:26.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.063+0000 7fe8ac1d9700 1 --2- 172.21.15.151:0/2557876860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a40ff240 0x7fe8a40ff660 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009910 tx=0x7fe8a0009c20 comp rx=0 tx=0).stop 2024-04-16T02:36:26.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.063+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/2557876860 shutdown_connections 2024-04-16T02:36:26.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.063+0000 7fe8ac1d9700 1 --2- 172.21.15.151:0/2557876860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4100440 0x7fe8a41008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:26.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.063+0000 7fe8ac1d9700 1 --2- 172.21.15.151:0/2557876860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a40ff240 0x7fe8a40ff660 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:26.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.063+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/2557876860 >> 172.21.15.151:0/2557876860 conn(0x7fe8a40fa7c0 msgr2=0x7fe8a40fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:26.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.063+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/2557876860 shutdown_connections 2024-04-16T02:36:26.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.063+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/2557876860 wait complete. 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8ac1d9700 1 Processor -- start 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8ac1d9700 1 -- start start 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8ac1d9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40ff240 0x7fe8a4071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8ac1d9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a4100440 0x7fe8a4072100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8ac1d9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a4072740 con 0x7fe8a4100440 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8ac1d9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a41a3f60 con 0x7fe8a40ff240 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8a9774700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a4100440 0x7fe8a4072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8a9f75700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40ff240 0x7fe8a4071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8a9f75700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40ff240 0x7fe8a4071bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:40080/0 (socket says 172.21.15.151:40080) 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8a9774700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a4100440 0x7fe8a4072100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47420/0 (socket says 172.21.15.151:47420) 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.064+0000 7fe8a9f75700 1 -- 172.21.15.151:0/3439066356 learned_addr learned my addr 172.21.15.151:0/3439066356 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.065+0000 7fe8a9f75700 1 -- 172.21.15.151:0/3439066356 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a4100440 msgr2=0x7fe8a4072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.065+0000 7fe8a9f75700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a4100440 0x7fe8a4072100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.065+0000 7fe8a9f75700 1 -- 172.21.15.151:0/3439066356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89400a630 con 0x7fe8a40ff240 2024-04-16T02:36:26.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.065+0000 7fe8a9f75700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40ff240 0x7fe8a4071bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0006010 tx=0x7fe8a000f9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:26.069 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.065+0000 7fe89affd700 1 -- 172.21.15.151:0/3439066356 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8a001d070 con 0x7fe8a40ff240 2024-04-16T02:36:26.069 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.065+0000 7fe89affd700 1 -- 172.21.15.151:0/3439066356 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe8a000b5a0 con 0x7fe8a40ff240 2024-04-16T02:36:26.069 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.065+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a00095f0 con 0x7fe8a40ff240 2024-04-16T02:36:26.069 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.065+0000 7fe89affd700 1 -- 172.21.15.151:0/3439066356 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8a0017630 con 0x7fe8a40ff240 2024-04-16T02:36:26.069 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.065+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a41a4310 con 0x7fe8a40ff240 2024-04-16T02:36:26.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.066+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a4107510 con 0x7fe8a40ff240 2024-04-16T02:36:26.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.067+0000 7fe89affd700 1 -- 172.21.15.151:0/3439066356 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7fe8a0003680 con 0x7fe8a40ff240 2024-04-16T02:36:26.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.068+0000 7fe89affd700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fe89006d840 0x7fe89006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:26.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.068+0000 7fe89affd700 1 -- 172.21.15.151:0/3439066356 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fe8a008ea00 con 0x7fe8a40ff240 2024-04-16T02:36:26.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.068+0000 7fe8a9774700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fe89006d840 0x7fe89006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:26.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.068+0000 7fe8a9774700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fe89006d840 0x7fe89006fd00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe89400a600 tx=0x7fe894010040 comp rx=0 tx=0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:26.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.076+0000 7fe89affd700 1 -- 172.21.15.151:0/3439066356 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fe8a005b6f0 con 0x7fe8a40ff240 2024-04-16T02:36:26.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:25 smithi153 ceph-mon[36747]: from='client.24425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:26.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:25 smithi153 ceph-mon[36747]: pgmap v40: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 14 MiB/s rd, 34 MiB/s wr, 231 op/s 2024-04-16T02:36:26.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:25 smithi153 ceph-mon[36747]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:26.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:25 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:36:26.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.451+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 --> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8a4104a70 con 0x7fe89006d840 2024-04-16T02:36:26.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.452+0000 7fe89affd700 1 -- 172.21.15.151:0/3439066356 <== mgr.24357 v2:172.21.15.153:6828/2287826646 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fe8a4104a70 con 0x7fe89006d840 2024-04-16T02:36:26.455 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:36:26.455 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:36:26.455 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:36:26.455 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-16T02:36:26.455 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [], 2024-04-16T02:36:26.455 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "1/2 daemons upgraded", 2024-04-16T02:36:26.455 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mgr daemons", 2024-04-16T02:36:26.455 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:36:26.455 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:36:26.457 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fe89006d840 msgr2=0x7fe89006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fe89006d840 0x7fe89006fd00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe89400a600 tx=0x7fe894010040 comp rx=0 tx=0).stop 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40ff240 msgr2=0x7fe8a4071bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40ff240 0x7fe8a4071bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0006010 tx=0x7fe8a000f9e0 comp rx=0 tx=0).stop 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 shutdown_connections 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:6828/2287826646,v1:172.21.15.153:6829/2287826646] conn(0x7fe89006d840 0x7fe89006fd00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40ff240 0x7fe8a4071bc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 --2- 172.21.15.151:0/3439066356 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a4100440 0x7fe8a4072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 >> 172.21.15.151:0/3439066356 conn(0x7fe8a40fa7c0 msgr2=0x7fe8a40fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 shutdown_connections 2024-04-16T02:36:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:26.455+0000 7fe8ac1d9700 1 -- 172.21.15.151:0/3439066356 wait complete. 2024-04-16T02:36:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:26 smithi151 ceph-mon[29003]: from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:27.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:26 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3712906757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:36:27.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:26 smithi153 ceph-mon[36747]: from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:27.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:26 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3712906757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:36:27.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:27 smithi151 ceph-mon[29003]: pgmap v41: 65 pgs: 65 active+clean; 2.1 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 14 MiB/s rd, 34 MiB/s wr, 283 op/s 2024-04-16T02:36:27.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:27 smithi151 ceph-mon[29003]: from='client.24441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:28.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:27 smithi153 ceph-mon[36747]: pgmap v41: 65 pgs: 65 active+clean; 2.1 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 14 MiB/s rd, 34 MiB/s wr, 283 op/s 2024-04-16T02:36:28.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:27 smithi153 ceph-mon[36747]: from='client.24441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:29.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:29 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:29.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:29 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:29.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:29 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:36:29.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:29 smithi151 ceph-mon[29003]: pgmap v42: 65 pgs: 65 active+clean; 1.7 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 11 MiB/s rd, 26 MiB/s wr, 357 op/s 2024-04-16T02:36:29.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:29 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:29.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:29 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:29.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:29 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:29.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:29 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:29.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:29 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:36:29.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:29 smithi153 ceph-mon[36747]: pgmap v42: 65 pgs: 65 active+clean; 1.7 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 11 MiB/s rd, 26 MiB/s wr, 357 op/s 2024-04-16T02:36:29.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:29 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:29.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:29 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:32.142 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:31 smithi151 ceph-mon[29003]: pgmap v43: 65 pgs: 65 active+clean; 1.7 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 3.8 MiB/s rd, 7.1 MiB/s wr, 276 op/s 2024-04-16T02:36:32.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:31 smithi153 ceph-mon[36747]: pgmap v43: 65 pgs: 65 active+clean; 1.7 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 3.8 MiB/s rd, 7.1 MiB/s wr, 276 op/s 2024-04-16T02:36:33.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:33 smithi151 ceph-mon[29003]: pgmap v44: 65 pgs: 65 active+clean; 1.2 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 4.4 MiB/s rd, 7.7 MiB/s wr, 402 op/s 2024-04-16T02:36:34.156 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:33 smithi153 ceph-mon[36747]: pgmap v44: 65 pgs: 65 active+clean; 1.2 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 4.4 MiB/s rd, 7.7 MiB/s wr, 402 op/s 2024-04-16T02:36:36.121 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:35 smithi151 ceph-mon[29003]: pgmap v45: 65 pgs: 65 active+clean; 1.2 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 282 op/s 2024-04-16T02:36:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:35 smithi153 ceph-mon[36747]: pgmap v45: 65 pgs: 65 active+clean; 1.2 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 282 op/s 2024-04-16T02:36:37.344 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-16T02:36:37.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-16T02:36:38.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:37 smithi151 ceph-mon[29003]: pgmap v46: 65 pgs: 65 active+clean; 1.0 GiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 310 op/s 2024-04-16T02:36:38.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:37 smithi153 ceph-mon[36747]: pgmap v46: 65 pgs: 65 active+clean; 1.0 GiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 310 op/s 2024-04-16T02:36:39.207 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:38 smithi153 ceph-mon[36747]: Standby manager daemon smithi151.uqqame restarted 2024-04-16T02:36:39.208 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:38 smithi153 ceph-mon[36747]: Standby manager daemon smithi151.uqqame started 2024-04-16T02:36:39.208 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:38 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi151.uqqame/crt"}]: dispatch 2024-04-16T02:36:39.208 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:38 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:36:39.208 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:38 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi151.uqqame/key"}]: dispatch 2024-04-16T02:36:39.208 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:38 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:36:39.208 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:38 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:39.208 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:38 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:38 smithi151 ceph-mon[29003]: Standby manager daemon smithi151.uqqame restarted 2024-04-16T02:36:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:38 smithi151 ceph-mon[29003]: Standby manager daemon smithi151.uqqame started 2024-04-16T02:36:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:38 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi151.uqqame/crt"}]: dispatch 2024-04-16T02:36:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:38 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:36:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:38 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi151.uqqame/key"}]: dispatch 2024-04-16T02:36:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:38 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:36:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:38 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:38 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:40.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:39 smithi151 ceph-mon[29003]: pgmap v47: 65 pgs: 65 active+clean; 634 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 372 op/s 2024-04-16T02:36:40.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:39 smithi151 ceph-mon[29003]: mgrmap e26: smithi153.esbtzf(active, since 88s), standbys: smithi151.uqqame 2024-04-16T02:36:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:39 smithi153 ceph-mon[36747]: pgmap v47: 65 pgs: 65 active+clean; 634 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 372 op/s 2024-04-16T02:36:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:39 smithi153 ceph-mon[36747]: mgrmap e26: smithi153.esbtzf(active, since 88s), standbys: smithi151.uqqame 2024-04-16T02:36:41.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:41 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:41.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:41 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:41.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:41 smithi151 ceph-mon[29003]: pgmap v48: 65 pgs: 65 active+clean; 634 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-04-16T02:36:41.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:41 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:36:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:41 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:41 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:41 smithi153 ceph-mon[36747]: pgmap v48: 65 pgs: 65 active+clean; 634 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-04-16T02:36:41.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:41 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:36:43.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:43 smithi151 ceph-mon[29003]: pgmap v49: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 416 op/s 2024-04-16T02:36:43.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:43 smithi151 ceph-mon[29003]: mgrmap e27: smithi153.esbtzf(active, since 92s), standbys: smithi151.uqqame 2024-04-16T02:36:43.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:43 smithi153 ceph-mon[36747]: pgmap v49: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 416 op/s 2024-04-16T02:36:43.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:43 smithi153 ceph-mon[36747]: mgrmap e27: smithi153.esbtzf(active, since 92s), standbys: smithi151.uqqame 2024-04-16T02:36:46.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:45 smithi151 ceph-mon[29003]: pgmap v50: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 290 op/s 2024-04-16T02:36:46.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:45 smithi153 ceph-mon[36747]: pgmap v50: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 290 op/s 2024-04-16T02:36:48.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:47 smithi151 ceph-mon[29003]: pgmap v51: 65 pgs: 65 active+clean; 291 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 380 op/s 2024-04-16T02:36:48.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:47 smithi153 ceph-mon[36747]: pgmap v51: 65 pgs: 65 active+clean; 291 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 380 op/s 2024-04-16T02:36:50.224 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:49 smithi151 ceph-mon[29003]: pgmap v52: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 488 op/s 2024-04-16T02:36:50.240 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:49 smithi153 ceph-mon[36747]: pgmap v52: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 488 op/s 2024-04-16T02:36:51.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr fail", "who": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr fail", "who": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd='[{"prefix": "mgr fail", "who": "smithi153.esbtzf"}]': finished 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: mgrmap e28: smithi151.uqqame(active, starting, since 0.0077179s) 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:36:51.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:36:51.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: Manager daemon smithi151.uqqame is now available 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/trash_purge_schedule"}]: dispatch 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: [16/Apr/2024:02:36:50] ENGINE Bus STARTING 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: [16/Apr/2024:02:36:50] ENGINE Serving on https://172.21.15.151:7150 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: [16/Apr/2024:02:36:51] ENGINE Serving on http://172.21.15.151:8765 2024-04-16T02:36:51.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:51 smithi153 ceph-mon[36747]: [16/Apr/2024:02:36:51] ENGINE Bus STARTED 2024-04-16T02:36:51.333 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:51.333 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.24357 172.21.15.153:0/3734022008' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr fail", "who": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "mgr fail", "who": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.24357 ' entity='mgr.smithi153.esbtzf' cmd='[{"prefix": "mgr fail", "who": "smithi153.esbtzf"}]': finished 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: mgrmap e28: smithi151.uqqame(active, starting, since 0.0077179s) 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:36:51.334 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:36:51.335 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: Manager daemon smithi151.uqqame is now available 2024-04-16T02:36:51.336 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:36:51.336 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:36:51.336 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:36:51.336 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/trash_purge_schedule"}]: dispatch 2024-04-16T02:36:51.336 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: [16/Apr/2024:02:36:50] ENGINE Bus STARTING 2024-04-16T02:36:51.336 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: [16/Apr/2024:02:36:50] ENGINE Serving on https://172.21.15.151:7150 2024-04-16T02:36:51.336 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: [16/Apr/2024:02:36:51] ENGINE Serving on http://172.21.15.151:8765 2024-04-16T02:36:51.336 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:51 smithi151 ceph-mon[29003]: [16/Apr/2024:02:36:51] ENGINE Bus STARTED 2024-04-16T02:36:52.346 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:52 smithi151 ceph-mon[29003]: mgrmap e29: smithi151.uqqame(active, since 1.00895s) 2024-04-16T02:36:52.346 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:52 smithi151 ceph-mon[29003]: pgmap v3: 65 pgs: 65 active+clean; 299 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:36:52.466 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:52 smithi153 ceph-mon[36747]: mgrmap e29: smithi151.uqqame(active, since 1.00895s) 2024-04-16T02:36:52.466 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:52 smithi153 ceph-mon[36747]: pgmap v3: 65 pgs: 65 active+clean; 299 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:36:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:53 smithi151 ceph-mon[29003]: pgmap v4: 65 pgs: 65 active+clean; 299 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:36:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:53 smithi151 ceph-mon[29003]: mgrmap e30: smithi151.uqqame(active, since 2s) 2024-04-16T02:36:53.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:53 smithi153 ceph-mon[36747]: pgmap v4: 65 pgs: 65 active+clean; 299 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:36:53.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:53 smithi153 ceph-mon[36747]: mgrmap e30: smithi151.uqqame(active, since 2s) 2024-04-16T02:36:55.363 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:55 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:36:55.364 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:55 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:36:55.364 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:55 smithi153 ceph-mon[36747]: pgmap v5: 65 pgs: 65 active+clean; 299 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:36:55.364 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:55 smithi153 ceph-mon[36747]: mgrmap e31: smithi151.uqqame(active, since 4s) 2024-04-16T02:36:55.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:55 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:36:55.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:55 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:36:55.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:55 smithi151 ceph-mon[29003]: pgmap v5: 65 pgs: 65 active+clean; 299 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:36:55.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:55 smithi151 ceph-mon[29003]: mgrmap e31: smithi151.uqqame(active, since 4s) 2024-04-16T02:36:56.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.598+0000 7f0a82ab1700 1 -- 172.21.15.151:0/1374472054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c102df0 msgr2=0x7f0a7c103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:56.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.598+0000 7f0a82ab1700 1 --2- 172.21.15.151:0/1374472054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c102df0 0x7f0a7c103210 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009910 tx=0x7f0a70009c20 comp rx=0 tx=0).stop 2024-04-16T02:36:56.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.598+0000 7f0a82ab1700 1 -- 172.21.15.151:0/1374472054 shutdown_connections 2024-04-16T02:36:56.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.598+0000 7f0a82ab1700 1 --2- 172.21.15.151:0/1374472054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a7c103ff0 0x7f0a7c104450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:56.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.598+0000 7f0a82ab1700 1 --2- 172.21.15.151:0/1374472054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c102df0 0x7f0a7c103210 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:56.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.598+0000 7f0a82ab1700 1 -- 172.21.15.151:0/1374472054 >> 172.21.15.151:0/1374472054 conn(0x7f0a7c0fe390 msgr2=0x7f0a7c1007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:56.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.598+0000 7f0a82ab1700 1 -- 172.21.15.151:0/1374472054 shutdown_connections 2024-04-16T02:36:56.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.598+0000 7f0a82ab1700 1 -- 172.21.15.151:0/1374472054 wait complete. 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a82ab1700 1 Processor -- start 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a82ab1700 1 -- start start 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a82ab1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a7c102df0 0x7f0a7c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a82ab1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c103ff0 0x7f0a7c198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a82ab1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c1990e0 con 0x7f0a7c103ff0 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a82ab1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c199220 con 0x7f0a7c102df0 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a7bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c103ff0 0x7f0a7c198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a8084d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a7c102df0 0x7f0a7c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a7bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c103ff0 0x7f0a7c198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54848/0 (socket says 172.21.15.151:54848) 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.599+0000 7f0a7bfff700 1 -- 172.21.15.151:0/4174543165 learned_addr learned my addr 172.21.15.151:0/4174543165 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a8084d700 1 -- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c103ff0 msgr2=0x7f0a7c198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a8084d700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c103ff0 0x7f0a7c198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a8084d700 1 -- 172.21.15.151:0/4174543165 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a700095f0 con 0x7f0a7c102df0 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a7bfff700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c103ff0 0x7f0a7c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:36:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a8084d700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a7c102df0 0x7f0a7c198580 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009910 tx=0x7f0a7000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:56.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a79ffb700 1 -- 172.21.15.151:0/4174543165 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a7001d070 con 0x7f0a7c102df0 2024-04-16T02:36:56.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c19dc70 con 0x7f0a7c102df0 2024-04-16T02:36:56.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a79ffb700 1 -- 172.21.15.151:0/4174543165 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0a70005dc0 con 0x7f0a7c102df0 2024-04-16T02:36:56.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a79ffb700 1 -- 172.21.15.151:0/4174543165 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a70017630 con 0x7f0a7c102df0 2024-04-16T02:36:56.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.600+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c19e160 con 0x7f0a7c102df0 2024-04-16T02:36:56.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.601+0000 7f0a79ffb700 1 -- 172.21.15.151:0/4174543165 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 31) v1 ==== 46699+0+0 (secure 0 0 0) 0x7f0a70017790 con 0x7f0a7c102df0 2024-04-16T02:36:56.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.602+0000 7f0a79ffb700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f0a6c0398c0 0x7f0a6c03bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:56.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.602+0000 7f0a79ffb700 1 -- 172.21.15.151:0/4174543165 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f0a7004f0d0 con 0x7f0a7c102df0 2024-04-16T02:36:56.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.602+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c10b950 con 0x7f0a7c102df0 2024-04-16T02:36:56.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.602+0000 7f0a7bfff700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f0a6c0398c0 0x7f0a6c03bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:56.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.602+0000 7f0a7bfff700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f0a6c0398c0 0x7f0a6c03bd80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0a68011550 tx=0x7f0a68009260 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:56.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.608+0000 7f0a79ffb700 1 -- 172.21.15.151:0/4174543165 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f0a7002b440 con 0x7f0a7c102df0 2024-04-16T02:36:56.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:56 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:36:56.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:56 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:36:56.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:56 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:36:56.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:56 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:36:56.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.991+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a7c108940 con 0x7f0a6c0398c0 2024-04-16T02:36:56.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.993+0000 7f0a79ffb700 1 -- 172.21.15.151:0/4174543165 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f0a7c108940 con 0x7f0a6c0398c0 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.995+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f0a6c0398c0 msgr2=0x7f0a6c03bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.995+0000 7f0a82ab1700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f0a6c0398c0 0x7f0a6c03bd80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0a68011550 tx=0x7f0a68009260 comp rx=0 tx=0).stop 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.995+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a7c102df0 msgr2=0x7f0a7c198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.995+0000 7f0a82ab1700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a7c102df0 0x7f0a7c198580 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009910 tx=0x7f0a7000f690 comp rx=0 tx=0).stop 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.995+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 shutdown_connections 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.995+0000 7f0a82ab1700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f0a6c0398c0 0x7f0a6c03bd80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.995+0000 7f0a82ab1700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a7c102df0 0x7f0a7c198580 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.995+0000 7f0a82ab1700 1 --2- 172.21.15.151:0/4174543165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a7c103ff0 0x7f0a7c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.995+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 >> 172.21.15.151:0/4174543165 conn(0x7f0a7c0fe390 msgr2=0x7f0a7c107220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.996+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 shutdown_connections 2024-04-16T02:36:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:56.996+0000 7f0a82ab1700 1 -- 172.21.15.151:0/4174543165 wait complete. 2024-04-16T02:36:57.009 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:36:57.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.149+0000 7f027f2db700 1 -- 172.21.15.151:0/3826512096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 msgr2=0x7f02781008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.149+0000 7f027f2db700 1 --2- 172.21.15.151:0/3826512096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 0x7f02781008a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0268009910 tx=0x7f0268009c20 comp rx=0 tx=0).stop 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.149+0000 7f027f2db700 1 -- 172.21.15.151:0/3826512096 shutdown_connections 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.149+0000 7f027f2db700 1 --2- 172.21.15.151:0/3826512096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 0x7f02781008a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.149+0000 7f027f2db700 1 --2- 172.21.15.151:0/3826512096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02780ff240 0x7f02780ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.149+0000 7f027f2db700 1 -- 172.21.15.151:0/3826512096 >> 172.21.15.151:0/3826512096 conn(0x7f02780fa7c0 msgr2=0x7f02780fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.149+0000 7f027f2db700 1 -- 172.21.15.151:0/3826512096 shutdown_connections 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.150+0000 7f027f2db700 1 -- 172.21.15.151:0/3826512096 wait complete. 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.150+0000 7f027f2db700 1 Processor -- start 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.150+0000 7f027f2db700 1 -- start start 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027f2db700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02780ff240 0x7f0278071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027f2db700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 0x7f02780722e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027f2db700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0278072900 con 0x7f0278100440 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027f2db700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02781a4130 con 0x7f02780ff240 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027c876700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 0x7f02780722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:57.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027c876700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 0x7f02780722e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54864/0 (socket says 172.21.15.151:54864) 2024-04-16T02:36:57.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027c876700 1 -- 172.21.15.151:0/2488612783 learned_addr learned my addr 172.21.15.151:0/2488612783 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:57.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027d077700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02780ff240 0x7f0278071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:57.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027c876700 1 -- 172.21.15.151:0/2488612783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02780ff240 msgr2=0x7f0278071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:57.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.151+0000 7f027c876700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02780ff240 0x7f0278071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.152+0000 7f027c876700 1 -- 172.21.15.151:0/2488612783 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680095f0 con 0x7f0278100440 2024-04-16T02:36:57.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.152+0000 7f027c876700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 0x7f02780722e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f026800bd70 tx=0x7f026800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:57.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.152+0000 7f026e7fc700 1 -- 172.21.15.151:0/2488612783 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f026801d070 con 0x7f0278100440 2024-04-16T02:36:57.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.152+0000 7f026e7fc700 1 -- 172.21.15.151:0/2488612783 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0268005dc0 con 0x7f0278100440 2024-04-16T02:36:57.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.152+0000 7f026e7fc700 1 -- 172.21.15.151:0/2488612783 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0268021410 con 0x7f0278100440 2024-04-16T02:36:57.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.152+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02781a42d0 con 0x7f0278100440 2024-04-16T02:36:57.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.153+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02781a4770 con 0x7f0278100440 2024-04-16T02:36:57.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.153+0000 7f026e7fc700 1 -- 172.21.15.151:0/2488612783 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 31) v1 ==== 46699+0+0 (secure 0 0 0) 0x7f026802b440 con 0x7f0278100440 2024-04-16T02:36:57.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.154+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0278107510 con 0x7f0278100440 2024-04-16T02:36:57.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.154+0000 7f026e7fc700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f02640398c0 0x7f026403bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:57.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.154+0000 7f026e7fc700 1 -- 172.21.15.151:0/2488612783 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f026804f000 con 0x7f0278100440 2024-04-16T02:36:57.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.154+0000 7f027d077700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f02640398c0 0x7f026403bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:57.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.155+0000 7f027d077700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f02640398c0 0x7f026403bd80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f02781002a0 tx=0x7f027400b510 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.160+0000 7f026e7fc700 1 -- 172.21.15.151:0/2488612783 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f02680296e0 con 0x7f0278100440 2024-04-16T02:36:57.469 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.467+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0278104d90 con 0x7f02640398c0 2024-04-16T02:36:57.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.468+0000 7f026e7fc700 1 -- 172.21.15.151:0/2488612783 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f0278104d90 con 0x7f02640398c0 2024-04-16T02:36:57.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.470+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f02640398c0 msgr2=0x7f026403bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:57.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.470+0000 7f027f2db700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f02640398c0 0x7f026403bd80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f02781002a0 tx=0x7f027400b510 comp rx=0 tx=0).stop 2024-04-16T02:36:57.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.470+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 msgr2=0x7f02780722e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:57.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.470+0000 7f027f2db700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 0x7f02780722e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f026800bd70 tx=0x7f026800bda0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.471+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 shutdown_connections 2024-04-16T02:36:57.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.471+0000 7f027f2db700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f02640398c0 0x7f026403bd80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.471+0000 7f027f2db700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02780ff240 0x7f0278071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.471+0000 7f027f2db700 1 --2- 172.21.15.151:0/2488612783 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0278100440 0x7f02780722e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.471+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 >> 172.21.15.151:0/2488612783 conn(0x7f02780fa7c0 msgr2=0x7f0278103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:57.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.471+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 shutdown_connections 2024-04-16T02:36:57.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.471+0000 7f027f2db700 1 -- 172.21.15.151:0/2488612783 wait complete. 2024-04-16T02:36:57.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.601+0000 7feb6da79700 1 -- 172.21.15.151:0/3300107664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb68101060 msgr2=0x7feb68101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:57.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.601+0000 7feb6da79700 1 --2- 172.21.15.151:0/3300107664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb68101060 0x7feb68101480 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7feb58009910 tx=0x7feb58009c20 comp rx=0 tx=0).stop 2024-04-16T02:36:57.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.601+0000 7feb6da79700 1 -- 172.21.15.151:0/3300107664 shutdown_connections 2024-04-16T02:36:57.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.601+0000 7feb6da79700 1 --2- 172.21.15.151:0/3300107664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb680ffcc0 0x7feb68100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.601+0000 7feb6da79700 1 --2- 172.21.15.151:0/3300107664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb68101060 0x7feb68101480 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.601+0000 7feb6da79700 1 -- 172.21.15.151:0/3300107664 >> 172.21.15.151:0/3300107664 conn(0x7feb680fa790 msgr2=0x7feb680fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:57.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.601+0000 7feb6da79700 1 -- 172.21.15.151:0/3300107664 shutdown_connections 2024-04-16T02:36:57.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.601+0000 7feb6da79700 1 -- 172.21.15.151:0/3300107664 wait complete. 2024-04-16T02:36:57.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.602+0000 7feb6da79700 1 Processor -- start 2024-04-16T02:36:57.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.602+0000 7feb6da79700 1 -- start start 2024-04-16T02:36:57.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb6da79700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb680ffcc0 0x7feb68194110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:57.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb6da79700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb68101060 0x7feb68194650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:57.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb6da79700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb68194c70 con 0x7feb680ffcc0 2024-04-16T02:36:57.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb6da79700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb68194db0 con 0x7feb68101060 2024-04-16T02:36:57.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb677fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb680ffcc0 0x7feb68194110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:57.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb677fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb680ffcc0 0x7feb68194110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54876/0 (socket says 172.21.15.151:54876) 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb677fe700 1 -- 172.21.15.151:0/861009139 learned_addr learned my addr 172.21.15.151:0/861009139 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb66ffd700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb68101060 0x7feb68194650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb677fe700 1 -- 172.21.15.151:0/861009139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb68101060 msgr2=0x7feb68194650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb677fe700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb68101060 0x7feb68194650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb677fe700 1 -- 172.21.15.151:0/861009139 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580095f0 con 0x7feb680ffcc0 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.603+0000 7feb66ffd700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb68101060 0x7feb68194650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.604+0000 7feb677fe700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb680ffcc0 0x7feb68194110 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7feb58009fd0 tx=0x7feb580049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.604+0000 7feb64ff9700 1 -- 172.21.15.151:0/861009139 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb5800d340 con 0x7feb680ffcc0 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.604+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb681996c0 con 0x7feb680ffcc0 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.604+0000 7feb64ff9700 1 -- 172.21.15.151:0/861009139 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feb58021e80 con 0x7feb680ffcc0 2024-04-16T02:36:57.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.604+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb68199bd0 con 0x7feb680ffcc0 2024-04-16T02:36:57.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.605+0000 7feb64ff9700 1 -- 172.21.15.151:0/861009139 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb58003b60 con 0x7feb680ffcc0 2024-04-16T02:36:57.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.605+0000 7feb64ff9700 1 -- 172.21.15.151:0/861009139 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 31) v1 ==== 46699+0+0 (secure 0 0 0) 0x7feb58015410 con 0x7feb680ffcc0 2024-04-16T02:36:57.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.606+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb68107510 con 0x7feb680ffcc0 2024-04-16T02:36:57.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.606+0000 7feb64ff9700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feb500398c0 0x7feb5003bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:57.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.606+0000 7feb64ff9700 1 -- 172.21.15.151:0/861009139 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7feb580570a0 con 0x7feb680ffcc0 2024-04-16T02:36:57.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.607+0000 7feb66ffd700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feb500398c0 0x7feb5003bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:57.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.608+0000 7feb66ffd700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feb500398c0 0x7feb5003bd80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb68100d90 tx=0x7feb5c009260 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:57.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:57.613+0000 7feb64ff9700 1 -- 172.21.15.151:0/861009139 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7feb5805c060 con 0x7feb680ffcc0 2024-04-16T02:36:57.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:57 smithi151 ceph-mon[29003]: pgmap v6: 65 pgs: 65 active+clean; 299 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 449 KiB/s rd, 272 KiB/s wr, 61 op/s 2024-04-16T02:36:57.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:57 smithi151 ceph-mon[29003]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:57.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:57 smithi153 ceph-mon[36747]: pgmap v6: 65 pgs: 65 active+clean; 299 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 449 KiB/s rd, 272 KiB/s wr, 61 op/s 2024-04-16T02:36:57.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:57 smithi153 ceph-mon[36747]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:58.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.003+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feb68105620 con 0x7feb500398c0 2024-04-16T02:36:58.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.018+0000 7feb64ff9700 1 -- 172.21.15.151:0/861009139 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7feb68105620 con 0x7feb500398c0 2024-04-16T02:36:58.020 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:36:58.020 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (8m) 19s ago 9m 24.8M - 0.25.0 c8568f914cd2 9f7a3dfb7aa4 2024-04-16T02:36:58.020 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (10m) 19s ago 10m 7868k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:36:58.020 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (9m) 3s ago 9m 7805k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:36:58.020 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (10m) 19s ago 10m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (8m) 3s ago 8m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (8m) 19s ago 9m 86.2M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (4m) 19s ago 4m 146M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (4m) 19s ago 4m 167M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (4m) 3s ago 4m 62.8M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (4m) 3s ago 4m 100M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (32s) 19s ago 11m 90.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (2m) 3s ago 8m 75.1M - 19.0.0-3084-g87dbfb63 a452a57266c7 cd0d253df44f 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (11m) 19s ago 11m 54.0M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (8m) 3s ago 8m 44.0M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (10m) 19s ago 9m 15.4M - 1.5.0 0da6a335fe13 7748f5130613 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (8m) 3s ago 8m 16.7M - 1.5.0 0da6a335fe13 6a289489fe28 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (8m) 19s ago 8m 196M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (7m) 19s ago 7m 226M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (7m) 19s ago 7m 175M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:36:58.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (6m) 3s ago 6m 243M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:36:58.022 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (6m) 3s ago 6m 214M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:36:58.022 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (5m) 3s ago 5m 226M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:36:58.022 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (42s) 19s ago 9m 40.1M - 2.43.0 a07b618ecd1d 9bc63a550cd3 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feb500398c0 msgr2=0x7feb5003bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feb500398c0 0x7feb5003bd80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb68100d90 tx=0x7feb5c009260 comp rx=0 tx=0).stop 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb680ffcc0 msgr2=0x7feb68194110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb680ffcc0 0x7feb68194110 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7feb58009fd0 tx=0x7feb580049b0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 shutdown_connections 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feb500398c0 0x7feb5003bd80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb680ffcc0 0x7feb68194110 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 --2- 172.21.15.151:0/861009139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb68101060 0x7feb68194650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 >> 172.21.15.151:0/861009139 conn(0x7feb680fa790 msgr2=0x7feb68103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 shutdown_connections 2024-04-16T02:36:58.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.021+0000 7feb6da79700 1 -- 172.21.15.151:0/861009139 wait complete. 2024-04-16T02:36:58.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.132+0000 7f81fa360700 1 -- 172.21.15.151:0/4174907699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ffdf0 msgr2=0x7f81f4100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.132+0000 7f81fa360700 1 --2- 172.21.15.151:0/4174907699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ffdf0 0x7f81f4100270 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009910 tx=0x7f81e8009c20 comp rx=0 tx=0).stop 2024-04-16T02:36:58.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.133+0000 7f81fa360700 1 -- 172.21.15.151:0/4174907699 shutdown_connections 2024-04-16T02:36:58.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.133+0000 7f81fa360700 1 --2- 172.21.15.151:0/4174907699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ffdf0 0x7f81f4100270 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.133+0000 7f81fa360700 1 --2- 172.21.15.151:0/4174907699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40ff490 0x7f81f40ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.133+0000 7f81fa360700 1 -- 172.21.15.151:0/4174907699 >> 172.21.15.151:0/4174907699 conn(0x7f81f40fb030 msgr2=0x7f81f40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:58.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.133+0000 7f81fa360700 1 -- 172.21.15.151:0/4174907699 shutdown_connections 2024-04-16T02:36:58.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.133+0000 7f81fa360700 1 -- 172.21.15.151:0/4174907699 wait complete. 2024-04-16T02:36:58.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.133+0000 7f81fa360700 1 Processor -- start 2024-04-16T02:36:58.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.134+0000 7f81fa360700 1 -- start start 2024-04-16T02:36:58.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.134+0000 7f81fa360700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ff490 0x7f81f4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:58.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.134+0000 7f81fa360700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40ffdf0 0x7f81f4198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:58.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.134+0000 7f81fa360700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f4199080 con 0x7f81f40ff490 2024-04-16T02:36:58.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.134+0000 7f81fa360700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f41991c0 con 0x7f81f40ffdf0 2024-04-16T02:36:58.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.134+0000 7f81f3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ff490 0x7f81f4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:58.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.134+0000 7f81f3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ff490 0x7f81f4198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54908/0 (socket says 172.21.15.151:54908) 2024-04-16T02:36:58.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.134+0000 7f81f3fff700 1 -- 172.21.15.151:0/3657894735 learned_addr learned my addr 172.21.15.151:0/3657894735 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:58.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.134+0000 7f81f37fe700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40ffdf0 0x7f81f4198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:58.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.135+0000 7f81f3fff700 1 -- 172.21.15.151:0/3657894735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40ffdf0 msgr2=0x7f81f4198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.135+0000 7f81f3fff700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40ffdf0 0x7f81f4198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.135+0000 7f81f3fff700 1 -- 172.21.15.151:0/3657894735 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e80095f0 con 0x7f81f40ff490 2024-04-16T02:36:58.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.135+0000 7f81f3fff700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ff490 0x7f81f4198520 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f81e400e920 tx=0x7f81e400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:58.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.135+0000 7f81f17fa700 1 -- 172.21.15.151:0/3657894735 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e400c9d0 con 0x7f81f40ff490 2024-04-16T02:36:58.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.135+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f419dc70 con 0x7f81f40ff490 2024-04-16T02:36:58.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.135+0000 7f81f17fa700 1 -- 172.21.15.151:0/3657894735 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f81e4010430 con 0x7f81f40ff490 2024-04-16T02:36:58.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.135+0000 7f81f17fa700 1 -- 172.21.15.151:0/3657894735 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e4018650 con 0x7f81f40ff490 2024-04-16T02:36:58.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.135+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f419e1c0 con 0x7f81f40ff490 2024-04-16T02:36:58.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.136+0000 7f81f17fa700 1 -- 172.21.15.151:0/3657894735 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 31) v1 ==== 46699+0+0 (secure 0 0 0) 0x7f81e4018910 con 0x7f81f40ff490 2024-04-16T02:36:58.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.136+0000 7f81f17fa700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f81dc0398c0 0x7f81dc03bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:58.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.136+0000 7f81f17fa700 1 -- 172.21.15.151:0/3657894735 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f81e4014070 con 0x7f81f40ff490 2024-04-16T02:36:58.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.137+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f410b910 con 0x7f81f40ff490 2024-04-16T02:36:58.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.137+0000 7f81f37fe700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f81dc0398c0 0x7f81dc03bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:58.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.137+0000 7f81f37fe700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f81dc0398c0 0x7f81dc03bd80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f81e800b6f0 tx=0x7f81e8005a90 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:58.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.142+0000 7f81f17fa700 1 -- 172.21.15.151:0/3657894735 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f81e401fdd0 con 0x7f81f40ff490 2024-04-16T02:36:58.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.488+0000 7f81f17fa700 1 -- 172.21.15.151:0/3657894735 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 32) v1 ==== 92702+0+0 (secure 0 0 0) 0x7f81e4018cb0 con 0x7f81f40ff490 2024-04-16T02:36:58.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.500+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f81f419e770 con 0x7f81f40ff490 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.501+0000 7f81f17fa700 1 -- 172.21.15.151:0/3657894735 <== mon.0 v2:172.21.15.151:3300/0 8 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f81e405f300 con 0x7f81f40ff490 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:36:58.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:36:58.504 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:36:58.504 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:36:58.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:36:58.504 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:36:58.504 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-16T02:36:58.504 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:36:58.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:36:58.504 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:36:58.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.503+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f81dc0398c0 msgr2=0x7f81dc03bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.503+0000 7f81fa360700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f81dc0398c0 0x7f81dc03bd80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f81e800b6f0 tx=0x7f81e8005a90 comp rx=0 tx=0).stop 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.503+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ff490 msgr2=0x7f81f4198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.503+0000 7f81fa360700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ff490 0x7f81f4198520 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f81e400e920 tx=0x7f81e400ec30 comp rx=0 tx=0).stop 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.503+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 shutdown_connections 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.503+0000 7f81fa360700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f81dc0398c0 0x7f81dc03bd80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.503+0000 7f81fa360700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f40ff490 0x7f81f4198520 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.503+0000 7f81fa360700 1 --2- 172.21.15.151:0/3657894735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40ffdf0 0x7f81f4198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.503+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 >> 172.21.15.151:0/3657894735 conn(0x7f81f40fb030 msgr2=0x7f81f4107ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.504+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 shutdown_connections 2024-04-16T02:36:58.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.504+0000 7f81fa360700 1 -- 172.21.15.151:0/3657894735 wait complete. 2024-04-16T02:36:58.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.641+0000 7fc804f9e700 1 -- 172.21.15.151:0/2601310983 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 msgr2=0x7fc8001020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.641+0000 7fc804f9e700 1 --2- 172.21.15.151:0/2601310983 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 0x7fc8001020e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009910 tx=0x7fc7f4009c20 comp rx=0 tx=0).stop 2024-04-16T02:36:58.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.641+0000 7fc804f9e700 1 -- 172.21.15.151:0/2601310983 shutdown_connections 2024-04-16T02:36:58.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.641+0000 7fc804f9e700 1 --2- 172.21.15.151:0/2601310983 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 0x7fc8001020e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.641+0000 7fc804f9e700 1 --2- 172.21.15.151:0/2601310983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc800100a80 0x7fc800100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.641+0000 7fc804f9e700 1 -- 172.21.15.151:0/2601310983 >> 172.21.15.151:0/2601310983 conn(0x7fc8000fc000 msgr2=0x7fc8000fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:58.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.641+0000 7fc804f9e700 1 -- 172.21.15.151:0/2601310983 shutdown_connections 2024-04-16T02:36:58.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.641+0000 7fc804f9e700 1 -- 172.21.15.151:0/2601310983 wait complete. 2024-04-16T02:36:58.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.642+0000 7fc804f9e700 1 Processor -- start 2024-04-16T02:36:58.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.642+0000 7fc804f9e700 1 -- start start 2024-04-16T02:36:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.642+0000 7fc804f9e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc800100a80 0x7fc8001961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.642+0000 7fc804f9e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 0x7fc8001966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.642+0000 7fc804f9e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc800196d00 con 0x7fc800101c80 2024-04-16T02:36:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.642+0000 7fc804f9e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc800196e40 con 0x7fc800100a80 2024-04-16T02:36:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.642+0000 7fc7feffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 0x7fc8001966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.642+0000 7fc7feffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 0x7fc8001966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54938/0 (socket says 172.21.15.151:54938) 2024-04-16T02:36:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.642+0000 7fc7feffd700 1 -- 172.21.15.151:0/2000972312 learned_addr learned my addr 172.21.15.151:0/2000972312 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:36:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc7ff7fe700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc800100a80 0x7fc8001961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc7feffd700 1 -- 172.21.15.151:0/2000972312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc800100a80 msgr2=0x7fc8001961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc7feffd700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc800100a80 0x7fc8001961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc7feffd700 1 -- 172.21.15.151:0/2000972312 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f40095f0 con 0x7fc800101c80 2024-04-16T02:36:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc7ff7fe700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc800100a80 0x7fc8001961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:36:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc7feffd700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 0x7fc8001966e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4000c00 tx=0x7fc7f400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc7fcff9700 1 -- 172.21.15.151:0/2000972312 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7f401d070 con 0x7fc800101c80 2024-04-16T02:36:58.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc80019b890 con 0x7fc800101c80 2024-04-16T02:36:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc7fcff9700 1 -- 172.21.15.151:0/2000972312 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc7f4005dc0 con 0x7fc800101c80 2024-04-16T02:36:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.643+0000 7fc7fcff9700 1 -- 172.21.15.151:0/2000972312 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7f40226a0 con 0x7fc800101c80 2024-04-16T02:36:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.644+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80019bd80 con 0x7fc800101c80 2024-04-16T02:36:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.645+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc800109570 con 0x7fc800101c80 2024-04-16T02:36:58.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.645+0000 7fc7fcff9700 1 -- 172.21.15.151:0/2000972312 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 32) v1 ==== 92702+0+0 (secure 0 0 0) 0x7fc7f4022800 con 0x7fc800101c80 2024-04-16T02:36:58.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.645+0000 7fc7fcff9700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fc7e806ea80 0x7fc7e8070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:36:58.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.646+0000 7fc7fcff9700 1 -- 172.21.15.151:0/2000972312 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fc7f408ff60 con 0x7fc800101c80 2024-04-16T02:36:58.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.646+0000 7fc7ff7fe700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fc7e806ea80 0x7fc7e8070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:36:58.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.646+0000 7fc7ff7fe700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fc7e806ea80 0x7fc7e8070f40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc800101ae0 tx=0x7fc7f0009260 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:36:58.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.651+0000 7fc7fcff9700 1 -- 172.21.15.151:0/2000972312 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fc7f40c4cb0 con 0x7fc800101c80 2024-04-16T02:36:58.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:58 smithi151 ceph-mon[29003]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:58.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:58 smithi151 ceph-mon[29003]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:58.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:58 smithi151 ceph-mon[29003]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:36:58.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:58 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/876336306' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:36:58.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:58 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/876336306' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:36:58.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:58 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/876336306' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:36:58.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:58 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/876336306' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:36:58.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:58 smithi153 ceph-mon[36747]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:58.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:58 smithi153 ceph-mon[36747]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:58.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:58 smithi153 ceph-mon[36747]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:36:58.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:58 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/876336306' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:36:58.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:58 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/876336306' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:36:58.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:58 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/876336306' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:36:58.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:58 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/876336306' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:36:58.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.945+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8001065d0 con 0x7fc7e806ea80 2024-04-16T02:36:58.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.947+0000 7fc7fcff9700 1 -- 172.21.15.151:0/2000972312 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fc8001065d0 con 0x7fc7e806ea80 2024-04-16T02:36:58.949 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:36:58.949 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:36:58.949 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:36:58.949 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-16T02:36:58.949 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:36:58.949 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr" 2024-04-16T02:36:58.949 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:36:58.949 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "2/2 daemons upgraded", 2024-04-16T02:36:58.950 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "", 2024-04-16T02:36:58.950 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:36:58.950 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:36:58.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.950+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fc7e806ea80 msgr2=0x7fc7e8070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.950+0000 7fc804f9e700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fc7e806ea80 0x7fc7e8070f40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc800101ae0 tx=0x7fc7f0009260 comp rx=0 tx=0).stop 2024-04-16T02:36:58.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.950+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 msgr2=0x7fc8001966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:36:58.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.950+0000 7fc804f9e700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 0x7fc8001966e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4000c00 tx=0x7fc7f400f690 comp rx=0 tx=0).stop 2024-04-16T02:36:58.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.951+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 shutdown_connections 2024-04-16T02:36:58.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.951+0000 7fc804f9e700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fc7e806ea80 0x7fc7e8070f40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.951+0000 7fc804f9e700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc800100a80 0x7fc8001961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.951+0000 7fc804f9e700 1 --2- 172.21.15.151:0/2000972312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc800101c80 0x7fc8001966e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:36:58.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.951+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 >> 172.21.15.151:0/2000972312 conn(0x7fc8000fc000 msgr2=0x7fc800104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:36:58.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.951+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 shutdown_connections 2024-04-16T02:36:58.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:36:58.951+0000 7fc804f9e700 1 -- 172.21.15.151:0/2000972312 wait complete. 2024-04-16T02:36:59.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:59 smithi151 ceph-mon[29003]: pgmap v7: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1000 KiB/s wr, 299 op/s 2024-04-16T02:36:59.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:59 smithi151 ceph-mon[29003]: mgrmap e32: smithi151.uqqame(active, since 8s), standbys: smithi153.esbtzf 2024-04-16T02:36:59.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi153.esbtzf", "id": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:36:59.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:59 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3657894735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:36:59.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:36:59 smithi151 ceph-mon[29003]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:36:59.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:59 smithi153 ceph-mon[36747]: pgmap v7: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1000 KiB/s wr, 299 op/s 2024-04-16T02:36:59.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:59 smithi153 ceph-mon[36747]: mgrmap e32: smithi151.uqqame(active, since 8s), standbys: smithi153.esbtzf 2024-04-16T02:36:59.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi153.esbtzf", "id": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:36:59.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:59 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3657894735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:36:59.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:36:59 smithi153 ceph-mon[36747]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:37:02.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:01 smithi151 ceph-mon[29003]: pgmap v8: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 881 KiB/s rd, 778 KiB/s wr, 232 op/s 2024-04-16T02:37:02.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:01 smithi153 ceph-mon[36747]: pgmap v8: 65 pgs: 65 active+clean; 300 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 881 KiB/s rd, 778 KiB/s wr, 232 op/s 2024-04-16T02:37:03.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:03 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:03.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:03 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:03.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:03 smithi151 ceph-mon[29003]: pgmap v9: 65 pgs: 65 active+clean; 296 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 361 op/s 2024-04-16T02:37:03.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:03 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:03.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:03 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:03.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:03 smithi153 ceph-mon[36747]: pgmap v9: 65 pgs: 65 active+clean; 296 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 361 op/s 2024-04-16T02:37:04.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:04 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:04.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:04 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:04.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:04 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:04.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:04 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:06.065 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:05 smithi151 ceph-mon[29003]: pgmap v10: 65 pgs: 65 active+clean; 296 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 331 op/s 2024-04-16T02:37:06.066 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:05 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:37:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:05 smithi153 ceph-mon[36747]: pgmap v10: 65 pgs: 65 active+clean; 296 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 331 op/s 2024-04-16T02:37:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:05 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:37:08.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:07 smithi151 ceph-mon[29003]: pgmap v11: 65 pgs: 65 active+clean; 294 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 367 op/s 2024-04-16T02:37:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:07 smithi153 ceph-mon[36747]: pgmap v11: 65 pgs: 65 active+clean; 294 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 367 op/s 2024-04-16T02:37:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:09 smithi153 ceph-mon[36747]: pgmap v12: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 503 op/s 2024-04-16T02:37:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:09 smithi151 ceph-mon[29003]: pgmap v12: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 503 op/s 2024-04-16T02:37:10.894 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:10 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:11.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:10 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:11.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:10 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:37:11.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:10 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:37:11.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:10 smithi151 ceph-mon[29003]: Updating smithi151:/etc/ceph/ceph.conf 2024-04-16T02:37:11.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:10 smithi151 ceph-mon[29003]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-16T02:37:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:10 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:10 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:10 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:37:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:10 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:37:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:10 smithi153 ceph-mon[36747]: Updating smithi151:/etc/ceph/ceph.conf 2024-04-16T02:37:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:10 smithi153 ceph-mon[36747]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: pgmap v13: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 354 op/s 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:11 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: pgmap v13: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 354 op/s 2024-04-16T02:37:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:37:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:37:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:37:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:37:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:37:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:37:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:11 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:12.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:12 smithi151 ceph-mon[29003]: Reconfiguring prometheus.smithi151 (dependencies changed)... 2024-04-16T02:37:12.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:12 smithi151 ceph-mon[29003]: Reconfiguring daemon prometheus.smithi151 on smithi151 2024-04-16T02:37:13.140 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-16T02:37:13.141 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-16T02:37:13.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:12 smithi153 ceph-mon[36747]: Reconfiguring prometheus.smithi151 (dependencies changed)... 2024-04-16T02:37:13.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:12 smithi153 ceph-mon[36747]: Reconfiguring daemon prometheus.smithi151 on smithi151 2024-04-16T02:37:13.439 DEBUG:teuthology.parallel:result is None 2024-04-16T02:37:14.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:13 smithi151 ceph-mon[29003]: pgmap v14: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 543 op/s 2024-04-16T02:37:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:13 smithi153 ceph-mon[36747]: pgmap v14: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 543 op/s 2024-04-16T02:37:16.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:15 smithi151 ceph-mon[29003]: pgmap v15: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 386 op/s 2024-04-16T02:37:16.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:15 smithi153 ceph-mon[36747]: pgmap v15: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 386 op/s 2024-04-16T02:37:18.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:17 smithi151 ceph-mon[29003]: pgmap v16: 65 pgs: 65 active+clean; 286 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 406 op/s 2024-04-16T02:37:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:17 smithi153 ceph-mon[36747]: pgmap v16: 65 pgs: 65 active+clean; 286 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 406 op/s 2024-04-16T02:37:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:19 smithi151 ceph-mon[29003]: pgmap v17: 65 pgs: 65 active+clean; 284 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 502 op/s 2024-04-16T02:37:20.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:19 smithi153 ceph-mon[36747]: pgmap v17: 65 pgs: 65 active+clean; 284 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 502 op/s 2024-04-16T02:37:21.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:20 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:37:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:20 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:37:22.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:21 smithi151 ceph-mon[29003]: pgmap v18: 65 pgs: 65 active+clean; 284 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 341 op/s 2024-04-16T02:37:22.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:21 smithi153 ceph-mon[36747]: pgmap v18: 65 pgs: 65 active+clean; 284 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 341 op/s 2024-04-16T02:37:24.161 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:23 smithi151 ceph-mon[29003]: pgmap v19: 65 pgs: 65 active+clean; 284 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 462 op/s 2024-04-16T02:37:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:23 smithi153 ceph-mon[36747]: pgmap v19: 65 pgs: 65 active+clean; 284 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 462 op/s 2024-04-16T02:37:26.220 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:25 smithi153 ceph-mon[36747]: pgmap v20: 65 pgs: 65 active+clean; 284 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 939 KiB/s rd, 933 KiB/s wr, 274 op/s 2024-04-16T02:37:26.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:26.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:26.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:37:26.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:37:26.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:25 smithi151 ceph-mon[29003]: pgmap v20: 65 pgs: 65 active+clean; 284 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 939 KiB/s rd, 933 KiB/s wr, 274 op/s 2024-04-16T02:37:26.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:26.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:26.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:37:26.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:37:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:26 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:37:27.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:26 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:37:28.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:27 smithi153 ceph-mon[36747]: pgmap v21: 65 pgs: 65 active+clean; 282 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 950 KiB/s rd, 945 KiB/s wr, 305 op/s 2024-04-16T02:37:28.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:27 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:37:28.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:27 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:37:28.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:27 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:37:28.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:27 smithi151 ceph-mon[29003]: pgmap v21: 65 pgs: 65 active+clean; 282 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 950 KiB/s rd, 945 KiB/s wr, 305 op/s 2024-04-16T02:37:28.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:27 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:37:28.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:27 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:37:28.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:27 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:37:29.062 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:28 smithi153 ceph-mon[36747]: Upgrade: Updating mgr.smithi153.esbtzf 2024-04-16T02:37:29.062 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:28 smithi153 ceph-mon[36747]: Deploying daemon mgr.smithi153.esbtzf on smithi153 2024-04-16T02:37:29.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.106+0000 7faeb4ebe700 1 -- 172.21.15.151:0/4108435073 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faea0004a00 con 0x7faeb0101db0 2024-04-16T02:37:29.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.107+0000 7faeb8124700 1 -- 172.21.15.151:0/4108435073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faeb0101db0 msgr2=0x7faeb0102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.107+0000 7faeb8124700 1 --2- 172.21.15.151:0/4108435073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faeb0101db0 0x7faeb0102210 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7faea0009910 tx=0x7faea0009c20 comp rx=0 tx=0).stop 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.107+0000 7faeb8124700 1 -- 172.21.15.151:0/4108435073 shutdown_connections 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.107+0000 7faeb8124700 1 --2- 172.21.15.151:0/4108435073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faeb0101db0 0x7faeb0102210 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.107+0000 7faeb8124700 1 --2- 172.21.15.151:0/4108435073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeb0100bb0 0x7faeb0100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.107+0000 7faeb8124700 1 -- 172.21.15.151:0/4108435073 >> 172.21.15.151:0/4108435073 conn(0x7faeb00fc110 msgr2=0x7faeb00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.107+0000 7faeb8124700 1 -- 172.21.15.151:0/4108435073 shutdown_connections 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.107+0000 7faeb8124700 1 -- 172.21.15.151:0/4108435073 wait complete. 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.108+0000 7faeb8124700 1 Processor -- start 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.108+0000 7faeb8124700 1 -- start start 2024-04-16T02:37:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.108+0000 7faeb8124700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faeb0100bb0 0x7faeb0074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:29.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.108+0000 7faeb8124700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeb0101db0 0x7faeb0073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:29.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.108+0000 7faeb8124700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb019a080 con 0x7faeb0100bb0 2024-04-16T02:37:29.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.108+0000 7faeb8124700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb019a1c0 con 0x7faeb0101db0 2024-04-16T02:37:29.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.109+0000 7faeb5ec0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faeb0100bb0 0x7faeb0074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:29.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.109+0000 7faeb5ec0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faeb0100bb0 0x7faeb0074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49912/0 (socket says 172.21.15.151:49912) 2024-04-16T02:37:29.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.109+0000 7faeb5ec0700 1 -- 172.21.15.151:0/1576099493 learned_addr learned my addr 172.21.15.151:0/1576099493 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:37:29.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.109+0000 7faeb56bf700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeb0101db0 0x7faeb0073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:29.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.110+0000 7faeb56bf700 1 -- 172.21.15.151:0/1576099493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faeb0100bb0 msgr2=0x7faeb0074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.110+0000 7faeb56bf700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faeb0100bb0 0x7faeb0074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.110+0000 7faeb56bf700 1 -- 172.21.15.151:0/1576099493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea00095f0 con 0x7faeb0101db0 2024-04-16T02:37:29.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.111+0000 7faeb56bf700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeb0101db0 0x7faeb0073180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faea0009fd0 tx=0x7faea0005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:29.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.111+0000 7faea6ffd700 1 -- 172.21.15.151:0/1576099493 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faea001d070 con 0x7faeb0101db0 2024-04-16T02:37:29.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.111+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb00736c0 con 0x7faeb0101db0 2024-04-16T02:37:29.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.111+0000 7faea6ffd700 1 -- 172.21.15.151:0/1576099493 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faea000bdc0 con 0x7faeb0101db0 2024-04-16T02:37:29.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.112+0000 7faea6ffd700 1 -- 172.21.15.151:0/1576099493 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faea0004a00 con 0x7faeb0101db0 2024-04-16T02:37:29.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.112+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb0073b80 con 0x7faeb0101db0 2024-04-16T02:37:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.113+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb0073df0 con 0x7faeb0101db0 2024-04-16T02:37:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.114+0000 7faea6ffd700 1 -- 172.21.15.151:0/1576099493 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 92702+0+0 (secure 0 0 0) 0x7faea0004320 con 0x7faeb0101db0 2024-04-16T02:37:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.114+0000 7faea6ffd700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fae9c06ead0 0x7fae9c070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.114+0000 7faea6ffd700 1 -- 172.21.15.151:0/1576099493 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7faea008fe10 con 0x7faeb0101db0 2024-04-16T02:37:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.115+0000 7faeb5ec0700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fae9c06ead0 0x7fae9c070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.115+0000 7faeb5ec0700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fae9c06ead0 0x7fae9c070f90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faeac009aa0 tx=0x7faeac009240 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:29.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.124+0000 7faea6ffd700 1 -- 172.21.15.151:0/1576099493 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7faea005c9e0 con 0x7faeb0101db0 2024-04-16T02:37:29.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:28 smithi151 ceph-mon[29003]: Upgrade: Updating mgr.smithi153.esbtzf 2024-04-16T02:37:29.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:28 smithi151 ceph-mon[29003]: Deploying daemon mgr.smithi153.esbtzf on smithi153 2024-04-16T02:37:29.312 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-16T02:37:29.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-16T02:37:29.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.404+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faeb0106700 con 0x7fae9c06ead0 2024-04-16T02:37:29.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.405+0000 7faea6ffd700 1 -- 172.21.15.151:0/1576099493 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7faeb0106700 con 0x7fae9c06ead0 2024-04-16T02:37:29.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fae9c06ead0 msgr2=0x7fae9c070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fae9c06ead0 0x7fae9c070f90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faeac009aa0 tx=0x7faeac009240 comp rx=0 tx=0).stop 2024-04-16T02:37:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeb0101db0 msgr2=0x7faeb0073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeb0101db0 0x7faeb0073180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faea0009fd0 tx=0x7faea0005e30 comp rx=0 tx=0).stop 2024-04-16T02:37:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 shutdown_connections 2024-04-16T02:37:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fae9c06ead0 0x7fae9c070f90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faeb0100bb0 0x7faeb0074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 --2- 172.21.15.151:0/1576099493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeb0101db0 0x7faeb0073180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 >> 172.21.15.151:0/1576099493 conn(0x7faeb00fc110 msgr2=0x7faeb0104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 shutdown_connections 2024-04-16T02:37:29.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.408+0000 7faeb8124700 1 -- 172.21.15.151:0/1576099493 wait complete. 2024-04-16T02:37:29.421 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:37:29.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.518+0000 7f2c1652b700 1 -- 172.21.15.151:0/3408529278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 msgr2=0x7f2c10100400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.518+0000 7f2c1652b700 1 --2- 172.21.15.151:0/3408529278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 0x7f2c10100400 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009910 tx=0x7f2bf8009c20 comp rx=0 tx=0).stop 2024-04-16T02:37:29.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.518+0000 7f2c1652b700 1 -- 172.21.15.151:0/3408529278 shutdown_connections 2024-04-16T02:37:29.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.518+0000 7f2c1652b700 1 --2- 172.21.15.151:0/3408529278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c10100d00 0x7f2c10101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.518+0000 7f2c1652b700 1 --2- 172.21.15.151:0/3408529278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 0x7f2c10100400 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.518+0000 7f2c1652b700 1 -- 172.21.15.151:0/3408529278 >> 172.21.15.151:0/3408529278 conn(0x7f2c100fa7c0 msgr2=0x7f2c100fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:29.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.518+0000 7f2c1652b700 1 -- 172.21.15.151:0/3408529278 shutdown_connections 2024-04-16T02:37:29.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.518+0000 7f2c1652b700 1 -- 172.21.15.151:0/3408529278 wait complete. 2024-04-16T02:37:29.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.518+0000 7f2c1652b700 1 Processor -- start 2024-04-16T02:37:29.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c1652b700 1 -- start start 2024-04-16T02:37:29.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c1652b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 0x7f2c100ff6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:29.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c1652b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c10100d00 0x7f2c1019b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:29.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c1652b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1019b700 con 0x7f2c100fffe0 2024-04-16T02:37:29.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c1652b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1019b840 con 0x7f2c10100d00 2024-04-16T02:37:29.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c0ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 0x7f2c100ff6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c0ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 0x7f2c100ff6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49936/0 (socket says 172.21.15.151:49936) 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c0ffff700 1 -- 172.21.15.151:0/2153357959 learned_addr learned my addr 172.21.15.151:0/2153357959 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c0f7fe700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c10100d00 0x7f2c1019b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c0ffff700 1 -- 172.21.15.151:0/2153357959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c10100d00 msgr2=0x7f2c1019b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c0ffff700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c10100d00 0x7f2c1019b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.519+0000 7f2c0ffff700 1 -- 172.21.15.151:0/2153357959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf80095f0 con 0x7f2c100fffe0 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.520+0000 7f2c0ffff700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 0x7f2c100ff6a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8000c00 tx=0x7f2bf800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.520+0000 7f2c0d7fa700 1 -- 172.21.15.151:0/2153357959 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bf801d070 con 0x7f2c100fffe0 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.520+0000 7f2c0d7fa700 1 -- 172.21.15.151:0/2153357959 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2bf8005dc0 con 0x7f2c100fffe0 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.520+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1019bac0 con 0x7f2c100fffe0 2024-04-16T02:37:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.520+0000 7f2c0d7fa700 1 -- 172.21.15.151:0/2153357959 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bf8022470 con 0x7f2c100fffe0 2024-04-16T02:37:29.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.520+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1019bfb0 con 0x7f2c100fffe0 2024-04-16T02:37:29.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.521+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c10108650 con 0x7f2c100fffe0 2024-04-16T02:37:29.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.521+0000 7f2c0d7fa700 1 -- 172.21.15.151:0/2153357959 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 32) v1 ==== 92702+0+0 (secure 0 0 0) 0x7f2bf800fa70 con 0x7f2c100fffe0 2024-04-16T02:37:29.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.522+0000 7f2c0d7fa700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f2bfc06ea80 0x7f2bfc070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:29.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.522+0000 7f2c0d7fa700 1 -- 172.21.15.151:0/2153357959 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f2bf80900f0 con 0x7f2c100fffe0 2024-04-16T02:37:29.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.522+0000 7f2c0f7fe700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f2bfc06ea80 0x7f2bfc070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:29.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.523+0000 7f2c0f7fe700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f2bfc06ea80 0x7f2bfc070f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2c100ffc20 tx=0x7f2c00009260 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:29.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.527+0000 7f2c0d7fa700 1 -- 172.21.15.151:0/2153357959 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f2bf805cce0 con 0x7f2c100fffe0 2024-04-16T02:37:29.620 DEBUG:teuthology.parallel:result is None 2024-04-16T02:37:29.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-16T02:37:29.668 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-16T02:37:29.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-16T02:37:29.708 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-16T02:37:29.709 DEBUG:teuthology.parallel:result is None 2024-04-16T02:37:29.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.763+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c10105650 con 0x7f2bfc06ea80 2024-04-16T02:37:29.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.764+0000 7f2c0d7fa700 1 -- 172.21.15.151:0/2153357959 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2c10105650 con 0x7f2bfc06ea80 2024-04-16T02:37:29.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.766+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f2bfc06ea80 msgr2=0x7f2bfc070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.766+0000 7f2c1652b700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f2bfc06ea80 0x7f2bfc070f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2c100ffc20 tx=0x7f2c00009260 comp rx=0 tx=0).stop 2024-04-16T02:37:29.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.766+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 msgr2=0x7f2c100ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.766+0000 7f2c1652b700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 0x7f2c100ff6a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8000c00 tx=0x7f2bf800f690 comp rx=0 tx=0).stop 2024-04-16T02:37:29.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.767+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 shutdown_connections 2024-04-16T02:37:29.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.767+0000 7f2c1652b700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f2bfc06ea80 0x7f2bfc070f40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.767+0000 7f2c1652b700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c100fffe0 0x7f2c100ff6a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.767+0000 7f2c1652b700 1 --2- 172.21.15.151:0/2153357959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c10100d00 0x7f2c1019b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.767+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 >> 172.21.15.151:0/2153357959 conn(0x7f2c100fa7c0 msgr2=0x7f2c10103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:29.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.767+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 shutdown_connections 2024-04-16T02:37:29.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.767+0000 7f2c1652b700 1 -- 172.21.15.151:0/2153357959 wait complete. 2024-04-16T02:37:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.874+0000 7f12e05ac700 1 -- 172.21.15.151:0/1473652087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 msgr2=0x7f12d8101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.874+0000 7f12e05ac700 1 --2- 172.21.15.151:0/1473652087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 0x7f12d8101120 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009910 tx=0x7f12c8009c20 comp rx=0 tx=0).stop 2024-04-16T02:37:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.874+0000 7f12e05ac700 1 -- 172.21.15.151:0/1473652087 shutdown_connections 2024-04-16T02:37:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.874+0000 7f12e05ac700 1 --2- 172.21.15.151:0/1473652087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 0x7f12d8101120 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.874+0000 7f12e05ac700 1 --2- 172.21.15.151:0/1473652087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d80ffac0 0x7f12d80ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.874+0000 7f12e05ac700 1 -- 172.21.15.151:0/1473652087 >> 172.21.15.151:0/1473652087 conn(0x7f12d80fb060 msgr2=0x7f12d80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.874+0000 7f12e05ac700 1 -- 172.21.15.151:0/1473652087 shutdown_connections 2024-04-16T02:37:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.874+0000 7f12e05ac700 1 -- 172.21.15.151:0/1473652087 wait complete. 2024-04-16T02:37:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12e05ac700 1 Processor -- start 2024-04-16T02:37:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12e05ac700 1 -- start start 2024-04-16T02:37:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12e05ac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d80ffac0 0x7f12d8069910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12e05ac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 0x7f12d819b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12e05ac700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d819b6a0 con 0x7f12d8100cc0 2024-04-16T02:37:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12e05ac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d819b7e0 con 0x7f12d80ffac0 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12ddb47700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 0x7f12d819b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12ddb47700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 0x7f12d819b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49964/0 (socket says 172.21.15.151:49964) 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12ddb47700 1 -- 172.21.15.151:0/3151187886 learned_addr learned my addr 172.21.15.151:0/3151187886 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12de348700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d80ffac0 0x7f12d8069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12ddb47700 1 -- 172.21.15.151:0/3151187886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d80ffac0 msgr2=0x7f12d8069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12ddb47700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d80ffac0 0x7f12d8069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12ddb47700 1 -- 172.21.15.151:0/3151187886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c80095f0 con 0x7f12d8100cc0 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12de348700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d80ffac0 0x7f12d8069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.875+0000 7f12ddb47700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 0x7f12d819b0d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f12c8000c00 tx=0x7f12c800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.876+0000 7f12cf7fe700 1 -- 172.21.15.151:0/3151187886 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12c801d070 con 0x7f12d8100cc0 2024-04-16T02:37:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.876+0000 7f12cf7fe700 1 -- 172.21.15.151:0/3151187886 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f12c8005dc0 con 0x7f12d8100cc0 2024-04-16T02:37:29.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.876+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d819ba60 con 0x7f12d8100cc0 2024-04-16T02:37:29.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.876+0000 7f12cf7fe700 1 -- 172.21.15.151:0/3151187886 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12c80226a0 con 0x7f12d8100cc0 2024-04-16T02:37:29.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.876+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d819bf50 con 0x7f12d8100cc0 2024-04-16T02:37:29.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.877+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d8108620 con 0x7f12d8100cc0 2024-04-16T02:37:29.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.877+0000 7f12cf7fe700 1 -- 172.21.15.151:0/3151187886 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 32) v1 ==== 92702+0+0 (secure 0 0 0) 0x7f12c800faf0 con 0x7f12d8100cc0 2024-04-16T02:37:29.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.878+0000 7f12cf7fe700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f12c406ea80 0x7f12c4070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:29.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.878+0000 7f12cf7fe700 1 -- 172.21.15.151:0/3151187886 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f12c808ff30 con 0x7f12d8100cc0 2024-04-16T02:37:29.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.878+0000 7f12de348700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f12c406ea80 0x7f12c4070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:29.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.878+0000 7f12de348700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f12c406ea80 0x7f12c4070f40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12d8100b20 tx=0x7f12d4009260 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:29.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:29.883+0000 7f12cf7fe700 1 -- 172.21.15.151:0/3151187886 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f12c80c4cb0 con 0x7f12d8100cc0 2024-04-16T02:37:30.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.131+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f12d8105610 con 0x7f12c406ea80 2024-04-16T02:37:30.133 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:29 smithi151 ceph-mon[29003]: pgmap v22: 65 pgs: 65 active+clean; 280 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 381 op/s 2024-04-16T02:37:30.146 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.143+0000 7f12cf7fe700 1 -- 172.21.15.151:0/3151187886 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f12d8105610 con 0x7f12c406ea80 2024-04-16T02:37:30.146 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:37:30.146 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (8m) 28s ago 10m 24.9M - 0.25.0 c8568f914cd2 9f7a3dfb7aa4 2024-04-16T02:37:30.146 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (10m) 28s ago 10m 7898k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:37:30.146 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (9m) 35s ago 9m 7805k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:37:30.146 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (10m) 28s ago 10m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (9m) 35s ago 9m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (8m) 28s ago 10m 86.2M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (4m) 28s ago 4m 131M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (4m) 28s ago 4m 172M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (4m) 35s ago 4m 62.8M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (4m) 35s ago 4m 100M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (64s) 28s ago 11m 479M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (2m) 35s ago 9m 75.1M - 19.0.0-3084-g87dbfb63 a452a57266c7 cd0d253df44f 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (11m) 28s ago 11m 56.0M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (9m) 35s ago 9m 44.0M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:37:30.147 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (10m) 28s ago 10m 15.9M - 1.5.0 0da6a335fe13 7748f5130613 2024-04-16T02:37:30.148 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (9m) 35s ago 9m 16.7M - 1.5.0 0da6a335fe13 6a289489fe28 2024-04-16T02:37:30.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (8m) 28s ago 8m 205M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:37:30.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (8m) 28s ago 8m 233M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:37:30.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (7m) 28s ago 7m 176M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:37:30.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (7m) 35s ago 7m 243M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:37:30.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (6m) 35s ago 6m 214M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:37:30.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (6m) 35s ago 6m 226M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:37:30.148 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 starting - - - - 2024-04-16T02:37:30.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.149+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f12c406ea80 msgr2=0x7f12c4070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:30.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.149+0000 7f12e05ac700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f12c406ea80 0x7f12c4070f40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12d8100b20 tx=0x7f12d4009260 comp rx=0 tx=0).stop 2024-04-16T02:37:30.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.149+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 msgr2=0x7f12d819b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:30.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.149+0000 7f12e05ac700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 0x7f12d819b0d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f12c8000c00 tx=0x7f12c800f690 comp rx=0 tx=0).stop 2024-04-16T02:37:30.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.150+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 shutdown_connections 2024-04-16T02:37:30.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.150+0000 7f12e05ac700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f12c406ea80 0x7f12c4070f40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.150+0000 7f12e05ac700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d80ffac0 0x7f12d8069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.150+0000 7f12e05ac700 1 --2- 172.21.15.151:0/3151187886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d8100cc0 0x7f12d819b0d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.150+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 >> 172.21.15.151:0/3151187886 conn(0x7f12d80fb060 msgr2=0x7f12d8103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:30.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.150+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 shutdown_connections 2024-04-16T02:37:30.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.150+0000 7f12e05ac700 1 -- 172.21.15.151:0/3151187886 wait complete. 2024-04-16T02:37:30.243 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:29 smithi153 ceph-mon[36747]: pgmap v22: 65 pgs: 65 active+clean; 280 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 381 op/s 2024-04-16T02:37:30.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.325+0000 7fce755af700 1 -- 172.21.15.151:0/497465863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 msgr2=0x7fce70100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:30.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.325+0000 7fce755af700 1 --2- 172.21.15.151:0/497465863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 0x7fce70100f40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fce64009910 tx=0x7fce64009c20 comp rx=0 tx=0).stop 2024-04-16T02:37:30.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.326+0000 7fce755af700 1 -- 172.21.15.151:0/497465863 shutdown_connections 2024-04-16T02:37:30.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.326+0000 7fce755af700 1 --2- 172.21.15.151:0/497465863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 0x7fce70100f40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.326+0000 7fce755af700 1 --2- 172.21.15.151:0/497465863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce700ff8e0 0x7fce700ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.326+0000 7fce755af700 1 -- 172.21.15.151:0/497465863 >> 172.21.15.151:0/497465863 conn(0x7fce700fae40 msgr2=0x7fce700fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:30.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.326+0000 7fce755af700 1 -- 172.21.15.151:0/497465863 shutdown_connections 2024-04-16T02:37:30.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.326+0000 7fce755af700 1 -- 172.21.15.151:0/497465863 wait complete. 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.326+0000 7fce755af700 1 Processor -- start 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.326+0000 7fce755af700 1 -- start start 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce755af700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce700ff8e0 0x7fce701961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce755af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 0x7fce701966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce755af700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce70196d00 con 0x7fce70100ae0 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce755af700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce70196e40 con 0x7fce700ff8e0 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 0x7fce701966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 0x7fce701966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49992/0 (socket says 172.21.15.151:49992) 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6f7fe700 1 -- 172.21.15.151:0/342552268 learned_addr learned my addr 172.21.15.151:0/342552268 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:37:30.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6f7fe700 1 -- 172.21.15.151:0/342552268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce700ff8e0 msgr2=0x7fce701961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:30.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6f7fe700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce700ff8e0 0x7fce701961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6f7fe700 1 -- 172.21.15.151:0/342552268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce640095f0 con 0x7fce70100ae0 2024-04-16T02:37:30.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6f7fe700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 0x7fce701966e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fce6400bd70 tx=0x7fce6400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:30.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6d7fa700 1 -- 172.21.15.151:0/342552268 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce6401d070 con 0x7fce70100ae0 2024-04-16T02:37:30.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7019b890 con 0x7fce70100ae0 2024-04-16T02:37:30.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7019bd80 con 0x7fce70100ae0 2024-04-16T02:37:30.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6d7fa700 1 -- 172.21.15.151:0/342552268 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fce6400f460 con 0x7fce70100ae0 2024-04-16T02:37:30.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.327+0000 7fce6d7fa700 1 -- 172.21.15.151:0/342552268 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce64021640 con 0x7fce70100ae0 2024-04-16T02:37:30.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.329+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce70108440 con 0x7fce70100ae0 2024-04-16T02:37:30.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.329+0000 7fce6d7fa700 1 -- 172.21.15.151:0/342552268 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 32) v1 ==== 92702+0+0 (secure 0 0 0) 0x7fce6400fa40 con 0x7fce70100ae0 2024-04-16T02:37:30.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.330+0000 7fce6d7fa700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fce5806eac0 0x7fce58070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:30.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.330+0000 7fce6d7fa700 1 -- 172.21.15.151:0/342552268 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fce6408fe00 con 0x7fce70100ae0 2024-04-16T02:37:30.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.330+0000 7fce6ffff700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fce5806eac0 0x7fce58070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:30.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.330+0000 7fce6ffff700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fce5806eac0 0x7fce58070f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fce60005fd0 tx=0x7fce60005d00 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:30.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.336+0000 7fce6d7fa700 1 -- 172.21.15.151:0/342552268 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fce6405c9f0 con 0x7fce70100ae0 2024-04-16T02:37:30.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.655+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fce7019c480 con 0x7fce70100ae0 2024-04-16T02:37:30.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.655+0000 7fce6d7fa700 1 -- 172.21.15.151:0/342552268 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fce6405c580 con 0x7fce70100ae0 2024-04-16T02:37:30.658 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:37:30.658 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:37:30.658 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:37:30.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:37:30.658 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:37:30.658 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:37:30.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:37:30.658 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:37:30.659 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:37:30.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.657+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fce5806eac0 msgr2=0x7fce58070f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:30.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.657+0000 7fce755af700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fce5806eac0 0x7fce58070f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fce60005fd0 tx=0x7fce60005d00 comp rx=0 tx=0).stop 2024-04-16T02:37:30.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.657+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 msgr2=0x7fce701966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:30.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.657+0000 7fce755af700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 0x7fce701966e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fce6400bd70 tx=0x7fce6400be50 comp rx=0 tx=0).stop 2024-04-16T02:37:30.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.658+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 shutdown_connections 2024-04-16T02:37:30.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.658+0000 7fce755af700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fce5806eac0 0x7fce58070f80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.658+0000 7fce755af700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce700ff8e0 0x7fce701961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.658+0000 7fce755af700 1 --2- 172.21.15.151:0/342552268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce70100ae0 0x7fce701966e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.658+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 >> 172.21.15.151:0/342552268 conn(0x7fce700fae40 msgr2=0x7fce70068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:30.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.658+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 shutdown_connections 2024-04-16T02:37:30.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.658+0000 7fce755af700 1 -- 172.21.15.151:0/342552268 wait complete. 2024-04-16T02:37:30.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.767+0000 7fe8ae03c700 1 -- 172.21.15.151:0/918915526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a81016a0 msgr2=0x7fe8a8103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.767+0000 7fe8ae03c700 1 --2- 172.21.15.151:0/918915526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a81016a0 0x7fe8a8103a90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe89800b0a0 tx=0x7fe89800b3b0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.767+0000 7fe8ae03c700 1 -- 172.21.15.151:0/918915526 shutdown_connections 2024-04-16T02:37:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.767+0000 7fe8ae03c700 1 --2- 172.21.15.151:0/918915526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a8103fd0 0x7fe8a81063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.767+0000 7fe8ae03c700 1 --2- 172.21.15.151:0/918915526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a81016a0 0x7fe8a8103a90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.767+0000 7fe8ae03c700 1 -- 172.21.15.151:0/918915526 >> 172.21.15.151:0/918915526 conn(0x7fe8a80fb0b0 msgr2=0x7fe8a80fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.767+0000 7fe8ae03c700 1 -- 172.21.15.151:0/918915526 shutdown_connections 2024-04-16T02:37:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.767+0000 7fe8ae03c700 1 -- 172.21.15.151:0/918915526 wait complete. 2024-04-16T02:37:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.767+0000 7fe8ae03c700 1 Processor -- start 2024-04-16T02:37:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8ae03c700 1 -- start start 2024-04-16T02:37:30.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8ae03c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a81016a0 0x7fe8a81963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8ae03c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a8103fd0 0x7fe8a8196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8ae03c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8196f50 con 0x7fe8a8103fd0 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8ae03c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8197090 con 0x7fe8a81016a0 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8a6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a8103fd0 0x7fe8a8196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8a6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a8103fd0 0x7fe8a8196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50004/0 (socket says 172.21.15.151:50004) 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8a6ffd700 1 -- 172.21.15.151:0/3349250935 learned_addr learned my addr 172.21.15.151:0/3349250935 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8a77fe700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a81016a0 0x7fe8a81963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8a6ffd700 1 -- 172.21.15.151:0/3349250935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a81016a0 msgr2=0x7fe8a81963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8a6ffd700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a81016a0 0x7fe8a81963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8a6ffd700 1 -- 172.21.15.151:0/3349250935 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898009d00 con 0x7fe8a8103fd0 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8a6ffd700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a8103fd0 0x7fe8a8196930 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00d900 tx=0x7fe89c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8a4ff9700 1 -- 172.21.15.151:0/3349250935 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89c0041f0 con 0x7fe8a8103fd0 2024-04-16T02:37:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.768+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a819bb40 con 0x7fe8a8103fd0 2024-04-16T02:37:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.769+0000 7fe8a4ff9700 1 -- 172.21.15.151:0/3349250935 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe89c004ce0 con 0x7fe8a8103fd0 2024-04-16T02:37:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.769+0000 7fe8a4ff9700 1 -- 172.21.15.151:0/3349250935 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89c00b750 con 0x7fe8a8103fd0 2024-04-16T02:37:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.769+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a819c090 con 0x7fe8a8103fd0 2024-04-16T02:37:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.769+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a8190580 con 0x7fe8a8103fd0 2024-04-16T02:37:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.770+0000 7fe8a4ff9700 1 -- 172.21.15.151:0/3349250935 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 32) v1 ==== 92702+0+0 (secure 0 0 0) 0x7fe89c004e50 con 0x7fe8a8103fd0 2024-04-16T02:37:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.770+0000 7fe8a4ff9700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe89006ea80 0x7fe890070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:37:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.771+0000 7fe8a4ff9700 1 -- 172.21.15.151:0/3349250935 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fe89c08e9a0 con 0x7fe8a8103fd0 2024-04-16T02:37:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.771+0000 7fe8a77fe700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe89006ea80 0x7fe890070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:37:30.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.771+0000 7fe8a77fe700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe89006ea80 0x7fe890070f40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe898000f80 tx=0x7fe8980060c0 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:37:30.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.775+0000 7fe8a4ff9700 1 -- 172.21.15.151:0/3349250935 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fe89c05b610 con 0x7fe8a8103fd0 2024-04-16T02:37:31.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.998+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8a80611d0 con 0x7fe89006ea80 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:30.999+0000 7fe8a4ff9700 1 -- 172.21.15.151:0/3349250935 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fe8a80611d0 con 0x7fe89006ea80 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr" 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "2/2 daemons upgraded", 2024-04-16T02:37:31.002 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mgr daemons", 2024-04-16T02:37:31.003 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:37:31.003 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.001+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe89006ea80 msgr2=0x7fe890070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.001+0000 7fe8ae03c700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe89006ea80 0x7fe890070f40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe898000f80 tx=0x7fe8980060c0 comp rx=0 tx=0).stop 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.001+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a8103fd0 msgr2=0x7fe8a8196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.001+0000 7fe8ae03c700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a8103fd0 0x7fe8a8196930 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00d900 tx=0x7fe89c00dc10 comp rx=0 tx=0).stop 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.001+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 shutdown_connections 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.001+0000 7fe8ae03c700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe89006ea80 0x7fe890070f40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.001+0000 7fe8ae03c700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a81016a0 0x7fe8a81963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.001+0000 7fe8ae03c700 1 --2- 172.21.15.151:0/3349250935 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8a8103fd0 0x7fe8a8196930 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.001+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 >> 172.21.15.151:0/3349250935 conn(0x7fe8a80fb0b0 msgr2=0x7fe8a80fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:37:31.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.002+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 shutdown_connections 2024-04-16T02:37:31.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:37:31.002+0000 7fe8ae03c700 1 -- 172.21.15.151:0/3349250935 wait complete. 2024-04-16T02:37:31.050 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:30 smithi153 ceph-mon[36747]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:37:31.050 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:30 smithi153 ceph-mon[36747]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:37:31.050 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:30 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/342552268' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:31.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:30 smithi151 ceph-mon[29003]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:37:31.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:30 smithi151 ceph-mon[29003]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:37:31.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:30 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/342552268' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:32.236 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:31 smithi151 ceph-mon[29003]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:37:32.237 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:31 smithi151 ceph-mon[29003]: pgmap v23: 65 pgs: 65 active+clean; 280 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 829 KiB/s rd, 814 KiB/s wr, 248 op/s 2024-04-16T02:37:32.237 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:31 smithi151 ceph-mon[29003]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:37:32.237 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:31 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:32.237 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:31 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:32.237 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:31 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:37:32.267 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:31 smithi153 ceph-mon[36747]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:37:32.267 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:31 smithi153 ceph-mon[36747]: pgmap v23: 65 pgs: 65 active+clean; 280 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 829 KiB/s rd, 814 KiB/s wr, 248 op/s 2024-04-16T02:37:32.267 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:31 smithi153 ceph-mon[36747]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:37:32.267 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:31 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:32.267 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:31 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:32.267 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:31 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:37:34.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:33 smithi151 ceph-mon[29003]: pgmap v24: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 372 op/s 2024-04-16T02:37:34.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:33 smithi153 ceph-mon[36747]: pgmap v24: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 372 op/s 2024-04-16T02:37:35.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:35 smithi153 ceph-mon[36747]: pgmap v25: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 753 KiB/s rd, 759 KiB/s wr, 251 op/s 2024-04-16T02:37:35.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:35.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:35.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:37:35.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:35 smithi151 ceph-mon[29003]: pgmap v25: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 753 KiB/s rd, 759 KiB/s wr, 251 op/s 2024-04-16T02:37:35.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:35.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:35.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:37:36.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:36 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:36.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:36 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:36 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:36 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:38.137 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:37 smithi153 ceph-mon[36747]: pgmap v26: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 754 KiB/s rd, 759 KiB/s wr, 255 op/s 2024-04-16T02:37:38.137 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:37 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:38.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:37 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:38.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:37 smithi151 ceph-mon[29003]: pgmap v26: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 754 KiB/s rd, 759 KiB/s wr, 255 op/s 2024-04-16T02:37:38.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:37 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:38.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:37 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:39.218 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:38 smithi153 ceph-mon[36747]: Standby manager daemon smithi153.esbtzf restarted 2024-04-16T02:37:39.219 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:38 smithi153 ceph-mon[36747]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:37:39.219 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:38 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/2518200516' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:37:39.219 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:38 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/2518200516' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:37:39.219 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:38 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/2518200516' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:37:39.219 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:38 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/2518200516' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:37:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:38 smithi151 ceph-mon[29003]: Standby manager daemon smithi153.esbtzf restarted 2024-04-16T02:37:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:38 smithi151 ceph-mon[29003]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:37:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:38 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/2518200516' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:37:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:38 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/2518200516' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:37:39.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:38 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/2518200516' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:37:39.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:38 smithi151 ceph-mon[29003]: from='mgr.? 172.21.15.153:0/2518200516' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: pgmap v27: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 747 KiB/s rd, 751 KiB/s wr, 227 op/s 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: mgrmap e33: smithi151.uqqame(active, since 48s), standbys: smithi153.esbtzf 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.098 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi151.uqqame"}]: dispatch 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi151.uqqame"}]': finished 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi153.esbtzf"}]: dispatch 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi153.esbtzf"}]': finished 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.100 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.100 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: pgmap v27: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 747 KiB/s rd, 751 KiB/s wr, 227 op/s 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: mgrmap e33: smithi151.uqqame(active, since 48s), standbys: smithi153.esbtzf 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi151.uqqame"}]: dispatch 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi151.uqqame"}]': finished 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi153.esbtzf"}]: dispatch 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi153.esbtzf"}]': finished 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:37:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:41.101 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:40 smithi151 ceph-mon[29003]: Upgrade: Setting container_image for all mgr 2024-04-16T02:37:41.101 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:40 smithi151 ceph-mon[29003]: Upgrade: Setting container_image for all rgw 2024-04-16T02:37:41.101 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:40 smithi151 ceph-mon[29003]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T02:37:41.102 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:40 smithi151 ceph-mon[29003]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-16T02:37:41.102 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:40 smithi151 ceph-mon[29003]: Upgrade: Setting container_image for all iscsi 2024-04-16T02:37:41.102 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:40 smithi151 ceph-mon[29003]: Upgrade: Setting container_image for all nfs 2024-04-16T02:37:41.102 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:40 smithi151 ceph-mon[29003]: Upgrade: Setting container_image for all nvmeof 2024-04-16T02:37:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:40 smithi153 ceph-mon[36747]: Upgrade: Setting container_image for all mgr 2024-04-16T02:37:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:40 smithi153 ceph-mon[36747]: Upgrade: Setting container_image for all rgw 2024-04-16T02:37:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:40 smithi153 ceph-mon[36747]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T02:37:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:40 smithi153 ceph-mon[36747]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-16T02:37:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:40 smithi153 ceph-mon[36747]: Upgrade: Setting container_image for all iscsi 2024-04-16T02:37:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:40 smithi153 ceph-mon[36747]: Upgrade: Setting container_image for all nfs 2024-04-16T02:37:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:40 smithi153 ceph-mon[36747]: Upgrade: Setting container_image for all nvmeof 2024-04-16T02:37:41.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:41 smithi151 ceph-mon[29003]: pgmap v28: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 342 KiB/s rd, 349 KiB/s wr, 132 op/s 2024-04-16T02:37:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:41 smithi153 ceph-mon[36747]: pgmap v28: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 342 KiB/s rd, 349 KiB/s wr, 132 op/s 2024-04-16T02:37:43.128 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:42 smithi151 ceph-mon[29003]: Upgrade: Updating node-exporter.smithi151 (1/2) 2024-04-16T02:37:43.128 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:42 smithi151 ceph-mon[29003]: Deploying daemon node-exporter.smithi151 on smithi151 2024-04-16T02:37:43.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:42 smithi153 ceph-mon[36747]: Upgrade: Updating node-exporter.smithi151 (1/2) 2024-04-16T02:37:43.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:42 smithi153 ceph-mon[36747]: Deploying daemon node-exporter.smithi151 on smithi151 2024-04-16T02:37:44.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:43 smithi153 ceph-mon[36747]: pgmap v29: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 349 KiB/s wr, 134 op/s 2024-04-16T02:37:44.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:43 smithi151 ceph-mon[29003]: pgmap v29: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 349 KiB/s wr, 134 op/s 2024-04-16T02:37:45.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:44 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:45.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:44 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:45.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:44 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:45.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:44 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:46.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:45 smithi151 ceph-mon[29003]: pgmap v30: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 5.2 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-04-16T02:37:46.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:45 smithi153 ceph-mon[36747]: pgmap v30: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 5.2 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-04-16T02:37:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:47 smithi151 ceph-mon[29003]: pgmap v31: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-04-16T02:37:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:47 smithi151 ceph-mon[29003]: Upgrade: Updating node-exporter.smithi153 (2/2) 2024-04-16T02:37:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:47 smithi151 ceph-mon[29003]: Deploying daemon node-exporter.smithi153 on smithi153 2024-04-16T02:37:48.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:47 smithi153 ceph-mon[36747]: pgmap v31: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-04-16T02:37:48.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:47 smithi153 ceph-mon[36747]: Upgrade: Updating node-exporter.smithi153 (2/2) 2024-04-16T02:37:48.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:47 smithi153 ceph-mon[36747]: Deploying daemon node-exporter.smithi153 on smithi153 2024-04-16T02:37:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:50 smithi151 ceph-mon[29003]: pgmap v32: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-16T02:37:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:50 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:50 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:50 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:37:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:50 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:50 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:50 smithi153 ceph-mon[36747]: pgmap v32: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-16T02:37:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:50 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:50 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:50 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:37:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:50 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:50 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:51.349 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:37:51.431 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:37:52.304 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:52 smithi153 ceph-mon[36747]: pgmap v33: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T02:37:52.315 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:52 smithi151 ceph-mon[29003]: pgmap v33: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T02:37:54.080 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:53 smithi151 ceph-mon[29003]: pgmap v34: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-16T02:37:54.080 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:53 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:54.081 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:53 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:53 smithi153 ceph-mon[36747]: pgmap v34: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-16T02:37:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:53 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:53 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:55.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:54 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:55.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:54 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:55.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:54 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:55.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:54 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:54 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:54 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:54 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:54 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:56.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:55 smithi153 ceph-mon[36747]: pgmap v35: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:37:56.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:55 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:56.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:55 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:56.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:55 smithi151 ceph-mon[29003]: pgmap v35: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:37:56.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:55 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:56.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:55 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:57 smithi151 ceph-mon[29003]: pgmap v36: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:37:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:57 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:57 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:57 smithi153 ceph-mon[36747]: pgmap v36: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:37:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:57 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:37:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:57 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: pgmap v37: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:38:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:37:59 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: pgmap v37: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:00.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:00.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:00.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:00.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:37:59 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:01.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.111+0000 7f313fb17700 1 -- 172.21.15.151:0/2917408905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138100bc0 msgr2=0x7f3138100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.111+0000 7f313fb17700 1 --2- 172.21.15.151:0/2917408905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138100bc0 0x7f3138100fe0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3134009910 tx=0x7f3134009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:01.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.111+0000 7f313fb17700 1 -- 172.21.15.151:0/2917408905 shutdown_connections 2024-04-16T02:38:01.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.111+0000 7f313fb17700 1 --2- 172.21.15.151:0/2917408905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3138101dc0 0x7f3138102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.111+0000 7f313fb17700 1 --2- 172.21.15.151:0/2917408905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138100bc0 0x7f3138100fe0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.111+0000 7f313fb17700 1 -- 172.21.15.151:0/2917408905 >> 172.21.15.151:0/2917408905 conn(0x7f31380fc140 msgr2=0x7f31380fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.111+0000 7f313fb17700 1 -- 172.21.15.151:0/2917408905 shutdown_connections 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.111+0000 7f313fb17700 1 -- 172.21.15.151:0/2917408905 wait complete. 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313fb17700 1 Processor -- start 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313fb17700 1 -- start start 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313fb17700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3138100bc0 0x7f3138196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313fb17700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138101dc0 0x7f31381968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313fb17700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3138196ef0 con 0x7f3138101dc0 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313fb17700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3138197030 con 0x7f3138100bc0 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313d0b2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138101dc0 0x7f31381968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313d0b2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138101dc0 0x7f31381968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53892/0 (socket says 172.21.15.151:53892) 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313d0b2700 1 -- 172.21.15.151:0/3990179306 learned_addr learned my addr 172.21.15.151:0/3990179306 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313d8b3700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3138100bc0 0x7f3138196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313d0b2700 1 -- 172.21.15.151:0/3990179306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3138100bc0 msgr2=0x7f3138196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313d0b2700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3138100bc0 0x7f3138196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313d0b2700 1 -- 172.21.15.151:0/3990179306 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31340095f0 con 0x7f3138101dc0 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313d8b3700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3138100bc0 0x7f3138196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.112+0000 7f313d0b2700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138101dc0 0x7f31381968d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f312800c8a0 tx=0x7f312800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.113+0000 7f312effd700 1 -- 172.21.15.151:0/3990179306 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f312800cea0 con 0x7f3138101dc0 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.113+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313819bae0 con 0x7f3138101dc0 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.113+0000 7f312effd700 1 -- 172.21.15.151:0/3990179306 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3128004830 con 0x7f3138101dc0 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.113+0000 7f312effd700 1 -- 172.21.15.151:0/3990179306 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31280055b0 con 0x7f3138101dc0 2024-04-16T02:38:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.113+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313819c030 con 0x7f3138101dc0 2024-04-16T02:38:01.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.114+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3138109720 con 0x7f3138101dc0 2024-04-16T02:38:01.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.114+0000 7f312effd700 1 -- 172.21.15.151:0/3990179306 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 33) v1 ==== 92702+0+0 (secure 0 0 0) 0x7f3128007ad0 con 0x7f3138101dc0 2024-04-16T02:38:01.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.115+0000 7f312effd700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f312406ea80 0x7f3124070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:01.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.115+0000 7f313d8b3700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f312406ea80 0x7f3124070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:01.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.115+0000 7f312effd700 1 -- 172.21.15.151:0/3990179306 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f312808e870 con 0x7f3138101dc0 2024-04-16T02:38:01.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.115+0000 7f313d8b3700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f312406ea80 0x7f3124070f40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3134006010 tx=0x7f3134005920 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:01.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.120+0000 7f312effd700 1 -- 172.21.15.151:0/3990179306 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f312805b460 con 0x7f3138101dc0 2024-04-16T02:38:01.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.358+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3138106710 con 0x7f312406ea80 2024-04-16T02:38:01.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.359+0000 7f312effd700 1 -- 172.21.15.151:0/3990179306 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f3138106710 con 0x7f312406ea80 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f312406ea80 msgr2=0x7f3124070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f312406ea80 0x7f3124070f40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3134006010 tx=0x7f3134005920 comp rx=0 tx=0).stop 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138101dc0 msgr2=0x7f31381968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138101dc0 0x7f31381968d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f312800c8a0 tx=0x7f312800cc60 comp rx=0 tx=0).stop 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 shutdown_connections 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f312406ea80 0x7f3124070f40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3138100bc0 0x7f3138196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 --2- 172.21.15.151:0/3990179306 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3138101dc0 0x7f31381968d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 >> 172.21.15.151:0/3990179306 conn(0x7f31380fc140 msgr2=0x7f3138104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 shutdown_connections 2024-04-16T02:38:01.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.361+0000 7f313fb17700 1 -- 172.21.15.151:0/3990179306 wait complete. 2024-04-16T02:38:01.375 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:38:01.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.477+0000 7fb6401e4700 1 -- 172.21.15.151:0/2605021500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638103f70 msgr2=0x7fb638106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.477+0000 7fb6401e4700 1 --2- 172.21.15.151:0/2605021500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638103f70 0x7fb638106360 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb628009910 tx=0x7fb628009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:01.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.477+0000 7fb6401e4700 1 -- 172.21.15.151:0/2605021500 shutdown_connections 2024-04-16T02:38:01.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.477+0000 7fb6401e4700 1 --2- 172.21.15.151:0/2605021500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638103f70 0x7fb638106360 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.477+0000 7fb6401e4700 1 --2- 172.21.15.151:0/2605021500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb638101640 0x7fb638103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.477+0000 7fb6401e4700 1 -- 172.21.15.151:0/2605021500 >> 172.21.15.151:0/2605021500 conn(0x7fb6380fb050 msgr2=0x7fb6380fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:01.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.478+0000 7fb6401e4700 1 -- 172.21.15.151:0/2605021500 shutdown_connections 2024-04-16T02:38:01.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.478+0000 7fb6401e4700 1 -- 172.21.15.151:0/2605021500 wait complete. 2024-04-16T02:38:01.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.478+0000 7fb6401e4700 1 Processor -- start 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.478+0000 7fb6401e4700 1 -- start start 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb6401e4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638101640 0x7fb638100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb6401e4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb638103f70 0x7fb6380ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb6401e4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb638199f70 con 0x7fb638101640 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb6401e4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63819a0b0 con 0x7fb638103f70 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb63df80700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638101640 0x7fb638100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb63df80700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638101640 0x7fb638100fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53910/0 (socket says 172.21.15.151:53910) 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb63df80700 1 -- 172.21.15.151:0/2789273780 learned_addr learned my addr 172.21.15.151:0/2789273780 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb63d77f700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb638103f70 0x7fb6380ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb63df80700 1 -- 172.21.15.151:0/2789273780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb638103f70 msgr2=0x7fb6380ff5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb63df80700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb638103f70 0x7fb6380ff5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb63df80700 1 -- 172.21.15.151:0/2789273780 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6280095f0 con 0x7fb638101640 2024-04-16T02:38:01.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.479+0000 7fb63df80700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638101640 0x7fb638100fa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb63400d8d0 tx=0x7fb63400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:01.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.480+0000 7fb62effd700 1 -- 172.21.15.151:0/2789273780 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb634009940 con 0x7fb638101640 2024-04-16T02:38:01.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.480+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6380ffb30 con 0x7fb638101640 2024-04-16T02:38:01.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.480+0000 7fb62effd700 1 -- 172.21.15.151:0/2789273780 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb634010460 con 0x7fb638101640 2024-04-16T02:38:01.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.480+0000 7fb62effd700 1 -- 172.21.15.151:0/2789273780 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63400f5f0 con 0x7fb638101640 2024-04-16T02:38:01.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.480+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb638100050 con 0x7fb638101640 2024-04-16T02:38:01.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.481+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb638190520 con 0x7fb638101640 2024-04-16T02:38:01.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.482+0000 7fb62effd700 1 -- 172.21.15.151:0/2789273780 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 33) v1 ==== 92702+0+0 (secure 0 0 0) 0x7fb6340105d0 con 0x7fb638101640 2024-04-16T02:38:01.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.483+0000 7fb62effd700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb62406ead0 0x7fb624070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:01.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.483+0000 7fb62effd700 1 -- 172.21.15.151:0/2789273780 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fb63408eb20 con 0x7fb638101640 2024-04-16T02:38:01.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.483+0000 7fb63d77f700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb62406ead0 0x7fb624070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:01.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.484+0000 7fb63d77f700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb62406ead0 0x7fb624070f90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb628006010 tx=0x7fb62800b580 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:01.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.487+0000 7fb62effd700 1 -- 172.21.15.151:0/2789273780 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fb63405b820 con 0x7fb638101640 2024-04-16T02:38:01.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.712+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6381003a0 con 0x7fb62406ead0 2024-04-16T02:38:01.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.714+0000 7fb62effd700 1 -- 172.21.15.151:0/2789273780 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fb6381003a0 con 0x7fb62406ead0 2024-04-16T02:38:01.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb62406ead0 msgr2=0x7fb624070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb62406ead0 0x7fb624070f90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb628006010 tx=0x7fb62800b580 comp rx=0 tx=0).stop 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638101640 msgr2=0x7fb638100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638101640 0x7fb638100fa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb63400d8d0 tx=0x7fb63400dc90 comp rx=0 tx=0).stop 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 shutdown_connections 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb62406ead0 0x7fb624070f90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb638101640 0x7fb638100fa0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 --2- 172.21.15.151:0/2789273780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb638103f70 0x7fb6380ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 >> 172.21.15.151:0/2789273780 conn(0x7fb6380fb050 msgr2=0x7fb6380fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 shutdown_connections 2024-04-16T02:38:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.716+0000 7fb6401e4700 1 -- 172.21.15.151:0/2789273780 wait complete. 2024-04-16T02:38:01.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.825+0000 7f6d572d2700 1 -- 172.21.15.151:0/2164567111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d500ffd50 msgr2=0x7f6d501001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.825+0000 7f6d572d2700 1 --2- 172.21.15.151:0/2164567111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d500ffd50 0x7f6d501001b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009910 tx=0x7f6d40009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:01.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.825+0000 7f6d572d2700 1 -- 172.21.15.151:0/2164567111 shutdown_connections 2024-04-16T02:38:01.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.825+0000 7f6d572d2700 1 --2- 172.21.15.151:0/2164567111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d500ffd50 0x7f6d501001b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.825+0000 7f6d572d2700 1 --2- 172.21.15.151:0/2164567111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d501010a0 0x7f6d501014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.825+0000 7f6d572d2700 1 -- 172.21.15.151:0/2164567111 >> 172.21.15.151:0/2164567111 conn(0x7f6d500fa7c0 msgr2=0x7f6d500fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d572d2700 1 -- 172.21.15.151:0/2164567111 shutdown_connections 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d572d2700 1 -- 172.21.15.151:0/2164567111 wait complete. 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d572d2700 1 Processor -- start 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d572d2700 1 -- start start 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d572d2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d500ffd50 0x7f6d500ff7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d572d2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d501010a0 0x7f6d5019b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d572d2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5019b680 con 0x7f6d501010a0 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d572d2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5019b7f0 con 0x7f6d500ffd50 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d5486d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d501010a0 0x7f6d5019b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d5486d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d501010a0 0x7f6d5019b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53922/0 (socket says 172.21.15.151:53922) 2024-04-16T02:38:01.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.826+0000 7f6d5486d700 1 -- 172.21.15.151:0/3739741292 learned_addr learned my addr 172.21.15.151:0/3739741292 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d5506e700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d500ffd50 0x7f6d500ff7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d5486d700 1 -- 172.21.15.151:0/3739741292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d500ffd50 msgr2=0x7f6d500ff7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d5486d700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d500ffd50 0x7f6d500ff7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d5486d700 1 -- 172.21.15.151:0/3739741292 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d400095f0 con 0x7f6d501010a0 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d5506e700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d500ffd50 0x7f6d500ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d5486d700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d501010a0 0x7f6d5019b0b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6d40006010 tx=0x7f6d4000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d467fc700 1 -- 172.21.15.151:0/3739741292 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d4001d070 con 0x7f6d501010a0 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d467fc700 1 -- 172.21.15.151:0/3739741292 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6d4000f460 con 0x7f6d501010a0 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5019ba70 con 0x7f6d501010a0 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d467fc700 1 -- 172.21.15.151:0/3739741292 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d40017630 con 0x7f6d501010a0 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.827+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5019bf60 con 0x7f6d501010a0 2024-04-16T02:38:01.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.828+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d50108620 con 0x7f6d501010a0 2024-04-16T02:38:01.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.829+0000 7f6d467fc700 1 -- 172.21.15.151:0/3739741292 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 33) v1 ==== 92702+0+0 (secure 0 0 0) 0x7f6d4000f5d0 con 0x7f6d501010a0 2024-04-16T02:38:01.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.829+0000 7f6d467fc700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6d3c06ea80 0x7f6d3c070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:01.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.829+0000 7f6d467fc700 1 -- 172.21.15.151:0/3739741292 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f6d40090020 con 0x7f6d501010a0 2024-04-16T02:38:01.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.829+0000 7f6d5506e700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6d3c06ea80 0x7f6d3c070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:01.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.830+0000 7f6d5506e700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6d3c06ea80 0x7f6d3c070f40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c005fd0 tx=0x7f6d4c005f00 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:01.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:01.834+0000 7f6d467fc700 1 -- 172.21.15.151:0/3739741292 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f6d4005cc10 con 0x7f6d501010a0 2024-04-16T02:38:02.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.056+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d501056b0 con 0x7f6d3c06ea80 2024-04-16T02:38:02.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.067+0000 7f6d467fc700 1 -- 172.21.15.151:0/3739741292 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6d501056b0 con 0x7f6d3c06ea80 2024-04-16T02:38:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:38:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (9m) 7s ago 10m 24.9M - 0.25.0 c8568f914cd2 9f7a3dfb7aa4 2024-04-16T02:38:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (11m) 7s ago 11m 7977k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:38:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (10m) 9s ago 10m 7884k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:38:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (11m) 7s ago 11m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (10m) 9s ago 10m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (9m) 7s ago 10m 82.4M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (5m) 7s ago 5m 121M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (5m) 7s ago 5m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (5m) 9s ago 5m 58.2M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (5m) 9s ago 5m 67.2M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (96s) 7s ago 12m 496M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (31s) 9s ago 9m 448M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (12m) 7s ago 12m 59.3M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (9m) 9s ago 9m 45.9M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (18s) 7s ago 11m 3661k - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (13s) 9s ago 9m 3678k - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:38:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (9m) 7s ago 9m 222M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:38:02.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (8m) 7s ago 8m 248M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:38:02.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (8m) 7s ago 8m 194M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:38:02.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (7m) 9s ago 7m 279M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:38:02.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (7m) 9s ago 7m 228M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:38:02.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (6m) 9s ago 6m 243M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:38:02.072 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (37s) 7s ago 10m 57.2M - 2.43.0 a07b618ecd1d b0477f4b8630 2024-04-16T02:38:02.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6d3c06ea80 msgr2=0x7f6d3c070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6d3c06ea80 0x7f6d3c070f40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c005fd0 tx=0x7f6d4c005f00 comp rx=0 tx=0).stop 2024-04-16T02:38:02.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d501010a0 msgr2=0x7f6d5019b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d501010a0 0x7f6d5019b0b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6d40006010 tx=0x7f6d4000be20 comp rx=0 tx=0).stop 2024-04-16T02:38:02.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 shutdown_connections 2024-04-16T02:38:02.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6d3c06ea80 0x7f6d3c070f40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d500ffd50 0x7f6d500ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 --2- 172.21.15.151:0/3739741292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d501010a0 0x7f6d5019b0b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 >> 172.21.15.151:0/3739741292 conn(0x7f6d500fa7c0 msgr2=0x7f6d50103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:02.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 shutdown_connections 2024-04-16T02:38:02.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.069+0000 7f6d572d2700 1 -- 172.21.15.151:0/3739741292 wait complete. 2024-04-16T02:38:02.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.174+0000 7f76d5740700 1 -- 172.21.15.151:0/2366105063 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 msgr2=0x7f76d0102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.174+0000 7f76d5740700 1 --2- 172.21.15.151:0/2366105063 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 0x7f76d0102260 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009910 tx=0x7f76c4009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:02.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.174+0000 7f76d5740700 1 -- 172.21.15.151:0/2366105063 shutdown_connections 2024-04-16T02:38:02.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.174+0000 7f76d5740700 1 --2- 172.21.15.151:0/2366105063 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 0x7f76d0102260 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.174+0000 7f76d5740700 1 --2- 172.21.15.151:0/2366105063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d0100c00 0x7f76d0101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.174+0000 7f76d5740700 1 -- 172.21.15.151:0/2366105063 >> 172.21.15.151:0/2366105063 conn(0x7f76d00fc180 msgr2=0x7f76d00fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:02.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.174+0000 7f76d5740700 1 -- 172.21.15.151:0/2366105063 shutdown_connections 2024-04-16T02:38:02.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.174+0000 7f76d5740700 1 -- 172.21.15.151:0/2366105063 wait complete. 2024-04-16T02:38:02.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.175+0000 7f76d5740700 1 Processor -- start 2024-04-16T02:38:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.175+0000 7f76d5740700 1 -- start start 2024-04-16T02:38:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.175+0000 7f76d5740700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d0100c00 0x7f76d0196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.175+0000 7f76d5740700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 0x7f76d01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.175+0000 7f76d5740700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0196ef0 con 0x7f76d0101e00 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.175+0000 7f76d5740700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0197030 con 0x7f76d0100c00 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.176+0000 7f76ce7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 0x7f76d01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.176+0000 7f76ce7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 0x7f76d01968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53930/0 (socket says 172.21.15.151:53930) 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.176+0000 7f76ce7fc700 1 -- 172.21.15.151:0/1974313094 learned_addr learned my addr 172.21.15.151:0/1974313094 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.176+0000 7f76ceffd700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d0100c00 0x7f76d0196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.176+0000 7f76ce7fc700 1 -- 172.21.15.151:0/1974313094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d0100c00 msgr2=0x7f76d0196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.176+0000 7f76ce7fc700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d0100c00 0x7f76d0196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.176+0000 7f76ce7fc700 1 -- 172.21.15.151:0/1974313094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c40095f0 con 0x7f76d0101e00 2024-04-16T02:38:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.176+0000 7f76ce7fc700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 0x7f76d01968d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f76c400bd70 tx=0x7f76c400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:02.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.176+0000 7f76b7fff700 1 -- 172.21.15.151:0/1974313094 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76c401d070 con 0x7f76d0101e00 2024-04-16T02:38:02.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.177+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d019ba80 con 0x7f76d0101e00 2024-04-16T02:38:02.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.177+0000 7f76b7fff700 1 -- 172.21.15.151:0/1974313094 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f76c400f460 con 0x7f76d0101e00 2024-04-16T02:38:02.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.177+0000 7f76b7fff700 1 -- 172.21.15.151:0/1974313094 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76c4021620 con 0x7f76d0101e00 2024-04-16T02:38:02.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.177+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d019bf70 con 0x7f76d0101e00 2024-04-16T02:38:02.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.178+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d0109760 con 0x7f76d0101e00 2024-04-16T02:38:02.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.179+0000 7f76b7fff700 1 -- 172.21.15.151:0/1974313094 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 33) v1 ==== 92702+0+0 (secure 0 0 0) 0x7f76c400fa40 con 0x7f76d0101e00 2024-04-16T02:38:02.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.179+0000 7f76b7fff700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f76b806eac0 0x7f76b8070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:02.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.179+0000 7f76b7fff700 1 -- 172.21.15.151:0/1974313094 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f76c408fea0 con 0x7f76d0101e00 2024-04-16T02:38:02.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.179+0000 7f76ceffd700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f76b806eac0 0x7f76b8070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:02.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.180+0000 7f76ceffd700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f76b806eac0 0x7f76b8070f80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f76c0005950 tx=0x7f76c00058e0 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:02.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.184+0000 7f76b7fff700 1 -- 172.21.15.151:0/1974313094 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f76c405ca90 con 0x7f76d0101e00 2024-04-16T02:38:02.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:01 smithi151 ceph-mon[29003]: pgmap v38: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:02.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:01 smithi153 ceph-mon[36747]: pgmap v38: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:02.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.479+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f76d0109920 con 0x7f76d0101e00 2024-04-16T02:38:02.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.480+0000 7f76b7fff700 1 -- 172.21.15.151:0/1974313094 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f76c405c620 con 0x7f76d0101e00 2024-04-16T02:38:02.484 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:38:02.484 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:38:02.484 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:38:02.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:38:02.484 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:38:02.485 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:38:02.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:38:02.485 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:38:02.485 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:38:02.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:38:02.485 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:38:02.485 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:38:02.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:38:02.486 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:38:02.486 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-16T02:38:02.486 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:38:02.486 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:38:02.486 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:38:02.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.482+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f76b806eac0 msgr2=0x7f76b8070f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.482+0000 7f76d5740700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f76b806eac0 0x7f76b8070f80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f76c0005950 tx=0x7f76c00058e0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.482+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 msgr2=0x7f76d01968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.482+0000 7f76d5740700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 0x7f76d01968d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f76c400bd70 tx=0x7f76c400be50 comp rx=0 tx=0).stop 2024-04-16T02:38:02.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.482+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 shutdown_connections 2024-04-16T02:38:02.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.482+0000 7f76d5740700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f76b806eac0 0x7f76b8070f80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.482+0000 7f76d5740700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d0100c00 0x7f76d0196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.482+0000 7f76d5740700 1 --2- 172.21.15.151:0/1974313094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76d0101e00 0x7f76d01968d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.482+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 >> 172.21.15.151:0/1974313094 conn(0x7f76d00fc180 msgr2=0x7f76d0105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:02.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.483+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 shutdown_connections 2024-04-16T02:38:02.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.483+0000 7f76d5740700 1 -- 172.21.15.151:0/1974313094 wait complete. 2024-04-16T02:38:02.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.592+0000 7f325ba32700 1 -- 172.21.15.151:0/2115803310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3254100cb0 msgr2=0x7f3254101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.592+0000 7f325ba32700 1 --2- 172.21.15.151:0/2115803310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3254100cb0 0x7f3254101110 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3244009910 tx=0x7f3244009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:02.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.592+0000 7f325ba32700 1 -- 172.21.15.151:0/2115803310 shutdown_connections 2024-04-16T02:38:02.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.592+0000 7f325ba32700 1 --2- 172.21.15.151:0/2115803310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3254100cb0 0x7f3254101110 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.592+0000 7f325ba32700 1 --2- 172.21.15.151:0/2115803310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32540ffab0 0x7f32540ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.592+0000 7f325ba32700 1 -- 172.21.15.151:0/2115803310 >> 172.21.15.151:0/2115803310 conn(0x7f32540fb030 msgr2=0x7f32540fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:02.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.592+0000 7f325ba32700 1 -- 172.21.15.151:0/2115803310 shutdown_connections 2024-04-16T02:38:02.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.592+0000 7f325ba32700 1 -- 172.21.15.151:0/2115803310 wait complete. 2024-04-16T02:38:02.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f325ba32700 1 Processor -- start 2024-04-16T02:38:02.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f325ba32700 1 -- start start 2024-04-16T02:38:02.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f325ba32700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32540ffab0 0x7f32541963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:02.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f325ba32700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3254100cb0 0x7f3254196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:02.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f325ba32700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3254196f40 con 0x7f32540ffab0 2024-04-16T02:38:02.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f325ba32700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3254197080 con 0x7f3254100cb0 2024-04-16T02:38:02.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f32597ce700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32540ffab0 0x7f32541963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:02.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f32597ce700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32540ffab0 0x7f32541963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53942/0 (socket says 172.21.15.151:53942) 2024-04-16T02:38:02.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f32597ce700 1 -- 172.21.15.151:0/2668585861 learned_addr learned my addr 172.21.15.151:0/2668585861 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:02.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f3258fcd700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3254100cb0 0x7f3254196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:02.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f32597ce700 1 -- 172.21.15.151:0/2668585861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3254100cb0 msgr2=0x7f3254196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f32597ce700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3254100cb0 0x7f3254196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.593+0000 7f32597ce700 1 -- 172.21.15.151:0/2668585861 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32440095f0 con 0x7f32540ffab0 2024-04-16T02:38:02.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.594+0000 7f32597ce700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32540ffab0 0x7f32541963e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f325000b480 tx=0x7f325000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:02.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.594+0000 7f324a7fc700 1 -- 172.21.15.151:0/2668585861 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3250011800 con 0x7f32540ffab0 2024-04-16T02:38:02.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.594+0000 7f324a7fc700 1 -- 172.21.15.151:0/2668585861 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3250011e10 con 0x7f32540ffab0 2024-04-16T02:38:02.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.594+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325419bb30 con 0x7f32540ffab0 2024-04-16T02:38:02.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.594+0000 7f324a7fc700 1 -- 172.21.15.151:0/2668585861 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3250010ea0 con 0x7f32540ffab0 2024-04-16T02:38:02.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.594+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325419c080 con 0x7f32540ffab0 2024-04-16T02:38:02.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.595+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32541086b0 con 0x7f32540ffab0 2024-04-16T02:38:02.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.595+0000 7f324a7fc700 1 -- 172.21.15.151:0/2668585861 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 33) v1 ==== 92702+0+0 (secure 0 0 0) 0x7f3250010380 con 0x7f32540ffab0 2024-04-16T02:38:02.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.596+0000 7f324a7fc700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f324006ea80 0x7f3240070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:02.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.596+0000 7f324a7fc700 1 -- 172.21.15.151:0/2668585861 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f325008e590 con 0x7f32540ffab0 2024-04-16T02:38:02.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.596+0000 7f3258fcd700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f324006ea80 0x7f3240070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:02.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.596+0000 7f3258fcd700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f324006ea80 0x7f3240070f40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3244006010 tx=0x7f324400b540 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:02.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.600+0000 7f324a7fc700 1 -- 172.21.15.151:0/2668585861 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f325005b290 con 0x7f32540ffab0 2024-04-16T02:38:02.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.822+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3254068b20 con 0x7f324006ea80 2024-04-16T02:38:02.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.824+0000 7f324a7fc700 1 -- 172.21.15.151:0/2668585861 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f3254068b20 con 0x7f324006ea80 2024-04-16T02:38:02.849 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:38:02.849 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:38:02.850 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:38:02.850 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-16T02:38:02.850 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:38:02.850 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr" 2024-04-16T02:38:02.850 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:38:02.850 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "2/2 daemons upgraded", 2024-04-16T02:38:02.850 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-16T02:38:02.850 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:38:02.850 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:38:02.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f324006ea80 msgr2=0x7f3240070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f324006ea80 0x7f3240070f40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3244006010 tx=0x7f324400b540 comp rx=0 tx=0).stop 2024-04-16T02:38:02.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32540ffab0 msgr2=0x7f32541963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:02.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32540ffab0 0x7f32541963e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f325000b480 tx=0x7f325000b790 comp rx=0 tx=0).stop 2024-04-16T02:38:02.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 shutdown_connections 2024-04-16T02:38:02.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f324006ea80 0x7f3240070f40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32540ffab0 0x7f32541963e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 --2- 172.21.15.151:0/2668585861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3254100cb0 0x7f3254196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:02.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 >> 172.21.15.151:0/2668585861 conn(0x7f32540fb030 msgr2=0x7f3254068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:02.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 shutdown_connections 2024-04-16T02:38:02.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:02.826+0000 7f325ba32700 1 -- 172.21.15.151:0/2668585861 wait complete. 2024-04-16T02:38:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:02 smithi151 ceph-mon[29003]: Upgrade: Updating prometheus.smithi151 2024-04-16T02:38:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:02 smithi151 ceph-mon[29003]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:02 smithi151 ceph-mon[29003]: Deploying daemon prometheus.smithi151 on smithi151 2024-04-16T02:38:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:02 smithi151 ceph-mon[29003]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:02 smithi151 ceph-mon[29003]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:02 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1974313094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:02 smithi153 ceph-mon[36747]: Upgrade: Updating prometheus.smithi151 2024-04-16T02:38:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:02 smithi153 ceph-mon[36747]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:02 smithi153 ceph-mon[36747]: Deploying daemon prometheus.smithi151 on smithi151 2024-04-16T02:38:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:02 smithi153 ceph-mon[36747]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:02 smithi153 ceph-mon[36747]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:02 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1974313094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:04.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:03 smithi151 ceph-mon[29003]: pgmap v39: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:38:04.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:03 smithi151 ceph-mon[29003]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:03 smithi153 ceph-mon[36747]: pgmap v39: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:38:04.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:03 smithi153 ceph-mon[36747]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:05 smithi151 ceph-mon[29003]: pgmap v40: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:05 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:38:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:05 smithi153 ceph-mon[36747]: pgmap v40: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:05 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:38:07.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:07 smithi151 ceph-mon[29003]: pgmap v41: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:38:07.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:07 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:07.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:07 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:07.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:07 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:08.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:07 smithi153 ceph-mon[36747]: pgmap v41: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:38:08.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:07 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:08.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:07 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:08.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:07 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:10.215 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:09 smithi153 ceph-mon[36747]: pgmap v42: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:09 smithi151 ceph-mon[29003]: pgmap v42: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:11 smithi151 ceph-mon[29003]: pgmap v43: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:12.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:11 smithi153 ceph-mon[36747]: pgmap v43: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:13.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:13 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:13.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:13 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:13.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:13 smithi151 ceph-mon[29003]: pgmap v44: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:38:13.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:13 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:13.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:13 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:13.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:13 smithi153 ceph-mon[36747]: pgmap v44: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:38:14.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:14 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:14.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:14 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:14.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:14 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:14.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:14 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:15 smithi153 ceph-mon[36747]: pgmap v45: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:16.096 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:15 smithi151 ceph-mon[29003]: pgmap v45: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:17.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: pgmap v46: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:38:17.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:17.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:17.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:38:17.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:38:17.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:17.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:38:17.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:38:17.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:17.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:17.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:17.998 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:17.998 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:17.998 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:17.998 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:17.998 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:17.998 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:17.998 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:17 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: pgmap v46: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-16T02:38:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:18.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:18.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:18.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:18.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:18.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:18.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:17 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:19 smithi151 ceph-mon[29003]: pgmap v47: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:19 smithi151 ceph-mon[29003]: Upgrade: Updating alertmanager.smithi151 2024-04-16T02:38:20.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:19 smithi151 ceph-mon[29003]: Deploying daemon alertmanager.smithi151 on smithi151 2024-04-16T02:38:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:19 smithi153 ceph-mon[36747]: pgmap v47: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:19 smithi153 ceph-mon[36747]: Upgrade: Updating alertmanager.smithi151 2024-04-16T02:38:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:19 smithi153 ceph-mon[36747]: Deploying daemon alertmanager.smithi151 on smithi151 2024-04-16T02:38:20.961 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:20 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:38:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:20 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:38:21.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:21 smithi151 ceph-mon[29003]: pgmap v48: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:21 smithi153 ceph-mon[36747]: pgmap v48: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-16T02:38:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:23 smithi151 ceph-mon[29003]: pgmap v49: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-16T02:38:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:23 smithi151 ceph-mon[29003]: mgrmap e34: smithi151.uqqame(active, since 92s), standbys: smithi153.esbtzf 2024-04-16T02:38:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:23 smithi153 ceph-mon[36747]: pgmap v49: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-16T02:38:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:23 smithi153 ceph-mon[36747]: mgrmap e34: smithi151.uqqame(active, since 92s), standbys: smithi153.esbtzf 2024-04-16T02:38:25.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:25 smithi151 ceph-mon[29003]: pgmap v50: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-16T02:38:25.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:25.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:25.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:25 smithi153 ceph-mon[36747]: pgmap v50: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-16T02:38:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:27.699 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:27 smithi151 ceph-mon[29003]: pgmap v51: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-16T02:38:27.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:27 smithi153 ceph-mon[36747]: pgmap v51: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-16T02:38:29.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:29 smithi151 ceph-mon[29003]: pgmap v52: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:38:29.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:29 smithi153 ceph-mon[36747]: pgmap v52: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:38:30.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:30 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:30.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:30 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:30.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:30 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:30.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:30 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:31.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:31 smithi151 ceph-mon[29003]: pgmap v53: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:38:31.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:31 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:31.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:31 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:31 smithi153 ceph-mon[36747]: pgmap v53: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-16T02:38:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:31 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:31 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:32.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 -- 172.21.15.151:0/3286668121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07040ffab0 msgr2=0x7f07040ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 --2- 172.21.15.151:0/3286668121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07040ffab0 0x7f07040ffed0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0700009910 tx=0x7f0700009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 -- 172.21.15.151:0/3286668121 shutdown_connections 2024-04-16T02:38:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 --2- 172.21.15.151:0/3286668121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0704100cb0 0x7f0704101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 --2- 172.21.15.151:0/3286668121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07040ffab0 0x7f07040ffed0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 -- 172.21.15.151:0/3286668121 >> 172.21.15.151:0/3286668121 conn(0x7f07040fb030 msgr2=0x7f07040fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 -- 172.21.15.151:0/3286668121 shutdown_connections 2024-04-16T02:38:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 -- 172.21.15.151:0/3286668121 wait complete. 2024-04-16T02:38:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 Processor -- start 2024-04-16T02:38:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.937+0000 7f070b966700 1 -- start start 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f070b966700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040ffab0 0x7f0704198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f070b966700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0704100cb0 0x7f0704198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f070b966700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0704199160 con 0x7f0704100cb0 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f070b966700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07041992a0 con 0x7f07040ffab0 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f0708f01700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0704100cb0 0x7f0704198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f0708f01700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0704100cb0 0x7f0704198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52624/0 (socket says 172.21.15.151:52624) 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f0708f01700 1 -- 172.21.15.151:0/3063738260 learned_addr learned my addr 172.21.15.151:0/3063738260 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f0709702700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040ffab0 0x7f0704198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f0708f01700 1 -- 172.21.15.151:0/3063738260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040ffab0 msgr2=0x7f0704198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f0708f01700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040ffab0 0x7f0704198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f0708f01700 1 -- 172.21.15.151:0/3063738260 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07000095f0 con 0x7f0704100cb0 2024-04-16T02:38:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.938+0000 7f0708f01700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0704100cb0 0x7f0704198b40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f06f400b4b0 tx=0x7f06f400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.939+0000 7f06fa7fc700 1 -- 172.21.15.151:0/3063738260 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06f4011800 con 0x7f0704100cb0 2024-04-16T02:38:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.939+0000 7f06fa7fc700 1 -- 172.21.15.151:0/3063738260 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f06f4011e10 con 0x7f0704100cb0 2024-04-16T02:38:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.939+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070419dd50 con 0x7f0704100cb0 2024-04-16T02:38:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.939+0000 7f06fa7fc700 1 -- 172.21.15.151:0/3063738260 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06f400f2e0 con 0x7f0704100cb0 2024-04-16T02:38:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.939+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070419e320 con 0x7f0704100cb0 2024-04-16T02:38:32.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.940+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0704108690 con 0x7f0704100cb0 2024-04-16T02:38:32.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.941+0000 7f06fa7fc700 1 -- 172.21.15.151:0/3063738260 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f06f4010380 con 0x7f0704100cb0 2024-04-16T02:38:32.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.941+0000 7f06fa7fc700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f06f006e940 0x7f06f0070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:32.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.941+0000 7f06fa7fc700 1 -- 172.21.15.151:0/3063738260 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f06f408e380 con 0x7f0704100cb0 2024-04-16T02:38:32.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.941+0000 7f0709702700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f06f006e940 0x7f06f0070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:32.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.942+0000 7f0709702700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f06f006e940 0x7f06f0070e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0700006010 tx=0x7f070000b540 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:32.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:32.946+0000 7f06fa7fc700 1 -- 172.21.15.151:0/3063738260 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f06f405b210 con 0x7f0704100cb0 2024-04-16T02:38:33.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.175+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0704068b20 con 0x7f06f006e940 2024-04-16T02:38:33.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.176+0000 7f06fa7fc700 1 -- 172.21.15.151:0/3063738260 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f0704068b20 con 0x7f06f006e940 2024-04-16T02:38:33.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f06f006e940 msgr2=0x7f06f0070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f06f006e940 0x7f06f0070e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0700006010 tx=0x7f070000b540 comp rx=0 tx=0).stop 2024-04-16T02:38:33.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0704100cb0 msgr2=0x7f0704198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0704100cb0 0x7f0704198b40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f06f400b4b0 tx=0x7f06f400b7c0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 shutdown_connections 2024-04-16T02:38:33.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f06f006e940 0x7f06f0070e00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040ffab0 0x7f0704198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 --2- 172.21.15.151:0/3063738260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0704100cb0 0x7f0704198b40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 >> 172.21.15.151:0/3063738260 conn(0x7f07040fb030 msgr2=0x7f0704068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:33.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 shutdown_connections 2024-04-16T02:38:33.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.178+0000 7f070b966700 1 -- 172.21.15.151:0/3063738260 wait complete. 2024-04-16T02:38:33.193 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:38:33.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.291+0000 7fe4a0f40700 1 -- 172.21.15.151:0/3433456022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c0ffad0 msgr2=0x7fe49c0ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.291+0000 7fe4a0f40700 1 --2- 172.21.15.151:0/3433456022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c0ffad0 0x7fe49c0ffef0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009910 tx=0x7fe48c009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:33.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.292+0000 7fe4a0f40700 1 -- 172.21.15.151:0/3433456022 shutdown_connections 2024-04-16T02:38:33.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.292+0000 7fe4a0f40700 1 --2- 172.21.15.151:0/3433456022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe49c100cd0 0x7fe49c101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.292+0000 7fe4a0f40700 1 --2- 172.21.15.151:0/3433456022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c0ffad0 0x7fe49c0ffef0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.292+0000 7fe4a0f40700 1 -- 172.21.15.151:0/3433456022 >> 172.21.15.151:0/3433456022 conn(0x7fe49c0fb050 msgr2=0x7fe49c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:33.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.292+0000 7fe4a0f40700 1 -- 172.21.15.151:0/3433456022 shutdown_connections 2024-04-16T02:38:33.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.292+0000 7fe4a0f40700 1 -- 172.21.15.151:0/3433456022 wait complete. 2024-04-16T02:38:33.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.292+0000 7fe4a0f40700 1 Processor -- start 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.292+0000 7fe4a0f40700 1 -- start start 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe4a0f40700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe49c0ffad0 0x7fe49c196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe4a0f40700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c100cd0 0x7fe49c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe4a0f40700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c196ef0 con 0x7fe49c100cd0 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe4a0f40700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c197030 con 0x7fe49c0ffad0 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe499d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c100cd0 0x7fe49c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe499d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c100cd0 0x7fe49c1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52642/0 (socket says 172.21.15.151:52642) 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe499d9b700 1 -- 172.21.15.151:0/932283963 learned_addr learned my addr 172.21.15.151:0/932283963 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe49a59c700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe49c0ffad0 0x7fe49c196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe499d9b700 1 -- 172.21.15.151:0/932283963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe49c0ffad0 msgr2=0x7fe49c196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe499d9b700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe49c0ffad0 0x7fe49c196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe499d9b700 1 -- 172.21.15.151:0/932283963 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c0095f0 con 0x7fe49c100cd0 2024-04-16T02:38:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.293+0000 7fe499d9b700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c100cd0 0x7fe49c1968d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe49000c8a0 tx=0x7fe49000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:33.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.294+0000 7fe48b7fe700 1 -- 172.21.15.151:0/932283963 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4900078e0 con 0x7fe49c100cd0 2024-04-16T02:38:33.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.294+0000 7fe48b7fe700 1 -- 172.21.15.151:0/932283963 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe49000f450 con 0x7fe49c100cd0 2024-04-16T02:38:33.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.294+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c19bae0 con 0x7fe49c100cd0 2024-04-16T02:38:33.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.294+0000 7fe48b7fe700 1 -- 172.21.15.151:0/932283963 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe49000e5e0 con 0x7fe49c100cd0 2024-04-16T02:38:33.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.294+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c19c030 con 0x7fe49c100cd0 2024-04-16T02:38:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.295+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49c108630 con 0x7fe49c100cd0 2024-04-16T02:38:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.295+0000 7fe48b7fe700 1 -- 172.21.15.151:0/932283963 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7fe49000f5c0 con 0x7fe49c100cd0 2024-04-16T02:38:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.296+0000 7fe48b7fe700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe48406e940 0x7fe484070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.296+0000 7fe48b7fe700 1 -- 172.21.15.151:0/932283963 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fe49008e850 con 0x7fe49c100cd0 2024-04-16T02:38:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.296+0000 7fe49a59c700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe48406e940 0x7fe484070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.296+0000 7fe49a59c700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe48406e940 0x7fe484070e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe48c006010 tx=0x7fe48c00b540 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:33.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.300+0000 7fe48b7fe700 1 -- 172.21.15.151:0/932283963 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fe49005b660 con 0x7fe49c100cd0 2024-04-16T02:38:33.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.533+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe49c069440 con 0x7fe48406e940 2024-04-16T02:38:33.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.534+0000 7fe48b7fe700 1 -- 172.21.15.151:0/932283963 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fe49c069440 con 0x7fe48406e940 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe48406e940 msgr2=0x7fe484070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe48406e940 0x7fe484070e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe48c006010 tx=0x7fe48c00b540 comp rx=0 tx=0).stop 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c100cd0 msgr2=0x7fe49c1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c100cd0 0x7fe49c1968d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe49000c8a0 tx=0x7fe49000cbb0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 shutdown_connections 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fe48406e940 0x7fe484070e00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe49c0ffad0 0x7fe49c196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 --2- 172.21.15.151:0/932283963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe49c100cd0 0x7fe49c1968d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 >> 172.21.15.151:0/932283963 conn(0x7fe49c0fb050 msgr2=0x7fe49c068d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 shutdown_connections 2024-04-16T02:38:33.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.536+0000 7fe4a0f40700 1 -- 172.21.15.151:0/932283963 wait complete. 2024-04-16T02:38:33.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.642+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/3956108706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98100bb0 msgr2=0x7f8d98100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.642+0000 7f8d9ddc6700 1 --2- 172.21.15.151:0/3956108706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98100bb0 0x7f8d98100fd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009910 tx=0x7f8d88009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:33.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.642+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/3956108706 shutdown_connections 2024-04-16T02:38:33.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.642+0000 7f8d9ddc6700 1 --2- 172.21.15.151:0/3956108706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d98101db0 0x7f8d98102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.642+0000 7f8d9ddc6700 1 --2- 172.21.15.151:0/3956108706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98100bb0 0x7f8d98100fd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.642+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/3956108706 >> 172.21.15.151:0/3956108706 conn(0x7f8d980fc110 msgr2=0x7f8d980fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:33.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.642+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/3956108706 shutdown_connections 2024-04-16T02:38:33.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.642+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/3956108706 wait complete. 2024-04-16T02:38:33.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d9ddc6700 1 Processor -- start 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d9ddc6700 1 -- start start 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d9ddc6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d98100bb0 0x7f8d98196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d9ddc6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98101db0 0x7f8d981968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d9ddc6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d98196ee0 con 0x7f8d98101db0 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d9ddc6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d98197020 con 0x7f8d98100bb0 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d8ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98101db0 0x7f8d981968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d8ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98101db0 0x7f8d981968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52666/0 (socket says 172.21.15.151:52666) 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d8ffff700 1 -- 172.21.15.151:0/839372775 learned_addr learned my addr 172.21.15.151:0/839372775 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d977fe700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d98100bb0 0x7f8d98196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.643+0000 7f8d8ffff700 1 -- 172.21.15.151:0/839372775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d98100bb0 msgr2=0x7f8d98196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.644+0000 7f8d8ffff700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d98100bb0 0x7f8d98196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.644+0000 7f8d8ffff700 1 -- 172.21.15.151:0/839372775 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d880095f0 con 0x7f8d98101db0 2024-04-16T02:38:33.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.644+0000 7f8d8ffff700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98101db0 0x7f8d981968c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000c8a0 tx=0x7f8d8000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.644+0000 7f8d957fa700 1 -- 172.21.15.151:0/839372775 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d800078e0 con 0x7f8d98101db0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.644+0000 7f8d957fa700 1 -- 172.21.15.151:0/839372775 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8d8000f450 con 0x7f8d98101db0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.644+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9819bad0 con 0x7f8d98101db0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.644+0000 7f8d957fa700 1 -- 172.21.15.151:0/839372775 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d8000e5e0 con 0x7f8d98101db0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.644+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9819c020 con 0x7f8d98101db0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.645+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d98109710 con 0x7f8d98101db0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.645+0000 7f8d957fa700 1 -- 172.21.15.151:0/839372775 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f8d8000f5c0 con 0x7f8d98101db0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.646+0000 7f8d957fa700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8d8406e940 0x7f8d84070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.646+0000 7f8d957fa700 1 -- 172.21.15.151:0/839372775 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f8d8008e740 con 0x7f8d98101db0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.646+0000 7f8d977fe700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8d8406e940 0x7f8d84070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.646+0000 7f8d977fe700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8d8406e940 0x7f8d84070e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8d88006010 tx=0x7f8d8800b540 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:33.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.650+0000 7f8d957fa700 1 -- 172.21.15.151:0/839372775 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f8d8005b5d0 con 0x7f8d98101db0 2024-04-16T02:38:33.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.870+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d98106700 con 0x7f8d8406e940 2024-04-16T02:38:33.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.881+0000 7f8d957fa700 1 -- 172.21.15.151:0/839372775 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8d98106700 con 0x7f8d8406e940 2024-04-16T02:38:33.883 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (10s) 4s ago 11m 17.5M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (11m) 4s ago 11m 8032k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (10m) 41s ago 10m 7884k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (11m) 4s ago 11m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (10m) 41s ago 10m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (9m) 4s ago 11m 83.5M - 9.4.7 954c08fa6188 f76af6f58171 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (5m) 4s ago 5m 121M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (5m) 4s ago 5m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (5m) 41s ago 5m 58.2M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (5m) 41s ago 5m 67.2M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (2m) 4s ago 12m 499M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (63s) 41s ago 10m 448M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (12m) 4s ago 12m 60.7M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:38:33.884 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (10m) 41s ago 10m 45.9M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:38:33.885 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (50s) 4s ago 11m 7977k - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:38:33.885 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (45s) 41s ago 10m 3678k - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:38:33.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (9m) 4s ago 9m 223M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:38:33.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (9m) 4s ago 9m 249M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:38:33.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (8m) 4s ago 8m 194M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:38:33.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (8m) 41s ago 8m 279M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:38:33.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (7m) 41s ago 7m 228M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:38:33.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (7m) 41s ago 7m 243M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:38:33.885 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (27s) 4s ago 10m 53.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:38:33.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8d8406e940 msgr2=0x7f8d84070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8d8406e940 0x7f8d84070e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8d88006010 tx=0x7f8d8800b540 comp rx=0 tx=0).stop 2024-04-16T02:38:33.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98101db0 msgr2=0x7f8d981968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98101db0 0x7f8d981968c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000c8a0 tx=0x7f8d8000cbb0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 shutdown_connections 2024-04-16T02:38:33.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8d8406e940 0x7f8d84070e00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d98100bb0 0x7f8d98196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 --2- 172.21.15.151:0/839372775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d98101db0 0x7f8d981968c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 >> 172.21.15.151:0/839372775 conn(0x7f8d980fc110 msgr2=0x7f8d98104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:33.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 shutdown_connections 2024-04-16T02:38:33.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.883+0000 7f8d9ddc6700 1 -- 172.21.15.151:0/839372775 wait complete. 2024-04-16T02:38:33.889 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:33 smithi151 ceph-mon[29003]: pgmap v54: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-16T02:38:33.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.988+0000 7fd9b963e700 1 -- 172.21.15.151:0/81551156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 msgr2=0x7fd9b41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.988+0000 7fd9b963e700 1 --2- 172.21.15.151:0/81551156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 0x7fd9b41008b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009910 tx=0x7fd9a8009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:33.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.988+0000 7fd9b963e700 1 -- 172.21.15.151:0/81551156 shutdown_connections 2024-04-16T02:38:33.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.988+0000 7fd9b963e700 1 --2- 172.21.15.151:0/81551156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 0x7fd9b41008b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.988+0000 7fd9b963e700 1 --2- 172.21.15.151:0/81551156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40ff250 0x7fd9b40ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.988+0000 7fd9b963e700 1 -- 172.21.15.151:0/81551156 >> 172.21.15.151:0/81551156 conn(0x7fd9b40fa7f0 msgr2=0x7fd9b40fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:33.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.989+0000 7fd9b963e700 1 -- 172.21.15.151:0/81551156 shutdown_connections 2024-04-16T02:38:33.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.989+0000 7fd9b963e700 1 -- 172.21.15.151:0/81551156 wait complete. 2024-04-16T02:38:33.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.989+0000 7fd9b963e700 1 Processor -- start 2024-04-16T02:38:33.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b963e700 1 -- start start 2024-04-16T02:38:33.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b963e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40ff250 0x7fd9b41021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:33.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b963e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 0x7fd9b419b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:33.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b963e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b419b730 con 0x7fd9b4100450 2024-04-16T02:38:33.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b963e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b419b8a0 con 0x7fd9b40ff250 2024-04-16T02:38:33.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 0x7fd9b419b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:33.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 0x7fd9b419b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52690/0 (socket says 172.21.15.151:52690) 2024-04-16T02:38:33.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b27fc700 1 -- 172.21.15.151:0/2037326748 learned_addr learned my addr 172.21.15.151:0/2037326748 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:33.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b2ffd700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40ff250 0x7fd9b41021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b27fc700 1 -- 172.21.15.151:0/2037326748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40ff250 msgr2=0x7fd9b41021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b27fc700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40ff250 0x7fd9b41021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b27fc700 1 -- 172.21.15.151:0/2037326748 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a80095f0 con 0x7fd9b4100450 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b2ffd700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40ff250 0x7fd9b41021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b27fc700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 0x7fd9b419b160 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800bd70 tx=0x7fd9a800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd99bfff700 1 -- 172.21.15.151:0/2037326748 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9a801d070 con 0x7fd9b4100450 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd99bfff700 1 -- 172.21.15.151:0/2037326748 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd9a8022470 con 0x7fd9b4100450 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.990+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b419bb20 con 0x7fd9b4100450 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.991+0000 7fd99bfff700 1 -- 172.21.15.151:0/2037326748 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9a800f670 con 0x7fd9b4100450 2024-04-16T02:38:33.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.991+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b419c090 con 0x7fd9b4100450 2024-04-16T02:38:33.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.992+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b4107e50 con 0x7fd9b4100450 2024-04-16T02:38:33.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.992+0000 7fd99bfff700 1 -- 172.21.15.151:0/2037326748 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7fd9a8022a50 con 0x7fd9b4100450 2024-04-16T02:38:33.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.993+0000 7fd99bfff700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd99c06e940 0x7fd99c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:33.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.993+0000 7fd99bfff700 1 -- 172.21.15.151:0/2037326748 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fd9a808ff70 con 0x7fd9b4100450 2024-04-16T02:38:33.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.993+0000 7fd9b2ffd700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd99c06e940 0x7fd99c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:33.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.993+0000 7fd9b2ffd700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd99c06e940 0x7fd99c070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd9b41002b0 tx=0x7fd9a4008040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:34.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:33.998+0000 7fd99bfff700 1 -- 172.21.15.151:0/2037326748 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fd9a80c4cb0 con 0x7fd9b4100450 2024-04-16T02:38:34.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:33 smithi153 ceph-mon[36747]: pgmap v54: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-16T02:38:34.291 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.288+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd9b419c6a0 con 0x7fd9b4100450 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.289+0000 7fd99bfff700 1 -- 172.21.15.151:0/2037326748 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fd9a8027070 con 0x7fd9b4100450 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:38:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:38:34.293 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:38:34.293 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:38:34.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:38:34.293 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:38:34.293 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-16T02:38:34.293 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:38:34.293 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:38:34.293 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:38:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd99c06e940 msgr2=0x7fd99c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd99c06e940 0x7fd99c070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd9b41002b0 tx=0x7fd9a4008040 comp rx=0 tx=0).stop 2024-04-16T02:38:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 msgr2=0x7fd9b419b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 0x7fd9b419b160 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800bd70 tx=0x7fd9a800be50 comp rx=0 tx=0).stop 2024-04-16T02:38:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 shutdown_connections 2024-04-16T02:38:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd99c06e940 0x7fd99c070e00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40ff250 0x7fd9b41021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 --2- 172.21.15.151:0/2037326748 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9b4100450 0x7fd9b419b160 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 >> 172.21.15.151:0/2037326748 conn(0x7fd9b40fa7f0 msgr2=0x7fd9b40fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:34.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 shutdown_connections 2024-04-16T02:38:34.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.291+0000 7fd9b963e700 1 -- 172.21.15.151:0/2037326748 wait complete. 2024-04-16T02:38:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.407+0000 7fb3de52c700 1 -- 172.21.15.151:0/1173051132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d8100cb0 msgr2=0x7fb3d8101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.407+0000 7fb3de52c700 1 --2- 172.21.15.151:0/1173051132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d8100cb0 0x7fb3d8101110 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009910 tx=0x7fb3cc009c20 comp rx=0 tx=0).stop 2024-04-16T02:38:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.407+0000 7fb3de52c700 1 -- 172.21.15.151:0/1173051132 shutdown_connections 2024-04-16T02:38:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.407+0000 7fb3de52c700 1 --2- 172.21.15.151:0/1173051132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d8100cb0 0x7fb3d8101110 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.407+0000 7fb3de52c700 1 --2- 172.21.15.151:0/1173051132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d80ffab0 0x7fb3d80ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.407+0000 7fb3de52c700 1 -- 172.21.15.151:0/1173051132 >> 172.21.15.151:0/1173051132 conn(0x7fb3d80fb050 msgr2=0x7fb3d80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.407+0000 7fb3de52c700 1 -- 172.21.15.151:0/1173051132 shutdown_connections 2024-04-16T02:38:34.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.407+0000 7fb3de52c700 1 -- 172.21.15.151:0/1173051132 wait complete. 2024-04-16T02:38:34.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.408+0000 7fb3de52c700 1 Processor -- start 2024-04-16T02:38:34.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.408+0000 7fb3de52c700 1 -- start start 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.408+0000 7fb3de52c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d80ffab0 0x7fb3d8069820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3de52c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8100cb0 0x7fb3d819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3de52c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d819b670 con 0x7fb3d80ffab0 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3de52c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d819b7e0 con 0x7fb3d8100cb0 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d80ffab0 0x7fb3d8069820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d80ffab0 0x7fb3d8069820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52710/0 (socket says 172.21.15.151:52710) 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d7fff700 1 -- 172.21.15.151:0/590604543 learned_addr learned my addr 172.21.15.151:0/590604543 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d77fe700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8100cb0 0x7fb3d819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d7fff700 1 -- 172.21.15.151:0/590604543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8100cb0 msgr2=0x7fb3d819b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d7fff700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8100cb0 0x7fb3d819b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d7fff700 1 -- 172.21.15.151:0/590604543 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc0095f0 con 0x7fb3d80ffab0 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d77fe700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8100cb0 0x7fb3d819b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d7fff700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d80ffab0 0x7fb3d8069820 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800d900 tx=0x7fb3c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d57fa700 1 -- 172.21.15.151:0/590604543 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3c80041f0 con 0x7fb3d80ffab0 2024-04-16T02:38:34.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d57fa700 1 -- 172.21.15.151:0/590604543 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb3c8004350 con 0x7fb3d80ffab0 2024-04-16T02:38:34.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d819bac0 con 0x7fb3d80ffab0 2024-04-16T02:38:34.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3d57fa700 1 -- 172.21.15.151:0/590604543 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3c8003da0 con 0x7fb3d80ffab0 2024-04-16T02:38:34.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.409+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d819c010 con 0x7fb3d80ffab0 2024-04-16T02:38:34.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.410+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d8108610 con 0x7fb3d80ffab0 2024-04-16T02:38:34.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.411+0000 7fb3d57fa700 1 -- 172.21.15.151:0/590604543 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7fb3c8009730 con 0x7fb3d80ffab0 2024-04-16T02:38:34.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.412+0000 7fb3d57fa700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb3c006e940 0x7fb3c0070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:38:34.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.412+0000 7fb3d57fa700 1 -- 172.21.15.151:0/590604543 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fb3c808e9f0 con 0x7fb3d80ffab0 2024-04-16T02:38:34.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.412+0000 7fb3d77fe700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb3c006e940 0x7fb3c0070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:38:34.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.412+0000 7fb3d77fe700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb3c006e940 0x7fb3c0070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8069d80 tx=0x7fb3cc005a90 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:38:34.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.416+0000 7fb3d57fa700 1 -- 172.21.15.151:0/590604543 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fb3c8090e60 con 0x7fb3d80ffab0 2024-04-16T02:38:34.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.646+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3d8105600 con 0x7fb3c006e940 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.647+0000 7fb3d57fa700 1 -- 172.21.15.151:0/590604543 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fb3d8105600 con 0x7fb3c006e940 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr" 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "2/2 daemons upgraded", 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading grafana daemons", 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:38:34.650 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:38:34.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb3c006e940 msgr2=0x7fb3c0070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:34.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb3c006e940 0x7fb3c0070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8069d80 tx=0x7fb3cc005a90 comp rx=0 tx=0).stop 2024-04-16T02:38:34.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d80ffab0 msgr2=0x7fb3d8069820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:38:34.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d80ffab0 0x7fb3d8069820 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800d900 tx=0x7fb3c800dcc0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 shutdown_connections 2024-04-16T02:38:34.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb3c006e940 0x7fb3c0070e00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d80ffab0 0x7fb3d8069820 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 --2- 172.21.15.151:0/590604543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8100cb0 0x7fb3d819b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:38:34.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 >> 172.21.15.151:0/590604543 conn(0x7fb3d80fb050 msgr2=0x7fb3d8103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:38:34.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 shutdown_connections 2024-04-16T02:38:34.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:38:34.650+0000 7fb3de52c700 1 -- 172.21.15.151:0/590604543 wait complete. 2024-04-16T02:38:34.949 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:34.949 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:34.949 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2037326748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.951 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:34.951 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:34 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:35.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:35.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:35.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:35.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:35.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:38:35.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2037326748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:35.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:34 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:38:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:35 smithi151 ceph-mon[29003]: pgmap v55: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:35 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T02:38:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:35 smithi151 ceph-mon[29003]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:38:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:35 smithi153 ceph-mon[36747]: pgmap v55: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:35 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T02:38:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:35 smithi153 ceph-mon[36747]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:38:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:38:37.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:36 smithi151 ceph-mon[29003]: Upgrade: Updating grafana.smithi151 2024-04-16T02:38:37.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:36 smithi151 ceph-mon[29003]: Deploying daemon grafana.smithi151 on smithi151 2024-04-16T02:38:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:36 smithi153 ceph-mon[36747]: Upgrade: Updating grafana.smithi151 2024-04-16T02:38:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:36 smithi153 ceph-mon[36747]: Deploying daemon grafana.smithi151 on smithi151 2024-04-16T02:38:38.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:37 smithi151 ceph-mon[29003]: pgmap v56: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:37 smithi153 ceph-mon[36747]: pgmap v56: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:40.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:39 smithi151 ceph-mon[29003]: pgmap v57: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:39 smithi153 ceph-mon[36747]: pgmap v57: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:42.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:41 smithi151 ceph-mon[29003]: pgmap v58: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:42.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:41 smithi153 ceph-mon[36747]: pgmap v58: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:44.092 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:43 smithi153 ceph-mon[36747]: pgmap v59: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:44.153 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:43 smithi151 ceph-mon[29003]: pgmap v59: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:46.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:45 smithi151 ceph-mon[29003]: pgmap v60: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:46.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:45 smithi153 ceph-mon[36747]: pgmap v60: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:47.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:47 smithi151 ceph-mon[29003]: pgmap v61: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:48.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:47 smithi153 ceph-mon[36747]: pgmap v61: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:50.099 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:49 smithi151 ceph-mon[29003]: pgmap v62: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:49 smithi153 ceph-mon[36747]: pgmap v62: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:51.118 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:51.118 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:51.118 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:51.118 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:51 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:38:51.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:51.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:51.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:38:51.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:51 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:38:52.316 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:52 smithi151 ceph-mon[29003]: pgmap v63: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:52.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:52 smithi153 ceph-mon[36747]: pgmap v63: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:54.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:54 smithi151 ceph-mon[29003]: pgmap v64: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:54.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:54 smithi153 ceph-mon[36747]: pgmap v64: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:56.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:56 smithi151 ceph-mon[29003]: pgmap v65: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:56.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:56 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:56.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:56 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:56.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:56 smithi153 ceph-mon[36747]: pgmap v65: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:38:56.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:56 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:56.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:56 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:57.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:57 smithi151 ceph-mon[29003]: pgmap v66: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:57.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:57 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:57.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:57 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:58.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:57 smithi153 ceph-mon[36747]: pgmap v66: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:38:58.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:57 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:58.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:57 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:38:59.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:38:59 smithi151 ceph-mon[29003]: pgmap v67: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:00.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:38:59 smithi153 ceph-mon[36747]: pgmap v67: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: Upgrade: Finalizing container_image settings 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:01.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:00 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: Upgrade: Finalizing container_image settings 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T02:39:01.147 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T02:39:01.148 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T02:39:01.149 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:01.150 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.150 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:01.150 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:01.150 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:01.150 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:01.150 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:01.150 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:01.150 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:00 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:02.172 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:01 smithi151 ceph-mon[29003]: Upgrade: Complete! 2024-04-16T02:39:02.172 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:01 smithi151 ceph-mon[29003]: pgmap v68: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:01 smithi153 ceph-mon[36747]: Upgrade: Complete! 2024-04-16T02:39:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:01 smithi153 ceph-mon[36747]: pgmap v68: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:04.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:03 smithi151 ceph-mon[29003]: pgmap v69: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:04.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:03 smithi153 ceph-mon[36747]: pgmap v69: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:04.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.769+0000 7f8ac20e9700 1 -- 172.21.15.151:0/1884669397 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc101640 msgr2=0x7f8abc103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.769+0000 7f8ac20e9700 1 --2- 172.21.15.151:0/1884669397 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc101640 0x7f8abc103a30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009910 tx=0x7f8aa4009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.769+0000 7f8ac20e9700 1 -- 172.21.15.151:0/1884669397 shutdown_connections 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.769+0000 7f8ac20e9700 1 --2- 172.21.15.151:0/1884669397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8abc103f70 0x7f8abc106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.769+0000 7f8ac20e9700 1 --2- 172.21.15.151:0/1884669397 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc101640 0x7f8abc103a30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.769+0000 7f8ac20e9700 1 -- 172.21.15.151:0/1884669397 >> 172.21.15.151:0/1884669397 conn(0x7f8abc0fb030 msgr2=0x7f8abc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.769+0000 7f8ac20e9700 1 -- 172.21.15.151:0/1884669397 shutdown_connections 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.769+0000 7f8ac20e9700 1 -- 172.21.15.151:0/1884669397 wait complete. 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.769+0000 7f8ac20e9700 1 Processor -- start 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8ac20e9700 1 -- start start 2024-04-16T02:39:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8ac20e9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8abc101640 0x7f8abc100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8ac20e9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc103f70 0x7f8abc0ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8ac20e9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc0ffb30 con 0x7f8abc103f70 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8ac20e9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc0ffc70 con 0x7f8abc101640 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8abaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc103f70 0x7f8abc0ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8abaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc103f70 0x7f8abc0ff5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52582/0 (socket says 172.21.15.151:52582) 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8abaffd700 1 -- 172.21.15.151:0/3617410497 learned_addr learned my addr 172.21.15.151:0/3617410497 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8abb7fe700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8abc101640 0x7f8abc100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8abaffd700 1 -- 172.21.15.151:0/3617410497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8abc101640 msgr2=0x7f8abc100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8abaffd700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8abc101640 0x7f8abc100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8abaffd700 1 -- 172.21.15.151:0/3617410497 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa40095f0 con 0x7f8abc103f70 2024-04-16T02:39:04.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8abb7fe700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8abc101640 0x7f8abc100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:39:04.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8abaffd700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc103f70 0x7f8abc0ff5f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00b4b0 tx=0x7f8aac00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:04.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.770+0000 7f8ab8ff9700 1 -- 172.21.15.151:0/3617410497 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8aac011800 con 0x7f8abc103f70 2024-04-16T02:39:04.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.771+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8abc0fff50 con 0x7f8abc103f70 2024-04-16T02:39:04.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.771+0000 7f8ab8ff9700 1 -- 172.21.15.151:0/3617410497 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8aac011e20 con 0x7f8abc103f70 2024-04-16T02:39:04.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.771+0000 7f8ab8ff9700 1 -- 172.21.15.151:0/3617410497 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8aac00f2e0 con 0x7f8abc103f70 2024-04-16T02:39:04.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.771+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8abc19e470 con 0x7f8abc103f70 2024-04-16T02:39:04.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.772+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8abc1905a0 con 0x7f8abc103f70 2024-04-16T02:39:04.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.772+0000 7f8ab8ff9700 1 -- 172.21.15.151:0/3617410497 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f8aac010380 con 0x7f8abc103f70 2024-04-16T02:39:04.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.773+0000 7f8ab8ff9700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8aa806e940 0x7f8aa8070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:04.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.773+0000 7f8ab8ff9700 1 -- 172.21.15.151:0/3617410497 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f8aac08e440 con 0x7f8abc103f70 2024-04-16T02:39:04.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.773+0000 7f8abb7fe700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8aa806e940 0x7f8aa8070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:04.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.773+0000 7f8abb7fe700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8aa806e940 0x7f8aa8070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8aa40059d0 tx=0x7f8aa4005960 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:04.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.777+0000 7f8ab8ff9700 1 -- 172.21.15.151:0/3617410497 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f8aac05b250 con 0x7f8abc103f70 2024-04-16T02:39:05.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.997+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8abc0611d0 con 0x7f8aa806e940 2024-04-16T02:39:05.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:04.998+0000 7f8ab8ff9700 1 -- 172.21.15.151:0/3617410497 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f8abc0611d0 con 0x7f8aa806e940 2024-04-16T02:39:05.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8aa806e940 msgr2=0x7f8aa8070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:05.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8aa806e940 0x7f8aa8070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8aa40059d0 tx=0x7f8aa4005960 comp rx=0 tx=0).stop 2024-04-16T02:39:05.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc103f70 msgr2=0x7f8abc0ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:05.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc103f70 0x7f8abc0ff5f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00b4b0 tx=0x7f8aac00b870 comp rx=0 tx=0).stop 2024-04-16T02:39:05.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 shutdown_connections 2024-04-16T02:39:05.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f8aa806e940 0x7f8aa8070e00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:05.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8abc101640 0x7f8abc100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:05.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 --2- 172.21.15.151:0/3617410497 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8abc103f70 0x7f8abc0ff5f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:05.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 >> 172.21.15.151:0/3617410497 conn(0x7f8abc0fb030 msgr2=0x7f8abc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:05.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 shutdown_connections 2024-04-16T02:39:05.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:05.001+0000 7f8ac20e9700 1 -- 172.21.15.151:0/3617410497 wait complete. 2024-04-16T02:39:05.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-16T02:39:05.917 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:05.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:05 smithi151 ceph-mon[29003]: pgmap v70: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:05.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:05 smithi151 ceph-mon[29003]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:05.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:05 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:39:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:05 smithi153 ceph-mon[36747]: pgmap v70: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:05 smithi153 ceph-mon[36747]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:05 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:39:07.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 -- 172.21.15.151:0/515777780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 msgr2=0x7feef0105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:07.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 --2- 172.21.15.151:0/515777780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 0x7feef0105870 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feee4009910 tx=0x7feee4009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:07.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 -- 172.21.15.151:0/515777780 shutdown_connections 2024-04-16T02:39:07.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 --2- 172.21.15.151:0/515777780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 0x7feef0105870 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:07.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 --2- 172.21.15.151:0/515777780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef00ff530 0x7feef00ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:07.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 -- 172.21.15.151:0/515777780 >> 172.21.15.151:0/515777780 conn(0x7feef0074df0 msgr2=0x7feef0075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:07.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 -- 172.21.15.151:0/515777780 shutdown_connections 2024-04-16T02:39:07.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 -- 172.21.15.151:0/515777780 wait complete. 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 Processor -- start 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.626+0000 7feef6f5e700 1 -- start start 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feef6f5e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef00ff530 0x7feef01004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feef6f5e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 0x7feef01009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feef6f5e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef0100f30 con 0x7feef00ffee0 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feef6f5e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef0101070 con 0x7feef00ff530 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeeffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 0x7feef01009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeeffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 0x7feef01009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48842/0 (socket says 172.21.15.151:48842) 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeeffff700 1 -- 172.21.15.151:0/1846647732 learned_addr learned my addr 172.21.15.151:0/1846647732 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeeffff700 1 -- 172.21.15.151:0/1846647732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef00ff530 msgr2=0x7feef01004b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeeffff700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef00ff530 0x7feef01004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeeffff700 1 -- 172.21.15.151:0/1846647732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee40095f0 con 0x7feef00ffee0 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeeffff700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 0x7feef01009f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feee4005f50 tx=0x7feee400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeedffb700 1 -- 172.21.15.151:0/1846647732 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feee401d070 con 0x7feef00ffee0 2024-04-16T02:39:07.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeedffb700 1 -- 172.21.15.151:0/1846647732 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feee400f460 con 0x7feef00ffee0 2024-04-16T02:39:07.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef01012f0 con 0x7feef00ffee0 2024-04-16T02:39:07.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.627+0000 7feeedffb700 1 -- 172.21.15.151:0/1846647732 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feee4017630 con 0x7feef00ffee0 2024-04-16T02:39:07.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.628+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef01a2cd0 con 0x7feef00ffee0 2024-04-16T02:39:07.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.628+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef0109150 con 0x7feef00ffee0 2024-04-16T02:39:07.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.629+0000 7feeedffb700 1 -- 172.21.15.151:0/1846647732 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7feee400f5d0 con 0x7feef00ffee0 2024-04-16T02:39:07.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.629+0000 7feeedffb700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feed806e940 0x7feed8070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:07.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.629+0000 7feeedffb700 1 -- 172.21.15.151:0/1846647732 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7feee408fdd0 con 0x7feef00ffee0 2024-04-16T02:39:07.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.630+0000 7feef4cfa700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feed806e940 0x7feed8070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:07.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.630+0000 7feef4cfa700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feed806e940 0x7feed8070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feee800b420 tx=0x7feee8008540 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:07.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.634+0000 7feeedffb700 1 -- 172.21.15.151:0/1846647732 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7feee4059130 con 0x7feef00ffee0 2024-04-16T02:39:07.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.927+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feef004ea90 con 0x7feef00ffee0 2024-04-16T02:39:07.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.927+0000 7feeedffb700 1 -- 172.21.15.151:0/1846647732 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7feee405c7f0 con 0x7feef00ffee0 2024-04-16T02:39:07.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feed806e940 msgr2=0x7feed8070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:07.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feed806e940 0x7feed8070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feee800b420 tx=0x7feee8008540 comp rx=0 tx=0).stop 2024-04-16T02:39:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 msgr2=0x7feef01009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 0x7feef01009f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feee4005f50 tx=0x7feee400be20 comp rx=0 tx=0).stop 2024-04-16T02:39:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 shutdown_connections 2024-04-16T02:39:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7feed806e940 0x7feed8070e00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feef00ff530 0x7feef01004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 --2- 172.21.15.151:0/1846647732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef00ffee0 0x7feef01009f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 >> 172.21.15.151:0/1846647732 conn(0x7feef0074df0 msgr2=0x7feef01050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 shutdown_connections 2024-04-16T02:39:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:07.930+0000 7feef6f5e700 1 -- 172.21.15.151:0/1846647732 wait complete. 2024-04-16T02:39:07.943 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:39:08.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:07 smithi151 ceph-mon[29003]: pgmap v71: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:08.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:07 smithi153 ceph-mon[36747]: pgmap v71: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:08.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-16T02:39:08.925 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:08.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:08 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1846647732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:09.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:08 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1846647732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:10.163 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:09 smithi151 ceph-mon[29003]: pgmap v72: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:10.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:09 smithi153 ceph-mon[36747]: pgmap v72: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:10.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.696+0000 7fb5c7d76700 1 -- 172.21.15.151:0/213358020 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073130 msgr2=0x7fb5c0073510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.696+0000 7fb5c7d76700 1 --2- 172.21.15.151:0/213358020 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073130 0x7fb5c0073510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc009910 tx=0x7fb5bc009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.696+0000 7fb5c7d76700 1 -- 172.21.15.151:0/213358020 shutdown_connections 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.696+0000 7fb5c7d76700 1 --2- 172.21.15.151:0/213358020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c0073a50 0x7fb5c0111630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.696+0000 7fb5c7d76700 1 --2- 172.21.15.151:0/213358020 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073130 0x7fb5c0073510 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.696+0000 7fb5c7d76700 1 -- 172.21.15.151:0/213358020 >> 172.21.15.151:0/213358020 conn(0x7fb5c00fc650 msgr2=0x7fb5c00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.697+0000 7fb5c7d76700 1 -- 172.21.15.151:0/213358020 shutdown_connections 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.697+0000 7fb5c7d76700 1 -- 172.21.15.151:0/213358020 wait complete. 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.697+0000 7fb5c7d76700 1 Processor -- start 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.697+0000 7fb5c7d76700 1 -- start start 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c7d76700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c0073130 0x7fb5c01a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c7d76700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073a50 0x7fb5c01a3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c7d76700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c01a36a0 con 0x7fb5c0073a50 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c7d76700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c019cba0 con 0x7fb5c0073130 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c5311700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073a50 0x7fb5c01a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c5311700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073a50 0x7fb5c01a3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48860/0 (socket says 172.21.15.151:48860) 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c5311700 1 -- 172.21.15.151:0/1432302300 learned_addr learned my addr 172.21.15.151:0/1432302300 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c5b12700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c0073130 0x7fb5c01a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c5311700 1 -- 172.21.15.151:0/1432302300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c0073130 msgr2=0x7fb5c01a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c5311700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c0073130 0x7fb5c01a2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c5311700 1 -- 172.21.15.151:0/1432302300 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc0095f0 con 0x7fb5c0073a50 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c5b12700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c0073130 0x7fb5c01a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c5311700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073a50 0x7fb5c01a3010 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000c8a0 tx=0x7fb5b000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5b6ffd700 1 -- 172.21.15.151:0/1432302300 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5b000cea0 con 0x7fb5c0073a50 2024-04-16T02:39:10.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5b6ffd700 1 -- 172.21.15.151:0/1432302300 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5b0004830 con 0x7fb5c0073a50 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c019ce80 con 0x7fb5c0073a50 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5b6ffd700 1 -- 172.21.15.151:0/1432302300 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5b00055b0 con 0x7fb5c0073a50 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.698+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c019d3d0 con 0x7fb5c0073a50 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.699+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c010edb0 con 0x7fb5c0073a50 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.700+0000 7fb5b6ffd700 1 -- 172.21.15.151:0/1432302300 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7fb5b00049a0 con 0x7fb5c0073a50 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.700+0000 7fb5b6ffd700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb5ac06e940 0x7fb5ac070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.700+0000 7fb5c5b12700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb5ac06e940 0x7fb5ac070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.701+0000 7fb5b6ffd700 1 -- 172.21.15.151:0/1432302300 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fb5b008e590 con 0x7fb5c0073a50 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.701+0000 7fb5c5b12700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb5ac06e940 0x7fb5ac070e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0059d0 tx=0x7fb5bc01a040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.705+0000 7fb5b6ffd700 1 -- 172.21.15.151:0/1432302300 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fb5b005b3a0 con 0x7fb5c0073a50 2024-04-16T02:39:10.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.984+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb5c004ea90 con 0x7fb5c0073a50 2024-04-16T02:39:10.987 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.984+0000 7fb5b6ffd700 1 -- 172.21.15.151:0/1432302300 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fb5b0004c50 con 0x7fb5c0073a50 2024-04-16T02:39:10.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.986+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb5ac06e940 msgr2=0x7fb5ac070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:10.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.986+0000 7fb5c7d76700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb5ac06e940 0x7fb5ac070e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0059d0 tx=0x7fb5bc01a040 comp rx=0 tx=0).stop 2024-04-16T02:39:10.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.986+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073a50 msgr2=0x7fb5c01a3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:10.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.986+0000 7fb5c7d76700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073a50 0x7fb5c01a3010 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000c8a0 tx=0x7fb5b000cc60 comp rx=0 tx=0).stop 2024-04-16T02:39:10.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.986+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 shutdown_connections 2024-04-16T02:39:10.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.986+0000 7fb5c7d76700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fb5ac06e940 0x7fb5ac070e00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:10.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.986+0000 7fb5c7d76700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c0073130 0x7fb5c01a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:10.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.986+0000 7fb5c7d76700 1 --2- 172.21.15.151:0/1432302300 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c0073a50 0x7fb5c01a3010 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:10.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.986+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 >> 172.21.15.151:0/1432302300 conn(0x7fb5c00fc650 msgr2=0x7fb5c0103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:10.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.987+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 shutdown_connections 2024-04-16T02:39:10.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:10.987+0000 7fb5c7d76700 1 -- 172.21.15.151:0/1432302300 wait complete. 2024-04-16T02:39:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)" 2024-04-16T02:39:11.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-16T02:39:11.909 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:11.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:11 smithi151 ceph-mon[29003]: pgmap v73: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:11 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/1432302300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:11 smithi153 ceph-mon[36747]: pgmap v73: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:11 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/1432302300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:13.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.564+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3383895033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 msgr2=0x7f6f481011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:13.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.564+0000 7f6f4e92f700 1 --2- 172.21.15.151:0/3383895033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 0x7f6f481011a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009910 tx=0x7f6f38009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:13.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.564+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3383895033 shutdown_connections 2024-04-16T02:39:13.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.564+0000 7f6f4e92f700 1 --2- 172.21.15.151:0/3383895033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48101770 0x7f6f481057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:13.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.564+0000 7f6f4e92f700 1 --2- 172.21.15.151:0/3383895033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 0x7f6f481011a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:13.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.564+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3383895033 >> 172.21.15.151:0/3383895033 conn(0x7f6f480fc650 msgr2=0x7f6f480fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:13.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.564+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3383895033 shutdown_connections 2024-04-16T02:39:13.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.564+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3383895033 wait complete. 2024-04-16T02:39:13.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f4e92f700 1 Processor -- start 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f4e92f700 1 -- start start 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f4e92f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 0x7f6f4819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f4e92f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48101770 0x7f6f4819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f4e92f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4819d110 con 0x7f6f48100dc0 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f4e92f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f481965c0 con 0x7f6f48101770 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f47fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 0x7f6f4819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f47fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 0x7f6f4819c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48870/0 (socket says 172.21.15.151:48870) 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f47fff700 1 -- 172.21.15.151:0/3782221076 learned_addr learned my addr 172.21.15.151:0/3782221076 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.565+0000 7f6f477fe700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48101770 0x7f6f4819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f47fff700 1 -- 172.21.15.151:0/3782221076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48101770 msgr2=0x7f6f4819ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f47fff700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48101770 0x7f6f4819ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f47fff700 1 -- 172.21.15.151:0/3782221076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380095f0 con 0x7f6f48100dc0 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f477fe700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48101770 0x7f6f4819ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:39:13.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f47fff700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 0x7f6f4819c540 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6f38006010 tx=0x7f6f3800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:13.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f457fa700 1 -- 172.21.15.151:0/3782221076 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f3801d070 con 0x7f6f48100dc0 2024-04-16T02:39:13.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f457fa700 1 -- 172.21.15.151:0/3782221076 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f38004fb0 con 0x7f6f48100dc0 2024-04-16T02:39:13.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f48196840 con 0x7f6f48100dc0 2024-04-16T02:39:13.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f457fa700 1 -- 172.21.15.151:0/3782221076 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f38022ea0 con 0x7f6f48100dc0 2024-04-16T02:39:13.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.566+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f48196d30 con 0x7f6f48100dc0 2024-04-16T02:39:13.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.567+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f48109160 con 0x7f6f48100dc0 2024-04-16T02:39:13.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.568+0000 7f6f457fa700 1 -- 172.21.15.151:0/3782221076 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f6f38022470 con 0x7f6f48100dc0 2024-04-16T02:39:13.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.568+0000 7f6f457fa700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6f3406e940 0x7f6f34070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:13.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.568+0000 7f6f457fa700 1 -- 172.21.15.151:0/3782221076 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f6f3808fe30 con 0x7f6f48100dc0 2024-04-16T02:39:13.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.568+0000 7f6f477fe700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6f3406e940 0x7f6f34070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:13.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.569+0000 7f6f477fe700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6f3406e940 0x7f6f34070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6f30005fd0 tx=0x7f6f30005dc0 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:13.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.573+0000 7f6f457fa700 1 -- 172.21.15.151:0/3782221076 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f6f3805ccf0 con 0x7f6f48100dc0 2024-04-16T02:39:13.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.854+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6f4802ccf0 con 0x7f6f48100dc0 2024-04-16T02:39:13.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.854+0000 7f6f457fa700 1 -- 172.21.15.151:0/3782221076 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f6f38022720 con 0x7f6f48100dc0 2024-04-16T02:39:13.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.856+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6f3406e940 msgr2=0x7f6f34070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:13.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.856+0000 7f6f4e92f700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6f3406e940 0x7f6f34070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6f30005fd0 tx=0x7f6f30005dc0 comp rx=0 tx=0).stop 2024-04-16T02:39:13.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.856+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 msgr2=0x7f6f4819c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:13.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.856+0000 7f6f4e92f700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 0x7f6f4819c540 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6f38006010 tx=0x7f6f3800be20 comp rx=0 tx=0).stop 2024-04-16T02:39:13.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.857+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 shutdown_connections 2024-04-16T02:39:13.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.857+0000 7f6f4e92f700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6f3406e940 0x7f6f34070e00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:13.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.857+0000 7f6f4e92f700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f48100dc0 0x7f6f4819c540 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:13.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.857+0000 7f6f4e92f700 1 --2- 172.21.15.151:0/3782221076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48101770 0x7f6f4819ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:13.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.857+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 >> 172.21.15.151:0/3782221076 conn(0x7f6f480fc650 msgr2=0x7f6f480fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:13.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.857+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 shutdown_connections 2024-04-16T02:39:13.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:13.857+0000 7f6f4e92f700 1 -- 172.21.15.151:0/3782221076 wait complete. 2024-04-16T02:39:13.870 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:39:14.169 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:13 smithi151 ceph-mon[29003]: pgmap v74: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:14.169 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:13 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/3782221076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:14.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:13 smithi153 ceph-mon[36747]: pgmap v74: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:14.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:13 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/3782221076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:14.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-16T02:39:14.697 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:16.097 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:15 smithi151 ceph-mon[29003]: pgmap v75: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:16.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:15 smithi153 ceph-mon[36747]: pgmap v75: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.316+0000 7f789f835700 1 -- 172.21.15.151:0/37991494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 msgr2=0x7f7898100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.316+0000 7f789f835700 1 --2- 172.21.15.151:0/37991494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 0x7f7898100050 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f788c009910 tx=0x7f788c009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.316+0000 7f789f835700 1 -- 172.21.15.151:0/37991494 shutdown_connections 2024-04-16T02:39:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.316+0000 7f789f835700 1 --2- 172.21.15.151:0/37991494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 0x7f7898100050 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.316+0000 7f789f835700 1 --2- 172.21.15.151:0/37991494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7898105a40 0x7f7898105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.316+0000 7f789f835700 1 -- 172.21.15.151:0/37991494 >> 172.21.15.151:0/37991494 conn(0x7f78980755a0 msgr2=0x7f78980759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.316+0000 7f789f835700 1 -- 172.21.15.151:0/37991494 shutdown_connections 2024-04-16T02:39:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.316+0000 7f789f835700 1 -- 172.21.15.151:0/37991494 wait complete. 2024-04-16T02:39:16.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789f835700 1 Processor -- start 2024-04-16T02:39:16.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789f835700 1 -- start start 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789f835700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 0x7f78981976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789f835700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7898105a40 0x7f7898197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789f835700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7898198200 con 0x7f78980ffbd0 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789f835700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7898198340 con 0x7f7898105a40 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789d5d1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 0x7f78981976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789d5d1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 0x7f78981976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48896/0 (socket says 172.21.15.151:48896) 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789d5d1700 1 -- 172.21.15.151:0/3239452142 learned_addr learned my addr 172.21.15.151:0/3239452142 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.317+0000 7f789cdd0700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7898105a40 0x7f7898197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f789d5d1700 1 -- 172.21.15.151:0/3239452142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7898105a40 msgr2=0x7f7898197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f789d5d1700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7898105a40 0x7f7898197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f789d5d1700 1 -- 172.21.15.151:0/3239452142 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c0095f0 con 0x7f78980ffbd0 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f789cdd0700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7898105a40 0x7f7898197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:39:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f789d5d1700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 0x7f78981976a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f789000ba50 tx=0x7f789000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:16.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f78967fc700 1 -- 172.21.15.151:0/3239452142 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f789000ca00 con 0x7f78980ffbd0 2024-04-16T02:39:16.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f78967fc700 1 -- 172.21.15.151:0/3239452142 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f789000cb60 con 0x7f78980ffbd0 2024-04-16T02:39:16.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78981a0710 con 0x7f78980ffbd0 2024-04-16T02:39:16.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f78967fc700 1 -- 172.21.15.151:0/3239452142 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78900075a0 con 0x7f78980ffbd0 2024-04-16T02:39:16.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.318+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78981a0bd0 con 0x7f78980ffbd0 2024-04-16T02:39:16.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.319+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7898108060 con 0x7f78980ffbd0 2024-04-16T02:39:16.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.319+0000 7f78967fc700 1 -- 172.21.15.151:0/3239452142 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f789000ccd0 con 0x7f78980ffbd0 2024-04-16T02:39:16.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.320+0000 7f78967fc700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f788006e940 0x7f7880070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:16.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.320+0000 7f78967fc700 1 -- 172.21.15.151:0/3239452142 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f7890006e40 con 0x7f78980ffbd0 2024-04-16T02:39:16.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.320+0000 7f789cdd0700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f788006e940 0x7f7880070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:16.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.320+0000 7f789cdd0700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f788006e940 0x7f7880070e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7898068cc0 tx=0x7f788c01a040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:16.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.325+0000 7f78967fc700 1 -- 172.21.15.151:0/3239452142 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f78900573c0 con 0x7f78980ffbd0 2024-04-16T02:39:16.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:16.543+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29", "target": ["mon-mgr", ""]}) v1 -- 0x7f789802cce0 con 0x7f788006e940 2024-04-16T02:39:18.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:17 smithi151 ceph-mon[29003]: pgmap v76: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:18.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:17 smithi151 ceph-mon[29003]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:17 smithi153 ceph-mon[36747]: pgmap v76: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:17 smithi153 ceph-mon[36747]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:18.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.713+0000 7f78967fc700 1 -- 172.21.15.151:0/3239452142 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5391 (secure 0 0 0) 0x7f789802cce0 con 0x7f788006e940 2024-04-16T02:39:18.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f788006e940 msgr2=0x7f7880070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:18.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f788006e940 0x7f7880070e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7898068cc0 tx=0x7f788c01a040 comp rx=0 tx=0).stop 2024-04-16T02:39:18.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 msgr2=0x7f78981976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:18.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 0x7f78981976a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f789000ba50 tx=0x7f789000be10 comp rx=0 tx=0).stop 2024-04-16T02:39:18.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 shutdown_connections 2024-04-16T02:39:18.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f788006e940 0x7f7880070e00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:18.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78980ffbd0 0x7f78981976a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:18.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 --2- 172.21.15.151:0/3239452142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7898105a40 0x7f7898197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:18.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 >> 172.21.15.151:0/3239452142 conn(0x7f78980755a0 msgr2=0x7f78980fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:18.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 shutdown_connections 2024-04-16T02:39:18.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:18.715+0000 7f789f835700 1 -- 172.21.15.151:0/3239452142 wait complete. 2024-04-16T02:39:18.730 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:39:19.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph orch ps' 2024-04-16T02:39:19.649 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:20.100 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:19 smithi151 ceph-mon[29003]: pgmap v77: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:19 smithi153 ceph-mon[36747]: pgmap v77: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:20.989 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:20 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:39:21.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.183+0000 7f631e9ad700 1 -- 172.21.15.151:0/592268428 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ff530 msgr2=0x7f63180ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:21.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.183+0000 7f631e9ad700 1 --2- 172.21.15.151:0/592268428 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ff530 0x7f63180ff910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6308009910 tx=0x7f6308009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:21.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.183+0000 7f631e9ad700 1 -- 172.21.15.151:0/592268428 shutdown_connections 2024-04-16T02:39:21.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.183+0000 7f631e9ad700 1 --2- 172.21.15.151:0/592268428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63180ffee0 0x7f6318105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:21.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.183+0000 7f631e9ad700 1 --2- 172.21.15.151:0/592268428 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ff530 0x7f63180ff910 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:21.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.183+0000 7f631e9ad700 1 -- 172.21.15.151:0/592268428 >> 172.21.15.151:0/592268428 conn(0x7f6318074df0 msgr2=0x7f6318075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:21.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.183+0000 7f631e9ad700 1 -- 172.21.15.151:0/592268428 shutdown_connections 2024-04-16T02:39:21.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.183+0000 7f631e9ad700 1 -- 172.21.15.151:0/592268428 wait complete. 2024-04-16T02:39:21.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.183+0000 7f631e9ad700 1 Processor -- start 2024-04-16T02:39:21.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f631e9ad700 1 -- start start 2024-04-16T02:39:21.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f631e9ad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63180ff530 0x7f631819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:21.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f631e9ad700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ffee0 0x7f631819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f631e9ad700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631819d1d0 con 0x7f63180ffee0 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f631e9ad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6318196680 con 0x7f63180ff530 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f63177fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ffee0 0x7f631819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f63177fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ffee0 0x7f631819cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57248/0 (socket says 172.21.15.151:57248) 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f63177fe700 1 -- 172.21.15.151:0/3980737174 learned_addr learned my addr 172.21.15.151:0/3980737174 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f6317fff700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63180ff530 0x7f631819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f63177fe700 1 -- 172.21.15.151:0/3980737174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63180ff530 msgr2=0x7f631819c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f63177fe700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63180ff530 0x7f631819c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.184+0000 7f63177fe700 1 -- 172.21.15.151:0/3980737174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080095f0 con 0x7f63180ffee0 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.185+0000 7f63177fe700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ffee0 0x7f631819cb40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f630c00b480 tx=0x7f630c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.185+0000 7f63157fa700 1 -- 172.21.15.151:0/3980737174 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f630c011800 con 0x7f63180ffee0 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.185+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6318196960 con 0x7f63180ffee0 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.185+0000 7f63157fa700 1 -- 172.21.15.151:0/3980737174 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f630c011e20 con 0x7f63180ffee0 2024-04-16T02:39:21.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.185+0000 7f63157fa700 1 -- 172.21.15.151:0/3980737174 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f630c010ea0 con 0x7f63180ffee0 2024-04-16T02:39:21.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.185+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6318196eb0 con 0x7f63180ffee0 2024-04-16T02:39:21.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.186+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63181091e0 con 0x7f63180ffee0 2024-04-16T02:39:21.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.186+0000 7f63157fa700 1 -- 172.21.15.151:0/3980737174 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f630c010380 con 0x7f63180ffee0 2024-04-16T02:39:21.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.187+0000 7f63157fa700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f630006e940 0x7f6300070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:21.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.187+0000 7f63157fa700 1 -- 172.21.15.151:0/3980737174 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f630c08e370 con 0x7f63180ffee0 2024-04-16T02:39:21.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.187+0000 7f6317fff700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f630006e940 0x7f6300070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:21.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.187+0000 7f6317fff700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f630006e940 0x7f6300070e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6308006010 tx=0x7f630801a040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:21.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.192+0000 7f63157fa700 1 -- 172.21.15.151:0/3980737174 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f630c05b180 con 0x7f63180ffee0 2024-04-16T02:39:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:20 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:39:21.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.404+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f63181971c0 con 0x7f630006e940 2024-04-16T02:39:21.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.415+0000 7f63157fa700 1 -- 172.21.15.151:0/3980737174 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f63181971c0 con 0x7f630006e940 2024-04-16T02:39:21.418 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:39:21.418 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (57s) 25s ago 12m 17.5M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (12m) 25s ago 12m 8066k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (11m) 88s ago 11m 7884k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (12m) 25s ago 12m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (11m) 88s ago 11m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (34s) 25s ago 11m 52.6M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (6m) 25s ago 6m 121M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (6m) 25s ago 6m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (6m) 88s ago 6m 58.2M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (6m) 88s ago 6m 67.2M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (2m) 25s ago 13m 507M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (110s) 88s ago 11m 448M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:39:21.419 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (13m) 25s ago 13m 61.6M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (11m) 88s ago 11m 45.9M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (97s) 25s ago 12m 13.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (92s) 88s ago 11m 3678k - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (10m) 25s ago 10m 223M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (9m) 25s ago 9m 249M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (9m) 25s ago 9m 194M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (9m) 88s ago 9m 279M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (8m) 88s ago 8m 228M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (8m) 88s ago 8m 243M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:39:21.420 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (75s) 25s ago 11m 60.2M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:39:21.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f630006e940 msgr2=0x7f6300070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:21.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f630006e940 0x7f6300070e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6308006010 tx=0x7f630801a040 comp rx=0 tx=0).stop 2024-04-16T02:39:21.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ffee0 msgr2=0x7f631819cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:21.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ffee0 0x7f631819cb40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f630c00b480 tx=0x7f630c00b790 comp rx=0 tx=0).stop 2024-04-16T02:39:21.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 shutdown_connections 2024-04-16T02:39:21.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f630006e940 0x7f6300070e00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:21.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63180ff530 0x7f631819c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:21.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 --2- 172.21.15.151:0/3980737174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63180ffee0 0x7f631819cb40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:21.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 >> 172.21.15.151:0/3980737174 conn(0x7f6318074df0 msgr2=0x7f63180feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:21.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 shutdown_connections 2024-04-16T02:39:21.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:21.417+0000 7f631e9ad700 1 -- 172.21.15.151:0/3980737174 wait complete. 2024-04-16T02:39:21.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:21 smithi151 ceph-mon[29003]: pgmap v78: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:22.102 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T02:39:22.103 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:39:22.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-16T02:39:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:21 smithi153 ceph-mon[36747]: pgmap v78: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:22.327 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:22.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:22 smithi151 ceph-mon[29003]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:23.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:22 smithi153 ceph-mon[36747]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:23.971 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:23 smithi151 ceph-mon[29003]: pgmap v79: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:24.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.185+0000 7fd78529a700 1 -- 172.21.15.151:0/1386290827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 msgr2=0x7fd7801057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:24.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.185+0000 7fd78529a700 1 --2- 172.21.15.151:0/1386290827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 0x7fd7801057d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd774009910 tx=0x7fd774009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:24.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.185+0000 7fd78529a700 1 -- 172.21.15.151:0/1386290827 shutdown_connections 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.185+0000 7fd78529a700 1 --2- 172.21.15.151:0/1386290827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 0x7fd7801057d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.185+0000 7fd78529a700 1 --2- 172.21.15.151:0/1386290827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd780100dd0 0x7fd7801011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.185+0000 7fd78529a700 1 -- 172.21.15.151:0/1386290827 >> 172.21.15.151:0/1386290827 conn(0x7fd7800fc640 msgr2=0x7fd7800fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.185+0000 7fd78529a700 1 -- 172.21.15.151:0/1386290827 shutdown_connections 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.185+0000 7fd78529a700 1 -- 172.21.15.151:0/1386290827 wait complete. 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.185+0000 7fd78529a700 1 Processor -- start 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd78529a700 1 -- start start 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd78529a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd780100dd0 0x7fd78019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd78529a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 0x7fd78019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd78529a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78019d240 con 0x7fd780101780 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd78529a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801966f0 con 0x7fd780100dd0 2024-04-16T02:39:24.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd77e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 0x7fd78019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:24.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd77e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 0x7fd78019cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57258/0 (socket says 172.21.15.151:57258) 2024-04-16T02:39:24.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd77e7fc700 1 -- 172.21.15.151:0/934801462 learned_addr learned my addr 172.21.15.151:0/934801462 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:24.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd77effd700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd780100dd0 0x7fd78019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:24.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd77e7fc700 1 -- 172.21.15.151:0/934801462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd780100dd0 msgr2=0x7fd78019c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:24.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd77e7fc700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd780100dd0 0x7fd78019c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:24.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd77e7fc700 1 -- 172.21.15.151:0/934801462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7740095f0 con 0x7fd780101780 2024-04-16T02:39:24.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd77e7fc700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 0x7fd78019cbb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd774005f50 tx=0x7fd77400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:24.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd767fff700 1 -- 172.21.15.151:0/934801462 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd77401d070 con 0x7fd780101780 2024-04-16T02:39:24.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd780196970 con 0x7fd780101780 2024-04-16T02:39:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.186+0000 7fd767fff700 1 -- 172.21.15.151:0/934801462 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd77400f460 con 0x7fd780101780 2024-04-16T02:39:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.187+0000 7fd767fff700 1 -- 172.21.15.151:0/934801462 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd774017630 con 0x7fd780101780 2024-04-16T02:39:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.187+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd780196e60 con 0x7fd780101780 2024-04-16T02:39:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.187+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd780109120 con 0x7fd780101780 2024-04-16T02:39:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.188+0000 7fd767fff700 1 -- 172.21.15.151:0/934801462 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7fd77400f5d0 con 0x7fd780101780 2024-04-16T02:39:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.189+0000 7fd767fff700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd76806e940 0x7fd768070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.189+0000 7fd767fff700 1 -- 172.21.15.151:0/934801462 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fd77408fea0 con 0x7fd780101780 2024-04-16T02:39:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.189+0000 7fd77effd700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd76806e940 0x7fd768070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.189+0000 7fd77effd700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd76806e940 0x7fd768070e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd770006fd0 tx=0x7fd770008040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:24.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.194+0000 7fd767fff700 1 -- 172.21.15.151:0/934801462 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fd77405ccb0 con 0x7fd780101780 2024-04-16T02:39:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:23 smithi153 ceph-mon[36747]: pgmap v79: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:24.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.390+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7fd78004ea90 con 0x7fd780101780 2024-04-16T02:39:24.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.395+0000 7fd767fff700 1 -- 172.21.15.151:0/934801462 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v44)=0 v44) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd77405c840 con 0x7fd780101780 2024-04-16T02:39:24.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd76806e940 msgr2=0x7fd768070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd76806e940 0x7fd768070e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd770006fd0 tx=0x7fd770008040 comp rx=0 tx=0).stop 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 msgr2=0x7fd78019cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 0x7fd78019cbb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd774005f50 tx=0x7fd77400be20 comp rx=0 tx=0).stop 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 shutdown_connections 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fd76806e940 0x7fd768070e00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd780100dd0 0x7fd78019c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 --2- 172.21.15.151:0/934801462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd780101780 0x7fd78019cbb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 >> 172.21.15.151:0/934801462 conn(0x7fd7800fc640 msgr2=0x7fd7800fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 shutdown_connections 2024-04-16T02:39:24.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:24.397+0000 7fd78529a700 1 -- 172.21.15.151:0/934801462 wait complete. 2024-04-16T02:39:25.152 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T02:39:25.153 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:39:25.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-16T02:39:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:25.647 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:25 smithi151 ceph-mon[29003]: pgmap v80: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:25.647 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:25 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/934801462' entity='client.admin' 2024-04-16T02:39:25.647 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:25.647 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:25.647 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:25.647 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:25 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:25 smithi153 ceph-mon[36747]: pgmap v80: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:25 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/934801462' entity='client.admin' 2024-04-16T02:39:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:25 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:27.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.010+0000 7f3d5be4b700 1 -- 172.21.15.151:0/2351150043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d54105a40 msgr2=0x7f3d54105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.010+0000 7f3d5be4b700 1 --2- 172.21.15.151:0/2351150043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d54105a40 0x7f3d54105e20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009910 tx=0x7f3d50009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.010+0000 7f3d5be4b700 1 -- 172.21.15.151:0/2351150043 shutdown_connections 2024-04-16T02:39:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.010+0000 7f3d5be4b700 1 --2- 172.21.15.151:0/2351150043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d540ffa40 0x7f3d540ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.010+0000 7f3d5be4b700 1 --2- 172.21.15.151:0/2351150043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d54105a40 0x7f3d54105e20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.010+0000 7f3d5be4b700 1 -- 172.21.15.151:0/2351150043 >> 172.21.15.151:0/2351150043 conn(0x7f3d540fb570 msgr2=0x7f3d540fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.010+0000 7f3d5be4b700 1 -- 172.21.15.151:0/2351150043 shutdown_connections 2024-04-16T02:39:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.010+0000 7f3d5be4b700 1 -- 172.21.15.151:0/2351150043 wait complete. 2024-04-16T02:39:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d5be4b700 1 Processor -- start 2024-04-16T02:39:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d5be4b700 1 -- start start 2024-04-16T02:39:27.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d5be4b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d540ffa40 0x7f3d54197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:27.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d5be4b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54105a40 0x7f3d54197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:27.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d5be4b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d541980f0 con 0x7f3d540ffa40 2024-04-16T02:39:27.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d5be4b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d54198260 con 0x7f3d54105a40 2024-04-16T02:39:27.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d59be7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d540ffa40 0x7f3d54197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:27.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d59be7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d540ffa40 0x7f3d54197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41756/0 (socket says 172.21.15.151:41756) 2024-04-16T02:39:27.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d59be7700 1 -- 172.21.15.151:0/935876368 learned_addr learned my addr 172.21.15.151:0/935876368 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d593e6700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54105a40 0x7f3d54197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d59be7700 1 -- 172.21.15.151:0/935876368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54105a40 msgr2=0x7f3d54197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.011+0000 7f3d59be7700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54105a40 0x7f3d54197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.012+0000 7f3d59be7700 1 -- 172.21.15.151:0/935876368 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d500095f0 con 0x7f3d540ffa40 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.012+0000 7f3d593e6700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54105a40 0x7f3d54197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.012+0000 7f3d59be7700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d540ffa40 0x7f3d54197590 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006010 tx=0x7f3d5000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.012+0000 7f3d4affd700 1 -- 172.21.15.151:0/935876368 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d5001d070 con 0x7f3d540ffa40 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.012+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d541a0600 con 0x7f3d540ffa40 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.012+0000 7f3d4affd700 1 -- 172.21.15.151:0/935876368 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d5000f460 con 0x7f3d540ffa40 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.012+0000 7f3d4affd700 1 -- 172.21.15.151:0/935876368 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d50017630 con 0x7f3d540ffa40 2024-04-16T02:39:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.012+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d541a0a70 con 0x7f3d540ffa40 2024-04-16T02:39:27.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.013+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d54108040 con 0x7f3d540ffa40 2024-04-16T02:39:27.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.013+0000 7f3d4affd700 1 -- 172.21.15.151:0/935876368 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f3d5000f5d0 con 0x7f3d540ffa40 2024-04-16T02:39:27.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.014+0000 7f3d4affd700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f3d4006e940 0x7f3d40070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:27.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.014+0000 7f3d593e6700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f3d4006e940 0x7f3d40070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:27.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.014+0000 7f3d4affd700 1 -- 172.21.15.151:0/935876368 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3d5008fd40 con 0x7f3d540ffa40 2024-04-16T02:39:27.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.014+0000 7f3d593e6700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f3d4006e940 0x7f3d40070e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3d54069660 tx=0x7f3d440092a0 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:27.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.019+0000 7f3d4affd700 1 -- 172.21.15.151:0/935876368 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f3d50059020 con 0x7f3d540ffa40 2024-04-16T02:39:27.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.217+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f3d5402cd60 con 0x7f3d540ffa40 2024-04-16T02:39:27.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.218+0000 7f3d4affd700 1 -- 172.21.15.151:0/935876368 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v44)=0 v44) v1 ==== 155+0+0 (secure 0 0 0) 0x7f3d5005c6e0 con 0x7f3d540ffa40 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f3d4006e940 msgr2=0x7f3d40070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f3d4006e940 0x7f3d40070e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3d54069660 tx=0x7f3d440092a0 comp rx=0 tx=0).stop 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d540ffa40 msgr2=0x7f3d54197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d540ffa40 0x7f3d54197590 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006010 tx=0x7f3d5000be20 comp rx=0 tx=0).stop 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 shutdown_connections 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f3d4006e940 0x7f3d40070e00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d540ffa40 0x7f3d54197590 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 --2- 172.21.15.151:0/935876368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54105a40 0x7f3d54197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 >> 172.21.15.151:0/935876368 conn(0x7f3d540fb570 msgr2=0x7f3d540fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:27.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 shutdown_connections 2024-04-16T02:39:27.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:27.220+0000 7f3d5be4b700 1 -- 172.21.15.151:0/935876368 wait complete. 2024-04-16T02:39:27.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:27 smithi151 ceph-mon[29003]: pgmap v81: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:27 smithi153 ceph-mon[36747]: pgmap v81: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:27.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-16T02:39:28.124 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:29.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:29 smithi151 ceph-mon[29003]: pgmap v82: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:29.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 -- 172.21.15.151:0/1883645349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8100de0 msgr2=0x7f04b81011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:29.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 --2- 172.21.15.151:0/1883645349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8100de0 0x7f04b81011c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009910 tx=0x7f04b4009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:29.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 -- 172.21.15.151:0/1883645349 shutdown_connections 2024-04-16T02:39:29.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 --2- 172.21.15.151:0/1883645349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b8101790 0x7f04b81057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:29.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 --2- 172.21.15.151:0/1883645349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8100de0 0x7f04b81011c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:29.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 -- 172.21.15.151:0/1883645349 >> 172.21.15.151:0/1883645349 conn(0x7f04b80fc670 msgr2=0x7f04b80fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:29.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 -- 172.21.15.151:0/1883645349 shutdown_connections 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 -- 172.21.15.151:0/1883645349 wait complete. 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 Processor -- start 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.726+0000 7f04c064b700 1 -- start start 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04c064b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b8100de0 0x7f04b819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04c064b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8101790 0x7f04b819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04c064b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b819f1d0 con 0x7f04b8101790 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04c064b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b8198780 con 0x7f04b8100de0 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04bdbe6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8101790 0x7f04b819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04bdbe6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8101790 0x7f04b819ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41768/0 (socket says 172.21.15.151:41768) 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04bdbe6700 1 -- 172.21.15.151:0/2837062491 learned_addr learned my addr 172.21.15.151:0/2837062491 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04be3e7700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b8100de0 0x7f04b819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04bdbe6700 1 -- 172.21.15.151:0/2837062491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b8100de0 msgr2=0x7f04b819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:29.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04bdbe6700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b8100de0 0x7f04b819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:29.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04bdbe6700 1 -- 172.21.15.151:0/2837062491 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40095f0 con 0x7f04b8101790 2024-04-16T02:39:29.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04bdbe6700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8101790 0x7f04b819ebd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f04a800c960 tx=0x7f04a800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:29.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04af7fe700 1 -- 172.21.15.151:0/2837062491 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04a8007a70 con 0x7f04b8101790 2024-04-16T02:39:29.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b8198a60 con 0x7f04b8101790 2024-04-16T02:39:29.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04af7fe700 1 -- 172.21.15.151:0/2837062491 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04a8007bd0 con 0x7f04b8101790 2024-04-16T02:39:29.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.727+0000 7f04af7fe700 1 -- 172.21.15.151:0/2837062491 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04a80186c0 con 0x7f04b8101790 2024-04-16T02:39:29.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.728+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b8198fb0 con 0x7f04b8101790 2024-04-16T02:39:29.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.729+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b8104f60 con 0x7f04b8101790 2024-04-16T02:39:29.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.729+0000 7f04af7fe700 1 -- 172.21.15.151:0/2837062491 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f04a800faa0 con 0x7f04b8101790 2024-04-16T02:39:29.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.730+0000 7f04af7fe700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f04a406e940 0x7f04a4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:29.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.730+0000 7f04be3e7700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f04a406e940 0x7f04a4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:29.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.730+0000 7f04af7fe700 1 -- 172.21.15.151:0/2837062491 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f04a808e8b0 con 0x7f04b8101790 2024-04-16T02:39:29.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.730+0000 7f04be3e7700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f04a406e940 0x7f04a4070e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f04b400b5c0 tx=0x7f04b401a040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:29.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.734+0000 7f04af7fe700 1 -- 172.21.15.151:0/2837062491 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f04a805b430 con 0x7f04b8101790 2024-04-16T02:39:29.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:29 smithi153 ceph-mon[36747]: pgmap v82: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:29.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.931+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f04b804ea90 con 0x7f04b8101790 2024-04-16T02:39:29.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.931+0000 7f04af7fe700 1 -- 172.21.15.151:0/2837062491 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v44)=0 v44) v1 ==== 163+0+0 (secure 0 0 0) 0x7f04a805b250 con 0x7f04b8101790 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f04a406e940 msgr2=0x7f04a4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f04a406e940 0x7f04a4070e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f04b400b5c0 tx=0x7f04b401a040 comp rx=0 tx=0).stop 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8101790 msgr2=0x7f04b819ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8101790 0x7f04b819ebd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f04a800c960 tx=0x7f04a800cd20 comp rx=0 tx=0).stop 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 shutdown_connections 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f04a406e940 0x7f04a4070e00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b8100de0 0x7f04b819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 --2- 172.21.15.151:0/2837062491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b8101790 0x7f04b819ebd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 >> 172.21.15.151:0/2837062491 conn(0x7f04b80fc670 msgr2=0x7f04b80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 shutdown_connections 2024-04-16T02:39:29.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:29.934+0000 7f04c064b700 1 -- 172.21.15.151:0/2837062491 wait complete. 2024-04-16T02:39:30.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-16T02:39:30.860 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:32.175 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:31 smithi151 ceph-mon[29003]: pgmap v83: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:31 smithi153 ceph-mon[36747]: pgmap v83: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.516+0000 7fab6ae9b700 1 -- 172.21.15.151:0/1526899607 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ffee0 msgr2=0x7fab64105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.516+0000 7fab6ae9b700 1 --2- 172.21.15.151:0/1526899607 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ffee0 0x7fab64105870 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fab50009910 tx=0x7fab50009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.516+0000 7fab6ae9b700 1 -- 172.21.15.151:0/1526899607 shutdown_connections 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.516+0000 7fab6ae9b700 1 --2- 172.21.15.151:0/1526899607 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ffee0 0x7fab64105870 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.516+0000 7fab6ae9b700 1 --2- 172.21.15.151:0/1526899607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab640ff530 0x7fab640ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.517+0000 7fab6ae9b700 1 -- 172.21.15.151:0/1526899607 >> 172.21.15.151:0/1526899607 conn(0x7fab64074df0 msgr2=0x7fab64075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.517+0000 7fab6ae9b700 1 -- 172.21.15.151:0/1526899607 shutdown_connections 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.517+0000 7fab6ae9b700 1 -- 172.21.15.151:0/1526899607 wait complete. 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.517+0000 7fab6ae9b700 1 Processor -- start 2024-04-16T02:39:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.517+0000 7fab6ae9b700 1 -- start start 2024-04-16T02:39:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab6ae9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ff530 0x7fab6419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab6ae9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab640ffee0 0x7fab6419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab6ae9b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6419d2a0 con 0x7fab640ff530 2024-04-16T02:39:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab6ae9b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab64196750 con 0x7fab640ffee0 2024-04-16T02:39:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab68c37700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ff530 0x7fab6419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab68c37700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ff530 0x7fab6419c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41788/0 (socket says 172.21.15.151:41788) 2024-04-16T02:39:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab68c37700 1 -- 172.21.15.151:0/2137807022 learned_addr learned my addr 172.21.15.151:0/2137807022 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab68c37700 1 -- 172.21.15.151:0/2137807022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab640ffee0 msgr2=0x7fab6419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab68c37700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab640ffee0 0x7fab6419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab68c37700 1 -- 172.21.15.151:0/2137807022 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500095f0 con 0x7fab640ff530 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab68c37700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ff530 0x7fab6419c6d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fab5800b4b0 tx=0x7fab5800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab61ffb700 1 -- 172.21.15.151:0/2137807022 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab58011800 con 0x7fab640ff530 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab61ffb700 1 -- 172.21.15.151:0/2137807022 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab58011e20 con 0x7fab640ff530 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab64196a30 con 0x7fab640ff530 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.518+0000 7fab61ffb700 1 -- 172.21.15.151:0/2137807022 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab5800f2e0 con 0x7fab640ff530 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.519+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab64196f80 con 0x7fab640ff530 2024-04-16T02:39:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.520+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab64109170 con 0x7fab640ff530 2024-04-16T02:39:32.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.520+0000 7fab61ffb700 1 -- 172.21.15.151:0/2137807022 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7fab58010380 con 0x7fab640ff530 2024-04-16T02:39:32.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.521+0000 7fab61ffb700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fab5406e940 0x7fab54070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:32.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.521+0000 7fab61ffb700 1 -- 172.21.15.151:0/2137807022 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fab5808e3d0 con 0x7fab640ff530 2024-04-16T02:39:32.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.521+0000 7fab63fff700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fab5406e940 0x7fab54070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:32.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.521+0000 7fab63fff700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fab5406e940 0x7fab54070e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fab5000b5c0 tx=0x7fab50005960 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:32.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.525+0000 7fab61ffb700 1 -- 172.21.15.151:0/2137807022 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fab5805b310 con 0x7fab640ff530 2024-04-16T02:39:32.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.722+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fab6404ea90 con 0x7fab640ff530 2024-04-16T02:39:32.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.722+0000 7fab61ffb700 1 -- 172.21.15.151:0/2137807022 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v44)=0 v44) v1 ==== 135+0+0 (secure 0 0 0) 0x7fab5805aea0 con 0x7fab640ff530 2024-04-16T02:39:32.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fab5406e940 msgr2=0x7fab54070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:32.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fab5406e940 0x7fab54070e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fab5000b5c0 tx=0x7fab50005960 comp rx=0 tx=0).stop 2024-04-16T02:39:32.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ff530 msgr2=0x7fab6419c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:32.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ff530 0x7fab6419c6d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fab5800b4b0 tx=0x7fab5800b870 comp rx=0 tx=0).stop 2024-04-16T02:39:32.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 shutdown_connections 2024-04-16T02:39:32.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fab5406e940 0x7fab54070e00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:32.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab640ff530 0x7fab6419c6d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:32.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 --2- 172.21.15.151:0/2137807022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab640ffee0 0x7fab6419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:32.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 >> 172.21.15.151:0/2137807022 conn(0x7fab64074df0 msgr2=0x7fab640fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:32.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 shutdown_connections 2024-04-16T02:39:32.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:32.725+0000 7fab6ae9b700 1 -- 172.21.15.151:0/2137807022 wait complete. 2024-04-16T02:39:33.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-16T02:39:33.591 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:33.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:33 smithi151 ceph-mon[29003]: pgmap v84: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:34.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:33 smithi153 ceph-mon[36747]: pgmap v84: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:35.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.175+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1093330227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4101770 msgr2=0x7fcfc41057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:35.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.175+0000 7fcfc96b9700 1 --2- 172.21.15.151:0/1093330227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4101770 0x7fcfc41057c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009910 tx=0x7fcfb4009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.175+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1093330227 shutdown_connections 2024-04-16T02:39:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.175+0000 7fcfc96b9700 1 --2- 172.21.15.151:0/1093330227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4101770 0x7fcfc41057c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.175+0000 7fcfc96b9700 1 --2- 172.21.15.151:0/1093330227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfc4100dc0 0x7fcfc41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.175+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1093330227 >> 172.21.15.151:0/1093330227 conn(0x7fcfc40fc650 msgr2=0x7fcfc40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.175+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1093330227 shutdown_connections 2024-04-16T02:39:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.175+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1093330227 wait complete. 2024-04-16T02:39:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.176+0000 7fcfc96b9700 1 Processor -- start 2024-04-16T02:39:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.176+0000 7fcfc96b9700 1 -- start start 2024-04-16T02:39:35.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.176+0000 7fcfc96b9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4100dc0 0x7fcfc419c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:35.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.176+0000 7fcfc96b9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfc4101770 0x7fcfc419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:35.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.176+0000 7fcfc96b9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc419d150 con 0x7fcfc4100dc0 2024-04-16T02:39:35.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.176+0000 7fcfc96b9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc4196600 con 0x7fcfc4101770 2024-04-16T02:39:35.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4100dc0 0x7fcfc419c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:35.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4100dc0 0x7fcfc419c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41810/0 (socket says 172.21.15.151:41810) 2024-04-16T02:39:35.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc2ffd700 1 -- 172.21.15.151:0/1845610381 learned_addr learned my addr 172.21.15.151:0/1845610381 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc27fc700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfc4101770 0x7fcfc419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc2ffd700 1 -- 172.21.15.151:0/1845610381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfc4101770 msgr2=0x7fcfc419cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc2ffd700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfc4101770 0x7fcfc419cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc2ffd700 1 -- 172.21.15.151:0/1845610381 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb40095f0 con 0x7fcfc4100dc0 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc2ffd700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4100dc0 0x7fcfc419c580 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00d8d0 tx=0x7fcfac00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfbbfff700 1 -- 172.21.15.151:0/1845610381 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfac0098a0 con 0x7fcfc4100dc0 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfbbfff700 1 -- 172.21.15.151:0/1845610381 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfac010460 con 0x7fcfc4100dc0 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc41968e0 con 0x7fcfc4100dc0 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfbbfff700 1 -- 172.21.15.151:0/1845610381 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfac00f5f0 con 0x7fcfc4100dc0 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.177+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc4196e30 con 0x7fcfc4100dc0 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.178+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc41091c0 con 0x7fcfc4100dc0 2024-04-16T02:39:35.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.179+0000 7fcfbbfff700 1 -- 172.21.15.151:0/1845610381 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7fcfac009a00 con 0x7fcfc4100dc0 2024-04-16T02:39:35.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.179+0000 7fcfbbfff700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fcfb006e940 0x7fcfb0070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:35.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.179+0000 7fcfbbfff700 1 -- 172.21.15.151:0/1845610381 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fcfac08e770 con 0x7fcfc4100dc0 2024-04-16T02:39:35.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.179+0000 7fcfc27fc700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fcfb006e940 0x7fcfb0070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:35.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.180+0000 7fcfc27fc700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fcfb006e940 0x7fcfb0070e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40fcea0 tx=0x7fcfb401a040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:35.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.184+0000 7fcfbbfff700 1 -- 172.21.15.151:0/1845610381 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7fcfac05b580 con 0x7fcfc4100dc0 2024-04-16T02:39:35.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.407+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfc41971c0 con 0x7fcfb006e940 2024-04-16T02:39:35.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.413+0000 7fcfbbfff700 1 -- 172.21.15.151:0/1845610381 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fcfc41971c0 con 0x7fcfb006e940 2024-04-16T02:39:35.415 INFO:teuthology.orchestra.run.smithi151.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:39:35.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fcfb006e940 msgr2=0x7fcfb0070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fcfb006e940 0x7fcfb0070e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40fcea0 tx=0x7fcfb401a040 comp rx=0 tx=0).stop 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4100dc0 msgr2=0x7fcfc419c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4100dc0 0x7fcfc419c580 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00d8d0 tx=0x7fcfac00dbe0 comp rx=0 tx=0).stop 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 shutdown_connections 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7fcfb006e940 0x7fcfb0070e00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc4100dc0 0x7fcfc419c580 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 --2- 172.21.15.151:0/1845610381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfc4101770 0x7fcfc419cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 >> 172.21.15.151:0/1845610381 conn(0x7fcfc40fc650 msgr2=0x7fcfc40fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 shutdown_connections 2024-04-16T02:39:35.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:35.415+0000 7fcfc96b9700 1 -- 172.21.15.151:0/1845610381 wait complete. 2024-04-16T02:39:36.123 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:35 smithi153 ceph-mon[36747]: pgmap v85: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:36.124 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:39:36.124 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:36.124 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:36.124 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:36.124 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:36.124 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:35 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:36.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:35 smithi151 ceph-mon[29003]: pgmap v85: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:36.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:39:36.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:36.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:36.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:36.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:39:36.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:35 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:36.174 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T02:39:36.175 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T02:39:36.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-16T02:39:36.408 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T02:39:37.096 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:36 smithi151 ceph-mon[29003]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:37.096 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:36 smithi151 ceph-mon[29003]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:39:37.096 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:36 smithi151 ceph-mon[29003]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:39:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:36 smithi153 ceph-mon[36747]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:36 smithi153 ceph-mon[36747]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:39:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:36 smithi153 ceph-mon[36747]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:87dbfb632e974cb2c28d17423071d2d833b53b29 2024-04-16T02:39:37.976 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:37 smithi151 ceph-mon[29003]: pgmap v86: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:37 smithi153 ceph-mon[36747]: pgmap v86: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:39:38.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.863+0000 7f99038eb700 1 -- 172.21.15.151:0/787756581 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc101770 msgr2=0x7f98fc1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:38.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.863+0000 7f99038eb700 1 --2- 172.21.15.151:0/787756581 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc101770 0x7f98fc1057c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009910 tx=0x7f98f0009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.863+0000 7f99038eb700 1 -- 172.21.15.151:0/787756581 shutdown_connections 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.863+0000 7f99038eb700 1 --2- 172.21.15.151:0/787756581 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc101770 0x7f98fc1057c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.863+0000 7f99038eb700 1 --2- 172.21.15.151:0/787756581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98fc100dc0 0x7f98fc1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.863+0000 7f99038eb700 1 -- 172.21.15.151:0/787756581 >> 172.21.15.151:0/787756581 conn(0x7f98fc0fc670 msgr2=0x7f98fc0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.864+0000 7f99038eb700 1 -- 172.21.15.151:0/787756581 shutdown_connections 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.864+0000 7f99038eb700 1 -- 172.21.15.151:0/787756581 wait complete. 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.864+0000 7f99038eb700 1 Processor -- start 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.864+0000 7f99038eb700 1 -- start start 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.864+0000 7f99038eb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc100dc0 0x7f98fc19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.864+0000 7f99038eb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98fc101770 0x7f98fc19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.864+0000 7f99038eb700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc19d1f0 con 0x7f98fc100dc0 2024-04-16T02:39:38.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.864+0000 7f99038eb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc1966a0 con 0x7f98fc101770 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.864+0000 7f9901687700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc100dc0 0x7f98fc19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f9901687700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc100dc0 0x7f98fc19c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33448/0 (socket says 172.21.15.151:33448) 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f9901687700 1 -- 172.21.15.151:0/1474359516 learned_addr learned my addr 172.21.15.151:0/1474359516 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f9900e86700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98fc101770 0x7f98fc19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f9901687700 1 -- 172.21.15.151:0/1474359516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98fc101770 msgr2=0x7f98fc19cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f9901687700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98fc101770 0x7f98fc19cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f9901687700 1 -- 172.21.15.151:0/1474359516 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f00095f0 con 0x7f98fc100dc0 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f9900e86700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98fc101770 0x7f98fc19cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f9901687700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc100dc0 0x7f98fc19c620 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00ba50 tx=0x7f98ec00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f98fa7fc700 1 -- 172.21.15.151:0/1474359516 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98ec00ca00 con 0x7f98fc100dc0 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc196980 con 0x7f98fc100dc0 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f98fa7fc700 1 -- 172.21.15.151:0/1474359516 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98ec00cb60 con 0x7f98fc100dc0 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f98fa7fc700 1 -- 172.21.15.151:0/1474359516 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98ec0075a0 con 0x7f98fc100dc0 2024-04-16T02:39:38.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.865+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc196ed0 con 0x7f98fc100dc0 2024-04-16T02:39:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.866+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc109110 con 0x7f98fc100dc0 2024-04-16T02:39:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.867+0000 7f98fa7fc700 1 -- 172.21.15.151:0/1474359516 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f98ec00ccd0 con 0x7f98fc100dc0 2024-04-16T02:39:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.867+0000 7f98fa7fc700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f98e406e940 0x7f98e4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.867+0000 7f98fa7fc700 1 -- 172.21.15.151:0/1474359516 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f98ec006e40 con 0x7f98fc100dc0 2024-04-16T02:39:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.867+0000 7f9900e86700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f98e406e940 0x7f98e4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:38.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.868+0000 7f9900e86700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f98e406e940 0x7f98e4070e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f98f000b5c0 tx=0x7f98f0005480 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:38.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:38.872+0000 7f98fa7fc700 1 -- 172.21.15.151:0/1474359516 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f98ec05aef0 con 0x7f98fc100dc0 2024-04-16T02:39:39.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.099+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98fc1971c0 con 0x7f98e406e940 2024-04-16T02:39:39.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.101+0000 7f98fa7fc700 1 -- 172.21.15.151:0/1474359516 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f98fc1971c0 con 0x7f98e406e940 2024-04-16T02:39:39.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f98e406e940 msgr2=0x7f98e4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f98e406e940 0x7f98e4070e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f98f000b5c0 tx=0x7f98f0005480 comp rx=0 tx=0).stop 2024-04-16T02:39:39.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc100dc0 msgr2=0x7f98fc19c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc100dc0 0x7f98fc19c620 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00ba50 tx=0x7f98ec00be10 comp rx=0 tx=0).stop 2024-04-16T02:39:39.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 shutdown_connections 2024-04-16T02:39:39.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f98e406e940 0x7f98e4070e00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98fc100dc0 0x7f98fc19c620 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 --2- 172.21.15.151:0/1474359516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98fc101770 0x7f98fc19cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 >> 172.21.15.151:0/1474359516 conn(0x7f98fc0fc670 msgr2=0x7f98fc0fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:39.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 shutdown_connections 2024-04-16T02:39:39.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.103+0000 7f99038eb700 1 -- 172.21.15.151:0/1474359516 wait complete. 2024-04-16T02:39:39.116 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:39:39.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.211+0000 7ff03e3bf700 1 -- 172.21.15.151:0/2496394654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 msgr2=0x7ff0380ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.211+0000 7ff03e3bf700 1 --2- 172.21.15.151:0/2496394654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 0x7ff0380ffeb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff020009910 tx=0x7ff020009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:39.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.211+0000 7ff03e3bf700 1 -- 172.21.15.151:0/2496394654 shutdown_connections 2024-04-16T02:39:39.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.211+0000 7ff03e3bf700 1 --2- 172.21.15.151:0/2496394654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 0x7ff0380ffeb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.211+0000 7ff03e3bf700 1 --2- 172.21.15.151:0/2496394654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038105a30 0x7ff038105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.211+0000 7ff03e3bf700 1 -- 172.21.15.151:0/2496394654 >> 172.21.15.151:0/2496394654 conn(0x7ff0380fb540 msgr2=0x7ff0380fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:39.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.211+0000 7ff03e3bf700 1 -- 172.21.15.151:0/2496394654 shutdown_connections 2024-04-16T02:39:39.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.211+0000 7ff03e3bf700 1 -- 172.21.15.151:0/2496394654 wait complete. 2024-04-16T02:39:39.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff03e3bf700 1 Processor -- start 2024-04-16T02:39:39.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff03e3bf700 1 -- start start 2024-04-16T02:39:39.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff03e3bf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 0x7ff03819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:39.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff03e3bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038105a30 0x7ff03819cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:39.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff03e3bf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03819d1c0 con 0x7ff0380ffa30 2024-04-16T02:39:39.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff03e3bf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff038196670 con 0x7ff038105a30 2024-04-16T02:39:39.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff037fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 0x7ff03819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff037fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 0x7ff03819c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33470/0 (socket says 172.21.15.151:33470) 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff037fff700 1 -- 172.21.15.151:0/656217909 learned_addr learned my addr 172.21.15.151:0/656217909 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff037fff700 1 -- 172.21.15.151:0/656217909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038105a30 msgr2=0x7ff03819cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff02ffff700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038105a30 0x7ff03819cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff037fff700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038105a30 0x7ff03819cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.212+0000 7ff037fff700 1 -- 172.21.15.151:0/656217909 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0200095f0 con 0x7ff0380ffa30 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.213+0000 7ff037fff700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 0x7ff03819c5f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff02800b4b0 tx=0x7ff02800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.213+0000 7ff035ffb700 1 -- 172.21.15.151:0/656217909 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff028011800 con 0x7ff0380ffa30 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.213+0000 7ff035ffb700 1 -- 172.21.15.151:0/656217909 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff028011e20 con 0x7ff0380ffa30 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.213+0000 7ff035ffb700 1 -- 172.21.15.151:0/656217909 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff02800f2e0 con 0x7ff0380ffa30 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.213+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff038196950 con 0x7ff0380ffa30 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.213+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff038196ea0 con 0x7ff0380ffa30 2024-04-16T02:39:39.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.214+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff038109060 con 0x7ff0380ffa30 2024-04-16T02:39:39.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.214+0000 7ff035ffb700 1 -- 172.21.15.151:0/656217909 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7ff028010380 con 0x7ff0380ffa30 2024-04-16T02:39:39.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.215+0000 7ff035ffb700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7ff02406e940 0x7ff024070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:39.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.215+0000 7ff035ffb700 1 -- 172.21.15.151:0/656217909 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7ff02808e440 con 0x7ff0380ffa30 2024-04-16T02:39:39.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.215+0000 7ff02ffff700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7ff02406e940 0x7ff024070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:39.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.215+0000 7ff02ffff700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7ff02406e940 0x7ff024070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff020006010 tx=0x7ff02000b540 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:39.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.219+0000 7ff035ffb700 1 -- 172.21.15.151:0/656217909 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7ff02805b250 con 0x7ff0380ffa30 2024-04-16T02:39:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.446+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff038197170 con 0x7ff02406e940 2024-04-16T02:39:39.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.448+0000 7ff035ffb700 1 -- 172.21.15.151:0/656217909 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7ff038197170 con 0x7ff02406e940 2024-04-16T02:39:39.452 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7ff02406e940 msgr2=0x7ff024070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7ff02406e940 0x7ff024070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff020006010 tx=0x7ff02000b540 comp rx=0 tx=0).stop 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 msgr2=0x7ff03819c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 0x7ff03819c5f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff02800b4b0 tx=0x7ff02800b870 comp rx=0 tx=0).stop 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 shutdown_connections 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7ff02406e940 0x7ff024070e00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0380ffa30 0x7ff03819c5f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 --2- 172.21.15.151:0/656217909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038105a30 0x7ff03819cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 >> 172.21.15.151:0/656217909 conn(0x7ff0380fb540 msgr2=0x7ff0380690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 shutdown_connections 2024-04-16T02:39:39.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.450+0000 7ff03e3bf700 1 -- 172.21.15.151:0/656217909 wait complete. 2024-04-16T02:39:39.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.560+0000 7f702b5e9700 1 -- 172.21.15.151:0/266558932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024101790 msgr2=0x7f70241057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.560+0000 7f702b5e9700 1 --2- 172.21.15.151:0/266558932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024101790 0x7f70241057e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7014009910 tx=0x7f7014009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:39.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.560+0000 7f702b5e9700 1 -- 172.21.15.151:0/266558932 shutdown_connections 2024-04-16T02:39:39.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.560+0000 7f702b5e9700 1 --2- 172.21.15.151:0/266558932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024101790 0x7f70241057e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.560+0000 7f702b5e9700 1 --2- 172.21.15.151:0/266558932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024100de0 0x7f70241011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.560+0000 7f702b5e9700 1 -- 172.21.15.151:0/266558932 >> 172.21.15.151:0/266558932 conn(0x7f70240fc670 msgr2=0x7f70240fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:39.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.560+0000 7f702b5e9700 1 -- 172.21.15.151:0/266558932 shutdown_connections 2024-04-16T02:39:39.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.560+0000 7f702b5e9700 1 -- 172.21.15.151:0/266558932 wait complete. 2024-04-16T02:39:39.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f702b5e9700 1 Processor -- start 2024-04-16T02:39:39.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f702b5e9700 1 -- start start 2024-04-16T02:39:39.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f702b5e9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024100de0 0x7f702419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:39.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f702b5e9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024101790 0x7f702419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:39.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f702b5e9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702419d130 con 0x7f7024100de0 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f702b5e9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70241965e0 con 0x7f7024101790 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f7029385700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024100de0 0x7f702419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f7029385700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024100de0 0x7f702419c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33480/0 (socket says 172.21.15.151:33480) 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f7029385700 1 -- 172.21.15.151:0/4044150207 learned_addr learned my addr 172.21.15.151:0/4044150207 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f7028b84700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024101790 0x7f702419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f7029385700 1 -- 172.21.15.151:0/4044150207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024101790 msgr2=0x7f702419caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f7029385700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024101790 0x7f702419caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.561+0000 7f7029385700 1 -- 172.21.15.151:0/4044150207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70140095f0 con 0x7f7024100de0 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.562+0000 7f7029385700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024100de0 0x7f702419c560 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f702000d8d0 tx=0x7f702000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.562+0000 7f701a7fc700 1 -- 172.21.15.151:0/4044150207 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7020009940 con 0x7f7024100de0 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.562+0000 7f701a7fc700 1 -- 172.21.15.151:0/4044150207 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7020010460 con 0x7f7024100de0 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.562+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70241968c0 con 0x7f7024100de0 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.562+0000 7f701a7fc700 1 -- 172.21.15.151:0/4044150207 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702000f5f0 con 0x7f7024100de0 2024-04-16T02:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.562+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7024196e10 con 0x7f7024100de0 2024-04-16T02:39:39.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.563+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70241091e0 con 0x7f7024100de0 2024-04-16T02:39:39.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.563+0000 7f701a7fc700 1 -- 172.21.15.151:0/4044150207 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f7020009aa0 con 0x7f7024100de0 2024-04-16T02:39:39.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.564+0000 7f701a7fc700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f701006e940 0x7f7010070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:39.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.564+0000 7f701a7fc700 1 -- 172.21.15.151:0/4044150207 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f702008e890 con 0x7f7024100de0 2024-04-16T02:39:39.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.564+0000 7f7028b84700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f701006e940 0x7f7010070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:39.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.565+0000 7f7028b84700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f701006e940 0x7f7010070e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70140059d0 tx=0x7f7014005960 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:39.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.569+0000 7f701a7fc700 1 -- 172.21.15.151:0/4044150207 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f702005b6a0 con 0x7f7024100de0 2024-04-16T02:39:39.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.783+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f70241970f0 con 0x7f701006e940 2024-04-16T02:39:39.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.794+0000 7f701a7fc700 1 -- 172.21.15.151:0/4044150207 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f70241970f0 con 0x7f701006e940 2024-04-16T02:39:39.797 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:39:39.797 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (75s) 44s ago 12m 17.5M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:39:39.797 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (12m) 44s ago 12m 8066k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:39:39.797 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (11m) 106s ago 11m 7884k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:39:39.797 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (12m) 44s ago 12m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:39:39.797 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (11m) 106s ago 11m 6651k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (52s) 44s ago 12m 52.6M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (7m) 44s ago 7m 121M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (6m) 44s ago 6m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (6m) 106s ago 6m 58.2M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (6m) 106s ago 6m 67.2M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (3m) 44s ago 13m 507M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (2m) 106s ago 11m 448M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (14m) 44s ago 14m 61.6M 2048M 18.2.1 5be31c24972a 051460b1db9f 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (11m) 106s ago 11m 45.9M 2048M 18.2.1 5be31c24972a 5f3186f6d9f9 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (116s) 44s ago 12m 13.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (111s) 106s ago 11m 3678k - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (10m) 44s ago 10m 223M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (10m) 44s ago 10m 249M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (9m) 44s ago 9m 194M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (9m) 106s ago 9m 279M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:39:39.798 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (8m) 106s ago 8m 228M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:39:39.799 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (8m) 106s ago 8m 243M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:39:39.799 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (93s) 44s ago 12m 60.2M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:39:39.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f701006e940 msgr2=0x7f7010070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f701006e940 0x7f7010070e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70140059d0 tx=0x7f7014005960 comp rx=0 tx=0).stop 2024-04-16T02:39:39.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024100de0 msgr2=0x7f702419c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024100de0 0x7f702419c560 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f702000d8d0 tx=0x7f702000dc90 comp rx=0 tx=0).stop 2024-04-16T02:39:39.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 shutdown_connections 2024-04-16T02:39:39.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f701006e940 0x7f7010070e00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7024100de0 0x7f702419c560 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 --2- 172.21.15.151:0/4044150207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024101790 0x7f702419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 >> 172.21.15.151:0/4044150207 conn(0x7f70240fc670 msgr2=0x7f70240fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:39.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 shutdown_connections 2024-04-16T02:39:39.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.797+0000 7f702b5e9700 1 -- 172.21.15.151:0/4044150207 wait complete. 2024-04-16T02:39:39.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.911+0000 7f1101768700 1 -- 172.21.15.151:0/4041104470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 msgr2=0x7f10fc0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.911+0000 7f1101768700 1 --2- 172.21.15.151:0/4041104470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 0x7f10fc0ffed0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009910 tx=0x7f10f0009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:39.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.911+0000 7f1101768700 1 -- 172.21.15.151:0/4041104470 shutdown_connections 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.911+0000 7f1101768700 1 --2- 172.21.15.151:0/4041104470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 0x7f10fc0ffed0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.911+0000 7f1101768700 1 --2- 172.21.15.151:0/4041104470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc105a50 0x7f10fc105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.911+0000 7f1101768700 1 -- 172.21.15.151:0/4041104470 >> 172.21.15.151:0/4041104470 conn(0x7f10fc0fb540 msgr2=0x7f10fc0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.911+0000 7f1101768700 1 -- 172.21.15.151:0/4041104470 shutdown_connections 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.911+0000 7f1101768700 1 -- 172.21.15.151:0/4041104470 wait complete. 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f1101768700 1 Processor -- start 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f1101768700 1 -- start start 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f1101768700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 0x7f10fc197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f1101768700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc105a50 0x7f10fc197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f1101768700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc198190 con 0x7f10fc0ffa50 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f1101768700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1982d0 con 0x7f10fc105a50 2024-04-16T02:39:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f10faffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 0x7f10fc197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f10faffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 0x7f10fc197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33490/0 (socket says 172.21.15.151:33490) 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f10faffd700 1 -- 172.21.15.151:0/2267136498 learned_addr learned my addr 172.21.15.151:0/2267136498 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.912+0000 7f10fa7fc700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc105a50 0x7f10fc197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.913+0000 7f10faffd700 1 -- 172.21.15.151:0/2267136498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc105a50 msgr2=0x7f10fc197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.913+0000 7f10faffd700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc105a50 0x7f10fc197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.913+0000 7f10faffd700 1 -- 172.21.15.151:0/2267136498 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00095f0 con 0x7f10fc0ffa50 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.913+0000 7f10faffd700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 0x7f10fc197570 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00c8f0 tx=0x7f10ec00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.913+0000 7f10e3fff700 1 -- 172.21.15.151:0/2267136498 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10ec004a00 con 0x7f10fc0ffa50 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.913+0000 7f10e3fff700 1 -- 172.21.15.151:0/2267136498 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10ec0054e0 con 0x7f10fc0ffa50 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.913+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc1a05e0 con 0x7f10fc0ffa50 2024-04-16T02:39:39.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.913+0000 7f10e3fff700 1 -- 172.21.15.151:0/2267136498 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10ec007cf0 con 0x7f10fc0ffa50 2024-04-16T02:39:39.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.913+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc1a0b00 con 0x7f10fc0ffa50 2024-04-16T02:39:39.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.914+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc04ea90 con 0x7f10fc0ffa50 2024-04-16T02:39:39.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.914+0000 7f10e3fff700 1 -- 172.21.15.151:0/2267136498 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f10ec005020 con 0x7f10fc0ffa50 2024-04-16T02:39:39.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.915+0000 7f10e3fff700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f10e406e940 0x7f10e4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:39.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.915+0000 7f10e3fff700 1 -- 172.21.15.151:0/2267136498 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f10ec013070 con 0x7f10fc0ffa50 2024-04-16T02:39:39.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.915+0000 7f10fa7fc700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f10e406e940 0x7f10e4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:39.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.915+0000 7f10fa7fc700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f10e406e940 0x7f10e4070e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10fc069e10 tx=0x7f10f001a040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:39.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:39.920+0000 7f10e3fff700 1 -- 172.21.15.151:0/2267136498 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f10ec0579c0 con 0x7f10fc0ffa50 2024-04-16T02:39:40.118 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:39 smithi151 ceph-mon[29003]: pgmap v87: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:40.118 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:40.118 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:40.118 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:40.119 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:40.119 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:39 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T02:39:40.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.211+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f10fc1a0f50 con 0x7f10fc0ffa50 2024-04-16T02:39:40.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.212+0000 7f10e3fff700 1 -- 172.21.15.151:0/2267136498 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f10ec05b080 con 0x7f10fc0ffa50 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:39:40.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:39:40.216 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:39:40.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:39:40.216 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:39:40.216 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-16T02:39:40.216 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:39:40.216 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:39:40.216 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f10e406e940 msgr2=0x7f10e4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f10e406e940 0x7f10e4070e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10fc069e10 tx=0x7f10f001a040 comp rx=0 tx=0).stop 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 msgr2=0x7f10fc197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 0x7f10fc197570 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00c8f0 tx=0x7f10ec00cc00 comp rx=0 tx=0).stop 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 shutdown_connections 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f10e406e940 0x7f10e4070e00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f10fc0ffa50 0x7f10fc197570 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 --2- 172.21.15.151:0/2267136498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc105a50 0x7f10fc197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 >> 172.21.15.151:0/2267136498 conn(0x7f10fc0fb540 msgr2=0x7f10fc0fbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:40.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 shutdown_connections 2024-04-16T02:39:40.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.214+0000 7f1101768700 1 -- 172.21.15.151:0/2267136498 wait complete. 2024-04-16T02:39:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:39 smithi153 ceph-mon[36747]: pgmap v87: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:39 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/1885944121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38101770 msgr2=0x7f6b381057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 --2- 172.21.15.151:0/1885944121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38101770 0x7f6b381057c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009910 tx=0x7f6b2c009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/1885944121 shutdown_connections 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 --2- 172.21.15.151:0/1885944121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38101770 0x7f6b381057c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 --2- 172.21.15.151:0/1885944121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b38100dc0 0x7f6b381011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/1885944121 >> 172.21.15.151:0/1885944121 conn(0x7f6b380fc670 msgr2=0x7f6b380fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/1885944121 shutdown_connections 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/1885944121 wait complete. 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 Processor -- start 2024-04-16T02:39:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.324+0000 7f6b3e6fb700 1 -- start start 2024-04-16T02:39:40.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b3e6fb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38100dc0 0x7f6b3819c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:40.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b3e6fb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b38101770 0x7f6b3819cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:40.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b3e6fb700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3819d0c0 con 0x7f6b38100dc0 2024-04-16T02:39:40.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b3e6fb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b38196600 con 0x7f6b38101770 2024-04-16T02:39:40.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b37fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38100dc0 0x7f6b3819c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:40.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b37fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38100dc0 0x7f6b3819c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33510/0 (socket says 172.21.15.151:33510) 2024-04-16T02:39:40.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b37fff700 1 -- 172.21.15.151:0/2281246915 learned_addr learned my addr 172.21.15.151:0/2281246915 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:40.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b377fe700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b38101770 0x7f6b3819cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:40.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b37fff700 1 -- 172.21.15.151:0/2281246915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b38101770 msgr2=0x7f6b3819cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b37fff700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b38101770 0x7f6b3819cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.325+0000 7f6b37fff700 1 -- 172.21.15.151:0/2281246915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c0095f0 con 0x7f6b38100dc0 2024-04-16T02:39:40.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.326+0000 7f6b37fff700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38100dc0 0x7f6b3819c580 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800c930 tx=0x7f6b2800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:40.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.326+0000 7f6b357fa700 1 -- 172.21.15.151:0/2281246915 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b28007ad0 con 0x7f6b38100dc0 2024-04-16T02:39:40.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.326+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b381968e0 con 0x7f6b38100dc0 2024-04-16T02:39:40.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.326+0000 7f6b357fa700 1 -- 172.21.15.151:0/2281246915 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b2800ce80 con 0x7f6b38100dc0 2024-04-16T02:39:40.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.326+0000 7f6b357fa700 1 -- 172.21.15.151:0/2281246915 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b28018630 con 0x7f6b38100dc0 2024-04-16T02:39:40.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.326+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b38196e30 con 0x7f6b38100dc0 2024-04-16T02:39:40.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.327+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b381091c0 con 0x7f6b38100dc0 2024-04-16T02:39:40.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.328+0000 7f6b357fa700 1 -- 172.21.15.151:0/2281246915 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f6b28007c30 con 0x7f6b38100dc0 2024-04-16T02:39:40.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.328+0000 7f6b357fa700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6b2006e940 0x7f6b20070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:40.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.328+0000 7f6b357fa700 1 -- 172.21.15.151:0/2281246915 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f6b2808e900 con 0x7f6b38100dc0 2024-04-16T02:39:40.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.328+0000 7f6b377fe700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6b2006e940 0x7f6b20070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:40.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.329+0000 7f6b377fe700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6b2006e940 0x7f6b20070e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6b380fce40 tx=0x7f6b2c01a040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:40.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.337+0000 7f6b357fa700 1 -- 172.21.15.151:0/2281246915 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f6b2805b710 con 0x7f6b38100dc0 2024-04-16T02:39:40.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.590+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6b3804ea90 con 0x7f6b38100dc0 2024-04-16T02:39:40.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.591+0000 7f6b357fa700 1 -- 172.21.15.151:0/2281246915 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1885 (secure 0 0 0) 0x7f6b2805b2a0 con 0x7f6b38100dc0 2024-04-16T02:39:40.594 INFO:teuthology.orchestra.run.smithi151.stdout:e20 2024-04-16T02:39:40.594 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:39:40.594 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:39:40.594 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:39:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 20 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:36:05.787890+0000 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:39:40.595 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 export targets 1 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:39:40.596 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 export targets 0 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:39:40.597 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:39:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:39:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:39:40.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6b2006e940 msgr2=0x7f6b20070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6b2006e940 0x7f6b20070e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6b380fce40 tx=0x7f6b2c01a040 comp rx=0 tx=0).stop 2024-04-16T02:39:40.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38100dc0 msgr2=0x7f6b3819c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38100dc0 0x7f6b3819c580 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800c930 tx=0x7f6b2800cc40 comp rx=0 tx=0).stop 2024-04-16T02:39:40.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 shutdown_connections 2024-04-16T02:39:40.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6b2006e940 0x7f6b20070e00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b38100dc0 0x7f6b3819c580 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 --2- 172.21.15.151:0/2281246915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b38101770 0x7f6b3819cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 >> 172.21.15.151:0/2281246915 conn(0x7f6b380fc670 msgr2=0x7f6b380fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:40.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 shutdown_connections 2024-04-16T02:39:40.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.593+0000 7f6b3e6fb700 1 -- 172.21.15.151:0/2281246915 wait complete. 2024-04-16T02:39:40.598 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 20 2024-04-16T02:39:40.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.702+0000 7f6fcebee700 1 -- 172.21.15.151:0/3435388499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 msgr2=0x7f6fc8068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.702+0000 7f6fcebee700 1 --2- 172.21.15.151:0/3435388499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 0x7f6fc8068a50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009910 tx=0x7f6fc4009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:40.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.702+0000 7f6fcebee700 1 -- 172.21.15.151:0/3435388499 shutdown_connections 2024-04-16T02:39:40.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.702+0000 7f6fcebee700 1 --2- 172.21.15.151:0/3435388499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fc8069020 0x7f6fc8105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.702+0000 7f6fcebee700 1 --2- 172.21.15.151:0/3435388499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 0x7f6fc8068a50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.702+0000 7f6fcebee700 1 -- 172.21.15.151:0/3435388499 >> 172.21.15.151:0/3435388499 conn(0x7f6fc80755a0 msgr2=0x7f6fc80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:40.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.702+0000 7f6fcebee700 1 -- 172.21.15.151:0/3435388499 shutdown_connections 2024-04-16T02:39:40.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.702+0000 7f6fcebee700 1 -- 172.21.15.151:0/3435388499 wait complete. 2024-04-16T02:39:40.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fcebee700 1 Processor -- start 2024-04-16T02:39:40.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fcebee700 1 -- start start 2024-04-16T02:39:40.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fcebee700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 0x7f6fc819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:40.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fcebee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fc8069020 0x7f6fc819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:40.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fcebee700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc819d290 con 0x7f6fc8068670 2024-04-16T02:39:40.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fcebee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc8196740 con 0x7f6fc8069020 2024-04-16T02:39:40.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fcc98a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 0x7f6fc819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fcc98a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 0x7f6fc819c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33520/0 (socket says 172.21.15.151:33520) 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fcc98a700 1 -- 172.21.15.151:0/2878375506 learned_addr learned my addr 172.21.15.151:0/2878375506 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.703+0000 7f6fbffff700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fc8069020 0x7f6fc819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.704+0000 7f6fcc98a700 1 -- 172.21.15.151:0/2878375506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fc8069020 msgr2=0x7f6fc819cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.704+0000 7f6fcc98a700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fc8069020 0x7f6fc819cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.704+0000 7f6fcc98a700 1 -- 172.21.15.151:0/2878375506 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc40095f0 con 0x7f6fc8068670 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.704+0000 7f6fbffff700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fc8069020 0x7f6fc819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.704+0000 7f6fcc98a700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 0x7f6fc819c6c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4006010 tx=0x7f6fc400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.704+0000 7f6fbdffb700 1 -- 172.21.15.151:0/2878375506 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc401d070 con 0x7f6fc8068670 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.704+0000 7f6fbdffb700 1 -- 172.21.15.151:0/2878375506 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fc4004fb0 con 0x7f6fc8068670 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.704+0000 7f6fbdffb700 1 -- 172.21.15.151:0/2878375506 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc4022ea0 con 0x7f6fc8068670 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.704+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc81969c0 con 0x7f6fc8068670 2024-04-16T02:39:40.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.705+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc8196eb0 con 0x7f6fc8068670 2024-04-16T02:39:40.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.706+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc8109130 con 0x7f6fc8068670 2024-04-16T02:39:40.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.706+0000 7f6fbdffb700 1 -- 172.21.15.151:0/2878375506 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f6fc4022470 con 0x7f6fc8068670 2024-04-16T02:39:40.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.707+0000 7f6fbdffb700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6fb006e940 0x7f6fb0070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:40.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.707+0000 7f6fbffff700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6fb006e940 0x7f6fb0070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:40.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.707+0000 7f6fbdffb700 1 -- 172.21.15.151:0/2878375506 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f6fc408fe30 con 0x7f6fc8068670 2024-04-16T02:39:40.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.707+0000 7f6fbffff700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6fb006e940 0x7f6fb0070e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8197bb0 tx=0x7f6fb8008040 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:40.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.712+0000 7f6fbdffb700 1 -- 172.21.15.151:0/2878375506 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f6fc405ccf0 con 0x7f6fc8068670 2024-04-16T02:39:40.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.936+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 --> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fc802cce0 con 0x7f6fb006e940 2024-04-16T02:39:40.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.937+0000 7f6fbdffb700 1 -- 172.21.15.151:0/2878375506 <== mgr.14692 v2:172.21.15.151:6800/1243156277 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f6fc802cce0 con 0x7f6fb006e940 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr" 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "2/23 daemons upgraded", 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mon daemons", 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:39:40.940 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:39:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.939+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6fb006e940 msgr2=0x7f6fb0070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.939+0000 7f6fcebee700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6fb006e940 0x7f6fb0070e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8197bb0 tx=0x7f6fb8008040 comp rx=0 tx=0).stop 2024-04-16T02:39:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.939+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 msgr2=0x7f6fc819c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.939+0000 7f6fcebee700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 0x7f6fc819c6c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4006010 tx=0x7f6fc400be20 comp rx=0 tx=0).stop 2024-04-16T02:39:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.939+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 shutdown_connections 2024-04-16T02:39:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.939+0000 7f6fcebee700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f6fb006e940 0x7f6fb0070e00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.939+0000 7f6fcebee700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6fc8068670 0x7f6fc819c6c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.939+0000 7f6fcebee700 1 --2- 172.21.15.151:0/2878375506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fc8069020 0x7f6fc819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:40.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.939+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 >> 172.21.15.151:0/2878375506 conn(0x7f6fc80755a0 msgr2=0x7f6fc80fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:40.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.940+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 shutdown_connections 2024-04-16T02:39:40.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:40.940+0000 7f6fcebee700 1 -- 172.21.15.151:0/2878375506 wait complete. 2024-04-16T02:39:40.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:40 smithi151 ceph-mon[29003]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:40.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:40 smithi151 ceph-mon[29003]: Upgrade: Target is version 19.0.0-3084-g87dbfb63 (squid) 2024-04-16T02:39:40.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:40 smithi151 ceph-mon[29003]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f'] 2024-04-16T02:39:40.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:40 smithi151 ceph-mon[29003]: Upgrade: Setting container_image for all mgr 2024-04-16T02:39:40.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:40 smithi151 ceph-mon[29003]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:40.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:40 smithi151 ceph-mon[29003]: from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:40.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:40 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2267136498' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:40.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:40 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2281246915' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:39:41.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.049+0000 7f45d2baf700 1 -- 172.21.15.151:0/1577630646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc105a40 msgr2=0x7f45cc105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:41.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.049+0000 7f45d2baf700 1 --2- 172.21.15.151:0/1577630646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc105a40 0x7f45cc105e20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009910 tx=0x7f45c0009c20 comp rx=0 tx=0).stop 2024-04-16T02:39:41.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.049+0000 7f45d2baf700 1 -- 172.21.15.151:0/1577630646 shutdown_connections 2024-04-16T02:39:41.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.049+0000 7f45d2baf700 1 --2- 172.21.15.151:0/1577630646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc0ffa40 0x7f45cc0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:41.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.049+0000 7f45d2baf700 1 --2- 172.21.15.151:0/1577630646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc105a40 0x7f45cc105e20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.049+0000 7f45d2baf700 1 -- 172.21.15.151:0/1577630646 >> 172.21.15.151:0/1577630646 conn(0x7f45cc0fb570 msgr2=0x7f45cc0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.049+0000 7f45d2baf700 1 -- 172.21.15.151:0/1577630646 shutdown_connections 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.049+0000 7f45d2baf700 1 -- 172.21.15.151:0/1577630646 wait complete. 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d2baf700 1 Processor -- start 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d2baf700 1 -- start start 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d2baf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc0ffa40 0x7f45cc195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d2baf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc105a40 0x7f45cc195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d2baf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc196270 con 0x7f45cc0ffa40 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d2baf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc19a000 con 0x7f45cc105a40 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d094b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc0ffa40 0x7f45cc195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d094b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc0ffa40 0x7f45cc195650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33546/0 (socket says 172.21.15.151:33546) 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d094b700 1 -- 172.21.15.151:0/2762418067 learned_addr learned my addr 172.21.15.151:0/2762418067 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45cbfff700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc105a40 0x7f45cc195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d094b700 1 -- 172.21.15.151:0/2762418067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc105a40 msgr2=0x7f45cc195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d094b700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc105a40 0x7f45cc195b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.050+0000 7f45d094b700 1 -- 172.21.15.151:0/2762418067 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c00095f0 con 0x7f45cc0ffa40 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.051+0000 7f45d094b700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc0ffa40 0x7f45cc195650 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f45c0006010 tx=0x7f45c000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.051+0000 7f45c9ffb700 1 -- 172.21.15.151:0/2762418067 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45c001d070 con 0x7f45cc0ffa40 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.051+0000 7f45c9ffb700 1 -- 172.21.15.151:0/2762418067 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45c0005dc0 con 0x7f45cc0ffa40 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.051+0000 7f45c9ffb700 1 -- 172.21.15.151:0/2762418067 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45c0017400 con 0x7f45cc0ffa40 2024-04-16T02:39:41.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.051+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc19a280 con 0x7f45cc0ffa40 2024-04-16T02:39:41.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.051+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc19a720 con 0x7f45cc0ffa40 2024-04-16T02:39:41.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.052+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45cc108040 con 0x7f45cc0ffa40 2024-04-16T02:39:41.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.053+0000 7f45c9ffb700 1 -- 172.21.15.151:0/2762418067 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 34) v1 ==== 92600+0+0 (secure 0 0 0) 0x7f45c0003a60 con 0x7f45cc0ffa40 2024-04-16T02:39:41.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.054+0000 7f45c9ffb700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f45bc06e940 0x7f45bc070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:39:41.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.054+0000 7f45c9ffb700 1 -- 172.21.15.151:0/2762418067 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f45c008fd30 con 0x7f45cc0ffa40 2024-04-16T02:39:41.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.054+0000 7f45cbfff700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f45bc06e940 0x7f45bc070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:39:41.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.054+0000 7f45cbfff700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f45bc06e940 0x7f45bc070e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f45cc196c70 tx=0x7f45b800b410 comp rx=0 tx=0).ready entity=mgr.14692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:39:41.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.058+0000 7f45c9ffb700 1 -- 172.21.15.151:0/2762418067 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187551 (secure 0 0 0) 0x7f45c0059010 con 0x7f45cc0ffa40 2024-04-16T02:39:41.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:40 smithi153 ceph-mon[36747]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:41.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:40 smithi153 ceph-mon[36747]: Upgrade: Target is version 19.0.0-3084-g87dbfb63 (squid) 2024-04-16T02:39:41.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:40 smithi153 ceph-mon[36747]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f'] 2024-04-16T02:39:41.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:40 smithi153 ceph-mon[36747]: Upgrade: Setting container_image for all mgr 2024-04-16T02:39:41.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:40 smithi153 ceph-mon[36747]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:41.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:40 smithi153 ceph-mon[36747]: from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:41.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:40 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2267136498' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:39:41.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:40 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2281246915' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:39:41.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.356+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f45cc04ea90 con 0x7f45cc0ffa40 2024-04-16T02:39:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.356+0000 7f45c9ffb700 1 -- 172.21.15.151:0/2762418067 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f45c005c6d0 con 0x7f45cc0ffa40 2024-04-16T02:39:41.360 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_OK 2024-04-16T02:39:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.360+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f45bc06e940 msgr2=0x7f45bc070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.360+0000 7f45d2baf700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f45bc06e940 0x7f45bc070e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f45cc196c70 tx=0x7f45b800b410 comp rx=0 tx=0).stop 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.360+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc0ffa40 msgr2=0x7f45cc195650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.360+0000 7f45d2baf700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc0ffa40 0x7f45cc195650 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f45c0006010 tx=0x7f45c000bd70 comp rx=0 tx=0).stop 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.360+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 shutdown_connections 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.360+0000 7f45d2baf700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:6800/1243156277,v1:172.21.15.151:6801/1243156277] conn(0x7f45bc06e940 0x7f45bc070e00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.360+0000 7f45d2baf700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f45cc0ffa40 0x7f45cc195650 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.360+0000 7f45d2baf700 1 --2- 172.21.15.151:0/2762418067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc105a40 0x7f45cc195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.360+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 >> 172.21.15.151:0/2762418067 conn(0x7f45cc0fb570 msgr2=0x7f45cc0fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.361+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 shutdown_connections 2024-04-16T02:39:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:39:41.361+0000 7f45d2baf700 1 -- 172.21.15.151:0/2762418067 wait complete. 2024-04-16T02:39:41.990 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:41 smithi151 ceph-mon[29003]: pgmap v88: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:41.990 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:41 smithi151 ceph-mon[29003]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:41.990 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:41 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:41.990 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:41 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:39:41.990 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:41 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:39:41.990 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:41 smithi151 ceph-mon[29003]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:41.990 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:41 smithi151 ceph-mon[29003]: from='client.? 172.21.15.151:0/2762418067' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:39:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:41 smithi153 ceph-mon[36747]: pgmap v88: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:41 smithi153 ceph-mon[36747]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:39:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:41 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:41 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:39:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:41 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:39:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:41 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:39:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:41 smithi153 ceph-mon[36747]: from='client.? 172.21.15.151:0/2762418067' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:39:42.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:42 smithi151 ceph-mon[29003]: Upgrade: Updating mon.smithi151 2024-04-16T02:39:42.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:42 smithi151 ceph-mon[29003]: Deploying daemon mon.smithi151 on smithi151 2024-04-16T02:39:43.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:42 smithi153 ceph-mon[36747]: Upgrade: Updating mon.smithi151 2024-04-16T02:39:43.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:42 smithi153 ceph-mon[36747]: Deploying daemon mon.smithi151 on smithi151 2024-04-16T02:39:43.451 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:43 smithi151 systemd[1]: Stopping Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:39:43.715 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:43 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[28999]: 2024-04-16T02:39:43.448+0000 7ffbcfde5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi151 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T02:39:43.716 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:43 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[28999]: 2024-04-16T02:39:43.448+0000 7ffbcfde5700 -1 mon.smithi151@0(leader) e2 *** Got Signal Terminated *** 2024-04-16T02:39:43.993 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:43 smithi151 podman[112054]: 2024-04-16 02:39:43.714051018 +0000 UTC m=+0.397330605 container died 051460b1db9f73ba31fe8575ade143d8b81ea32bf1082dff6ce456e8dc968247 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-04-16T02:39:44.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:44 smithi151 podman[112054]: 2024-04-16 02:39:44.074165471 +0000 UTC m=+0.757445071 container cleanup 051460b1db9f73ba31fe8575ade143d8b81ea32bf1082dff6ce456e8dc968247 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-16T02:39:44.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:44 smithi151 bash[112054]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151 2024-04-16T02:39:44.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:44 smithi151 podman[112079]: 2024-04-16 02:39:44.3758567 +0000 UTC m=+0.657354307 container remove 051460b1db9f73ba31fe8575ade143d8b81ea32bf1082dff6ce456e8dc968247 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1) 2024-04-16T02:39:44.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:44 smithi151 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151.service: Deactivated successfully. 2024-04-16T02:39:44.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:44 smithi151 systemd[1]: Stopped Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:39:44.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:44 smithi151 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151.service: Consumed 14.136s CPU time. 2024-04-16T02:39:44.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:44 smithi151 systemd[1]: Starting Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:39:44.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:44 smithi151 podman[112214]: 2024-04-16 02:39:44.864994977 +0000 UTC m=+0.018498665 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:39:44.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:44 smithi151 podman[112214]: 2024-04-16 02:39:44.985962947 +0000 UTC m=+0.139466614 container create 420c23cf907e8fb172d44e933f14c1ff0d7a82f1d9d6039e6a827d276613fc2b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-16T02:39:45.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 podman[112214]: 2024-04-16 02:39:45.214231263 +0000 UTC m=+0.367734930 container init 420c23cf907e8fb172d44e933f14c1ff0d7a82f1d9d6039e6a827d276613fc2b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, GIT_CLEAN=True, RELEASE=main-87dbfb6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 podman[112214]: 2024-04-16 02:39:45.217550484 +0000 UTC m=+0.371054162 container start 420c23cf907e8fb172d44e933f14c1ff0d7a82f1d9d6039e6a827d276613fc2b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6) 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev), process ceph-mon, pid 2 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: pidfile_write: ignore empty --pid-file 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: load: jerasure load: lrc 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: RocksDB version: 7.9.2 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Git sha 0 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Compile date 2024-04-14 15:12:02 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: DB SUMMARY 2024-04-16T02:39:45.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: DB Session ID: V5O3SDAJLJ3Y00PWC4BD 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: CURRENT file: CURRENT 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1509 Bytes 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi151/store.db dir, Total Num: 1, files: 000032.sst 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi151/store.db: 000030.log size: 4953252 ; 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.error_if_exists: 0 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.create_if_missing: 0 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.paranoid_checks: 1 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.env: 0x55a086c88cc0 2024-04-16T02:39:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T02:39:45.499 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.info_log: 0x55a0889e9ac0 2024-04-16T02:39:45.499 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T02:39:45.500 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.statistics: (nil) 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.use_fsync: 0 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_log_file_size: 0 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.allow_fallocate: 1 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.use_direct_reads: 0 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T02:39:45.501 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.db_log_dir: 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.wal_dir: 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.write_buffer_manager: 0x55a0889b81e0 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T02:39:45.502 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.unordered_write: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.row_cache: None 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.wal_filter: None 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.two_write_queues: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.wal_compression: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.atomic_flush: 0 2024-04-16T02:39:45.503 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.log_readahead_size: 0 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_background_jobs: 2 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_background_compactions: -1 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_subcompactions: 1 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T02:39:45.504 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_open_files: -1 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_background_flushes: -1 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Compression algorithms supported: 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: kZSTD supported: 0 2024-04-16T02:39:45.505 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: kXpressCompression supported: 0 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: kZlibCompression supported: 1 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: kSnappyCompression supported: 1 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: kLZ4Compression supported: 1 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: kBZip2Compression supported: 0 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi151/store.db/MANIFEST-000015 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.merge_operator: 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_filter: None 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T02:39:45.506 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a0889e9ca0) 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: index_type: 0 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: data_block_index_type: 0 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: index_shortening: 1 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: checksum: 4 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: no_block_cache: 0 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_cache: 0x55a0889eb090 2024-04-16T02:39:45.507 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_cache_options: 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: capacity : 536870912 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: num_shard_bits : 4 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: strict_capacity_limit : 0 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_cache_compressed: (nil) 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: persistent_cache: (nil) 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_size: 4096 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_size_deviation: 10 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_restart_interval: 16 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: index_block_restart_interval: 1 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: metadata_block_size: 4096 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: partition_filters: 0 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: use_delta_encoding: 1 2024-04-16T02:39:45.508 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: filter_policy: bloomfilter 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: whole_key_filtering: 1 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: verify_compression: 0 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: format_version: 5 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: enable_index_compression: 1 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: block_align: 0 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: max_auto_readahead_size: 262144 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: prepopulate_block_cache: 0 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: initial_auto_readahead_size: 8192 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression: NoCompression 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T02:39:45.509 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.num_levels: 7 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T02:39:45.510 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T02:39:45.511 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T02:39:45.512 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.inplace_update_support: 0 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.bloom_locality: 0 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.max_successive_merges: 0 2024-04-16T02:39:45.513 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.ttl: 2592000 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.enable_blob_files: false 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.min_blob_size: 0 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T02:39:45.514 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi151/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12077, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 494c98a9-92d5-4767-8003-d4a132cd652d 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713235185247311, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713235185260991, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 4400040, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12078, "largest_seqno": 15384, "table_properties": {"data_size": 4387089, "index_size": 8128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3781, "raw_key_size": 32801, "raw_average_key_size": 22, "raw_value_size": 4357823, "raw_average_value_size": 2924, "num_data_blocks": 368, "num_entries": 1490, "num_filter_entries": 1490, "num_deletions": 259, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713235185, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "494c98a9-92d5-4767-8003-d4a132cd652d", "db_session_id": "V5O3SDAJLJ3Y00PWC4BD", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713235185261094, "job": 1, "event": "recovery_finished"} 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T02:39:45.515 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi151/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a088ad8000 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: DB pointer 0x55a088ac2000 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ** DB Stats ** 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ** Compaction Stats [default] ** 2024-04-16T02:39:45.516 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: L0 1/0 4.20 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 688.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: L6 1/0 5.10 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Sum 2/0 9.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 688.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 688.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ** Compaction Stats [default] ** 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 688.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-16T02:39:45.517 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Cumulative compaction: 0.00 GB write, 223.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Interval compaction: 0.00 GB write, 223.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Block cache BinnedLRUCache@0x55a0889eb090#2 capacity: 512.00 MB usage: 12.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: starting mon.smithi151 rank 0 at public addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] at bind addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi151 fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: mon.smithi151@-1(???) e2 preinit fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: mon.smithi151@-1(???).mds e20 new map 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: mon.smithi151@-1(???).mds e20 print_map 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: e20 2024-04-16T02:39:45.518 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: legacy client fscid: 1 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: Filesystem 'cephfs' (1) 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: fs_name cephfs 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: epoch 20 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: modified 2024-04-16T02:36:05.787890+0000 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: tableserver 0 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: root 0 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: session_timeout 60 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: session_autoclose 300 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: max_file_size 1099511627776 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: max_xattr_size 65536 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: required_client_features {} 2024-04-16T02:39:45.519 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: last_failure 0 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: last_failure_osd_epoch 47 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: max_mds 2 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: in 0,1 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: up {0=14478,1=14490} 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: failed 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: damaged 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: stopped 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: data_pools [3] 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: metadata_pool 2 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: inline_data disabled 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: balancer 2024-04-16T02:39:45.520 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: bal_rank_mask -1 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: standby_count_wanted 1 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: qdb_cluster leader: 0 members: 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: [mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 export targets 1 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: [mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: [mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 export targets 0 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: [mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout: 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: mon.smithi151@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: mon.smithi151@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: mon.smithi151@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: mon.smithi151@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 ceph-mon[112233]: mon.smithi151@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 bash[112214]: 420c23cf907e8fb172d44e933f14c1ff0d7a82f1d9d6039e6a827d276613fc2b 2024-04-16T02:39:45.521 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:45 smithi151 systemd[1]: Started Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:39:46.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:39:46.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:39:46.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: pgmap v90: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:46.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: mon.smithi151 calling monitor election 2024-04-16T02:39:46.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: mon.smithi151 is new leader, mons smithi151,smithi153 in quorum (ranks 0,1) 2024-04-16T02:39:46.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: monmap epoch 2 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: last_changed 2024-04-16T02:28:12.026363+0000 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: created 2024-04-16T02:25:31.524258+0000 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: min_mon_release 18 (reef) 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: election_strategy: 1 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.smithi151 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.smithi153 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: mgrmap e34: smithi151.uqqame(active, since 2m), standbys: smithi153.esbtzf 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: overall HEALTH_OK 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: from='mgr.14692 ' entity='' 2024-04-16T02:39:46.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:46 smithi151 ceph-mon[112233]: mgrmap e35: smithi151.uqqame(active, since 2m), standbys: smithi153.esbtzf 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: from='mgr.14692 172.21.15.151:0/1644797291' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: pgmap v90: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: mon.smithi151 calling monitor election 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: mon.smithi151 is new leader, mons smithi151,smithi153 in quorum (ranks 0,1) 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: monmap epoch 2 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: last_changed 2024-04-16T02:28:12.026363+0000 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: created 2024-04-16T02:25:31.524258+0000 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: min_mon_release 18 (reef) 2024-04-16T02:39:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: election_strategy: 1 2024-04-16T02:39:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.smithi151 2024-04-16T02:39:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.smithi153 2024-04-16T02:39:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:39:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T02:39:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: mgrmap e34: smithi151.uqqame(active, since 2m), standbys: smithi153.esbtzf 2024-04-16T02:39:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: overall HEALTH_OK 2024-04-16T02:39:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: from='mgr.14692 ' entity='' 2024-04-16T02:39:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:46 smithi153 ceph-mon[36747]: mgrmap e35: smithi151.uqqame(active, since 2m), standbys: smithi153.esbtzf 2024-04-16T02:39:48.007 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:47 smithi151 ceph-mon[112233]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:39:48.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:47 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:39:53.701 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:53 smithi151 ceph-mon[112233]: Standby manager daemon smithi153.esbtzf restarted 2024-04-16T02:39:53.701 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:53 smithi151 ceph-mon[112233]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:39:53.701 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:53 smithi151 ceph-mon[112233]: from='mgr.? 172.21.15.153:0/3239022065' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:39:53.701 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:53 smithi151 ceph-mon[112233]: from='mgr.? 172.21.15.153:0/3239022065' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:39:53.701 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:53 smithi151 ceph-mon[112233]: from='mgr.? 172.21.15.153:0/3239022065' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:39:53.701 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:53 smithi151 ceph-mon[112233]: from='mgr.? 172.21.15.153:0/3239022065' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:39:53.819 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:53 smithi153 ceph-mon[36747]: Standby manager daemon smithi153.esbtzf restarted 2024-04-16T02:39:53.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:53 smithi153 ceph-mon[36747]: Standby manager daemon smithi153.esbtzf started 2024-04-16T02:39:53.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:53 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3239022065' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/crt"}]: dispatch 2024-04-16T02:39:53.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:53 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3239022065' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T02:39:53.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:53 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3239022065' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.esbtzf/key"}]: dispatch 2024-04-16T02:39:53.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:53 smithi153 ceph-mon[36747]: from='mgr.? 172.21.15.153:0/3239022065' entity='mgr.smithi153.esbtzf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: mgrmap e36: smithi151.uqqame(active, since 3m), standbys: smithi153.esbtzf 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: Active manager daemon smithi151.uqqame restarted 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: Activating manager daemon smithi151.uqqame 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: mgrmap e37: smithi151.uqqame(active, starting, since 0.00920006s), standbys: smithi153.esbtzf 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:39:54.894 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi153.esbtzf", "id": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: Manager daemon smithi151.uqqame is now available 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:39:54.895 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:39:54.896 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:54 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/trash_purge_schedule"}]: dispatch 2024-04-16T02:39:54.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: mgrmap e36: smithi151.uqqame(active, since 3m), standbys: smithi153.esbtzf 2024-04-16T02:39:54.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: Active manager daemon smithi151.uqqame restarted 2024-04-16T02:39:54.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: Activating manager daemon smithi151.uqqame 2024-04-16T02:39:54.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T02:39:54.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: mgrmap e37: smithi151.uqqame(active, starting, since 0.00920006s), standbys: smithi153.esbtzf 2024-04-16T02:39:54.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:39:54.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:39:54.916 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi151.uqqame", "id": "smithi151.uqqame"}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr metadata", "who": "smithi153.esbtzf", "id": "smithi153.esbtzf"}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: Manager daemon smithi151.uqqame is now available 2024-04-16T02:39:54.917 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:39:54.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:39:54.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/mirror_snapshot_schedule"}]: dispatch 2024-04-16T02:39:54.918 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi151.uqqame/trash_purge_schedule"}]: dispatch 2024-04-16T02:39:55.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:55 smithi151 ceph-mon[112233]: mgrmap e38: smithi151.uqqame(active, since 1.0098s), standbys: smithi153.esbtzf 2024-04-16T02:39:55.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:55 smithi151 ceph-mon[112233]: pgmap v3: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:39:56.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:55 smithi153 ceph-mon[36747]: mgrmap e38: smithi151.uqqame(active, since 1.0098s), standbys: smithi153.esbtzf 2024-04-16T02:39:56.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:55 smithi153 ceph-mon[36747]: pgmap v3: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:39:56.806 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:56 smithi151 ceph-mon[112233]: [16/Apr/2024:02:39:54] ENGINE Bus STARTING 2024-04-16T02:39:56.806 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:56 smithi151 ceph-mon[112233]: [16/Apr/2024:02:39:55] ENGINE Serving on https://172.21.15.151:7150 2024-04-16T02:39:56.807 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:56 smithi151 ceph-mon[112233]: [16/Apr/2024:02:39:55] ENGINE Serving on http://172.21.15.151:8765 2024-04-16T02:39:56.807 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:56 smithi151 ceph-mon[112233]: [16/Apr/2024:02:39:55] ENGINE Bus STARTED 2024-04-16T02:39:56.807 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:56 smithi151 ceph-mon[112233]: pgmap v4: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:39:56.807 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:56 smithi151 ceph-mon[112233]: mgrmap e39: smithi151.uqqame(active, since 2s), standbys: smithi153.esbtzf 2024-04-16T02:39:57.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:56 smithi153 ceph-mon[36747]: [16/Apr/2024:02:39:54] ENGINE Bus STARTING 2024-04-16T02:39:57.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:56 smithi153 ceph-mon[36747]: [16/Apr/2024:02:39:55] ENGINE Serving on https://172.21.15.151:7150 2024-04-16T02:39:57.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:56 smithi153 ceph-mon[36747]: [16/Apr/2024:02:39:55] ENGINE Serving on http://172.21.15.151:8765 2024-04-16T02:39:57.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:56 smithi153 ceph-mon[36747]: [16/Apr/2024:02:39:55] ENGINE Bus STARTED 2024-04-16T02:39:57.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:56 smithi153 ceph-mon[36747]: pgmap v4: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:39:57.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:56 smithi153 ceph-mon[36747]: mgrmap e39: smithi151.uqqame(active, since 2s), standbys: smithi153.esbtzf 2024-04-16T02:39:58.209 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:57 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:58.209 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:57 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:58.209 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:57 smithi151 ceph-mon[112233]: pgmap v5: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:39:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:57 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:57 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:57 smithi153 ceph-mon[36747]: pgmap v5: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:39:59.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:58 smithi151 ceph-mon[112233]: mgrmap e40: smithi151.uqqame(active, since 4s), standbys: smithi153.esbtzf 2024-04-16T02:39:59.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:58 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:59.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:39:58 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:59.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:58 smithi153 ceph-mon[36747]: mgrmap e40: smithi151.uqqame(active, since 4s), standbys: smithi153.esbtzf 2024-04-16T02:39:59.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:58 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:39:59.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:39:58 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:00.585 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:00 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:00.585 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:00 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:00.585 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:00 smithi153 ceph-mon[36747]: pgmap v6: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 43 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-16T02:40:00.585 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:00 smithi153 ceph-mon[36747]: overall HEALTH_OK 2024-04-16T02:40:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:00 smithi151 ceph-mon[112233]: pgmap v6: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 43 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-16T02:40:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:00 smithi151 ceph-mon[112233]: overall HEALTH_OK 2024-04-16T02:40:01.956 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:01.956 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:02.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:01 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:02.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:01 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:02.870 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:02 smithi151 ceph-mon[112233]: Detected new or changed devices on smithi153 2024-04-16T02:40:02.871 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:02 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:02.871 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:02 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:02.871 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:02 smithi151 ceph-mon[112233]: pgmap v7: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-16T02:40:03.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:02 smithi153 ceph-mon[36747]: Detected new or changed devices on smithi153 2024-04-16T02:40:03.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:02 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:03.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:02 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:03.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:02 smithi153 ceph-mon[36747]: pgmap v7: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-16T02:40:04.060 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:03 smithi151 ceph-mon[112233]: pgmap v8: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-16T02:40:04.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:03 smithi153 ceph-mon[36747]: pgmap v8: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-16T02:40:05.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:05 smithi151 ceph-mon[112233]: Detected new or changed devices on smithi151 2024-04-16T02:40:05.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:05.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:05.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:05.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:40:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:05 smithi151 ceph-mon[112233]: Updating smithi151:/etc/ceph/ceph.conf 2024-04-16T02:40:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:05 smithi151 ceph-mon[112233]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-16T02:40:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:05 smithi151 ceph-mon[112233]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:40:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:05 smithi151 ceph-mon[112233]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:40:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:05 smithi153 ceph-mon[36747]: Detected new or changed devices on smithi151 2024-04-16T02:40:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:05 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:05 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:05 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:05 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:40:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:05 smithi153 ceph-mon[36747]: Updating smithi151:/etc/ceph/ceph.conf 2024-04-16T02:40:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:05 smithi153 ceph-mon[36747]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-16T02:40:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:05 smithi153 ceph-mon[36747]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:40:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:05 smithi153 ceph-mon[36747]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T02:40:06.477 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:40:06.477 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:40:06.477 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:40:06.477 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:40:06.477 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:40:06.478 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.478 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.478 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.478 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.478 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.478 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:40:06.478 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:06.478 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T02:40:06.478 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:06 smithi153 ceph-mon[36747]: pgmap v9: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-16T02:40:06.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:40:06.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-16T02:40:06.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: Updating smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: Updating smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T02:40:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:06 smithi151 ceph-mon[112233]: pgmap v9: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-16T02:40:08.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:08 smithi153 ceph-mon[36747]: Upgrade: Updating mon.smithi153 2024-04-16T02:40:08.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:08 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:08.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:08 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:40:08.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:08 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:40:08.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:08 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:08.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:08 smithi153 ceph-mon[36747]: Deploying daemon mon.smithi153 on smithi153 2024-04-16T02:40:08.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:08 smithi153 ceph-mon[36747]: pgmap v10: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T02:40:08.523 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:08 smithi151 ceph-mon[112233]: Upgrade: Updating mon.smithi153 2024-04-16T02:40:08.523 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:08.523 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:40:08.523 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:40:08.523 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:08.523 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:08 smithi151 ceph-mon[112233]: Deploying daemon mon.smithi153 on smithi153 2024-04-16T02:40:08.523 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:08 smithi151 ceph-mon[112233]: pgmap v10: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T02:40:09.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:09 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:40:09.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:09 smithi153 ceph-mon[36747]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:40:09.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:09 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153[36743]: 2024-04-16T02:40:09.448+0000 7f53c1f13700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T02:40:09.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:09 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153[36743]: 2024-04-16T02:40:09.448+0000 7f53c1f13700 -1 mon.smithi153@1(peon) e2 *** Got Signal Terminated *** 2024-04-16T02:40:09.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:40:09.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:09 smithi153 podman[87692]: 2024-04-16 02:40:09.520570779 +0000 UTC m=+0.170015993 container died 5f3186f6d9f908f44962f139f9d40629028d9ed81e4938887f9b9e8cbdcf907a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-16T02:40:10.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:09 smithi153 podman[87692]: 2024-04-16 02:40:09.923610933 +0000 UTC m=+0.573056143 container cleanup 5f3186f6d9f908f44962f139f9d40629028d9ed81e4938887f9b9e8cbdcf907a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS) 2024-04-16T02:40:10.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:09 smithi153 bash[87692]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153 2024-04-16T02:40:10.445 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 podman[87706]: 2024-04-16 02:40:10.141754215 +0000 UTC m=+0.617005491 container remove 5f3186f6d9f908f44962f139f9d40629028d9ed81e4938887f9b9e8cbdcf907a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-16T02:40:10.445 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi153.service: Deactivated successfully. 2024-04-16T02:40:10.445 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 systemd[1]: Stopped Ceph mon.smithi153 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:40:10.445 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi153.service: Consumed 7.830s CPU time. 2024-04-16T02:40:10.665 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 systemd[1]: Starting Ceph mon.smithi153 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:40:10.665 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 podman[87839]: 2024-04-16 02:40:10.565517157 +0000 UTC m=+0.018781778 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:40:10.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 podman[87839]: 2024-04-16 02:40:10.667816562 +0000 UTC m=+0.121081169 container create cc3cd0d47f4ee34c97e15c656cc323f6c0f5393a2db1ac0c529f79383c0d9c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, RELEASE=main-87dbfb6, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-16T02:40:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 podman[87839]: 2024-04-16 02:40:10.903487728 +0000 UTC m=+0.356752322 container init cc3cd0d47f4ee34c97e15c656cc323f6c0f5393a2db1ac0c529f79383c0d9c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, org.label-schema.schema-version=1.0, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-16T02:40:11.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 podman[87839]: 2024-04-16 02:40:10.906798753 +0000 UTC m=+0.360063347 container start cc3cd0d47f4ee34c97e15c656cc323f6c0f5393a2db1ac0c529f79383c0d9c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-16T02:40:11.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T02:40:11.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev), process ceph-mon, pid 2 2024-04-16T02:40:11.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: pidfile_write: ignore empty --pid-file 2024-04-16T02:40:11.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: load: jerasure load: lrc 2024-04-16T02:40:11.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: RocksDB version: 7.9.2 2024-04-16T02:40:11.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Git sha 0 2024-04-16T02:40:11.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Compile date 2024-04-14 15:12:02 2024-04-16T02:40:11.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: DB SUMMARY 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: DB Session ID: BYBTG052B3BP8VEWSE8R 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: CURRENT file: CURRENT 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1403 Bytes 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi153/store.db dir, Total Num: 1, files: 000027.sst 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi153/store.db: 000025.log size: 10462208 ; 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.error_if_exists: 0 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.create_if_missing: 0 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.paranoid_checks: 1 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T02:40:11.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.env: 0x560d9620fcc0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.info_log: 0x560d976b7ac0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.statistics: (nil) 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.use_fsync: 0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_log_file_size: 0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.allow_fallocate: 1 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.use_direct_reads: 0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T02:40:11.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.db_log_dir: 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.wal_dir: 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.write_buffer_manager: 0x560d976861e0 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T02:40:11.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.unordered_write: 0 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.row_cache: None 2024-04-16T02:40:11.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.wal_filter: None 2024-04-16T02:40:11.278 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.two_write_queues: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.wal_compression: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.atomic_flush: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.log_readahead_size: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T02:40:11.279 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_background_jobs: 2 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_background_compactions: -1 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_subcompactions: 1 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_open_files: -1 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T02:40:11.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_background_flushes: -1 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Compression algorithms supported: 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: kZSTD supported: 0 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: kXpressCompression supported: 0 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: kZlibCompression supported: 1 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: kSnappyCompression supported: 1 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: kLZ4Compression supported: 1 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: kBZip2Compression supported: 0 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000010 2024-04-16T02:40:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.merge_operator: 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_filter: None 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d976b7ca0) 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_type: 0 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_index_type: 0 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_shortening: 1 2024-04-16T02:40:11.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: checksum: 4 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: no_block_cache: 0 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache: 0x560d976b9090 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_options: 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: capacity : 536870912 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_shard_bits : 4 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: strict_capacity_limit : 0 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_compressed: (nil) 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: persistent_cache: (nil) 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size: 4096 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size_deviation: 10 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_restart_interval: 16 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_block_restart_interval: 1 2024-04-16T02:40:11.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: metadata_block_size: 4096 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: partition_filters: 0 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: use_delta_encoding: 1 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: filter_policy: bloomfilter 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: whole_key_filtering: 1 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: verify_compression: 0 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: format_version: 5 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_index_compression: 1 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_align: 0 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_auto_readahead_size: 262144 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: prepopulate_block_cache: 0 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: initial_auto_readahead_size: 8192 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression: NoCompression 2024-04-16T02:40:11.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.num_levels: 7 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T02:40:11.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T02:40:11.286 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T02:40:11.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.inplace_update_support: 0 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T02:40:11.288 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.bloom_locality: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.max_successive_merges: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.ttl: 2592000 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.enable_blob_files: false 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.min_blob_size: 0 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T02:40:11.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12111, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fb9d9b85-0cc5-4a26-8461-7ecbdf8de661 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713235210937237, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713235210966146, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 6196046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12116, "largest_seqno": 16317, "table_properties": {"data_size": 6180897, "index_size": 9750, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4357, "raw_key_size": 38061, "raw_average_key_size": 22, "raw_value_size": 6146899, "raw_average_value_size": 3580, "num_data_blocks": 437, "num_entries": 1717, "num_filter_entries": 1717, "num_deletions": 259, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713235210, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fb9d9b85-0cc5-4a26-8461-7ecbdf8de661", "db_session_id": "BYBTG052B3BP8VEWSE8R", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713235210966266, "job": 1, "event": "recovery_finished"} 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-16T02:40:11.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi153/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560d977a6000 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: DB pointer 0x560d97790000 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** DB Stats ** 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-16T02:40:11.291 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 1/0 5.91 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 487.2 0.01 0.00 1 0.012 0 0 0.0 0.0 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L6 1/0 5.10 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 2/0 11.01 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 487.2 0.01 0.00 1 0.012 0 0 0.0 0.0 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 487.2 0.01 0.00 1 0.012 0 0 0.0 0.0 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 487.2 0.01 0.00 1 0.012 0 0 0.0 0.0 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.006, interval 0.006 2024-04-16T02:40:11.292 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.01 GB write, 183.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.01 GB write, 183.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Block cache BinnedLRUCache@0x560d976b9090#2 capacity: 512.00 MB usage: 14.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.53 KB,0.000864267%) IndexBlock(1,10.08 KB,0.00192225%) Misc(1,0.00 KB,0%) 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: starting mon.smithi153 rank 1 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi153 fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: mon.smithi153@-1(???) e2 preinit fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: mon.smithi153@-1(???).mds e22 new map 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: mon.smithi153@-1(???).mds e22 print_map 2024-04-16T02:40:11.293 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: e22 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: legacy client fscid: 1 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Filesystem 'cephfs' (1) 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: fs_name cephfs 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: epoch 22 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: modified 2024-04-16T02:40:05.114230+0000 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: tableserver 0 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: root 0 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: session_timeout 60 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: session_autoclose 300 2024-04-16T02:40:11.294 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_file_size 1099511627776 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_xattr_size 65536 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: required_client_features {} 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: last_failure 0 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: last_failure_osd_epoch 47 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_mds 2 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: in 0,1 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: up {0=14478,1=14490} 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: failed 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: damaged 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stopped 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_pools [3] 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: metadata_pool 2 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: inline_data disabled 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: balancer 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: bal_rank_mask -1 2024-04-16T02:40:11.295 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: standby_count_wanted 1 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: qdb_cluster leader: 14478 members: 14490,14478 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: [mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: [mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: [mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: [mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: mon.smithi153@-1(???).osd e50 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: mon.smithi153@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: mon.smithi153@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: mon.smithi153@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 ceph-mon[87858]: mon.smithi153@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 bash[87839]: cc3cd0d47f4ee34c97e15c656cc323f6c0f5393a2db1ac0c529f79383c0d9c2d 2024-04-16T02:40:11.296 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:10 smithi153 systemd[1]: Started Ceph mon.smithi153 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:40:11.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.481+0000 7f1e853f4700 1 -- 172.21.15.151:0/1829299039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e80105a40 msgr2=0x7f1e80105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:11.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.481+0000 7f1e853f4700 1 --2- 172.21.15.151:0/1829299039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e80105a40 0x7f1e80105e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009910 tx=0x7f1e70009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:11.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.481+0000 7f1e853f4700 1 -- 172.21.15.151:0/1829299039 shutdown_connections 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.481+0000 7f1e853f4700 1 --2- 172.21.15.151:0/1829299039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e800ffa40 0x7f1e800ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.481+0000 7f1e853f4700 1 --2- 172.21.15.151:0/1829299039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e80105a40 0x7f1e80105e20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.481+0000 7f1e853f4700 1 -- 172.21.15.151:0/1829299039 >> 172.21.15.151:0/1829299039 conn(0x7f1e800fb570 msgr2=0x7f1e800fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.481+0000 7f1e853f4700 1 -- 172.21.15.151:0/1829299039 shutdown_connections 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.481+0000 7f1e853f4700 1 -- 172.21.15.151:0/1829299039 wait complete. 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.482+0000 7f1e853f4700 1 Processor -- start 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.482+0000 7f1e853f4700 1 -- start start 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.482+0000 7f1e853f4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e800ffa40 0x7f1e80101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.482+0000 7f1e853f4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e80105a40 0x7f1e80101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e853f4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e80102090 con 0x7f1e800ffa40 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e853f4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e801021d0 con 0x7f1e80105a40 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e800ffa40 0x7f1e80101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e800ffa40 0x7f1e80101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46274/0 (socket says 172.21.15.151:46274) 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7effd700 1 -- 172.21.15.151:0/474653427 learned_addr learned my addr 172.21.15.151:0/474653427 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e767fc700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e80105a40 0x7f1e80101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e767fc700 1 -- 172.21.15.151:0/474653427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e80105a40 msgr2=0x7f1e80101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e767fc700 1 -- 172.21.15.151:0/474653427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e80105a40 msgr2=0x7f1e80101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e767fc700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e80105a40 0x7f1e80101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e767fc700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e80105a40 0x7f1e80101b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7effd700 1 -- 172.21.15.151:0/474653427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e80105a40 msgr2=0x7f1e80101b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7effd700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e80105a40 0x7f1e80101b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7effd700 1 -- 172.21.15.151:0/474653427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e700095f0 con 0x7f1e800ffa40 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7effd700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e800ffa40 0x7f1e80101610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e70006010 tx=0x7f1e7000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:11.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7cff9700 1 -- 172.21.15.151:0/474653427 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e7001d070 con 0x7f1e800ffa40 2024-04-16T02:40:11.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7cff9700 1 -- 172.21.15.151:0/474653427 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e7000f460 con 0x7f1e800ffa40 2024-04-16T02:40:11.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e7cff9700 1 -- 172.21.15.151:0/474653427 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e70017630 con 0x7f1e800ffa40 2024-04-16T02:40:11.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.483+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e80102450 con 0x7f1e800ffa40 2024-04-16T02:40:11.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.484+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e801a2c80 con 0x7f1e800ffa40 2024-04-16T02:40:11.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.484+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8004ea90 con 0x7f1e800ffa40 2024-04-16T02:40:11.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.485+0000 7f1e7cff9700 1 -- 172.21.15.151:0/474653427 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f1e7000f5d0 con 0x7f1e800ffa40 2024-04-16T02:40:11.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.485+0000 7f1e7cff9700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e6c06ee00 0x7f1e6c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:11.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.486+0000 7f1e7cff9700 1 -- 172.21.15.151:0/474653427 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1e700904e0 con 0x7f1e800ffa40 2024-04-16T02:40:11.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.486+0000 7f1e767fc700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e6c06ee00 0x7f1e6c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:11.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.486+0000 7f1e767fc700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e6c06ee00 0x7f1e6c0712c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1e68005950 tx=0x7f1e680058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.491+0000 7f1e7cff9700 1 -- 172.21.15.151:0/474653427 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1e70059120 con 0x7f1e800ffa40 2024-04-16T02:40:11.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.709+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e800fbdd0 con 0x7f1e6c06ee00 2024-04-16T02:40:11.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.711+0000 7f1e7cff9700 1 -- 172.21.15.151:0/474653427 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f1e800fbdd0 con 0x7f1e6c06ee00 2024-04-16T02:40:11.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e6c06ee00 msgr2=0x7f1e6c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:11.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e6c06ee00 0x7f1e6c0712c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1e68005950 tx=0x7f1e680058e0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e800ffa40 msgr2=0x7f1e80101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:11.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e800ffa40 0x7f1e80101610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e70006010 tx=0x7f1e7000be20 comp rx=0 tx=0).stop 2024-04-16T02:40:11.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 shutdown_connections 2024-04-16T02:40:11.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e6c06ee00 0x7f1e6c0712c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e800ffa40 0x7f1e80101610 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 --2- 172.21.15.151:0/474653427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e80105a40 0x7f1e80101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 >> 172.21.15.151:0/474653427 conn(0x7f1e800fb570 msgr2=0x7f1e80103830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:11.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 shutdown_connections 2024-04-16T02:40:11.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.713+0000 7f1e853f4700 1 -- 172.21.15.151:0/474653427 wait complete. 2024-04-16T02:40:11.726 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 -- 172.21.15.151:0/3199697837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c100dc0 msgr2=0x7f161c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 --2- 172.21.15.151:0/3199697837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c100dc0 0x7f161c1011a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1618009910 tx=0x7f1618009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 -- 172.21.15.151:0/3199697837 shutdown_connections 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 --2- 172.21.15.151:0/3199697837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c101770 0x7f161c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 --2- 172.21.15.151:0/3199697837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c100dc0 0x7f161c1011a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 -- 172.21.15.151:0/3199697837 >> 172.21.15.151:0/3199697837 conn(0x7f161c0fc650 msgr2=0x7f161c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 -- 172.21.15.151:0/3199697837 shutdown_connections 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 -- 172.21.15.151:0/3199697837 wait complete. 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 Processor -- start 2024-04-16T02:40:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.821+0000 7f1623c8e700 1 -- start start 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1623c8e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c100dc0 0x7f161c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1623c8e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c101770 0x7f161c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1623c8e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c19d110 con 0x7f161c101770 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1623c8e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c1965c0 con 0x7f161c100dc0 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621229700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c101770 0x7f161c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621229700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c101770 0x7f161c19ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46306/0 (socket says 172.21.15.151:46306) 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621229700 1 -- 172.21.15.151:0/3787943530 learned_addr learned my addr 172.21.15.151:0/3787943530 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621a2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c100dc0 0x7f161c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621a2a700 1 -- 172.21.15.151:0/3787943530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c100dc0 msgr2=0x7f161c19c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621a2a700 1 -- 172.21.15.151:0/3787943530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c100dc0 msgr2=0x7f161c19c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T02:40:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621a2a700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c100dc0 0x7f161c19c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621a2a700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c100dc0 0x7f161c19c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621229700 1 -- 172.21.15.151:0/3787943530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c100dc0 msgr2=0x7f161c19c540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621229700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c100dc0 0x7f161c19c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621229700 1 -- 172.21.15.151:0/3787943530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16180095f0 con 0x7f161c101770 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1621229700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c101770 0x7f161c19ca80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f160c00d900 tx=0x7f160c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1612ffd700 1 -- 172.21.15.151:0/3787943530 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160c009900 con 0x7f161c101770 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161c1968a0 con 0x7f161c101770 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1612ffd700 1 -- 172.21.15.151:0/3787943530 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f160c010460 con 0x7f161c101770 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1612ffd700 1 -- 172.21.15.151:0/3787943530 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160c00f5f0 con 0x7f161c101770 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.822+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161c196df0 con 0x7f161c101770 2024-04-16T02:40:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.823+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161c109160 con 0x7f161c101770 2024-04-16T02:40:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.824+0000 7f1612ffd700 1 -- 172.21.15.151:0/3787943530 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f160c0105d0 con 0x7f161c101770 2024-04-16T02:40:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.825+0000 7f1612ffd700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f160806ee00 0x7f16080712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.825+0000 7f1621a2a700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f160806ee00 0x7f16080712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.825+0000 7f1612ffd700 1 -- 172.21.15.151:0/3787943530 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f160c08ee90 con 0x7f161c101770 2024-04-16T02:40:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.825+0000 7f1621a2a700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f160806ee00 0x7f16080712c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f161800b5c0 tx=0x7f161801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:11.830+0000 7f1612ffd700 1 -- 172.21.15.151:0/3787943530 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f160c05b600 con 0x7f161c101770 2024-04-16T02:40:12.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.047+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f161c1970b0 con 0x7f160806ee00 2024-04-16T02:40:12.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.048+0000 7f1612ffd700 1 -- 172.21.15.151:0/3787943530 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f161c1970b0 con 0x7f160806ee00 2024-04-16T02:40:12.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f160806ee00 msgr2=0x7f16080712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:12.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f160806ee00 0x7f16080712c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f161800b5c0 tx=0x7f161801a040 comp rx=0 tx=0).stop 2024-04-16T02:40:12.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c101770 msgr2=0x7f161c19ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:12.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c101770 0x7f161c19ca80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f160c00d900 tx=0x7f160c00dcc0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 shutdown_connections 2024-04-16T02:40:12.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f160806ee00 0x7f16080712c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f161c100dc0 0x7f161c19c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 --2- 172.21.15.151:0/3787943530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f161c101770 0x7f161c19ca80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 >> 172.21.15.151:0/3787943530 conn(0x7f161c0fc650 msgr2=0x7f161c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:12.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.050+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 shutdown_connections 2024-04-16T02:40:12.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.051+0000 7f1623c8e700 1 -- 172.21.15.151:0/3787943530 wait complete. 2024-04-16T02:40:12.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.160+0000 7f996a523700 1 -- 172.21.15.151:0/3528489292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 msgr2=0x7f9964105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:12.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.160+0000 7f996a523700 1 --2- 172.21.15.151:0/3528489292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 0x7f9964105840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f994c009910 tx=0x7f994c009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:12.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.160+0000 7f996a523700 1 -- 172.21.15.151:0/3528489292 shutdown_connections 2024-04-16T02:40:12.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.160+0000 7f996a523700 1 --2- 172.21.15.151:0/3528489292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 0x7f9964105840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.160+0000 7f996a523700 1 --2- 172.21.15.151:0/3528489292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 0x7f99640ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.160+0000 7f996a523700 1 -- 172.21.15.151:0/3528489292 >> 172.21.15.151:0/3528489292 conn(0x7f9964074d80 msgr2=0x7f9964075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:12.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.160+0000 7f996a523700 1 -- 172.21.15.151:0/3528489292 shutdown_connections 2024-04-16T02:40:12.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.160+0000 7f996a523700 1 -- 172.21.15.151:0/3528489292 wait complete. 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f996a523700 1 Processor -- start 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f996a523700 1 -- start start 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f996a523700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 0x7f9964100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f996a523700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 0x7f99641009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f996a523700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964100f00 con 0x7f99640ffeb0 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f996a523700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964101040 con 0x7f99640ff500 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f995bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 0x7f99641009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f995bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 0x7f99641009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46334/0 (socket says 172.21.15.151:46334) 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f995bfff700 1 -- 172.21.15.151:0/2836747287 learned_addr learned my addr 172.21.15.151:0/2836747287 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f9963fff700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 0x7f9964100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f9963fff700 1 -- 172.21.15.151:0/2836747287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 msgr2=0x7f9964100480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-16T02:40:12.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f9963fff700 1 -- 172.21.15.151:0/2836747287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 msgr2=0x7f9964100480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f9963fff700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 0x7f9964100480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f9963fff700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 0x7f9964100480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f995bfff700 1 -- 172.21.15.151:0/2836747287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 msgr2=0x7f9964100480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f995bfff700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 0x7f9964100480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.161+0000 7f995bfff700 1 -- 172.21.15.151:0/2836747287 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0095f0 con 0x7f99640ffeb0 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.162+0000 7f995bfff700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 0x7f99641009c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f994c00bd70 tx=0x7f994c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.162+0000 7f9961ffb700 1 -- 172.21.15.151:0/2836747287 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994c01d070 con 0x7f99640ffeb0 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.162+0000 7f9961ffb700 1 -- 172.21.15.151:0/2836747287 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f994c005dc0 con 0x7f99640ffeb0 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.162+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99641012c0 con 0x7f99640ffeb0 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.162+0000 7f9961ffb700 1 -- 172.21.15.151:0/2836747287 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994c017400 con 0x7f99640ffeb0 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.162+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99641a2c50 con 0x7f99640ffeb0 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.163+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99641090e0 con 0x7f99640ffeb0 2024-04-16T02:40:12.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.163+0000 7f9961ffb700 1 -- 172.21.15.151:0/2836747287 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f994c00f7f0 con 0x7f99640ffeb0 2024-04-16T02:40:12.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.164+0000 7f9961ffb700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f995006edf0 0x7f99500712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:12.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.164+0000 7f9961ffb700 1 -- 172.21.15.151:0/2836747287 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f994c0905f0 con 0x7f99640ffeb0 2024-04-16T02:40:12.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.164+0000 7f9963fff700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f995006edf0 0x7f99500712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:12.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.164+0000 7f9963fff700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f995006edf0 0x7f99500712b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9954005950 tx=0x7f99540058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:12.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.168+0000 7f9961ffb700 1 -- 172.21.15.151:0/2836747287 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f994c059190 con 0x7f99640ffeb0 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: mgrmap e40: smithi151.uqqame(active, since 17s), standbys: smithi153.esbtzf 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: overall HEALTH_OK 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: mon.smithi151 calling monitor election 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: mon.smithi153 calling monitor election 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: mon.smithi151 is new leader, mons smithi151,smithi153 in quorum (ranks 0,1) 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: monmap epoch 3 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: last_changed 2024-04-16T02:40:11.093095+0000 2024-04-16T02:40:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: created 2024-04-16T02:25:31.524258+0000 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: min_mon_release 19 (squid) 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: election_strategy: 1 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.smithi151 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.smithi153 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: mgrmap e40: smithi151.uqqame(active, since 17s), standbys: smithi153.esbtzf 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: overall HEALTH_OK 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:12.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:12 smithi153 ceph-mon[87858]: pgmap v12: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:12.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.378+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f996402cce0 con 0x7f995006edf0 2024-04-16T02:40:12.381 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T02:40:12.381 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: mgrmap e40: smithi151.uqqame(active, since 17s), standbys: smithi153.esbtzf 2024-04-16T02:40:12.381 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: overall HEALTH_OK 2024-04-16T02:40:12.381 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: mon.smithi151 calling monitor election 2024-04-16T02:40:12.381 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: mon.smithi153 calling monitor election 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: mon.smithi151 is new leader, mons smithi151,smithi153 in quorum (ranks 0,1) 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: monmap epoch 3 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: last_changed 2024-04-16T02:40:11.093095+0000 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: created 2024-04-16T02:25:31.524258+0000 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: min_mon_release 19 (squid) 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: election_strategy: 1 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.smithi151 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.smithi153 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: osdmap e50: 6 total, 6 up, 6 in 2024-04-16T02:40:12.382 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: mgrmap e40: smithi151.uqqame(active, since 17s), standbys: smithi153.esbtzf 2024-04-16T02:40:12.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: overall HEALTH_OK 2024-04-16T02:40:12.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi151"}]: dispatch 2024-04-16T02:40:12.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:12.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-16T02:40:12.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:12.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:40:12.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:12.383 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:12 smithi151 ceph-mon[112233]: pgmap v12: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:12.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.389+0000 7f9961ffb700 1 -- 172.21.15.151:0/2836747287 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f996402cce0 con 0x7f995006edf0 2024-04-16T02:40:12.392 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:40:12.392 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (108s) 12s ago 13m 17.5M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:40:12.392 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (13m) 12s ago 13m 8174k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:40:12.392 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (12m) 15s ago 12m 8044k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:40:12.392 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (13m) 12s ago 13m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:40:12.392 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (12m) 15s ago 12m 6656k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:40:12.392 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (85s) 12s ago 12m 75.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (7m) 12s ago 7m 120M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (7m) 12s ago 7m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (7m) 15s ago 7m 58.4M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (7m) 15s ago 7m 67.2M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (3m) 12s ago 14m 473M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (2m) 15s ago 12m 447M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (27s) 12s ago 14m 29.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 starting - - - 2048M 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (2m) 12s ago 13m 15.6M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (2m) 15s ago 12m 16.4M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (11m) 12s ago 11m 224M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (10m) 12s ago 10m 250M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (10m) 12s ago 10m 195M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (9m) 15s ago 9m 281M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:40:12.393 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (9m) 15s ago 9m 230M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:40:12.394 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (9m) 15s ago 9m 245M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:40:12.394 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (2m) 12s ago 12m 68.8M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:40:12.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f995006edf0 msgr2=0x7f99500712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f995006edf0 0x7f99500712b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9954005950 tx=0x7f99540058e0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 msgr2=0x7f99641009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 0x7f99641009c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f994c00bd70 tx=0x7f994c00be50 comp rx=0 tx=0).stop 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 shutdown_connections 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f995006edf0 0x7f99500712b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99640ff500 0x7f9964100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 --2- 172.21.15.151:0/2836747287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99640ffeb0 0x7f99641009c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 >> 172.21.15.151:0/2836747287 conn(0x7f9964074d80 msgr2=0x7f99641050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 shutdown_connections 2024-04-16T02:40:12.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.392+0000 7f996a523700 1 -- 172.21.15.151:0/2836747287 wait complete. 2024-04-16T02:40:12.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.495+0000 7f440d123700 1 -- 172.21.15.151:0/2144902234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 msgr2=0x7f4408100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:12.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.495+0000 7f440d123700 1 --2- 172.21.15.151:0/2144902234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 0x7f4408100050 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009910 tx=0x7f43fc009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:12.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.495+0000 7f440d123700 1 -- 172.21.15.151:0/2144902234 shutdown_connections 2024-04-16T02:40:12.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.496+0000 7f440d123700 1 --2- 172.21.15.151:0/2144902234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 0x7f4408100050 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.496+0000 7f440d123700 1 --2- 172.21.15.151:0/2144902234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 0x7f4408105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.496+0000 7f440d123700 1 -- 172.21.15.151:0/2144902234 >> 172.21.15.151:0/2144902234 conn(0x7f44080755a0 msgr2=0x7f44080759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:12.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.496+0000 7f440d123700 1 -- 172.21.15.151:0/2144902234 shutdown_connections 2024-04-16T02:40:12.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.496+0000 7f440d123700 1 -- 172.21.15.151:0/2144902234 wait complete. 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.496+0000 7f440d123700 1 Processor -- start 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440d123700 1 -- start start 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440d123700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 0x7f4408196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440d123700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 0x7f4408196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440d123700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44081972e0 con 0x7f44080ffbd0 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440d123700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440819a010 con 0x7f4408105a40 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f4406d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 0x7f4408196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f4406d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 0x7f4408196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46360/0 (socket says 172.21.15.151:46360) 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f4406d9d700 1 -- 172.21.15.151:0/42825215 learned_addr learned my addr 172.21.15.151:0/42825215 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440659c700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 0x7f4408196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440659c700 1 -- 172.21.15.151:0/42825215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 msgr2=0x7f4408196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-16T02:40:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440659c700 1 -- 172.21.15.151:0/42825215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 msgr2=0x7f4408196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440659c700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 0x7f4408196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f440659c700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 0x7f4408196c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f4406d9d700 1 -- 172.21.15.151:0/42825215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 msgr2=0x7f4408196c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f4406d9d700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 0x7f4408196c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.497+0000 7f4406d9d700 1 -- 172.21.15.151:0/42825215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc0095f0 con 0x7f44080ffbd0 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.498+0000 7f4406d9d700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 0x7f4408196710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43f800d8d0 tx=0x7f43f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.498+0000 7f43effff700 1 -- 172.21.15.151:0/42825215 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43f8004a00 con 0x7f44080ffbd0 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.498+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440819a2f0 con 0x7f44080ffbd0 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.498+0000 7f43effff700 1 -- 172.21.15.151:0/42825215 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43f80054e0 con 0x7f44080ffbd0 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.498+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440819a840 con 0x7f44080ffbd0 2024-04-16T02:40:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.498+0000 7f43effff700 1 -- 172.21.15.151:0/42825215 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43f8009d50 con 0x7f44080ffbd0 2024-04-16T02:40:12.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.499+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440804ea90 con 0x7f44080ffbd0 2024-04-16T02:40:12.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.499+0000 7f43effff700 1 -- 172.21.15.151:0/42825215 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f43f8005000 con 0x7f44080ffbd0 2024-04-16T02:40:12.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.500+0000 7f43effff700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f43f006ee00 0x7f43f00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:12.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.500+0000 7f440659c700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f43f006ee00 0x7f43f00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:12.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.500+0000 7f43effff700 1 -- 172.21.15.151:0/42825215 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f43f808efd0 con 0x7f44080ffbd0 2024-04-16T02:40:12.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.500+0000 7f440659c700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f43f006ee00 0x7f43f00712c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4408197d30 tx=0x7f43fc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:12.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.505+0000 7f43effff700 1 -- 172.21.15.151:0/42825215 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f43f8057c10 con 0x7f44080ffbd0 2024-04-16T02:40:12.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.788+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4408197a70 con 0x7f44080ffbd0 2024-04-16T02:40:12.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.789+0000 7f43effff700 1 -- 172.21.15.151:0/42825215 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f43f805b2d0 con 0x7f44080ffbd0 2024-04-16T02:40:12.791 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:40:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T02:40:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:40:12.793 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f43f006ee00 msgr2=0x7f43f00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f43f006ee00 0x7f43f00712c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4408197d30 tx=0x7f43fc01a040 comp rx=0 tx=0).stop 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 msgr2=0x7f4408196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 0x7f4408196710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43f800d8d0 tx=0x7f43f800dc90 comp rx=0 tx=0).stop 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 shutdown_connections 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f43f006ee00 0x7f43f00712c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44080ffbd0 0x7f4408196710 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 --2- 172.21.15.151:0/42825215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4408105a40 0x7f4408196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 >> 172.21.15.151:0/42825215 conn(0x7f44080755a0 msgr2=0x7f44080684b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 shutdown_connections 2024-04-16T02:40:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.791+0000 7f440d123700 1 -- 172.21.15.151:0/42825215 wait complete. 2024-04-16T02:40:12.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.896+0000 7febd5666700 1 -- 172.21.15.151:0/95347769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0100da0 msgr2=0x7febd0101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:12.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.896+0000 7febd5666700 1 --2- 172.21.15.151:0/95347769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0100da0 0x7febd0101180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7febc4009910 tx=0x7febc4009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:12.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.896+0000 7febd5666700 1 -- 172.21.15.151:0/95347769 shutdown_connections 2024-04-16T02:40:12.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.896+0000 7febd5666700 1 --2- 172.21.15.151:0/95347769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0101750 0x7febd01057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.896+0000 7febd5666700 1 --2- 172.21.15.151:0/95347769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0100da0 0x7febd0101180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.896+0000 7febd5666700 1 -- 172.21.15.151:0/95347769 >> 172.21.15.151:0/95347769 conn(0x7febd00fc650 msgr2=0x7febd00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:12.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.896+0000 7febd5666700 1 -- 172.21.15.151:0/95347769 shutdown_connections 2024-04-16T02:40:12.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.896+0000 7febd5666700 1 -- 172.21.15.151:0/95347769 wait complete. 2024-04-16T02:40:12.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febd5666700 1 Processor -- start 2024-04-16T02:40:12.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febd5666700 1 -- start start 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febd5666700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0100da0 0x7febd0198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febd5666700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0101750 0x7febd0198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febd5666700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd0199520 con 0x7febd0101750 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febd5666700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd019d330 con 0x7febd0100da0 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febce7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0101750 0x7febd0198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febce7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0101750 0x7febd0198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46386/0 (socket says 172.21.15.151:46386) 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febce7fc700 1 -- 172.21.15.151:0/4209534555 learned_addr learned my addr 172.21.15.151:0/4209534555 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.897+0000 7febceffd700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0100da0 0x7febd0198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febceffd700 1 -- 172.21.15.151:0/4209534555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0100da0 msgr2=0x7febd0198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febceffd700 1 -- 172.21.15.151:0/4209534555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0100da0 msgr2=0x7febd0198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febceffd700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0100da0 0x7febd0198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febceffd700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0100da0 0x7febd0198950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:40:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febce7fc700 1 -- 172.21.15.151:0/4209534555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0100da0 msgr2=0x7febd0198950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:40:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febce7fc700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0100da0 0x7febd0198950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febce7fc700 1 -- 172.21.15.151:0/4209534555 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40095f0 con 0x7febd0101750 2024-04-16T02:40:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febce7fc700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0101750 0x7febd0198e90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febc000d900 tx=0x7febc000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febb7fff700 1 -- 172.21.15.151:0/4209534555 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febc0009900 con 0x7febd0101750 2024-04-16T02:40:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febb7fff700 1 -- 172.21.15.151:0/4209534555 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febc0010460 con 0x7febd0101750 2024-04-16T02:40:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febb7fff700 1 -- 172.21.15.151:0/4209534555 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febc000f5f0 con 0x7febd0101750 2024-04-16T02:40:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd019d610 con 0x7febd0101750 2024-04-16T02:40:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.898+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd019db60 con 0x7febd0101750 2024-04-16T02:40:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.899+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd0104f20 con 0x7febd0101750 2024-04-16T02:40:12.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.900+0000 7febb7fff700 1 -- 172.21.15.151:0/4209534555 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7febc00105d0 con 0x7febd0101750 2024-04-16T02:40:12.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.900+0000 7febb7fff700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7febb806ee00 0x7febb80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:12.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.900+0000 7febceffd700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7febb806ee00 0x7febb80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:12.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.900+0000 7febb7fff700 1 -- 172.21.15.151:0/4209534555 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7febc008eea0 con 0x7febd0101750 2024-04-16T02:40:12.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.901+0000 7febceffd700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7febb806ee00 0x7febb80712c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7febc400b5c0 tx=0x7febc401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:12.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:12.905+0000 7febb7fff700 1 -- 172.21.15.151:0/4209534555 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7febc005b610 con 0x7febd0101750 2024-04-16T02:40:13.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.149+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7febd004ea90 con 0x7febd0101750 2024-04-16T02:40:13.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.149+0000 7febb7fff700 1 -- 172.21.15.151:0/4209534555 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1919 (secure 0 0 0) 0x7febc005b1a0 con 0x7febd0101750 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:e22 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 22 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:40:05.114230+0000 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:40:13.152 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:40:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 14478 members: 14490,14478 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:40:13.154 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:40:13.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7febb806ee00 msgr2=0x7febb80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:13.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7febb806ee00 0x7febb80712c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7febc400b5c0 tx=0x7febc401a040 comp rx=0 tx=0).stop 2024-04-16T02:40:13.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0101750 msgr2=0x7febd0198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:13.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0101750 0x7febd0198e90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febc000d900 tx=0x7febc000dcc0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 shutdown_connections 2024-04-16T02:40:13.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7febb806ee00 0x7febb80712c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febd0100da0 0x7febd0198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 --2- 172.21.15.151:0/4209534555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febd0101750 0x7febd0198e90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 >> 172.21.15.151:0/4209534555 conn(0x7febd00fc650 msgr2=0x7febd00fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:13.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 shutdown_connections 2024-04-16T02:40:13.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.152+0000 7febd5666700 1 -- 172.21.15.151:0/4209534555 wait complete. 2024-04-16T02:40:13.156 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 22 2024-04-16T02:40:13.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 -- 172.21.15.151:0/3459384052 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40100dc0 msgr2=0x7fea401011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:13.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 --2- 172.21.15.151:0/3459384052 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40100dc0 0x7fea401011a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea34009910 tx=0x7fea34009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:13.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 -- 172.21.15.151:0/3459384052 shutdown_connections 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 --2- 172.21.15.151:0/3459384052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40101770 0x7fea401057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 --2- 172.21.15.151:0/3459384052 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40100dc0 0x7fea401011a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 -- 172.21.15.151:0/3459384052 >> 172.21.15.151:0/3459384052 conn(0x7fea400fc650 msgr2=0x7fea400fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 -- 172.21.15.151:0/3459384052 shutdown_connections 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 -- 172.21.15.151:0/3459384052 wait complete. 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 Processor -- start 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.258+0000 7fea46c73700 1 -- start start 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea46c73700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40100dc0 0x7fea4019c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea46c73700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40101770 0x7fea4019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea46c73700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4019d110 con 0x7fea40101770 2024-04-16T02:40:13.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea46c73700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea401965c0 con 0x7fea40100dc0 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40101770 0x7fea4019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40101770 0x7fea4019ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46404/0 (socket says 172.21.15.151:46404) 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3ffff700 1 -- 172.21.15.151:0/4152061374 learned_addr learned my addr 172.21.15.151:0/4152061374 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea44a0f700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40100dc0 0x7fea4019c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea44a0f700 1 -- 172.21.15.151:0/4152061374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40100dc0 msgr2=0x7fea4019c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea44a0f700 1 -- 172.21.15.151:0/4152061374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40100dc0 msgr2=0x7fea4019c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea44a0f700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40100dc0 0x7fea4019c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea44a0f700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40100dc0 0x7fea4019c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3ffff700 1 -- 172.21.15.151:0/4152061374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40100dc0 msgr2=0x7fea4019c540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3ffff700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40100dc0 0x7fea4019c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3ffff700 1 -- 172.21.15.151:0/4152061374 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea340095f0 con 0x7fea40101770 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3ffff700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40101770 0x7fea4019ca80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00d900 tx=0x7fea2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:13.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3dffb700 1 -- 172.21.15.151:0/4152061374 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea2c009900 con 0x7fea40101770 2024-04-16T02:40:13.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3dffb700 1 -- 172.21.15.151:0/4152061374 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea2c010460 con 0x7fea40101770 2024-04-16T02:40:13.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea401968a0 con 0x7fea40101770 2024-04-16T02:40:13.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.259+0000 7fea3dffb700 1 -- 172.21.15.151:0/4152061374 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea2c00f5f0 con 0x7fea40101770 2024-04-16T02:40:13.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.260+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea40196df0 con 0x7fea40101770 2024-04-16T02:40:13.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.260+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea40109160 con 0x7fea40101770 2024-04-16T02:40:13.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.261+0000 7fea3dffb700 1 -- 172.21.15.151:0/4152061374 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7fea2c0105d0 con 0x7fea40101770 2024-04-16T02:40:13.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.261+0000 7fea3dffb700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fea3006ee00 0x7fea300712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:13.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.261+0000 7fea3dffb700 1 -- 172.21.15.151:0/4152061374 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fea2c08eea0 con 0x7fea40101770 2024-04-16T02:40:13.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.262+0000 7fea44a0f700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fea3006ee00 0x7fea300712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:13.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.262+0000 7fea44a0f700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fea3006ee00 0x7fea300712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fea3400b5c0 tx=0x7fea3401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:13.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.266+0000 7fea3dffb700 1 -- 172.21.15.151:0/4152061374 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fea2c05b610 con 0x7fea40101770 2024-04-16T02:40:13.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.484+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea40197090 con 0x7fea3006ee00 2024-04-16T02:40:13.487 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:13 smithi151 ceph-mon[112233]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:13.487 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:13 smithi151 ceph-mon[112233]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:13.487 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:13 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/42825215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:13.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.485+0000 7fea3dffb700 1 -- 172.21.15.151:0/4152061374 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fea40197090 con 0x7fea3006ee00 2024-04-16T02:40:13.488 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:40:13.488 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:40:13.488 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:40:13.488 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:40:13.489 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:40:13.489 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr" 2024-04-16T02:40:13.489 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:40:13.489 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "3/23 daemons upgraded", 2024-04-16T02:40:13.489 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mon daemons", 2024-04-16T02:40:13.489 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:40:13.489 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:40:13.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fea3006ee00 msgr2=0x7fea300712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:13.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fea3006ee00 0x7fea300712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fea3400b5c0 tx=0x7fea3401a040 comp rx=0 tx=0).stop 2024-04-16T02:40:13.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40101770 msgr2=0x7fea4019ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:13.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40101770 0x7fea4019ca80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00d900 tx=0x7fea2c00dcc0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 shutdown_connections 2024-04-16T02:40:13.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fea3006ee00 0x7fea300712c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea40100dc0 0x7fea4019c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 --2- 172.21.15.151:0/4152061374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea40101770 0x7fea4019ca80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 >> 172.21.15.151:0/4152061374 conn(0x7fea400fc650 msgr2=0x7fea400fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:13.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 shutdown_connections 2024-04-16T02:40:13.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.488+0000 7fea46c73700 1 -- 172.21.15.151:0/4152061374 wait complete. 2024-04-16T02:40:13.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:13 smithi153 ceph-mon[87858]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:13.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:13 smithi153 ceph-mon[87858]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:13.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:13 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/42825215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:13.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.599+0000 7f64649d5700 1 -- 172.21.15.151:0/3552449958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6460105a40 msgr2=0x7f6460105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:13.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.599+0000 7f64649d5700 1 --2- 172.21.15.151:0/3552449958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6460105a40 0x7f6460105e20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6448009910 tx=0x7f6448009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:13.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.599+0000 7f64649d5700 1 -- 172.21.15.151:0/3552449958 shutdown_connections 2024-04-16T02:40:13.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.599+0000 7f64649d5700 1 --2- 172.21.15.151:0/3552449958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64600ffbd0 0x7f6460100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.599+0000 7f64649d5700 1 --2- 172.21.15.151:0/3552449958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6460105a40 0x7f6460105e20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.599+0000 7f64649d5700 1 -- 172.21.15.151:0/3552449958 >> 172.21.15.151:0/3552449958 conn(0x7f64600755a0 msgr2=0x7f64600759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:13.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.599+0000 7f64649d5700 1 -- 172.21.15.151:0/3552449958 shutdown_connections 2024-04-16T02:40:13.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.599+0000 7f64649d5700 1 -- 172.21.15.151:0/3552449958 wait complete. 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f64649d5700 1 Processor -- start 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f64649d5700 1 -- start start 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f64649d5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600ffbd0 0x7f64601953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f64649d5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460105a40 0x7f64601958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f64649d5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6460195e20 con 0x7f64600ffbd0 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f64649d5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6460195f90 con 0x7f6460105a40 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f645e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600ffbd0 0x7f64601953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f645e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600ffbd0 0x7f64601953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46418/0 (socket says 172.21.15.151:46418) 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f645e59c700 1 -- 172.21.15.151:0/1636903149 learned_addr learned my addr 172.21.15.151:0/1636903149 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f645dd9b700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460105a40 0x7f64601958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f645dd9b700 1 -- 172.21.15.151:0/1636903149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460105a40 msgr2=0x7f64601958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f645dd9b700 1 -- 172.21.15.151:0/1636903149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460105a40 msgr2=0x7f64601958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f645dd9b700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460105a40 0x7f64601958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.600+0000 7f645dd9b700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460105a40 0x7f64601958e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.601+0000 7f645e59c700 1 -- 172.21.15.151:0/1636903149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460105a40 msgr2=0x7f64601958e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.601+0000 7f645e59c700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460105a40 0x7f64601958e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.601+0000 7f645e59c700 1 -- 172.21.15.151:0/1636903149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64480095f0 con 0x7f64600ffbd0 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.601+0000 7f645e59c700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600ffbd0 0x7f64601953a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6448006010 tx=0x7f644800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.601+0000 7f64577fe700 1 -- 172.21.15.151:0/1636903149 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f644801d070 con 0x7f64600ffbd0 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.601+0000 7f64577fe700 1 -- 172.21.15.151:0/1636903149 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f644800f460 con 0x7f64600ffbd0 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.601+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6460196210 con 0x7f64600ffbd0 2024-04-16T02:40:13.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.601+0000 7f64577fe700 1 -- 172.21.15.151:0/1636903149 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6448017630 con 0x7f64600ffbd0 2024-04-16T02:40:13.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.601+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64601a2c90 con 0x7f64600ffbd0 2024-04-16T02:40:13.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.602+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6460108040 con 0x7f64600ffbd0 2024-04-16T02:40:13.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.603+0000 7f64577fe700 1 -- 172.21.15.151:0/1636903149 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f644800f5d0 con 0x7f64600ffbd0 2024-04-16T02:40:13.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.604+0000 7f64577fe700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f644c06ee00 0x7f644c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:13.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.604+0000 7f64577fe700 1 -- 172.21.15.151:0/1636903149 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6448090360 con 0x7f64600ffbd0 2024-04-16T02:40:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.604+0000 7f645dd9b700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f644c06ee00 0x7f644c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.604+0000 7f645dd9b700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f644c06ee00 0x7f644c0712c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6460068d70 tx=0x7f645000b2c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:13.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.609+0000 7f64577fe700 1 -- 172.21.15.151:0/1636903149 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f644805cad0 con 0x7f64600ffbd0 2024-04-16T02:40:13.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.893+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6460196210 con 0x7f64600ffbd0 2024-04-16T02:40:13.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.894+0000 7f64577fe700 1 -- 172.21.15.151:0/1636903149 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f6460196210 con 0x7f64600ffbd0 2024-04-16T02:40:13.896 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_OK 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f644c06ee00 msgr2=0x7f644c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f644c06ee00 0x7f644c0712c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6460068d70 tx=0x7f645000b2c0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600ffbd0 msgr2=0x7f64601953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600ffbd0 0x7f64601953a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6448006010 tx=0x7f644800be20 comp rx=0 tx=0).stop 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 shutdown_connections 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f644c06ee00 0x7f644c0712c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600ffbd0 0x7f64601953a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 --2- 172.21.15.151:0/1636903149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460105a40 0x7f64601958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 >> 172.21.15.151:0/1636903149 conn(0x7f64600755a0 msgr2=0x7f64600fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 shutdown_connections 2024-04-16T02:40:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:13.896+0000 7f64649d5700 1 -- 172.21.15.151:0/1636903149 wait complete. 2024-04-16T02:40:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:14 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4209534555' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:40:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:14 smithi151 ceph-mon[112233]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:14 smithi151 ceph-mon[112233]: pgmap v13: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:14 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1636903149' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:40:14.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:14 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4209534555' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:40:14.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:14 smithi153 ceph-mon[87858]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:14.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:14 smithi153 ceph-mon[87858]: pgmap v13: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:14.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:14 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1636903149' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:40:15.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:15.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:15.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:15.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:16.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:16.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:16.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:16 smithi153 ceph-mon[87858]: pgmap v14: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:16.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:16.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:16.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:16 smithi151 ceph-mon[112233]: pgmap v14: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:18.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:17 smithi151 ceph-mon[112233]: pgmap v15: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:18.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:17 smithi153 ceph-mon[87858]: pgmap v15: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:40:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:40:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:40:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:40:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:40:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:40:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:20 smithi151 ceph-mon[112233]: Reconfiguring mon.smithi151 (monmap changed)... 2024-04-16T02:40:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:20 smithi151 ceph-mon[112233]: Reconfiguring daemon mon.smithi151 on smithi151 2024-04-16T02:40:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:20 smithi151 ceph-mon[112233]: pgmap v16: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:20.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:20.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:20.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi151.uqqame", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:40:20.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:40:20.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:20 smithi153 ceph-mon[87858]: Reconfiguring mon.smithi151 (monmap changed)... 2024-04-16T02:40:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:20 smithi153 ceph-mon[87858]: Reconfiguring daemon mon.smithi151 on smithi151 2024-04-16T02:40:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:20 smithi153 ceph-mon[87858]: pgmap v16: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi151.uqqame", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:40:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:40:21.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:21.984 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:21 smithi151 ceph-mon[112233]: Reconfiguring mgr.smithi151.uqqame (monmap changed)... 2024-04-16T02:40:21.984 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:21 smithi151 ceph-mon[112233]: Reconfiguring daemon mgr.smithi151.uqqame on smithi151 2024-04-16T02:40:22.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:21 smithi153 ceph-mon[87858]: Reconfiguring mgr.smithi151.uqqame (monmap changed)... 2024-04-16T02:40:22.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:21 smithi153 ceph-mon[87858]: Reconfiguring daemon mgr.smithi151.uqqame on smithi151 2024-04-16T02:40:22.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:22 smithi151 ceph-mon[112233]: pgmap v17: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:22.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:22 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:22.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:22 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:22.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:22 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:40:22.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:22 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:40:22.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:22 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-16T02:40:22.922 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:22 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi151"}]: dispatch 2024-04-16T02:40:22.923 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:22 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:23.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:22 smithi153 ceph-mon[87858]: pgmap v17: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:23.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:22 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:23.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:22 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:23.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:22 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:40:23.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:22 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:40:23.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:22 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-16T02:40:23.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:22 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi151"}]: dispatch 2024-04-16T02:40:23.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:22 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:23 smithi151 ceph-mon[112233]: Reconfiguring ceph-exporter.smithi151 (monmap changed)... 2024-04-16T02:40:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:23 smithi151 ceph-mon[112233]: Unable to update caps for client.ceph-exporter.smithi151 2024-04-16T02:40:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:23 smithi151 ceph-mon[112233]: Reconfiguring daemon ceph-exporter.smithi151 on smithi151 2024-04-16T02:40:24.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:23 smithi153 ceph-mon[87858]: Reconfiguring ceph-exporter.smithi151 (monmap changed)... 2024-04-16T02:40:24.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:23 smithi153 ceph-mon[87858]: Unable to update caps for client.ceph-exporter.smithi151 2024-04-16T02:40:24.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:23 smithi153 ceph-mon[87858]: Reconfiguring daemon ceph-exporter.smithi151 on smithi151 2024-04-16T02:40:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:24 smithi151 ceph-mon[112233]: pgmap v18: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:24 smithi151 ceph-mon[112233]: Reconfiguring crash.smithi151 (monmap changed)... 2024-04-16T02:40:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi151", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:40:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:24 smithi151 ceph-mon[112233]: Reconfiguring daemon crash.smithi151 on smithi151 2024-04-16T02:40:24.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:40:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:24 smithi153 ceph-mon[87858]: pgmap v18: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:24 smithi153 ceph-mon[87858]: Reconfiguring crash.smithi151 (monmap changed)... 2024-04-16T02:40:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi151", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:40:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:24 smithi153 ceph-mon[87858]: Reconfiguring daemon crash.smithi151 on smithi151 2024-04-16T02:40:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:40:26.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:26.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:26.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:26 smithi151 ceph-mon[112233]: Reconfiguring osd.0 (monmap changed)... 2024-04-16T02:40:26.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T02:40:26.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:26.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:26 smithi151 ceph-mon[112233]: Reconfiguring daemon osd.0 on smithi151 2024-04-16T02:40:26.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:26 smithi151 ceph-mon[112233]: pgmap v19: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:26.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:26.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:26.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:26 smithi153 ceph-mon[87858]: Reconfiguring osd.0 (monmap changed)... 2024-04-16T02:40:26.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T02:40:26.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:26.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:26 smithi153 ceph-mon[87858]: Reconfiguring daemon osd.0 on smithi151 2024-04-16T02:40:26.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:26 smithi153 ceph-mon[87858]: pgmap v19: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:28.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:28.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:28.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:28 smithi151 ceph-mon[112233]: Reconfiguring osd.1 (monmap changed)... 2024-04-16T02:40:28.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T02:40:28.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:28.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:28 smithi151 ceph-mon[112233]: Reconfiguring daemon osd.1 on smithi151 2024-04-16T02:40:28.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:28 smithi151 ceph-mon[112233]: pgmap v20: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:28.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:28.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:28.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:28 smithi153 ceph-mon[87858]: Reconfiguring osd.1 (monmap changed)... 2024-04-16T02:40:28.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T02:40:28.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:28.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:28 smithi153 ceph-mon[87858]: Reconfiguring daemon osd.1 on smithi151 2024-04-16T02:40:28.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:28 smithi153 ceph-mon[87858]: pgmap v20: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:30.168 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:29 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:30.168 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:29 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:30.168 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:29 smithi151 ceph-mon[112233]: Reconfiguring osd.2 (monmap changed)... 2024-04-16T02:40:30.168 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:29 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T02:40:30.169 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:29 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:30.169 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:29 smithi151 ceph-mon[112233]: Reconfiguring daemon osd.2 on smithi151 2024-04-16T02:40:30.169 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:29 smithi151 ceph-mon[112233]: pgmap v21: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:40:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:29 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:29 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:29 smithi153 ceph-mon[87858]: Reconfiguring osd.2 (monmap changed)... 2024-04-16T02:40:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:29 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T02:40:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:29 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:29 smithi153 ceph-mon[87858]: Reconfiguring daemon osd.2 on smithi151 2024-04-16T02:40:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:29 smithi153 ceph-mon[87858]: pgmap v21: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:40:31.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:31.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:31.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:31 smithi151 ceph-mon[112233]: Reconfiguring mds.cephfs.smithi151.bdpnxq (monmap changed)... 2024-04-16T02:40:31.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.bdpnxq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:40:31.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:31.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:31 smithi151 ceph-mon[112233]: Reconfiguring daemon mds.cephfs.smithi151.bdpnxq on smithi151 2024-04-16T02:40:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:31 smithi153 ceph-mon[87858]: Reconfiguring mds.cephfs.smithi151.bdpnxq (monmap changed)... 2024-04-16T02:40:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.bdpnxq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:40:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:31 smithi153 ceph-mon[87858]: Reconfiguring daemon mds.cephfs.smithi151.bdpnxq on smithi151 2024-04-16T02:40:32.899 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:32 smithi151 ceph-mon[112233]: pgmap v22: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-16T02:40:32.900 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:32 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:32.900 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:32 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:32.900 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:32 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.slrqog", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:40:32.900 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:32 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:33.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:32 smithi153 ceph-mon[87858]: pgmap v22: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-16T02:40:33.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:32 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:33.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:32 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:33.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:32 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.slrqog", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:40:33.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:32 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:33.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:33 smithi151 ceph-mon[112233]: Reconfiguring mds.cephfs.smithi151.slrqog (monmap changed)... 2024-04-16T02:40:33.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:33 smithi151 ceph-mon[112233]: Reconfiguring daemon mds.cephfs.smithi151.slrqog on smithi151 2024-04-16T02:40:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:33 smithi153 ceph-mon[87858]: Reconfiguring mds.cephfs.smithi151.slrqog (monmap changed)... 2024-04-16T02:40:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:33 smithi153 ceph-mon[87858]: Reconfiguring daemon mds.cephfs.smithi151.slrqog on smithi151 2024-04-16T02:40:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:34 smithi153 ceph-mon[87858]: pgmap v23: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-16T02:40:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:34 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:34 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:34 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:40:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:34 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:40:34.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:34 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-16T02:40:34.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:34 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi153"}]: dispatch 2024-04-16T02:40:34.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:34 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:34.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:34 smithi151 ceph-mon[112233]: pgmap v23: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-16T02:40:34.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:34 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:34.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:34 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:34.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:34 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:40:34.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:34 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T02:40:34.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:34 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-16T02:40:34.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:34 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi153"}]: dispatch 2024-04-16T02:40:34.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:34 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:35 smithi153 ceph-mon[87858]: Reconfiguring ceph-exporter.smithi153 (monmap changed)... 2024-04-16T02:40:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:35 smithi153 ceph-mon[87858]: Unable to update caps for client.ceph-exporter.smithi153 2024-04-16T02:40:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:35 smithi153 ceph-mon[87858]: Reconfiguring daemon ceph-exporter.smithi153 on smithi153 2024-04-16T02:40:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:35 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:35 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:35 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:40:35.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:35 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:35.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:35 smithi151 ceph-mon[112233]: Reconfiguring ceph-exporter.smithi153 (monmap changed)... 2024-04-16T02:40:35.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:35 smithi151 ceph-mon[112233]: Unable to update caps for client.ceph-exporter.smithi153 2024-04-16T02:40:35.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:35 smithi151 ceph-mon[112233]: Reconfiguring daemon ceph-exporter.smithi153 on smithi153 2024-04-16T02:40:35.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:35 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:35.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:35 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:35.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:35 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:40:35.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:35 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:36.749 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:36 smithi153 ceph-mon[87858]: Reconfiguring crash.smithi153 (monmap changed)... 2024-04-16T02:40:36.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:36 smithi153 ceph-mon[87858]: Reconfiguring daemon crash.smithi153 on smithi153 2024-04-16T02:40:36.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:36 smithi153 ceph-mon[87858]: pgmap v24: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:40:36.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:36 smithi151 ceph-mon[112233]: Reconfiguring crash.smithi153 (monmap changed)... 2024-04-16T02:40:36.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:36 smithi151 ceph-mon[112233]: Reconfiguring daemon crash.smithi153 on smithi153 2024-04-16T02:40:36.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:36 smithi151 ceph-mon[112233]: pgmap v24: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:40:38.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:38.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:38.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:38 smithi153 ceph-mon[87858]: Reconfiguring mgr.smithi153.esbtzf (monmap changed)... 2024-04-16T02:40:38.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:40:38.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:40:38.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:38.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:38 smithi153 ceph-mon[87858]: Reconfiguring daemon mgr.smithi153.esbtzf on smithi153 2024-04-16T02:40:38.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:38 smithi153 ceph-mon[87858]: pgmap v25: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-16T02:40:38.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:38.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:38.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:38 smithi151 ceph-mon[112233]: Reconfiguring mgr.smithi153.esbtzf (monmap changed)... 2024-04-16T02:40:38.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.esbtzf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T02:40:38.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T02:40:38.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:38.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:38 smithi151 ceph-mon[112233]: Reconfiguring daemon mgr.smithi153.esbtzf on smithi153 2024-04-16T02:40:38.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:38 smithi151 ceph-mon[112233]: pgmap v25: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-16T02:40:39.564 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:40:39.564 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:39.564 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:39.564 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:40:39.564 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:40:39.564 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:39.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:40:39.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:39.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:39.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T02:40:39.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T02:40:39.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:40.353 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:40 smithi153 ceph-mon[87858]: Reconfiguring mon.smithi153 (monmap changed)... 2024-04-16T02:40:40.353 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:40 smithi153 ceph-mon[87858]: Reconfiguring daemon mon.smithi153 on smithi153 2024-04-16T02:40:40.353 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:40 smithi153 ceph-mon[87858]: pgmap v26: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:40:40.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:40 smithi151 ceph-mon[112233]: Reconfiguring mon.smithi153 (monmap changed)... 2024-04-16T02:40:40.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:40 smithi151 ceph-mon[112233]: Reconfiguring daemon mon.smithi153 on smithi153 2024-04-16T02:40:40.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:40 smithi151 ceph-mon[112233]: pgmap v26: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:40:41.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:41 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:41.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:41 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:41.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:41 smithi151 ceph-mon[112233]: Reconfiguring osd.3 (monmap changed)... 2024-04-16T02:40:41.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:41 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T02:40:41.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:41 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:41.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:41 smithi151 ceph-mon[112233]: Reconfiguring daemon osd.3 on smithi153 2024-04-16T02:40:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:41 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:41 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:41 smithi153 ceph-mon[87858]: Reconfiguring osd.3 (monmap changed)... 2024-04-16T02:40:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:41 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T02:40:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:41 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:41 smithi153 ceph-mon[87858]: Reconfiguring daemon osd.3 on smithi153 2024-04-16T02:40:42.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:42 smithi151 ceph-mon[112233]: pgmap v27: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:42.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:42 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:42.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:42 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:42.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:42 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T02:40:42.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:42 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:42.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:42 smithi153 ceph-mon[87858]: pgmap v27: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:42.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:42 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:42.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:42 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:42.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:42 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T02:40:42.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:42 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:43.575 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:43 smithi153 ceph-mon[87858]: Reconfiguring osd.4 (monmap changed)... 2024-04-16T02:40:43.575 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:43 smithi153 ceph-mon[87858]: Reconfiguring daemon osd.4 on smithi153 2024-04-16T02:40:43.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:43 smithi151 ceph-mon[112233]: Reconfiguring osd.4 (monmap changed)... 2024-04-16T02:40:43.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:43 smithi151 ceph-mon[112233]: Reconfiguring daemon osd.4 on smithi153 2024-04-16T02:40:44.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.022+0000 7f877d5e8700 1 -- 172.21.15.151:0/1605262605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 msgr2=0x7f8778104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.022+0000 7f877d5e8700 1 --2- 172.21.15.151:0/1605262605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 0x7f8778104e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8768009910 tx=0x7f8768009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:44.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.022+0000 7f877d5e8700 1 -- 172.21.15.151:0/1605262605 shutdown_connections 2024-04-16T02:40:44.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.022+0000 7f877d5e8700 1 --2- 172.21.15.151:0/1605262605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778100140 0x7f87781005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.022+0000 7f877d5e8700 1 --2- 172.21.15.151:0/1605262605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 0x7f8778104e30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.023+0000 7f877d5e8700 1 -- 172.21.15.151:0/1605262605 >> 172.21.15.151:0/1605262605 conn(0x7f8778075e30 msgr2=0x7f87780fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:44.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.023+0000 7f877d5e8700 1 -- 172.21.15.151:0/1605262605 shutdown_connections 2024-04-16T02:40:44.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.023+0000 7f877d5e8700 1 -- 172.21.15.151:0/1605262605 wait complete. 2024-04-16T02:40:44.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.023+0000 7f877d5e8700 1 Processor -- start 2024-04-16T02:40:44.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.023+0000 7f877d5e8700 1 -- start start 2024-04-16T02:40:44.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.023+0000 7f877d5e8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778100140 0x7f87781955a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:44.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.023+0000 7f877d5e8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 0x7f8778195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:44.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.023+0000 7f877d5e8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87781961c0 con 0x7f8778104a50 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.023+0000 7f877d5e8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8778199f50 con 0x7f8778100140 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f876ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 0x7f8778195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f876ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 0x7f8778195ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38790/0 (socket says 172.21.15.151:38790) 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f876ffff700 1 -- 172.21.15.151:0/601700978 learned_addr learned my addr 172.21.15.151:0/601700978 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f8776ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778100140 0x7f87781955a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f876ffff700 1 -- 172.21.15.151:0/601700978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778100140 msgr2=0x7f87781955a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f876ffff700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778100140 0x7f87781955a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f876ffff700 1 -- 172.21.15.151:0/601700978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680095f0 con 0x7f8778104a50 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f876ffff700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 0x7f8778195ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f876000e940 tx=0x7f876000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f8774ff9700 1 -- 172.21.15.151:0/601700978 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f876000cba0 con 0x7f8778104a50 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f8774ff9700 1 -- 172.21.15.151:0/601700978 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f876000ee90 con 0x7f8778104a50 2024-04-16T02:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877819a230 con 0x7f8778104a50 2024-04-16T02:40:44.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f8774ff9700 1 -- 172.21.15.151:0/601700978 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87600055d0 con 0x7f8778104a50 2024-04-16T02:40:44.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.024+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877819a800 con 0x7f8778104a50 2024-04-16T02:40:44.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.025+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8778108020 con 0x7f8778104a50 2024-04-16T02:40:44.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.026+0000 7f8774ff9700 1 -- 172.21.15.151:0/601700978 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f8760004750 con 0x7f8778104a50 2024-04-16T02:40:44.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.026+0000 7f8774ff9700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f876406ee00 0x7f87640712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:44.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.026+0000 7f8774ff9700 1 -- 172.21.15.151:0/601700978 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8760014070 con 0x7f8778104a50 2024-04-16T02:40:44.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.027+0000 7f8776ffd700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f876406ee00 0x7f87640712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:44.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.027+0000 7f8776ffd700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f876406ee00 0x7f87640712c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f87680059d0 tx=0x7f8768005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:44.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.031+0000 7f8774ff9700 1 -- 172.21.15.151:0/601700978 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8760057fd0 con 0x7f8778104a50 2024-04-16T02:40:44.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.252+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f877819ab40 con 0x7f876406ee00 2024-04-16T02:40:44.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.254+0000 7f8774ff9700 1 -- 172.21.15.151:0/601700978 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f877819ab40 con 0x7f876406ee00 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f876406ee00 msgr2=0x7f87640712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f876406ee00 0x7f87640712c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f87680059d0 tx=0x7f8768005940 comp rx=0 tx=0).stop 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 msgr2=0x7f8778195ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 0x7f8778195ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f876000e940 tx=0x7f876000ec50 comp rx=0 tx=0).stop 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 shutdown_connections 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f876406ee00 0x7f87640712c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778100140 0x7f87781955a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 --2- 172.21.15.151:0/601700978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8778104a50 0x7f8778195ae0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 >> 172.21.15.151:0/601700978 conn(0x7f8778075e30 msgr2=0x7f8778103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:44.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 shutdown_connections 2024-04-16T02:40:44.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.256+0000 7f877d5e8700 1 -- 172.21.15.151:0/601700978 wait complete. 2024-04-16T02:40:44.269 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:40:44.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 -- 172.21.15.151:0/1028979908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 msgr2=0x7f1d8c1057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 --2- 172.21.15.151:0/1028979908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 0x7f1d8c1057d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c009910 tx=0x7f1d7c009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:44.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 -- 172.21.15.151:0/1028979908 shutdown_connections 2024-04-16T02:40:44.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 --2- 172.21.15.151:0/1028979908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 0x7f1d8c1057d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 --2- 172.21.15.151:0/1028979908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c100dd0 0x7f1d8c1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 -- 172.21.15.151:0/1028979908 >> 172.21.15.151:0/1028979908 conn(0x7f1d8c0fc640 msgr2=0x7f1d8c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:44.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 -- 172.21.15.151:0/1028979908 shutdown_connections 2024-04-16T02:40:44.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 -- 172.21.15.151:0/1028979908 wait complete. 2024-04-16T02:40:44.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 Processor -- start 2024-04-16T02:40:44.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.363+0000 7f1d94564700 1 -- start start 2024-04-16T02:40:44.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d94564700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c100dd0 0x7f1d8c19c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:44.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d94564700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 0x7f1d8c19cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:44.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d94564700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c19d160 con 0x7f1d8c101780 2024-04-16T02:40:44.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d94564700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c196610 con 0x7f1d8c100dd0 2024-04-16T02:40:44.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d91aff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 0x7f1d8c19cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:44.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d91aff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 0x7f1d8c19cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38802/0 (socket says 172.21.15.151:38802) 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d91aff700 1 -- 172.21.15.151:0/2921656718 learned_addr learned my addr 172.21.15.151:0/2921656718 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d91aff700 1 -- 172.21.15.151:0/2921656718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c100dd0 msgr2=0x7f1d8c19c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d91aff700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c100dd0 0x7f1d8c19c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d91aff700 1 -- 172.21.15.151:0/2921656718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0095f0 con 0x7f1d8c101780 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d91aff700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 0x7f1d8c19cad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0092e0 tx=0x7f1d7c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d837fe700 1 -- 172.21.15.151:0/2921656718 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7c01d070 con 0x7f1d8c101780 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d837fe700 1 -- 172.21.15.151:0/2921656718 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d7c005dc0 con 0x7f1d8c101780 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c196890 con 0x7f1d8c101780 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d837fe700 1 -- 172.21.15.151:0/2921656718 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7c017400 con 0x7f1d8c101780 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.364+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8c196d80 con 0x7f1d8c101780 2024-04-16T02:40:44.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.365+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d8c1091d0 con 0x7f1d8c101780 2024-04-16T02:40:44.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.366+0000 7f1d837fe700 1 -- 172.21.15.151:0/2921656718 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f1d7c00f460 con 0x7f1d8c101780 2024-04-16T02:40:44.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.366+0000 7f1d837fe700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ee00 0x7f1d780712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:44.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.366+0000 7f1d837fe700 1 -- 172.21.15.151:0/2921656718 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1d7c090560 con 0x7f1d8c101780 2024-04-16T02:40:44.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.366+0000 7f1d92300700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ee00 0x7f1d780712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:44.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.367+0000 7f1d92300700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ee00 0x7f1d780712c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1d84005950 tx=0x7f1d840058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:44.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.371+0000 7f1d837fe700 1 -- 172.21.15.151:0/2921656718 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1d7c05ccd0 con 0x7f1d8c101780 2024-04-16T02:40:44.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.589+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d8c197060 con 0x7f1d7806ee00 2024-04-16T02:40:44.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.590+0000 7f1d837fe700 1 -- 172.21.15.151:0/2921656718 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f1d8c197060 con 0x7f1d7806ee00 2024-04-16T02:40:44.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ee00 msgr2=0x7f1d780712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ee00 0x7f1d780712c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1d84005950 tx=0x7f1d840058e0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 msgr2=0x7f1d8c19cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 0x7f1d8c19cad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0092e0 tx=0x7f1d7c00be20 comp rx=0 tx=0).stop 2024-04-16T02:40:44.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 shutdown_connections 2024-04-16T02:40:44.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ee00 0x7f1d780712c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c100dd0 0x7f1d8c19c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 --2- 172.21.15.151:0/2921656718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c101780 0x7f1d8c19cad0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 >> 172.21.15.151:0/2921656718 conn(0x7f1d8c0fc640 msgr2=0x7f1d8c0fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 shutdown_connections 2024-04-16T02:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.592+0000 7f1d94564700 1 -- 172.21.15.151:0/2921656718 wait complete. 2024-04-16T02:40:44.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.711+0000 7f3f711d6700 1 -- 172.21.15.151:0/4102634088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c100da0 msgr2=0x7f3f6c101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.711+0000 7f3f711d6700 1 --2- 172.21.15.151:0/4102634088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c100da0 0x7f3f6c101180 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c009910 tx=0x7f3f5c009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:44.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.711+0000 7f3f711d6700 1 -- 172.21.15.151:0/4102634088 shutdown_connections 2024-04-16T02:40:44.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.711+0000 7f3f711d6700 1 --2- 172.21.15.151:0/4102634088 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f6c101750 0x7f3f6c1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.711+0000 7f3f711d6700 1 --2- 172.21.15.151:0/4102634088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c100da0 0x7f3f6c101180 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.711+0000 7f3f711d6700 1 -- 172.21.15.151:0/4102634088 >> 172.21.15.151:0/4102634088 conn(0x7f3f6c0fc650 msgr2=0x7f3f6c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:44.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.711+0000 7f3f711d6700 1 -- 172.21.15.151:0/4102634088 shutdown_connections 2024-04-16T02:40:44.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.711+0000 7f3f711d6700 1 -- 172.21.15.151:0/4102634088 wait complete. 2024-04-16T02:40:44.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.712+0000 7f3f711d6700 1 Processor -- start 2024-04-16T02:40:44.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.712+0000 7f3f711d6700 1 -- start start 2024-04-16T02:40:44.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.712+0000 7f3f711d6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f6c100da0 0x7f3f6c19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:44.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.712+0000 7f3f711d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c101750 0x7f3f6c19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:44.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.712+0000 7f3f711d6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c19d1f0 con 0x7f3f6c100da0 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.712+0000 7f3f711d6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c1966a0 con 0x7f3f6c101750 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.712+0000 7f3f6a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c101750 0x7f3f6c19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.712+0000 7f3f6a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c101750 0x7f3f6c19cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:33570/0 (socket says 172.21.15.151:33570) 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f6a59c700 1 -- 172.21.15.151:0/3387626407 learned_addr learned my addr 172.21.15.151:0/3387626407 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f6a59c700 1 -- 172.21.15.151:0/3387626407 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f6c100da0 msgr2=0x7f3f6c19c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f6a59c700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f6c100da0 0x7f3f6c19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f6a59c700 1 -- 172.21.15.151:0/3387626407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c0095f0 con 0x7f3f6c101750 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f6a59c700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c101750 0x7f3f6c19cb60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000b480 tx=0x7f3f6000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f53fff700 1 -- 172.21.15.151:0/3387626407 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f600118c0 con 0x7f3f6c101750 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c196980 con 0x7f3f6c101750 2024-04-16T02:40:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f53fff700 1 -- 172.21.15.151:0/3387626407 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f60010380 con 0x7f3f6c101750 2024-04-16T02:40:44.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f53fff700 1 -- 172.21.15.151:0/3387626407 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f6000f4f0 con 0x7f3f6c101750 2024-04-16T02:40:44.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.713+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c196ed0 con 0x7f3f6c101750 2024-04-16T02:40:44.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.714+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c1090d0 con 0x7f3f6c101750 2024-04-16T02:40:44.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.715+0000 7f3f53fff700 1 -- 172.21.15.151:0/3387626407 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f3f60011a20 con 0x7f3f6c101750 2024-04-16T02:40:44.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.715+0000 7f3f53fff700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3f5406ee00 0x7f3f540712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:44.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.715+0000 7f3f6ad9d700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3f5406ee00 0x7f3f540712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:44.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.715+0000 7f3f53fff700 1 -- 172.21.15.151:0/3387626407 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3f6008eb70 con 0x7f3f6c101750 2024-04-16T02:40:44.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.716+0000 7f3f6ad9d700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3f5406ee00 0x7f3f540712c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00b5c0 tx=0x7f3f5c005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:44.722 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.720+0000 7f3f53fff700 1 -- 172.21.15.151:0/3387626407 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3f6005b3f0 con 0x7f3f6c101750 2024-04-16T02:40:44.898 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:44 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:44.898 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:44 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:44.898 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:44 smithi153 ceph-mon[87858]: Reconfiguring osd.5 (monmap changed)... 2024-04-16T02:40:44.898 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:44 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T02:40:44.898 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:44 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:44.898 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:44 smithi153 ceph-mon[87858]: Reconfiguring daemon osd.5 on smithi153 2024-04-16T02:40:44.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:44 smithi153 ceph-mon[87858]: pgmap v28: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:44.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.928+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f6c197150 con 0x7f3f5406ee00 2024-04-16T02:40:44.931 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:44 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:44.931 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:44 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:44.931 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:44 smithi151 ceph-mon[112233]: Reconfiguring osd.5 (monmap changed)... 2024-04-16T02:40:44.931 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:44 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T02:40:44.931 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:44 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:44.931 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:44 smithi151 ceph-mon[112233]: Reconfiguring daemon osd.5 on smithi153 2024-04-16T02:40:44.931 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:44 smithi151 ceph-mon[112233]: pgmap v28: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.940+0000 7f3f53fff700 1 -- 172.21.15.151:0/3387626407 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f3f6c197150 con 0x7f3f5406ee00 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (2m) 45s ago 13m 17.5M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (13m) 45s ago 13m 8174k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (12m) 30s ago 12m 8074k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (13m) 45s ago 13m 6665k - 18.2.1 5be31c24972a b207f70adbf2 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (12m) 30s ago 12m 6656k - 18.2.1 5be31c24972a 4336c153c36d 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (117s) 45s ago 13m 75.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (8m) 45s ago 8m 120M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:40:44.943 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (8m) 45s ago 8m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (7m) 30s ago 7m 58.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (8m) 30s ago 8m 67.5M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (4m) 45s ago 15m 473M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (3m) 30s ago 12m 448M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (60s) 45s ago 15m 29.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (34s) 30s ago 12m 30.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (3m) 45s ago 13m 15.6M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (2m) 30s ago 12m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (11m) 45s ago 11m 224M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (11m) 45s ago 11m 250M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (10m) 45s ago 10m 195M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:40:44.944 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (10m) 30s ago 10m 282M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:40:44.945 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (9m) 30s ago 9m 231M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:40:44.945 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (9m) 30s ago 9m 246M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:40:44.945 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (2m) 45s ago 13m 68.8M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3f5406ee00 msgr2=0x7f3f540712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3f5406ee00 0x7f3f540712c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00b5c0 tx=0x7f3f5c005960 comp rx=0 tx=0).stop 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c101750 msgr2=0x7f3f6c19cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c101750 0x7f3f6c19cb60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000b480 tx=0x7f3f6000b840 comp rx=0 tx=0).stop 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 shutdown_connections 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3f5406ee00 0x7f3f540712c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f6c100da0 0x7f3f6c19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 --2- 172.21.15.151:0/3387626407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c101750 0x7f3f6c19cb60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 >> 172.21.15.151:0/3387626407 conn(0x7f3f6c0fc650 msgr2=0x7f3f6c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 shutdown_connections 2024-04-16T02:40:44.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:44.942+0000 7f3f711d6700 1 -- 172.21.15.151:0/3387626407 wait complete. 2024-04-16T02:40:45.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.046+0000 7f40ae878700 1 -- 172.21.15.151:0/2520754888 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a80ffa40 msgr2=0x7f40a80ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.046+0000 7f40ae878700 1 --2- 172.21.15.151:0/2520754888 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a80ffa40 0x7f40a80ffec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f409c009910 tx=0x7f409c009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:45.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.046+0000 7f40ae878700 1 -- 172.21.15.151:0/2520754888 shutdown_connections 2024-04-16T02:40:45.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.046+0000 7f40ae878700 1 --2- 172.21.15.151:0/2520754888 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a80ffa40 0x7f40a80ffec0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.046+0000 7f40ae878700 1 --2- 172.21.15.151:0/2520754888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40a8105a40 0x7f40a8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.046+0000 7f40ae878700 1 -- 172.21.15.151:0/2520754888 >> 172.21.15.151:0/2520754888 conn(0x7f40a80fb570 msgr2=0x7f40a80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:45.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.046+0000 7f40ae878700 1 -- 172.21.15.151:0/2520754888 shutdown_connections 2024-04-16T02:40:45.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.046+0000 7f40ae878700 1 -- 172.21.15.151:0/2520754888 wait complete. 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.047+0000 7f40ae878700 1 Processor -- start 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.047+0000 7f40ae878700 1 -- start start 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.047+0000 7f40ae878700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40a80ffa40 0x7f40a819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.047+0000 7f40ae878700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a8105a40 0x7f40a819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.047+0000 7f40ae878700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a819d0c0 con 0x7f40a8105a40 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.047+0000 7f40ae878700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a8196570 con 0x7f40a80ffa40 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a8105a40 0x7f40a819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a8105a40 0x7f40a819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38830/0 (socket says 172.21.15.151:38830) 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a77fe700 1 -- 172.21.15.151:0/1630496684 learned_addr learned my addr 172.21.15.151:0/1630496684 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:45.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a7fff700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40a80ffa40 0x7f40a819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a77fe700 1 -- 172.21.15.151:0/1630496684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40a80ffa40 msgr2=0x7f40a819c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a77fe700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40a80ffa40 0x7f40a819c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a77fe700 1 -- 172.21.15.151:0/1630496684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c0095f0 con 0x7f40a8105a40 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a77fe700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a8105a40 0x7f40a819ca30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f409c006010 tx=0x7f409c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a57fa700 1 -- 172.21.15.151:0/1630496684 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f409c01d070 con 0x7f40a8105a40 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a81967f0 con 0x7f40a8105a40 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a57fa700 1 -- 172.21.15.151:0/1630496684 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f409c004fb0 con 0x7f40a8105a40 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40a57fa700 1 -- 172.21.15.151:0/1630496684 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f409c022ea0 con 0x7f40a8105a40 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.048+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a8196ce0 con 0x7f40a8105a40 2024-04-16T02:40:45.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.049+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a8108fe0 con 0x7f40a8105a40 2024-04-16T02:40:45.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.050+0000 7f40a57fa700 1 -- 172.21.15.151:0/1630496684 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f409c022470 con 0x7f40a8105a40 2024-04-16T02:40:45.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.051+0000 7f40a57fa700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f409006ee00 0x7f40900712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:45.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.051+0000 7f40a57fa700 1 -- 172.21.15.151:0/1630496684 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f409c090a60 con 0x7f40a8105a40 2024-04-16T02:40:45.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.051+0000 7f40a7fff700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f409006ee00 0x7f40900712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:45.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.051+0000 7f40a7fff700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f409006ee00 0x7f40900712c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4098005950 tx=0x7f4098005eb0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:45.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.059+0000 7f40a57fa700 1 -- 172.21.15.151:0/1630496684 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f409c05d300 con 0x7f40a8105a40 2024-04-16T02:40:45.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.347+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f40a8197a80 con 0x7f40a8105a40 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.348+0000 7f40a57fa700 1 -- 172.21.15.151:0/1630496684 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f409c0312c0 con 0x7f40a8105a40 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-16T02:40:45.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:40:45.352 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:40:45.352 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:40:45.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:40:45.352 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:40:45.352 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-04-16T02:40:45.352 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T02:40:45.352 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:40:45.352 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:40:45.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f409006ee00 msgr2=0x7f40900712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f409006ee00 0x7f40900712c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4098005950 tx=0x7f4098005eb0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a8105a40 msgr2=0x7f40a819ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a8105a40 0x7f40a819ca30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f409c006010 tx=0x7f409c00be20 comp rx=0 tx=0).stop 2024-04-16T02:40:45.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 shutdown_connections 2024-04-16T02:40:45.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f409006ee00 0x7f40900712c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40a80ffa40 0x7f40a819c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 --2- 172.21.15.151:0/1630496684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40a8105a40 0x7f40a819ca30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 >> 172.21.15.151:0/1630496684 conn(0x7f40a80fb570 msgr2=0x7f40a8068f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:45.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 shutdown_connections 2024-04-16T02:40:45.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.350+0000 7f40ae878700 1 -- 172.21.15.151:0/1630496684 wait complete. 2024-04-16T02:40:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.455+0000 7f37b05da700 1 -- 172.21.15.151:0/2534548415 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8100dc0 msgr2=0x7f37a81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.455+0000 7f37b05da700 1 --2- 172.21.15.151:0/2534548415 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8100dc0 0x7f37a81011a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f37a4009910 tx=0x7f37a4009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.455+0000 7f37b05da700 1 -- 172.21.15.151:0/2534548415 shutdown_connections 2024-04-16T02:40:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.455+0000 7f37b05da700 1 --2- 172.21.15.151:0/2534548415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37a8101770 0x7f37a81057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.455+0000 7f37b05da700 1 --2- 172.21.15.151:0/2534548415 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8100dc0 0x7f37a81011a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.455+0000 7f37b05da700 1 -- 172.21.15.151:0/2534548415 >> 172.21.15.151:0/2534548415 conn(0x7f37a80fc650 msgr2=0x7f37a80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.455+0000 7f37b05da700 1 -- 172.21.15.151:0/2534548415 shutdown_connections 2024-04-16T02:40:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.455+0000 7f37b05da700 1 -- 172.21.15.151:0/2534548415 wait complete. 2024-04-16T02:40:45.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37b05da700 1 Processor -- start 2024-04-16T02:40:45.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37b05da700 1 -- start start 2024-04-16T02:40:45.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37b05da700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37a8100dc0 0x7f37a819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:45.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37b05da700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8101770 0x7f37a819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:45.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37b05da700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a819d110 con 0x7f37a8101770 2024-04-16T02:40:45.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37b05da700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a81965c0 con 0x7f37a8100dc0 2024-04-16T02:40:45.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37adb75700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8101770 0x7f37a819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37adb75700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8101770 0x7f37a819ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38852/0 (socket says 172.21.15.151:38852) 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37adb75700 1 -- 172.21.15.151:0/3008422635 learned_addr learned my addr 172.21.15.151:0/3008422635 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37ae376700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37a8100dc0 0x7f37a819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37adb75700 1 -- 172.21.15.151:0/3008422635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37a8100dc0 msgr2=0x7f37a819c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37adb75700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37a8100dc0 0x7f37a819c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37adb75700 1 -- 172.21.15.151:0/3008422635 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a40095f0 con 0x7f37a8101770 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.456+0000 7f37adb75700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8101770 0x7f37a819ca80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f379800d8d0 tx=0x7f379800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.457+0000 7f379f7fe700 1 -- 172.21.15.151:0/3008422635 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37980041f0 con 0x7f37a8101770 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.457+0000 7f379f7fe700 1 -- 172.21.15.151:0/3008422635 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3798007620 con 0x7f37a8101770 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.457+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a81968a0 con 0x7f37a8101770 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.457+0000 7f379f7fe700 1 -- 172.21.15.151:0/3008422635 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37980056b0 con 0x7f37a8101770 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.457+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a8196df0 con 0x7f37a8101770 2024-04-16T02:40:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.458+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a8109160 con 0x7f37a8101770 2024-04-16T02:40:45.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.458+0000 7f379f7fe700 1 -- 172.21.15.151:0/3008422635 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f3798004750 con 0x7f37a8101770 2024-04-16T02:40:45.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.459+0000 7f379f7fe700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f379406ee00 0x7f37940712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:45.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.459+0000 7f37ae376700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f379406ee00 0x7f37940712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:45.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.459+0000 7f379f7fe700 1 -- 172.21.15.151:0/3008422635 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f379808ee10 con 0x7f37a8101770 2024-04-16T02:40:45.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.459+0000 7f37ae376700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f379406ee00 0x7f37940712c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f37a4006010 tx=0x7f37a401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:45.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.463+0000 7f379f7fe700 1 -- 172.21.15.151:0/3008422635 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f379805b600 con 0x7f37a8101770 2024-04-16T02:40:45.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.706+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f37a804ea90 con 0x7f37a8101770 2024-04-16T02:40:45.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.707+0000 7f379f7fe700 1 -- 172.21.15.151:0/3008422635 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1919 (secure 0 0 0) 0x7f3798016790 con 0x7f37a8101770 2024-04-16T02:40:45.709 INFO:teuthology.orchestra.run.smithi151.stdout:e22 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 22 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:40:05.114230+0000 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:40:45.710 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:40:45.711 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 14478 members: 14490,14478 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:40:45.712 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:40:45.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f379406ee00 msgr2=0x7f37940712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f379406ee00 0x7f37940712c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f37a4006010 tx=0x7f37a401a040 comp rx=0 tx=0).stop 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8101770 msgr2=0x7f37a819ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8101770 0x7f37a819ca80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f379800d8d0 tx=0x7f379800dbe0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 shutdown_connections 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f379406ee00 0x7f37940712c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37a8100dc0 0x7f37a819c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 --2- 172.21.15.151:0/3008422635 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37a8101770 0x7f37a819ca80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 >> 172.21.15.151:0/3008422635 conn(0x7f37a80fc650 msgr2=0x7f37a80fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 shutdown_connections 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.709+0000 7f37b05da700 1 -- 172.21.15.151:0/3008422635 wait complete. 2024-04-16T02:40:45.714 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 22 2024-04-16T02:40:45.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.816+0000 7f4b45d92700 1 -- 172.21.15.151:0/4260643436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 msgr2=0x7f4b401057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.816+0000 7f4b45d92700 1 --2- 172.21.15.151:0/4260643436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 0x7f4b401057c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009910 tx=0x7f4b28009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:45.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.816+0000 7f4b45d92700 1 -- 172.21.15.151:0/4260643436 shutdown_connections 2024-04-16T02:40:45.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.816+0000 7f4b45d92700 1 --2- 172.21.15.151:0/4260643436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 0x7f4b401057c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.816+0000 7f4b45d92700 1 --2- 172.21.15.151:0/4260643436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b40100dc0 0x7f4b401011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.816+0000 7f4b45d92700 1 -- 172.21.15.151:0/4260643436 >> 172.21.15.151:0/4260643436 conn(0x7f4b400fc650 msgr2=0x7f4b400fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:45.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.816+0000 7f4b45d92700 1 -- 172.21.15.151:0/4260643436 shutdown_connections 2024-04-16T02:40:45.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.816+0000 7f4b45d92700 1 -- 172.21.15.151:0/4260643436 wait complete. 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b45d92700 1 Processor -- start 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b45d92700 1 -- start start 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b45d92700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b40100dc0 0x7f4b4019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b45d92700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 0x7f4b4019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b45d92700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4019d130 con 0x7f4b40101770 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b45d92700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b401965e0 con 0x7f4b40100dc0 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b37fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 0x7f4b4019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b37fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 0x7f4b4019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38878/0 (socket says 172.21.15.151:38878) 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b37fff700 1 -- 172.21.15.151:0/4091546666 learned_addr learned my addr 172.21.15.151:0/4091546666 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b3f7fe700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b40100dc0 0x7f4b4019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b37fff700 1 -- 172.21.15.151:0/4091546666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b40100dc0 msgr2=0x7f4b4019c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b37fff700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b40100dc0 0x7f4b4019c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:45.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b37fff700 1 -- 172.21.15.151:0/4091546666 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280095f0 con 0x7f4b40101770 2024-04-16T02:40:45.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b37fff700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 0x7f4b4019caa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b280092e0 tx=0x7f4b2800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:45.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.817+0000 7f4b3d7fa700 1 -- 172.21.15.151:0/4091546666 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b2801d070 con 0x7f4b40101770 2024-04-16T02:40:45.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.818+0000 7f4b3d7fa700 1 -- 172.21.15.151:0/4091546666 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b28005dc0 con 0x7f4b40101770 2024-04-16T02:40:45.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.818+0000 7f4b3d7fa700 1 -- 172.21.15.151:0/4091546666 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b28017400 con 0x7f4b40101770 2024-04-16T02:40:45.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.818+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b40196860 con 0x7f4b40101770 2024-04-16T02:40:45.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.818+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b40196d00 con 0x7f4b40101770 2024-04-16T02:40:45.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.819+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b401091c0 con 0x7f4b40101770 2024-04-16T02:40:45.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.819+0000 7f4b3d7fa700 1 -- 172.21.15.151:0/4091546666 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f4b2800f6f0 con 0x7f4b40101770 2024-04-16T02:40:45.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.820+0000 7f4b3d7fa700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4b2c06ee00 0x7f4b2c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:45.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.820+0000 7f4b3d7fa700 1 -- 172.21.15.151:0/4091546666 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4b28090430 con 0x7f4b40101770 2024-04-16T02:40:45.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.820+0000 7f4b3f7fe700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4b2c06ee00 0x7f4b2c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:45.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.820+0000 7f4b3f7fe700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4b2c06ee00 0x7f4b2c0712c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4b30005950 tx=0x7f4b3000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:45.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:45.824+0000 7f4b3d7fa700 1 -- 172.21.15.151:0/4091546666 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4b2805cba0 con 0x7f4b40101770 2024-04-16T02:40:45.980 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:45 smithi153 ceph-mon[87858]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:45.980 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:45 smithi153 ceph-mon[87858]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:45.980 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:45 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:45.980 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:45 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:45.980 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:45 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ogdcjk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:40:45.981 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:45 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:45.981 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:45 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1630496684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:45 smithi151 ceph-mon[112233]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:45 smithi151 ceph-mon[112233]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:45 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:45 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:45 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ogdcjk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:40:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:45 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:45 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1630496684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:46.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.041+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b4002ccf0 con 0x7f4b2c06ee00 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.042+0000 7f4b3d7fa700 1 -- 172.21.15.151:0/4091546666 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f4b4002ccf0 con 0x7f4b2c06ee00 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "4/23 daemons upgraded", 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mon daemons", 2024-04-16T02:40:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:40:46.046 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:40:46.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.044+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4b2c06ee00 msgr2=0x7f4b2c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:46.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.044+0000 7f4b45d92700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4b2c06ee00 0x7f4b2c0712c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4b30005950 tx=0x7f4b3000b410 comp rx=0 tx=0).stop 2024-04-16T02:40:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.044+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 msgr2=0x7f4b4019caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.044+0000 7f4b45d92700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 0x7f4b4019caa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b280092e0 tx=0x7f4b2800bd70 comp rx=0 tx=0).stop 2024-04-16T02:40:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.045+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 shutdown_connections 2024-04-16T02:40:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.045+0000 7f4b45d92700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4b2c06ee00 0x7f4b2c0712c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.045+0000 7f4b45d92700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b40100dc0 0x7f4b4019c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.045+0000 7f4b45d92700 1 --2- 172.21.15.151:0/4091546666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b40101770 0x7f4b4019caa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.045+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 >> 172.21.15.151:0/4091546666 conn(0x7f4b400fc650 msgr2=0x7f4b400fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.045+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 shutdown_connections 2024-04-16T02:40:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.045+0000 7f4b45d92700 1 -- 172.21.15.151:0/4091546666 wait complete. 2024-04-16T02:40:46.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.151+0000 7f896ce29700 1 -- 172.21.15.151:0/678391057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ff530 msgr2=0x7f89680ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:46.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.151+0000 7f896ce29700 1 --2- 172.21.15.151:0/678391057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ff530 0x7f89680ff910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8958009910 tx=0x7f8958009c20 comp rx=0 tx=0).stop 2024-04-16T02:40:46.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.151+0000 7f896ce29700 1 -- 172.21.15.151:0/678391057 shutdown_connections 2024-04-16T02:40:46.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.151+0000 7f896ce29700 1 --2- 172.21.15.151:0/678391057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89680ffee0 0x7f8968105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:46.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.151+0000 7f896ce29700 1 --2- 172.21.15.151:0/678391057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ff530 0x7f89680ff910 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:46.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.151+0000 7f896ce29700 1 -- 172.21.15.151:0/678391057 >> 172.21.15.151:0/678391057 conn(0x7f8968074df0 msgr2=0x7f8968075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:46.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.151+0000 7f896ce29700 1 -- 172.21.15.151:0/678391057 shutdown_connections 2024-04-16T02:40:46.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.151+0000 7f896ce29700 1 -- 172.21.15.151:0/678391057 wait complete. 2024-04-16T02:40:46.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.151+0000 7f896ce29700 1 Processor -- start 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f896ce29700 1 -- start start 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f896ce29700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89680ff530 0x7f896819c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f896ce29700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ffee0 0x7f896819cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f896ce29700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896819d0d0 con 0x7f89680ffee0 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f896ce29700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8968196610 con 0x7f89680ff530 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f8965d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ffee0 0x7f896819cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f8965d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ffee0 0x7f896819cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38892/0 (socket says 172.21.15.151:38892) 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f8965d9b700 1 -- 172.21.15.151:0/1369314797 learned_addr learned my addr 172.21.15.151:0/1369314797 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f896659c700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89680ff530 0x7f896819c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f8965d9b700 1 -- 172.21.15.151:0/1369314797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89680ff530 msgr2=0x7f896819c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f8965d9b700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89680ff530 0x7f896819c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f8965d9b700 1 -- 172.21.15.151:0/1369314797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89580095f0 con 0x7f89680ffee0 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f8965d9b700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ffee0 0x7f896819cad0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f895c00c960 tx=0x7f895c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.152+0000 7f89577fe700 1 -- 172.21.15.151:0/1369314797 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f895c007a70 con 0x7f89680ffee0 2024-04-16T02:40:46.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.153+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89681968f0 con 0x7f89680ffee0 2024-04-16T02:40:46.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.153+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8968196e40 con 0x7f89680ffee0 2024-04-16T02:40:46.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.153+0000 7f89577fe700 1 -- 172.21.15.151:0/1369314797 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f895c007bd0 con 0x7f89680ffee0 2024-04-16T02:40:46.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.153+0000 7f89577fe700 1 -- 172.21.15.151:0/1369314797 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f895c0186c0 con 0x7f89680ffee0 2024-04-16T02:40:46.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.153+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8948005320 con 0x7f89680ffee0 2024-04-16T02:40:46.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.154+0000 7f89577fe700 1 -- 172.21.15.151:0/1369314797 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f895c00f450 con 0x7f89680ffee0 2024-04-16T02:40:46.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.154+0000 7f89577fe700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f895006ee00 0x7f89500712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:40:46.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.155+0000 7f89577fe700 1 -- 172.21.15.151:0/1369314797 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f895c08ee70 con 0x7f89680ffee0 2024-04-16T02:40:46.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.155+0000 7f896659c700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f895006ee00 0x7f89500712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:40:46.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.155+0000 7f896659c700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f895006ee00 0x7f89500712c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f895800b5c0 tx=0x7f8958005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:40:46.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.159+0000 7f89577fe700 1 -- 172.21.15.151:0/1369314797 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f895c05b3d0 con 0x7f89680ffee0 2024-04-16T02:40:46.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.441+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8948005190 con 0x7f89680ffee0 2024-04-16T02:40:46.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.442+0000 7f89577fe700 1 -- 172.21.15.151:0/1369314797 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f895c05b1f0 con 0x7f89680ffee0 2024-04-16T02:40:46.445 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_OK 2024-04-16T02:40:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f895006ee00 msgr2=0x7f89500712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f895006ee00 0x7f89500712c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f895800b5c0 tx=0x7f8958005960 comp rx=0 tx=0).stop 2024-04-16T02:40:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ffee0 msgr2=0x7f896819cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:40:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ffee0 0x7f896819cad0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f895c00c960 tx=0x7f895c00cd20 comp rx=0 tx=0).stop 2024-04-16T02:40:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 shutdown_connections 2024-04-16T02:40:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f895006ee00 0x7f89500712c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89680ff530 0x7f896819c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 --2- 172.21.15.151:0/1369314797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89680ffee0 0x7f896819cad0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:40:46.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 >> 172.21.15.151:0/1369314797 conn(0x7f8968074df0 msgr2=0x7f89680feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:40:46.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 shutdown_connections 2024-04-16T02:40:46.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:40:46.444+0000 7f896ce29700 1 -- 172.21.15.151:0/1369314797 wait complete. 2024-04-16T02:40:46.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:46 smithi151 ceph-mon[112233]: from='client.44107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:46.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:46 smithi151 ceph-mon[112233]: Reconfiguring mds.cephfs.smithi153.ogdcjk (monmap changed)... 2024-04-16T02:40:46.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:46 smithi151 ceph-mon[112233]: Reconfiguring daemon mds.cephfs.smithi153.ogdcjk on smithi153 2024-04-16T02:40:46.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:46 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3008422635' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:40:46.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:46 smithi151 ceph-mon[112233]: pgmap v29: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:46.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:46 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1369314797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:40:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:46 smithi153 ceph-mon[87858]: from='client.44107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:46 smithi153 ceph-mon[87858]: Reconfiguring mds.cephfs.smithi153.ogdcjk (monmap changed)... 2024-04-16T02:40:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:46 smithi153 ceph-mon[87858]: Reconfiguring daemon mds.cephfs.smithi153.ogdcjk on smithi153 2024-04-16T02:40:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:46 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3008422635' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:40:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:46 smithi153 ceph-mon[87858]: pgmap v29: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:46 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1369314797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:40:47.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:47 smithi153 ceph-mon[87858]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:47.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:47 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:47.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:47 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:47.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:47 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.mqbgav", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:40:47.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:47 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:47.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:47 smithi151 ceph-mon[112233]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:40:47.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:47 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:47.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:47 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:47.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:47 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.mqbgav", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:40:47.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:47 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:48.961 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: Reconfiguring mds.cephfs.smithi153.mqbgav (monmap changed)... 2024-04-16T02:40:48.961 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: Reconfiguring daemon mds.cephfs.smithi153.mqbgav on smithi153 2024-04-16T02:40:48.961 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: pgmap v30: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:48.961 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:48.961 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:48.961 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:40:48.962 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:48.962 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:48.962 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:48.962 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi151"}]: dispatch 2024-04-16T02:40:48.962 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi151"}]': finished 2024-04-16T02:40:48.962 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi153"}]: dispatch 2024-04-16T02:40:48.962 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi153"}]': finished 2024-04-16T02:40:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: Reconfiguring mds.cephfs.smithi153.mqbgav (monmap changed)... 2024-04-16T02:40:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: Reconfiguring daemon mds.cephfs.smithi153.mqbgav on smithi153 2024-04-16T02:40:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: pgmap v30: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:40:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:49.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:40:49.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:49.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi151"}]: dispatch 2024-04-16T02:40:49.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi151"}]': finished 2024-04-16T02:40:49.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi153"}]: dispatch 2024-04-16T02:40:49.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi153"}]': finished 2024-04-16T02:40:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:49 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all mon 2024-04-16T02:40:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:49 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all mon 2024-04-16T02:40:50.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:50 smithi151 ceph-mon[112233]: pgmap v31: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:50.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:50 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:50.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:50 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi151", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:40:50.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:50 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:51.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:50 smithi153 ceph-mon[87858]: pgmap v31: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:51.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:50 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:51.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:50 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi151", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:40:51.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:50 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:51.935 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:51 smithi151 ceph-mon[112233]: Upgrade: Updating crash.smithi151 (1/2) 2024-04-16T02:40:51.935 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:51 smithi151 ceph-mon[112233]: Deploying daemon crash.smithi151 on smithi151 2024-04-16T02:40:52.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:51 smithi153 ceph-mon[87858]: Upgrade: Updating crash.smithi151 (1/2) 2024-04-16T02:40:52.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:51 smithi153 ceph-mon[87858]: Deploying daemon crash.smithi151 on smithi151 2024-04-16T02:40:52.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:52 smithi151 ceph-mon[112233]: pgmap v32: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:53.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:52 smithi153 ceph-mon[87858]: pgmap v32: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:54.115 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:53 smithi151 ceph-mon[112233]: pgmap v33: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:54.115 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:40:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:53 smithi153 ceph-mon[87858]: pgmap v33: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:40:55.486 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:55 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:55.486 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:55 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:55.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:55 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:55.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:55 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:56.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:56 smithi153 ceph-mon[87858]: pgmap v34: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:56.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:56 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:56.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:56 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:40:56.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:56 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:56.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:56 smithi151 ceph-mon[112233]: pgmap v34: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:40:56.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:56 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:40:56.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:56 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T02:40:56.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:56 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:40:57.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:57 smithi153 ceph-mon[87858]: Upgrade: Updating crash.smithi153 (2/2) 2024-04-16T02:40:57.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:57 smithi153 ceph-mon[87858]: Deploying daemon crash.smithi153 on smithi153 2024-04-16T02:40:57.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:57 smithi151 ceph-mon[112233]: Upgrade: Updating crash.smithi153 (2/2) 2024-04-16T02:40:57.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:57 smithi151 ceph-mon[112233]: Deploying daemon crash.smithi153 on smithi153 2024-04-16T02:40:58.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:40:58 smithi153 ceph-mon[87858]: pgmap v35: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:40:58.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:40:58 smithi151 ceph-mon[112233]: pgmap v35: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:00.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:00.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:00.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:00.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:00 smithi153 ceph-mon[87858]: pgmap v36: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:00.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:00.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:00.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:00.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:00 smithi151 ceph-mon[112233]: pgmap v36: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:01.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:01 smithi153 ceph-mon[87858]: pgmap v37: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:02.156 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:01 smithi151 ceph-mon[112233]: pgmap v37: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:03.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:03.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:04.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:04.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:04.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:04 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:04.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:04 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:04.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:04 smithi151 ceph-mon[112233]: pgmap v38: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:05.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:04 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:05.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:04 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:05.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:04 smithi153 ceph-mon[87858]: pgmap v38: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:05 smithi151 ceph-mon[112233]: pgmap v39: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:05 smithi153 ceph-mon[87858]: pgmap v39: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:07.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:07.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:07.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:07.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:08.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:08.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:08.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:08 smithi151 ceph-mon[112233]: pgmap v40: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:08.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:08.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:08.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:08 smithi153 ceph-mon[87858]: pgmap v40: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:09.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:41:09.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:41:10.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:10.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:10.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:41:10.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:41:10.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: pgmap v41: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi151"}]: dispatch 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi151"}]': finished 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi153"}]: dispatch 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi153"}]': finished 2024-04-16T02:41:10.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T02:41:11.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:11.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:11.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:41:11.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: pgmap v41: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi151"}]: dispatch 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi151"}]': finished 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi153"}]: dispatch 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi153"}]': finished 2024-04-16T02:41:11.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T02:41:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:11 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all crash 2024-04-16T02:41:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:11 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T02:41:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:11 smithi151 ceph-mon[112233]: Upgrade: osd.0 is safe to restart 2024-04-16T02:41:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:11 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:11 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T02:41:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:11 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:41:12.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:11 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all crash 2024-04-16T02:41:12.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:11 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T02:41:12.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:11 smithi153 ceph-mon[87858]: Upgrade: osd.0 is safe to restart 2024-04-16T02:41:12.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:11 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:12.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:11 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T02:41:12.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:11 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:41:12.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:12 smithi151 ceph-mon[112233]: Upgrade: Updating osd.0 2024-04-16T02:41:12.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:12 smithi151 ceph-mon[112233]: Deploying daemon osd.0 on smithi151 2024-04-16T02:41:12.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:12 smithi151 ceph-mon[112233]: pgmap v42: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:13.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:12 smithi153 ceph-mon[87858]: Upgrade: Updating osd.0 2024-04-16T02:41:13.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:12 smithi153 ceph-mon[87858]: Deploying daemon osd.0 on smithi151 2024-04-16T02:41:13.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:12 smithi153 ceph-mon[87858]: pgmap v42: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:13.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:13 smithi151 ceph-mon[112233]: pgmap v43: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:14.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:13 smithi153 ceph-mon[87858]: pgmap v43: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:15.277 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:14 smithi151 systemd[1]: Stopping Ceph osd.0 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:41:15.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:15 smithi151 ceph-mon[112233]: osd.0 marked itself down and dead 2024-04-16T02:41:15.746 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:15 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[47348]: 2024-04-16T02:41:15.273+0000 7f26c3b33700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T02:41:15.746 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:15 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[47348]: 2024-04-16T02:41:15.273+0000 7f26c3b33700 -1 osd.0 50 *** Got signal Terminated *** 2024-04-16T02:41:15.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:15 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[47348]: 2024-04-16T02:41:15.273+0000 7f26c3b33700 -1 osd.0 50 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T02:41:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:15 smithi153 ceph-mon[87858]: osd.0 marked itself down and dead 2024-04-16T02:41:16.100 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:15 smithi151 podman[123341]: 2024-04-16 02:41:15.746199568 +0000 UTC m=+0.600127531 container died 1fbb4ea1979ba4a8f2a0be8acfc0ae5aa48ef0ac68e141e323bea068878b2679 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) 2024-04-16T02:41:16.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:16 smithi151 ceph-mon[112233]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:41:16.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:16 smithi151 ceph-mon[112233]: osdmap e51: 6 total, 5 up, 6 in 2024-04-16T02:41:16.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:16 smithi151 ceph-mon[112233]: pgmap v45: 65 pgs: 9 stale+active+clean, 56 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:41:16.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:16 smithi151 ceph-mon[112233]: osdmap e52: 6 total, 5 up, 6 in 2024-04-16T02:41:16.497 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:16 smithi151 podman[123341]: 2024-04-16 02:41:16.291801762 +0000 UTC m=+1.145729712 container cleanup 1fbb4ea1979ba4a8f2a0be8acfc0ae5aa48ef0ac68e141e323bea068878b2679 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-16T02:41:16.497 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:16 smithi151 bash[123341]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0 2024-04-16T02:41:16.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.556+0000 7f9f466e5700 1 -- 172.21.15.151:0/175388314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 msgr2=0x7f9f40105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:16.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.556+0000 7f9f466e5700 1 --2- 172.21.15.151:0/175388314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 0x7f9f40105e10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9f30009910 tx=0x7f9f30009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:16.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.556+0000 7f9f466e5700 1 -- 172.21.15.151:0/175388314 shutdown_connections 2024-04-16T02:41:16.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.556+0000 7f9f466e5700 1 --2- 172.21.15.151:0/175388314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f400ffa30 0x7f9f400ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.556+0000 7f9f466e5700 1 --2- 172.21.15.151:0/175388314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 0x7f9f40105e10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.556+0000 7f9f466e5700 1 -- 172.21.15.151:0/175388314 >> 172.21.15.151:0/175388314 conn(0x7f9f400fb540 msgr2=0x7f9f400fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:16.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.556+0000 7f9f466e5700 1 -- 172.21.15.151:0/175388314 shutdown_connections 2024-04-16T02:41:16.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.556+0000 7f9f466e5700 1 -- 172.21.15.151:0/175388314 wait complete. 2024-04-16T02:41:16.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f466e5700 1 Processor -- start 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f466e5700 1 -- start start 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f466e5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f400ffa30 0x7f9f401976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f466e5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 0x7f9f40197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f466e5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f40198200 con 0x7f9f40105a30 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f466e5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f40198340 con 0x7f9f400ffa30 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f37fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 0x7f9f40197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f37fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 0x7f9f40197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45642/0 (socket says 172.21.15.151:45642) 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f37fff700 1 -- 172.21.15.151:0/1570212234 learned_addr learned my addr 172.21.15.151:0/1570212234 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f3ffff700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f400ffa30 0x7f9f401976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f37fff700 1 -- 172.21.15.151:0/1570212234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f400ffa30 msgr2=0x7f9f401976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:16.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f37fff700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f400ffa30 0x7f9f401976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f37fff700 1 -- 172.21.15.151:0/1570212234 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300095f0 con 0x7f9f40105a30 2024-04-16T02:41:16.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.557+0000 7f9f37fff700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 0x7f9f40197be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9f2800b4b0 tx=0x7f9f2800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:16.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.558+0000 7f9f3dffb700 1 -- 172.21.15.151:0/1570212234 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f28011800 con 0x7f9f40105a30 2024-04-16T02:41:16.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.558+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f401a0710 con 0x7f9f40105a30 2024-04-16T02:41:16.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.558+0000 7f9f3dffb700 1 -- 172.21.15.151:0/1570212234 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f28011e20 con 0x7f9f40105a30 2024-04-16T02:41:16.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.558+0000 7f9f3dffb700 1 -- 172.21.15.151:0/1570212234 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f2800f2e0 con 0x7f9f40105a30 2024-04-16T02:41:16.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.558+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f401a0bd0 con 0x7f9f40105a30 2024-04-16T02:41:16.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.559+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f40108030 con 0x7f9f40105a30 2024-04-16T02:41:16.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.559+0000 7f9f3dffb700 1 -- 172.21.15.151:0/1570212234 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f9f28010380 con 0x7f9f40105a30 2024-04-16T02:41:16.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.560+0000 7f9f3dffb700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ee00 0x7f9f2c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:16.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.560+0000 7f9f3dffb700 1 -- 172.21.15.151:0/1570212234 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9f2808ea60 con 0x7f9f40105a30 2024-04-16T02:41:16.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.560+0000 7f9f3ffff700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ee00 0x7f9f2c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:16.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.560+0000 7f9f3ffff700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ee00 0x7f9f2c0712c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9f30006010 tx=0x7f9f3001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:16.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.565+0000 7f9f3dffb700 1 -- 172.21.15.151:0/1570212234 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9f280577b0 con 0x7f9f40105a30 2024-04-16T02:41:16.673 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:16 smithi153 ceph-mon[87858]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:41:16.674 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:16 smithi153 ceph-mon[87858]: osdmap e51: 6 total, 5 up, 6 in 2024-04-16T02:41:16.674 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:16 smithi153 ceph-mon[87858]: pgmap v45: 65 pgs: 9 stale+active+clean, 56 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:41:16.674 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:16 smithi153 ceph-mon[87858]: osdmap e52: 6 total, 5 up, 6 in 2024-04-16T02:41:16.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.805+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f4002cce0 con 0x7f9f2c06ee00 2024-04-16T02:41:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.807+0000 7f9f3dffb700 1 -- 172.21.15.151:0/1570212234 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9f4002cce0 con 0x7f9f2c06ee00 2024-04-16T02:41:16.810 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:16 smithi151 podman[123356]: 2024-04-16 02:41:16.602038239 +0000 UTC m=+0.855001510 container remove 1fbb4ea1979ba4a8f2a0be8acfc0ae5aa48ef0ac68e141e323bea068878b2679 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-16T02:41:16.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.810+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ee00 msgr2=0x7f9f2c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:16.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.810+0000 7f9f466e5700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ee00 0x7f9f2c0712c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9f30006010 tx=0x7f9f3001a040 comp rx=0 tx=0).stop 2024-04-16T02:41:16.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.810+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 msgr2=0x7f9f40197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:16.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.810+0000 7f9f466e5700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 0x7f9f40197be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9f2800b4b0 tx=0x7f9f2800b7c0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.810+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 shutdown_connections 2024-04-16T02:41:16.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.810+0000 7f9f466e5700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ee00 0x7f9f2c0712c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.810+0000 7f9f466e5700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f400ffa30 0x7f9f401976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.810+0000 7f9f466e5700 1 --2- 172.21.15.151:0/1570212234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40105a30 0x7f9f40197be0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.810+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 >> 172.21.15.151:0/1570212234 conn(0x7f9f400fb540 msgr2=0x7f9f400fc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:16.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.811+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 shutdown_connections 2024-04-16T02:41:16.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.811+0000 7f9f466e5700 1 -- 172.21.15.151:0/1570212234 wait complete. 2024-04-16T02:41:16.826 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:41:16.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.928+0000 7f919232b700 1 -- 172.21.15.151:0/3917870573 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 msgr2=0x7f918c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.928+0000 7f919232b700 1 --2- 172.21.15.151:0/3917870573 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 0x7f918c1057c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9174009910 tx=0x7f9174009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.928+0000 7f919232b700 1 -- 172.21.15.151:0/3917870573 shutdown_connections 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.928+0000 7f919232b700 1 --2- 172.21.15.151:0/3917870573 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 0x7f918c1057c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.928+0000 7f919232b700 1 --2- 172.21.15.151:0/3917870573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f918c100dc0 0x7f918c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.928+0000 7f919232b700 1 -- 172.21.15.151:0/3917870573 >> 172.21.15.151:0/3917870573 conn(0x7f918c0fc650 msgr2=0x7f918c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.928+0000 7f919232b700 1 -- 172.21.15.151:0/3917870573 shutdown_connections 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.928+0000 7f919232b700 1 -- 172.21.15.151:0/3917870573 wait complete. 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f919232b700 1 Processor -- start 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f919232b700 1 -- start start 2024-04-16T02:41:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f919232b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f918c100dc0 0x7f918c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f919232b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 0x7f918c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f919232b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c073b10 con 0x7f918c101770 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f919232b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c073c50 con 0x7f918c100dc0 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f9183fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 0x7f918c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f9183fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 0x7f918c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45658/0 (socket says 172.21.15.151:45658) 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f9183fff700 1 -- 172.21.15.151:0/2354656810 learned_addr learned my addr 172.21.15.151:0/2354656810 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f918bfff700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f918c100dc0 0x7f918c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f9183fff700 1 -- 172.21.15.151:0/2354656810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f918c100dc0 msgr2=0x7f918c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f9183fff700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f918c100dc0 0x7f918c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f9183fff700 1 -- 172.21.15.151:0/2354656810 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91740095f0 con 0x7f918c101770 2024-04-16T02:41:16.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.929+0000 7f9183fff700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 0x7f918c0735d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91740092e0 tx=0x7f917400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:16.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.930+0000 7f9189ffb700 1 -- 172.21.15.151:0/2354656810 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f917401d070 con 0x7f918c101770 2024-04-16T02:41:16.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.930+0000 7f9189ffb700 1 -- 172.21.15.151:0/2354656810 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9174005dc0 con 0x7f918c101770 2024-04-16T02:41:16.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.930+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c073ed0 con 0x7f918c101770 2024-04-16T02:41:16.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.930+0000 7f9189ffb700 1 -- 172.21.15.151:0/2354656810 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9174017400 con 0x7f918c101770 2024-04-16T02:41:16.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.930+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c1a2c90 con 0x7f918c101770 2024-04-16T02:41:16.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.931+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c1091c0 con 0x7f918c101770 2024-04-16T02:41:16.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.931+0000 7f9189ffb700 1 -- 172.21.15.151:0/2354656810 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f917400f6f0 con 0x7f918c101770 2024-04-16T02:41:16.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.932+0000 7f9189ffb700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917806edf0 0x7f91780712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:16.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.932+0000 7f918bfff700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917806edf0 0x7f91780712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:16.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.932+0000 7f9189ffb700 1 -- 172.21.15.151:0/2354656810 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9174090410 con 0x7f918c101770 2024-04-16T02:41:16.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.932+0000 7f918bfff700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917806edf0 0x7f91780712b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f917c005950 tx=0x7f917c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:16.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:16.936+0000 7f9189ffb700 1 -- 172.21.15.151:0/2354656810 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9174059050 con 0x7f918c101770 2024-04-16T02:41:17.154 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:16 smithi151 podman[123472]: 2024-04-16 02:41:16.884274431 +0000 UTC m=+0.018694565 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:41:17.154 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:17 smithi151 podman[123472]: 2024-04-16 02:41:17.003552887 +0000 UTC m=+0.137973030 container create 4e5929a70655bffc177c5da12ecf13a219121ce0de2079b9560eed53bab84332 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=main-87dbfb6) 2024-04-16T02:41:17.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.165+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f918c0611d0 con 0x7f917806edf0 2024-04-16T02:41:17.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.166+0000 7f9189ffb700 1 -- 172.21.15.151:0/2354656810 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f918c0611d0 con 0x7f917806edf0 2024-04-16T02:41:17.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.168+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917806edf0 msgr2=0x7f91780712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.168+0000 7f919232b700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917806edf0 0x7f91780712b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f917c005950 tx=0x7f917c00b410 comp rx=0 tx=0).stop 2024-04-16T02:41:17.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.168+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 msgr2=0x7f918c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.168+0000 7f919232b700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 0x7f918c0735d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91740092e0 tx=0x7f917400bd70 comp rx=0 tx=0).stop 2024-04-16T02:41:17.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.168+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 shutdown_connections 2024-04-16T02:41:17.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.168+0000 7f919232b700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917806edf0 0x7f91780712b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.168+0000 7f919232b700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f918c100dc0 0x7f918c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.168+0000 7f919232b700 1 --2- 172.21.15.151:0/2354656810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f918c101770 0x7f918c0735d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.168+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 >> 172.21.15.151:0/2354656810 conn(0x7f918c0fc650 msgr2=0x7f918c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:17.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.169+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 shutdown_connections 2024-04-16T02:41:17.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.169+0000 7f919232b700 1 -- 172.21.15.151:0/2354656810 wait complete. 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.275+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3193330235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 msgr2=0x7f58b8101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.275+0000 7f58bd5e7700 1 --2- 172.21.15.151:0/3193330235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 0x7f58b8101180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009910 tx=0x7f58ac009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.276+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3193330235 shutdown_connections 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.276+0000 7f58bd5e7700 1 --2- 172.21.15.151:0/3193330235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b8101750 0x7f58b81057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.276+0000 7f58bd5e7700 1 --2- 172.21.15.151:0/3193330235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 0x7f58b8101180 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.276+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3193330235 >> 172.21.15.151:0/3193330235 conn(0x7f58b80fc630 msgr2=0x7f58b80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.276+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3193330235 shutdown_connections 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.276+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3193330235 wait complete. 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.276+0000 7f58bd5e7700 1 Processor -- start 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.276+0000 7f58bd5e7700 1 -- start start 2024-04-16T02:41:17.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58bd5e7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 0x7f58b819c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58bd5e7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b8101750 0x7f58b819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58bd5e7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b819d170 con 0x7f58b8100da0 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58bd5e7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b8196620 con 0x7f58b8101750 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 0x7f58b819c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 0x7f58b819c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45674/0 (socket says 172.21.15.151:45674) 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58b6ffd700 1 -- 172.21.15.151:0/3775783786 learned_addr learned my addr 172.21.15.151:0/3775783786 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58b67fc700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b8101750 0x7f58b819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58b6ffd700 1 -- 172.21.15.151:0/3775783786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b8101750 msgr2=0x7f58b819cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58b6ffd700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b8101750 0x7f58b819cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58b6ffd700 1 -- 172.21.15.151:0/3775783786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac0095f0 con 0x7f58b8100da0 2024-04-16T02:41:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58b67fc700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b8101750 0x7f58b819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:41:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f58b6ffd700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 0x7f58b819c5a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f58ac005f50 tx=0x7f58ac00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f589ffff700 1 -- 172.21.15.151:0/3775783786 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58ac01d070 con 0x7f58b8100da0 2024-04-16T02:41:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.277+0000 7f589ffff700 1 -- 172.21.15.151:0/3775783786 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58ac00b610 con 0x7f58b8100da0 2024-04-16T02:41:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.278+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b81968a0 con 0x7f58b8100da0 2024-04-16T02:41:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.278+0000 7f589ffff700 1 -- 172.21.15.151:0/3775783786 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58ac022470 con 0x7f58b8100da0 2024-04-16T02:41:17.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.278+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b8196d90 con 0x7f58b8100da0 2024-04-16T02:41:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.279+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b8109140 con 0x7f58b8100da0 2024-04-16T02:41:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.279+0000 7f589ffff700 1 -- 172.21.15.151:0/3775783786 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f58ac003780 con 0x7f58b8100da0 2024-04-16T02:41:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.280+0000 7f589ffff700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58a006ee00 0x7f58a00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.280+0000 7f589ffff700 1 -- 172.21.15.151:0/3775783786 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f58ac0903d0 con 0x7f58b8100da0 2024-04-16T02:41:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.280+0000 7f58b67fc700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58a006ee00 0x7f58a00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:17.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.280+0000 7f58b67fc700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58a006ee00 0x7f58a00712c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f58b80fcdb0 tx=0x7f58a8009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:17.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.284+0000 7f589ffff700 1 -- 172.21.15.151:0/3775783786 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f58ac05cb40 con 0x7f58b8100da0 2024-04-16T02:41:17.496 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:17 smithi151 podman[123472]: 2024-04-16 02:41:17.290998708 +0000 UTC m=+0.425418852 container init 4e5929a70655bffc177c5da12ecf13a219121ce0de2079b9560eed53bab84332 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T02:41:17.496 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:17 smithi151 podman[123472]: 2024-04-16 02:41:17.294598553 +0000 UTC m=+0.429018691 container start 4e5929a70655bffc177c5da12ecf13a219121ce0de2079b9560eed53bab84332 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-04-16T02:41:17.496 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:17 smithi151 podman[123472]: 2024-04-16 02:41:17.390850018 +0000 UTC m=+0.525270177 container attach 4e5929a70655bffc177c5da12ecf13a219121ce0de2079b9560eed53bab84332 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T02:41:17.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.503+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f58b8197070 con 0x7f58a006ee00 2024-04-16T02:41:17.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.515+0000 7f589ffff700 1 -- 172.21.15.151:0/3775783786 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f58b8197070 con 0x7f58a006ee00 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (2m) 12s ago 14m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (14m) 12s ago 14m 8258k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (13m) 14s ago 13m 8132k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (23s) 12s ago 14m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (18s) 14s ago 13m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (2m) 12s ago 13m 76.2M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (8m) 12s ago 8m 120M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (8m) 12s ago 8m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:41:17.518 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (8m) 14s ago 8m 58.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (8m) 14s ago 8m 67.4M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (4m) 12s ago 15m 497M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (3m) 14s ago 13m 450M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (92s) 12s ago 15m 43.4M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (66s) 14s ago 13m 41.2M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (3m) 12s ago 14m 15.9M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (3m) 14s ago 13m 15.9M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (12m) 12s ago 12m 225M 1216M 18.2.1 5be31c24972a 1fbb4ea1979b 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (11m) 12s ago 11m 250M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (11m) 12s ago 11m 195M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (10m) 14s ago 10m 282M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:41:17.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (10m) 14s ago 10m 231M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:41:17.520 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (10m) 14s ago 10m 245M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:41:17.520 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (3m) 12s ago 13m 77.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:41:17.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58a006ee00 msgr2=0x7f58a00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58a006ee00 0x7f58a00712c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f58b80fcdb0 tx=0x7f58a8009260 comp rx=0 tx=0).stop 2024-04-16T02:41:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 msgr2=0x7f58b819c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 0x7f58b819c5a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f58ac005f50 tx=0x7f58ac00f690 comp rx=0 tx=0).stop 2024-04-16T02:41:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 shutdown_connections 2024-04-16T02:41:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58a006ee00 0x7f58a00712c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58b8100da0 0x7f58b819c5a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 --2- 172.21.15.151:0/3775783786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b8101750 0x7f58b819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 >> 172.21.15.151:0/3775783786 conn(0x7f58b80fc630 msgr2=0x7f58b80fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 shutdown_connections 2024-04-16T02:41:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.517+0000 7f58bd5e7700 1 -- 172.21.15.151:0/3775783786 wait complete. 2024-04-16T02:41:17.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.628+0000 7f9be03c4700 1 -- 172.21.15.151:0/1485519904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd80ffbd0 msgr2=0x7f9bd8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.628+0000 7f9be03c4700 1 --2- 172.21.15.151:0/1485519904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd80ffbd0 0x7f9bd8100050 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009910 tx=0x7f9bc8009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:17.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 -- 172.21.15.151:0/1485519904 shutdown_connections 2024-04-16T02:41:17.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 --2- 172.21.15.151:0/1485519904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd80ffbd0 0x7f9bd8100050 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 --2- 172.21.15.151:0/1485519904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd8105a40 0x7f9bd8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 -- 172.21.15.151:0/1485519904 >> 172.21.15.151:0/1485519904 conn(0x7f9bd80755a0 msgr2=0x7f9bd80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 -- 172.21.15.151:0/1485519904 shutdown_connections 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 -- 172.21.15.151:0/1485519904 wait complete. 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 Processor -- start 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 -- start start 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd80ffbd0 0x7f9bd8101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd8105a40 0x7f9bd8101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8102090 con 0x7f9bd8105a40 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.629+0000 7f9be03c4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd81021d0 con 0x7f9bd80ffbd0 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bdd95f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd8105a40 0x7f9bd8101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bdd95f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd8105a40 0x7f9bd8101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45698/0 (socket says 172.21.15.151:45698) 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bdd95f700 1 -- 172.21.15.151:0/333768140 learned_addr learned my addr 172.21.15.151:0/333768140 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bde160700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd80ffbd0 0x7f9bd8101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bdd95f700 1 -- 172.21.15.151:0/333768140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd80ffbd0 msgr2=0x7f9bd8101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bdd95f700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd80ffbd0 0x7f9bd8101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bdd95f700 1 -- 172.21.15.151:0/333768140 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc80095f0 con 0x7f9bd8105a40 2024-04-16T02:41:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bdd95f700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd8105a40 0x7f9bd8101b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8005f50 tx=0x7f9bc800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bcf7fe700 1 -- 172.21.15.151:0/333768140 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc801d070 con 0x7f9bd8105a40 2024-04-16T02:41:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd8102450 con 0x7f9bd8105a40 2024-04-16T02:41:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bcf7fe700 1 -- 172.21.15.151:0/333768140 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bc800b610 con 0x7f9bd8105a40 2024-04-16T02:41:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9bcf7fe700 1 -- 172.21.15.151:0/333768140 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc8022470 con 0x7f9bd8105a40 2024-04-16T02:41:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.630+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd81a2c30 con 0x7f9bd8105a40 2024-04-16T02:41:17.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.631+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd8108fe0 con 0x7f9bd8105a40 2024-04-16T02:41:17.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.632+0000 7f9bcf7fe700 1 -- 172.21.15.151:0/333768140 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7f9bc8003780 con 0x7f9bd8105a40 2024-04-16T02:41:17.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.632+0000 7f9bcf7fe700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bc406ee00 0x7f9bc40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:17.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.632+0000 7f9bcf7fe700 1 -- 172.21.15.151:0/333768140 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9bc8090400 con 0x7f9bd8105a40 2024-04-16T02:41:17.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.632+0000 7f9bde160700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bc406ee00 0x7f9bc40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:17.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.633+0000 7f9bde160700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bc406ee00 0x7f9bc40712c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009c60 tx=0x7f9bd4009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:17.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.637+0000 7f9bcf7fe700 1 -- 172.21.15.151:0/333768140 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9bc8059040 con 0x7f9bd8105a40 2024-04-16T02:41:17.820 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:17 smithi151 podman[123472]: 2024-04-16 02:41:17.600677543 +0000 UTC m=+0.735097685 container died 4e5929a70655bffc177c5da12ecf13a219121ce0de2079b9560eed53bab84332 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-16T02:41:17.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.927+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9bd804ea90 con 0x7f9bd8105a40 2024-04-16T02:41:17.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.927+0000 7f9bcf7fe700 1 -- 172.21.15.151:0/333768140 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+687 (secure 0 0 0) 0x7f9bc805c700 con 0x7f9bd8105a40 2024-04-16T02:41:17.930 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:41:17.930 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:41:17.930 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:41:17.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:41:17.930 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 9, 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:41:17.931 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bc406ee00 msgr2=0x7f9bc40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bc406ee00 0x7f9bc40712c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009c60 tx=0x7f9bd4009260 comp rx=0 tx=0).stop 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd8105a40 msgr2=0x7f9bd8101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd8105a40 0x7f9bd8101b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8005f50 tx=0x7f9bc800f690 comp rx=0 tx=0).stop 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 shutdown_connections 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bc406ee00 0x7f9bc40712c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd80ffbd0 0x7f9bd8101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 --2- 172.21.15.151:0/333768140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd8105a40 0x7f9bd8101b50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 >> 172.21.15.151:0/333768140 conn(0x7f9bd80755a0 msgr2=0x7f9bd8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 shutdown_connections 2024-04-16T02:41:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:17.930+0000 7f9be03c4700 1 -- 172.21.15.151:0/333768140 wait complete. 2024-04-16T02:41:18.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.037+0000 7fe3e518f700 1 -- 172.21.15.151:0/277813654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 msgr2=0x7fe3e0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.037+0000 7fe3e518f700 1 --2- 172.21.15.151:0/277813654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 0x7fe3e0105e20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009910 tx=0x7fe3d0009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:18.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.037+0000 7fe3e518f700 1 -- 172.21.15.151:0/277813654 shutdown_connections 2024-04-16T02:41:18.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.037+0000 7fe3e518f700 1 --2- 172.21.15.151:0/277813654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e00ffbd0 0x7fe3e0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.037+0000 7fe3e518f700 1 --2- 172.21.15.151:0/277813654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 0x7fe3e0105e20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.037+0000 7fe3e518f700 1 -- 172.21.15.151:0/277813654 >> 172.21.15.151:0/277813654 conn(0x7fe3e00755a0 msgr2=0x7fe3e00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:18.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.037+0000 7fe3e518f700 1 -- 172.21.15.151:0/277813654 shutdown_connections 2024-04-16T02:41:18.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.037+0000 7fe3e518f700 1 -- 172.21.15.151:0/277813654 wait complete. 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3e518f700 1 Processor -- start 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3e518f700 1 -- start start 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3e518f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e00ffbd0 0x7fe3e0197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3e518f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 0x7fe3e0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3e518f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e01980f0 con 0x7fe3e0105a40 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3e518f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e0198260 con 0x7fe3e00ffbd0 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3de59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 0x7fe3e0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3de59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 0x7fe3e0197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45714/0 (socket says 172.21.15.151:45714) 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3de59c700 1 -- 172.21.15.151:0/4262317514 learned_addr learned my addr 172.21.15.151:0/4262317514 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.038+0000 7fe3ded9d700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e00ffbd0 0x7fe3e0197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.039+0000 7fe3de59c700 1 -- 172.21.15.151:0/4262317514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e00ffbd0 msgr2=0x7fe3e0197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.039+0000 7fe3de59c700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e00ffbd0 0x7fe3e0197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.039+0000 7fe3de59c700 1 -- 172.21.15.151:0/4262317514 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d00095f0 con 0x7fe3e0105a40 2024-04-16T02:41:18.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.039+0000 7fe3de59c700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 0x7fe3e0197ad0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400d8d0 tx=0x7fe3d400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:18.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.039+0000 7fe3c7fff700 1 -- 172.21.15.151:0/4262317514 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3d40098a0 con 0x7fe3e0105a40 2024-04-16T02:41:18.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.039+0000 7fe3c7fff700 1 -- 172.21.15.151:0/4262317514 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3d4010460 con 0x7fe3e0105a40 2024-04-16T02:41:18.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.039+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e01a0600 con 0x7fe3e0105a40 2024-04-16T02:41:18.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.039+0000 7fe3c7fff700 1 -- 172.21.15.151:0/4262317514 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3d400f5f0 con 0x7fe3e0105a40 2024-04-16T02:41:18.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.039+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e01a0af0 con 0x7fe3e0105a40 2024-04-16T02:41:18.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.040+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e0108040 con 0x7fe3e0105a40 2024-04-16T02:41:18.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.041+0000 7fe3c7fff700 1 -- 172.21.15.151:0/4262317514 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7fe3d4009a00 con 0x7fe3e0105a40 2024-04-16T02:41:18.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.041+0000 7fe3c7fff700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3c806ee00 0x7fe3c80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:18.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.041+0000 7fe3ded9d700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3c806ee00 0x7fe3c80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:18.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.041+0000 7fe3c7fff700 1 -- 172.21.15.151:0/4262317514 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe3d408ef10 con 0x7fe3e0105a40 2024-04-16T02:41:18.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.041+0000 7fe3ded9d700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3c806ee00 0x7fe3c80712c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0006010 tx=0x7fe3d001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:18.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.046+0000 7fe3c7fff700 1 -- 172.21.15.151:0/4262317514 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe3d4057b50 con 0x7fe3e0105a40 2024-04-16T02:41:18.223 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:17 smithi151 ceph-mon[112233]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:18.223 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:17 smithi151 ceph-mon[112233]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:18.224 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:17 smithi151 ceph-mon[112233]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:18.224 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:17 smithi151 ceph-mon[112233]: pgmap v47: 65 pgs: 9 stale+active+clean, 56 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-16T02:41:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:17 smithi153 ceph-mon[87858]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:17 smithi153 ceph-mon[87858]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:17 smithi153 ceph-mon[87858]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:17 smithi153 ceph-mon[87858]: pgmap v47: 65 pgs: 9 stale+active+clean, 56 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-16T02:41:18.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.321+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe3e004ea90 con 0x7fe3e0105a40 2024-04-16T02:41:18.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.321+0000 7fe3c7fff700 1 -- 172.21.15.151:0/4262317514 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1919 (secure 0 0 0) 0x7fe3d405b210 con 0x7fe3e0105a40 2024-04-16T02:41:18.324 INFO:teuthology.orchestra.run.smithi151.stdout:e22 2024-04-16T02:41:18.324 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:41:18.324 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 22 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:40:05.114230+0000 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:41:18.325 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:41:18.326 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:41:18.327 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:41:18.327 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 14478 members: 14490,14478 2024-04-16T02:41:18.327 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:41:18.327 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:41:18.327 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:41:18.327 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:41:18.327 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:41:18.327 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:41:18.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.323+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3c806ee00 msgr2=0x7fe3c80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.323+0000 7fe3e518f700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3c806ee00 0x7fe3c80712c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0006010 tx=0x7fe3d001a040 comp rx=0 tx=0).stop 2024-04-16T02:41:18.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.323+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 msgr2=0x7fe3e0197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.323+0000 7fe3e518f700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 0x7fe3e0197ad0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400d8d0 tx=0x7fe3d400dbe0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.323+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 shutdown_connections 2024-04-16T02:41:18.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.323+0000 7fe3e518f700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3c806ee00 0x7fe3c80712c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.323+0000 7fe3e518f700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e00ffbd0 0x7fe3e0197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.323+0000 7fe3e518f700 1 --2- 172.21.15.151:0/4262317514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3e0105a40 0x7fe3e0197ad0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.323+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 >> 172.21.15.151:0/4262317514 conn(0x7fe3e00755a0 msgr2=0x7fe3e00fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:18.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.324+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 shutdown_connections 2024-04-16T02:41:18.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.324+0000 7fe3e518f700 1 -- 172.21.15.151:0/4262317514 wait complete. 2024-04-16T02:41:18.329 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 22 2024-04-16T02:41:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.432+0000 7fe57a386700 1 -- 172.21.15.151:0/2615347948 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 msgr2=0x7fe5740ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.432+0000 7fe57a386700 1 --2- 172.21.15.151:0/2615347948 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 0x7fe5740ffed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe564009910 tx=0x7fe564009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.432+0000 7fe57a386700 1 -- 172.21.15.151:0/2615347948 shutdown_connections 2024-04-16T02:41:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.432+0000 7fe57a386700 1 --2- 172.21.15.151:0/2615347948 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 0x7fe5740ffed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.432+0000 7fe57a386700 1 --2- 172.21.15.151:0/2615347948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe574105a50 0x7fe574105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.432+0000 7fe57a386700 1 -- 172.21.15.151:0/2615347948 >> 172.21.15.151:0/2615347948 conn(0x7fe5740fb540 msgr2=0x7fe5740fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.432+0000 7fe57a386700 1 -- 172.21.15.151:0/2615347948 shutdown_connections 2024-04-16T02:41:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.432+0000 7fe57a386700 1 -- 172.21.15.151:0/2615347948 wait complete. 2024-04-16T02:41:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.432+0000 7fe57a386700 1 Processor -- start 2024-04-16T02:41:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.433+0000 7fe57a386700 1 -- start start 2024-04-16T02:41:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.433+0000 7fe57a386700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 0x7fe5741953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.433+0000 7fe57a386700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe574105a50 0x7fe5741958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.433+0000 7fe57a386700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe574195e20 con 0x7fe5740ffa50 2024-04-16T02:41:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.433+0000 7fe57a386700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe574195f90 con 0x7fe574105a50 2024-04-16T02:41:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.433+0000 7fe573fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 0x7fe5741953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.433+0000 7fe573fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 0x7fe5741953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45724/0 (socket says 172.21.15.151:45724) 2024-04-16T02:41:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.433+0000 7fe573fff700 1 -- 172.21.15.151:0/2139917317 learned_addr learned my addr 172.21.15.151:0/2139917317 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.433+0000 7fe5737fe700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe574105a50 0x7fe5741958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.434+0000 7fe573fff700 1 -- 172.21.15.151:0/2139917317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe574105a50 msgr2=0x7fe5741958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.434+0000 7fe573fff700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe574105a50 0x7fe5741958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.434+0000 7fe573fff700 1 -- 172.21.15.151:0/2139917317 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5640095f0 con 0x7fe5740ffa50 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.434+0000 7fe573fff700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 0x7fe5741953a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00e920 tx=0x7fe55c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.434+0000 7fe5717fa700 1 -- 172.21.15.151:0/2139917317 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe55c00c9d0 con 0x7fe5740ffa50 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.434+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe574196210 con 0x7fe5740ffa50 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.434+0000 7fe5717fa700 1 -- 172.21.15.151:0/2139917317 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe55c010430 con 0x7fe5740ffa50 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.434+0000 7fe5717fa700 1 -- 172.21.15.151:0/2139917317 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe55c018630 con 0x7fe5740ffa50 2024-04-16T02:41:18.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.434+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5741a2d40 con 0x7fe5740ffa50 2024-04-16T02:41:18.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.435+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe574107fe0 con 0x7fe5740ffa50 2024-04-16T02:41:18.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.436+0000 7fe5717fa700 1 -- 172.21.15.151:0/2139917317 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7fe55c0105a0 con 0x7fe5740ffa50 2024-04-16T02:41:18.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.436+0000 7fe5717fa700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56006ee00 0x7fe5600712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:18.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.436+0000 7fe5717fa700 1 -- 172.21.15.151:0/2139917317 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe55c014070 con 0x7fe5740ffa50 2024-04-16T02:41:18.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.436+0000 7fe5737fe700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56006ee00 0x7fe5600712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:18.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.437+0000 7fe5737fe700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56006ee00 0x7fe5600712c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe574069780 tx=0x7fe564005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:18.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.445+0000 7fe5717fa700 1 -- 172.21.15.151:0/2139917317 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe55c053ff0 con 0x7fe5740ffa50 2024-04-16T02:41:18.476 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:18 smithi151 podman[123544]: 2024-04-16 02:41:18.221592223 +0000 UTC m=+0.615736094 container remove 4e5929a70655bffc177c5da12ecf13a219121ce0de2079b9560eed53bab84332 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-16T02:41:18.476 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:18 smithi151 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-16T02:41:18.476 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:18 smithi151 systemd[1]: Stopped Ceph osd.0 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:41:18.476 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:18 smithi151 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.0.service: Consumed 1min 2.868s CPU time. 2024-04-16T02:41:18.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.679+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe574196210 con 0x7fe56006ee00 2024-04-16T02:41:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.680+0000 7fe5717fa700 1 -- 172.21.15.151:0/2139917317 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe574196210 con 0x7fe56006ee00 2024-04-16T02:41:18.683 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:41:18.683 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:41:18.683 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:41:18.683 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:41:18.683 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:41:18.683 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:41:18.683 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:41:18.683 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:41:18.684 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:41:18.684 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "6/23 daemons upgraded", 2024-04-16T02:41:18.684 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T02:41:18.684 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:41:18.684 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:41:18.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.682+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56006ee00 msgr2=0x7fe5600712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.682+0000 7fe57a386700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56006ee00 0x7fe5600712c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe574069780 tx=0x7fe564005940 comp rx=0 tx=0).stop 2024-04-16T02:41:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.682+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 msgr2=0x7fe5741953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.682+0000 7fe57a386700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 0x7fe5741953a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00e920 tx=0x7fe55c00ec30 comp rx=0 tx=0).stop 2024-04-16T02:41:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.682+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 shutdown_connections 2024-04-16T02:41:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.682+0000 7fe57a386700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56006ee00 0x7fe5600712c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.682+0000 7fe57a386700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5740ffa50 0x7fe5741953a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.682+0000 7fe57a386700 1 --2- 172.21.15.151:0/2139917317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe574105a50 0x7fe5741958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.682+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 >> 172.21.15.151:0/2139917317 conn(0x7fe5740fb540 msgr2=0x7fe5740fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.683+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 shutdown_connections 2024-04-16T02:41:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.683+0000 7fe57a386700 1 -- 172.21.15.151:0/2139917317 wait complete. 2024-04-16T02:41:18.746 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:18 smithi151 systemd[1]: Starting Ceph osd.0 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:41:18.746 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:18 smithi151 podman[123713]: 2024-04-16 02:41:18.598461753 +0000 UTC m=+0.018977711 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:41:18.746 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:18 smithi151 podman[123713]: 2024-04-16 02:41:18.698626703 +0000 UTC m=+0.119142636 container create 04e7c2a9de794dad47a6150a40c7fdb79537eda580c3840f51fc609031e688ab (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-16T02:41:18.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 -- 172.21.15.151:0/2301922189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 msgr2=0x7fa2fc105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 --2- 172.21.15.151:0/2301922189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 0x7fa2fc105790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009910 tx=0x7fa2e4009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:18.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 -- 172.21.15.151:0/2301922189 shutdown_connections 2024-04-16T02:41:18.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 --2- 172.21.15.151:0/2301922189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 0x7fa2fc105790 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 --2- 172.21.15.151:0/2301922189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2fc068670 0x7fa2fc068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 -- 172.21.15.151:0/2301922189 >> 172.21.15.151:0/2301922189 conn(0x7fa2fc0755a0 msgr2=0x7fa2fc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:18.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 -- 172.21.15.151:0/2301922189 shutdown_connections 2024-04-16T02:41:18.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 -- 172.21.15.151:0/2301922189 wait complete. 2024-04-16T02:41:18.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 Processor -- start 2024-04-16T02:41:18.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.792+0000 7fa302554700 1 -- start start 2024-04-16T02:41:18.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa302554700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2fc068670 0x7fa2fc196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:18.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa302554700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 0x7fa2fc196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:18.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa302554700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc1972d0 con 0x7fa2fc069020 2024-04-16T02:41:18.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa302554700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc19a000 con 0x7fa2fc068670 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 0x7fa2fc196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 0x7fa2fc196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45744/0 (socket says 172.21.15.151:45744) 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f3fff700 1 -- 172.21.15.151:0/1522064828 learned_addr learned my addr 172.21.15.151:0/1522064828 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2fbfff700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2fc068670 0x7fa2fc196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f3fff700 1 -- 172.21.15.151:0/1522064828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2fc068670 msgr2=0x7fa2fc196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f3fff700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2fc068670 0x7fa2fc196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f3fff700 1 -- 172.21.15.151:0/1522064828 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40095f0 con 0x7fa2fc069020 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f3fff700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 0x7fa2fc196c40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40092e0 tx=0x7fa2e400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f9ffb700 1 -- 172.21.15.151:0/1522064828 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2e401d070 con 0x7fa2fc069020 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f9ffb700 1 -- 172.21.15.151:0/1522064828 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2e400f460 con 0x7fa2fc069020 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc19a280 con 0x7fa2fc069020 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.793+0000 7fa2f9ffb700 1 -- 172.21.15.151:0/1522064828 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2e40175f0 con 0x7fa2fc069020 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.794+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc19a770 con 0x7fa2fc069020 2024-04-16T02:41:18.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.794+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc1090a0 con 0x7fa2fc069020 2024-04-16T02:41:18.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.795+0000 7fa2f9ffb700 1 -- 172.21.15.151:0/1522064828 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 40) v1 ==== 92849+0+0 (secure 0 0 0) 0x7fa2e400f5d0 con 0x7fa2fc069020 2024-04-16T02:41:18.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.796+0000 7fa2f9ffb700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2e806ee00 0x7fa2e80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:18.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.796+0000 7fa2fbfff700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2e806ee00 0x7fa2e80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:18.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.796+0000 7fa2f9ffb700 1 -- 172.21.15.151:0/1522064828 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa2e40904e0 con 0x7fa2fc069020 2024-04-16T02:41:18.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.796+0000 7fa2fbfff700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2e806ee00 0x7fa2e80712c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec005fd0 tx=0x7fa2ec00a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:18.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:18.800+0000 7fa2f9ffb700 1 -- 172.21.15.151:0/1522064828 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa2e40591a0 con 0x7fa2fc069020 2024-04-16T02:41:19.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.101+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa2fc02cc70 con 0x7fa2fc069020 2024-04-16T02:41:19.104 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:18 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/333768140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:19.104 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:18 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4262317514' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:41:19.105 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:18 smithi151 ceph-mon[112233]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:19.105 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:18 smithi151 podman[123713]: 2024-04-16 02:41:18.960527113 +0000 UTC m=+0.381043047 container init 04e7c2a9de794dad47a6150a40c7fdb79537eda580c3840f51fc609031e688ab (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate, ceph=True, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:41:19.105 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:18 smithi151 podman[123713]: 2024-04-16 02:41:18.964064389 +0000 UTC m=+0.384580326 container start 04e7c2a9de794dad47a6150a40c7fdb79537eda580c3840f51fc609031e688ab (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-16T02:41:19.105 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 podman[123713]: 2024-04-16 02:41:19.035540794 +0000 UTC m=+0.456056741 container attach 04e7c2a9de794dad47a6150a40c7fdb79537eda580c3840f51fc609031e688ab (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True) 2024-04-16T02:41:19.105 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.102+0000 7fa2f9ffb700 1 -- 172.21.15.151:0/1522064828 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+100 (secure 0 0 0) 0x7fa2e405c860 con 0x7fa2fc069020 2024-04-16T02:41:19.105 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_WARN 1 osds down 2024-04-16T02:41:19.106 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-16T02:41:19.106 INFO:teuthology.orchestra.run.smithi151.stdout: osd.0 (root=default,host=smithi151) is down 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2e806ee00 msgr2=0x7fa2e80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2e806ee00 0x7fa2e80712c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec005fd0 tx=0x7fa2ec00a560 comp rx=0 tx=0).stop 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 msgr2=0x7fa2fc196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 0x7fa2fc196c40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40092e0 tx=0x7fa2e400be20 comp rx=0 tx=0).stop 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 shutdown_connections 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2e806ee00 0x7fa2e80712c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2fc068670 0x7fa2fc196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 --2- 172.21.15.151:0/1522064828 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2fc069020 0x7fa2fc196c40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 >> 172.21.15.151:0/1522064828 conn(0x7fa2fc0755a0 msgr2=0x7fa2fc105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 shutdown_connections 2024-04-16T02:41:19.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:19.104+0000 7fa302554700 1 -- 172.21.15.151:0/1522064828 wait complete. 2024-04-16T02:41:19.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:18 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/333768140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:19.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:18 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4262317514' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:41:19.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:18 smithi153 ceph-mon[87858]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:19.746 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate[123750]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate[123750]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 bash[123713]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 bash[123713]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 bash[123713]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 bash[123713]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 bash[123713]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 bash[123713]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 bash[123713]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate[123750]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate[123750]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T02:41:19.747 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate[123750]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-16T02:41:19.748 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate[123750]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T02:41:19.748 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate[123750]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-16T02:41:19.748 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:19 smithi151 podman[123713]: 2024-04-16 02:41:19.449844589 +0000 UTC m=+0.870360531 container died 04e7c2a9de794dad47a6150a40c7fdb79537eda580c3840f51fc609031e688ab (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T02:41:20.077 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:19 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1522064828' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:41:20.077 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:19 smithi151 ceph-mon[112233]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 64/336 objects degraded (19.048%) 2024-04-16T02:41:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:19 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1522064828' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:41:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:19 smithi153 ceph-mon[87858]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 64/336 objects degraded (19.048%) 2024-04-16T02:41:20.336 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:20 smithi151 podman[123878]: 2024-04-16 02:41:20.075004394 +0000 UTC m=+0.619832895 container remove 04e7c2a9de794dad47a6150a40c7fdb79537eda580c3840f51fc609031e688ab (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-16T02:41:20.337 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:20 smithi151 podman[123938]: 2024-04-16 02:41:20.191324041 +0000 UTC m=+0.018037397 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:41:20.337 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:20 smithi151 podman[123938]: 2024-04-16 02:41:20.334650076 +0000 UTC m=+0.161363473 container create fee11db8c57b7e47cae3509050a716d3548bdde89e218d71f973bcb1e9610165 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T02:41:20.596 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:20 smithi151 podman[123938]: 2024-04-16 02:41:20.570713478 +0000 UTC m=+0.397426833 container init fee11db8c57b7e47cae3509050a716d3548bdde89e218d71f973bcb1e9610165 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6) 2024-04-16T02:41:20.596 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:20 smithi151 podman[123938]: 2024-04-16 02:41:20.57388421 +0000 UTC m=+0.400597565 container start fee11db8c57b7e47cae3509050a716d3548bdde89e218d71f973bcb1e9610165 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-16T02:41:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:20 smithi151 ceph-mon[112233]: Health check failed: Degraded data redundancy: 64/336 objects degraded (19.048%), 19 pgs degraded (PG_DEGRADED) 2024-04-16T02:41:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:20.997 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:20 smithi151 ceph-osd[123957]: -- 172.21.15.151:0/1455959654 <== mon.0 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561655b67680 con 0x561655b31000 2024-04-16T02:41:20.997 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:20 smithi151 bash[123938]: fee11db8c57b7e47cae3509050a716d3548bdde89e218d71f973bcb1e9610165 2024-04-16T02:41:20.997 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:20 smithi151 systemd[1]: Started Ceph osd.0 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:41:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:20 smithi153 ceph-mon[87858]: Health check failed: Degraded data redundancy: 64/336 objects degraded (19.048%), 19 pgs degraded (PG_DEGRADED) 2024-04-16T02:41:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:21.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:21.496 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:21 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[123953]: 2024-04-16T02:41:21.113+0000 7f2a4fc277c0 -1 Falling back to public interface 2024-04-16T02:41:22.215 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:21 smithi153 ceph-mon[87858]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 64/336 objects degraded (19.048%) 2024-04-16T02:41:22.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:21 smithi151 ceph-mon[112233]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 64/336 objects degraded (19.048%) 2024-04-16T02:41:24.116 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:23 smithi151 ceph-mon[112233]: pgmap v50: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 64/336 objects degraded (19.048%) 2024-04-16T02:41:24.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:24.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:41:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:23 smithi153 ceph-mon[87858]: pgmap v50: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 64/336 objects degraded (19.048%) 2024-04-16T02:41:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:41:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:25 smithi151 ceph-mon[112233]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 614 B/s wr, 2 op/s; 64/336 objects degraded (19.048%) 2024-04-16T02:41:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:25 smithi151 ceph-mon[112233]: mgrmap e41: smithi151.uqqame(active, since 92s), standbys: smithi153.esbtzf 2024-04-16T02:41:25.996 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:25 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[123953]: 2024-04-16T02:41:25.630+0000 7f2a4fc277c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-16T02:41:26.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:25 smithi153 ceph-mon[87858]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 614 B/s wr, 2 op/s; 64/336 objects degraded (19.048%) 2024-04-16T02:41:26.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:25 smithi153 ceph-mon[87858]: mgrmap e41: smithi151.uqqame(active, since 92s), standbys: smithi153.esbtzf 2024-04-16T02:41:26.496 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:26 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[123953]: 2024-04-16T02:41:26.114+0000 7f2a4fc277c0 -1 osd.0 50 log_to_monitors true 2024-04-16T02:41:27.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:26 smithi151 ceph-mon[112233]: from='osd.0 [v2:172.21.15.151:6802/1997175184,v1:172.21.15.151:6803/1997175184]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T02:41:27.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:27.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:27.247 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 02:41:26 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[123953]: 2024-04-16T02:41:26.851+0000 7f2a4107f700 -1 osd.0 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:41:27.268 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:26 smithi153 ceph-mon[87858]: from='osd.0 [v2:172.21.15.151:6802/1997175184,v1:172.21.15.151:6803/1997175184]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T02:41:27.269 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:27.269 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:28.191 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:27 smithi151 ceph-mon[112233]: from='osd.0 [v2:172.21.15.151:6802/1997175184,v1:172.21.15.151:6803/1997175184]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T02:41:28.191 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:27 smithi151 ceph-mon[112233]: osdmap e53: 6 total, 5 up, 6 in 2024-04-16T02:41:28.191 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:27 smithi151 ceph-mon[112233]: from='osd.0 [v2:172.21.15.151:6802/1997175184,v1:172.21.15.151:6803/1997175184]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:41:28.192 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:27 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:28.192 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:27 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:28.192 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:27 smithi151 ceph-mon[112233]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 614 B/s wr, 2 op/s; 64/336 objects degraded (19.048%) 2024-04-16T02:41:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:27 smithi153 ceph-mon[87858]: from='osd.0 [v2:172.21.15.151:6802/1997175184,v1:172.21.15.151:6803/1997175184]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T02:41:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:27 smithi153 ceph-mon[87858]: osdmap e53: 6 total, 5 up, 6 in 2024-04-16T02:41:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:27 smithi153 ceph-mon[87858]: from='osd.0 [v2:172.21.15.151:6802/1997175184,v1:172.21.15.151:6803/1997175184]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:41:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:27 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:27 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:27 smithi153 ceph-mon[87858]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 614 B/s wr, 2 op/s; 64/336 objects degraded (19.048%) 2024-04-16T02:41:29.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:28 smithi151 ceph-mon[112233]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:41:29.159 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:28 smithi151 ceph-mon[112233]: osd.0 [v2:172.21.15.151:6802/1997175184,v1:172.21.15.151:6803/1997175184] boot 2024-04-16T02:41:29.159 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:28 smithi151 ceph-mon[112233]: osdmap e54: 6 total, 6 up, 6 in 2024-04-16T02:41:29.159 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:41:29.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:28 smithi153 ceph-mon[87858]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:41:29.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:28 smithi153 ceph-mon[87858]: osd.0 [v2:172.21.15.151:6802/1997175184,v1:172.21.15.151:6803/1997175184] boot 2024-04-16T02:41:29.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:28 smithi153 ceph-mon[87858]: osdmap e54: 6 total, 6 up, 6 in 2024-04-16T02:41:29.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T02:41:30.132 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:29 smithi151 ceph-mon[112233]: osdmap e55: 6 total, 6 up, 6 in 2024-04-16T02:41:30.132 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:29 smithi151 ceph-mon[112233]: pgmap v56: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s; 11/336 objects degraded (3.274%); 46 B/s, 0 objects/s recovering 2024-04-16T02:41:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:29 smithi153 ceph-mon[87858]: osdmap e55: 6 total, 6 up, 6 in 2024-04-16T02:41:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:29 smithi153 ceph-mon[87858]: pgmap v56: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s; 11/336 objects degraded (3.274%); 46 B/s, 0 objects/s recovering 2024-04-16T02:41:31.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:30 smithi151 ceph-mon[112233]: Health check update: Degraded data redundancy: 11/336 objects degraded (3.274%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T02:41:31.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:30 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:31.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:30 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:31.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:30 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:41:31.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:30 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:41:31.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:30 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:31.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:30 smithi153 ceph-mon[87858]: Health check update: Degraded data redundancy: 11/336 objects degraded (3.274%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T02:41:31.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:30 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:31.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:30 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:31.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:30 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:41:31.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:30 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:41:31.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:30 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:32.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:32.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:32.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:32.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:32.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T02:41:32.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:31 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T02:41:32.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:31 smithi151 ceph-mon[112233]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-16T02:41:32.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:31 smithi151 ceph-mon[112233]: pgmap v57: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 11/336 objects degraded (3.274%); 46 B/s, 0 objects/s recovering 2024-04-16T02:41:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T02:41:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:31 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T02:41:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:31 smithi153 ceph-mon[87858]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-16T02:41:32.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:31 smithi153 ceph-mon[87858]: pgmap v57: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 11/336 objects degraded (3.274%); 46 B/s, 0 objects/s recovering 2024-04-16T02:41:34.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:33 smithi151 ceph-mon[112233]: pgmap v58: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 40 B/s, 0 objects/s recovering 2024-04-16T02:41:34.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:33 smithi151 ceph-mon[112233]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/336 objects degraded (3.274%), 3 pgs degraded) 2024-04-16T02:41:34.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:33 smithi151 ceph-mon[112233]: Cluster is now healthy 2024-04-16T02:41:34.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:33 smithi153 ceph-mon[87858]: pgmap v58: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 40 B/s, 0 objects/s recovering 2024-04-16T02:41:34.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:33 smithi153 ceph-mon[87858]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/336 objects degraded (3.274%), 3 pgs degraded) 2024-04-16T02:41:34.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:33 smithi153 ceph-mon[87858]: Cluster is now healthy 2024-04-16T02:41:36.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:36 smithi151 ceph-mon[112233]: pgmap v59: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s; 35 B/s, 0 objects/s recovering 2024-04-16T02:41:36.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:36 smithi153 ceph-mon[87858]: pgmap v59: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s; 35 B/s, 0 objects/s recovering 2024-04-16T02:41:38.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:37 smithi151 ceph-mon[112233]: pgmap v60: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s; 28 B/s, 0 objects/s recovering 2024-04-16T02:41:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:37 smithi153 ceph-mon[87858]: pgmap v60: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s; 28 B/s, 0 objects/s recovering 2024-04-16T02:41:40.216 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:40.217 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:41:40.217 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:40.217 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:39 smithi151 ceph-mon[112233]: pgmap v61: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 5 op/s; 25 B/s, 0 objects/s recovering 2024-04-16T02:41:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:41:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:39 smithi153 ceph-mon[87858]: pgmap v61: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 5 op/s; 25 B/s, 0 objects/s recovering 2024-04-16T02:41:42.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:41 smithi151 ceph-mon[112233]: pgmap v62: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:41 smithi153 ceph-mon[87858]: pgmap v62: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:44.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:43 smithi151 ceph-mon[112233]: pgmap v63: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:44.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:43 smithi153 ceph-mon[87858]: pgmap v63: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:46.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:45 smithi153 ceph-mon[87858]: pgmap v64: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:46.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:45 smithi151 ceph-mon[112233]: pgmap v64: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:46.837 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T02:41:46.837 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:46 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T02:41:46.837 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:46 smithi151 ceph-mon[112233]: Upgrade: osd.1 is safe to restart 2024-04-16T02:41:47.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T02:41:47.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:46 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T02:41:47.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:46 smithi153 ceph-mon[87858]: Upgrade: osd.1 is safe to restart 2024-04-16T02:41:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:48 smithi151 ceph-mon[112233]: Upgrade: Updating osd.1 2024-04-16T02:41:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T02:41:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:48 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:41:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:48 smithi151 ceph-mon[112233]: Deploying daemon osd.1 on smithi151 2024-04-16T02:41:48.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:48 smithi151 ceph-mon[112233]: pgmap v65: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:48 smithi153 ceph-mon[87858]: Upgrade: Updating osd.1 2024-04-16T02:41:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T02:41:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:48 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:41:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:48 smithi153 ceph-mon[87858]: Deploying daemon osd.1 on smithi151 2024-04-16T02:41:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:48 smithi153 ceph-mon[87858]: pgmap v65: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:49.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.219+0000 7f9c0556f700 1 -- 172.21.15.151:0/4226888517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c00105a60 msgr2=0x7f9c00105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.219+0000 7f9c0556f700 1 --2- 172.21.15.151:0/4226888517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c00105a60 0x7f9c00105e40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009910 tx=0x7f9bf0009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:49.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.219+0000 7f9c0556f700 1 -- 172.21.15.151:0/4226888517 shutdown_connections 2024-04-16T02:41:49.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.219+0000 7f9c0556f700 1 --2- 172.21.15.151:0/4226888517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c000ffa60 0x7f9c000ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.219+0000 7f9c0556f700 1 --2- 172.21.15.151:0/4226888517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c00105a60 0x7f9c00105e40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.219+0000 7f9c0556f700 1 -- 172.21.15.151:0/4226888517 >> 172.21.15.151:0/4226888517 conn(0x7f9c000fb550 msgr2=0x7f9c000fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:49.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.219+0000 7f9c0556f700 1 -- 172.21.15.151:0/4226888517 shutdown_connections 2024-04-16T02:41:49.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.219+0000 7f9c0556f700 1 -- 172.21.15.151:0/4226888517 wait complete. 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.219+0000 7f9c0556f700 1 Processor -- start 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9c0556f700 1 -- start start 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9c0556f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c000ffa60 0x7f9c0019e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9c0556f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c00105a60 0x7f9c0019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9c0556f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0019f2e0 con 0x7f9c000ffa60 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9c0556f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c00198790 con 0x7f9c00105a60 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9bfeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c000ffa60 0x7f9c0019e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9bfeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c000ffa60 0x7f9c0019e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53660/0 (socket says 172.21.15.151:53660) 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9bfeffd700 1 -- 172.21.15.151:0/3228645502 learned_addr learned my addr 172.21.15.151:0/3228645502 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9bf67fc700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c00105a60 0x7f9c0019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9bfeffd700 1 -- 172.21.15.151:0/3228645502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c00105a60 msgr2=0x7f9c0019ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9bfeffd700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c00105a60 0x7f9c0019ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.220+0000 7f9bfeffd700 1 -- 172.21.15.151:0/3228645502 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00095f0 con 0x7f9c000ffa60 2024-04-16T02:41:49.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.221+0000 7f9bfeffd700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c000ffa60 0x7f9c0019e710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0006010 tx=0x7f9bf000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:49.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.221+0000 7f9bfcff9700 1 -- 172.21.15.151:0/3228645502 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bf001d070 con 0x7f9c000ffa60 2024-04-16T02:41:49.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.221+0000 7f9bfcff9700 1 -- 172.21.15.151:0/3228645502 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bf0004fb0 con 0x7f9c000ffa60 2024-04-16T02:41:49.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.221+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c00198a10 con 0x7f9c000ffa60 2024-04-16T02:41:49.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.221+0000 7f9bfcff9700 1 -- 172.21.15.151:0/3228645502 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bf0022ea0 con 0x7f9c000ffa60 2024-04-16T02:41:49.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.221+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c00198f00 con 0x7f9c000ffa60 2024-04-16T02:41:49.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.222+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0004ea90 con 0x7f9c000ffa60 2024-04-16T02:41:49.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.222+0000 7f9bfcff9700 1 -- 172.21.15.151:0/3228645502 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9bf0022470 con 0x7f9c000ffa60 2024-04-16T02:41:49.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.223+0000 7f9bfcff9700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bec06ecc0 0x7f9bec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:49.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.223+0000 7f9bfcff9700 1 -- 172.21.15.151:0/3228645502 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9bf0090340 con 0x7f9c000ffa60 2024-04-16T02:41:49.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.223+0000 7f9bf67fc700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bec06ecc0 0x7f9bec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:49.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.223+0000 7f9bf67fc700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bec06ecc0 0x7f9bec071180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9be8005fd0 tx=0x7f9be800a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:49.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.228+0000 7f9bfcff9700 1 -- 172.21.15.151:0/3228645502 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9bf005cd80 con 0x7f9c000ffa60 2024-04-16T02:41:49.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.468+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c00199350 con 0x7f9bec06ecc0 2024-04-16T02:41:49.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.469+0000 7f9bfcff9700 1 -- 172.21.15.151:0/3228645502 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9c00199350 con 0x7f9bec06ecc0 2024-04-16T02:41:49.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bec06ecc0 msgr2=0x7f9bec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bec06ecc0 0x7f9bec071180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9be8005fd0 tx=0x7f9be800a560 comp rx=0 tx=0).stop 2024-04-16T02:41:49.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c000ffa60 msgr2=0x7f9c0019e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c000ffa60 0x7f9c0019e710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0006010 tx=0x7f9bf000be20 comp rx=0 tx=0).stop 2024-04-16T02:41:49.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 shutdown_connections 2024-04-16T02:41:49.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bec06ecc0 0x7f9bec071180 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c000ffa60 0x7f9c0019e710 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 --2- 172.21.15.151:0/3228645502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c00105a60 0x7f9c0019ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 >> 172.21.15.151:0/3228645502 conn(0x7f9c000fb550 msgr2=0x7f9c00103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:49.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 shutdown_connections 2024-04-16T02:41:49.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.471+0000 7f9c0556f700 1 -- 172.21.15.151:0/3228645502 wait complete. 2024-04-16T02:41:49.485 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:41:49.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.579+0000 7fadf47ac700 1 -- 172.21.15.151:0/4065363224 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec105a70 msgr2=0x7fadec105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.579+0000 7fadf47ac700 1 --2- 172.21.15.151:0/4065363224 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec105a70 0x7fadec105e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fade8009910 tx=0x7fade8009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:49.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.579+0000 7fadf47ac700 1 -- 172.21.15.151:0/4065363224 shutdown_connections 2024-04-16T02:41:49.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.579+0000 7fadf47ac700 1 --2- 172.21.15.151:0/4065363224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadec0ffa70 0x7fadec0ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.579+0000 7fadf47ac700 1 --2- 172.21.15.151:0/4065363224 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec105a70 0x7fadec105e50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.579+0000 7fadf47ac700 1 -- 172.21.15.151:0/4065363224 >> 172.21.15.151:0/4065363224 conn(0x7fadec0fb560 msgr2=0x7fadec0fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:49.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.579+0000 7fadf47ac700 1 -- 172.21.15.151:0/4065363224 shutdown_connections 2024-04-16T02:41:49.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.579+0000 7fadf47ac700 1 -- 172.21.15.151:0/4065363224 wait complete. 2024-04-16T02:41:49.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf47ac700 1 Processor -- start 2024-04-16T02:41:49.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf47ac700 1 -- start start 2024-04-16T02:41:49.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf47ac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec0ffa70 0x7fadec19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:49.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf47ac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadec105a70 0x7fadec19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:49.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf47ac700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec19d100 con 0x7fadec0ffa70 2024-04-16T02:41:49.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf47ac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec196560 con 0x7fadec105a70 2024-04-16T02:41:49.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf2548700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec0ffa70 0x7fadec19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:49.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf2548700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec0ffa70 0x7fadec19c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53676/0 (socket says 172.21.15.151:53676) 2024-04-16T02:41:49.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf2548700 1 -- 172.21.15.151:0/952501492 learned_addr learned my addr 172.21.15.151:0/952501492 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf1d47700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadec105a70 0x7fadec19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf2548700 1 -- 172.21.15.151:0/952501492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadec105a70 msgr2=0x7fadec19cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf2548700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadec105a70 0x7fadec19cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf2548700 1 -- 172.21.15.151:0/952501492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade80095f0 con 0x7fadec0ffa70 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.580+0000 7fadf2548700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec0ffa70 0x7fadec19c570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fade8006010 tx=0x7fade800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.581+0000 7fade37fe700 1 -- 172.21.15.151:0/952501492 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fade801d070 con 0x7fadec0ffa70 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.581+0000 7fade37fe700 1 -- 172.21.15.151:0/952501492 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fade800f460 con 0x7fadec0ffa70 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.581+0000 7fade37fe700 1 -- 172.21.15.151:0/952501492 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fade8017630 con 0x7fadec0ffa70 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.581+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadec1967e0 con 0x7fadec0ffa70 2024-04-16T02:41:49.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.581+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadec196cd0 con 0x7fadec0ffa70 2024-04-16T02:41:49.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.582+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadec109080 con 0x7fadec0ffa70 2024-04-16T02:41:49.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.582+0000 7fade37fe700 1 -- 172.21.15.151:0/952501492 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fade800f5d0 con 0x7fadec0ffa70 2024-04-16T02:41:49.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.583+0000 7fade37fe700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fadd806ecc0 0x7fadd8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:49.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.583+0000 7fadf1d47700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fadd806ecc0 0x7fadd8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:49.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.583+0000 7fade37fe700 1 -- 172.21.15.151:0/952501492 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fade8090310 con 0x7fadec0ffa70 2024-04-16T02:41:49.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.583+0000 7fadf1d47700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fadd806ecc0 0x7fadd8071180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fadec197e00 tx=0x7faddc00a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:49.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.589+0000 7fade37fe700 1 -- 172.21.15.151:0/952501492 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fade8059170 con 0x7fadec0ffa70 2024-04-16T02:41:49.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.834+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fadec196fb0 con 0x7fadd806ecc0 2024-04-16T02:41:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.835+0000 7fade37fe700 1 -- 172.21.15.151:0/952501492 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fadec196fb0 con 0x7fadd806ecc0 2024-04-16T02:41:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.837+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fadd806ecc0 msgr2=0x7fadd8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.837+0000 7fadf47ac700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fadd806ecc0 0x7fadd8071180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fadec197e00 tx=0x7faddc00a380 comp rx=0 tx=0).stop 2024-04-16T02:41:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.837+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec0ffa70 msgr2=0x7fadec19c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.837+0000 7fadf47ac700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec0ffa70 0x7fadec19c570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fade8006010 tx=0x7fade800bd70 comp rx=0 tx=0).stop 2024-04-16T02:41:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.837+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 shutdown_connections 2024-04-16T02:41:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.837+0000 7fadf47ac700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fadd806ecc0 0x7fadd8071180 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.837+0000 7fadf47ac700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadec0ffa70 0x7fadec19c570 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.837+0000 7fadf47ac700 1 --2- 172.21.15.151:0/952501492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadec105a70 0x7fadec19cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.837+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 >> 172.21.15.151:0/952501492 conn(0x7fadec0fb560 msgr2=0x7fadec068fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.838+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 shutdown_connections 2024-04-16T02:41:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.838+0000 7fadf47ac700 1 -- 172.21.15.151:0/952501492 wait complete. 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.947+0000 7f407696e700 1 -- 172.21.15.151:0/4004452414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070068670 msgr2=0x7f4070068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.947+0000 7f407696e700 1 --2- 172.21.15.151:0/4004452414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070068670 0x7f4070068a50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4060009910 tx=0x7f4060009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.947+0000 7f407696e700 1 -- 172.21.15.151:0/4004452414 shutdown_connections 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.947+0000 7f407696e700 1 --2- 172.21.15.151:0/4004452414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4070069020 0x7f4070105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.947+0000 7f407696e700 1 --2- 172.21.15.151:0/4004452414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070068670 0x7f4070068a50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.947+0000 7f407696e700 1 -- 172.21.15.151:0/4004452414 >> 172.21.15.151:0/4004452414 conn(0x7f40700755a0 msgr2=0x7f40700759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.948+0000 7f407696e700 1 -- 172.21.15.151:0/4004452414 shutdown_connections 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.948+0000 7f407696e700 1 -- 172.21.15.151:0/4004452414 wait complete. 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.948+0000 7f407696e700 1 Processor -- start 2024-04-16T02:41:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.948+0000 7f407696e700 1 -- start start 2024-04-16T02:41:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.948+0000 7f407696e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4070068670 0x7f407019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f407696e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070069020 0x7f407019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f407696e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407019d050 con 0x7f4070069020 2024-04-16T02:41:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f407696e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4070196590 con 0x7f4070068670 2024-04-16T02:41:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070069020 0x7f407019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070069020 0x7f407019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53694/0 (socket says 172.21.15.151:53694) 2024-04-16T02:41:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406f7fe700 1 -- 172.21.15.151:0/3635978123 learned_addr learned my addr 172.21.15.151:0/3635978123 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406ffff700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4070068670 0x7f407019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:49.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406f7fe700 1 -- 172.21.15.151:0/3635978123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4070068670 msgr2=0x7f407019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:49.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406f7fe700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4070068670 0x7f407019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:49.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406f7fe700 1 -- 172.21.15.151:0/3635978123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40600095f0 con 0x7f4070069020 2024-04-16T02:41:49.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406f7fe700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070069020 0x7f407019ca50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f406400c960 tx=0x7f406400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:49.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406d7fa700 1 -- 172.21.15.151:0/3635978123 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4064007a70 con 0x7f4070069020 2024-04-16T02:41:49.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4070196870 con 0x7f4070069020 2024-04-16T02:41:49.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406d7fa700 1 -- 172.21.15.151:0/3635978123 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4064007bd0 con 0x7f4070069020 2024-04-16T02:41:49.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f406d7fa700 1 -- 172.21.15.151:0/3635978123 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40640186c0 con 0x7f4070069020 2024-04-16T02:41:49.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.949+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4070196dc0 con 0x7f4070069020 2024-04-16T02:41:49.955 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.951+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4070109130 con 0x7f4070069020 2024-04-16T02:41:49.955 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.951+0000 7f406d7fa700 1 -- 172.21.15.151:0/3635978123 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f406400faa0 con 0x7f4070069020 2024-04-16T02:41:49.955 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.951+0000 7f406d7fa700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f405806ecc0 0x7f4058071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:49.955 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.952+0000 7f406d7fa700 1 -- 172.21.15.151:0/3635978123 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f406408edc0 con 0x7f4070069020 2024-04-16T02:41:49.955 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.952+0000 7f406ffff700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f405806ecc0 0x7f4058071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.952+0000 7f406ffff700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f405806ecc0 0x7f4058071180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4060006010 tx=0x7f406001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:49.959 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:49.956+0000 7f406d7fa700 1 -- 172.21.15.151:0/3635978123 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f406405b4c0 con 0x7f4070069020 2024-04-16T02:41:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:49 smithi151 ceph-mon[112233]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:49 smithi151 ceph-mon[112233]: pgmap v66: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:50.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.172+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4070197040 con 0x7f405806ecc0 2024-04-16T02:41:50.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.182+0000 7f406d7fa700 1 -- 172.21.15.151:0/3635978123 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4070197040 con 0x7f405806ecc0 2024-04-16T02:41:50.185 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:41:50.185 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (3m) 24s ago 14m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (15m) 24s ago 14m 8288k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (13m) 47s ago 13m 8132k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (56s) 24s ago 14m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (51s) 47s ago 13m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (3m) 24s ago 14m 76.4M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (9m) 24s ago 9m 120M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (9m) 24s ago 9m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (9m) 47s ago 9m 58.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (9m) 47s ago 9m 67.4M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (5m) 24s ago 16m 499M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (4m) 47s ago 13m 450M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (2m) 24s ago 16m 44.7M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (99s) 47s ago 13m 41.2M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:41:50.186 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (4m) 24s ago 14m 15.9M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:41:50.187 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (4m) 47s ago 13m 15.9M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:41:50.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (29s) 24s ago 12m 12.7M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:41:50.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (12m) 24s ago 12m 251M 1216M 18.2.1 5be31c24972a 6b1afb23424e 2024-04-16T02:41:50.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (11m) 24s ago 11m 195M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:41:50.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (11m) 47s ago 11m 282M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:41:50.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (11m) 47s ago 11m 231M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:41:50.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (10m) 47s ago 10m 245M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:41:50.187 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (3m) 24s ago 14m 77.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:41:50.188 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f405806ecc0 msgr2=0x7f4058071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:50.188 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f405806ecc0 0x7f4058071180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4060006010 tx=0x7f406001a040 comp rx=0 tx=0).stop 2024-04-16T02:41:50.188 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070069020 msgr2=0x7f407019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:50.188 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070069020 0x7f407019ca50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f406400c960 tx=0x7f406400cd20 comp rx=0 tx=0).stop 2024-04-16T02:41:50.188 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 shutdown_connections 2024-04-16T02:41:50.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f405806ecc0 0x7f4058071180 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4070068670 0x7f407019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 --2- 172.21.15.151:0/3635978123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4070069020 0x7f407019ca50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 >> 172.21.15.151:0/3635978123 conn(0x7f40700755a0 msgr2=0x7f40700fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:50.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 shutdown_connections 2024-04-16T02:41:50.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.185+0000 7f407696e700 1 -- 172.21.15.151:0/3635978123 wait complete. 2024-04-16T02:41:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:49 smithi153 ceph-mon[87858]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:49 smithi153 ceph-mon[87858]: pgmap v66: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:41:50.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.290+0000 7fbd63bba700 1 -- 172.21.15.151:0/3855023078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 msgr2=0x7fbd5c10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:50.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.290+0000 7fbd63bba700 1 --2- 172.21.15.151:0/3855023078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 0x7fbd5c10ab10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009910 tx=0x7fbd50009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:50.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.290+0000 7fbd63bba700 1 -- 172.21.15.151:0/3855023078 shutdown_connections 2024-04-16T02:41:50.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.290+0000 7fbd63bba700 1 --2- 172.21.15.151:0/3855023078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 0x7fbd5c10ab10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.290+0000 7fbd63bba700 1 --2- 172.21.15.151:0/3855023078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd5c101c70 0x7fbd5c102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.290+0000 7fbd63bba700 1 -- 172.21.15.151:0/3855023078 >> 172.21.15.151:0/3855023078 conn(0x7fbd5c0fb570 msgr2=0x7fbd5c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:50.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.290+0000 7fbd63bba700 1 -- 172.21.15.151:0/3855023078 shutdown_connections 2024-04-16T02:41:50.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.291+0000 7fbd63bba700 1 -- 172.21.15.151:0/3855023078 wait complete. 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.291+0000 7fbd63bba700 1 Processor -- start 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.291+0000 7fbd63bba700 1 -- start start 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd63bba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd5c101c70 0x7fbd5c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd63bba700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 0x7fbd5c19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd63bba700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c19d0e0 con 0x7fbd5c102620 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd63bba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c196590 con 0x7fbd5c101c70 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd61155700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 0x7fbd5c19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd61155700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 0x7fbd5c19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53718/0 (socket says 172.21.15.151:53718) 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd61155700 1 -- 172.21.15.151:0/3517495803 learned_addr learned my addr 172.21.15.151:0/3517495803 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd61155700 1 -- 172.21.15.151:0/3517495803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd5c101c70 msgr2=0x7fbd5c19c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd61155700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd5c101c70 0x7fbd5c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd61155700 1 -- 172.21.15.151:0/3517495803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd500095f0 con 0x7fbd5c102620 2024-04-16T02:41:50.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd61155700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 0x7fbd5c19ca50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbd500092e0 tx=0x7fbd5000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:50.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd5affd700 1 -- 172.21.15.151:0/3517495803 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd5001d070 con 0x7fbd5c102620 2024-04-16T02:41:50.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd5affd700 1 -- 172.21.15.151:0/3517495803 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd5000f460 con 0x7fbd5c102620 2024-04-16T02:41:50.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c196810 con 0x7fbd5c102620 2024-04-16T02:41:50.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c196d00 con 0x7fbd5c102620 2024-04-16T02:41:50.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.292+0000 7fbd5affd700 1 -- 172.21.15.151:0/3517495803 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd500175f0 con 0x7fbd5c102620 2024-04-16T02:41:50.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.293+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c108210 con 0x7fbd5c102620 2024-04-16T02:41:50.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.294+0000 7fbd5affd700 1 -- 172.21.15.151:0/3517495803 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fbd5000f5d0 con 0x7fbd5c102620 2024-04-16T02:41:50.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.294+0000 7fbd5affd700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd4406ecc0 0x7fbd44071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:50.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.294+0000 7fbd5affd700 1 -- 172.21.15.151:0/3517495803 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbd500902f0 con 0x7fbd5c102620 2024-04-16T02:41:50.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.294+0000 7fbd61956700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd4406ecc0 0x7fbd44071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:50.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.295+0000 7fbd61956700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd4406ecc0 0x7fbd44071180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00b420 tx=0x7fbd4c008540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:50.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.299+0000 7fbd5affd700 1 -- 172.21.15.151:0/3517495803 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbd5005cc80 con 0x7fbd5c102620 2024-04-16T02:41:50.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.588+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbd5c04ea90 con 0x7fbd5c102620 2024-04-16T02:41:50.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.589+0000 7fbd5affd700 1 -- 172.21.15.151:0/3517495803 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fbd5005c810 con 0x7fbd5c102620 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 1 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:41:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:41:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:41:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 9, 2024-04-16T02:41:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 5 2024-04-16T02:41:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:41:50.593 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:41:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd4406ecc0 msgr2=0x7fbd44071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd4406ecc0 0x7fbd44071180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00b420 tx=0x7fbd4c008540 comp rx=0 tx=0).stop 2024-04-16T02:41:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 msgr2=0x7fbd5c19ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 0x7fbd5c19ca50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbd500092e0 tx=0x7fbd5000be20 comp rx=0 tx=0).stop 2024-04-16T02:41:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 shutdown_connections 2024-04-16T02:41:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd4406ecc0 0x7fbd44071180 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd5c101c70 0x7fbd5c19c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 --2- 172.21.15.151:0/3517495803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd5c102620 0x7fbd5c19ca50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 >> 172.21.15.151:0/3517495803 conn(0x7fbd5c0fb570 msgr2=0x7fbd5c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:50.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 shutdown_connections 2024-04-16T02:41:50.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.591+0000 7fbd63bba700 1 -- 172.21.15.151:0/3517495803 wait complete. 2024-04-16T02:41:50.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.697+0000 7f457d7d7700 1 -- 172.21.15.151:0/1269680414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 msgr2=0x7f4578105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:50.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.697+0000 7f457d7d7700 1 --2- 172.21.15.151:0/1269680414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 0x7f4578105e20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f456800b0a0 tx=0x7f456800b3b0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.697+0000 7f457d7d7700 1 -- 172.21.15.151:0/1269680414 shutdown_connections 2024-04-16T02:41:50.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.697+0000 7f457d7d7700 1 --2- 172.21.15.151:0/1269680414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45780ffa40 0x7f45780ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.697+0000 7f457d7d7700 1 --2- 172.21.15.151:0/1269680414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 0x7f4578105e20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.697+0000 7f457d7d7700 1 -- 172.21.15.151:0/1269680414 >> 172.21.15.151:0/1269680414 conn(0x7f45780fb570 msgr2=0x7f45780fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:50.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.697+0000 7f457d7d7700 1 -- 172.21.15.151:0/1269680414 shutdown_connections 2024-04-16T02:41:50.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.697+0000 7f457d7d7700 1 -- 172.21.15.151:0/1269680414 wait complete. 2024-04-16T02:41:50.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.698+0000 7f457d7d7700 1 Processor -- start 2024-04-16T02:41:50.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.698+0000 7f457d7d7700 1 -- start start 2024-04-16T02:41:50.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f457d7d7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45780ffa40 0x7f4578195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:50.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f457d7d7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 0x7f4578195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:50.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f457d7d7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4578196270 con 0x7f4578105a40 2024-04-16T02:41:50.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f457d7d7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457819a000 con 0x7f45780ffa40 2024-04-16T02:41:50.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f45767fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 0x7f4578195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:50.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f45767fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 0x7f4578195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53738/0 (socket says 172.21.15.151:53738) 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f45767fc700 1 -- 172.21.15.151:0/4232933683 learned_addr learned my addr 172.21.15.151:0/4232933683 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f45767fc700 1 -- 172.21.15.151:0/4232933683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45780ffa40 msgr2=0x7f4578195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f4576ffd700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45780ffa40 0x7f4578195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f45767fc700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45780ffa40 0x7f4578195650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f45767fc700 1 -- 172.21.15.151:0/4232933683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4568009d00 con 0x7f4578105a40 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f45767fc700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 0x7f4578195b90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f456000d8d0 tx=0x7f456000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f456ffff700 1 -- 172.21.15.151:0/4232933683 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4560009940 con 0x7f4578105a40 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f456ffff700 1 -- 172.21.15.151:0/4232933683 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4560010460 con 0x7f4578105a40 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457819a2e0 con 0x7f4578105a40 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.699+0000 7f456ffff700 1 -- 172.21.15.151:0/4232933683 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f456000f5f0 con 0x7f4578105a40 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.700+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457819a830 con 0x7f4578105a40 2024-04-16T02:41:50.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.700+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4578108040 con 0x7f4578105a40 2024-04-16T02:41:50.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.701+0000 7f456ffff700 1 -- 172.21.15.151:0/4232933683 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f45600105d0 con 0x7f4578105a40 2024-04-16T02:41:50.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.701+0000 7f456ffff700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f456406ecc0 0x7f4564071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:50.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.702+0000 7f4576ffd700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f456406ecc0 0x7f4564071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:50.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.702+0000 7f456ffff700 1 -- 172.21.15.151:0/4232933683 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f456008edd0 con 0x7f4578105a40 2024-04-16T02:41:50.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.702+0000 7f4576ffd700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f456406ecc0 0x7f4564071180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f456800b830 tx=0x7f4568006000 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:50.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:50.706+0000 7f456ffff700 1 -- 172.21.15.151:0/4232933683 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4560057c30 con 0x7f4578105a40 2024-04-16T02:41:50.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:50 smithi151 ceph-mon[112233]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:50.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:50 smithi151 ceph-mon[112233]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:50.914 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:50 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3517495803' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:51.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.016+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f457804ea90 con 0x7f4578105a40 2024-04-16T02:41:51.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.017+0000 7f456ffff700 1 -- 172.21.15.151:0/4232933683 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1919 (secure 0 0 0) 0x7f456005b2f0 con 0x7f4578105a40 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:e22 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 22 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:40:05.114230+0000 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:41:51.021 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:41:51.022 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 14478 members: 14490,14478 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:41:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f456406ecc0 msgr2=0x7f4564071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f456406ecc0 0x7f4564071180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f456800b830 tx=0x7f4568006000 comp rx=0 tx=0).stop 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 msgr2=0x7f4578195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 0x7f4578195b90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f456000d8d0 tx=0x7f456000dc90 comp rx=0 tx=0).stop 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 shutdown_connections 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f456406ecc0 0x7f4564071180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45780ffa40 0x7f4578195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 --2- 172.21.15.151:0/4232933683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4578105a40 0x7f4578195b90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 >> 172.21.15.151:0/4232933683 conn(0x7f45780fb570 msgr2=0x7f45780fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:51.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 shutdown_connections 2024-04-16T02:41:51.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.019+0000 7f457d7d7700 1 -- 172.21.15.151:0/4232933683 wait complete. 2024-04-16T02:41:51.025 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 22 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.130+0000 7fa8d191c700 1 -- 172.21.15.151:0/2823980903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc105b20 msgr2=0x7fa8cc105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.130+0000 7fa8d191c700 1 --2- 172.21.15.151:0/2823980903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc105b20 0x7fa8cc105f00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc009910 tx=0x7fa8bc009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.130+0000 7fa8d191c700 1 -- 172.21.15.151:0/2823980903 shutdown_connections 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.130+0000 7fa8d191c700 1 --2- 172.21.15.151:0/2823980903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8cc0feff0 0x7fa8cc0ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.130+0000 7fa8d191c700 1 --2- 172.21.15.151:0/2823980903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc105b20 0x7fa8cc105f00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.130+0000 7fa8d191c700 1 -- 172.21.15.151:0/2823980903 >> 172.21.15.151:0/2823980903 conn(0x7fa8cc075e30 msgr2=0x7fa8cc0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.130+0000 7fa8d191c700 1 -- 172.21.15.151:0/2823980903 shutdown_connections 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.130+0000 7fa8d191c700 1 -- 172.21.15.151:0/2823980903 wait complete. 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.131+0000 7fa8d191c700 1 Processor -- start 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.131+0000 7fa8d191c700 1 -- start start 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.131+0000 7fa8d191c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc0feff0 0x7fa8cc19c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.131+0000 7fa8d191c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8cc105b20 0x7fa8cc19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.131+0000 7fa8d191c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc19d120 con 0x7fa8cc0feff0 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.131+0000 7fa8d191c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc1965d0 con 0x7fa8cc105b20 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.131+0000 7fa8caffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc0feff0 0x7fa8cc19c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.131+0000 7fa8caffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc0feff0 0x7fa8cc19c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53762/0 (socket says 172.21.15.151:53762) 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.131+0000 7fa8caffd700 1 -- 172.21.15.151:0/2203230430 learned_addr learned my addr 172.21.15.151:0/2203230430 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8c3fff700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8cc105b20 0x7fa8cc19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8caffd700 1 -- 172.21.15.151:0/2203230430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8cc105b20 msgr2=0x7fa8cc19ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8caffd700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8cc105b20 0x7fa8cc19ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8caffd700 1 -- 172.21.15.151:0/2203230430 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc0095f0 con 0x7fa8cc0feff0 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8c3fff700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8cc105b20 0x7fa8cc19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8caffd700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc0feff0 0x7fa8cc19c550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00bd70 tx=0x7fa8bc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8d091a700 1 -- 172.21.15.151:0/2203230430 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8bc01d070 con 0x7fa8cc0feff0 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8d091a700 1 -- 172.21.15.151:0/2203230430 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8bc00f460 con 0x7fa8cc0feff0 2024-04-16T02:41:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc196850 con 0x7fa8cc0feff0 2024-04-16T02:41:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8d091a700 1 -- 172.21.15.151:0/2203230430 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8bc017630 con 0x7fa8cc0feff0 2024-04-16T02:41:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.132+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc196d40 con 0x7fa8cc0feff0 2024-04-16T02:41:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.133+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc109060 con 0x7fa8cc0feff0 2024-04-16T02:41:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.134+0000 7fa8d091a700 1 -- 172.21.15.151:0/2203230430 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa8bc00f5d0 con 0x7fa8cc0feff0 2024-04-16T02:41:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.134+0000 7fa8d091a700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8b806ecc0 0x7fa8b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.134+0000 7fa8c3fff700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8b806ecc0 0x7fa8b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.134+0000 7fa8d091a700 1 -- 172.21.15.151:0/2203230430 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa8bc090400 con 0x7fa8cc0feff0 2024-04-16T02:41:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.135+0000 7fa8c3fff700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8b806ecc0 0x7fa8b8071180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4005fd0 tx=0x7fa8b4005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:51.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.139+0000 7fa8d091a700 1 -- 172.21.15.151:0/2203230430 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa8bc05cd90 con 0x7fa8cc0feff0 2024-04-16T02:41:51.246 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:51 smithi151 systemd[1]: Stopping Ceph osd.1 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:41:51.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:50 smithi153 ceph-mon[87858]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:51.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:50 smithi153 ceph-mon[87858]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:51.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:50 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3517495803' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:41:51.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.376+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8cc196fc0 con 0x7fa8b806ecc0 2024-04-16T02:41:51.381 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:41:51.381 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:41:51.381 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:41:51.381 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:41:51.381 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:41:51.381 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:41:51.381 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:41:51.381 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:41:51.382 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:41:51.382 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "7/23 daemons upgraded", 2024-04-16T02:41:51.382 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T02:41:51.382 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:41:51.382 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:41:51.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.378+0000 7fa8d091a700 1 -- 172.21.15.151:0/2203230430 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fa8cc196fc0 con 0x7fa8b806ecc0 2024-04-16T02:41:51.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8b806ecc0 msgr2=0x7fa8b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8b806ecc0 0x7fa8b8071180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4005fd0 tx=0x7fa8b4005dc0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc0feff0 msgr2=0x7fa8cc19c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc0feff0 0x7fa8cc19c550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00bd70 tx=0x7fa8bc00be50 comp rx=0 tx=0).stop 2024-04-16T02:41:51.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 shutdown_connections 2024-04-16T02:41:51.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8b806ecc0 0x7fa8b8071180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8cc0feff0 0x7fa8cc19c550 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 --2- 172.21.15.151:0/2203230430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8cc105b20 0x7fa8cc19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 >> 172.21.15.151:0/2203230430 conn(0x7fa8cc075e30 msgr2=0x7fa8cc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:51.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 shutdown_connections 2024-04-16T02:41:51.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.380+0000 7fa8d191c700 1 -- 172.21.15.151:0/2203230430 wait complete. 2024-04-16T02:41:51.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.487+0000 7f178862a700 1 -- 172.21.15.151:0/2025366481 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 msgr2=0x7f1780106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.487+0000 7f178862a700 1 --2- 172.21.15.151:0/2025366481 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 0x7f1780106fe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f177c009910 tx=0x7f177c009c20 comp rx=0 tx=0).stop 2024-04-16T02:41:51.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.487+0000 7f178862a700 1 -- 172.21.15.151:0/2025366481 shutdown_connections 2024-04-16T02:41:51.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.487+0000 7f178862a700 1 --2- 172.21.15.151:0/2025366481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1780100b50 0x7f1780100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.487+0000 7f178862a700 1 --2- 172.21.15.151:0/2025366481 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 0x7f1780106fe0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.487+0000 7f178862a700 1 -- 172.21.15.151:0/2025366481 >> 172.21.15.151:0/2025366481 conn(0x7f17800fc640 msgr2=0x7f17800fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:51.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.487+0000 7f178862a700 1 -- 172.21.15.151:0/2025366481 shutdown_connections 2024-04-16T02:41:51.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.487+0000 7f178862a700 1 -- 172.21.15.151:0/2025366481 wait complete. 2024-04-16T02:41:51.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.488+0000 7f178862a700 1 Processor -- start 2024-04-16T02:41:51.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.488+0000 7f178862a700 1 -- start start 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.488+0000 7f178862a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1780100b50 0x7f178019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.488+0000 7f178862a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 0x7f178019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.488+0000 7f178862a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178019d0e0 con 0x7f1780106c00 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.488+0000 7f178862a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1780196590 con 0x7f1780100b50 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.488+0000 7f1785bc5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 0x7f178019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.488+0000 7f1785bc5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 0x7f178019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53792/0 (socket says 172.21.15.151:53792) 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.488+0000 7f1785bc5700 1 -- 172.21.15.151:0/1848019499 learned_addr learned my addr 172.21.15.151:0/1848019499 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f17863c6700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1780100b50 0x7f178019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f1785bc5700 1 -- 172.21.15.151:0/1848019499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1780100b50 msgr2=0x7f178019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f1785bc5700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1780100b50 0x7f178019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f1785bc5700 1 -- 172.21.15.151:0/1848019499 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177c0095f0 con 0x7f1780106c00 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f1785bc5700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 0x7f178019ca50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f177400b4b0 tx=0x7f177400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f17737fe700 1 -- 172.21.15.151:0/1848019499 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1774011800 con 0x7f1780106c00 2024-04-16T02:41:51.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f17737fe700 1 -- 172.21.15.151:0/1848019499 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1774011e20 con 0x7f1780106c00 2024-04-16T02:41:51.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1780196870 con 0x7f1780106c00 2024-04-16T02:41:51.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1780196e40 con 0x7f1780106c00 2024-04-16T02:41:51.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.489+0000 7f17737fe700 1 -- 172.21.15.151:0/1848019499 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f177400f2e0 con 0x7f1780106c00 2024-04-16T02:41:51.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.490+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1780109030 con 0x7f1780106c00 2024-04-16T02:41:51.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.491+0000 7f17737fe700 1 -- 172.21.15.151:0/1848019499 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1774010380 con 0x7f1780106c00 2024-04-16T02:41:51.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.491+0000 7f17737fe700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f176c06ecc0 0x7f176c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:41:51.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.491+0000 7f17737fe700 1 -- 172.21.15.151:0/1848019499 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f177408e980 con 0x7f1780106c00 2024-04-16T02:41:51.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.491+0000 7f17863c6700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f176c06ecc0 0x7f176c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:41:51.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.492+0000 7f17863c6700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f176c06ecc0 0x7f176c071180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f177c00b5c0 tx=0x7f177c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:41:51.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.496+0000 7f17737fe700 1 -- 172.21.15.151:0/1848019499 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f177405b310 con 0x7f1780106c00 2024-04-16T02:41:51.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.779+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f178004ea90 con 0x7f1780106c00 2024-04-16T02:41:51.783 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:51 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[55137]: 2024-04-16T02:41:51.501+0000 7fc67d30d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T02:41:51.783 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:51 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[55137]: 2024-04-16T02:41:51.501+0000 7fc67d30d700 -1 osd.1 55 *** Got signal Terminated *** 2024-04-16T02:41:51.783 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:51 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[55137]: 2024-04-16T02:41:51.501+0000 7fc67d30d700 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T02:41:51.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.779+0000 7f17737fe700 1 -- 172.21.15.151:0/1848019499 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f177405aea0 con 0x7f1780106c00 2024-04-16T02:41:51.784 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_OK 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f176c06ecc0 msgr2=0x7f176c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f176c06ecc0 0x7f176c071180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f177c00b5c0 tx=0x7f177c01a040 comp rx=0 tx=0).stop 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 msgr2=0x7f178019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 0x7f178019ca50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f177400b4b0 tx=0x7f177400b870 comp rx=0 tx=0).stop 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 shutdown_connections 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f176c06ecc0 0x7f176c071180 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1780100b50 0x7f178019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 --2- 172.21.15.151:0/1848019499 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1780106c00 0x7f178019ca50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 >> 172.21.15.151:0/1848019499 conn(0x7f17800fc640 msgr2=0x7f17800fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 shutdown_connections 2024-04-16T02:41:51.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:41:51.782+0000 7f178862a700 1 -- 172.21.15.151:0/1848019499 wait complete. 2024-04-16T02:41:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:51 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4232933683' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:41:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:51 smithi151 ceph-mon[112233]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:51 smithi151 ceph-mon[112233]: osd.1 marked itself down and dead 2024-04-16T02:41:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:51 smithi151 ceph-mon[112233]: pgmap v67: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:51 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1848019499' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:41:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:51 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4232933683' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:41:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:51 smithi153 ceph-mon[87858]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:41:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:51 smithi153 ceph-mon[87858]: osd.1 marked itself down and dead 2024-04-16T02:41:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:51 smithi153 ceph-mon[87858]: pgmap v67: 65 pgs: 65 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:41:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:51 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1848019499' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:41:52.662 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:52 smithi151 podman[128863]: 2024-04-16 02:41:52.366815114 +0000 UTC m=+1.082393209 container died 6b1afb23424ec02a3bfe398dc1701491e35cd0fd47bf2872af1ca4406128160b (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-04-16T02:41:52.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:52 smithi151 ceph-mon[112233]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:41:52.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:52 smithi151 ceph-mon[112233]: osdmap e56: 6 total, 5 up, 6 in 2024-04-16T02:41:52.996 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:52 smithi151 podman[128863]: 2024-04-16 02:41:52.759528083 +0000 UTC m=+1.475106215 container cleanup 6b1afb23424ec02a3bfe398dc1701491e35cd0fd47bf2872af1ca4406128160b (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , ceph=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-16T02:41:52.996 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:52 smithi151 bash[128863]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1 2024-04-16T02:41:53.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:52 smithi153 ceph-mon[87858]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:41:53.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:52 smithi153 ceph-mon[87858]: osdmap e56: 6 total, 5 up, 6 in 2024-04-16T02:41:53.326 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:53 smithi151 podman[128909]: 2024-04-16 02:41:53.028557287 +0000 UTC m=+0.657233450 container remove 6b1afb23424ec02a3bfe398dc1701491e35cd0fd47bf2872af1ca4406128160b (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-16T02:41:53.585 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:53 smithi151 podman[128999]: 2024-04-16 02:41:53.227606168 +0000 UTC m=+0.018645172 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:41:53.585 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:53 smithi151 podman[128999]: 2024-04-16 02:41:53.357030392 +0000 UTC m=+0.148069398 container create 75e463f7ae09a6df44937381b93437bb7d3fb1e4198af1bd61847c75b534f867 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-16T02:41:53.585 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:53 smithi151 podman[128999]: 2024-04-16 02:41:53.582850584 +0000 UTC m=+0.373889588 container init 75e463f7ae09a6df44937381b93437bb7d3fb1e4198af1bd61847c75b534f867 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:41:53.847 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:53 smithi151 podman[128999]: 2024-04-16 02:41:53.587805334 +0000 UTC m=+0.378844338 container start 75e463f7ae09a6df44937381b93437bb7d3fb1e4198af1bd61847c75b534f867 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:41:53.847 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:53 smithi151 podman[128999]: 2024-04-16 02:41:53.666275841 +0000 UTC m=+0.457314849 container attach 75e463f7ae09a6df44937381b93437bb7d3fb1e4198af1bd61847c75b534f867 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T02:41:54.197 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:53 smithi151 podman[128999]: 2024-04-16 02:41:53.892300943 +0000 UTC m=+0.683339947 container died 75e463f7ae09a6df44937381b93437bb7d3fb1e4198af1bd61847c75b534f867 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-16T02:41:54.198 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:53 smithi151 ceph-mon[112233]: osdmap e57: 6 total, 5 up, 6 in 2024-04-16T02:41:54.198 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:53 smithi151 ceph-mon[112233]: pgmap v70: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-16T02:41:54.198 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:54.198 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:41:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:53 smithi153 ceph-mon[87858]: osdmap e57: 6 total, 5 up, 6 in 2024-04-16T02:41:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:53 smithi153 ceph-mon[87858]: pgmap v70: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-16T02:41:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:41:54.746 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:54 smithi151 podman[129029]: 2024-04-16 02:41:54.479770944 +0000 UTC m=+0.577204770 container remove 75e463f7ae09a6df44937381b93437bb7d3fb1e4198af1bd61847c75b534f867 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True) 2024-04-16T02:41:54.746 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:54 smithi151 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-16T02:41:54.747 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:54 smithi151 systemd[1]: Stopped Ceph osd.1 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:41:54.747 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:54 smithi151 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.1.service: Consumed 1min 10.867s CPU time. 2024-04-16T02:41:54.747 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:54 smithi151 systemd[1]: Starting Ceph osd.1 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:41:55.104 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:54 smithi151 ceph-mon[112233]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-16T02:41:55.104 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:54 smithi151 podman[129140]: 2024-04-16 02:41:54.83881519 +0000 UTC m=+0.018152836 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:41:55.105 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:54 smithi151 podman[129140]: 2024-04-16 02:41:54.973600266 +0000 UTC m=+0.152937948 container create 315aa2b153add7707767e1b2bcb5f04cf159ae11d5ff822db6636080620397fc (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-16T02:41:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:54 smithi153 ceph-mon[87858]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-16T02:41:55.496 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 podman[129140]: 2024-04-16 02:41:55.22697757 +0000 UTC m=+0.406315194 container init 315aa2b153add7707767e1b2bcb5f04cf159ae11d5ff822db6636080620397fc (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-16T02:41:55.496 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 podman[129140]: 2024-04-16 02:41:55.230637718 +0000 UTC m=+0.409975343 container start 315aa2b153add7707767e1b2bcb5f04cf159ae11d5ff822db6636080620397fc (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-87dbfb6, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T02:41:55.496 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 podman[129140]: 2024-04-16 02:41:55.302009839 +0000 UTC m=+0.481347474 container attach 315aa2b153add7707767e1b2bcb5f04cf159ae11d5ff822db6636080620397fc (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T02:41:55.992 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:55 smithi151 ceph-mon[112233]: pgmap v71: 65 pgs: 14 active+undersized+degraded, 5 peering, 15 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 46/336 objects degraded (13.690%) 2024-04-16T02:41:55.992 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate[129154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T02:41:55.992 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate[129154]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T02:41:55.992 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 bash[129140]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T02:41:55.992 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 bash[129140]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T02:41:55.992 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 bash[129140]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T02:41:55.992 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 bash[129140]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T02:41:55.993 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 bash[129140]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-16T02:41:55.993 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 bash[129140]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T02:41:55.993 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 bash[129140]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-16T02:41:55.993 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate[129154]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T02:41:55.993 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate[129154]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T02:41:55.993 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate[129154]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-16T02:41:55.993 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate[129154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T02:41:55.993 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate[129154]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-16T02:41:55.993 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:55 smithi151 podman[129140]: 2024-04-16 02:41:55.707377113 +0000 UTC m=+0.886714747 container died 315aa2b153add7707767e1b2bcb5f04cf159ae11d5ff822db6636080620397fc (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T02:41:56.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:55 smithi153 ceph-mon[87858]: pgmap v71: 65 pgs: 14 active+undersized+degraded, 5 peering, 15 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 46/336 objects degraded (13.690%) 2024-04-16T02:41:56.666 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:56 smithi151 podman[129294]: 2024-04-16 02:41:56.315953631 +0000 UTC m=+0.603077952 container remove 315aa2b153add7707767e1b2bcb5f04cf159ae11d5ff822db6636080620397fc (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-activate, GIT_CLEAN=True, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-16T02:41:56.666 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:56 smithi151 podman[129356]: 2024-04-16 02:41:56.431788133 +0000 UTC m=+0.018087189 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:41:56.666 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:56 smithi151 podman[129356]: 2024-04-16 02:41:56.558693973 +0000 UTC m=+0.144993005 container create e3b70d49369dd6efd1f93ae266e6763d6f57c9df7f2f6a03442035ec441ec2bd (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1, ceph=True, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-16T02:41:56.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:56 smithi151 ceph-mon[112233]: Health check failed: Degraded data redundancy: 46/336 objects degraded (13.690%), 14 pgs degraded (PG_DEGRADED) 2024-04-16T02:41:56.996 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:56 smithi151 podman[129356]: 2024-04-16 02:41:56.761989863 +0000 UTC m=+0.348288896 container init e3b70d49369dd6efd1f93ae266e6763d6f57c9df7f2f6a03442035ec441ec2bd (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T02:41:56.996 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:56 smithi151 podman[129356]: 2024-04-16 02:41:56.765353386 +0000 UTC m=+0.351652419 container start e3b70d49369dd6efd1f93ae266e6763d6f57c9df7f2f6a03442035ec441ec2bd (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:41:56.997 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:56 smithi151 ceph-osd[129375]: -- 172.21.15.151:0/2817861158 <== mon.0 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5606d47b1680 con 0x5606d477ac00 2024-04-16T02:41:56.997 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:56 smithi151 bash[129356]: e3b70d49369dd6efd1f93ae266e6763d6f57c9df7f2f6a03442035ec441ec2bd 2024-04-16T02:41:56.997 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:56 smithi151 systemd[1]: Started Ceph osd.1 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:41:57.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:56 smithi153 ceph-mon[87858]: Health check failed: Degraded data redundancy: 46/336 objects degraded (13.690%), 14 pgs degraded (PG_DEGRADED) 2024-04-16T02:41:57.719 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:41:57 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[129371]: 2024-04-16T02:41:57.308+0000 7f3e4959f7c0 -1 Falling back to public interface 2024-04-16T02:41:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:57 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:57 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:57 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:57 smithi151 ceph-mon[112233]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 52/336 objects degraded (15.476%) 2024-04-16T02:41:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:57 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:57 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:41:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:57 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:41:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:57 smithi153 ceph-mon[87858]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 52/336 objects degraded (15.476%) 2024-04-16T02:41:59.200 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:58 smithi151 ceph-mon[112233]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-16T02:41:59.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:58 smithi153 ceph-mon[87858]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-16T02:42:00.031 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:41:59 smithi151 ceph-mon[112233]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/336 objects degraded (15.476%) 2024-04-16T02:42:00.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:41:59 smithi153 ceph-mon[87858]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/336 objects degraded (15.476%) 2024-04-16T02:42:02.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:01 smithi151 ceph-mon[112233]: pgmap v74: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/336 objects degraded (15.476%) 2024-04-16T02:42:02.246 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:42:01 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[129371]: 2024-04-16T02:42:01.895+0000 7f3e4959f7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-16T02:42:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:01 smithi153 ceph-mon[87858]: pgmap v74: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/336 objects degraded (15.476%) 2024-04-16T02:42:02.746 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:42:02 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[129371]: 2024-04-16T02:42:02.449+0000 7f3e4959f7c0 -1 osd.1 55 log_to_monitors true 2024-04-16T02:42:03.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:03.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:03.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:03 smithi151 ceph-mon[112233]: from='osd.1 [v2:172.21.15.151:6810/4084991031,v1:172.21.15.151:6811/4084991031]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T02:42:03.497 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 02:42:03 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[129371]: 2024-04-16T02:42:03.140+0000 7f3e3a9f7700 -1 osd.1 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:42:03.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:03.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:03.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:03 smithi153 ceph-mon[87858]: from='osd.1 [v2:172.21.15.151:6810/4084991031,v1:172.21.15.151:6811/4084991031]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T02:42:04.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:04 smithi151 ceph-mon[112233]: from='osd.1 [v2:172.21.15.151:6810/4084991031,v1:172.21.15.151:6811/4084991031]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T02:42:04.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:04 smithi151 ceph-mon[112233]: osdmap e58: 6 total, 5 up, 6 in 2024-04-16T02:42:04.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:04 smithi151 ceph-mon[112233]: from='osd.1 [v2:172.21.15.151:6810/4084991031,v1:172.21.15.151:6811/4084991031]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:42:04.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:04 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:04.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:04 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:04.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:04 smithi151 ceph-mon[112233]: pgmap v76: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 52/336 objects degraded (15.476%) 2024-04-16T02:42:04.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:04 smithi153 ceph-mon[87858]: from='osd.1 [v2:172.21.15.151:6810/4084991031,v1:172.21.15.151:6811/4084991031]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T02:42:04.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:04 smithi153 ceph-mon[87858]: osdmap e58: 6 total, 5 up, 6 in 2024-04-16T02:42:04.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:04 smithi153 ceph-mon[87858]: from='osd.1 [v2:172.21.15.151:6810/4084991031,v1:172.21.15.151:6811/4084991031]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:42:04.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:04 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:04.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:04 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:04.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:04 smithi153 ceph-mon[87858]: pgmap v76: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 52/336 objects degraded (15.476%) 2024-04-16T02:42:05.410 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:05 smithi151 ceph-mon[112233]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:42:05.410 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:05 smithi151 ceph-mon[112233]: osd.1 [v2:172.21.15.151:6810/4084991031,v1:172.21.15.151:6811/4084991031] boot 2024-04-16T02:42:05.410 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:05 smithi151 ceph-mon[112233]: osdmap e59: 6 total, 6 up, 6 in 2024-04-16T02:42:05.410 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:42:05.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:05 smithi153 ceph-mon[87858]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:42:05.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:05 smithi153 ceph-mon[87858]: osd.1 [v2:172.21.15.151:6810/4084991031,v1:172.21.15.151:6811/4084991031] boot 2024-04-16T02:42:05.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:05 smithi153 ceph-mon[87858]: osdmap e59: 6 total, 6 up, 6 in 2024-04-16T02:42:05.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T02:42:06.455 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:06 smithi151 ceph-mon[112233]: osdmap e60: 6 total, 6 up, 6 in 2024-04-16T02:42:06.455 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:06 smithi151 ceph-mon[112233]: Health check update: Degraded data redundancy: 52/336 objects degraded (15.476%), 16 pgs degraded (PG_DEGRADED) 2024-04-16T02:42:06.455 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:06 smithi151 ceph-mon[112233]: pgmap v79: 65 pgs: 20 peering, 7 active+undersized+degraded, 7 active+undersized, 31 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 21/336 objects degraded (6.250%) 2024-04-16T02:42:06.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:06 smithi153 ceph-mon[87858]: osdmap e60: 6 total, 6 up, 6 in 2024-04-16T02:42:06.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:06 smithi153 ceph-mon[87858]: Health check update: Degraded data redundancy: 52/336 objects degraded (15.476%), 16 pgs degraded (PG_DEGRADED) 2024-04-16T02:42:06.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:06 smithi153 ceph-mon[87858]: pgmap v79: 65 pgs: 20 peering, 7 active+undersized+degraded, 7 active+undersized, 31 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 21/336 objects degraded (6.250%) 2024-04-16T02:42:08.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:08.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-16T02:42:08.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:07 smithi153 ceph-mon[87858]: pgmap v80: 65 pgs: 20 peering, 7 active+undersized+degraded, 5 active+undersized, 33 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 21/336 objects degraded (6.250%) 2024-04-16T02:42:08.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-16T02:42:08.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:07 smithi151 ceph-mon[112233]: pgmap v80: 65 pgs: 20 peering, 7 active+undersized+degraded, 5 active+undersized, 33 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 21/336 objects degraded (6.250%) 2024-04-16T02:42:10.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:10.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:42:10.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:09 smithi151 ceph-mon[112233]: pgmap v81: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-16T02:42:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:42:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:09 smithi153 ceph-mon[87858]: pgmap v81: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-16T02:42:11.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:10 smithi151 ceph-mon[112233]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/336 objects degraded (6.250%), 7 pgs degraded) 2024-04-16T02:42:11.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:10 smithi151 ceph-mon[112233]: Cluster is now healthy 2024-04-16T02:42:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:10 smithi153 ceph-mon[87858]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/336 objects degraded (6.250%), 7 pgs degraded) 2024-04-16T02:42:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:10 smithi153 ceph-mon[87858]: Cluster is now healthy 2024-04-16T02:42:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:11 smithi151 ceph-mon[112233]: pgmap v82: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:42:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:11 smithi153 ceph-mon[87858]: pgmap v82: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:42:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:13 smithi151 ceph-mon[112233]: pgmap v83: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:42:14.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:13 smithi153 ceph-mon[87858]: pgmap v83: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:42:16.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:15 smithi151 ceph-mon[112233]: pgmap v84: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T02:42:16.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:15 smithi153 ceph-mon[87858]: pgmap v84: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T02:42:18.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:17 smithi151 ceph-mon[112233]: pgmap v85: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:42:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:17 smithi153 ceph-mon[87858]: pgmap v85: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:42:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:19 smithi151 ceph-mon[112233]: pgmap v86: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:42:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:19 smithi153 ceph-mon[87858]: pgmap v86: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:42:21.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.898+0000 7f9dedcf0700 1 -- 172.21.15.151:0/769280409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de81018e0 msgr2=0x7f9de8101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:21.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.898+0000 7f9dedcf0700 1 --2- 172.21.15.151:0/769280409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de81018e0 0x7f9de8101cc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8009910 tx=0x7f9dd8009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:21.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.898+0000 7f9dedcf0700 1 -- 172.21.15.151:0/769280409 shutdown_connections 2024-04-16T02:42:21.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.898+0000 7f9dedcf0700 1 --2- 172.21.15.151:0/769280409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de80ff260 0x7f9de80ff6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:21.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.898+0000 7f9dedcf0700 1 --2- 172.21.15.151:0/769280409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de81018e0 0x7f9de8101cc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:21.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.898+0000 7f9dedcf0700 1 -- 172.21.15.151:0/769280409 >> 172.21.15.151:0/769280409 conn(0x7f9de8074ce0 msgr2=0x7f9de80750f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:21.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.898+0000 7f9dedcf0700 1 -- 172.21.15.151:0/769280409 shutdown_connections 2024-04-16T02:42:21.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.898+0000 7f9dedcf0700 1 -- 172.21.15.151:0/769280409 wait complete. 2024-04-16T02:42:21.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9dedcf0700 1 Processor -- start 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9dedcf0700 1 -- start start 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9dedcf0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de80ff260 0x7f9de8198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9dedcf0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de81018e0 0x7f9de8198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9dedcf0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de8199550 con 0x7f9de80ff260 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9dedcf0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de819d360 con 0x7f9de81018e0 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9de77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de80ff260 0x7f9de8198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9de77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de80ff260 0x7f9de8198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52488/0 (socket says 172.21.15.151:52488) 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9de77fe700 1 -- 172.21.15.151:0/3698605576 learned_addr learned my addr 172.21.15.151:0/3698605576 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.899+0000 7f9de6ffd700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de81018e0 0x7f9de8198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.900+0000 7f9de77fe700 1 -- 172.21.15.151:0/3698605576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de81018e0 msgr2=0x7f9de8198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.900+0000 7f9de77fe700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de81018e0 0x7f9de8198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.900+0000 7f9de77fe700 1 -- 172.21.15.151:0/3698605576 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd80095f0 con 0x7f9de80ff260 2024-04-16T02:42:21.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.900+0000 7f9de77fe700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de80ff260 0x7f9de8198980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8006010 tx=0x7f9dd800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:21.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.900+0000 7f9de4ff9700 1 -- 172.21.15.151:0/3698605576 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd801d070 con 0x7f9de80ff260 2024-04-16T02:42:21.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.900+0000 7f9de4ff9700 1 -- 172.21.15.151:0/3698605576 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dd8005dc0 con 0x7f9de80ff260 2024-04-16T02:42:21.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.900+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de819d5e0 con 0x7f9de80ff260 2024-04-16T02:42:21.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.900+0000 7f9de4ff9700 1 -- 172.21.15.151:0/3698605576 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd8017400 con 0x7f9de80ff260 2024-04-16T02:42:21.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.900+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de819dad0 con 0x7f9de80ff260 2024-04-16T02:42:21.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.901+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de804ea90 con 0x7f9de80ff260 2024-04-16T02:42:21.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.902+0000 7f9de4ff9700 1 -- 172.21.15.151:0/3698605576 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9dd8003a60 con 0x7f9de80ff260 2024-04-16T02:42:21.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.902+0000 7f9de4ff9700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dd006ecc0 0x7f9dd0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:21.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.902+0000 7f9de4ff9700 1 -- 172.21.15.151:0/3698605576 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9dd8090150 con 0x7f9de80ff260 2024-04-16T02:42:21.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.902+0000 7f9de6ffd700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dd006ecc0 0x7f9dd0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:21.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.903+0000 7f9de6ffd700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dd006ecc0 0x7f9dd0071180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9de8199fa0 tx=0x7f9ddc00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:21.910 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:21.907+0000 7f9de4ff9700 1 -- 172.21.15.151:0/3698605576 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9dd805cb60 con 0x7f9de80ff260 2024-04-16T02:42:22.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.148+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9de819dec0 con 0x7f9dd006ecc0 2024-04-16T02:42:22.151 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:21 smithi151 ceph-mon[112233]: pgmap v87: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:42:22.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.150+0000 7f9de4ff9700 1 -- 172.21.15.151:0/3698605576 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9de819dec0 con 0x7f9dd006ecc0 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dd006ecc0 msgr2=0x7f9dd0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dd006ecc0 0x7f9dd0071180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9de8199fa0 tx=0x7f9ddc00b410 comp rx=0 tx=0).stop 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de80ff260 msgr2=0x7f9de8198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de80ff260 0x7f9de8198980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8006010 tx=0x7f9dd800bd70 comp rx=0 tx=0).stop 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 shutdown_connections 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dd006ecc0 0x7f9dd0071180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9de80ff260 0x7f9de8198980 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 --2- 172.21.15.151:0/3698605576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de81018e0 0x7f9de8198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 >> 172.21.15.151:0/3698605576 conn(0x7f9de8074ce0 msgr2=0x7f9de8106a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 shutdown_connections 2024-04-16T02:42:22.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.152+0000 7f9dedcf0700 1 -- 172.21.15.151:0/3698605576 wait complete. 2024-04-16T02:42:22.166 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:42:22.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.265+0000 7f3e48d2a700 1 -- 172.21.15.151:0/511584680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44068670 msgr2=0x7f3e44068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.265+0000 7f3e48d2a700 1 --2- 172.21.15.151:0/511584680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44068670 0x7f3e44068a50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009910 tx=0x7f3e34009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:22.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.265+0000 7f3e48d2a700 1 -- 172.21.15.151:0/511584680 shutdown_connections 2024-04-16T02:42:22.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.265+0000 7f3e48d2a700 1 --2- 172.21.15.151:0/511584680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e44069020 0x7f3e44105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.265+0000 7f3e48d2a700 1 --2- 172.21.15.151:0/511584680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44068670 0x7f3e44068a50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.265+0000 7f3e48d2a700 1 -- 172.21.15.151:0/511584680 >> 172.21.15.151:0/511584680 conn(0x7f3e440755a0 msgr2=0x7f3e440759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:22.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.265+0000 7f3e48d2a700 1 -- 172.21.15.151:0/511584680 shutdown_connections 2024-04-16T02:42:22.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.265+0000 7f3e48d2a700 1 -- 172.21.15.151:0/511584680 wait complete. 2024-04-16T02:42:22.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.266+0000 7f3e48d2a700 1 Processor -- start 2024-04-16T02:42:22.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.266+0000 7f3e48d2a700 1 -- start start 2024-04-16T02:42:22.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.266+0000 7f3e48d2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e44068670 0x7f3e440ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.266+0000 7f3e48d2a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44069020 0x7f3e441001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.266+0000 7f3e48d2a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e44100720 con 0x7f3e44069020 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.266+0000 7f3e48d2a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e44100860 con 0x7f3e44068670 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.266+0000 7f3e41d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44069020 0x7f3e441001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.266+0000 7f3e41d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44069020 0x7f3e441001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52506/0 (socket says 172.21.15.151:52506) 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e41d9b700 1 -- 172.21.15.151:0/966866560 learned_addr learned my addr 172.21.15.151:0/966866560 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e4259c700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e44068670 0x7f3e440ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e41d9b700 1 -- 172.21.15.151:0/966866560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e44068670 msgr2=0x7f3e440ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e41d9b700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e44068670 0x7f3e440ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e41d9b700 1 -- 172.21.15.151:0/966866560 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e340095f0 con 0x7f3e44069020 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e41d9b700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44069020 0x7f3e441001e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e3800c8f0 tx=0x7f3e3800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e337fe700 1 -- 172.21.15.151:0/966866560 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e380078a0 con 0x7f3e44069020 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e48d2a700 1 -- 172.21.15.151:0/966866560 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e441a2910 con 0x7f3e44069020 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e337fe700 1 -- 172.21.15.151:0/966866560 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e3800f450 con 0x7f3e44069020 2024-04-16T02:42:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e337fe700 1 -- 172.21.15.151:0/966866560 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e38018610 con 0x7f3e44069020 2024-04-16T02:42:22.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.267+0000 7f3e48d2a700 1 -- 172.21.15.151:0/966866560 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e441a2d20 con 0x7f3e44069020 2024-04-16T02:42:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:21 smithi153 ceph-mon[87858]: pgmap v87: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:42:22.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.269+0000 7f3e317fa700 1 -- 172.21.15.151:0/966866560 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e280052f0 con 0x7f3e44069020 2024-04-16T02:42:22.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.269+0000 7f3e337fe700 1 -- 172.21.15.151:0/966866560 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3e3800f5c0 con 0x7f3e44069020 2024-04-16T02:42:22.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.270+0000 7f3e337fe700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:22.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.270+0000 7f3e337fe700 1 -- 172.21.15.151:0/966866560 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3e3808ed30 con 0x7f3e44069020 2024-04-16T02:42:22.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.270+0000 7f3e4259c700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:22.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.271+0000 7f3e4259c700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e34006010 tx=0x7f3e3401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:22.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.275+0000 7f3e337fe700 1 -- 172.21.15.151:0/966866560 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3e38057b70 con 0x7f3e44069020 2024-04-16T02:42:22.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.501+0000 7f3e317fa700 1 -- 172.21.15.151:0/966866560 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e28000bc0 con 0x7f3e2c06ecc0 2024-04-16T02:42:22.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.503+0000 7f3e337fe700 1 -- 172.21.15.151:0/966866560 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f3e28000bc0 con 0x7f3e2c06ecc0 2024-04-16T02:42:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.506+0000 7f3e317fa700 1 -- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 msgr2=0x7f3e2c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.506+0000 7f3e317fa700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e34006010 tx=0x7f3e3401a040 comp rx=0 tx=0).stop 2024-04-16T02:42:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.506+0000 7f3e317fa700 1 -- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44069020 msgr2=0x7f3e441001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.506+0000 7f3e317fa700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44069020 0x7f3e441001e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e3800c8f0 tx=0x7f3e3800cc00 comp rx=0 tx=0).stop 2024-04-16T02:42:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.506+0000 7f3e317fa700 1 -- 172.21.15.151:0/966866560 shutdown_connections 2024-04-16T02:42:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.506+0000 7f3e317fa700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.506+0000 7f3e317fa700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e44068670 0x7f3e440ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.506+0000 7f3e317fa700 1 --2- 172.21.15.151:0/966866560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e44069020 0x7f3e441001e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.506+0000 7f3e317fa700 1 -- 172.21.15.151:0/966866560 >> 172.21.15.151:0/966866560 conn(0x7f3e440755a0 msgr2=0x7f3e440fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:22.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.507+0000 7f3e317fa700 1 -- 172.21.15.151:0/966866560 shutdown_connections 2024-04-16T02:42:22.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.507+0000 7f3e317fa700 1 -- 172.21.15.151:0/966866560 wait complete. 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.622+0000 7efd14580700 1 -- 172.21.15.151:0/2767620048 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 msgr2=0x7efd0c10cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.622+0000 7efd14580700 1 --2- 172.21.15.151:0/2767620048 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 0x7efd0c10cee0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efd08009910 tx=0x7efd08009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.622+0000 7efd14580700 1 -- 172.21.15.151:0/2767620048 shutdown_connections 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.622+0000 7efd14580700 1 --2- 172.21.15.151:0/2767620048 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 0x7efd0c10cee0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.622+0000 7efd14580700 1 --2- 172.21.15.151:0/2767620048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0c073160 0x7efd0c073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.622+0000 7efd14580700 1 -- 172.21.15.151:0/2767620048 >> 172.21.15.151:0/2767620048 conn(0x7efd0c0fc670 msgr2=0x7efd0c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.622+0000 7efd14580700 1 -- 172.21.15.151:0/2767620048 shutdown_connections 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.622+0000 7efd14580700 1 -- 172.21.15.151:0/2767620048 wait complete. 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd14580700 1 Processor -- start 2024-04-16T02:42:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd14580700 1 -- start start 2024-04-16T02:42:22.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd14580700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0c073160 0x7efd0c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:22.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd14580700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 0x7efd0c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:22.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd14580700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c19f260 con 0x7efd0c073a80 2024-04-16T02:42:22.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd14580700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c198780 con 0x7efd0c073160 2024-04-16T02:42:22.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd11b1b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 0x7efd0c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:22.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd11b1b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 0x7efd0c19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52528/0 (socket says 172.21.15.151:52528) 2024-04-16T02:42:22.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd11b1b700 1 -- 172.21.15.151:0/3374260929 learned_addr learned my addr 172.21.15.151:0/3374260929 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:22.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.623+0000 7efd1231c700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0c073160 0x7efd0c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd11b1b700 1 -- 172.21.15.151:0/3374260929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0c073160 msgr2=0x7efd0c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd11b1b700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0c073160 0x7efd0c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd11b1b700 1 -- 172.21.15.151:0/3374260929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd080095f0 con 0x7efd0c073a80 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd1231c700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0c073160 0x7efd0c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd11b1b700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 0x7efd0c19ebd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efd0800b7d0 tx=0x7efd0800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd037fe700 1 -- 172.21.15.151:0/3374260929 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd0801d070 con 0x7efd0c073a80 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd037fe700 1 -- 172.21.15.151:0/3374260929 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd080052a0 con 0x7efd0c073a80 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd037fe700 1 -- 172.21.15.151:0/3374260929 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd08022470 con 0x7efd0c073a80 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c198a00 con 0x7efd0c073a80 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.624+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c198ef0 con 0x7efd0c073a80 2024-04-16T02:42:22.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.625+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0c10a5e0 con 0x7efd0c073a80 2024-04-16T02:42:22.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.626+0000 7efd037fe700 1 -- 172.21.15.151:0/3374260929 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efd0800f850 con 0x7efd0c073a80 2024-04-16T02:42:22.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.627+0000 7efd037fe700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efcf806ecc0 0x7efcf8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:22.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.627+0000 7efd037fe700 1 -- 172.21.15.151:0/3374260929 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7efd080901a0 con 0x7efd0c073a80 2024-04-16T02:42:22.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.627+0000 7efd1231c700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efcf806ecc0 0x7efcf8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:22.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.628+0000 7efd1231c700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efcf806ecc0 0x7efcf8071180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efcfc011490 tx=0x7efcfc009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.632+0000 7efd037fe700 1 -- 172.21.15.151:0/3374260929 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efd0805cbb0 con 0x7efd0c073a80 2024-04-16T02:42:22.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.853+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efd0c02cce0 con 0x7efcf806ecc0 2024-04-16T02:42:22.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.865+0000 7efd037fe700 1 -- 172.21.15.151:0/3374260929 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7efd0c02cce0 con 0x7efcf806ecc0 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (3m) 20s ago 15m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (15m) 20s ago 15m 8502k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (14m) 80s ago 14m 8132k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (89s) 20s ago 15m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (83s) 80s ago 14m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (3m) 20s ago 14m 81.6M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (9m) 20s ago 9m 120M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (9m) 20s ago 9m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (9m) 80s ago 9m 58.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (9m) 80s ago 9m 67.4M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:42:22.869 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (5m) 20s ago 16m 504M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (4m) 80s ago 14m 450M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (2m) 20s ago 16m 46.7M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (2m) 80s ago 14m 41.2M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (4m) 20s ago 15m 16.0M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (4m) 80s ago 14m 15.9M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (62s) 20s ago 13m 87.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (26s) 20s ago 12m 12.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (12m) 20s ago 12m 196M 1216M 18.2.1 5be31c24972a 682d81bb816c 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (12m) 80s ago 12m 282M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (11m) 80s ago 11m 231M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (11m) 80s ago 11m 245M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:42:22.870 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (4m) 20s ago 14m 77.4M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:42:22.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efcf806ecc0 msgr2=0x7efcf8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efcf806ecc0 0x7efcf8071180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efcfc011490 tx=0x7efcfc009260 comp rx=0 tx=0).stop 2024-04-16T02:42:22.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 msgr2=0x7efd0c19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 0x7efd0c19ebd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efd0800b7d0 tx=0x7efd0800bf90 comp rx=0 tx=0).stop 2024-04-16T02:42:22.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 shutdown_connections 2024-04-16T02:42:22.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efcf806ecc0 0x7efcf8071180 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0c073160 0x7efd0c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 --2- 172.21.15.151:0/3374260929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd0c073a80 0x7efd0c19ebd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 >> 172.21.15.151:0/3374260929 conn(0x7efd0c0fc670 msgr2=0x7efd0c107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:22.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 shutdown_connections 2024-04-16T02:42:22.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.868+0000 7efd14580700 1 -- 172.21.15.151:0/3374260929 wait complete. 2024-04-16T02:42:22.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.973+0000 7f8655a76700 1 -- 172.21.15.151:0/3998346022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 msgr2=0x7f86500ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.973+0000 7f8655a76700 1 --2- 172.21.15.151:0/3998346022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 0x7f86500ff7e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f864800b0a0 tx=0x7f864800b3b0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.973+0000 7f8655a76700 1 -- 172.21.15.151:0/3998346022 shutdown_connections 2024-04-16T02:42:22.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.973+0000 7f8655a76700 1 --2- 172.21.15.151:0/3998346022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86500ffdb0 0x7f8650103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.973+0000 7f8655a76700 1 --2- 172.21.15.151:0/3998346022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 0x7f86500ff7e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.973+0000 7f8655a76700 1 -- 172.21.15.151:0/3998346022 >> 172.21.15.151:0/3998346022 conn(0x7f8650075e30 msgr2=0x7f86500fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:22.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.973+0000 7f8655a76700 1 -- 172.21.15.151:0/3998346022 shutdown_connections 2024-04-16T02:42:22.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.973+0000 7f8655a76700 1 -- 172.21.15.151:0/3998346022 wait complete. 2024-04-16T02:42:22.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.974+0000 7f8655a76700 1 Processor -- start 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.974+0000 7f8655a76700 1 -- start start 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.974+0000 7f8655a76700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 0x7f865019b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.974+0000 7f8655a76700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86500ffdb0 0x7f865019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.974+0000 7f8655a76700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865019c0e0 con 0x7f86500ff400 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.974+0000 7f8655a76700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8650195590 con 0x7f86500ffdb0 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.974+0000 7f864f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 0x7f865019b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.974+0000 7f864f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 0x7f865019b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52558/0 (socket says 172.21.15.151:52558) 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.974+0000 7f864f7fe700 1 -- 172.21.15.151:0/207744305 learned_addr learned my addr 172.21.15.151:0/207744305 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f864effd700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86500ffdb0 0x7f865019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f864f7fe700 1 -- 172.21.15.151:0/207744305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86500ffdb0 msgr2=0x7f865019ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:22.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f864f7fe700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86500ffdb0 0x7f865019ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:22.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f864f7fe700 1 -- 172.21.15.151:0/207744305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8648009d00 con 0x7f86500ff400 2024-04-16T02:42:22.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f864effd700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86500ffdb0 0x7f865019ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:42:22.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f864f7fe700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 0x7f865019b510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8648000f80 tx=0x7f864800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:22.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f864cff9700 1 -- 172.21.15.151:0/207744305 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f864800e040 con 0x7f86500ff400 2024-04-16T02:42:22.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f864cff9700 1 -- 172.21.15.151:0/207744305 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8648003e60 con 0x7f86500ff400 2024-04-16T02:42:22.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8650195810 con 0x7f86500ff400 2024-04-16T02:42:22.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f864cff9700 1 -- 172.21.15.151:0/207744305 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f864801d910 con 0x7f86500ff400 2024-04-16T02:42:22.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.975+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8650195d00 con 0x7f86500ff400 2024-04-16T02:42:22.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.976+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86501077a0 con 0x7f86500ff400 2024-04-16T02:42:22.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.977+0000 7f864cff9700 1 -- 172.21.15.151:0/207744305 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8648004040 con 0x7f86500ff400 2024-04-16T02:42:22.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.977+0000 7f864cff9700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:22.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.977+0000 7f864cff9700 1 -- 172.21.15.151:0/207744305 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f86480906b0 con 0x7f86500ff400 2024-04-16T02:42:22.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.977+0000 7f864effd700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:22.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.978+0000 7f864effd700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86501041c0 tx=0x7f8638008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:22.984 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:22.982+0000 7f864cff9700 1 -- 172.21.15.151:0/207744305 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f864805d040 con 0x7f86500ff400 2024-04-16T02:42:23.093 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:22 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T02:42:23.093 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:22 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T02:42:23.093 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:22 smithi151 ceph-mon[112233]: Upgrade: osd.2 is safe to restart 2024-04-16T02:42:23.093 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:22 smithi151 ceph-mon[112233]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:23.093 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:22 smithi151 ceph-mon[112233]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:23.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:22 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T02:42:23.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:22 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T02:42:23.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:22 smithi153 ceph-mon[87858]: Upgrade: osd.2 is safe to restart 2024-04-16T02:42:23.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:22 smithi153 ceph-mon[87858]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:23.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:22 smithi153 ceph-mon[87858]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:23.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.274+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f865004ea90 con 0x7f86500ff400 2024-04-16T02:42:23.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.274+0000 7f864cff9700 1 -- 172.21.15.151:0/207744305 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f864805cbd0 con 0x7f86500ff400 2024-04-16T02:42:23.277 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:42:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:42:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:42:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:42:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:42:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 8, 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:42:23.278 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 msgr2=0x7f863c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86501041c0 tx=0x7f8638008040 comp rx=0 tx=0).stop 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 msgr2=0x7f865019b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 0x7f865019b510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8648000f80 tx=0x7f864800cdd0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 shutdown_connections 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ff400 0x7f865019b510 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 --2- 172.21.15.151:0/207744305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86500ffdb0 0x7f865019ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 >> 172.21.15.151:0/207744305 conn(0x7f8650075e30 msgr2=0x7f86500fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:23.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 shutdown_connections 2024-04-16T02:42:23.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.277+0000 7f8655a76700 1 -- 172.21.15.151:0/207744305 wait complete. 2024-04-16T02:42:23.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2308609797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 msgr2=0x7ff1bc105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:23.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 --2- 172.21.15.151:0/2308609797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 0x7ff1bc105850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009910 tx=0x7ff1a4009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:23.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2308609797 shutdown_connections 2024-04-16T02:42:23.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 --2- 172.21.15.151:0/2308609797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 0x7ff1bc105850 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 --2- 172.21.15.151:0/2308609797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1bc0ff510 0x7ff1bc0ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2308609797 >> 172.21.15.151:0/2308609797 conn(0x7ff1bc074d80 msgr2=0x7ff1bc075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:23.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2308609797 shutdown_connections 2024-04-16T02:42:23.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2308609797 wait complete. 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 Processor -- start 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.388+0000 7ff1c29b2700 1 -- start start 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1c29b2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1bc0ff510 0x7ff1bc19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1c29b2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 0x7ff1bc19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1c29b2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc19d0e0 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1c29b2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc196590 con 0x7ff1bc0ff510 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 0x7ff1bc19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 0x7ff1bc19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52588/0 (socket says 172.21.15.151:52588) 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b3fff700 1 -- 172.21.15.151:0/2090757033 learned_addr learned my addr 172.21.15.151:0/2090757033 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b3fff700 1 -- 172.21.15.151:0/2090757033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1bc0ff510 msgr2=0x7ff1bc19c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1bbfff700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1bc0ff510 0x7ff1bc19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b3fff700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1bc0ff510 0x7ff1bc19c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b3fff700 1 -- 172.21.15.151:0/2090757033 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a40095f0 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b3fff700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 0x7ff1bc19ca50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400bd70 tx=0x7ff1a400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:23.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b9ffb700 1 -- 172.21.15.151:0/2090757033 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a401d070 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc196810 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b9ffb700 1 -- 172.21.15.151:0/2090757033 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1a4005dc0 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1b9ffb700 1 -- 172.21.15.151:0/2090757033 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a4017400 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.389+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc196d00 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.390+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc1091a0 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.391+0000 7ff1b9ffb700 1 -- 172.21.15.151:0/2090757033 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff1a400f7b0 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.392+0000 7ff1b9ffb700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff1a806ecc0 0x7ff1a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:23.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.392+0000 7ff1b9ffb700 1 -- 172.21.15.151:0/2090757033 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff1a4090280 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.392+0000 7ff1bbfff700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff1a806ecc0 0x7ff1a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:23.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.392+0000 7ff1bbfff700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff1a806ecc0 0x7ff1a8071180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac005950 tx=0x7ff1ac0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:23.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.397+0000 7ff1b9ffb700 1 -- 172.21.15.151:0/2090757033 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff1a405cc10 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.644+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff1bc04ea90 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.645+0000 7ff1b9ffb700 1 -- 172.21.15.151:0/2090757033 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1919 (secure 0 0 0) 0x7ff1a405c7a0 con 0x7ff1bc0ffec0 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:e22 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 22 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:40:05.114230+0000 2024-04-16T02:42:23.648 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:42:23.649 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 14478 members: 14490,14478 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:42:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:42:23.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff1a806ecc0 msgr2=0x7ff1a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:23.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff1a806ecc0 0x7ff1a8071180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac005950 tx=0x7ff1ac0058e0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 msgr2=0x7ff1bc19ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:23.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 0x7ff1bc19ca50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400bd70 tx=0x7ff1a400be50 comp rx=0 tx=0).stop 2024-04-16T02:42:23.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 shutdown_connections 2024-04-16T02:42:23.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff1a806ecc0 0x7ff1a8071180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1bc0ff510 0x7ff1bc19c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 --2- 172.21.15.151:0/2090757033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1bc0ffec0 0x7ff1bc19ca50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 >> 172.21.15.151:0/2090757033 conn(0x7ff1bc074d80 msgr2=0x7ff1bc0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:23.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 shutdown_connections 2024-04-16T02:42:23.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.647+0000 7ff1c29b2700 1 -- 172.21.15.151:0/2090757033 wait complete. 2024-04-16T02:42:23.652 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 22 2024-04-16T02:42:23.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.757+0000 7f2b2da49700 1 -- 172.21.15.151:0/1782718078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 msgr2=0x7f2b28102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.757+0000 7f2b2da49700 1 --2- 172.21.15.151:0/1782718078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 0x7f2b28102140 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009940 tx=0x7f2b18009c50 comp rx=0 tx=0).stop 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.757+0000 7f2b2da49700 1 -- 172.21.15.151:0/1782718078 shutdown_connections 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.757+0000 7f2b2da49700 1 --2- 172.21.15.151:0/1782718078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b28102680 0x7f2b2810ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.757+0000 7f2b2da49700 1 --2- 172.21.15.151:0/1782718078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 0x7f2b28102140 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.757+0000 7f2b2da49700 1 -- 172.21.15.151:0/1782718078 >> 172.21.15.151:0/1782718078 conn(0x7f2b280fb570 msgr2=0x7f2b280fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.757+0000 7f2b2da49700 1 -- 172.21.15.151:0/1782718078 shutdown_connections 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.757+0000 7f2b2da49700 1 -- 172.21.15.151:0/1782718078 wait complete. 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.757+0000 7f2b2da49700 1 Processor -- start 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b2da49700 1 -- start start 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b2da49700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 0x7f2b2819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b2da49700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b28102680 0x7f2b2819d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b2da49700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2819d960 con 0x7f2b28101d60 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b2da49700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b281a16f0 con 0x7f2b28102680 2024-04-16T02:42:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b26ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 0x7f2b2819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b26ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 0x7f2b2819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52606/0 (socket says 172.21.15.151:52606) 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b26ffd700 1 -- 172.21.15.151:0/855647323 learned_addr learned my addr 172.21.15.151:0/855647323 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b267fc700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b28102680 0x7f2b2819d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b26ffd700 1 -- 172.21.15.151:0/855647323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b28102680 msgr2=0x7f2b2819d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b26ffd700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b28102680 0x7f2b2819d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b26ffd700 1 -- 172.21.15.151:0/855647323 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b180095f0 con 0x7f2b28101d60 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.758+0000 7f2b26ffd700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 0x7f2b2819cd40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800b580 tx=0x7f2b180055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.759+0000 7f2b2ca47700 1 -- 172.21.15.151:0/855647323 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b1801d070 con 0x7f2b28101d60 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.759+0000 7f2b2ca47700 1 -- 172.21.15.151:0/855647323 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b18005dc0 con 0x7f2b28101d60 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.759+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b281a1970 con 0x7f2b28101d60 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.759+0000 7f2b2ca47700 1 -- 172.21.15.151:0/855647323 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b18021410 con 0x7f2b28101d60 2024-04-16T02:42:23.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.759+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b281a1e60 con 0x7f2b28101d60 2024-04-16T02:42:23.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.760+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b28108270 con 0x7f2b28101d60 2024-04-16T02:42:23.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.760+0000 7f2b2ca47700 1 -- 172.21.15.151:0/855647323 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2b1800f7f0 con 0x7f2b28101d60 2024-04-16T02:42:23.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.761+0000 7f2b2ca47700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b1006ecc0 0x7f2b10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:23.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.761+0000 7f2b267fc700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b1006ecc0 0x7f2b10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:23.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.761+0000 7f2b2ca47700 1 -- 172.21.15.151:0/855647323 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2b18090000 con 0x7f2b28101d60 2024-04-16T02:42:23.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.761+0000 7f2b267fc700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b1006ecc0 0x7f2b10071180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b2819e360 tx=0x7f2b1c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:23.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.765+0000 7f2b2ca47700 1 -- 172.21.15.151:0/855647323 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2b18058e60 con 0x7f2b28101d60 2024-04-16T02:42:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:23 smithi151 ceph-mon[112233]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:23 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/207744305' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:23 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2090757033' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:42:23.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:23 smithi151 ceph-mon[112233]: pgmap v88: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:42:23.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:23.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:42:23.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:23.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.994+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b2802cd60 con 0x7f2b1006ecc0 2024-04-16T02:42:23.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.995+0000 7f2b2ca47700 1 -- 172.21.15.151:0/855647323 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2b2802cd60 con 0x7f2b1006ecc0 2024-04-16T02:42:23.998 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:42:23.998 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:42:23.998 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:42:23.998 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:42:23.998 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:42:23.998 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:42:23.999 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:42:23.999 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:42:23.999 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:42:23.999 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "8/23 daemons upgraded", 2024-04-16T02:42:23.999 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T02:42:23.999 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:42:23.999 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b1006ecc0 msgr2=0x7f2b10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b1006ecc0 0x7f2b10071180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b2819e360 tx=0x7f2b1c00b410 comp rx=0 tx=0).stop 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 msgr2=0x7f2b2819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 0x7f2b2819cd40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800b580 tx=0x7f2b180055f0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 shutdown_connections 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b1006ecc0 0x7f2b10071180 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b28101d60 0x7f2b2819cd40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 --2- 172.21.15.151:0/855647323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b28102680 0x7f2b2819d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 >> 172.21.15.151:0/855647323 conn(0x7f2b280fb570 msgr2=0x7f2b281053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 shutdown_connections 2024-04-16T02:42:24.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:23.998+0000 7f2b2da49700 1 -- 172.21.15.151:0/855647323 wait complete. 2024-04-16T02:42:24.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.115+0000 7fd0c9632700 1 -- 172.21.15.151:0/3432011423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4069020 msgr2=0x7fd0c4105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:24.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.115+0000 7fd0c9632700 1 --2- 172.21.15.151:0/3432011423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4069020 0x7fd0c4105790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009910 tx=0x7fd0b4009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:24.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.115+0000 7fd0c9632700 1 -- 172.21.15.151:0/3432011423 shutdown_connections 2024-04-16T02:42:24.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.115+0000 7fd0c9632700 1 --2- 172.21.15.151:0/3432011423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4069020 0x7fd0c4105790 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.115+0000 7fd0c9632700 1 --2- 172.21.15.151:0/3432011423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0c4068670 0x7fd0c4068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.115+0000 7fd0c9632700 1 -- 172.21.15.151:0/3432011423 >> 172.21.15.151:0/3432011423 conn(0x7fd0c40755a0 msgr2=0x7fd0c40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:24.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.115+0000 7fd0c9632700 1 -- 172.21.15.151:0/3432011423 shutdown_connections 2024-04-16T02:42:24.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.115+0000 7fd0c9632700 1 -- 172.21.15.151:0/3432011423 wait complete. 2024-04-16T02:42:24.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.116+0000 7fd0c9632700 1 Processor -- start 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.116+0000 7fd0c9632700 1 -- start start 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.116+0000 7fd0c9632700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4068670 0x7fd0c4196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.116+0000 7fd0c9632700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0c4069020 0x7fd0c4196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.116+0000 7fd0c9632700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c41972d0 con 0x7fd0c4068670 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.116+0000 7fd0c9632700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c419a000 con 0x7fd0c4069020 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.116+0000 7fd0c2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4068670 0x7fd0c4196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.116+0000 7fd0c2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4068670 0x7fd0c4196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52610/0 (socket says 172.21.15.151:52610) 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.116+0000 7fd0c2ffd700 1 -- 172.21.15.151:0/2068594378 learned_addr learned my addr 172.21.15.151:0/2068594378 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0c27fc700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0c4069020 0x7fd0c4196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0c2ffd700 1 -- 172.21.15.151:0/2068594378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0c4069020 msgr2=0x7fd0c4196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0c2ffd700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0c4069020 0x7fd0c4196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0c2ffd700 1 -- 172.21.15.151:0/2068594378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b40095f0 con 0x7fd0c4068670 2024-04-16T02:42:24.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0c2ffd700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4068670 0x7fd0c4196700 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd0b800c8a0 tx=0x7fd0b800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:24.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0abfff700 1 -- 172.21.15.151:0/2068594378 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0b800cea0 con 0x7fd0c4068670 2024-04-16T02:42:24.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c419a2e0 con 0x7fd0c4068670 2024-04-16T02:42:24.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0abfff700 1 -- 172.21.15.151:0/2068594378 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0b8004830 con 0x7fd0c4068670 2024-04-16T02:42:24.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c419a800 con 0x7fd0c4068670 2024-04-16T02:42:24.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.117+0000 7fd0abfff700 1 -- 172.21.15.151:0/2068594378 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0b80055b0 con 0x7fd0c4068670 2024-04-16T02:42:24.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.118+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c41090a0 con 0x7fd0c4068670 2024-04-16T02:42:24.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.119+0000 7fd0abfff700 1 -- 172.21.15.151:0/2068594378 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd0b80049a0 con 0x7fd0c4068670 2024-04-16T02:42:24.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.119+0000 7fd0abfff700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd0ac0730d0 0x7fd0ac075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:24.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.119+0000 7fd0abfff700 1 -- 172.21.15.151:0/2068594378 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd0b808ea70 con 0x7fd0c4068670 2024-04-16T02:42:24.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.119+0000 7fd0c27fc700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd0ac0730d0 0x7fd0ac075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:24.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.120+0000 7fd0c27fc700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd0ac0730d0 0x7fd0ac075590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4197e20 tx=0x7fd0b401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:24.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.128+0000 7fd0abfff700 1 -- 172.21.15.151:0/2068594378 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd0b80578d0 con 0x7fd0c4068670 2024-04-16T02:42:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:23 smithi153 ceph-mon[87858]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:23 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/207744305' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:23 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2090757033' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:42:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:23 smithi153 ceph-mon[87858]: pgmap v88: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:42:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:42:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:24.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.421+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd0c404ea90 con 0x7fd0c4068670 2024-04-16T02:42:24.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.421+0000 7fd0abfff700 1 -- 172.21.15.151:0/2068594378 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fd0b801d020 con 0x7fd0c4068670 2024-04-16T02:42:24.425 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_OK 2024-04-16T02:42:24.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.424+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd0ac0730d0 msgr2=0x7fd0ac075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:24.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.424+0000 7fd0c9632700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd0ac0730d0 0x7fd0ac075590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4197e20 tx=0x7fd0b401a040 comp rx=0 tx=0).stop 2024-04-16T02:42:24.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.424+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4068670 msgr2=0x7fd0c4196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:24.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.424+0000 7fd0c9632700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4068670 0x7fd0c4196700 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd0b800c8a0 tx=0x7fd0b800cc60 comp rx=0 tx=0).stop 2024-04-16T02:42:24.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.424+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 shutdown_connections 2024-04-16T02:42:24.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.424+0000 7fd0c9632700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd0ac0730d0 0x7fd0ac075590 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.424+0000 7fd0c9632700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0c4068670 0x7fd0c4196700 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.424+0000 7fd0c9632700 1 --2- 172.21.15.151:0/2068594378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0c4069020 0x7fd0c4196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:24.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.424+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 >> 172.21.15.151:0/2068594378 conn(0x7fd0c40755a0 msgr2=0x7fd0c4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:24.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.425+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 shutdown_connections 2024-04-16T02:42:24.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:24.425+0000 7fd0c9632700 1 -- 172.21.15.151:0/2068594378 wait complete. 2024-04-16T02:42:25.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:24 smithi151 ceph-mon[112233]: Upgrade: Updating osd.2 2024-04-16T02:42:25.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T02:42:25.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:42:25.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:24 smithi151 ceph-mon[112233]: Deploying daemon osd.2 on smithi151 2024-04-16T02:42:25.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:24 smithi151 ceph-mon[112233]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:25.050 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:24 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2068594378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:42:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:24 smithi153 ceph-mon[87858]: Upgrade: Updating osd.2 2024-04-16T02:42:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T02:42:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:42:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:24 smithi153 ceph-mon[87858]: Deploying daemon osd.2 on smithi151 2024-04-16T02:42:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:24 smithi153 ceph-mon[87858]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:24 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2068594378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:42:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:25 smithi151 ceph-mon[112233]: pgmap v89: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:42:26.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:25 smithi153 ceph-mon[87858]: pgmap v89: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:42:27.409 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:27 smithi151 systemd[1]: Stopping Ceph osd.2 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:42:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:27 smithi151 ceph-mon[112233]: osd.2 marked itself down and dead 2024-04-16T02:42:27.747 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:27 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[63000]: 2024-04-16T02:42:27.405+0000 7f6a9243c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T02:42:27.747 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:27 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[63000]: 2024-04-16T02:42:27.405+0000 7f6a9243c700 -1 osd.2 60 *** Got signal Terminated *** 2024-04-16T02:42:27.747 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:27 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[63000]: 2024-04-16T02:42:27.405+0000 7f6a9243c700 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T02:42:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:27 smithi153 ceph-mon[87858]: osd.2 marked itself down and dead 2024-04-16T02:42:28.176 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:27 smithi151 podman[134474]: 2024-04-16 02:42:27.905825095 +0000 UTC m=+0.661698632 container died 682d81bb816cf407a7e6856cbe728c1364abfe384288f9472b3ca4014969744f (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1) 2024-04-16T02:42:28.475 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:28 smithi151 podman[134474]: 2024-04-16 02:42:28.264422874 +0000 UTC m=+1.020296404 container cleanup 682d81bb816cf407a7e6856cbe728c1364abfe384288f9472b3ca4014969744f (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-04-16T02:42:28.476 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:28 smithi151 bash[134474]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2 2024-04-16T02:42:28.746 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:28 smithi151 podman[134491]: 2024-04-16 02:42:28.541091951 +0000 UTC m=+0.630582258 container remove 682d81bb816cf407a7e6856cbe728c1364abfe384288f9472b3ca4014969744f (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1) 2024-04-16T02:42:28.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:28 smithi151 ceph-mon[112233]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:42:28.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:28 smithi151 ceph-mon[112233]: osdmap e61: 6 total, 5 up, 6 in 2024-04-16T02:42:28.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:28 smithi151 ceph-mon[112233]: pgmap v91: 65 pgs: 9 stale+active+clean, 56 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:42:28.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:28 smithi151 ceph-mon[112233]: osdmap e62: 6 total, 5 up, 6 in 2024-04-16T02:42:28.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:28 smithi153 ceph-mon[87858]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:42:28.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:28 smithi153 ceph-mon[87858]: osdmap e61: 6 total, 5 up, 6 in 2024-04-16T02:42:28.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:28 smithi153 ceph-mon[87858]: pgmap v91: 65 pgs: 9 stale+active+clean, 56 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:42:28.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:28 smithi153 ceph-mon[87858]: osdmap e62: 6 total, 5 up, 6 in 2024-04-16T02:42:29.164 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:28 smithi151 podman[134580]: 2024-04-16 02:42:28.782139944 +0000 UTC m=+0.018407415 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:42:29.164 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:28 smithi151 podman[134580]: 2024-04-16 02:42:28.900420168 +0000 UTC m=+0.136687668 container create 136854a148c392dea43bbe9e72a3ce2fcb1d40ab3083662abfeaa4cc80bd86b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-16T02:42:29.164 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:29 smithi151 podman[134580]: 2024-04-16 02:42:29.161503992 +0000 UTC m=+0.397771450 container init 136854a148c392dea43bbe9e72a3ce2fcb1d40ab3083662abfeaa4cc80bd86b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-16T02:42:29.462 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:29 smithi151 podman[134580]: 2024-04-16 02:42:29.164960734 +0000 UTC m=+0.401228201 container start 136854a148c392dea43bbe9e72a3ce2fcb1d40ab3083662abfeaa4cc80bd86b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6) 2024-04-16T02:42:29.462 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:29 smithi151 podman[134580]: 2024-04-16 02:42:29.278281765 +0000 UTC m=+0.514549246 container attach 136854a148c392dea43bbe9e72a3ce2fcb1d40ab3083662abfeaa4cc80bd86b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-16T02:42:29.746 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:29 smithi151 podman[134580]: 2024-04-16 02:42:29.460573647 +0000 UTC m=+0.696841104 container died 136854a148c392dea43bbe9e72a3ce2fcb1d40ab3083662abfeaa4cc80bd86b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-16T02:42:30.090 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:29 smithi151 ceph-mon[112233]: pgmap v93: 65 pgs: 5 active+undersized+degraded, 7 peering, 3 stale+active+clean, 7 active+undersized, 43 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 15/336 objects degraded (4.464%) 2024-04-16T02:42:30.090 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:29 smithi151 ceph-mon[112233]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-04-16T02:42:30.090 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:29 smithi151 ceph-mon[112233]: Health check failed: Degraded data redundancy: 15/336 objects degraded (4.464%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T02:42:30.090 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 podman[134606]: 2024-04-16 02:42:30.073658843 +0000 UTC m=+0.607254526 container remove 136854a148c392dea43bbe9e72a3ce2fcb1d40ab3083662abfeaa4cc80bd86b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:42:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:29 smithi153 ceph-mon[87858]: pgmap v93: 65 pgs: 5 active+undersized+degraded, 7 peering, 3 stale+active+clean, 7 active+undersized, 43 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 15/336 objects degraded (4.464%) 2024-04-16T02:42:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:29 smithi153 ceph-mon[87858]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-04-16T02:42:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:29 smithi153 ceph-mon[87858]: Health check failed: Degraded data redundancy: 15/336 objects degraded (4.464%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T02:42:30.496 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-16T02:42:30.496 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 systemd[1]: Stopped Ceph osd.2 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:42:30.496 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.2.service: Consumed 56.884s CPU time. 2024-04-16T02:42:30.496 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 systemd[1]: Starting Ceph osd.2 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:42:30.880 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 podman[134709]: 2024-04-16 02:42:30.431377426 +0000 UTC m=+0.018003839 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:42:30.880 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 podman[134709]: 2024-04-16 02:42:30.533500568 +0000 UTC m=+0.120127009 container create 291b42166798bf2abfc379eb6c1ece7a098fc9bf114c24d096636b138f81aa19 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:42:30.880 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 podman[134709]: 2024-04-16 02:42:30.752443679 +0000 UTC m=+0.339070074 container init 291b42166798bf2abfc379eb6c1ece7a098fc9bf114c24d096636b138f81aa19 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240409) 2024-04-16T02:42:30.880 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 podman[134709]: 2024-04-16 02:42:30.755961859 +0000 UTC m=+0.342588255 container start 291b42166798bf2abfc379eb6c1ece7a098fc9bf114c24d096636b138f81aa19 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-16T02:42:30.880 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:30 smithi151 podman[134709]: 2024-04-16 02:42:30.877682609 +0000 UTC m=+0.464309005 container attach 291b42166798bf2abfc379eb6c1ece7a098fc9bf114c24d096636b138f81aa19 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-16T02:42:31.496 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate[134724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate[134724]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 bash[134709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 bash[134709]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 bash[134709]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 bash[134709]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 bash[134709]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 bash[134709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 bash[134709]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate[134724]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate[134724]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate[134724]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate[134724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate[134724]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-16T02:42:31.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 podman[134709]: 2024-04-16 02:42:31.235499741 +0000 UTC m=+0.822126145 container died 291b42166798bf2abfc379eb6c1ece7a098fc9bf114c24d096636b138f81aa19 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:42:31.883 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:31 smithi151 ceph-mon[112233]: pgmap v94: 65 pgs: 5 active+undersized+degraded, 7 peering, 3 stale+active+clean, 7 active+undersized, 43 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 15/336 objects degraded (4.464%) 2024-04-16T02:42:32.219 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:31 smithi151 podman[134848]: 2024-04-16 02:42:31.88134877 +0000 UTC m=+0.639912694 container remove 291b42166798bf2abfc379eb6c1ece7a098fc9bf114c24d096636b138f81aa19 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-16T02:42:32.219 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:32 smithi151 podman[134910]: 2024-04-16 02:42:32.090643754 +0000 UTC m=+0.108044732 container create 4f046ee57e957df22e18ec5044171e29a94855786fc83303d7c36a01356f3fd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:42:32.219 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:32 smithi151 podman[134910]: 2024-04-16 02:42:32.001004451 +0000 UTC m=+0.018405442 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:42:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:31 smithi153 ceph-mon[87858]: pgmap v94: 65 pgs: 5 active+undersized+degraded, 7 peering, 3 stale+active+clean, 7 active+undersized, 43 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 15/336 objects degraded (4.464%) 2024-04-16T02:42:32.496 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:32 smithi151 podman[134910]: 2024-04-16 02:42:32.334539323 +0000 UTC m=+0.351940300 container init 4f046ee57e957df22e18ec5044171e29a94855786fc83303d7c36a01356f3fd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-16T02:42:32.496 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:32 smithi151 podman[134910]: 2024-04-16 02:42:32.337828701 +0000 UTC m=+0.355229679 container start 4f046ee57e957df22e18ec5044171e29a94855786fc83303d7c36a01356f3fd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T02:42:32.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:32 smithi151 ceph-osd[134929]: -- 172.21.15.151:0/160837242 <== mon.0 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559128f83680 con 0x559128f4d000 2024-04-16T02:42:32.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:32 smithi151 bash[134910]: 4f046ee57e957df22e18ec5044171e29a94855786fc83303d7c36a01356f3fd3 2024-04-16T02:42:32.497 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:32 smithi151 systemd[1]: Started Ceph osd.2 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:42:33.246 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:32 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[134925]: 2024-04-16T02:42:32.878+0000 7f65230367c0 -1 Falling back to public interface 2024-04-16T02:42:33.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:33 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:33.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:33 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:33.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:33 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:42:33.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:33 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:33.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:33 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:33.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:33 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:42:34.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:34 smithi151 ceph-mon[112233]: pgmap v95: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 41/336 objects degraded (12.202%) 2024-04-16T02:42:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:34 smithi153 ceph-mon[87858]: pgmap v95: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 41/336 objects degraded (12.202%) 2024-04-16T02:42:35.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:35 smithi151 ceph-mon[112233]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-16T02:42:35.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:35 smithi151 ceph-mon[112233]: Health check update: Degraded data redundancy: 41/336 objects degraded (12.202%), 12 pgs degraded (PG_DEGRADED) 2024-04-16T02:42:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:35 smithi153 ceph-mon[87858]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-16T02:42:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:35 smithi153 ceph-mon[87858]: Health check update: Degraded data redundancy: 41/336 objects degraded (12.202%), 12 pgs degraded (PG_DEGRADED) 2024-04-16T02:42:36.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:36 smithi151 ceph-mon[112233]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 41/336 objects degraded (12.202%) 2024-04-16T02:42:36.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:36 smithi153 ceph-mon[87858]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 41/336 objects degraded (12.202%) 2024-04-16T02:42:37.496 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:37 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[134925]: 2024-04-16T02:42:37.291+0000 7f65230367c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-16T02:42:37.996 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:37 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[134925]: 2024-04-16T02:42:37.736+0000 7f65230367c0 -1 osd.2 60 log_to_monitors true 2024-04-16T02:42:37.996 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 02:42:37 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[134925]: 2024-04-16T02:42:37.814+0000 7f651448e700 -1 osd.2 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:42:37.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:37 smithi151 ceph-mon[112233]: from='osd.2 [v2:172.21.15.151:6818/3234791793,v1:172.21.15.151:6819/3234791793]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T02:42:37.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:37 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:37.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:37 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:37.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:37 smithi151 ceph-mon[112233]: pgmap v97: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 41/336 objects degraded (12.202%) 2024-04-16T02:42:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:37 smithi153 ceph-mon[87858]: from='osd.2 [v2:172.21.15.151:6818/3234791793,v1:172.21.15.151:6819/3234791793]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T02:42:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:37 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:37 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:38.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:37 smithi153 ceph-mon[87858]: pgmap v97: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 41/336 objects degraded (12.202%) 2024-04-16T02:42:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:38 smithi151 ceph-mon[112233]: from='osd.2 [v2:172.21.15.151:6818/3234791793,v1:172.21.15.151:6819/3234791793]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T02:42:39.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:38 smithi151 ceph-mon[112233]: osdmap e63: 6 total, 5 up, 6 in 2024-04-16T02:42:39.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:38 smithi151 ceph-mon[112233]: from='osd.2 [v2:172.21.15.151:6818/3234791793,v1:172.21.15.151:6819/3234791793]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:42:39.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:39.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:42:39.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:38 smithi153 ceph-mon[87858]: from='osd.2 [v2:172.21.15.151:6818/3234791793,v1:172.21.15.151:6819/3234791793]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T02:42:39.307 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:38 smithi153 ceph-mon[87858]: osdmap e63: 6 total, 5 up, 6 in 2024-04-16T02:42:39.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:38 smithi153 ceph-mon[87858]: from='osd.2 [v2:172.21.15.151:6818/3234791793,v1:172.21.15.151:6819/3234791793]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-04-16T02:42:39.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:39.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:42:40.217 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:39 smithi151 ceph-mon[112233]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:42:40.217 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:39 smithi151 ceph-mon[112233]: osd.2 [v2:172.21.15.151:6818/3234791793,v1:172.21.15.151:6819/3234791793] boot 2024-04-16T02:42:40.218 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:39 smithi151 ceph-mon[112233]: osdmap e64: 6 total, 6 up, 6 in 2024-04-16T02:42:40.218 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:42:40.218 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:40.218 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:40.218 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:39 smithi151 ceph-mon[112233]: pgmap v100: 65 pgs: 10 peering, 7 active+undersized+degraded, 10 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 26/336 objects degraded (7.738%) 2024-04-16T02:42:40.218 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:39 smithi151 ceph-mon[112233]: osdmap e65: 6 total, 6 up, 6 in 2024-04-16T02:42:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:39 smithi153 ceph-mon[87858]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:42:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:39 smithi153 ceph-mon[87858]: osd.2 [v2:172.21.15.151:6818/3234791793,v1:172.21.15.151:6819/3234791793] boot 2024-04-16T02:42:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:39 smithi153 ceph-mon[87858]: osdmap e64: 6 total, 6 up, 6 in 2024-04-16T02:42:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T02:42:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:39 smithi153 ceph-mon[87858]: pgmap v100: 65 pgs: 10 peering, 7 active+undersized+degraded, 10 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 26/336 objects degraded (7.738%) 2024-04-16T02:42:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:39 smithi153 ceph-mon[87858]: osdmap e65: 6 total, 6 up, 6 in 2024-04-16T02:42:41.136 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:40 smithi151 ceph-mon[112233]: Health check update: Degraded data redundancy: 26/336 objects degraded (7.738%), 7 pgs degraded (PG_DEGRADED) 2024-04-16T02:42:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:40 smithi153 ceph-mon[87858]: Health check update: Degraded data redundancy: 26/336 objects degraded (7.738%), 7 pgs degraded (PG_DEGRADED) 2024-04-16T02:42:42.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:41 smithi151 ceph-mon[112233]: pgmap v102: 65 pgs: 10 peering, 7 active+undersized+degraded, 10 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 26/336 objects degraded (7.738%) 2024-04-16T02:42:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:41 smithi153 ceph-mon[87858]: pgmap v102: 65 pgs: 10 peering, 7 active+undersized+degraded, 10 active+undersized, 38 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 26/336 objects degraded (7.738%) 2024-04-16T02:42:43.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T02:42:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:43 smithi151 ceph-mon[112233]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-16T02:42:44.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:44.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:44.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:42:44.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:42:44.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:44.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:42:44.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:44.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:44.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:44.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T02:42:44.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T02:42:44.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:43 smithi153 ceph-mon[87858]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-16T02:42:44.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:44 smithi151 ceph-mon[112233]: pgmap v103: 65 pgs: 10 peering, 55 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-16T02:42:45.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:44 smithi153 ceph-mon[87858]: pgmap v103: 65 pgs: 10 peering, 55 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-16T02:42:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:45 smithi151 ceph-mon[112233]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/336 objects degraded (7.738%), 7 pgs degraded) 2024-04-16T02:42:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:45 smithi151 ceph-mon[112233]: Cluster is now healthy 2024-04-16T02:42:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:45 smithi153 ceph-mon[87858]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/336 objects degraded (7.738%), 7 pgs degraded) 2024-04-16T02:42:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:45 smithi153 ceph-mon[87858]: Cluster is now healthy 2024-04-16T02:42:46.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:46 smithi153 ceph-mon[87858]: pgmap v104: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-16T02:42:46.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:46 smithi151 ceph-mon[112233]: pgmap v104: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-16T02:42:48.129 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:47 smithi153 ceph-mon[87858]: pgmap v105: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-16T02:42:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:47 smithi151 ceph-mon[112233]: pgmap v105: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-16T02:42:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:49 smithi151 ceph-mon[112233]: pgmap v106: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:42:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:49 smithi153 ceph-mon[87858]: pgmap v106: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:42:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:51 smithi151 ceph-mon[112233]: pgmap v107: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-16T02:42:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:51 smithi153 ceph-mon[87858]: pgmap v107: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-16T02:42:54.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.545+0000 7f632d533700 1 -- 172.21.15.151:0/2867492625 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 msgr2=0x7f6328101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:54.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.545+0000 7f632d533700 1 --2- 172.21.15.151:0/2867492625 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 0x7f6328101180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6318009910 tx=0x7f6318009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:54.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.545+0000 7f632d533700 1 -- 172.21.15.151:0/2867492625 shutdown_connections 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.545+0000 7f632d533700 1 --2- 172.21.15.151:0/2867492625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6328101750 0x7f63281057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.545+0000 7f632d533700 1 --2- 172.21.15.151:0/2867492625 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 0x7f6328101180 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.545+0000 7f632d533700 1 -- 172.21.15.151:0/2867492625 >> 172.21.15.151:0/2867492625 conn(0x7f63280fc650 msgr2=0x7f63280fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.545+0000 7f632d533700 1 -- 172.21.15.151:0/2867492625 shutdown_connections 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.545+0000 7f632d533700 1 -- 172.21.15.151:0/2867492625 wait complete. 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.546+0000 7f632d533700 1 Processor -- start 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.546+0000 7f632d533700 1 -- start start 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.546+0000 7f632d533700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 0x7f6328198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.546+0000 7f632d533700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6328101750 0x7f6328198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.546+0000 7f632d533700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328199520 con 0x7f6328100da0 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.546+0000 7f632d533700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632819d330 con 0x7f6328101750 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.546+0000 7f6326ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 0x7f6328198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f6326ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 0x7f6328198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57694/0 (socket says 172.21.15.151:57694) 2024-04-16T02:42:54.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f6326ffd700 1 -- 172.21.15.151:0/1122527529 learned_addr learned my addr 172.21.15.151:0/1122527529 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f63267fc700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6328101750 0x7f6328198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f6326ffd700 1 -- 172.21.15.151:0/1122527529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6328101750 msgr2=0x7f6328198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f6326ffd700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6328101750 0x7f6328198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f6326ffd700 1 -- 172.21.15.151:0/1122527529 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63180095f0 con 0x7f6328100da0 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f63267fc700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6328101750 0x7f6328198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f6326ffd700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 0x7f6328198950 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6318005f50 tx=0x7f631800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f630ffff700 1 -- 172.21.15.151:0/1122527529 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f631801d070 con 0x7f6328100da0 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f630ffff700 1 -- 172.21.15.151:0/1122527529 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f631800b610 con 0x7f6328100da0 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632819d5b0 con 0x7f6328100da0 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f630ffff700 1 -- 172.21.15.151:0/1122527529 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6318022470 con 0x7f6328100da0 2024-04-16T02:42:54.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.547+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632819daa0 con 0x7f6328100da0 2024-04-16T02:42:54.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.548+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6328104f20 con 0x7f6328100da0 2024-04-16T02:42:54.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.549+0000 7f630ffff700 1 -- 172.21.15.151:0/1122527529 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f631800fca0 con 0x7f6328100da0 2024-04-16T02:42:54.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.549+0000 7f630ffff700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f631006ecc0 0x7f6310071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:54.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.549+0000 7f63267fc700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f631006ecc0 0x7f6310071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:54.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.549+0000 7f630ffff700 1 -- 172.21.15.151:0/1122527529 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6318090220 con 0x7f6328100da0 2024-04-16T02:42:54.554 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.550+0000 7f63267fc700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f631006ecc0 0x7f6310071180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6328199f70 tx=0x7f631c009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:54.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.554+0000 7f630ffff700 1 -- 172.21.15.151:0/1122527529 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f631805cbb0 con 0x7f6328100da0 2024-04-16T02:42:54.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.772+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f632819dcc0 con 0x7f631006ecc0 2024-04-16T02:42:54.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.774+0000 7f630ffff700 1 -- 172.21.15.151:0/1122527529 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f632819dcc0 con 0x7f631006ecc0 2024-04-16T02:42:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.776+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f631006ecc0 msgr2=0x7f6310071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.776+0000 7f632d533700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f631006ecc0 0x7f6310071180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6328199f70 tx=0x7f631c009260 comp rx=0 tx=0).stop 2024-04-16T02:42:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.776+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 msgr2=0x7f6328198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.776+0000 7f632d533700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 0x7f6328198950 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6318005f50 tx=0x7f631800f740 comp rx=0 tx=0).stop 2024-04-16T02:42:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.776+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 shutdown_connections 2024-04-16T02:42:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.776+0000 7f632d533700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f631006ecc0 0x7f6310071180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.776+0000 7f632d533700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6328100da0 0x7f6328198950 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.776+0000 7f632d533700 1 --2- 172.21.15.151:0/1122527529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6328101750 0x7f6328198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.776+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 >> 172.21.15.151:0/1122527529 conn(0x7f63280fc650 msgr2=0x7f63280fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.777+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 shutdown_connections 2024-04-16T02:42:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.777+0000 7f632d533700 1 -- 172.21.15.151:0/1122527529 wait complete. 2024-04-16T02:42:54.791 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:42:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.884+0000 7f17dff18700 1 -- 172.21.15.151:0/693550038 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ffdb0 msgr2=0x7f17d8103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.884+0000 7f17dff18700 1 --2- 172.21.15.151:0/693550038 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ffdb0 0x7f17d8103e00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009910 tx=0x7f17d4009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.884+0000 7f17dff18700 1 -- 172.21.15.151:0/693550038 shutdown_connections 2024-04-16T02:42:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.884+0000 7f17dff18700 1 --2- 172.21.15.151:0/693550038 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ffdb0 0x7f17d8103e00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.884+0000 7f17dff18700 1 --2- 172.21.15.151:0/693550038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d80ff400 0x7f17d80ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.884+0000 7f17dff18700 1 -- 172.21.15.151:0/693550038 >> 172.21.15.151:0/693550038 conn(0x7f17d8075e30 msgr2=0x7f17d80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:54.888 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:54 smithi151 ceph-mon[112233]: pgmap v108: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-16T02:42:54.889 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:54.889 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:42:54.889 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.884+0000 7f17dff18700 1 -- 172.21.15.151:0/693550038 shutdown_connections 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.884+0000 7f17dff18700 1 -- 172.21.15.151:0/693550038 wait complete. 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.885+0000 7f17dff18700 1 Processor -- start 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.885+0000 7f17dff18700 1 -- start start 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.885+0000 7f17dff18700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ff400 0x7f17d819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.885+0000 7f17dff18700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d80ffdb0 0x7f17d819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.885+0000 7f17dff18700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d819d0c0 con 0x7f17d80ff400 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.885+0000 7f17dff18700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d8196570 con 0x7f17d80ffdb0 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.885+0000 7f17ddcb4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ff400 0x7f17d819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.885+0000 7f17ddcb4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ff400 0x7f17d819c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57708/0 (socket says 172.21.15.151:57708) 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.885+0000 7f17ddcb4700 1 -- 172.21.15.151:0/2778035669 learned_addr learned my addr 172.21.15.151:0/2778035669 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17dd4b3700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d80ffdb0 0x7f17d819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17ddcb4700 1 -- 172.21.15.151:0/2778035669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d80ffdb0 msgr2=0x7f17d819ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17ddcb4700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d80ffdb0 0x7f17d819ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17ddcb4700 1 -- 172.21.15.151:0/2778035669 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d40095f0 con 0x7f17d80ff400 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17ddcb4700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ff400 0x7f17d819c4f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17c800dd70 tx=0x7f17c800a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17ceffd700 1 -- 172.21.15.151:0/2778035669 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17c800ad50 con 0x7f17d80ff400 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17ceffd700 1 -- 172.21.15.151:0/2778035669 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17c8004cf0 con 0x7f17d80ff400 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17ceffd700 1 -- 172.21.15.151:0/2778035669 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17c800e680 con 0x7f17d80ff400 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d8196850 con 0x7f17d80ff400 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.886+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d8196da0 con 0x7f17d80ff400 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.887+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d8109720 con 0x7f17d80ff400 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.888+0000 7f17ceffd700 1 -- 172.21.15.151:0/2778035669 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f17c8004750 con 0x7f17d80ff400 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.888+0000 7f17ceffd700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17c406ecc0 0x7f17c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.888+0000 7f17ceffd700 1 -- 172.21.15.151:0/2778035669 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f17c808ed00 con 0x7f17d80ff400 2024-04-16T02:42:54.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.888+0000 7f17dd4b3700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17c406ecc0 0x7f17c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:54.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.889+0000 7f17dd4b3700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17c406ecc0 0x7f17c4071180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17d8103110 tx=0x7f17d401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:54.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:54.893+0000 7f17ceffd700 1 -- 172.21.15.151:0/2778035669 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f17c8092030 con 0x7f17d80ff400 2024-04-16T02:42:55.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.109+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f17d81970d0 con 0x7f17c406ecc0 2024-04-16T02:42:55.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.111+0000 7f17ceffd700 1 -- 172.21.15.151:0/2778035669 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f17d81970d0 con 0x7f17c406ecc0 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17c406ecc0 msgr2=0x7f17c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17c406ecc0 0x7f17c4071180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17d8103110 tx=0x7f17d401a040 comp rx=0 tx=0).stop 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ff400 msgr2=0x7f17d819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ff400 0x7f17d819c4f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17c800dd70 tx=0x7f17c800a530 comp rx=0 tx=0).stop 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 shutdown_connections 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17c406ecc0 0x7f17c4071180 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17d80ff400 0x7f17d819c4f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 --2- 172.21.15.151:0/2778035669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d80ffdb0 0x7f17d819ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 >> 172.21.15.151:0/2778035669 conn(0x7f17d8075e30 msgr2=0x7f17d80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 shutdown_connections 2024-04-16T02:42:55.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.113+0000 7f17dff18700 1 -- 172.21.15.151:0/2778035669 wait complete. 2024-04-16T02:42:55.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.218+0000 7fb3fead3700 1 -- 172.21.15.151:0/3050258646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 msgr2=0x7fb3f8101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.218+0000 7fb3fead3700 1 --2- 172.21.15.151:0/3050258646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 0x7fb3f8101180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009910 tx=0x7fb3ec009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:55.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.218+0000 7fb3fead3700 1 -- 172.21.15.151:0/3050258646 shutdown_connections 2024-04-16T02:42:55.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.218+0000 7fb3fead3700 1 --2- 172.21.15.151:0/3050258646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8101750 0x7fb3f81057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.218+0000 7fb3fead3700 1 --2- 172.21.15.151:0/3050258646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 0x7fb3f8101180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.218+0000 7fb3fead3700 1 -- 172.21.15.151:0/3050258646 >> 172.21.15.151:0/3050258646 conn(0x7fb3f80fc630 msgr2=0x7fb3f80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:55.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.218+0000 7fb3fead3700 1 -- 172.21.15.151:0/3050258646 shutdown_connections 2024-04-16T02:42:55.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.218+0000 7fb3fead3700 1 -- 172.21.15.151:0/3050258646 wait complete. 2024-04-16T02:42:55.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fead3700 1 Processor -- start 2024-04-16T02:42:55.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fead3700 1 -- start start 2024-04-16T02:42:55.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fead3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 0x7fb3f8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:55.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fead3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8101750 0x7fb3f80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:55.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fead3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8073b10 con 0x7fb3f8100da0 2024-04-16T02:42:55.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fead3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8073c50 con 0x7fb3f8101750 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fc86f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 0x7fb3f8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fc86f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 0x7fb3f8073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57726/0 (socket says 172.21.15.151:57726) 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fc86f700 1 -- 172.21.15.151:0/4175080205 learned_addr learned my addr 172.21.15.151:0/4175080205 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3f7fff700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8101750 0x7fb3f80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fc86f700 1 -- 172.21.15.151:0/4175080205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8101750 msgr2=0x7fb3f80735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fc86f700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8101750 0x7fb3f80735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.219+0000 7fb3fc86f700 1 -- 172.21.15.151:0/4175080205 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec0095f0 con 0x7fb3f8100da0 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.220+0000 7fb3fc86f700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 0x7fb3f8073090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00f690 tx=0x7fb3ec00f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.220+0000 7fb3f5ffb700 1 -- 172.21.15.151:0/4175080205 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3ec01d070 con 0x7fb3f8100da0 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.220+0000 7fb3f5ffb700 1 -- 172.21.15.151:0/4175080205 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3ec0050d0 con 0x7fb3f8100da0 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.220+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f8073ed0 con 0x7fb3f8100da0 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.220+0000 7fb3f5ffb700 1 -- 172.21.15.151:0/4175080205 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3ec022470 con 0x7fb3f8100da0 2024-04-16T02:42:55.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.220+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f81a2cd0 con 0x7fb3f8100da0 2024-04-16T02:42:55.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.221+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f8109140 con 0x7fb3f8100da0 2024-04-16T02:42:55.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.222+0000 7fb3f5ffb700 1 -- 172.21.15.151:0/4175080205 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb3ec003780 con 0x7fb3f8100da0 2024-04-16T02:42:55.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.222+0000 7fb3f5ffb700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb3e806ecc0 0x7fb3e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:55.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.223+0000 7fb3f5ffb700 1 -- 172.21.15.151:0/4175080205 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb3ec090190 con 0x7fb3f8100da0 2024-04-16T02:42:55.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.223+0000 7fb3f7fff700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb3e806ecc0 0x7fb3e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:55.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.223+0000 7fb3f7fff700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb3e806ecc0 0x7fb3e8071180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8074880 tx=0x7fb3e4009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:55.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.227+0000 7fb3f5ffb700 1 -- 172.21.15.151:0/4175080205 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb3ec059120 con 0x7fb3f8100da0 2024-04-16T02:42:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:54 smithi153 ceph-mon[87858]: pgmap v108: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-16T02:42:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:42:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:42:55.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.460+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3f80611d0 con 0x7fb3e806ecc0 2024-04-16T02:42:55.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.476+0000 7fb3f5ffb700 1 -- 172.21.15.151:0/4175080205 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fb3f80611d0 con 0x7fb3e806ecc0 2024-04-16T02:42:55.479 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:42:55.479 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (4m) 17s ago 15m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:42:55.479 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (16m) 17s ago 16m 8740k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (15m) 112s ago 15m 8132k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (2m) 17s ago 16m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (116s) 112s ago 14m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (4m) 17s ago 15m 81.1M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (10m) 17s ago 10m 120M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (10m) 17s ago 10m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (10m) 112s ago 10m 58.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (10m) 112s ago 10m 67.4M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (6m) 17s ago 17m 505M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (5m) 112s ago 14m 450M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (3m) 17s ago 17m 48.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (2m) 112s ago 14m 41.2M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:42:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (5m) 17s ago 15m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:42:55.481 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (5m) 112s ago 14m 15.9M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:42:55.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (95s) 17s ago 13m 88.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:42:55.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (59s) 17s ago 13m 96.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:42:55.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (23s) 17s ago 13m 12.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:42:55.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (12m) 112s ago 12m 282M 2240M 18.2.1 5be31c24972a 52e427295fb9 2024-04-16T02:42:55.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (12m) 112s ago 12m 231M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:42:55.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (11m) 112s ago 11m 245M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:42:55.481 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (4m) 17s ago 15m 73.7M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:42:55.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb3e806ecc0 msgr2=0x7fb3e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb3e806ecc0 0x7fb3e8071180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8074880 tx=0x7fb3e4009260 comp rx=0 tx=0).stop 2024-04-16T02:42:55.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 msgr2=0x7fb3f8073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 0x7fb3f8073090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00f690 tx=0x7fb3ec00f6c0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 shutdown_connections 2024-04-16T02:42:55.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb3e806ecc0 0x7fb3e8071180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3f8100da0 0x7fb3f8073090 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 --2- 172.21.15.151:0/4175080205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8101750 0x7fb3f80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 >> 172.21.15.151:0/4175080205 conn(0x7fb3f80fc630 msgr2=0x7fb3f80fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:55.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 shutdown_connections 2024-04-16T02:42:55.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.479+0000 7fb3fead3700 1 -- 172.21.15.151:0/4175080205 wait complete. 2024-04-16T02:42:55.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.583+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/3389691933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 msgr2=0x7fa4ec10ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.583+0000 7fa4f3a6f700 1 --2- 172.21.15.151:0/3389691933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 0x7fa4ec10ab60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009910 tx=0x7fa4e0009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:55.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.583+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/3389691933 shutdown_connections 2024-04-16T02:42:55.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.583+0000 7fa4f3a6f700 1 --2- 172.21.15.151:0/3389691933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 0x7fa4ec10ab60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.583+0000 7fa4f3a6f700 1 --2- 172.21.15.151:0/3389691933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec101d50 0x7fa4ec102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.583+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/3389691933 >> 172.21.15.151:0/3389691933 conn(0x7fa4ec0fb540 msgr2=0x7fa4ec0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:55.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.583+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/3389691933 shutdown_connections 2024-04-16T02:42:55.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.583+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/3389691933 wait complete. 2024-04-16T02:42:55.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f3a6f700 1 Processor -- start 2024-04-16T02:42:55.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f3a6f700 1 -- start start 2024-04-16T02:42:55.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f3a6f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec101d50 0x7fa4ec0ff9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:55.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f3a6f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 0x7fa4ec0ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:55.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f3a6f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec100430 con 0x7fa4ec102670 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f3a6f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec100570 con 0x7fa4ec101d50 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f100a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 0x7fa4ec0ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f100a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 0x7fa4ec0ffef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57738/0 (socket says 172.21.15.151:57738) 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f100a700 1 -- 172.21.15.151:0/282452953 learned_addr learned my addr 172.21.15.151:0/282452953 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f180b700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec101d50 0x7fa4ec0ff9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f100a700 1 -- 172.21.15.151:0/282452953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec101d50 msgr2=0x7fa4ec0ff9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f100a700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec101d50 0x7fa4ec0ff9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.584+0000 7fa4f100a700 1 -- 172.21.15.151:0/282452953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e00095f0 con 0x7fa4ec102670 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.585+0000 7fa4f100a700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 0x7fa4ec0ffef0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00092e0 tx=0x7fa4e000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.585+0000 7fa4eaffd700 1 -- 172.21.15.151:0/282452953 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4e001d070 con 0x7fa4ec102670 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.585+0000 7fa4eaffd700 1 -- 172.21.15.151:0/282452953 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4e000f460 con 0x7fa4ec102670 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.585+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ec1007f0 con 0x7fa4ec102670 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.585+0000 7fa4eaffd700 1 -- 172.21.15.151:0/282452953 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4e00175d0 con 0x7fa4ec102670 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.585+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec1a2c40 con 0x7fa4ec102670 2024-04-16T02:42:55.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.586+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ec1081f0 con 0x7fa4ec102670 2024-04-16T02:42:55.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.586+0000 7fa4eaffd700 1 -- 172.21.15.151:0/282452953 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa4e000f5d0 con 0x7fa4ec102670 2024-04-16T02:42:55.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.587+0000 7fa4eaffd700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4d406ecc0 0x7fa4d4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:55.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.587+0000 7fa4eaffd700 1 -- 172.21.15.151:0/282452953 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa4e0090130 con 0x7fa4ec102670 2024-04-16T02:42:55.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.587+0000 7fa4f180b700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4d406ecc0 0x7fa4d4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:55.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.587+0000 7fa4f180b700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4d406ecc0 0x7fa4d4071180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40060b0 tx=0x7fa4e400c040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:55.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.592+0000 7fa4eaffd700 1 -- 172.21.15.151:0/282452953 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa4e0059010 con 0x7fa4ec102670 2024-04-16T02:42:55.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.879+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa4ec04ea90 con 0x7fa4ec102670 2024-04-16T02:42:55.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.880+0000 7fa4eaffd700 1 -- 172.21.15.151:0/282452953 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fa4e005c6d0 con 0x7fa4ec102670 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-04-16T02:42:55.883 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:42:55.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:42:55.884 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:42:55.884 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:42:55.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:42:55.884 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:42:55.884 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 7, 2024-04-16T02:42:55.884 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 7 2024-04-16T02:42:55.884 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:42:55.884 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4d406ecc0 msgr2=0x7fa4d4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4d406ecc0 0x7fa4d4071180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40060b0 tx=0x7fa4e400c040 comp rx=0 tx=0).stop 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 msgr2=0x7fa4ec0ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 0x7fa4ec0ffef0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00092e0 tx=0x7fa4e000bd70 comp rx=0 tx=0).stop 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 shutdown_connections 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4d406ecc0 0x7fa4d4071180 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec101d50 0x7fa4ec0ff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 --2- 172.21.15.151:0/282452953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4ec102670 0x7fa4ec0ffef0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 >> 172.21.15.151:0/282452953 conn(0x7fa4ec0fb540 msgr2=0x7fa4ec105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 shutdown_connections 2024-04-16T02:42:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.882+0000 7fa4f3a6f700 1 -- 172.21.15.151:0/282452953 wait complete. 2024-04-16T02:42:55.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.988+0000 7f8ab2815700 1 -- 172.21.15.151:0/920325981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 msgr2=0x7f8aac0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.988+0000 7f8ab2815700 1 --2- 172.21.15.151:0/920325981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 0x7f8aac0ffec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009910 tx=0x7f8a94009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:55.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.988+0000 7f8ab2815700 1 -- 172.21.15.151:0/920325981 shutdown_connections 2024-04-16T02:42:55.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.988+0000 7f8ab2815700 1 --2- 172.21.15.151:0/920325981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 0x7f8aac0ffec0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.988+0000 7f8ab2815700 1 --2- 172.21.15.151:0/920325981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aac105a40 0x7f8aac105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.988+0000 7f8ab2815700 1 -- 172.21.15.151:0/920325981 >> 172.21.15.151:0/920325981 conn(0x7f8aac0fb530 msgr2=0x7f8aac0fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:55.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.988+0000 7f8ab2815700 1 -- 172.21.15.151:0/920325981 shutdown_connections 2024-04-16T02:42:55.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.988+0000 7f8ab2815700 1 -- 172.21.15.151:0/920325981 wait complete. 2024-04-16T02:42:55.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.988+0000 7f8ab2815700 1 Processor -- start 2024-04-16T02:42:55.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8ab2815700 1 -- start start 2024-04-16T02:42:55.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8ab2815700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 0x7f8aac19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:55.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8ab2815700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aac105a40 0x7f8aac19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:55.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8ab2815700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac19f2d0 con 0x7f8aac0ffa40 2024-04-16T02:42:55.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8ab2815700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac198780 con 0x7f8aac105a40 2024-04-16T02:42:55.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8aabfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 0x7f8aac19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:55.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8aabfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 0x7f8aac19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57756/0 (socket says 172.21.15.151:57756) 2024-04-16T02:42:55.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8aabfff700 1 -- 172.21.15.151:0/1759842492 learned_addr learned my addr 172.21.15.151:0/1759842492 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:55.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8aab7fe700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aac105a40 0x7f8aac19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8aabfff700 1 -- 172.21.15.151:0/1759842492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aac105a40 msgr2=0x7f8aac19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8aabfff700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aac105a40 0x7f8aac19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8aabfff700 1 -- 172.21.15.151:0/1759842492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a940095f0 con 0x7f8aac0ffa40 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8aab7fe700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aac105a40 0x7f8aac19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.989+0000 7f8aabfff700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 0x7f8aac19e700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00e920 tx=0x7f8a9c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.990+0000 7f8aa97fa700 1 -- 172.21.15.151:0/1759842492 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a9c009790 con 0x7f8aac0ffa40 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.990+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac198a60 con 0x7f8aac0ffa40 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.990+0000 7f8aa97fa700 1 -- 172.21.15.151:0/1759842492 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a9c00cba0 con 0x7f8aac0ffa40 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.990+0000 7f8aa97fa700 1 -- 172.21.15.151:0/1759842492 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a9c018650 con 0x7f8aac0ffa40 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.990+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac198fb0 con 0x7f8aac0ffa40 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.991+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac04ea90 con 0x7f8aac0ffa40 2024-04-16T02:42:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.991+0000 7f8aa97fa700 1 -- 172.21.15.151:0/1759842492 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8a9c00cd10 con 0x7f8aac0ffa40 2024-04-16T02:42:55.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.992+0000 7f8aa97fa700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a9806ecc0 0x7f8a98071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:55.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.992+0000 7f8aab7fe700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a9806ecc0 0x7f8a98071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:55.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.992+0000 7f8aa97fa700 1 -- 172.21.15.151:0/1759842492 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8a9c014070 con 0x7f8aac0ffa40 2024-04-16T02:42:55.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.992+0000 7f8aab7fe700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a9806ecc0 0x7f8a98071180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a94000c00 tx=0x7f8a94009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:55.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:55.997+0000 7f8aa97fa700 1 -- 172.21.15.151:0/1759842492 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8a9c053ff0 con 0x7f8aac0ffa40 2024-04-16T02:42:56.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.242+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8aac1993d0 con 0x7f8aac0ffa40 2024-04-16T02:42:56.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:55 smithi151 ceph-mon[112233]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:56.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.242+0000 7f8aa97fa700 1 -- 172.21.15.151:0/1759842492 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1919 (secure 0 0 0) 0x7f8a9c05b8b0 con 0x7f8aac0ffa40 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:e22 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 22 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:40:05.114230+0000 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:42:56.246 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:42:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 14478 members: 14490,14478 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:42:56.248 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:42:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a9806ecc0 msgr2=0x7f8a98071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a9806ecc0 0x7f8a98071180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a94000c00 tx=0x7f8a94009f70 comp rx=0 tx=0).stop 2024-04-16T02:42:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 msgr2=0x7f8aac19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 0x7f8aac19e700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00e920 tx=0x7f8a9c00ece0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 shutdown_connections 2024-04-16T02:42:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a9806ecc0 0x7f8a98071180 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8aac0ffa40 0x7f8aac19e700 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 --2- 172.21.15.151:0/1759842492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aac105a40 0x7f8aac19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 >> 172.21.15.151:0/1759842492 conn(0x7f8aac0fb530 msgr2=0x7f8aac068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:56.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 shutdown_connections 2024-04-16T02:42:56.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.245+0000 7f8ab2815700 1 -- 172.21.15.151:0/1759842492 wait complete. 2024-04-16T02:42:56.250 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 22 2024-04-16T02:42:56.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:55 smithi153 ceph-mon[87858]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.351+0000 7f58e6157700 1 -- 172.21.15.151:0/920784431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0100dc0 msgr2=0x7f58e01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.351+0000 7f58e6157700 1 --2- 172.21.15.151:0/920784431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0100dc0 0x7f58e01011a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009910 tx=0x7f58d0009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.352+0000 7f58e6157700 1 -- 172.21.15.151:0/920784431 shutdown_connections 2024-04-16T02:42:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.352+0000 7f58e6157700 1 --2- 172.21.15.151:0/920784431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58e0101770 0x7f58e01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.352+0000 7f58e6157700 1 --2- 172.21.15.151:0/920784431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0100dc0 0x7f58e01011a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.352+0000 7f58e6157700 1 -- 172.21.15.151:0/920784431 >> 172.21.15.151:0/920784431 conn(0x7f58e00fc650 msgr2=0x7f58e00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.352+0000 7f58e6157700 1 -- 172.21.15.151:0/920784431 shutdown_connections 2024-04-16T02:42:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.352+0000 7f58e6157700 1 -- 172.21.15.151:0/920784431 wait complete. 2024-04-16T02:42:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.352+0000 7f58e6157700 1 Processor -- start 2024-04-16T02:42:56.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.352+0000 7f58e6157700 1 -- start start 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58e6157700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58e0100dc0 0x7f58e019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58e6157700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0101770 0x7f58e019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58e6157700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e019d210 con 0x7f58e0101770 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58e6157700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e01966c0 con 0x7f58e0100dc0 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58deffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0101770 0x7f58e019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58deffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0101770 0x7f58e019cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57780/0 (socket says 172.21.15.151:57780) 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58deffd700 1 -- 172.21.15.151:0/2318103021 learned_addr learned my addr 172.21.15.151:0/2318103021 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58df7fe700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58e0100dc0 0x7f58e019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58deffd700 1 -- 172.21.15.151:0/2318103021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58e0100dc0 msgr2=0x7f58e019c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58deffd700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58e0100dc0 0x7f58e019c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58deffd700 1 -- 172.21.15.151:0/2318103021 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d00095f0 con 0x7f58e0101770 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58deffd700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0101770 0x7f58e019cb80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f58d400b480 tx=0x7f58d400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58dcff9700 1 -- 172.21.15.151:0/2318103021 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58d40118c0 con 0x7f58e0101770 2024-04-16T02:42:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58dcff9700 1 -- 172.21.15.151:0/2318103021 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58d4010380 con 0x7f58e0101770 2024-04-16T02:42:56.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e01969a0 con 0x7f58e0101770 2024-04-16T02:42:56.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58dcff9700 1 -- 172.21.15.151:0/2318103021 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58d400f4f0 con 0x7f58e0101770 2024-04-16T02:42:56.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.353+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e0196ef0 con 0x7f58e0101770 2024-04-16T02:42:56.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.354+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e0109160 con 0x7f58e0101770 2024-04-16T02:42:56.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.355+0000 7f58dcff9700 1 -- 172.21.15.151:0/2318103021 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f58d40104f0 con 0x7f58e0101770 2024-04-16T02:42:56.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.355+0000 7f58dcff9700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58c806ecc0 0x7f58c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:56.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.355+0000 7f58dcff9700 1 -- 172.21.15.151:0/2318103021 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f58d408e920 con 0x7f58e0101770 2024-04-16T02:42:56.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.356+0000 7f58df7fe700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58c806ecc0 0x7f58c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:56.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.356+0000 7f58df7fe700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58c806ecc0 0x7f58c8071180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58d000b5c0 tx=0x7f58d001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:56.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.360+0000 7f58dcff9700 1 -- 172.21.15.151:0/2318103021 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f58d405b330 con 0x7f58e0101770 2024-04-16T02:42:56.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.577+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f58e0197230 con 0x7f58c806ecc0 2024-04-16T02:42:56.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.578+0000 7f58dcff9700 1 -- 172.21.15.151:0/2318103021 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f58e0197230 con 0x7f58c806ecc0 2024-04-16T02:42:56.581 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:42:56.581 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:42:56.581 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:42:56.581 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:42:56.582 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:42:56.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:42:56.582 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:42:56.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:42:56.582 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:42:56.582 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "9/23 daemons upgraded", 2024-04-16T02:42:56.582 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T02:42:56.582 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:42:56.582 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.580+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58c806ecc0 msgr2=0x7f58c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.580+0000 7f58e6157700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58c806ecc0 0x7f58c8071180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58d000b5c0 tx=0x7f58d001a040 comp rx=0 tx=0).stop 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.580+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0101770 msgr2=0x7f58e019cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.580+0000 7f58e6157700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0101770 0x7f58e019cb80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f58d400b480 tx=0x7f58d400b840 comp rx=0 tx=0).stop 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.580+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 shutdown_connections 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.580+0000 7f58e6157700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58c806ecc0 0x7f58c8071180 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.580+0000 7f58e6157700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58e0100dc0 0x7f58e019c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.580+0000 7f58e6157700 1 --2- 172.21.15.151:0/2318103021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58e0101770 0x7f58e019cb80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.580+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 >> 172.21.15.151:0/2318103021 conn(0x7f58e00fc650 msgr2=0x7f58e00fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.581+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 shutdown_connections 2024-04-16T02:42:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.581+0000 7f58e6157700 1 -- 172.21.15.151:0/2318103021 wait complete. 2024-04-16T02:42:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.685+0000 7f56d7f9e700 1 -- 172.21.15.151:0/960955205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d00ffa60 msgr2=0x7f56d00ffe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.685+0000 7f56d7f9e700 1 --2- 172.21.15.151:0/960955205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d00ffa60 0x7f56d00ffe40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009910 tx=0x7f56cc009c20 comp rx=0 tx=0).stop 2024-04-16T02:42:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.685+0000 7f56d7f9e700 1 -- 172.21.15.151:0/960955205 shutdown_connections 2024-04-16T02:42:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.685+0000 7f56d7f9e700 1 --2- 172.21.15.151:0/960955205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56d0100410 0x7f56d01048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.685+0000 7f56d7f9e700 1 --2- 172.21.15.151:0/960955205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d00ffa60 0x7f56d00ffe40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.685+0000 7f56d7f9e700 1 -- 172.21.15.151:0/960955205 >> 172.21.15.151:0/960955205 conn(0x7f56d00fb550 msgr2=0x7f56d00fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.685+0000 7f56d7f9e700 1 -- 172.21.15.151:0/960955205 shutdown_connections 2024-04-16T02:42:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.685+0000 7f56d7f9e700 1 -- 172.21.15.151:0/960955205 wait complete. 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.685+0000 7f56d7f9e700 1 Processor -- start 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d7f9e700 1 -- start start 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d7f9e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56d00ffa60 0x7f56d019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d7f9e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d0100410 0x7f56d019d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d7f9e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d019d960 con 0x7f56d0100410 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d7f9e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d01a16f0 con 0x7f56d00ffa60 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d5539700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d0100410 0x7f56d019d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d5539700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d0100410 0x7f56d019d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38006/0 (socket says 172.21.15.151:38006) 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d5539700 1 -- 172.21.15.151:0/3148533195 learned_addr learned my addr 172.21.15.151:0/3148533195 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:42:56.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d5d3a700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56d00ffa60 0x7f56d019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:56.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d5539700 1 -- 172.21.15.151:0/3148533195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56d00ffa60 msgr2=0x7f56d019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d5539700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56d00ffa60 0x7f56d019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d5539700 1 -- 172.21.15.151:0/3148533195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc0095f0 con 0x7f56d0100410 2024-04-16T02:42:56.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.686+0000 7f56d5539700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d0100410 0x7f56d019d280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f56c000d900 tx=0x7f56c000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:56.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.687+0000 7f56c6ffd700 1 -- 172.21.15.151:0/3148533195 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56c0009900 con 0x7f56d0100410 2024-04-16T02:42:56.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.687+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d01a19d0 con 0x7f56d0100410 2024-04-16T02:42:56.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.687+0000 7f56c6ffd700 1 -- 172.21.15.151:0/3148533195 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56c0010460 con 0x7f56d0100410 2024-04-16T02:42:56.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.687+0000 7f56c6ffd700 1 -- 172.21.15.151:0/3148533195 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56c000f5f0 con 0x7f56d0100410 2024-04-16T02:42:56.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.687+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d01a1f20 con 0x7f56d0100410 2024-04-16T02:42:56.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.688+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d0102040 con 0x7f56d0100410 2024-04-16T02:42:56.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.688+0000 7f56c6ffd700 1 -- 172.21.15.151:0/3148533195 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f56c00105d0 con 0x7f56d0100410 2024-04-16T02:42:56.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.689+0000 7f56c6ffd700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56bc06ecc0 0x7f56bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:42:56.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.689+0000 7f56d5d3a700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56bc06ecc0 0x7f56bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:42:56.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.689+0000 7f56c6ffd700 1 -- 172.21.15.151:0/3148533195 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f56c008e980 con 0x7f56d0100410 2024-04-16T02:42:56.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.689+0000 7f56d5d3a700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56bc06ecc0 0x7f56bc071180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f56cc006010 tx=0x7f56cc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:42:56.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.694+0000 7f56c6ffd700 1 -- 172.21.15.151:0/3148533195 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f56c00577e0 con 0x7f56d0100410 2024-04-16T02:42:56.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.974+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f56d004ea90 con 0x7f56d0100410 2024-04-16T02:42:56.977 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:56 smithi151 ceph-mon[112233]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:56.977 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:56 smithi151 ceph-mon[112233]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:56.977 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:56 smithi151 ceph-mon[112233]: pgmap v109: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:42:56.978 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:56 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/282452953' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:56.978 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:56 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1759842492' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:42:56.978 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:56 smithi151 ceph-mon[112233]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.974+0000 7f56c6ffd700 1 -- 172.21.15.151:0/3148533195 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f56c005aea0 con 0x7f56d0100410 2024-04-16T02:42:56.978 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_OK 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56bc06ecc0 msgr2=0x7f56bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56bc06ecc0 0x7f56bc071180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f56cc006010 tx=0x7f56cc01a040 comp rx=0 tx=0).stop 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d0100410 msgr2=0x7f56d019d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d0100410 0x7f56d019d280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f56c000d900 tx=0x7f56c000dc10 comp rx=0 tx=0).stop 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 shutdown_connections 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56bc06ecc0 0x7f56bc071180 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56d00ffa60 0x7f56d019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 --2- 172.21.15.151:0/3148533195 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56d0100410 0x7f56d019d280 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 >> 172.21.15.151:0/3148533195 conn(0x7f56d00fb550 msgr2=0x7f56d0110f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 shutdown_connections 2024-04-16T02:42:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:42:56.977+0000 7f56d7f9e700 1 -- 172.21.15.151:0/3148533195 wait complete. 2024-04-16T02:42:57.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:56 smithi153 ceph-mon[87858]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:57.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:56 smithi153 ceph-mon[87858]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:57.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:56 smithi153 ceph-mon[87858]: pgmap v109: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:42:57.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:56 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/282452953' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:42:57.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:56 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1759842492' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:42:57.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:56 smithi153 ceph-mon[87858]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:42:58.235 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:57 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3148533195' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:42:58.235 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:57 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T02:42:58.235 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:57 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T02:42:58.235 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:57 smithi153 ceph-mon[87858]: Upgrade: osd.3 is safe to restart 2024-04-16T02:42:58.235 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:42:57 smithi153 ceph-mon[87858]: pgmap v110: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:42:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:57 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3148533195' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:42:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:57 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T02:42:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:57 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T02:42:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:57 smithi151 ceph-mon[112233]: Upgrade: osd.3 is safe to restart 2024-04-16T02:42:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:42:57 smithi151 ceph-mon[112233]: pgmap v110: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:43:00.665 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:00 smithi153 ceph-mon[87858]: Upgrade: Updating osd.3 2024-04-16T02:43:00.665 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:00.665 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T02:43:00.665 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:43:00.665 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:00 smithi153 ceph-mon[87858]: Deploying daemon osd.3 on smithi153 2024-04-16T02:43:00.665 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:00 smithi153 ceph-mon[87858]: pgmap v111: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:43:00.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:00 smithi151 ceph-mon[112233]: Upgrade: Updating osd.3 2024-04-16T02:43:00.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:00.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T02:43:00.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:43:00.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:00 smithi151 ceph-mon[112233]: Deploying daemon osd.3 on smithi153 2024-04-16T02:43:00.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:00 smithi151 ceph-mon[112233]: pgmap v111: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:43:02.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:01 smithi153 ceph-mon[87858]: pgmap v112: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:43:02.175 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:01 smithi151 ceph-mon[112233]: pgmap v112: 65 pgs: 65 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:43:02.934 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:02 smithi153 systemd[1]: Stopping Ceph osd.3 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:43:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:03 smithi153 ceph-mon[87858]: osd.3 marked itself down and dead 2024-04-16T02:43:03.272 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:02 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[42707]: 2024-04-16T02:43:02.950+0000 7f80d07be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T02:43:03.272 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:02 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[42707]: 2024-04-16T02:43:02.950+0000 7f80d07be700 -1 osd.3 65 *** Got signal Terminated *** 2024-04-16T02:43:03.273 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:02 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[42707]: 2024-04-16T02:43:02.950+0000 7f80d07be700 -1 osd.3 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T02:43:03.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:03 smithi151 ceph-mon[112233]: osd.3 marked itself down and dead 2024-04-16T02:43:03.895 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:03 smithi153 podman[96708]: 2024-04-16 02:43:03.584129447 +0000 UTC m=+0.760761573 container died 52e427295fb9d6588ff128969c57e00628c4924f7f6ddcd75676ab288a8a7c29 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-04-16T02:43:04.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:04 smithi153 ceph-mon[87858]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:43:04.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:04 smithi153 ceph-mon[87858]: osdmap e66: 6 total, 5 up, 6 in 2024-04-16T02:43:04.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:04 smithi153 ceph-mon[87858]: pgmap v114: 65 pgs: 9 peering, 14 stale+active+clean, 42 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s 2024-04-16T02:43:04.273 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:04 smithi153 podman[96708]: 2024-04-16 02:43:04.076605507 +0000 UTC m=+1.253237644 container cleanup 52e427295fb9d6588ff128969c57e00628c4924f7f6ddcd75676ab288a8a7c29 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1) 2024-04-16T02:43:04.273 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:04 smithi153 bash[96708]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3 2024-04-16T02:43:04.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:04 smithi151 ceph-mon[112233]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:43:04.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:04 smithi151 ceph-mon[112233]: osdmap e66: 6 total, 5 up, 6 in 2024-04-16T02:43:04.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:04 smithi151 ceph-mon[112233]: pgmap v114: 65 pgs: 9 peering, 14 stale+active+clean, 42 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s 2024-04-16T02:43:04.582 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:04 smithi153 podman[96735]: 2024-04-16 02:43:04.286275444 +0000 UTC m=+0.702717613 container remove 52e427295fb9d6588ff128969c57e00628c4924f7f6ddcd75676ab288a8a7c29 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.29.1, ceph=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-16T02:43:04.833 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:04 smithi153 podman[96823]: 2024-04-16 02:43:04.483095848 +0000 UTC m=+0.018670216 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:43:04.834 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:04 smithi153 podman[96823]: 2024-04-16 02:43:04.587146117 +0000 UTC m=+0.122720474 container create e93dd8b413d67748179f9917a75579da2b8b194f546aba8153d2c590243a1ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-16T02:43:04.834 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:04 smithi153 podman[96823]: 2024-04-16 02:43:04.831647569 +0000 UTC m=+0.367221927 container init e93dd8b413d67748179f9917a75579da2b8b194f546aba8153d2c590243a1ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-16T02:43:05.136 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:05 smithi153 ceph-mon[87858]: osdmap e67: 6 total, 5 up, 6 in 2024-04-16T02:43:05.137 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:05 smithi153 ceph-mon[87858]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-16T02:43:05.137 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:04 smithi153 podman[96823]: 2024-04-16 02:43:04.835583156 +0000 UTC m=+0.371157518 container start e93dd8b413d67748179f9917a75579da2b8b194f546aba8153d2c590243a1ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-16T02:43:05.137 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:04 smithi153 podman[96823]: 2024-04-16 02:43:04.97342662 +0000 UTC m=+0.509000992 container attach e93dd8b413d67748179f9917a75579da2b8b194f546aba8153d2c590243a1ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-16T02:43:05.446 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:05 smithi153 podman[96823]: 2024-04-16 02:43:05.135082951 +0000 UTC m=+0.670657308 container died e93dd8b413d67748179f9917a75579da2b8b194f546aba8153d2c590243a1ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-16T02:43:05.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:05 smithi151 ceph-mon[112233]: osdmap e67: 6 total, 5 up, 6 in 2024-04-16T02:43:05.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:05 smithi151 ceph-mon[112233]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-16T02:43:05.698 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:05 smithi153 podman[96849]: 2024-04-16 02:43:05.693916825 +0000 UTC m=+0.550043535 container remove e93dd8b413d67748179f9917a75579da2b8b194f546aba8153d2c590243a1ee5 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:43:06.022 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:05 smithi153 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-16T02:43:06.022 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:05 smithi153 systemd[1]: Stopped Ceph osd.3 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:43:06.022 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:05 smithi153 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.3.service: Consumed 1min 17.048s CPU time. 2024-04-16T02:43:06.023 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:05 smithi153 systemd[1]: Starting Ceph osd.3 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:43:06.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:06 smithi153 ceph-mon[87858]: pgmap v116: 65 pgs: 9 active+undersized+degraded, 9 peering, 4 stale+active+clean, 9 active+undersized, 34 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 41/336 objects degraded (12.202%) 2024-04-16T02:43:06.288 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 podman[96948]: 2024-04-16 02:43:06.046353025 +0000 UTC m=+0.017505577 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:43:06.288 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 podman[96948]: 2024-04-16 02:43:06.169811885 +0000 UTC m=+0.140964421 container create 1784e2fa30b62dc5dcdf154ef6820b52745159add729aa8a9847790e4c2b848a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=main-87dbfb6, maintainer=Guillaume Abrioux ) 2024-04-16T02:43:06.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:06 smithi151 ceph-mon[112233]: pgmap v116: 65 pgs: 9 active+undersized+degraded, 9 peering, 4 stale+active+clean, 9 active+undersized, 34 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 41/336 objects degraded (12.202%) 2024-04-16T02:43:06.772 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 podman[96948]: 2024-04-16 02:43:06.414196894 +0000 UTC m=+0.385349430 container init 1784e2fa30b62dc5dcdf154ef6820b52745159add729aa8a9847790e4c2b848a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T02:43:06.772 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 podman[96948]: 2024-04-16 02:43:06.41786833 +0000 UTC m=+0.389020867 container start 1784e2fa30b62dc5dcdf154ef6820b52745159add729aa8a9847790e4c2b848a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-16T02:43:06.773 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 podman[96948]: 2024-04-16 02:43:06.497499385 +0000 UTC m=+0.468651942 container attach 1784e2fa30b62dc5dcdf154ef6820b52745159add729aa8a9847790e4c2b848a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-16T02:43:07.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:07 smithi153 ceph-mon[87858]: Health check failed: Degraded data redundancy: 41/336 objects degraded (12.202%), 9 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate[96963]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate[96963]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate[96963]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate[96963]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate[96963]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate[96963]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate[96963]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 bash[96948]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 bash[96948]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 bash[96948]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 bash[96948]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T02:43:07.228 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 bash[96948]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-16T02:43:07.229 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 bash[96948]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T02:43:07.229 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 bash[96948]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-16T02:43:07.229 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:06 smithi153 podman[96948]: 2024-04-16 02:43:06.894967494 +0000 UTC m=+0.866120023 container died 1784e2fa30b62dc5dcdf154ef6820b52745159add729aa8a9847790e4c2b848a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-16T02:43:07.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:07 smithi151 ceph-mon[112233]: Health check failed: Degraded data redundancy: 41/336 objects degraded (12.202%), 9 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:07.876 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:07 smithi153 podman[97083]: 2024-04-16 02:43:07.543513038 +0000 UTC m=+0.643172184 container remove 1784e2fa30b62dc5dcdf154ef6820b52745159add729aa8a9847790e4c2b848a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-activate, ceph=True, GIT_CLEAN=True, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-16T02:43:07.876 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:07 smithi153 podman[97146]: 2024-04-16 02:43:07.664144488 +0000 UTC m=+0.018243357 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:43:07.876 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:07 smithi153 podman[97146]: 2024-04-16 02:43:07.777586166 +0000 UTC m=+0.131685025 container create a731606ec80f696227676ac1863f0bd846bc24cf3648e284a4e127ec7b5a7786 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-16T02:43:08.132 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:08 smithi153 ceph-mon[87858]: pgmap v117: 65 pgs: 12 active+undersized+degraded, 9 peering, 1 stale+active+clean, 13 active+undersized, 30 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 48/336 objects degraded (14.286%) 2024-04-16T02:43:08.132 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:08 smithi153 podman[97146]: 2024-04-16 02:43:08.038344193 +0000 UTC m=+0.392443051 container init a731606ec80f696227676ac1863f0bd846bc24cf3648e284a4e127ec7b5a7786 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-16T02:43:08.132 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:08 smithi153 podman[97146]: 2024-04-16 02:43:08.041737201 +0000 UTC m=+0.395836068 container start a731606ec80f696227676ac1863f0bd846bc24cf3648e284a4e127ec7b5a7786 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-16T02:43:08.132 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:08 smithi153 ceph-osd[97166]: -- 172.21.15.153:0/2213998564 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a671e61680 con 0x55a671e2b000 2024-04-16T02:43:08.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:08 smithi151 ceph-mon[112233]: pgmap v117: 65 pgs: 12 active+undersized+degraded, 9 peering, 1 stale+active+clean, 13 active+undersized, 30 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 48/336 objects degraded (14.286%) 2024-04-16T02:43:08.522 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:08 smithi153 bash[97146]: a731606ec80f696227676ac1863f0bd846bc24cf3648e284a4e127ec7b5a7786 2024-04-16T02:43:08.522 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:08 smithi153 systemd[1]: Started Ceph osd.3 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:43:09.012 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:08 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T02:43:08.596+0000 7f0bdd6127c0 -1 Falling back to public interface 2024-04-16T02:43:09.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:09.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:09.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:43:09.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:09.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:43:09.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:09.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:09.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:43:09.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:09.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:43:10.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:10 smithi151 ceph-mon[112233]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/336 objects degraded (20.833%) 2024-04-16T02:43:10.508 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:10 smithi153 ceph-mon[87858]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/336 objects degraded (20.833%) 2024-04-16T02:43:11.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:11 smithi151 ceph-mon[112233]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-16T02:43:11.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:11 smithi153 ceph-mon[87858]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-16T02:43:12.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:12 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:12.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:12 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:12.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:12 smithi151 ceph-mon[112233]: pgmap v119: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/336 objects degraded (20.833%) 2024-04-16T02:43:12.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:12 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:12.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:12 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:12.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:12 smithi153 ceph-mon[87858]: pgmap v119: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/336 objects degraded (20.833%) 2024-04-16T02:43:13.390 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:13 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T02:43:13.269+0000 7f0bdd6127c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-16T02:43:13.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:13 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:13.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:13 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:13.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:13 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:13.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:13 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:14.106 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:13 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T02:43:13.812+0000 7f0bdd6127c0 -1 osd.3 65 log_to_monitors true 2024-04-16T02:43:14.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:14 smithi153 ceph-mon[87858]: pgmap v120: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/336 objects degraded (20.833%) 2024-04-16T02:43:14.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:14 smithi153 ceph-mon[87858]: from='osd.3 [v2:172.21.15.153:6800/1577343223,v1:172.21.15.153:6801/1577343223]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T02:43:14.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:14 smithi153 ceph-mon[87858]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T02:43:14.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:14 smithi151 ceph-mon[112233]: pgmap v120: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 275 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 70/336 objects degraded (20.833%) 2024-04-16T02:43:14.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:14 smithi151 ceph-mon[112233]: from='osd.3 [v2:172.21.15.153:6800/1577343223,v1:172.21.15.153:6801/1577343223]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T02:43:14.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:14 smithi151 ceph-mon[112233]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T02:43:15.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:15 smithi151 ceph-mon[112233]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T02:43:15.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:15 smithi151 ceph-mon[112233]: from='osd.3 [v2:172.21.15.153:6800/1577343223,v1:172.21.15.153:6801/1577343223]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:43:15.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:15 smithi151 ceph-mon[112233]: osdmap e68: 6 total, 5 up, 6 in 2024-04-16T02:43:15.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:15 smithi151 ceph-mon[112233]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:43:15.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:15 smithi151 ceph-mon[112233]: Health check update: Degraded data redundancy: 70/336 objects degraded (20.833%), 19 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:15.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:15 smithi151 ceph-mon[112233]: from='osd.3 ' entity='osd.3' 2024-04-16T02:43:15.772 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 02:43:15 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T02:43:15.390+0000 7f0bcaa62700 -1 osd.3 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:43:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:15 smithi153 ceph-mon[87858]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T02:43:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:15 smithi153 ceph-mon[87858]: from='osd.3 [v2:172.21.15.153:6800/1577343223,v1:172.21.15.153:6801/1577343223]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:43:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:15 smithi153 ceph-mon[87858]: osdmap e68: 6 total, 5 up, 6 in 2024-04-16T02:43:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:15 smithi153 ceph-mon[87858]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:43:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:15 smithi153 ceph-mon[87858]: Health check update: Degraded data redundancy: 70/336 objects degraded (20.833%), 19 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:15.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:15 smithi153 ceph-mon[87858]: from='osd.3 ' entity='osd.3' 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: pgmap v122: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 70/336 objects degraded (20.833%) 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T02:43:16.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:43:16.748 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: osd.3 [v2:172.21.15.153:6800/1577343223,v1:172.21.15.153:6801/1577343223] boot 2024-04-16T02:43:16.748 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: osdmap e69: 6 total, 6 up, 6 in 2024-04-16T02:43:16.748 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:16 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:43:16.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: pgmap v122: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 70/336 objects degraded (20.833%) 2024-04-16T02:43:16.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: osd.3 [v2:172.21.15.153:6800/1577343223,v1:172.21.15.153:6801/1577343223] boot 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: osdmap e69: 6 total, 6 up, 6 in 2024-04-16T02:43:16.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:16 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T02:43:17.563 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:17 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T02:43:17.563 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:17 smithi153 ceph-mon[87858]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-16T02:43:17.563 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:17 smithi153 ceph-mon[87858]: osdmap e70: 6 total, 6 up, 6 in 2024-04-16T02:43:17.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:17 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T02:43:17.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:17 smithi151 ceph-mon[112233]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-16T02:43:17.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:17 smithi151 ceph-mon[112233]: osdmap e70: 6 total, 6 up, 6 in 2024-04-16T02:43:18.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:18 smithi151 ceph-mon[112233]: pgmap v125: 65 pgs: 4 peering, 18 active+undersized+degraded, 16 active+undersized, 27 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 67/336 objects degraded (19.940%) 2024-04-16T02:43:18.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:18 smithi153 ceph-mon[87858]: pgmap v125: 65 pgs: 4 peering, 18 active+undersized+degraded, 16 active+undersized, 27 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 67/336 objects degraded (19.940%) 2024-04-16T02:43:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:19 smithi151 ceph-mon[112233]: pgmap v126: 65 pgs: 4 peering, 61 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-16T02:43:20.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:19 smithi151 ceph-mon[112233]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/336 objects degraded (19.940%), 18 pgs degraded) 2024-04-16T02:43:20.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:19 smithi151 ceph-mon[112233]: Cluster is now healthy 2024-04-16T02:43:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:19 smithi153 ceph-mon[87858]: pgmap v126: 65 pgs: 4 peering, 61 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-16T02:43:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:19 smithi153 ceph-mon[87858]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/336 objects degraded (19.940%), 18 pgs degraded) 2024-04-16T02:43:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:19 smithi153 ceph-mon[87858]: Cluster is now healthy 2024-04-16T02:43:22.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:21 smithi151 ceph-mon[112233]: pgmap v127: 65 pgs: 4 peering, 61 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:43:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:21 smithi153 ceph-mon[87858]: pgmap v127: 65 pgs: 4 peering, 61 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:43:25.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:24 smithi151 ceph-mon[112233]: pgmap v128: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 2 op/s 2024-04-16T02:43:25.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:25.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:43:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:24 smithi153 ceph-mon[87858]: pgmap v128: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 2 op/s 2024-04-16T02:43:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:43:27.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.098+0000 7f67721bc700 1 -- 172.21.15.151:0/3260124666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c105a40 msgr2=0x7f676c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.098+0000 7f67721bc700 1 --2- 172.21.15.151:0/3260124666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c105a40 0x7f676c105e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f675c009910 tx=0x7f675c009c20 comp rx=0 tx=0).stop 2024-04-16T02:43:27.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.098+0000 7f67721bc700 1 -- 172.21.15.151:0/3260124666 shutdown_connections 2024-04-16T02:43:27.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.098+0000 7f67721bc700 1 --2- 172.21.15.151:0/3260124666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0ffa40 0x7f676c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.098+0000 7f67721bc700 1 --2- 172.21.15.151:0/3260124666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c105a40 0x7f676c105e20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.098+0000 7f67721bc700 1 -- 172.21.15.151:0/3260124666 >> 172.21.15.151:0/3260124666 conn(0x7f676c0fb570 msgr2=0x7f676c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:27.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.098+0000 7f67721bc700 1 -- 172.21.15.151:0/3260124666 shutdown_connections 2024-04-16T02:43:27.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.098+0000 7f67721bc700 1 -- 172.21.15.151:0/3260124666 wait complete. 2024-04-16T02:43:27.103 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:26 smithi151 ceph-mon[112233]: pgmap v129: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T02:43:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.099+0000 7f67721bc700 1 Processor -- start 2024-04-16T02:43:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.099+0000 7f67721bc700 1 -- start start 2024-04-16T02:43:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.099+0000 7f67721bc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c0ffa40 0x7f676c1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.099+0000 7f67721bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c105a40 0x7f676c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.099+0000 7f67721bc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c198100 con 0x7f676c0ffa40 2024-04-16T02:43:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.099+0000 7f67721bc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c198270 con 0x7f676c105a40 2024-04-16T02:43:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.099+0000 7f676b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c0ffa40 0x7f676c1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.099+0000 7f676b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c0ffa40 0x7f676c1975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50028/0 (socket says 172.21.15.151:50028) 2024-04-16T02:43:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f676b7fe700 1 -- 172.21.15.151:0/2443180220 learned_addr learned my addr 172.21.15.151:0/2443180220 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:43:27.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f6763fff700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c105a40 0x7f676c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:27.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f676b7fe700 1 -- 172.21.15.151:0/2443180220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c105a40 msgr2=0x7f676c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f676b7fe700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c105a40 0x7f676c197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f676b7fe700 1 -- 172.21.15.151:0/2443180220 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c0095f0 con 0x7f676c0ffa40 2024-04-16T02:43:27.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f676b7fe700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c0ffa40 0x7f676c1975a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f675c006010 tx=0x7f675c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:27.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f67697fa700 1 -- 172.21.15.151:0/2443180220 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f675c01d070 con 0x7f676c0ffa40 2024-04-16T02:43:27.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f67697fa700 1 -- 172.21.15.151:0/2443180220 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f675c00f460 con 0x7f676c0ffa40 2024-04-16T02:43:27.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c1a0610 con 0x7f676c0ffa40 2024-04-16T02:43:27.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f67697fa700 1 -- 172.21.15.151:0/2443180220 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f675c017610 con 0x7f676c0ffa40 2024-04-16T02:43:27.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.100+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c1a0a80 con 0x7f676c0ffa40 2024-04-16T02:43:27.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.101+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c108040 con 0x7f676c0ffa40 2024-04-16T02:43:27.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.102+0000 7f67697fa700 1 -- 172.21.15.151:0/2443180220 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f675c00f5d0 con 0x7f676c0ffa40 2024-04-16T02:43:27.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.102+0000 7f67697fa700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:27.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.102+0000 7f67697fa700 1 -- 172.21.15.151:0/2443180220 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f675c090260 con 0x7f676c0ffa40 2024-04-16T02:43:27.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.103+0000 7f6763fff700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:27.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.103+0000 7f6763fff700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f676c069570 tx=0x7f675400a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:27.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.108+0000 7f67697fa700 1 -- 172.21.15.151:0/2443180220 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f675c059140 con 0x7f676c0ffa40 2024-04-16T02:43:27.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:26 smithi153 ceph-mon[87858]: pgmap v129: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T02:43:27.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.325+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f676c02cd30 con 0x7f675806ecc0 2024-04-16T02:43:27.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.326+0000 7f67697fa700 1 -- 172.21.15.151:0/2443180220 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f676c02cd30 con 0x7f675806ecc0 2024-04-16T02:43:27.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 msgr2=0x7f6758071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f676c069570 tx=0x7f675400a300 comp rx=0 tx=0).stop 2024-04-16T02:43:27.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c0ffa40 msgr2=0x7f676c1975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c0ffa40 0x7f676c1975a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f675c006010 tx=0x7f675c00bd70 comp rx=0 tx=0).stop 2024-04-16T02:43:27.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 shutdown_connections 2024-04-16T02:43:27.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c0ffa40 0x7f676c1975a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 --2- 172.21.15.151:0/2443180220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c105a40 0x7f676c197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 >> 172.21.15.151:0/2443180220 conn(0x7f676c0fb570 msgr2=0x7f676c0fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:27.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 shutdown_connections 2024-04-16T02:43:27.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.328+0000 7f67721bc700 1 -- 172.21.15.151:0/2443180220 wait complete. 2024-04-16T02:43:27.342 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:43:27.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.435+0000 7f8551e6b700 1 -- 172.21.15.151:0/3200966258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 msgr2=0x7f854c102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.435+0000 7f8551e6b700 1 --2- 172.21.15.151:0/3200966258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 0x7f854c102180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f853c009910 tx=0x7f853c009c20 comp rx=0 tx=0).stop 2024-04-16T02:43:27.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.435+0000 7f8551e6b700 1 -- 172.21.15.151:0/3200966258 shutdown_connections 2024-04-16T02:43:27.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.435+0000 7f8551e6b700 1 --2- 172.21.15.151:0/3200966258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f854c1026c0 0x7f854c10abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.435+0000 7f8551e6b700 1 --2- 172.21.15.151:0/3200966258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 0x7f854c102180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.435+0000 7f8551e6b700 1 -- 172.21.15.151:0/3200966258 >> 172.21.15.151:0/3200966258 conn(0x7f854c0fb570 msgr2=0x7f854c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:27.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.435+0000 7f8551e6b700 1 -- 172.21.15.151:0/3200966258 shutdown_connections 2024-04-16T02:43:27.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.435+0000 7f8551e6b700 1 -- 172.21.15.151:0/3200966258 wait complete. 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.436+0000 7f8551e6b700 1 Processor -- start 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.436+0000 7f8551e6b700 1 -- start start 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f8551e6b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 0x7f854c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f8551e6b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f854c1026c0 0x7f854c19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f8551e6b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c19d0e0 con 0x7f854c101da0 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f8551e6b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c196590 con 0x7f854c1026c0 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f854b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 0x7f854c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f854b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 0x7f854c19c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50044/0 (socket says 172.21.15.151:50044) 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f854b7fe700 1 -- 172.21.15.151:0/714511529 learned_addr learned my addr 172.21.15.151:0/714511529 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f854affd700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f854c1026c0 0x7f854c19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f854b7fe700 1 -- 172.21.15.151:0/714511529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f854c1026c0 msgr2=0x7f854c19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f854b7fe700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f854c1026c0 0x7f854c19ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f854b7fe700 1 -- 172.21.15.151:0/714511529 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0095f0 con 0x7f854c101da0 2024-04-16T02:43:27.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.437+0000 7f854b7fe700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 0x7f854c19c510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f853c00bd70 tx=0x7f853c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:27.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.438+0000 7f8548ff9700 1 -- 172.21.15.151:0/714511529 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f853c01d070 con 0x7f854c101da0 2024-04-16T02:43:27.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.438+0000 7f8548ff9700 1 -- 172.21.15.151:0/714511529 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f853c00f460 con 0x7f854c101da0 2024-04-16T02:43:27.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.438+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c196810 con 0x7f854c101da0 2024-04-16T02:43:27.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.438+0000 7f8548ff9700 1 -- 172.21.15.151:0/714511529 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f853c017630 con 0x7f854c101da0 2024-04-16T02:43:27.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.438+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c196d00 con 0x7f854c101da0 2024-04-16T02:43:27.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.439+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c1082b0 con 0x7f854c101da0 2024-04-16T02:43:27.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.440+0000 7f8548ff9700 1 -- 172.21.15.151:0/714511529 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f853c00f5d0 con 0x7f854c101da0 2024-04-16T02:43:27.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.440+0000 7f8548ff9700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f853406ecc0 0x7f8534071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:27.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.440+0000 7f8548ff9700 1 -- 172.21.15.151:0/714511529 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f853c0903a0 con 0x7f854c101da0 2024-04-16T02:43:27.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.440+0000 7f854affd700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f853406ecc0 0x7f8534071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:27.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.441+0000 7f854affd700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f853406ecc0 0x7f8534071180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8540005fd0 tx=0x7f854000a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:27.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.445+0000 7f8548ff9700 1 -- 172.21.15.151:0/714511529 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f853c05cd30 con 0x7f854c101da0 2024-04-16T02:43:27.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.662+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f854c197030 con 0x7f853406ecc0 2024-04-16T02:43:27.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.663+0000 7f8548ff9700 1 -- 172.21.15.151:0/714511529 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f854c197030 con 0x7f853406ecc0 2024-04-16T02:43:27.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f853406ecc0 msgr2=0x7f8534071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f853406ecc0 0x7f8534071180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8540005fd0 tx=0x7f854000a560 comp rx=0 tx=0).stop 2024-04-16T02:43:27.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 msgr2=0x7f854c19c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 0x7f854c19c510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f853c00bd70 tx=0x7f853c00be50 comp rx=0 tx=0).stop 2024-04-16T02:43:27.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 shutdown_connections 2024-04-16T02:43:27.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f853406ecc0 0x7f8534071180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f854c101da0 0x7f854c19c510 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 --2- 172.21.15.151:0/714511529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f854c1026c0 0x7f854c19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 >> 172.21.15.151:0/714511529 conn(0x7f854c0fb570 msgr2=0x7f854c1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:27.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 shutdown_connections 2024-04-16T02:43:27.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.665+0000 7f8551e6b700 1 -- 172.21.15.151:0/714511529 wait complete. 2024-04-16T02:43:27.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 -- 172.21.15.151:0/1120943047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598101770 msgr2=0x7f15981057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 --2- 172.21.15.151:0/1120943047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598101770 0x7f15981057c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1584009910 tx=0x7f1584009c20 comp rx=0 tx=0).stop 2024-04-16T02:43:27.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 -- 172.21.15.151:0/1120943047 shutdown_connections 2024-04-16T02:43:27.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 --2- 172.21.15.151:0/1120943047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598101770 0x7f15981057c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 --2- 172.21.15.151:0/1120943047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1598100dc0 0x7f15981011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 -- 172.21.15.151:0/1120943047 >> 172.21.15.151:0/1120943047 conn(0x7f15980fc650 msgr2=0x7f15980fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:27.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 -- 172.21.15.151:0/1120943047 shutdown_connections 2024-04-16T02:43:27.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 -- 172.21.15.151:0/1120943047 wait complete. 2024-04-16T02:43:27.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 Processor -- start 2024-04-16T02:43:27.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.771+0000 7f159f0ba700 1 -- start start 2024-04-16T02:43:27.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159f0ba700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598100dc0 0x7f159819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:27.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159f0ba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1598101770 0x7f159819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:27.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159f0ba700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159819d130 con 0x7f1598100dc0 2024-04-16T02:43:27.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159f0ba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981965e0 con 0x7f1598101770 2024-04-16T02:43:27.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159ce56700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598100dc0 0x7f159819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:27.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159ce56700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598100dc0 0x7f159819c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50070/0 (socket says 172.21.15.151:50070) 2024-04-16T02:43:27.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159ce56700 1 -- 172.21.15.151:0/4035247910 learned_addr learned my addr 172.21.15.151:0/4035247910 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f1597fff700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1598101770 0x7f159819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159ce56700 1 -- 172.21.15.151:0/4035247910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1598101770 msgr2=0x7f159819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159ce56700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1598101770 0x7f159819caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159ce56700 1 -- 172.21.15.151:0/4035247910 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15840095f0 con 0x7f1598100dc0 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159ce56700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598100dc0 0x7f159819c560 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f158c00d8d0 tx=0x7f158c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f1595ffb700 1 -- 172.21.15.151:0/4035247910 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f158c0098a0 con 0x7f1598100dc0 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f1595ffb700 1 -- 172.21.15.151:0/4035247910 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f158c010460 con 0x7f1598100dc0 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15981968c0 con 0x7f1598100dc0 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.772+0000 7f1595ffb700 1 -- 172.21.15.151:0/4035247910 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f158c00f5f0 con 0x7f1598100dc0 2024-04-16T02:43:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.773+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1598196e10 con 0x7f1598100dc0 2024-04-16T02:43:27.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.774+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15981091c0 con 0x7f1598100dc0 2024-04-16T02:43:27.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.774+0000 7f1595ffb700 1 -- 172.21.15.151:0/4035247910 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f158c009a00 con 0x7f1598100dc0 2024-04-16T02:43:27.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.774+0000 7f1595ffb700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f158806ecc0 0x7f1588071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:27.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.774+0000 7f1595ffb700 1 -- 172.21.15.151:0/4035247910 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f158c08eb80 con 0x7f1598100dc0 2024-04-16T02:43:27.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.774+0000 7f1597fff700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f158806ecc0 0x7f1588071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:27.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.775+0000 7f1597fff700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f158806ecc0 0x7f1588071180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f15980fcea0 tx=0x7f158401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:27.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.779+0000 7f1595ffb700 1 -- 172.21.15.151:0/4035247910 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f158c05b590 con 0x7f1598100dc0 2024-04-16T02:43:27.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:27.992+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1598197100 con 0x7f158806ecc0 2024-04-16T02:43:28.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.002+0000 7f1595ffb700 1 -- 172.21.15.151:0/4035247910 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1598197100 con 0x7f158806ecc0 2024-04-16T02:43:28.005 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (5m) 50s ago 16m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (16m) 50s ago 16m 8740k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (15m) 16s ago 15m 8308k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (2m) 50s ago 16m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (2m) 16s ago 15m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (4m) 50s ago 16m 81.1M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (10m) 50s ago 10m 120M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (10m) 50s ago 10m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (10m) 16s ago 10m 58.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (10m) 16s ago 10m 67.5M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (7m) 50s ago 17m 505M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:43:28.006 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (5m) 16s ago 15m 452M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (3m) 50s ago 17m 48.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (3m) 16s ago 15m 41.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (5m) 50s ago 16m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (5m) 16s ago 15m 15.6M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (2m) 50s ago 14m 88.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (91s) 50s ago 14m 96.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (55s) 50s ago 13m 12.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (20s) 16s ago 13m 12.7M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (12m) 16s ago 12m 231M 2240M 18.2.1 5be31c24972a 06f646d752f3 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (12m) 16s ago 12m 241M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:43:28.007 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (5m) 50s ago 15m 73.7M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:43:28.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f158806ecc0 msgr2=0x7f1588071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f158806ecc0 0x7f1588071180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f15980fcea0 tx=0x7f158401a040 comp rx=0 tx=0).stop 2024-04-16T02:43:28.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598100dc0 msgr2=0x7f159819c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598100dc0 0x7f159819c560 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f158c00d8d0 tx=0x7f158c00dbe0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 shutdown_connections 2024-04-16T02:43:28.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f158806ecc0 0x7f1588071180 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1598100dc0 0x7f159819c560 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 --2- 172.21.15.151:0/4035247910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1598101770 0x7f159819caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 >> 172.21.15.151:0/4035247910 conn(0x7f15980fc650 msgr2=0x7f15980fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:28.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 shutdown_connections 2024-04-16T02:43:28.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.005+0000 7f159f0ba700 1 -- 172.21.15.151:0/4035247910 wait complete. 2024-04-16T02:43:28.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.110+0000 7f02b6323700 1 -- 172.21.15.151:0/3847349994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 msgr2=0x7f02b01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.110+0000 7f02b6323700 1 --2- 172.21.15.151:0/3847349994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 0x7f02b01011a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009910 tx=0x7f02a0009c20 comp rx=0 tx=0).stop 2024-04-16T02:43:28.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.110+0000 7f02b6323700 1 -- 172.21.15.151:0/3847349994 shutdown_connections 2024-04-16T02:43:28.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.110+0000 7f02b6323700 1 --2- 172.21.15.151:0/3847349994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b0101770 0x7f02b01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.110+0000 7f02b6323700 1 --2- 172.21.15.151:0/3847349994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 0x7f02b01011a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.110+0000 7f02b6323700 1 -- 172.21.15.151:0/3847349994 >> 172.21.15.151:0/3847349994 conn(0x7f02b00fc650 msgr2=0x7f02b00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.111+0000 7f02b6323700 1 -- 172.21.15.151:0/3847349994 shutdown_connections 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.111+0000 7f02b6323700 1 -- 172.21.15.151:0/3847349994 wait complete. 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.111+0000 7f02b6323700 1 Processor -- start 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.111+0000 7f02b6323700 1 -- start start 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.111+0000 7f02b6323700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 0x7f02b019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.111+0000 7f02b6323700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b0101770 0x7f02b019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.111+0000 7f02b6323700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b019d210 con 0x7f02b0100dc0 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.111+0000 7f02b6323700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01966c0 con 0x7f02b0101770 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02affff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 0x7f02b019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02affff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 0x7f02b019c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50092/0 (socket says 172.21.15.151:50092) 2024-04-16T02:43:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02affff700 1 -- 172.21.15.151:0/1650527110 learned_addr learned my addr 172.21.15.151:0/1650527110 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02affff700 1 -- 172.21.15.151:0/1650527110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b0101770 msgr2=0x7f02b019cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02a7fff700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b0101770 0x7f02b019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02affff700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b0101770 0x7f02b019cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02affff700 1 -- 172.21.15.151:0/1650527110 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a00095f0 con 0x7f02b0100dc0 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02affff700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 0x7f02b019c640 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f02a0006010 tx=0x7f02a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02adffb700 1 -- 172.21.15.151:0/1650527110 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02a001d070 con 0x7f02b0100dc0 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02adffb700 1 -- 172.21.15.151:0/1650527110 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02a0004fb0 con 0x7f02b0100dc0 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b0196940 con 0x7f02b0100dc0 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02adffb700 1 -- 172.21.15.151:0/1650527110 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02a0022ea0 con 0x7f02b0100dc0 2024-04-16T02:43:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.112+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b0196e30 con 0x7f02b0100dc0 2024-04-16T02:43:28.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.114+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b0109160 con 0x7f02b0100dc0 2024-04-16T02:43:28.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.114+0000 7f02adffb700 1 -- 172.21.15.151:0/1650527110 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f02a0022470 con 0x7f02b0100dc0 2024-04-16T02:43:28.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.115+0000 7f02adffb700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f029c06ecc0 0x7f029c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:28.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.115+0000 7f02adffb700 1 -- 172.21.15.151:0/1650527110 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f02a0090340 con 0x7f02b0100dc0 2024-04-16T02:43:28.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.115+0000 7f02a7fff700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f029c06ecc0 0x7f029c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:28.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.115+0000 7f02a7fff700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f029c06ecc0 0x7f029c071180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0298006fd0 tx=0x7f0298008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:28.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.119+0000 7f02adffb700 1 -- 172.21.15.151:0/1650527110 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f02a005cd80 con 0x7f02b0100dc0 2024-04-16T02:43:28.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.412+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f02b004ea90 con 0x7f02b0100dc0 2024-04-16T02:43:28.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.413+0000 7f02adffb700 1 -- 172.21.15.151:0/1650527110 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f02a0022720 con 0x7f02b0100dc0 2024-04-16T02:43:28.416 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:43:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:43:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:43:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6, 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 8 2024-04-16T02:43:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:43:28.418 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:43:28.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f029c06ecc0 msgr2=0x7f029c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f029c06ecc0 0x7f029c071180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0298006fd0 tx=0x7f0298008040 comp rx=0 tx=0).stop 2024-04-16T02:43:28.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 msgr2=0x7f02b019c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 0x7f02b019c640 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f02a0006010 tx=0x7f02a000be20 comp rx=0 tx=0).stop 2024-04-16T02:43:28.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 shutdown_connections 2024-04-16T02:43:28.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f029c06ecc0 0x7f029c071180 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b0100dc0 0x7f02b019c640 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 --2- 172.21.15.151:0/1650527110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b0101770 0x7f02b019cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 >> 172.21.15.151:0/1650527110 conn(0x7f02b00fc650 msgr2=0x7f02b00fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:28.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 shutdown_connections 2024-04-16T02:43:28.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.415+0000 7f02b6323700 1 -- 172.21.15.151:0/1650527110 wait complete. 2024-04-16T02:43:28.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.520+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1219839868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec0ffa40 msgr2=0x7ffbec0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.520+0000 7ffbf3f87700 1 --2- 172.21.15.151:0/1219839868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec0ffa40 0x7ffbec0ffec0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffbe8009910 tx=0x7ffbe8009c20 comp rx=0 tx=0).stop 2024-04-16T02:43:28.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.520+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1219839868 shutdown_connections 2024-04-16T02:43:28.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.520+0000 7ffbf3f87700 1 --2- 172.21.15.151:0/1219839868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec0ffa40 0x7ffbec0ffec0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.520+0000 7ffbf3f87700 1 --2- 172.21.15.151:0/1219839868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec105a40 0x7ffbec105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.520+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1219839868 >> 172.21.15.151:0/1219839868 conn(0x7ffbec0fb550 msgr2=0x7ffbec0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:28.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.520+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1219839868 shutdown_connections 2024-04-16T02:43:28.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.520+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1219839868 wait complete. 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf3f87700 1 Processor -- start 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf3f87700 1 -- start start 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf3f87700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec0ffa40 0x7ffbec0fbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf3f87700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec105a40 0x7ffbec0fc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf3f87700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec0fc940 con 0x7ffbec105a40 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf3f87700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec0fca80 con 0x7ffbec0ffa40 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf1522700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec105a40 0x7ffbec0fc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf1522700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec105a40 0x7ffbec0fc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50118/0 (socket says 172.21.15.151:50118) 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf1522700 1 -- 172.21.15.151:0/1998090912 learned_addr learned my addr 172.21.15.151:0/1998090912 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.521+0000 7ffbf1d23700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec0ffa40 0x7ffbec0fbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.522+0000 7ffbf1522700 1 -- 172.21.15.151:0/1998090912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec0ffa40 msgr2=0x7ffbec0fbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.522+0000 7ffbf1522700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec0ffa40 0x7ffbec0fbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.522+0000 7ffbf1522700 1 -- 172.21.15.151:0/1998090912 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe80095f0 con 0x7ffbec105a40 2024-04-16T02:43:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.522+0000 7ffbf1522700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec105a40 0x7ffbec0fc400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffbe8006010 tx=0x7ffbe800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.522+0000 7ffbe2ffd700 1 -- 172.21.15.151:0/1998090912 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbe801d070 con 0x7ffbec105a40 2024-04-16T02:43:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.522+0000 7ffbe2ffd700 1 -- 172.21.15.151:0/1998090912 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbe800f460 con 0x7ffbec105a40 2024-04-16T02:43:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.522+0000 7ffbe2ffd700 1 -- 172.21.15.151:0/1998090912 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbe8017610 con 0x7ffbec105a40 2024-04-16T02:43:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.522+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec0fcd00 con 0x7ffbec105a40 2024-04-16T02:43:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.522+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec1a2d00 con 0x7ffbec105a40 2024-04-16T02:43:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.523+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbec109070 con 0x7ffbec105a40 2024-04-16T02:43:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.523+0000 7ffbe2ffd700 1 -- 172.21.15.151:0/1998090912 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ffbe800f5d0 con 0x7ffbec105a40 2024-04-16T02:43:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.524+0000 7ffbe2ffd700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffbd806ecc0 0x7ffbd8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.524+0000 7ffbe2ffd700 1 -- 172.21.15.151:0/1998090912 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ffbe80902f0 con 0x7ffbec105a40 2024-04-16T02:43:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.524+0000 7ffbf1d23700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffbd806ecc0 0x7ffbd8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.524+0000 7ffbf1d23700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffbd806ecc0 0x7ffbd8071180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009a10 tx=0x7ffbdc009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:28.531 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.529+0000 7ffbe2ffd700 1 -- 172.21.15.151:0/1998090912 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ffbe8059150 con 0x7ffbec105a40 2024-04-16T02:43:28.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.772+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffbec04ea90 con 0x7ffbec105a40 2024-04-16T02:43:28.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.772+0000 7ffbe2ffd700 1 -- 172.21.15.151:0/1998090912 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1919 (secure 0 0 0) 0x7ffbe805c810 con 0x7ffbec105a40 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:e22 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 22 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:40:05.114230+0000 2024-04-16T02:43:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:43:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 14478 members: 14490,14478 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:43:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:43:28.779 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:43:28.779 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:43:28.779 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.774+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffbd806ecc0 msgr2=0x7ffbd8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.774+0000 7ffbf3f87700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffbd806ecc0 0x7ffbd8071180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009a10 tx=0x7ffbdc009190 comp rx=0 tx=0).stop 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.774+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec105a40 msgr2=0x7ffbec0fc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.774+0000 7ffbf3f87700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec105a40 0x7ffbec0fc400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffbe8006010 tx=0x7ffbe800bd70 comp rx=0 tx=0).stop 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.775+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 shutdown_connections 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.775+0000 7ffbf3f87700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffbd806ecc0 0x7ffbd8071180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.775+0000 7ffbf3f87700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec0ffa40 0x7ffbec0fbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.775+0000 7ffbf3f87700 1 --2- 172.21.15.151:0/1998090912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec105a40 0x7ffbec0fc400 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.775+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 >> 172.21.15.151:0/1998090912 conn(0x7ffbec0fb550 msgr2=0x7ffbec069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.775+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 shutdown_connections 2024-04-16T02:43:28.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.775+0000 7ffbf3f87700 1 -- 172.21.15.151:0/1998090912 wait complete. 2024-04-16T02:43:28.781 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 22 2024-04-16T02:43:28.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.882+0000 7f36990f2700 1 -- 172.21.15.151:0/2639011774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36940ffbd0 msgr2=0x7f3694100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.882+0000 7f36990f2700 1 --2- 172.21.15.151:0/2639011774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36940ffbd0 0x7f3694100050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3684009910 tx=0x7f3684009c20 comp rx=0 tx=0).stop 2024-04-16T02:43:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.882+0000 7f36990f2700 1 -- 172.21.15.151:0/2639011774 shutdown_connections 2024-04-16T02:43:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.882+0000 7f36990f2700 1 --2- 172.21.15.151:0/2639011774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36940ffbd0 0x7f3694100050 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.882+0000 7f36990f2700 1 --2- 172.21.15.151:0/2639011774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3694105a40 0x7f3694105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.882+0000 7f36990f2700 1 -- 172.21.15.151:0/2639011774 >> 172.21.15.151:0/2639011774 conn(0x7f36940755a0 msgr2=0x7f36940759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.882+0000 7f36990f2700 1 -- 172.21.15.151:0/2639011774 shutdown_connections 2024-04-16T02:43:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.882+0000 7f36990f2700 1 -- 172.21.15.151:0/2639011774 wait complete. 2024-04-16T02:43:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f36990f2700 1 Processor -- start 2024-04-16T02:43:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f36990f2700 1 -- start start 2024-04-16T02:43:28.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f36990f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36940ffbd0 0x7f3694197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:28.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f36990f2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3694105a40 0x7f3694197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:28.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f36990f2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36941981b0 con 0x7f3694105a40 2024-04-16T02:43:28.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f36990f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3694198320 con 0x7f36940ffbd0 2024-04-16T02:43:28.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f369259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3694105a40 0x7f3694197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:28.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f369259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3694105a40 0x7f3694197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50134/0 (socket says 172.21.15.151:50134) 2024-04-16T02:43:28.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f369259c700 1 -- 172.21.15.151:0/458923683 learned_addr learned my addr 172.21.15.151:0/458923683 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:43:28.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.883+0000 7f3692d9d700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36940ffbd0 0x7f3694197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:28.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.884+0000 7f369259c700 1 -- 172.21.15.151:0/458923683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36940ffbd0 msgr2=0x7f3694197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.884+0000 7f369259c700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36940ffbd0 0x7f3694197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.884+0000 7f369259c700 1 -- 172.21.15.151:0/458923683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36840095f0 con 0x7f3694105a40 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.884+0000 7f369259c700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3694105a40 0x7f3694197ad0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f36840092e0 tx=0x7f368400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.884+0000 7f367bfff700 1 -- 172.21.15.151:0/458923683 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f368401d070 con 0x7f3694105a40 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.884+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36941a0600 con 0x7f3694105a40 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.884+0000 7f367bfff700 1 -- 172.21.15.151:0/458923683 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3684005e00 con 0x7f3694105a40 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.884+0000 7f367bfff700 1 -- 172.21.15.151:0/458923683 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3684017400 con 0x7f3694105a40 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.884+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36941a0a70 con 0x7f3694105a40 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.885+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3694107fd0 con 0x7f3694105a40 2024-04-16T02:43:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.886+0000 7f367bfff700 1 -- 172.21.15.151:0/458923683 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f368400f6f0 con 0x7f3694105a40 2024-04-16T02:43:28.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.886+0000 7f367bfff700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f367c06ecc0 0x7f367c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:28.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.886+0000 7f367bfff700 1 -- 172.21.15.151:0/458923683 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3684090280 con 0x7f3694105a40 2024-04-16T02:43:28.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.886+0000 7f3692d9d700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f367c06ecc0 0x7f367c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:28.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.887+0000 7f3692d9d700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f367c06ecc0 0x7f367c071180 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3688005950 tx=0x7f368800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:28.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:28.891+0000 7f367bfff700 1 -- 172.21.15.151:0/458923683 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f36840590e0 con 0x7f3694105a40 2024-04-16T02:43:29.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:28 smithi153 ceph-mon[87858]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:43:29.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:28 smithi153 ceph-mon[87858]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:43:29.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:28 smithi153 ceph-mon[87858]: pgmap v130: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:43:29.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:28 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1650527110' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:29.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:28 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1998090912' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:43:29.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.107+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f369402cce0 con 0x7f367c06ecc0 2024-04-16T02:43:29.110 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:28 smithi151 ceph-mon[112233]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:43:29.110 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:28 smithi151 ceph-mon[112233]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:43:29.110 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:28 smithi151 ceph-mon[112233]: pgmap v130: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:43:29.110 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:28 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1650527110' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:29.111 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:28 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1998090912' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:43:29.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.108+0000 7f367bfff700 1 -- 172.21.15.151:0/458923683 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f369402cce0 con 0x7f367c06ecc0 2024-04-16T02:43:29.111 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:43:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:43:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:43:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:43:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:43:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:43:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:43:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:43:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:43:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "10/23 daemons upgraded", 2024-04-16T02:43:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T02:43:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:43:29.112 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:43:29.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f367c06ecc0 msgr2=0x7f367c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f367c06ecc0 0x7f367c071180 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3688005950 tx=0x7f368800b410 comp rx=0 tx=0).stop 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3694105a40 msgr2=0x7f3694197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3694105a40 0x7f3694197ad0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f36840092e0 tx=0x7f368400bd70 comp rx=0 tx=0).stop 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 shutdown_connections 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f367c06ecc0 0x7f367c071180 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36940ffbd0 0x7f3694197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 --2- 172.21.15.151:0/458923683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3694105a40 0x7f3694197ad0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 >> 172.21.15.151:0/458923683 conn(0x7f36940755a0 msgr2=0x7f36940fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 shutdown_connections 2024-04-16T02:43:29.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.110+0000 7f36990f2700 1 -- 172.21.15.151:0/458923683 wait complete. 2024-04-16T02:43:29.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.216+0000 7f228d724700 1 -- 172.21.15.151:0/1149553060 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 msgr2=0x7f2288068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:29.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.216+0000 7f228d724700 1 --2- 172.21.15.151:0/1149553060 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 0x7f2288068a50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2278009910 tx=0x7f2278009c20 comp rx=0 tx=0).stop 2024-04-16T02:43:29.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.216+0000 7f228d724700 1 -- 172.21.15.151:0/1149553060 shutdown_connections 2024-04-16T02:43:29.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.216+0000 7f228d724700 1 --2- 172.21.15.151:0/1149553060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2288069020 0x7f2288105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.216+0000 7f228d724700 1 --2- 172.21.15.151:0/1149553060 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 0x7f2288068a50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.216+0000 7f228d724700 1 -- 172.21.15.151:0/1149553060 >> 172.21.15.151:0/1149553060 conn(0x7f22880755a0 msgr2=0x7f22880759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:29.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.216+0000 7f228d724700 1 -- 172.21.15.151:0/1149553060 shutdown_connections 2024-04-16T02:43:29.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.216+0000 7f228d724700 1 -- 172.21.15.151:0/1149553060 wait complete. 2024-04-16T02:43:29.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f228d724700 1 Processor -- start 2024-04-16T02:43:29.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f228d724700 1 -- start start 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f228d724700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 0x7f22880ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f228d724700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2288069020 0x7f22881001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f228d724700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2288100720 con 0x7f2288068670 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f228d724700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2288100860 con 0x7f2288069020 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f2286ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 0x7f22880ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f2286ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 0x7f22880ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50144/0 (socket says 172.21.15.151:50144) 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f2286ffd700 1 -- 172.21.15.151:0/4229192320 learned_addr learned my addr 172.21.15.151:0/4229192320 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f22867fc700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2288069020 0x7f22881001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f2286ffd700 1 -- 172.21.15.151:0/4229192320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2288069020 msgr2=0x7f22881001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f2286ffd700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2288069020 0x7f22881001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.217+0000 7f2286ffd700 1 -- 172.21.15.151:0/4229192320 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22780095f0 con 0x7f2288068670 2024-04-16T02:43:29.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.218+0000 7f2286ffd700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 0x7f22880ffca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f22780092e0 tx=0x7f227800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:29.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.218+0000 7f226ffff700 1 -- 172.21.15.151:0/4229192320 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f227801d070 con 0x7f2288068670 2024-04-16T02:43:29.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.218+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2288100ae0 con 0x7f2288068670 2024-04-16T02:43:29.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.218+0000 7f226ffff700 1 -- 172.21.15.151:0/4229192320 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2278005e00 con 0x7f2288068670 2024-04-16T02:43:29.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.218+0000 7f226ffff700 1 -- 172.21.15.151:0/4229192320 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2278017400 con 0x7f2288068670 2024-04-16T02:43:29.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.218+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22881a2c40 con 0x7f2288068670 2024-04-16T02:43:29.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.219+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2288109130 con 0x7f2288068670 2024-04-16T02:43:29.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.219+0000 7f226ffff700 1 -- 172.21.15.151:0/4229192320 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f227800f460 con 0x7f2288068670 2024-04-16T02:43:29.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.220+0000 7f226ffff700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227006ecc0 0x7f2270071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:29.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.220+0000 7f22867fc700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227006ecc0 0x7f2270071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:29.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.220+0000 7f226ffff700 1 -- 172.21.15.151:0/4229192320 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f22780902a0 con 0x7f2288068670 2024-04-16T02:43:29.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.220+0000 7f22867fc700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227006ecc0 0x7f2270071180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f227c005950 tx=0x7f227c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:29.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.225+0000 7f226ffff700 1 -- 172.21.15.151:0/4229192320 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2278059100 con 0x7f2288068670 2024-04-16T02:43:29.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.506+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f228804ea90 con 0x7f2288068670 2024-04-16T02:43:29.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.506+0000 7f226ffff700 1 -- 172.21.15.151:0/4229192320 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f227805c7c0 con 0x7f2288068670 2024-04-16T02:43:29.509 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_OK 2024-04-16T02:43:29.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.508+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227006ecc0 msgr2=0x7f2270071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.508+0000 7f228d724700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227006ecc0 0x7f2270071180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f227c005950 tx=0x7f227c0058e0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.508+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 msgr2=0x7f22880ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.508+0000 7f228d724700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 0x7f22880ffca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f22780092e0 tx=0x7f227800be20 comp rx=0 tx=0).stop 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.508+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 shutdown_connections 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.508+0000 7f228d724700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227006ecc0 0x7f2270071180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.508+0000 7f228d724700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2288068670 0x7f22880ffca0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.508+0000 7f228d724700 1 --2- 172.21.15.151:0/4229192320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2288069020 0x7f22881001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.508+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 >> 172.21.15.151:0/4229192320 conn(0x7f22880755a0 msgr2=0x7f22880fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.509+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 shutdown_connections 2024-04-16T02:43:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:29.509+0000 7f228d724700 1 -- 172.21.15.151:0/4229192320 wait complete. 2024-04-16T02:43:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:29 smithi151 ceph-mon[112233]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:43:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:29 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:30.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:29 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4229192320' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:43:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:29 smithi153 ceph-mon[87858]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:43:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:29 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:29 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4229192320' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:43:31.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:30 smithi151 ceph-mon[112233]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:43:31.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:30 smithi151 ceph-mon[112233]: pgmap v131: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-16T02:43:31.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:30 smithi153 ceph-mon[87858]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:43:31.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:30 smithi153 ceph-mon[87858]: pgmap v131: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-16T02:43:32.046 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:31 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T02:43:32.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:31 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T02:43:33.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:32 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T02:43:33.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:32 smithi151 ceph-mon[112233]: Upgrade: osd.4 is safe to restart 2024-04-16T02:43:33.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:32 smithi151 ceph-mon[112233]: pgmap v132: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:43:33.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:32 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T02:43:33.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:32 smithi153 ceph-mon[87858]: Upgrade: osd.4 is safe to restart 2024-04-16T02:43:33.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:32 smithi153 ceph-mon[87858]: pgmap v132: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:43:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:33 smithi153 ceph-mon[87858]: Upgrade: Updating osd.4 2024-04-16T02:43:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:33 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:33 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T02:43:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:33 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:43:34.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:33 smithi153 ceph-mon[87858]: Deploying daemon osd.4 on smithi153 2024-04-16T02:43:34.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:33 smithi153 ceph-mon[87858]: pgmap v133: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:43:34.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:33 smithi151 ceph-mon[112233]: Upgrade: Updating osd.4 2024-04-16T02:43:34.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:33 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:34.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:33 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T02:43:34.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:33 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:43:34.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:33 smithi151 ceph-mon[112233]: Deploying daemon osd.4 on smithi153 2024-04-16T02:43:34.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:33 smithi151 ceph-mon[112233]: pgmap v133: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:43:36.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:35 smithi153 ceph-mon[87858]: pgmap v134: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:43:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:35 smithi151 ceph-mon[112233]: pgmap v134: 65 pgs: 65 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:43:36.513 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:36 smithi153 systemd[1]: Stopping Ceph osd.4 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:43:36.772 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:36 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[49579]: 2024-04-16T02:43:36.508+0000 7f07748a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T02:43:36.772 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:36 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[49579]: 2024-04-16T02:43:36.508+0000 7f07748a4700 -1 osd.4 70 *** Got signal Terminated *** 2024-04-16T02:43:36.772 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:36 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[49579]: 2024-04-16T02:43:36.508+0000 7f07748a4700 -1 osd.4 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T02:43:37.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:36 smithi151 ceph-mon[112233]: osd.4 marked itself down and dead 2024-04-16T02:43:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:36 smithi153 ceph-mon[87858]: osd.4 marked itself down and dead 2024-04-16T02:43:37.628 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:37 smithi153 podman[101925]: 2024-04-16 02:43:37.33634124 +0000 UTC m=+1.026693149 container died 06f646d752f34b546d3b4c1b0a884a0a16939b9d051cac7b46e9d3b4647596a9 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-16T02:43:37.969 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:37 smithi153 ceph-mon[87858]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:43:37.970 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:37 smithi153 ceph-mon[87858]: osdmap e71: 6 total, 5 up, 6 in 2024-04-16T02:43:37.970 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:37 smithi153 ceph-mon[87858]: pgmap v136: 65 pgs: 6 stale+active+clean, 59 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:43:37.970 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:37 smithi153 podman[101925]: 2024-04-16 02:43:37.724953121 +0000 UTC m=+1.415305029 container cleanup 06f646d752f34b546d3b4c1b0a884a0a16939b9d051cac7b46e9d3b4647596a9 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-16T02:43:37.970 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:37 smithi153 bash[101925]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4 2024-04-16T02:43:38.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:37 smithi151 ceph-mon[112233]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:43:38.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:37 smithi151 ceph-mon[112233]: osdmap e71: 6 total, 5 up, 6 in 2024-04-16T02:43:38.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:37 smithi151 ceph-mon[112233]: pgmap v136: 65 pgs: 6 stale+active+clean, 59 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:43:38.272 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:37 smithi153 podman[101943]: 2024-04-16 02:43:37.9678617 +0000 UTC m=+0.635539974 container remove 06f646d752f34b546d3b4c1b0a884a0a16939b9d051cac7b46e9d3b4647596a9 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-16T02:43:38.548 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:38 smithi153 podman[102037]: 2024-04-16 02:43:38.195888443 +0000 UTC m=+0.018252373 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:43:38.549 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:38 smithi153 podman[102037]: 2024-04-16 02:43:38.31904264 +0000 UTC m=+0.141406604 container create b5e76f56ed4630ab1fe57ba4901074fcfaafdf2efd64c020ca241d8eac11aecf (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-16T02:43:38.549 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:38 smithi153 podman[102037]: 2024-04-16 02:43:38.546720571 +0000 UTC m=+0.369084496 container init b5e76f56ed4630ab1fe57ba4901074fcfaafdf2efd64c020ca241d8eac11aecf (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-16T02:43:38.860 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:38 smithi153 podman[102037]: 2024-04-16 02:43:38.550316068 +0000 UTC m=+0.372679996 container start b5e76f56ed4630ab1fe57ba4901074fcfaafdf2efd64c020ca241d8eac11aecf (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-16T02:43:38.860 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:38 smithi153 podman[102037]: 2024-04-16 02:43:38.632227808 +0000 UTC m=+0.454591749 container attach b5e76f56ed4630ab1fe57ba4901074fcfaafdf2efd64c020ca241d8eac11aecf (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2) 2024-04-16T02:43:39.162 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:38 smithi153 podman[102037]: 2024-04-16 02:43:38.858768413 +0000 UTC m=+0.681132337 container died b5e76f56ed4630ab1fe57ba4901074fcfaafdf2efd64c020ca241d8eac11aecf (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-87dbfb6, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-16T02:43:39.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:38 smithi153 ceph-mon[87858]: osdmap e72: 6 total, 5 up, 6 in 2024-04-16T02:43:39.162 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:39.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:43:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:38 smithi151 ceph-mon[112233]: osdmap e72: 6 total, 5 up, 6 in 2024-04-16T02:43:39.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:39.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:43:39.772 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:39 smithi153 podman[102064]: 2024-04-16 02:43:39.410822291 +0000 UTC m=+0.540650730 container remove b5e76f56ed4630ab1fe57ba4901074fcfaafdf2efd64c020ca241d8eac11aecf (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:43:39.773 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:39 smithi153 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-16T02:43:39.773 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:39 smithi153 systemd[1]: Stopped Ceph osd.4 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:43:39.773 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:39 smithi153 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.4.service: Consumed 1min 4.851s CPU time. 2024-04-16T02:43:39.773 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:39 smithi153 systemd[1]: Starting Ceph osd.4 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:43:40.191 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:39 smithi153 ceph-mon[87858]: pgmap v138: 65 pgs: 12 active+undersized+degraded, 2 peering, 2 stale+active+clean, 12 active+undersized, 37 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 42/336 objects degraded (12.500%) 2024-04-16T02:43:40.191 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:39 smithi153 podman[102169]: 2024-04-16 02:43:39.760496629 +0000 UTC m=+0.018521525 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:43:40.191 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:39 smithi153 podman[102169]: 2024-04-16 02:43:39.878315674 +0000 UTC m=+0.136340542 container create d3f087fda678b30e41dbbd275e502c3d1078404d6e3105010f82dbb1330d9fcb (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-16T02:43:40.191 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 podman[102169]: 2024-04-16 02:43:40.098045703 +0000 UTC m=+0.356070572 container init d3f087fda678b30e41dbbd275e502c3d1078404d6e3105010f82dbb1330d9fcb (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T02:43:40.191 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 podman[102169]: 2024-04-16 02:43:40.101688163 +0000 UTC m=+0.359713040 container start d3f087fda678b30e41dbbd275e502c3d1078404d6e3105010f82dbb1330d9fcb (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-16T02:43:40.191 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 podman[102169]: 2024-04-16 02:43:40.188976268 +0000 UTC m=+0.447001145 container attach d3f087fda678b30e41dbbd275e502c3d1078404d6e3105010f82dbb1330d9fcb (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-87dbfb6) 2024-04-16T02:43:40.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:39 smithi151 ceph-mon[112233]: pgmap v138: 65 pgs: 12 active+undersized+degraded, 2 peering, 2 stale+active+clean, 12 active+undersized, 37 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 42/336 objects degraded (12.500%) 2024-04-16T02:43:40.869 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate[102183]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T02:43:40.869 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate[102183]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T02:43:40.869 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 bash[102169]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 bash[102169]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 bash[102169]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 bash[102169]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 bash[102169]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 bash[102169]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 bash[102169]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate[102183]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate[102183]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate[102183]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate[102183]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate[102183]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-16T02:43:40.870 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:40 smithi153 podman[102169]: 2024-04-16 02:43:40.580344643 +0000 UTC m=+0.838369522 container died d3f087fda678b30e41dbbd275e502c3d1078404d6e3105010f82dbb1330d9fcb (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-87dbfb6, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:43:41.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:40 smithi151 ceph-mon[112233]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-16T02:43:41.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:40 smithi151 ceph-mon[112233]: Health check failed: Degraded data redundancy: 42/336 objects degraded (12.500%), 12 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:41.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:40 smithi153 ceph-mon[87858]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-16T02:43:41.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:40 smithi153 ceph-mon[87858]: Health check failed: Degraded data redundancy: 42/336 objects degraded (12.500%), 12 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:41.522 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:41 smithi153 podman[102310]: 2024-04-16 02:43:41.226424464 +0000 UTC m=+0.640140602 container remove d3f087fda678b30e41dbbd275e502c3d1078404d6e3105010f82dbb1330d9fcb (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-16T02:43:41.522 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:41 smithi153 podman[102372]: 2024-04-16 02:43:41.348702348 +0000 UTC m=+0.018691545 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:43:41.522 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:41 smithi153 podman[102372]: 2024-04-16 02:43:41.452316466 +0000 UTC m=+0.122305687 container create 184b926e242cf992b5265d1eae5185442002b53423a5064e5efd247ebaf40a36 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-16T02:43:41.878 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:41 smithi153 podman[102372]: 2024-04-16 02:43:41.705141932 +0000 UTC m=+0.375131111 container init 184b926e242cf992b5265d1eae5185442002b53423a5064e5efd247ebaf40a36 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-16T02:43:41.879 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:41 smithi153 podman[102372]: 2024-04-16 02:43:41.708534284 +0000 UTC m=+0.378523465 container start 184b926e242cf992b5265d1eae5185442002b53423a5064e5efd247ebaf40a36 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-16T02:43:41.879 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:41 smithi153 bash[102372]: 184b926e242cf992b5265d1eae5185442002b53423a5064e5efd247ebaf40a36 2024-04-16T02:43:41.879 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:41 smithi153 systemd[1]: Started Ceph osd.4 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:43:42.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:41 smithi151 ceph-mon[112233]: pgmap v139: 65 pgs: 12 active+undersized+degraded, 2 peering, 2 stale+active+clean, 12 active+undersized, 37 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 42/336 objects degraded (12.500%) 2024-04-16T02:43:42.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:41 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:42.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:41 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:42.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:41 smithi153 ceph-mon[87858]: pgmap v139: 65 pgs: 12 active+undersized+degraded, 2 peering, 2 stale+active+clean, 12 active+undersized, 37 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 42/336 objects degraded (12.500%) 2024-04-16T02:43:42.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:41 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:42.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:41 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:42.522 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:42 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T02:43:42.262+0000 7f43ad0a97c0 -1 Falling back to public interface 2024-04-16T02:43:43.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:42 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:43:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:42 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:43:43.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:43 smithi153 ceph-mon[87858]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 55/336 objects degraded (16.369%) 2024-04-16T02:43:44.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:43 smithi151 ceph-mon[112233]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 55/336 objects degraded (16.369%) 2024-04-16T02:43:45.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:44 smithi151 ceph-mon[112233]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-16T02:43:45.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:44 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:45.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:44 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:45.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:44 smithi153 ceph-mon[87858]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-16T02:43:45.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:44 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:45.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:44 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:46.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:45 smithi153 ceph-mon[87858]: Health check update: Degraded data redundancy: 55/336 objects degraded (16.369%), 16 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:46.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:45 smithi153 ceph-mon[87858]: pgmap v141: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 55/336 objects degraded (16.369%) 2024-04-16T02:43:46.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:45 smithi151 ceph-mon[112233]: Health check update: Degraded data redundancy: 55/336 objects degraded (16.369%), 16 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:46.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:45 smithi151 ceph-mon[112233]: pgmap v141: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 55/336 objects degraded (16.369%) 2024-04-16T02:43:47.022 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:46 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T02:43:46.820+0000 7f43ad0a97c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-16T02:43:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:47.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:47.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:47.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:47.605 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:47 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T02:43:47.338+0000 7f43ad0a97c0 -1 osd.4 70 log_to_monitors true 2024-04-16T02:43:48.197 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 02:43:47 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T02:43:47.978+0000 7f439e501700 -1 osd.4 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:43:48.197 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:47 smithi153 ceph-mon[87858]: from='osd.4 [v2:172.21.15.153:6808/290685586,v1:172.21.15.153:6809/290685586]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T02:43:48.197 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:47 smithi153 ceph-mon[87858]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T02:43:48.197 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:47 smithi153 ceph-mon[87858]: pgmap v142: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 55/336 objects degraded (16.369%) 2024-04-16T02:43:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:47 smithi151 ceph-mon[112233]: from='osd.4 [v2:172.21.15.153:6808/290685586,v1:172.21.15.153:6809/290685586]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T02:43:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:47 smithi151 ceph-mon[112233]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T02:43:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:47 smithi151 ceph-mon[112233]: pgmap v142: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 275 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 55/336 objects degraded (16.369%) 2024-04-16T02:43:49.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:48 smithi151 ceph-mon[112233]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T02:43:49.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:48 smithi151 ceph-mon[112233]: osdmap e73: 6 total, 5 up, 6 in 2024-04-16T02:43:49.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:48 smithi151 ceph-mon[112233]: from='osd.4 [v2:172.21.15.153:6808/290685586,v1:172.21.15.153:6809/290685586]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:43:49.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:48 smithi151 ceph-mon[112233]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:43:49.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:48 smithi153 ceph-mon[87858]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T02:43:49.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:48 smithi153 ceph-mon[87858]: osdmap e73: 6 total, 5 up, 6 in 2024-04-16T02:43:49.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:48 smithi153 ceph-mon[87858]: from='osd.4 [v2:172.21.15.153:6808/290685586,v1:172.21.15.153:6809/290685586]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:43:49.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:48 smithi153 ceph-mon[87858]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:43:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:43:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: osd.4 [v2:172.21.15.153:6808/290685586,v1:172.21.15.153:6809/290685586] boot 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: osdmap e74: 6 total, 6 up, 6 in 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: pgmap v145: 65 pgs: 14 peering, 10 active+undersized+degraded, 10 active+undersized, 31 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 28/336 objects degraded (8.333%) 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T02:43:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:43:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: osd.4 [v2:172.21.15.153:6808/290685586,v1:172.21.15.153:6809/290685586] boot 2024-04-16T02:43:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: osdmap e74: 6 total, 6 up, 6 in 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: pgmap v145: 65 pgs: 14 peering, 10 active+undersized+degraded, 10 active+undersized, 31 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 28/336 objects degraded (8.333%) 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:43:50.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T02:43:51.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:50 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T02:43:51.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:50 smithi151 ceph-mon[112233]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-16T02:43:51.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:50 smithi151 ceph-mon[112233]: osdmap e75: 6 total, 6 up, 6 in 2024-04-16T02:43:51.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:50 smithi151 ceph-mon[112233]: Health check update: Degraded data redundancy: 28/336 objects degraded (8.333%), 10 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:51.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:50 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T02:43:51.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:50 smithi153 ceph-mon[87858]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-16T02:43:51.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:50 smithi153 ceph-mon[87858]: osdmap e75: 6 total, 6 up, 6 in 2024-04-16T02:43:51.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:50 smithi153 ceph-mon[87858]: Health check update: Degraded data redundancy: 28/336 objects degraded (8.333%), 10 pgs degraded (PG_DEGRADED) 2024-04-16T02:43:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:51 smithi151 ceph-mon[112233]: pgmap v147: 65 pgs: 14 peering, 10 active+undersized+degraded, 10 active+undersized, 31 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 28/336 objects degraded (8.333%) 2024-04-16T02:43:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:51 smithi153 ceph-mon[87858]: pgmap v147: 65 pgs: 14 peering, 10 active+undersized+degraded, 10 active+undersized, 31 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 28/336 objects degraded (8.333%) 2024-04-16T02:43:55.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:54 smithi151 ceph-mon[112233]: pgmap v148: 65 pgs: 14 peering, 51 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-16T02:43:55.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:55.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:43:55.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:54 smithi151 ceph-mon[112233]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/336 objects degraded (8.333%), 10 pgs degraded) 2024-04-16T02:43:55.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:54 smithi151 ceph-mon[112233]: Cluster is now healthy 2024-04-16T02:43:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:54 smithi153 ceph-mon[87858]: pgmap v148: 65 pgs: 14 peering, 51 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-16T02:43:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:43:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:43:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:54 smithi153 ceph-mon[87858]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/336 objects degraded (8.333%), 10 pgs degraded) 2024-04-16T02:43:55.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:54 smithi153 ceph-mon[87858]: Cluster is now healthy 2024-04-16T02:43:57.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:56 smithi153 ceph-mon[87858]: pgmap v149: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:43:57.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:56 smithi151 ceph-mon[112233]: pgmap v149: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-16T02:43:59.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:43:58 smithi151 ceph-mon[112233]: pgmap v150: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-16T02:43:59.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:43:58 smithi153 ceph-mon[87858]: pgmap v150: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-16T02:43:59.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.627+0000 7faf2197f700 1 -- 172.21.15.151:0/274379070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c101c70 msgr2=0x7faf1c102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:59.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.627+0000 7faf2197f700 1 --2- 172.21.15.151:0/274379070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c101c70 0x7faf1c102050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faf0c009910 tx=0x7faf0c009c20 comp rx=0 tx=0).stop 2024-04-16T02:43:59.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.627+0000 7faf2197f700 1 -- 172.21.15.151:0/274379070 shutdown_connections 2024-04-16T02:43:59.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.627+0000 7faf2197f700 1 --2- 172.21.15.151:0/274379070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c102620 0x7faf1c10ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.627+0000 7faf2197f700 1 --2- 172.21.15.151:0/274379070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c101c70 0x7faf1c102050 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.627+0000 7faf2197f700 1 -- 172.21.15.151:0/274379070 >> 172.21.15.151:0/274379070 conn(0x7faf1c0fb550 msgr2=0x7faf1c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.627+0000 7faf2197f700 1 -- 172.21.15.151:0/274379070 shutdown_connections 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.627+0000 7faf2197f700 1 -- 172.21.15.151:0/274379070 wait complete. 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf2197f700 1 Processor -- start 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf2197f700 1 -- start start 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf2197f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c101c70 0x7faf1c0ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf2197f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c102620 0x7faf1c0ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf2197f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c1003e0 con 0x7faf1c102620 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf2197f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c100520 con 0x7faf1c101c70 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf1a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c102620 0x7faf1c0ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf1a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c102620 0x7faf1c0ffea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42182/0 (socket says 172.21.15.151:42182) 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf1a7fc700 1 -- 172.21.15.151:0/74883519 learned_addr learned my addr 172.21.15.151:0/74883519 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.628+0000 7faf1affd700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c101c70 0x7faf1c0ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.629+0000 7faf1a7fc700 1 -- 172.21.15.151:0/74883519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c101c70 msgr2=0x7faf1c0ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:59.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.629+0000 7faf1a7fc700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c101c70 0x7faf1c0ff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:59.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.629+0000 7faf1a7fc700 1 -- 172.21.15.151:0/74883519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c0095f0 con 0x7faf1c102620 2024-04-16T02:43:59.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.629+0000 7faf1a7fc700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c102620 0x7faf1c0ffea0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faf1000d900 tx=0x7faf1000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:59.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.629+0000 7faf2097d700 1 -- 172.21.15.151:0/74883519 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf10009900 con 0x7faf1c102620 2024-04-16T02:43:59.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.629+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1c1a28f0 con 0x7faf1c102620 2024-04-16T02:43:59.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.629+0000 7faf2097d700 1 -- 172.21.15.151:0/74883519 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf10010460 con 0x7faf1c102620 2024-04-16T02:43:59.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.629+0000 7faf2097d700 1 -- 172.21.15.151:0/74883519 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf1000f5f0 con 0x7faf1c102620 2024-04-16T02:43:59.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.629+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1c1a2d00 con 0x7faf1c102620 2024-04-16T02:43:59.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.630+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1c108210 con 0x7faf1c102620 2024-04-16T02:43:59.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.631+0000 7faf2097d700 1 -- 172.21.15.151:0/74883519 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7faf100105d0 con 0x7faf1c102620 2024-04-16T02:43:59.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.631+0000 7faf2097d700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faf0406ecc0 0x7faf04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:59.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.631+0000 7faf2097d700 1 -- 172.21.15.151:0/74883519 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7faf1008e980 con 0x7faf1c102620 2024-04-16T02:43:59.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.631+0000 7faf1affd700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faf0406ecc0 0x7faf04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:59.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.632+0000 7faf1affd700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faf0406ecc0 0x7faf04071180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00b5c0 tx=0x7faf0c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:59.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.636+0000 7faf2097d700 1 -- 172.21.15.151:0/74883519 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7faf100577e0 con 0x7faf1c102620 2024-04-16T02:43:59.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.855+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faf1c02cda0 con 0x7faf0406ecc0 2024-04-16T02:43:59.858 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.856+0000 7faf2097d700 1 -- 172.21.15.151:0/74883519 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7faf1c02cda0 con 0x7faf0406ecc0 2024-04-16T02:43:59.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faf0406ecc0 msgr2=0x7faf04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:59.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faf0406ecc0 0x7faf04071180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00b5c0 tx=0x7faf0c01a040 comp rx=0 tx=0).stop 2024-04-16T02:43:59.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c102620 msgr2=0x7faf1c0ffea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:59.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c102620 0x7faf1c0ffea0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faf1000d900 tx=0x7faf1000dc10 comp rx=0 tx=0).stop 2024-04-16T02:43:59.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 shutdown_connections 2024-04-16T02:43:59.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faf0406ecc0 0x7faf04071180 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:59.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf1c101c70 0x7faf1c0ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:59.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 --2- 172.21.15.151:0/74883519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf1c102620 0x7faf1c0ffea0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:59.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 >> 172.21.15.151:0/74883519 conn(0x7faf1c0fb550 msgr2=0x7faf1c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:59.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 shutdown_connections 2024-04-16T02:43:59.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.858+0000 7faf2197f700 1 -- 172.21.15.151:0/74883519 wait complete. 2024-04-16T02:43:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:43:59.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.964+0000 7f71d7a13700 1 -- 172.21.15.151:0/3279348909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 msgr2=0x7f71d0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:59.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.964+0000 7f71d7a13700 1 --2- 172.21.15.151:0/3279348909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 0x7f71d0105e20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009910 tx=0x7f71cc009c20 comp rx=0 tx=0).stop 2024-04-16T02:43:59.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.964+0000 7f71d7a13700 1 -- 172.21.15.151:0/3279348909 shutdown_connections 2024-04-16T02:43:59.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.964+0000 7f71d7a13700 1 --2- 172.21.15.151:0/3279348909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71d00ffbd0 0x7f71d0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:59.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.964+0000 7f71d7a13700 1 --2- 172.21.15.151:0/3279348909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 0x7f71d0105e20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:59.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.964+0000 7f71d7a13700 1 -- 172.21.15.151:0/3279348909 >> 172.21.15.151:0/3279348909 conn(0x7f71d00755a0 msgr2=0x7f71d00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:43:59.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.965+0000 7f71d7a13700 1 -- 172.21.15.151:0/3279348909 shutdown_connections 2024-04-16T02:43:59.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.965+0000 7f71d7a13700 1 -- 172.21.15.151:0/3279348909 wait complete. 2024-04-16T02:43:59.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.965+0000 7f71d7a13700 1 Processor -- start 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.965+0000 7f71d7a13700 1 -- start start 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d7a13700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71d00ffbd0 0x7f71d019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d7a13700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 0x7f71d019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d7a13700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d019d100 con 0x7f71d0105a40 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d7a13700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d01965b0 con 0x7f71d00ffbd0 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d4fae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 0x7f71d019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d4fae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 0x7f71d019ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42206/0 (socket says 172.21.15.151:42206) 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d4fae700 1 -- 172.21.15.151:0/1472397359 learned_addr learned my addr 172.21.15.151:0/1472397359 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d57af700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71d00ffbd0 0x7f71d019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d4fae700 1 -- 172.21.15.151:0/1472397359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71d00ffbd0 msgr2=0x7f71d019c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d4fae700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71d00ffbd0 0x7f71d019c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d4fae700 1 -- 172.21.15.151:0/1472397359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc0095f0 con 0x7f71d0105a40 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d4fae700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 0x7f71d019ca70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c8a0 tx=0x7f71c000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:59.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71c67fc700 1 -- 172.21.15.151:0/1472397359 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71c00078e0 con 0x7f71d0105a40 2024-04-16T02:43:59.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71c67fc700 1 -- 172.21.15.151:0/1472397359 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71c000f450 con 0x7f71d0105a40 2024-04-16T02:43:59.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d0196890 con 0x7f71d0105a40 2024-04-16T02:43:59.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.966+0000 7f71c67fc700 1 -- 172.21.15.151:0/1472397359 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71c000e5e0 con 0x7f71d0105a40 2024-04-16T02:43:59.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.967+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d0196de0 con 0x7f71d0105a40 2024-04-16T02:43:59.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.968+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d0108040 con 0x7f71d0105a40 2024-04-16T02:43:59.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.968+0000 7f71c67fc700 1 -- 172.21.15.151:0/1472397359 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f71c000f5c0 con 0x7f71d0105a40 2024-04-16T02:43:59.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.969+0000 7f71c67fc700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f71bc06ecc0 0x7f71bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:43:59.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.969+0000 7f71c67fc700 1 -- 172.21.15.151:0/1472397359 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f71c008edd0 con 0x7f71d0105a40 2024-04-16T02:43:59.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.969+0000 7f71d57af700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f71bc06ecc0 0x7f71bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:43:59.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.969+0000 7f71d57af700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f71bc06ecc0 0x7f71bc071180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f71cc005f50 tx=0x7f71cc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:43:59.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:43:59.974+0000 7f71c67fc700 1 -- 172.21.15.151:0/1472397359 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f71c005b760 con 0x7f71d0105a40 2024-04-16T02:44:00.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.195+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f71d01970e0 con 0x7f71bc06ecc0 2024-04-16T02:44:00.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.196+0000 7f71c67fc700 1 -- 172.21.15.151:0/1472397359 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f71d01970e0 con 0x7f71bc06ecc0 2024-04-16T02:44:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f71bc06ecc0 msgr2=0x7f71bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f71bc06ecc0 0x7f71bc071180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f71cc005f50 tx=0x7f71cc01a040 comp rx=0 tx=0).stop 2024-04-16T02:44:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 msgr2=0x7f71d019ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 0x7f71d019ca70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c8a0 tx=0x7f71c000cbb0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 shutdown_connections 2024-04-16T02:44:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f71bc06ecc0 0x7f71bc071180 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71d00ffbd0 0x7f71d019c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 --2- 172.21.15.151:0/1472397359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71d0105a40 0x7f71d019ca70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 >> 172.21.15.151:0/1472397359 conn(0x7f71d00755a0 msgr2=0x7f71d00fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:00.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 shutdown_connections 2024-04-16T02:44:00.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.199+0000 7f71d7a13700 1 -- 172.21.15.151:0/1472397359 wait complete. 2024-04-16T02:44:00.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.305+0000 7f536dd53700 1 -- 172.21.15.151:0/2612719716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368068670 msgr2=0x7f5368068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.305+0000 7f536dd53700 1 --2- 172.21.15.151:0/2612719716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368068670 0x7f5368068a50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5350009910 tx=0x7f5350009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:00.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.305+0000 7f536dd53700 1 -- 172.21.15.151:0/2612719716 shutdown_connections 2024-04-16T02:44:00.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.305+0000 7f536dd53700 1 --2- 172.21.15.151:0/2612719716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5368069020 0x7f5368105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.305+0000 7f536dd53700 1 --2- 172.21.15.151:0/2612719716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368068670 0x7f5368068a50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.305+0000 7f536dd53700 1 -- 172.21.15.151:0/2612719716 >> 172.21.15.151:0/2612719716 conn(0x7f53680755a0 msgr2=0x7f53680759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.305+0000 7f536dd53700 1 -- 172.21.15.151:0/2612719716 shutdown_connections 2024-04-16T02:44:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.305+0000 7f536dd53700 1 -- 172.21.15.151:0/2612719716 wait complete. 2024-04-16T02:44:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f536dd53700 1 Processor -- start 2024-04-16T02:44:00.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f536dd53700 1 -- start start 2024-04-16T02:44:00.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f536dd53700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5368068670 0x7f536819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:00.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f536dd53700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368069020 0x7f536819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f536dd53700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536819d0e0 con 0x7f5368069020 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f536dd53700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368196590 con 0x7f5368068670 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f5366ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368069020 0x7f536819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f5366ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368069020 0x7f536819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42218/0 (socket says 172.21.15.151:42218) 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f5366ffd700 1 -- 172.21.15.151:0/3273320116 learned_addr learned my addr 172.21.15.151:0/3273320116 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.306+0000 7f53677fe700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5368068670 0x7f536819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f5366ffd700 1 -- 172.21.15.151:0/3273320116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5368068670 msgr2=0x7f536819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f5366ffd700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5368068670 0x7f536819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f5366ffd700 1 -- 172.21.15.151:0/3273320116 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53500095f0 con 0x7f5368069020 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f53677fe700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5368068670 0x7f536819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f5366ffd700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368069020 0x7f536819ca50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f535800d8d0 tx=0x7f535800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f5364ff9700 1 -- 172.21.15.151:0/3273320116 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53580041f0 con 0x7f5368069020 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f5364ff9700 1 -- 172.21.15.151:0/3273320116 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5358007620 con 0x7f5368069020 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5368196870 con 0x7f5368069020 2024-04-16T02:44:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f5364ff9700 1 -- 172.21.15.151:0/3273320116 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5358003c50 con 0x7f5368069020 2024-04-16T02:44:00.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.307+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5368196dc0 con 0x7f5368069020 2024-04-16T02:44:00.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.308+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5368109130 con 0x7f5368069020 2024-04-16T02:44:00.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.308+0000 7f5364ff9700 1 -- 172.21.15.151:0/3273320116 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5358004350 con 0x7f5368069020 2024-04-16T02:44:00.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.309+0000 7f5364ff9700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f535406ecc0 0x7f5354071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:00.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.309+0000 7f5364ff9700 1 -- 172.21.15.151:0/3273320116 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f535808e920 con 0x7f5368069020 2024-04-16T02:44:00.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.309+0000 7f53677fe700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f535406ecc0 0x7f5354071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:00.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.309+0000 7f53677fe700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f535406ecc0 0x7f5354071180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5350005b40 tx=0x7f535001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.314+0000 7f5364ff9700 1 -- 172.21.15.151:0/3273320116 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f535805b2b0 con 0x7f5368069020 2024-04-16T02:44:00.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.521+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f53681970a0 con 0x7f535406ecc0 2024-04-16T02:44:00.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.532+0000 7f5364ff9700 1 -- 172.21.15.151:0/3273320116 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f53681970a0 con 0x7f535406ecc0 2024-04-16T02:44:00.535 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:44:00.535 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (5m) 82s ago 16m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:44:00.535 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (17m) 82s ago 17m 8740k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (16m) 15s ago 16m 8598k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (3m) 82s ago 17m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (3m) 15s ago 16m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (5m) 82s ago 16m 81.1M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (11m) 82s ago 11m 120M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (11m) 82s ago 11m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (11m) 15s ago 11m 58.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (11m) 15s ago 11m 67.3M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (7m) 82s ago 18m 505M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (6m) 15s ago 15m 452M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (4m) 82s ago 18m 48.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (3m) 15s ago 15m 42.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (6m) 82s ago 17m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (6m) 15s ago 15m 16.0M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (2m) 82s ago 15m 88.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:44:00.536 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (2m) 82s ago 14m 96.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:44:00.537 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (88s) 82s ago 14m 12.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:44:00.537 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (52s) 15s ago 13m 107M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:44:00.537 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (19s) 15s ago 13m 11.6M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:44:00.537 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (12m) 15s ago 12m 241M 2240M 18.2.1 5be31c24972a 4bcb732928d4 2024-04-16T02:44:00.537 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (5m) 82s ago 16m 73.7M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:44:00.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f535406ecc0 msgr2=0x7f5354071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f535406ecc0 0x7f5354071180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5350005b40 tx=0x7f535001a040 comp rx=0 tx=0).stop 2024-04-16T02:44:00.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368069020 msgr2=0x7f536819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368069020 0x7f536819ca50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f535800d8d0 tx=0x7f535800dc90 comp rx=0 tx=0).stop 2024-04-16T02:44:00.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 shutdown_connections 2024-04-16T02:44:00.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f535406ecc0 0x7f5354071180 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5368068670 0x7f536819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 --2- 172.21.15.151:0/3273320116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5368069020 0x7f536819ca50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 >> 172.21.15.151:0/3273320116 conn(0x7f53680755a0 msgr2=0x7f53680fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:00.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 shutdown_connections 2024-04-16T02:44:00.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.535+0000 7f536dd53700 1 -- 172.21.15.151:0/3273320116 wait complete. 2024-04-16T02:44:00.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.638+0000 7f1b888ac700 1 -- 172.21.15.151:0/3933957916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84100dc0 msgr2=0x7f1b841011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.638+0000 7f1b888ac700 1 --2- 172.21.15.151:0/3933957916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84100dc0 0x7f1b841011a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009910 tx=0x7f1b74009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:00.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.638+0000 7f1b888ac700 1 -- 172.21.15.151:0/3933957916 shutdown_connections 2024-04-16T02:44:00.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.638+0000 7f1b888ac700 1 --2- 172.21.15.151:0/3933957916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84101770 0x7f1b841057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.638+0000 7f1b888ac700 1 --2- 172.21.15.151:0/3933957916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84100dc0 0x7f1b841011a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.638+0000 7f1b888ac700 1 -- 172.21.15.151:0/3933957916 >> 172.21.15.151:0/3933957916 conn(0x7f1b840fc650 msgr2=0x7f1b840fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:00.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.638+0000 7f1b888ac700 1 -- 172.21.15.151:0/3933957916 shutdown_connections 2024-04-16T02:44:00.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.638+0000 7f1b888ac700 1 -- 172.21.15.151:0/3933957916 wait complete. 2024-04-16T02:44:00.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.639+0000 7f1b888ac700 1 Processor -- start 2024-04-16T02:44:00.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.639+0000 7f1b888ac700 1 -- start start 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.639+0000 7f1b888ac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84100dc0 0x7f1b84073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.639+0000 7f1b888ac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84101770 0x7f1b840735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.639+0000 7f1b888ac700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b84073b10 con 0x7f1b84101770 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.639+0000 7f1b888ac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b84073c50 con 0x7f1b84100dc0 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b81d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84101770 0x7f1b840735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b81d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84101770 0x7f1b840735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42240/0 (socket says 172.21.15.151:42240) 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b81d9b700 1 -- 172.21.15.151:0/3336368379 learned_addr learned my addr 172.21.15.151:0/3336368379 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b8259c700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84100dc0 0x7f1b84073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b81d9b700 1 -- 172.21.15.151:0/3336368379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84100dc0 msgr2=0x7f1b84073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b81d9b700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84100dc0 0x7f1b84073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b81d9b700 1 -- 172.21.15.151:0/3336368379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b740095f0 con 0x7f1b84101770 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b81d9b700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84101770 0x7f1b840735d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800c8f0 tx=0x7f1b7800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b737fe700 1 -- 172.21.15.151:0/3336368379 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b780078a0 con 0x7f1b84101770 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b737fe700 1 -- 172.21.15.151:0/3336368379 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b7800f450 con 0x7f1b84101770 2024-04-16T02:44:00.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b737fe700 1 -- 172.21.15.151:0/3336368379 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b78018610 con 0x7f1b84101770 2024-04-16T02:44:00.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b841a2940 con 0x7f1b84101770 2024-04-16T02:44:00.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.640+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b841a2d50 con 0x7f1b84101770 2024-04-16T02:44:00.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.642+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b84109160 con 0x7f1b84101770 2024-04-16T02:44:00.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.642+0000 7f1b737fe700 1 -- 172.21.15.151:0/3336368379 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1b7800f5c0 con 0x7f1b84101770 2024-04-16T02:44:00.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.643+0000 7f1b737fe700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b6c06ecc0 0x7f1b6c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:00.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.643+0000 7f1b737fe700 1 -- 172.21.15.151:0/3336368379 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1b7808ece0 con 0x7f1b84101770 2024-04-16T02:44:00.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.643+0000 7f1b8259c700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b6c06ecc0 0x7f1b6c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:00.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.643+0000 7f1b8259c700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b6c06ecc0 0x7f1b6c071180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b74006010 tx=0x7f1b7401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:00.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.647+0000 7f1b737fe700 1 -- 172.21.15.151:0/3336368379 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1b78057b40 con 0x7f1b84101770 2024-04-16T02:44:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.932+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1b84073e10 con 0x7f1b84101770 2024-04-16T02:44:00.936 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:00 smithi151 ceph-mon[112233]: pgmap v151: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:44:00.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.933+0000 7f1b737fe700 1 -- 172.21.15.151:0/3336368379 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f1b7805b200 con 0x7f1b84101770 2024-04-16T02:44:00.936 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:44:00.936 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:44:00.936 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:44:00.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:44:00.936 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:44:00.936 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 5 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 9 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:44:00.937 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:44:00.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b6c06ecc0 msgr2=0x7f1b6c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b6c06ecc0 0x7f1b6c071180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b74006010 tx=0x7f1b7401a040 comp rx=0 tx=0).stop 2024-04-16T02:44:00.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84101770 msgr2=0x7f1b840735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:00.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84101770 0x7f1b840735d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800c8f0 tx=0x7f1b7800cc00 comp rx=0 tx=0).stop 2024-04-16T02:44:00.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 shutdown_connections 2024-04-16T02:44:00.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b6c06ecc0 0x7f1b6c071180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84100dc0 0x7f1b84073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 --2- 172.21.15.151:0/3336368379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b84101770 0x7f1b840735d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:00.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 >> 172.21.15.151:0/3336368379 conn(0x7f1b840fc650 msgr2=0x7f1b840fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:00.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 shutdown_connections 2024-04-16T02:44:00.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:00.935+0000 7f1b888ac700 1 -- 172.21.15.151:0/3336368379 wait complete. 2024-04-16T02:44:01.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.040+0000 7f9198304700 1 -- 172.21.15.151:0/240849527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 msgr2=0x7f91900ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 --2- 172.21.15.151:0/240849527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 0x7f91900ffe30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f918c009910 tx=0x7f918c009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:01.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 -- 172.21.15.151:0/240849527 shutdown_connections 2024-04-16T02:44:01.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 --2- 172.21.15.151:0/240849527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9190100400 0x7f91901048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 --2- 172.21.15.151:0/240849527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 0x7f91900ffe30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 -- 172.21.15.151:0/240849527 >> 172.21.15.151:0/240849527 conn(0x7f91900fb520 msgr2=0x7f91900fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:01.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 -- 172.21.15.151:0/240849527 shutdown_connections 2024-04-16T02:44:01.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 -- 172.21.15.151:0/240849527 wait complete. 2024-04-16T02:44:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 Processor -- start 2024-04-16T02:44:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 -- start start 2024-04-16T02:44:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.041+0000 7f9198304700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 0x7f91901a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f9198304700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9190100400 0x7f91901a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f9198304700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91901a3680 con 0x7f91900ffa50 2024-04-16T02:44:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f9198304700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919019cb30 con 0x7f9190100400 2024-04-16T02:44:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91960a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 0x7f91901a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91960a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 0x7f91901a2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42258/0 (socket says 172.21.15.151:42258) 2024-04-16T02:44:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91960a0700 1 -- 172.21.15.151:0/1007312171 learned_addr learned my addr 172.21.15.151:0/1007312171 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91960a0700 1 -- 172.21.15.151:0/1007312171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9190100400 msgr2=0x7f91901a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f919589f700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9190100400 0x7f91901a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91960a0700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9190100400 0x7f91901a2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91960a0700 1 -- 172.21.15.151:0/1007312171 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c0095f0 con 0x7f91900ffa50 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f919589f700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9190100400 0x7f91901a2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91960a0700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 0x7f91901a2ab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f918c000c00 tx=0x7f918c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91877fe700 1 -- 172.21.15.151:0/1007312171 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918c01d070 con 0x7f91900ffa50 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91877fe700 1 -- 172.21.15.151:0/1007312171 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f918c00f460 con 0x7f91900ffa50 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f91877fe700 1 -- 172.21.15.151:0/1007312171 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918c017630 con 0x7f91900ffa50 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919019cdb0 con 0x7f91900ffa50 2024-04-16T02:44:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.042+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919019d2a0 con 0x7f91900ffa50 2024-04-16T02:44:01.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.043+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9190102030 con 0x7f91900ffa50 2024-04-16T02:44:01.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.044+0000 7f91877fe700 1 -- 172.21.15.151:0/1007312171 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f918c00f5d0 con 0x7f91900ffa50 2024-04-16T02:44:01.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.044+0000 7f91877fe700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917c06ecc0 0x7f917c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:01.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.044+0000 7f91877fe700 1 -- 172.21.15.151:0/1007312171 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f918c0903d0 con 0x7f91900ffa50 2024-04-16T02:44:01.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.044+0000 7f919589f700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917c06ecc0 0x7f917c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:01.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.045+0000 7f919589f700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917c06ecc0 0x7f917c071180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9180005fd0 tx=0x7f9180005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:01.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.049+0000 7f91877fe700 1 -- 172.21.15.151:0/1007312171 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f918c05cd60 con 0x7f91900ffa50 2024-04-16T02:44:01.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:00 smithi153 ceph-mon[87858]: pgmap v151: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:44:01.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.301+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f919002cd30 con 0x7f91900ffa50 2024-04-16T02:44:01.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.301+0000 7f91877fe700 1 -- 172.21.15.151:0/1007312171 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1919 (secure 0 0 0) 0x7f918c05c8f0 con 0x7f91900ffa50 2024-04-16T02:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout:e22 2024-04-16T02:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 22 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:40:05.114230+0000 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 47 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=14478,1=14490} 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 14478 members: 14490,14478 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:14478} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.151:6826/1662886502,v1:172.21.15.151:6827/1662886502] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{0:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.153:6826/2749127439,v1:172.21.15.153:6827/2749127439] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:14490} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.151:6828/37453862,v1:172.21.15.151:6829/37453862] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{1:14502} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/3333609621,v1:172.21.15.153:6825/3333609621] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:44:01.307 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:44:01.307 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:44:01.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917c06ecc0 msgr2=0x7f917c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917c06ecc0 0x7f917c071180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9180005fd0 tx=0x7f9180005dc0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 msgr2=0x7f91901a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 0x7f91901a2ab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f918c000c00 tx=0x7f918c00be20 comp rx=0 tx=0).stop 2024-04-16T02:44:01.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 shutdown_connections 2024-04-16T02:44:01.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f917c06ecc0 0x7f917c071180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91900ffa50 0x7f91901a2ab0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 --2- 172.21.15.151:0/1007312171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9190100400 0x7f91901a2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 >> 172.21.15.151:0/1007312171 conn(0x7f91900fb520 msgr2=0x7f9190110f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:01.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 shutdown_connections 2024-04-16T02:44:01.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.304+0000 7f9198304700 1 -- 172.21.15.151:0/1007312171 wait complete. 2024-04-16T02:44:01.308 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 22 2024-04-16T02:44:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.410+0000 7f78ecbfa700 1 -- 172.21.15.151:0/1533426390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e8105b50 msgr2=0x7f78e8105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.410+0000 7f78ecbfa700 1 --2- 172.21.15.151:0/1533426390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e8105b50 0x7f78e8105f30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f78d8009910 tx=0x7f78d8009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.410+0000 7f78ecbfa700 1 -- 172.21.15.151:0/1533426390 shutdown_connections 2024-04-16T02:44:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.410+0000 7f78ecbfa700 1 --2- 172.21.15.151:0/1533426390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e80fef70 0x7f78e80ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.411+0000 7f78ecbfa700 1 --2- 172.21.15.151:0/1533426390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e8105b50 0x7f78e8105f30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.411+0000 7f78ecbfa700 1 -- 172.21.15.151:0/1533426390 >> 172.21.15.151:0/1533426390 conn(0x7f78e8075e30 msgr2=0x7f78e80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.411+0000 7f78ecbfa700 1 -- 172.21.15.151:0/1533426390 shutdown_connections 2024-04-16T02:44:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.411+0000 7f78ecbfa700 1 -- 172.21.15.151:0/1533426390 wait complete. 2024-04-16T02:44:01.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.411+0000 7f78ecbfa700 1 Processor -- start 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.411+0000 7f78ecbfa700 1 -- start start 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78ecbfa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e80fef70 0x7f78e819c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78ecbfa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e8105b50 0x7f78e819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78ecbfa700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e819d140 con 0x7f78e80fef70 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78ecbfa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e81965f0 con 0x7f78e8105b50 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78e659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e80fef70 0x7f78e819c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78e659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e80fef70 0x7f78e819c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42272/0 (socket says 172.21.15.151:42272) 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78e659c700 1 -- 172.21.15.151:0/454194324 learned_addr learned my addr 172.21.15.151:0/454194324 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78e659c700 1 -- 172.21.15.151:0/454194324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e8105b50 msgr2=0x7f78e819cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78e659c700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e8105b50 0x7f78e819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78e659c700 1 -- 172.21.15.151:0/454194324 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80095f0 con 0x7f78e80fef70 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78e659c700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e80fef70 0x7f78e819c570 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f78d8006010 tx=0x7f78d800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78d77fe700 1 -- 172.21.15.151:0/454194324 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78d801d070 con 0x7f78e80fef70 2024-04-16T02:44:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e8196870 con 0x7f78e80fef70 2024-04-16T02:44:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78d77fe700 1 -- 172.21.15.151:0/454194324 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78d800f460 con 0x7f78e80fef70 2024-04-16T02:44:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78d77fe700 1 -- 172.21.15.151:0/454194324 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78d8017630 con 0x7f78e80fef70 2024-04-16T02:44:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.412+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e8196d60 con 0x7f78e80fef70 2024-04-16T02:44:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.414+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e8109120 con 0x7f78e80fef70 2024-04-16T02:44:01.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.414+0000 7f78d77fe700 1 -- 172.21.15.151:0/454194324 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f78d80178f0 con 0x7f78e80fef70 2024-04-16T02:44:01.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.414+0000 7f78d77fe700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f78d006ecc0 0x7f78d0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:01.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.414+0000 7f78d77fe700 1 -- 172.21.15.151:0/454194324 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f78d8090270 con 0x7f78e80fef70 2024-04-16T02:44:01.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.414+0000 7f78e5d9b700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f78d006ecc0 0x7f78d0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:01.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.415+0000 7f78e5d9b700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f78d006ecc0 0x7f78d0071180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78dc005950 tx=0x7f78dc0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:01.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.419+0000 7f78d77fe700 1 -- 172.21.15.151:0/454194324 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f78d805cc70 con 0x7f78e80fef70 2024-04-16T02:44:01.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.636+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f78e8196f80 con 0x7f78d006ecc0 2024-04-16T02:44:01.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.637+0000 7f78d77fe700 1 -- 172.21.15.151:0/454194324 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f78e8196f80 con 0x7f78d006ecc0 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "11/23 daemons upgraded", 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:44:01.641 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:44:01.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f78d006ecc0 msgr2=0x7f78d0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f78d006ecc0 0x7f78d0071180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78dc005950 tx=0x7f78dc0058e0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e80fef70 msgr2=0x7f78e819c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e80fef70 0x7f78e819c570 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f78d8006010 tx=0x7f78d800be20 comp rx=0 tx=0).stop 2024-04-16T02:44:01.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 shutdown_connections 2024-04-16T02:44:01.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f78d006ecc0 0x7f78d0071180 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78e80fef70 0x7f78e819c570 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 --2- 172.21.15.151:0/454194324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78e8105b50 0x7f78e819cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 >> 172.21.15.151:0/454194324 conn(0x7f78e8075e30 msgr2=0x7f78e8104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:01.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 shutdown_connections 2024-04-16T02:44:01.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.640+0000 7f78ecbfa700 1 -- 172.21.15.151:0/454194324 wait complete. 2024-04-16T02:44:01.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.746+0000 7efdbc20a700 1 -- 172.21.15.151:0/1442982579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073130 msgr2=0x7efdb4073510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.746+0000 7efdbc20a700 1 --2- 172.21.15.151:0/1442982579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073130 0x7efdb4073510 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009910 tx=0x7efdb0009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:01.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.746+0000 7efdbc20a700 1 -- 172.21.15.151:0/1442982579 shutdown_connections 2024-04-16T02:44:01.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.746+0000 7efdbc20a700 1 --2- 172.21.15.151:0/1442982579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4073a50 0x7efdb4111610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.746+0000 7efdbc20a700 1 --2- 172.21.15.151:0/1442982579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073130 0x7efdb4073510 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.746+0000 7efdbc20a700 1 -- 172.21.15.151:0/1442982579 >> 172.21.15.151:0/1442982579 conn(0x7efdb40fc650 msgr2=0x7efdb40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:01.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.746+0000 7efdbc20a700 1 -- 172.21.15.151:0/1442982579 shutdown_connections 2024-04-16T02:44:01.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.746+0000 7efdbc20a700 1 -- 172.21.15.151:0/1442982579 wait complete. 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdbc20a700 1 Processor -- start 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdbc20a700 1 -- start start 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdbc20a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4073130 0x7efdb419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdbc20a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073a50 0x7efdb419d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdbc20a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb419d870 con 0x7efdb4073a50 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdbc20a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb41a05a0 con 0x7efdb4073130 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdb97a5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073a50 0x7efdb419d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdb97a5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073a50 0x7efdb419d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42286/0 (socket says 172.21.15.151:42286) 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdb97a5700 1 -- 172.21.15.151:0/2181637966 learned_addr learned my addr 172.21.15.151:0/2181637966 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdb9fa6700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4073130 0x7efdb419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdb97a5700 1 -- 172.21.15.151:0/2181637966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4073130 msgr2=0x7efdb419cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdb97a5700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4073130 0x7efdb419cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdb97a5700 1 -- 172.21.15.151:0/2181637966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb00095f0 con 0x7efdb4073a50 2024-04-16T02:44:01.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.747+0000 7efdb97a5700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073a50 0x7efdb419d1e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efda400c8a0 tx=0x7efda400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:01.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.748+0000 7efdaaffd700 1 -- 172.21.15.151:0/2181637966 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda40078e0 con 0x7efdb4073a50 2024-04-16T02:44:01.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.748+0000 7efdaaffd700 1 -- 172.21.15.151:0/2181637966 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efda400f450 con 0x7efdb4073a50 2024-04-16T02:44:01.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.748+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb41a0880 con 0x7efdb4073a50 2024-04-16T02:44:01.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.748+0000 7efdaaffd700 1 -- 172.21.15.151:0/2181637966 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda400e5e0 con 0x7efdb4073a50 2024-04-16T02:44:01.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.748+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb41a0dd0 con 0x7efdb4073a50 2024-04-16T02:44:01.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.749+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb410ed90 con 0x7efdb4073a50 2024-04-16T02:44:01.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.749+0000 7efdaaffd700 1 -- 172.21.15.151:0/2181637966 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efda400f5c0 con 0x7efdb4073a50 2024-04-16T02:44:01.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.750+0000 7efdaaffd700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efda006ecc0 0x7efda0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:01.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.750+0000 7efdaaffd700 1 -- 172.21.15.151:0/2181637966 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7efda408edd0 con 0x7efdb4073a50 2024-04-16T02:44:01.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.750+0000 7efdb9fa6700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efda006ecc0 0x7efda0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:01.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.750+0000 7efdb9fa6700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efda006ecc0 0x7efda0071180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efdb0006010 tx=0x7efdb001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:01.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:01.755+0000 7efdaaffd700 1 -- 172.21.15.151:0/2181637966 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efda4057c30 con 0x7efdb4073a50 2024-04-16T02:44:01.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:01 smithi151 ceph-mon[112233]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:01.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:01 smithi151 ceph-mon[112233]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:01.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:01 smithi151 ceph-mon[112233]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:01.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:01 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3336368379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:01.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:01 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1007312171' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:44:02.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.043+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efdb404ea90 con 0x7efdb4073a50 2024-04-16T02:44:02.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.044+0000 7efdaaffd700 1 -- 172.21.15.151:0/2181637966 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7efda405b2f0 con 0x7efdb4073a50 2024-04-16T02:44:02.047 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_OK 2024-04-16T02:44:02.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efda006ecc0 msgr2=0x7efda0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:02.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efda006ecc0 0x7efda0071180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efdb0006010 tx=0x7efdb001a040 comp rx=0 tx=0).stop 2024-04-16T02:44:02.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073a50 msgr2=0x7efdb419d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:02.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073a50 0x7efdb419d1e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efda400c8a0 tx=0x7efda400cbb0 comp rx=0 tx=0).stop 2024-04-16T02:44:02.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 shutdown_connections 2024-04-16T02:44:02.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efda006ecc0 0x7efda0071180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:02.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4073130 0x7efdb419cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:02.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 --2- 172.21.15.151:0/2181637966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efdb4073a50 0x7efdb419d1e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:02.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 >> 172.21.15.151:0/2181637966 conn(0x7efdb40fc650 msgr2=0x7efdb4103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:02.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 shutdown_connections 2024-04-16T02:44:02.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:02.046+0000 7efdbc20a700 1 -- 172.21.15.151:0/2181637966 wait complete. 2024-04-16T02:44:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:01 smithi153 ceph-mon[87858]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:01 smithi153 ceph-mon[87858]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:01 smithi153 ceph-mon[87858]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:02.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:01 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3336368379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:02.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:01 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1007312171' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:44:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:02 smithi151 ceph-mon[112233]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:03.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:02 smithi151 ceph-mon[112233]: pgmap v152: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:44:03.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:02 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2181637966' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:44:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:02 smithi153 ceph-mon[87858]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:02 smithi153 ceph-mon[87858]: pgmap v152: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:44:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:02 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2181637966' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:44:04.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:03 smithi151 ceph-mon[112233]: pgmap v153: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:44:04.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:03 smithi153 ceph-mon[87858]: pgmap v153: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:44:05.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:04 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:05.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:04 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T02:44:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:05 smithi153 ceph-mon[87858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T02:44:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:05 smithi153 ceph-mon[87858]: Upgrade: osd.5 is safe to restart 2024-04-16T02:44:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:05 smithi153 ceph-mon[87858]: pgmap v154: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:44:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T02:44:06.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:05 smithi151 ceph-mon[112233]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T02:44:06.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:05 smithi151 ceph-mon[112233]: Upgrade: osd.5 is safe to restart 2024-04-16T02:44:06.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:05 smithi151 ceph-mon[112233]: pgmap v154: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:44:07.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:07 smithi153 ceph-mon[87858]: Upgrade: Updating osd.5 2024-04-16T02:44:07.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:07.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T02:44:07.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:07 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:07.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:07 smithi153 ceph-mon[87858]: Deploying daemon osd.5 on smithi153 2024-04-16T02:44:07.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:07 smithi151 ceph-mon[112233]: Upgrade: Updating osd.5 2024-04-16T02:44:07.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:07.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T02:44:07.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:07 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:07.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:07 smithi151 ceph-mon[112233]: Deploying daemon osd.5 on smithi153 2024-04-16T02:44:08.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:08 smithi153 ceph-mon[87858]: pgmap v155: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:44:08.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:08 smithi151 ceph-mon[112233]: pgmap v155: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T02:44:09.951 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:09 smithi153 systemd[1]: Stopping Ceph osd.5 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:44:09.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:09.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:44:10.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:10.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:44:10.272 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:09 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[56540]: 2024-04-16T02:44:09.947+0000 7f843f1c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T02:44:10.272 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:09 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[56540]: 2024-04-16T02:44:09.947+0000 7f843f1c8700 -1 osd.5 75 *** Got signal Terminated *** 2024-04-16T02:44:10.272 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:09 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[56540]: 2024-04-16T02:44:09.947+0000 7f843f1c8700 -1 osd.5 75 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T02:44:11.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:10 smithi151 ceph-mon[112233]: pgmap v156: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:44:11.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:10 smithi151 ceph-mon[112233]: osd.5 marked itself down and dead 2024-04-16T02:44:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:10 smithi153 ceph-mon[87858]: pgmap v156: 65 pgs: 65 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-16T02:44:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:10 smithi153 ceph-mon[87858]: osd.5 marked itself down and dead 2024-04-16T02:44:11.603 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:11 smithi153 podman[107269]: 2024-04-16 02:44:11.313345707 +0000 UTC m=+1.467401334 container died 4bcb732928d4ff11f0e55aed6db87d6e1d4751e92563d53ee6b0254b5039dd33 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1) 2024-04-16T02:44:11.952 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:11 smithi153 podman[107269]: 2024-04-16 02:44:11.690432496 +0000 UTC m=+1.844488121 container cleanup 4bcb732928d4ff11f0e55aed6db87d6e1d4751e92563d53ee6b0254b5039dd33 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1) 2024-04-16T02:44:11.952 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:11 smithi153 bash[107269]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5 2024-04-16T02:44:11.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:11 smithi153 ceph-mon[87858]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:44:11.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:11 smithi153 ceph-mon[87858]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-16T02:44:11.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:11 smithi153 ceph-mon[87858]: osdmap e76: 6 total, 5 up, 6 in 2024-04-16T02:44:11.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:11 smithi153 ceph-mon[87858]: pgmap v158: 65 pgs: 13 stale+active+clean, 52 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:44:11.953 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:11 smithi153 ceph-mon[87858]: osdmap e77: 6 total, 5 up, 6 in 2024-04-16T02:44:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:11 smithi151 ceph-mon[112233]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T02:44:12.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:11 smithi151 ceph-mon[112233]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-16T02:44:12.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:11 smithi151 ceph-mon[112233]: osdmap e76: 6 total, 5 up, 6 in 2024-04-16T02:44:12.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:11 smithi151 ceph-mon[112233]: pgmap v158: 65 pgs: 13 stale+active+clean, 52 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-16T02:44:12.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:11 smithi151 ceph-mon[112233]: osdmap e77: 6 total, 5 up, 6 in 2024-04-16T02:44:12.272 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:11 smithi153 podman[107299]: 2024-04-16 02:44:11.950026428 +0000 UTC m=+0.633359237 container remove 4bcb732928d4ff11f0e55aed6db87d6e1d4751e92563d53ee6b0254b5039dd33 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1) 2024-04-16T02:44:12.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:12 smithi153 podman[107389]: 2024-04-16 02:44:12.176371141 +0000 UTC m=+0.018668494 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:44:12.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:12 smithi153 podman[107389]: 2024-04-16 02:44:12.284519835 +0000 UTC m=+0.126817207 container create 850fcf201e9aedd606d19e047043912069de0bca2a4ceb06eca67426db98748e (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-16T02:44:12.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:12 smithi153 podman[107389]: 2024-04-16 02:44:12.545138651 +0000 UTC m=+0.387435981 container init 850fcf201e9aedd606d19e047043912069de0bca2a4ceb06eca67426db98748e (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=) 2024-04-16T02:44:12.841 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:12 smithi153 podman[107389]: 2024-04-16 02:44:12.54857954 +0000 UTC m=+0.390876873 container start 850fcf201e9aedd606d19e047043912069de0bca2a4ceb06eca67426db98748e (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-16T02:44:12.841 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:12 smithi153 podman[107389]: 2024-04-16 02:44:12.637096175 +0000 UTC m=+0.479393520 container attach 850fcf201e9aedd606d19e047043912069de0bca2a4ceb06eca67426db98748e (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-16T02:44:13.201 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:12 smithi153 podman[107389]: 2024-04-16 02:44:12.839263605 +0000 UTC m=+0.681560949 container died 850fcf201e9aedd606d19e047043912069de0bca2a4ceb06eca67426db98748e (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-16T02:44:13.772 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:13 smithi153 podman[107416]: 2024-04-16 02:44:13.474088119 +0000 UTC m=+0.628583076 container remove 850fcf201e9aedd606d19e047043912069de0bca2a4ceb06eca67426db98748e (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, ceph=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-16T02:44:13.772 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:13 smithi153 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-16T02:44:13.772 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:13 smithi153 systemd[1]: Stopped Ceph osd.5 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:44:13.773 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:13 smithi153 systemd[1]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.5.service: Consumed 1min 9.860s CPU time. 2024-04-16T02:44:13.773 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:13 smithi153 systemd[1]: Starting Ceph osd.5 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T02:44:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:13 smithi151 ceph-mon[112233]: pgmap v160: 65 pgs: 8 active+undersized+degraded, 3 stale+active+clean, 10 active+undersized, 44 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 22/336 objects degraded (6.548%) 2024-04-16T02:44:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:13 smithi151 ceph-mon[112233]: Health check failed: Degraded data redundancy: 22/336 objects degraded (6.548%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T02:44:14.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:13 smithi153 ceph-mon[87858]: pgmap v160: 65 pgs: 8 active+undersized+degraded, 3 stale+active+clean, 10 active+undersized, 44 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 22/336 objects degraded (6.548%) 2024-04-16T02:44:14.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:13 smithi153 ceph-mon[87858]: Health check failed: Degraded data redundancy: 22/336 objects degraded (6.548%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T02:44:14.255 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:13 smithi153 podman[107521]: 2024-04-16 02:44:13.827148268 +0000 UTC m=+0.018376106 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:44:14.255 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:13 smithi153 podman[107521]: 2024-04-16 02:44:13.958870649 +0000 UTC m=+0.150098512 container create 4208a44f16fc89c01d74928a9c7460cfd2edd23688c16d77538bb6bc37bcda0a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-87dbfb6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-16T02:44:14.256 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 podman[107521]: 2024-04-16 02:44:14.25330618 +0000 UTC m=+0.444534001 container init 4208a44f16fc89c01d74928a9c7460cfd2edd23688c16d77538bb6bc37bcda0a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-16T02:44:14.522 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 podman[107521]: 2024-04-16 02:44:14.257035246 +0000 UTC m=+0.448263070 container start 4208a44f16fc89c01d74928a9c7460cfd2edd23688c16d77538bb6bc37bcda0a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-16T02:44:14.522 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 podman[107521]: 2024-04-16 02:44:14.336636847 +0000 UTC m=+0.527864677 container attach 4208a44f16fc89c01d74928a9c7460cfd2edd23688c16d77538bb6bc37bcda0a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-87dbfb6) 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate[107535]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate[107535]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 bash[107521]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 bash[107521]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 bash[107521]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 bash[107521]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 bash[107521]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 bash[107521]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 bash[107521]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate[107535]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T02:44:15.023 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate[107535]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T02:44:15.024 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate[107535]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-16T02:44:15.024 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate[107535]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T02:44:15.024 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate[107535]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-16T02:44:15.024 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:14 smithi153 podman[107521]: 2024-04-16 02:44:14.736655464 +0000 UTC m=+0.927883286 container died 4208a44f16fc89c01d74928a9c7460cfd2edd23688c16d77538bb6bc37bcda0a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate, RELEASE=main-87dbfb6, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T02:44:15.772 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:15 smithi153 podman[107666]: 2024-04-16 02:44:15.365549995 +0000 UTC m=+0.622828270 container remove 4208a44f16fc89c01d74928a9c7460cfd2edd23688c16d77538bb6bc37bcda0a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-16T02:44:15.772 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:15 smithi153 podman[107730]: 2024-04-16 02:44:15.48887283 +0000 UTC m=+0.018263811 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T02:44:15.772 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:15 smithi153 podman[107730]: 2024-04-16 02:44:15.608179391 +0000 UTC m=+0.137570348 container create a3f762d46f2455afcd55ec640107288391d9e8816e9f51bb3dc212818b663b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-87dbfb6, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:44:16.121 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:15 smithi153 ceph-mon[87858]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/336 objects degraded (16.071%) 2024-04-16T02:44:16.121 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:16 smithi153 podman[107730]: 2024-04-16 02:44:16.036302295 +0000 UTC m=+0.565693265 container init a3f762d46f2455afcd55ec640107288391d9e8816e9f51bb3dc212818b663b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_CLEAN=True, ceph=True) 2024-04-16T02:44:16.121 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:16 smithi153 podman[107730]: 2024-04-16 02:44:16.039969397 +0000 UTC m=+0.569360350 container start a3f762d46f2455afcd55ec640107288391d9e8816e9f51bb3dc212818b663b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T02:44:16.121 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:16 smithi153 ceph-osd[107748]: -- 172.21.15.153:0/1472550020 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd283a9680 con 0x55bd28373000 2024-04-16T02:44:16.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:15 smithi151 ceph-mon[112233]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/336 objects degraded (16.071%) 2024-04-16T02:44:16.522 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:16 smithi153 bash[107730]: a3f762d46f2455afcd55ec640107288391d9e8816e9f51bb3dc212818b663b8a 2024-04-16T02:44:16.523 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:16 smithi153 systemd[1]: Started Ceph osd.5 for 80d1e958-fb98-11ee-bc8f-c7b262605968. 2024-04-16T02:44:17.010 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:16 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T02:44:16.582+0000 7fb9798307c0 -1 Falling back to public interface 2024-04-16T02:44:17.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:17 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:17.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:17 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:17.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:17 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:17 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:17 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:17.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:17 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:18.470 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:18 smithi153 ceph-mon[87858]: pgmap v162: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/336 objects degraded (16.071%) 2024-04-16T02:44:18.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:18 smithi151 ceph-mon[112233]: pgmap v162: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/336 objects degraded (16.071%) 2024-04-16T02:44:20.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:20.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:20.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:20 smithi151 ceph-mon[112233]: pgmap v163: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/336 objects degraded (16.071%) 2024-04-16T02:44:20.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:20.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:20.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:20 smithi153 ceph-mon[87858]: pgmap v163: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/336 objects degraded (16.071%) 2024-04-16T02:44:21.210 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:21 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T02:44:21.069+0000 7fb9798307c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-16T02:44:21.397 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:21 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:21.397 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:21 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:21.397 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:21 smithi151 ceph-mon[112233]: Health check update: Degraded data redundancy: 54/336 objects degraded (16.071%), 14 pgs degraded (PG_DEGRADED) 2024-04-16T02:44:21.507 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:21 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:21.508 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:21 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:21.508 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:21 smithi153 ceph-mon[87858]: Health check update: Degraded data redundancy: 54/336 objects degraded (16.071%), 14 pgs degraded (PG_DEGRADED) 2024-04-16T02:44:21.772 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:21 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T02:44:21.505+0000 7fb9798307c0 -1 osd.5 75 log_to_monitors true 2024-04-16T02:44:22.452 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:22 smithi153 ceph-mon[87858]: from='osd.5 [v2:172.21.15.153:6816/45452282,v1:172.21.15.153:6817/45452282]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T02:44:22.452 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:22 smithi153 ceph-mon[87858]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T02:44:22.452 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:22 smithi153 ceph-mon[87858]: pgmap v164: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/336 objects degraded (16.071%) 2024-04-16T02:44:22.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:22 smithi151 ceph-mon[112233]: from='osd.5 [v2:172.21.15.153:6816/45452282,v1:172.21.15.153:6817/45452282]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T02:44:22.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:22 smithi151 ceph-mon[112233]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T02:44:22.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:22 smithi151 ceph-mon[112233]: pgmap v164: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 54/336 objects degraded (16.071%) 2024-04-16T02:44:23.522 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 02:44:23 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T02:44:23.238+0000 7fb966c80700 -1 osd.5 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T02:44:23.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:23 smithi153 ceph-mon[87858]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T02:44:23.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:23 smithi153 ceph-mon[87858]: osdmap e78: 6 total, 5 up, 6 in 2024-04-16T02:44:23.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:23 smithi153 ceph-mon[87858]: from='osd.5 [v2:172.21.15.153:6816/45452282,v1:172.21.15.153:6817/45452282]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:44:23.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:23 smithi153 ceph-mon[87858]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:44:23.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:23 smithi153 ceph-mon[87858]: from='osd.5 ' entity='osd.5' 2024-04-16T02:44:23.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:23 smithi151 ceph-mon[112233]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T02:44:23.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:23 smithi151 ceph-mon[112233]: osdmap e78: 6 total, 5 up, 6 in 2024-04-16T02:44:23.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:23 smithi151 ceph-mon[112233]: from='osd.5 [v2:172.21.15.153:6816/45452282,v1:172.21.15.153:6817/45452282]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:44:23.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:23 smithi151 ceph-mon[112233]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-16T02:44:23.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:23 smithi151 ceph-mon[112233]: from='osd.5 ' entity='osd.5' 2024-04-16T02:44:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: pgmap v166: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 54/336 objects degraded (16.071%) 2024-04-16T02:44:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:44:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:25.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-16T02:44:25.248 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: osd.5 [v2:172.21.15.153:6816/45452282,v1:172.21.15.153:6817/45452282] boot 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: osdmap e79: 6 total, 6 up, 6 in 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-16T02:44:25.249 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:24 smithi151 ceph-mon[112233]: osdmap e80: 6 total, 6 up, 6 in 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: pgmap v166: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 54/336 objects degraded (16.071%) 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:25.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-16T02:44:25.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-16T02:44:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T02:44:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-16T02:44:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-16T02:44:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: osd.5 [v2:172.21.15.153:6816/45452282,v1:172.21.15.153:6817/45452282] boot 2024-04-16T02:44:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: osdmap e79: 6 total, 6 up, 6 in 2024-04-16T02:44:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T02:44:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-16T02:44:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:24 smithi153 ceph-mon[87858]: osdmap e80: 6 total, 6 up, 6 in 2024-04-16T02:44:25.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T02:44:25.408+0000 7f7be12de700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T02:44:26.050 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all osd 2024-04-16T02:44:26.050 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: Upgrade: Setting require_osd_release to 19 squid 2024-04-16T02:44:26.051 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-16T02:44:26.051 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-16T02:44:26.051 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-16T02:44:26.051 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2024-04-16T02:44:26.051 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:26.051 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:26.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:44:26.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:26.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: osdmap e81: 6 total, 6 up, 6 in 2024-04-16T02:44:26.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:26.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:26.052 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:26.053 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:26.053 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:26.053 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-16T02:44:26.053 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T02:44:26.053 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T02:44:26.053 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-16T02:44:26.054 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-16T02:44:26.054 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: osdmap e82: 6 total, 6 up, 6 in 2024-04-16T02:44:26.054 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.153:6824/1306847002,v1:172.21.15.153:6825/1306847002] up:boot 2024-04-16T02:44:26.054 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] up:boot 2024-04-16T02:44:26.054 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-16T02:44:26.054 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:44:26.055 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:44:26.175 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all osd 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: Upgrade: Setting require_osd_release to 19 squid 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: osdmap e81: 6 total, 6 up, 6 in 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:26.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: osdmap e82: 6 total, 6 up, 6 in 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.153:6824/1306847002,v1:172.21.15.153:6825/1306847002] up:boot 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] up:boot 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:44:26.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T02:44:27.048 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:26 smithi151 ceph-mon[112233]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-16T02:44:27.048 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:26 smithi151 ceph-mon[112233]: pgmap v171: 65 pgs: 9 active+undersized+degraded, 12 active+undersized, 44 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s; 26/336 objects degraded (7.738%) 2024-04-16T02:44:27.048 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:26 smithi151 ceph-mon[112233]: Health check update: Degraded data redundancy: 26/336 objects degraded (7.738%), 9 pgs degraded (PG_DEGRADED) 2024-04-16T02:44:27.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:26 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6826/4272338874,v1:172.21.15.151:6828/4272338874] up:boot 2024-04-16T02:44:27.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:26 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6827/271070112,v1:172.21.15.151:6829/271070112] up:boot 2024-04-16T02:44:27.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:26 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T02:44:27.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:44:27.049 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:44:27.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:26 smithi153 ceph-mon[87858]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-16T02:44:27.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:26 smithi153 ceph-mon[87858]: pgmap v171: 65 pgs: 9 active+undersized+degraded, 12 active+undersized, 44 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s; 26/336 objects degraded (7.738%) 2024-04-16T02:44:27.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:26 smithi153 ceph-mon[87858]: Health check update: Degraded data redundancy: 26/336 objects degraded (7.738%), 9 pgs degraded (PG_DEGRADED) 2024-04-16T02:44:27.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:26 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6826/4272338874,v1:172.21.15.151:6828/4272338874] up:boot 2024-04-16T02:44:27.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:26 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6827/271070112,v1:172.21.15.151:6829/271070112] up:boot 2024-04-16T02:44:27.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:26 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T02:44:27.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:44:27.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:44:28.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:28 smithi151 ceph-mon[112233]: Upgrade: Updating mds.cephfs.smithi151.bdpnxq 2024-04-16T02:44:28.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:28.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.bdpnxq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:44:28.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:28.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:28 smithi151 ceph-mon[112233]: Deploying daemon mds.cephfs.smithi151.bdpnxq on smithi151 2024-04-16T02:44:28.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:28 smithi151 ceph-mon[112233]: pgmap v172: 65 pgs: 4 active+undersized+degraded, 3 active+undersized, 58 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 10/336 objects degraded (2.976%) 2024-04-16T02:44:28.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:28 smithi153 ceph-mon[87858]: Upgrade: Updating mds.cephfs.smithi151.bdpnxq 2024-04-16T02:44:28.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:28.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.bdpnxq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:44:28.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:28.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:28 smithi153 ceph-mon[87858]: Deploying daemon mds.cephfs.smithi151.bdpnxq on smithi151 2024-04-16T02:44:28.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:28 smithi153 ceph-mon[87858]: pgmap v172: 65 pgs: 4 active+undersized+degraded, 3 active+undersized, 58 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 10/336 objects degraded (2.976%) 2024-04-16T02:44:30.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:30 smithi151 ceph-mon[112233]: osdmap e83: 6 total, 6 up, 6 in 2024-04-16T02:44:30.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:30 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-16T02:44:30.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:30 smithi151 ceph-mon[112233]: pgmap v174: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-16T02:44:30.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:30 smithi153 ceph-mon[87858]: osdmap e83: 6 total, 6 up, 6 in 2024-04-16T02:44:30.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:30 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-16T02:44:30.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:30 smithi153 ceph-mon[87858]: pgmap v174: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-16T02:44:31.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:31 smithi151 ceph-mon[112233]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/336 objects degraded (2.976%), 4 pgs degraded) 2024-04-16T02:44:31.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:31 smithi153 ceph-mon[87858]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/336 objects degraded (2.976%), 4 pgs degraded) 2024-04-16T02:44:32.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.167+0000 7f18aff23700 1 -- 172.21.15.151:0/3765904742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 msgr2=0x7f18a81057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.167+0000 7f18aff23700 1 --2- 172.21.15.151:0/3765904742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 0x7f18a81057c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009910 tx=0x7f18a4009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:32.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.167+0000 7f18aff23700 1 -- 172.21.15.151:0/3765904742 shutdown_connections 2024-04-16T02:44:32.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.167+0000 7f18aff23700 1 --2- 172.21.15.151:0/3765904742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 0x7f18a81057c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.167+0000 7f18aff23700 1 --2- 172.21.15.151:0/3765904742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a8100dc0 0x7f18a81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.167+0000 7f18aff23700 1 -- 172.21.15.151:0/3765904742 >> 172.21.15.151:0/3765904742 conn(0x7f18a80fc650 msgr2=0x7f18a80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:32.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.168+0000 7f18aff23700 1 -- 172.21.15.151:0/3765904742 shutdown_connections 2024-04-16T02:44:32.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.168+0000 7f18aff23700 1 -- 172.21.15.151:0/3765904742 wait complete. 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.168+0000 7f18aff23700 1 Processor -- start 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.168+0000 7f18aff23700 1 -- start start 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18aff23700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a8100dc0 0x7f18a819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18aff23700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 0x7f18a819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18aff23700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a819d240 con 0x7f18a8101770 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18aff23700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a81966f0 con 0x7f18a8100dc0 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18ad4be700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 0x7f18a819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18ad4be700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 0x7f18a819cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53378/0 (socket says 172.21.15.151:53378) 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18ad4be700 1 -- 172.21.15.151:0/2197642886 learned_addr learned my addr 172.21.15.151:0/2197642886 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18adcbf700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a8100dc0 0x7f18a819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18ad4be700 1 -- 172.21.15.151:0/2197642886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a8100dc0 msgr2=0x7f18a819c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18ad4be700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a8100dc0 0x7f18a819c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18ad4be700 1 -- 172.21.15.151:0/2197642886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a40095f0 con 0x7f18a8101770 2024-04-16T02:44:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18adcbf700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a8100dc0 0x7f18a819c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:44:32.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18ad4be700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 0x7f18a819cbb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f18a40092e0 tx=0x7f18a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:32.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f189effd700 1 -- 172.21.15.151:0/2197642886 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a401d070 con 0x7f18a8101770 2024-04-16T02:44:32.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f189effd700 1 -- 172.21.15.151:0/2197642886 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18a400f460 con 0x7f18a8101770 2024-04-16T02:44:32.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a8196970 con 0x7f18a8101770 2024-04-16T02:44:32.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f189effd700 1 -- 172.21.15.151:0/2197642886 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a40175f0 con 0x7f18a8101770 2024-04-16T02:44:32.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.169+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a8196e60 con 0x7f18a8101770 2024-04-16T02:44:32.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.170+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a8109110 con 0x7f18a8101770 2024-04-16T02:44:32.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.171+0000 7f189effd700 1 -- 172.21.15.151:0/2197642886 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f18a400f5d0 con 0x7f18a8101770 2024-04-16T02:44:32.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.171+0000 7f189effd700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f189406ecc0 0x7f1894071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:32.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.172+0000 7f189effd700 1 -- 172.21.15.151:0/2197642886 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f18a40903c0 con 0x7f18a8101770 2024-04-16T02:44:32.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.172+0000 7f18adcbf700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f189406ecc0 0x7f1894071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:32.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.172+0000 7f18adcbf700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f189406ecc0 0x7f1894071180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18980099d0 tx=0x7f1898008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:32.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.176+0000 7f189effd700 1 -- 172.21.15.151:0/2197642886 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f18a405cca0 con 0x7f18a8101770 2024-04-16T02:44:32.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.400+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f18a802cce0 con 0x7f189406ecc0 2024-04-16T02:44:32.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.402+0000 7f189effd700 1 -- 172.21.15.151:0/2197642886 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f18a802cce0 con 0x7f189406ecc0 2024-04-16T02:44:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f189406ecc0 msgr2=0x7f1894071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f189406ecc0 0x7f1894071180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18980099d0 tx=0x7f1898008040 comp rx=0 tx=0).stop 2024-04-16T02:44:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 msgr2=0x7f18a819cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 0x7f18a819cbb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f18a40092e0 tx=0x7f18a400be20 comp rx=0 tx=0).stop 2024-04-16T02:44:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 shutdown_connections 2024-04-16T02:44:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f189406ecc0 0x7f1894071180 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a8100dc0 0x7f18a819c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 --2- 172.21.15.151:0/2197642886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18a8101770 0x7f18a819cbb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 >> 172.21.15.151:0/2197642886 conn(0x7f18a80fc650 msgr2=0x7f18a80fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 shutdown_connections 2024-04-16T02:44:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.404+0000 7f18aff23700 1 -- 172.21.15.151:0/2197642886 wait complete. 2024-04-16T02:44:32.419 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:44:32.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:32 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:44:32.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:32 smithi151 ceph-mon[112233]: pgmap v175: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.511+0000 7f3a3d811700 1 -- 172.21.15.151:0/851200988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 msgr2=0x7f3a38100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.511+0000 7f3a3d811700 1 --2- 172.21.15.151:0/851200988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 0x7f3a38100050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009910 tx=0x7f3a2c009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.511+0000 7f3a3d811700 1 -- 172.21.15.151:0/851200988 shutdown_connections 2024-04-16T02:44:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.511+0000 7f3a3d811700 1 --2- 172.21.15.151:0/851200988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 0x7f3a38100050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.511+0000 7f3a3d811700 1 --2- 172.21.15.151:0/851200988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a38105a40 0x7f3a38105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.511+0000 7f3a3d811700 1 -- 172.21.15.151:0/851200988 >> 172.21.15.151:0/851200988 conn(0x7f3a380755a0 msgr2=0x7f3a380759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.511+0000 7f3a3d811700 1 -- 172.21.15.151:0/851200988 shutdown_connections 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.511+0000 7f3a3d811700 1 -- 172.21.15.151:0/851200988 wait complete. 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a3d811700 1 Processor -- start 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a3d811700 1 -- start start 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a3d811700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 0x7f3a381976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a3d811700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a38105a40 0x7f3a38197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a3d811700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38198200 con 0x7f3a380ffbd0 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a3d811700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38198340 con 0x7f3a38105a40 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a36ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 0x7f3a381976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a36ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 0x7f3a381976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53410/0 (socket says 172.21.15.151:53410) 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a36ffd700 1 -- 172.21.15.151:0/2699040517 learned_addr learned my addr 172.21.15.151:0/2699040517 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a367fc700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a38105a40 0x7f3a38197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a36ffd700 1 -- 172.21.15.151:0/2699040517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a38105a40 msgr2=0x7f3a38197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a36ffd700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a38105a40 0x7f3a38197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.512+0000 7f3a36ffd700 1 -- 172.21.15.151:0/2699040517 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c0095f0 con 0x7f3a380ffbd0 2024-04-16T02:44:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.513+0000 7f3a36ffd700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 0x7f3a381976a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800b4b0 tx=0x7f3a2800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.513+0000 7f3a3c80f700 1 -- 172.21.15.151:0/2699040517 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a28011800 con 0x7f3a380ffbd0 2024-04-16T02:44:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.513+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a381a0710 con 0x7f3a380ffbd0 2024-04-16T02:44:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.513+0000 7f3a3c80f700 1 -- 172.21.15.151:0/2699040517 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a28011e20 con 0x7f3a380ffbd0 2024-04-16T02:44:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.513+0000 7f3a3c80f700 1 -- 172.21.15.151:0/2699040517 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a2800f2e0 con 0x7f3a380ffbd0 2024-04-16T02:44:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.513+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a381a0bd0 con 0x7f3a380ffbd0 2024-04-16T02:44:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.514+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a38108060 con 0x7f3a380ffbd0 2024-04-16T02:44:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.514+0000 7f3a3c80f700 1 -- 172.21.15.151:0/2699040517 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3a28011960 con 0x7f3a380ffbd0 2024-04-16T02:44:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.515+0000 7f3a3c80f700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3a2006ecc0 0x7f3a20071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.515+0000 7f3a3c80f700 1 -- 172.21.15.151:0/2699040517 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3a2808e8b0 con 0x7f3a380ffbd0 2024-04-16T02:44:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.515+0000 7f3a367fc700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3a2006ecc0 0x7f3a20071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.515+0000 7f3a367fc700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3a2006ecc0 0x7f3a20071180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a38068cc0 tx=0x7f3a2c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.520+0000 7f3a3c80f700 1 -- 172.21.15.151:0/2699040517 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3a280576e0 con 0x7f3a380ffbd0 2024-04-16T02:44:32.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.739+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a3802cce0 con 0x7f3a2006ecc0 2024-04-16T02:44:32.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.740+0000 7f3a3c80f700 1 -- 172.21.15.151:0/2699040517 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3a3802cce0 con 0x7f3a2006ecc0 2024-04-16T02:44:32.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3a2006ecc0 msgr2=0x7f3a20071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3a2006ecc0 0x7f3a20071180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a38068cc0 tx=0x7f3a2c01a040 comp rx=0 tx=0).stop 2024-04-16T02:44:32.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 msgr2=0x7f3a381976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 0x7f3a381976a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800b4b0 tx=0x7f3a2800b7c0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 shutdown_connections 2024-04-16T02:44:32.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3a2006ecc0 0x7f3a20071180 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a380ffbd0 0x7f3a381976a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 --2- 172.21.15.151:0/2699040517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a38105a40 0x7f3a38197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 >> 172.21.15.151:0/2699040517 conn(0x7f3a380755a0 msgr2=0x7f3a380fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:32.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 shutdown_connections 2024-04-16T02:44:32.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.743+0000 7f3a3d811700 1 -- 172.21.15.151:0/2699040517 wait complete. 2024-04-16T02:44:32.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:32 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:44:32.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:32 smithi153 ceph-mon[87858]: pgmap v175: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:32.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.848+0000 7fd8ab089700 1 -- 172.21.15.151:0/1962121234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ffec0 msgr2=0x7fd8a4105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.848+0000 7fd8ab089700 1 --2- 172.21.15.151:0/1962121234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ffec0 0x7fd8a4105850 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd890009910 tx=0x7fd890009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:32.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.849+0000 7fd8ab089700 1 -- 172.21.15.151:0/1962121234 shutdown_connections 2024-04-16T02:44:32.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.849+0000 7fd8ab089700 1 --2- 172.21.15.151:0/1962121234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ffec0 0x7fd8a4105850 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.849+0000 7fd8ab089700 1 --2- 172.21.15.151:0/1962121234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a40ff510 0x7fd8a40ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.849+0000 7fd8ab089700 1 -- 172.21.15.151:0/1962121234 >> 172.21.15.151:0/1962121234 conn(0x7fd8a4074d80 msgr2=0x7fd8a4075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:32.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.849+0000 7fd8ab089700 1 -- 172.21.15.151:0/1962121234 shutdown_connections 2024-04-16T02:44:32.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.849+0000 7fd8ab089700 1 -- 172.21.15.151:0/1962121234 wait complete. 2024-04-16T02:44:32.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.849+0000 7fd8ab089700 1 Processor -- start 2024-04-16T02:44:32.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.849+0000 7fd8ab089700 1 -- start start 2024-04-16T02:44:32.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8ab089700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ff510 0x7fd8a419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:32.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8ab089700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a40ffec0 0x7fd8a419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:32.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8ab089700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a419d0e0 con 0x7fd8a40ff510 2024-04-16T02:44:32.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8ab089700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a4196590 con 0x7fd8a40ffec0 2024-04-16T02:44:32.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a8e25700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ff510 0x7fd8a419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:32.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a8e25700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ff510 0x7fd8a419c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53418/0 (socket says 172.21.15.151:53418) 2024-04-16T02:44:32.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a8e25700 1 -- 172.21.15.151:0/4108241307 learned_addr learned my addr 172.21.15.151:0/4108241307 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a3fff700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a40ffec0 0x7fd8a419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a8e25700 1 -- 172.21.15.151:0/4108241307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a40ffec0 msgr2=0x7fd8a419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a8e25700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a40ffec0 0x7fd8a419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a8e25700 1 -- 172.21.15.151:0/4108241307 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8900095f0 con 0x7fd8a40ff510 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a8e25700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ff510 0x7fd8a419c510 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd89800d8d0 tx=0x7fd89800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a1ffb700 1 -- 172.21.15.151:0/4108241307 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8980098a0 con 0x7fd8a40ff510 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a1ffb700 1 -- 172.21.15.151:0/4108241307 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd898010460 con 0x7fd8a40ff510 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a4196870 con 0x7fd8a40ff510 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.850+0000 7fd8a1ffb700 1 -- 172.21.15.151:0/4108241307 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd89800f5f0 con 0x7fd8a40ff510 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.851+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a4196d90 con 0x7fd8a40ff510 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.852+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a4109180 con 0x7fd8a40ff510 2024-04-16T02:44:32.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.852+0000 7fd8a1ffb700 1 -- 172.21.15.151:0/4108241307 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd898009a00 con 0x7fd8a40ff510 2024-04-16T02:44:32.858 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.853+0000 7fd8a1ffb700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd89406ecc0 0x7fd894071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:32.858 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.853+0000 7fd8a1ffb700 1 -- 172.21.15.151:0/4108241307 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd89808ec30 con 0x7fd8a40ff510 2024-04-16T02:44:32.858 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.853+0000 7fd8a3fff700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd89406ecc0 0x7fd894071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:32.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.853+0000 7fd8a3fff700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd89406ecc0 0x7fd894071180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd890006010 tx=0x7fd89001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:32.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:32.858+0000 7fd8a1ffb700 1 -- 172.21.15.151:0/4108241307 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd89805b590 con 0x7fd8a40ff510 2024-04-16T02:44:33.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.067+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8a4197030 con 0x7fd89406ecc0 2024-04-16T02:44:33.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.078+0000 7fd8a1ffb700 1 -- 172.21.15.151:0/4108241307 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd8a4197030 con 0x7fd89406ecc0 2024-04-16T02:44:33.082 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:44:33.082 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (6m) 115s ago 17m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:44:33.082 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (17m) 115s ago 17m 8740k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:44:33.082 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (16m) 13s ago 16m 8804k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (3m) 115s ago 17m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (3m) 13s ago 16m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (5m) 115s ago 17m 81.1M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (11m) 115s ago 11m 120M - 18.2.1 5be31c24972a 9e440473d11f 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (11m) 115s ago 11m 130M - 18.2.1 5be31c24972a 884d7ff26476 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (11m) 13s ago 11m 58.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (11m) 13s ago 11m 67.5M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (8m) 115s ago 18m 505M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (7m) 13s ago 16m 452M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (4m) 115s ago 18m 48.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (4m) 13s ago 16m 43.5M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (6m) 115s ago 17m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (6m) 13s ago 16m 16.0M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (3m) 115s ago 15m 88.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (2m) 115s ago 15m 96.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:44:33.083 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (2m) 115s ago 14m 12.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:44:33.084 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (85s) 13s ago 14m 109M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:44:33.084 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (51s) 13s ago 13m 90.8M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:44:33.084 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (17s) 13s ago 13m 12.6M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:44:33.084 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (6m) 115s ago 16m 73.7M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd89406ecc0 msgr2=0x7fd894071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd89406ecc0 0x7fd894071180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd890006010 tx=0x7fd89001a040 comp rx=0 tx=0).stop 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ff510 msgr2=0x7fd8a419c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ff510 0x7fd8a419c510 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd89800d8d0 tx=0x7fd89800dbe0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 shutdown_connections 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd89406ecc0 0x7fd894071180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8a40ff510 0x7fd8a419c510 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 --2- 172.21.15.151:0/4108241307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a40ffec0 0x7fd8a419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 >> 172.21.15.151:0/4108241307 conn(0x7fd8a4074d80 msgr2=0x7fd8a40feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 shutdown_connections 2024-04-16T02:44:33.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.082+0000 7fd8ab089700 1 -- 172.21.15.151:0/4108241307 wait complete. 2024-04-16T02:44:33.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.187+0000 7f475655a700 1 -- 172.21.15.151:0/4245782494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ff510 msgr2=0x7f47500ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.187+0000 7f475655a700 1 --2- 172.21.15.151:0/4245782494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ff510 0x7f47500ff8f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f474000aa20 tx=0x7f474000ad30 comp rx=0 tx=0).stop 2024-04-16T02:44:33.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.187+0000 7f475655a700 1 -- 172.21.15.151:0/4245782494 shutdown_connections 2024-04-16T02:44:33.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.187+0000 7f475655a700 1 --2- 172.21.15.151:0/4245782494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47500ffec0 0x7f4750103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.187+0000 7f475655a700 1 --2- 172.21.15.151:0/4245782494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ff510 0x7f47500ff8f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.187+0000 7f475655a700 1 -- 172.21.15.151:0/4245782494 >> 172.21.15.151:0/4245782494 conn(0x7f4750074d80 msgr2=0x7f4750075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:33.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.187+0000 7f475655a700 1 -- 172.21.15.151:0/4245782494 shutdown_connections 2024-04-16T02:44:33.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.187+0000 7f475655a700 1 -- 172.21.15.151:0/4245782494 wait complete. 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.187+0000 7f475655a700 1 Processor -- start 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f475655a700 1 -- start start 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f475655a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47500ff510 0x7f4750194550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f475655a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ffec0 0x7f4750194a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f475655a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4750195170 con 0x7f47500ffec0 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f475655a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4750198f00 con 0x7f47500ff510 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f474f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ffec0 0x7f4750194a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f474f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ffec0 0x7f4750194a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53442/0 (socket says 172.21.15.151:53442) 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f474f7fe700 1 -- 172.21.15.151:0/31829782 learned_addr learned my addr 172.21.15.151:0/31829782 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f474ffff700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47500ff510 0x7f4750194550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:33.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f474f7fe700 1 -- 172.21.15.151:0/31829782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47500ff510 msgr2=0x7f4750194550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f474f7fe700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47500ff510 0x7f4750194550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.188+0000 7f474f7fe700 1 -- 172.21.15.151:0/31829782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474000a700 con 0x7f47500ffec0 2024-04-16T02:44:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.189+0000 7f474f7fe700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ffec0 0x7f4750194a90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f474400d900 tx=0x7f474400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.189+0000 7f474d7fa700 1 -- 172.21.15.151:0/31829782 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4744009900 con 0x7f47500ffec0 2024-04-16T02:44:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.189+0000 7f474d7fa700 1 -- 172.21.15.151:0/31829782 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4744010460 con 0x7f47500ffec0 2024-04-16T02:44:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.189+0000 7f474d7fa700 1 -- 172.21.15.151:0/31829782 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f474400f5f0 con 0x7f47500ffec0 2024-04-16T02:44:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.189+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47501991e0 con 0x7f47500ffec0 2024-04-16T02:44:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.189+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4750199730 con 0x7f47500ffec0 2024-04-16T02:44:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.190+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47501077d0 con 0x7f47500ffec0 2024-04-16T02:44:33.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.190+0000 7f474d7fa700 1 -- 172.21.15.151:0/31829782 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f47440105d0 con 0x7f47500ffec0 2024-04-16T02:44:33.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.191+0000 7f474d7fa700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f473806ecc0 0x7f4738071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:33.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.191+0000 7f474d7fa700 1 -- 172.21.15.151:0/31829782 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f474408ea30 con 0x7f47500ffec0 2024-04-16T02:44:33.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.191+0000 7f474ffff700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f473806ecc0 0x7f4738071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:33.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.192+0000 7f474ffff700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f473806ecc0 0x7f4738071180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f474000afd0 tx=0x7f4740005de0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:33.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.196+0000 7f474d7fa700 1 -- 172.21.15.151:0/31829782 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f47440577e0 con 0x7f47500ffec0 2024-04-16T02:44:33.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:33 smithi151 ceph-mon[112233]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:33.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:33 smithi151 ceph-mon[112233]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:33.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.498+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f475004ea90 con 0x7f47500ffec0 2024-04-16T02:44:33.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.499+0000 7f474d7fa700 1 -- 172.21.15.151:0/31829782 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+701 (secure 0 0 0) 0x7f474405aea0 con 0x7f47500ffec0 2024-04-16T02:44:33.503 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:44:33.503 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:44:33.503 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:44:33.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:44:33.503 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:44:33.503 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:44:33.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:44:33.503 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 10 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:44:33.504 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f473806ecc0 msgr2=0x7f4738071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f473806ecc0 0x7f4738071180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f474000afd0 tx=0x7f4740005de0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ffec0 msgr2=0x7f4750194a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ffec0 0x7f4750194a90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f474400d900 tx=0x7f474400dc10 comp rx=0 tx=0).stop 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 shutdown_connections 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f473806ecc0 0x7f4738071180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47500ff510 0x7f4750194550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 --2- 172.21.15.151:0/31829782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47500ffec0 0x7f4750194a90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 >> 172.21.15.151:0/31829782 conn(0x7f4750074d80 msgr2=0x7f47500fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 shutdown_connections 2024-04-16T02:44:33.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.502+0000 7f475655a700 1 -- 172.21.15.151:0/31829782 wait complete. 2024-04-16T02:44:33.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.608+0000 7fe21ae46700 1 -- 172.21.15.151:0/822687653 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2140ffbd0 msgr2=0x7fe214100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.608+0000 7fe21ae46700 1 --2- 172.21.15.151:0/822687653 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2140ffbd0 0x7fe214100050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe200009910 tx=0x7fe200009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:33.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.608+0000 7fe21ae46700 1 -- 172.21.15.151:0/822687653 shutdown_connections 2024-04-16T02:44:33.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.608+0000 7fe21ae46700 1 --2- 172.21.15.151:0/822687653 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2140ffbd0 0x7fe214100050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.608+0000 7fe21ae46700 1 --2- 172.21.15.151:0/822687653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe214105a40 0x7fe214105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.608+0000 7fe21ae46700 1 -- 172.21.15.151:0/822687653 >> 172.21.15.151:0/822687653 conn(0x7fe2140755a0 msgr2=0x7fe2140759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:33.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.608+0000 7fe21ae46700 1 -- 172.21.15.151:0/822687653 shutdown_connections 2024-04-16T02:44:33.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.608+0000 7fe21ae46700 1 -- 172.21.15.151:0/822687653 wait complete. 2024-04-16T02:44:33.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.609+0000 7fe21ae46700 1 Processor -- start 2024-04-16T02:44:33.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.609+0000 7fe21ae46700 1 -- start start 2024-04-16T02:44:33.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.609+0000 7fe21ae46700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2140ffbd0 0x7fe214197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:33.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.609+0000 7fe21ae46700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe214105a40 0x7fe214197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:33.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.609+0000 7fe21ae46700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe214198190 con 0x7fe214105a40 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.609+0000 7fe21ae46700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2141982d0 con 0x7fe2140ffbd0 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.609+0000 7fe213fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe214105a40 0x7fe214197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.609+0000 7fe213fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe214105a40 0x7fe214197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53450/0 (socket says 172.21.15.151:53450) 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.609+0000 7fe213fff700 1 -- 172.21.15.151:0/763076909 learned_addr learned my addr 172.21.15.151:0/763076909 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe218be2700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2140ffbd0 0x7fe214197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe213fff700 1 -- 172.21.15.151:0/763076909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2140ffbd0 msgr2=0x7fe214197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe213fff700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2140ffbd0 0x7fe214197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe213fff700 1 -- 172.21.15.151:0/763076909 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2000095f0 con 0x7fe214105a40 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe213fff700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe214105a40 0x7fe214197bc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe200006010 tx=0x7fe20000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe211ffb700 1 -- 172.21.15.151:0/763076909 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20001d070 con 0x7fe214105a40 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2141a06f0 con 0x7fe214105a40 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe211ffb700 1 -- 172.21.15.151:0/763076909 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe200004fb0 con 0x7fe214105a40 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe211ffb700 1 -- 172.21.15.151:0/763076909 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe200022ea0 con 0x7fe214105a40 2024-04-16T02:44:33.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.610+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2141a0aa0 con 0x7fe214105a40 2024-04-16T02:44:33.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.611+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe214108060 con 0x7fe214105a40 2024-04-16T02:44:33.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.612+0000 7fe211ffb700 1 -- 172.21.15.151:0/763076909 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe200022470 con 0x7fe214105a40 2024-04-16T02:44:33.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.612+0000 7fe211ffb700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe20406ecc0 0x7fe204071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:33.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.612+0000 7fe211ffb700 1 -- 172.21.15.151:0/763076909 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe2000903f0 con 0x7fe214105a40 2024-04-16T02:44:33.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.612+0000 7fe218be2700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe20406ecc0 0x7fe204071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:33.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.613+0000 7fe218be2700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe20406ecc0 0x7fe204071180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe208006fd0 tx=0x7fe208008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:33.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.618+0000 7fe211ffb700 1 -- 172.21.15.151:0/763076909 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe200059250 con 0x7fe214105a40 2024-04-16T02:44:33.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:33 smithi153 ceph-mon[87858]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:33.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:33 smithi153 ceph-mon[87858]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:33.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.884+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe21404ea90 con 0x7fe214105a40 2024-04-16T02:44:33.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.884+0000 7fe211ffb700 1 -- 172.21.15.151:0/763076909 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1695 (secure 0 0 0) 0x7fe2000276b0 con 0x7fe214105a40 2024-04-16T02:44:33.888 INFO:teuthology.orchestra.run.smithi151.stdout:e26 2024-04-16T02:44:33.888 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:44:33.889 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:44:33.890 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1306847002,v1:172.21.15.153:6825/1306847002] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:44:33.891 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:44255} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/271070112,v1:172.21.15.151:6829/271070112] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:44:33.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe20406ecc0 msgr2=0x7fe204071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe20406ecc0 0x7fe204071180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe208006fd0 tx=0x7fe208008040 comp rx=0 tx=0).stop 2024-04-16T02:44:33.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe214105a40 msgr2=0x7fe214197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:33.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe214105a40 0x7fe214197bc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe200006010 tx=0x7fe20000be20 comp rx=0 tx=0).stop 2024-04-16T02:44:33.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 shutdown_connections 2024-04-16T02:44:33.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe20406ecc0 0x7fe204071180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2140ffbd0 0x7fe214197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 --2- 172.21.15.151:0/763076909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe214105a40 0x7fe214197bc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 >> 172.21.15.151:0/763076909 conn(0x7fe2140755a0 msgr2=0x7fe2140fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 shutdown_connections 2024-04-16T02:44:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.887+0000 7fe21ae46700 1 -- 172.21.15.151:0/763076909 wait complete. 2024-04-16T02:44:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 26 2024-04-16T02:44:34.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.995+0000 7f06bf30a700 1 -- 172.21.15.151:0/2570865914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 msgr2=0x7f06b8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:34.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.995+0000 7f06bf30a700 1 --2- 172.21.15.151:0/2570865914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 0x7f06b8100050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009910 tx=0x7f06b4009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:34.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.995+0000 7f06bf30a700 1 -- 172.21.15.151:0/2570865914 shutdown_connections 2024-04-16T02:44:34.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.995+0000 7f06bf30a700 1 --2- 172.21.15.151:0/2570865914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 0x7f06b8100050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.995+0000 7f06bf30a700 1 --2- 172.21.15.151:0/2570865914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8105a40 0x7f06b8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.995+0000 7f06bf30a700 1 -- 172.21.15.151:0/2570865914 >> 172.21.15.151:0/2570865914 conn(0x7f06b80755a0 msgr2=0x7f06b80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:34.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.995+0000 7f06bf30a700 1 -- 172.21.15.151:0/2570865914 shutdown_connections 2024-04-16T02:44:34.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.995+0000 7f06bf30a700 1 -- 172.21.15.151:0/2570865914 wait complete. 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bf30a700 1 Processor -- start 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bf30a700 1 -- start start 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bf30a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 0x7f06b819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bf30a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8105a40 0x7f06b819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bf30a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b819d0d0 con 0x7f06b80ffbd0 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bf30a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b8196580 con 0x7f06b8105a40 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bd0a6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 0x7f06b819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bd0a6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 0x7f06b819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53458/0 (socket says 172.21.15.151:53458) 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bd0a6700 1 -- 172.21.15.151:0/4249787099 learned_addr learned my addr 172.21.15.151:0/4249787099 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bc8a5700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8105a40 0x7f06b819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bd0a6700 1 -- 172.21.15.151:0/4249787099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8105a40 msgr2=0x7f06b819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:34.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bd0a6700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8105a40 0x7f06b819ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.996+0000 7f06bd0a6700 1 -- 172.21.15.151:0/4249787099 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40095f0 con 0x7f06b80ffbd0 2024-04-16T02:44:34.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.997+0000 7f06bd0a6700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 0x7f06b819c500 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00d8d0 tx=0x7f06ac00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:34.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.997+0000 7f06aa7fc700 1 -- 172.21.15.151:0/4249787099 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06ac0098a0 con 0x7f06b80ffbd0 2024-04-16T02:44:34.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.997+0000 7f06aa7fc700 1 -- 172.21.15.151:0/4249787099 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06ac010460 con 0x7f06b80ffbd0 2024-04-16T02:44:34.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.997+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b8196860 con 0x7f06b80ffbd0 2024-04-16T02:44:34.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.997+0000 7f06aa7fc700 1 -- 172.21.15.151:0/4249787099 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06ac00f5f0 con 0x7f06b80ffbd0 2024-04-16T02:44:34.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.997+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b8196db0 con 0x7f06b80ffbd0 2024-04-16T02:44:34.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.998+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b8108fe0 con 0x7f06b80ffbd0 2024-04-16T02:44:34.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.998+0000 7f06aa7fc700 1 -- 172.21.15.151:0/4249787099 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f06ac009a00 con 0x7f06b80ffbd0 2024-04-16T02:44:34.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.999+0000 7f06aa7fc700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f06a406ecc0 0x7f06a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:34.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.999+0000 7f06aa7fc700 1 -- 172.21.15.151:0/4249787099 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f06ac08ed30 con 0x7f06b80ffbd0 2024-04-16T02:44:34.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.999+0000 7f06bc8a5700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f06a406ecc0 0x7f06a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:34.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:33.999+0000 7f06bc8a5700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f06a406ecc0 0x7f06a4071180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f06b8104d90 tx=0x7f06b401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:34.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.004+0000 7f06aa7fc700 1 -- 172.21.15.151:0/4249787099 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f06ac05b610 con 0x7f06b80ffbd0 2024-04-16T02:44:34.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.260+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f06b8197050 con 0x7f06a406ecc0 2024-04-16T02:44:34.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.262+0000 7f06aa7fc700 1 -- 172.21.15.151:0/4249787099 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f06b8197050 con 0x7f06a406ecc0 2024-04-16T02:44:34.266 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "12/23 daemons upgraded", 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:44:34.267 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:44:34.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f06a406ecc0 msgr2=0x7f06a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:34.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f06a406ecc0 0x7f06a4071180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f06b8104d90 tx=0x7f06b401a040 comp rx=0 tx=0).stop 2024-04-16T02:44:34.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 msgr2=0x7f06b819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:34.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 0x7f06b819c500 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00d8d0 tx=0x7f06ac00dbe0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 shutdown_connections 2024-04-16T02:44:34.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f06a406ecc0 0x7f06a4071180 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f06b80ffbd0 0x7f06b819c500 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 --2- 172.21.15.151:0/4249787099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8105a40 0x7f06b819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 >> 172.21.15.151:0/4249787099 conn(0x7f06b80755a0 msgr2=0x7f06b8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:34.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 shutdown_connections 2024-04-16T02:44:34.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.265+0000 7f06bf30a700 1 -- 172.21.15.151:0/4249787099 wait complete. 2024-04-16T02:44:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.372+0000 7f68584bf700 1 -- 172.21.15.151:0/3628415217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 msgr2=0x7f6850104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.372+0000 7f68584bf700 1 --2- 172.21.15.151:0/3628415217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 0x7f6850104ec0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f684c009910 tx=0x7f684c009c20 comp rx=0 tx=0).stop 2024-04-16T02:44:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.372+0000 7f68584bf700 1 -- 172.21.15.151:0/3628415217 shutdown_connections 2024-04-16T02:44:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.372+0000 7f68584bf700 1 --2- 172.21.15.151:0/3628415217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68501003e0 0x7f6850100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.372+0000 7f68584bf700 1 --2- 172.21.15.151:0/3628415217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 0x7f6850104ec0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.372+0000 7f68584bf700 1 -- 172.21.15.151:0/3628415217 >> 172.21.15.151:0/3628415217 conn(0x7f6850074df0 msgr2=0x7f6850075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.372+0000 7f68584bf700 1 -- 172.21.15.151:0/3628415217 shutdown_connections 2024-04-16T02:44:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.372+0000 7f68584bf700 1 -- 172.21.15.151:0/3628415217 wait complete. 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f68584bf700 1 Processor -- start 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f68584bf700 1 -- start start 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f68584bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68501003e0 0x7f68501017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f68584bf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 0x7f6850101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f68584bf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6850102260 con 0x7f6850104ae0 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f68584bf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68501023a0 con 0x7f68501003e0 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f6855a5a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 0x7f6850101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f6855a5a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 0x7f6850101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53486/0 (socket says 172.21.15.151:53486) 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f6855a5a700 1 -- 172.21.15.151:0/1213359337 learned_addr learned my addr 172.21.15.151:0/1213359337 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.373+0000 7f685625b700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68501003e0 0x7f68501017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.374+0000 7f6855a5a700 1 -- 172.21.15.151:0/1213359337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68501003e0 msgr2=0x7f68501017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.374+0000 7f6855a5a700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68501003e0 0x7f68501017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.374+0000 7f6855a5a700 1 -- 172.21.15.151:0/1213359337 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c0095f0 con 0x7f6850104ae0 2024-04-16T02:44:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.374+0000 7f6855a5a700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 0x7f6850101d20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f684000d8d0 tx=0x7f684000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.374+0000 7f68477fe700 1 -- 172.21.15.151:0/1213359337 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68400098a0 con 0x7f6850104ae0 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.374+0000 7f68477fe700 1 -- 172.21.15.151:0/1213359337 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6840010460 con 0x7f6850104ae0 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.374+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68501a2a00 con 0x7f6850104ae0 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.374+0000 7f68477fe700 1 -- 172.21.15.151:0/1213359337 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f684000f5f0 con 0x7f6850104ae0 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.374+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68501a2e10 con 0x7f6850104ae0 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.375+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68501090e0 con 0x7f6850104ae0 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.376+0000 7f68477fe700 1 -- 172.21.15.151:0/1213359337 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6840009a00 con 0x7f6850104ae0 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.377+0000 7f68477fe700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f683c06ecc0 0x7f683c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.377+0000 7f68477fe700 1 -- 172.21.15.151:0/1213359337 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f684008ed30 con 0x7f6850104ae0 2024-04-16T02:44:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.377+0000 7f685625b700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f683c06ecc0 0x7f683c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:44:34.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.377+0000 7f685625b700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f683c06ecc0 0x7f683c071180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f684c00b5c0 tx=0x7f684c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:44:34.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.382+0000 7f68477fe700 1 -- 172.21.15.151:0/1213359337 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6840057ae0 con 0x7f6850104ae0 2024-04-16T02:44:34.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:34 smithi151 ceph-mon[112233]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:34.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:34 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/31829782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:34.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:34 smithi151 ceph-mon[112233]: pgmap v176: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:34.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:34 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/763076909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:44:34.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.682+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6850102560 con 0x7f6850104ae0 2024-04-16T02:44:34.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.682+0000 7f68477fe700 1 -- 172.21.15.151:0/1213359337 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6850102560 con 0x7f6850104ae0 2024-04-16T02:44:34.686 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:44:34.686 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:44:34.686 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:44:34.686 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:44:34.686 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:44:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f683c06ecc0 msgr2=0x7f683c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f683c06ecc0 0x7f683c071180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f684c00b5c0 tx=0x7f684c01a040 comp rx=0 tx=0).stop 2024-04-16T02:44:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 msgr2=0x7f6850101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:44:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 0x7f6850101d20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f684000d8d0 tx=0x7f684000dbe0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 shutdown_connections 2024-04-16T02:44:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f683c06ecc0 0x7f683c071180 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68501003e0 0x7f68501017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 --2- 172.21.15.151:0/1213359337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6850104ae0 0x7f6850101d20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:44:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 >> 172.21.15.151:0/1213359337 conn(0x7f6850074df0 msgr2=0x7f68500ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:44:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 shutdown_connections 2024-04-16T02:44:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:44:34.684+0000 7f68584bf700 1 -- 172.21.15.151:0/1213359337 wait complete. 2024-04-16T02:44:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:34 smithi153 ceph-mon[87858]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:34 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/31829782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:34 smithi153 ceph-mon[87858]: pgmap v176: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:34.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:34 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/763076909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:44:35.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:35 smithi151 ceph-mon[112233]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:35.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:35 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1213359337' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:44:35.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:35 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:35.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:35 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:35.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:35 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:35 smithi153 ceph-mon[87858]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:44:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:35 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1213359337' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:44:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:35 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:35.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:35 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:35.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:35 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:36.621 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:36 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:boot 2024-04-16T02:44:36.621 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:36 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T02:44:36.621 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:36 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:44:36.621 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:36 smithi151 ceph-mon[112233]: pgmap v177: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:36.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:36 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:boot 2024-04-16T02:44:36.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:36 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T02:44:36.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:36 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T02:44:36.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:36 smithi153 ceph-mon[87858]: pgmap v177: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:37.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:37 smithi151 ceph-mon[112233]: pgmap v178: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-16T02:44:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:37 smithi153 ceph-mon[87858]: pgmap v178: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-16T02:44:40.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:40.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:44:40.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:44:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:41.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:40 smithi151 ceph-mon[112233]: pgmap v179: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 97 B/s rd, 0 op/s 2024-04-16T02:44:41.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:40 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:41.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:40 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:40 smithi153 ceph-mon[87858]: pgmap v179: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 97 B/s rd, 0 op/s 2024-04-16T02:44:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:40 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:41.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:40 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:42.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:42 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:42.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:42 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:42.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:42 smithi151 ceph-mon[112233]: pgmap v180: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:42.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:42 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:42.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:42 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:42.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:42 smithi153 ceph-mon[87858]: pgmap v180: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:44.186 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:43 smithi151 ceph-mon[112233]: pgmap v181: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:44.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:43 smithi153 ceph-mon[87858]: pgmap v181: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:46.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:46.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:46.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:46.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:44:46.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:46.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:46.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:46.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:46.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:46.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:46.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:46 smithi151 ceph-mon[112233]: pgmap v182: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:46.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:44:46.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:46 smithi153 ceph-mon[87858]: pgmap v182: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:47.942 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:47 smithi151 ceph-mon[112233]: Upgrade: Updating mds.cephfs.smithi151.slrqog 2024-04-16T02:44:47.942 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:47 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:47.943 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:47 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.slrqog", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:44:47.943 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:47 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:47.943 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:47 smithi151 ceph-mon[112233]: Deploying daemon mds.cephfs.smithi151.slrqog on smithi151 2024-04-16T02:44:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:47 smithi153 ceph-mon[87858]: Upgrade: Updating mds.cephfs.smithi151.slrqog 2024-04-16T02:44:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:47 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:47 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi151.slrqog", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:44:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:47 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:44:48.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:47 smithi153 ceph-mon[87858]: Deploying daemon mds.cephfs.smithi151.slrqog on smithi151 2024-04-16T02:44:48.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:48 smithi151 ceph-mon[112233]: pgmap v183: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:48 smithi153 ceph-mon[87858]: pgmap v183: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:49 smithi153 ceph-mon[87858]: osdmap e84: 6 total, 6 up, 6 in 2024-04-16T02:44:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:49 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-16T02:44:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:49 smithi151 ceph-mon[112233]: osdmap e84: 6 total, 6 up, 6 in 2024-04-16T02:44:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:49 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-16T02:44:51.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:50 smithi151 ceph-mon[112233]: pgmap v185: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:51.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:50 smithi153 ceph-mon[87858]: pgmap v185: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:51 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:44:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:51 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:44:52.963 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:52 smithi151 ceph-mon[112233]: pgmap v186: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:53.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:52 smithi153 ceph-mon[87858]: pgmap v186: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:53.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:44:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:44:55.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:54 smithi153 ceph-mon[87858]: pgmap v187: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:55.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:55.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:55.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:55.123 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:54 smithi151 ceph-mon[112233]: pgmap v187: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:55.123 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:55.123 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:44:55.123 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:44:56.115 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:55 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:boot 2024-04-16T02:44:56.115 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:55 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T02:44:56.115 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:55 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:44:56.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:55 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:boot 2024-04-16T02:44:56.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:55 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T02:44:56.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:55 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T02:44:56.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:56 smithi151 ceph-mon[112233]: pgmap v188: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:57.232 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:56 smithi153 ceph-mon[87858]: pgmap v188: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:59.154 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:44:58 smithi151 ceph-mon[112233]: pgmap v189: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:44:59.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:44:58 smithi153 ceph-mon[87858]: pgmap v189: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:00 smithi151 ceph-mon[112233]: pgmap v190: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:00 smithi153 ceph-mon[87858]: pgmap v190: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:01.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:01.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:01.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:01 smithi151 ceph-mon[112233]: pgmap v191: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:01 smithi153 ceph-mon[87858]: pgmap v191: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:04.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:03 smithi151 ceph-mon[112233]: pgmap v192: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:04.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:03 smithi153 ceph-mon[87858]: pgmap v192: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:04.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.793+0000 7f8024762700 1 -- 172.21.15.151:0/835799262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 msgr2=0x7f801c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:04.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.793+0000 7f8024762700 1 --2- 172.21.15.151:0/835799262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 0x7f801c1011a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8018009910 tx=0x7f8018009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.793+0000 7f8024762700 1 -- 172.21.15.151:0/835799262 shutdown_connections 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.793+0000 7f8024762700 1 --2- 172.21.15.151:0/835799262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f801c101770 0x7f801c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.793+0000 7f8024762700 1 --2- 172.21.15.151:0/835799262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 0x7f801c1011a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.793+0000 7f8024762700 1 -- 172.21.15.151:0/835799262 >> 172.21.15.151:0/835799262 conn(0x7f801c0fc650 msgr2=0x7f801c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.793+0000 7f8024762700 1 -- 172.21.15.151:0/835799262 shutdown_connections 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.793+0000 7f8024762700 1 -- 172.21.15.151:0/835799262 wait complete. 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f8024762700 1 Processor -- start 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f8024762700 1 -- start start 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f8024762700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 0x7f801c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f8024762700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f801c101770 0x7f801c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f8024762700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c19d110 con 0x7f801c100dc0 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f8024762700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c1965c0 con 0x7f801c101770 2024-04-16T02:45:04.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f80224fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 0x7f801c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f80224fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 0x7f801c19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57792/0 (socket says 172.21.15.151:57792) 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f80224fe700 1 -- 172.21.15.151:0/2357567193 learned_addr learned my addr 172.21.15.151:0/2357567193 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.794+0000 7f8021cfd700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f801c101770 0x7f801c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.795+0000 7f80224fe700 1 -- 172.21.15.151:0/2357567193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f801c101770 msgr2=0x7f801c19ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.795+0000 7f80224fe700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f801c101770 0x7f801c19ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.795+0000 7f80224fe700 1 -- 172.21.15.151:0/2357567193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80180095f0 con 0x7f801c100dc0 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.795+0000 7f80224fe700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 0x7f801c19c540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8018006010 tx=0x7f801800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.795+0000 7f80137fe700 1 -- 172.21.15.151:0/2357567193 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f801801d070 con 0x7f801c100dc0 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.795+0000 7f80137fe700 1 -- 172.21.15.151:0/2357567193 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8018005dc0 con 0x7f801c100dc0 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.795+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c196840 con 0x7f801c100dc0 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.795+0000 7f80137fe700 1 -- 172.21.15.151:0/2357567193 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8018017400 con 0x7f801c100dc0 2024-04-16T02:45:04.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.795+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c196d30 con 0x7f801c100dc0 2024-04-16T02:45:04.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.796+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c109160 con 0x7f801c100dc0 2024-04-16T02:45:04.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.796+0000 7f80137fe700 1 -- 172.21.15.151:0/2357567193 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8018003a60 con 0x7f801c100dc0 2024-04-16T02:45:04.802 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.797+0000 7f80137fe700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f800806ecc0 0x7f8008071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:04.802 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.797+0000 7f80137fe700 1 -- 172.21.15.151:0/2357567193 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f80180902f0 con 0x7f801c100dc0 2024-04-16T02:45:04.802 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.797+0000 7f8021cfd700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f800806ecc0 0x7f8008071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:04.802 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.797+0000 7f8021cfd700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f800806ecc0 0x7f8008071180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f801c1059b0 tx=0x7f800c0073d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:04.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:04.802+0000 7f80137fe700 1 -- 172.21.15.151:0/2357567193 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f801805cbd0 con 0x7f801c100dc0 2024-04-16T02:45:05.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.020+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f801c197010 con 0x7f800806ecc0 2024-04-16T02:45:05.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.021+0000 7f80137fe700 1 -- 172.21.15.151:0/2357567193 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f801c197010 con 0x7f800806ecc0 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.023+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f800806ecc0 msgr2=0x7f8008071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.023+0000 7f8024762700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f800806ecc0 0x7f8008071180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f801c1059b0 tx=0x7f800c0073d0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.023+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 msgr2=0x7f801c19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.023+0000 7f8024762700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 0x7f801c19c540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8018006010 tx=0x7f801800bd70 comp rx=0 tx=0).stop 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.023+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 shutdown_connections 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.023+0000 7f8024762700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f800806ecc0 0x7f8008071180 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.024+0000 7f8024762700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801c100dc0 0x7f801c19c540 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.024+0000 7f8024762700 1 --2- 172.21.15.151:0/2357567193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f801c101770 0x7f801c19ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.024+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 >> 172.21.15.151:0/2357567193 conn(0x7f801c0fc650 msgr2=0x7f801c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.024+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 shutdown_connections 2024-04-16T02:45:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.024+0000 7f8024762700 1 -- 172.21.15.151:0/2357567193 wait complete. 2024-04-16T02:45:05.037 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:45:05.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.130+0000 7f1150a75700 1 -- 172.21.15.151:0/1593208491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c101750 msgr2=0x7f114c1057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.130+0000 7f1150a75700 1 --2- 172.21.15.151:0/1593208491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c101750 0x7f114c1057a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f113c009910 tx=0x7f113c009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.130+0000 7f1150a75700 1 -- 172.21.15.151:0/1593208491 shutdown_connections 2024-04-16T02:45:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.130+0000 7f1150a75700 1 --2- 172.21.15.151:0/1593208491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c101750 0x7f114c1057a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.130+0000 7f1150a75700 1 --2- 172.21.15.151:0/1593208491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114c100da0 0x7f114c101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.130+0000 7f1150a75700 1 -- 172.21.15.151:0/1593208491 >> 172.21.15.151:0/1593208491 conn(0x7f114c0fc630 msgr2=0x7f114c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.130+0000 7f1150a75700 1 -- 172.21.15.151:0/1593208491 shutdown_connections 2024-04-16T02:45:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.130+0000 7f1150a75700 1 -- 172.21.15.151:0/1593208491 wait complete. 2024-04-16T02:45:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f1150a75700 1 Processor -- start 2024-04-16T02:45:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f1150a75700 1 -- start start 2024-04-16T02:45:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f1150a75700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c100da0 0x7f114c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f1150a75700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114c101750 0x7f114c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f1150a75700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c19d0d0 con 0x7f114c100da0 2024-04-16T02:45:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f1150a75700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c196580 con 0x7f114c101750 2024-04-16T02:45:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f114a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c100da0 0x7f114c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f114a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c100da0 0x7f114c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57812/0 (socket says 172.21.15.151:57812) 2024-04-16T02:45:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f114a59c700 1 -- 172.21.15.151:0/1834507561 learned_addr learned my addr 172.21.15.151:0/1834507561 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.131+0000 7f1149d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114c101750 0x7f114c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f1149d9b700 1 -- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c100da0 msgr2=0x7f114c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f1149d9b700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c100da0 0x7f114c19c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f1149d9b700 1 -- 172.21.15.151:0/1834507561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0095f0 con 0x7f114c101750 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f114a59c700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c100da0 0x7f114c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f1149d9b700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114c101750 0x7f114c19ca40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f113c006010 tx=0x7f113c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f113b7fe700 1 -- 172.21.15.151:0/1834507561 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f113c01d070 con 0x7f114c101750 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c196800 con 0x7f114c101750 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f113b7fe700 1 -- 172.21.15.151:0/1834507561 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f113c00b610 con 0x7f114c101750 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f113b7fe700 1 -- 172.21.15.151:0/1834507561 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f113c022470 con 0x7f114c101750 2024-04-16T02:45:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.132+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114c196cc0 con 0x7f114c101750 2024-04-16T02:45:05.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.133+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114c1091a0 con 0x7f114c101750 2024-04-16T02:45:05.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.134+0000 7f113b7fe700 1 -- 172.21.15.151:0/1834507561 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f113c0050d0 con 0x7f114c101750 2024-04-16T02:45:05.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.134+0000 7f113b7fe700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f113406ecc0 0x7f1134071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:05.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.134+0000 7f114a59c700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f113406ecc0 0x7f1134071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:05.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.134+0000 7f113b7fe700 1 -- 172.21.15.151:0/1834507561 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f113c0903d0 con 0x7f114c101750 2024-04-16T02:45:05.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.135+0000 7f114a59c700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f113406ecc0 0x7f1134071180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1140011550 tx=0x7f1140009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:05.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.140+0000 7f113b7fe700 1 -- 172.21.15.151:0/1834507561 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f113c05cc90 con 0x7f114c101750 2024-04-16T02:45:05.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.358+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f114c196f30 con 0x7f113406ecc0 2024-04-16T02:45:05.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.359+0000 7f113b7fe700 1 -- 172.21.15.151:0/1834507561 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f114c196f30 con 0x7f113406ecc0 2024-04-16T02:45:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f113406ecc0 msgr2=0x7f1134071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f113406ecc0 0x7f1134071180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1140011550 tx=0x7f1140009260 comp rx=0 tx=0).stop 2024-04-16T02:45:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114c101750 msgr2=0x7f114c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114c101750 0x7f114c19ca40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f113c006010 tx=0x7f113c00f690 comp rx=0 tx=0).stop 2024-04-16T02:45:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 shutdown_connections 2024-04-16T02:45:05.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f113406ecc0 0x7f1134071180 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f114c100da0 0x7f114c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 --2- 172.21.15.151:0/1834507561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114c101750 0x7f114c19ca40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 >> 172.21.15.151:0/1834507561 conn(0x7f114c0fc630 msgr2=0x7f114c0fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:05.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 shutdown_connections 2024-04-16T02:45:05.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.361+0000 7f1150a75700 1 -- 172.21.15.151:0/1834507561 wait complete. 2024-04-16T02:45:05.469 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.466+0000 7fe68bbea700 1 -- 172.21.15.151:0/2955309494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ff530 msgr2=0x7fe6840ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.466+0000 7fe68bbea700 1 --2- 172.21.15.151:0/2955309494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ff530 0x7fe6840ff910 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe680009910 tx=0x7fe680009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.466+0000 7fe68bbea700 1 -- 172.21.15.151:0/2955309494 shutdown_connections 2024-04-16T02:45:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.466+0000 7fe68bbea700 1 --2- 172.21.15.151:0/2955309494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6840ffee0 0x7fe684105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.466+0000 7fe68bbea700 1 --2- 172.21.15.151:0/2955309494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ff530 0x7fe6840ff910 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.466+0000 7fe68bbea700 1 -- 172.21.15.151:0/2955309494 >> 172.21.15.151:0/2955309494 conn(0x7fe684074df0 msgr2=0x7fe684075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.466+0000 7fe68bbea700 1 -- 172.21.15.151:0/2955309494 shutdown_connections 2024-04-16T02:45:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.466+0000 7fe68bbea700 1 -- 172.21.15.151:0/2955309494 wait complete. 2024-04-16T02:45:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.467+0000 7fe68bbea700 1 Processor -- start 2024-04-16T02:45:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.467+0000 7fe68bbea700 1 -- start start 2024-04-16T02:45:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.467+0000 7fe68bbea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6840ff530 0x7fe6841004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.467+0000 7fe68bbea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ffee0 0x7fe6841009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.467+0000 7fe68bbea700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe684100f30 con 0x7fe6840ffee0 2024-04-16T02:45:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.467+0000 7fe68bbea700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe684101070 con 0x7fe6840ff530 2024-04-16T02:45:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.467+0000 7fe689185700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ffee0 0x7fe6841009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.467+0000 7fe689185700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ffee0 0x7fe6841009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57820/0 (socket says 172.21.15.151:57820) 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe689185700 1 -- 172.21.15.151:0/285818588 learned_addr learned my addr 172.21.15.151:0/285818588 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe689986700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6840ff530 0x7fe6841004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe689185700 1 -- 172.21.15.151:0/285818588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6840ff530 msgr2=0x7fe6841004b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe689185700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6840ff530 0x7fe6841004b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe689185700 1 -- 172.21.15.151:0/285818588 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6800095f0 con 0x7fe6840ffee0 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe689185700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ffee0 0x7fe6841009f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe67400e8f0 tx=0x7fe67400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe67affd700 1 -- 172.21.15.151:0/285818588 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe674009790 con 0x7fe6840ffee0 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe67affd700 1 -- 172.21.15.151:0/285818588 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe67400c810 con 0x7fe6840ffee0 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6841a2990 con 0x7fe6840ffee0 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe67affd700 1 -- 172.21.15.151:0/285818588 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe674005580 con 0x7fe6840ffee0 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.468+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6841a2da0 con 0x7fe6840ffee0 2024-04-16T02:45:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.469+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6841091e0 con 0x7fe6840ffee0 2024-04-16T02:45:05.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.470+0000 7fe67affd700 1 -- 172.21.15.151:0/285818588 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe67400c980 con 0x7fe6840ffee0 2024-04-16T02:45:05.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.470+0000 7fe67affd700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe67006ecc0 0x7fe670071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:05.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.470+0000 7fe67affd700 1 -- 172.21.15.151:0/285818588 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe674014070 con 0x7fe6840ffee0 2024-04-16T02:45:05.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.470+0000 7fe689986700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe67006ecc0 0x7fe670071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:05.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.471+0000 7fe689986700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe67006ecc0 0x7fe670071180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe6800059d0 tx=0x7fe680005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:05.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.475+0000 7fe67affd700 1 -- 172.21.15.151:0/285818588 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe674057ed0 con 0x7fe6840ffee0 2024-04-16T02:45:05.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.685+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe68402cce0 con 0x7fe67006ecc0 2024-04-16T02:45:05.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.696+0000 7fe67affd700 1 -- 172.21.15.151:0/285818588 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fe68402cce0 con 0x7fe67006ecc0 2024-04-16T02:45:05.699 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:45:05.699 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (6m) 5s ago 17m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:45:05.699 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (18m) 5s ago 18m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:45:05.699 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (17m) 46s ago 17m 8804k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:45:05.699 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (4m) 5s ago 18m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:45:05.699 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (4m) 46s ago 17m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:45:05.699 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (6m) 5s ago 17m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:45:05.699 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (31s) 5s ago 12m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (11s) 5s ago 12m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (12m) 46s ago 12m 58.5M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (12m) 46s ago 12m 67.5M - 18.2.1 5be31c24972a 7e19f4c34e96 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (8m) 5s ago 19m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (7m) 46s ago 16m 452M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (5m) 5s ago 19m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (4m) 46s ago 16m 43.5M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (7m) 5s ago 18m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 46s ago 17m 16.0M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (3m) 5s ago 16m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (3m) 5s ago 15m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (2m) 5s ago 15m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (118s) 46s ago 14m 109M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (84s) 46s ago 14m 90.8M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:45:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (50s) 46s ago 13m 12.6M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:45:05.701 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (6m) 5s ago 17m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:45:05.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe67006ecc0 msgr2=0x7fe670071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe67006ecc0 0x7fe670071180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe6800059d0 tx=0x7fe680005940 comp rx=0 tx=0).stop 2024-04-16T02:45:05.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ffee0 msgr2=0x7fe6841009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ffee0 0x7fe6841009f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe67400e8f0 tx=0x7fe67400ec00 comp rx=0 tx=0).stop 2024-04-16T02:45:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 shutdown_connections 2024-04-16T02:45:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe67006ecc0 0x7fe670071180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6840ff530 0x7fe6841004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 --2- 172.21.15.151:0/285818588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe6840ffee0 0x7fe6841009f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 >> 172.21.15.151:0/285818588 conn(0x7fe684074df0 msgr2=0x7fe6840feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 shutdown_connections 2024-04-16T02:45:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.698+0000 7fe68bbea700 1 -- 172.21.15.151:0/285818588 wait complete. 2024-04-16T02:45:05.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:05.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:45:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:45:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:45:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:05.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:05.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:05.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.805+0000 7f8a3a772700 1 -- 172.21.15.151:0/1139674039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a340ffa40 msgr2=0x7f8a340ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.805+0000 7f8a3a772700 1 --2- 172.21.15.151:0/1139674039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a340ffa40 0x7f8a340ffec0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009910 tx=0x7f8a1c009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.805+0000 7f8a3a772700 1 -- 172.21.15.151:0/1139674039 shutdown_connections 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.805+0000 7f8a3a772700 1 --2- 172.21.15.151:0/1139674039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a340ffa40 0x7f8a340ffec0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.805+0000 7f8a3a772700 1 --2- 172.21.15.151:0/1139674039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a34105a40 0x7f8a34105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.805+0000 7f8a3a772700 1 -- 172.21.15.151:0/1139674039 >> 172.21.15.151:0/1139674039 conn(0x7f8a340fb550 msgr2=0x7f8a340fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.805+0000 7f8a3a772700 1 -- 172.21.15.151:0/1139674039 shutdown_connections 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.805+0000 7f8a3a772700 1 -- 172.21.15.151:0/1139674039 wait complete. 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.805+0000 7f8a3a772700 1 Processor -- start 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a3a772700 1 -- start start 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a3a772700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a340ffa40 0x7f8a3419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a3a772700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a34105a40 0x7f8a3419ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a3a772700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3419d0f0 con 0x7f8a34105a40 2024-04-16T02:45:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a3a772700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a341965a0 con 0x7f8a340ffa40 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a2bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a34105a40 0x7f8a3419ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a2bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a34105a40 0x7f8a3419ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57842/0 (socket says 172.21.15.151:57842) 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a2bfff700 1 -- 172.21.15.151:0/680516623 learned_addr learned my addr 172.21.15.151:0/680516623 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a33fff700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a340ffa40 0x7f8a3419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a2bfff700 1 -- 172.21.15.151:0/680516623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a340ffa40 msgr2=0x7f8a3419c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a2bfff700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a340ffa40 0x7f8a3419c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a2bfff700 1 -- 172.21.15.151:0/680516623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0095f0 con 0x7f8a34105a40 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a2bfff700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a34105a40 0x7f8a3419ca60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c006010 tx=0x7f8a1c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.806+0000 7f8a31ffb700 1 -- 172.21.15.151:0/680516623 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a1c01d070 con 0x7f8a34105a40 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.807+0000 7f8a31ffb700 1 -- 172.21.15.151:0/680516623 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a1c00f460 con 0x7f8a34105a40 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.807+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a34196820 con 0x7f8a34105a40 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.807+0000 7f8a31ffb700 1 -- 172.21.15.151:0/680516623 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a1c017610 con 0x7f8a34105a40 2024-04-16T02:45:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.807+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a34196d10 con 0x7f8a34105a40 2024-04-16T02:45:05.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.807+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a34108fe0 con 0x7f8a34105a40 2024-04-16T02:45:05.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.808+0000 7f8a31ffb700 1 -- 172.21.15.151:0/680516623 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8a1c00f5d0 con 0x7f8a34105a40 2024-04-16T02:45:05.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.809+0000 7f8a31ffb700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a2006ecc0 0x7f8a20071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:05.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.809+0000 7f8a33fff700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a2006ecc0 0x7f8a20071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:05.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.809+0000 7f8a31ffb700 1 -- 172.21.15.151:0/680516623 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8a1c0903d0 con 0x7f8a34105a40 2024-04-16T02:45:05.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.809+0000 7f8a33fff700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a2006ecc0 0x7f8a20071180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8a24005950 tx=0x7f8a2400a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:05.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:05.813+0000 7f8a31ffb700 1 -- 172.21.15.151:0/680516623 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8a1c05ccb0 con 0x7f8a34105a40 2024-04-16T02:45:06.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.099+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8a34196ff0 con 0x7f8a34105a40 2024-04-16T02:45:06.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.099+0000 7f8a31ffb700 1 -- 172.21.15.151:0/680516623 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f8a1c05c840 con 0x7f8a34105a40 2024-04-16T02:45:06.103 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:45:06.103 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:45:06.103 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:45:06.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:45:06.103 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:45:06.103 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:45:06.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:45:06.103 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:45:06.103 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 12 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:45:06.104 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:45:06.105 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a2006ecc0 msgr2=0x7f8a20071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.105 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a2006ecc0 0x7f8a20071180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8a24005950 tx=0x7f8a2400a300 comp rx=0 tx=0).stop 2024-04-16T02:45:06.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a34105a40 msgr2=0x7f8a3419ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a34105a40 0x7f8a3419ca60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c006010 tx=0x7f8a1c00bd70 comp rx=0 tx=0).stop 2024-04-16T02:45:06.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 shutdown_connections 2024-04-16T02:45:06.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a2006ecc0 0x7f8a20071180 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a340ffa40 0x7f8a3419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 --2- 172.21.15.151:0/680516623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a34105a40 0x7f8a3419ca60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 >> 172.21.15.151:0/680516623 conn(0x7f8a340fb550 msgr2=0x7f8a34069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:06.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 shutdown_connections 2024-04-16T02:45:06.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.102+0000 7f8a3a772700 1 -- 172.21.15.151:0/680516623 wait complete. 2024-04-16T02:45:06.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.208+0000 7f9330149700 1 -- 172.21.15.151:0/1076272872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f93280ffbd0 msgr2=0x7f9328100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.208+0000 7f9330149700 1 --2- 172.21.15.151:0/1076272872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f93280ffbd0 0x7f9328100050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9324009910 tx=0x7f9324009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:06.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.208+0000 7f9330149700 1 -- 172.21.15.151:0/1076272872 shutdown_connections 2024-04-16T02:45:06.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.208+0000 7f9330149700 1 --2- 172.21.15.151:0/1076272872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f93280ffbd0 0x7f9328100050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.208+0000 7f9330149700 1 --2- 172.21.15.151:0/1076272872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9328105a40 0x7f9328105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.208+0000 7f9330149700 1 -- 172.21.15.151:0/1076272872 >> 172.21.15.151:0/1076272872 conn(0x7f93280755a0 msgr2=0x7f93280759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.208+0000 7f9330149700 1 -- 172.21.15.151:0/1076272872 shutdown_connections 2024-04-16T02:45:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.208+0000 7f9330149700 1 -- 172.21.15.151:0/1076272872 wait complete. 2024-04-16T02:45:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f9330149700 1 Processor -- start 2024-04-16T02:45:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f9330149700 1 -- start start 2024-04-16T02:45:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f9330149700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93280ffbd0 0x7f932819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f9330149700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9328105a40 0x7f932819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f9330149700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932819d200 con 0x7f9328105a40 2024-04-16T02:45:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f9330149700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93281966b0 con 0x7f93280ffbd0 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f932d6e4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9328105a40 0x7f932819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f932d6e4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9328105a40 0x7f932819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57864/0 (socket says 172.21.15.151:57864) 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f932d6e4700 1 -- 172.21.15.151:0/3567572785 learned_addr learned my addr 172.21.15.151:0/3567572785 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f932dee5700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93280ffbd0 0x7f932819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f932d6e4700 1 -- 172.21.15.151:0/3567572785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93280ffbd0 msgr2=0x7f932819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f932d6e4700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93280ffbd0 0x7f932819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.209+0000 7f932d6e4700 1 -- 172.21.15.151:0/3567572785 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93240095f0 con 0x7f9328105a40 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.210+0000 7f932d6e4700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9328105a40 0x7f932819cb70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9324006010 tx=0x7f932400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.210+0000 7f931affd700 1 -- 172.21.15.151:0/3567572785 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f932401d070 con 0x7f9328105a40 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.210+0000 7f931affd700 1 -- 172.21.15.151:0/3567572785 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9324004fb0 con 0x7f9328105a40 2024-04-16T02:45:06.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.210+0000 7f931affd700 1 -- 172.21.15.151:0/3567572785 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9324022ea0 con 0x7f9328105a40 2024-04-16T02:45:06.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.210+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9328196930 con 0x7f9328105a40 2024-04-16T02:45:06.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.210+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9328196e20 con 0x7f9328105a40 2024-04-16T02:45:06.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.211+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9328109070 con 0x7f9328105a40 2024-04-16T02:45:06.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.211+0000 7f931affd700 1 -- 172.21.15.151:0/3567572785 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9324022470 con 0x7f9328105a40 2024-04-16T02:45:06.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.212+0000 7f931affd700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f931406ecc0 0x7f9314071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:06.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.212+0000 7f931affd700 1 -- 172.21.15.151:0/3567572785 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f93240903f0 con 0x7f9328105a40 2024-04-16T02:45:06.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.212+0000 7f932dee5700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f931406ecc0 0x7f9314071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:06.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.212+0000 7f932dee5700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f931406ecc0 0x7f9314071180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f931c006fd0 tx=0x7f931c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:06.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.216+0000 7f931affd700 1 -- 172.21.15.151:0/3567572785 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f932405cd80 con 0x7f9328105a40 2024-04-16T02:45:06.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.464+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f932804ea90 con 0x7f9328105a40 2024-04-16T02:45:06.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.465+0000 7f931affd700 1 -- 172.21.15.151:0/3567572785 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1877 (secure 0 0 0) 0x7f9324027770 con 0x7f9328105a40 2024-04-16T02:45:06.468 INFO:teuthology.orchestra.run.smithi151.stdout:e29 2024-04-16T02:45:06.468 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:45:06.468 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:45:06.468 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:45:06.468 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:45:06.468 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:45:06.468 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:45:06.469 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:45:06.470 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:45:06.472 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:45:06.472 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1306847002,v1:172.21.15.153:6825/1306847002] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:45:06.472 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:45:06.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f931406ecc0 msgr2=0x7f9314071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f931406ecc0 0x7f9314071180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f931c006fd0 tx=0x7f931c008040 comp rx=0 tx=0).stop 2024-04-16T02:45:06.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9328105a40 msgr2=0x7f932819cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9328105a40 0x7f932819cb70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9324006010 tx=0x7f932400be20 comp rx=0 tx=0).stop 2024-04-16T02:45:06.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 shutdown_connections 2024-04-16T02:45:06.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f931406ecc0 0x7f9314071180 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93280ffbd0 0x7f932819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 --2- 172.21.15.151:0/3567572785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9328105a40 0x7f932819cb70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 >> 172.21.15.151:0/3567572785 conn(0x7f93280755a0 msgr2=0x7f9328068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:06.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 shutdown_connections 2024-04-16T02:45:06.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.467+0000 7f9330149700 1 -- 172.21.15.151:0/3567572785 wait complete. 2024-04-16T02:45:06.474 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 29 2024-04-16T02:45:06.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:06 smithi153 ceph-mon[87858]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:06.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:06 smithi153 ceph-mon[87858]: from='client.44291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:06.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:06 smithi153 ceph-mon[87858]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:06.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:06 smithi153 ceph-mon[87858]: pgmap v193: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:06.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:06 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/680516623' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:06.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:06 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:06.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:06 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ogdcjk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:45:06.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:06 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:45:06.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:06 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3567572785' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:45:06.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.575+0000 7f5d5d020700 1 -- 172.21.15.151:0/1067935114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d58102680 msgr2=0x7f5d5810ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.575+0000 7f5d5d020700 1 --2- 172.21.15.151:0/1067935114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d58102680 0x7f5d5810ab70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009940 tx=0x7f5d40009c50 comp rx=0 tx=0).stop 2024-04-16T02:45:06.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.575+0000 7f5d5d020700 1 -- 172.21.15.151:0/1067935114 shutdown_connections 2024-04-16T02:45:06.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.575+0000 7f5d5d020700 1 --2- 172.21.15.151:0/1067935114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d58102680 0x7f5d5810ab70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.575+0000 7f5d5d020700 1 --2- 172.21.15.151:0/1067935114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d58101d60 0x7f5d58102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.575+0000 7f5d5d020700 1 -- 172.21.15.151:0/1067935114 >> 172.21.15.151:0/1067935114 conn(0x7f5d580fb570 msgr2=0x7f5d580fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:06.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.575+0000 7f5d5d020700 1 -- 172.21.15.151:0/1067935114 shutdown_connections 2024-04-16T02:45:06.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.575+0000 7f5d5d020700 1 -- 172.21.15.151:0/1067935114 wait complete. 2024-04-16T02:45:06.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.575+0000 7f5d5d020700 1 Processor -- start 2024-04-16T02:45:06.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d5d020700 1 -- start start 2024-04-16T02:45:06.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d5d020700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d58101d60 0x7f5d5819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:06.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d5d020700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d58102680 0x7f5d5819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:06.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d5d020700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5819d130 con 0x7f5d58102680 2024-04-16T02:45:06.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d5d020700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d581965e0 con 0x7f5d58101d60 2024-04-16T02:45:06.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d5659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d58102680 0x7f5d5819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:06.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d5659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d58102680 0x7f5d5819caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58098/0 (socket says 172.21.15.151:58098) 2024-04-16T02:45:06.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d56d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d58101d60 0x7f5d5819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:06.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d5659c700 1 -- 172.21.15.151:0/2880433399 learned_addr learned my addr 172.21.15.151:0/2880433399 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d56d9d700 1 -- 172.21.15.151:0/2880433399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d58102680 msgr2=0x7f5d5819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d56d9d700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d58102680 0x7f5d5819caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.576+0000 7f5d56d9d700 1 -- 172.21.15.151:0/2880433399 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d400095f0 con 0x7f5d58101d60 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.577+0000 7f5d56d9d700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d58101d60 0x7f5d5819c560 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d4800ef10 tx=0x7f5d4800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.577+0000 7f5d4ffff700 1 -- 172.21.15.151:0/2880433399 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d4800c9f0 con 0x7f5d58101d60 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.577+0000 7f5d4ffff700 1 -- 172.21.15.151:0/2880433399 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d4800f040 con 0x7f5d58101d60 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.577+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d58196860 con 0x7f5d58101d60 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.577+0000 7f5d4ffff700 1 -- 172.21.15.151:0/2880433399 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d480146b0 con 0x7f5d58101d60 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.577+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d58196db0 con 0x7f5d58101d60 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.578+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d58108270 con 0x7f5d58101d60 2024-04-16T02:45:06.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.578+0000 7f5d4ffff700 1 -- 172.21.15.151:0/2880433399 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5d4800cb50 con 0x7f5d58101d60 2024-04-16T02:45:06.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.579+0000 7f5d4ffff700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d4406ecc0 0x7f5d44071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:06.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.579+0000 7f5d4ffff700 1 -- 172.21.15.151:0/2880433399 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5d4808f280 con 0x7f5d58101d60 2024-04-16T02:45:06.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.579+0000 7f5d5659c700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d4406ecc0 0x7f5d44071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:06.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.579+0000 7f5d5659c700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d4406ecc0 0x7f5d44071180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5d580fcce0 tx=0x7f5d4000b5a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:06.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.584+0000 7f5d4ffff700 1 -- 172.21.15.151:0/2880433399 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5d4805bac0 con 0x7f5d58101d60 2024-04-16T02:45:06.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:06 smithi151 ceph-mon[112233]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:06.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:06 smithi151 ceph-mon[112233]: from='client.44291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:06.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:06 smithi151 ceph-mon[112233]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:06.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:06 smithi151 ceph-mon[112233]: pgmap v193: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:06 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/680516623' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ogdcjk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T02:45:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:06 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T02:45:06.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:06 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3567572785' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:45:06.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.803+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d58196fd0 con 0x7f5d4406ecc0 2024-04-16T02:45:06.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.804+0000 7f5d4ffff700 1 -- 172.21.15.151:0/2880433399 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5d58196fd0 con 0x7f5d4406ecc0 2024-04-16T02:45:06.807 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "14/23 daemons upgraded", 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:45:06.808 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:45:06.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d4406ecc0 msgr2=0x7f5d44071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d4406ecc0 0x7f5d44071180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5d580fcce0 tx=0x7f5d4000b5a0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d58101d60 msgr2=0x7f5d5819c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d58101d60 0x7f5d5819c560 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d4800ef10 tx=0x7f5d4800c3a0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 shutdown_connections 2024-04-16T02:45:06.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d4406ecc0 0x7f5d44071180 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d58101d60 0x7f5d5819c560 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 --2- 172.21.15.151:0/2880433399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d58102680 0x7f5d5819caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 >> 172.21.15.151:0/2880433399 conn(0x7f5d580fb570 msgr2=0x7f5d581053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:06.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 shutdown_connections 2024-04-16T02:45:06.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.806+0000 7f5d5d020700 1 -- 172.21.15.151:0/2880433399 wait complete. 2024-04-16T02:45:06.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.912+0000 7f276154c700 1 -- 172.21.15.151:0/897172623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c100e10 msgr2=0x7f275c1011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.912+0000 7f276154c700 1 --2- 172.21.15.151:0/897172623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c100e10 0x7f275c1011f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f274c009910 tx=0x7f274c009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:06.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.912+0000 7f276154c700 1 -- 172.21.15.151:0/897172623 shutdown_connections 2024-04-16T02:45:06.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.912+0000 7f276154c700 1 --2- 172.21.15.151:0/897172623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f275c1017c0 0x7f275c105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.912+0000 7f276154c700 1 --2- 172.21.15.151:0/897172623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c100e10 0x7f275c1011f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.912+0000 7f276154c700 1 -- 172.21.15.151:0/897172623 >> 172.21.15.151:0/897172623 conn(0x7f275c0fc6c0 msgr2=0x7f275c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:06.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.913+0000 7f276154c700 1 -- 172.21.15.151:0/897172623 shutdown_connections 2024-04-16T02:45:06.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.913+0000 7f276154c700 1 -- 172.21.15.151:0/897172623 wait complete. 2024-04-16T02:45:06.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.913+0000 7f276154c700 1 Processor -- start 2024-04-16T02:45:06.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.913+0000 7f276154c700 1 -- start start 2024-04-16T02:45:06.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f276154c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f275c100e10 0x7f275c19c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:06.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f276154c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c1017c0 0x7f275c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:06.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f276154c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c19d190 con 0x7f275c1017c0 2024-04-16T02:45:06.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f276154c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c196640 con 0x7f275c100e10 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f275a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c1017c0 0x7f275c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f275a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c1017c0 0x7f275c19cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58126/0 (socket says 172.21.15.151:58126) 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f275a7fc700 1 -- 172.21.15.151:0/2289282378 learned_addr learned my addr 172.21.15.151:0/2289282378 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f275affd700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f275c100e10 0x7f275c19c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f275a7fc700 1 -- 172.21.15.151:0/2289282378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f275c100e10 msgr2=0x7f275c19c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f275a7fc700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f275c100e10 0x7f275c19c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f275a7fc700 1 -- 172.21.15.151:0/2289282378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c0095f0 con 0x7f275c1017c0 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f275a7fc700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c1017c0 0x7f275c19cb00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f275000e8f0 tx=0x7f275000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f2743fff700 1 -- 172.21.15.151:0/2289282378 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2750009790 con 0x7f275c1017c0 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275c196920 con 0x7f275c1017c0 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f2743fff700 1 -- 172.21.15.151:0/2289282378 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f275000c810 con 0x7f275c1017c0 2024-04-16T02:45:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f2743fff700 1 -- 172.21.15.151:0/2289282378 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2750005580 con 0x7f275c1017c0 2024-04-16T02:45:06.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.914+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f275c196e70 con 0x7f275c1017c0 2024-04-16T02:45:06.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.915+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275c1091b0 con 0x7f275c1017c0 2024-04-16T02:45:06.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.916+0000 7f2743fff700 1 -- 172.21.15.151:0/2289282378 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f275000c980 con 0x7f275c1017c0 2024-04-16T02:45:06.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.916+0000 7f2743fff700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f274406ecc0 0x7f2744071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:06.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.917+0000 7f275affd700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f274406ecc0 0x7f2744071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:06.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.917+0000 7f2743fff700 1 -- 172.21.15.151:0/2289282378 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2750014070 con 0x7f275c1017c0 2024-04-16T02:45:06.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.917+0000 7f275affd700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f274406ecc0 0x7f2744071180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f274c00b760 tx=0x7f274c005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:06.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:06.922+0000 7f2743fff700 1 -- 172.21.15.151:0/2289282378 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f275005ba00 con 0x7f275c1017c0 2024-04-16T02:45:07.211 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.208+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f275c1970f0 con 0x7f275c1017c0 2024-04-16T02:45:07.211 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.208+0000 7f2743fff700 1 -- 172.21.15.151:0/2289282378 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2750062030 con 0x7f275c1017c0 2024-04-16T02:45:07.211 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:45:07.212 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:45:07.212 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:45:07.212 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:45:07.212 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.211+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f274406ecc0 msgr2=0x7f2744071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.211+0000 7f276154c700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f274406ecc0 0x7f2744071180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f274c00b760 tx=0x7f274c005a90 comp rx=0 tx=0).stop 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.211+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c1017c0 msgr2=0x7f275c19cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.211+0000 7f276154c700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c1017c0 0x7f275c19cb00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f275000e8f0 tx=0x7f275000ec00 comp rx=0 tx=0).stop 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.211+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 shutdown_connections 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.211+0000 7f276154c700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f274406ecc0 0x7f2744071180 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.211+0000 7f276154c700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f275c100e10 0x7f275c19c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.211+0000 7f276154c700 1 --2- 172.21.15.151:0/2289282378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f275c1017c0 0x7f275c19cb00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.211+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 >> 172.21.15.151:0/2289282378 conn(0x7f275c0fc6c0 msgr2=0x7f275c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.212+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 shutdown_connections 2024-04-16T02:45:07.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:07.212+0000 7f276154c700 1 -- 172.21.15.151:0/2289282378 wait complete. 2024-04-16T02:45:07.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:07 smithi151 ceph-mon[112233]: Upgrade: Updating mds.cephfs.smithi153.ogdcjk 2024-04-16T02:45:07.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:07 smithi151 ceph-mon[112233]: Deploying daemon mds.cephfs.smithi153.ogdcjk on smithi153 2024-04-16T02:45:07.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:07 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2289282378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:45:07.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:07 smithi153 ceph-mon[87858]: Upgrade: Updating mds.cephfs.smithi153.ogdcjk 2024-04-16T02:45:07.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:07 smithi153 ceph-mon[87858]: Deploying daemon mds.cephfs.smithi153.ogdcjk on smithi153 2024-04-16T02:45:07.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:07 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2289282378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:45:08.774 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:08 smithi153 ceph-mon[87858]: from='client.44303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:08.774 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:08 smithi153 ceph-mon[87858]: pgmap v194: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:08.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:08 smithi151 ceph-mon[112233]: from='client.44303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:08.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:08 smithi151 ceph-mon[112233]: pgmap v194: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:09.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:09 smithi153 ceph-mon[87858]: osdmap e85: 6 total, 6 up, 6 in 2024-04-16T02:45:09.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:09 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-16T02:45:09.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:45:09.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:09 smithi151 ceph-mon[112233]: osdmap e85: 6 total, 6 up, 6 in 2024-04-16T02:45:09.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:09 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-16T02:45:09.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:45:10.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:10 smithi153 ceph-mon[87858]: pgmap v196: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:10.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:10 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:45:10.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:10 smithi151 ceph-mon[112233]: pgmap v196: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:10.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:10 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:45:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:11 smithi151 ceph-mon[112233]: pgmap v197: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:11 smithi153 ceph-mon[87858]: pgmap v197: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:14.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:13 smithi151 ceph-mon[112233]: pgmap v198: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:14.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:13 smithi153 ceph-mon[87858]: pgmap v198: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:15.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:15.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:15.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:45:15.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:15 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] up:boot 2024-04-16T02:45:15.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:15 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T02:45:15.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:45:15.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:15.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:15.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T02:45:15.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:15 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] up:boot 2024-04-16T02:45:15.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:15 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T02:45:15.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T02:45:16.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:16 smithi153 ceph-mon[87858]: pgmap v199: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:16.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:16 smithi151 ceph-mon[112233]: pgmap v199: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:18.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:18 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:18.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:18 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:18.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:18 smithi151 ceph-mon[112233]: pgmap v200: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:18.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:18 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:18.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:18 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:18.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:18 smithi153 ceph-mon[87858]: pgmap v200: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:19.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:19.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:19 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:19 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T02:45:20.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:20 smithi153 ceph-mon[87858]: pgmap v201: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:20.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:20 smithi151 ceph-mon[112233]: pgmap v201: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:22.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:21 smithi151 ceph-mon[112233]: pgmap v202: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:21 smithi153 ceph-mon[87858]: pgmap v202: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:45:23.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:23 smithi151 ceph-mon[112233]: pgmap v203: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:45:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:23 smithi153 ceph-mon[87858]: pgmap v203: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:26.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:25 smithi151 ceph-mon[112233]: pgmap v204: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:26.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:25 smithi153 ceph-mon[87858]: pgmap v204: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:28.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:27 smithi151 ceph-mon[112233]: pgmap v205: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:27 smithi153 ceph-mon[87858]: pgmap v205: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:29 smithi151 ceph-mon[112233]: pgmap v206: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:29 smithi153 ceph-mon[87858]: pgmap v206: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:32.193 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:31 smithi151 ceph-mon[112233]: pgmap v207: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:31 smithi153 ceph-mon[87858]: pgmap v207: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:34.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:33 smithi151 ceph-mon[112233]: pgmap v208: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:34.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:33 smithi153 ceph-mon[87858]: pgmap v208: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:36.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:36 smithi151 ceph-mon[112233]: pgmap v209: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:36.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:36 smithi153 ceph-mon[87858]: pgmap v209: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:37.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.335+0000 7fea156e9700 1 -- 172.21.15.151:0/1965519873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10100de0 msgr2=0x7fea101011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:37.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.335+0000 7fea156e9700 1 --2- 172.21.15.151:0/1965519873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10100de0 0x7fea101011c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fea00009910 tx=0x7fea00009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:37.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.336+0000 7fea156e9700 1 -- 172.21.15.151:0/1965519873 shutdown_connections 2024-04-16T02:45:37.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.336+0000 7fea156e9700 1 --2- 172.21.15.151:0/1965519873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea10101790 0x7fea101057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.336+0000 7fea156e9700 1 --2- 172.21.15.151:0/1965519873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10100de0 0x7fea101011c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.336+0000 7fea156e9700 1 -- 172.21.15.151:0/1965519873 >> 172.21.15.151:0/1965519873 conn(0x7fea100fc670 msgr2=0x7fea100fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:37.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.336+0000 7fea156e9700 1 -- 172.21.15.151:0/1965519873 shutdown_connections 2024-04-16T02:45:37.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.336+0000 7fea156e9700 1 -- 172.21.15.151:0/1965519873 wait complete. 2024-04-16T02:45:37.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.336+0000 7fea156e9700 1 Processor -- start 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.336+0000 7fea156e9700 1 -- start start 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea156e9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea10100de0 0x7fea1019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea156e9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10101790 0x7fea1019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea156e9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1019d100 con 0x7fea10101790 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea156e9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea101965b0 con 0x7fea10100de0 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea07fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10101790 0x7fea1019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea0effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea10100de0 0x7fea1019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea07fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10101790 0x7fea1019ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34182/0 (socket says 172.21.15.151:34182) 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea07fff700 1 -- 172.21.15.151:0/2974852958 learned_addr learned my addr 172.21.15.151:0/2974852958 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea07fff700 1 -- 172.21.15.151:0/2974852958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea10100de0 msgr2=0x7fea1019c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea07fff700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea10100de0 0x7fea1019c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea07fff700 1 -- 172.21.15.151:0/2974852958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea000095f0 con 0x7fea10101790 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea07fff700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10101790 0x7fea1019ca70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800d900 tx=0x7fe9f800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.337+0000 7fea0cff9700 1 -- 172.21.15.151:0/2974852958 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9f8009900 con 0x7fea10101790 2024-04-16T02:45:37.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.338+0000 7fea0cff9700 1 -- 172.21.15.151:0/2974852958 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9f8010460 con 0x7fea10101790 2024-04-16T02:45:37.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.338+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea10196890 con 0x7fea10101790 2024-04-16T02:45:37.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.338+0000 7fea0cff9700 1 -- 172.21.15.151:0/2974852958 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9f800f5f0 con 0x7fea10101790 2024-04-16T02:45:37.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.338+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea10196de0 con 0x7fea10101790 2024-04-16T02:45:37.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.338+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea10109180 con 0x7fea10101790 2024-04-16T02:45:37.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.339+0000 7fea0cff9700 1 -- 172.21.15.151:0/2974852958 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe9f80105d0 con 0x7fea10101790 2024-04-16T02:45:37.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.340+0000 7fea0cff9700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9fc06ecc0 0x7fe9fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:37.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.340+0000 7fea0cff9700 1 -- 172.21.15.151:0/2974852958 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe9f808ea30 con 0x7fea10101790 2024-04-16T02:45:37.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.340+0000 7fea0effd700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9fc06ecc0 0x7fe9fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:37.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.340+0000 7fea0effd700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9fc06ecc0 0x7fe9fc071180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fea0000b5c0 tx=0x7fea0001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:37.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.344+0000 7fea0cff9700 1 -- 172.21.15.151:0/2974852958 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe9f805b310 con 0x7fea10101790 2024-04-16T02:45:37.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.561+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea10197030 con 0x7fe9fc06ecc0 2024-04-16T02:45:37.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.563+0000 7fea0cff9700 1 -- 172.21.15.151:0/2974852958 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fea10197030 con 0x7fe9fc06ecc0 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.565+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9fc06ecc0 msgr2=0x7fe9fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.565+0000 7fea156e9700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9fc06ecc0 0x7fe9fc071180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fea0000b5c0 tx=0x7fea0001a040 comp rx=0 tx=0).stop 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.565+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10101790 msgr2=0x7fea1019ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.565+0000 7fea156e9700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10101790 0x7fea1019ca70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800d900 tx=0x7fe9f800dc10 comp rx=0 tx=0).stop 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.565+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 shutdown_connections 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.565+0000 7fea156e9700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9fc06ecc0 0x7fe9fc071180 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.565+0000 7fea156e9700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea10100de0 0x7fea1019c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.565+0000 7fea156e9700 1 --2- 172.21.15.151:0/2974852958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea10101790 0x7fea1019ca70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.565+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 >> 172.21.15.151:0/2974852958 conn(0x7fea100fc670 msgr2=0x7fea100fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:37.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.566+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 shutdown_connections 2024-04-16T02:45:37.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.566+0000 7fea156e9700 1 -- 172.21.15.151:0/2974852958 wait complete. 2024-04-16T02:45:37.579 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:45:37.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.671+0000 7fdd450bc700 1 -- 172.21.15.151:0/4291547840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 msgr2=0x7fdd401057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:37.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.671+0000 7fdd450bc700 1 --2- 172.21.15.151:0/4291547840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 0x7fdd401057c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009910 tx=0x7fdd28009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:37.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.671+0000 7fdd450bc700 1 -- 172.21.15.151:0/4291547840 shutdown_connections 2024-04-16T02:45:37.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.671+0000 7fdd450bc700 1 --2- 172.21.15.151:0/4291547840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 0x7fdd401057c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.671+0000 7fdd450bc700 1 --2- 172.21.15.151:0/4291547840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd40100dc0 0x7fdd401011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.671+0000 7fdd450bc700 1 -- 172.21.15.151:0/4291547840 >> 172.21.15.151:0/4291547840 conn(0x7fdd400fc650 msgr2=0x7fdd400fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:37.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.671+0000 7fdd450bc700 1 -- 172.21.15.151:0/4291547840 shutdown_connections 2024-04-16T02:45:37.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.671+0000 7fdd450bc700 1 -- 172.21.15.151:0/4291547840 wait complete. 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd450bc700 1 Processor -- start 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd450bc700 1 -- start start 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd450bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd40100dc0 0x7fdd40073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd450bc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 0x7fdd400735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd450bc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd40073b10 con 0x7fdd40101770 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd450bc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd40073c50 con 0x7fdd40100dc0 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd37fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 0x7fdd400735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd37fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 0x7fdd400735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34198/0 (socket says 172.21.15.151:34198) 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd37fff700 1 -- 172.21.15.151:0/100803659 learned_addr learned my addr 172.21.15.151:0/100803659 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd3ed9d700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd40100dc0 0x7fdd40073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd37fff700 1 -- 172.21.15.151:0/100803659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd40100dc0 msgr2=0x7fdd40073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd37fff700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd40100dc0 0x7fdd40073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd37fff700 1 -- 172.21.15.151:0/100803659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd280095f0 con 0x7fdd40101770 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.672+0000 7fdd3ed9d700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd40100dc0 0x7fdd40073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.673+0000 7fdd37fff700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 0x7fdd400735d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd28000c00 tx=0x7fdd2800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.673+0000 7fdd3cd99700 1 -- 172.21.15.151:0/100803659 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd2801d070 con 0x7fdd40101770 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.673+0000 7fdd3cd99700 1 -- 172.21.15.151:0/100803659 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd28004fb0 con 0x7fdd40101770 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.673+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd40073ed0 con 0x7fdd40101770 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.673+0000 7fdd3cd99700 1 -- 172.21.15.151:0/100803659 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd2800f460 con 0x7fdd40101770 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.673+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd401a2da0 con 0x7fdd40101770 2024-04-16T02:45:37.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.674+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd40109110 con 0x7fdd40101770 2024-04-16T02:45:37.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.674+0000 7fdd3cd99700 1 -- 172.21.15.151:0/100803659 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fdd280228a0 con 0x7fdd40101770 2024-04-16T02:45:37.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.675+0000 7fdd3cd99700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdd2c06ecc0 0x7fdd2c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:37.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.675+0000 7fdd3cd99700 1 -- 172.21.15.151:0/100803659 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdd28090430 con 0x7fdd40101770 2024-04-16T02:45:37.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.675+0000 7fdd3ed9d700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdd2c06ecc0 0x7fdd2c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:37.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.675+0000 7fdd3ed9d700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdd2c06ecc0 0x7fdd2c071180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdd300099d0 tx=0x7fdd30008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:37.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.679+0000 7fdd3cd99700 1 -- 172.21.15.151:0/100803659 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdd28059290 con 0x7fdd40101770 2024-04-16T02:45:37.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.900+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd400611d0 con 0x7fdd2c06ecc0 2024-04-16T02:45:37.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.901+0000 7fdd3cd99700 1 -- 172.21.15.151:0/100803659 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdd400611d0 con 0x7fdd2c06ecc0 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.903+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdd2c06ecc0 msgr2=0x7fdd2c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.903+0000 7fdd450bc700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdd2c06ecc0 0x7fdd2c071180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdd300099d0 tx=0x7fdd30008040 comp rx=0 tx=0).stop 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.903+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 msgr2=0x7fdd400735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.903+0000 7fdd450bc700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 0x7fdd400735d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd28000c00 tx=0x7fdd2800be20 comp rx=0 tx=0).stop 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.903+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 shutdown_connections 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.903+0000 7fdd450bc700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdd2c06ecc0 0x7fdd2c071180 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.903+0000 7fdd450bc700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd40100dc0 0x7fdd40073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.903+0000 7fdd450bc700 1 --2- 172.21.15.151:0/100803659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd40101770 0x7fdd400735d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.903+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 >> 172.21.15.151:0/100803659 conn(0x7fdd400fc650 msgr2=0x7fdd400fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:37.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.904+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 shutdown_connections 2024-04-16T02:45:37.908 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:37.904+0000 7fdd450bc700 1 -- 172.21.15.151:0/100803659 wait complete. 2024-04-16T02:45:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.008+0000 7ff2a061d700 1 -- 172.21.15.151:0/746073546 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298100dc0 msgr2=0x7ff2981011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.008+0000 7ff2a061d700 1 --2- 172.21.15.151:0/746073546 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298100dc0 0x7ff2981011a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff294009910 tx=0x7ff294009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.008+0000 7ff2a061d700 1 -- 172.21.15.151:0/746073546 shutdown_connections 2024-04-16T02:45:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.009+0000 7ff2a061d700 1 --2- 172.21.15.151:0/746073546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff298101770 0x7ff2981057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.009+0000 7ff2a061d700 1 --2- 172.21.15.151:0/746073546 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298100dc0 0x7ff2981011a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.009+0000 7ff2a061d700 1 -- 172.21.15.151:0/746073546 >> 172.21.15.151:0/746073546 conn(0x7ff2980fc650 msgr2=0x7ff2980fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.009+0000 7ff2a061d700 1 -- 172.21.15.151:0/746073546 shutdown_connections 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.009+0000 7ff2a061d700 1 -- 172.21.15.151:0/746073546 wait complete. 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.009+0000 7ff2a061d700 1 Processor -- start 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.009+0000 7ff2a061d700 1 -- start start 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff2a061d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff298100dc0 0x7ff29819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff2a061d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298101770 0x7ff29819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff2a061d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29819d110 con 0x7ff298101770 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff2a061d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2981965c0 con 0x7ff298100dc0 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff29dbb8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298101770 0x7ff29819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff29dbb8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298101770 0x7ff29819ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34216/0 (socket says 172.21.15.151:34216) 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff29dbb8700 1 -- 172.21.15.151:0/1755103357 learned_addr learned my addr 172.21.15.151:0/1755103357 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff29e3b9700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff298100dc0 0x7ff29819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff29dbb8700 1 -- 172.21.15.151:0/1755103357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff298100dc0 msgr2=0x7ff29819c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff29dbb8700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff298100dc0 0x7ff29819c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff29dbb8700 1 -- 172.21.15.151:0/1755103357 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2940095f0 con 0x7ff298101770 2024-04-16T02:45:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff29dbb8700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298101770 0x7ff29819ca80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff28800d900 tx=0x7ff28800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff28f7fe700 1 -- 172.21.15.151:0/1755103357 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff288009900 con 0x7ff298101770 2024-04-16T02:45:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff28f7fe700 1 -- 172.21.15.151:0/1755103357 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff288010460 con 0x7ff298101770 2024-04-16T02:45:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2981968a0 con 0x7ff298101770 2024-04-16T02:45:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff28f7fe700 1 -- 172.21.15.151:0/1755103357 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28800f5f0 con 0x7ff298101770 2024-04-16T02:45:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.010+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff298196df0 con 0x7ff298101770 2024-04-16T02:45:38.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.011+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff298109160 con 0x7ff298101770 2024-04-16T02:45:38.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.012+0000 7ff28f7fe700 1 -- 172.21.15.151:0/1755103357 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff2880105d0 con 0x7ff298101770 2024-04-16T02:45:38.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.012+0000 7ff28f7fe700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff28406ecc0 0x7ff284071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:38.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.013+0000 7ff29e3b9700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff28406ecc0 0x7ff284071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:38.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.013+0000 7ff28f7fe700 1 -- 172.21.15.151:0/1755103357 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff28808e670 con 0x7ff298101770 2024-04-16T02:45:38.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.013+0000 7ff29e3b9700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff28406ecc0 0x7ff284071180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff294006010 tx=0x7ff29401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.017+0000 7ff28f7fe700 1 -- 172.21.15.151:0/1755103357 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff28805afd0 con 0x7ff298101770 2024-04-16T02:45:38.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.226+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2981970b0 con 0x7ff28406ecc0 2024-04-16T02:45:38.229 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:37 smithi151 ceph-mon[112233]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:38.229 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:37 smithi151 ceph-mon[112233]: pgmap v210: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.237+0000 7ff28f7fe700 1 -- 172.21.15.151:0/1755103357 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7ff2981970b0 con 0x7ff28406ecc0 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (7m) 38s ago 18m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (18m) 38s ago 18m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (17m) 21s ago 17m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (4m) 38s ago 18m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (4m) 21s ago 17m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (6m) 38s ago 18m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (63s) 38s ago 12m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (44s) 38s ago 12m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (12m) 21s ago 12m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:45:38.240 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (24s) 21s ago 12m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (9m) 38s ago 19m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (8m) 21s ago 17m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (5m) 38s ago 20m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (5m) 21s ago 17m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (7m) 38s ago 18m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 21s ago 17m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (4m) 38s ago 16m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (3m) 38s ago 16m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (3m) 38s ago 15m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (2m) 21s ago 15m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (116s) 21s ago 14m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (82s) 21s ago 14m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:45:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (7m) 38s ago 18m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:45:38.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff28406ecc0 msgr2=0x7ff284071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff28406ecc0 0x7ff284071180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff294006010 tx=0x7ff29401a040 comp rx=0 tx=0).stop 2024-04-16T02:45:38.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298101770 msgr2=0x7ff29819ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298101770 0x7ff29819ca80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff28800d900 tx=0x7ff28800dc10 comp rx=0 tx=0).stop 2024-04-16T02:45:38.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 shutdown_connections 2024-04-16T02:45:38.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff28406ecc0 0x7ff284071180 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff298100dc0 0x7ff29819c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 --2- 172.21.15.151:0/1755103357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff298101770 0x7ff29819ca80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 >> 172.21.15.151:0/1755103357 conn(0x7ff2980fc650 msgr2=0x7ff2980fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:38.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 shutdown_connections 2024-04-16T02:45:38.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.239+0000 7ff2a061d700 1 -- 172.21.15.151:0/1755103357 wait complete. 2024-04-16T02:45:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:37 smithi153 ceph-mon[87858]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:37 smithi153 ceph-mon[87858]: pgmap v210: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:38.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.343+0000 7f67728a8700 1 -- 172.21.15.151:0/2779291581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c1026c0 msgr2=0x7f676c10abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.343+0000 7f67728a8700 1 --2- 172.21.15.151:0/2779291581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c1026c0 0x7f676c10abb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6754009910 tx=0x7f6754009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:38.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.343+0000 7f67728a8700 1 -- 172.21.15.151:0/2779291581 shutdown_connections 2024-04-16T02:45:38.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.343+0000 7f67728a8700 1 --2- 172.21.15.151:0/2779291581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c1026c0 0x7f676c10abb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.343+0000 7f67728a8700 1 --2- 172.21.15.151:0/2779291581 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c101da0 0x7f676c102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.343+0000 7f67728a8700 1 -- 172.21.15.151:0/2779291581 >> 172.21.15.151:0/2779291581 conn(0x7f676c0fb550 msgr2=0x7f676c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:38.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.343+0000 7f67728a8700 1 -- 172.21.15.151:0/2779291581 shutdown_connections 2024-04-16T02:45:38.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.343+0000 7f67728a8700 1 -- 172.21.15.151:0/2779291581 wait complete. 2024-04-16T02:45:38.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.344+0000 7f67728a8700 1 Processor -- start 2024-04-16T02:45:38.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.344+0000 7f67728a8700 1 -- start start 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.344+0000 7f67728a8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c101da0 0x7f676c19c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.344+0000 7f67728a8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c1026c0 0x7f676c19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.344+0000 7f67728a8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c19d100 con 0x7f676c101da0 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.344+0000 7f67728a8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c1965b0 con 0x7f676c1026c0 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.344+0000 7f676bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c101da0 0x7f676c19c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.344+0000 7f676bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c101da0 0x7f676c19c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34228/0 (socket says 172.21.15.151:34228) 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.344+0000 7f676bfff700 1 -- 172.21.15.151:0/52049954 learned_addr learned my addr 172.21.15.151:0/52049954 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f676b7fe700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c1026c0 0x7f676c19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f676bfff700 1 -- 172.21.15.151:0/52049954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c1026c0 msgr2=0x7f676c19ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f676bfff700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c1026c0 0x7f676c19ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f676bfff700 1 -- 172.21.15.151:0/52049954 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67540095f0 con 0x7f676c101da0 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f676bfff700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c101da0 0x7f676c19c530 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f675c00e920 tx=0x7f675c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f67697fa700 1 -- 172.21.15.151:0/52049954 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f675c00c9d0 con 0x7f676c101da0 2024-04-16T02:45:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c196890 con 0x7f676c101da0 2024-04-16T02:45:38.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f67697fa700 1 -- 172.21.15.151:0/52049954 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f675c010430 con 0x7f676c101da0 2024-04-16T02:45:38.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f67697fa700 1 -- 172.21.15.151:0/52049954 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f675c018630 con 0x7f676c101da0 2024-04-16T02:45:38.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.345+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c196db0 con 0x7f676c101da0 2024-04-16T02:45:38.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.346+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c1082b0 con 0x7f676c101da0 2024-04-16T02:45:38.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.347+0000 7f67697fa700 1 -- 172.21.15.151:0/52049954 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f675c0105a0 con 0x7f676c101da0 2024-04-16T02:45:38.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.347+0000 7f67697fa700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:38.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.347+0000 7f67697fa700 1 -- 172.21.15.151:0/52049954 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f675c014070 con 0x7f676c101da0 2024-04-16T02:45:38.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.347+0000 7f676b7fe700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:38.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.348+0000 7f676b7fe700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f67540059d0 tx=0x7f6754005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:38.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.352+0000 7f67697fa700 1 -- 172.21.15.151:0/52049954 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f675c053ff0 con 0x7f676c101da0 2024-04-16T02:45:38.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.633+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f676c197090 con 0x7f676c101da0 2024-04-16T02:45:38.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.633+0000 7f67697fa700 1 -- 172.21.15.151:0/52049954 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f675c062070 con 0x7f676c101da0 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:45:38.637 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:45:38.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:45:38.638 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:45:38.638 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:45:38.638 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:45:38.638 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:45:38.638 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.635+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 msgr2=0x7f6758071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.635+0000 7f67728a8700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f67540059d0 tx=0x7f6754005940 comp rx=0 tx=0).stop 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.635+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c101da0 msgr2=0x7f676c19c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.635+0000 7f67728a8700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c101da0 0x7f676c19c530 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f675c00e920 tx=0x7f675c00ec30 comp rx=0 tx=0).stop 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.636+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 shutdown_connections 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.636+0000 7f67728a8700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f675806ecc0 0x7f6758071180 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.636+0000 7f67728a8700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f676c101da0 0x7f676c19c530 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.636+0000 7f67728a8700 1 --2- 172.21.15.151:0/52049954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c1026c0 0x7f676c19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.636+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 >> 172.21.15.151:0/52049954 conn(0x7f676c0fb550 msgr2=0x7f676c1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.636+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 shutdown_connections 2024-04-16T02:45:38.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.636+0000 7f67728a8700 1 -- 172.21.15.151:0/52049954 wait complete. 2024-04-16T02:45:38.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.742+0000 7f417dc86700 1 -- 172.21.15.151:0/3223689754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 msgr2=0x7f41781057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.742+0000 7f417dc86700 1 --2- 172.21.15.151:0/3223689754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 0x7f41781057a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4160009910 tx=0x7f4160009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:38.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.742+0000 7f417dc86700 1 -- 172.21.15.151:0/3223689754 shutdown_connections 2024-04-16T02:45:38.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.742+0000 7f417dc86700 1 --2- 172.21.15.151:0/3223689754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 0x7f41781057a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.742+0000 7f417dc86700 1 --2- 172.21.15.151:0/3223689754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4178100da0 0x7f4178101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.742+0000 7f417dc86700 1 -- 172.21.15.151:0/3223689754 >> 172.21.15.151:0/3223689754 conn(0x7f41780fc630 msgr2=0x7f41780fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:38.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.742+0000 7f417dc86700 1 -- 172.21.15.151:0/3223689754 shutdown_connections 2024-04-16T02:45:38.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.742+0000 7f417dc86700 1 -- 172.21.15.151:0/3223689754 wait complete. 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.743+0000 7f417dc86700 1 Processor -- start 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.743+0000 7f417dc86700 1 -- start start 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.743+0000 7f417dc86700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4178100da0 0x7f4178073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.743+0000 7f417dc86700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 0x7f41780735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.743+0000 7f417dc86700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4178073b10 con 0x7f4178101750 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.743+0000 7f417dc86700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4178073c50 con 0x7f4178100da0 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.743+0000 7f416effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 0x7f41780735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.743+0000 7f416effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 0x7f41780735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34258/0 (socket says 172.21.15.151:34258) 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.743+0000 7f416effd700 1 -- 172.21.15.151:0/393321996 learned_addr learned my addr 172.21.15.151:0/393321996 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f41777fe700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4178100da0 0x7f4178073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f416effd700 1 -- 172.21.15.151:0/393321996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4178100da0 msgr2=0x7f4178073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f416effd700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4178100da0 0x7f4178073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f416effd700 1 -- 172.21.15.151:0/393321996 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41600095f0 con 0x7f4178101750 2024-04-16T02:45:38.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f416effd700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 0x7f41780735d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41600059d0 tx=0x7f416000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:38.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f41757fa700 1 -- 172.21.15.151:0/393321996 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f416001d070 con 0x7f4178101750 2024-04-16T02:45:38.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f41757fa700 1 -- 172.21.15.151:0/393321996 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f416000b610 con 0x7f4178101750 2024-04-16T02:45:38.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f41757fa700 1 -- 172.21.15.151:0/393321996 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4160022470 con 0x7f4178101750 2024-04-16T02:45:38.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4178073ed0 con 0x7f4178101750 2024-04-16T02:45:38.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.744+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41781a2c30 con 0x7f4178101750 2024-04-16T02:45:38.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.745+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41781091a0 con 0x7f4178101750 2024-04-16T02:45:38.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.746+0000 7f41757fa700 1 -- 172.21.15.151:0/393321996 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4160003710 con 0x7f4178101750 2024-04-16T02:45:38.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.746+0000 7f41757fa700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416406ecc0 0x7f4164071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:38.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.746+0000 7f41757fa700 1 -- 172.21.15.151:0/393321996 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f41600902e0 con 0x7f4178101750 2024-04-16T02:45:38.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.746+0000 7f41777fe700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416406ecc0 0x7f4164071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:38.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.747+0000 7f41777fe700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416406ecc0 0x7f4164071180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4168009c60 tx=0x7f4168009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:38.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.751+0000 7f41757fa700 1 -- 172.21.15.151:0/393321996 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4160059090 con 0x7f4178101750 2024-04-16T02:45:38.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:38 smithi151 ceph-mon[112233]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:38.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:38 smithi151 ceph-mon[112233]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:38.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:38 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/52049954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:38.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:45:38.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.996+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f417802cc70 con 0x7f4178101750 2024-04-16T02:45:39.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.997+0000 7f41757fa700 1 -- 172.21.15.151:0/393321996 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f416005c750 con 0x7f4178101750 2024-04-16T02:45:39.000 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:45:39.000 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:45:39.000 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:45:39.000 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:45:39.000 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:45:39.000 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:45:39.000 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:45:39.001 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:45:39.002 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:45:39.003 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:45:39.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416406ecc0 msgr2=0x7f4164071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416406ecc0 0x7f4164071180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4168009c60 tx=0x7f4168009260 comp rx=0 tx=0).stop 2024-04-16T02:45:39.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 msgr2=0x7f41780735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 0x7f41780735d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41600059d0 tx=0x7f416000f690 comp rx=0 tx=0).stop 2024-04-16T02:45:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 shutdown_connections 2024-04-16T02:45:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416406ecc0 0x7f4164071180 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4178100da0 0x7f4178073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 --2- 172.21.15.151:0/393321996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178101750 0x7f41780735d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 >> 172.21.15.151:0/393321996 conn(0x7f41780fc630 msgr2=0x7f41780fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 shutdown_connections 2024-04-16T02:45:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:38.999+0000 7f417dc86700 1 -- 172.21.15.151:0/393321996 wait complete. 2024-04-16T02:45:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:45:39.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.106+0000 7fb20a7ee700 1 -- 172.21.15.151:0/158374581 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb204105a40 msgr2=0x7fb204105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.106+0000 7fb20a7ee700 1 --2- 172.21.15.151:0/158374581 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb204105a40 0x7fb204105e20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009910 tx=0x7fb1f4009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:39.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.106+0000 7fb20a7ee700 1 -- 172.21.15.151:0/158374581 shutdown_connections 2024-04-16T02:45:39.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.106+0000 7fb20a7ee700 1 --2- 172.21.15.151:0/158374581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2040ffa40 0x7fb2040ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.106+0000 7fb20a7ee700 1 --2- 172.21.15.151:0/158374581 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb204105a40 0x7fb204105e20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.106+0000 7fb20a7ee700 1 -- 172.21.15.151:0/158374581 >> 172.21.15.151:0/158374581 conn(0x7fb2040fb550 msgr2=0x7fb2040fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:39.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.106+0000 7fb20a7ee700 1 -- 172.21.15.151:0/158374581 shutdown_connections 2024-04-16T02:45:39.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.106+0000 7fb20a7ee700 1 -- 172.21.15.151:0/158374581 wait complete. 2024-04-16T02:45:39.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.107+0000 7fb20a7ee700 1 Processor -- start 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.107+0000 7fb20a7ee700 1 -- start start 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.107+0000 7fb20a7ee700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2040ffa40 0x7fb204197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.107+0000 7fb20a7ee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb204105a40 0x7fb204197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.107+0000 7fb20a7ee700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2041980d0 con 0x7fb2040ffa40 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.107+0000 7fb20a7ee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb204198240 con 0x7fb204105a40 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb203fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2040ffa40 0x7fb204197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb203fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2040ffa40 0x7fb204197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34276/0 (socket says 172.21.15.151:34276) 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb203fff700 1 -- 172.21.15.151:0/1606221493 learned_addr learned my addr 172.21.15.151:0/1606221493 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb2037fe700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb204105a40 0x7fb204197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb203fff700 1 -- 172.21.15.151:0/1606221493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb204105a40 msgr2=0x7fb204197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb203fff700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb204105a40 0x7fb204197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb203fff700 1 -- 172.21.15.151:0/1606221493 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f40095f0 con 0x7fb2040ffa40 2024-04-16T02:45:39.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb203fff700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2040ffa40 0x7fb204197570 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4006010 tx=0x7fb1f400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:39.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb2017fa700 1 -- 172.21.15.151:0/1606221493 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f401d070 con 0x7fb2040ffa40 2024-04-16T02:45:39.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb2017fa700 1 -- 172.21.15.151:0/1606221493 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1f4005dc0 con 0x7fb2040ffa40 2024-04-16T02:45:39.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb2017fa700 1 -- 172.21.15.151:0/1606221493 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f4017400 con 0x7fb2040ffa40 2024-04-16T02:45:39.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2041a05e0 con 0x7fb2040ffa40 2024-04-16T02:45:39.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.108+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2041a0a50 con 0x7fb2040ffa40 2024-04-16T02:45:39.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.109+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb204108040 con 0x7fb2040ffa40 2024-04-16T02:45:39.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.110+0000 7fb2017fa700 1 -- 172.21.15.151:0/1606221493 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb1f4003a60 con 0x7fb2040ffa40 2024-04-16T02:45:39.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.110+0000 7fb2017fa700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb1f006ecc0 0x7fb1f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:39.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.110+0000 7fb2017fa700 1 -- 172.21.15.151:0/1606221493 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb1f4090200 con 0x7fb2040ffa40 2024-04-16T02:45:39.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.110+0000 7fb2037fe700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb1f006ecc0 0x7fb1f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:39.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.111+0000 7fb2037fe700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb1f006ecc0 0x7fb1f0071180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb204069680 tx=0x7fb1ec00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:39.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.115+0000 7fb2017fa700 1 -- 172.21.15.151:0/1606221493 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb1f4059030 con 0x7fb2040ffa40 2024-04-16T02:45:39.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:38 smithi153 ceph-mon[87858]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:39.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:38 smithi153 ceph-mon[87858]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:39.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:38 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/52049954' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:45:39.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:45:39.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.332+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb20402cd30 con 0x7fb1f006ecc0 2024-04-16T02:45:39.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.334+0000 7fb2017fa700 1 -- 172.21.15.151:0/1606221493 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb20402cd30 con 0x7fb1f006ecc0 2024-04-16T02:45:39.337 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:45:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:45:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:45:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:45:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:45:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:45:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:45:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:45:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:45:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:45:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:45:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:45:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:45:39.338 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:45:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb1f006ecc0 msgr2=0x7fb1f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb1f006ecc0 0x7fb1f0071180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb204069680 tx=0x7fb1ec00b410 comp rx=0 tx=0).stop 2024-04-16T02:45:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2040ffa40 msgr2=0x7fb204197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2040ffa40 0x7fb204197570 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4006010 tx=0x7fb1f400bd70 comp rx=0 tx=0).stop 2024-04-16T02:45:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 shutdown_connections 2024-04-16T02:45:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb1f006ecc0 0x7fb1f0071180 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2040ffa40 0x7fb204197570 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 --2- 172.21.15.151:0/1606221493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb204105a40 0x7fb204197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 >> 172.21.15.151:0/1606221493 conn(0x7fb2040fb550 msgr2=0x7fb2040fbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:39.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 shutdown_connections 2024-04-16T02:45:39.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.337+0000 7fb20a7ee700 1 -- 172.21.15.151:0/1606221493 wait complete. 2024-04-16T02:45:39.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.442+0000 7f264ced8700 1 -- 172.21.15.151:0/3383764522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 msgr2=0x7f26481011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.442+0000 7f264ced8700 1 --2- 172.21.15.151:0/3383764522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 0x7f26481011b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2638009910 tx=0x7f2638009c20 comp rx=0 tx=0).stop 2024-04-16T02:45:39.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.442+0000 7f264ced8700 1 -- 172.21.15.151:0/3383764522 shutdown_connections 2024-04-16T02:45:39.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.442+0000 7f264ced8700 1 --2- 172.21.15.151:0/3383764522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2648101780 0x7f26481057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.442+0000 7f264ced8700 1 --2- 172.21.15.151:0/3383764522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 0x7f26481011b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.442+0000 7f264ced8700 1 -- 172.21.15.151:0/3383764522 >> 172.21.15.151:0/3383764522 conn(0x7f26480fc640 msgr2=0x7f26480fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:39.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.442+0000 7f264ced8700 1 -- 172.21.15.151:0/3383764522 shutdown_connections 2024-04-16T02:45:39.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.442+0000 7f264ced8700 1 -- 172.21.15.151:0/3383764522 wait complete. 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.443+0000 7f264ced8700 1 Processor -- start 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.443+0000 7f264ced8700 1 -- start start 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.443+0000 7f264ced8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 0x7f264819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.443+0000 7f264ced8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2648101780 0x7f264819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.443+0000 7f264ced8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264819d100 con 0x7f2648100dd0 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.443+0000 7f264ced8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26481965b0 con 0x7f2648101780 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f264659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 0x7f264819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f264659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 0x7f264819c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34288/0 (socket says 172.21.15.151:34288) 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f264659c700 1 -- 172.21.15.151:0/380343841 learned_addr learned my addr 172.21.15.151:0/380343841 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f2645d9b700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2648101780 0x7f264819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f264659c700 1 -- 172.21.15.151:0/380343841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2648101780 msgr2=0x7f264819ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f264659c700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2648101780 0x7f264819ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f264659c700 1 -- 172.21.15.151:0/380343841 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26380095f0 con 0x7f2648100dd0 2024-04-16T02:45:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f264659c700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 0x7f264819c530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2638006010 tx=0x7f263800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f26377fe700 1 -- 172.21.15.151:0/380343841 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f263801d070 con 0x7f2648100dd0 2024-04-16T02:45:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2648196830 con 0x7f2648100dd0 2024-04-16T02:45:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f26377fe700 1 -- 172.21.15.151:0/380343841 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2638004fb0 con 0x7f2648100dd0 2024-04-16T02:45:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f26377fe700 1 -- 172.21.15.151:0/380343841 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2638022ea0 con 0x7f2648100dd0 2024-04-16T02:45:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.444+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2648196d20 con 0x7f2648100dd0 2024-04-16T02:45:39.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.445+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2648109170 con 0x7f2648100dd0 2024-04-16T02:45:39.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.446+0000 7f26377fe700 1 -- 172.21.15.151:0/380343841 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2638022470 con 0x7f2648100dd0 2024-04-16T02:45:39.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.446+0000 7f26377fe700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f263006ecc0 0x7f2630071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:45:39.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.446+0000 7f26377fe700 1 -- 172.21.15.151:0/380343841 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f26380903f0 con 0x7f2648100dd0 2024-04-16T02:45:39.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.446+0000 7f2645d9b700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f263006ecc0 0x7f2630071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:45:39.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.447+0000 7f2645d9b700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f263006ecc0 0x7f2630071180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f26480fce60 tx=0x7f263c005eb0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:45:39.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.451+0000 7f26377fe700 1 -- 172.21.15.151:0/380343841 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f263805cd80 con 0x7f2648100dd0 2024-04-16T02:45:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.735+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f264804ea90 con 0x7f2648100dd0 2024-04-16T02:45:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.736+0000 7f26377fe700 1 -- 172.21.15.151:0/380343841 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2638022720 con 0x7f2648100dd0 2024-04-16T02:45:39.739 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:45:39.739 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:45:39.739 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:45:39.739 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:45:39.739 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:45:39.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f263006ecc0 msgr2=0x7f2630071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f263006ecc0 0x7f2630071180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f26480fce60 tx=0x7f263c005eb0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 msgr2=0x7f264819c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:45:39.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 0x7f264819c530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2638006010 tx=0x7f263800be20 comp rx=0 tx=0).stop 2024-04-16T02:45:39.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 shutdown_connections 2024-04-16T02:45:39.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f263006ecc0 0x7f2630071180 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2648100dd0 0x7f264819c530 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 --2- 172.21.15.151:0/380343841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2648101780 0x7f264819ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:45:39.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 >> 172.21.15.151:0/380343841 conn(0x7f26480fc640 msgr2=0x7f26480fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:45:39.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 shutdown_connections 2024-04-16T02:45:39.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:45:39.738+0000 7f264ced8700 1 -- 172.21.15.151:0/380343841 wait complete. 2024-04-16T02:45:39.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:39 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/393321996' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:45:39.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:39 smithi151 ceph-mon[112233]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:39.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:39 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/380343841' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:45:39.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:39 smithi151 ceph-mon[112233]: pgmap v211: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:39 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/393321996' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:45:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:39 smithi153 ceph-mon[87858]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:45:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:39 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/380343841' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:45:40.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:39 smithi153 ceph-mon[87858]: pgmap v211: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:42.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:41 smithi151 ceph-mon[112233]: pgmap v212: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:41 smithi153 ceph-mon[87858]: pgmap v212: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:44.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:43 smithi151 ceph-mon[112233]: pgmap v213: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:44.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:43 smithi153 ceph-mon[87858]: pgmap v213: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:46.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:45 smithi151 ceph-mon[112233]: pgmap v214: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:46.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:45 smithi153 ceph-mon[87858]: pgmap v214: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:47 smithi151 ceph-mon[112233]: pgmap v215: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:48.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:47 smithi153 ceph-mon[87858]: pgmap v215: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:50.129 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:49 smithi153 ceph-mon[87858]: pgmap v216: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:49 smithi151 ceph-mon[112233]: pgmap v216: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:51 smithi151 ceph-mon[112233]: pgmap v217: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:51 smithi153 ceph-mon[87858]: pgmap v217: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:54.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:45:54.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:53 smithi151 ceph-mon[112233]: pgmap v218: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:45:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:53 smithi153 ceph-mon[87858]: pgmap v218: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:56.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:55 smithi151 ceph-mon[112233]: pgmap v219: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:56.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:55 smithi153 ceph-mon[87858]: pgmap v219: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:57 smithi151 ceph-mon[112233]: pgmap v220: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:45:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:57 smithi153 ceph-mon[87858]: pgmap v220: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:00.235 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:45:59 smithi153 ceph-mon[87858]: pgmap v221: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:45:59 smithi151 ceph-mon[112233]: pgmap v221: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:02.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:01 smithi151 ceph-mon[112233]: pgmap v222: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:01 smithi153 ceph-mon[87858]: pgmap v222: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:04.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:03 smithi151 ceph-mon[112233]: pgmap v223: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:04.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:03 smithi153 ceph-mon[87858]: pgmap v223: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:05 smithi151 ceph-mon[112233]: pgmap v224: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:05 smithi153 ceph-mon[87858]: pgmap v224: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:08.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:07 smithi151 ceph-mon[112233]: pgmap v225: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:08.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:07 smithi153 ceph-mon[87858]: pgmap v225: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:09.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:46:09.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:46:09.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.857+0000 7f6254cf7700 1 -- 172.21.15.151:0/2388519917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 msgr2=0x7f62501011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:09.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.857+0000 7f6254cf7700 1 --2- 172.21.15.151:0/2388519917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 0x7f62501011a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f624000b0a0 tx=0x7f624000b3b0 comp rx=0 tx=0).stop 2024-04-16T02:46:09.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.857+0000 7f6254cf7700 1 -- 172.21.15.151:0/2388519917 shutdown_connections 2024-04-16T02:46:09.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.857+0000 7f6254cf7700 1 --2- 172.21.15.151:0/2388519917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6250101770 0x7f62501057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:09.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.857+0000 7f6254cf7700 1 --2- 172.21.15.151:0/2388519917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 0x7f62501011a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:09.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.857+0000 7f6254cf7700 1 -- 172.21.15.151:0/2388519917 >> 172.21.15.151:0/2388519917 conn(0x7f62500fc650 msgr2=0x7f62500fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:09.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.857+0000 7f6254cf7700 1 -- 172.21.15.151:0/2388519917 shutdown_connections 2024-04-16T02:46:09.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.857+0000 7f6254cf7700 1 -- 172.21.15.151:0/2388519917 wait complete. 2024-04-16T02:46:09.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.858+0000 7f6254cf7700 1 Processor -- start 2024-04-16T02:46:09.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f6254cf7700 1 -- start start 2024-04-16T02:46:09.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f6254cf7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 0x7f6250073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f6254cf7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6250101770 0x7f62500735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f6254cf7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6250073b10 con 0x7f6250100dc0 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f6254cf7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6250073c50 con 0x7f6250101770 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f624e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 0x7f6250073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f624e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 0x7f6250073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50234/0 (socket says 172.21.15.151:50234) 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f624e59c700 1 -- 172.21.15.151:0/2405094054 learned_addr learned my addr 172.21.15.151:0/2405094054 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f6245d9b700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6250101770 0x7f62500735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f624e59c700 1 -- 172.21.15.151:0/2405094054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6250101770 msgr2=0x7f62500735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f624e59c700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6250101770 0x7f62500735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.859+0000 7f624e59c700 1 -- 172.21.15.151:0/2405094054 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6240009d00 con 0x7f6250100dc0 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.860+0000 7f624e59c700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 0x7f6250073090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6240000f80 tx=0x7f624000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.860+0000 7f6247fff700 1 -- 172.21.15.151:0/2405094054 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f624000e040 con 0x7f6250100dc0 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.860+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6250073ed0 con 0x7f6250100dc0 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.860+0000 7f6247fff700 1 -- 172.21.15.151:0/2405094054 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6240003de0 con 0x7f6250100dc0 2024-04-16T02:46:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.860+0000 7f6247fff700 1 -- 172.21.15.151:0/2405094054 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f624001d7d0 con 0x7f6250100dc0 2024-04-16T02:46:09.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.860+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62501a2c20 con 0x7f6250100dc0 2024-04-16T02:46:09.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.861+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6250109160 con 0x7f6250100dc0 2024-04-16T02:46:09.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.862+0000 7f6247fff700 1 -- 172.21.15.151:0/2405094054 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6240004040 con 0x7f6250100dc0 2024-04-16T02:46:09.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.862+0000 7f6247fff700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f623c06ecc0 0x7f623c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:09.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.862+0000 7f6247fff700 1 -- 172.21.15.151:0/2405094054 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f62400907a0 con 0x7f6250100dc0 2024-04-16T02:46:09.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.862+0000 7f6245d9b700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f623c06ecc0 0x7f623c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:09.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.863+0000 7f6245d9b700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f623c06ecc0 0x7f623c071180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6250074880 tx=0x7f623800a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:09.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:09.867+0000 7f6247fff700 1 -- 172.21.15.151:0/2405094054 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6240059550 con 0x7f6250100dc0 2024-04-16T02:46:10.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.083+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f625002cce0 con 0x7f623c06ecc0 2024-04-16T02:46:10.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.085+0000 7f6247fff700 1 -- 172.21.15.151:0/2405094054 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f625002cce0 con 0x7f623c06ecc0 2024-04-16T02:46:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f623c06ecc0 msgr2=0x7f623c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f623c06ecc0 0x7f623c071180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6250074880 tx=0x7f623800a380 comp rx=0 tx=0).stop 2024-04-16T02:46:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 msgr2=0x7f6250073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 0x7f6250073090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6240000f80 tx=0x7f624000cd20 comp rx=0 tx=0).stop 2024-04-16T02:46:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 shutdown_connections 2024-04-16T02:46:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f623c06ecc0 0x7f623c071180 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6250100dc0 0x7f6250073090 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 --2- 172.21.15.151:0/2405094054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6250101770 0x7f62500735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 >> 172.21.15.151:0/2405094054 conn(0x7f62500fc650 msgr2=0x7f62500fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 shutdown_connections 2024-04-16T02:46:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.087+0000 7f6254cf7700 1 -- 172.21.15.151:0/2405094054 wait complete. 2024-04-16T02:46:10.101 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:46:10.197 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.194+0000 7fd82a8e0700 1 -- 172.21.15.151:0/1704085428 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 msgr2=0x7fd8240ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.197 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.194+0000 7fd82a8e0700 1 --2- 172.21.15.151:0/1704085428 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 0x7fd8240ffc70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd814009910 tx=0x7fd814009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:10.197 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:09 smithi151 ceph-mon[112233]: pgmap v226: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.194+0000 7fd82a8e0700 1 -- 172.21.15.151:0/1704085428 shutdown_connections 2024-04-16T02:46:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.194+0000 7fd82a8e0700 1 --2- 172.21.15.151:0/1704085428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8241001b0 0x7fd82410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.194+0000 7fd82a8e0700 1 --2- 172.21.15.151:0/1704085428 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 0x7fd8240ffc70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.194+0000 7fd82a8e0700 1 -- 172.21.15.151:0/1704085428 >> 172.21.15.151:0/1704085428 conn(0x7fd8240fb530 msgr2=0x7fd8240fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.194+0000 7fd82a8e0700 1 -- 172.21.15.151:0/1704085428 shutdown_connections 2024-04-16T02:46:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.194+0000 7fd82a8e0700 1 -- 172.21.15.151:0/1704085428 wait complete. 2024-04-16T02:46:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd82a8e0700 1 Processor -- start 2024-04-16T02:46:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd82a8e0700 1 -- start start 2024-04-16T02:46:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd82a8e0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 0x7fd82419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd82a8e0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8241001b0 0x7fd82419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd82a8e0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82419f270 con 0x7fd8240ff890 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd82a8e0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd824198790 con 0x7fd8241001b0 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd823fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 0x7fd82419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd823fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 0x7fd82419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50244/0 (socket says 172.21.15.151:50244) 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd823fff700 1 -- 172.21.15.151:0/3140097039 learned_addr learned my addr 172.21.15.151:0/3140097039 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd823fff700 1 -- 172.21.15.151:0/3140097039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8241001b0 msgr2=0x7fd82419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd823fff700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8241001b0 0x7fd82419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd823fff700 1 -- 172.21.15.151:0/3140097039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8140095f0 con 0x7fd8240ff890 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd823fff700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 0x7fd82419e6a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd814006010 tx=0x7fd81400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd821ffb700 1 -- 172.21.15.151:0/3140097039 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd81401d070 con 0x7fd8240ff890 2024-04-16T02:46:10.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd821ffb700 1 -- 172.21.15.151:0/3140097039 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd81400f460 con 0x7fd8240ff890 2024-04-16T02:46:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.195+0000 7fd821ffb700 1 -- 172.21.15.151:0/3140097039 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd814017630 con 0x7fd8240ff890 2024-04-16T02:46:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.196+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd824198a10 con 0x7fd8240ff890 2024-04-16T02:46:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.196+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd824198eb0 con 0x7fd8240ff890 2024-04-16T02:46:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.197+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82410a670 con 0x7fd8240ff890 2024-04-16T02:46:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.197+0000 7fd821ffb700 1 -- 172.21.15.151:0/3140097039 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd81400f5d0 con 0x7fd8240ff890 2024-04-16T02:46:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.198+0000 7fd821ffb700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ecc0 0x7fd810071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.198+0000 7fd821ffb700 1 -- 172.21.15.151:0/3140097039 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd814090290 con 0x7fd8240ff890 2024-04-16T02:46:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.198+0000 7fd81bfff700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ecc0 0x7fd810071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.198+0000 7fd81bfff700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ecc0 0x7fd810071180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd80c005950 tx=0x7fd80c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:10.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.202+0000 7fd821ffb700 1 -- 172.21.15.151:0/3140097039 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd81405cbf0 con 0x7fd8240ff890 2024-04-16T02:46:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:09 smithi153 ceph-mon[87858]: pgmap v226: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.420+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd82402cce0 con 0x7fd81006ecc0 2024-04-16T02:46:10.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.422+0000 7fd821ffb700 1 -- 172.21.15.151:0/3140097039 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd82402cce0 con 0x7fd81006ecc0 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ecc0 msgr2=0x7fd810071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ecc0 0x7fd810071180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd80c005950 tx=0x7fd80c0058e0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 msgr2=0x7fd82419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 0x7fd82419e6a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd814006010 tx=0x7fd81400be20 comp rx=0 tx=0).stop 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 shutdown_connections 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ecc0 0x7fd810071180 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8240ff890 0x7fd82419e6a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 --2- 172.21.15.151:0/3140097039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8241001b0 0x7fd82419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 >> 172.21.15.151:0/3140097039 conn(0x7fd8240fb530 msgr2=0x7fd8240fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:10.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 shutdown_connections 2024-04-16T02:46:10.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.424+0000 7fd82a8e0700 1 -- 172.21.15.151:0/3140097039 wait complete. 2024-04-16T02:46:10.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.530+0000 7fc275119700 1 -- 172.21.15.151:0/991253562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 msgr2=0x7fc27010cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.530+0000 7fc275119700 1 --2- 172.21.15.151:0/991253562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 0x7fc27010cec0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc264009910 tx=0x7fc264009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:10.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.530+0000 7fc275119700 1 -- 172.21.15.151:0/991253562 shutdown_connections 2024-04-16T02:46:10.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.530+0000 7fc275119700 1 --2- 172.21.15.151:0/991253562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 0x7fc27010cec0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.530+0000 7fc275119700 1 --2- 172.21.15.151:0/991253562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc270073160 0x7fc270073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.530+0000 7fc275119700 1 -- 172.21.15.151:0/991253562 >> 172.21.15.151:0/991253562 conn(0x7fc2700fc650 msgr2=0x7fc2700fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:10.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.530+0000 7fc275119700 1 -- 172.21.15.151:0/991253562 shutdown_connections 2024-04-16T02:46:10.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.530+0000 7fc275119700 1 -- 172.21.15.151:0/991253562 wait complete. 2024-04-16T02:46:10.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.531+0000 7fc275119700 1 Processor -- start 2024-04-16T02:46:10.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.531+0000 7fc275119700 1 -- start start 2024-04-16T02:46:10.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.531+0000 7fc275119700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc270073160 0x7fc27019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:10.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.531+0000 7fc275119700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 0x7fc27019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:10.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.531+0000 7fc275119700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27019f350 con 0x7fc270073a80 2024-04-16T02:46:10.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.531+0000 7fc275119700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc270198800 con 0x7fc270073160 2024-04-16T02:46:10.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc26e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 0x7fc27019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc26e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 0x7fc27019ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50264/0 (socket says 172.21.15.151:50264) 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc26e59c700 1 -- 172.21.15.151:0/3823220548 learned_addr learned my addr 172.21.15.151:0/3823220548 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc26ed9d700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc270073160 0x7fc27019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc26e59c700 1 -- 172.21.15.151:0/3823220548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc270073160 msgr2=0x7fc27019e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc26e59c700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc270073160 0x7fc27019e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc26e59c700 1 -- 172.21.15.151:0/3823220548 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2640095f0 con 0x7fc270073a80 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc26ed9d700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc270073160 0x7fc27019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc26e59c700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 0x7fc27019ecc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc26400bd70 tx=0x7fc26400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc257fff700 1 -- 172.21.15.151:0/3823220548 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc26401d070 con 0x7fc270073a80 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc257fff700 1 -- 172.21.15.151:0/3823220548 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc26400f460 con 0x7fc270073a80 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc270198a80 con 0x7fc270073a80 2024-04-16T02:46:10.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc257fff700 1 -- 172.21.15.151:0/3823220548 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc264017630 con 0x7fc270073a80 2024-04-16T02:46:10.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.532+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc270198f70 con 0x7fc270073a80 2024-04-16T02:46:10.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.533+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27010a5c0 con 0x7fc270073a80 2024-04-16T02:46:10.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.534+0000 7fc257fff700 1 -- 172.21.15.151:0/3823220548 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc264003a60 con 0x7fc270073a80 2024-04-16T02:46:10.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.534+0000 7fc257fff700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc25806ecc0 0x7fc258071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:10.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.534+0000 7fc257fff700 1 -- 172.21.15.151:0/3823220548 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc264090480 con 0x7fc270073a80 2024-04-16T02:46:10.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.534+0000 7fc26ed9d700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc25806ecc0 0x7fc258071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:10.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.535+0000 7fc26ed9d700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc25806ecc0 0x7fc258071180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc260005950 tx=0x7fc260005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:10.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.539+0000 7fc257fff700 1 -- 172.21.15.151:0/3823220548 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc26405cd60 con 0x7fc270073a80 2024-04-16T02:46:10.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.746+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc270199250 con 0x7fc25806ecc0 2024-04-16T02:46:10.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.757+0000 7fc257fff700 1 -- 172.21.15.151:0/3823220548 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fc270199250 con 0x7fc25806ecc0 2024-04-16T02:46:10.760 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:46:10.760 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (7m) 71s ago 19m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (19m) 71s ago 19m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (18m) 53s ago 18m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (5m) 71s ago 19m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (5m) 53s ago 18m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (7m) 71s ago 18m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (96s) 71s ago 13m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (76s) 71s ago 13m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (13m) 53s ago 13m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (57s) 53s ago 13m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (9m) 71s ago 20m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (8m) 53s ago 18m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (6m) 71s ago 20m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (6m) 53s ago 17m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (8m) 71s ago 19m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (8m) 53s ago 18m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (4m) 71s ago 17m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:46:10.761 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (4m) 71s ago 16m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:46:10.762 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (3m) 71s ago 16m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:46:10.762 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (3m) 53s ago 15m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:46:10.762 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (2m) 53s ago 15m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:46:10.762 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (115s) 53s ago 14m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:46:10.762 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (8m) 71s ago 18m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.759+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc25806ecc0 msgr2=0x7fc258071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.759+0000 7fc275119700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc25806ecc0 0x7fc258071180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc260005950 tx=0x7fc260005f50 comp rx=0 tx=0).stop 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.759+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 msgr2=0x7fc27019ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.759+0000 7fc275119700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 0x7fc27019ecc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc26400bd70 tx=0x7fc26400be50 comp rx=0 tx=0).stop 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.759+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 shutdown_connections 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.759+0000 7fc275119700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc25806ecc0 0x7fc258071180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.759+0000 7fc275119700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc270073160 0x7fc27019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.759+0000 7fc275119700 1 --2- 172.21.15.151:0/3823220548 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc270073a80 0x7fc27019ecc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.759+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 >> 172.21.15.151:0/3823220548 conn(0x7fc2700fc650 msgr2=0x7fc270107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:10.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.760+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 shutdown_connections 2024-04-16T02:46:10.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.760+0000 7fc275119700 1 -- 172.21.15.151:0/3823220548 wait complete. 2024-04-16T02:46:10.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.864+0000 7f34d7da3700 1 -- 172.21.15.151:0/754586917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d00ffa30 msgr2=0x7f34d00ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.864+0000 7f34d7da3700 1 --2- 172.21.15.151:0/754586917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d00ffa30 0x7f34d00ffeb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009910 tx=0x7f34c4009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:10.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.864+0000 7f34d7da3700 1 -- 172.21.15.151:0/754586917 shutdown_connections 2024-04-16T02:46:10.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.864+0000 7f34d7da3700 1 --2- 172.21.15.151:0/754586917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d00ffa30 0x7f34d00ffeb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.864+0000 7f34d7da3700 1 --2- 172.21.15.151:0/754586917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d0105a30 0x7f34d0105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.864+0000 7f34d7da3700 1 -- 172.21.15.151:0/754586917 >> 172.21.15.151:0/754586917 conn(0x7f34d00fb540 msgr2=0x7f34d00fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:10.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.864+0000 7f34d7da3700 1 -- 172.21.15.151:0/754586917 shutdown_connections 2024-04-16T02:46:10.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.864+0000 7f34d7da3700 1 -- 172.21.15.151:0/754586917 wait complete. 2024-04-16T02:46:10.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.864+0000 7f34d7da3700 1 Processor -- start 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d7da3700 1 -- start start 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d7da3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d00ffa30 0x7f34d00fbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d7da3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d0105a30 0x7f34d00fc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d7da3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00fc910 con 0x7f34d0105a30 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d7da3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00fca50 con 0x7f34d00ffa30 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d533e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d0105a30 0x7f34d00fc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d533e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d0105a30 0x7f34d00fc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50280/0 (socket says 172.21.15.151:50280) 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d533e700 1 -- 172.21.15.151:0/4268958216 learned_addr learned my addr 172.21.15.151:0/4268958216 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d5b3f700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d00ffa30 0x7f34d00fbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d533e700 1 -- 172.21.15.151:0/4268958216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d00ffa30 msgr2=0x7f34d00fbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d533e700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d00ffa30 0x7f34d00fbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d533e700 1 -- 172.21.15.151:0/4268958216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c40095f0 con 0x7f34d0105a30 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.865+0000 7f34d533e700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d0105a30 0x7f34d00fc3d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f34c4006010 tx=0x7f34c400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.866+0000 7f34ceffd700 1 -- 172.21.15.151:0/4268958216 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34c401d070 con 0x7f34d0105a30 2024-04-16T02:46:10.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.866+0000 7f34ceffd700 1 -- 172.21.15.151:0/4268958216 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34c4004fb0 con 0x7f34d0105a30 2024-04-16T02:46:10.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.866+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d00fccd0 con 0x7f34d0105a30 2024-04-16T02:46:10.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.866+0000 7f34ceffd700 1 -- 172.21.15.151:0/4268958216 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34c4022ea0 con 0x7f34d0105a30 2024-04-16T02:46:10.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.866+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d01a2c10 con 0x7f34d0105a30 2024-04-16T02:46:10.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.867+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d0108fd0 con 0x7f34d0105a30 2024-04-16T02:46:10.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.868+0000 7f34ceffd700 1 -- 172.21.15.151:0/4268958216 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f34c4022470 con 0x7f34d0105a30 2024-04-16T02:46:10.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.868+0000 7f34ceffd700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34b806ecc0 0x7f34b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:10.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.868+0000 7f34ceffd700 1 -- 172.21.15.151:0/4268958216 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f34c4090350 con 0x7f34d0105a30 2024-04-16T02:46:10.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.868+0000 7f34d5b3f700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34b806ecc0 0x7f34b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:10.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.869+0000 7f34d5b3f700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34b806ecc0 0x7f34b8071180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f34c00060b0 tx=0x7f34c0007b80 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:10.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:10.873+0000 7f34ceffd700 1 -- 172.21.15.151:0/4268958216 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f34c4059230 con 0x7f34d0105a30 2024-04-16T02:46:11.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:10 smithi151 ceph-mon[112233]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:11.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:10 smithi151 ceph-mon[112233]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:11.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:10 smithi151 ceph-mon[112233]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:11.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.155+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f34d004ea90 con 0x7f34d0105a30 2024-04-16T02:46:11.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.155+0000 7f34ceffd700 1 -- 172.21.15.151:0/4268958216 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f34c4022720 con 0x7f34d0105a30 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:46:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:46:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:46:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:46:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:46:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:46:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:46:11.160 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34b806ecc0 msgr2=0x7f34b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34b806ecc0 0x7f34b8071180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f34c00060b0 tx=0x7f34c0007b80 comp rx=0 tx=0).stop 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d0105a30 msgr2=0x7f34d00fc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d0105a30 0x7f34d00fc3d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f34c4006010 tx=0x7f34c400be20 comp rx=0 tx=0).stop 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 shutdown_connections 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34b806ecc0 0x7f34b8071180 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d00ffa30 0x7f34d00fbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 --2- 172.21.15.151:0/4268958216 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34d0105a30 0x7f34d00fc3d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 >> 172.21.15.151:0/4268958216 conn(0x7f34d00fb540 msgr2=0x7f34d00690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 shutdown_connections 2024-04-16T02:46:11.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.158+0000 7f34d7da3700 1 -- 172.21.15.151:0/4268958216 wait complete. 2024-04-16T02:46:11.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.262+0000 7f0037179700 1 -- 172.21.15.151:0/991543438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100360 msgr2=0x7f0030100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.262+0000 7f0037179700 1 --2- 172.21.15.151:0/991543438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100360 0x7f0030100740 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0024009910 tx=0x7f0024009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 -- 172.21.15.151:0/991543438 shutdown_connections 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 --2- 172.21.15.151:0/991543438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0030100d10 0x7f00301046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 --2- 172.21.15.151:0/991543438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100360 0x7f0030100740 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 -- 172.21.15.151:0/991543438 >> 172.21.15.151:0/991543438 conn(0x7f0030075e30 msgr2=0x7f00300fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 -- 172.21.15.151:0/991543438 shutdown_connections 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 -- 172.21.15.151:0/991543438 wait complete. 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 Processor -- start 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 -- start start 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0030100360 0x7f0030196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.263+0000 7f0037179700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100d10 0x7f0030196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f0037179700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00301971d0 con 0x7f0030100d10 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f0037179700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0030197340 con 0x7f0030100360 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100d10 0x7f0030196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100d10 0x7f0030196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50304/0 (socket says 172.21.15.151:50304) 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002ffff700 1 -- 172.21.15.151:0/825294577 learned_addr learned my addr 172.21.15.151:0/825294577 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f0034f15700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0030100360 0x7f0030196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002ffff700 1 -- 172.21.15.151:0/825294577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0030100360 msgr2=0x7f0030196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002ffff700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0030100360 0x7f0030196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002ffff700 1 -- 172.21.15.151:0/825294577 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00240095f0 con 0x7f0030100d10 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002ffff700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100d10 0x7f0030196bd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f001c00c930 tx=0x7f001c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002dffb700 1 -- 172.21.15.151:0/825294577 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f001c007ad0 con 0x7f0030100d10 2024-04-16T02:46:11.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002dffb700 1 -- 172.21.15.151:0/825294577 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f001c00ce80 con 0x7f0030100d10 2024-04-16T02:46:11.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f002dffb700 1 -- 172.21.15.151:0/825294577 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f001c018630 con 0x7f0030100d10 2024-04-16T02:46:11.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.264+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003019a0e0 con 0x7f0030100d10 2024-04-16T02:46:11.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.265+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003019a6b0 con 0x7f0030100d10 2024-04-16T02:46:11.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.266+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0030109010 con 0x7f0030100d10 2024-04-16T02:46:11.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.266+0000 7f002dffb700 1 -- 172.21.15.151:0/825294577 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f001c007c30 con 0x7f0030100d10 2024-04-16T02:46:11.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.267+0000 7f002dffb700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f002006ecc0 0x7f0020071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:11.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.267+0000 7f002dffb700 1 -- 172.21.15.151:0/825294577 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f001c08edd0 con 0x7f0030100d10 2024-04-16T02:46:11.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.267+0000 7f0034f15700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f002006ecc0 0x7f0020071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:11.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.267+0000 7f0034f15700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f002006ecc0 0x7f0020071180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0024006010 tx=0x7f002401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:10 smithi153 ceph-mon[87858]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:10 smithi153 ceph-mon[87858]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:10 smithi153 ceph-mon[87858]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:11.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.271+0000 7f002dffb700 1 -- 172.21.15.151:0/825294577 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f001c057c00 con 0x7f0030100d10 2024-04-16T02:46:11.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.515+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f003004ea90 con 0x7f0030100d10 2024-04-16T02:46:11.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.515+0000 7f002dffb700 1 -- 172.21.15.151:0/825294577 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f001c05b2c0 con 0x7f0030100d10 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:46:11.519 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:46:11.520 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:46:11.521 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:46:11.522 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:46:11.522 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:46:11.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f002006ecc0 msgr2=0x7f0020071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f002006ecc0 0x7f0020071180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0024006010 tx=0x7f002401a040 comp rx=0 tx=0).stop 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100d10 msgr2=0x7f0030196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100d10 0x7f0030196bd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f001c00c930 tx=0x7f001c00cc40 comp rx=0 tx=0).stop 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 shutdown_connections 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f002006ecc0 0x7f0020071180 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0030100360 0x7f0030196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 --2- 172.21.15.151:0/825294577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0030100d10 0x7f0030196bd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 >> 172.21.15.151:0/825294577 conn(0x7f0030075e30 msgr2=0x7f00300ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 shutdown_connections 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.518+0000 7f0037179700 1 -- 172.21.15.151:0/825294577 wait complete. 2024-04-16T02:46:11.523 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:46:11.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.624+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1902384384 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0069020 msgr2=0x7fd5e0105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.624+0000 7fd5e84c7700 1 --2- 172.21.15.151:0/1902384384 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0069020 0x7fd5e0105790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009910 tx=0x7fd5d0009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:11.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.624+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1902384384 shutdown_connections 2024-04-16T02:46:11.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.624+0000 7fd5e84c7700 1 --2- 172.21.15.151:0/1902384384 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0069020 0x7fd5e0105790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.624+0000 7fd5e84c7700 1 --2- 172.21.15.151:0/1902384384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e0068670 0x7fd5e0068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.624+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1902384384 >> 172.21.15.151:0/1902384384 conn(0x7fd5e00755a0 msgr2=0x7fd5e00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.625+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1902384384 shutdown_connections 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.625+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1902384384 wait complete. 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.625+0000 7fd5e84c7700 1 Processor -- start 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.625+0000 7fd5e84c7700 1 -- start start 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.625+0000 7fd5e84c7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0068670 0x7fd5e00ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.625+0000 7fd5e84c7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e0069020 0x7fd5e01001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.625+0000 7fd5e84c7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e0100720 con 0x7fd5e0068670 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.625+0000 7fd5e84c7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e0100860 con 0x7fd5e0069020 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.625+0000 7fd5e6263700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0068670 0x7fd5e00ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5e6263700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0068670 0x7fd5e00ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50322/0 (socket says 172.21.15.151:50322) 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5e6263700 1 -- 172.21.15.151:0/1932540231 learned_addr learned my addr 172.21.15.151:0/1932540231 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5e5a62700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e0069020 0x7fd5e01001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5e6263700 1 -- 172.21.15.151:0/1932540231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e0069020 msgr2=0x7fd5e01001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5e6263700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e0069020 0x7fd5e01001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5e6263700 1 -- 172.21.15.151:0/1932540231 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d00095f0 con 0x7fd5e0068670 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5e6263700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0068670 0x7fd5e00ffca0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00e920 tx=0x7fd5dc00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5d77fe700 1 -- 172.21.15.151:0/1932540231 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5dc009790 con 0x7fd5e0068670 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5d77fe700 1 -- 172.21.15.151:0/1932540231 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5dc00cba0 con 0x7fd5e0068670 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5d77fe700 1 -- 172.21.15.151:0/1932540231 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5dc018650 con 0x7fd5e0068670 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e01a28d0 con 0x7fd5e0068670 2024-04-16T02:46:11.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.626+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e01a2c80 con 0x7fd5e0068670 2024-04-16T02:46:11.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.627+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e01090a0 con 0x7fd5e0068670 2024-04-16T02:46:11.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.628+0000 7fd5d77fe700 1 -- 172.21.15.151:0/1932540231 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd5dc00cd10 con 0x7fd5e0068670 2024-04-16T02:46:11.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.628+0000 7fd5d77fe700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd5cc06ecc0 0x7fd5cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:11.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.628+0000 7fd5e5a62700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd5cc06ecc0 0x7fd5cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:11.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.628+0000 7fd5d77fe700 1 -- 172.21.15.151:0/1932540231 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd5dc014070 con 0x7fd5e0068670 2024-04-16T02:46:11.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.629+0000 7fd5e5a62700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd5cc06ecc0 0x7fd5cc071180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009fd0 tx=0x7fd5d0005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:11.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.633+0000 7fd5d77fe700 1 -- 172.21.15.151:0/1932540231 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd5dc053ff0 con 0x7fd5e0068670 2024-04-16T02:46:11.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.853+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5e002cdd0 con 0x7fd5cc06ecc0 2024-04-16T02:46:11.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.854+0000 7fd5d77fe700 1 -- 172.21.15.151:0/1932540231 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd5e002cdd0 con 0x7fd5cc06ecc0 2024-04-16T02:46:11.857 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:46:11.857 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:46:11.857 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:46:11.857 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:46:11.857 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:46:11.857 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:46:11.857 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:46:11.857 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:46:11.858 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:46:11.858 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:46:11.858 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:46:11.858 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:46:11.858 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:46:11.858 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:46:11.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd5cc06ecc0 msgr2=0x7fd5cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd5cc06ecc0 0x7fd5cc071180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009fd0 tx=0x7fd5d0005940 comp rx=0 tx=0).stop 2024-04-16T02:46:11.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0068670 msgr2=0x7fd5e00ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0068670 0x7fd5e00ffca0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00e920 tx=0x7fd5dc00ece0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 shutdown_connections 2024-04-16T02:46:11.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd5cc06ecc0 0x7fd5cc071180 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5e0068670 0x7fd5e00ffca0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 --2- 172.21.15.151:0/1932540231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e0069020 0x7fd5e01001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 >> 172.21.15.151:0/1932540231 conn(0x7fd5e00755a0 msgr2=0x7fd5e0105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:11.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 shutdown_connections 2024-04-16T02:46:11.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.856+0000 7fd5e84c7700 1 -- 172.21.15.151:0/1932540231 wait complete. 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.962+0000 7f4308704700 1 -- 172.21.15.151:0/3793263920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 msgr2=0x7f4300100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.962+0000 7f4308704700 1 --2- 172.21.15.151:0/3793263920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 0x7f4300100050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009910 tx=0x7f42f0009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.962+0000 7f4308704700 1 -- 172.21.15.151:0/3793263920 shutdown_connections 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.962+0000 7f4308704700 1 --2- 172.21.15.151:0/3793263920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 0x7f4300100050 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.962+0000 7f4308704700 1 --2- 172.21.15.151:0/3793263920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4300105a40 0x7f4300105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.962+0000 7f4308704700 1 -- 172.21.15.151:0/3793263920 >> 172.21.15.151:0/3793263920 conn(0x7f43000755a0 msgr2=0x7f43000759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.962+0000 7f4308704700 1 -- 172.21.15.151:0/3793263920 shutdown_connections 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.962+0000 7f4308704700 1 -- 172.21.15.151:0/3793263920 wait complete. 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.962+0000 7f4308704700 1 Processor -- start 2024-04-16T02:46:11.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f4308704700 1 -- start start 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f4308704700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 0x7f430019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f4308704700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4300105a40 0x7f430019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f4308704700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430019d040 con 0x7f43000ffbd0 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f4308704700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4300196580 con 0x7f4300105a40 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f43064a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 0x7f430019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f43064a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 0x7f430019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50338/0 (socket says 172.21.15.151:50338) 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f43064a0700 1 -- 172.21.15.151:0/2524536215 learned_addr learned my addr 172.21.15.151:0/2524536215 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f4305c9f700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4300105a40 0x7f430019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f43064a0700 1 -- 172.21.15.151:0/2524536215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4300105a40 msgr2=0x7f430019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f43064a0700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4300105a40 0x7f430019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f43064a0700 1 -- 172.21.15.151:0/2524536215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f00095f0 con 0x7f43000ffbd0 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f4305c9f700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4300105a40 0x7f430019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:46:11.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f43064a0700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 0x7f430019c500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00c930 tx=0x7f42fc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:11.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f42f77fe700 1 -- 172.21.15.151:0/2524536215 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42fc007ad0 con 0x7f43000ffbd0 2024-04-16T02:46:11.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4300196860 con 0x7f43000ffbd0 2024-04-16T02:46:11.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.963+0000 7f42f77fe700 1 -- 172.21.15.151:0/2524536215 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42fc007c30 con 0x7f43000ffbd0 2024-04-16T02:46:11.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.964+0000 7f42f77fe700 1 -- 172.21.15.151:0/2524536215 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42fc018610 con 0x7f43000ffbd0 2024-04-16T02:46:11.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.964+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4300196db0 con 0x7f43000ffbd0 2024-04-16T02:46:11.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.964+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4300108fe0 con 0x7f43000ffbd0 2024-04-16T02:46:11.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.965+0000 7f42f77fe700 1 -- 172.21.15.151:0/2524536215 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f42fc00f450 con 0x7f43000ffbd0 2024-04-16T02:46:11.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.966+0000 7f42f77fe700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f42ec06ecc0 0x7f42ec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:11.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.966+0000 7f4305c9f700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f42ec06ecc0 0x7f42ec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:11.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.966+0000 7f42f77fe700 1 -- 172.21.15.151:0/2524536215 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f42fc08f080 con 0x7f43000ffbd0 2024-04-16T02:46:11.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.966+0000 7f4305c9f700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f42ec06ecc0 0x7f42ec071180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f42f0006010 tx=0x7f42f000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:11.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:11.970+0000 7f42f77fe700 1 -- 172.21.15.151:0/2524536215 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f42fc05b960 con 0x7f43000ffbd0 2024-04-16T02:46:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:11 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4268958216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:46:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:11 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/825294577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:46:12.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:11 smithi151 ceph-mon[112233]: pgmap v227: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:12.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.253+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f430004ea90 con 0x7f43000ffbd0 2024-04-16T02:46:12.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.253+0000 7f42f77fe700 1 -- 172.21.15.151:0/2524536215 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f42fc05b4f0 con 0x7f43000ffbd0 2024-04-16T02:46:12.256 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:46:12.256 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:46:12.256 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:46:12.256 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:46:12.256 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:46:12.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f42ec06ecc0 msgr2=0x7f42ec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:12.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f42ec06ecc0 0x7f42ec071180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f42f0006010 tx=0x7f42f000b540 comp rx=0 tx=0).stop 2024-04-16T02:46:12.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 msgr2=0x7f430019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:12.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 0x7f430019c500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00c930 tx=0x7f42fc00ccf0 comp rx=0 tx=0).stop 2024-04-16T02:46:12.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 shutdown_connections 2024-04-16T02:46:12.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f42ec06ecc0 0x7f42ec071180 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:12.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43000ffbd0 0x7f430019c500 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:12.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 --2- 172.21.15.151:0/2524536215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4300105a40 0x7f430019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:12.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 >> 172.21.15.151:0/2524536215 conn(0x7f43000755a0 msgr2=0x7f4300068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:12.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 shutdown_connections 2024-04-16T02:46:12.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:12.255+0000 7f4308704700 1 -- 172.21.15.151:0/2524536215 wait complete. 2024-04-16T02:46:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:11 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4268958216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:46:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:11 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/825294577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:46:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:11 smithi153 ceph-mon[87858]: pgmap v227: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:13.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:12 smithi151 ceph-mon[112233]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:13.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:12 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2524536215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:46:13.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:12 smithi153 ceph-mon[87858]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:13.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:12 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2524536215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:46:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:13 smithi151 ceph-mon[112233]: pgmap v228: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:14.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:13 smithi153 ceph-mon[87858]: pgmap v228: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:16.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:15 smithi151 ceph-mon[112233]: pgmap v229: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:16.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:15 smithi153 ceph-mon[87858]: pgmap v229: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:17.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:17 smithi151 ceph-mon[112233]: pgmap v230: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:17 smithi153 ceph-mon[87858]: pgmap v230: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:19 smithi151 ceph-mon[112233]: pgmap v231: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:19 smithi153 ceph-mon[87858]: pgmap v231: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:22.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:21 smithi151 ceph-mon[112233]: pgmap v232: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:21 smithi153 ceph-mon[87858]: pgmap v232: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:46:24.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:23 smithi151 ceph-mon[112233]: pgmap v233: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:46:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:23 smithi153 ceph-mon[87858]: pgmap v233: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:26.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:25 smithi151 ceph-mon[112233]: pgmap v234: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:26.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:25 smithi153 ceph-mon[87858]: pgmap v234: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:28.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:27 smithi151 ceph-mon[112233]: pgmap v235: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:27 smithi153 ceph-mon[87858]: pgmap v235: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:29 smithi151 ceph-mon[112233]: pgmap v236: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:29 smithi153 ceph-mon[87858]: pgmap v236: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:32.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:31 smithi151 ceph-mon[112233]: pgmap v237: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:32.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:31 smithi153 ceph-mon[87858]: pgmap v237: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:34.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:33 smithi151 ceph-mon[112233]: pgmap v238: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:34.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:33 smithi153 ceph-mon[87858]: pgmap v238: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:36.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:35 smithi151 ceph-mon[112233]: pgmap v239: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:35 smithi153 ceph-mon[87858]: pgmap v239: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:38.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:37 smithi151 ceph-mon[112233]: pgmap v240: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:37 smithi153 ceph-mon[87858]: pgmap v240: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:46:39.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:46:40.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:39 smithi151 ceph-mon[112233]: pgmap v241: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:39 smithi153 ceph-mon[87858]: pgmap v241: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:42.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:41 smithi151 ceph-mon[112233]: pgmap v242: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:41 smithi153 ceph-mon[87858]: pgmap v242: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:42.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.375+0000 7feffd3f4700 1 -- 172.21.15.151:0/715594456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8068670 msgr2=0x7feff8068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:42.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.375+0000 7feffd3f4700 1 --2- 172.21.15.151:0/715594456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8068670 0x7feff8068a50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009910 tx=0x7fefe8009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:42.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.375+0000 7feffd3f4700 1 -- 172.21.15.151:0/715594456 shutdown_connections 2024-04-16T02:46:42.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.375+0000 7feffd3f4700 1 --2- 172.21.15.151:0/715594456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff8069020 0x7feff8105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.375+0000 7feffd3f4700 1 --2- 172.21.15.151:0/715594456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8068670 0x7feff8068a50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.375+0000 7feffd3f4700 1 -- 172.21.15.151:0/715594456 >> 172.21.15.151:0/715594456 conn(0x7feff80755a0 msgr2=0x7feff80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:42.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.375+0000 7feffd3f4700 1 -- 172.21.15.151:0/715594456 shutdown_connections 2024-04-16T02:46:42.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.375+0000 7feffd3f4700 1 -- 172.21.15.151:0/715594456 wait complete. 2024-04-16T02:46:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.376+0000 7feffd3f4700 1 Processor -- start 2024-04-16T02:46:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.376+0000 7feffd3f4700 1 -- start start 2024-04-16T02:46:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.376+0000 7feffd3f4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff8068670 0x7feff819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.376+0000 7feffd3f4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8069020 0x7feff819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.376+0000 7feffd3f4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff819d0e0 con 0x7feff8069020 2024-04-16T02:46:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.376+0000 7feffd3f4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff8196590 con 0x7feff8068670 2024-04-16T02:46:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.376+0000 7fefee7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8069020 0x7feff819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7fefee7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8069020 0x7feff819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36012/0 (socket says 172.21.15.151:36012) 2024-04-16T02:46:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7fefee7fc700 1 -- 172.21.15.151:0/472539965 learned_addr learned my addr 172.21.15.151:0/472539965 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7feff6ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff8068670 0x7feff819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7fefee7fc700 1 -- 172.21.15.151:0/472539965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff8068670 msgr2=0x7feff819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7fefee7fc700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff8068670 0x7feff819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7fefee7fc700 1 -- 172.21.15.151:0/472539965 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe80095f0 con 0x7feff8069020 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7fefee7fc700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8069020 0x7feff819ca50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fefe000d900 tx=0x7fefe000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7feff4ff9700 1 -- 172.21.15.151:0/472539965 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe0009900 con 0x7feff8069020 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7feff4ff9700 1 -- 172.21.15.151:0/472539965 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefe0010460 con 0x7feff8069020 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff8196870 con 0x7feff8069020 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7feff4ff9700 1 -- 172.21.15.151:0/472539965 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe000f5f0 con 0x7feff8069020 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.377+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff8196dc0 con 0x7feff8069020 2024-04-16T02:46:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.378+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff8109130 con 0x7feff8069020 2024-04-16T02:46:42.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.379+0000 7feff4ff9700 1 -- 172.21.15.151:0/472539965 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fefe00105d0 con 0x7feff8069020 2024-04-16T02:46:42.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.379+0000 7feff4ff9700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefe406ecc0 0x7fefe4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:42.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.379+0000 7feff4ff9700 1 -- 172.21.15.151:0/472539965 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fefe008e910 con 0x7feff8069020 2024-04-16T02:46:42.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.379+0000 7feff6ffd700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefe406ecc0 0x7fefe4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:42.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.380+0000 7feff6ffd700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefe406ecc0 0x7fefe4071180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fefe8006010 tx=0x7fefe801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:42.387 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.384+0000 7feff4ff9700 1 -- 172.21.15.151:0/472539965 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fefe005b270 con 0x7feff8069020 2024-04-16T02:46:42.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.601+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feff81970c0 con 0x7fefe406ecc0 2024-04-16T02:46:42.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.602+0000 7feff4ff9700 1 -- 172.21.15.151:0/472539965 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7feff81970c0 con 0x7fefe406ecc0 2024-04-16T02:46:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefe406ecc0 msgr2=0x7fefe4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefe406ecc0 0x7fefe4071180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fefe8006010 tx=0x7fefe801a040 comp rx=0 tx=0).stop 2024-04-16T02:46:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8069020 msgr2=0x7feff819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8069020 0x7feff819ca50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fefe000d900 tx=0x7fefe000dc10 comp rx=0 tx=0).stop 2024-04-16T02:46:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 shutdown_connections 2024-04-16T02:46:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefe406ecc0 0x7fefe4071180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff8068670 0x7feff819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 --2- 172.21.15.151:0/472539965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feff8069020 0x7feff819ca50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 >> 172.21.15.151:0/472539965 conn(0x7feff80755a0 msgr2=0x7feff80fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 shutdown_connections 2024-04-16T02:46:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.604+0000 7feffd3f4700 1 -- 172.21.15.151:0/472539965 wait complete. 2024-04-16T02:46:42.618 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:46:42.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.711+0000 7f14bbabe700 1 -- 172.21.15.151:0/4253648032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 msgr2=0x7f14b4068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:42.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.711+0000 7f14bbabe700 1 --2- 172.21.15.151:0/4253648032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 0x7f14b4068a50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009910 tx=0x7f14a8009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:42.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.711+0000 7f14bbabe700 1 -- 172.21.15.151:0/4253648032 shutdown_connections 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.711+0000 7f14bbabe700 1 --2- 172.21.15.151:0/4253648032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b4069020 0x7f14b4105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.711+0000 7f14bbabe700 1 --2- 172.21.15.151:0/4253648032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 0x7f14b4068a50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.711+0000 7f14bbabe700 1 -- 172.21.15.151:0/4253648032 >> 172.21.15.151:0/4253648032 conn(0x7f14b40755a0 msgr2=0x7f14b40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.711+0000 7f14bbabe700 1 -- 172.21.15.151:0/4253648032 shutdown_connections 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.711+0000 7f14bbabe700 1 -- 172.21.15.151:0/4253648032 wait complete. 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14bbabe700 1 Processor -- start 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14bbabe700 1 -- start start 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14bbabe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 0x7f14b419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14bbabe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b4069020 0x7f14b419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14bbabe700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b419d0e0 con 0x7f14b4068670 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14bbabe700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b4196590 con 0x7f14b4069020 2024-04-16T02:46:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14b985a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 0x7f14b419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14b985a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 0x7f14b419c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36022/0 (socket says 172.21.15.151:36022) 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14b985a700 1 -- 172.21.15.151:0/876333496 learned_addr learned my addr 172.21.15.151:0/876333496 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.712+0000 7f14b9059700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b4069020 0x7f14b419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14b985a700 1 -- 172.21.15.151:0/876333496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b4069020 msgr2=0x7f14b419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14b985a700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b4069020 0x7f14b419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14b985a700 1 -- 172.21.15.151:0/876333496 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a80095f0 con 0x7f14b4068670 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14b9059700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b4069020 0x7f14b419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14b985a700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 0x7f14b419c510 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f14a8006010 tx=0x7f14a800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14a6ffd700 1 -- 172.21.15.151:0/876333496 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14a801d070 con 0x7f14b4068670 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14a6ffd700 1 -- 172.21.15.151:0/876333496 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14a800f460 con 0x7f14b4068670 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b4196810 con 0x7f14b4068670 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14a6ffd700 1 -- 172.21.15.151:0/876333496 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14a8017630 con 0x7f14b4068670 2024-04-16T02:46:42.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.713+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b4196d00 con 0x7f14b4068670 2024-04-16T02:46:42.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.714+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b4109130 con 0x7f14b4068670 2024-04-16T02:46:42.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.714+0000 7f14a6ffd700 1 -- 172.21.15.151:0/876333496 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f14a800f5d0 con 0x7f14b4068670 2024-04-16T02:46:42.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.715+0000 7f14a6ffd700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14a006ecc0 0x7f14a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:42.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.715+0000 7f14b9059700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14a006ecc0 0x7f14a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:42.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.715+0000 7f14a6ffd700 1 -- 172.21.15.151:0/876333496 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f14a8090300 con 0x7f14b4068670 2024-04-16T02:46:42.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.716+0000 7f14b9059700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14a006ecc0 0x7f14a0071180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f14b0005fd0 tx=0x7f14b0005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:42.723 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.720+0000 7f14a6ffd700 1 -- 172.21.15.151:0/876333496 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f14a805cbe0 con 0x7f14b4068670 2024-04-16T02:46:42.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.940+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f14b402ccf0 con 0x7f14a006ecc0 2024-04-16T02:46:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.942+0000 7f14a6ffd700 1 -- 172.21.15.151:0/876333496 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f14b402ccf0 con 0x7f14a006ecc0 2024-04-16T02:46:42.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14a006ecc0 msgr2=0x7f14a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:42.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14a006ecc0 0x7f14a0071180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f14b0005fd0 tx=0x7f14b0005ee0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 msgr2=0x7f14b419c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:42.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 0x7f14b419c510 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f14a8006010 tx=0x7f14a800be20 comp rx=0 tx=0).stop 2024-04-16T02:46:42.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 shutdown_connections 2024-04-16T02:46:42.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14a006ecc0 0x7f14a0071180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14b4068670 0x7f14b419c510 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 --2- 172.21.15.151:0/876333496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b4069020 0x7f14b419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:42.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 >> 172.21.15.151:0/876333496 conn(0x7f14b40755a0 msgr2=0x7f14b40fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:42.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 shutdown_connections 2024-04-16T02:46:42.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:42.944+0000 7f14bbabe700 1 -- 172.21.15.151:0/876333496 wait complete. 2024-04-16T02:46:43.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 -- 172.21.15.151:0/3800040084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684101d60 msgr2=0x7f6684102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 --2- 172.21.15.151:0/3800040084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684101d60 0x7f6684102140 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6678009940 tx=0x7f6678009c50 comp rx=0 tx=0).stop 2024-04-16T02:46:43.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 -- 172.21.15.151:0/3800040084 shutdown_connections 2024-04-16T02:46:43.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 --2- 172.21.15.151:0/3800040084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6684102680 0x7f668410ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 --2- 172.21.15.151:0/3800040084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684101d60 0x7f6684102140 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 -- 172.21.15.151:0/3800040084 >> 172.21.15.151:0/3800040084 conn(0x7f66840fb550 msgr2=0x7f66840fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:43.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 -- 172.21.15.151:0/3800040084 shutdown_connections 2024-04-16T02:46:43.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 -- 172.21.15.151:0/3800040084 wait complete. 2024-04-16T02:46:43.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 Processor -- start 2024-04-16T02:46:43.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.051+0000 7f668afe6700 1 -- start start 2024-04-16T02:46:43.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f668afe6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6684101d60 0x7f668419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:43.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f668afe6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684102680 0x7f668419ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:43.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f668afe6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668419d060 con 0x7f6684102680 2024-04-16T02:46:43.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f668afe6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66841965a0 con 0x7f6684101d60 2024-04-16T02:46:43.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6683fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684102680 0x7f668419ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:43.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6683fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684102680 0x7f668419ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36036/0 (socket says 172.21.15.151:36036) 2024-04-16T02:46:43.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6683fff700 1 -- 172.21.15.151:0/3250539670 learned_addr learned my addr 172.21.15.151:0/3250539670 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:43.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6688d82700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6684101d60 0x7f668419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:43.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6683fff700 1 -- 172.21.15.151:0/3250539670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6684101d60 msgr2=0x7f668419c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6683fff700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6684101d60 0x7f668419c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6683fff700 1 -- 172.21.15.151:0/3250539670 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780095f0 con 0x7f6684102680 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6688d82700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6684101d60 0x7f668419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6683fff700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684102680 0x7f668419ca60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f667000c8f0 tx=0x7f667000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6681ffb700 1 -- 172.21.15.151:0/3250539670 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66700041f0 con 0x7f6684102680 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6684196880 con 0x7f6684102680 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6681ffb700 1 -- 172.21.15.151:0/3250539670 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6670007dc0 con 0x7f6684102680 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.052+0000 7f6681ffb700 1 -- 172.21.15.151:0/3250539670 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6670003ea0 con 0x7f6684102680 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.053+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6684196dd0 con 0x7f6684102680 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.054+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6684108270 con 0x7f6684102680 2024-04-16T02:46:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.054+0000 7f6681ffb700 1 -- 172.21.15.151:0/3250539670 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6670004350 con 0x7f6684102680 2024-04-16T02:46:43.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.054+0000 7f6681ffb700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f667406ecc0 0x7f6674071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:43.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.055+0000 7f6681ffb700 1 -- 172.21.15.151:0/3250539670 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f667008eba0 con 0x7f6684102680 2024-04-16T02:46:43.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.055+0000 7f6688d82700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f667406ecc0 0x7f6674071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:43.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.055+0000 7f6688d82700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f667406ecc0 0x7f6674071180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6678009910 tx=0x7f6678005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:43.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.059+0000 7f6681ffb700 1 -- 172.21.15.151:0/3250539670 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f667005b480 con 0x7f6684102680 2024-04-16T02:46:43.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:42 smithi151 ceph-mon[112233]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:43.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:42 smithi153 ceph-mon[87858]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:43.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.270+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f66841970a0 con 0x7f667406ecc0 2024-04-16T02:46:43.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.281+0000 7f6681ffb700 1 -- 172.21.15.151:0/3250539670 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f66841970a0 con 0x7f667406ecc0 2024-04-16T02:46:43.284 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:46:43.284 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (8m) 103s ago 19m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:46:43.284 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (19m) 103s ago 19m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (18m) 86s ago 18m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (5m) 103s ago 19m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (5m) 86s ago 18m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (7m) 103s ago 19m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (2m) 103s ago 14m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (109s) 103s ago 13m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (13m) 86s ago 13m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (89s) 86s ago 14m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (10m) 103s ago 21m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (9m) 86s ago 18m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (6m) 103s ago 21m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (6m) 86s ago 18m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (8m) 103s ago 19m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (8m) 86s ago 18m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (5m) 103s ago 17m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:46:43.285 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (4m) 103s ago 17m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:46:43.286 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (4m) 103s ago 16m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:46:43.286 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (3m) 86s ago 16m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:46:43.286 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (3m) 86s ago 15m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:46:43.286 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (2m) 86s ago 15m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:46:43.286 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (8m) 103s ago 19m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f667406ecc0 msgr2=0x7f6674071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f667406ecc0 0x7f6674071180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6678009910 tx=0x7f6678005fd0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684102680 msgr2=0x7f668419ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684102680 0x7f668419ca60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f667000c8f0 tx=0x7f667000ccb0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 shutdown_connections 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f667406ecc0 0x7f6674071180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6684101d60 0x7f668419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 --2- 172.21.15.151:0/3250539670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6684102680 0x7f668419ca60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 >> 172.21.15.151:0/3250539670 conn(0x7f66840fb550 msgr2=0x7f66841053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 shutdown_connections 2024-04-16T02:46:43.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.284+0000 7f668afe6700 1 -- 172.21.15.151:0/3250539670 wait complete. 2024-04-16T02:46:43.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.387+0000 7f722c583700 1 -- 172.21.15.151:0/1543907673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 msgr2=0x7f72241057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.387+0000 7f722c583700 1 --2- 172.21.15.151:0/1543907673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 0x7f72241057c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7218009910 tx=0x7f7218009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:43.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.387+0000 7f722c583700 1 -- 172.21.15.151:0/1543907673 shutdown_connections 2024-04-16T02:46:43.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.387+0000 7f722c583700 1 --2- 172.21.15.151:0/1543907673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 0x7f72241057c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.387+0000 7f722c583700 1 --2- 172.21.15.151:0/1543907673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7224100dc0 0x7f72241011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.387+0000 7f722c583700 1 -- 172.21.15.151:0/1543907673 >> 172.21.15.151:0/1543907673 conn(0x7f72240fc650 msgr2=0x7f72240fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:43.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f722c583700 1 -- 172.21.15.151:0/1543907673 shutdown_connections 2024-04-16T02:46:43.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f722c583700 1 -- 172.21.15.151:0/1543907673 wait complete. 2024-04-16T02:46:43.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f722c583700 1 Processor -- start 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f722c583700 1 -- start start 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f722c583700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7224100dc0 0x7f7224073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f722c583700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 0x7f72240735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f722c583700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7224073b10 con 0x7f7224101770 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f722c583700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7224073c50 con 0x7f7224100dc0 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f7229b1e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 0x7f72240735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f7229b1e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 0x7f72240735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36044/0 (socket says 172.21.15.151:36044) 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.388+0000 7f7229b1e700 1 -- 172.21.15.151:0/301712396 learned_addr learned my addr 172.21.15.151:0/301712396 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f722a31f700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7224100dc0 0x7f7224073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f7229b1e700 1 -- 172.21.15.151:0/301712396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7224100dc0 msgr2=0x7f7224073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f7229b1e700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7224100dc0 0x7f7224073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f7229b1e700 1 -- 172.21.15.151:0/301712396 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72180095f0 con 0x7f7224101770 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f722a31f700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7224100dc0 0x7f7224073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f7229b1e700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 0x7f72240735d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f721800bda0 tx=0x7f721800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f72237fe700 1 -- 172.21.15.151:0/301712396 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f721801d070 con 0x7f7224101770 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f72237fe700 1 -- 172.21.15.151:0/301712396 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7218004fb0 con 0x7f7224101770 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7224073ed0 con 0x7f7224101770 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f72237fe700 1 -- 172.21.15.151:0/301712396 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f721800f460 con 0x7f7224101770 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.389+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72241a2c40 con 0x7f7224101770 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.390+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72241091c0 con 0x7f7224101770 2024-04-16T02:46:43.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.391+0000 7f72237fe700 1 -- 172.21.15.151:0/301712396 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7218022800 con 0x7f7224101770 2024-04-16T02:46:43.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.391+0000 7f72237fe700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f720c06ecc0 0x7f720c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:43.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.391+0000 7f722a31f700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f720c06ecc0 0x7f720c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:43.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.391+0000 7f72237fe700 1 -- 172.21.15.151:0/301712396 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f72180903a0 con 0x7f7224101770 2024-04-16T02:46:43.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.392+0000 7f722a31f700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f720c06ecc0 0x7f720c071180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f721c00e300 tx=0x7f721c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:43.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.396+0000 7f72237fe700 1 -- 172.21.15.151:0/301712396 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7218059320 con 0x7f7224101770 2024-04-16T02:46:43.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.679+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f722402cc70 con 0x7f7224101770 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.680+0000 7f72237fe700 1 -- 172.21.15.151:0/301712396 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f7218027340 con 0x7f7224101770 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:46:43.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:46:43.684 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:46:43.684 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:46:43.684 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:46:43.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:46:43.684 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:46:43.684 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:46:43.684 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:46:43.684 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:46:43.684 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:46:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f720c06ecc0 msgr2=0x7f720c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f720c06ecc0 0x7f720c071180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f721c00e300 tx=0x7f721c009040 comp rx=0 tx=0).stop 2024-04-16T02:46:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 msgr2=0x7f72240735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 0x7f72240735d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f721800bda0 tx=0x7f721800be80 comp rx=0 tx=0).stop 2024-04-16T02:46:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 shutdown_connections 2024-04-16T02:46:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f720c06ecc0 0x7f720c071180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7224100dc0 0x7f7224073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 --2- 172.21.15.151:0/301712396 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7224101770 0x7f72240735d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 >> 172.21.15.151:0/301712396 conn(0x7f72240fc650 msgr2=0x7f72240fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 shutdown_connections 2024-04-16T02:46:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.682+0000 7f722c583700 1 -- 172.21.15.151:0/301712396 wait complete. 2024-04-16T02:46:43.790 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.787+0000 7f74dea39700 1 -- 172.21.15.151:0/505193722 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d80ffa40 msgr2=0x7f74d80ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.787+0000 7f74dea39700 1 --2- 172.21.15.151:0/505193722 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d80ffa40 0x7f74d80ffec0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009910 tx=0x7f74cc009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:43.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.787+0000 7f74dea39700 1 -- 172.21.15.151:0/505193722 shutdown_connections 2024-04-16T02:46:43.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.787+0000 7f74dea39700 1 --2- 172.21.15.151:0/505193722 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d80ffa40 0x7f74d80ffec0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.787+0000 7f74dea39700 1 --2- 172.21.15.151:0/505193722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d8105a40 0x7f74d8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.787+0000 7f74dea39700 1 -- 172.21.15.151:0/505193722 >> 172.21.15.151:0/505193722 conn(0x7f74d80fb550 msgr2=0x7f74d80fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:43.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.788+0000 7f74dea39700 1 -- 172.21.15.151:0/505193722 shutdown_connections 2024-04-16T02:46:43.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.788+0000 7f74dea39700 1 -- 172.21.15.151:0/505193722 wait complete. 2024-04-16T02:46:43.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.788+0000 7f74dea39700 1 Processor -- start 2024-04-16T02:46:43.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.788+0000 7f74dea39700 1 -- start start 2024-04-16T02:46:43.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74dea39700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d80ffa40 0x7f74d819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:43.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74dea39700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d8105a40 0x7f74d819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:43.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74dea39700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d819d110 con 0x7f74d8105a40 2024-04-16T02:46:43.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74dea39700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d81965c0 con 0x7f74d80ffa40 2024-04-16T02:46:43.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d8105a40 0x7f74d819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d8105a40 0x7f74d819ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36056/0 (socket says 172.21.15.151:36056) 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d77fe700 1 -- 172.21.15.151:0/3278487760 learned_addr learned my addr 172.21.15.151:0/3278487760 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d7fff700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d80ffa40 0x7f74d819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d77fe700 1 -- 172.21.15.151:0/3278487760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d80ffa40 msgr2=0x7f74d819c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d77fe700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d80ffa40 0x7f74d819c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d77fe700 1 -- 172.21.15.151:0/3278487760 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc0095f0 con 0x7f74d8105a40 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d77fe700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d8105a40 0x7f74d819ca80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f74cc006010 tx=0x7f74cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d57fa700 1 -- 172.21.15.151:0/3278487760 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74cc01d070 con 0x7f74d8105a40 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d57fa700 1 -- 172.21.15.151:0/3278487760 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74cc004fb0 con 0x7f74d8105a40 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d8196840 con 0x7f74d8105a40 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74d57fa700 1 -- 172.21.15.151:0/3278487760 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74cc022ea0 con 0x7f74d8105a40 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.789+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d8196d30 con 0x7f74d8105a40 2024-04-16T02:46:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.790+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d804ea90 con 0x7f74d8105a40 2024-04-16T02:46:43.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.791+0000 7f74d57fa700 1 -- 172.21.15.151:0/3278487760 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f74cc022470 con 0x7f74d8105a40 2024-04-16T02:46:43.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.791+0000 7f74d57fa700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f74c006ecc0 0x7f74c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:43.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.791+0000 7f74d57fa700 1 -- 172.21.15.151:0/3278487760 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f74cc0903f0 con 0x7f74d8105a40 2024-04-16T02:46:43.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.791+0000 7f74d7fff700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f74c006ecc0 0x7f74c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:43.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.792+0000 7f74d7fff700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f74c006ecc0 0x7f74c0071180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f74c8005fd0 tx=0x7f74c800a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:43.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:43.796+0000 7f74d57fa700 1 -- 172.21.15.151:0/3278487760 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f74cc05cd80 con 0x7f74d8105a40 2024-04-16T02:46:43.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:43 smithi151 ceph-mon[112233]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:43.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:43 smithi151 ceph-mon[112233]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:43.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:43 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/301712396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:46:43.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:43 smithi151 ceph-mon[112233]: pgmap v243: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:44.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.045+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f74d8197ad0 con 0x7f74d8105a40 2024-04-16T02:46:44.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.045+0000 7f74d57fa700 1 -- 172.21.15.151:0/3278487760 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f74cc027770 con 0x7f74d8105a40 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:46:44.049 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:46:44.050 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:46:44.051 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:46:44.052 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:46:44.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.048+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f74c006ecc0 msgr2=0x7f74c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.048+0000 7f74dea39700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f74c006ecc0 0x7f74c0071180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f74c8005fd0 tx=0x7f74c800a560 comp rx=0 tx=0).stop 2024-04-16T02:46:44.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.048+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d8105a40 msgr2=0x7f74d819ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.048+0000 7f74dea39700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d8105a40 0x7f74d819ca80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f74cc006010 tx=0x7f74cc00be20 comp rx=0 tx=0).stop 2024-04-16T02:46:44.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.049+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 shutdown_connections 2024-04-16T02:46:44.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.049+0000 7f74dea39700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f74c006ecc0 0x7f74c0071180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.049+0000 7f74dea39700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d80ffa40 0x7f74d819c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.049+0000 7f74dea39700 1 --2- 172.21.15.151:0/3278487760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d8105a40 0x7f74d819ca80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.049+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 >> 172.21.15.151:0/3278487760 conn(0x7f74d80fb550 msgr2=0x7f74d8103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:44.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.049+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 shutdown_connections 2024-04-16T02:46:44.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.049+0000 7f74dea39700 1 -- 172.21.15.151:0/3278487760 wait complete. 2024-04-16T02:46:44.053 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:46:44.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.155+0000 7fb048863700 1 -- 172.21.15.151:0/1223697038 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 msgr2=0x7fb0441021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.155+0000 7fb048863700 1 --2- 172.21.15.151:0/1223697038 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 0x7fb0441021a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb034009910 tx=0x7fb034009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.155+0000 7fb048863700 1 -- 172.21.15.151:0/1223697038 shutdown_connections 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.155+0000 7fb048863700 1 --2- 172.21.15.151:0/1223697038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0441026e0 0x7fb04410abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.155+0000 7fb048863700 1 --2- 172.21.15.151:0/1223697038 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 0x7fb0441021a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.155+0000 7fb048863700 1 -- 172.21.15.151:0/1223697038 >> 172.21.15.151:0/1223697038 conn(0x7fb0440fb570 msgr2=0x7fb0440fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.155+0000 7fb048863700 1 -- 172.21.15.151:0/1223697038 shutdown_connections 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.155+0000 7fb048863700 1 -- 172.21.15.151:0/1223697038 wait complete. 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.156+0000 7fb048863700 1 Processor -- start 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.156+0000 7fb048863700 1 -- start start 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.156+0000 7fb048863700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 0x7fb0440ffa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.156+0000 7fb048863700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0441026e0 0x7fb0440fff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:44.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.156+0000 7fb048863700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0441004a0 con 0x7fb044101dc0 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.156+0000 7fb048863700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0441005e0 con 0x7fb0441026e0 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.156+0000 7fb04259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 0x7fb0440ffa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb04259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 0x7fb0440ffa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36082/0 (socket says 172.21.15.151:36082) 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb04259c700 1 -- 172.21.15.151:0/811449393 learned_addr learned my addr 172.21.15.151:0/811449393 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb041d9b700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0441026e0 0x7fb0440fff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb04259c700 1 -- 172.21.15.151:0/811449393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0441026e0 msgr2=0x7fb0440fff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb04259c700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0441026e0 0x7fb0440fff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb04259c700 1 -- 172.21.15.151:0/811449393 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0340095f0 con 0x7fb044101dc0 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb041d9b700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0441026e0 0x7fb0440fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb04259c700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 0x7fb0440ffa20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb034006010 tx=0x7fb03400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb0337fe700 1 -- 172.21.15.151:0/811449393 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb03401d070 con 0x7fb044101dc0 2024-04-16T02:46:44.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb044100860 con 0x7fb044101dc0 2024-04-16T02:46:44.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb0337fe700 1 -- 172.21.15.151:0/811449393 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb03400f460 con 0x7fb044101dc0 2024-04-16T02:46:44.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb0337fe700 1 -- 172.21.15.151:0/811449393 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb034017630 con 0x7fb044101dc0 2024-04-16T02:46:44.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.157+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0441a2c50 con 0x7fb044101dc0 2024-04-16T02:46:44.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.158+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb044108220 con 0x7fb044101dc0 2024-04-16T02:46:44.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.159+0000 7fb0337fe700 1 -- 172.21.15.151:0/811449393 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb034003a60 con 0x7fb044101dc0 2024-04-16T02:46:44.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.159+0000 7fb0337fe700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb02c06ecc0 0x7fb02c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:44.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.159+0000 7fb041d9b700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb02c06ecc0 0x7fb02c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:44.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.159+0000 7fb0337fe700 1 -- 172.21.15.151:0/811449393 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb0340902d0 con 0x7fb044101dc0 2024-04-16T02:46:44.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.160+0000 7fb041d9b700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb02c06ecc0 0x7fb02c071180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb038005fd0 tx=0x7fb038005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:44.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.164+0000 7fb0337fe700 1 -- 172.21.15.151:0/811449393 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb034059100 con 0x7fb044101dc0 2024-04-16T02:46:44.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:43 smithi153 ceph-mon[87858]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:44.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:43 smithi153 ceph-mon[87858]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:44.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:43 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/301712396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:46:44.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:43 smithi153 ceph-mon[87858]: pgmap v243: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:44.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.382+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0440611d0 con 0x7fb02c06ecc0 2024-04-16T02:46:44.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.383+0000 7fb0337fe700 1 -- 172.21.15.151:0/811449393 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb0440611d0 con 0x7fb02c06ecc0 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:46:44.387 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:46:44.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb02c06ecc0 msgr2=0x7fb02c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb02c06ecc0 0x7fb02c071180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb038005fd0 tx=0x7fb038005ee0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 msgr2=0x7fb0440ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 0x7fb0440ffa20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb034006010 tx=0x7fb03400be20 comp rx=0 tx=0).stop 2024-04-16T02:46:44.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 shutdown_connections 2024-04-16T02:46:44.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb02c06ecc0 0x7fb02c071180 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb044101dc0 0x7fb0440ffa20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 --2- 172.21.15.151:0/811449393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0441026e0 0x7fb0440fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 >> 172.21.15.151:0/811449393 conn(0x7fb0440fb570 msgr2=0x7fb044105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:44.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 shutdown_connections 2024-04-16T02:46:44.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.386+0000 7fb048863700 1 -- 172.21.15.151:0/811449393 wait complete. 2024-04-16T02:46:44.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.491+0000 7f5449109700 1 -- 172.21.15.151:0/896214033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444105a40 msgr2=0x7f5444105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.491+0000 7f5449109700 1 --2- 172.21.15.151:0/896214033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444105a40 0x7f5444105e20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5438009910 tx=0x7f5438009c20 comp rx=0 tx=0).stop 2024-04-16T02:46:44.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.491+0000 7f5449109700 1 -- 172.21.15.151:0/896214033 shutdown_connections 2024-04-16T02:46:44.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.491+0000 7f5449109700 1 --2- 172.21.15.151:0/896214033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54440ffa40 0x7f54440ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.491+0000 7f5449109700 1 --2- 172.21.15.151:0/896214033 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444105a40 0x7f5444105e20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.491+0000 7f5449109700 1 -- 172.21.15.151:0/896214033 >> 172.21.15.151:0/896214033 conn(0x7f54440fb570 msgr2=0x7f54440fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:44.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.491+0000 7f5449109700 1 -- 172.21.15.151:0/896214033 shutdown_connections 2024-04-16T02:46:44.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.491+0000 7f5449109700 1 -- 172.21.15.151:0/896214033 wait complete. 2024-04-16T02:46:44.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f5449109700 1 Processor -- start 2024-04-16T02:46:44.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f5449109700 1 -- start start 2024-04-16T02:46:44.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f5449109700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440ffa40 0x7f544419c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:44.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f5449109700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444105a40 0x7f544419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:44.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f5449109700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544419d110 con 0x7f54440ffa40 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f5449109700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54441965c0 con 0x7f5444105a40 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f5442d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440ffa40 0x7f544419c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f5442d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440ffa40 0x7f544419c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36092/0 (socket says 172.21.15.151:36092) 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f5442d9d700 1 -- 172.21.15.151:0/4018882259 learned_addr learned my addr 172.21.15.151:0/4018882259 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.492+0000 7f544259c700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444105a40 0x7f544419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.493+0000 7f5442d9d700 1 -- 172.21.15.151:0/4018882259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444105a40 msgr2=0x7f544419ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.493+0000 7f5442d9d700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444105a40 0x7f544419ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.493+0000 7f5442d9d700 1 -- 172.21.15.151:0/4018882259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54380095f0 con 0x7f54440ffa40 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.493+0000 7f5442d9d700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440ffa40 0x7f544419c540 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5438006010 tx=0x7f543800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.493+0000 7f542bfff700 1 -- 172.21.15.151:0/4018882259 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f543801d070 con 0x7f54440ffa40 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.493+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5444196840 con 0x7f54440ffa40 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.493+0000 7f542bfff700 1 -- 172.21.15.151:0/4018882259 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5438004fb0 con 0x7f54440ffa40 2024-04-16T02:46:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.493+0000 7f542bfff700 1 -- 172.21.15.151:0/4018882259 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f543800f690 con 0x7f54440ffa40 2024-04-16T02:46:44.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.493+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5444196d30 con 0x7f54440ffa40 2024-04-16T02:46:44.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.494+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544404ea90 con 0x7f54440ffa40 2024-04-16T02:46:44.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.494+0000 7f542bfff700 1 -- 172.21.15.151:0/4018882259 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f54380228a0 con 0x7f54440ffa40 2024-04-16T02:46:44.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.495+0000 7f542bfff700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ecc0 0x7f542c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:46:44.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.495+0000 7f542bfff700 1 -- 172.21.15.151:0/4018882259 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f54380906d0 con 0x7f54440ffa40 2024-04-16T02:46:44.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.495+0000 7f544259c700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ecc0 0x7f542c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:46:44.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.495+0000 7f544259c700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ecc0 0x7f542c071180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5434005fd0 tx=0x7f543400a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:46:44.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.499+0000 7f542bfff700 1 -- 172.21.15.151:0/4018882259 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f543805cfb0 con 0x7f54440ffa40 2024-04-16T02:46:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.777+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f54441971e0 con 0x7f54440ffa40 2024-04-16T02:46:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.778+0000 7f542bfff700 1 -- 172.21.15.151:0/4018882259 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f543805cb40 con 0x7f54440ffa40 2024-04-16T02:46:44.781 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:46:44.781 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:46:44.781 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:46:44.781 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:46:44.781 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:46:44.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ecc0 msgr2=0x7f542c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ecc0 0x7f542c071180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5434005fd0 tx=0x7f543400a560 comp rx=0 tx=0).stop 2024-04-16T02:46:44.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440ffa40 msgr2=0x7f544419c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:46:44.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440ffa40 0x7f544419c540 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5438006010 tx=0x7f543800be20 comp rx=0 tx=0).stop 2024-04-16T02:46:44.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 shutdown_connections 2024-04-16T02:46:44.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ecc0 0x7f542c071180 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440ffa40 0x7f544419c540 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 --2- 172.21.15.151:0/4018882259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444105a40 0x7f544419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:46:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 >> 172.21.15.151:0/4018882259 conn(0x7f54440fb570 msgr2=0x7f5444103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:46:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 shutdown_connections 2024-04-16T02:46:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:46:44.780+0000 7f5449109700 1 -- 172.21.15.151:0/4018882259 wait complete. 2024-04-16T02:46:45.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:44 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3278487760' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:46:45.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:44 smithi151 ceph-mon[112233]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:45.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:44 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4018882259' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:46:45.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:44 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3278487760' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:46:45.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:44 smithi153 ceph-mon[87858]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:46:45.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:44 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4018882259' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:46:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:45 smithi153 ceph-mon[87858]: pgmap v244: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:46.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:45 smithi151 ceph-mon[112233]: pgmap v244: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:47 smithi151 ceph-mon[112233]: pgmap v245: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:48.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:47 smithi153 ceph-mon[87858]: pgmap v245: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:49 smithi151 ceph-mon[112233]: pgmap v246: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:49 smithi153 ceph-mon[87858]: pgmap v246: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:51 smithi151 ceph-mon[112233]: pgmap v247: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:51 smithi153 ceph-mon[87858]: pgmap v247: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:54.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:46:54.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:53 smithi151 ceph-mon[112233]: pgmap v248: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:46:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:53 smithi153 ceph-mon[87858]: pgmap v248: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:56.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:55 smithi151 ceph-mon[112233]: pgmap v249: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:56.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:55 smithi153 ceph-mon[87858]: pgmap v249: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:57 smithi151 ceph-mon[112233]: pgmap v250: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:46:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:57 smithi153 ceph-mon[87858]: pgmap v250: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:46:59 smithi151 ceph-mon[112233]: pgmap v251: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:00.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:46:59 smithi153 ceph-mon[87858]: pgmap v251: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:02.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:01 smithi151 ceph-mon[112233]: pgmap v252: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:01 smithi153 ceph-mon[87858]: pgmap v252: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:04.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:03 smithi151 ceph-mon[112233]: pgmap v253: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:04.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:03 smithi153 ceph-mon[87858]: pgmap v253: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:05 smithi151 ceph-mon[112233]: pgmap v254: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:06.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:05 smithi153 ceph-mon[87858]: pgmap v254: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:08.213 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:07 smithi151 ceph-mon[112233]: pgmap v255: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:08.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:07 smithi153 ceph-mon[87858]: pgmap v255: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:09.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:47:09.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:47:10.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:09 smithi151 ceph-mon[112233]: pgmap v256: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:09 smithi153 ceph-mon[87858]: pgmap v256: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:11 smithi151 ceph-mon[112233]: pgmap v257: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:11 smithi153 ceph-mon[87858]: pgmap v257: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:13 smithi151 ceph-mon[112233]: pgmap v258: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:14.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:13 smithi153 ceph-mon[87858]: pgmap v258: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:14.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.900+0000 7f91b0c46700 1 -- 172.21.15.151:0/2772906377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac105a40 msgr2=0x7f91ac105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:14.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.900+0000 7f91b0c46700 1 --2- 172.21.15.151:0/2772906377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac105a40 0x7f91ac105e20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f919c009910 tx=0x7f919c009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:14.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.900+0000 7f91b0c46700 1 -- 172.21.15.151:0/2772906377 shutdown_connections 2024-04-16T02:47:14.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.900+0000 7f91b0c46700 1 --2- 172.21.15.151:0/2772906377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ac0ffa40 0x7f91ac0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:14.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.900+0000 7f91b0c46700 1 --2- 172.21.15.151:0/2772906377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac105a40 0x7f91ac105e20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:14.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.900+0000 7f91b0c46700 1 -- 172.21.15.151:0/2772906377 >> 172.21.15.151:0/2772906377 conn(0x7f91ac0fb530 msgr2=0x7f91ac0fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:14.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.900+0000 7f91b0c46700 1 -- 172.21.15.151:0/2772906377 shutdown_connections 2024-04-16T02:47:14.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.900+0000 7f91b0c46700 1 -- 172.21.15.151:0/2772906377 wait complete. 2024-04-16T02:47:14.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.900+0000 7f91b0c46700 1 Processor -- start 2024-04-16T02:47:14.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91b0c46700 1 -- start start 2024-04-16T02:47:14.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91b0c46700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac0ffa40 0x7f91ac1955f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91b0c46700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ac105a40 0x7f91ac195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91b0c46700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac196210 con 0x7f91ac0ffa40 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91b0c46700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac199fa0 con 0x7f91ac105a40 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91aa59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac0ffa40 0x7f91ac1955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91aa59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac0ffa40 0x7f91ac1955f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46752/0 (socket says 172.21.15.151:46752) 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91aa59c700 1 -- 172.21.15.151:0/746264200 learned_addr learned my addr 172.21.15.151:0/746264200 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91a3fff700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ac105a40 0x7f91ac195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91aa59c700 1 -- 172.21.15.151:0/746264200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ac105a40 msgr2=0x7f91ac195b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91aa59c700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ac105a40 0x7f91ac195b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91aa59c700 1 -- 172.21.15.151:0/746264200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c0095f0 con 0x7f91ac0ffa40 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.901+0000 7f91aa59c700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac0ffa40 0x7f91ac1955f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f919c0092e0 tx=0x7f919c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:14.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.902+0000 7f91a37fe700 1 -- 172.21.15.151:0/746264200 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919c01d070 con 0x7f91ac0ffa40 2024-04-16T02:47:14.908 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.902+0000 7f91a37fe700 1 -- 172.21.15.151:0/746264200 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f919c005e00 con 0x7f91ac0ffa40 2024-04-16T02:47:14.908 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.902+0000 7f91a37fe700 1 -- 172.21.15.151:0/746264200 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919c017400 con 0x7f91ac0ffa40 2024-04-16T02:47:14.908 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.902+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac19a220 con 0x7f91ac0ffa40 2024-04-16T02:47:14.908 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.902+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac19a6c0 con 0x7f91ac0ffa40 2024-04-16T02:47:14.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.903+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ac04ea90 con 0x7f91ac0ffa40 2024-04-16T02:47:14.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.903+0000 7f91a37fe700 1 -- 172.21.15.151:0/746264200 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f919c00f460 con 0x7f91ac0ffa40 2024-04-16T02:47:14.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.904+0000 7f91a37fe700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f919806ecc0 0x7f9198071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:14.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.904+0000 7f91a37fe700 1 -- 172.21.15.151:0/746264200 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f919c090380 con 0x7f91ac0ffa40 2024-04-16T02:47:14.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.904+0000 7f91a3fff700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f919806ecc0 0x7f9198071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:14.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.904+0000 7f91a3fff700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f919806ecc0 0x7f9198071180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9194005fd0 tx=0x7f9194005f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:14.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:14.910+0000 7f91a37fe700 1 -- 172.21.15.151:0/746264200 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f919c059130 con 0x7f91ac0ffa40 2024-04-16T02:47:15.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.133+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f91ac19abd0 con 0x7f919806ecc0 2024-04-16T02:47:15.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.134+0000 7f91a37fe700 1 -- 172.21.15.151:0/746264200 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f91ac19abd0 con 0x7f919806ecc0 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.136+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f919806ecc0 msgr2=0x7f9198071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.136+0000 7f91b0c46700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f919806ecc0 0x7f9198071180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9194005fd0 tx=0x7f9194005f60 comp rx=0 tx=0).stop 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.136+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac0ffa40 msgr2=0x7f91ac1955f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.136+0000 7f91b0c46700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac0ffa40 0x7f91ac1955f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f919c0092e0 tx=0x7f919c00be20 comp rx=0 tx=0).stop 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.136+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 shutdown_connections 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.136+0000 7f91b0c46700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f919806ecc0 0x7f9198071180 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.136+0000 7f91b0c46700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91ac0ffa40 0x7f91ac1955f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.136+0000 7f91b0c46700 1 --2- 172.21.15.151:0/746264200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ac105a40 0x7f91ac195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.136+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 >> 172.21.15.151:0/746264200 conn(0x7f91ac0fb530 msgr2=0x7f91ac0fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.137+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 shutdown_connections 2024-04-16T02:47:15.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.137+0000 7f91b0c46700 1 -- 172.21.15.151:0/746264200 wait complete. 2024-04-16T02:47:15.152 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:47:15.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.243+0000 7f8c55550700 1 -- 172.21.15.151:0/3362624071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c50105a40 msgr2=0x7f8c50105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.243+0000 7f8c55550700 1 --2- 172.21.15.151:0/3362624071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c50105a40 0x7f8c50105e20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800b0a0 tx=0x7f8c4800b3b0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.243+0000 7f8c55550700 1 -- 172.21.15.151:0/3362624071 shutdown_connections 2024-04-16T02:47:15.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.243+0000 7f8c55550700 1 --2- 172.21.15.151:0/3362624071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c500ffbd0 0x7f8c50100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.243+0000 7f8c55550700 1 --2- 172.21.15.151:0/3362624071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c50105a40 0x7f8c50105e20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.243+0000 7f8c55550700 1 -- 172.21.15.151:0/3362624071 >> 172.21.15.151:0/3362624071 conn(0x7f8c500755a0 msgr2=0x7f8c500759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:15.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.243+0000 7f8c55550700 1 -- 172.21.15.151:0/3362624071 shutdown_connections 2024-04-16T02:47:15.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.243+0000 7f8c55550700 1 -- 172.21.15.151:0/3362624071 wait complete. 2024-04-16T02:47:15.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.244+0000 7f8c55550700 1 Processor -- start 2024-04-16T02:47:15.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.244+0000 7f8c55550700 1 -- start start 2024-04-16T02:47:15.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.244+0000 7f8c55550700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c500ffbd0 0x7f8c5019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:15.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.244+0000 7f8c55550700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c50105a40 0x7f8c5019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:15.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.244+0000 7f8c55550700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5019d060 con 0x7f8c500ffbd0 2024-04-16T02:47:15.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.244+0000 7f8c55550700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c50196520 con 0x7f8c50105a40 2024-04-16T02:47:15.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.244+0000 7f8c4effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c500ffbd0 0x7f8c5019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:15.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.244+0000 7f8c4effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c500ffbd0 0x7f8c5019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46776/0 (socket says 172.21.15.151:46776) 2024-04-16T02:47:15.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.244+0000 7f8c4effd700 1 -- 172.21.15.151:0/352252310 learned_addr learned my addr 172.21.15.151:0/352252310 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:15.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c4e7fc700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c50105a40 0x7f8c5019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:15.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c4effd700 1 -- 172.21.15.151:0/352252310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c50105a40 msgr2=0x7f8c5019c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c4effd700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c50105a40 0x7f8c5019c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c4effd700 1 -- 172.21.15.151:0/352252310 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c48009d00 con 0x7f8c500ffbd0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c4effd700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c500ffbd0 0x7f8c5019c440 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8c48000f80 tx=0x7f8c4800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c47fff700 1 -- 172.21.15.151:0/352252310 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c4800e040 con 0x7f8c500ffbd0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c501967a0 con 0x7f8c500ffbd0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c47fff700 1 -- 172.21.15.151:0/352252310 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c48003de0 con 0x7f8c500ffbd0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c47fff700 1 -- 172.21.15.151:0/352252310 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c4801d790 con 0x7f8c500ffbd0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.245+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c50196c90 con 0x7f8c500ffbd0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.246+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c50109050 con 0x7f8c500ffbd0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.246+0000 7f8c47fff700 1 -- 172.21.15.151:0/352252310 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8c48004040 con 0x7f8c500ffbd0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.247+0000 7f8c47fff700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c3c06ecc0 0x7f8c3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.247+0000 7f8c47fff700 1 -- 172.21.15.151:0/352252310 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8c48090670 con 0x7f8c500ffbd0 2024-04-16T02:47:15.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.247+0000 7f8c4e7fc700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c3c06ecc0 0x7f8c3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:15.253 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.247+0000 7f8c4e7fc700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c3c06ecc0 0x7f8c3c071180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8c38005950 tx=0x7f8c380058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:15.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.252+0000 7f8c47fff700 1 -- 172.21.15.151:0/352252310 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8c4805cf50 con 0x7f8c500ffbd0 2024-04-16T02:47:15.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.466+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c50196eb0 con 0x7f8c3c06ecc0 2024-04-16T02:47:15.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.468+0000 7f8c47fff700 1 -- 172.21.15.151:0/352252310 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8c50196eb0 con 0x7f8c3c06ecc0 2024-04-16T02:47:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c3c06ecc0 msgr2=0x7f8c3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c3c06ecc0 0x7f8c3c071180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8c38005950 tx=0x7f8c380058e0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c500ffbd0 msgr2=0x7f8c5019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c500ffbd0 0x7f8c5019c440 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8c48000f80 tx=0x7f8c4800cdd0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 shutdown_connections 2024-04-16T02:47:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c3c06ecc0 0x7f8c3c071180 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c500ffbd0 0x7f8c5019c440 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 --2- 172.21.15.151:0/352252310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c50105a40 0x7f8c5019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 >> 172.21.15.151:0/352252310 conn(0x7f8c500755a0 msgr2=0x7f8c500685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:15.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 shutdown_connections 2024-04-16T02:47:15.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.471+0000 7f8c55550700 1 -- 172.21.15.151:0/352252310 wait complete. 2024-04-16T02:47:15.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.575+0000 7f3de9865700 1 -- 172.21.15.151:0/488998040 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de40ffa40 msgr2=0x7f3de40ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.575+0000 7f3de9865700 1 --2- 172.21.15.151:0/488998040 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de40ffa40 0x7f3de40ffec0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4009910 tx=0x7f3dd4009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:15.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.575+0000 7f3de9865700 1 -- 172.21.15.151:0/488998040 shutdown_connections 2024-04-16T02:47:15.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.575+0000 7f3de9865700 1 --2- 172.21.15.151:0/488998040 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de40ffa40 0x7f3de40ffec0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.575+0000 7f3de9865700 1 --2- 172.21.15.151:0/488998040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de4105a40 0x7f3de4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.575+0000 7f3de9865700 1 -- 172.21.15.151:0/488998040 >> 172.21.15.151:0/488998040 conn(0x7f3de40fb550 msgr2=0x7f3de40fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:15.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.575+0000 7f3de9865700 1 -- 172.21.15.151:0/488998040 shutdown_connections 2024-04-16T02:47:15.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.575+0000 7f3de9865700 1 -- 172.21.15.151:0/488998040 wait complete. 2024-04-16T02:47:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.576+0000 7f3de9865700 1 Processor -- start 2024-04-16T02:47:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.576+0000 7f3de9865700 1 -- start start 2024-04-16T02:47:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.576+0000 7f3de9865700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40ffa40 0x7f3de41975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.576+0000 7f3de9865700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de4105a40 0x7f3de4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.576+0000 7f3de9865700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de4198110 con 0x7f3de4105a40 2024-04-16T02:47:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.576+0000 7f3de9865700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de4198250 con 0x7f3de40ffa40 2024-04-16T02:47:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.576+0000 7f3de27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de4105a40 0x7f3de4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.576+0000 7f3de27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de4105a40 0x7f3de4197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46802/0 (socket says 172.21.15.151:46802) 2024-04-16T02:47:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.576+0000 7f3de27fc700 1 -- 172.21.15.151:0/71527996 learned_addr learned my addr 172.21.15.151:0/71527996 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de2ffd700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40ffa40 0x7f3de41975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de27fc700 1 -- 172.21.15.151:0/71527996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40ffa40 msgr2=0x7f3de41975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de27fc700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40ffa40 0x7f3de41975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de27fc700 1 -- 172.21.15.151:0/71527996 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd40095f0 con 0x7f3de4105a40 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de27fc700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de4105a40 0x7f3de4197af0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4006010 tx=0x7f3dd400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de8863700 1 -- 172.21.15.151:0/71527996 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dd401d070 con 0x7f3de4105a40 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de8863700 1 -- 172.21.15.151:0/71527996 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dd400f460 con 0x7f3de4105a40 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de41a0620 con 0x7f3de4105a40 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de8863700 1 -- 172.21.15.151:0/71527996 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dd4017610 con 0x7f3de4105a40 2024-04-16T02:47:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.577+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de41a0a30 con 0x7f3de4105a40 2024-04-16T02:47:15.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.579+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de4107fd0 con 0x7f3de4105a40 2024-04-16T02:47:15.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.579+0000 7f3de8863700 1 -- 172.21.15.151:0/71527996 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3dd400f5d0 con 0x7f3de4105a40 2024-04-16T02:47:15.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.579+0000 7f3de8863700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3dcc06ecc0 0x7f3dcc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:15.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.579+0000 7f3de8863700 1 -- 172.21.15.151:0/71527996 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3dd40903e0 con 0x7f3de4105a40 2024-04-16T02:47:15.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.579+0000 7f3de2ffd700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3dcc06ecc0 0x7f3dcc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:15.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.580+0000 7f3de2ffd700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3dcc06ecc0 0x7f3dcc071180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8005950 tx=0x7f3dd800a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:15.588 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.584+0000 7f3de8863700 1 -- 172.21.15.151:0/71527996 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3dd4059190 con 0x7f3de4105a40 2024-04-16T02:47:15.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.792+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3de402cd30 con 0x7f3dcc06ecc0 2024-04-16T02:47:15.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.802+0000 7f3de8863700 1 -- 172.21.15.151:0/71527996 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3de402cd30 con 0x7f3dcc06ecc0 2024-04-16T02:47:15.806 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (8m) 2m ago 20m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (20m) 2m ago 20m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (19m) 118s ago 19m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (6m) 2m ago 20m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (6m) 118s ago 19m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (8m) 2m ago 19m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (2m) 2m ago 14m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (2m) 2m ago 14m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (14m) 118s ago 14m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (2m) 118s ago 14m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (10m) 2m ago 21m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (9m) 118s ago 19m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (7m) 2m ago 21m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:47:15.807 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (7m) 118s ago 19m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:47:15.808 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (9m) 2m ago 20m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:47:15.808 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (9m) 118s ago 19m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:47:15.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (5m) 2m ago 18m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:47:15.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (5m) 2m ago 17m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:47:15.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (4m) 2m ago 17m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:47:15.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (4m) 118s ago 16m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:47:15.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (3m) 118s ago 16m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:47:15.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (3m) 118s ago 16m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:47:15.808 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (9m) 2m ago 19m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3dcc06ecc0 msgr2=0x7f3dcc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3dcc06ecc0 0x7f3dcc071180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8005950 tx=0x7f3dd800a300 comp rx=0 tx=0).stop 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de4105a40 msgr2=0x7f3de4197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de4105a40 0x7f3de4197af0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4006010 tx=0x7f3dd400bd70 comp rx=0 tx=0).stop 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 shutdown_connections 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3dcc06ecc0 0x7f3dcc071180 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40ffa40 0x7f3de41975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 --2- 172.21.15.151:0/71527996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3de4105a40 0x7f3de4197af0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 >> 172.21.15.151:0/71527996 conn(0x7f3de40fb550 msgr2=0x7f3de40fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 shutdown_connections 2024-04-16T02:47:15.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.805+0000 7f3de9865700 1 -- 172.21.15.151:0/71527996 wait complete. 2024-04-16T02:47:15.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.911+0000 7f9cfa711700 1 -- 172.21.15.151:0/4176784710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40730b0 msgr2=0x7f9cf4073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.911+0000 7f9cfa711700 1 --2- 172.21.15.151:0/4176784710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40730b0 0x7f9cf4073490 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009910 tx=0x7f9ce4009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:15.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.911+0000 7f9cfa711700 1 -- 172.21.15.151:0/4176784710 shutdown_connections 2024-04-16T02:47:15.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.911+0000 7f9cfa711700 1 --2- 172.21.15.151:0/4176784710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cf40739d0 0x7f9cf410cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.911+0000 7f9cfa711700 1 --2- 172.21.15.151:0/4176784710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40730b0 0x7f9cf4073490 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.911+0000 7f9cfa711700 1 -- 172.21.15.151:0/4176784710 >> 172.21.15.151:0/4176784710 conn(0x7f9cf40fc650 msgr2=0x7f9cf40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:15.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.911+0000 7f9cfa711700 1 -- 172.21.15.151:0/4176784710 shutdown_connections 2024-04-16T02:47:15.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.911+0000 7f9cfa711700 1 -- 172.21.15.151:0/4176784710 wait complete. 2024-04-16T02:47:15.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.912+0000 7f9cfa711700 1 Processor -- start 2024-04-16T02:47:15.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.912+0000 7f9cfa711700 1 -- start start 2024-04-16T02:47:15.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.912+0000 7f9cfa711700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cf40730b0 0x7f9cf4198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:15.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.912+0000 7f9cfa711700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40739d0 0x7f9cf4198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:15.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.912+0000 7f9cfa711700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf4199530 con 0x7f9cf40739d0 2024-04-16T02:47:15.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.912+0000 7f9cfa711700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf419d340 con 0x7f9cf40730b0 2024-04-16T02:47:15.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.912+0000 7f9cf37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40739d0 0x7f9cf4198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.912+0000 7f9cf37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40739d0 0x7f9cf4198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46820/0 (socket says 172.21.15.151:46820) 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.912+0000 7f9cf37fe700 1 -- 172.21.15.151:0/2683498084 learned_addr learned my addr 172.21.15.151:0/2683498084 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cf3fff700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cf40730b0 0x7f9cf4198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cf37fe700 1 -- 172.21.15.151:0/2683498084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cf40730b0 msgr2=0x7f9cf4198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cf37fe700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cf40730b0 0x7f9cf4198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cf37fe700 1 -- 172.21.15.151:0/2683498084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce40095f0 con 0x7f9cf40739d0 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cf37fe700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40739d0 0x7f9cf4198ea0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00d8d0 tx=0x7f9cdc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cf17fa700 1 -- 172.21.15.151:0/2683498084 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cdc009940 con 0x7f9cf40739d0 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cf17fa700 1 -- 172.21.15.151:0/2683498084 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cdc010460 con 0x7f9cf40739d0 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf419d620 con 0x7f9cf40739d0 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cf17fa700 1 -- 172.21.15.151:0/2683498084 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cdc00f5f0 con 0x7f9cf40739d0 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.913+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf419db70 con 0x7f9cf40739d0 2024-04-16T02:47:15.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.914+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf410a5e0 con 0x7f9cf40739d0 2024-04-16T02:47:15.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.915+0000 7f9cf17fa700 1 -- 172.21.15.151:0/2683498084 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9cdc0105d0 con 0x7f9cf40739d0 2024-04-16T02:47:15.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.915+0000 7f9cf17fa700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9ce006ecc0 0x7f9ce0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:15.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.915+0000 7f9cf17fa700 1 -- 172.21.15.151:0/2683498084 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9cdc08ef20 con 0x7f9cf40739d0 2024-04-16T02:47:15.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.915+0000 7f9cf3fff700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9ce006ecc0 0x7f9ce0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:15.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.915+0000 7f9cf3fff700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9ce006ecc0 0x7f9ce0071180 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4006010 tx=0x7f9ce401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:15.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:15.920+0000 7f9cf17fa700 1 -- 172.21.15.151:0/2683498084 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9cdc05b760 con 0x7f9cf40739d0 2024-04-16T02:47:16.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.201+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9cf404ea90 con 0x7f9cf40739d0 2024-04-16T02:47:16.206 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:15 smithi151 ceph-mon[112233]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:16.206 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:15 smithi151 ceph-mon[112233]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:16.206 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:15 smithi151 ceph-mon[112233]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:16.206 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:15 smithi151 ceph-mon[112233]: pgmap v259: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:16.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.202+0000 7f9cf17fa700 1 -- 172.21.15.151:0/2683498084 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f9cdc05b2f0 con 0x7f9cf40739d0 2024-04-16T02:47:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:47:16.207 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:47:16.208 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:47:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:47:16.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.204+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9ce006ecc0 msgr2=0x7f9ce0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.204+0000 7f9cfa711700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9ce006ecc0 0x7f9ce0071180 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4006010 tx=0x7f9ce401a040 comp rx=0 tx=0).stop 2024-04-16T02:47:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.204+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40739d0 msgr2=0x7f9cf4198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.204+0000 7f9cfa711700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40739d0 0x7f9cf4198ea0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00d8d0 tx=0x7f9cdc00dc90 comp rx=0 tx=0).stop 2024-04-16T02:47:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.204+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 shutdown_connections 2024-04-16T02:47:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.204+0000 7f9cfa711700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9ce006ecc0 0x7f9ce0071180 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.204+0000 7f9cfa711700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cf40730b0 0x7f9cf4198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.204+0000 7f9cfa711700 1 --2- 172.21.15.151:0/2683498084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cf40739d0 0x7f9cf4198ea0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.205+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 >> 172.21.15.151:0/2683498084 conn(0x7f9cf40fc650 msgr2=0x7f9cf4107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.205+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 shutdown_connections 2024-04-16T02:47:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.205+0000 7f9cfa711700 1 -- 172.21.15.151:0/2683498084 wait complete. 2024-04-16T02:47:16.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:15 smithi153 ceph-mon[87858]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:16.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:15 smithi153 ceph-mon[87858]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:16.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:15 smithi153 ceph-mon[87858]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:16.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:15 smithi153 ceph-mon[87858]: pgmap v259: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:16.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 -- 172.21.15.151:0/23108532 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 msgr2=0x7f1bac0ff330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 --2- 172.21.15.151:0/23108532 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 0x7f1bac0ff330 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009940 tx=0x7f1ba8009c50 comp rx=0 tx=0).stop 2024-04-16T02:47:16.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 -- 172.21.15.151:0/23108532 shutdown_connections 2024-04-16T02:47:16.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 --2- 172.21.15.151:0/23108532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac0ff900 0x7f1bac10ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 --2- 172.21.15.151:0/23108532 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 0x7f1bac0ff330 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 -- 172.21.15.151:0/23108532 >> 172.21.15.151:0/23108532 conn(0x7f1bac075dd0 msgr2=0x7f1bac0761e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:16.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 -- 172.21.15.151:0/23108532 shutdown_connections 2024-04-16T02:47:16.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 -- 172.21.15.151:0/23108532 wait complete. 2024-04-16T02:47:16.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 Processor -- start 2024-04-16T02:47:16.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.310+0000 7f1bb32b0700 1 -- start start 2024-04-16T02:47:16.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb32b0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 0x7f1bac19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:16.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb32b0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac0ff900 0x7f1bac19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb32b0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac19f270 con 0x7f1bac0fef50 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb32b0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac198720 con 0x7f1bac0ff900 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb104c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 0x7f1bac19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb104c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 0x7f1bac19e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46830/0 (socket says 172.21.15.151:46830) 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb104c700 1 -- 172.21.15.151:0/453679582 learned_addr learned my addr 172.21.15.151:0/453679582 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb084b700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac0ff900 0x7f1bac19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb104c700 1 -- 172.21.15.151:0/453679582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac0ff900 msgr2=0x7f1bac19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb104c700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac0ff900 0x7f1bac19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb104c700 1 -- 172.21.15.151:0/453679582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba80095f0 con 0x7f1bac0fef50 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb104c700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 0x7f1bac19e6a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b580 tx=0x7f1ba80055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1ba27fc700 1 -- 172.21.15.151:0/453679582 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba801d070 con 0x7f1bac0fef50 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1ba27fc700 1 -- 172.21.15.151:0/453679582 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ba8005dc0 con 0x7f1bac0fef50 2024-04-16T02:47:16.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac1989a0 con 0x7f1bac0fef50 2024-04-16T02:47:16.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1ba27fc700 1 -- 172.21.15.151:0/453679582 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba8021410 con 0x7f1bac0fef50 2024-04-16T02:47:16.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.311+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bac198f10 con 0x7f1bac0fef50 2024-04-16T02:47:16.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.312+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bac10a460 con 0x7f1bac0fef50 2024-04-16T02:47:16.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.313+0000 7f1ba27fc700 1 -- 172.21.15.151:0/453679582 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1ba800f7f0 con 0x7f1bac0fef50 2024-04-16T02:47:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.314+0000 7f1ba27fc700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b9806ecc0 0x7f1b98071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.314+0000 7f1bb084b700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b9806ecc0 0x7f1b98071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.314+0000 7f1ba27fc700 1 -- 172.21.15.151:0/453679582 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1ba8090000 con 0x7f1bac0fef50 2024-04-16T02:47:16.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.314+0000 7f1bb084b700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b9806ecc0 0x7f1b98071180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1bac199b90 tx=0x7f1b9c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:16.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.318+0000 7f1ba27fc700 1 -- 172.21.15.151:0/453679582 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1ba805c960 con 0x7f1bac0fef50 2024-04-16T02:47:16.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.560+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1bac04ea90 con 0x7f1bac0fef50 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.562+0000 7f1ba27fc700 1 -- 172.21.15.151:0/453679582 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f1ba805c4f0 con 0x7f1bac0fef50 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:47:16.565 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:47:16.566 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:47:16.567 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:47:16.568 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:47:16.568 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:47:16.568 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:47:16.568 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:47:16.568 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b9806ecc0 msgr2=0x7f1b98071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b9806ecc0 0x7f1b98071180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1bac199b90 tx=0x7f1b9c00b410 comp rx=0 tx=0).stop 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 msgr2=0x7f1bac19e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 0x7f1bac19e6a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b580 tx=0x7f1ba80055f0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 shutdown_connections 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b9806ecc0 0x7f1b98071180 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1bac0fef50 0x7f1bac19e6a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 --2- 172.21.15.151:0/453679582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac0ff900 0x7f1bac19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 >> 172.21.15.151:0/453679582 conn(0x7f1bac075dd0 msgr2=0x7f1bac0fc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 shutdown_connections 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.564+0000 7f1bb32b0700 1 -- 172.21.15.151:0/453679582 wait complete. 2024-04-16T02:47:16.569 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:47:16.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.671+0000 7f2596a63700 1 -- 172.21.15.151:0/2318276943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590101cc0 msgr2=0x7f25901020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.671+0000 7f2596a63700 1 --2- 172.21.15.151:0/2318276943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590101cc0 0x7f25901020a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2580009910 tx=0x7f2580009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:16.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.671+0000 7f2596a63700 1 -- 172.21.15.151:0/2318276943 shutdown_connections 2024-04-16T02:47:16.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.671+0000 7f2596a63700 1 --2- 172.21.15.151:0/2318276943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25901025e0 0x7f259010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.671+0000 7f2596a63700 1 --2- 172.21.15.151:0/2318276943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590101cc0 0x7f25901020a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.671+0000 7f2596a63700 1 -- 172.21.15.151:0/2318276943 >> 172.21.15.151:0/2318276943 conn(0x7f25900fb570 msgr2=0x7f25900fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:16.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.671+0000 7f2596a63700 1 -- 172.21.15.151:0/2318276943 shutdown_connections 2024-04-16T02:47:16.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.671+0000 7f2596a63700 1 -- 172.21.15.151:0/2318276943 wait complete. 2024-04-16T02:47:16.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.671+0000 7f2596a63700 1 Processor -- start 2024-04-16T02:47:16.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f2596a63700 1 -- start start 2024-04-16T02:47:16.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f2596a63700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101cc0 0x7f25901966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:16.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f2596a63700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25901025e0 0x7f2590196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f2596a63700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2590197290 con 0x7f25901025e0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f2596a63700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2590199fc0 con 0x7f2590101cc0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f258f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25901025e0 0x7f2590196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f258f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25901025e0 0x7f2590196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46970/0 (socket says 172.21.15.151:46970) 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f258f7fe700 1 -- 172.21.15.151:0/1877667737 learned_addr learned my addr 172.21.15.151:0/1877667737 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f258ffff700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101cc0 0x7f25901966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f258f7fe700 1 -- 172.21.15.151:0/1877667737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101cc0 msgr2=0x7f25901966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f258f7fe700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101cc0 0x7f25901966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.672+0000 7f258f7fe700 1 -- 172.21.15.151:0/1877667737 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25800095f0 con 0x7f25901025e0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.673+0000 7f258f7fe700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25901025e0 0x7f2590196c00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f258400d6d0 tx=0x7f258400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.673+0000 7f258d7fa700 1 -- 172.21.15.151:0/1877667737 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f258400f860 con 0x7f25901025e0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.673+0000 7f258d7fa700 1 -- 172.21.15.151:0/1877667737 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f258400fe80 con 0x7f25901025e0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.673+0000 7f258d7fa700 1 -- 172.21.15.151:0/1877667737 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2584010ea0 con 0x7f25901025e0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.673+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259019a2a0 con 0x7f25901025e0 2024-04-16T02:47:16.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.673+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259019a7f0 con 0x7f25901025e0 2024-04-16T02:47:16.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.674+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25901081d0 con 0x7f25901025e0 2024-04-16T02:47:16.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.674+0000 7f258d7fa700 1 -- 172.21.15.151:0/1877667737 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f258400e700 con 0x7f25901025e0 2024-04-16T02:47:16.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.675+0000 7f258d7fa700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:16.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.675+0000 7f258d7fa700 1 -- 172.21.15.151:0/1877667737 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f258408ea50 con 0x7f25901025e0 2024-04-16T02:47:16.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.675+0000 7f258ffff700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:16.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.675+0000 7f258ffff700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f258000b5c0 tx=0x7f258001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:16.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.679+0000 7f258d7fa700 1 -- 172.21.15.151:0/1877667737 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2584057870 con 0x7f25901025e0 2024-04-16T02:47:16.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.902+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f259019aad0 con 0x7f257806ecc0 2024-04-16T02:47:16.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.903+0000 7f258d7fa700 1 -- 172.21.15.151:0/1877667737 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f259019aad0 con 0x7f257806ecc0 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:47:16.907 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 msgr2=0x7f2578071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f258000b5c0 tx=0x7f258001a040 comp rx=0 tx=0).stop 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25901025e0 msgr2=0x7f2590196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25901025e0 0x7f2590196c00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f258400d6d0 tx=0x7f258400da90 comp rx=0 tx=0).stop 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 shutdown_connections 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101cc0 0x7f25901966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 --2- 172.21.15.151:0/1877667737 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25901025e0 0x7f2590196c00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 >> 172.21.15.151:0/1877667737 conn(0x7f25900fb570 msgr2=0x7f2590105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 shutdown_connections 2024-04-16T02:47:16.910 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:16.906+0000 7f2596a63700 1 -- 172.21.15.151:0/1877667737 wait complete. 2024-04-16T02:47:17.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.010+0000 7fef861af700 1 -- 172.21.15.151:0/2219318467 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80101770 msgr2=0x7fef801057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:17.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.010+0000 7fef861af700 1 --2- 172.21.15.151:0/2219318467 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80101770 0x7fef801057c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fef68009910 tx=0x7fef68009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:17.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.011+0000 7fef861af700 1 -- 172.21.15.151:0/2219318467 shutdown_connections 2024-04-16T02:47:17.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.011+0000 7fef861af700 1 --2- 172.21.15.151:0/2219318467 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80101770 0x7fef801057c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:17.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.011+0000 7fef861af700 1 --2- 172.21.15.151:0/2219318467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef80100dc0 0x7fef801011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:17.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.011+0000 7fef861af700 1 -- 172.21.15.151:0/2219318467 >> 172.21.15.151:0/2219318467 conn(0x7fef800fc650 msgr2=0x7fef800fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:17.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.011+0000 7fef861af700 1 -- 172.21.15.151:0/2219318467 shutdown_connections 2024-04-16T02:47:17.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.011+0000 7fef861af700 1 -- 172.21.15.151:0/2219318467 wait complete. 2024-04-16T02:47:17.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.011+0000 7fef861af700 1 Processor -- start 2024-04-16T02:47:17.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.011+0000 7fef861af700 1 -- start start 2024-04-16T02:47:17.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef861af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80100dc0 0x7fef8019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:17.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef861af700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef80101770 0x7fef8019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef861af700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8019d130 con 0x7fef80100dc0 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef861af700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef801965e0 con 0x7fef80101770 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80100dc0 0x7fef8019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80100dc0 0x7fef8019c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46988/0 (socket says 172.21.15.151:46988) 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7f7fe700 1 -- 172.21.15.151:0/1572919477 learned_addr learned my addr 172.21.15.151:0/1572919477 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7f7fe700 1 -- 172.21.15.151:0/1572919477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef80101770 msgr2=0x7fef8019caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7effd700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef80101770 0x7fef8019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7f7fe700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef80101770 0x7fef8019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7f7fe700 1 -- 172.21.15.151:0/1572919477 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef680095f0 con 0x7fef80100dc0 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7f7fe700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80100dc0 0x7fef8019c560 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fef7000c8a0 tx=0x7fef7000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7cff9700 1 -- 172.21.15.151:0/1572919477 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef7000cea0 con 0x7fef80100dc0 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7cff9700 1 -- 172.21.15.151:0/1572919477 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef70004830 con 0x7fef80100dc0 2024-04-16T02:47:17.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef801968c0 con 0x7fef80100dc0 2024-04-16T02:47:17.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.012+0000 7fef7cff9700 1 -- 172.21.15.151:0/1572919477 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef700055b0 con 0x7fef80100dc0 2024-04-16T02:47:17.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.013+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef80196e10 con 0x7fef80100dc0 2024-04-16T02:47:17.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.013+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef801091c0 con 0x7fef80100dc0 2024-04-16T02:47:17.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.014+0000 7fef7cff9700 1 -- 172.21.15.151:0/1572919477 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fef700049a0 con 0x7fef80100dc0 2024-04-16T02:47:17.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.014+0000 7fef7cff9700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fef6c06ecc0 0x7fef6c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:17.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.015+0000 7fef7cff9700 1 -- 172.21.15.151:0/1572919477 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fef7008ea80 con 0x7fef80100dc0 2024-04-16T02:47:17.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.015+0000 7fef7effd700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fef6c06ecc0 0x7fef6c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:17.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.015+0000 7fef7effd700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fef6c06ecc0 0x7fef6c071180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fef680059d0 tx=0x7fef68005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:17.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.019+0000 7fef7cff9700 1 -- 172.21.15.151:0/1572919477 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fef7005b3e0 con 0x7fef80100dc0 2024-04-16T02:47:17.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:16 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2683498084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:47:17.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:16 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/453679582' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:47:17.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:16 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2683498084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:47:17.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:16 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/453679582' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:47:17.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.302+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fef8004ea90 con 0x7fef80100dc0 2024-04-16T02:47:17.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.302+0000 7fef7cff9700 1 -- 172.21.15.151:0/1572919477 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fef7001d090 con 0x7fef80100dc0 2024-04-16T02:47:17.305 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:47:17.306 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:47:17.306 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:47:17.306 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:47:17.306 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:47:17.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.304+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fef6c06ecc0 msgr2=0x7fef6c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.304+0000 7fef861af700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fef6c06ecc0 0x7fef6c071180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fef680059d0 tx=0x7fef68005960 comp rx=0 tx=0).stop 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.304+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80100dc0 msgr2=0x7fef8019c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.304+0000 7fef861af700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80100dc0 0x7fef8019c560 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fef7000c8a0 tx=0x7fef7000cc60 comp rx=0 tx=0).stop 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.304+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 shutdown_connections 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.304+0000 7fef861af700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fef6c06ecc0 0x7fef6c071180 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.304+0000 7fef861af700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef80100dc0 0x7fef8019c560 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.304+0000 7fef861af700 1 --2- 172.21.15.151:0/1572919477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef80101770 0x7fef8019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.304+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 >> 172.21.15.151:0/1572919477 conn(0x7fef800fc650 msgr2=0x7fef800fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.305+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 shutdown_connections 2024-04-16T02:47:17.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:17.305+0000 7fef861af700 1 -- 172.21.15.151:0/1572919477 wait complete. 2024-04-16T02:47:18.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:17 smithi151 ceph-mon[112233]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:18.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:17 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1572919477' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:47:18.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:17 smithi151 ceph-mon[112233]: pgmap v260: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:17 smithi153 ceph-mon[87858]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:17 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1572919477' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:47:18.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:17 smithi153 ceph-mon[87858]: pgmap v260: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:19 smithi151 ceph-mon[112233]: pgmap v261: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:19 smithi153 ceph-mon[87858]: pgmap v261: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:22.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:21 smithi151 ceph-mon[112233]: pgmap v262: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:21 smithi153 ceph-mon[87858]: pgmap v262: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:24.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:47:24.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:23 smithi151 ceph-mon[112233]: pgmap v263: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:47:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:23 smithi153 ceph-mon[87858]: pgmap v263: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:26.145 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:25 smithi153 ceph-mon[87858]: pgmap v264: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:26.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:25 smithi151 ceph-mon[112233]: pgmap v264: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:28.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:27 smithi151 ceph-mon[112233]: pgmap v265: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:27 smithi153 ceph-mon[87858]: pgmap v265: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:31.146 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:30 smithi151 ceph-mon[112233]: pgmap v266: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:31.198 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:30 smithi153 ceph-mon[87858]: pgmap v266: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:33.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:32 smithi151 ceph-mon[112233]: pgmap v267: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:33.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:32 smithi153 ceph-mon[87858]: pgmap v267: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:35.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:34 smithi151 ceph-mon[112233]: pgmap v268: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:35.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:34 smithi153 ceph-mon[87858]: pgmap v268: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:37.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:37 smithi151 ceph-mon[112233]: pgmap v269: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:37.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:37 smithi153 ceph-mon[87858]: pgmap v269: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:39.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:39 smithi151 ceph-mon[112233]: pgmap v270: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:39.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:47:39.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:39 smithi153 ceph-mon[87858]: pgmap v270: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:39.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:47:41.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:41 smithi151 ceph-mon[112233]: pgmap v271: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:41 smithi153 ceph-mon[87858]: pgmap v271: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:43.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:43 smithi151 ceph-mon[112233]: pgmap v272: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:43.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:43 smithi153 ceph-mon[87858]: pgmap v272: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:45.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:45 smithi151 ceph-mon[112233]: pgmap v273: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:45.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:45 smithi153 ceph-mon[87858]: pgmap v273: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:47.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.425+0000 7fd2d7fda700 1 -- 172.21.15.151:0/849219342 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 msgr2=0x7fd2d01057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:47.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.425+0000 7fd2d7fda700 1 --2- 172.21.15.151:0/849219342 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 0x7fd2d01057d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009910 tx=0x7fd2c0009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:47.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.425+0000 7fd2d7fda700 1 -- 172.21.15.151:0/849219342 shutdown_connections 2024-04-16T02:47:47.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.425+0000 7fd2d7fda700 1 --2- 172.21.15.151:0/849219342 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 0x7fd2d01057d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.425+0000 7fd2d7fda700 1 --2- 172.21.15.151:0/849219342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d0100dd0 0x7fd2d01011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.425+0000 7fd2d7fda700 1 -- 172.21.15.151:0/849219342 >> 172.21.15.151:0/849219342 conn(0x7fd2d00fc640 msgr2=0x7fd2d00fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.426+0000 7fd2d7fda700 1 -- 172.21.15.151:0/849219342 shutdown_connections 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.426+0000 7fd2d7fda700 1 -- 172.21.15.151:0/849219342 wait complete. 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.426+0000 7fd2d7fda700 1 Processor -- start 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.426+0000 7fd2d7fda700 1 -- start start 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.426+0000 7fd2d7fda700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d0100dd0 0x7fd2d019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.426+0000 7fd2d7fda700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 0x7fd2d019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.426+0000 7fd2d7fda700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d019d240 con 0x7fd2d0101780 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.426+0000 7fd2d7fda700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d01966f0 con 0x7fd2d0100dd0 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d5575700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 0x7fd2d019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d5575700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 0x7fd2d019cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53636/0 (socket says 172.21.15.151:53636) 2024-04-16T02:47:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d5575700 1 -- 172.21.15.151:0/1606613066 learned_addr learned my addr 172.21.15.151:0/1606613066 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d5d76700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d0100dd0 0x7fd2d019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d5575700 1 -- 172.21.15.151:0/1606613066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d0100dd0 msgr2=0x7fd2d019c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d5575700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d0100dd0 0x7fd2d019c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d5575700 1 -- 172.21.15.151:0/1606613066 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c00095f0 con 0x7fd2d0101780 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d5575700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 0x7fd2d019cbb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000bda0 tx=0x7fd2c000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2c6ffd700 1 -- 172.21.15.151:0/1606613066 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2c001d070 con 0x7fd2d0101780 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d0196970 con 0x7fd2d0101780 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2c6ffd700 1 -- 172.21.15.151:0/1606613066 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2c0004fb0 con 0x7fd2d0101780 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2c6ffd700 1 -- 172.21.15.151:0/1606613066 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2c000f460 con 0x7fd2d0101780 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.427+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d0196e60 con 0x7fd2d0101780 2024-04-16T02:47:47.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.428+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d0109120 con 0x7fd2d0101780 2024-04-16T02:47:47.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.429+0000 7fd2c6ffd700 1 -- 172.21.15.151:0/1606613066 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd2c0022800 con 0x7fd2d0101780 2024-04-16T02:47:47.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.429+0000 7fd2c6ffd700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd2bc06ecc0 0x7fd2bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:47.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.430+0000 7fd2c6ffd700 1 -- 172.21.15.151:0/1606613066 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd2c00903a0 con 0x7fd2d0101780 2024-04-16T02:47:47.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.430+0000 7fd2d5d76700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd2bc06ecc0 0x7fd2bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:47.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.430+0000 7fd2d5d76700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd2bc06ecc0 0x7fd2bc071180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc006fd0 tx=0x7fd2cc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:47.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.434+0000 7fd2c6ffd700 1 -- 172.21.15.151:0/1606613066 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd2c005ce50 con 0x7fd2d0101780 2024-04-16T02:47:47.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.654+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2d0197140 con 0x7fd2bc06ecc0 2024-04-16T02:47:47.657 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:47 smithi151 ceph-mon[112233]: pgmap v274: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:47.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.655+0000 7fd2c6ffd700 1 -- 172.21.15.151:0/1606613066 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd2d0197140 con 0x7fd2bc06ecc0 2024-04-16T02:47:47.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd2bc06ecc0 msgr2=0x7fd2bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd2bc06ecc0 0x7fd2bc071180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc006fd0 tx=0x7fd2cc008040 comp rx=0 tx=0).stop 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 msgr2=0x7fd2d019cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 0x7fd2d019cbb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000bda0 tx=0x7fd2c000be80 comp rx=0 tx=0).stop 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 shutdown_connections 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd2bc06ecc0 0x7fd2bc071180 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d0100dd0 0x7fd2d019c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 --2- 172.21.15.151:0/1606613066 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2d0101780 0x7fd2d019cbb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 >> 172.21.15.151:0/1606613066 conn(0x7fd2d00fc640 msgr2=0x7fd2d00fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 shutdown_connections 2024-04-16T02:47:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.657+0000 7fd2d7fda700 1 -- 172.21.15.151:0/1606613066 wait complete. 2024-04-16T02:47:47.671 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:47:47.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.764+0000 7f71b7f4d700 1 -- 172.21.15.151:0/3395869767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 msgr2=0x7f71b0105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:47.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.764+0000 7f71b7f4d700 1 --2- 172.21.15.151:0/3395869767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 0x7f71b0105790 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009910 tx=0x7f71ac009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:47.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.764+0000 7f71b7f4d700 1 -- 172.21.15.151:0/3395869767 shutdown_connections 2024-04-16T02:47:47.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.764+0000 7f71b7f4d700 1 --2- 172.21.15.151:0/3395869767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 0x7f71b0105790 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.764+0000 7f71b7f4d700 1 --2- 172.21.15.151:0/3395869767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b0068670 0x7f71b0068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.764+0000 7f71b7f4d700 1 -- 172.21.15.151:0/3395869767 >> 172.21.15.151:0/3395869767 conn(0x7f71b00755a0 msgr2=0x7f71b00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:47.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.764+0000 7f71b7f4d700 1 -- 172.21.15.151:0/3395869767 shutdown_connections 2024-04-16T02:47:47.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.764+0000 7f71b7f4d700 1 -- 172.21.15.151:0/3395869767 wait complete. 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b7f4d700 1 Processor -- start 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b7f4d700 1 -- start start 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b7f4d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b0068670 0x7f71b00ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b7f4d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 0x7f71b01001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b7f4d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b0100720 con 0x7f71b0069020 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b7f4d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b0100860 con 0x7f71b0068670 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b54e8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 0x7f71b01001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b54e8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 0x7f71b01001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53652/0 (socket says 172.21.15.151:53652) 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b54e8700 1 -- 172.21.15.151:0/198271979 learned_addr learned my addr 172.21.15.151:0/198271979 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.765+0000 7f71b5ce9700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b0068670 0x7f71b00ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.766+0000 7f71b54e8700 1 -- 172.21.15.151:0/198271979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b0068670 msgr2=0x7f71b00ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.766+0000 7f71b54e8700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b0068670 0x7f71b00ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.766+0000 7f71b54e8700 1 -- 172.21.15.151:0/198271979 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac0095f0 con 0x7f71b0069020 2024-04-16T02:47:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.766+0000 7f71b54e8700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 0x7f71b01001e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f71ac0092e0 tx=0x7f71ac00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.766+0000 7f71a2ffd700 1 -- 172.21.15.151:0/198271979 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71ac01d070 con 0x7f71b0069020 2024-04-16T02:47:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.766+0000 7f71a2ffd700 1 -- 172.21.15.151:0/198271979 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71ac005dc0 con 0x7f71b0069020 2024-04-16T02:47:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.766+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b0100ae0 con 0x7f71b0069020 2024-04-16T02:47:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.766+0000 7f71a2ffd700 1 -- 172.21.15.151:0/198271979 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71ac017400 con 0x7f71b0069020 2024-04-16T02:47:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.766+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b01a2c00 con 0x7f71b0069020 2024-04-16T02:47:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.767+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b01090a0 con 0x7f71b0069020 2024-04-16T02:47:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.767+0000 7f71a2ffd700 1 -- 172.21.15.151:0/198271979 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f71ac00f6f0 con 0x7f71b0069020 2024-04-16T02:47:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.768+0000 7f71a2ffd700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f719c06ecc0 0x7f719c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:47.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.768+0000 7f71a2ffd700 1 -- 172.21.15.151:0/198271979 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f71ac090190 con 0x7f71b0069020 2024-04-16T02:47:47.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.768+0000 7f71b5ce9700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f719c06ecc0 0x7f719c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:47.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.768+0000 7f71b5ce9700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f719c06ecc0 0x7f719c071180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f71a4005950 tx=0x7f71a400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:47.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:47 smithi153 ceph-mon[87858]: pgmap v274: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:47.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.773+0000 7f71a2ffd700 1 -- 172.21.15.151:0/198271979 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f71ac058fc0 con 0x7f71b0069020 2024-04-16T02:47:47.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.991+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f71b00611d0 con 0x7f719c06ecc0 2024-04-16T02:47:47.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.992+0000 7f71a2ffd700 1 -- 172.21.15.151:0/198271979 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f71b00611d0 con 0x7f719c06ecc0 2024-04-16T02:47:47.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f719c06ecc0 msgr2=0x7f719c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:47.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f719c06ecc0 0x7f719c071180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f71a4005950 tx=0x7f71a400b410 comp rx=0 tx=0).stop 2024-04-16T02:47:47.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 msgr2=0x7f71b01001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:47.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 0x7f71b01001e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f71ac0092e0 tx=0x7f71ac00bd70 comp rx=0 tx=0).stop 2024-04-16T02:47:47.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 shutdown_connections 2024-04-16T02:47:47.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f719c06ecc0 0x7f719c071180 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b0068670 0x7f71b00ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 --2- 172.21.15.151:0/198271979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71b0069020 0x7f71b01001e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:47.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 >> 172.21.15.151:0/198271979 conn(0x7f71b00755a0 msgr2=0x7f71b0105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:47.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 shutdown_connections 2024-04-16T02:47:47.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:47.995+0000 7f71b7f4d700 1 -- 172.21.15.151:0/198271979 wait complete. 2024-04-16T02:47:48.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.102+0000 7f60c5bad700 1 -- 172.21.15.151:0/2532234498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 msgr2=0x7f60c0105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.102+0000 7f60c5bad700 1 --2- 172.21.15.151:0/2532234498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 0x7f60c0105790 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009910 tx=0x7f60a8009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:48.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.102+0000 7f60c5bad700 1 -- 172.21.15.151:0/2532234498 shutdown_connections 2024-04-16T02:47:48.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.102+0000 7f60c5bad700 1 --2- 172.21.15.151:0/2532234498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 0x7f60c0105790 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.102+0000 7f60c5bad700 1 --2- 172.21.15.151:0/2532234498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c0068670 0x7f60c0068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.102+0000 7f60c5bad700 1 -- 172.21.15.151:0/2532234498 >> 172.21.15.151:0/2532234498 conn(0x7f60c00755a0 msgr2=0x7f60c00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.102+0000 7f60c5bad700 1 -- 172.21.15.151:0/2532234498 shutdown_connections 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.102+0000 7f60c5bad700 1 -- 172.21.15.151:0/2532234498 wait complete. 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60c5bad700 1 Processor -- start 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60c5bad700 1 -- start start 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60c5bad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c0068670 0x7f60c019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60c5bad700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 0x7f60c019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60c5bad700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c019d220 con 0x7f60c0069020 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60c5bad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c01966d0 con 0x7f60c0068670 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60beffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 0x7f60c019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60beffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 0x7f60c019cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53668/0 (socket says 172.21.15.151:53668) 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60beffd700 1 -- 172.21.15.151:0/3055890024 learned_addr learned my addr 172.21.15.151:0/3055890024 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60bf7fe700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c0068670 0x7f60c019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60beffd700 1 -- 172.21.15.151:0/3055890024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c0068670 msgr2=0x7f60c019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60beffd700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c0068670 0x7f60c019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.103+0000 7f60beffd700 1 -- 172.21.15.151:0/3055890024 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a80095f0 con 0x7f60c0069020 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.104+0000 7f60beffd700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 0x7f60c019cb90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f60a8005f50 tx=0x7f60a800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.104+0000 7f60bcff9700 1 -- 172.21.15.151:0/3055890024 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60a801d070 con 0x7f60c0069020 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.104+0000 7f60bcff9700 1 -- 172.21.15.151:0/3055890024 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60a800f460 con 0x7f60c0069020 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.104+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c0196950 con 0x7f60c0069020 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.104+0000 7f60bcff9700 1 -- 172.21.15.151:0/3055890024 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60a8017630 con 0x7f60c0069020 2024-04-16T02:47:48.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.104+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c0196e40 con 0x7f60c0069020 2024-04-16T02:47:48.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.105+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c01090e0 con 0x7f60c0069020 2024-04-16T02:47:48.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.105+0000 7f60bcff9700 1 -- 172.21.15.151:0/3055890024 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f60a800f5d0 con 0x7f60c0069020 2024-04-16T02:47:48.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.106+0000 7f60bcff9700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f60ac06ecc0 0x7f60ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:48.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.106+0000 7f60bcff9700 1 -- 172.21.15.151:0/3055890024 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f60a80903a0 con 0x7f60c0069020 2024-04-16T02:47:48.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.106+0000 7f60bf7fe700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f60ac06ecc0 0x7f60ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:48.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.106+0000 7f60bf7fe700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f60ac06ecc0 0x7f60ac071180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f60b800a400 tx=0x7f60b800d040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:48.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.111+0000 7f60bcff9700 1 -- 172.21.15.151:0/3055890024 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f60a805cc80 con 0x7f60c0069020 2024-04-16T02:47:48.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.319+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f60c0197120 con 0x7f60ac06ecc0 2024-04-16T02:47:48.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.330+0000 7f60bcff9700 1 -- 172.21.15.151:0/3055890024 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f60c0197120 con 0x7f60ac06ecc0 2024-04-16T02:47:48.333 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:47:48.333 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (9m) 2m ago 20m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:47:48.333 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (20m) 2m ago 20m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:47:48.333 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (19m) 2m ago 19m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:47:48.333 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (6m) 2m ago 20m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:47:48.333 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (6m) 2m ago 19m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:47:48.333 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (9m) 2m ago 20m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:47:48.333 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (3m) 2m ago 15m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (2m) 2m ago 15m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (15m) 2m ago 15m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (2m) 2m ago 15m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (11m) 2m ago 22m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (10m) 2m ago 19m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (8m) 2m ago 22m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (7m) 2m ago 19m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (10m) 2m ago 20m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (9m) 2m ago 19m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (6m) 2m ago 18m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (5m) 2m ago 18m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (5m) 2m ago 17m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:47:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (4m) 2m ago 17m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:47:48.335 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (4m) 2m ago 17m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:47:48.335 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (3m) 2m ago 16m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:47:48.335 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (9m) 2m ago 20m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:47:48.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f60ac06ecc0 msgr2=0x7f60ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f60ac06ecc0 0x7f60ac071180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f60b800a400 tx=0x7f60b800d040 comp rx=0 tx=0).stop 2024-04-16T02:47:48.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 msgr2=0x7f60c019cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 0x7f60c019cb90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f60a8005f50 tx=0x7f60a800bd70 comp rx=0 tx=0).stop 2024-04-16T02:47:48.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 shutdown_connections 2024-04-16T02:47:48.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f60ac06ecc0 0x7f60ac071180 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60c0068670 0x7f60c019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 --2- 172.21.15.151:0/3055890024 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60c0069020 0x7f60c019cb90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 >> 172.21.15.151:0/3055890024 conn(0x7f60c00755a0 msgr2=0x7f60c00fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:48.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 shutdown_connections 2024-04-16T02:47:48.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.333+0000 7f60c5bad700 1 -- 172.21.15.151:0/3055890024 wait complete. 2024-04-16T02:47:48.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 -- 172.21.15.151:0/2972111841 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c102680 msgr2=0x7efd7c10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 --2- 172.21.15.151:0/2972111841 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c102680 0x7efd7c10ab70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efd70009940 tx=0x7efd70009c50 comp rx=0 tx=0).stop 2024-04-16T02:47:48.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 -- 172.21.15.151:0/2972111841 shutdown_connections 2024-04-16T02:47:48.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 --2- 172.21.15.151:0/2972111841 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c102680 0x7efd7c10ab70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 --2- 172.21.15.151:0/2972111841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7c101d60 0x7efd7c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 -- 172.21.15.151:0/2972111841 >> 172.21.15.151:0/2972111841 conn(0x7efd7c0fb570 msgr2=0x7efd7c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:48.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 -- 172.21.15.151:0/2972111841 shutdown_connections 2024-04-16T02:47:48.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 -- 172.21.15.151:0/2972111841 wait complete. 2024-04-16T02:47:48.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 Processor -- start 2024-04-16T02:47:48.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.439+0000 7efd828af700 1 -- start start 2024-04-16T02:47:48.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd828af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c101d60 0x7efd7c19c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd828af700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7c102680 0x7efd7c19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd828af700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c19d070 con 0x7efd7c101d60 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd828af700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c1965b0 con 0x7efd7c102680 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd7bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c101d60 0x7efd7c19c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd7bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c101d60 0x7efd7c19c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53694/0 (socket says 172.21.15.151:53694) 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd7bfff700 1 -- 172.21.15.151:0/3386715927 learned_addr learned my addr 172.21.15.151:0/3386715927 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd7b7fe700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7c102680 0x7efd7c19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd7bfff700 1 -- 172.21.15.151:0/3386715927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7c102680 msgr2=0x7efd7c19ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd7bfff700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7c102680 0x7efd7c19ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd7bfff700 1 -- 172.21.15.151:0/3386715927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd700095f0 con 0x7efd7c101d60 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd7b7fe700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7c102680 0x7efd7c19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd7bfff700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c101d60 0x7efd7c19c530 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00c930 tx=0x7efd6c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:48.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.440+0000 7efd797fa700 1 -- 172.21.15.151:0/3386715927 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd6c007ad0 con 0x7efd7c101d60 2024-04-16T02:47:48.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.441+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c196890 con 0x7efd7c101d60 2024-04-16T02:47:48.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.441+0000 7efd797fa700 1 -- 172.21.15.151:0/3386715927 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd6c007c30 con 0x7efd7c101d60 2024-04-16T02:47:48.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.441+0000 7efd797fa700 1 -- 172.21.15.151:0/3386715927 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd6c018610 con 0x7efd7c101d60 2024-04-16T02:47:48.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.441+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c196de0 con 0x7efd7c101d60 2024-04-16T02:47:48.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.441+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c108270 con 0x7efd7c101d60 2024-04-16T02:47:48.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.442+0000 7efd797fa700 1 -- 172.21.15.151:0/3386715927 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efd6c00f450 con 0x7efd7c101d60 2024-04-16T02:47:48.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.442+0000 7efd797fa700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd6406ecc0 0x7efd64071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:48.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.443+0000 7efd797fa700 1 -- 172.21.15.151:0/3386715927 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efd6c08efb0 con 0x7efd7c101d60 2024-04-16T02:47:48.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.443+0000 7efd7b7fe700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd6406ecc0 0x7efd64071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:48.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.443+0000 7efd7b7fe700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd6406ecc0 0x7efd64071180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0fccb0 tx=0x7efd70005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:48.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.447+0000 7efd797fa700 1 -- 172.21.15.151:0/3386715927 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efd6c0c3ff0 con 0x7efd7c101d60 2024-04-16T02:47:48.644 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:48 smithi151 ceph-mon[112233]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.738+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efd7c197060 con 0x7efd7c101d60 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.739+0000 7efd797fa700 1 -- 172.21.15.151:0/3386715927 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7efd6c05b910 con 0x7efd7c101d60 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:47:48.742 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:47:48.743 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:47:48.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd6406ecc0 msgr2=0x7efd64071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd6406ecc0 0x7efd64071180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0fccb0 tx=0x7efd70005fd0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c101d60 msgr2=0x7efd7c19c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c101d60 0x7efd7c19c530 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00c930 tx=0x7efd6c00ccf0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 shutdown_connections 2024-04-16T02:47:48.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd6406ecc0 0x7efd64071180 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd7c101d60 0x7efd7c19c530 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 --2- 172.21.15.151:0/3386715927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7c102680 0x7efd7c19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 >> 172.21.15.151:0/3386715927 conn(0x7efd7c0fb570 msgr2=0x7efd7c1053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:48.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 shutdown_connections 2024-04-16T02:47:48.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.741+0000 7efd828af700 1 -- 172.21.15.151:0/3386715927 wait complete. 2024-04-16T02:47:48.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:48 smithi153 ceph-mon[87858]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:48.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.846+0000 7f521b90b700 1 -- 172.21.15.151:0/3797483785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 msgr2=0x7f5214068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.846+0000 7f521b90b700 1 --2- 172.21.15.151:0/3797483785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 0x7f5214068a50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5210009910 tx=0x7f5210009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:48.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.846+0000 7f521b90b700 1 -- 172.21.15.151:0/3797483785 shutdown_connections 2024-04-16T02:47:48.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.846+0000 7f521b90b700 1 --2- 172.21.15.151:0/3797483785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214069020 0x7f5214105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.846+0000 7f521b90b700 1 --2- 172.21.15.151:0/3797483785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 0x7f5214068a50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.846+0000 7f521b90b700 1 -- 172.21.15.151:0/3797483785 >> 172.21.15.151:0/3797483785 conn(0x7f52140755a0 msgr2=0x7f52140759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:48.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.846+0000 7f521b90b700 1 -- 172.21.15.151:0/3797483785 shutdown_connections 2024-04-16T02:47:48.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.846+0000 7f521b90b700 1 -- 172.21.15.151:0/3797483785 wait complete. 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f521b90b700 1 Processor -- start 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f521b90b700 1 -- start start 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f521b90b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 0x7f52140ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f521b90b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214069020 0x7f52141001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f521b90b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5214100720 con 0x7f5214068670 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f521b90b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5214100860 con 0x7f5214069020 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f52196a7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 0x7f52140ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f52196a7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 0x7f52140ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53708/0 (socket says 172.21.15.151:53708) 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f52196a7700 1 -- 172.21.15.151:0/1253444854 learned_addr learned my addr 172.21.15.151:0/1253444854 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f5218ea6700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214069020 0x7f52141001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f52196a7700 1 -- 172.21.15.151:0/1253444854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214069020 msgr2=0x7f52141001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f52196a7700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214069020 0x7f52141001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.847+0000 7f52196a7700 1 -- 172.21.15.151:0/1253444854 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100095f0 con 0x7f5214068670 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.848+0000 7f52196a7700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 0x7f52140ffca0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5210006010 tx=0x7f521000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.848+0000 7f52067fc700 1 -- 172.21.15.151:0/1253444854 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f521001d070 con 0x7f5214068670 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.848+0000 7f52067fc700 1 -- 172.21.15.151:0/1253444854 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5210004fb0 con 0x7f5214068670 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.848+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5214100ae0 con 0x7f5214068670 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.848+0000 7f52067fc700 1 -- 172.21.15.151:0/1253444854 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5210022ea0 con 0x7f5214068670 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.848+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52141a2bf0 con 0x7f5214068670 2024-04-16T02:47:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.849+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5214109130 con 0x7f5214068670 2024-04-16T02:47:48.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.849+0000 7f52067fc700 1 -- 172.21.15.151:0/1253444854 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5210022470 con 0x7f5214068670 2024-04-16T02:47:48.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.850+0000 7f52067fc700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f520006ecc0 0x7f5200071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:48.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.850+0000 7f5218ea6700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f520006ecc0 0x7f5200071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:48.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.850+0000 7f52067fc700 1 -- 172.21.15.151:0/1253444854 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5210090400 con 0x7f5214068670 2024-04-16T02:47:48.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.851+0000 7f5218ea6700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f520006ecc0 0x7f5200071180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5208005ea0 tx=0x7f5208005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:48.858 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:48.855+0000 7f52067fc700 1 -- 172.21.15.151:0/1253444854 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5210059260 con 0x7f5214068670 2024-04-16T02:47:49.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.098+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f521404ea90 con 0x7f5214068670 2024-04-16T02:47:49.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.099+0000 7f52067fc700 1 -- 172.21.15.151:0/1253444854 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f5210027770 con 0x7f5214068670 2024-04-16T02:47:49.102 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:47:49.102 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:47:49.103 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:47:49.104 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:47:49.105 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:47:49.105 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:47:49.105 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:47:49.105 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:47:49.105 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:47:49.105 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:47:49.105 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:47:49.105 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:47:49.105 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:47:49.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f520006ecc0 msgr2=0x7f5200071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f520006ecc0 0x7f5200071180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5208005ea0 tx=0x7f5208005e30 comp rx=0 tx=0).stop 2024-04-16T02:47:49.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 msgr2=0x7f52140ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 0x7f52140ffca0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5210006010 tx=0x7f521000be20 comp rx=0 tx=0).stop 2024-04-16T02:47:49.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 shutdown_connections 2024-04-16T02:47:49.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f520006ecc0 0x7f5200071180 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5214068670 0x7f52140ffca0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 --2- 172.21.15.151:0/1253444854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214069020 0x7f52141001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 >> 172.21.15.151:0/1253444854 conn(0x7f52140755a0 msgr2=0x7f52140fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:49.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 shutdown_connections 2024-04-16T02:47:49.107 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.102+0000 7f521b90b700 1 -- 172.21.15.151:0/1253444854 wait complete. 2024-04-16T02:47:49.107 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:47:49.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.208+0000 7f165860e700 1 -- 172.21.15.151:0/672086161 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 msgr2=0x7f1650100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.208+0000 7f165860e700 1 --2- 172.21.15.151:0/672086161 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 0x7f1650100050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f164c009910 tx=0x7f164c009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:49.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.208+0000 7f165860e700 1 -- 172.21.15.151:0/672086161 shutdown_connections 2024-04-16T02:47:49.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.208+0000 7f165860e700 1 --2- 172.21.15.151:0/672086161 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 0x7f1650100050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.208+0000 7f165860e700 1 --2- 172.21.15.151:0/672086161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1650105a40 0x7f1650105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.208+0000 7f165860e700 1 -- 172.21.15.151:0/672086161 >> 172.21.15.151:0/672086161 conn(0x7f16500755a0 msgr2=0x7f16500759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.208+0000 7f165860e700 1 -- 172.21.15.151:0/672086161 shutdown_connections 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.208+0000 7f165860e700 1 -- 172.21.15.151:0/672086161 wait complete. 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f165860e700 1 Processor -- start 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f165860e700 1 -- start start 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f165860e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 0x7f1650101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f165860e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1650105a40 0x7f1650101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f165860e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1650102090 con 0x7f16500ffbd0 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f165860e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16501021d0 con 0x7f1650105a40 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f16563aa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 0x7f1650101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f16563aa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 0x7f1650101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53730/0 (socket says 172.21.15.151:53730) 2024-04-16T02:47:49.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f16563aa700 1 -- 172.21.15.151:0/3011542564 learned_addr learned my addr 172.21.15.151:0/3011542564 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f1655ba9700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1650105a40 0x7f1650101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f16563aa700 1 -- 172.21.15.151:0/3011542564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1650105a40 msgr2=0x7f1650101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f16563aa700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1650105a40 0x7f1650101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.209+0000 7f16563aa700 1 -- 172.21.15.151:0/3011542564 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c0095f0 con 0x7f16500ffbd0 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.210+0000 7f16563aa700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 0x7f1650101610 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f164000d8d0 tx=0x7f164000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.210+0000 7f16477fe700 1 -- 172.21.15.151:0/3011542564 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16400098a0 con 0x7f16500ffbd0 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.210+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16501a2900 con 0x7f16500ffbd0 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.210+0000 7f16477fe700 1 -- 172.21.15.151:0/3011542564 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1640010460 con 0x7f16500ffbd0 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.210+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16501a2cb0 con 0x7f16500ffbd0 2024-04-16T02:47:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.210+0000 7f16477fe700 1 -- 172.21.15.151:0/3011542564 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f164000f5f0 con 0x7f16500ffbd0 2024-04-16T02:47:49.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.211+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16340052f0 con 0x7f16500ffbd0 2024-04-16T02:47:49.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.211+0000 7f16477fe700 1 -- 172.21.15.151:0/3011542564 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1640009a00 con 0x7f16500ffbd0 2024-04-16T02:47:49.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.212+0000 7f16477fe700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f163c06ecc0 0x7f163c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:49.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.212+0000 7f16477fe700 1 -- 172.21.15.151:0/3011542564 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f164008ec20 con 0x7f16500ffbd0 2024-04-16T02:47:49.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.212+0000 7f1655ba9700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f163c06ecc0 0x7f163c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:49.216 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.212+0000 7f1655ba9700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f163c06ecc0 0x7f163c071180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f164c00b5c0 tx=0x7f164c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:49.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.216+0000 7f16477fe700 1 -- 172.21.15.151:0/3011542564 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1640057a50 con 0x7f16500ffbd0 2024-04-16T02:47:49.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.433+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1634000bc0 con 0x7f163c06ecc0 2024-04-16T02:47:49.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.434+0000 7f16477fe700 1 -- 172.21.15.151:0/3011542564 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1634000bc0 con 0x7f163c06ecc0 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:47:49.438 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:47:49.439 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:47:49.439 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:47:49.439 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:47:49.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f163c06ecc0 msgr2=0x7f163c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f163c06ecc0 0x7f163c071180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f164c00b5c0 tx=0x7f164c01a040 comp rx=0 tx=0).stop 2024-04-16T02:47:49.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 msgr2=0x7f1650101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 0x7f1650101610 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f164000d8d0 tx=0x7f164000dbe0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 shutdown_connections 2024-04-16T02:47:49.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f163c06ecc0 0x7f163c071180 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16500ffbd0 0x7f1650101610 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 --2- 172.21.15.151:0/3011542564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1650105a40 0x7f1650101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 >> 172.21.15.151:0/3011542564 conn(0x7f16500755a0 msgr2=0x7f1650068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:49.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 shutdown_connections 2024-04-16T02:47:49.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.437+0000 7f165860e700 1 -- 172.21.15.151:0/3011542564 wait complete. 2024-04-16T02:47:49.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.542+0000 7f310caa1700 1 -- 172.21.15.151:0/388981949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 msgr2=0x7f31081057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.542+0000 7f310caa1700 1 --2- 172.21.15.151:0/388981949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 0x7f31081057c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009910 tx=0x7f30fc009c20 comp rx=0 tx=0).stop 2024-04-16T02:47:49.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.542+0000 7f310caa1700 1 -- 172.21.15.151:0/388981949 shutdown_connections 2024-04-16T02:47:49.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.542+0000 7f310caa1700 1 --2- 172.21.15.151:0/388981949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 0x7f31081057c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.542+0000 7f310caa1700 1 --2- 172.21.15.151:0/388981949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3108100dc0 0x7f31081011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.542+0000 7f310caa1700 1 -- 172.21.15.151:0/388981949 >> 172.21.15.151:0/388981949 conn(0x7f31080fc650 msgr2=0x7f31080fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:49.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.542+0000 7f310caa1700 1 -- 172.21.15.151:0/388981949 shutdown_connections 2024-04-16T02:47:49.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.542+0000 7f310caa1700 1 -- 172.21.15.151:0/388981949 wait complete. 2024-04-16T02:47:49.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f310caa1700 1 Processor -- start 2024-04-16T02:47:49.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f310caa1700 1 -- start start 2024-04-16T02:47:49.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f310caa1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3108100dc0 0x7f3108073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:49.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f310caa1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 0x7f31080735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f310caa1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3108073b10 con 0x7f3108101770 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f310caa1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3108073c50 con 0x7f3108100dc0 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f3105d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 0x7f31080735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f3105d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 0x7f31080735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53754/0 (socket says 172.21.15.151:53754) 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f3105d9b700 1 -- 172.21.15.151:0/2987120427 learned_addr learned my addr 172.21.15.151:0/2987120427 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f310659c700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3108100dc0 0x7f3108073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f3105d9b700 1 -- 172.21.15.151:0/2987120427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3108100dc0 msgr2=0x7f3108073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f3105d9b700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3108100dc0 0x7f3108073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f3105d9b700 1 -- 172.21.15.151:0/2987120427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc0095f0 con 0x7f3108101770 2024-04-16T02:47:49.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.543+0000 7f310659c700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3108100dc0 0x7f3108073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:47:49.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.544+0000 7f3105d9b700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 0x7f31080735d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00bda0 tx=0x7f30fc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:49.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.544+0000 7f30f77fe700 1 -- 172.21.15.151:0/2987120427 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30fc01d070 con 0x7f3108101770 2024-04-16T02:47:49.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.544+0000 7f30f77fe700 1 -- 172.21.15.151:0/2987120427 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30fc00f460 con 0x7f3108101770 2024-04-16T02:47:49.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.544+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3108073ed0 con 0x7f3108101770 2024-04-16T02:47:49.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.544+0000 7f30f77fe700 1 -- 172.21.15.151:0/2987120427 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30fc017630 con 0x7f3108101770 2024-04-16T02:47:49.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.544+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31081a2c90 con 0x7f3108101770 2024-04-16T02:47:49.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.545+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31081091c0 con 0x7f3108101770 2024-04-16T02:47:49.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.545+0000 7f30f77fe700 1 -- 172.21.15.151:0/2987120427 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f30fc00fa40 con 0x7f3108101770 2024-04-16T02:47:49.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.546+0000 7f30f77fe700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30f006ecc0 0x7f30f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:47:49.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.546+0000 7f30f77fe700 1 -- 172.21.15.151:0/2987120427 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f30fc0902a0 con 0x7f3108101770 2024-04-16T02:47:49.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.546+0000 7f310659c700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30f006ecc0 0x7f30f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:47:49.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.546+0000 7f310659c700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30f006ecc0 0x7f30f0071180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f30f800bc10 tx=0x7f30f800b5f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:47:49.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.550+0000 7f30f77fe700 1 -- 172.21.15.151:0/2987120427 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f30fc059050 con 0x7f3108101770 2024-04-16T02:47:49.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:49 smithi151 ceph-mon[112233]: pgmap v275: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:49.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:49 smithi151 ceph-mon[112233]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:49.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:49 smithi151 ceph-mon[112233]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:49.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:49 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3386715927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:47:49.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:49 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1253444854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:47:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:49 smithi153 ceph-mon[87858]: pgmap v275: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:49.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:49 smithi153 ceph-mon[87858]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:49.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:49 smithi153 ceph-mon[87858]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:49.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:49 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3386715927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:47:49.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:49 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1253444854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:47:49.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.831+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f310804ea90 con 0x7f3108101770 2024-04-16T02:47:49.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.832+0000 7f30f77fe700 1 -- 172.21.15.151:0/2987120427 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f30fc05c710 con 0x7f3108101770 2024-04-16T02:47:49.835 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:47:49.835 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:47:49.835 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:47:49.836 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:47:49.836 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:47:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30f006ecc0 msgr2=0x7f30f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30f006ecc0 0x7f30f0071180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f30f800bc10 tx=0x7f30f800b5f0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 msgr2=0x7f31080735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:47:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 0x7f31080735d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00bda0 tx=0x7f30fc00be80 comp rx=0 tx=0).stop 2024-04-16T02:47:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 shutdown_connections 2024-04-16T02:47:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30f006ecc0 0x7f30f0071180 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3108100dc0 0x7f3108073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 --2- 172.21.15.151:0/2987120427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3108101770 0x7f31080735d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:47:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 >> 172.21.15.151:0/2987120427 conn(0x7f31080fc650 msgr2=0x7f31080fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:47:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 shutdown_connections 2024-04-16T02:47:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:47:49.834+0000 7f310caa1700 1 -- 172.21.15.151:0/2987120427 wait complete. 2024-04-16T02:47:50.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:50 smithi151 ceph-mon[112233]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:50.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:50 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2987120427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:47:50.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:50 smithi153 ceph-mon[87858]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:47:50.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:50 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2987120427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:47:51.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:51 smithi151 ceph-mon[112233]: pgmap v276: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:51 smithi153 ceph-mon[87858]: pgmap v276: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:53.699 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:53 smithi151 ceph-mon[112233]: pgmap v277: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:53.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:53 smithi153 ceph-mon[87858]: pgmap v277: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:54.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:47:54.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:47:55.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:55 smithi151 ceph-mon[112233]: pgmap v278: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:55.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:55 smithi153 ceph-mon[87858]: pgmap v278: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:57.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:57 smithi151 ceph-mon[112233]: pgmap v279: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:57.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:57 smithi153 ceph-mon[87858]: pgmap v279: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:59.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:47:59 smithi151 ceph-mon[112233]: pgmap v280: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:47:59.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:47:59 smithi153 ceph-mon[87858]: pgmap v280: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:01.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:01 smithi151 ceph-mon[112233]: pgmap v281: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:01.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:01 smithi153 ceph-mon[87858]: pgmap v281: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:03.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:03 smithi151 ceph-mon[112233]: pgmap v282: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:03.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:03 smithi153 ceph-mon[87858]: pgmap v282: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:05.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:05 smithi151 ceph-mon[112233]: pgmap v283: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:05 smithi153 ceph-mon[87858]: pgmap v283: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:07.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:07 smithi151 ceph-mon[112233]: pgmap v284: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:07.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:07 smithi153 ceph-mon[87858]: pgmap v284: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:09.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:09 smithi151 ceph-mon[112233]: pgmap v285: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:09.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:48:09.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:09 smithi153 ceph-mon[87858]: pgmap v285: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:09.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:48:11.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:11 smithi151 ceph-mon[112233]: pgmap v286: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:11.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:11 smithi153 ceph-mon[87858]: pgmap v286: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:13.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:13 smithi151 ceph-mon[112233]: pgmap v287: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:13.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:13 smithi153 ceph-mon[87858]: pgmap v287: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:15.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:15 smithi151 ceph-mon[112233]: pgmap v288: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:15 smithi153 ceph-mon[87858]: pgmap v288: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:17.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:17 smithi151 ceph-mon[112233]: pgmap v289: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:17.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:17 smithi153 ceph-mon[87858]: pgmap v289: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:19 smithi151 ceph-mon[112233]: pgmap v290: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:19.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:19 smithi153 ceph-mon[87858]: pgmap v290: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:19.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 -- 172.21.15.151:0/3035449624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 msgr2=0x7f1a580ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:19.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 --2- 172.21.15.151:0/3035449624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 0x7f1a580ffec0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009910 tx=0x7f1a4c009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:19.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 -- 172.21.15.151:0/3035449624 shutdown_connections 2024-04-16T02:48:19.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 --2- 172.21.15.151:0/3035449624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 0x7f1a580ffec0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:19.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 --2- 172.21.15.151:0/3035449624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a58105a40 0x7f1a58105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:19.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 -- 172.21.15.151:0/3035449624 >> 172.21.15.151:0/3035449624 conn(0x7f1a580fb570 msgr2=0x7f1a580fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:19.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 -- 172.21.15.151:0/3035449624 shutdown_connections 2024-04-16T02:48:19.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 -- 172.21.15.151:0/3035449624 wait complete. 2024-04-16T02:48:19.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 Processor -- start 2024-04-16T02:48:19.959 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.954+0000 7f1a5e439700 1 -- start start 2024-04-16T02:48:19.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a5e439700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 0x7f1a580fbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a5e439700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a58105a40 0x7f1a580fc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a5e439700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580fc910 con 0x7f1a580ffa40 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a5e439700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580fca50 con 0x7f1a58105a40 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a57fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 0x7f1a580fbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a57fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 0x7f1a580fbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58236/0 (socket says 172.21.15.151:58236) 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a57fff700 1 -- 172.21.15.151:0/1895146812 learned_addr learned my addr 172.21.15.151:0/1895146812 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a577fe700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a58105a40 0x7f1a580fc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a57fff700 1 -- 172.21.15.151:0/1895146812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a58105a40 msgr2=0x7f1a580fc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a57fff700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a58105a40 0x7f1a580fc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.955+0000 7f1a57fff700 1 -- 172.21.15.151:0/1895146812 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c0095f0 con 0x7f1a580ffa40 2024-04-16T02:48:19.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.956+0000 7f1a57fff700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 0x7f1a580fbe90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800d8d0 tx=0x7f1a4800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:19.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.956+0000 7f1a557fa700 1 -- 172.21.15.151:0/1895146812 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a48009940 con 0x7f1a580ffa40 2024-04-16T02:48:19.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.956+0000 7f1a557fa700 1 -- 172.21.15.151:0/1895146812 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a48010460 con 0x7f1a580ffa40 2024-04-16T02:48:19.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.956+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a581a28e0 con 0x7f1a580ffa40 2024-04-16T02:48:19.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.956+0000 7f1a557fa700 1 -- 172.21.15.151:0/1895146812 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a4800f5f0 con 0x7f1a580ffa40 2024-04-16T02:48:19.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.956+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a581a2cf0 con 0x7f1a580ffa40 2024-04-16T02:48:19.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.957+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a58108fe0 con 0x7f1a580ffa40 2024-04-16T02:48:19.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.957+0000 7f1a557fa700 1 -- 172.21.15.151:0/1895146812 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1a480105d0 con 0x7f1a580ffa40 2024-04-16T02:48:19.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.958+0000 7f1a557fa700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1a4006ecc0 0x7f1a40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:19.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.958+0000 7f1a557fa700 1 -- 172.21.15.151:0/1895146812 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1a4808ee80 con 0x7f1a580ffa40 2024-04-16T02:48:19.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.958+0000 7f1a577fe700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1a4006ecc0 0x7f1a40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:19.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.958+0000 7f1a577fe700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1a4006ecc0 0x7f1a40071180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1a580fd680 tx=0x7f1a4c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:19.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:19.963+0000 7f1a557fa700 1 -- 172.21.15.151:0/1895146812 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1a48057c30 con 0x7f1a580ffa40 2024-04-16T02:48:20.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.182+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a5802ccc0 con 0x7f1a4006ecc0 2024-04-16T02:48:20.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.183+0000 7f1a557fa700 1 -- 172.21.15.151:0/1895146812 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1a5802ccc0 con 0x7f1a4006ecc0 2024-04-16T02:48:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1a4006ecc0 msgr2=0x7f1a40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1a4006ecc0 0x7f1a40071180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1a580fd680 tx=0x7f1a4c01a040 comp rx=0 tx=0).stop 2024-04-16T02:48:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 msgr2=0x7f1a580fbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 0x7f1a580fbe90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800d8d0 tx=0x7f1a4800dc90 comp rx=0 tx=0).stop 2024-04-16T02:48:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 shutdown_connections 2024-04-16T02:48:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1a4006ecc0 0x7f1a40071180 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a580ffa40 0x7f1a580fbe90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 --2- 172.21.15.151:0/1895146812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a58105a40 0x7f1a580fc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 >> 172.21.15.151:0/1895146812 conn(0x7f1a580fb570 msgr2=0x7f1a58069070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:20.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 shutdown_connections 2024-04-16T02:48:20.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.186+0000 7f1a5e439700 1 -- 172.21.15.151:0/1895146812 wait complete. 2024-04-16T02:48:20.199 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:48:20.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 -- 172.21.15.151:0/2887534154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268102620 msgr2=0x7f326810ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 --2- 172.21.15.151:0/2887534154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268102620 0x7f326810ab10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3254009910 tx=0x7f3254009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:20.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 -- 172.21.15.151:0/2887534154 shutdown_connections 2024-04-16T02:48:20.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 --2- 172.21.15.151:0/2887534154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268102620 0x7f326810ab10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 --2- 172.21.15.151:0/2887534154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3268101c70 0x7f3268102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 -- 172.21.15.151:0/2887534154 >> 172.21.15.151:0/2887534154 conn(0x7f32680fb550 msgr2=0x7f32680fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:20.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 -- 172.21.15.151:0/2887534154 shutdown_connections 2024-04-16T02:48:20.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 -- 172.21.15.151:0/2887534154 wait complete. 2024-04-16T02:48:20.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 Processor -- start 2024-04-16T02:48:20.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.293+0000 7f326ed2b700 1 -- start start 2024-04-16T02:48:20.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326ed2b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268101c70 0x7f326819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:20.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326ed2b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3268102620 0x7f326819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:20.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326ed2b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326819d100 con 0x7f3268101c70 2024-04-16T02:48:20.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326ed2b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32681965b0 con 0x7f3268102620 2024-04-16T02:48:20.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326cac7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268101c70 0x7f326819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326cac7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268101c70 0x7f326819c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58260/0 (socket says 172.21.15.151:58260) 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326cac7700 1 -- 172.21.15.151:0/485931465 learned_addr learned my addr 172.21.15.151:0/485931465 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f3267fff700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3268102620 0x7f326819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326cac7700 1 -- 172.21.15.151:0/485931465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3268102620 msgr2=0x7f326819ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326cac7700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3268102620 0x7f326819ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326cac7700 1 -- 172.21.15.151:0/485931465 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32540095f0 con 0x7f3268101c70 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326cac7700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268101c70 0x7f326819c530 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f325c00d8d0 tx=0x7f325c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f3265ffb700 1 -- 172.21.15.151:0/485931465 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f325c009940 con 0x7f3268101c70 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f3265ffb700 1 -- 172.21.15.151:0/485931465 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f325c010460 con 0x7f3268101c70 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3268196890 con 0x7f3268101c70 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.294+0000 7f3265ffb700 1 -- 172.21.15.151:0/485931465 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f325c00f5f0 con 0x7f3268101c70 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.295+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3268196de0 con 0x7f3268101c70 2024-04-16T02:48:20.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.296+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3268108210 con 0x7f3268101c70 2024-04-16T02:48:20.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.296+0000 7f3265ffb700 1 -- 172.21.15.151:0/485931465 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f325c0105d0 con 0x7f3268101c70 2024-04-16T02:48:20.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.297+0000 7f3265ffb700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:20.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.297+0000 7f3267fff700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:20.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.297+0000 7f3265ffb700 1 -- 172.21.15.151:0/485931465 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f325c08ee80 con 0x7f3268101c70 2024-04-16T02:48:20.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.297+0000 7f3267fff700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f32540059d0 tx=0x7f3254005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:20.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.302+0000 7f3265ffb700 1 -- 172.21.15.151:0/485931465 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f325c05b760 con 0x7f3268101c70 2024-04-16T02:48:20.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.519+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3268197080 con 0x7f325806ecc0 2024-04-16T02:48:20.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.521+0000 7f3265ffb700 1 -- 172.21.15.151:0/485931465 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3268197080 con 0x7f325806ecc0 2024-04-16T02:48:20.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 msgr2=0x7f3258071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f32540059d0 tx=0x7f3254005960 comp rx=0 tx=0).stop 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268101c70 msgr2=0x7f326819c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268101c70 0x7f326819c530 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f325c00d8d0 tx=0x7f325c00dc90 comp rx=0 tx=0).stop 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 shutdown_connections 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3268101c70 0x7f326819c530 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 --2- 172.21.15.151:0/485931465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3268102620 0x7f326819ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 >> 172.21.15.151:0/485931465 conn(0x7f32680fb550 msgr2=0x7f3268105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 shutdown_connections 2024-04-16T02:48:20.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.523+0000 7f326ed2b700 1 -- 172.21.15.151:0/485931465 wait complete. 2024-04-16T02:48:20.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.628+0000 7fb240206700 1 -- 172.21.15.151:0/3758048638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100fd0 msgr2=0x7fb2381046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.628+0000 7fb240206700 1 --2- 172.21.15.151:0/3758048638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100fd0 0x7fb2381046e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb234009910 tx=0x7fb234009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:20.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.628+0000 7fb240206700 1 -- 172.21.15.151:0/3758048638 shutdown_connections 2024-04-16T02:48:20.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.628+0000 7fb240206700 1 --2- 172.21.15.151:0/3758048638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100fd0 0x7fb2381046e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.628+0000 7fb240206700 1 --2- 172.21.15.151:0/3758048638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb238100620 0x7fb238100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.628+0000 7fb240206700 1 -- 172.21.15.151:0/3758048638 >> 172.21.15.151:0/3758048638 conn(0x7fb238074d80 msgr2=0x7fb238075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:20.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.628+0000 7fb240206700 1 -- 172.21.15.151:0/3758048638 shutdown_connections 2024-04-16T02:48:20.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.628+0000 7fb240206700 1 -- 172.21.15.151:0/3758048638 wait complete. 2024-04-16T02:48:20.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb240206700 1 Processor -- start 2024-04-16T02:48:20.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb240206700 1 -- start start 2024-04-16T02:48:20.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb240206700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100620 0x7fb23819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:20.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb240206700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb238100fd0 0x7fb23819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:20.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb240206700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23819f2b0 con 0x7fb238100620 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb240206700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb238198760 con 0x7fb238100fd0 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb23dfa2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100620 0x7fb23819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb23dfa2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100620 0x7fb23819e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58276/0 (socket says 172.21.15.151:58276) 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb23dfa2700 1 -- 172.21.15.151:0/2973282538 learned_addr learned my addr 172.21.15.151:0/2973282538 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.629+0000 7fb23d7a1700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb238100fd0 0x7fb23819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.630+0000 7fb23dfa2700 1 -- 172.21.15.151:0/2973282538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb238100fd0 msgr2=0x7fb23819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.630+0000 7fb23dfa2700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb238100fd0 0x7fb23819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.630+0000 7fb23dfa2700 1 -- 172.21.15.151:0/2973282538 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2340095f0 con 0x7fb238100620 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.630+0000 7fb23dfa2700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100620 0x7fb23819e6e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00d8d0 tx=0x7fb22c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.630+0000 7fb22affd700 1 -- 172.21.15.151:0/2973282538 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb22c009940 con 0x7fb238100620 2024-04-16T02:48:20.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.630+0000 7fb22affd700 1 -- 172.21.15.151:0/2973282538 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb22c010460 con 0x7fb238100620 2024-04-16T02:48:20.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.630+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb238198a40 con 0x7fb238100620 2024-04-16T02:48:20.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.630+0000 7fb22affd700 1 -- 172.21.15.151:0/2973282538 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb22c00f5f0 con 0x7fb238100620 2024-04-16T02:48:20.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.630+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb238198f90 con 0x7fb238100620 2024-04-16T02:48:20.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.631+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23804ea90 con 0x7fb238100620 2024-04-16T02:48:20.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.632+0000 7fb22affd700 1 -- 172.21.15.151:0/2973282538 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb22c0105d0 con 0x7fb238100620 2024-04-16T02:48:20.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.632+0000 7fb22affd700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb22406ecc0 0x7fb224071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:20.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.632+0000 7fb22affd700 1 -- 172.21.15.151:0/2973282538 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb22c08ee80 con 0x7fb238100620 2024-04-16T02:48:20.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.632+0000 7fb23d7a1700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb22406ecc0 0x7fb224071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:20.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.633+0000 7fb23d7a1700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb22406ecc0 0x7fb224071180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb238199bd0 tx=0x7fb23401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:20.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.637+0000 7fb22affd700 1 -- 172.21.15.151:0/2973282538 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb22c05b760 con 0x7fb238100620 2024-04-16T02:48:20.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.847+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb238199380 con 0x7fb22406ecc0 2024-04-16T02:48:20.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.857+0000 7fb22affd700 1 -- 172.21.15.151:0/2973282538 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb238199380 con 0x7fb22406ecc0 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (9m) 3m ago 21m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (21m) 3m ago 21m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (20m) 3m ago 20m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (7m) 3m ago 21m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (7m) 3m ago 20m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (9m) 3m ago 20m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (3m) 3m ago 15m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (3m) 3m ago 15m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (15m) 3m ago 15m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (3m) 3m ago 15m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (11m) 3m ago 22m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (10m) 3m ago 20m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (8m) 3m ago 22m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:48:20.861 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (8m) 3m ago 20m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:48:20.862 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (10m) 3m ago 21m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:48:20.862 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (10m) 3m ago 20m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:48:20.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (7m) 3m ago 19m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:48:20.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (6m) 3m ago 18m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:48:20.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (5m) 3m ago 18m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:48:20.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (5m) 3m ago 18m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:48:20.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (4m) 3m ago 17m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:48:20.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (4m) 3m ago 17m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:48:20.862 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (10m) 3m ago 20m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.859+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb22406ecc0 msgr2=0x7fb224071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.859+0000 7fb240206700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb22406ecc0 0x7fb224071180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb238199bd0 tx=0x7fb23401a040 comp rx=0 tx=0).stop 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.859+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100620 msgr2=0x7fb23819e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.859+0000 7fb240206700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100620 0x7fb23819e6e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00d8d0 tx=0x7fb22c00dc90 comp rx=0 tx=0).stop 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.860+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 shutdown_connections 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.860+0000 7fb240206700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb22406ecc0 0x7fb224071180 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.860+0000 7fb240206700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb238100620 0x7fb23819e6e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.860+0000 7fb240206700 1 --2- 172.21.15.151:0/2973282538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb238100fd0 0x7fb23819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.860+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 >> 172.21.15.151:0/2973282538 conn(0x7fb238074d80 msgr2=0x7fb2380fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.860+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 shutdown_connections 2024-04-16T02:48:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.860+0000 7fb240206700 1 -- 172.21.15.151:0/2973282538 wait complete. 2024-04-16T02:48:20.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.963+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/910488408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4100dc0 msgr2=0x7fa2e41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.963+0000 7fa2e8f4f700 1 --2- 172.21.15.151:0/910488408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4100dc0 0x7fa2e41011a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009910 tx=0x7fa2d8009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:20.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.963+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/910488408 shutdown_connections 2024-04-16T02:48:20.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.963+0000 7fa2e8f4f700 1 --2- 172.21.15.151:0/910488408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e4101770 0x7fa2e41057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.963+0000 7fa2e8f4f700 1 --2- 172.21.15.151:0/910488408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4100dc0 0x7fa2e41011a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.963+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/910488408 >> 172.21.15.151:0/910488408 conn(0x7fa2e40fc650 msgr2=0x7fa2e40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:20.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.963+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/910488408 shutdown_connections 2024-04-16T02:48:20.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.963+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/910488408 wait complete. 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.964+0000 7fa2e8f4f700 1 Processor -- start 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.964+0000 7fa2e8f4f700 1 -- start start 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.964+0000 7fa2e8f4f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e4100dc0 0x7fa2e4198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.964+0000 7fa2e8f4f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4101770 0x7fa2e4198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.964+0000 7fa2e8f4f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e4199520 con 0x7fa2e4101770 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.964+0000 7fa2e8f4f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e419d330 con 0x7fa2e4100dc0 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.964+0000 7fa2e1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4101770 0x7fa2e4198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2e1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4101770 0x7fa2e4198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58300/0 (socket says 172.21.15.151:58300) 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2e1d9b700 1 -- 172.21.15.151:0/2182765126 learned_addr learned my addr 172.21.15.151:0/2182765126 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:20.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2e259c700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e4100dc0 0x7fa2e4198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:20.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2e1d9b700 1 -- 172.21.15.151:0/2182765126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e4100dc0 msgr2=0x7fa2e4198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:20.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2e1d9b700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e4100dc0 0x7fa2e4198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:20.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2e1d9b700 1 -- 172.21.15.151:0/2182765126 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d80095f0 con 0x7fa2e4101770 2024-04-16T02:48:20.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2e1d9b700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4101770 0x7fa2e4198e90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400c8f0 tx=0x7fa2d400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:20.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2d37fe700 1 -- 172.21.15.151:0/2182765126 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d40078a0 con 0x7fa2e4101770 2024-04-16T02:48:20.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2d37fe700 1 -- 172.21.15.151:0/2182765126 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2d400f450 con 0x7fa2e4101770 2024-04-16T02:48:20.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2d37fe700 1 -- 172.21.15.151:0/2182765126 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d4018610 con 0x7fa2e4101770 2024-04-16T02:48:20.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e419d610 con 0x7fa2e4101770 2024-04-16T02:48:20.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.965+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e419db60 con 0x7fa2e4101770 2024-04-16T02:48:20.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.966+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4104f40 con 0x7fa2e4101770 2024-04-16T02:48:20.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.967+0000 7fa2d37fe700 1 -- 172.21.15.151:0/2182765126 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa2d400f5c0 con 0x7fa2e4101770 2024-04-16T02:48:20.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.967+0000 7fa2d37fe700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2cc06ecc0 0x7fa2cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:20.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.967+0000 7fa2d37fe700 1 -- 172.21.15.151:0/2182765126 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa2d408edd0 con 0x7fa2e4101770 2024-04-16T02:48:20.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.967+0000 7fa2e259c700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2cc06ecc0 0x7fa2cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:20.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.968+0000 7fa2e259c700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2cc06ecc0 0x7fa2cc071180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8005f50 tx=0x7fa2d801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:20.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:20.972+0000 7fa2d37fe700 1 -- 172.21.15.151:0/2182765126 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa2d405b6b0 con 0x7fa2e4101770 2024-04-16T02:48:21.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.253+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa2e404ea90 con 0x7fa2e4101770 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.254+0000 7fa2d37fe700 1 -- 172.21.15.151:0/2182765126 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fa2d405b240 con 0x7fa2e4101770 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:48:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:48:21.258 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:48:21.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2cc06ecc0 msgr2=0x7fa2cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2cc06ecc0 0x7fa2cc071180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8005f50 tx=0x7fa2d801a040 comp rx=0 tx=0).stop 2024-04-16T02:48:21.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4101770 msgr2=0x7fa2e4198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4101770 0x7fa2e4198e90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400c8f0 tx=0x7fa2d400cc00 comp rx=0 tx=0).stop 2024-04-16T02:48:21.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 shutdown_connections 2024-04-16T02:48:21.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa2cc06ecc0 0x7fa2cc071180 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e4100dc0 0x7fa2e4198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 --2- 172.21.15.151:0/2182765126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2e4101770 0x7fa2e4198e90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 >> 172.21.15.151:0/2182765126 conn(0x7fa2e40fc650 msgr2=0x7fa2e40fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:21.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 shutdown_connections 2024-04-16T02:48:21.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.256+0000 7fa2e8f4f700 1 -- 172.21.15.151:0/2182765126 wait complete. 2024-04-16T02:48:21.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.362+0000 7f3ce15a4700 1 -- 172.21.15.151:0/715547998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 msgr2=0x7f3cdc073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.362+0000 7f3ce15a4700 1 --2- 172.21.15.151:0/715547998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 0x7f3cdc073490 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00b0a0 tx=0x7f3ccc00b3b0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.362+0000 7f3ce15a4700 1 -- 172.21.15.151:0/715547998 shutdown_connections 2024-04-16T02:48:21.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.362+0000 7f3ce15a4700 1 --2- 172.21.15.151:0/715547998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cdc0739d0 0x7f3cdc10cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.362+0000 7f3ce15a4700 1 --2- 172.21.15.151:0/715547998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 0x7f3cdc073490 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.362+0000 7f3ce15a4700 1 -- 172.21.15.151:0/715547998 >> 172.21.15.151:0/715547998 conn(0x7f3cdc0fc650 msgr2=0x7f3cdc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:21.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.362+0000 7f3ce15a4700 1 -- 172.21.15.151:0/715547998 shutdown_connections 2024-04-16T02:48:21.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.362+0000 7f3ce15a4700 1 -- 172.21.15.151:0/715547998 wait complete. 2024-04-16T02:48:21.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.363+0000 7f3ce15a4700 1 Processor -- start 2024-04-16T02:48:21.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.363+0000 7f3ce15a4700 1 -- start start 2024-04-16T02:48:21.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.363+0000 7f3ce15a4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 0x7f3cdc198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:21.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.363+0000 7f3ce15a4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cdc0739d0 0x7f3cdc198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:21.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.363+0000 7f3ce15a4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc199530 con 0x7f3cdc0730b0 2024-04-16T02:48:21.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.363+0000 7f3ce15a4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc19d340 con 0x7f3cdc0739d0 2024-04-16T02:48:21.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.363+0000 7f3cdaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 0x7f3cdc198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:21.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.363+0000 7f3cdaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 0x7f3cdc198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58310/0 (socket says 172.21.15.151:58310) 2024-04-16T02:48:21.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cdaffd700 1 -- 172.21.15.151:0/720187105 learned_addr learned my addr 172.21.15.151:0/720187105 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cda7fc700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cdc0739d0 0x7f3cdc198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cdaffd700 1 -- 172.21.15.151:0/720187105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cdc0739d0 msgr2=0x7f3cdc198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cdaffd700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cdc0739d0 0x7f3cdc198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cdaffd700 1 -- 172.21.15.151:0/720187105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc009d00 con 0x7f3cdc0730b0 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cda7fc700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cdc0739d0 0x7f3cdc198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cdaffd700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 0x7f3cdc198960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc000f80 tx=0x7f3ccc003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cd3fff700 1 -- 172.21.15.151:0/720187105 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ccc00e040 con 0x7f3cdc0730b0 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cdc19d5c0 con 0x7f3cdc0730b0 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cd3fff700 1 -- 172.21.15.151:0/720187105 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ccc007c00 con 0x7f3cdc0730b0 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3cd3fff700 1 -- 172.21.15.151:0/720187105 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ccc01d6b0 con 0x7f3cdc0730b0 2024-04-16T02:48:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.364+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc19dab0 con 0x7f3cdc0730b0 2024-04-16T02:48:21.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.365+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cdc10a5e0 con 0x7f3cdc0730b0 2024-04-16T02:48:21.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.366+0000 7f3cd3fff700 1 -- 172.21.15.151:0/720187105 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3ccc004040 con 0x7f3cdc0730b0 2024-04-16T02:48:21.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.366+0000 7f3cd3fff700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3cc806ecc0 0x7f3cc8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:21.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.366+0000 7f3cd3fff700 1 -- 172.21.15.151:0/720187105 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3ccc090560 con 0x7f3cdc0730b0 2024-04-16T02:48:21.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.366+0000 7f3cda7fc700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3cc806ecc0 0x7f3cc8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:21.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.367+0000 7f3cda7fc700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3cc806ecc0 0x7f3cc8071180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4005950 tx=0x7f3cc40058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:21.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.371+0000 7f3cd3fff700 1 -- 172.21.15.151:0/720187105 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3ccc05ce40 con 0x7f3cdc0730b0 2024-04-16T02:48:21.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.613+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3cdc04ea90 con 0x7f3cdc0730b0 2024-04-16T02:48:21.617 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:21 smithi151 ceph-mon[112233]: pgmap v291: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:21.617 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:21 smithi151 ceph-mon[112233]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:21.617 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:21 smithi151 ceph-mon[112233]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:21.617 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:21 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2182765126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:48:21.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.614+0000 7f3cd3fff700 1 -- 172.21.15.151:0/720187105 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f3ccc05c9d0 con 0x7f3cdc0730b0 2024-04-16T02:48:21.617 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:48:21.618 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:48:21.619 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:48:21.620 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:48:21.621 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:48:21.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.616+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3cc806ecc0 msgr2=0x7f3cc8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.616+0000 7f3ce15a4700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3cc806ecc0 0x7f3cc8071180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4005950 tx=0x7f3cc40058e0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.616+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 msgr2=0x7f3cdc198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.616+0000 7f3ce15a4700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 0x7f3cdc198960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc000f80 tx=0x7f3ccc003ce0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.616+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 shutdown_connections 2024-04-16T02:48:21.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.616+0000 7f3ce15a4700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3cc806ecc0 0x7f3cc8071180 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.616+0000 7f3ce15a4700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cdc0730b0 0x7f3cdc198960 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.616+0000 7f3ce15a4700 1 --2- 172.21.15.151:0/720187105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cdc0739d0 0x7f3cdc198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.616+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 >> 172.21.15.151:0/720187105 conn(0x7f3cdc0fc650 msgr2=0x7f3cdc107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:21.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.617+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 shutdown_connections 2024-04-16T02:48:21.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.617+0000 7f3ce15a4700 1 -- 172.21.15.151:0/720187105 wait complete. 2024-04-16T02:48:21.622 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:48:21.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.723+0000 7f5f08dea700 1 -- 172.21.15.151:0/952846203 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 msgr2=0x7f5f04105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.723+0000 7f5f08dea700 1 --2- 172.21.15.151:0/952846203 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 0x7f5f04105840 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009910 tx=0x7f5ef8009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:21.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.723+0000 7f5f08dea700 1 -- 172.21.15.151:0/952846203 shutdown_connections 2024-04-16T02:48:21.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.723+0000 7f5f08dea700 1 --2- 172.21.15.151:0/952846203 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 0x7f5f04105840 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.723+0000 7f5f08dea700 1 --2- 172.21.15.151:0/952846203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f040ff500 0x7f5f040ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.723+0000 7f5f08dea700 1 -- 172.21.15.151:0/952846203 >> 172.21.15.151:0/952846203 conn(0x7f5f04074d80 msgr2=0x7f5f04075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:21.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.723+0000 7f5f08dea700 1 -- 172.21.15.151:0/952846203 shutdown_connections 2024-04-16T02:48:21.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.723+0000 7f5f08dea700 1 -- 172.21.15.151:0/952846203 wait complete. 2024-04-16T02:48:21.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.724+0000 7f5f08dea700 1 Processor -- start 2024-04-16T02:48:21.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.724+0000 7f5f08dea700 1 -- start start 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.724+0000 7f5f08dea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f040ff500 0x7f5f04100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.724+0000 7f5f08dea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 0x7f5f041009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.724+0000 7f5f08dea700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f04100f00 con 0x7f5f040ffeb0 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.724+0000 7f5f08dea700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f04101040 con 0x7f5f040ff500 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.724+0000 7f5f01d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 0x7f5f041009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.724+0000 7f5f01d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 0x7f5f041009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58336/0 (socket says 172.21.15.151:58336) 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.724+0000 7f5f01d9b700 1 -- 172.21.15.151:0/901258378 learned_addr learned my addr 172.21.15.151:0/901258378 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5f0259c700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f040ff500 0x7f5f04100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5f01d9b700 1 -- 172.21.15.151:0/901258378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f040ff500 msgr2=0x7f5f04100480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5f01d9b700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f040ff500 0x7f5f04100480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5f01d9b700 1 -- 172.21.15.151:0/901258378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef80095f0 con 0x7f5f040ffeb0 2024-04-16T02:48:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5f01d9b700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 0x7f5f041009c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80092e0 tx=0x7f5ef800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5ef37fe700 1 -- 172.21.15.151:0/901258378 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ef801d070 con 0x7f5f040ffeb0 2024-04-16T02:48:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f041012c0 con 0x7f5f040ffeb0 2024-04-16T02:48:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5ef37fe700 1 -- 172.21.15.151:0/901258378 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ef800f460 con 0x7f5f040ffeb0 2024-04-16T02:48:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5ef37fe700 1 -- 172.21.15.151:0/901258378 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ef80175f0 con 0x7f5f040ffeb0 2024-04-16T02:48:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.725+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f041a2d90 con 0x7f5f040ffeb0 2024-04-16T02:48:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.726+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f041091b0 con 0x7f5f040ffeb0 2024-04-16T02:48:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.727+0000 7f5ef37fe700 1 -- 172.21.15.151:0/901258378 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5ef800f5d0 con 0x7f5f040ffeb0 2024-04-16T02:48:21.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.727+0000 7f5ef37fe700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5eec06ecc0 0x7f5eec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:21.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.727+0000 7f5f0259c700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5eec06ecc0 0x7f5eec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:21.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.727+0000 7f5ef37fe700 1 -- 172.21.15.151:0/901258378 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5ef8090390 con 0x7f5f040ffeb0 2024-04-16T02:48:21.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.728+0000 7f5f0259c700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5eec06ecc0 0x7f5eec071180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4006fd0 tx=0x7f5ef4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:21.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.732+0000 7f5ef37fe700 1 -- 172.21.15.151:0/901258378 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5ef8059140 con 0x7f5f040ffeb0 2024-04-16T02:48:21.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:21 smithi153 ceph-mon[87858]: pgmap v291: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:21.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:21 smithi153 ceph-mon[87858]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:21.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:21 smithi153 ceph-mon[87858]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:21.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:21 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2182765126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:48:21.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.954+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f040611d0 con 0x7f5eec06ecc0 2024-04-16T02:48:21.959 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.956+0000 7f5ef37fe700 1 -- 172.21.15.151:0/901258378 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5f040611d0 con 0x7f5eec06ecc0 2024-04-16T02:48:21.959 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:48:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:48:21.960 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:48:21.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5eec06ecc0 msgr2=0x7f5eec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5eec06ecc0 0x7f5eec071180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4006fd0 tx=0x7f5ef4008040 comp rx=0 tx=0).stop 2024-04-16T02:48:21.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 msgr2=0x7f5f041009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:21.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 0x7f5f041009c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80092e0 tx=0x7f5ef800be20 comp rx=0 tx=0).stop 2024-04-16T02:48:21.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 shutdown_connections 2024-04-16T02:48:21.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5eec06ecc0 0x7f5eec071180 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f040ff500 0x7f5f04100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 --2- 172.21.15.151:0/901258378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f040ffeb0 0x7f5f041009c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:21.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 >> 172.21.15.151:0/901258378 conn(0x7f5f04074d80 msgr2=0x7f5f040fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:21.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 shutdown_connections 2024-04-16T02:48:21.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:21.959+0000 7f5f08dea700 1 -- 172.21.15.151:0/901258378 wait complete. 2024-04-16T02:48:22.069 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.066+0000 7fa3e2676700 1 -- 172.21.15.151:0/162260121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 msgr2=0x7fa3dc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.066+0000 7fa3e2676700 1 --2- 172.21.15.151:0/162260121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 0x7fa3dc100050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009910 tx=0x7fa3cc009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.066+0000 7fa3e2676700 1 -- 172.21.15.151:0/162260121 shutdown_connections 2024-04-16T02:48:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.066+0000 7fa3e2676700 1 --2- 172.21.15.151:0/162260121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 0x7fa3dc100050 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.066+0000 7fa3e2676700 1 --2- 172.21.15.151:0/162260121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3dc105a40 0x7fa3dc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.066+0000 7fa3e2676700 1 -- 172.21.15.151:0/162260121 >> 172.21.15.151:0/162260121 conn(0x7fa3dc0755a0 msgr2=0x7fa3dc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.066+0000 7fa3e2676700 1 -- 172.21.15.151:0/162260121 shutdown_connections 2024-04-16T02:48:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.066+0000 7fa3e2676700 1 -- 172.21.15.151:0/162260121 wait complete. 2024-04-16T02:48:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3e2676700 1 Processor -- start 2024-04-16T02:48:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3e2676700 1 -- start start 2024-04-16T02:48:22.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3e2676700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 0x7fa3dc197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:22.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3e2676700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3dc105a40 0x7fa3dc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:22.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3e2676700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc1981b0 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3e2676700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc198320 con 0x7fa3dc105a40 2024-04-16T02:48:22.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3dbfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 0x7fa3dc197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:22.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3dbfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 0x7fa3dc197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58348/0 (socket says 172.21.15.151:58348) 2024-04-16T02:48:22.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3dbfff700 1 -- 172.21.15.151:0/444877390 learned_addr learned my addr 172.21.15.151:0/444877390 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:22.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.067+0000 7fa3db7fe700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3dc105a40 0x7fa3dc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.068+0000 7fa3dbfff700 1 -- 172.21.15.151:0/444877390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3dc105a40 msgr2=0x7fa3dc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.068+0000 7fa3dbfff700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3dc105a40 0x7fa3dc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.068+0000 7fa3dbfff700 1 -- 172.21.15.151:0/444877390 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0095f0 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.068+0000 7fa3dbfff700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 0x7fa3dc197590 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400d8d0 tx=0x7fa3c400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.068+0000 7fa3d97fa700 1 -- 172.21.15.151:0/444877390 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3c40098a0 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.068+0000 7fa3d97fa700 1 -- 172.21.15.151:0/444877390 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3c4010460 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.068+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc1a0600 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.068+0000 7fa3d97fa700 1 -- 172.21.15.151:0/444877390 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3c400f5f0 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.068+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc1a0b50 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.069+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc107fd0 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.069+0000 7fa3d97fa700 1 -- 172.21.15.151:0/444877390 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa3c4009a00 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.070+0000 7fa3d97fa700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3c806ecc0 0x7fa3c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:22.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.070+0000 7fa3d97fa700 1 -- 172.21.15.151:0/444877390 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa3c408eda0 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.070+0000 7fa3db7fe700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3c806ecc0 0x7fa3c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:22.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.070+0000 7fa3db7fe700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3c806ecc0 0x7fa3c8071180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc068cc0 tx=0x7fa3cc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:22.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.075+0000 7fa3d97fa700 1 -- 172.21.15.151:0/444877390 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa3c4057b50 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.354+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa3dc04ea90 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.355+0000 7fa3d97fa700 1 -- 172.21.15.151:0/444877390 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa3c405b210 con 0x7fa3dc0ffbd0 2024-04-16T02:48:22.358 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:48:22.358 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:48:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:48:22.358 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:48:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:48:22.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3c806ecc0 msgr2=0x7fa3c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:22.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3c806ecc0 0x7fa3c8071180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc068cc0 tx=0x7fa3cc01a040 comp rx=0 tx=0).stop 2024-04-16T02:48:22.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 msgr2=0x7fa3dc197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:22.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 0x7fa3dc197590 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400d8d0 tx=0x7fa3c400dbe0 comp rx=0 tx=0).stop 2024-04-16T02:48:22.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 shutdown_connections 2024-04-16T02:48:22.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3c806ecc0 0x7fa3c8071180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:22.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3dc0ffbd0 0x7fa3dc197590 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:22.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 --2- 172.21.15.151:0/444877390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3dc105a40 0x7fa3dc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:22.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 >> 172.21.15.151:0/444877390 conn(0x7fa3dc0755a0 msgr2=0x7fa3dc0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:22.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 shutdown_connections 2024-04-16T02:48:22.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:22.357+0000 7fa3e2676700 1 -- 172.21.15.151:0/444877390 wait complete. 2024-04-16T02:48:22.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:22 smithi151 ceph-mon[112233]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:22.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:22 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/720187105' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:48:22.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:22 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/444877390' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:48:22.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:22 smithi153 ceph-mon[87858]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:22.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:22 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/720187105' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:48:22.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:22 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/444877390' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:48:23.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:23 smithi151 ceph-mon[112233]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:23.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:23 smithi151 ceph-mon[112233]: pgmap v292: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:23.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:23 smithi153 ceph-mon[87858]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:23.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:23 smithi153 ceph-mon[87858]: pgmap v292: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:24.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:48:24.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:48:25.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:25 smithi151 ceph-mon[112233]: pgmap v293: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:25 smithi153 ceph-mon[87858]: pgmap v293: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:27 smithi151 ceph-mon[112233]: pgmap v294: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:27 smithi153 ceph-mon[87858]: pgmap v294: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:29.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:29 smithi151 ceph-mon[112233]: pgmap v295: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:29.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:29 smithi153 ceph-mon[87858]: pgmap v295: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:31.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:31 smithi151 ceph-mon[112233]: pgmap v296: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:31.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:31 smithi153 ceph-mon[87858]: pgmap v296: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:33.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:33 smithi151 ceph-mon[112233]: pgmap v297: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:33.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:33 smithi153 ceph-mon[87858]: pgmap v297: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:35.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:35 smithi151 ceph-mon[112233]: pgmap v298: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:35 smithi153 ceph-mon[87858]: pgmap v298: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:37.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:37 smithi151 ceph-mon[112233]: pgmap v299: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:37.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:37 smithi153 ceph-mon[87858]: pgmap v299: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:39.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:39 smithi151 ceph-mon[112233]: pgmap v300: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:39.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:48:39.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:39 smithi153 ceph-mon[87858]: pgmap v300: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:39.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:48:41.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:41 smithi151 ceph-mon[112233]: pgmap v301: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:41 smithi153 ceph-mon[87858]: pgmap v301: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:43.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:43 smithi151 ceph-mon[112233]: pgmap v302: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:43.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:43 smithi153 ceph-mon[87858]: pgmap v302: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:45.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:45 smithi151 ceph-mon[112233]: pgmap v303: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:45.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:45 smithi153 ceph-mon[87858]: pgmap v303: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:47.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:47 smithi151 ceph-mon[112233]: pgmap v304: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:47.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:47 smithi153 ceph-mon[87858]: pgmap v304: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:49.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:49 smithi151 ceph-mon[112233]: pgmap v305: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:49 smithi153 ceph-mon[87858]: pgmap v305: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:51.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:51 smithi151 ceph-mon[112233]: pgmap v306: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:51 smithi153 ceph-mon[87858]: pgmap v306: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:52.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.475+0000 7f0358569700 1 -- 172.21.15.151:0/788966331 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0350105a50 msgr2=0x7f0350105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:52.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.475+0000 7f0358569700 1 --2- 172.21.15.151:0/788966331 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0350105a50 0x7f0350105e30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f034c009910 tx=0x7f034c009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:52.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.475+0000 7f0358569700 1 -- 172.21.15.151:0/788966331 shutdown_connections 2024-04-16T02:48:52.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.475+0000 7f0358569700 1 --2- 172.21.15.151:0/788966331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03500ffa50 0x7f03500ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.475+0000 7f0358569700 1 --2- 172.21.15.151:0/788966331 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0350105a50 0x7f0350105e30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.475+0000 7f0358569700 1 -- 172.21.15.151:0/788966331 >> 172.21.15.151:0/788966331 conn(0x7f03500fb540 msgr2=0x7f03500fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.475+0000 7f0358569700 1 -- 172.21.15.151:0/788966331 shutdown_connections 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.475+0000 7f0358569700 1 -- 172.21.15.151:0/788966331 wait complete. 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0358569700 1 Processor -- start 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0358569700 1 -- start start 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0358569700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03500ffa50 0x7f03501953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0358569700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0350105a50 0x7f0350195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0358569700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0350195e60 con 0x7f03500ffa50 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0358569700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0350195fd0 con 0x7f0350105a50 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0356305700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03500ffa50 0x7f03501953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0356305700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03500ffa50 0x7f03501953e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49392/0 (socket says 172.21.15.151:49392) 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0356305700 1 -- 172.21.15.151:0/2966709281 learned_addr learned my addr 172.21.15.151:0/2966709281 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.476+0000 7f0355b04700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0350105a50 0x7f0350195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.477+0000 7f0356305700 1 -- 172.21.15.151:0/2966709281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0350105a50 msgr2=0x7f0350195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.477+0000 7f0356305700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0350105a50 0x7f0350195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.477+0000 7f0356305700 1 -- 172.21.15.151:0/2966709281 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c0095f0 con 0x7f03500ffa50 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.477+0000 7f0356305700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03500ffa50 0x7f03501953e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f034c0092e0 tx=0x7f034c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.477+0000 7f03477fe700 1 -- 172.21.15.151:0/2966709281 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f034c01d070 con 0x7f03500ffa50 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.477+0000 7f03477fe700 1 -- 172.21.15.151:0/2966709281 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f034c005e00 con 0x7f03500ffa50 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.477+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0350196250 con 0x7f03500ffa50 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.477+0000 7f03477fe700 1 -- 172.21.15.151:0/2966709281 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f034c017400 con 0x7f03500ffa50 2024-04-16T02:48:52.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.477+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03501a2cd0 con 0x7f03500ffa50 2024-04-16T02:48:52.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.478+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0350108050 con 0x7f03500ffa50 2024-04-16T02:48:52.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.479+0000 7f03477fe700 1 -- 172.21.15.151:0/2966709281 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f034c00f460 con 0x7f03500ffa50 2024-04-16T02:48:52.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.479+0000 7f03477fe700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f033c06ecc0 0x7f033c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:52.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.479+0000 7f03477fe700 1 -- 172.21.15.151:0/2966709281 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f034c0900f0 con 0x7f03500ffa50 2024-04-16T02:48:52.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.479+0000 7f0355b04700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f033c06ecc0 0x7f033c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:52.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.480+0000 7f0355b04700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f033c06ecc0 0x7f033c071180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0340005950 tx=0x7f03400058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:52.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.484+0000 7f03477fe700 1 -- 172.21.15.151:0/2966709281 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f034c05ca50 con 0x7f03500ffa50 2024-04-16T02:48:52.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.700+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0350196250 con 0x7f033c06ecc0 2024-04-16T02:48:52.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.701+0000 7f03477fe700 1 -- 172.21.15.151:0/2966709281 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0350196250 con 0x7f033c06ecc0 2024-04-16T02:48:52.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f033c06ecc0 msgr2=0x7f033c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:52.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f033c06ecc0 0x7f033c071180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0340005950 tx=0x7f03400058e0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03500ffa50 msgr2=0x7f03501953e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:52.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03500ffa50 0x7f03501953e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f034c0092e0 tx=0x7f034c00be20 comp rx=0 tx=0).stop 2024-04-16T02:48:52.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 shutdown_connections 2024-04-16T02:48:52.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f033c06ecc0 0x7f033c071180 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03500ffa50 0x7f03501953e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 --2- 172.21.15.151:0/2966709281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0350105a50 0x7f0350195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 >> 172.21.15.151:0/2966709281 conn(0x7f03500fb540 msgr2=0x7f03500fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:52.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 shutdown_connections 2024-04-16T02:48:52.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.704+0000 7f0358569700 1 -- 172.21.15.151:0/2966709281 wait complete. 2024-04-16T02:48:52.718 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:48:52.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.809+0000 7f330fec9700 1 -- 172.21.15.151:0/349024990 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 msgr2=0x7f33081057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:52.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.809+0000 7f330fec9700 1 --2- 172.21.15.151:0/349024990 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 0x7f33081057c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3304009910 tx=0x7f3304009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:52.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.809+0000 7f330fec9700 1 -- 172.21.15.151:0/349024990 shutdown_connections 2024-04-16T02:48:52.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.809+0000 7f330fec9700 1 --2- 172.21.15.151:0/349024990 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 0x7f33081057c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.809+0000 7f330fec9700 1 --2- 172.21.15.151:0/349024990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3308100dc0 0x7f33081011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.809+0000 7f330fec9700 1 -- 172.21.15.151:0/349024990 >> 172.21.15.151:0/349024990 conn(0x7f33080fc650 msgr2=0x7f33080fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:52.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.809+0000 7f330fec9700 1 -- 172.21.15.151:0/349024990 shutdown_connections 2024-04-16T02:48:52.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.809+0000 7f330fec9700 1 -- 172.21.15.151:0/349024990 wait complete. 2024-04-16T02:48:52.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330fec9700 1 Processor -- start 2024-04-16T02:48:52.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330fec9700 1 -- start start 2024-04-16T02:48:52.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330fec9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3308100dc0 0x7f330819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:52.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330fec9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 0x7f330819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:52.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330fec9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330819d130 con 0x7f3308101770 2024-04-16T02:48:52.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330fec9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33081965e0 con 0x7f3308100dc0 2024-04-16T02:48:52.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330d464700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 0x7f330819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330d464700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 0x7f330819caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49402/0 (socket says 172.21.15.151:49402) 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330d464700 1 -- 172.21.15.151:0/478073563 learned_addr learned my addr 172.21.15.151:0/478073563 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.810+0000 7f330dc65700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3308100dc0 0x7f330819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.811+0000 7f330d464700 1 -- 172.21.15.151:0/478073563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3308100dc0 msgr2=0x7f330819c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.811+0000 7f330d464700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3308100dc0 0x7f330819c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.811+0000 7f330d464700 1 -- 172.21.15.151:0/478073563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33040095f0 con 0x7f3308101770 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.811+0000 7f330d464700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 0x7f330819caa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f33040092e0 tx=0x7f330400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.811+0000 7f32feffd700 1 -- 172.21.15.151:0/478073563 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f330401d070 con 0x7f3308101770 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.811+0000 7f32feffd700 1 -- 172.21.15.151:0/478073563 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f330400f460 con 0x7f3308101770 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.811+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3308196860 con 0x7f3308101770 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.811+0000 7f32feffd700 1 -- 172.21.15.151:0/478073563 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33040175d0 con 0x7f3308101770 2024-04-16T02:48:52.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.811+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3308196d50 con 0x7f3308101770 2024-04-16T02:48:52.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.812+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33081091c0 con 0x7f3308101770 2024-04-16T02:48:52.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.813+0000 7f32feffd700 1 -- 172.21.15.151:0/478073563 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f330400f5d0 con 0x7f3308101770 2024-04-16T02:48:52.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.813+0000 7f32feffd700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32f406ecc0 0x7f32f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:52.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.813+0000 7f32feffd700 1 -- 172.21.15.151:0/478073563 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f33040902c0 con 0x7f3308101770 2024-04-16T02:48:52.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.813+0000 7f330dc65700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32f406ecc0 0x7f32f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:52.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.814+0000 7f330dc65700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32f406ecc0 0x7f32f4071180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f32f8005950 tx=0x7f32f800a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:52.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:52.818+0000 7f32feffd700 1 -- 172.21.15.151:0/478073563 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f330405cba0 con 0x7f3308101770 2024-04-16T02:48:53.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.037+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3308197060 con 0x7f32f406ecc0 2024-04-16T02:48:53.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.038+0000 7f32feffd700 1 -- 172.21.15.151:0/478073563 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3308197060 con 0x7f32f406ecc0 2024-04-16T02:48:53.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32f406ecc0 msgr2=0x7f32f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32f406ecc0 0x7f32f4071180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f32f8005950 tx=0x7f32f800a300 comp rx=0 tx=0).stop 2024-04-16T02:48:53.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 msgr2=0x7f330819caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 0x7f330819caa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f33040092e0 tx=0x7f330400bd70 comp rx=0 tx=0).stop 2024-04-16T02:48:53.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 shutdown_connections 2024-04-16T02:48:53.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32f406ecc0 0x7f32f4071180 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3308100dc0 0x7f330819c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 --2- 172.21.15.151:0/478073563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3308101770 0x7f330819caa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 >> 172.21.15.151:0/478073563 conn(0x7f33080fc650 msgr2=0x7f33080fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:53.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 shutdown_connections 2024-04-16T02:48:53.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.041+0000 7f330fec9700 1 -- 172.21.15.151:0/478073563 wait complete. 2024-04-16T02:48:53.148 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.145+0000 7f8da5d04700 1 -- 172.21.15.151:0/2853738916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da00ffbd0 msgr2=0x7f8da0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.148 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.145+0000 7f8da5d04700 1 --2- 172.21.15.151:0/2853738916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da00ffbd0 0x7f8da0100050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009910 tx=0x7f8d90009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:53.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.145+0000 7f8da5d04700 1 -- 172.21.15.151:0/2853738916 shutdown_connections 2024-04-16T02:48:53.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.145+0000 7f8da5d04700 1 --2- 172.21.15.151:0/2853738916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da00ffbd0 0x7f8da0100050 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.145+0000 7f8da5d04700 1 --2- 172.21.15.151:0/2853738916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8da0105a40 0x7f8da0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.145+0000 7f8da5d04700 1 -- 172.21.15.151:0/2853738916 >> 172.21.15.151:0/2853738916 conn(0x7f8da00755a0 msgr2=0x7f8da00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:53.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.145+0000 7f8da5d04700 1 -- 172.21.15.151:0/2853738916 shutdown_connections 2024-04-16T02:48:53.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.145+0000 7f8da5d04700 1 -- 172.21.15.151:0/2853738916 wait complete. 2024-04-16T02:48:53.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8da5d04700 1 Processor -- start 2024-04-16T02:48:53.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8da5d04700 1 -- start start 2024-04-16T02:48:53.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8da5d04700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8da00ffbd0 0x7f8da0195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:53.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8da5d04700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da0105a40 0x7f8da0195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:53.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8da5d04700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0196240 con 0x7f8da0105a40 2024-04-16T02:48:53.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8da5d04700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0199fd0 con 0x7f8da00ffbd0 2024-04-16T02:48:53.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8d9effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da0105a40 0x7f8da0195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:53.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8d9effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da0105a40 0x7f8da0195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49420/0 (socket says 172.21.15.151:49420) 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8d9effd700 1 -- 172.21.15.151:0/3677520279 learned_addr learned my addr 172.21.15.151:0/3677520279 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8d9f7fe700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8da00ffbd0 0x7f8da0195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8d9effd700 1 -- 172.21.15.151:0/3677520279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8da00ffbd0 msgr2=0x7f8da0195620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.146+0000 7f8d9effd700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8da00ffbd0 0x7f8da0195620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.147+0000 7f8d9effd700 1 -- 172.21.15.151:0/3677520279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900095f0 con 0x7f8da0105a40 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.147+0000 7f8d9effd700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da0105a40 0x7f8da0195b60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d90006010 tx=0x7f8d9000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.147+0000 7f8d9cff9700 1 -- 172.21.15.151:0/3677520279 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d9001d070 con 0x7f8da0105a40 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.147+0000 7f8d9cff9700 1 -- 172.21.15.151:0/3677520279 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d9000f460 con 0x7f8da0105a40 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.147+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da019a250 con 0x7f8da0105a40 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.147+0000 7f8d9cff9700 1 -- 172.21.15.151:0/3677520279 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d90017610 con 0x7f8da0105a40 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.147+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da019a740 con 0x7f8da0105a40 2024-04-16T02:48:53.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.148+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da0107fd0 con 0x7f8da0105a40 2024-04-16T02:48:53.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.148+0000 7f8d9cff9700 1 -- 172.21.15.151:0/3677520279 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8d9000f5d0 con 0x7f8da0105a40 2024-04-16T02:48:53.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.149+0000 7f8d9cff9700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d8806ecc0 0x7f8d88071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:53.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.149+0000 7f8d9cff9700 1 -- 172.21.15.151:0/3677520279 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8d900903b0 con 0x7f8da0105a40 2024-04-16T02:48:53.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.149+0000 7f8d9f7fe700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d8806ecc0 0x7f8d88071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:53.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.149+0000 7f8d9f7fe700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d8806ecc0 0x7f8d88071180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d94005950 tx=0x7f8d9400a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:53.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.154+0000 7f8d9cff9700 1 -- 172.21.15.151:0/3677520279 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8d90059160 con 0x7f8da0105a40 2024-04-16T02:48:53.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.361+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8da002cce0 con 0x7f8d8806ecc0 2024-04-16T02:48:53.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.372+0000 7f8d9cff9700 1 -- 172.21.15.151:0/3677520279 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8da002cce0 con 0x7f8d8806ecc0 2024-04-16T02:48:53.375 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (10m) 3m ago 21m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (22m) 3m ago 22m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (20m) 3m ago 20m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (7m) 3m ago 21m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (7m) 3m ago 20m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (10m) 3m ago 21m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (4m) 3m ago 16m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (3m) 3m ago 16m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (16m) 3m ago 16m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (3m) 3m ago 16m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (12m) 3m ago 23m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (11m) 3m ago 20m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (9m) 3m ago 23m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (8m) 3m ago 20m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:48:53.376 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (11m) 3m ago 21m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:48:53.377 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (11m) 3m ago 20m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:48:53.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (7m) 3m ago 19m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:48:53.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (6m) 3m ago 19m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:48:53.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (6m) 3m ago 18m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:48:53.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (5m) 3m ago 18m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:48:53.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (5m) 3m ago 18m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:48:53.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (4m) 3m ago 17m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:48:53.377 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (10m) 3m ago 21m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:48:53.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d8806ecc0 msgr2=0x7f8d88071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d8806ecc0 0x7f8d88071180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d94005950 tx=0x7f8d9400a300 comp rx=0 tx=0).stop 2024-04-16T02:48:53.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da0105a40 msgr2=0x7f8da0195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da0105a40 0x7f8da0195b60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d90006010 tx=0x7f8d9000bd70 comp rx=0 tx=0).stop 2024-04-16T02:48:53.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 shutdown_connections 2024-04-16T02:48:53.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d8806ecc0 0x7f8d88071180 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8da00ffbd0 0x7f8da0195620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 --2- 172.21.15.151:0/3677520279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8da0105a40 0x7f8da0195b60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 >> 172.21.15.151:0/3677520279 conn(0x7f8da00755a0 msgr2=0x7f8da00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:53.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.374+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 shutdown_connections 2024-04-16T02:48:53.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.375+0000 7f8da5d04700 1 -- 172.21.15.151:0/3677520279 wait complete. 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.479+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1862783925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 msgr2=0x7fe7d81020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.479+0000 7fe7df0d9700 1 --2- 172.21.15.151:0/1862783925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 0x7fe7d81020e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009910 tx=0x7fe7cc009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1862783925 shutdown_connections 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 --2- 172.21.15.151:0/1862783925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d8102620 0x7fe7d810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 --2- 172.21.15.151:0/1862783925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 0x7fe7d81020e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1862783925 >> 172.21.15.151:0/1862783925 conn(0x7fe7d80fb570 msgr2=0x7fe7d80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1862783925 shutdown_connections 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1862783925 wait complete. 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 Processor -- start 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 -- start start 2024-04-16T02:48:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 0x7fe7d819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d8102620 0x7fe7d819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d819d100 con 0x7fe7d8101d00 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.480+0000 7fe7df0d9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d81965b0 con 0x7fe7d8102620 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7dce75700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 0x7fe7d819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7dce75700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 0x7fe7d819c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49440/0 (socket says 172.21.15.151:49440) 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7dce75700 1 -- 172.21.15.151:0/1208040405 learned_addr learned my addr 172.21.15.151:0/1208040405 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7d7fff700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d8102620 0x7fe7d819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7dce75700 1 -- 172.21.15.151:0/1208040405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d8102620 msgr2=0x7fe7d819ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7dce75700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d8102620 0x7fe7d819ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7dce75700 1 -- 172.21.15.151:0/1208040405 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc0095f0 con 0x7fe7d8101d00 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7dce75700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 0x7fe7d819c530 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc006010 tx=0x7fe7cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7d5ffb700 1 -- 172.21.15.151:0/1208040405 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7cc01d070 con 0x7fe7d8101d00 2024-04-16T02:48:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7d5ffb700 1 -- 172.21.15.151:0/1208040405 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7cc004fb0 con 0x7fe7d8101d00 2024-04-16T02:48:53.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d8196830 con 0x7fe7d8101d00 2024-04-16T02:48:53.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7d5ffb700 1 -- 172.21.15.151:0/1208040405 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7cc022ea0 con 0x7fe7d8101d00 2024-04-16T02:48:53.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.481+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d8196d20 con 0x7fe7d8101d00 2024-04-16T02:48:53.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.482+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d8108210 con 0x7fe7d8101d00 2024-04-16T02:48:53.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.483+0000 7fe7d5ffb700 1 -- 172.21.15.151:0/1208040405 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe7cc022470 con 0x7fe7d8101d00 2024-04-16T02:48:53.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.483+0000 7fe7d5ffb700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe7c806ecc0 0x7fe7c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:53.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.483+0000 7fe7d5ffb700 1 -- 172.21.15.151:0/1208040405 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe7cc090350 con 0x7fe7d8101d00 2024-04-16T02:48:53.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.483+0000 7fe7d7fff700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe7c806ecc0 0x7fe7c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:53.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.484+0000 7fe7d7fff700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe7c806ecc0 0x7fe7c8071180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80fcc90 tx=0x7fe7c4005f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:53.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.488+0000 7fe7d5ffb700 1 -- 172.21.15.151:0/1208040405 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe7cc05cd60 con 0x7fe7d8101d00 2024-04-16T02:48:53.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:53 smithi151 ceph-mon[112233]: pgmap v307: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:53.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:53 smithi151 ceph-mon[112233]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:53.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:53 smithi153 ceph-mon[87858]: pgmap v307: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:53.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:53 smithi153 ceph-mon[87858]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:53.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.771+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe7d8197050 con 0x7fe7d8101d00 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.772+0000 7fe7d5ffb700 1 -- 172.21.15.151:0/1208040405 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fe7cc022720 con 0x7fe7d8101d00 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:48:53.775 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:48:53.776 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:48:53.776 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:48:53.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:48:53.776 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:48:53.776 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:48:53.776 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:48:53.776 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:48:53.776 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:48:53.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe7c806ecc0 msgr2=0x7fe7c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe7c806ecc0 0x7fe7c8071180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80fcc90 tx=0x7fe7c4005f90 comp rx=0 tx=0).stop 2024-04-16T02:48:53.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 msgr2=0x7fe7d819c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 0x7fe7d819c530 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc006010 tx=0x7fe7cc00be20 comp rx=0 tx=0).stop 2024-04-16T02:48:53.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 shutdown_connections 2024-04-16T02:48:53.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe7c806ecc0 0x7fe7c8071180 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe7d8101d00 0x7fe7d819c530 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 --2- 172.21.15.151:0/1208040405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d8102620 0x7fe7d819ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 >> 172.21.15.151:0/1208040405 conn(0x7fe7d80fb570 msgr2=0x7fe7d8105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:53.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 shutdown_connections 2024-04-16T02:48:53.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.774+0000 7fe7df0d9700 1 -- 172.21.15.151:0/1208040405 wait complete. 2024-04-16T02:48:53.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.880+0000 7fd743a11700 1 -- 172.21.15.151:0/219341895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c0ffa40 msgr2=0x7fd73c0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.880+0000 7fd743a11700 1 --2- 172.21.15.151:0/219341895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c0ffa40 0x7fd73c0ffec0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009910 tx=0x7fd72c009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:53.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.880+0000 7fd743a11700 1 -- 172.21.15.151:0/219341895 shutdown_connections 2024-04-16T02:48:53.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.880+0000 7fd743a11700 1 --2- 172.21.15.151:0/219341895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c0ffa40 0x7fd73c0ffec0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.880+0000 7fd743a11700 1 --2- 172.21.15.151:0/219341895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd73c105a40 0x7fd73c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.880+0000 7fd743a11700 1 -- 172.21.15.151:0/219341895 >> 172.21.15.151:0/219341895 conn(0x7fd73c0fb550 msgr2=0x7fd73c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:53.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.880+0000 7fd743a11700 1 -- 172.21.15.151:0/219341895 shutdown_connections 2024-04-16T02:48:53.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.880+0000 7fd743a11700 1 -- 172.21.15.151:0/219341895 wait complete. 2024-04-16T02:48:53.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd743a11700 1 Processor -- start 2024-04-16T02:48:53.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd743a11700 1 -- start start 2024-04-16T02:48:53.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd743a11700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd73c0ffa40 0x7fd73c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:53.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd743a11700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c105a40 0x7fd73c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd743a11700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c19d110 con 0x7fd73c105a40 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd743a11700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c1965c0 con 0x7fd73c0ffa40 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd740fac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c105a40 0x7fd73c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd740fac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c105a40 0x7fd73c19ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49464/0 (socket says 172.21.15.151:49464) 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd740fac700 1 -- 172.21.15.151:0/1425649200 learned_addr learned my addr 172.21.15.151:0/1425649200 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd7417ad700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd73c0ffa40 0x7fd73c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd740fac700 1 -- 172.21.15.151:0/1425649200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd73c0ffa40 msgr2=0x7fd73c19c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd740fac700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd73c0ffa40 0x7fd73c19c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd740fac700 1 -- 172.21.15.151:0/1425649200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c0095f0 con 0x7fd73c105a40 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.881+0000 7fd740fac700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c105a40 0x7fd73c19ca80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0092e0 tx=0x7fd72c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:53.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.882+0000 7fd7327fc700 1 -- 172.21.15.151:0/1425649200 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd72c01d070 con 0x7fd73c105a40 2024-04-16T02:48:53.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.882+0000 7fd7327fc700 1 -- 172.21.15.151:0/1425649200 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd72c005e00 con 0x7fd73c105a40 2024-04-16T02:48:53.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.882+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c196840 con 0x7fd73c105a40 2024-04-16T02:48:53.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.882+0000 7fd7327fc700 1 -- 172.21.15.151:0/1425649200 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd72c017400 con 0x7fd73c105a40 2024-04-16T02:48:53.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.882+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c196d30 con 0x7fd73c105a40 2024-04-16T02:48:53.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.883+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73c04ea90 con 0x7fd73c105a40 2024-04-16T02:48:53.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.883+0000 7fd7327fc700 1 -- 172.21.15.151:0/1425649200 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd72c00f6f0 con 0x7fd73c105a40 2024-04-16T02:48:53.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.884+0000 7fd7327fc700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd72806ecc0 0x7fd728071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:53.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.884+0000 7fd7327fc700 1 -- 172.21.15.151:0/1425649200 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd72c090340 con 0x7fd73c105a40 2024-04-16T02:48:53.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.884+0000 7fd7417ad700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd72806ecc0 0x7fd728071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:53.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.884+0000 7fd7417ad700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd72806ecc0 0x7fd728071180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd738005950 tx=0x7fd73800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:53.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:53.888+0000 7fd7327fc700 1 -- 172.21.15.151:0/1425649200 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd72c05cc20 con 0x7fd73c105a40 2024-04-16T02:48:54.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.138+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd73c1970f0 con 0x7fd73c105a40 2024-04-16T02:48:54.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.139+0000 7fd7327fc700 1 -- 172.21.15.151:0/1425649200 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7fd72c05c7b0 con 0x7fd73c105a40 2024-04-16T02:48:54.142 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:48:54.142 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:48:54.142 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:48:54.142 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:48:54.142 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:48:54.142 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:48:54.142 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:48:54.142 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:48:54.143 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:48:54.144 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:48:54.145 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:48:54.146 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd72806ecc0 msgr2=0x7fd728071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.146 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd72806ecc0 0x7fd728071180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd738005950 tx=0x7fd73800b410 comp rx=0 tx=0).stop 2024-04-16T02:48:54.146 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c105a40 msgr2=0x7fd73c19ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.146 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c105a40 0x7fd73c19ca80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0092e0 tx=0x7fd72c00bd70 comp rx=0 tx=0).stop 2024-04-16T02:48:54.146 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 shutdown_connections 2024-04-16T02:48:54.147 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd72806ecc0 0x7fd728071180 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.147 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd73c0ffa40 0x7fd73c19c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.147 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 --2- 172.21.15.151:0/1425649200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd73c105a40 0x7fd73c19ca80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.147 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 >> 172.21.15.151:0/1425649200 conn(0x7fd73c0fb550 msgr2=0x7fd73c103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:54.147 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 shutdown_connections 2024-04-16T02:48:54.147 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.142+0000 7fd743a11700 1 -- 172.21.15.151:0/1425649200 wait complete. 2024-04-16T02:48:54.147 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:48:54.254 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.249+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2623075230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 msgr2=0x7f3b580ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.249+0000 7f3b5f7be700 1 --2- 172.21.15.151:0/2623075230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 0x7f3b580ffec0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3b54009910 tx=0x7f3b54009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.249+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2623075230 shutdown_connections 2024-04-16T02:48:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.249+0000 7f3b5f7be700 1 --2- 172.21.15.151:0/2623075230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 0x7f3b580ffec0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.249+0000 7f3b5f7be700 1 --2- 172.21.15.151:0/2623075230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b58105a40 0x7f3b58105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.249+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2623075230 >> 172.21.15.151:0/2623075230 conn(0x7f3b580fb550 msgr2=0x7f3b580fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.249+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2623075230 shutdown_connections 2024-04-16T02:48:54.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.249+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2623075230 wait complete. 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.250+0000 7f3b5f7be700 1 Processor -- start 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.250+0000 7f3b5f7be700 1 -- start start 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.250+0000 7f3b5f7be700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 0x7f3b581975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.250+0000 7f3b5f7be700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b58105a40 0x7f3b58197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.250+0000 7f3b5f7be700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b58198080 con 0x7f3b580ffa40 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.250+0000 7f3b5f7be700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b581981c0 con 0x7f3b58105a40 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.250+0000 7f3b5d55a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 0x7f3b581975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.250+0000 7f3b5d55a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 0x7f3b581975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49494/0 (socket says 172.21.15.151:49494) 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.250+0000 7f3b5d55a700 1 -- 172.21.15.151:0/2904791750 learned_addr learned my addr 172.21.15.151:0/2904791750 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b5cd59700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b58105a40 0x7f3b58197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b5d55a700 1 -- 172.21.15.151:0/2904791750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b58105a40 msgr2=0x7f3b58197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b5d55a700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b58105a40 0x7f3b58197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b5d55a700 1 -- 172.21.15.151:0/2904791750 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b540095f0 con 0x7f3b580ffa40 2024-04-16T02:48:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b5cd59700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b58105a40 0x7f3b58197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:48:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b5d55a700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 0x7f3b581975b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800c930 tx=0x7f3b4800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b4e7fc700 1 -- 172.21.15.151:0/2904791750 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b48007ad0 con 0x7f3b580ffa40 2024-04-16T02:48:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b4e7fc700 1 -- 172.21.15.151:0/2904791750 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b48007c30 con 0x7f3b580ffa40 2024-04-16T02:48:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b581a0620 con 0x7f3b580ffa40 2024-04-16T02:48:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b581a0a90 con 0x7f3b580ffa40 2024-04-16T02:48:54.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.251+0000 7f3b4e7fc700 1 -- 172.21.15.151:0/2904791750 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b48018610 con 0x7f3b580ffa40 2024-04-16T02:48:54.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.252+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b58107fd0 con 0x7f3b580ffa40 2024-04-16T02:48:54.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.253+0000 7f3b4e7fc700 1 -- 172.21.15.151:0/2904791750 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3b4800f450 con 0x7f3b580ffa40 2024-04-16T02:48:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.253+0000 7f3b4e7fc700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b4406ecc0 0x7f3b44071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.253+0000 7f3b4e7fc700 1 -- 172.21.15.151:0/2904791750 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3b4808f0a0 con 0x7f3b580ffa40 2024-04-16T02:48:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.253+0000 7f3b5cd59700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b4406ecc0 0x7f3b44071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:54.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.254+0000 7f3b5cd59700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b4406ecc0 0x7f3b44071180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3b58069660 tx=0x7f3b5401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:54.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.258+0000 7f3b4e7fc700 1 -- 172.21.15.151:0/2904791750 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3b48057e50 con 0x7f3b580ffa40 2024-04-16T02:48:54.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.484+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b5802cce0 con 0x7f3b4406ecc0 2024-04-16T02:48:54.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.485+0000 7f3b4e7fc700 1 -- 172.21.15.151:0/2904791750 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3b5802cce0 con 0x7f3b4406ecc0 2024-04-16T02:48:54.488 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:48:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:48:54.489 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.487+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b4406ecc0 msgr2=0x7f3b44071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.487+0000 7f3b5f7be700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b4406ecc0 0x7f3b44071180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3b58069660 tx=0x7f3b5401a040 comp rx=0 tx=0).stop 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.487+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 msgr2=0x7f3b581975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.487+0000 7f3b5f7be700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 0x7f3b581975b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800c930 tx=0x7f3b4800ccf0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.487+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 shutdown_connections 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.487+0000 7f3b5f7be700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b4406ecc0 0x7f3b44071180 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.487+0000 7f3b5f7be700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b580ffa40 0x7f3b581975b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.487+0000 7f3b5f7be700 1 --2- 172.21.15.151:0/2904791750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b58105a40 0x7f3b58197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.487+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 >> 172.21.15.151:0/2904791750 conn(0x7f3b580fb550 msgr2=0x7f3b580fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.488+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 shutdown_connections 2024-04-16T02:48:54.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.488+0000 7f3b5f7be700 1 -- 172.21.15.151:0/2904791750 wait complete. 2024-04-16T02:48:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.593+0000 7f2595713700 1 -- 172.21.15.151:0/3381039717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 msgr2=0x7f25901011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.593+0000 7f2595713700 1 --2- 172.21.15.151:0/3381039717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 0x7f25901011a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2584009910 tx=0x7f2584009c20 comp rx=0 tx=0).stop 2024-04-16T02:48:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.593+0000 7f2595713700 1 -- 172.21.15.151:0/3381039717 shutdown_connections 2024-04-16T02:48:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.593+0000 7f2595713700 1 --2- 172.21.15.151:0/3381039717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101770 0x7f25901057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.593+0000 7f2595713700 1 --2- 172.21.15.151:0/3381039717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 0x7f25901011a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.593+0000 7f2595713700 1 -- 172.21.15.151:0/3381039717 >> 172.21.15.151:0/3381039717 conn(0x7f25900fc650 msgr2=0x7f25900fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.593+0000 7f2595713700 1 -- 172.21.15.151:0/3381039717 shutdown_connections 2024-04-16T02:48:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.593+0000 7f2595713700 1 -- 172.21.15.151:0/3381039717 wait complete. 2024-04-16T02:48:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f2595713700 1 Processor -- start 2024-04-16T02:48:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f2595713700 1 -- start start 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f2595713700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 0x7f2590073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f2595713700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101770 0x7f25900735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f2595713700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2590073b10 con 0x7f2590100dc0 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f2595713700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2590073c50 con 0x7f2590101770 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f258effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 0x7f2590073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f258effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 0x7f2590073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49512/0 (socket says 172.21.15.151:49512) 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f258effd700 1 -- 172.21.15.151:0/804681460 learned_addr learned my addr 172.21.15.151:0/804681460 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.594+0000 7f258e7fc700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101770 0x7f25900735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.595+0000 7f258effd700 1 -- 172.21.15.151:0/804681460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101770 msgr2=0x7f25900735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.595+0000 7f258effd700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101770 0x7f25900735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.595+0000 7f258effd700 1 -- 172.21.15.151:0/804681460 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25840095f0 con 0x7f2590100dc0 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.595+0000 7f258effd700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 0x7f2590073090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2584006010 tx=0x7f258400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.595+0000 7f2577fff700 1 -- 172.21.15.151:0/804681460 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f258401d070 con 0x7f2590100dc0 2024-04-16T02:48:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.595+0000 7f2577fff700 1 -- 172.21.15.151:0/804681460 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2584005dc0 con 0x7f2590100dc0 2024-04-16T02:48:54.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.595+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2590073ed0 con 0x7f2590100dc0 2024-04-16T02:48:54.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.595+0000 7f2577fff700 1 -- 172.21.15.151:0/804681460 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2584017400 con 0x7f2590100dc0 2024-04-16T02:48:54.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.595+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25901a2c70 con 0x7f2590100dc0 2024-04-16T02:48:54.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.596+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2590109160 con 0x7f2590100dc0 2024-04-16T02:48:54.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.597+0000 7f2577fff700 1 -- 172.21.15.151:0/804681460 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2584003a60 con 0x7f2590100dc0 2024-04-16T02:48:54.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.597+0000 7f2577fff700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:48:54.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.597+0000 7f2577fff700 1 -- 172.21.15.151:0/804681460 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f25840902e0 con 0x7f2590100dc0 2024-04-16T02:48:54.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.597+0000 7f258e7fc700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:48:54.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.598+0000 7f258e7fc700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2590074880 tx=0x7f258000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:48:54.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.602+0000 7f2577fff700 1 -- 172.21.15.151:0/804681460 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2584059090 con 0x7f2590100dc0 2024-04-16T02:48:54.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:54 smithi151 ceph-mon[112233]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:54.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:54 smithi151 ceph-mon[112233]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:54.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:54 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1208040405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:48:54.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:48:54.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:54 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1425649200' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:48:54.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:54 smithi153 ceph-mon[87858]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:54.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:54 smithi153 ceph-mon[87858]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:54.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:54 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1208040405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:48:54.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:48:54.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:54 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1425649200' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:48:54.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.885+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f259004ea90 con 0x7f2590100dc0 2024-04-16T02:48:54.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.886+0000 7f2577fff700 1 -- 172.21.15.151:0/804681460 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f258405c750 con 0x7f2590100dc0 2024-04-16T02:48:54.889 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:48:54.889 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:48:54.889 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:48:54.889 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:48:54.889 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:48:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 msgr2=0x7f2578071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2590074880 tx=0x7f258000b410 comp rx=0 tx=0).stop 2024-04-16T02:48:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 msgr2=0x7f2590073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:48:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 0x7f2590073090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2584006010 tx=0x7f258400bd70 comp rx=0 tx=0).stop 2024-04-16T02:48:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 shutdown_connections 2024-04-16T02:48:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f257806ecc0 0x7f2578071180 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2590100dc0 0x7f2590073090 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 --2- 172.21.15.151:0/804681460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2590101770 0x7f25900735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:48:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 >> 172.21.15.151:0/804681460 conn(0x7f25900fc650 msgr2=0x7f25900fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:48:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 shutdown_connections 2024-04-16T02:48:54.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:48:54.888+0000 7f2595713700 1 -- 172.21.15.151:0/804681460 wait complete. 2024-04-16T02:48:55.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:55 smithi153 ceph-mon[87858]: pgmap v308: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:55.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:55 smithi153 ceph-mon[87858]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:55.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:55 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/804681460' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:48:55.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:55 smithi151 ceph-mon[112233]: pgmap v308: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:55.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:55 smithi151 ceph-mon[112233]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:48:55.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:55 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/804681460' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:48:57.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:57 smithi153 ceph-mon[87858]: pgmap v309: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:57 smithi151 ceph-mon[112233]: pgmap v309: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:59.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:48:59 smithi151 ceph-mon[112233]: pgmap v310: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:48:59.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:48:59 smithi153 ceph-mon[87858]: pgmap v310: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:01.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:01 smithi153 ceph-mon[87858]: pgmap v311: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:01.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:01 smithi151 ceph-mon[112233]: pgmap v311: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:03.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:03 smithi153 ceph-mon[87858]: pgmap v312: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:03.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:03 smithi151 ceph-mon[112233]: pgmap v312: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:05 smithi153 ceph-mon[87858]: pgmap v313: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:05.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:05 smithi151 ceph-mon[112233]: pgmap v313: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:07.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:07 smithi151 ceph-mon[112233]: pgmap v314: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:08.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:07 smithi153 ceph-mon[87858]: pgmap v314: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:09 smithi151 ceph-mon[112233]: pgmap v315: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:09.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:49:10.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:09 smithi153 ceph-mon[87858]: pgmap v315: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:10.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:49:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:11 smithi151 ceph-mon[112233]: pgmap v316: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:12.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:11 smithi153 ceph-mon[87858]: pgmap v316: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:13.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:13 smithi151 ceph-mon[112233]: pgmap v317: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:14.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:13 smithi153 ceph-mon[87858]: pgmap v317: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:15.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:15 smithi151 ceph-mon[112233]: pgmap v318: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:16.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:15 smithi153 ceph-mon[87858]: pgmap v318: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:17.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:17 smithi153 ceph-mon[87858]: pgmap v319: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:17.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:17 smithi151 ceph-mon[112233]: pgmap v319: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:19 smithi151 ceph-mon[112233]: pgmap v320: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:19 smithi153 ceph-mon[87858]: pgmap v320: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:21.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:21 smithi151 ceph-mon[112233]: pgmap v321: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:22.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:21 smithi153 ceph-mon[87858]: pgmap v321: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:23 smithi151 ceph-mon[112233]: pgmap v322: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:24.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:23 smithi153 ceph-mon[87858]: pgmap v322: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:49:25.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.005+0000 7fc5839de700 1 -- 172.21.15.151:0/127681289 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 msgr2=0x7fc57c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.005+0000 7fc5839de700 1 --2- 172.21.15.151:0/127681289 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 0x7fc57c105e20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc570009910 tx=0x7fc570009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:25.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.005+0000 7fc5839de700 1 -- 172.21.15.151:0/127681289 shutdown_connections 2024-04-16T02:49:25.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.005+0000 7fc5839de700 1 --2- 172.21.15.151:0/127681289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc57c0ffa40 0x7fc57c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.005+0000 7fc5839de700 1 --2- 172.21.15.151:0/127681289 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 0x7fc57c105e20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.005+0000 7fc5839de700 1 -- 172.21.15.151:0/127681289 >> 172.21.15.151:0/127681289 conn(0x7fc57c0fb570 msgr2=0x7fc57c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:25.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.005+0000 7fc5839de700 1 -- 172.21.15.151:0/127681289 shutdown_connections 2024-04-16T02:49:25.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.005+0000 7fc5839de700 1 -- 172.21.15.151:0/127681289 wait complete. 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.006+0000 7fc5839de700 1 Processor -- start 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.006+0000 7fc5839de700 1 -- start start 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.006+0000 7fc5839de700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc57c0ffa40 0x7fc57c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.006+0000 7fc5839de700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 0x7fc57c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.006+0000 7fc5839de700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c19d080 con 0x7fc57c105a40 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.006+0000 7fc5839de700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c196540 con 0x7fc57c0ffa40 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.006+0000 7fc580f79700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 0x7fc57c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.006+0000 7fc580f79700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 0x7fc57c19c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60892/0 (socket says 172.21.15.151:60892) 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc580f79700 1 -- 172.21.15.151:0/3677939952 learned_addr learned my addr 172.21.15.151:0/3677939952 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc58177a700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc57c0ffa40 0x7fc57c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc580f79700 1 -- 172.21.15.151:0/3677939952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc57c0ffa40 msgr2=0x7fc57c19c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc580f79700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc57c0ffa40 0x7fc57c19c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc580f79700 1 -- 172.21.15.151:0/3677939952 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700095f0 con 0x7fc57c105a40 2024-04-16T02:49:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc580f79700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 0x7fc57c19c9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc57800e920 tx=0x7fc57800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc56e7fc700 1 -- 172.21.15.151:0/3677939952 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc57800c9d0 con 0x7fc57c105a40 2024-04-16T02:49:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc56e7fc700 1 -- 172.21.15.151:0/3677939952 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc578010430 con 0x7fc57c105a40 2024-04-16T02:49:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57c196820 con 0x7fc57c105a40 2024-04-16T02:49:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc56e7fc700 1 -- 172.21.15.151:0/3677939952 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc578018630 con 0x7fc57c105a40 2024-04-16T02:49:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.007+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57c196d70 con 0x7fc57c105a40 2024-04-16T02:49:25.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.008+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57c109050 con 0x7fc57c105a40 2024-04-16T02:49:25.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.009+0000 7fc56e7fc700 1 -- 172.21.15.151:0/3677939952 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc5780105a0 con 0x7fc57c105a40 2024-04-16T02:49:25.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.009+0000 7fc56e7fc700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc56806ecc0 0x7fc568071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:25.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.009+0000 7fc56e7fc700 1 -- 172.21.15.151:0/3677939952 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc578014070 con 0x7fc57c105a40 2024-04-16T02:49:25.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.009+0000 7fc58177a700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc56806ecc0 0x7fc568071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:25.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.010+0000 7fc58177a700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc56806ecc0 0x7fc568071180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc57000b760 tx=0x7fc570005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:25.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.014+0000 7fc56e7fc700 1 -- 172.21.15.151:0/3677939952 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc578053ff0 con 0x7fc57c105a40 2024-04-16T02:49:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:49:25.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.231+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc57c197050 con 0x7fc56806ecc0 2024-04-16T02:49:25.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.233+0000 7fc56e7fc700 1 -- 172.21.15.151:0/3677939952 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc57c197050 con 0x7fc56806ecc0 2024-04-16T02:49:25.238 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc56806ecc0 msgr2=0x7fc568071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.238 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc56806ecc0 0x7fc568071180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc57000b760 tx=0x7fc570005a90 comp rx=0 tx=0).stop 2024-04-16T02:49:25.238 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 msgr2=0x7fc57c19c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 0x7fc57c19c9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc57800e920 tx=0x7fc57800ec30 comp rx=0 tx=0).stop 2024-04-16T02:49:25.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 shutdown_connections 2024-04-16T02:49:25.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc56806ecc0 0x7fc568071180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc57c0ffa40 0x7fc57c19c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 --2- 172.21.15.151:0/3677939952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc57c105a40 0x7fc57c19c9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 >> 172.21.15.151:0/3677939952 conn(0x7fc57c0fb570 msgr2=0x7fc57c068fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:25.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 shutdown_connections 2024-04-16T02:49:25.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.235+0000 7fc5839de700 1 -- 172.21.15.151:0/3677939952 wait complete. 2024-04-16T02:49:25.249 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:49:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.342+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2604002658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb88105a50 msgr2=0x7fcb88105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.342+0000 7fcb8ce47700 1 --2- 172.21.15.151:0/2604002658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb88105a50 0x7fcb88105e30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000b0a0 tx=0x7fcb8000b3b0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.342+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2604002658 shutdown_connections 2024-04-16T02:49:25.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.342+0000 7fcb8ce47700 1 --2- 172.21.15.151:0/2604002658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb880ffa50 0x7fcb880ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.342+0000 7fcb8ce47700 1 --2- 172.21.15.151:0/2604002658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb88105a50 0x7fcb88105e30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.342+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2604002658 >> 172.21.15.151:0/2604002658 conn(0x7fcb880fb540 msgr2=0x7fcb880fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.342+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2604002658 shutdown_connections 2024-04-16T02:49:25.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.342+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2604002658 wait complete. 2024-04-16T02:49:25.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.343+0000 7fcb8ce47700 1 Processor -- start 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.343+0000 7fcb8ce47700 1 -- start start 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.343+0000 7fcb8ce47700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb880ffa50 0x7fcb88197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.343+0000 7fcb8ce47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb88105a50 0x7fcb88197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.343+0000 7fcb8ce47700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb881981e0 con 0x7fcb880ffa50 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.343+0000 7fcb8ce47700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88198350 con 0x7fcb88105a50 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.343+0000 7fcb8659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb880ffa50 0x7fcb88197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb8659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb880ffa50 0x7fcb88197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60910/0 (socket says 172.21.15.151:60910) 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb8659c700 1 -- 172.21.15.151:0/2802740530 learned_addr learned my addr 172.21.15.151:0/2802740530 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb85d9b700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb88105a50 0x7fcb88197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb8659c700 1 -- 172.21.15.151:0/2802740530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb88105a50 msgr2=0x7fcb88197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb8659c700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb88105a50 0x7fcb88197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb8659c700 1 -- 172.21.15.151:0/2802740530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb80009d00 con 0x7fcb880ffa50 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb8659c700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb880ffa50 0x7fcb88197680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcb80000f80 tx=0x7fcb8000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb7f7fe700 1 -- 172.21.15.151:0/2802740530 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb8000e040 con 0x7fcb880ffa50 2024-04-16T02:49:25.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb7f7fe700 1 -- 172.21.15.151:0/2802740530 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb80003de0 con 0x7fcb880ffa50 2024-04-16T02:49:25.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb7f7fe700 1 -- 172.21.15.151:0/2802740530 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb8001d7d0 con 0x7fcb880ffa50 2024-04-16T02:49:25.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb881a06f0 con 0x7fcb880ffa50 2024-04-16T02:49:25.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.344+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb881a0b60 con 0x7fcb880ffa50 2024-04-16T02:49:25.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.345+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb88108050 con 0x7fcb880ffa50 2024-04-16T02:49:25.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.346+0000 7fcb7f7fe700 1 -- 172.21.15.151:0/2802740530 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fcb80004040 con 0x7fcb880ffa50 2024-04-16T02:49:25.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.346+0000 7fcb7f7fe700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcb7406ecc0 0x7fcb74071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:25.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.346+0000 7fcb7f7fe700 1 -- 172.21.15.151:0/2802740530 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcb80090780 con 0x7fcb880ffa50 2024-04-16T02:49:25.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.347+0000 7fcb85d9b700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcb7406ecc0 0x7fcb74071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:25.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.347+0000 7fcb85d9b700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcb7406ecc0 0x7fcb74071180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb88069600 tx=0x7fcb70009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:25.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.351+0000 7fcb7f7fe700 1 -- 172.21.15.151:0/2802740530 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcb80059530 con 0x7fcb880ffa50 2024-04-16T02:49:25.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.572+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb8802cd30 con 0x7fcb7406ecc0 2024-04-16T02:49:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.574+0000 7fcb7f7fe700 1 -- 172.21.15.151:0/2802740530 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcb8802cd30 con 0x7fcb7406ecc0 2024-04-16T02:49:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcb7406ecc0 msgr2=0x7fcb74071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcb7406ecc0 0x7fcb74071180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb88069600 tx=0x7fcb70009190 comp rx=0 tx=0).stop 2024-04-16T02:49:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb880ffa50 msgr2=0x7fcb88197680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb880ffa50 0x7fcb88197680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcb80000f80 tx=0x7fcb8000cd20 comp rx=0 tx=0).stop 2024-04-16T02:49:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 shutdown_connections 2024-04-16T02:49:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcb7406ecc0 0x7fcb74071180 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb880ffa50 0x7fcb88197680 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 --2- 172.21.15.151:0/2802740530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb88105a50 0x7fcb88197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 >> 172.21.15.151:0/2802740530 conn(0x7fcb880fb540 msgr2=0x7fcb880fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:25.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 shutdown_connections 2024-04-16T02:49:25.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.576+0000 7fcb8ce47700 1 -- 172.21.15.151:0/2802740530 wait complete. 2024-04-16T02:49:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.680+0000 7f1af8762700 1 -- 172.21.15.151:0/2493688103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 msgr2=0x7f1af0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.680+0000 7f1af8762700 1 --2- 172.21.15.151:0/2493688103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 0x7f1af0100050 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009910 tx=0x7f1ae0009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.680+0000 7f1af8762700 1 -- 172.21.15.151:0/2493688103 shutdown_connections 2024-04-16T02:49:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.680+0000 7f1af8762700 1 --2- 172.21.15.151:0/2493688103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 0x7f1af0100050 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.680+0000 7f1af8762700 1 --2- 172.21.15.151:0/2493688103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1af0105a40 0x7f1af0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.680+0000 7f1af8762700 1 -- 172.21.15.151:0/2493688103 >> 172.21.15.151:0/2493688103 conn(0x7f1af00755a0 msgr2=0x7f1af00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:25.684 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:25 smithi151 ceph-mon[112233]: pgmap v323: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:25.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.680+0000 7f1af8762700 1 -- 172.21.15.151:0/2493688103 shutdown_connections 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.680+0000 7f1af8762700 1 -- 172.21.15.151:0/2493688103 wait complete. 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af8762700 1 Processor -- start 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af8762700 1 -- start start 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af8762700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 0x7f1af0101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af8762700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1af0105a40 0x7f1af0101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af8762700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af0102090 con 0x7f1af00ffbd0 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af8762700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af01021d0 con 0x7f1af0105a40 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af64fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 0x7f1af0101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af64fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 0x7f1af0101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60920/0 (socket says 172.21.15.151:60920) 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af64fe700 1 -- 172.21.15.151:0/2024565335 learned_addr learned my addr 172.21.15.151:0/2024565335 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af5cfd700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1af0105a40 0x7f1af0101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af64fe700 1 -- 172.21.15.151:0/2024565335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1af0105a40 msgr2=0x7f1af0101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.681+0000 7f1af64fe700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1af0105a40 0x7f1af0101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.682+0000 7f1af64fe700 1 -- 172.21.15.151:0/2024565335 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae00095f0 con 0x7f1af00ffbd0 2024-04-16T02:49:25.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.682+0000 7f1af64fe700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 0x7f1af0101610 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00b4b0 tx=0x7f1aec00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:25.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.682+0000 7f1ae77fe700 1 -- 172.21.15.151:0/2024565335 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1aec011800 con 0x7f1af00ffbd0 2024-04-16T02:49:25.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.682+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af01a2a30 con 0x7f1af00ffbd0 2024-04-16T02:49:25.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.682+0000 7f1ae77fe700 1 -- 172.21.15.151:0/2024565335 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1aec011e20 con 0x7f1af00ffbd0 2024-04-16T02:49:25.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.682+0000 7f1ae77fe700 1 -- 172.21.15.151:0/2024565335 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1aec00f2e0 con 0x7f1af00ffbd0 2024-04-16T02:49:25.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.682+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af01a2e40 con 0x7f1af00ffbd0 2024-04-16T02:49:25.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.683+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af0109070 con 0x7f1af00ffbd0 2024-04-16T02:49:25.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.683+0000 7f1ae77fe700 1 -- 172.21.15.151:0/2024565335 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1aec010380 con 0x7f1af00ffbd0 2024-04-16T02:49:25.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.684+0000 7f1ae77fe700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1adc06ecc0 0x7f1adc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:25.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.684+0000 7f1ae77fe700 1 -- 172.21.15.151:0/2024565335 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1aec08e980 con 0x7f1af00ffbd0 2024-04-16T02:49:25.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.684+0000 7f1af5cfd700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1adc06ecc0 0x7f1adc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:25.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.684+0000 7f1af5cfd700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1adc06ecc0 0x7f1adc071180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1af0102e00 tx=0x7f1ae001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:25.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.689+0000 7f1ae77fe700 1 -- 172.21.15.151:0/2024565335 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1aec057730 con 0x7f1af00ffbd0 2024-04-16T02:49:25.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.897+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1af002cd30 con 0x7f1adc06ecc0 2024-04-16T02:49:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.908+0000 7f1ae77fe700 1 -- 172.21.15.151:0/2024565335 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1af002cd30 con 0x7f1adc06ecc0 2024-04-16T02:49:25.911 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:49:25.911 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (11m) 4m ago 22m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:49:25.911 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (22m) 4m ago 22m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:49:25.911 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (21m) 4m ago 21m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:49:25.911 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (8m) 4m ago 22m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (8m) 4m ago 21m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (10m) 4m ago 22m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (4m) 4m ago 16m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (4m) 4m ago 16m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (16m) 4m ago 16m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (4m) 4m ago 16m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (13m) 4m ago 23m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (11m) 4m ago 21m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (9m) 4m ago 23m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (9m) 4m ago 21m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (11m) 4m ago 22m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (11m) 4m ago 21m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (8m) 4m ago 20m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (7m) 4m ago 19m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:49:25.912 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (6m) 4m ago 19m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:49:25.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (6m) 4m ago 19m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:49:25.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (5m) 4m ago 18m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:49:25.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (5m) 4m ago 18m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:49:25.913 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (11m) 4m ago 21m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:49:25.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1adc06ecc0 msgr2=0x7f1adc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1adc06ecc0 0x7f1adc071180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1af0102e00 tx=0x7f1ae001a040 comp rx=0 tx=0).stop 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 msgr2=0x7f1af0101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 0x7f1af0101610 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00b4b0 tx=0x7f1aec00b7c0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 shutdown_connections 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1adc06ecc0 0x7f1adc071180 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1af00ffbd0 0x7f1af0101610 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 --2- 172.21.15.151:0/2024565335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1af0105a40 0x7f1af0101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 >> 172.21.15.151:0/2024565335 conn(0x7f1af00755a0 msgr2=0x7f1af0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 shutdown_connections 2024-04-16T02:49:25.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:25.910+0000 7f1af8762700 1 -- 172.21.15.151:0/2024565335 wait complete. 2024-04-16T02:49:26.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.015+0000 7f12d88ae700 1 -- 172.21.15.151:0/1945043822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ff500 msgr2=0x7f12d40ff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.015+0000 7f12d88ae700 1 --2- 172.21.15.151:0/1945043822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ff500 0x7f12d40ff8e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009910 tx=0x7f12c8009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:26.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.015+0000 7f12d88ae700 1 -- 172.21.15.151:0/1945043822 shutdown_connections 2024-04-16T02:49:26.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.015+0000 7f12d88ae700 1 --2- 172.21.15.151:0/1945043822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d40ffeb0 0x7f12d4105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.015+0000 7f12d88ae700 1 --2- 172.21.15.151:0/1945043822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ff500 0x7f12d40ff8e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.015+0000 7f12d88ae700 1 -- 172.21.15.151:0/1945043822 >> 172.21.15.151:0/1945043822 conn(0x7f12d4074d80 msgr2=0x7f12d4075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:26.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.015+0000 7f12d88ae700 1 -- 172.21.15.151:0/1945043822 shutdown_connections 2024-04-16T02:49:26.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.016+0000 7f12d88ae700 1 -- 172.21.15.151:0/1945043822 wait complete. 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.016+0000 7f12d88ae700 1 Processor -- start 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.016+0000 7f12d88ae700 1 -- start start 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.016+0000 7f12d88ae700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d40ff500 0x7f12d4100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.016+0000 7f12d88ae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ffeb0 0x7f12d41009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.016+0000 7f12d88ae700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d4100f00 con 0x7f12d40ffeb0 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.016+0000 7f12d88ae700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d4101040 con 0x7f12d40ff500 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ffeb0 0x7f12d41009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ffeb0 0x7f12d41009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60944/0 (socket says 172.21.15.151:60944) 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d1d9b700 1 -- 172.21.15.151:0/488110895 learned_addr learned my addr 172.21.15.151:0/488110895 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d259c700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d40ff500 0x7f12d4100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d1d9b700 1 -- 172.21.15.151:0/488110895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d40ff500 msgr2=0x7f12d4100480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d1d9b700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d40ff500 0x7f12d4100480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d1d9b700 1 -- 172.21.15.151:0/488110895 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c80095f0 con 0x7f12d40ffeb0 2024-04-16T02:49:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d1d9b700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ffeb0 0x7f12d41009c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f12c400d900 tx=0x7f12c400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12c37fe700 1 -- 172.21.15.151:0/488110895 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12c4009900 con 0x7f12d40ffeb0 2024-04-16T02:49:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12c37fe700 1 -- 172.21.15.151:0/488110895 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12c4010460 con 0x7f12d40ffeb0 2024-04-16T02:49:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12c37fe700 1 -- 172.21.15.151:0/488110895 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12c400f5f0 con 0x7f12d40ffeb0 2024-04-16T02:49:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d41a2a10 con 0x7f12d40ffeb0 2024-04-16T02:49:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.017+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d41a2e20 con 0x7f12d40ffeb0 2024-04-16T02:49:26.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:25 smithi153 ceph-mon[87858]: pgmap v323: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:26.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.018+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d4109190 con 0x7f12d40ffeb0 2024-04-16T02:49:26.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.019+0000 7f12c37fe700 1 -- 172.21.15.151:0/488110895 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f12c40105d0 con 0x7f12d40ffeb0 2024-04-16T02:49:26.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.019+0000 7f12c37fe700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f12bc06ecc0 0x7f12bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:26.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.019+0000 7f12c37fe700 1 -- 172.21.15.151:0/488110895 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f12c408e920 con 0x7f12d40ffeb0 2024-04-16T02:49:26.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.019+0000 7f12d259c700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f12bc06ecc0 0x7f12bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:26.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.020+0000 7f12d259c700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f12bc06ecc0 0x7f12bc071180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12c8006010 tx=0x7f12c801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:26.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.024+0000 7f12c37fe700 1 -- 172.21.15.151:0/488110895 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f12c4057750 con 0x7f12d40ffeb0 2024-04-16T02:49:26.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.311+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f12d404ea90 con 0x7f12d40ffeb0 2024-04-16T02:49:26.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.312+0000 7f12c37fe700 1 -- 172.21.15.151:0/488110895 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f12c405ae10 con 0x7f12d40ffeb0 2024-04-16T02:49:26.315 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:49:26.315 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:49:26.315 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:49:26.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:49:26.315 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:49:26.315 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:49:26.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:49:26.316 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:49:26.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f12bc06ecc0 msgr2=0x7f12bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f12bc06ecc0 0x7f12bc071180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12c8006010 tx=0x7f12c801a040 comp rx=0 tx=0).stop 2024-04-16T02:49:26.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ffeb0 msgr2=0x7f12d41009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ffeb0 0x7f12d41009c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f12c400d900 tx=0x7f12c400dc10 comp rx=0 tx=0).stop 2024-04-16T02:49:26.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 shutdown_connections 2024-04-16T02:49:26.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f12bc06ecc0 0x7f12bc071180 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d40ff500 0x7f12d4100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 --2- 172.21.15.151:0/488110895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12d40ffeb0 0x7f12d41009c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 >> 172.21.15.151:0/488110895 conn(0x7f12d4074d80 msgr2=0x7f12d40fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:26.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 shutdown_connections 2024-04-16T02:49:26.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.314+0000 7f12d88ae700 1 -- 172.21.15.151:0/488110895 wait complete. 2024-04-16T02:49:26.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.419+0000 7ff8a07f2700 1 -- 172.21.15.151:0/561376759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 msgr2=0x7ff898101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.419+0000 7ff8a07f2700 1 --2- 172.21.15.151:0/561376759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 0x7ff898101180 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff890009910 tx=0x7ff890009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:26.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.419+0000 7ff8a07f2700 1 -- 172.21.15.151:0/561376759 shutdown_connections 2024-04-16T02:49:26.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.419+0000 7ff8a07f2700 1 --2- 172.21.15.151:0/561376759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff898101750 0x7ff8981057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.419+0000 7ff8a07f2700 1 --2- 172.21.15.151:0/561376759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 0x7ff898101180 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.419+0000 7ff8a07f2700 1 -- 172.21.15.151:0/561376759 >> 172.21.15.151:0/561376759 conn(0x7ff8980fc650 msgr2=0x7ff8980fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:26.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.419+0000 7ff8a07f2700 1 -- 172.21.15.151:0/561376759 shutdown_connections 2024-04-16T02:49:26.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.419+0000 7ff8a07f2700 1 -- 172.21.15.151:0/561376759 wait complete. 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff8a07f2700 1 Processor -- start 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff8a07f2700 1 -- start start 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff8a07f2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 0x7ff89819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff8a07f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff898101750 0x7ff89819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff8a07f2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89819d0f0 con 0x7ff898100da0 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff8a07f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8981965a0 con 0x7ff898101750 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff89e58e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 0x7ff89819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff89e58e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 0x7ff89819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60970/0 (socket says 172.21.15.151:60970) 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff89e58e700 1 -- 172.21.15.151:0/3414891041 learned_addr learned my addr 172.21.15.151:0/3414891041 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff89dd8d700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff898101750 0x7ff89819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff89e58e700 1 -- 172.21.15.151:0/3414891041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff898101750 msgr2=0x7ff89819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff89e58e700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff898101750 0x7ff89819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.420+0000 7ff89e58e700 1 -- 172.21.15.151:0/3414891041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8900095f0 con 0x7ff898100da0 2024-04-16T02:49:26.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.421+0000 7ff89e58e700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 0x7ff89819c520 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff890006010 tx=0x7ff89000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:26.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.421+0000 7ff8977fe700 1 -- 172.21.15.151:0/3414891041 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff89001d070 con 0x7ff898100da0 2024-04-16T02:49:26.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.421+0000 7ff8977fe700 1 -- 172.21.15.151:0/3414891041 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff890005dc0 con 0x7ff898100da0 2024-04-16T02:49:26.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.421+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff898196820 con 0x7ff898100da0 2024-04-16T02:49:26.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.421+0000 7ff8977fe700 1 -- 172.21.15.151:0/3414891041 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff890017400 con 0x7ff898100da0 2024-04-16T02:49:26.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.421+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff898196d10 con 0x7ff898100da0 2024-04-16T02:49:26.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.422+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff898109140 con 0x7ff898100da0 2024-04-16T02:49:26.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.422+0000 7ff8977fe700 1 -- 172.21.15.151:0/3414891041 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff890003a60 con 0x7ff898100da0 2024-04-16T02:49:26.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.423+0000 7ff8977fe700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff88006ecc0 0x7ff880071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:26.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.423+0000 7ff89dd8d700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff88006ecc0 0x7ff880071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:26.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.423+0000 7ff8977fe700 1 -- 172.21.15.151:0/3414891041 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff8900902f0 con 0x7ff898100da0 2024-04-16T02:49:26.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.423+0000 7ff89dd8d700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff88006ecc0 0x7ff880071180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff8980fce20 tx=0x7ff88c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:26.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.428+0000 7ff8977fe700 1 -- 172.21.15.151:0/3414891041 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff89005cbd0 con 0x7ff898100da0 2024-04-16T02:49:26.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.676+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff89804ea90 con 0x7ff898100da0 2024-04-16T02:49:26.680 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:26 smithi151 ceph-mon[112233]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:26.680 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:26 smithi151 ceph-mon[112233]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:26.680 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:26 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/488110895' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:49:26.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.677+0000 7ff8977fe700 1 -- 172.21.15.151:0/3414891041 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7ff89005c760 con 0x7ff898100da0 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:49:26.681 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:49:26.682 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:49:26.683 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:49:26.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff88006ecc0 msgr2=0x7ff880071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff88006ecc0 0x7ff880071180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff8980fce20 tx=0x7ff88c00b410 comp rx=0 tx=0).stop 2024-04-16T02:49:26.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 msgr2=0x7ff89819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 0x7ff89819c520 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff890006010 tx=0x7ff89000bd70 comp rx=0 tx=0).stop 2024-04-16T02:49:26.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 shutdown_connections 2024-04-16T02:49:26.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff88006ecc0 0x7ff880071180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff898100da0 0x7ff89819c520 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 --2- 172.21.15.151:0/3414891041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff898101750 0x7ff89819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 >> 172.21.15.151:0/3414891041 conn(0x7ff8980fc650 msgr2=0x7ff8980fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:26.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 shutdown_connections 2024-04-16T02:49:26.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.679+0000 7ff8a07f2700 1 -- 172.21.15.151:0/3414891041 wait complete. 2024-04-16T02:49:26.685 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:49:26.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.785+0000 7f91207eb700 1 -- 172.21.15.151:0/3580092390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 msgr2=0x7f911810d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.785+0000 7f91207eb700 1 --2- 172.21.15.151:0/3580092390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 0x7f911810d240 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9114009910 tx=0x7f9114009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:26.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.785+0000 7f91207eb700 1 -- 172.21.15.151:0/3580092390 shutdown_connections 2024-04-16T02:49:26.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.785+0000 7f91207eb700 1 --2- 172.21.15.151:0/3580092390 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 0x7f911810d240 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.785+0000 7f91207eb700 1 --2- 172.21.15.151:0/3580092390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9118068410 0x7f91180687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.785+0000 7f91207eb700 1 -- 172.21.15.151:0/3580092390 >> 172.21.15.151:0/3580092390 conn(0x7f91180755a0 msgr2=0x7f91180759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:26.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.785+0000 7f91207eb700 1 -- 172.21.15.151:0/3580092390 shutdown_connections 2024-04-16T02:49:26.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.785+0000 7f91207eb700 1 -- 172.21.15.151:0/3580092390 wait complete. 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f91207eb700 1 Processor -- start 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f91207eb700 1 -- start start 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f91207eb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9118068410 0x7f911819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f91207eb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 0x7f911819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f91207eb700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911819f220 con 0x7f9118068d30 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f91207eb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9118198760 con 0x7f9118068410 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f911dd86700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 0x7f911819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f911dd86700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 0x7f911819ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34094/0 (socket says 172.21.15.151:34094) 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f911dd86700 1 -- 172.21.15.151:0/1742715764 learned_addr learned my addr 172.21.15.151:0/1742715764 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f911e587700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9118068410 0x7f911819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f911dd86700 1 -- 172.21.15.151:0/1742715764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9118068410 msgr2=0x7f911819e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:26.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f911dd86700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9118068410 0x7f911819e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:26.792 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f911dd86700 1 -- 172.21.15.151:0/1742715764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91140095f0 con 0x7f9118068d30 2024-04-16T02:49:26.792 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.786+0000 7f911dd86700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 0x7f911819ec20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9114006010 tx=0x7f911400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:26.792 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.787+0000 7f910f7fe700 1 -- 172.21.15.151:0/1742715764 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f911401d070 con 0x7f9118068d30 2024-04-16T02:49:26.792 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.787+0000 7f910f7fe700 1 -- 172.21.15.151:0/1742715764 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9114004fb0 con 0x7f9118068d30 2024-04-16T02:49:26.792 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.787+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9118198a40 con 0x7f9118068d30 2024-04-16T02:49:26.792 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.787+0000 7f910f7fe700 1 -- 172.21.15.151:0/1742715764 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9114022ea0 con 0x7f9118068d30 2024-04-16T02:49:26.792 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.787+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9118198f90 con 0x7f9118068d30 2024-04-16T02:49:26.792 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.788+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911810a9b0 con 0x7f9118068d30 2024-04-16T02:49:26.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.788+0000 7f910f7fe700 1 -- 172.21.15.151:0/1742715764 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9114022470 con 0x7f9118068d30 2024-04-16T02:49:26.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.789+0000 7f910f7fe700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f910406ecc0 0x7f9104071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:26.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.789+0000 7f911e587700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f910406ecc0 0x7f9104071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:26.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.789+0000 7f910f7fe700 1 -- 172.21.15.151:0/1742715764 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f91140903b0 con 0x7f9118068d30 2024-04-16T02:49:26.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.789+0000 7f911e587700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f910406ecc0 0x7f9104071180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9108005fd0 tx=0x7f9108005e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:26.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:26.794+0000 7f910f7fe700 1 -- 172.21.15.151:0/1742715764 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f911405cd40 con 0x7f9118068d30 2024-04-16T02:49:27.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.009+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9118199270 con 0x7f910406ecc0 2024-04-16T02:49:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.011+0000 7f910f7fe700 1 -- 172.21.15.151:0/1742715764 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9118199270 con 0x7f910406ecc0 2024-04-16T02:49:27.014 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:49:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:49:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:49:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:49:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:49:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:49:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:49:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:49:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:49:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:49:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:49:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:49:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:49:27.015 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:49:27.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f910406ecc0 msgr2=0x7f9104071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:27.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f910406ecc0 0x7f9104071180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9108005fd0 tx=0x7f9108005e50 comp rx=0 tx=0).stop 2024-04-16T02:49:27.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 msgr2=0x7f911819ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:27.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 0x7f911819ec20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9114006010 tx=0x7f911400be20 comp rx=0 tx=0).stop 2024-04-16T02:49:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 shutdown_connections 2024-04-16T02:49:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f910406ecc0 0x7f9104071180 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9118068410 0x7f911819e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 --2- 172.21.15.151:0/1742715764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9118068d30 0x7f911819ec20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 >> 172.21.15.151:0/1742715764 conn(0x7f91180755a0 msgr2=0x7f91180fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 shutdown_connections 2024-04-16T02:49:27.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.013+0000 7f91207eb700 1 -- 172.21.15.151:0/1742715764 wait complete. 2024-04-16T02:49:27.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:26 smithi153 ceph-mon[87858]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:27.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:26 smithi153 ceph-mon[87858]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:27.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:26 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/488110895' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 -- 172.21.15.151:0/3288348292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 msgr2=0x7f35181057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 --2- 172.21.15.151:0/3288348292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 0x7f35181057a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3508009910 tx=0x7f3508009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 -- 172.21.15.151:0/3288348292 shutdown_connections 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 --2- 172.21.15.151:0/3288348292 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 0x7f35181057a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 --2- 172.21.15.151:0/3288348292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518100da0 0x7f3518101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 -- 172.21.15.151:0/3288348292 >> 172.21.15.151:0/3288348292 conn(0x7f35180fc630 msgr2=0x7f35180fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 -- 172.21.15.151:0/3288348292 shutdown_connections 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 -- 172.21.15.151:0/3288348292 wait complete. 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 Processor -- start 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.120+0000 7f35202fe700 1 -- start start 2024-04-16T02:49:27.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f35202fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518100da0 0x7f3518073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f35202fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 0x7f35180735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f35202fe700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518073b10 con 0x7f3518101750 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f35202fe700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518073c50 con 0x7f3518100da0 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f351d899700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 0x7f35180735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f351d899700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 0x7f35180735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34120/0 (socket says 172.21.15.151:34120) 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f351d899700 1 -- 172.21.15.151:0/1334241530 learned_addr learned my addr 172.21.15.151:0/1334241530 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f351e09a700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518100da0 0x7f3518073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f351d899700 1 -- 172.21.15.151:0/1334241530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518100da0 msgr2=0x7f3518073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f351d899700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518100da0 0x7f3518073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f351d899700 1 -- 172.21.15.151:0/1334241530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35080095f0 con 0x7f3518101750 2024-04-16T02:49:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f351d899700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 0x7f35180735d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f350800bda0 tx=0x7f350800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:27.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f350f7fe700 1 -- 172.21.15.151:0/1334241530 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f350801d070 con 0x7f3518101750 2024-04-16T02:49:27.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.121+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3518073ed0 con 0x7f3518101750 2024-04-16T02:49:27.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.122+0000 7f350f7fe700 1 -- 172.21.15.151:0/1334241530 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3508005dc0 con 0x7f3518101750 2024-04-16T02:49:27.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.122+0000 7f350f7fe700 1 -- 172.21.15.151:0/1334241530 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3508017400 con 0x7f3518101750 2024-04-16T02:49:27.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.122+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35181a2c30 con 0x7f3518101750 2024-04-16T02:49:27.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.122+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35181091a0 con 0x7f3518101750 2024-04-16T02:49:27.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.123+0000 7f350f7fe700 1 -- 172.21.15.151:0/1334241530 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f350800f460 con 0x7f3518101750 2024-04-16T02:49:27.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.123+0000 7f350f7fe700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:27.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.124+0000 7f350f7fe700 1 -- 172.21.15.151:0/1334241530 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3508090230 con 0x7f3518101750 2024-04-16T02:49:27.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.124+0000 7f351e09a700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:27.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.124+0000 7f351e09a700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3514005950 tx=0x7f351400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:27.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.128+0000 7f350f7fe700 1 -- 172.21.15.151:0/1334241530 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3508058fe0 con 0x7f3518101750 2024-04-16T02:49:27.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.408+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3518108b80 con 0x7f3518101750 2024-04-16T02:49:27.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.409+0000 7f350f7fe700 1 -- 172.21.15.151:0/1334241530 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f350805c6a0 con 0x7f3518101750 2024-04-16T02:49:27.412 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:49:27.412 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:49:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:49:27.412 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:49:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:49:27.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 msgr2=0x7f3504071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:27.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3514005950 tx=0x7f351400b410 comp rx=0 tx=0).stop 2024-04-16T02:49:27.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 msgr2=0x7f35180735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:27.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 0x7f35180735d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f350800bda0 tx=0x7f350800bdd0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 shutdown_connections 2024-04-16T02:49:27.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518100da0 0x7f3518073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 --2- 172.21.15.151:0/1334241530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3518101750 0x7f35180735d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:27.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 >> 172.21.15.151:0/1334241530 conn(0x7f35180fc630 msgr2=0x7f35180fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:27.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 shutdown_connections 2024-04-16T02:49:27.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:27.411+0000 7f35202fe700 1 -- 172.21.15.151:0/1334241530 wait complete. 2024-04-16T02:49:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:27 smithi151 ceph-mon[112233]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:27.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:27 smithi151 ceph-mon[112233]: pgmap v324: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:27.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:27 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3414891041' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:49:27.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:27 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1334241530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:49:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:27 smithi153 ceph-mon[87858]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:27 smithi153 ceph-mon[87858]: pgmap v324: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:27 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3414891041' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:49:27.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:27 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1334241530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:49:28.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:28 smithi151 ceph-mon[112233]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:29.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:28 smithi153 ceph-mon[87858]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:29.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:29 smithi151 ceph-mon[112233]: pgmap v325: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:30.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:29 smithi153 ceph-mon[87858]: pgmap v325: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:31.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:31 smithi151 ceph-mon[112233]: pgmap v326: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:31 smithi153 ceph-mon[87858]: pgmap v326: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:33.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:33 smithi151 ceph-mon[112233]: pgmap v327: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:33 smithi153 ceph-mon[87858]: pgmap v327: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:35.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:35 smithi151 ceph-mon[112233]: pgmap v328: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:36.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:35 smithi153 ceph-mon[87858]: pgmap v328: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:37.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:37 smithi151 ceph-mon[112233]: pgmap v329: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:38.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:37 smithi153 ceph-mon[87858]: pgmap v329: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:39.838 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:39 smithi151 ceph-mon[112233]: pgmap v330: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:39.838 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:49:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:39 smithi153 ceph-mon[87858]: pgmap v330: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:49:41.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:41 smithi151 ceph-mon[112233]: pgmap v331: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:42.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:41 smithi153 ceph-mon[87858]: pgmap v331: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:43.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:43 smithi151 ceph-mon[112233]: pgmap v332: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:44.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:43 smithi153 ceph-mon[87858]: pgmap v332: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:45 smithi151 ceph-mon[112233]: pgmap v333: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:45 smithi153 ceph-mon[87858]: pgmap v333: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:47.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:47 smithi151 ceph-mon[112233]: pgmap v334: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:47 smithi153 ceph-mon[87858]: pgmap v334: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:49.945 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:49 smithi151 ceph-mon[112233]: pgmap v335: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:49 smithi153 ceph-mon[87858]: pgmap v335: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:51.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:51 smithi151 ceph-mon[112233]: pgmap v336: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:52.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:51 smithi153 ceph-mon[87858]: pgmap v336: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:53.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:53 smithi151 ceph-mon[112233]: pgmap v337: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:54.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:53 smithi153 ceph-mon[87858]: pgmap v337: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:54.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:49:55.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:49:55.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:55 smithi151 ceph-mon[112233]: pgmap v338: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:56.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:55 smithi153 ceph-mon[87858]: pgmap v338: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:57.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.530+0000 7f7ac0439700 1 -- 172.21.15.151:0/3868735680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8101770 msgr2=0x7f7ab81057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:57.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.530+0000 7f7ac0439700 1 --2- 172.21.15.151:0/3868735680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8101770 0x7f7ab81057c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009910 tx=0x7f7aa8009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:57.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.531+0000 7f7ac0439700 1 -- 172.21.15.151:0/3868735680 shutdown_connections 2024-04-16T02:49:57.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.531+0000 7f7ac0439700 1 --2- 172.21.15.151:0/3868735680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8101770 0x7f7ab81057c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:57.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.531+0000 7f7ac0439700 1 --2- 172.21.15.151:0/3868735680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ab8100dc0 0x7f7ab81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:57.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.531+0000 7f7ac0439700 1 -- 172.21.15.151:0/3868735680 >> 172.21.15.151:0/3868735680 conn(0x7f7ab80fc650 msgr2=0x7f7ab80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:57.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.531+0000 7f7ac0439700 1 -- 172.21.15.151:0/3868735680 shutdown_connections 2024-04-16T02:49:57.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.531+0000 7f7ac0439700 1 -- 172.21.15.151:0/3868735680 wait complete. 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.531+0000 7f7ac0439700 1 Processor -- start 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.531+0000 7f7ac0439700 1 -- start start 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7ac0439700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8100dc0 0x7f7ab819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7ac0439700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ab8101770 0x7f7ab819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7ac0439700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab819d240 con 0x7f7ab8100dc0 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7ac0439700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab81966f0 con 0x7f7ab8101770 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7abe1d5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8100dc0 0x7f7ab819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7abe1d5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8100dc0 0x7f7ab819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43716/0 (socket says 172.21.15.151:43716) 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7abe1d5700 1 -- 172.21.15.151:0/1994965902 learned_addr learned my addr 172.21.15.151:0/1994965902 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7abd9d4700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ab8101770 0x7f7ab819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7abe1d5700 1 -- 172.21.15.151:0/1994965902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ab8101770 msgr2=0x7f7ab819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7abe1d5700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ab8101770 0x7f7ab819cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7abe1d5700 1 -- 172.21.15.151:0/1994965902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80095f0 con 0x7f7ab8100dc0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7abe1d5700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8100dc0 0x7f7ab819c670 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400b4b0 tx=0x7f7ab400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7aaf7fe700 1 -- 172.21.15.151:0/1994965902 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab4011800 con 0x7f7ab8100dc0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.532+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab81969d0 con 0x7f7ab8100dc0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.533+0000 7f7aaf7fe700 1 -- 172.21.15.151:0/1994965902 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ab4011e20 con 0x7f7ab8100dc0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.533+0000 7f7aaf7fe700 1 -- 172.21.15.151:0/1994965902 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab400f2e0 con 0x7f7ab8100dc0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.533+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab8196f20 con 0x7f7ab8100dc0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.533+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab81090f0 con 0x7f7ab8100dc0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.534+0000 7f7aaf7fe700 1 -- 172.21.15.151:0/1994965902 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7ab4010380 con 0x7f7ab8100dc0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.535+0000 7f7aaf7fe700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7aa406ecc0 0x7f7aa4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.535+0000 7f7aaf7fe700 1 -- 172.21.15.151:0/1994965902 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7ab408e920 con 0x7f7ab8100dc0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.535+0000 7f7abd9d4700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7aa406ecc0 0x7f7aa4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.535+0000 7f7abd9d4700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7aa406ecc0 0x7f7aa4071180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80059d0 tx=0x7f7aa8005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:57.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.539+0000 7f7aaf7fe700 1 -- 172.21.15.151:0/1994965902 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7ab405b280 con 0x7f7ab8100dc0 2024-04-16T02:49:57.758 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.755+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ab81972b0 con 0x7f7aa406ecc0 2024-04-16T02:49:57.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.756+0000 7f7aaf7fe700 1 -- 172.21.15.151:0/1994965902 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7ab81972b0 con 0x7f7aa406ecc0 2024-04-16T02:49:57.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7aa406ecc0 msgr2=0x7f7aa4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:57.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7aa406ecc0 0x7f7aa4071180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80059d0 tx=0x7f7aa8005960 comp rx=0 tx=0).stop 2024-04-16T02:49:57.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8100dc0 msgr2=0x7f7ab819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:57.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8100dc0 0x7f7ab819c670 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400b4b0 tx=0x7f7ab400b870 comp rx=0 tx=0).stop 2024-04-16T02:49:57.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 shutdown_connections 2024-04-16T02:49:57.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7aa406ecc0 0x7f7aa4071180 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:57.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8100dc0 0x7f7ab819c670 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:57.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 --2- 172.21.15.151:0/1994965902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ab8101770 0x7f7ab819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:57.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 >> 172.21.15.151:0/1994965902 conn(0x7f7ab80fc650 msgr2=0x7f7ab80fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:57.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 shutdown_connections 2024-04-16T02:49:57.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.759+0000 7f7ac0439700 1 -- 172.21.15.151:0/1994965902 wait complete. 2024-04-16T02:49:57.773 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:49:57.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.865+0000 7ff90037f700 1 -- 172.21.15.151:0/2239553556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 msgr2=0x7ff8f81057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:57.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.865+0000 7ff90037f700 1 --2- 172.21.15.151:0/2239553556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 0x7ff8f81057e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009910 tx=0x7ff8e8009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:57.869 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:57 smithi151 ceph-mon[112233]: pgmap v339: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:57.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.865+0000 7ff90037f700 1 -- 172.21.15.151:0/2239553556 shutdown_connections 2024-04-16T02:49:57.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.865+0000 7ff90037f700 1 --2- 172.21.15.151:0/2239553556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 0x7ff8f81057e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:57.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.865+0000 7ff90037f700 1 --2- 172.21.15.151:0/2239553556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8100de0 0x7ff8f81011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:57.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.865+0000 7ff90037f700 1 -- 172.21.15.151:0/2239553556 >> 172.21.15.151:0/2239553556 conn(0x7ff8f80fc6b0 msgr2=0x7ff8f80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:57.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.865+0000 7ff90037f700 1 -- 172.21.15.151:0/2239553556 shutdown_connections 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.865+0000 7ff90037f700 1 -- 172.21.15.151:0/2239553556 wait complete. 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.866+0000 7ff90037f700 1 Processor -- start 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.866+0000 7ff90037f700 1 -- start start 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.866+0000 7ff90037f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8100de0 0x7ff8f8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.866+0000 7ff90037f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 0x7ff8f80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.866+0000 7ff90037f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8073b10 con 0x7ff8f8101790 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.866+0000 7ff90037f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8073c50 con 0x7ff8f8100de0 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.866+0000 7ff8fd91a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 0x7ff8f80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.866+0000 7ff8fd91a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 0x7ff8f80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43726/0 (socket says 172.21.15.151:43726) 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.866+0000 7ff8fd91a700 1 -- 172.21.15.151:0/4123882770 learned_addr learned my addr 172.21.15.151:0/4123882770 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff8fe11b700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8100de0 0x7ff8f8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:57.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff8fd91a700 1 -- 172.21.15.151:0/4123882770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8100de0 msgr2=0x7ff8f8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:57.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff8fd91a700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8100de0 0x7ff8f8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:57.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff8fd91a700 1 -- 172.21.15.151:0/4123882770 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e80095f0 con 0x7ff8f8101790 2024-04-16T02:49:57.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff8fd91a700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 0x7ff8f80735d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80092e0 tx=0x7ff8e800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:57.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff8ef7fe700 1 -- 172.21.15.151:0/4123882770 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8e801d070 con 0x7ff8f8101790 2024-04-16T02:49:57.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f8073ed0 con 0x7ff8f8101790 2024-04-16T02:49:57.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff8ef7fe700 1 -- 172.21.15.151:0/4123882770 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8e8005dc0 con 0x7ff8f8101790 2024-04-16T02:49:57.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff8ef7fe700 1 -- 172.21.15.151:0/4123882770 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8e8017400 con 0x7ff8f8101790 2024-04-16T02:49:57.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.867+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f81a2c50 con 0x7ff8f8101790 2024-04-16T02:49:57.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.868+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f81091e0 con 0x7ff8f8101790 2024-04-16T02:49:57.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.869+0000 7ff8ef7fe700 1 -- 172.21.15.151:0/4123882770 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff8e800f6f0 con 0x7ff8f8101790 2024-04-16T02:49:57.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.869+0000 7ff8ef7fe700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff8e406ecc0 0x7ff8e4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:57.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.869+0000 7ff8ef7fe700 1 -- 172.21.15.151:0/4123882770 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff8e8090250 con 0x7ff8f8101790 2024-04-16T02:49:57.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.869+0000 7ff8fe11b700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff8e406ecc0 0x7ff8e4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:57.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.870+0000 7ff8fe11b700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff8e406ecc0 0x7ff8e4071180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4005950 tx=0x7ff8f400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:57.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:57.874+0000 7ff8ef7fe700 1 -- 172.21.15.151:0/4123882770 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff8e8059000 con 0x7ff8f8101790 2024-04-16T02:49:58.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:57 smithi153 ceph-mon[87858]: pgmap v339: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:58.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.093+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8f802cce0 con 0x7ff8e406ecc0 2024-04-16T02:49:58.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.095+0000 7ff8ef7fe700 1 -- 172.21.15.151:0/4123882770 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff8f802cce0 con 0x7ff8e406ecc0 2024-04-16T02:49:58.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff8e406ecc0 msgr2=0x7ff8e4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff8e406ecc0 0x7ff8e4071180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4005950 tx=0x7ff8f400b410 comp rx=0 tx=0).stop 2024-04-16T02:49:58.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 msgr2=0x7ff8f80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 0x7ff8f80735d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80092e0 tx=0x7ff8e800bd70 comp rx=0 tx=0).stop 2024-04-16T02:49:58.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 shutdown_connections 2024-04-16T02:49:58.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff8e406ecc0 0x7ff8e4071180 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f8100de0 0x7ff8f8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 --2- 172.21.15.151:0/4123882770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8f8101790 0x7ff8f80735d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 >> 172.21.15.151:0/4123882770 conn(0x7ff8f80fc6b0 msgr2=0x7ff8f80fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:58.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 shutdown_connections 2024-04-16T02:49:58.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.097+0000 7ff90037f700 1 -- 172.21.15.151:0/4123882770 wait complete. 2024-04-16T02:49:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.201+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/141082013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68100da0 msgr2=0x7f7c68101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.201+0000 7f7c6e6bf700 1 --2- 172.21.15.151:0/141082013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68100da0 0x7f7c68101180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009910 tx=0x7f7c58009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.201+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/141082013 shutdown_connections 2024-04-16T02:49:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.201+0000 7f7c6e6bf700 1 --2- 172.21.15.151:0/141082013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c68101750 0x7f7c681057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.201+0000 7f7c6e6bf700 1 --2- 172.21.15.151:0/141082013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68100da0 0x7f7c68101180 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.201+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/141082013 >> 172.21.15.151:0/141082013 conn(0x7f7c680fc650 msgr2=0x7f7c680fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.201+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/141082013 shutdown_connections 2024-04-16T02:49:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.201+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/141082013 wait complete. 2024-04-16T02:49:58.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.202+0000 7f7c6e6bf700 1 Processor -- start 2024-04-16T02:49:58.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.202+0000 7f7c6e6bf700 1 -- start start 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.202+0000 7f7c6e6bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c68100da0 0x7f7c68073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.202+0000 7f7c6e6bf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68101750 0x7f7c680735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.202+0000 7f7c6e6bf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68073b10 con 0x7f7c68101750 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.202+0000 7f7c6e6bf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68073c50 con 0x7f7c68100da0 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.202+0000 7f7c677fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68101750 0x7f7c680735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c677fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68101750 0x7f7c680735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43750/0 (socket says 172.21.15.151:43750) 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c677fe700 1 -- 172.21.15.151:0/237860146 learned_addr learned my addr 172.21.15.151:0/237860146 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c67fff700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c68100da0 0x7f7c68073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c677fe700 1 -- 172.21.15.151:0/237860146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c68100da0 msgr2=0x7f7c68073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c677fe700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c68100da0 0x7f7c68073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c677fe700 1 -- 172.21.15.151:0/237860146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c580095f0 con 0x7f7c68101750 2024-04-16T02:49:58.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c67fff700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c68100da0 0x7f7c68073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:49:58.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c677fe700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68101750 0x7f7c680735d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c8f0 tx=0x7f7c5c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:58.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c657fa700 1 -- 172.21.15.151:0/237860146 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c5c007a50 con 0x7f7c68101750 2024-04-16T02:49:58.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c657fa700 1 -- 172.21.15.151:0/237860146 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c5c007bb0 con 0x7f7c68101750 2024-04-16T02:49:58.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c681a2920 con 0x7f7c68101750 2024-04-16T02:49:58.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c657fa700 1 -- 172.21.15.151:0/237860146 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c5c018630 con 0x7f7c68101750 2024-04-16T02:49:58.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.203+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c681a2d30 con 0x7f7c68101750 2024-04-16T02:49:58.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.204+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c68109140 con 0x7f7c68101750 2024-04-16T02:49:58.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.205+0000 7f7c657fa700 1 -- 172.21.15.151:0/237860146 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7c5c007d20 con 0x7f7c68101750 2024-04-16T02:49:58.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.205+0000 7f7c657fa700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7c5006ecc0 0x7f7c50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:58.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.206+0000 7f7c657fa700 1 -- 172.21.15.151:0/237860146 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7c5c08eeb0 con 0x7f7c68101750 2024-04-16T02:49:58.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.206+0000 7f7c67fff700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7c5006ecc0 0x7f7c50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:58.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.206+0000 7f7c67fff700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7c5006ecc0 0x7f7c50071180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7c580059d0 tx=0x7f7c5801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:58.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.210+0000 7f7c657fa700 1 -- 172.21.15.151:0/237860146 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7c5c057bc0 con 0x7f7c68101750 2024-04-16T02:49:58.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.419+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c6802cd30 con 0x7f7c5006ecc0 2024-04-16T02:49:58.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.430+0000 7f7c657fa700 1 -- 172.21.15.151:0/237860146 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7c6802cd30 con 0x7f7c5006ecc0 2024-04-16T02:49:58.433 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:49:58.433 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (11m) 4m ago 22m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:49:58.433 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (23m) 4m ago 23m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:49:58.433 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (22m) 4m ago 22m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (9m) 4m ago 23m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (8m) 4m ago 21m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (11m) 4m ago 22m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (5m) 4m ago 17m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (5m) 4m ago 17m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (17m) 4m ago 17m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (4m) 4m ago 17m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (13m) 4m ago 24m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (12m) 4m ago 21m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (10m) 4m ago 24m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (9m) 4m ago 21m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (12m) 4m ago 23m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (12m) 4m ago 21m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (8m) 4m ago 21m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (8m) 4m ago 20m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:49:58.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (7m) 4m ago 20m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:49:58.435 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (6m) 4m ago 19m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:49:58.435 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (6m) 4m ago 19m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:49:58.435 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (5m) 4m ago 18m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:49:58.435 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (11m) 4m ago 22m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7c5006ecc0 msgr2=0x7f7c50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7c5006ecc0 0x7f7c50071180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7c580059d0 tx=0x7f7c5801a040 comp rx=0 tx=0).stop 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68101750 msgr2=0x7f7c680735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68101750 0x7f7c680735d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c8f0 tx=0x7f7c5c00ccb0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 shutdown_connections 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7c5006ecc0 0x7f7c50071180 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c68100da0 0x7f7c68073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 --2- 172.21.15.151:0/237860146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7c68101750 0x7f7c680735d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 >> 172.21.15.151:0/237860146 conn(0x7f7c680fc650 msgr2=0x7f7c680fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 shutdown_connections 2024-04-16T02:49:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.432+0000 7f7c6e6bf700 1 -- 172.21.15.151:0/237860146 wait complete. 2024-04-16T02:49:58.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.537+0000 7fec33b42700 1 -- 172.21.15.151:0/575008629 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c105a40 msgr2=0x7fec2c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.537+0000 7fec33b42700 1 --2- 172.21.15.151:0/575008629 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c105a40 0x7fec2c105e20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec28009910 tx=0x7fec28009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:58.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.537+0000 7fec33b42700 1 -- 172.21.15.151:0/575008629 shutdown_connections 2024-04-16T02:49:58.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.537+0000 7fec33b42700 1 --2- 172.21.15.151:0/575008629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec2c0ffa40 0x7fec2c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.537+0000 7fec33b42700 1 --2- 172.21.15.151:0/575008629 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c105a40 0x7fec2c105e20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.537+0000 7fec33b42700 1 -- 172.21.15.151:0/575008629 >> 172.21.15.151:0/575008629 conn(0x7fec2c0fb570 msgr2=0x7fec2c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:58.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.538+0000 7fec33b42700 1 -- 172.21.15.151:0/575008629 shutdown_connections 2024-04-16T02:49:58.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.538+0000 7fec33b42700 1 -- 172.21.15.151:0/575008629 wait complete. 2024-04-16T02:49:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.538+0000 7fec33b42700 1 Processor -- start 2024-04-16T02:49:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.538+0000 7fec33b42700 1 -- start start 2024-04-16T02:49:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.538+0000 7fec33b42700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c0ffa40 0x7fec2c0fbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.538+0000 7fec33b42700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec2c105a40 0x7fec2c0fc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.538+0000 7fec33b42700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0fc900 con 0x7fec2c0ffa40 2024-04-16T02:49:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.538+0000 7fec33b42700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0fca70 con 0x7fec2c105a40 2024-04-16T02:49:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec318de700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c0ffa40 0x7fec2c0fbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec318de700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c0ffa40 0x7fec2c0fbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43760/0 (socket says 172.21.15.151:43760) 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec318de700 1 -- 172.21.15.151:0/1341885232 learned_addr learned my addr 172.21.15.151:0/1341885232 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec310dd700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec2c105a40 0x7fec2c0fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec318de700 1 -- 172.21.15.151:0/1341885232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec2c105a40 msgr2=0x7fec2c0fc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec318de700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec2c105a40 0x7fec2c0fc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec318de700 1 -- 172.21.15.151:0/1341885232 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec280095f0 con 0x7fec2c0ffa40 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec318de700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c0ffa40 0x7fec2c0fbdf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fec280092e0 tx=0x7fec2800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec1effd700 1 -- 172.21.15.151:0/1341885232 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec2801d070 con 0x7fec2c0ffa40 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c1a28e0 con 0x7fec2c0ffa40 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec1effd700 1 -- 172.21.15.151:0/1341885232 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec28005e00 con 0x7fec2c0ffa40 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec1effd700 1 -- 172.21.15.151:0/1341885232 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec28017400 con 0x7fec2c0ffa40 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.539+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c1a2c90 con 0x7fec2c0ffa40 2024-04-16T02:49:58.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.540+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c109050 con 0x7fec2c0ffa40 2024-04-16T02:49:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.541+0000 7fec1effd700 1 -- 172.21.15.151:0/1341885232 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fec2800f460 con 0x7fec2c0ffa40 2024-04-16T02:49:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.541+0000 7fec1effd700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec1806ecc0 0x7fec18071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.541+0000 7fec1effd700 1 -- 172.21.15.151:0/1341885232 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fec28090290 con 0x7fec2c0ffa40 2024-04-16T02:49:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.541+0000 7fec310dd700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec1806ecc0 0x7fec18071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.542+0000 7fec310dd700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec1806ecc0 0x7fec18071180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fec20005fd0 tx=0x7fec20005f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:58.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.546+0000 7fec1effd700 1 -- 172.21.15.151:0/1341885232 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fec280590c0 con 0x7fec2c0ffa40 2024-04-16T02:49:58.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.830+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fec2c04ea90 con 0x7fec2c0ffa40 2024-04-16T02:49:58.833 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:58 smithi151 ceph-mon[112233]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.830+0000 7fec1effd700 1 -- 172.21.15.151:0/1341885232 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fec2805c780 con 0x7fec2c0ffa40 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:49:58.834 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:49:58.835 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.833+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec1806ecc0 msgr2=0x7fec18071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.833+0000 7fec33b42700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec1806ecc0 0x7fec18071180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fec20005fd0 tx=0x7fec20005f60 comp rx=0 tx=0).stop 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.833+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c0ffa40 msgr2=0x7fec2c0fbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.833+0000 7fec33b42700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c0ffa40 0x7fec2c0fbdf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fec280092e0 tx=0x7fec2800be20 comp rx=0 tx=0).stop 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.834+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 shutdown_connections 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.834+0000 7fec33b42700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec1806ecc0 0x7fec18071180 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.834+0000 7fec33b42700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec2c0ffa40 0x7fec2c0fbdf0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.834+0000 7fec33b42700 1 --2- 172.21.15.151:0/1341885232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec2c105a40 0x7fec2c0fc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.834+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 >> 172.21.15.151:0/1341885232 conn(0x7fec2c0fb570 msgr2=0x7fec2c068fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.834+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 shutdown_connections 2024-04-16T02:49:58.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.834+0000 7fec33b42700 1 -- 172.21.15.151:0/1341885232 wait complete. 2024-04-16T02:49:58.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.939+0000 7fddd4739700 1 -- 172.21.15.151:0/2402030121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc0ffbd0 msgr2=0x7fddcc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.939+0000 7fddd4739700 1 --2- 172.21.15.151:0/2402030121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc0ffbd0 0x7fddcc100050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009910 tx=0x7fddc8009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:58.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.939+0000 7fddd4739700 1 -- 172.21.15.151:0/2402030121 shutdown_connections 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.939+0000 7fddd4739700 1 --2- 172.21.15.151:0/2402030121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc0ffbd0 0x7fddcc100050 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.939+0000 7fddd4739700 1 --2- 172.21.15.151:0/2402030121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddcc105a40 0x7fddcc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.939+0000 7fddd4739700 1 -- 172.21.15.151:0/2402030121 >> 172.21.15.151:0/2402030121 conn(0x7fddcc0755a0 msgr2=0x7fddcc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.939+0000 7fddd4739700 1 -- 172.21.15.151:0/2402030121 shutdown_connections 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.939+0000 7fddd4739700 1 -- 172.21.15.151:0/2402030121 wait complete. 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd4739700 1 Processor -- start 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd4739700 1 -- start start 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd4739700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddcc0ffbd0 0x7fddcc195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd4739700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc105a40 0x7fddcc195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd4739700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc196240 con 0x7fddcc105a40 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd4739700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc199fd0 con 0x7fddcc0ffbd0 2024-04-16T02:49:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd1cd4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc105a40 0x7fddcc195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd1cd4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc105a40 0x7fddcc195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43776/0 (socket says 172.21.15.151:43776) 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd1cd4700 1 -- 172.21.15.151:0/405275495 learned_addr learned my addr 172.21.15.151:0/405275495 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd24d5700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddcc0ffbd0 0x7fddcc195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd1cd4700 1 -- 172.21.15.151:0/405275495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddcc0ffbd0 msgr2=0x7fddcc195620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.940+0000 7fddd1cd4700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddcc0ffbd0 0x7fddcc195620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.941+0000 7fddd1cd4700 1 -- 172.21.15.151:0/405275495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc80095f0 con 0x7fddcc105a40 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.941+0000 7fddd1cd4700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc105a40 0x7fddcc195b60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fddc80092e0 tx=0x7fddc800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.941+0000 7fddc37fe700 1 -- 172.21.15.151:0/405275495 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc801f070 con 0x7fddcc105a40 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.941+0000 7fddc37fe700 1 -- 172.21.15.151:0/405275495 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddc8019610 con 0x7fddcc105a40 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.941+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc19a250 con 0x7fddcc105a40 2024-04-16T02:49:58.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.941+0000 7fddc37fe700 1 -- 172.21.15.151:0/405275495 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc800f670 con 0x7fddcc105a40 2024-04-16T02:49:58.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.941+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddcc19a740 con 0x7fddcc105a40 2024-04-16T02:49:58.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.942+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddcc04ea90 con 0x7fddcc105a40 2024-04-16T02:49:58.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.943+0000 7fddc37fe700 1 -- 172.21.15.151:0/405275495 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fddc8019780 con 0x7fddcc105a40 2024-04-16T02:49:58.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.943+0000 7fddc37fe700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddb806ecc0 0x7fddb8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:58.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.943+0000 7fddc37fe700 1 -- 172.21.15.151:0/405275495 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fddc8092630 con 0x7fddcc105a40 2024-04-16T02:49:58.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.943+0000 7fddd24d5700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddb806ecc0 0x7fddb8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:58.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.944+0000 7fddd24d5700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddb806ecc0 0x7fddb8071180 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fddbc005950 tx=0x7fddbc00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:58.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:58.948+0000 7fddc37fe700 1 -- 172.21.15.151:0/405275495 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fddc805b340 con 0x7fddcc105a40 2024-04-16T02:49:59.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:58 smithi153 ceph-mon[87858]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:59.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.191+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fddcc196980 con 0x7fddcc105a40 2024-04-16T02:49:59.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.191+0000 7fddc37fe700 1 -- 172.21.15.151:0/405275495 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7fddc805ea00 con 0x7fddcc105a40 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:49:59.195 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:49:59.196 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:49:59.197 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:49:59.198 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:49:59.198 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:49:59.198 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:49:59.198 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:49:59.198 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:49:59.198 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:49:59.198 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:49:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddb806ecc0 msgr2=0x7fddb8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddb806ecc0 0x7fddb8071180 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fddbc005950 tx=0x7fddbc00b410 comp rx=0 tx=0).stop 2024-04-16T02:49:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc105a40 msgr2=0x7fddcc195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc105a40 0x7fddcc195b60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fddc80092e0 tx=0x7fddc800bd70 comp rx=0 tx=0).stop 2024-04-16T02:49:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 shutdown_connections 2024-04-16T02:49:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddb806ecc0 0x7fddb8071180 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddcc0ffbd0 0x7fddcc195620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 --2- 172.21.15.151:0/405275495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddcc105a40 0x7fddcc195b60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 >> 172.21.15.151:0/405275495 conn(0x7fddcc0755a0 msgr2=0x7fddcc0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:59.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 shutdown_connections 2024-04-16T02:49:59.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.194+0000 7fddd4739700 1 -- 172.21.15.151:0/405275495 wait complete. 2024-04-16T02:49:59.200 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:49:59.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.300+0000 7f3e43109700 1 -- 172.21.15.151:0/3217838330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c0ffbd0 msgr2=0x7f3e3c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.300+0000 7f3e43109700 1 --2- 172.21.15.151:0/3217838330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c0ffbd0 0x7f3e3c100050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009910 tx=0x7f3e28009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:59.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.300+0000 7f3e43109700 1 -- 172.21.15.151:0/3217838330 shutdown_connections 2024-04-16T02:49:59.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.300+0000 7f3e43109700 1 --2- 172.21.15.151:0/3217838330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c0ffbd0 0x7f3e3c100050 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.300+0000 7f3e43109700 1 --2- 172.21.15.151:0/3217838330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c105a40 0x7f3e3c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.300+0000 7f3e43109700 1 -- 172.21.15.151:0/3217838330 >> 172.21.15.151:0/3217838330 conn(0x7f3e3c0755a0 msgr2=0x7f3e3c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:59.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.300+0000 7f3e43109700 1 -- 172.21.15.151:0/3217838330 shutdown_connections 2024-04-16T02:49:59.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.300+0000 7f3e43109700 1 -- 172.21.15.151:0/3217838330 wait complete. 2024-04-16T02:49:59.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e43109700 1 Processor -- start 2024-04-16T02:49:59.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e43109700 1 -- start start 2024-04-16T02:49:59.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e43109700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c0ffbd0 0x7f3e3c197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:59.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e43109700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c105a40 0x7f3e3c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:59.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e43109700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1980c0 con 0x7f3e3c105a40 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e43109700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c198230 con 0x7f3e3c0ffbd0 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e3bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c105a40 0x7f3e3c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e3bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c105a40 0x7f3e3c197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43790/0 (socket says 172.21.15.151:43790) 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e3bfff700 1 -- 172.21.15.151:0/3960736579 learned_addr learned my addr 172.21.15.151:0/3960736579 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e40ea5700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c0ffbd0 0x7f3e3c197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e3bfff700 1 -- 172.21.15.151:0/3960736579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c0ffbd0 msgr2=0x7f3e3c197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e3bfff700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c0ffbd0 0x7f3e3c197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e3bfff700 1 -- 172.21.15.151:0/3960736579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e280095f0 con 0x7f3e3c105a40 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.301+0000 7f3e3bfff700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c105a40 0x7f3e3c197aa0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3e28006010 tx=0x7f3e2800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.302+0000 7f3e39ffb700 1 -- 172.21.15.151:0/3960736579 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e2801d070 con 0x7f3e3c105a40 2024-04-16T02:49:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.302+0000 7f3e39ffb700 1 -- 172.21.15.151:0/3960736579 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e2800f460 con 0x7f3e3c105a40 2024-04-16T02:49:59.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.302+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c1a05d0 con 0x7f3e3c105a40 2024-04-16T02:49:59.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.302+0000 7f3e39ffb700 1 -- 172.21.15.151:0/3960736579 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e28017610 con 0x7f3e3c105a40 2024-04-16T02:49:59.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.302+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c1a0a40 con 0x7f3e3c105a40 2024-04-16T02:49:59.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.303+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c107fd0 con 0x7f3e3c105a40 2024-04-16T02:49:59.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.303+0000 7f3e39ffb700 1 -- 172.21.15.151:0/3960736579 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3e2800f5d0 con 0x7f3e3c105a40 2024-04-16T02:49:59.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.304+0000 7f3e39ffb700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:59.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.304+0000 7f3e39ffb700 1 -- 172.21.15.151:0/3960736579 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3e280903e0 con 0x7f3e3c105a40 2024-04-16T02:49:59.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.304+0000 7f3e40ea5700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:59.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.304+0000 7f3e40ea5700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3e30005950 tx=0x7f3e3000a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:59.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.309+0000 7f3e39ffb700 1 -- 172.21.15.151:0/3960736579 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3e28059190 con 0x7f3e3c105a40 2024-04-16T02:49:59.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.527+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e3c02cd30 con 0x7f3e2c06ecc0 2024-04-16T02:49:59.531 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.528+0000 7f3e39ffb700 1 -- 172.21.15.151:0/3960736579 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3e3c02cd30 con 0x7f3e2c06ecc0 2024-04-16T02:49:59.531 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:49:59.531 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:49:59.531 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:49:59.531 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:49:59.531 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:49:59.531 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:49:59.532 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:49:59.532 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:49:59.532 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:49:59.532 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:49:59.532 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:49:59.532 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:49:59.532 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:49:59.532 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:49:59.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 msgr2=0x7f3e2c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3e30005950 tx=0x7f3e3000a300 comp rx=0 tx=0).stop 2024-04-16T02:49:59.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c105a40 msgr2=0x7f3e3c197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c105a40 0x7f3e3c197aa0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3e28006010 tx=0x7f3e2800bd70 comp rx=0 tx=0).stop 2024-04-16T02:49:59.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 shutdown_connections 2024-04-16T02:49:59.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e2c06ecc0 0x7f3e2c071180 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c0ffbd0 0x7f3e3c197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 --2- 172.21.15.151:0/3960736579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e3c105a40 0x7f3e3c197aa0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 >> 172.21.15.151:0/3960736579 conn(0x7f3e3c0755a0 msgr2=0x7f3e3c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:59.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 shutdown_connections 2024-04-16T02:49:59.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.530+0000 7f3e43109700 1 -- 172.21.15.151:0/3960736579 wait complete. 2024-04-16T02:49:59.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.637+0000 7f6bbe694700 1 -- 172.21.15.151:0/3353219364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 msgr2=0x7f6bb8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.637+0000 7f6bbe694700 1 --2- 172.21.15.151:0/3353219364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 0x7f6bb8100050 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009910 tx=0x7f6ba0009c20 comp rx=0 tx=0).stop 2024-04-16T02:49:59.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.637+0000 7f6bbe694700 1 -- 172.21.15.151:0/3353219364 shutdown_connections 2024-04-16T02:49:59.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.637+0000 7f6bbe694700 1 --2- 172.21.15.151:0/3353219364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 0x7f6bb8100050 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.637+0000 7f6bbe694700 1 --2- 172.21.15.151:0/3353219364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8105a40 0x7f6bb8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.637+0000 7f6bbe694700 1 -- 172.21.15.151:0/3353219364 >> 172.21.15.151:0/3353219364 conn(0x7f6bb80755a0 msgr2=0x7f6bb80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:59.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.637+0000 7f6bbe694700 1 -- 172.21.15.151:0/3353219364 shutdown_connections 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.637+0000 7f6bbe694700 1 -- 172.21.15.151:0/3353219364 wait complete. 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bbe694700 1 Processor -- start 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bbe694700 1 -- start start 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bbe694700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 0x7f6bb819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bbe694700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8105a40 0x7f6bb819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bbe694700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb819d0d0 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bbe694700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8196580 con 0x7f6bb8105a40 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bb7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 0x7f6bb819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bb7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 0x7f6bb819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43810/0 (socket says 172.21.15.151:43810) 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bb7fff700 1 -- 172.21.15.151:0/3562736438 learned_addr learned my addr 172.21.15.151:0/3562736438 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6baffff700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8105a40 0x7f6bb819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bb7fff700 1 -- 172.21.15.151:0/3562736438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8105a40 msgr2=0x7f6bb819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bb7fff700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8105a40 0x7f6bb819ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.638+0000 7f6bb7fff700 1 -- 172.21.15.151:0/3562736438 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba00095f0 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.639+0000 7f6bb7fff700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 0x7f6bb819c500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800c8a0 tx=0x7f6ba800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:59.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.639+0000 7f6bb5ffb700 1 -- 172.21.15.151:0/3562736438 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba80078e0 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.639+0000 7f6bb5ffb700 1 -- 172.21.15.151:0/3562736438 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ba800f450 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.639+0000 7f6bb5ffb700 1 -- 172.21.15.151:0/3562736438 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba800e5e0 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.639+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb8196860 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.639+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb8196db0 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.640+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb8108fe0 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.640+0000 7f6bb5ffb700 1 -- 172.21.15.151:0/3562736438 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6ba800f5c0 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.641+0000 7f6bb5ffb700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ba406ecc0 0x7f6ba4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:49:59.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.641+0000 7f6baffff700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ba406ecc0 0x7f6ba4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:49:59.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.641+0000 7f6bb5ffb700 1 -- 172.21.15.151:0/3562736438 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6ba808ecc0 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.641+0000 7f6baffff700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ba406ecc0 0x7f6ba4071180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8104d90 tx=0x7f6ba001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:49:59.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.646+0000 7f6bb5ffb700 1 -- 172.21.15.151:0/3562736438 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6ba805b620 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.925+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6bb8197100 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:59 smithi151 ceph-mon[112233]: pgmap v340: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:49:59.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:59 smithi151 ceph-mon[112233]: from='client.34682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:59.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:59 smithi151 ceph-mon[112233]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:49:59.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:59 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1341885232' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:49:59.929 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:49:59 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/405275495' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:49:59.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.926+0000 7f6bb5ffb700 1 -- 172.21.15.151:0/3562736438 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6ba805b1b0 con 0x7f6bb80ffbd0 2024-04-16T02:49:59.929 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:49:59.930 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:49:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:49:59.930 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:49:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:49:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ba406ecc0 msgr2=0x7f6ba4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ba406ecc0 0x7f6ba4071180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8104d90 tx=0x7f6ba001a040 comp rx=0 tx=0).stop 2024-04-16T02:49:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 msgr2=0x7f6bb819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:49:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 0x7f6bb819c500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800c8a0 tx=0x7f6ba800cbb0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 shutdown_connections 2024-04-16T02:49:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ba406ecc0 0x7f6ba4071180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bb80ffbd0 0x7f6bb819c500 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 --2- 172.21.15.151:0/3562736438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8105a40 0x7f6bb819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:49:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 >> 172.21.15.151:0/3562736438 conn(0x7f6bb80755a0 msgr2=0x7f6bb8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:49:59.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 shutdown_connections 2024-04-16T02:49:59.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:49:59.928+0000 7f6bbe694700 1 -- 172.21.15.151:0/3562736438 wait complete. 2024-04-16T02:50:00.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:59 smithi153 ceph-mon[87858]: pgmap v340: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:00.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:59 smithi153 ceph-mon[87858]: from='client.34682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:00.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:59 smithi153 ceph-mon[87858]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:00.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:59 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1341885232' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:50:00.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:49:59 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/405275495' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:50:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T02:49:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:50:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T02:49:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:50:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T02:49:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-16T02:50:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T02:49:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:50:00.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T02:49:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-16T02:50:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-mon[112233]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3562736438' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:50:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-mon[112233]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:50:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-mon[112233]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:50:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-mon[112233]: fs cephfs is degraded 2024-04-16T02:50:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-mon[112233]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:50:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:00 smithi151 ceph-mon[112233]: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:50:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:00 smithi153 ceph-mon[87858]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:00 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3562736438' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:50:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:00 smithi153 ceph-mon[87858]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:50:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:00 smithi153 ceph-mon[87858]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:50:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:00 smithi153 ceph-mon[87858]: fs cephfs is degraded 2024-04-16T02:50:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:00 smithi153 ceph-mon[87858]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:50:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:00 smithi153 ceph-mon[87858]: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:50:01.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:01 smithi151 ceph-mon[112233]: pgmap v341: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:02.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:01 smithi153 ceph-mon[87858]: pgmap v341: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:03.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:03 smithi151 ceph-mon[112233]: pgmap v342: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:04.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:03 smithi153 ceph-mon[87858]: pgmap v342: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:05.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:05 smithi151 ceph-mon[112233]: pgmap v343: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:05 smithi153 ceph-mon[87858]: pgmap v343: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:07.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:07 smithi151 ceph-mon[112233]: pgmap v344: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:08.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:07 smithi153 ceph-mon[87858]: pgmap v344: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:09 smithi151 ceph-mon[112233]: pgmap v345: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:09.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:50:10.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:09 smithi153 ceph-mon[87858]: pgmap v345: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:10.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:50:11.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:11 smithi151 ceph-mon[112233]: pgmap v346: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:12.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:11 smithi153 ceph-mon[87858]: pgmap v346: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:13.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:13 smithi151 ceph-mon[112233]: pgmap v347: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:14.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:13 smithi153 ceph-mon[87858]: pgmap v347: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:15.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:15 smithi151 ceph-mon[112233]: pgmap v348: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:16.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:15 smithi153 ceph-mon[87858]: pgmap v348: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:17.981 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:17 smithi153 ceph-mon[87858]: pgmap v349: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:17.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:17 smithi151 ceph-mon[112233]: pgmap v349: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:19.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:19 smithi151 ceph-mon[112233]: pgmap v350: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:19 smithi153 ceph-mon[87858]: pgmap v350: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:21.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:21 smithi151 ceph-mon[112233]: pgmap v351: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:22.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:21 smithi153 ceph-mon[87858]: pgmap v351: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:23 smithi151 ceph-mon[112233]: pgmap v352: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:24.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:23 smithi153 ceph-mon[87858]: pgmap v352: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:50:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:50:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:25 smithi151 ceph-mon[112233]: pgmap v353: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:26.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:25 smithi153 ceph-mon[87858]: pgmap v353: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:27.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:27 smithi151 ceph-mon[112233]: pgmap v354: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:28.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:27 smithi153 ceph-mon[87858]: pgmap v354: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:29.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:29 smithi151 ceph-mon[112233]: pgmap v355: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:30.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:29 smithi153 ceph-mon[87858]: pgmap v355: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.052+0000 7f55e7faf700 1 -- 172.21.15.151:0/3467160105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e0100d70 msgr2=0x7f55e01048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.052+0000 7f55e7faf700 1 --2- 172.21.15.151:0/3467160105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e0100d70 0x7f55e01048a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f55d4009910 tx=0x7f55d4009c20 comp rx=0 tx=0).stop 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.052+0000 7f55e7faf700 1 -- 172.21.15.151:0/3467160105 shutdown_connections 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.052+0000 7f55e7faf700 1 --2- 172.21.15.151:0/3467160105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e0100d70 0x7f55e01048a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.052+0000 7f55e7faf700 1 --2- 172.21.15.151:0/3467160105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e01003c0 0x7f55e01007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.052+0000 7f55e7faf700 1 -- 172.21.15.151:0/3467160105 >> 172.21.15.151:0/3467160105 conn(0x7f55e0075e30 msgr2=0x7f55e00fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.053+0000 7f55e7faf700 1 -- 172.21.15.151:0/3467160105 shutdown_connections 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.053+0000 7f55e7faf700 1 -- 172.21.15.151:0/3467160105 wait complete. 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.053+0000 7f55e7faf700 1 Processor -- start 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e7faf700 1 -- start start 2024-04-16T02:50:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e7faf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e01003c0 0x7f55e019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e7faf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e0100d70 0x7f55e019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e7faf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e019d2c0 con 0x7f55e01003c0 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e7faf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e0196770 con 0x7f55e0100d70 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e5d4b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e01003c0 0x7f55e019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e5d4b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e01003c0 0x7f55e019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40618/0 (socket says 172.21.15.151:40618) 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e5d4b700 1 -- 172.21.15.151:0/447180943 learned_addr learned my addr 172.21.15.151:0/447180943 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e554a700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e0100d70 0x7f55e019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e5d4b700 1 -- 172.21.15.151:0/447180943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e0100d70 msgr2=0x7f55e019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e5d4b700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e0100d70 0x7f55e019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.054+0000 7f55e5d4b700 1 -- 172.21.15.151:0/447180943 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40095f0 con 0x7f55e01003c0 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.055+0000 7f55e5d4b700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e01003c0 0x7f55e019c6f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f55d00060b0 tx=0x7f55d000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.055+0000 7f55deffd700 1 -- 172.21.15.151:0/447180943 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55d000dda0 con 0x7f55e01003c0 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.055+0000 7f55deffd700 1 -- 172.21.15.151:0/447180943 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55d000f040 con 0x7f55e01003c0 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.055+0000 7f55deffd700 1 -- 172.21.15.151:0/447180943 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55d0014790 con 0x7f55e01003c0 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.055+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e0196a50 con 0x7f55e01003c0 2024-04-16T02:50:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.055+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e0196fa0 con 0x7f55e01003c0 2024-04-16T02:50:30.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.056+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e004ea90 con 0x7f55e01003c0 2024-04-16T02:50:30.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.057+0000 7f55deffd700 1 -- 172.21.15.151:0/447180943 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f55d0014980 con 0x7f55e01003c0 2024-04-16T02:50:30.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.057+0000 7f55deffd700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55c806ecc0 0x7f55c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:30.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.057+0000 7f55deffd700 1 -- 172.21.15.151:0/447180943 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f55d008e900 con 0x7f55e01003c0 2024-04-16T02:50:30.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.057+0000 7f55e554a700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55c806ecc0 0x7f55c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:30.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.058+0000 7f55e554a700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55c806ecc0 0x7f55c8071180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f55e0197be0 tx=0x7f55d401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:30.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.062+0000 7f55deffd700 1 -- 172.21.15.151:0/447180943 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f55d005b290 con 0x7f55e01003c0 2024-04-16T02:50:30.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.281+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f55e0197220 con 0x7f55c806ecc0 2024-04-16T02:50:30.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.282+0000 7f55deffd700 1 -- 172.21.15.151:0/447180943 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f55e0197220 con 0x7f55c806ecc0 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.284+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55c806ecc0 msgr2=0x7f55c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.284+0000 7f55e7faf700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55c806ecc0 0x7f55c8071180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f55e0197be0 tx=0x7f55d401a040 comp rx=0 tx=0).stop 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.284+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e01003c0 msgr2=0x7f55e019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.284+0000 7f55e7faf700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e01003c0 0x7f55e019c6f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f55d00060b0 tx=0x7f55d000d3b0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.285+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 shutdown_connections 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.285+0000 7f55e7faf700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55c806ecc0 0x7f55c8071180 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.285+0000 7f55e7faf700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55e01003c0 0x7f55e019c6f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.285+0000 7f55e7faf700 1 --2- 172.21.15.151:0/447180943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e0100d70 0x7f55e019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.285+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 >> 172.21.15.151:0/447180943 conn(0x7f55e0075e30 msgr2=0x7f55e00fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.285+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 shutdown_connections 2024-04-16T02:50:30.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.285+0000 7f55e7faf700 1 -- 172.21.15.151:0/447180943 wait complete. 2024-04-16T02:50:30.298 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:50:30.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.391+0000 7f8c05aba700 1 -- 172.21.15.151:0/4255081507 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ff530 msgr2=0x7f8c000ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.391+0000 7f8c05aba700 1 --2- 172.21.15.151:0/4255081507 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ff530 0x7f8c000ff910 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0009910 tx=0x7f8bf0009c20 comp rx=0 tx=0).stop 2024-04-16T02:50:30.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.392+0000 7f8c05aba700 1 -- 172.21.15.151:0/4255081507 shutdown_connections 2024-04-16T02:50:30.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.392+0000 7f8c05aba700 1 --2- 172.21.15.151:0/4255081507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c000ffee0 0x7f8c00105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.392+0000 7f8c05aba700 1 --2- 172.21.15.151:0/4255081507 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ff530 0x7f8c000ff910 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.392+0000 7f8c05aba700 1 -- 172.21.15.151:0/4255081507 >> 172.21.15.151:0/4255081507 conn(0x7f8c00074df0 msgr2=0x7f8c00075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:30.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.392+0000 7f8c05aba700 1 -- 172.21.15.151:0/4255081507 shutdown_connections 2024-04-16T02:50:30.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.392+0000 7f8c05aba700 1 -- 172.21.15.151:0/4255081507 wait complete. 2024-04-16T02:50:30.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.392+0000 7f8c05aba700 1 Processor -- start 2024-04-16T02:50:30.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8c05aba700 1 -- start start 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8c05aba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c000ff530 0x7f8c001004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8c05aba700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ffee0 0x7f8c001009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8c05aba700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c00100f30 con 0x7f8c000ffee0 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8c05aba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c00101070 con 0x7f8c000ff530 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8bfeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ffee0 0x7f8c001009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8bfeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ffee0 0x7f8c001009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40632/0 (socket says 172.21.15.151:40632) 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8bfeffd700 1 -- 172.21.15.151:0/3690727958 learned_addr learned my addr 172.21.15.151:0/3690727958 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8bff7fe700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c000ff530 0x7f8c001004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8bfeffd700 1 -- 172.21.15.151:0/3690727958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c000ff530 msgr2=0x7f8c001004b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8bfeffd700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c000ff530 0x7f8c001004b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8bfeffd700 1 -- 172.21.15.151:0/3690727958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf00095f0 con 0x7f8c000ffee0 2024-04-16T02:50:30.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8bff7fe700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c000ff530 0x7f8c001004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:50:30.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.393+0000 7f8bfeffd700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ffee0 0x7f8c001009f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400c960 tx=0x7f8bf400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:30.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.394+0000 7f8bfcff9700 1 -- 172.21.15.151:0/3690727958 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bf4007a70 con 0x7f8c000ffee0 2024-04-16T02:50:30.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.394+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c001a2990 con 0x7f8c000ffee0 2024-04-16T02:50:30.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.394+0000 7f8bfcff9700 1 -- 172.21.15.151:0/3690727958 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bf4007bd0 con 0x7f8c000ffee0 2024-04-16T02:50:30.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.394+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c001a2cf0 con 0x7f8c000ffee0 2024-04-16T02:50:30.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.394+0000 7f8bfcff9700 1 -- 172.21.15.151:0/3690727958 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bf40186c0 con 0x7f8c000ffee0 2024-04-16T02:50:30.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.395+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c001091e0 con 0x7f8c000ffee0 2024-04-16T02:50:30.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.395+0000 7f8bfcff9700 1 -- 172.21.15.151:0/3690727958 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8bf400faa0 con 0x7f8c000ffee0 2024-04-16T02:50:30.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.396+0000 7f8bfcff9700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8be806ecc0 0x7f8be8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:30.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.396+0000 7f8bfcff9700 1 -- 172.21.15.151:0/3690727958 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8bf408ed60 con 0x7f8c000ffee0 2024-04-16T02:50:30.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.396+0000 7f8bff7fe700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8be806ecc0 0x7f8be8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:30.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.396+0000 7f8bff7fe700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8be806ecc0 0x7f8be8071180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0006010 tx=0x7f8bf000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:30.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.401+0000 7f8bfcff9700 1 -- 172.21.15.151:0/3690727958 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8bf405b6c0 con 0x7f8c000ffee0 2024-04-16T02:50:30.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.620+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c0002cce0 con 0x7f8be806ecc0 2024-04-16T02:50:30.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.621+0000 7f8bfcff9700 1 -- 172.21.15.151:0/3690727958 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8c0002cce0 con 0x7f8be806ecc0 2024-04-16T02:50:30.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.623+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8be806ecc0 msgr2=0x7f8be8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.623+0000 7f8c05aba700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8be806ecc0 0x7f8be8071180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0006010 tx=0x7f8bf000b540 comp rx=0 tx=0).stop 2024-04-16T02:50:30.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.623+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ffee0 msgr2=0x7f8c001009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.623+0000 7f8c05aba700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ffee0 0x7f8c001009f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400c960 tx=0x7f8bf400cd20 comp rx=0 tx=0).stop 2024-04-16T02:50:30.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.623+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 shutdown_connections 2024-04-16T02:50:30.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.623+0000 7f8c05aba700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8be806ecc0 0x7f8be8071180 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.623+0000 7f8c05aba700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c000ff530 0x7f8c001004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.623+0000 7f8c05aba700 1 --2- 172.21.15.151:0/3690727958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c000ffee0 0x7f8c001009f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.623+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 >> 172.21.15.151:0/3690727958 conn(0x7f8c00074df0 msgr2=0x7f8c000feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:30.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.624+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 shutdown_connections 2024-04-16T02:50:30.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.624+0000 7f8c05aba700 1 -- 172.21.15.151:0/3690727958 wait complete. 2024-04-16T02:50:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 -- 172.21.15.151:0/122145490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 msgr2=0x7ff7000ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 --2- 172.21.15.151:0/122145490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 0x7ff7000ffec0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009910 tx=0x7ff6f4009c20 comp rx=0 tx=0).stop 2024-04-16T02:50:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 -- 172.21.15.151:0/122145490 shutdown_connections 2024-04-16T02:50:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 --2- 172.21.15.151:0/122145490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 0x7ff7000ffec0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 --2- 172.21.15.151:0/122145490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff700105a40 0x7ff700105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 -- 172.21.15.151:0/122145490 >> 172.21.15.151:0/122145490 conn(0x7ff7000fb550 msgr2=0x7ff7000fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 -- 172.21.15.151:0/122145490 shutdown_connections 2024-04-16T02:50:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 -- 172.21.15.151:0/122145490 wait complete. 2024-04-16T02:50:30.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 Processor -- start 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.731+0000 7ff707744700 1 -- start start 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff707744700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 0x7ff7001953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff707744700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff700105a40 0x7ff7001958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff707744700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff700195e20 con 0x7ff7000ffa40 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff707744700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff700195f90 con 0x7ff700105a40 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff7054e0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 0x7ff7001953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff7054e0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 0x7ff7001953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40650/0 (socket says 172.21.15.151:40650) 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff7054e0700 1 -- 172.21.15.151:0/2740143625 learned_addr learned my addr 172.21.15.151:0/2740143625 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff704cdf700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff700105a40 0x7ff7001958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff7054e0700 1 -- 172.21.15.151:0/2740143625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff700105a40 msgr2=0x7ff7001958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff7054e0700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff700105a40 0x7ff7001958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff7054e0700 1 -- 172.21.15.151:0/2740143625 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f40095f0 con 0x7ff7000ffa40 2024-04-16T02:50:30.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff7054e0700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 0x7ff7001953a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00060b0 tx=0x7ff6f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:30.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff6fe7fc700 1 -- 172.21.15.151:0/2740143625 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6f000dda0 con 0x7ff7000ffa40 2024-04-16T02:50:30.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff6fe7fc700 1 -- 172.21.15.151:0/2740143625 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6f000f040 con 0x7ff7000ffa40 2024-04-16T02:50:30.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7001a2a10 con 0x7ff7000ffa40 2024-04-16T02:50:30.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.732+0000 7ff6fe7fc700 1 -- 172.21.15.151:0/2740143625 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6f00146f0 con 0x7ff7000ffa40 2024-04-16T02:50:30.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.733+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7001a2d70 con 0x7ff7000ffa40 2024-04-16T02:50:30.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.733+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff700107fd0 con 0x7ff7000ffa40 2024-04-16T02:50:30.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.734+0000 7ff6fe7fc700 1 -- 172.21.15.151:0/2740143625 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff6f0015400 con 0x7ff7000ffa40 2024-04-16T02:50:30.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.735+0000 7ff6fe7fc700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff6e806ecc0 0x7ff6e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:30.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.735+0000 7ff704cdf700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff6e806ecc0 0x7ff6e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:30.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.735+0000 7ff6fe7fc700 1 -- 172.21.15.151:0/2740143625 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff6f008e740 con 0x7ff7000ffa40 2024-04-16T02:50:30.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.735+0000 7ff704cdf700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff6e806ecc0 0x7ff6e8071180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff700069710 tx=0x7ff6f401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:30.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.739+0000 7ff6fe7fc700 1 -- 172.21.15.151:0/2740143625 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff6f005b150 con 0x7ff7000ffa40 2024-04-16T02:50:30.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.947+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff70002cd80 con 0x7ff6e806ecc0 2024-04-16T02:50:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.959+0000 7ff6fe7fc700 1 -- 172.21.15.151:0/2740143625 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff70002cd80 con 0x7ff6e806ecc0 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (12m) 5m ago 23m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (23m) 5m ago 23m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (22m) 5m ago 22m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (9m) 5m ago 23m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (9m) 5m ago 22m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (11m) 5m ago 23m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (5m) 5m ago 17m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (5m) 5m ago 17m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (17m) 5m ago 17m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (5m) 5m ago 17m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (14m) 5m ago 24m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:50:30.962 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (13m) 5m ago 22m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (10m) 5m ago 24m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (10m) 5m ago 22m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (12m) 5m ago 23m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (12m) 5m ago 22m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (9m) 5m ago 21m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (8m) 5m ago 21m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (7m) 5m ago 20m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (7m) 5m ago 20m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (6m) 5m ago 19m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (6m) 5m ago 19m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:50:30.963 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (12m) 5m ago 22m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:50:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff6e806ecc0 msgr2=0x7ff6e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff6e806ecc0 0x7ff6e8071180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff700069710 tx=0x7ff6f401a040 comp rx=0 tx=0).stop 2024-04-16T02:50:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 msgr2=0x7ff7001953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 0x7ff7001953a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00060b0 tx=0x7ff6f000d3b0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 shutdown_connections 2024-04-16T02:50:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff6e806ecc0 0x7ff6e8071180 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7000ffa40 0x7ff7001953a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 --2- 172.21.15.151:0/2740143625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff700105a40 0x7ff7001958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 >> 172.21.15.151:0/2740143625 conn(0x7ff7000fb550 msgr2=0x7ff7000fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:30.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 shutdown_connections 2024-04-16T02:50:30.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:30.961+0000 7ff707744700 1 -- 172.21.15.151:0/2740143625 wait complete. 2024-04-16T02:50:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.065+0000 7f5511b21700 1 -- 172.21.15.151:0/4112535004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 msgr2=0x7f550c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.065+0000 7f5511b21700 1 --2- 172.21.15.151:0/4112535004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 0x7f550c106ff0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00b0a0 tx=0x7f54fc00b3b0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.065+0000 7f5511b21700 1 -- 172.21.15.151:0/4112535004 shutdown_connections 2024-04-16T02:50:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.065+0000 7f5511b21700 1 --2- 172.21.15.151:0/4112535004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c0ff310 0x7f550c0ff770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.065+0000 7f5511b21700 1 --2- 172.21.15.151:0/4112535004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 0x7f550c106ff0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.065+0000 7f5511b21700 1 -- 172.21.15.151:0/4112535004 >> 172.21.15.151:0/4112535004 conn(0x7f550c074bf0 msgr2=0x7f550c075000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.065+0000 7f5511b21700 1 -- 172.21.15.151:0/4112535004 shutdown_connections 2024-04-16T02:50:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.065+0000 7f5511b21700 1 -- 172.21.15.151:0/4112535004 wait complete. 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.065+0000 7f5511b21700 1 Processor -- start 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f5511b21700 1 -- start start 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f5511b21700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c0ff310 0x7f550c1966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f5511b21700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 0x7f550c196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f5511b21700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c1972c0 con 0x7f550c106c10 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f5511b21700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c199ff0 con 0x7f550c0ff310 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f550affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 0x7f550c196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f550affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 0x7f550c196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40676/0 (socket says 172.21.15.151:40676) 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f550affd700 1 -- 172.21.15.151:0/3106410286 learned_addr learned my addr 172.21.15.151:0/3106410286 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f550b7fe700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c0ff310 0x7f550c1966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f550affd700 1 -- 172.21.15.151:0/3106410286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c0ff310 msgr2=0x7f550c1966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f550affd700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c0ff310 0x7f550c1966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f550affd700 1 -- 172.21.15.151:0/3106410286 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc009d00 con 0x7f550c106c10 2024-04-16T02:50:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.066+0000 7f550affd700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 0x7f550c196c30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f550000d900 tx=0x7f550000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.067+0000 7f5508ff9700 1 -- 172.21.15.151:0/3106410286 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55000041f0 con 0x7f550c106c10 2024-04-16T02:50:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.067+0000 7f5508ff9700 1 -- 172.21.15.151:0/3106410286 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5500004cf0 con 0x7f550c106c10 2024-04-16T02:50:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.067+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550c19a2d0 con 0x7f550c106c10 2024-04-16T02:50:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.067+0000 7f5508ff9700 1 -- 172.21.15.151:0/3106410286 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f550000b750 con 0x7f550c106c10 2024-04-16T02:50:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.067+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550c19a820 con 0x7f550c106c10 2024-04-16T02:50:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.068+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550c109110 con 0x7f550c106c10 2024-04-16T02:50:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.068+0000 7f5508ff9700 1 -- 172.21.15.151:0/3106410286 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5500004750 con 0x7f550c106c10 2024-04-16T02:50:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.069+0000 7f5508ff9700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f54f406ecc0 0x7f54f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.069+0000 7f550b7fe700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f54f406ecc0 0x7f54f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.069+0000 7f5508ff9700 1 -- 172.21.15.151:0/3106410286 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f550008ec60 con 0x7f550c106c10 2024-04-16T02:50:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.069+0000 7f550b7fe700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f54f406ecc0 0x7f54f4071180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f54fc000f80 tx=0x7f54fc0060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:31.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.074+0000 7f5508ff9700 1 -- 172.21.15.151:0/3106410286 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5500057a90 con 0x7f550c106c10 2024-04-16T02:50:31.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.361+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f550c04ea90 con 0x7f550c106c10 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.361+0000 7f5508ff9700 1 -- 172.21.15.151:0/3106410286 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f5500016340 con 0x7f550c106c10 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:50:31.364 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:50:31.365 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:50:31.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f54f406ecc0 msgr2=0x7f54f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f54f406ecc0 0x7f54f4071180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f54fc000f80 tx=0x7f54fc0060c0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 msgr2=0x7f550c196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 0x7f550c196c30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f550000d900 tx=0x7f550000dc10 comp rx=0 tx=0).stop 2024-04-16T02:50:31.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 shutdown_connections 2024-04-16T02:50:31.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f54f406ecc0 0x7f54f4071180 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c0ff310 0x7f550c1966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 --2- 172.21.15.151:0/3106410286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f550c106c10 0x7f550c196c30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 >> 172.21.15.151:0/3106410286 conn(0x7f550c074bf0 msgr2=0x7f550c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:31.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 shutdown_connections 2024-04-16T02:50:31.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.363+0000 7f5511b21700 1 -- 172.21.15.151:0/3106410286 wait complete. 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.469+0000 7fd44ff27700 1 -- 172.21.15.151:0/629263862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 msgr2=0x7fd4481057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.469+0000 7fd44ff27700 1 --2- 172.21.15.151:0/629263862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 0x7fd4481057e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd438009910 tx=0x7fd438009c20 comp rx=0 tx=0).stop 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.469+0000 7fd44ff27700 1 -- 172.21.15.151:0/629263862 shutdown_connections 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.469+0000 7fd44ff27700 1 --2- 172.21.15.151:0/629263862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 0x7fd4481057e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.469+0000 7fd44ff27700 1 --2- 172.21.15.151:0/629263862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd448100de0 0x7fd4481011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.469+0000 7fd44ff27700 1 -- 172.21.15.151:0/629263862 >> 172.21.15.151:0/629263862 conn(0x7fd4480fc670 msgr2=0x7fd4480fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.469+0000 7fd44ff27700 1 -- 172.21.15.151:0/629263862 shutdown_connections 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.469+0000 7fd44ff27700 1 -- 172.21.15.151:0/629263862 wait complete. 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44ff27700 1 Processor -- start 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44ff27700 1 -- start start 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44ff27700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd448100de0 0x7fd448073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44ff27700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 0x7fd4480735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:31.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44ff27700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd448073b10 con 0x7fd448101790 2024-04-16T02:50:31.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44ff27700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd448073c50 con 0x7fd448100de0 2024-04-16T02:50:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44d4c2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 0x7fd4480735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44d4c2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 0x7fd4480735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40686/0 (socket says 172.21.15.151:40686) 2024-04-16T02:50:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44d4c2700 1 -- 172.21.15.151:0/3465451094 learned_addr learned my addr 172.21.15.151:0/3465451094 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:50:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44dcc3700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd448100de0 0x7fd448073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44d4c2700 1 -- 172.21.15.151:0/3465451094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd448100de0 msgr2=0x7fd448073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.470+0000 7fd44d4c2700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd448100de0 0x7fd448073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.471+0000 7fd44d4c2700 1 -- 172.21.15.151:0/3465451094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4380095f0 con 0x7fd448101790 2024-04-16T02:50:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.471+0000 7fd44d4c2700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 0x7fd4480735d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd4380092e0 tx=0x7fd43800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.471+0000 7fd43effd700 1 -- 172.21.15.151:0/3465451094 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd43801d070 con 0x7fd448101790 2024-04-16T02:50:31.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.471+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd448073ed0 con 0x7fd448101790 2024-04-16T02:50:31.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.471+0000 7fd43effd700 1 -- 172.21.15.151:0/3465451094 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd43800f460 con 0x7fd448101790 2024-04-16T02:50:31.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.471+0000 7fd43effd700 1 -- 172.21.15.151:0/3465451094 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4380175f0 con 0x7fd448101790 2024-04-16T02:50:31.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.471+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4481a2c90 con 0x7fd448101790 2024-04-16T02:50:31.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.472+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4481091e0 con 0x7fd448101790 2024-04-16T02:50:31.477 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.472+0000 7fd43effd700 1 -- 172.21.15.151:0/3465451094 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd43800f5d0 con 0x7fd448101790 2024-04-16T02:50:31.477 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.473+0000 7fd43effd700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:31.477 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.473+0000 7fd43effd700 1 -- 172.21.15.151:0/3465451094 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd438090390 con 0x7fd448101790 2024-04-16T02:50:31.477 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.473+0000 7fd44dcc3700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:31.477 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.473+0000 7fd44dcc3700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd444005fd0 tx=0x7fd44400a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:31.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.478+0000 7fd43effd700 1 -- 172.21.15.151:0/3465451094 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd438059140 con 0x7fd448101790 2024-04-16T02:50:31.722 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.719+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd44804ea90 con 0x7fd448101790 2024-04-16T02:50:31.722 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.720+0000 7fd43effd700 1 -- 172.21.15.151:0/3465451094 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7fd43805c800 con 0x7fd448101790 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:50:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:50:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:50:31.725 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:50:31.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 msgr2=0x7fd434071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd444005fd0 tx=0x7fd44400a560 comp rx=0 tx=0).stop 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 msgr2=0x7fd4480735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 0x7fd4480735d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd4380092e0 tx=0x7fd43800be20 comp rx=0 tx=0).stop 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 shutdown_connections 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd448100de0 0x7fd448073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 --2- 172.21.15.151:0/3465451094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd448101790 0x7fd4480735d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 >> 172.21.15.151:0/3465451094 conn(0x7fd4480fc670 msgr2=0x7fd4480fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 shutdown_connections 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.722+0000 7fd44ff27700 1 -- 172.21.15.151:0/3465451094 wait complete. 2024-04-16T02:50:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:50:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.827+0000 7fec4f114700 1 -- 172.21.15.151:0/1825518710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec48105a40 msgr2=0x7fec48105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.827+0000 7fec4f114700 1 --2- 172.21.15.151:0/1825518710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec48105a40 0x7fec48105e20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009910 tx=0x7fec3c009c20 comp rx=0 tx=0).stop 2024-04-16T02:50:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.827+0000 7fec4f114700 1 -- 172.21.15.151:0/1825518710 shutdown_connections 2024-04-16T02:50:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.827+0000 7fec4f114700 1 --2- 172.21.15.151:0/1825518710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec480ffbd0 0x7fec48100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.827+0000 7fec4f114700 1 --2- 172.21.15.151:0/1825518710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec48105a40 0x7fec48105e20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.827+0000 7fec4f114700 1 -- 172.21.15.151:0/1825518710 >> 172.21.15.151:0/1825518710 conn(0x7fec480755a0 msgr2=0x7fec480759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:31.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.827+0000 7fec4f114700 1 -- 172.21.15.151:0/1825518710 shutdown_connections 2024-04-16T02:50:31.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.827+0000 7fec4f114700 1 -- 172.21.15.151:0/1825518710 wait complete. 2024-04-16T02:50:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4f114700 1 Processor -- start 2024-04-16T02:50:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4f114700 1 -- start start 2024-04-16T02:50:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4f114700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec480ffbd0 0x7fec48101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4f114700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48105a40 0x7fec48101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4f114700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec48102090 con 0x7fec480ffbd0 2024-04-16T02:50:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4f114700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec48102200 con 0x7fec48105a40 2024-04-16T02:50:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4ceb0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec480ffbd0 0x7fec48101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4ceb0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec480ffbd0 0x7fec48101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40698/0 (socket says 172.21.15.151:40698) 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4ceb0700 1 -- 172.21.15.151:0/1666008064 learned_addr learned my addr 172.21.15.151:0/1666008064 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4ceb0700 1 -- 172.21.15.151:0/1666008064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48105a40 msgr2=0x7fec48101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec47fff700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48105a40 0x7fec48101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4ceb0700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48105a40 0x7fec48101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.828+0000 7fec4ceb0700 1 -- 172.21.15.151:0/1666008064 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c0095f0 con 0x7fec480ffbd0 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.829+0000 7fec4ceb0700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec480ffbd0 0x7fec48101610 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00b5c0 tx=0x7fec3c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.829+0000 7fec45ffb700 1 -- 172.21.15.151:0/1666008064 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec3c01d070 con 0x7fec480ffbd0 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.829+0000 7fec45ffb700 1 -- 172.21.15.151:0/1666008064 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec3c00f460 con 0x7fec480ffbd0 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.829+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec48102420 con 0x7fec480ffbd0 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.829+0000 7fec45ffb700 1 -- 172.21.15.151:0/1666008064 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec3c017610 con 0x7fec480ffbd0 2024-04-16T02:50:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.829+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec481a2bd0 con 0x7fec480ffbd0 2024-04-16T02:50:31.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.830+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec48109050 con 0x7fec480ffbd0 2024-04-16T02:50:31.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.831+0000 7fec45ffb700 1 -- 172.21.15.151:0/1666008064 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fec3c00f5d0 con 0x7fec480ffbd0 2024-04-16T02:50:31.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.831+0000 7fec45ffb700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec3806ecc0 0x7fec38071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:31.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.831+0000 7fec45ffb700 1 -- 172.21.15.151:0/1666008064 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fec3c0903a0 con 0x7fec480ffbd0 2024-04-16T02:50:31.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.831+0000 7fec47fff700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec3806ecc0 0x7fec38071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:31.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.832+0000 7fec47fff700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec3806ecc0 0x7fec38071180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fec48102e00 tx=0x7fec34005d50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:31.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:31.836+0000 7fec45ffb700 1 -- 172.21.15.151:0/1666008064 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fec3c0591d0 con 0x7fec480ffbd0 2024-04-16T02:50:31.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:31 smithi151 ceph-mon[112233]: pgmap v356: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:31.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:31 smithi151 ceph-mon[112233]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:31.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:31 smithi151 ceph-mon[112233]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:31.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:31 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3106410286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:50:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:31 smithi153 ceph-mon[87858]: pgmap v356: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:31 smithi153 ceph-mon[87858]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:31 smithi153 ceph-mon[87858]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:31 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3106410286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:50:32.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.055+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fec480611d0 con 0x7fec3806ecc0 2024-04-16T02:50:32.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.056+0000 7fec45ffb700 1 -- 172.21.15.151:0/1666008064 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fec480611d0 con 0x7fec3806ecc0 2024-04-16T02:50:32.059 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:50:32.059 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:50:32.059 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:50:32.059 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:50:32.059 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:50:32.059 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:50:32.059 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:50:32.060 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:50:32.060 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:50:32.060 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:50:32.060 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:50:32.060 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:50:32.060 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:50:32.060 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:50:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec3806ecc0 msgr2=0x7fec38071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec3806ecc0 0x7fec38071180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fec48102e00 tx=0x7fec34005d50 comp rx=0 tx=0).stop 2024-04-16T02:50:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec480ffbd0 msgr2=0x7fec48101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec480ffbd0 0x7fec48101610 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00b5c0 tx=0x7fec3c00be00 comp rx=0 tx=0).stop 2024-04-16T02:50:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 shutdown_connections 2024-04-16T02:50:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fec3806ecc0 0x7fec38071180 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec480ffbd0 0x7fec48101610 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 --2- 172.21.15.151:0/1666008064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48105a40 0x7fec48101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 >> 172.21.15.151:0/1666008064 conn(0x7fec480755a0 msgr2=0x7fec480685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 shutdown_connections 2024-04-16T02:50:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.059+0000 7fec4f114700 1 -- 172.21.15.151:0/1666008064 wait complete. 2024-04-16T02:50:32.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.164+0000 7f8ebad68700 1 -- 172.21.15.151:0/23517938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4101790 msgr2=0x7f8eb41057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:32.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.164+0000 7f8ebad68700 1 --2- 172.21.15.151:0/23517938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4101790 0x7f8eb41057e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0009910 tx=0x7f8ea0009c20 comp rx=0 tx=0).stop 2024-04-16T02:50:32.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.164+0000 7f8ebad68700 1 -- 172.21.15.151:0/23517938 shutdown_connections 2024-04-16T02:50:32.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.164+0000 7f8ebad68700 1 --2- 172.21.15.151:0/23517938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4101790 0x7f8eb41057e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.164+0000 7f8ebad68700 1 --2- 172.21.15.151:0/23517938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb4100de0 0x7f8eb41011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.164+0000 7f8ebad68700 1 -- 172.21.15.151:0/23517938 >> 172.21.15.151:0/23517938 conn(0x7f8eb40fc670 msgr2=0x7f8eb40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:32.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.164+0000 7f8ebad68700 1 -- 172.21.15.151:0/23517938 shutdown_connections 2024-04-16T02:50:32.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.164+0000 7f8ebad68700 1 -- 172.21.15.151:0/23517938 wait complete. 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.164+0000 7f8ebad68700 1 Processor -- start 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8ebad68700 1 -- start start 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8ebad68700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4100de0 0x7f8eb419c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8ebad68700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb4101790 0x7f8eb419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8ebad68700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb419d120 con 0x7f8eb4100de0 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8ebad68700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41965d0 con 0x7f8eb4101790 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8eb8b04700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4100de0 0x7f8eb419c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8eb8b04700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4100de0 0x7f8eb419c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40702/0 (socket says 172.21.15.151:40702) 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8eb8b04700 1 -- 172.21.15.151:0/3286118787 learned_addr learned my addr 172.21.15.151:0/3286118787 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8eb3fff700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb4101790 0x7f8eb419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8eb8b04700 1 -- 172.21.15.151:0/3286118787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb4101790 msgr2=0x7f8eb419ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:32.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8eb8b04700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb4101790 0x7f8eb419ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8eb8b04700 1 -- 172.21.15.151:0/3286118787 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea00095f0 con 0x7f8eb4100de0 2024-04-16T02:50:32.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8eb8b04700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4100de0 0x7f8eb419c550 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800d900 tx=0x7f8ea800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:32.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.165+0000 7f8eb1ffb700 1 -- 172.21.15.151:0/3286118787 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea8004a00 con 0x7f8eb4100de0 2024-04-16T02:50:32.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.166+0000 7f8eb1ffb700 1 -- 172.21.15.151:0/3286118787 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ea80054e0 con 0x7f8eb4100de0 2024-04-16T02:50:32.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.166+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb41968b0 con 0x7f8eb4100de0 2024-04-16T02:50:32.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.166+0000 7f8eb1ffb700 1 -- 172.21.15.151:0/3286118787 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea8009d60 con 0x7f8eb4100de0 2024-04-16T02:50:32.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.166+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb4196e00 con 0x7f8eb4100de0 2024-04-16T02:50:32.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.167+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb41090f0 con 0x7f8eb4100de0 2024-04-16T02:50:32.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.167+0000 7f8eb1ffb700 1 -- 172.21.15.151:0/3286118787 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8ea8005020 con 0x7f8eb4100de0 2024-04-16T02:50:32.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.168+0000 7f8eb1ffb700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8ea406ecc0 0x7f8ea4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:50:32.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.168+0000 7f8eb3fff700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8ea406ecc0 0x7f8ea4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:50:32.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.168+0000 7f8eb1ffb700 1 -- 172.21.15.151:0/3286118787 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8ea808f030 con 0x7f8eb4100de0 2024-04-16T02:50:32.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.168+0000 7f8eb3fff700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8ea406ecc0 0x7f8ea4071180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40fdfc0 tx=0x7f8ea001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:50:32.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.172+0000 7f8eb1ffb700 1 -- 172.21.15.151:0/3286118787 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8ea805b910 con 0x7f8eb4100de0 2024-04-16T02:50:32.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.456+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8eb4197070 con 0x7f8eb4100de0 2024-04-16T02:50:32.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.456+0000 7f8eb1ffb700 1 -- 172.21.15.151:0/3286118787 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f8ea805b4a0 con 0x7f8eb4100de0 2024-04-16T02:50:32.460 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:50:32.460 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:50:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:50:32.460 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:50:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:50:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8ea406ecc0 msgr2=0x7f8ea4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8ea406ecc0 0x7f8ea4071180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40fdfc0 tx=0x7f8ea001a040 comp rx=0 tx=0).stop 2024-04-16T02:50:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4100de0 msgr2=0x7f8eb419c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:50:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4100de0 0x7f8eb419c550 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800d900 tx=0x7f8ea800dcc0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 shutdown_connections 2024-04-16T02:50:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8ea406ecc0 0x7f8ea4071180 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8eb4100de0 0x7f8eb419c550 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 --2- 172.21.15.151:0/3286118787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb4101790 0x7f8eb419ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:50:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 >> 172.21.15.151:0/3286118787 conn(0x7f8eb40fc670 msgr2=0x7f8eb4105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:50:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 shutdown_connections 2024-04-16T02:50:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:50:32.459+0000 7f8ebad68700 1 -- 172.21.15.151:0/3286118787 wait complete. 2024-04-16T02:50:32.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:32 smithi151 ceph-mon[112233]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:32.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:32 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3465451094' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:50:32.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:32 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3286118787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:50:33.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:32 smithi153 ceph-mon[87858]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:33.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:32 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3465451094' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:50:33.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:32 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3286118787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:50:33.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:33 smithi151 ceph-mon[112233]: from='client.34726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:33.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:33 smithi151 ceph-mon[112233]: pgmap v357: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:33 smithi153 ceph-mon[87858]: from='client.34726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:50:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:33 smithi153 ceph-mon[87858]: pgmap v357: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:35.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:35 smithi151 ceph-mon[112233]: pgmap v358: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:36.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:35 smithi153 ceph-mon[87858]: pgmap v358: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:37.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:37 smithi151 ceph-mon[112233]: pgmap v359: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:38.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:37 smithi153 ceph-mon[87858]: pgmap v359: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:39.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:39 smithi151 ceph-mon[112233]: pgmap v360: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:39.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:50:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:39 smithi153 ceph-mon[87858]: pgmap v360: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:50:41.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:41 smithi151 ceph-mon[112233]: pgmap v361: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:42.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:41 smithi153 ceph-mon[87858]: pgmap v361: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:43.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:43 smithi151 ceph-mon[112233]: pgmap v362: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:44.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:43 smithi153 ceph-mon[87858]: pgmap v362: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:45 smithi151 ceph-mon[112233]: pgmap v363: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:45 smithi153 ceph-mon[87858]: pgmap v363: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:47.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:47 smithi151 ceph-mon[112233]: pgmap v364: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:47 smithi153 ceph-mon[87858]: pgmap v364: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:49 smithi151 ceph-mon[112233]: pgmap v365: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:49 smithi153 ceph-mon[87858]: pgmap v365: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:51.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:51 smithi151 ceph-mon[112233]: pgmap v366: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:52.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:51 smithi153 ceph-mon[87858]: pgmap v366: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:53.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:53 smithi151 ceph-mon[112233]: pgmap v367: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:54.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:53 smithi153 ceph-mon[87858]: pgmap v367: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:54.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:50:55.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:50:55.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:55 smithi151 ceph-mon[112233]: pgmap v368: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:56.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:55 smithi153 ceph-mon[87858]: pgmap v368: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:57 smithi151 ceph-mon[112233]: pgmap v369: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:58.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:57 smithi153 ceph-mon[87858]: pgmap v369: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:50:59.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:50:59 smithi151 ceph-mon[112233]: pgmap v370: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:00.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:50:59 smithi153 ceph-mon[87858]: pgmap v370: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:01.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:01 smithi151 ceph-mon[112233]: pgmap v371: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:02.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:01 smithi153 ceph-mon[87858]: pgmap v371: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:02.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.577+0000 7fbac37bc700 1 -- 172.21.15.151:0/2788520042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc100da0 msgr2=0x7fbabc101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.577+0000 7fbac37bc700 1 --2- 172.21.15.151:0/2788520042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc100da0 0x7fbabc101180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009910 tx=0x7fbab4009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.578+0000 7fbac37bc700 1 -- 172.21.15.151:0/2788520042 shutdown_connections 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.578+0000 7fbac37bc700 1 --2- 172.21.15.151:0/2788520042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc101750 0x7fbabc1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.578+0000 7fbac37bc700 1 --2- 172.21.15.151:0/2788520042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc100da0 0x7fbabc101180 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.578+0000 7fbac37bc700 1 -- 172.21.15.151:0/2788520042 >> 172.21.15.151:0/2788520042 conn(0x7fbabc0fc650 msgr2=0x7fbabc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.578+0000 7fbac37bc700 1 -- 172.21.15.151:0/2788520042 shutdown_connections 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.578+0000 7fbac37bc700 1 -- 172.21.15.151:0/2788520042 wait complete. 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.578+0000 7fbac37bc700 1 Processor -- start 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.578+0000 7fbac37bc700 1 -- start start 2024-04-16T02:51:02.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac37bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc100da0 0x7fbabc19e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac37bc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc101750 0x7fbabc19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac37bc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc19f360 con 0x7fbabc101750 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac37bc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc198810 con 0x7fbabc100da0 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac0d57700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc101750 0x7fbabc19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac0d57700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc101750 0x7fbabc19ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47486/0 (socket says 172.21.15.151:47486) 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac0d57700 1 -- 172.21.15.151:0/825420728 learned_addr learned my addr 172.21.15.151:0/825420728 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac1558700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc100da0 0x7fbabc19e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac0d57700 1 -- 172.21.15.151:0/825420728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc100da0 msgr2=0x7fbabc19e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac0d57700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc100da0 0x7fbabc19e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac0d57700 1 -- 172.21.15.151:0/825420728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab40095f0 con 0x7fbabc101750 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbac0d57700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc101750 0x7fbabc19ecd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbab000c8f0 tx=0x7fbab000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:02.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbaba7fc700 1 -- 172.21.15.151:0/825420728 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbab00078a0 con 0x7fbabc101750 2024-04-16T02:51:02.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.579+0000 7fbaba7fc700 1 -- 172.21.15.151:0/825420728 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbab000f450 con 0x7fbabc101750 2024-04-16T02:51:02.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.580+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbabc198af0 con 0x7fbabc101750 2024-04-16T02:51:02.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.580+0000 7fbaba7fc700 1 -- 172.21.15.151:0/825420728 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbab0018610 con 0x7fbabc101750 2024-04-16T02:51:02.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.580+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc199040 con 0x7fbabc101750 2024-04-16T02:51:02.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.581+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbabc104f20 con 0x7fbabc101750 2024-04-16T02:51:02.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.581+0000 7fbaba7fc700 1 -- 172.21.15.151:0/825420728 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fbab000f5c0 con 0x7fbabc101750 2024-04-16T02:51:02.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.582+0000 7fbaba7fc700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbaa406ecc0 0x7fbaa4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:02.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.582+0000 7fbac1558700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbaa406ecc0 0x7fbaa4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:02.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.582+0000 7fbaba7fc700 1 -- 172.21.15.151:0/825420728 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbab008edd0 con 0x7fbabc101750 2024-04-16T02:51:02.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.582+0000 7fbac1558700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbaa406ecc0 0x7fbaa4071180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbab4005f50 tx=0x7fbab401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:02.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.586+0000 7fbaba7fc700 1 -- 172.21.15.151:0/825420728 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbab005b6b0 con 0x7fbabc101750 2024-04-16T02:51:02.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.805+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbabc199330 con 0x7fbaa406ecc0 2024-04-16T02:51:02.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.806+0000 7fbaba7fc700 1 -- 172.21.15.151:0/825420728 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbabc199330 con 0x7fbaa406ecc0 2024-04-16T02:51:02.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.808+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbaa406ecc0 msgr2=0x7fbaa4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:02.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.808+0000 7fbac37bc700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbaa406ecc0 0x7fbaa4071180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbab4005f50 tx=0x7fbab401a040 comp rx=0 tx=0).stop 2024-04-16T02:51:02.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.808+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc101750 msgr2=0x7fbabc19ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:02.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.808+0000 7fbac37bc700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc101750 0x7fbabc19ecd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbab000c8f0 tx=0x7fbab000cc00 comp rx=0 tx=0).stop 2024-04-16T02:51:02.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.808+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 shutdown_connections 2024-04-16T02:51:02.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.808+0000 7fbac37bc700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbaa406ecc0 0x7fbaa4071180 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:02.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.808+0000 7fbac37bc700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc100da0 0x7fbabc19e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:02.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.808+0000 7fbac37bc700 1 --2- 172.21.15.151:0/825420728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbabc101750 0x7fbabc19ecd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:02.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.808+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 >> 172.21.15.151:0/825420728 conn(0x7fbabc0fc650 msgr2=0x7fbabc0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:02.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.809+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 shutdown_connections 2024-04-16T02:51:02.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.809+0000 7fbac37bc700 1 -- 172.21.15.151:0/825420728 wait complete. 2024-04-16T02:51:02.823 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:51:02.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.915+0000 7fe14d1ce700 1 -- 172.21.15.151:0/3773315770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 msgr2=0x7fe148105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:02.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.915+0000 7fe14d1ce700 1 --2- 172.21.15.151:0/3773315770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 0x7fe148105e20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe138009910 tx=0x7fe138009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:02.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.915+0000 7fe14d1ce700 1 -- 172.21.15.151:0/3773315770 shutdown_connections 2024-04-16T02:51:02.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.915+0000 7fe14d1ce700 1 --2- 172.21.15.151:0/3773315770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1480ffbd0 0x7fe148100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:02.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.915+0000 7fe14d1ce700 1 --2- 172.21.15.151:0/3773315770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 0x7fe148105e20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:02.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.915+0000 7fe14d1ce700 1 -- 172.21.15.151:0/3773315770 >> 172.21.15.151:0/3773315770 conn(0x7fe1480755a0 msgr2=0x7fe1480759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:02.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.915+0000 7fe14d1ce700 1 -- 172.21.15.151:0/3773315770 shutdown_connections 2024-04-16T02:51:02.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.915+0000 7fe14d1ce700 1 -- 172.21.15.151:0/3773315770 wait complete. 2024-04-16T02:51:02.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe14d1ce700 1 Processor -- start 2024-04-16T02:51:02.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe14d1ce700 1 -- start start 2024-04-16T02:51:02.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe14d1ce700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1480ffbd0 0x7fe148197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe14d1ce700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 0x7fe148197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe14d1ce700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe148198170 con 0x7fe148105a40 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe14d1ce700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1481982b0 con 0x7fe1480ffbd0 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe13ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 0x7fe148197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe13ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 0x7fe148197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47504/0 (socket says 172.21.15.151:47504) 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe13ffff700 1 -- 172.21.15.151:0/1863229676 learned_addr learned my addr 172.21.15.151:0/1863229676 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.916+0000 7fe146d9d700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1480ffbd0 0x7fe148197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.917+0000 7fe13ffff700 1 -- 172.21.15.151:0/1863229676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1480ffbd0 msgr2=0x7fe148197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.917+0000 7fe13ffff700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1480ffbd0 0x7fe148197550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:02.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.917+0000 7fe13ffff700 1 -- 172.21.15.151:0/1863229676 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1380095f0 con 0x7fe148105a40 2024-04-16T02:51:02.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.917+0000 7fe13ffff700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 0x7fe148197a90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c8a0 tx=0x7fe13000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:02.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.917+0000 7fe144d99700 1 -- 172.21.15.151:0/1863229676 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe13000cea0 con 0x7fe148105a40 2024-04-16T02:51:02.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.917+0000 7fe144d99700 1 -- 172.21.15.151:0/1863229676 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe130004830 con 0x7fe148105a40 2024-04-16T02:51:02.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.917+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1481a05c0 con 0x7fe148105a40 2024-04-16T02:51:02.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.917+0000 7fe144d99700 1 -- 172.21.15.151:0/1863229676 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1300055b0 con 0x7fe148105a40 2024-04-16T02:51:02.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.917+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1481a0ae0 con 0x7fe148105a40 2024-04-16T02:51:02.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.918+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe148108040 con 0x7fe148105a40 2024-04-16T02:51:02.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.918+0000 7fe144d99700 1 -- 172.21.15.151:0/1863229676 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe1300049a0 con 0x7fe148105a40 2024-04-16T02:51:02.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.919+0000 7fe144d99700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe13406ecc0 0x7fe134071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:02.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.919+0000 7fe144d99700 1 -- 172.21.15.151:0/1863229676 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe13008eaf0 con 0x7fe148105a40 2024-04-16T02:51:02.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.919+0000 7fe146d9d700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe13406ecc0 0x7fe134071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:02.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.920+0000 7fe146d9d700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe13406ecc0 0x7fe134071180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe13800b5c0 tx=0x7fe13801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:02.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:02.924+0000 7fe144d99700 1 -- 172.21.15.151:0/1863229676 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe130057880 con 0x7fe148105a40 2024-04-16T02:51:03.145 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.141+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe14802cd10 con 0x7fe13406ecc0 2024-04-16T02:51:03.146 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.143+0000 7fe144d99700 1 -- 172.21.15.151:0/1863229676 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe14802cd10 con 0x7fe13406ecc0 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe13406ecc0 msgr2=0x7fe134071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe13406ecc0 0x7fe134071180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe13800b5c0 tx=0x7fe13801a040 comp rx=0 tx=0).stop 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 msgr2=0x7fe148197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 0x7fe148197a90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c8a0 tx=0x7fe13000cc60 comp rx=0 tx=0).stop 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 shutdown_connections 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe13406ecc0 0x7fe134071180 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1480ffbd0 0x7fe148197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 --2- 172.21.15.151:0/1863229676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe148105a40 0x7fe148197a90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 >> 172.21.15.151:0/1863229676 conn(0x7fe1480755a0 msgr2=0x7fe1480fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 shutdown_connections 2024-04-16T02:51:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.145+0000 7fe14d1ce700 1 -- 172.21.15.151:0/1863229676 wait complete. 2024-04-16T02:51:03.254 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 -- 172.21.15.151:0/3134430879 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 msgr2=0x7f677c1048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 --2- 172.21.15.151:0/3134430879 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 0x7f677c1048d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f676c009910 tx=0x7f676c009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:03.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 -- 172.21.15.151:0/3134430879 shutdown_connections 2024-04-16T02:51:03.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 --2- 172.21.15.151:0/3134430879 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 0x7f677c1048d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 --2- 172.21.15.151:0/3134430879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677c0ffa50 0x7f677c0ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 -- 172.21.15.151:0/3134430879 >> 172.21.15.151:0/3134430879 conn(0x7f677c0fb520 msgr2=0x7f677c0fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:03.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 -- 172.21.15.151:0/3134430879 shutdown_connections 2024-04-16T02:51:03.255 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 -- 172.21.15.151:0/3134430879 wait complete. 2024-04-16T02:51:03.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 Processor -- start 2024-04-16T02:51:03.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.251+0000 7f6783b3c700 1 -- start start 2024-04-16T02:51:03.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f6783b3c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677c0ffa50 0x7f677c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:03.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f6783b3c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 0x7f677c19d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:03.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f6783b3c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c19d940 con 0x7f677c100400 2024-04-16T02:51:03.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f6783b3c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c1a16d0 con 0x7f677c0ffa50 2024-04-16T02:51:03.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f67810d7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 0x7f677c19d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:03.257 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f67810d7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 0x7f677c19d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47526/0 (socket says 172.21.15.151:47526) 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f67810d7700 1 -- 172.21.15.151:0/1388094130 learned_addr learned my addr 172.21.15.151:0/1388094130 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f67818d8700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677c0ffa50 0x7f677c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f67810d7700 1 -- 172.21.15.151:0/1388094130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677c0ffa50 msgr2=0x7f677c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f67810d7700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677c0ffa50 0x7f677c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f67810d7700 1 -- 172.21.15.151:0/1388094130 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c0095f0 con 0x7f677c100400 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f67810d7700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 0x7f677c19d260 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f676c00bd70 tx=0x7f676c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f6772ffd700 1 -- 172.21.15.151:0/1388094130 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f676c01d070 con 0x7f677c100400 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f6772ffd700 1 -- 172.21.15.151:0/1388094130 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f676c005dc0 con 0x7f677c100400 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677c1a1950 con 0x7f677c100400 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f6772ffd700 1 -- 172.21.15.151:0/1388094130 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f676c017400 con 0x7f677c100400 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.252+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677c1a1e40 con 0x7f677c100400 2024-04-16T02:51:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.253+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c102030 con 0x7f677c100400 2024-04-16T02:51:03.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.254+0000 7f6772ffd700 1 -- 172.21.15.151:0/1388094130 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f676c00f7f0 con 0x7f677c100400 2024-04-16T02:51:03.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.254+0000 7f6772ffd700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f676806ecc0 0x7f6768071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:03.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.254+0000 7f6772ffd700 1 -- 172.21.15.151:0/1388094130 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f676c090370 con 0x7f677c100400 2024-04-16T02:51:03.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.254+0000 7f67818d8700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f676806ecc0 0x7f6768071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:03.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.255+0000 7f67818d8700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f676806ecc0 0x7f6768071180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6778005950 tx=0x7f67780058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:03.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.259+0000 7f6772ffd700 1 -- 172.21.15.151:0/1388094130 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f676c059120 con 0x7f677c100400 2024-04-16T02:51:03.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.466+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f677c02cdb0 con 0x7f676806ecc0 2024-04-16T02:51:03.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.476+0000 7f6772ffd700 1 -- 172.21.15.151:0/1388094130 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f677c02cdb0 con 0x7f676806ecc0 2024-04-16T02:51:03.480 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:51:03.480 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (12m) 6m ago 23m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:51:03.480 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (24m) 6m ago 24m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (23m) 5m ago 23m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (10m) 6m ago 24m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (10m) 5m ago 23m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (12m) 6m ago 23m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (6m) 6m ago 18m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (6m) 6m ago 18m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (18m) 5m ago 18m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (5m) 5m ago 18m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (14m) 6m ago 25m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (13m) 5m ago 22m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (11m) 6m ago 25m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (10m) 5m ago 22m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (13m) 6m ago 24m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (13m) 5m ago 22m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (9m) 6m ago 22m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:51:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (9m) 6m ago 21m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:51:03.482 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (8m) 6m ago 21m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:51:03.482 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (7m) 5m ago 20m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:51:03.482 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (7m) 5m ago 20m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:51:03.482 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (6m) 5m ago 19m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:51:03.482 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (12m) 6m ago 23m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:51:03.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f676806ecc0 msgr2=0x7f6768071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f676806ecc0 0x7f6768071180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6778005950 tx=0x7f67780058e0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 msgr2=0x7f677c19d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 0x7f677c19d260 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f676c00bd70 tx=0x7f676c00be50 comp rx=0 tx=0).stop 2024-04-16T02:51:03.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 shutdown_connections 2024-04-16T02:51:03.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f676806ecc0 0x7f6768071180 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677c0ffa50 0x7f677c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 --2- 172.21.15.151:0/1388094130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f677c100400 0x7f677c19d260 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 >> 172.21.15.151:0/1388094130 conn(0x7f677c0fb520 msgr2=0x7f677c110f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:03.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 shutdown_connections 2024-04-16T02:51:03.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.479+0000 7f6783b3c700 1 -- 172.21.15.151:0/1388094130 wait complete. 2024-04-16T02:51:03.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.585+0000 7f6dcbe22700 1 -- 172.21.15.151:0/1072611606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ffe30 msgr2=0x7f6dc4111200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.585+0000 7f6dcbe22700 1 --2- 172.21.15.151:0/1072611606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ffe30 0x7f6dc4111200 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009940 tx=0x7f6db4009c50 comp rx=0 tx=0).stop 2024-04-16T02:51:03.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.585+0000 7f6dcbe22700 1 -- 172.21.15.151:0/1072611606 shutdown_connections 2024-04-16T02:51:03.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.585+0000 7f6dcbe22700 1 --2- 172.21.15.151:0/1072611606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ffe30 0x7f6dc4111200 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.585+0000 7f6dcbe22700 1 --2- 172.21.15.151:0/1072611606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc40ff510 0x7f6dc40ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.585+0000 7f6dcbe22700 1 -- 172.21.15.151:0/1072611606 >> 172.21.15.151:0/1072611606 conn(0x7f6dc4074d80 msgr2=0x7f6dc4075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:03.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.585+0000 7f6dcbe22700 1 -- 172.21.15.151:0/1072611606 shutdown_connections 2024-04-16T02:51:03.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.585+0000 7f6dcbe22700 1 -- 172.21.15.151:0/1072611606 wait complete. 2024-04-16T02:51:03.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.585+0000 7f6dcbe22700 1 Processor -- start 2024-04-16T02:51:03.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dcbe22700 1 -- start start 2024-04-16T02:51:03.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dcbe22700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ff510 0x7f6dc419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:03.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dcbe22700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc40ffe30 0x7f6dc419d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:03.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dcbe22700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc419d930 con 0x7f6dc40ff510 2024-04-16T02:51:03.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dcbe22700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc41a16c0 con 0x7f6dc40ffe30 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dc9bbe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ff510 0x7f6dc419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dc9bbe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ff510 0x7f6dc419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47548/0 (socket says 172.21.15.151:47548) 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dc9bbe700 1 -- 172.21.15.151:0/2296102207 learned_addr learned my addr 172.21.15.151:0/2296102207 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dc93bd700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc40ffe30 0x7f6dc419d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dc9bbe700 1 -- 172.21.15.151:0/2296102207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc40ffe30 msgr2=0x7f6dc419d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dc9bbe700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc40ffe30 0x7f6dc419d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dc9bbe700 1 -- 172.21.15.151:0/2296102207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db40095f0 con 0x7f6dc40ff510 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dc93bd700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc40ffe30 0x7f6dc419d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dc9bbe700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ff510 0x7f6dc419cd10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000d8d0 tx=0x7f6dc000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.586+0000 7f6dbaffd700 1 -- 172.21.15.151:0/2296102207 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc0009940 con 0x7f6dc40ff510 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.587+0000 7f6dbaffd700 1 -- 172.21.15.151:0/2296102207 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dc0010460 con 0x7f6dc40ff510 2024-04-16T02:51:03.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.587+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc41a19a0 con 0x7f6dc40ff510 2024-04-16T02:51:03.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.587+0000 7f6dbaffd700 1 -- 172.21.15.151:0/2296102207 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc000f5f0 con 0x7f6dc40ff510 2024-04-16T02:51:03.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.587+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc41a1ef0 con 0x7f6dc40ff510 2024-04-16T02:51:03.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.588+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc4101d40 con 0x7f6dc40ff510 2024-04-16T02:51:03.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.588+0000 7f6dbaffd700 1 -- 172.21.15.151:0/2296102207 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6dc00105d0 con 0x7f6dc40ff510 2024-04-16T02:51:03.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.589+0000 7f6dbaffd700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6db006ecc0 0x7f6db0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:03.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.589+0000 7f6dc93bd700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6db006ecc0 0x7f6db0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:03.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.589+0000 7f6dbaffd700 1 -- 172.21.15.151:0/2296102207 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6dc008ed70 con 0x7f6dc40ff510 2024-04-16T02:51:03.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.589+0000 7f6dc93bd700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6db006ecc0 0x7f6db0071180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6db4006010 tx=0x7f6db4005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:03.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.594+0000 7f6dbaffd700 1 -- 172.21.15.151:0/2296102207 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6dc0057ba0 con 0x7f6dc40ff510 2024-04-16T02:51:03.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.880+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6dc41a20d0 con 0x7f6dc40ff510 2024-04-16T02:51:03.885 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:03 smithi151 ceph-mon[112233]: pgmap v372: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:03.885 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:03 smithi151 ceph-mon[112233]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:03.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.881+0000 7f6dbaffd700 1 -- 172.21.15.151:0/2296102207 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f6dc005b260 con 0x7f6dc40ff510 2024-04-16T02:51:03.885 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:51:03.885 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:51:03.885 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:51:03.886 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:51:03.887 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:51:03.887 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:51:03.887 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:51:03.887 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:51:03.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6db006ecc0 msgr2=0x7f6db0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6db006ecc0 0x7f6db0071180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6db4006010 tx=0x7f6db4005a90 comp rx=0 tx=0).stop 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ff510 msgr2=0x7f6dc419cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ff510 0x7f6dc419cd10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000d8d0 tx=0x7f6dc000dc90 comp rx=0 tx=0).stop 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 shutdown_connections 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6db006ecc0 0x7f6db0071180 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc40ff510 0x7f6dc419cd10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 --2- 172.21.15.151:0/2296102207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc40ffe30 0x7f6dc419d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 >> 172.21.15.151:0/2296102207 conn(0x7f6dc4074d80 msgr2=0x7f6dc410f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 shutdown_connections 2024-04-16T02:51:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.883+0000 7f6dcbe22700 1 -- 172.21.15.151:0/2296102207 wait complete. 2024-04-16T02:51:03.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.989+0000 7f1c97e52700 1 -- 172.21.15.151:0/3574889583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 msgr2=0x7f1c900ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.989+0000 7f1c97e52700 1 --2- 172.21.15.151:0/3574889583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 0x7f1c900ffed0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009910 tx=0x7f1c84009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:03.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.989+0000 7f1c97e52700 1 -- 172.21.15.151:0/3574889583 shutdown_connections 2024-04-16T02:51:03.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.989+0000 7f1c97e52700 1 --2- 172.21.15.151:0/3574889583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 0x7f1c900ffed0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.989+0000 7f1c97e52700 1 --2- 172.21.15.151:0/3574889583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c90105a50 0x7f1c90105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.989+0000 7f1c97e52700 1 -- 172.21.15.151:0/3574889583 >> 172.21.15.151:0/3574889583 conn(0x7f1c900fb540 msgr2=0x7f1c900fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:03.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.989+0000 7f1c97e52700 1 -- 172.21.15.151:0/3574889583 shutdown_connections 2024-04-16T02:51:03.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.989+0000 7f1c97e52700 1 -- 172.21.15.151:0/3574889583 wait complete. 2024-04-16T02:51:03.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c97e52700 1 Processor -- start 2024-04-16T02:51:03.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c97e52700 1 -- start start 2024-04-16T02:51:03.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c97e52700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 0x7f1c900fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:03.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c97e52700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c90105a50 0x7f1c900fc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:03.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c97e52700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c900fc890 con 0x7f1c900ffa50 2024-04-16T02:51:03.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c97e52700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c900fc9d0 con 0x7f1c90105a50 2024-04-16T02:51:03.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c95bee700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 0x7f1c900fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:03.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c95bee700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 0x7f1c900fbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47570/0 (socket says 172.21.15.151:47570) 2024-04-16T02:51:03.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c95bee700 1 -- 172.21.15.151:0/1804537268 learned_addr learned my addr 172.21.15.151:0/1804537268 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.990+0000 7f1c953ed700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c90105a50 0x7f1c900fc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c95bee700 1 -- 172.21.15.151:0/1804537268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c90105a50 msgr2=0x7f1c900fc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c95bee700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c90105a50 0x7f1c900fc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c95bee700 1 -- 172.21.15.151:0/1804537268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c840095f0 con 0x7f1c900ffa50 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c953ed700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c90105a50 0x7f1c900fc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c95bee700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 0x7f1c900fbe10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1c880060b0 tx=0x7f1c8800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c8effd700 1 -- 172.21.15.151:0/1804537268 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c8800dda0 con 0x7f1c900ffa50 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c8effd700 1 -- 172.21.15.151:0/1804537268 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c8800f040 con 0x7f1c900ffa50 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c901a28b0 con 0x7f1c900ffa50 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c8effd700 1 -- 172.21.15.151:0/1804537268 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c88014790 con 0x7f1c900ffa50 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.991+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c901a2cc0 con 0x7f1c900ffa50 2024-04-16T02:51:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.992+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c90108ff0 con 0x7f1c900ffa50 2024-04-16T02:51:03.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.992+0000 7f1c8effd700 1 -- 172.21.15.151:0/1804537268 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1c880148f0 con 0x7f1c900ffa50 2024-04-16T02:51:03.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.993+0000 7f1c8effd700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c7806ecc0 0x7f1c78071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:03.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.993+0000 7f1c8effd700 1 -- 172.21.15.151:0/1804537268 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1c8808e810 con 0x7f1c900ffa50 2024-04-16T02:51:03.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.993+0000 7f1c953ed700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c7806ecc0 0x7f1c78071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:03.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.993+0000 7f1c953ed700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c7806ecc0 0x7f1c78071180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c84005b40 tx=0x7f1c8401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:04.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:03.998+0000 7f1c8effd700 1 -- 172.21.15.151:0/1804537268 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1c880576f0 con 0x7f1c900ffa50 2024-04-16T02:51:04.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:03 smithi153 ceph-mon[87858]: pgmap v372: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:04.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:03 smithi153 ceph-mon[87858]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:04.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.238+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1c9002cdd0 con 0x7f1c900ffa50 2024-04-16T02:51:04.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.239+0000 7f1c8effd700 1 -- 172.21.15.151:0/1804537268 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f1c8805adb0 con 0x7f1c900ffa50 2024-04-16T02:51:04.243 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:51:04.243 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:51:04.243 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:51:04.243 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:51:04.243 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:51:04.243 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:51:04.244 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:51:04.245 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:51:04.246 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:51:04.247 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:51:04.247 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:51:04.247 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c7806ecc0 msgr2=0x7f1c78071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c7806ecc0 0x7f1c78071180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c84005b40 tx=0x7f1c8401a040 comp rx=0 tx=0).stop 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 msgr2=0x7f1c900fbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 0x7f1c900fbe10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1c880060b0 tx=0x7f1c8800d3b0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 shutdown_connections 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c7806ecc0 0x7f1c78071180 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c900ffa50 0x7f1c900fbe10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 --2- 172.21.15.151:0/1804537268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c90105a50 0x7f1c900fc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 >> 172.21.15.151:0/1804537268 conn(0x7f1c900fb540 msgr2=0x7f1c90068fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 shutdown_connections 2024-04-16T02:51:04.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.241+0000 7f1c97e52700 1 -- 172.21.15.151:0/1804537268 wait complete. 2024-04-16T02:51:04.249 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:51:04.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.349+0000 7ff15b30c700 1 -- 172.21.15.151:0/2585310994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 msgr2=0x7ff154105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.349+0000 7ff15b30c700 1 --2- 172.21.15.151:0/2585310994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 0x7ff154105870 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff144009910 tx=0x7ff144009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:04.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.349+0000 7ff15b30c700 1 -- 172.21.15.151:0/2585310994 shutdown_connections 2024-04-16T02:51:04.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.349+0000 7ff15b30c700 1 --2- 172.21.15.151:0/2585310994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 0x7ff154105870 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.349+0000 7ff15b30c700 1 --2- 172.21.15.151:0/2585310994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1540ff530 0x7ff1540ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.349+0000 7ff15b30c700 1 -- 172.21.15.151:0/2585310994 >> 172.21.15.151:0/2585310994 conn(0x7ff154074df0 msgr2=0x7ff154075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:04.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.349+0000 7ff15b30c700 1 -- 172.21.15.151:0/2585310994 shutdown_connections 2024-04-16T02:51:04.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.349+0000 7ff15b30c700 1 -- 172.21.15.151:0/2585310994 wait complete. 2024-04-16T02:51:04.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff15b30c700 1 Processor -- start 2024-04-16T02:51:04.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff15b30c700 1 -- start start 2024-04-16T02:51:04.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff15b30c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1540ff530 0x7ff1541004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:04.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff15b30c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 0x7ff1541009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:04.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff15b30c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff154100f30 con 0x7ff1540ffee0 2024-04-16T02:51:04.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff15b30c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff154101070 con 0x7ff1540ff530 2024-04-16T02:51:04.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff1588a7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 0x7ff1541009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff1588a7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 0x7ff1541009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47588/0 (socket says 172.21.15.151:47588) 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff1588a7700 1 -- 172.21.15.151:0/706340990 learned_addr learned my addr 172.21.15.151:0/706340990 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff1590a8700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1540ff530 0x7ff1541004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff1588a7700 1 -- 172.21.15.151:0/706340990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1540ff530 msgr2=0x7ff1541004b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff1588a7700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1540ff530 0x7ff1541004b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.350+0000 7ff1588a7700 1 -- 172.21.15.151:0/706340990 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1440095f0 con 0x7ff1540ffee0 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.351+0000 7ff1588a7700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 0x7ff1541009f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff1440092e0 tx=0x7ff14400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.351+0000 7ff14a7fc700 1 -- 172.21.15.151:0/706340990 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff14401d070 con 0x7ff1540ffee0 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.351+0000 7ff14a7fc700 1 -- 172.21.15.151:0/706340990 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff144005dc0 con 0x7ff1540ffee0 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.351+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1541012f0 con 0x7ff1540ffee0 2024-04-16T02:51:04.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.351+0000 7ff14a7fc700 1 -- 172.21.15.151:0/706340990 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff144017400 con 0x7ff1540ffee0 2024-04-16T02:51:04.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.351+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1541a2cd0 con 0x7ff1540ffee0 2024-04-16T02:51:04.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.352+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff154109150 con 0x7ff1540ffee0 2024-04-16T02:51:04.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.353+0000 7ff14a7fc700 1 -- 172.21.15.151:0/706340990 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff14400f6f0 con 0x7ff1540ffee0 2024-04-16T02:51:04.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.353+0000 7ff14a7fc700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff14006ecc0 0x7ff140071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:04.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.353+0000 7ff14a7fc700 1 -- 172.21.15.151:0/706340990 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff144090350 con 0x7ff1540ffee0 2024-04-16T02:51:04.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.353+0000 7ff1590a8700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff14006ecc0 0x7ff140071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:04.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.354+0000 7ff1590a8700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff14006ecc0 0x7ff140071180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff150005950 tx=0x7ff15000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:04.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.357+0000 7ff14a7fc700 1 -- 172.21.15.151:0/706340990 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff144059180 con 0x7ff1540ffee0 2024-04-16T02:51:04.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.574+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff15402cce0 con 0x7ff14006ecc0 2024-04-16T02:51:04.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.576+0000 7ff14a7fc700 1 -- 172.21.15.151:0/706340990 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff15402cce0 con 0x7ff14006ecc0 2024-04-16T02:51:04.579 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:51:04.580 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff14006ecc0 msgr2=0x7ff140071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff14006ecc0 0x7ff140071180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff150005950 tx=0x7ff15000b410 comp rx=0 tx=0).stop 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 msgr2=0x7ff1541009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 0x7ff1541009f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff1440092e0 tx=0x7ff14400bd70 comp rx=0 tx=0).stop 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 shutdown_connections 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff14006ecc0 0x7ff140071180 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1540ff530 0x7ff1541004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 --2- 172.21.15.151:0/706340990 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1540ffee0 0x7ff1541009f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 >> 172.21.15.151:0/706340990 conn(0x7ff154074df0 msgr2=0x7ff1541050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:04.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 shutdown_connections 2024-04-16T02:51:04.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.578+0000 7ff15b30c700 1 -- 172.21.15.151:0/706340990 wait complete. 2024-04-16T02:51:04.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.683+0000 7f4a59a39700 1 -- 172.21.15.151:0/2942595826 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a54105a50 msgr2=0x7f4a54105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.683+0000 7f4a59a39700 1 --2- 172.21.15.151:0/2942595826 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a54105a50 0x7f4a54105e30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009910 tx=0x7f4a48009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:04.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.683+0000 7f4a59a39700 1 -- 172.21.15.151:0/2942595826 shutdown_connections 2024-04-16T02:51:04.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.683+0000 7f4a59a39700 1 --2- 172.21.15.151:0/2942595826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a540ffa50 0x7f4a540ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.683+0000 7f4a59a39700 1 --2- 172.21.15.151:0/2942595826 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a54105a50 0x7f4a54105e30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.683+0000 7f4a59a39700 1 -- 172.21.15.151:0/2942595826 >> 172.21.15.151:0/2942595826 conn(0x7f4a540fb540 msgr2=0x7f4a540fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:04.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.683+0000 7f4a59a39700 1 -- 172.21.15.151:0/2942595826 shutdown_connections 2024-04-16T02:51:04.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.683+0000 7f4a59a39700 1 -- 172.21.15.151:0/2942595826 wait complete. 2024-04-16T02:51:04.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a59a39700 1 Processor -- start 2024-04-16T02:51:04.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a59a39700 1 -- start start 2024-04-16T02:51:04.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a59a39700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a540ffa50 0x7f4a540fbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:04.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a59a39700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54105a50 0x7f4a540fc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:04.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a59a39700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540fc910 con 0x7f4a540ffa50 2024-04-16T02:51:04.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a59a39700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540fca80 con 0x7f4a54105a50 2024-04-16T02:51:04.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a52ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a540ffa50 0x7f4a540fbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a52ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a540ffa50 0x7f4a540fbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47610/0 (socket says 172.21.15.151:47610) 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a52ffd700 1 -- 172.21.15.151:0/3955657939 learned_addr learned my addr 172.21.15.151:0/3955657939 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.684+0000 7f4a527fc700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54105a50 0x7f4a540fc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a52ffd700 1 -- 172.21.15.151:0/3955657939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54105a50 msgr2=0x7f4a540fc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a52ffd700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54105a50 0x7f4a540fc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a52ffd700 1 -- 172.21.15.151:0/3955657939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a480095f0 con 0x7f4a540ffa50 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a527fc700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54105a50 0x7f4a540fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a52ffd700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a540ffa50 0x7f4a540fbe00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4a48006010 tx=0x7f4a4800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a58a37700 1 -- 172.21.15.151:0/3955657939 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a4801d070 con 0x7f4a540ffa50 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a541a28f0 con 0x7f4a540ffa50 2024-04-16T02:51:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a58a37700 1 -- 172.21.15.151:0/3955657939 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a48004fb0 con 0x7f4a540ffa50 2024-04-16T02:51:04.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a58a37700 1 -- 172.21.15.151:0/3955657939 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a48022ea0 con 0x7f4a540ffa50 2024-04-16T02:51:04.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.685+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a541a2ca0 con 0x7f4a540ffa50 2024-04-16T02:51:04.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.686+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a54109060 con 0x7f4a540ffa50 2024-04-16T02:51:04.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.687+0000 7f4a58a37700 1 -- 172.21.15.151:0/3955657939 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4a48022470 con 0x7f4a540ffa50 2024-04-16T02:51:04.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.687+0000 7f4a58a37700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a3c06ecc0 0x7f4a3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:04.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.687+0000 7f4a58a37700 1 -- 172.21.15.151:0/3955657939 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4a480903f0 con 0x7f4a540ffa50 2024-04-16T02:51:04.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.687+0000 7f4a527fc700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a3c06ecc0 0x7f4a3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:04.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.688+0000 7f4a527fc700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a3c06ecc0 0x7f4a3c071180 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4a540fd5f0 tx=0x7f4a44005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:04.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.692+0000 7f4a58a37700 1 -- 172.21.15.151:0/3955657939 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4a48059250 con 0x7f4a540ffa50 2024-04-16T02:51:04.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.973+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4a5404ea90 con 0x7f4a540ffa50 2024-04-16T02:51:04.977 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:04 smithi151 ceph-mon[112233]: from='client.34738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:04.977 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:04 smithi151 ceph-mon[112233]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:04.977 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:04 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2296102207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:51:04.978 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:04 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1804537268' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:51:04.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.973+0000 7f4a58a37700 1 -- 172.21.15.151:0/3955657939 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4a48022720 con 0x7f4a540ffa50 2024-04-16T02:51:04.978 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:51:04.978 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:51:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:51:04.978 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:51:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a3c06ecc0 msgr2=0x7f4a3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a3c06ecc0 0x7f4a3c071180 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4a540fd5f0 tx=0x7f4a44005f50 comp rx=0 tx=0).stop 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a540ffa50 msgr2=0x7f4a540fbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a540ffa50 0x7f4a540fbe00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4a48006010 tx=0x7f4a4800be20 comp rx=0 tx=0).stop 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 shutdown_connections 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a3c06ecc0 0x7f4a3c071180 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a540ffa50 0x7f4a540fbe00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 --2- 172.21.15.151:0/3955657939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54105a50 0x7f4a540fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 >> 172.21.15.151:0/3955657939 conn(0x7f4a540fb540 msgr2=0x7f4a54068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 shutdown_connections 2024-04-16T02:51:04.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:04.976+0000 7f4a59a39700 1 -- 172.21.15.151:0/3955657939 wait complete. 2024-04-16T02:51:05.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:04 smithi153 ceph-mon[87858]: from='client.34738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:05.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:04 smithi153 ceph-mon[87858]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:05.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:04 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2296102207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:51:05.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:04 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1804537268' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:51:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:05 smithi153 ceph-mon[87858]: pgmap v373: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:05 smithi153 ceph-mon[87858]: from='client.34754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:05 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3955657939' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:51:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:05 smithi151 ceph-mon[112233]: pgmap v373: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:05 smithi151 ceph-mon[112233]: from='client.34754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:05 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3955657939' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:51:08.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:07 smithi153 ceph-mon[87858]: pgmap v374: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:08.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:07 smithi151 ceph-mon[112233]: pgmap v374: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:10.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:09 smithi153 ceph-mon[87858]: pgmap v375: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:10.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:51:10.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:09 smithi151 ceph-mon[112233]: pgmap v375: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:10.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:51:12.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:11 smithi153 ceph-mon[87858]: pgmap v376: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:11 smithi151 ceph-mon[112233]: pgmap v376: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:14.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:13 smithi153 ceph-mon[87858]: pgmap v377: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:13 smithi151 ceph-mon[112233]: pgmap v377: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:16.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:15 smithi153 ceph-mon[87858]: pgmap v378: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:16.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:15 smithi151 ceph-mon[112233]: pgmap v378: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:18.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:17 smithi153 ceph-mon[87858]: pgmap v379: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:18.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:17 smithi151 ceph-mon[112233]: pgmap v379: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:19 smithi151 ceph-mon[112233]: pgmap v380: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:20.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:19 smithi153 ceph-mon[87858]: pgmap v380: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:22.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:21 smithi151 ceph-mon[112233]: pgmap v381: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:21 smithi153 ceph-mon[87858]: pgmap v381: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:24.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:23 smithi153 ceph-mon[87858]: pgmap v382: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:23 smithi151 ceph-mon[112233]: pgmap v382: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:25.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:51:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:51:26.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:25 smithi151 ceph-mon[112233]: pgmap v383: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:26.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:25 smithi153 ceph-mon[87858]: pgmap v383: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:28.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:27 smithi151 ceph-mon[112233]: pgmap v384: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:28.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:27 smithi153 ceph-mon[87858]: pgmap v384: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:29 smithi151 ceph-mon[112233]: pgmap v385: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:30.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:29 smithi153 ceph-mon[87858]: pgmap v385: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:32.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:31 smithi151 ceph-mon[112233]: pgmap v386: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:32.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:31 smithi153 ceph-mon[87858]: pgmap v386: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:34.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:33 smithi151 ceph-mon[112233]: pgmap v387: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:34.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:33 smithi153 ceph-mon[87858]: pgmap v387: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:35.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.095+0000 7fd452146700 1 -- 172.21.15.151:0/3646358813 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c100dc0 msgr2=0x7fd44c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.095+0000 7fd452146700 1 --2- 172.21.15.151:0/3646358813 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c100dc0 0x7fd44c1011a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd43c009910 tx=0x7fd43c009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:35.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.096+0000 7fd452146700 1 -- 172.21.15.151:0/3646358813 shutdown_connections 2024-04-16T02:51:35.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.096+0000 7fd452146700 1 --2- 172.21.15.151:0/3646358813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c101770 0x7fd44c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.096+0000 7fd452146700 1 --2- 172.21.15.151:0/3646358813 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c100dc0 0x7fd44c1011a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.096+0000 7fd452146700 1 -- 172.21.15.151:0/3646358813 >> 172.21.15.151:0/3646358813 conn(0x7fd44c0fc650 msgr2=0x7fd44c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:35.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.096+0000 7fd452146700 1 -- 172.21.15.151:0/3646358813 shutdown_connections 2024-04-16T02:51:35.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.096+0000 7fd452146700 1 -- 172.21.15.151:0/3646358813 wait complete. 2024-04-16T02:51:35.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.096+0000 7fd452146700 1 Processor -- start 2024-04-16T02:51:35.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd452146700 1 -- start start 2024-04-16T02:51:35.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd452146700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c100dc0 0x7fd44c19c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:35.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd452146700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c101770 0x7fd44c19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:35.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd452146700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c19d170 con 0x7fd44c101770 2024-04-16T02:51:35.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd452146700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c196620 con 0x7fd44c100dc0 2024-04-16T02:51:35.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd44affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c101770 0x7fd44c19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:35.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd44affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c101770 0x7fd44c19cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55370/0 (socket says 172.21.15.151:55370) 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd44affd700 1 -- 172.21.15.151:0/3154112210 learned_addr learned my addr 172.21.15.151:0/3154112210 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd44b7fe700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c100dc0 0x7fd44c19c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd44affd700 1 -- 172.21.15.151:0/3154112210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c100dc0 msgr2=0x7fd44c19c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd44affd700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c100dc0 0x7fd44c19c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.097+0000 7fd44affd700 1 -- 172.21.15.151:0/3154112210 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43c0095f0 con 0x7fd44c101770 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.098+0000 7fd44affd700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c101770 0x7fd44c19cae0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd44000d900 tx=0x7fd44000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.098+0000 7fd448ff9700 1 -- 172.21.15.151:0/3154112210 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd440009900 con 0x7fd44c101770 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.098+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44c196900 con 0x7fd44c101770 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.098+0000 7fd448ff9700 1 -- 172.21.15.151:0/3154112210 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd440010460 con 0x7fd44c101770 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.098+0000 7fd448ff9700 1 -- 172.21.15.151:0/3154112210 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd44000f5f0 con 0x7fd44c101770 2024-04-16T02:51:35.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.098+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44c196e50 con 0x7fd44c101770 2024-04-16T02:51:35.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.099+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd44c109160 con 0x7fd44c101770 2024-04-16T02:51:35.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.099+0000 7fd448ff9700 1 -- 172.21.15.151:0/3154112210 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd4400105d0 con 0x7fd44c101770 2024-04-16T02:51:35.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.100+0000 7fd448ff9700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:35.105 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.100+0000 7fd448ff9700 1 -- 172.21.15.151:0/3154112210 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd44008eaa0 con 0x7fd44c101770 2024-04-16T02:51:35.105 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.100+0000 7fd44b7fe700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:35.105 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.100+0000 7fd44b7fe700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd43c006010 tx=0x7fd43c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:35.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.105+0000 7fd448ff9700 1 -- 172.21.15.151:0/3154112210 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd44005b2e0 con 0x7fd44c101770 2024-04-16T02:51:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.322+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd44c197070 con 0x7fd43406ecc0 2024-04-16T02:51:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.323+0000 7fd448ff9700 1 -- 172.21.15.151:0/3154112210 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd44c197070 con 0x7fd43406ecc0 2024-04-16T02:51:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.325+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 msgr2=0x7fd434071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd43c006010 tx=0x7fd43c01a040 comp rx=0 tx=0).stop 2024-04-16T02:51:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c101770 msgr2=0x7fd44c19cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c101770 0x7fd44c19cae0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd44000d900 tx=0x7fd44000dc10 comp rx=0 tx=0).stop 2024-04-16T02:51:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 shutdown_connections 2024-04-16T02:51:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd43406ecc0 0x7fd434071180 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c100dc0 0x7fd44c19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 --2- 172.21.15.151:0/3154112210 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd44c101770 0x7fd44c19cae0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 >> 172.21.15.151:0/3154112210 conn(0x7fd44c0fc650 msgr2=0x7fd44c0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 shutdown_connections 2024-04-16T02:51:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.326+0000 7fd452146700 1 -- 172.21.15.151:0/3154112210 wait complete. 2024-04-16T02:51:35.339 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:51:35.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.431+0000 7f2e399d1700 1 -- 172.21.15.151:0/2351828190 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 msgr2=0x7f2e341011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.431+0000 7f2e399d1700 1 --2- 172.21.15.151:0/2351828190 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 0x7f2e341011b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009910 tx=0x7f2e24009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:35.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.431+0000 7f2e399d1700 1 -- 172.21.15.151:0/2351828190 shutdown_connections 2024-04-16T02:51:35.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.431+0000 7f2e399d1700 1 --2- 172.21.15.151:0/2351828190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e34101780 0x7f2e341057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.431+0000 7f2e399d1700 1 --2- 172.21.15.151:0/2351828190 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 0x7f2e341011b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.431+0000 7f2e399d1700 1 -- 172.21.15.151:0/2351828190 >> 172.21.15.151:0/2351828190 conn(0x7f2e340fc640 msgr2=0x7f2e340fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.431+0000 7f2e399d1700 1 -- 172.21.15.151:0/2351828190 shutdown_connections 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.431+0000 7f2e399d1700 1 -- 172.21.15.151:0/2351828190 wait complete. 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.432+0000 7f2e399d1700 1 Processor -- start 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.432+0000 7f2e399d1700 1 -- start start 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.432+0000 7f2e399d1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 0x7f2e34073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.432+0000 7f2e399d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e34101780 0x7f2e340735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.432+0000 7f2e399d1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34073b10 con 0x7f2e34100dd0 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.432+0000 7f2e399d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34073c50 con 0x7f2e34101780 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.432+0000 7f2e32ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 0x7f2e34073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.432+0000 7f2e32ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 0x7f2e34073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55392/0 (socket says 172.21.15.151:55392) 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e32ffd700 1 -- 172.21.15.151:0/2849044760 learned_addr learned my addr 172.21.15.151:0/2849044760 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e2a7fc700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e34101780 0x7f2e340735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e32ffd700 1 -- 172.21.15.151:0/2849044760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e34101780 msgr2=0x7f2e340735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e32ffd700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e34101780 0x7f2e340735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e32ffd700 1 -- 172.21.15.151:0/2849044760 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e240095f0 con 0x7f2e34100dd0 2024-04-16T02:51:35.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e2a7fc700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e34101780 0x7f2e340735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:51:35.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e32ffd700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 0x7f2e34073090 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2e24006010 tx=0x7f2e2400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:35.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e389cf700 1 -- 172.21.15.151:0/2849044760 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e2401d070 con 0x7f2e34100dd0 2024-04-16T02:51:35.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e389cf700 1 -- 172.21.15.151:0/2849044760 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e2400f460 con 0x7f2e34100dd0 2024-04-16T02:51:35.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e34073ed0 con 0x7f2e34100dd0 2024-04-16T02:51:35.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e389cf700 1 -- 172.21.15.151:0/2849044760 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e24017630 con 0x7f2e34100dd0 2024-04-16T02:51:35.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.433+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e341a2c40 con 0x7f2e34100dd0 2024-04-16T02:51:35.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.434+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e34109170 con 0x7f2e34100dd0 2024-04-16T02:51:35.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.435+0000 7f2e389cf700 1 -- 172.21.15.151:0/2849044760 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2e240178a0 con 0x7f2e34100dd0 2024-04-16T02:51:35.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.435+0000 7f2e389cf700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e2006ecc0 0x7f2e20071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:35.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.435+0000 7f2e389cf700 1 -- 172.21.15.151:0/2849044760 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2e24090260 con 0x7f2e34100dd0 2024-04-16T02:51:35.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.435+0000 7f2e2a7fc700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e2006ecc0 0x7f2e20071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:35.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.436+0000 7f2e2a7fc700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e2006ecc0 0x7f2e20071180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c005d90 tx=0x7f2e1c005d00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:35.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.440+0000 7f2e389cf700 1 -- 172.21.15.151:0/2849044760 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2e24059170 con 0x7f2e34100dd0 2024-04-16T02:51:35.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.658+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e340611d0 con 0x7f2e2006ecc0 2024-04-16T02:51:35.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.660+0000 7f2e389cf700 1 -- 172.21.15.151:0/2849044760 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2e340611d0 con 0x7f2e2006ecc0 2024-04-16T02:51:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e2006ecc0 msgr2=0x7f2e20071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e2006ecc0 0x7f2e20071180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c005d90 tx=0x7f2e1c005d00 comp rx=0 tx=0).stop 2024-04-16T02:51:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 msgr2=0x7f2e34073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 0x7f2e34073090 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2e24006010 tx=0x7f2e2400be20 comp rx=0 tx=0).stop 2024-04-16T02:51:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 shutdown_connections 2024-04-16T02:51:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e2006ecc0 0x7f2e20071180 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e34100dd0 0x7f2e34073090 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 --2- 172.21.15.151:0/2849044760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e34101780 0x7f2e340735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 >> 172.21.15.151:0/2849044760 conn(0x7f2e340fc640 msgr2=0x7f2e340fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:35.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 shutdown_connections 2024-04-16T02:51:35.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.662+0000 7f2e399d1700 1 -- 172.21.15.151:0/2849044760 wait complete. 2024-04-16T02:51:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.767+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/908182106 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0101c70 msgr2=0x7f0aa0102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.767+0000 7f0aa4dcc700 1 --2- 172.21.15.151:0/908182106 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0101c70 0x7f0aa0102050 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009910 tx=0x7f0a90009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:35.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.768+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/908182106 shutdown_connections 2024-04-16T02:51:35.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.768+0000 7f0aa4dcc700 1 --2- 172.21.15.151:0/908182106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0102620 0x7f0aa010ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.768+0000 7f0aa4dcc700 1 --2- 172.21.15.151:0/908182106 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0101c70 0x7f0aa0102050 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.768+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/908182106 >> 172.21.15.151:0/908182106 conn(0x7f0aa00fb530 msgr2=0x7f0aa00fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:35.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.768+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/908182106 shutdown_connections 2024-04-16T02:51:35.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.768+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/908182106 wait complete. 2024-04-16T02:51:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.768+0000 7f0aa4dcc700 1 Processor -- start 2024-04-16T02:51:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.768+0000 7f0aa4dcc700 1 -- start start 2024-04-16T02:51:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0aa4dcc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0101c70 0x7f0aa019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0aa4dcc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0102620 0x7f0aa019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0aa4dcc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa019d0c0 con 0x7f0aa0102620 2024-04-16T02:51:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0aa4dcc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa0196570 con 0x7f0aa0101c70 2024-04-16T02:51:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a9dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0102620 0x7f0aa019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a9dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0102620 0x7f0aa019ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55406/0 (socket says 172.21.15.151:55406) 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a9dd9b700 1 -- 172.21.15.151:0/2753636442 learned_addr learned my addr 172.21.15.151:0/2753636442 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a9e59c700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0101c70 0x7f0aa019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a9dd9b700 1 -- 172.21.15.151:0/2753636442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0101c70 msgr2=0x7f0aa019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a9dd9b700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0101c70 0x7f0aa019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a9dd9b700 1 -- 172.21.15.151:0/2753636442 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a900095f0 con 0x7f0aa0102620 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a9e59c700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0101c70 0x7f0aa019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a9dd9b700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0102620 0x7f0aa019ca30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400d900 tx=0x7f0a9400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.769+0000 7f0a8f7fe700 1 -- 172.21.15.151:0/2753636442 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a94009900 con 0x7f0aa0102620 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.770+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa0196850 con 0x7f0aa0102620 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.770+0000 7f0a8f7fe700 1 -- 172.21.15.151:0/2753636442 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a94010460 con 0x7f0aa0102620 2024-04-16T02:51:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.770+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa0196da0 con 0x7f0aa0102620 2024-04-16T02:51:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.770+0000 7f0a8f7fe700 1 -- 172.21.15.151:0/2753636442 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a9401b5d0 con 0x7f0aa0102620 2024-04-16T02:51:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.771+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa0108210 con 0x7f0aa0102620 2024-04-16T02:51:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.771+0000 7f0a8f7fe700 1 -- 172.21.15.151:0/2753636442 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f0a940105d0 con 0x7f0aa0102620 2024-04-16T02:51:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.771+0000 7f0a8f7fe700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0a8806ecc0 0x7f0a88071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.772+0000 7f0a8f7fe700 1 -- 172.21.15.151:0/2753636442 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0a9408ece0 con 0x7f0aa0102620 2024-04-16T02:51:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.772+0000 7f0a9e59c700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0a8806ecc0 0x7f0a88071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:35.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.772+0000 7f0a9e59c700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0a8806ecc0 0x7f0a88071180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000b5c0 tx=0x7f0a9001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:35.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.776+0000 7f0a8f7fe700 1 -- 172.21.15.151:0/2753636442 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0a9405b5c0 con 0x7f0aa0102620 2024-04-16T02:51:35.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.986+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0aa0197090 con 0x7f0a8806ecc0 2024-04-16T02:51:36.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.997+0000 7f0a8f7fe700 1 -- 172.21.15.151:0/2753636442 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0aa0197090 con 0x7f0a8806ecc0 2024-04-16T02:51:36.000 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:51:36.000 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (13m) 6m ago 24m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:51:36.000 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (24m) 6m ago 24m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:51:36.000 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (23m) 6m ago 23m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:51:36.000 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (10m) 6m ago 24m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (10m) 6m ago 23m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (12m) 6m ago 24m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (7m) 6m ago 18m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (6m) 6m ago 18m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (18m) 6m ago 18m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (6m) 6m ago 18m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (15m) 6m ago 25m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (14m) 6m ago 23m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (11m) 6m ago 26m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (11m) 6m ago 23m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (13m) 6m ago 24m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (13m) 6m ago 23m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (10m) 6m ago 22m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (9m) 6m ago 22m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:51:36.001 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (9m) 6m ago 21m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:51:36.002 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (8m) 6m ago 21m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:51:36.002 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (7m) 6m ago 20m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:51:36.002 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (7m) 6m ago 20m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:51:36.002 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (13m) 6m ago 24m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0a8806ecc0 msgr2=0x7f0a88071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0a8806ecc0 0x7f0a88071180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000b5c0 tx=0x7f0a9001a040 comp rx=0 tx=0).stop 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0102620 msgr2=0x7f0aa019ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0102620 0x7f0aa019ca30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400d900 tx=0x7f0a9400dc10 comp rx=0 tx=0).stop 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 shutdown_connections 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0a8806ecc0 0x7f0a88071180 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0101c70 0x7f0aa019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 --2- 172.21.15.151:0/2753636442 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0aa0102620 0x7f0aa019ca30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 >> 172.21.15.151:0/2753636442 conn(0x7f0aa00fb530 msgr2=0x7f0aa0105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:35.999+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 shutdown_connections 2024-04-16T02:51:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.000+0000 7f0aa4dcc700 1 -- 172.21.15.151:0/2753636442 wait complete. 2024-04-16T02:51:36.069 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:35 smithi151 ceph-mon[112233]: pgmap v388: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:36.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 -- 172.21.15.151:0/2377149073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 msgr2=0x7f228c102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 --2- 172.21.15.151:0/2377149073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 0x7f228c102150 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2288009940 tx=0x7f2288009c50 comp rx=0 tx=0).stop 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 -- 172.21.15.151:0/2377149073 shutdown_connections 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 --2- 172.21.15.151:0/2377149073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f228c102690 0x7f228c10ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 --2- 172.21.15.151:0/2377149073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 0x7f228c102150 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 -- 172.21.15.151:0/2377149073 >> 172.21.15.151:0/2377149073 conn(0x7f228c0fb540 msgr2=0x7f228c0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 -- 172.21.15.151:0/2377149073 shutdown_connections 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 -- 172.21.15.151:0/2377149073 wait complete. 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 Processor -- start 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.124+0000 7f2293427700 1 -- start start 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f2293427700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 0x7f228c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f2293427700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f228c102690 0x7f228c19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f2293427700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c19f2b0 con 0x7f228c101d70 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f2293427700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c198760 con 0x7f228c102690 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22911c3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 0x7f228c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:36.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22911c3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 0x7f228c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55430/0 (socket says 172.21.15.151:55430) 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22911c3700 1 -- 172.21.15.151:0/3551371752 learned_addr learned my addr 172.21.15.151:0/3551371752 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22909c2700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f228c102690 0x7f228c19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22911c3700 1 -- 172.21.15.151:0/3551371752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f228c102690 msgr2=0x7f228c19ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22911c3700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f228c102690 0x7f228c19ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22911c3700 1 -- 172.21.15.151:0/3551371752 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880095f0 con 0x7f228c101d70 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22909c2700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f228c102690 0x7f228c19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22911c3700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 0x7f228c19e6e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f22880055f0 tx=0x7f22880056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.125+0000 7f22827fc700 1 -- 172.21.15.151:0/3551371752 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f228801d070 con 0x7f228c101d70 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.126+0000 7f22827fc700 1 -- 172.21.15.151:0/3551371752 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f228800f460 con 0x7f228c101d70 2024-04-16T02:51:36.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.126+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c1989e0 con 0x7f228c101d70 2024-04-16T02:51:36.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.126+0000 7f22827fc700 1 -- 172.21.15.151:0/3551371752 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2288021640 con 0x7f228c101d70 2024-04-16T02:51:36.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.126+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c198ed0 con 0x7f228c101d70 2024-04-16T02:51:36.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.127+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c10a6e0 con 0x7f228c101d70 2024-04-16T02:51:36.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.127+0000 7f22827fc700 1 -- 172.21.15.151:0/3551371752 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f228800fab0 con 0x7f228c101d70 2024-04-16T02:51:36.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.128+0000 7f22827fc700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227806ecc0 0x7f2278071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:36.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.128+0000 7f22827fc700 1 -- 172.21.15.151:0/3551371752 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f22880900c0 con 0x7f228c101d70 2024-04-16T02:51:36.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.128+0000 7f22909c2700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227806ecc0 0x7f2278071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:36.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.128+0000 7f22909c2700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227806ecc0 0x7f2278071180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f227c005950 tx=0x7f227c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:36.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.132+0000 7f22827fc700 1 -- 172.21.15.151:0/3551371752 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f228805c9a0 con 0x7f228c101d70 2024-04-16T02:51:36.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:35 smithi153 ceph-mon[87858]: pgmap v388: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:36.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.413+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f228c04ea90 con 0x7f228c101d70 2024-04-16T02:51:36.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.414+0000 7f22827fc700 1 -- 172.21.15.151:0/3551371752 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f228805c530 con 0x7f228c101d70 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:51:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:51:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:51:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:51:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:51:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:51:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:51:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:51:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:51:36.418 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:51:36.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227806ecc0 msgr2=0x7f2278071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227806ecc0 0x7f2278071180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f227c005950 tx=0x7f227c0058e0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 msgr2=0x7f228c19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 0x7f228c19e6e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f22880055f0 tx=0x7f22880056d0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 shutdown_connections 2024-04-16T02:51:36.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f227806ecc0 0x7f2278071180 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f228c101d70 0x7f228c19e6e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 --2- 172.21.15.151:0/3551371752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f228c102690 0x7f228c19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 >> 172.21.15.151:0/3551371752 conn(0x7f228c0fb540 msgr2=0x7f228c100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:36.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 shutdown_connections 2024-04-16T02:51:36.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.416+0000 7f2293427700 1 -- 172.21.15.151:0/3551371752 wait complete. 2024-04-16T02:51:36.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.522+0000 7f1e06b48700 1 -- 172.21.15.151:0/2465960226 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e00101d80 msgr2=0x7f1e00102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.522+0000 7f1e06b48700 1 --2- 172.21.15.151:0/2465960226 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e00101d80 0x7f1e00102160 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00b0d0 tx=0x7f1dfc00b3e0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.522+0000 7f1e06b48700 1 -- 172.21.15.151:0/2465960226 shutdown_connections 2024-04-16T02:51:36.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.522+0000 7f1e06b48700 1 --2- 172.21.15.151:0/2465960226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e001026a0 0x7f1e0010ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.522+0000 7f1e06b48700 1 --2- 172.21.15.151:0/2465960226 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e00101d80 0x7f1e00102160 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.522+0000 7f1e06b48700 1 -- 172.21.15.151:0/2465960226 >> 172.21.15.151:0/2465960226 conn(0x7f1e000fb570 msgr2=0x7f1e000fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:36.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.522+0000 7f1e06b48700 1 -- 172.21.15.151:0/2465960226 shutdown_connections 2024-04-16T02:51:36.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.522+0000 7f1e06b48700 1 -- 172.21.15.151:0/2465960226 wait complete. 2024-04-16T02:51:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1e06b48700 1 Processor -- start 2024-04-16T02:51:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1e06b48700 1 -- start start 2024-04-16T02:51:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1e06b48700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e00101d80 0x7f1e0019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1e06b48700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e001026a0 0x7f1e0019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1e06b48700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0019d100 con 0x7f1e001026a0 2024-04-16T02:51:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1e06b48700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e001965b0 con 0x7f1e00101d80 2024-04-16T02:51:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1df7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e001026a0 0x7f1e0019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1df7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e001026a0 0x7f1e0019ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35712/0 (socket says 172.21.15.151:35712) 2024-04-16T02:51:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1df7fff700 1 -- 172.21.15.151:0/1236212519 learned_addr learned my addr 172.21.15.151:0/1236212519 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1e048e4700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e00101d80 0x7f1e0019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1df7fff700 1 -- 172.21.15.151:0/1236212519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e00101d80 msgr2=0x7f1e0019c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1df7fff700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e00101d80 0x7f1e0019c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1df7fff700 1 -- 172.21.15.151:0/1236212519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc009d00 con 0x7f1e001026a0 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.523+0000 7f1df7fff700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e001026a0 0x7f1e0019ca70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1df000d8d0 tx=0x7f1df000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.524+0000 7f1df5ffb700 1 -- 172.21.15.151:0/1236212519 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df0004a00 con 0x7f1e001026a0 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.524+0000 7f1df5ffb700 1 -- 172.21.15.151:0/1236212519 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1df00054e0 con 0x7f1e001026a0 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.524+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e00196890 con 0x7f1e001026a0 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.524+0000 7f1df5ffb700 1 -- 172.21.15.151:0/1236212519 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df000fc20 con 0x7f1e001026a0 2024-04-16T02:51:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.524+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e00196de0 con 0x7f1e001026a0 2024-04-16T02:51:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.525+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e00108290 con 0x7f1e001026a0 2024-04-16T02:51:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.525+0000 7f1df5ffb700 1 -- 172.21.15.151:0/1236212519 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1df0005020 con 0x7f1e001026a0 2024-04-16T02:51:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.526+0000 7f1df5ffb700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1de806ecc0 0x7f1de8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.526+0000 7f1df5ffb700 1 -- 172.21.15.151:0/1236212519 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1df008e7a0 con 0x7f1e001026a0 2024-04-16T02:51:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.526+0000 7f1e048e4700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1de806ecc0 0x7f1de8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:36.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.526+0000 7f1e048e4700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1de806ecc0 0x7f1de8071180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009fd0 tx=0x7f1dfc006000 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:36.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.531+0000 7f1df5ffb700 1 -- 172.21.15.151:0/1236212519 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1df005afe0 con 0x7f1e001026a0 2024-04-16T02:51:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.777+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1e0004ea90 con 0x7f1e001026a0 2024-04-16T02:51:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.777+0000 7f1df5ffb700 1 -- 172.21.15.151:0/1236212519 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f1df0017770 con 0x7f1e001026a0 2024-04-16T02:51:36.780 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:51:36.780 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:51:36.781 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:51:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:51:36.783 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:51:36.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1de806ecc0 msgr2=0x7f1de8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1de806ecc0 0x7f1de8071180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009fd0 tx=0x7f1dfc006000 comp rx=0 tx=0).stop 2024-04-16T02:51:36.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e001026a0 msgr2=0x7f1e0019ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e001026a0 0x7f1e0019ca70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1df000d8d0 tx=0x7f1df000dbe0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 shutdown_connections 2024-04-16T02:51:36.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1de806ecc0 0x7f1de8071180 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e00101d80 0x7f1e0019c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 --2- 172.21.15.151:0/1236212519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e001026a0 0x7f1e0019ca70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 >> 172.21.15.151:0/1236212519 conn(0x7f1e000fb570 msgr2=0x7f1e001053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:36.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 shutdown_connections 2024-04-16T02:51:36.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.780+0000 7f1e06b48700 1 -- 172.21.15.151:0/1236212519 wait complete. 2024-04-16T02:51:36.785 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:51:36.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.887+0000 7f17f94b2700 1 -- 172.21.15.151:0/884162158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f4105a50 msgr2=0x7f17f4105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.887+0000 7f17f94b2700 1 --2- 172.21.15.151:0/884162158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f4105a50 0x7f17f4105e30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009910 tx=0x7f17e8009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.887+0000 7f17f94b2700 1 -- 172.21.15.151:0/884162158 shutdown_connections 2024-04-16T02:51:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.887+0000 7f17f94b2700 1 --2- 172.21.15.151:0/884162158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f40ffa50 0x7f17f40ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.887+0000 7f17f94b2700 1 --2- 172.21.15.151:0/884162158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f4105a50 0x7f17f4105e30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.887+0000 7f17f94b2700 1 -- 172.21.15.151:0/884162158 >> 172.21.15.151:0/884162158 conn(0x7f17f40fb540 msgr2=0x7f17f40fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.887+0000 7f17f94b2700 1 -- 172.21.15.151:0/884162158 shutdown_connections 2024-04-16T02:51:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.887+0000 7f17f94b2700 1 -- 172.21.15.151:0/884162158 wait complete. 2024-04-16T02:51:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.888+0000 7f17f94b2700 1 Processor -- start 2024-04-16T02:51:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.888+0000 7f17f94b2700 1 -- start start 2024-04-16T02:51:36.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.888+0000 7f17f94b2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f40ffa50 0x7f17f419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:36.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.888+0000 7f17f94b2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4105a50 0x7f17f419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:36.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.888+0000 7f17f94b2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f4196550 con 0x7f17f40ffa50 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.888+0000 7f17f94b2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f41966c0 con 0x7f17f4105a50 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.888+0000 7f17f2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f40ffa50 0x7f17f419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.888+0000 7f17f2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f40ffa50 0x7f17f419c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35736/0 (socket says 172.21.15.151:35736) 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.888+0000 7f17f2ffd700 1 -- 172.21.15.151:0/3725055076 learned_addr learned my addr 172.21.15.151:0/3725055076 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17f27fc700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4105a50 0x7f17f419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17f2ffd700 1 -- 172.21.15.151:0/3725055076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4105a50 msgr2=0x7f17f419caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17f2ffd700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4105a50 0x7f17f419caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17f2ffd700 1 -- 172.21.15.151:0/3725055076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80095f0 con 0x7f17f40ffa50 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17f2ffd700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f40ffa50 0x7f17f419c560 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f17e8006010 tx=0x7f17e800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17dbfff700 1 -- 172.21.15.151:0/3725055076 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17e801d070 con 0x7f17f40ffa50 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f4196940 con 0x7f17f40ffa50 2024-04-16T02:51:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17dbfff700 1 -- 172.21.15.151:0/3725055076 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17e8005dc0 con 0x7f17f40ffa50 2024-04-16T02:51:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17dbfff700 1 -- 172.21.15.151:0/3725055076 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17e8017400 con 0x7f17f40ffa50 2024-04-16T02:51:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.889+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f4196e30 con 0x7f17f40ffa50 2024-04-16T02:51:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.890+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f4109060 con 0x7f17f40ffa50 2024-04-16T02:51:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.891+0000 7f17dbfff700 1 -- 172.21.15.151:0/3725055076 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f17e8003a60 con 0x7f17f40ffa50 2024-04-16T02:51:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.891+0000 7f17dbfff700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17dc06ecc0 0x7f17dc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.891+0000 7f17dbfff700 1 -- 172.21.15.151:0/3725055076 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f17e80902f0 con 0x7f17f40ffa50 2024-04-16T02:51:36.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.891+0000 7f17f27fc700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17dc06ecc0 0x7f17dc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:36.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.892+0000 7f17f27fc700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17dc06ecc0 0x7f17dc071180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f17f4197df0 tx=0x7f17e400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:36.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:36.896+0000 7f17dbfff700 1 -- 172.21.15.151:0/3725055076 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f17e80590a0 con 0x7f17f40ffa50 2024-04-16T02:51:37.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.113+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f17f4197110 con 0x7f17dc06ecc0 2024-04-16T02:51:37.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:36 smithi151 ceph-mon[112233]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:37.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:36 smithi151 ceph-mon[112233]: from='client.34766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:37.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:36 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3551371752' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:51:37.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:36 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1236212519' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:51:37.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.115+0000 7f17dbfff700 1 -- 172.21.15.151:0/3725055076 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f17f4197110 con 0x7f17dc06ecc0 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:51:37.118 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:51:37.119 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:51:37.119 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:51:37.119 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:51:37.119 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:51:37.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17dc06ecc0 msgr2=0x7f17dc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:37.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17dc06ecc0 0x7f17dc071180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f17f4197df0 tx=0x7f17e400b410 comp rx=0 tx=0).stop 2024-04-16T02:51:37.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f40ffa50 msgr2=0x7f17f419c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:37.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f40ffa50 0x7f17f419c560 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f17e8006010 tx=0x7f17e800bd70 comp rx=0 tx=0).stop 2024-04-16T02:51:37.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 shutdown_connections 2024-04-16T02:51:37.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f17dc06ecc0 0x7f17dc071180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17f40ffa50 0x7f17f419c560 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 --2- 172.21.15.151:0/3725055076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4105a50 0x7f17f419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 >> 172.21.15.151:0/3725055076 conn(0x7f17f40fb540 msgr2=0x7f17f4068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:37.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 shutdown_connections 2024-04-16T02:51:37.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.117+0000 7f17f94b2700 1 -- 172.21.15.151:0/3725055076 wait complete. 2024-04-16T02:51:37.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.222+0000 7f2379703700 1 -- 172.21.15.151:0/4187680693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 msgr2=0x7f2374100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:37.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.222+0000 7f2379703700 1 --2- 172.21.15.151:0/4187680693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 0x7f2374100050 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2368009910 tx=0x7f2368009c20 comp rx=0 tx=0).stop 2024-04-16T02:51:37.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.222+0000 7f2379703700 1 -- 172.21.15.151:0/4187680693 shutdown_connections 2024-04-16T02:51:37.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.222+0000 7f2379703700 1 --2- 172.21.15.151:0/4187680693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 0x7f2374100050 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.222+0000 7f2379703700 1 --2- 172.21.15.151:0/4187680693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2374105a40 0x7f2374105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.222+0000 7f2379703700 1 -- 172.21.15.151:0/4187680693 >> 172.21.15.151:0/4187680693 conn(0x7f23740755a0 msgr2=0x7f23740759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:37.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.222+0000 7f2379703700 1 -- 172.21.15.151:0/4187680693 shutdown_connections 2024-04-16T02:51:37.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.222+0000 7f2379703700 1 -- 172.21.15.151:0/4187680693 wait complete. 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2379703700 1 Processor -- start 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2379703700 1 -- start start 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2379703700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 0x7f237419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2379703700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2374105a40 0x7f237419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2379703700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237419f2c0 con 0x7f23740ffbd0 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2379703700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2374198770 con 0x7f2374105a40 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2372ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 0x7f237419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2372ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 0x7f237419e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35756/0 (socket says 172.21.15.151:35756) 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2372ffd700 1 -- 172.21.15.151:0/513984004 learned_addr learned my addr 172.21.15.151:0/513984004 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2363fff700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2374105a40 0x7f237419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2372ffd700 1 -- 172.21.15.151:0/513984004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2374105a40 msgr2=0x7f237419ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:37.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2372ffd700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2374105a40 0x7f237419ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2372ffd700 1 -- 172.21.15.151:0/513984004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23680095f0 con 0x7f23740ffbd0 2024-04-16T02:51:37.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.223+0000 7f2372ffd700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 0x7f237419e6f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f23640060b0 tx=0x7f236400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:37.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.224+0000 7f2370ff9700 1 -- 172.21.15.151:0/513984004 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f236400dda0 con 0x7f23740ffbd0 2024-04-16T02:51:37.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.224+0000 7f2370ff9700 1 -- 172.21.15.151:0/513984004 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f236400f040 con 0x7f23740ffbd0 2024-04-16T02:51:37.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.224+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2374198a50 con 0x7f23740ffbd0 2024-04-16T02:51:37.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.224+0000 7f2370ff9700 1 -- 172.21.15.151:0/513984004 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2364014790 con 0x7f23740ffbd0 2024-04-16T02:51:37.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.224+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2374198fa0 con 0x7f23740ffbd0 2024-04-16T02:51:37.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.225+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237404ea90 con 0x7f23740ffbd0 2024-04-16T02:51:37.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.225+0000 7f2370ff9700 1 -- 172.21.15.151:0/513984004 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f23640148f0 con 0x7f23740ffbd0 2024-04-16T02:51:37.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.226+0000 7f2370ff9700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f235c06ecc0 0x7f235c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:51:37.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.226+0000 7f2370ff9700 1 -- 172.21.15.151:0/513984004 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f236408e830 con 0x7f23740ffbd0 2024-04-16T02:51:37.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.226+0000 7f2363fff700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f235c06ecc0 0x7f235c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:51:37.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.226+0000 7f2363fff700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f235c06ecc0 0x7f235c071180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f236800b5c0 tx=0x7f2368005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:51:37.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.230+0000 7f2370ff9700 1 -- 172.21.15.151:0/513984004 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f236405b240 con 0x7f23740ffbd0 2024-04-16T02:51:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:36 smithi153 ceph-mon[87858]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:36 smithi153 ceph-mon[87858]: from='client.34766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:36 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3551371752' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:51:37.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:36 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1236212519' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:51:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.514+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2374199c80 con 0x7f23740ffbd0 2024-04-16T02:51:37.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.515+0000 7f2370ff9700 1 -- 172.21.15.151:0/513984004 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f236408ce60 con 0x7f23740ffbd0 2024-04-16T02:51:37.518 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:51:37.518 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:51:37.518 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:51:37.518 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:51:37.518 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:51:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f235c06ecc0 msgr2=0x7f235c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f235c06ecc0 0x7f235c071180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f236800b5c0 tx=0x7f2368005960 comp rx=0 tx=0).stop 2024-04-16T02:51:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 msgr2=0x7f237419e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:51:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 0x7f237419e6f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f23640060b0 tx=0x7f236400d3b0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 shutdown_connections 2024-04-16T02:51:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f235c06ecc0 0x7f235c071180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23740ffbd0 0x7f237419e6f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 --2- 172.21.15.151:0/513984004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2374105a40 0x7f237419ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:51:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 >> 172.21.15.151:0/513984004 conn(0x7f23740755a0 msgr2=0x7f2374068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:51:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 shutdown_connections 2024-04-16T02:51:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:51:37.517+0000 7f2379703700 1 -- 172.21.15.151:0/513984004 wait complete. 2024-04-16T02:51:38.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:37 smithi151 ceph-mon[112233]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:38.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:37 smithi151 ceph-mon[112233]: pgmap v389: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:38.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:37 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/513984004' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:51:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:37 smithi153 ceph-mon[87858]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:37 smithi153 ceph-mon[87858]: pgmap v389: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:38.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:37 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/513984004' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:51:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:38 smithi151 ceph-mon[112233]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:51:39.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:38 smithi153 ceph-mon[87858]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:51:39.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:51:40.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:39 smithi151 ceph-mon[112233]: pgmap v390: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:40.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:39 smithi153 ceph-mon[87858]: pgmap v390: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:42.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:41 smithi151 ceph-mon[112233]: pgmap v391: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:41 smithi153 ceph-mon[87858]: pgmap v391: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:44.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:43 smithi151 ceph-mon[112233]: pgmap v392: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:44.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:43 smithi153 ceph-mon[87858]: pgmap v392: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:46.187 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:45 smithi151 ceph-mon[112233]: pgmap v393: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:46.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:45 smithi153 ceph-mon[87858]: pgmap v393: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:48.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:47 smithi151 ceph-mon[112233]: pgmap v394: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:48.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:47 smithi153 ceph-mon[87858]: pgmap v394: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:50.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:49 smithi151 ceph-mon[112233]: pgmap v395: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:49 smithi153 ceph-mon[87858]: pgmap v395: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:52.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:51 smithi151 ceph-mon[112233]: pgmap v396: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:52.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:51 smithi153 ceph-mon[87858]: pgmap v396: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:54.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:53 smithi151 ceph-mon[112233]: pgmap v397: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:54.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:51:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:53 smithi153 ceph-mon[87858]: pgmap v397: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:54.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:51:56.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:55 smithi151 ceph-mon[112233]: pgmap v398: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:56.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:55 smithi153 ceph-mon[87858]: pgmap v398: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:57 smithi151 ceph-mon[112233]: pgmap v399: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:51:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:57 smithi153 ceph-mon[87858]: pgmap v399: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:51:59 smithi151 ceph-mon[112233]: pgmap v400: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:00.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:51:59 smithi153 ceph-mon[87858]: pgmap v400: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:02.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:01 smithi151 ceph-mon[112233]: pgmap v401: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:02.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:01 smithi153 ceph-mon[87858]: pgmap v401: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:04.109 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:03 smithi153 ceph-mon[87858]: pgmap v402: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:04.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:03 smithi151 ceph-mon[112233]: pgmap v402: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:05 smithi153 ceph-mon[87858]: pgmap v403: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:06.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:05 smithi151 ceph-mon[112233]: pgmap v403: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:07.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.637+0000 7faff0210700 1 -- 172.21.15.151:0/1319752404 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 msgr2=0x7fafe81057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.637+0000 7faff0210700 1 --2- 172.21.15.151:0/1319752404 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 0x7fafe81057c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009910 tx=0x7fafd8009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.637+0000 7faff0210700 1 -- 172.21.15.151:0/1319752404 shutdown_connections 2024-04-16T02:52:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.637+0000 7faff0210700 1 --2- 172.21.15.151:0/1319752404 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 0x7fafe81057c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.637+0000 7faff0210700 1 --2- 172.21.15.151:0/1319752404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8100dc0 0x7fafe81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.637+0000 7faff0210700 1 -- 172.21.15.151:0/1319752404 >> 172.21.15.151:0/1319752404 conn(0x7fafe80fc650 msgr2=0x7fafe80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.637+0000 7faff0210700 1 -- 172.21.15.151:0/1319752404 shutdown_connections 2024-04-16T02:52:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.637+0000 7faff0210700 1 -- 172.21.15.151:0/1319752404 wait complete. 2024-04-16T02:52:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.638+0000 7faff0210700 1 Processor -- start 2024-04-16T02:52:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.638+0000 7faff0210700 1 -- start start 2024-04-16T02:52:07.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.638+0000 7faff0210700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8100dc0 0x7fafe8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:07.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.638+0000 7faff0210700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 0x7fafe80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:07.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.638+0000 7faff0210700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8073b10 con 0x7fafe8101770 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.638+0000 7faff0210700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8073c50 con 0x7fafe8100dc0 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.638+0000 7fafed7ab700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 0x7fafe80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.638+0000 7fafed7ab700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 0x7fafe80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46328/0 (socket says 172.21.15.151:46328) 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.638+0000 7fafed7ab700 1 -- 172.21.15.151:0/2427582338 learned_addr learned my addr 172.21.15.151:0/2427582338 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7fafedfac700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8100dc0 0x7fafe8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7fafed7ab700 1 -- 172.21.15.151:0/2427582338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8100dc0 msgr2=0x7fafe8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7fafed7ab700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8100dc0 0x7fafe8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7fafed7ab700 1 -- 172.21.15.151:0/2427582338 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd80095f0 con 0x7fafe8101770 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7fafed7ab700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 0x7fafe80735d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fafd80092e0 tx=0x7fafd800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7fafdeffd700 1 -- 172.21.15.151:0/2427582338 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd801d070 con 0x7fafe8101770 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7fafdeffd700 1 -- 172.21.15.151:0/2427582338 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafd8005dc0 con 0x7fafe8101770 2024-04-16T02:52:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7fafdeffd700 1 -- 172.21.15.151:0/2427582338 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd8017400 con 0x7fafe8101770 2024-04-16T02:52:07.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe8073ed0 con 0x7fafe8101770 2024-04-16T02:52:07.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.639+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe81a2c90 con 0x7fafe8101770 2024-04-16T02:52:07.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.640+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe81091c0 con 0x7fafe8101770 2024-04-16T02:52:07.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.641+0000 7fafdeffd700 1 -- 172.21.15.151:0/2427582338 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fafd800f460 con 0x7fafe8101770 2024-04-16T02:52:07.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.641+0000 7fafdeffd700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fafd406ecc0 0x7fafd4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:07.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.641+0000 7fafedfac700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fafd406ecc0 0x7fafd4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:07.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.641+0000 7fafdeffd700 1 -- 172.21.15.151:0/2427582338 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fafd8090360 con 0x7fafe8101770 2024-04-16T02:52:07.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.642+0000 7fafedfac700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fafd406ecc0 0x7fafd4071180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009d10 tx=0x7fafe4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:07.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.646+0000 7fafdeffd700 1 -- 172.21.15.151:0/2427582338 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fafd8059110 con 0x7fafe8101770 2024-04-16T02:52:07.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.863+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fafe80611d0 con 0x7fafd406ecc0 2024-04-16T02:52:07.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.864+0000 7fafdeffd700 1 -- 172.21.15.151:0/2427582338 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fafe80611d0 con 0x7fafd406ecc0 2024-04-16T02:52:07.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.866+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fafd406ecc0 msgr2=0x7fafd4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.866+0000 7faff0210700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fafd406ecc0 0x7fafd4071180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009d10 tx=0x7fafe4009040 comp rx=0 tx=0).stop 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.866+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 msgr2=0x7fafe80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.866+0000 7faff0210700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 0x7fafe80735d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fafd80092e0 tx=0x7fafd800be20 comp rx=0 tx=0).stop 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.867+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 shutdown_connections 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.867+0000 7faff0210700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fafd406ecc0 0x7fafd4071180 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.867+0000 7faff0210700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8100dc0 0x7fafe8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.867+0000 7faff0210700 1 --2- 172.21.15.151:0/2427582338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafe8101770 0x7fafe80735d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.867+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 >> 172.21.15.151:0/2427582338 conn(0x7fafe80fc650 msgr2=0x7fafe80fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.867+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 shutdown_connections 2024-04-16T02:52:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.867+0000 7faff0210700 1 -- 172.21.15.151:0/2427582338 wait complete. 2024-04-16T02:52:07.880 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:52:07.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.974+0000 7f363a603700 1 -- 172.21.15.151:0/1495447521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 msgr2=0x7f36341057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:07.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.974+0000 7f363a603700 1 --2- 172.21.15.151:0/1495447521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 0x7f36341057c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f361c009910 tx=0x7f361c009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:07.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.974+0000 7f363a603700 1 -- 172.21.15.151:0/1495447521 shutdown_connections 2024-04-16T02:52:07.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.974+0000 7f363a603700 1 --2- 172.21.15.151:0/1495447521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 0x7f36341057c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:07.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.974+0000 7f363a603700 1 --2- 172.21.15.151:0/1495447521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3634100dc0 0x7f36341011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:07.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.974+0000 7f363a603700 1 -- 172.21.15.151:0/1495447521 >> 172.21.15.151:0/1495447521 conn(0x7f36340fc650 msgr2=0x7f36340fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:07.977 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:07 smithi151 ceph-mon[112233]: pgmap v404: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:07.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.974+0000 7f363a603700 1 -- 172.21.15.151:0/1495447521 shutdown_connections 2024-04-16T02:52:07.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.974+0000 7f363a603700 1 -- 172.21.15.151:0/1495447521 wait complete. 2024-04-16T02:52:07.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f363a603700 1 Processor -- start 2024-04-16T02:52:07.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f363a603700 1 -- start start 2024-04-16T02:52:07.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f363a603700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3634100dc0 0x7f363419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f363a603700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 0x7f363419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f363a603700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363419d130 con 0x7f3634101770 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f363a603700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36341965e0 con 0x7f3634100dc0 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f362bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 0x7f363419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f362bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 0x7f363419caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46340/0 (socket says 172.21.15.151:46340) 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f362bfff700 1 -- 172.21.15.151:0/901423605 learned_addr learned my addr 172.21.15.151:0/901423605 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f3633fff700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3634100dc0 0x7f363419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f362bfff700 1 -- 172.21.15.151:0/901423605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3634100dc0 msgr2=0x7f363419c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f362bfff700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3634100dc0 0x7f363419c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.975+0000 7f362bfff700 1 -- 172.21.15.151:0/901423605 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0095f0 con 0x7f3634101770 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.976+0000 7f362bfff700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 0x7f363419caa0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f361c0092e0 tx=0x7f361c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:07.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.976+0000 7f3631ffb700 1 -- 172.21.15.151:0/901423605 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f361c01d070 con 0x7f3634101770 2024-04-16T02:52:07.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.976+0000 7f3631ffb700 1 -- 172.21.15.151:0/901423605 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f361c005dc0 con 0x7f3634101770 2024-04-16T02:52:07.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.976+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3634196860 con 0x7f3634101770 2024-04-16T02:52:07.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.976+0000 7f3631ffb700 1 -- 172.21.15.151:0/901423605 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f361c017400 con 0x7f3634101770 2024-04-16T02:52:07.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.976+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3634196d50 con 0x7f3634101770 2024-04-16T02:52:07.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.977+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36341091c0 con 0x7f3634101770 2024-04-16T02:52:07.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.977+0000 7f3631ffb700 1 -- 172.21.15.151:0/901423605 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f361c00f460 con 0x7f3634101770 2024-04-16T02:52:07.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.978+0000 7f3631ffb700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f362006ecc0 0x7f3620071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:07.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.978+0000 7f3631ffb700 1 -- 172.21.15.151:0/901423605 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f361c090410 con 0x7f3634101770 2024-04-16T02:52:07.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.978+0000 7f3633fff700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f362006ecc0 0x7f3620071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:07.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.978+0000 7f3633fff700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f362006ecc0 0x7f3620071180 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3624005950 tx=0x7f36240058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:07.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:07.983+0000 7f3631ffb700 1 -- 172.21.15.151:0/901423605 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f361c05cc50 con 0x7f3634101770 2024-04-16T02:52:08.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.200+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3634197030 con 0x7f362006ecc0 2024-04-16T02:52:08.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.201+0000 7f3631ffb700 1 -- 172.21.15.151:0/901423605 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3634197030 con 0x7f362006ecc0 2024-04-16T02:52:08.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.203+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f362006ecc0 msgr2=0x7f3620071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.203+0000 7f363a603700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f362006ecc0 0x7f3620071180 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3624005950 tx=0x7f36240058e0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.203+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 msgr2=0x7f363419caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.203+0000 7f363a603700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 0x7f363419caa0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f361c0092e0 tx=0x7f361c00be20 comp rx=0 tx=0).stop 2024-04-16T02:52:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.203+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 shutdown_connections 2024-04-16T02:52:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.203+0000 7f363a603700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f362006ecc0 0x7f3620071180 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.203+0000 7f363a603700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3634100dc0 0x7f363419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.203+0000 7f363a603700 1 --2- 172.21.15.151:0/901423605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3634101770 0x7f363419caa0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.203+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 >> 172.21.15.151:0/901423605 conn(0x7f36340fc650 msgr2=0x7f36340fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.204+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 shutdown_connections 2024-04-16T02:52:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.204+0000 7f363a603700 1 -- 172.21.15.151:0/901423605 wait complete. 2024-04-16T02:52:08.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:07 smithi153 ceph-mon[87858]: pgmap v404: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:08.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.310+0000 7fcbe71ea700 1 -- 172.21.15.151:0/474910559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 msgr2=0x7fcbe0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.310+0000 7fcbe71ea700 1 --2- 172.21.15.151:0/474910559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 0x7fcbe0105e20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009910 tx=0x7fcbd4009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:08.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.311+0000 7fcbe71ea700 1 -- 172.21.15.151:0/474910559 shutdown_connections 2024-04-16T02:52:08.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.311+0000 7fcbe71ea700 1 --2- 172.21.15.151:0/474910559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbe00ffa40 0x7fcbe00ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.311+0000 7fcbe71ea700 1 --2- 172.21.15.151:0/474910559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 0x7fcbe0105e20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.311+0000 7fcbe71ea700 1 -- 172.21.15.151:0/474910559 >> 172.21.15.151:0/474910559 conn(0x7fcbe00fb550 msgr2=0x7fcbe00fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:08.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.311+0000 7fcbe71ea700 1 -- 172.21.15.151:0/474910559 shutdown_connections 2024-04-16T02:52:08.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.311+0000 7fcbe71ea700 1 -- 172.21.15.151:0/474910559 wait complete. 2024-04-16T02:52:08.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.311+0000 7fcbe71ea700 1 Processor -- start 2024-04-16T02:52:08.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.311+0000 7fcbe71ea700 1 -- start start 2024-04-16T02:52:08.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbe71ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbe00ffa40 0x7fcbe0197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbe71ea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 0x7fcbe0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbe71ea700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe01980f0 con 0x7fcbe0105a40 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbe71ea700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe0198230 con 0x7fcbe00ffa40 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbdffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 0x7fcbe0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbdffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 0x7fcbe0197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46350/0 (socket says 172.21.15.151:46350) 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbdffff700 1 -- 172.21.15.151:0/2314037486 learned_addr learned my addr 172.21.15.151:0/2314037486 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbe4f86700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbe00ffa40 0x7fcbe0197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbdffff700 1 -- 172.21.15.151:0/2314037486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbe00ffa40 msgr2=0x7fcbe0197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbdffff700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbe00ffa40 0x7fcbe0197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbdffff700 1 -- 172.21.15.151:0/2314037486 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd40095f0 con 0x7fcbe0105a40 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbdffff700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 0x7fcbe0197ad0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00d8d0 tx=0x7fcbcc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbddffb700 1 -- 172.21.15.151:0/2314037486 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbcc0098a0 con 0x7fcbe0105a40 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbddffb700 1 -- 172.21.15.151:0/2314037486 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbcc010460 con 0x7fcbe0105a40 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbddffb700 1 -- 172.21.15.151:0/2314037486 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbcc00f5f0 con 0x7fcbe0105a40 2024-04-16T02:52:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.312+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe01a0600 con 0x7fcbe0105a40 2024-04-16T02:52:08.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.313+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe01a0a70 con 0x7fcbe0105a40 2024-04-16T02:52:08.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.314+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe0108040 con 0x7fcbe0105a40 2024-04-16T02:52:08.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.314+0000 7fcbddffb700 1 -- 172.21.15.151:0/2314037486 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fcbcc009a00 con 0x7fcbe0105a40 2024-04-16T02:52:08.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.314+0000 7fcbddffb700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcbd006ecc0 0x7fcbd0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:08.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.315+0000 7fcbddffb700 1 -- 172.21.15.151:0/2314037486 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcbcc08ed30 con 0x7fcbe0105a40 2024-04-16T02:52:08.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.315+0000 7fcbe4f86700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcbd006ecc0 0x7fcbd0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:08.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.315+0000 7fcbe4f86700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcbd006ecc0 0x7fcbd0071180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4006010 tx=0x7fcbd401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:08.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.319+0000 7fcbddffb700 1 -- 172.21.15.151:0/2314037486 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcbcc057ae0 con 0x7fcbe0105a40 2024-04-16T02:52:08.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.525+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbe002cd60 con 0x7fcbd006ecc0 2024-04-16T02:52:08.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.536+0000 7fcbddffb700 1 -- 172.21.15.151:0/2314037486 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fcbe002cd60 con 0x7fcbd006ecc0 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (13m) 7m ago 24m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (25m) 7m ago 25m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (24m) 6m ago 24m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (11m) 7m ago 25m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (11m) 6m ago 24m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (13m) 7m ago 24m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (7m) 7m ago 19m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (7m) 7m ago 19m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (19m) 6m ago 19m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (6m) 6m ago 19m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (15m) 7m ago 26m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (14m) 6m ago 24m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:52:08.539 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (12m) 7m ago 26m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (11m) 6m ago 23m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (14m) 7m ago 25m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (14m) 6m ago 24m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (10m) 7m ago 23m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (10m) 7m ago 22m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (9m) 7m ago 22m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (9m) 6m ago 21m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (8m) 6m ago 21m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (7m) 6m ago 20m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:52:08.540 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (14m) 7m ago 24m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:52:08.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcbd006ecc0 msgr2=0x7fcbd0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcbd006ecc0 0x7fcbd0071180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4006010 tx=0x7fcbd401a040 comp rx=0 tx=0).stop 2024-04-16T02:52:08.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 msgr2=0x7fcbe0197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 0x7fcbe0197ad0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00d8d0 tx=0x7fcbcc00dbe0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 shutdown_connections 2024-04-16T02:52:08.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcbd006ecc0 0x7fcbd0071180 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbe00ffa40 0x7fcbe0197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 --2- 172.21.15.151:0/2314037486 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbe0105a40 0x7fcbe0197ad0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 >> 172.21.15.151:0/2314037486 conn(0x7fcbe00fb550 msgr2=0x7fcbe00fbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:08.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 shutdown_connections 2024-04-16T02:52:08.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.538+0000 7fcbe71ea700 1 -- 172.21.15.151:0/2314037486 wait complete. 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.642+0000 7f0d22e2a700 1 -- 172.21.15.151:0/3420148854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c100dc0 msgr2=0x7f0d1c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.642+0000 7f0d22e2a700 1 --2- 172.21.15.151:0/3420148854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c100dc0 0x7f0d1c1011a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009910 tx=0x7f0d0c009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.642+0000 7f0d22e2a700 1 -- 172.21.15.151:0/3420148854 shutdown_connections 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.642+0000 7f0d22e2a700 1 --2- 172.21.15.151:0/3420148854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1c101770 0x7f0d1c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.642+0000 7f0d22e2a700 1 --2- 172.21.15.151:0/3420148854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c100dc0 0x7f0d1c1011a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.642+0000 7f0d22e2a700 1 -- 172.21.15.151:0/3420148854 >> 172.21.15.151:0/3420148854 conn(0x7f0d1c0fc650 msgr2=0x7f0d1c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.643+0000 7f0d22e2a700 1 -- 172.21.15.151:0/3420148854 shutdown_connections 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.643+0000 7f0d22e2a700 1 -- 172.21.15.151:0/3420148854 wait complete. 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.643+0000 7f0d22e2a700 1 Processor -- start 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.643+0000 7f0d22e2a700 1 -- start start 2024-04-16T02:52:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.643+0000 7f0d22e2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1c100dc0 0x7f0d1c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.643+0000 7f0d22e2a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c101770 0x7f0d1c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.643+0000 7f0d22e2a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c073b10 con 0x7f0d1c101770 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.643+0000 7f0d22e2a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c073c50 con 0x7f0d1c100dc0 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.643+0000 7f0d1bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c101770 0x7f0d1c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d1bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c101770 0x7f0d1c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46376/0 (socket says 172.21.15.151:46376) 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d1bfff700 1 -- 172.21.15.151:0/1534118923 learned_addr learned my addr 172.21.15.151:0/1534118923 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d20bc6700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1c100dc0 0x7f0d1c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d1bfff700 1 -- 172.21.15.151:0/1534118923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1c100dc0 msgr2=0x7f0d1c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d1bfff700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1c100dc0 0x7f0d1c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d1bfff700 1 -- 172.21.15.151:0/1534118923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c0095f0 con 0x7f0d1c101770 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d20bc6700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1c100dc0 0x7f0d1c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d1bfff700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c101770 0x7f0d1c0735d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400d900 tx=0x7f0d0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d19ffb700 1 -- 172.21.15.151:0/1534118923 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d04009900 con 0x7f0d1c101770 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d19ffb700 1 -- 172.21.15.151:0/1534118923 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d04010460 con 0x7f0d1c101770 2024-04-16T02:52:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c1a2940 con 0x7f0d1c101770 2024-04-16T02:52:08.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d19ffb700 1 -- 172.21.15.151:0/1534118923 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d0400f5f0 con 0x7f0d1c101770 2024-04-16T02:52:08.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.644+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c1a2d50 con 0x7f0d1c101770 2024-04-16T02:52:08.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.646+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c109160 con 0x7f0d1c101770 2024-04-16T02:52:08.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.646+0000 7f0d19ffb700 1 -- 172.21.15.151:0/1534118923 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f0d040105d0 con 0x7f0d1c101770 2024-04-16T02:52:08.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.646+0000 7f0d19ffb700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d0806ecc0 0x7f0d08071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:08.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.646+0000 7f0d19ffb700 1 -- 172.21.15.151:0/1534118923 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0d0408ecc0 con 0x7f0d1c101770 2024-04-16T02:52:08.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.646+0000 7f0d20bc6700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d0806ecc0 0x7f0d08071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:08.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.647+0000 7f0d20bc6700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d0806ecc0 0x7f0d08071180 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c005b40 tx=0x7f0d0c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.651+0000 7f0d19ffb700 1 -- 172.21.15.151:0/1534118923 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0d04057a70 con 0x7f0d1c101770 2024-04-16T02:52:08.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.935+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0d1c04ea90 con 0x7f0d1c101770 2024-04-16T02:52:08.938 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:08 smithi151 ceph-mon[112233]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:08.939 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:52:08.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.936+0000 7f0d19ffb700 1 -- 172.21.15.151:0/1534118923 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f0d0405b130 con 0x7f0d1c101770 2024-04-16T02:52:08.939 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:52:08.939 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:52:08.939 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:52:08.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:52:08.939 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:52:08.940 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:52:08.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d0806ecc0 msgr2=0x7f0d08071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d0806ecc0 0x7f0d08071180 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c005b40 tx=0x7f0d0c01a040 comp rx=0 tx=0).stop 2024-04-16T02:52:08.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c101770 msgr2=0x7f0d1c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:08.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c101770 0x7f0d1c0735d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400d900 tx=0x7f0d0400dcc0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 shutdown_connections 2024-04-16T02:52:08.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d0806ecc0 0x7f0d08071180 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1c100dc0 0x7f0d1c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 --2- 172.21.15.151:0/1534118923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d1c101770 0x7f0d1c0735d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:08.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 >> 172.21.15.151:0/1534118923 conn(0x7f0d1c0fc650 msgr2=0x7f0d1c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:08.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 shutdown_connections 2024-04-16T02:52:08.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:08.938+0000 7f0d22e2a700 1 -- 172.21.15.151:0/1534118923 wait complete. 2024-04-16T02:52:09.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.044+0000 7f30f7073700 1 -- 172.21.15.151:0/1733990010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0069020 msgr2=0x7f30f0105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.044+0000 7f30f7073700 1 --2- 172.21.15.151:0/1733990010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0069020 0x7f30f0105790 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009910 tx=0x7f30dc009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:09.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.044+0000 7f30f7073700 1 -- 172.21.15.151:0/1733990010 shutdown_connections 2024-04-16T02:52:09.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.044+0000 7f30f7073700 1 --2- 172.21.15.151:0/1733990010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0069020 0x7f30f0105790 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.044+0000 7f30f7073700 1 --2- 172.21.15.151:0/1733990010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f0068670 0x7f30f0068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.044+0000 7f30f7073700 1 -- 172.21.15.151:0/1733990010 >> 172.21.15.151:0/1733990010 conn(0x7f30f00755a0 msgr2=0x7f30f00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:09.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.044+0000 7f30f7073700 1 -- 172.21.15.151:0/1733990010 shutdown_connections 2024-04-16T02:52:09.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.044+0000 7f30f7073700 1 -- 172.21.15.151:0/1733990010 wait complete. 2024-04-16T02:52:09.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.045+0000 7f30f7073700 1 Processor -- start 2024-04-16T02:52:09.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.045+0000 7f30f7073700 1 -- start start 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.045+0000 7f30f7073700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0068670 0x7f30f0196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.045+0000 7f30f7073700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f0069020 0x7f30f0196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.045+0000 7f30f7073700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0197240 con 0x7f30f0068670 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.045+0000 7f30f7073700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f01973b0 con 0x7f30f0069020 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.045+0000 7f30f4e0f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0068670 0x7f30f0196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30f4e0f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0068670 0x7f30f0196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46388/0 (socket says 172.21.15.151:46388) 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30f4e0f700 1 -- 172.21.15.151:0/1359283430 learned_addr learned my addr 172.21.15.151:0/1359283430 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30effff700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f0069020 0x7f30f0196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30f4e0f700 1 -- 172.21.15.151:0/1359283430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f0069020 msgr2=0x7f30f0196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30f4e0f700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f0069020 0x7f30f0196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30f4e0f700 1 -- 172.21.15.151:0/1359283430 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30dc0095f0 con 0x7f30f0068670 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30effff700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f0069020 0x7f30f0196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:52:09.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30f4e0f700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0068670 0x7f30f0196700 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f30e400c930 tx=0x7f30e400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:09.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30edffb700 1 -- 172.21.15.151:0/1359283430 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30e4007ad0 con 0x7f30f0068670 2024-04-16T02:52:09.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f019a150 con 0x7f30f0068670 2024-04-16T02:52:09.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30edffb700 1 -- 172.21.15.151:0/1359283430 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30e4007c30 con 0x7f30f0068670 2024-04-16T02:52:09.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.046+0000 7f30edffb700 1 -- 172.21.15.151:0/1359283430 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30e4018610 con 0x7f30f0068670 2024-04-16T02:52:09.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.047+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f019a670 con 0x7f30f0068670 2024-04-16T02:52:09.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.048+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f01090a0 con 0x7f30f0068670 2024-04-16T02:52:09.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.048+0000 7f30edffb700 1 -- 172.21.15.151:0/1359283430 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f30e400f450 con 0x7f30f0068670 2024-04-16T02:52:09.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.049+0000 7f30edffb700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30e006ecc0 0x7f30e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:09.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.049+0000 7f30edffb700 1 -- 172.21.15.151:0/1359283430 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f30e408f0e0 con 0x7f30f0068670 2024-04-16T02:52:09.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.049+0000 7f30effff700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30e006ecc0 0x7f30e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:09.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.049+0000 7f30effff700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30e006ecc0 0x7f30e0071180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f30f0197e20 tx=0x7f30dc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:09.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.053+0000 7f30edffb700 1 -- 172.21.15.151:0/1359283430 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f30e4057e30 con 0x7f30f0068670 2024-04-16T02:52:09.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:08 smithi153 ceph-mon[87858]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:09.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:52:09.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.296+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f30f004ea90 con 0x7f30f0068670 2024-04-16T02:52:09.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.296+0000 7f30edffb700 1 -- 172.21.15.151:0/1359283430 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f30e405b4f0 con 0x7f30f0068670 2024-04-16T02:52:09.299 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:52:09.299 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:52:09.299 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:52:09.300 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:52:09.301 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:52:09.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.298+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30e006ecc0 msgr2=0x7f30e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.298+0000 7f30f7073700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30e006ecc0 0x7f30e0071180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f30f0197e20 tx=0x7f30dc01a040 comp rx=0 tx=0).stop 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.298+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0068670 msgr2=0x7f30f0196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.298+0000 7f30f7073700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0068670 0x7f30f0196700 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f30e400c930 tx=0x7f30e400ccf0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.299+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 shutdown_connections 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.299+0000 7f30f7073700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f30e006ecc0 0x7f30e0071180 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.299+0000 7f30f7073700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30f0068670 0x7f30f0196700 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.299+0000 7f30f7073700 1 --2- 172.21.15.151:0/1359283430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f0069020 0x7f30f0196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.299+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 >> 172.21.15.151:0/1359283430 conn(0x7f30f00755a0 msgr2=0x7f30f0105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.299+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 shutdown_connections 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.299+0000 7f30f7073700 1 -- 172.21.15.151:0/1359283430 wait complete. 2024-04-16T02:52:09.304 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:52:09.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.405+0000 7f5c659a6700 1 -- 172.21.15.151:0/3469534083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c60073ae0 msgr2=0x7f5c6010ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.405+0000 7f5c659a6700 1 --2- 172.21.15.151:0/3469534083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c60073ae0 0x7f5c6010ce70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009910 tx=0x7f5c48009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.405+0000 7f5c659a6700 1 -- 172.21.15.151:0/3469534083 shutdown_connections 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.405+0000 7f5c659a6700 1 --2- 172.21.15.151:0/3469534083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c60073ae0 0x7f5c6010ce70 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.405+0000 7f5c659a6700 1 --2- 172.21.15.151:0/3469534083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c600731c0 0x7f5c600735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.405+0000 7f5c659a6700 1 -- 172.21.15.151:0/3469534083 >> 172.21.15.151:0/3469534083 conn(0x7f5c600fc640 msgr2=0x7f5c600fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.405+0000 7f5c659a6700 1 -- 172.21.15.151:0/3469534083 shutdown_connections 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.405+0000 7f5c659a6700 1 -- 172.21.15.151:0/3469534083 wait complete. 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c659a6700 1 Processor -- start 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c659a6700 1 -- start start 2024-04-16T02:52:09.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c659a6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c600731c0 0x7f5c6019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c659a6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c60073ae0 0x7f5c6019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c659a6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6019f2b0 con 0x7f5c600731c0 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c659a6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c60198760 con 0x7f5c60073ae0 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c5effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c600731c0 0x7f5c6019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c5effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c600731c0 0x7f5c6019e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46400/0 (socket says 172.21.15.151:46400) 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c5effd700 1 -- 172.21.15.151:0/526425416 learned_addr learned my addr 172.21.15.151:0/526425416 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c57fff700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c60073ae0 0x7f5c6019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c5effd700 1 -- 172.21.15.151:0/526425416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c60073ae0 msgr2=0x7f5c6019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c5effd700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c60073ae0 0x7f5c6019ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.406+0000 7f5c5effd700 1 -- 172.21.15.151:0/526425416 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c480095f0 con 0x7f5c600731c0 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.407+0000 7f5c5effd700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c600731c0 0x7f5c6019e6e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000d8d0 tx=0x7f5c5000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.407+0000 7f5c649a4700 1 -- 172.21.15.151:0/526425416 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c500098a0 con 0x7f5c600731c0 2024-04-16T02:52:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.407+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c60198a40 con 0x7f5c600731c0 2024-04-16T02:52:09.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.407+0000 7f5c649a4700 1 -- 172.21.15.151:0/526425416 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c50010460 con 0x7f5c600731c0 2024-04-16T02:52:09.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.407+0000 7f5c649a4700 1 -- 172.21.15.151:0/526425416 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c5000f5f0 con 0x7f5c600731c0 2024-04-16T02:52:09.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.407+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c60198f90 con 0x7f5c600731c0 2024-04-16T02:52:09.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.408+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6010a570 con 0x7f5c600731c0 2024-04-16T02:52:09.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.408+0000 7f5c649a4700 1 -- 172.21.15.151:0/526425416 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5c50009a00 con 0x7f5c600731c0 2024-04-16T02:52:09.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.409+0000 7f5c649a4700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4c06ecc0 0x7f5c4c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:09.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.409+0000 7f5c57fff700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4c06ecc0 0x7f5c4c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:09.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.409+0000 7f5c649a4700 1 -- 172.21.15.151:0/526425416 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5c5008ed30 con 0x7f5c600731c0 2024-04-16T02:52:09.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.410+0000 7f5c57fff700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4c06ecc0 0x7f5c4c071180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5c60199bd0 tx=0x7f5c4801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:09.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.414+0000 7f5c649a4700 1 -- 172.21.15.151:0/526425416 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5c5005b610 con 0x7f5c600731c0 2024-04-16T02:52:09.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.630+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c601991b0 con 0x7f5c4c06ecc0 2024-04-16T02:52:09.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.631+0000 7f5c649a4700 1 -- 172.21.15.151:0/526425416 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5c601991b0 con 0x7f5c4c06ecc0 2024-04-16T02:52:09.634 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:52:09.634 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:52:09.634 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:52:09.635 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:52:09.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4c06ecc0 msgr2=0x7f5c4c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4c06ecc0 0x7f5c4c071180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5c60199bd0 tx=0x7f5c4801a040 comp rx=0 tx=0).stop 2024-04-16T02:52:09.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c600731c0 msgr2=0x7f5c6019e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c600731c0 0x7f5c6019e6e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000d8d0 tx=0x7f5c5000dbe0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 shutdown_connections 2024-04-16T02:52:09.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4c06ecc0 0x7f5c4c071180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c600731c0 0x7f5c6019e6e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 --2- 172.21.15.151:0/526425416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c60073ae0 0x7f5c6019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 >> 172.21.15.151:0/526425416 conn(0x7f5c600fc640 msgr2=0x7f5c601076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:09.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 shutdown_connections 2024-04-16T02:52:09.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.634+0000 7f5c659a6700 1 -- 172.21.15.151:0/526425416 wait complete. 2024-04-16T02:52:09.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.740+0000 7f73a7440700 1 -- 172.21.15.151:0/3541424566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 msgr2=0x7f73a0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.740+0000 7f73a7440700 1 --2- 172.21.15.151:0/3541424566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 0x7f73a0105e20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7390009910 tx=0x7f7390009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:09.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.740+0000 7f73a7440700 1 -- 172.21.15.151:0/3541424566 shutdown_connections 2024-04-16T02:52:09.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.740+0000 7f73a7440700 1 --2- 172.21.15.151:0/3541424566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00ffbd0 0x7f73a0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.740+0000 7f73a7440700 1 --2- 172.21.15.151:0/3541424566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 0x7f73a0105e20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.740+0000 7f73a7440700 1 -- 172.21.15.151:0/3541424566 >> 172.21.15.151:0/3541424566 conn(0x7f73a00755a0 msgr2=0x7f73a00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:09.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.740+0000 7f73a7440700 1 -- 172.21.15.151:0/3541424566 shutdown_connections 2024-04-16T02:52:09.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.740+0000 7f73a7440700 1 -- 172.21.15.151:0/3541424566 wait complete. 2024-04-16T02:52:09.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.741+0000 7f73a7440700 1 Processor -- start 2024-04-16T02:52:09.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.741+0000 7f73a7440700 1 -- start start 2024-04-16T02:52:09.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.741+0000 7f73a7440700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00ffbd0 0x7f73a019b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:09.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.741+0000 7f73a7440700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 0x7f73a019bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:09.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.741+0000 7f73a7440700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a01955c0 con 0x7f73a0105a40 2024-04-16T02:52:09.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.741+0000 7f73a7440700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a0195730 con 0x7f73a00ffbd0 2024-04-16T02:52:09.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.741+0000 7f73a49db700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 0x7f73a019bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:09.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.741+0000 7f73a49db700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 0x7f73a019bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46408/0 (socket says 172.21.15.151:46408) 2024-04-16T02:52:09.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73a49db700 1 -- 172.21.15.151:0/1906441244 learned_addr learned my addr 172.21.15.151:0/1906441244 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73a51dc700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00ffbd0 0x7f73a019b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73a49db700 1 -- 172.21.15.151:0/1906441244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00ffbd0 msgr2=0x7f73a019b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73a49db700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00ffbd0 0x7f73a019b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73a49db700 1 -- 172.21.15.151:0/1906441244 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73900095f0 con 0x7f73a0105a40 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73a51dc700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00ffbd0 0x7f73a019b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73a49db700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 0x7f73a019bb10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f739c00b4b0 tx=0x7f739c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73967fc700 1 -- 172.21.15.151:0/1906441244 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f739c011800 con 0x7f73a0105a40 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a0195a10 con 0x7f73a0105a40 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73967fc700 1 -- 172.21.15.151:0/1906441244 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f739c011e20 con 0x7f73a0105a40 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73967fc700 1 -- 172.21.15.151:0/1906441244 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f739c00f2e0 con 0x7f73a0105a40 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.742+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a0195f60 con 0x7f73a0105a40 2024-04-16T02:52:09.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.743+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a0108040 con 0x7f73a0105a40 2024-04-16T02:52:09.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.744+0000 7f73967fc700 1 -- 172.21.15.151:0/1906441244 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f739c010380 con 0x7f73a0105a40 2024-04-16T02:52:09.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.744+0000 7f73967fc700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f738c06ecc0 0x7f738c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:09.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.744+0000 7f73967fc700 1 -- 172.21.15.151:0/1906441244 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f739c08ea00 con 0x7f73a0105a40 2024-04-16T02:52:09.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.744+0000 7f73a51dc700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f738c06ecc0 0x7f738c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:09.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.745+0000 7f73a51dc700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f738c06ecc0 0x7f738c071180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f739000b5c0 tx=0x7f7390005480 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:09.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:09.749+0000 7f73967fc700 1 -- 172.21.15.151:0/1906441244 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f739c0577b0 con 0x7f73a0105a40 2024-04-16T02:52:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:09 smithi151 ceph-mon[112233]: pgmap v405: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:09 smithi151 ceph-mon[112233]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:09 smithi151 ceph-mon[112233]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:09 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1534118923' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:52:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:09 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1359283430' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:52:10.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.034+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f73a0196bc0 con 0x7f73a0105a40 2024-04-16T02:52:10.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.034+0000 7f73967fc700 1 -- 172.21.15.151:0/1906441244 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f739c05ae70 con 0x7f73a0105a40 2024-04-16T02:52:10.037 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:52:10.037 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:52:10.037 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:52:10.037 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:52:10.037 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:52:10.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.036+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f738c06ecc0 msgr2=0x7f738c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:10.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.036+0000 7f73a7440700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f738c06ecc0 0x7f738c071180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f739000b5c0 tx=0x7f7390005480 comp rx=0 tx=0).stop 2024-04-16T02:52:10.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.036+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 msgr2=0x7f73a019bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:10.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.036+0000 7f73a7440700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 0x7f73a019bb10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f739c00b4b0 tx=0x7f739c00b870 comp rx=0 tx=0).stop 2024-04-16T02:52:10.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.036+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 shutdown_connections 2024-04-16T02:52:10.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.036+0000 7f73a7440700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f738c06ecc0 0x7f738c071180 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:10.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.036+0000 7f73a7440700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00ffbd0 0x7f73a019b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:10.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.036+0000 7f73a7440700 1 --2- 172.21.15.151:0/1906441244 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73a0105a40 0x7f73a019bb10 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:10.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.036+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 >> 172.21.15.151:0/1906441244 conn(0x7f73a00755a0 msgr2=0x7f73a00fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:10.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.037+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 shutdown_connections 2024-04-16T02:52:10.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:10.037+0000 7f73a7440700 1 -- 172.21.15.151:0/1906441244 wait complete. 2024-04-16T02:52:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:09 smithi153 ceph-mon[87858]: pgmap v405: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:09 smithi153 ceph-mon[87858]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:09 smithi153 ceph-mon[87858]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:09 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1534118923' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:52:10.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:09 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1359283430' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:52:11.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:10 smithi151 ceph-mon[112233]: from='client.34810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:11.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:10 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1906441244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:52:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:10 smithi153 ceph-mon[87858]: from='client.34810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:11.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:10 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1906441244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:52:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:11 smithi151 ceph-mon[112233]: pgmap v406: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:12.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:11 smithi153 ceph-mon[87858]: pgmap v406: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:13.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:12 smithi151 ceph-mon[112233]: pgmap v407: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:13.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:12 smithi153 ceph-mon[87858]: pgmap v407: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:15.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:15 smithi151 ceph-mon[112233]: pgmap v408: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:15.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:15 smithi153 ceph-mon[87858]: pgmap v408: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:17.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:17 smithi151 ceph-mon[112233]: pgmap v409: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:17.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:17 smithi153 ceph-mon[87858]: pgmap v409: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:19.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:19 smithi151 ceph-mon[112233]: pgmap v410: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:19.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:19 smithi153 ceph-mon[87858]: pgmap v410: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:21.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:21 smithi151 ceph-mon[112233]: pgmap v411: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:21.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:21 smithi153 ceph-mon[87858]: pgmap v411: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:23.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:23 smithi151 ceph-mon[112233]: pgmap v412: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:23.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:23 smithi153 ceph-mon[87858]: pgmap v412: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:24.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:52:24.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:52:25.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:25 smithi151 ceph-mon[112233]: pgmap v413: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:25.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:25 smithi153 ceph-mon[87858]: pgmap v413: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:27.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:27 smithi151 ceph-mon[112233]: pgmap v414: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:27.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:27 smithi153 ceph-mon[87858]: pgmap v414: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:29.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:29 smithi151 ceph-mon[112233]: pgmap v415: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:29.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:29 smithi153 ceph-mon[87858]: pgmap v415: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:31.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:31 smithi151 ceph-mon[112233]: pgmap v416: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:31.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:31 smithi153 ceph-mon[87858]: pgmap v416: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:33.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:33 smithi151 ceph-mon[112233]: pgmap v417: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:33.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:33 smithi153 ceph-mon[87858]: pgmap v417: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:35.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:35 smithi151 ceph-mon[112233]: pgmap v418: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:35.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:35 smithi153 ceph-mon[87858]: pgmap v418: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:37.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:37 smithi151 ceph-mon[112233]: pgmap v419: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:37.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:37 smithi153 ceph-mon[87858]: pgmap v419: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:39.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:39 smithi153 ceph-mon[87858]: pgmap v420: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:39.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:52:39.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:39 smithi151 ceph-mon[112233]: pgmap v420: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:39.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:52:40.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.155+0000 7f77f481a700 1 -- 172.21.15.151:0/1602663931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 msgr2=0x7f77f010ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.155+0000 7f77f481a700 1 --2- 172.21.15.151:0/1602663931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 0x7f77f010ab00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009910 tx=0x7f77e4009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:40.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.155+0000 7f77f481a700 1 -- 172.21.15.151:0/1602663931 shutdown_connections 2024-04-16T02:52:40.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.155+0000 7f77f481a700 1 --2- 172.21.15.151:0/1602663931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 0x7f77f010ab00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.155+0000 7f77f481a700 1 --2- 172.21.15.151:0/1602663931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77f0101c60 0x7f77f0102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.155+0000 7f77f481a700 1 -- 172.21.15.151:0/1602663931 >> 172.21.15.151:0/1602663931 conn(0x7f77f00fb520 msgr2=0x7f77f00fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:40.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.155+0000 7f77f481a700 1 -- 172.21.15.151:0/1602663931 shutdown_connections 2024-04-16T02:52:40.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.155+0000 7f77f481a700 1 -- 172.21.15.151:0/1602663931 wait complete. 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.156+0000 7f77f481a700 1 Processor -- start 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77f481a700 1 -- start start 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77f481a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77f0101c60 0x7f77f019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77f481a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 0x7f77f019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77f481a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f019d0e0 con 0x7f77f0102610 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77f481a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f0196590 con 0x7f77f0101c60 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77edd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 0x7f77f019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77edd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 0x7f77f019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45832/0 (socket says 172.21.15.151:45832) 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77edd9b700 1 -- 172.21.15.151:0/4195029670 learned_addr learned my addr 172.21.15.151:0/4195029670 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77ee59c700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77f0101c60 0x7f77f019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:40.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77edd9b700 1 -- 172.21.15.151:0/4195029670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77f0101c60 msgr2=0x7f77f019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77edd9b700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77f0101c60 0x7f77f019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77edd9b700 1 -- 172.21.15.151:0/4195029670 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e40095f0 con 0x7f77f0102610 2024-04-16T02:52:40.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.157+0000 7f77edd9b700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 0x7f77f019ca50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f77e4005f50 tx=0x7f77e400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:40.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.158+0000 7f77df7fe700 1 -- 172.21.15.151:0/4195029670 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77e401d070 con 0x7f77f0102610 2024-04-16T02:52:40.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.158+0000 7f77df7fe700 1 -- 172.21.15.151:0/4195029670 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77e400f460 con 0x7f77f0102610 2024-04-16T02:52:40.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.158+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f0196810 con 0x7f77f0102610 2024-04-16T02:52:40.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.158+0000 7f77df7fe700 1 -- 172.21.15.151:0/4195029670 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77e4017630 con 0x7f77f0102610 2024-04-16T02:52:40.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.158+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f0196d00 con 0x7f77f0102610 2024-04-16T02:52:40.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.159+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f0108200 con 0x7f77f0102610 2024-04-16T02:52:40.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.159+0000 7f77df7fe700 1 -- 172.21.15.151:0/4195029670 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f77e400f5d0 con 0x7f77f0102610 2024-04-16T02:52:40.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.160+0000 7f77df7fe700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f77d806ecc0 0x7f77d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:40.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.160+0000 7f77df7fe700 1 -- 172.21.15.151:0/4195029670 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f77e40903a0 con 0x7f77f0102610 2024-04-16T02:52:40.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.160+0000 7f77ee59c700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f77d806ecc0 0x7f77d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:40.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.160+0000 7f77ee59c700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f77d806ecc0 0x7f77d8071180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f77e0005950 tx=0x7f77e000a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:40.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.165+0000 7f77df7fe700 1 -- 172.21.15.151:0/4195029670 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f77e405cc80 con 0x7f77f0102610 2024-04-16T02:52:40.387 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.384+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f77f002ccf0 con 0x7f77d806ecc0 2024-04-16T02:52:40.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.385+0000 7f77df7fe700 1 -- 172.21.15.151:0/4195029670 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f77f002ccf0 con 0x7f77d806ecc0 2024-04-16T02:52:40.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f77d806ecc0 msgr2=0x7f77d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f77d806ecc0 0x7f77d8071180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f77e0005950 tx=0x7f77e000a300 comp rx=0 tx=0).stop 2024-04-16T02:52:40.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 msgr2=0x7f77f019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 0x7f77f019ca50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f77e4005f50 tx=0x7f77e400bd70 comp rx=0 tx=0).stop 2024-04-16T02:52:40.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 shutdown_connections 2024-04-16T02:52:40.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f77d806ecc0 0x7f77d8071180 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77f0101c60 0x7f77f019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 --2- 172.21.15.151:0/4195029670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77f0102610 0x7f77f019ca50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 >> 172.21.15.151:0/4195029670 conn(0x7f77f00fb520 msgr2=0x7f77f0105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:40.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 shutdown_connections 2024-04-16T02:52:40.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.387+0000 7f77f481a700 1 -- 172.21.15.151:0/4195029670 wait complete. 2024-04-16T02:52:40.402 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:52:40.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.496+0000 7f79a0f36700 1 -- 172.21.15.151:0/1884043293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c0ffa70 msgr2=0x7f799c0ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.496+0000 7f79a0f36700 1 --2- 172.21.15.151:0/1884043293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c0ffa70 0x7f799c0ffef0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7990009910 tx=0x7f7990009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:40.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.496+0000 7f79a0f36700 1 -- 172.21.15.151:0/1884043293 shutdown_connections 2024-04-16T02:52:40.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.496+0000 7f79a0f36700 1 --2- 172.21.15.151:0/1884043293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c0ffa70 0x7f799c0ffef0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.496+0000 7f79a0f36700 1 --2- 172.21.15.151:0/1884043293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f799c105a70 0x7f799c105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.496+0000 7f79a0f36700 1 -- 172.21.15.151:0/1884043293 >> 172.21.15.151:0/1884043293 conn(0x7f799c0fb5a0 msgr2=0x7f799c0fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:40.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.497+0000 7f79a0f36700 1 -- 172.21.15.151:0/1884043293 shutdown_connections 2024-04-16T02:52:40.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.497+0000 7f79a0f36700 1 -- 172.21.15.151:0/1884043293 wait complete. 2024-04-16T02:52:40.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.497+0000 7f79a0f36700 1 Processor -- start 2024-04-16T02:52:40.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.497+0000 7f79a0f36700 1 -- start start 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f79a0f36700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f799c0ffa70 0x7f799c1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f79a0f36700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c105a70 0x7f799c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f79a0f36700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c198130 con 0x7f799c105a70 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f79a0f36700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c198270 con 0x7f799c0ffa70 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f7999d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c105a70 0x7f799c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f7999d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c105a70 0x7f799c197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45848/0 (socket says 172.21.15.151:45848) 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f7999d9b700 1 -- 172.21.15.151:0/4086713915 learned_addr learned my addr 172.21.15.151:0/4086713915 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f799a59c700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f799c0ffa70 0x7f799c1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f7999d9b700 1 -- 172.21.15.151:0/4086713915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f799c0ffa70 msgr2=0x7f799c1975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f7999d9b700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f799c0ffa70 0x7f799c1975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f7999d9b700 1 -- 172.21.15.151:0/4086713915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79900095f0 con 0x7f799c105a70 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f799a59c700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f799c0ffa70 0x7f799c1975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:52:40.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f7999d9b700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c105a70 0x7f799c197b10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7990006010 tx=0x7f799000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:40.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f798b7fe700 1 -- 172.21.15.151:0/4086713915 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f799001d070 con 0x7f799c105a70 2024-04-16T02:52:40.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799c1a0640 con 0x7f799c105a70 2024-04-16T02:52:40.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f798b7fe700 1 -- 172.21.15.151:0/4086713915 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7990004fb0 con 0x7f799c105a70 2024-04-16T02:52:40.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.498+0000 7f798b7fe700 1 -- 172.21.15.151:0/4086713915 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7990022ea0 con 0x7f799c105a70 2024-04-16T02:52:40.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.499+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c1a0a50 con 0x7f799c105a70 2024-04-16T02:52:40.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.499+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799c108000 con 0x7f799c105a70 2024-04-16T02:52:40.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.500+0000 7f798b7fe700 1 -- 172.21.15.151:0/4086713915 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7990022470 con 0x7f799c105a70 2024-04-16T02:52:40.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.500+0000 7f798b7fe700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f798406ecc0 0x7f7984071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:40.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.500+0000 7f798b7fe700 1 -- 172.21.15.151:0/4086713915 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f79900903f0 con 0x7f799c105a70 2024-04-16T02:52:40.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.500+0000 7f799a59c700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f798406ecc0 0x7f7984071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:40.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.501+0000 7f799a59c700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f798406ecc0 0x7f7984071180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f798c005fd0 tx=0x7f798c005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:40.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.505+0000 7f798b7fe700 1 -- 172.21.15.151:0/4086713915 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7990059250 con 0x7f799c105a70 2024-04-16T02:52:40.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.724+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f799c02cd30 con 0x7f798406ecc0 2024-04-16T02:52:40.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.726+0000 7f798b7fe700 1 -- 172.21.15.151:0/4086713915 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f799c02cd30 con 0x7f798406ecc0 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f798406ecc0 msgr2=0x7f7984071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f798406ecc0 0x7f7984071180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f798c005fd0 tx=0x7f798c005dc0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c105a70 msgr2=0x7f799c197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c105a70 0x7f799c197b10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7990006010 tx=0x7f799000be20 comp rx=0 tx=0).stop 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 shutdown_connections 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f798406ecc0 0x7f7984071180 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f799c0ffa70 0x7f799c1975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 --2- 172.21.15.151:0/4086713915 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f799c105a70 0x7f799c197b10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 >> 172.21.15.151:0/4086713915 conn(0x7f799c0fb5a0 msgr2=0x7f799c0fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:40.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 shutdown_connections 2024-04-16T02:52:40.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.728+0000 7f79a0f36700 1 -- 172.21.15.151:0/4086713915 wait complete. 2024-04-16T02:52:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.834+0000 7f04b5853700 1 -- 172.21.15.151:0/3778728083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 msgr2=0x7f04b0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.834+0000 7f04b5853700 1 --2- 172.21.15.151:0/3778728083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 0x7f04b0100050 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0498009910 tx=0x7f0498009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.834+0000 7f04b5853700 1 -- 172.21.15.151:0/3778728083 shutdown_connections 2024-04-16T02:52:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.834+0000 7f04b5853700 1 --2- 172.21.15.151:0/3778728083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 0x7f04b0100050 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.834+0000 7f04b5853700 1 --2- 172.21.15.151:0/3778728083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b0105a40 0x7f04b0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.834+0000 7f04b5853700 1 -- 172.21.15.151:0/3778728083 >> 172.21.15.151:0/3778728083 conn(0x7f04b00755a0 msgr2=0x7f04b00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.834+0000 7f04b5853700 1 -- 172.21.15.151:0/3778728083 shutdown_connections 2024-04-16T02:52:40.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.834+0000 7f04b5853700 1 -- 172.21.15.151:0/3778728083 wait complete. 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04b5853700 1 Processor -- start 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04b5853700 1 -- start start 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04b5853700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 0x7f04b019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04b5853700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b0105a40 0x7f04b019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04b5853700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b019d0d0 con 0x7f04b00ffbd0 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04b5853700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0196580 con 0x7f04b0105a40 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04aeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 0x7f04b019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04aeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 0x7f04b019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45862/0 (socket says 172.21.15.151:45862) 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04aeffd700 1 -- 172.21.15.151:0/2810101380 learned_addr learned my addr 172.21.15.151:0/2810101380 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:40.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.835+0000 7f04a67fc700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b0105a40 0x7f04b019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.836+0000 7f04aeffd700 1 -- 172.21.15.151:0/2810101380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b0105a40 msgr2=0x7f04b019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.836+0000 7f04aeffd700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b0105a40 0x7f04b019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.836+0000 7f04aeffd700 1 -- 172.21.15.151:0/2810101380 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04980095f0 con 0x7f04b00ffbd0 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.836+0000 7f04aeffd700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 0x7f04b019c500 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f04a000d900 tx=0x7f04a000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.836+0000 7f04b4851700 1 -- 172.21.15.151:0/2810101380 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04a00041f0 con 0x7f04b00ffbd0 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.836+0000 7f04b4851700 1 -- 172.21.15.151:0/2810101380 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04a0004cf0 con 0x7f04b00ffbd0 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.836+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b0196860 con 0x7f04b00ffbd0 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.836+0000 7f04b4851700 1 -- 172.21.15.151:0/2810101380 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04a000b750 con 0x7f04b00ffbd0 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.836+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b0196db0 con 0x7f04b00ffbd0 2024-04-16T02:52:40.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.837+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b0108fe0 con 0x7f04b00ffbd0 2024-04-16T02:52:40.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.838+0000 7f04b4851700 1 -- 172.21.15.151:0/2810101380 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f04a0004750 con 0x7f04b00ffbd0 2024-04-16T02:52:40.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.839+0000 7f04b4851700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f049c06ecc0 0x7f049c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:40.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.839+0000 7f04b4851700 1 -- 172.21.15.151:0/2810101380 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f04a008ee00 con 0x7f04b00ffbd0 2024-04-16T02:52:40.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.839+0000 7f04a67fc700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f049c06ecc0 0x7f049c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:40.843 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.839+0000 7f04a67fc700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f049c06ecc0 0x7f049c071180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f04b0104d90 tx=0x7f049801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:40.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:40.843+0000 7f04b4851700 1 -- 172.21.15.151:0/2810101380 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f04a005b640 con 0x7f04b00ffbd0 2024-04-16T02:52:41.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.051+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f04b0197070 con 0x7f049c06ecc0 2024-04-16T02:52:41.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.061+0000 7f04b4851700 1 -- 172.21.15.151:0/2810101380 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f04b0197070 con 0x7f049c06ecc0 2024-04-16T02:52:41.064 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:52:41.064 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (14m) 7m ago 25m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:52:41.064 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (25m) 7m ago 25m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:52:41.064 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (24m) 7m ago 24m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (11m) 7m ago 25m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (11m) 7m ago 24m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (13m) 7m ago 25m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (8m) 7m ago 20m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (7m) 7m ago 19m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (19m) 7m ago 19m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (7m) 7m ago 19m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (16m) 7m ago 26m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (15m) 7m ago 24m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (12m) 7m ago 27m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (12m) 7m ago 24m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (14m) 7m ago 25m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:52:41.065 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (14m) 7m ago 24m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:52:41.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (11m) 7m ago 23m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:52:41.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (10m) 7m ago 23m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:52:41.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (10m) 7m ago 22m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:52:41.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (9m) 7m ago 22m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:52:41.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (8m) 7m ago 21m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:52:41.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (8m) 7m ago 21m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:52:41.066 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (14m) 7m ago 25m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:52:41.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f049c06ecc0 msgr2=0x7f049c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f049c06ecc0 0x7f049c071180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f04b0104d90 tx=0x7f049801a040 comp rx=0 tx=0).stop 2024-04-16T02:52:41.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 msgr2=0x7f04b019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 0x7f04b019c500 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f04a000d900 tx=0x7f04a000dc10 comp rx=0 tx=0).stop 2024-04-16T02:52:41.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 shutdown_connections 2024-04-16T02:52:41.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f049c06ecc0 0x7f049c071180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04b00ffbd0 0x7f04b019c500 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 --2- 172.21.15.151:0/2810101380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b0105a40 0x7f04b019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 >> 172.21.15.151:0/2810101380 conn(0x7f04b00755a0 msgr2=0x7f04b0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:41.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 shutdown_connections 2024-04-16T02:52:41.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.064+0000 7f04b5853700 1 -- 172.21.15.151:0/2810101380 wait complete. 2024-04-16T02:52:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.168+0000 7f5849bbe700 1 -- 172.21.15.151:0/1627258130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 msgr2=0x7f58441011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.168+0000 7f5849bbe700 1 --2- 172.21.15.151:0/1627258130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 0x7f58441011a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5834009910 tx=0x7f5834009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.168+0000 7f5849bbe700 1 -- 172.21.15.151:0/1627258130 shutdown_connections 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.168+0000 7f5849bbe700 1 --2- 172.21.15.151:0/1627258130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5844101770 0x7f58441057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.168+0000 7f5849bbe700 1 --2- 172.21.15.151:0/1627258130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 0x7f58441011a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.168+0000 7f5849bbe700 1 -- 172.21.15.151:0/1627258130 >> 172.21.15.151:0/1627258130 conn(0x7f58440fc650 msgr2=0x7f58440fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.169+0000 7f5849bbe700 1 -- 172.21.15.151:0/1627258130 shutdown_connections 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.169+0000 7f5849bbe700 1 -- 172.21.15.151:0/1627258130 wait complete. 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.169+0000 7f5849bbe700 1 Processor -- start 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.169+0000 7f5849bbe700 1 -- start start 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.169+0000 7f5849bbe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 0x7f584419c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.169+0000 7f5849bbe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5844101770 0x7f584419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f5849bbe700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584419d110 con 0x7f5844100dc0 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f5849bbe700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441965c0 con 0x7f5844101770 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f58437fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 0x7f584419c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f58437fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 0x7f584419c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45878/0 (socket says 172.21.15.151:45878) 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f58437fe700 1 -- 172.21.15.151:0/2366369404 learned_addr learned my addr 172.21.15.151:0/2366369404 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f58437fe700 1 -- 172.21.15.151:0/2366369404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5844101770 msgr2=0x7f584419ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f5842ffd700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5844101770 0x7f584419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f58437fe700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5844101770 0x7f584419ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f58437fe700 1 -- 172.21.15.151:0/2366369404 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340095f0 con 0x7f5844100dc0 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f5842ffd700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5844101770 0x7f584419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f58437fe700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 0x7f584419c540 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5834006010 tx=0x7f583400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f5840ff9700 1 -- 172.21.15.151:0/2366369404 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f583401d070 con 0x7f5844100dc0 2024-04-16T02:52:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5844196840 con 0x7f5844100dc0 2024-04-16T02:52:41.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f5840ff9700 1 -- 172.21.15.151:0/2366369404 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f583400f460 con 0x7f5844100dc0 2024-04-16T02:52:41.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f5840ff9700 1 -- 172.21.15.151:0/2366369404 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5834017630 con 0x7f5844100dc0 2024-04-16T02:52:41.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.170+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5844196d30 con 0x7f5844100dc0 2024-04-16T02:52:41.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.171+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5844109160 con 0x7f5844100dc0 2024-04-16T02:52:41.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.172+0000 7f5840ff9700 1 -- 172.21.15.151:0/2366369404 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f583400f5d0 con 0x7f5844100dc0 2024-04-16T02:52:41.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.172+0000 7f5840ff9700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f582c06ecc0 0x7f582c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:41.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.172+0000 7f5840ff9700 1 -- 172.21.15.151:0/2366369404 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5834090310 con 0x7f5844100dc0 2024-04-16T02:52:41.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.172+0000 7f5842ffd700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f582c06ecc0 0x7f582c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:41.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.173+0000 7f5842ffd700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f582c06ecc0 0x7f582c071180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5838005fd0 tx=0x7f5838005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:41.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.177+0000 7f5840ff9700 1 -- 172.21.15.151:0/2366369404 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f583405cbf0 con 0x7f5844100dc0 2024-04-16T02:52:41.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.460+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5844197010 con 0x7f5844100dc0 2024-04-16T02:52:41.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.461+0000 7f5840ff9700 1 -- 172.21.15.151:0/2366369404 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f583405c780 con 0x7f5844100dc0 2024-04-16T02:52:41.464 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:52:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:52:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:52:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:52:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:52:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:52:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:52:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:52:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:52:41.466 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:52:41.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.463+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f582c06ecc0 msgr2=0x7f582c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.463+0000 7f5849bbe700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f582c06ecc0 0x7f582c071180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5838005fd0 tx=0x7f5838005ee0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.463+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 msgr2=0x7f584419c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.463+0000 7f5849bbe700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 0x7f584419c540 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5834006010 tx=0x7f583400be20 comp rx=0 tx=0).stop 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.464+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 shutdown_connections 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.464+0000 7f5849bbe700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f582c06ecc0 0x7f582c071180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.464+0000 7f5849bbe700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5844100dc0 0x7f584419c540 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.464+0000 7f5849bbe700 1 --2- 172.21.15.151:0/2366369404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5844101770 0x7f584419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.464+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 >> 172.21.15.151:0/2366369404 conn(0x7f58440fc650 msgr2=0x7f58440fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.464+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 shutdown_connections 2024-04-16T02:52:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.464+0000 7f5849bbe700 1 -- 172.21.15.151:0/2366369404 wait complete. 2024-04-16T02:52:41.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.569+0000 7ff41a8a7700 1 -- 172.21.15.151:0/3219774256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414100dc0 msgr2=0x7ff4141011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.569+0000 7ff41a8a7700 1 --2- 172.21.15.151:0/3219774256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414100dc0 0x7ff4141011a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff408009910 tx=0x7ff408009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:41.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.569+0000 7ff41a8a7700 1 -- 172.21.15.151:0/3219774256 shutdown_connections 2024-04-16T02:52:41.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.569+0000 7ff41a8a7700 1 --2- 172.21.15.151:0/3219774256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff414101770 0x7ff4141057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.569+0000 7ff41a8a7700 1 --2- 172.21.15.151:0/3219774256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414100dc0 0x7ff4141011a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.569+0000 7ff41a8a7700 1 -- 172.21.15.151:0/3219774256 >> 172.21.15.151:0/3219774256 conn(0x7ff4140fc650 msgr2=0x7ff4140fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:41.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.569+0000 7ff41a8a7700 1 -- 172.21.15.151:0/3219774256 shutdown_connections 2024-04-16T02:52:41.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.569+0000 7ff41a8a7700 1 -- 172.21.15.151:0/3219774256 wait complete. 2024-04-16T02:52:41.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.570+0000 7ff41a8a7700 1 Processor -- start 2024-04-16T02:52:41.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.570+0000 7ff41a8a7700 1 -- start start 2024-04-16T02:52:41.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.570+0000 7ff41a8a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff414100dc0 0x7ff41419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:41.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.570+0000 7ff41a8a7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414101770 0x7ff41419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:41.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.570+0000 7ff41a8a7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41419d070 con 0x7ff414101770 2024-04-16T02:52:41.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.570+0000 7ff41a8a7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4141965b0 con 0x7ff414100dc0 2024-04-16T02:52:41.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.570+0000 7ff4137fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414101770 0x7ff41419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:41.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.570+0000 7ff4137fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414101770 0x7ff41419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45896/0 (socket says 172.21.15.151:45896) 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.570+0000 7ff4137fe700 1 -- 172.21.15.151:0/2009328984 learned_addr learned my addr 172.21.15.151:0/2009328984 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff413fff700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff414100dc0 0x7ff41419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff4137fe700 1 -- 172.21.15.151:0/2009328984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff414100dc0 msgr2=0x7ff41419c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff4137fe700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff414100dc0 0x7ff41419c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff4137fe700 1 -- 172.21.15.151:0/2009328984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4080095f0 con 0x7ff414101770 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff413fff700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff414100dc0 0x7ff41419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff4137fe700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414101770 0x7ff41419ca70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff40400c960 tx=0x7ff40400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff4117fa700 1 -- 172.21.15.151:0/2009328984 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff404007a70 con 0x7ff414101770 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff4117fa700 1 -- 172.21.15.151:0/2009328984 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff404007bd0 con 0x7ff414101770 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff414196890 con 0x7ff414101770 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff4117fa700 1 -- 172.21.15.151:0/2009328984 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4040186c0 con 0x7ff414101770 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.571+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff414196de0 con 0x7ff414101770 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.572+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff414109160 con 0x7ff414101770 2024-04-16T02:52:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.573+0000 7ff4117fa700 1 -- 172.21.15.151:0/2009328984 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff40400faa0 con 0x7ff414101770 2024-04-16T02:52:41.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.573+0000 7ff4117fa700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3fc06ecc0 0x7ff3fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:41.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.573+0000 7ff4117fa700 1 -- 172.21.15.151:0/2009328984 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff40408ee70 con 0x7ff414101770 2024-04-16T02:52:41.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.573+0000 7ff413fff700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3fc06ecc0 0x7ff3fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:41.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.574+0000 7ff413fff700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3fc06ecc0 0x7ff3fc071180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff408005b40 tx=0x7ff40801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:41.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.578+0000 7ff4117fa700 1 -- 172.21.15.151:0/2009328984 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff40405b4c0 con 0x7ff414101770 2024-04-16T02:52:41.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:41 smithi151 ceph-mon[112233]: pgmap v421: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:41.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:41 smithi151 ceph-mon[112233]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:41.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:41 smithi151 ceph-mon[112233]: from='client.34822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:41 smithi153 ceph-mon[87858]: pgmap v421: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:41 smithi153 ceph-mon[87858]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:41.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:41 smithi153 ceph-mon[87858]: from='client.34822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:41.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.829+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff41404ea90 con 0x7ff414101770 2024-04-16T02:52:41.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.830+0000 7ff4117fa700 1 -- 172.21.15.151:0/2009328984 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7ff40405b2e0 con 0x7ff414101770 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:52:41.833 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:52:41.834 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:52:41.835 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:52:41.836 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:52:41.836 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3fc06ecc0 msgr2=0x7ff3fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3fc06ecc0 0x7ff3fc071180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff408005b40 tx=0x7ff40801a040 comp rx=0 tx=0).stop 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414101770 msgr2=0x7ff41419ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414101770 0x7ff41419ca70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff40400c960 tx=0x7ff40400cd20 comp rx=0 tx=0).stop 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 shutdown_connections 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3fc06ecc0 0x7ff3fc071180 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff414100dc0 0x7ff41419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 --2- 172.21.15.151:0/2009328984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff414101770 0x7ff41419ca70 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 >> 172.21.15.151:0/2009328984 conn(0x7ff4140fc650 msgr2=0x7ff4140fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 shutdown_connections 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.833+0000 7ff41a8a7700 1 -- 172.21.15.151:0/2009328984 wait complete. 2024-04-16T02:52:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:52:41.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.941+0000 7ff975df6700 1 -- 172.21.15.151:0/670035593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 msgr2=0x7ff9701057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.941+0000 7ff975df6700 1 --2- 172.21.15.151:0/670035593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 0x7ff9701057c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff964009910 tx=0x7ff964009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:41.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.941+0000 7ff975df6700 1 -- 172.21.15.151:0/670035593 shutdown_connections 2024-04-16T02:52:41.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.941+0000 7ff975df6700 1 --2- 172.21.15.151:0/670035593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 0x7ff9701057c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.941+0000 7ff975df6700 1 --2- 172.21.15.151:0/670035593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff970100dc0 0x7ff9701011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.941+0000 7ff975df6700 1 -- 172.21.15.151:0/670035593 >> 172.21.15.151:0/670035593 conn(0x7ff9700fc650 msgr2=0x7ff9700fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:41.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.941+0000 7ff975df6700 1 -- 172.21.15.151:0/670035593 shutdown_connections 2024-04-16T02:52:41.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.941+0000 7ff975df6700 1 -- 172.21.15.151:0/670035593 wait complete. 2024-04-16T02:52:41.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.941+0000 7ff975df6700 1 Processor -- start 2024-04-16T02:52:41.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff975df6700 1 -- start start 2024-04-16T02:52:41.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff975df6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff970100dc0 0x7ff970198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:41.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff975df6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 0x7ff970198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:41.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff975df6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff970199550 con 0x7ff970101770 2024-04-16T02:52:41.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff975df6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97019d360 con 0x7ff970100dc0 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff96effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 0x7ff970198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff96effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 0x7ff970198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45910/0 (socket says 172.21.15.151:45910) 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff96effd700 1 -- 172.21.15.151:0/805950256 learned_addr learned my addr 172.21.15.151:0/805950256 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff96f7fe700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff970100dc0 0x7ff970198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff96effd700 1 -- 172.21.15.151:0/805950256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff970100dc0 msgr2=0x7ff970198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff96effd700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff970100dc0 0x7ff970198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff96effd700 1 -- 172.21.15.151:0/805950256 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9640095f0 con 0x7ff970101770 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.942+0000 7ff96effd700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 0x7ff970198ec0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff96400bda0 tx=0x7ff96400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.943+0000 7ff96cff9700 1 -- 172.21.15.151:0/805950256 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff96401d070 con 0x7ff970101770 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.943+0000 7ff96cff9700 1 -- 172.21.15.151:0/805950256 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff964005dc0 con 0x7ff970101770 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.943+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97019d5e0 con 0x7ff970101770 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.943+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97019dad0 con 0x7ff970101770 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.943+0000 7ff96cff9700 1 -- 172.21.15.151:0/805950256 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff964017400 con 0x7ff970101770 2024-04-16T02:52:41.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.944+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff970104f40 con 0x7ff970101770 2024-04-16T02:52:41.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.944+0000 7ff96cff9700 1 -- 172.21.15.151:0/805950256 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff96400f460 con 0x7ff970101770 2024-04-16T02:52:41.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.945+0000 7ff96cff9700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff95806ecc0 0x7ff958071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:41.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.945+0000 7ff96cff9700 1 -- 172.21.15.151:0/805950256 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff964090200 con 0x7ff970101770 2024-04-16T02:52:41.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.945+0000 7ff96f7fe700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff95806ecc0 0x7ff958071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:41.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.945+0000 7ff96f7fe700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff95806ecc0 0x7ff958071180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff960009550 tx=0x7ff9600091b0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:41.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:41.949+0000 7ff96cff9700 1 -- 172.21.15.151:0/805950256 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff96405cae0 con 0x7ff970101770 2024-04-16T02:52:42.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.178+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff97019de00 con 0x7ff95806ecc0 2024-04-16T02:52:42.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.179+0000 7ff96cff9700 1 -- 172.21.15.151:0/805950256 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff97019de00 con 0x7ff95806ecc0 2024-04-16T02:52:42.181 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:52:42.182 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff95806ecc0 msgr2=0x7ff958071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff95806ecc0 0x7ff958071180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff960009550 tx=0x7ff9600091b0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 msgr2=0x7ff970198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 0x7ff970198ec0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff96400bda0 tx=0x7ff96400bdd0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 shutdown_connections 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff95806ecc0 0x7ff958071180 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff970100dc0 0x7ff970198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 --2- 172.21.15.151:0/805950256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff970101770 0x7ff970198ec0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 >> 172.21.15.151:0/805950256 conn(0x7ff9700fc650 msgr2=0x7ff9700fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 shutdown_connections 2024-04-16T02:52:42.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.181+0000 7ff975df6700 1 -- 172.21.15.151:0/805950256 wait complete. 2024-04-16T02:52:42.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.287+0000 7fc38a5d4700 1 -- 172.21.15.151:0/1877642199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 msgr2=0x7fc3840ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:42.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.287+0000 7fc38a5d4700 1 --2- 172.21.15.151:0/1877642199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 0x7fc3840ffec0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc378009910 tx=0x7fc378009c20 comp rx=0 tx=0).stop 2024-04-16T02:52:42.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.287+0000 7fc38a5d4700 1 -- 172.21.15.151:0/1877642199 shutdown_connections 2024-04-16T02:52:42.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.287+0000 7fc38a5d4700 1 --2- 172.21.15.151:0/1877642199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 0x7fc3840ffec0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.287+0000 7fc38a5d4700 1 --2- 172.21.15.151:0/1877642199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc384105a40 0x7fc384105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.287+0000 7fc38a5d4700 1 -- 172.21.15.151:0/1877642199 >> 172.21.15.151:0/1877642199 conn(0x7fc3840fb550 msgr2=0x7fc3840fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:42.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.287+0000 7fc38a5d4700 1 -- 172.21.15.151:0/1877642199 shutdown_connections 2024-04-16T02:52:42.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.287+0000 7fc38a5d4700 1 -- 172.21.15.151:0/1877642199 wait complete. 2024-04-16T02:52:42.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.288+0000 7fc38a5d4700 1 Processor -- start 2024-04-16T02:52:42.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.288+0000 7fc38a5d4700 1 -- start start 2024-04-16T02:52:42.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.288+0000 7fc38a5d4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 0x7fc384197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.288+0000 7fc38a5d4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc384105a40 0x7fc384197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.288+0000 7fc38a5d4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3841980f0 con 0x7fc3840ffa40 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.288+0000 7fc38a5d4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc384198260 con 0x7fc384105a40 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.288+0000 7fc383fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 0x7fc384197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.288+0000 7fc383fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 0x7fc384197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45932/0 (socket says 172.21.15.151:45932) 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.288+0000 7fc383fff700 1 -- 172.21.15.151:0/876830169 learned_addr learned my addr 172.21.15.151:0/876830169 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc3837fe700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc384105a40 0x7fc384197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc383fff700 1 -- 172.21.15.151:0/876830169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc384105a40 msgr2=0x7fc384197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc383fff700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc384105a40 0x7fc384197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc383fff700 1 -- 172.21.15.151:0/876830169 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780095f0 con 0x7fc3840ffa40 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc383fff700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 0x7fc384197590 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc37400d8d0 tx=0x7fc37400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc3817fa700 1 -- 172.21.15.151:0/876830169 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc374009940 con 0x7fc3840ffa40 2024-04-16T02:52:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3841a0600 con 0x7fc3840ffa40 2024-04-16T02:52:42.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc3817fa700 1 -- 172.21.15.151:0/876830169 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc374010460 con 0x7fc3840ffa40 2024-04-16T02:52:42.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc3817fa700 1 -- 172.21.15.151:0/876830169 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc37400f5f0 con 0x7fc3840ffa40 2024-04-16T02:52:42.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.289+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3841a0af0 con 0x7fc3840ffa40 2024-04-16T02:52:42.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.290+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc384107fd0 con 0x7fc3840ffa40 2024-04-16T02:52:42.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.290+0000 7fc3817fa700 1 -- 172.21.15.151:0/876830169 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc374009aa0 con 0x7fc3840ffa40 2024-04-16T02:52:42.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.291+0000 7fc3817fa700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc36c06ecc0 0x7fc36c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:52:42.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.291+0000 7fc3817fa700 1 -- 172.21.15.151:0/876830169 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc37408ee50 con 0x7fc3840ffa40 2024-04-16T02:52:42.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.291+0000 7fc3837fe700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc36c06ecc0 0x7fc36c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:52:42.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.291+0000 7fc3837fe700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc36c06ecc0 0x7fc36c071180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc378005b40 tx=0x7fc378005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:52:42.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.296+0000 7fc3817fa700 1 -- 172.21.15.151:0/876830169 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc374057c00 con 0x7fc3840ffa40 2024-04-16T02:52:42.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:42 smithi151 ceph-mon[112233]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:42.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:42 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2366369404' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:52:42.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:42 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2009328984' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:52:42.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.579+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc38404ea90 con 0x7fc3840ffa40 2024-04-16T02:52:42.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.580+0000 7fc3817fa700 1 -- 172.21.15.151:0/876830169 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc37405b2c0 con 0x7fc3840ffa40 2024-04-16T02:52:42.583 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:52:42.583 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:52:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:52:42.583 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:52:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:52:42.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc36c06ecc0 msgr2=0x7fc36c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:42.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc36c06ecc0 0x7fc36c071180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc378005b40 tx=0x7fc378005a90 comp rx=0 tx=0).stop 2024-04-16T02:52:42.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 msgr2=0x7fc384197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:52:42.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 0x7fc384197590 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc37400d8d0 tx=0x7fc37400dc90 comp rx=0 tx=0).stop 2024-04-16T02:52:42.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 shutdown_connections 2024-04-16T02:52:42.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc36c06ecc0 0x7fc36c071180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3840ffa40 0x7fc384197590 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 --2- 172.21.15.151:0/876830169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc384105a40 0x7fc384197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:52:42.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 >> 172.21.15.151:0/876830169 conn(0x7fc3840fb550 msgr2=0x7fc3840fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:52:42.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 shutdown_connections 2024-04-16T02:52:42.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:52:42.582+0000 7fc38a5d4700 1 -- 172.21.15.151:0/876830169 wait complete. 2024-04-16T02:52:42.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:42 smithi153 ceph-mon[87858]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:42.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:42 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2366369404' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:52:42.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:42 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2009328984' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:52:43.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:43 smithi151 ceph-mon[112233]: pgmap v422: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:43.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:43 smithi151 ceph-mon[112233]: from='client.34838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:43.747 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:43 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/876830169' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:52:43.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:43 smithi153 ceph-mon[87858]: pgmap v422: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:43.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:43 smithi153 ceph-mon[87858]: from='client.34838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:52:43.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:43 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/876830169' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:52:45.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:45 smithi151 ceph-mon[112233]: pgmap v423: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:45.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:45 smithi153 ceph-mon[87858]: pgmap v423: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:47.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:47 smithi151 ceph-mon[112233]: pgmap v424: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:47.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:47 smithi153 ceph-mon[87858]: pgmap v424: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:49.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:49 smithi151 ceph-mon[112233]: pgmap v425: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:49 smithi153 ceph-mon[87858]: pgmap v425: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:51.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:51 smithi151 ceph-mon[112233]: pgmap v426: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:51.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:51 smithi153 ceph-mon[87858]: pgmap v426: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:53.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:53 smithi151 ceph-mon[112233]: pgmap v427: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:53.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:53 smithi153 ceph-mon[87858]: pgmap v427: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:54.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:52:54.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:52:55.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:55 smithi151 ceph-mon[112233]: pgmap v428: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:55.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:55 smithi153 ceph-mon[87858]: pgmap v428: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:57.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:57 smithi151 ceph-mon[112233]: pgmap v429: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:57.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:57 smithi153 ceph-mon[87858]: pgmap v429: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:59.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:52:59 smithi151 ceph-mon[112233]: pgmap v430: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:52:59.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:52:59 smithi153 ceph-mon[87858]: pgmap v430: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:01.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:01 smithi151 ceph-mon[112233]: pgmap v431: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:01.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:01 smithi153 ceph-mon[87858]: pgmap v431: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:03.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:03 smithi151 ceph-mon[112233]: pgmap v432: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:03.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:03 smithi153 ceph-mon[87858]: pgmap v432: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:05.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:05 smithi151 ceph-mon[112233]: pgmap v433: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:05 smithi153 ceph-mon[87858]: pgmap v433: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:07.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:07 smithi151 ceph-mon[112233]: pgmap v434: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:07.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:07 smithi153 ceph-mon[87858]: pgmap v434: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:09.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:09 smithi151 ceph-mon[112233]: pgmap v435: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:09.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:53:09.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:09 smithi153 ceph-mon[87858]: pgmap v435: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:09.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:53:11.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:11 smithi151 ceph-mon[112233]: pgmap v436: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:11.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:11 smithi153 ceph-mon[87858]: pgmap v436: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.701+0000 7f2198b15700 1 -- 172.21.15.151:0/3044882636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 msgr2=0x7f2194105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.701+0000 7f2198b15700 1 --2- 172.21.15.151:0/3044882636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 0x7f2194105e20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2184009910 tx=0x7f2184009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.701+0000 7f2198b15700 1 -- 172.21.15.151:0/3044882636 shutdown_connections 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.701+0000 7f2198b15700 1 --2- 172.21.15.151:0/3044882636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21940ffa40 0x7f21940ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.701+0000 7f2198b15700 1 --2- 172.21.15.151:0/3044882636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 0x7f2194105e20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.701+0000 7f2198b15700 1 -- 172.21.15.151:0/3044882636 >> 172.21.15.151:0/3044882636 conn(0x7f21940fb570 msgr2=0x7f21940fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.701+0000 7f2198b15700 1 -- 172.21.15.151:0/3044882636 shutdown_connections 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.702+0000 7f2198b15700 1 -- 172.21.15.151:0/3044882636 wait complete. 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.702+0000 7f2198b15700 1 Processor -- start 2024-04-16T02:53:12.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.702+0000 7f2198b15700 1 -- start start 2024-04-16T02:53:12.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2198b15700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21940ffa40 0x7f2194197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:12.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2198b15700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 0x7f2194197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:12.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2198b15700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21941980f0 con 0x7f2194105a40 2024-04-16T02:53:12.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2198b15700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2194198260 con 0x7f21940ffa40 2024-04-16T02:53:12.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2191d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 0x7f2194197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:12.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2191d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 0x7f2194197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:37232/0 (socket says 172.21.15.151:37232) 2024-04-16T02:53:12.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2191d9b700 1 -- 172.21.15.151:0/4148089774 learned_addr learned my addr 172.21.15.151:0/4148089774 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:12.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f219259c700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21940ffa40 0x7f2194197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:12.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2191d9b700 1 -- 172.21.15.151:0/4148089774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21940ffa40 msgr2=0x7f2194197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2191d9b700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21940ffa40 0x7f2194197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2191d9b700 1 -- 172.21.15.151:0/4148089774 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21840095f0 con 0x7f2194105a40 2024-04-16T02:53:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2191d9b700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 0x7f2194197ad0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f218800d8d0 tx=0x7f218800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f21837fe700 1 -- 172.21.15.151:0/4148089774 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2188009940 con 0x7f2194105a40 2024-04-16T02:53:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f21837fe700 1 -- 172.21.15.151:0/4148089774 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2188010460 con 0x7f2194105a40 2024-04-16T02:53:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21941a0600 con 0x7f2194105a40 2024-04-16T02:53:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.703+0000 7f21837fe700 1 -- 172.21.15.151:0/4148089774 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f218800f5f0 con 0x7f2194105a40 2024-04-16T02:53:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.704+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21941a0af0 con 0x7f2194105a40 2024-04-16T02:53:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.704+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2194108040 con 0x7f2194105a40 2024-04-16T02:53:12.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.705+0000 7f21837fe700 1 -- 172.21.15.151:0/4148089774 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f21880105d0 con 0x7f2194105a40 2024-04-16T02:53:12.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.706+0000 7f21837fe700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f217c06ecc0 0x7f217c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:12.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.706+0000 7f21837fe700 1 -- 172.21.15.151:0/4148089774 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f218808ee80 con 0x7f2194105a40 2024-04-16T02:53:12.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.706+0000 7f219259c700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f217c06ecc0 0x7f217c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:12.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.706+0000 7f219259c700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f217c06ecc0 0x7f217c071180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2184006010 tx=0x7f218401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:12.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.711+0000 7f21837fe700 1 -- 172.21.15.151:0/4148089774 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2188057c30 con 0x7f2194105a40 2024-04-16T02:53:12.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.926+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f219402cdb0 con 0x7f217c06ecc0 2024-04-16T02:53:12.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.927+0000 7f21837fe700 1 -- 172.21.15.151:0/4148089774 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f219402cdb0 con 0x7f217c06ecc0 2024-04-16T02:53:12.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f217c06ecc0 msgr2=0x7f217c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:12.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f217c06ecc0 0x7f217c071180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2184006010 tx=0x7f218401a040 comp rx=0 tx=0).stop 2024-04-16T02:53:12.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 msgr2=0x7f2194197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:12.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 0x7f2194197ad0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f218800d8d0 tx=0x7f218800dc90 comp rx=0 tx=0).stop 2024-04-16T02:53:12.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 shutdown_connections 2024-04-16T02:53:12.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f217c06ecc0 0x7f217c071180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:12.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21940ffa40 0x7f2194197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:12.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 --2- 172.21.15.151:0/4148089774 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2194105a40 0x7f2194197ad0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:12.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 >> 172.21.15.151:0/4148089774 conn(0x7f21940fb570 msgr2=0x7f21940fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:12.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 shutdown_connections 2024-04-16T02:53:12.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:12.930+0000 7f2198b15700 1 -- 172.21.15.151:0/4148089774 wait complete. 2024-04-16T02:53:12.943 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:53:13.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.036+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3422645131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48100140 msgr2=0x7f0f481005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.036+0000 7f0f4e4a1700 1 --2- 172.21.15.151:0/3422645131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48100140 0x7f0f481005c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009910 tx=0x7f0f30009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.036+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3422645131 shutdown_connections 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.036+0000 7f0f4e4a1700 1 --2- 172.21.15.151:0/3422645131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48100140 0x7f0f481005c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.036+0000 7f0f4e4a1700 1 --2- 172.21.15.151:0/3422645131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f48104a50 0x7f0f48104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.036+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3422645131 >> 172.21.15.151:0/3422645131 conn(0x7f0f48075e30 msgr2=0x7f0f480fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.036+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3422645131 shutdown_connections 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.036+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3422645131 wait complete. 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f4e4a1700 1 Processor -- start 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f4e4a1700 1 -- start start 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f4e4a1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f48100140 0x7f0f480fb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f4e4a1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48104a50 0x7f0f480fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f4e4a1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f480fbd20 con 0x7f0f48104a50 2024-04-16T02:53:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f4e4a1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f480fbe60 con 0x7f0f48100140 2024-04-16T02:53:13.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f3ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48104a50 0x7f0f480fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:13.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f3ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48104a50 0x7f0f480fb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:37252/0 (socket says 172.21.15.151:37252) 2024-04-16T02:53:13.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f3ffff700 1 -- 172.21.15.151:0/3129540946 learned_addr learned my addr 172.21.15.151:0/3129540946 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f47fff700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f48100140 0x7f0f480fb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f3ffff700 1 -- 172.21.15.151:0/3129540946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f48100140 msgr2=0x7f0f480fb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.037+0000 7f0f3ffff700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f48100140 0x7f0f480fb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.038+0000 7f0f3ffff700 1 -- 172.21.15.151:0/3129540946 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f300095f0 con 0x7f0f48104a50 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.038+0000 7f0f3ffff700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48104a50 0x7f0f480fb7e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0f300092e0 tx=0x7f0f3000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.038+0000 7f0f45ffb700 1 -- 172.21.15.151:0/3129540946 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3001d070 con 0x7f0f48104a50 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.038+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f480fc0e0 con 0x7f0f48104a50 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.038+0000 7f0f45ffb700 1 -- 172.21.15.151:0/3129540946 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f30005e00 con 0x7f0f48104a50 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.038+0000 7f0f45ffb700 1 -- 172.21.15.151:0/3129540946 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f30017400 con 0x7f0f48104a50 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.038+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f481a2bd0 con 0x7f0f48104a50 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.039+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4804ea90 con 0x7f0f48104a50 2024-04-16T02:53:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.039+0000 7f0f45ffb700 1 -- 172.21.15.151:0/3129540946 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f0f3000f460 con 0x7f0f48104a50 2024-04-16T02:53:13.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.040+0000 7f0f45ffb700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0f3406ecc0 0x7f0f34071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:13.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.040+0000 7f0f45ffb700 1 -- 172.21.15.151:0/3129540946 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0f300902b0 con 0x7f0f48104a50 2024-04-16T02:53:13.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.040+0000 7f0f47fff700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0f3406ecc0 0x7f0f34071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:13.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.040+0000 7f0f47fff700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0f3406ecc0 0x7f0f34071180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f38005950 tx=0x7f0f380058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:13.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.044+0000 7f0f45ffb700 1 -- 172.21.15.151:0/3129540946 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0f300590e0 con 0x7f0f48104a50 2024-04-16T02:53:13.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.266+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f48102890 con 0x7f0f3406ecc0 2024-04-16T02:53:13.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.267+0000 7f0f45ffb700 1 -- 172.21.15.151:0/3129540946 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0f48102890 con 0x7f0f3406ecc0 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0f3406ecc0 msgr2=0x7f0f34071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0f3406ecc0 0x7f0f34071180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f38005950 tx=0x7f0f380058e0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48104a50 msgr2=0x7f0f480fb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48104a50 0x7f0f480fb7e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0f300092e0 tx=0x7f0f3000be20 comp rx=0 tx=0).stop 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 shutdown_connections 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0f3406ecc0 0x7f0f34071180 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f48100140 0x7f0f480fb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 --2- 172.21.15.151:0/3129540946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f48104a50 0x7f0f480fb7e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 >> 172.21.15.151:0/3129540946 conn(0x7f0f48075e30 msgr2=0x7f0f480fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 shutdown_connections 2024-04-16T02:53:13.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.269+0000 7f0f4e4a1700 1 -- 172.21.15.151:0/3129540946 wait complete. 2024-04-16T02:53:13.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.375+0000 7f923e62c700 1 -- 172.21.15.151:0/3098094508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 msgr2=0x7f9238100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.375+0000 7f923e62c700 1 --2- 172.21.15.151:0/3098094508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 0x7f9238100050 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9228009910 tx=0x7f9228009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:13.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.375+0000 7f923e62c700 1 -- 172.21.15.151:0/3098094508 shutdown_connections 2024-04-16T02:53:13.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.375+0000 7f923e62c700 1 --2- 172.21.15.151:0/3098094508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 0x7f9238100050 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.375+0000 7f923e62c700 1 --2- 172.21.15.151:0/3098094508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9238105a40 0x7f9238105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.375+0000 7f923e62c700 1 -- 172.21.15.151:0/3098094508 >> 172.21.15.151:0/3098094508 conn(0x7f92380755a0 msgr2=0x7f92380759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:13.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.375+0000 7f923e62c700 1 -- 172.21.15.151:0/3098094508 shutdown_connections 2024-04-16T02:53:13.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.375+0000 7f923e62c700 1 -- 172.21.15.151:0/3098094508 wait complete. 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.376+0000 7f923e62c700 1 Processor -- start 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.376+0000 7f923e62c700 1 -- start start 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.376+0000 7f923e62c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 0x7f923819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.376+0000 7f923e62c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9238105a40 0x7f923819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.376+0000 7f923e62c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923819d0d0 con 0x7f92380ffbd0 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.376+0000 7f923e62c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9238196580 con 0x7f9238105a40 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.376+0000 7f9237fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 0x7f923819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.376+0000 7f9237fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 0x7f923819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:37268/0 (socket says 172.21.15.151:37268) 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.376+0000 7f9237fff700 1 -- 172.21.15.151:0/1757324085 learned_addr learned my addr 172.21.15.151:0/1757324085 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f92377fe700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9238105a40 0x7f923819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:13.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f9237fff700 1 -- 172.21.15.151:0/1757324085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9238105a40 msgr2=0x7f923819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f9237fff700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9238105a40 0x7f923819ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f9237fff700 1 -- 172.21.15.151:0/1757324085 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92280095f0 con 0x7f92380ffbd0 2024-04-16T02:53:13.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f9237fff700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 0x7f923819c500 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f922000c8a0 tx=0x7f922000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:13.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f92357fa700 1 -- 172.21.15.151:0/1757324085 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922000cea0 con 0x7f92380ffbd0 2024-04-16T02:53:13.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f92357fa700 1 -- 172.21.15.151:0/1757324085 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9220004830 con 0x7f92380ffbd0 2024-04-16T02:53:13.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9238196860 con 0x7f92380ffbd0 2024-04-16T02:53:13.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f92357fa700 1 -- 172.21.15.151:0/1757324085 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92200055b0 con 0x7f92380ffbd0 2024-04-16T02:53:13.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.377+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9238196db0 con 0x7f92380ffbd0 2024-04-16T02:53:13.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.378+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9238108fe0 con 0x7f92380ffbd0 2024-04-16T02:53:13.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.379+0000 7f92357fa700 1 -- 172.21.15.151:0/1757324085 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f92200049a0 con 0x7f92380ffbd0 2024-04-16T02:53:13.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.379+0000 7f92357fa700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f922406ecc0 0x7f9224071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:13.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.379+0000 7f92357fa700 1 -- 172.21.15.151:0/1757324085 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f922008eb50 con 0x7f92380ffbd0 2024-04-16T02:53:13.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.380+0000 7f92377fe700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f922406ecc0 0x7f9224071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:13.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.380+0000 7f92377fe700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f922406ecc0 0x7f9224071180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f92280059d0 tx=0x7f922801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:13.387 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.384+0000 7f92357fa700 1 -- 172.21.15.151:0/1757324085 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f922005b430 con 0x7f92380ffbd0 2024-04-16T02:53:13.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.592+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f92381970b0 con 0x7f922406ecc0 2024-04-16T02:53:13.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.603+0000 7f92357fa700 1 -- 172.21.15.151:0/1757324085 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f92381970b0 con 0x7f922406ecc0 2024-04-16T02:53:13.605 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (14m) 8m ago 26m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (26m) 8m ago 26m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (25m) 7m ago 25m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (12m) 8m ago 26m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (12m) 7m ago 25m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (14m) 8m ago 25m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (8m) 8m ago 20m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (8m) 8m ago 20m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (20m) 7m ago 20m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (7m) 7m ago 20m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (16m) 8m ago 27m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (15m) 7m ago 25m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:53:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (13m) 8m ago 27m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (13m) 7m ago 25m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (15m) 8m ago 26m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (15m) 7m ago 25m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (11m) 8m ago 24m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (11m) 8m ago 23m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (10m) 8m ago 23m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (10m) 7m ago 22m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (9m) 7m ago 22m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (8m) 7m ago 22m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:53:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (15m) 8m ago 25m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:53:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f922406ecc0 msgr2=0x7f9224071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f922406ecc0 0x7f9224071180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f92280059d0 tx=0x7f922801a040 comp rx=0 tx=0).stop 2024-04-16T02:53:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 msgr2=0x7f923819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 0x7f923819c500 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f922000c8a0 tx=0x7f922000cc60 comp rx=0 tx=0).stop 2024-04-16T02:53:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 shutdown_connections 2024-04-16T02:53:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f922406ecc0 0x7f9224071180 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92380ffbd0 0x7f923819c500 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 --2- 172.21.15.151:0/1757324085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9238105a40 0x7f923819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 >> 172.21.15.151:0/1757324085 conn(0x7f92380755a0 msgr2=0x7f9238068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:13.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 shutdown_connections 2024-04-16T02:53:13.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.605+0000 7f923e62c700 1 -- 172.21.15.151:0/1757324085 wait complete. 2024-04-16T02:53:13.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.710+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1784880962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c0ffbd0 msgr2=0x7f859c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.710+0000 7f85a3ec5700 1 --2- 172.21.15.151:0/1784880962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c0ffbd0 0x7f859c100050 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f858c009910 tx=0x7f858c009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:13.712 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:13 smithi151 ceph-mon[112233]: pgmap v437: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:13.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.710+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1784880962 shutdown_connections 2024-04-16T02:53:13.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.710+0000 7f85a3ec5700 1 --2- 172.21.15.151:0/1784880962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c0ffbd0 0x7f859c100050 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.710+0000 7f85a3ec5700 1 --2- 172.21.15.151:0/1784880962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f859c105a40 0x7f859c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.710+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1784880962 >> 172.21.15.151:0/1784880962 conn(0x7f859c0755a0 msgr2=0x7f859c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:13.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.710+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1784880962 shutdown_connections 2024-04-16T02:53:13.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.710+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1784880962 wait complete. 2024-04-16T02:53:13.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a3ec5700 1 Processor -- start 2024-04-16T02:53:13.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a3ec5700 1 -- start start 2024-04-16T02:53:13.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a3ec5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f859c0ffbd0 0x7f859c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a3ec5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c105a40 0x7f859c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a3ec5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c1981b0 con 0x7f859c105a40 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a3ec5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c198320 con 0x7f859c0ffbd0 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a1460700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c105a40 0x7f859c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a1460700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c105a40 0x7f859c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:37286/0 (socket says 172.21.15.151:37286) 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a1460700 1 -- 172.21.15.151:0/1799932599 learned_addr learned my addr 172.21.15.151:0/1799932599 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.711+0000 7f85a1c61700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f859c0ffbd0 0x7f859c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.712+0000 7f85a1460700 1 -- 172.21.15.151:0/1799932599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f859c0ffbd0 msgr2=0x7f859c197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.712+0000 7f85a1460700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f859c0ffbd0 0x7f859c197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.712+0000 7f85a1460700 1 -- 172.21.15.151:0/1799932599 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c0095f0 con 0x7f859c105a40 2024-04-16T02:53:13.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.712+0000 7f85a1460700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c105a40 0x7f859c197ad0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f858c006010 tx=0x7f858c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:13.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.712+0000 7f8592ffd700 1 -- 172.21.15.151:0/1799932599 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f858c01d070 con 0x7f859c105a40 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.712+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859c1a0600 con 0x7f859c105a40 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.712+0000 7f8592ffd700 1 -- 172.21.15.151:0/1799932599 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f858c005dc0 con 0x7f859c105a40 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.712+0000 7f8592ffd700 1 -- 172.21.15.151:0/1799932599 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f858c017400 con 0x7f859c105a40 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.712+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859c1a0a70 con 0x7f859c105a40 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.713+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859c107fd0 con 0x7f859c105a40 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.714+0000 7f8592ffd700 1 -- 172.21.15.151:0/1799932599 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f858c003a60 con 0x7f859c105a40 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.714+0000 7f8592ffd700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f858806ecc0 0x7f8588071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.714+0000 7f8592ffd700 1 -- 172.21.15.151:0/1799932599 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f858c090200 con 0x7f859c105a40 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.714+0000 7f85a1c61700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f858806ecc0 0x7f8588071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:13.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.715+0000 7f85a1c61700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f858806ecc0 0x7f8588071180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8598005950 tx=0x7f859800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:13.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:13.719+0000 7f8592ffd700 1 -- 172.21.15.151:0/1799932599 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f858c059030 con 0x7f859c105a40 2024-04-16T02:53:13.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:13 smithi153 ceph-mon[87858]: pgmap v437: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:14.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.001+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f859c04ea90 con 0x7f859c105a40 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.001+0000 7f8592ffd700 1 -- 172.21.15.151:0/1799932599 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f858c05c6f0 con 0x7f859c105a40 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:53:14.004 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:53:14.005 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:53:14.005 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:53:14.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:53:14.005 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:53:14.005 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:53:14.005 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:53:14.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:53:14.005 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:53:14.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f858806ecc0 msgr2=0x7f8588071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f858806ecc0 0x7f8588071180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8598005950 tx=0x7f859800b410 comp rx=0 tx=0).stop 2024-04-16T02:53:14.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c105a40 msgr2=0x7f859c197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c105a40 0x7f859c197ad0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f858c006010 tx=0x7f858c00bd70 comp rx=0 tx=0).stop 2024-04-16T02:53:14.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 shutdown_connections 2024-04-16T02:53:14.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f858806ecc0 0x7f8588071180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f859c0ffbd0 0x7f859c197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 --2- 172.21.15.151:0/1799932599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f859c105a40 0x7f859c197ad0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 >> 172.21.15.151:0/1799932599 conn(0x7f859c0755a0 msgr2=0x7f859c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:14.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.003+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 shutdown_connections 2024-04-16T02:53:14.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.004+0000 7f85a3ec5700 1 -- 172.21.15.151:0/1799932599 wait complete. 2024-04-16T02:53:14.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.108+0000 7f3009d16700 1 -- 172.21.15.151:0/156206132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 msgr2=0x7f30040ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.108+0000 7f3009d16700 1 --2- 172.21.15.151:0/156206132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 0x7f30040ff910 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009910 tx=0x7f2ff4009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:14.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.108+0000 7f3009d16700 1 -- 172.21.15.151:0/156206132 shutdown_connections 2024-04-16T02:53:14.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.108+0000 7f3009d16700 1 --2- 172.21.15.151:0/156206132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040ffee0 0x7f3004105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.108+0000 7f3009d16700 1 --2- 172.21.15.151:0/156206132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 0x7f30040ff910 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.108+0000 7f3009d16700 1 -- 172.21.15.151:0/156206132 >> 172.21.15.151:0/156206132 conn(0x7f3004074df0 msgr2=0x7f3004075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:14.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.108+0000 7f3009d16700 1 -- 172.21.15.151:0/156206132 shutdown_connections 2024-04-16T02:53:14.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.108+0000 7f3009d16700 1 -- 172.21.15.151:0/156206132 wait complete. 2024-04-16T02:53:14.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.109+0000 7f3009d16700 1 Processor -- start 2024-04-16T02:53:14.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.109+0000 7f3009d16700 1 -- start start 2024-04-16T02:53:14.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.109+0000 7f3009d16700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 0x7f300419c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:14.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f3009d16700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040ffee0 0x7f300419caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:14.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f3009d16700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300419d180 con 0x7f30040ff530 2024-04-16T02:53:14.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f3009d16700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3004196630 con 0x7f30040ffee0 2024-04-16T02:53:14.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f30037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 0x7f300419c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:14.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f30037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 0x7f300419c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:37296/0 (socket says 172.21.15.151:37296) 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f30037fe700 1 -- 172.21.15.151:0/2453815037 learned_addr learned my addr 172.21.15.151:0/2453815037 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f30037fe700 1 -- 172.21.15.151:0/2453815037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040ffee0 msgr2=0x7f300419caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f3002ffd700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040ffee0 0x7f300419caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f30037fe700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040ffee0 0x7f300419caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f30037fe700 1 -- 172.21.15.151:0/2453815037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff40095f0 con 0x7f30040ff530 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f30037fe700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 0x7f300419c5b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4006010 tx=0x7f2ff400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f3000ff9700 1 -- 172.21.15.151:0/2453815037 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ff401d070 con 0x7f30040ff530 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30041968b0 con 0x7f30040ff530 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f3000ff9700 1 -- 172.21.15.151:0/2453815037 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ff4004fb0 con 0x7f30040ff530 2024-04-16T02:53:14.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f3000ff9700 1 -- 172.21.15.151:0/2453815037 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ff4022ea0 con 0x7f30040ff530 2024-04-16T02:53:14.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.110+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3004196da0 con 0x7f30040ff530 2024-04-16T02:53:14.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.111+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30041091e0 con 0x7f30040ff530 2024-04-16T02:53:14.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.112+0000 7f3000ff9700 1 -- 172.21.15.151:0/2453815037 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2ff4022470 con 0x7f30040ff530 2024-04-16T02:53:14.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.112+0000 7f3000ff9700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fec06ecc0 0x7f2fec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:14.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.112+0000 7f3000ff9700 1 -- 172.21.15.151:0/2453815037 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2ff4090400 con 0x7f30040ff530 2024-04-16T02:53:14.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.112+0000 7f3002ffd700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fec06ecc0 0x7f2fec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:14.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.113+0000 7f3002ffd700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fec06ecc0 0x7f2fec071180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f30041071f0 tx=0x7f2ff8005cd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:14.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.117+0000 7f3000ff9700 1 -- 172.21.15.151:0/2453815037 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2ff405cd90 con 0x7f30040ff530 2024-04-16T02:53:14.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.358+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f300404ea90 con 0x7f30040ff530 2024-04-16T02:53:14.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.359+0000 7f3000ff9700 1 -- 172.21.15.151:0/2453815037 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f2ff4027770 con 0x7f30040ff530 2024-04-16T02:53:14.361 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:53:14.361 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:53:14.362 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:53:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:53:14.364 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fec06ecc0 msgr2=0x7f2fec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fec06ecc0 0x7f2fec071180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f30041071f0 tx=0x7f2ff8005cd0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 msgr2=0x7f300419c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 0x7f300419c5b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4006010 tx=0x7f2ff400be20 comp rx=0 tx=0).stop 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 shutdown_connections 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fec06ecc0 0x7f2fec071180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30040ff530 0x7f300419c5b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 --2- 172.21.15.151:0/2453815037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040ffee0 0x7f300419caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 >> 172.21.15.151:0/2453815037 conn(0x7f3004074df0 msgr2=0x7f30040feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 shutdown_connections 2024-04-16T02:53:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.362+0000 7f3009d16700 1 -- 172.21.15.151:0/2453815037 wait complete. 2024-04-16T02:53:14.366 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:53:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.469+0000 7f94a67bd700 1 -- 172.21.15.151:0/967042746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 msgr2=0x7f94a00ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.469+0000 7f94a67bd700 1 --2- 172.21.15.151:0/967042746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 0x7f94a00ff910 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9490009910 tx=0x7f9490009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.469+0000 7f94a67bd700 1 -- 172.21.15.151:0/967042746 shutdown_connections 2024-04-16T02:53:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.469+0000 7f94a67bd700 1 --2- 172.21.15.151:0/967042746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a00ffee0 0x7f94a0103eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.469+0000 7f94a67bd700 1 --2- 172.21.15.151:0/967042746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 0x7f94a00ff910 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.469+0000 7f94a67bd700 1 -- 172.21.15.151:0/967042746 >> 172.21.15.151:0/967042746 conn(0x7f94a0074df0 msgr2=0x7f94a0075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.469+0000 7f94a67bd700 1 -- 172.21.15.151:0/967042746 shutdown_connections 2024-04-16T02:53:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.469+0000 7f94a67bd700 1 -- 172.21.15.151:0/967042746 wait complete. 2024-04-16T02:53:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f94a67bd700 1 Processor -- start 2024-04-16T02:53:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f94a67bd700 1 -- start start 2024-04-16T02:53:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f94a67bd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 0x7f94a019c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f94a67bd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a00ffee0 0x7f94a019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f94a67bd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a019d110 con 0x7f94a00ff530 2024-04-16T02:53:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f94a67bd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a01965c0 con 0x7f94a00ffee0 2024-04-16T02:53:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f949ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 0x7f94a019c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f949ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 0x7f94a019c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:37306/0 (socket says 172.21.15.151:37306) 2024-04-16T02:53:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f949ffff700 1 -- 172.21.15.151:0/4106191371 learned_addr learned my addr 172.21.15.151:0/4106191371 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.470+0000 7f9497fff700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a00ffee0 0x7f94a019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.471+0000 7f949ffff700 1 -- 172.21.15.151:0/4106191371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a00ffee0 msgr2=0x7f94a019ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.471+0000 7f949ffff700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a00ffee0 0x7f94a019ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.471+0000 7f949ffff700 1 -- 172.21.15.151:0/4106191371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94900095f0 con 0x7f94a00ff530 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.471+0000 7f949ffff700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 0x7f94a019c540 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9490000c00 tx=0x7f949000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.471+0000 7f949dffb700 1 -- 172.21.15.151:0/4106191371 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f949001d070 con 0x7f94a00ff530 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.471+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a0196840 con 0x7f94a00ff530 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.471+0000 7f949dffb700 1 -- 172.21.15.151:0/4106191371 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9490005dc0 con 0x7f94a00ff530 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.471+0000 7f949dffb700 1 -- 172.21.15.151:0/4106191371 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9490017400 con 0x7f94a00ff530 2024-04-16T02:53:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.471+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a0196d30 con 0x7f94a00ff530 2024-04-16T02:53:14.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.472+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a0109830 con 0x7f94a00ff530 2024-04-16T02:53:14.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.472+0000 7f949dffb700 1 -- 172.21.15.151:0/4106191371 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f949000f7f0 con 0x7f94a00ff530 2024-04-16T02:53:14.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.473+0000 7f949dffb700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f948c06ecc0 0x7f948c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:14.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.473+0000 7f949dffb700 1 -- 172.21.15.151:0/4106191371 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9490090380 con 0x7f94a00ff530 2024-04-16T02:53:14.477 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.473+0000 7f9497fff700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f948c06ecc0 0x7f948c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:14.477 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.473+0000 7f9497fff700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f948c06ecc0 0x7f948c071180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9488005950 tx=0x7f94880058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:14.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.478+0000 7f949dffb700 1 -- 172.21.15.151:0/4106191371 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f949005cc60 con 0x7f94a00ff530 2024-04-16T02:53:14.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.699+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f94a002ccc0 con 0x7f948c06ecc0 2024-04-16T02:53:14.701 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:14 smithi151 ceph-mon[112233]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:14.701 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:14 smithi151 ceph-mon[112233]: from='client.34850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:14.702 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:14 smithi151 ceph-mon[112233]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:14.702 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:14 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1799932599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:53:14.702 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:14 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2453815037' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:53:14.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.700+0000 7f949dffb700 1 -- 172.21.15.151:0/4106191371 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f94a002ccc0 con 0x7f948c06ecc0 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:53:14.703 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:53:14.704 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f948c06ecc0 msgr2=0x7f948c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f948c06ecc0 0x7f948c071180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9488005950 tx=0x7f94880058e0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 msgr2=0x7f94a019c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 0x7f94a019c540 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9490000c00 tx=0x7f949000be20 comp rx=0 tx=0).stop 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 shutdown_connections 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f948c06ecc0 0x7f948c071180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94a00ff530 0x7f94a019c540 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 --2- 172.21.15.151:0/4106191371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a00ffee0 0x7f94a019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 >> 172.21.15.151:0/4106191371 conn(0x7f94a0074df0 msgr2=0x7f94a01033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 shutdown_connections 2024-04-16T02:53:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.702+0000 7f94a67bd700 1 -- 172.21.15.151:0/4106191371 wait complete. 2024-04-16T02:53:14.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:14 smithi153 ceph-mon[87858]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:14.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:14 smithi153 ceph-mon[87858]: from='client.34850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:14.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:14 smithi153 ceph-mon[87858]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:14.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:14 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1799932599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:53:14.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:14 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2453815037' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:53:14.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.806+0000 7f1850032700 1 -- 172.21.15.151:0/397347353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 msgr2=0x7f18480ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.806+0000 7f1850032700 1 --2- 172.21.15.151:0/397347353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 0x7f18480ffec0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1838009910 tx=0x7f1838009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:14.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.807+0000 7f1850032700 1 -- 172.21.15.151:0/397347353 shutdown_connections 2024-04-16T02:53:14.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.807+0000 7f1850032700 1 --2- 172.21.15.151:0/397347353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 0x7f18480ffec0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.807+0000 7f1850032700 1 --2- 172.21.15.151:0/397347353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848105a40 0x7f1848105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.807+0000 7f1850032700 1 -- 172.21.15.151:0/397347353 >> 172.21.15.151:0/397347353 conn(0x7f18480fb550 msgr2=0x7f18480fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:14.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.807+0000 7f1850032700 1 -- 172.21.15.151:0/397347353 shutdown_connections 2024-04-16T02:53:14.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.807+0000 7f1850032700 1 -- 172.21.15.151:0/397347353 wait complete. 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.807+0000 7f1850032700 1 Processor -- start 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.807+0000 7f1850032700 1 -- start start 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.807+0000 7f1850032700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 0x7f184819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f1850032700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848105a40 0x7f184819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f1850032700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184819d0f0 con 0x7f18480ffa40 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f1850032700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18481965a0 con 0x7f1848105a40 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f184ddce700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 0x7f184819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f184ddce700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 0x7f184819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:37314/0 (socket says 172.21.15.151:37314) 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f184ddce700 1 -- 172.21.15.151:0/4051143513 learned_addr learned my addr 172.21.15.151:0/4051143513 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f184d5cd700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848105a40 0x7f184819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f184ddce700 1 -- 172.21.15.151:0/4051143513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848105a40 msgr2=0x7f184819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f184ddce700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848105a40 0x7f184819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f184ddce700 1 -- 172.21.15.151:0/4051143513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18380095f0 con 0x7f18480ffa40 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f184d5cd700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848105a40 0x7f184819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f184ddce700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 0x7f184819c520 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f184400c8a0 tx=0x7f184400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f183effd700 1 -- 172.21.15.151:0/4051143513 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f184400cea0 con 0x7f18480ffa40 2024-04-16T02:53:14.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f183effd700 1 -- 172.21.15.151:0/4051143513 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1844004830 con 0x7f18480ffa40 2024-04-16T02:53:14.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1848196880 con 0x7f18480ffa40 2024-04-16T02:53:14.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f183effd700 1 -- 172.21.15.151:0/4051143513 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18440055b0 con 0x7f18480ffa40 2024-04-16T02:53:14.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.808+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1848196dd0 con 0x7f18480ffa40 2024-04-16T02:53:14.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.809+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1848108fe0 con 0x7f18480ffa40 2024-04-16T02:53:14.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.810+0000 7f183effd700 1 -- 172.21.15.151:0/4051143513 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f18440049a0 con 0x7f18480ffa40 2024-04-16T02:53:14.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.810+0000 7f183effd700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f183406ecc0 0x7f1834071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:14.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.811+0000 7f183effd700 1 -- 172.21.15.151:0/4051143513 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f184408eb50 con 0x7f18480ffa40 2024-04-16T02:53:14.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.811+0000 7f184d5cd700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f183406ecc0 0x7f1834071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:14.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.811+0000 7f184d5cd700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f183406ecc0 0x7f1834071180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f183800b5c0 tx=0x7f1838005480 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:14.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:14.815+0000 7f183effd700 1 -- 172.21.15.151:0/4051143513 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f184405b430 con 0x7f18480ffa40 2024-04-16T02:53:15.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.103+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f184804ea90 con 0x7f18480ffa40 2024-04-16T02:53:15.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.104+0000 7f183effd700 1 -- 172.21.15.151:0/4051143513 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f184401d100 con 0x7f18480ffa40 2024-04-16T02:53:15.106 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:53:15.106 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:53:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:53:15.106 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:53:15.107 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:53:15.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f183406ecc0 msgr2=0x7f1834071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f183406ecc0 0x7f1834071180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f183800b5c0 tx=0x7f1838005480 comp rx=0 tx=0).stop 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 msgr2=0x7f184819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 0x7f184819c520 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f184400c8a0 tx=0x7f184400cc60 comp rx=0 tx=0).stop 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 shutdown_connections 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f183406ecc0 0x7f1834071180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f18480ffa40 0x7f184819c520 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 --2- 172.21.15.151:0/4051143513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848105a40 0x7f184819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 >> 172.21.15.151:0/4051143513 conn(0x7f18480fb550 msgr2=0x7f1848069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 shutdown_connections 2024-04-16T02:53:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:15.106+0000 7f1850032700 1 -- 172.21.15.151:0/4051143513 wait complete. 2024-04-16T02:53:15.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:15 smithi151 ceph-mon[112233]: pgmap v438: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:15.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:15 smithi151 ceph-mon[112233]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:15.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:15 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4051143513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:53:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:15 smithi153 ceph-mon[87858]: pgmap v438: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:15 smithi153 ceph-mon[87858]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:15 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4051143513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:53:17.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:17 smithi151 ceph-mon[112233]: pgmap v439: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:17.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:17 smithi153 ceph-mon[87858]: pgmap v439: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:19 smithi151 ceph-mon[112233]: pgmap v440: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:19.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:19 smithi153 ceph-mon[87858]: pgmap v440: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:21.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:21 smithi151 ceph-mon[112233]: pgmap v441: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:21.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:21 smithi153 ceph-mon[87858]: pgmap v441: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:23.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:23 smithi153 ceph-mon[87858]: pgmap v442: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:23 smithi151 ceph-mon[112233]: pgmap v442: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:24.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:53:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:53:25.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:25 smithi153 ceph-mon[87858]: pgmap v443: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:25 smithi151 ceph-mon[112233]: pgmap v443: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:27.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:27 smithi151 ceph-mon[112233]: pgmap v444: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:27.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:27 smithi153 ceph-mon[87858]: pgmap v444: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:29.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:29 smithi153 ceph-mon[87858]: pgmap v445: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:29.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:29 smithi151 ceph-mon[112233]: pgmap v445: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:31.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:31 smithi153 ceph-mon[87858]: pgmap v446: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:31.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:31 smithi151 ceph-mon[112233]: pgmap v446: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:33.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:33 smithi151 ceph-mon[112233]: pgmap v447: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:34.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:33 smithi153 ceph-mon[87858]: pgmap v447: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:35.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:35 smithi151 ceph-mon[112233]: pgmap v448: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:36.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:35 smithi153 ceph-mon[87858]: pgmap v448: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:37.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:37 smithi151 ceph-mon[112233]: pgmap v449: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:38.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:37 smithi153 ceph-mon[87858]: pgmap v449: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:39.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:39 smithi151 ceph-mon[112233]: pgmap v450: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:39.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:53:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:39 smithi153 ceph-mon[87858]: pgmap v450: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:53:41.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:41 smithi151 ceph-mon[112233]: pgmap v451: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:42.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:41 smithi153 ceph-mon[87858]: pgmap v451: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:43.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:43 smithi151 ceph-mon[112233]: pgmap v452: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:44.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:43 smithi153 ceph-mon[87858]: pgmap v452: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.223+0000 7efc9275f700 1 -- 172.21.15.151:0/1432046042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 msgr2=0x7efc8c068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.223+0000 7efc9275f700 1 --2- 172.21.15.151:0/1432046042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 0x7efc8c068a50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009910 tx=0x7efc7c009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.223+0000 7efc9275f700 1 -- 172.21.15.151:0/1432046042 shutdown_connections 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.223+0000 7efc9275f700 1 --2- 172.21.15.151:0/1432046042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc8c069020 0x7efc8c105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.223+0000 7efc9275f700 1 --2- 172.21.15.151:0/1432046042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 0x7efc8c068a50 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.223+0000 7efc9275f700 1 -- 172.21.15.151:0/1432046042 >> 172.21.15.151:0/1432046042 conn(0x7efc8c0755a0 msgr2=0x7efc8c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.223+0000 7efc9275f700 1 -- 172.21.15.151:0/1432046042 shutdown_connections 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.223+0000 7efc9275f700 1 -- 172.21.15.151:0/1432046042 wait complete. 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc9275f700 1 Processor -- start 2024-04-16T02:53:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc9275f700 1 -- start start 2024-04-16T02:53:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc9275f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 0x7efc8c19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc9275f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc8c069020 0x7efc8c19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc9275f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c19f2e0 con 0x7efc8c068670 2024-04-16T02:53:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc9275f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c198790 con 0x7efc8c069020 2024-04-16T02:53:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc8bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 0x7efc8c19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc8bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 0x7efc8c19e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55188/0 (socket says 172.21.15.151:55188) 2024-04-16T02:53:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc8bfff700 1 -- 172.21.15.151:0/1351738420 learned_addr learned my addr 172.21.15.151:0/1351738420 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc8b7fe700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc8c069020 0x7efc8c19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc8bfff700 1 -- 172.21.15.151:0/1351738420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc8c069020 msgr2=0x7efc8c19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc8bfff700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc8c069020 0x7efc8c19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.224+0000 7efc8bfff700 1 -- 172.21.15.151:0/1351738420 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0095f0 con 0x7efc8c068670 2024-04-16T02:53:45.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.225+0000 7efc8bfff700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 0x7efc8c19e710 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efc7c006010 tx=0x7efc7c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:45.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.225+0000 7efc897fa700 1 -- 172.21.15.151:0/1351738420 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc7c01d070 con 0x7efc8c068670 2024-04-16T02:53:45.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.225+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8c198a10 con 0x7efc8c068670 2024-04-16T02:53:45.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.225+0000 7efc897fa700 1 -- 172.21.15.151:0/1351738420 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc7c005dc0 con 0x7efc8c068670 2024-04-16T02:53:45.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.225+0000 7efc897fa700 1 -- 172.21.15.151:0/1351738420 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc7c017400 con 0x7efc8c068670 2024-04-16T02:53:45.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.225+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8c198f00 con 0x7efc8c068670 2024-04-16T02:53:45.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.226+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8c04ea90 con 0x7efc8c068670 2024-04-16T02:53:45.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.227+0000 7efc897fa700 1 -- 172.21.15.151:0/1351738420 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efc7c003a60 con 0x7efc8c068670 2024-04-16T02:53:45.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.227+0000 7efc897fa700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc7406ecc0 0x7efc74071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:45.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.227+0000 7efc897fa700 1 -- 172.21.15.151:0/1351738420 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efc7c0902f0 con 0x7efc8c068670 2024-04-16T02:53:45.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.227+0000 7efc8b7fe700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc7406ecc0 0x7efc74071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:45.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.228+0000 7efc8b7fe700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc7406ecc0 0x7efc74071180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efc8c199c00 tx=0x7efc800073d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:45.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.233+0000 7efc897fa700 1 -- 172.21.15.151:0/1351738420 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efc7c05cbd0 con 0x7efc8c068670 2024-04-16T02:53:45.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.451+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efc8c1993f0 con 0x7efc7406ecc0 2024-04-16T02:53:45.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.452+0000 7efc897fa700 1 -- 172.21.15.151:0/1351738420 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efc8c1993f0 con 0x7efc7406ecc0 2024-04-16T02:53:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.454+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc7406ecc0 msgr2=0x7efc74071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.454+0000 7efc9275f700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc7406ecc0 0x7efc74071180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efc8c199c00 tx=0x7efc800073d0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.455+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 msgr2=0x7efc8c19e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.455+0000 7efc9275f700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 0x7efc8c19e710 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efc7c006010 tx=0x7efc7c00bd70 comp rx=0 tx=0).stop 2024-04-16T02:53:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.455+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 shutdown_connections 2024-04-16T02:53:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.455+0000 7efc9275f700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc7406ecc0 0x7efc74071180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.455+0000 7efc9275f700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc8c068670 0x7efc8c19e710 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.458 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.455+0000 7efc9275f700 1 --2- 172.21.15.151:0/1351738420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc8c069020 0x7efc8c19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.455+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 >> 172.21.15.151:0/1351738420 conn(0x7efc8c0755a0 msgr2=0x7efc8c102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:45.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.455+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 shutdown_connections 2024-04-16T02:53:45.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.455+0000 7efc9275f700 1 -- 172.21.15.151:0/1351738420 wait complete. 2024-04-16T02:53:45.469 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:53:45.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.562+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3974998495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 msgr2=0x7fd4a4068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.562+0000 7fd4a8f93700 1 --2- 172.21.15.151:0/3974998495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 0x7fd4a4068a50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd494009910 tx=0x7fd494009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:45.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.562+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3974998495 shutdown_connections 2024-04-16T02:53:45.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.562+0000 7fd4a8f93700 1 --2- 172.21.15.151:0/3974998495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4a4069020 0x7fd4a4105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.562+0000 7fd4a8f93700 1 --2- 172.21.15.151:0/3974998495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 0x7fd4a4068a50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.562+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3974998495 >> 172.21.15.151:0/3974998495 conn(0x7fd4a40755a0 msgr2=0x7fd4a40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:45.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.562+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3974998495 shutdown_connections 2024-04-16T02:53:45.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.562+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3974998495 wait complete. 2024-04-16T02:53:45.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.562+0000 7fd4a8f93700 1 Processor -- start 2024-04-16T02:53:45.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a8f93700 1 -- start start 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a8f93700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 0x7fd4a419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a8f93700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4a4069020 0x7fd4a419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a8f93700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a419f2e0 con 0x7fd4a4068670 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a8f93700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a4198790 con 0x7fd4a4069020 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 0x7fd4a419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 0x7fd4a419e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55200/0 (socket says 172.21.15.151:55200) 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a259c700 1 -- 172.21.15.151:0/3257358800 learned_addr learned my addr 172.21.15.151:0/3257358800 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd499d9b700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4a4069020 0x7fd4a419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a259c700 1 -- 172.21.15.151:0/3257358800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4a4069020 msgr2=0x7fd4a419ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a259c700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4a4069020 0x7fd4a419ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.563+0000 7fd4a259c700 1 -- 172.21.15.151:0/3257358800 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4940095f0 con 0x7fd4a4068670 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.564+0000 7fd4a259c700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 0x7fd4a419e710 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd494006010 tx=0x7fd49400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.564+0000 7fd49bfff700 1 -- 172.21.15.151:0/3257358800 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd49401d070 con 0x7fd4a4068670 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.564+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a4198a10 con 0x7fd4a4068670 2024-04-16T02:53:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.564+0000 7fd49bfff700 1 -- 172.21.15.151:0/3257358800 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd494005dc0 con 0x7fd4a4068670 2024-04-16T02:53:45.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.564+0000 7fd49bfff700 1 -- 172.21.15.151:0/3257358800 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd494017400 con 0x7fd4a4068670 2024-04-16T02:53:45.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.564+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a4198f00 con 0x7fd4a4068670 2024-04-16T02:53:45.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.565+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a404ea90 con 0x7fd4a4068670 2024-04-16T02:53:45.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.566+0000 7fd49bfff700 1 -- 172.21.15.151:0/3257358800 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd494003a60 con 0x7fd4a4068670 2024-04-16T02:53:45.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.566+0000 7fd49bfff700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd49006ecc0 0x7fd490071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:45.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.566+0000 7fd49bfff700 1 -- 172.21.15.151:0/3257358800 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd4940902f0 con 0x7fd4a4068670 2024-04-16T02:53:45.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.566+0000 7fd499d9b700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd49006ecc0 0x7fd490071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:45.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.566+0000 7fd499d9b700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd49006ecc0 0x7fd490071180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4199c00 tx=0x7fd48c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:45.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.571+0000 7fd49bfff700 1 -- 172.21.15.151:0/3257358800 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd49405cbd0 con 0x7fd4a4068670 2024-04-16T02:53:45.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.788+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4a4199340 con 0x7fd49006ecc0 2024-04-16T02:53:45.792 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:45 smithi151 ceph-mon[112233]: pgmap v453: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:45.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.789+0000 7fd49bfff700 1 -- 172.21.15.151:0/3257358800 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd4a4199340 con 0x7fd49006ecc0 2024-04-16T02:53:45.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd49006ecc0 msgr2=0x7fd490071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd49006ecc0 0x7fd490071180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4199c00 tx=0x7fd48c00b410 comp rx=0 tx=0).stop 2024-04-16T02:53:45.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 msgr2=0x7fd4a419e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 0x7fd4a419e710 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd494006010 tx=0x7fd49400bd70 comp rx=0 tx=0).stop 2024-04-16T02:53:45.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 shutdown_connections 2024-04-16T02:53:45.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd49006ecc0 0x7fd490071180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4a4068670 0x7fd4a419e710 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 --2- 172.21.15.151:0/3257358800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4a4069020 0x7fd4a419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 >> 172.21.15.151:0/3257358800 conn(0x7fd4a40755a0 msgr2=0x7fd4a4102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:45.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 shutdown_connections 2024-04-16T02:53:45.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.792+0000 7fd4a8f93700 1 -- 172.21.15.151:0/3257358800 wait complete. 2024-04-16T02:53:45.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.896+0000 7ff45feac700 1 -- 172.21.15.151:0/2573509246 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff458105b20 msgr2=0x7ff458105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.896+0000 7ff45feac700 1 --2- 172.21.15.151:0/2573509246 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff458105b20 0x7ff458105f00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff448009910 tx=0x7ff448009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.896+0000 7ff45feac700 1 -- 172.21.15.151:0/2573509246 shutdown_connections 2024-04-16T02:53:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.896+0000 7ff45feac700 1 --2- 172.21.15.151:0/2573509246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4580ff050 0x7ff4580ff4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.896+0000 7ff45feac700 1 --2- 172.21.15.151:0/2573509246 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff458105b20 0x7ff458105f00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.896+0000 7ff45feac700 1 -- 172.21.15.151:0/2573509246 >> 172.21.15.151:0/2573509246 conn(0x7ff458075e30 msgr2=0x7ff4580fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.896+0000 7ff45feac700 1 -- 172.21.15.151:0/2573509246 shutdown_connections 2024-04-16T02:53:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.896+0000 7ff45feac700 1 -- 172.21.15.151:0/2573509246 wait complete. 2024-04-16T02:53:45.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45feac700 1 Processor -- start 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45feac700 1 -- start start 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45feac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4580ff050 0x7ff458100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45feac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff458105b20 0x7ff458100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45feac700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff458100cb0 con 0x7ff4580ff050 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45feac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff458100df0 con 0x7ff458105b20 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45dc48700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4580ff050 0x7ff458100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45dc48700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4580ff050 0x7ff458100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55210/0 (socket says 172.21.15.151:55210) 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45dc48700 1 -- 172.21.15.151:0/944192318 learned_addr learned my addr 172.21.15.151:0/944192318 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45d447700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff458105b20 0x7ff458100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45dc48700 1 -- 172.21.15.151:0/944192318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff458105b20 msgr2=0x7ff458100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45dc48700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff458105b20 0x7ff458100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.897+0000 7ff45dc48700 1 -- 172.21.15.151:0/944192318 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4480095f0 con 0x7ff4580ff050 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.898+0000 7ff45dc48700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4580ff050 0x7ff458100230 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff448000c00 tx=0x7ff44800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.898+0000 7ff44effd700 1 -- 172.21.15.151:0/944192318 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff44801d070 con 0x7ff4580ff050 2024-04-16T02:53:45.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.898+0000 7ff44effd700 1 -- 172.21.15.151:0/944192318 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff44800f460 con 0x7ff4580ff050 2024-04-16T02:53:45.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.898+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff458101070 con 0x7ff4580ff050 2024-04-16T02:53:45.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.898+0000 7ff44effd700 1 -- 172.21.15.151:0/944192318 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff448005e20 con 0x7ff4580ff050 2024-04-16T02:53:45.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.898+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4581a2c40 con 0x7ff4580ff050 2024-04-16T02:53:45.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.899+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff458109060 con 0x7ff4580ff050 2024-04-16T02:53:45.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.899+0000 7ff44effd700 1 -- 172.21.15.151:0/944192318 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff44800f5d0 con 0x7ff4580ff050 2024-04-16T02:53:45.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.900+0000 7ff44effd700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44406ecc0 0x7ff444071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:45.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.900+0000 7ff44effd700 1 -- 172.21.15.151:0/944192318 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff4480902d0 con 0x7ff4580ff050 2024-04-16T02:53:45.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.900+0000 7ff45d447700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44406ecc0 0x7ff444071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:45.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.900+0000 7ff45d447700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44406ecc0 0x7ff444071180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff454005950 tx=0x7ff4540058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:45.908 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:45.905+0000 7ff44effd700 1 -- 172.21.15.151:0/944192318 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff4480591c0 con 0x7ff4580ff050 2024-04-16T02:53:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:45 smithi153 ceph-mon[87858]: pgmap v453: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:46.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.111+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4580611d0 con 0x7ff44406ecc0 2024-04-16T02:53:46.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.123+0000 7ff44effd700 1 -- 172.21.15.151:0/944192318 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff4580611d0 con 0x7ff44406ecc0 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (15m) 8m ago 26m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (26m) 8m ago 26m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (25m) 8m ago 25m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (12m) 8m ago 26m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (12m) 8m ago 25m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (14m) 8m ago 26m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (9m) 8m ago 21m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (8m) 8m ago 21m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (20m) 8m ago 20m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (8m) 8m ago 21m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (17m) 8m ago 28m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:53:46.126 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (16m) 8m ago 25m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (14m) 8m ago 28m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (13m) 8m ago 25m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (16m) 8m ago 26m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (15m) 8m ago 25m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (12m) 8m ago 24m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (11m) 8m ago 24m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (11m) 8m ago 23m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (10m) 8m ago 23m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (10m) 8m ago 23m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (9m) 8m ago 22m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:53:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (15m) 8m ago 26m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:53:46.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44406ecc0 msgr2=0x7ff444071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44406ecc0 0x7ff444071180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff454005950 tx=0x7ff4540058e0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4580ff050 msgr2=0x7ff458100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4580ff050 0x7ff458100230 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff448000c00 tx=0x7ff44800be20 comp rx=0 tx=0).stop 2024-04-16T02:53:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 shutdown_connections 2024-04-16T02:53:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44406ecc0 0x7ff444071180 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4580ff050 0x7ff458100230 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 --2- 172.21.15.151:0/944192318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff458105b20 0x7ff458100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 >> 172.21.15.151:0/944192318 conn(0x7ff458075e30 msgr2=0x7ff4580fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 shutdown_connections 2024-04-16T02:53:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.126+0000 7ff45feac700 1 -- 172.21.15.151:0/944192318 wait complete. 2024-04-16T02:53:46.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.228+0000 7f47fd434700 1 -- 172.21.15.151:0/320775449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 msgr2=0x7f47f810cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.228+0000 7f47fd434700 1 --2- 172.21.15.151:0/320775449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 0x7f47f810cee0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009910 tx=0x7f47e8009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:46.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.229+0000 7f47fd434700 1 -- 172.21.15.151:0/320775449 shutdown_connections 2024-04-16T02:53:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.229+0000 7f47fd434700 1 --2- 172.21.15.151:0/320775449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 0x7f47f810cee0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.229+0000 7f47fd434700 1 --2- 172.21.15.151:0/320775449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47f80730b0 0x7f47f8073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.229+0000 7f47fd434700 1 -- 172.21.15.151:0/320775449 >> 172.21.15.151:0/320775449 conn(0x7f47f80fc650 msgr2=0x7f47f80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.229+0000 7f47fd434700 1 -- 172.21.15.151:0/320775449 shutdown_connections 2024-04-16T02:53:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.229+0000 7f47fd434700 1 -- 172.21.15.151:0/320775449 wait complete. 2024-04-16T02:53:46.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.229+0000 7f47fd434700 1 Processor -- start 2024-04-16T02:53:46.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.229+0000 7f47fd434700 1 -- start start 2024-04-16T02:53:46.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47fd434700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47f80730b0 0x7f47f819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:46.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47fd434700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 0x7f47f819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47fd434700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f819f2e0 con 0x7f47f80739d0 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47fd434700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8198790 con 0x7f47f80730b0 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47f67fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 0x7f47f819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47f67fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 0x7f47f819ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55228/0 (socket says 172.21.15.151:55228) 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47f67fc700 1 -- 172.21.15.151:0/4010219761 learned_addr learned my addr 172.21.15.151:0/4010219761 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47f6ffd700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47f80730b0 0x7f47f819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47f67fc700 1 -- 172.21.15.151:0/4010219761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47f80730b0 msgr2=0x7f47f819e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47f67fc700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47f80730b0 0x7f47f819e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47f67fc700 1 -- 172.21.15.151:0/4010219761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e80095f0 con 0x7f47f80739d0 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47f67fc700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 0x7f47f819ec50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f47e80092e0 tx=0x7f47e800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47dffff700 1 -- 172.21.15.151:0/4010219761 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47e801d070 con 0x7f47f80739d0 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47dffff700 1 -- 172.21.15.151:0/4010219761 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47e8005e00 con 0x7f47f80739d0 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.230+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f8198a10 con 0x7f47f80739d0 2024-04-16T02:53:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.231+0000 7f47dffff700 1 -- 172.21.15.151:0/4010219761 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47e8017400 con 0x7f47f80739d0 2024-04-16T02:53:46.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.231+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f8198f00 con 0x7f47f80739d0 2024-04-16T02:53:46.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.232+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f810a5e0 con 0x7f47f80739d0 2024-04-16T02:53:46.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.232+0000 7f47dffff700 1 -- 172.21.15.151:0/4010219761 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f47e800f6f0 con 0x7f47f80739d0 2024-04-16T02:53:46.237 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.232+0000 7f47dffff700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47e006ecc0 0x7f47e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:46.237 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.233+0000 7f47dffff700 1 -- 172.21.15.151:0/4010219761 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47e8090330 con 0x7f47f80739d0 2024-04-16T02:53:46.237 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.233+0000 7f47f6ffd700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47e006ecc0 0x7f47e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:46.237 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.233+0000 7f47f6ffd700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47e006ecc0 0x7f47e0071180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005950 tx=0x7f47ec00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:46.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.238+0000 7f47dffff700 1 -- 172.21.15.151:0/4010219761 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f47e805cc10 con 0x7f47f80739d0 2024-04-16T02:53:46.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.520+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f47f804ea90 con 0x7f47f80739d0 2024-04-16T02:53:46.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.521+0000 7f47dffff700 1 -- 172.21.15.151:0/4010219761 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f47e805c7a0 con 0x7f47f80739d0 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:53:46.524 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:53:46.525 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:53:46.525 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:53:46.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:53:46.525 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:53:46.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47e006ecc0 msgr2=0x7f47e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47e006ecc0 0x7f47e0071180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005950 tx=0x7f47ec00b410 comp rx=0 tx=0).stop 2024-04-16T02:53:46.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 msgr2=0x7f47f819ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 0x7f47f819ec50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f47e80092e0 tx=0x7f47e800bd70 comp rx=0 tx=0).stop 2024-04-16T02:53:46.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 shutdown_connections 2024-04-16T02:53:46.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47e006ecc0 0x7f47e0071180 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47f80730b0 0x7f47f819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 --2- 172.21.15.151:0/4010219761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f80739d0 0x7f47f819ec50 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 >> 172.21.15.151:0/4010219761 conn(0x7f47f80fc650 msgr2=0x7f47f8107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:46.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 shutdown_connections 2024-04-16T02:53:46.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.524+0000 7f47fd434700 1 -- 172.21.15.151:0/4010219761 wait complete. 2024-04-16T02:53:46.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.628+0000 7fd02f9ce700 1 -- 172.21.15.151:0/3737724781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0280ffa60 msgr2=0x7fd0280ffe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.628+0000 7fd02f9ce700 1 --2- 172.21.15.151:0/3737724781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0280ffa60 0x7fd0280ffe40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd024009910 tx=0x7fd024009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:46.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 -- 172.21.15.151:0/3737724781 shutdown_connections 2024-04-16T02:53:46.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 --2- 172.21.15.151:0/3737724781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd028100410 0x7fd0281048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 --2- 172.21.15.151:0/3737724781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0280ffa60 0x7fd0280ffe40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 -- 172.21.15.151:0/3737724781 >> 172.21.15.151:0/3737724781 conn(0x7fd0280fb530 msgr2=0x7fd0280fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:46.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 -- 172.21.15.151:0/3737724781 shutdown_connections 2024-04-16T02:53:46.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 -- 172.21.15.151:0/3737724781 wait complete. 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 Processor -- start 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 -- start start 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0280ffa60 0x7fd02819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.629+0000 7fd02f9ce700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd028100410 0x7fd02819d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02f9ce700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02819d960 con 0x7fd028100410 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02f9ce700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0281a16f0 con 0x7fd0280ffa60 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02cf69700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd028100410 0x7fd02819d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02cf69700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd028100410 0x7fd02819d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55318/0 (socket says 172.21.15.151:55318) 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02cf69700 1 -- 172.21.15.151:0/192812198 learned_addr learned my addr 172.21.15.151:0/192812198 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02d76a700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0280ffa60 0x7fd02819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02cf69700 1 -- 172.21.15.151:0/192812198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0280ffa60 msgr2=0x7fd02819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02cf69700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0280ffa60 0x7fd02819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02cf69700 1 -- 172.21.15.151:0/192812198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0240095f0 con 0x7fd028100410 2024-04-16T02:53:46.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02cf69700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd028100410 0x7fd02819d280 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd01800d900 tx=0x7fd01800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:46.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd01e7fc700 1 -- 172.21.15.151:0/192812198 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd018009900 con 0x7fd028100410 2024-04-16T02:53:46.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd01e7fc700 1 -- 172.21.15.151:0/192812198 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd018010460 con 0x7fd028100410 2024-04-16T02:53:46.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0281a19d0 con 0x7fd028100410 2024-04-16T02:53:46.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd01e7fc700 1 -- 172.21.15.151:0/192812198 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd01800f5f0 con 0x7fd028100410 2024-04-16T02:53:46.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.630+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0281a1f20 con 0x7fd028100410 2024-04-16T02:53:46.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.631+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd028102040 con 0x7fd028100410 2024-04-16T02:53:46.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.632+0000 7fd01e7fc700 1 -- 172.21.15.151:0/192812198 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd0180105d0 con 0x7fd028100410 2024-04-16T02:53:46.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.632+0000 7fd01e7fc700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd01406ecc0 0x7fd014071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:46.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.632+0000 7fd01e7fc700 1 -- 172.21.15.151:0/192812198 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd01808eb60 con 0x7fd028100410 2024-04-16T02:53:46.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.632+0000 7fd02d76a700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd01406ecc0 0x7fd014071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:46.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.633+0000 7fd02d76a700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd01406ecc0 0x7fd014071180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd024006010 tx=0x7fd02400b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:46.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.637+0000 7fd01e7fc700 1 -- 172.21.15.151:0/192812198 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd018057870 con 0x7fd028100410 2024-04-16T02:53:46.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.880+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd02804ea90 con 0x7fd028100410 2024-04-16T02:53:46.883 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:46 smithi151 ceph-mon[112233]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:46.883 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:46 smithi151 ceph-mon[112233]: from='client.34878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:46.884 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:46 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4010219761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:53:46.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.881+0000 7fd01e7fc700 1 -- 172.21.15.151:0/192812198 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7fd018017770 con 0x7fd028100410 2024-04-16T02:53:46.884 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:53:46.884 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:53:46.884 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:53:46.884 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:53:46.884 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:53:46.884 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:53:46.884 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:53:46.885 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:53:46.886 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:53:46.887 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:53:46.887 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:53:46.887 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:53:46.887 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:53:46.887 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:53:46.887 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd01406ecc0 msgr2=0x7fd014071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd01406ecc0 0x7fd014071180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd024006010 tx=0x7fd02400b540 comp rx=0 tx=0).stop 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd028100410 msgr2=0x7fd02819d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd028100410 0x7fd02819d280 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd01800d900 tx=0x7fd01800dcc0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 shutdown_connections 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd01406ecc0 0x7fd014071180 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0280ffa60 0x7fd02819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 --2- 172.21.15.151:0/192812198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd028100410 0x7fd02819d280 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 >> 172.21.15.151:0/192812198 conn(0x7fd0280fb530 msgr2=0x7fd028110f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 shutdown_connections 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.884+0000 7fd02f9ce700 1 -- 172.21.15.151:0/192812198 wait complete. 2024-04-16T02:53:46.888 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:53:46.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.991+0000 7f2e61267700 1 -- 172.21.15.151:0/3120018464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c069020 msgr2=0x7f2e5c105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.991+0000 7f2e61267700 1 --2- 172.21.15.151:0/3120018464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c069020 0x7f2e5c105790 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2e50009910 tx=0x7f2e50009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:46.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.991+0000 7f2e61267700 1 -- 172.21.15.151:0/3120018464 shutdown_connections 2024-04-16T02:53:46.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.991+0000 7f2e61267700 1 --2- 172.21.15.151:0/3120018464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c069020 0x7f2e5c105790 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.991+0000 7f2e61267700 1 --2- 172.21.15.151:0/3120018464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e5c068670 0x7f2e5c068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.991+0000 7f2e61267700 1 -- 172.21.15.151:0/3120018464 >> 172.21.15.151:0/3120018464 conn(0x7f2e5c0755a0 msgr2=0x7f2e5c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:46.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.991+0000 7f2e61267700 1 -- 172.21.15.151:0/3120018464 shutdown_connections 2024-04-16T02:53:46.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.992+0000 7f2e61267700 1 -- 172.21.15.151:0/3120018464 wait complete. 2024-04-16T02:53:46.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.992+0000 7f2e61267700 1 Processor -- start 2024-04-16T02:53:46.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.992+0000 7f2e61267700 1 -- start start 2024-04-16T02:53:46.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.992+0000 7f2e61267700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c068670 0x7f2e5c19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:46.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e61267700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e5c069020 0x7f2e5c19ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:46.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e61267700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c19f3b0 con 0x7f2e5c068670 2024-04-16T02:53:46.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e61267700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c198860 con 0x7f2e5c069020 2024-04-16T02:53:46.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e5affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c068670 0x7f2e5c19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e5affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c068670 0x7f2e5c19e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55326/0 (socket says 172.21.15.151:55326) 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e5affd700 1 -- 172.21.15.151:0/3030298816 learned_addr learned my addr 172.21.15.151:0/3030298816 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e5a7fc700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e5c069020 0x7f2e5c19ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e5affd700 1 -- 172.21.15.151:0/3030298816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e5c069020 msgr2=0x7f2e5c19ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e5affd700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e5c069020 0x7f2e5c19ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e5affd700 1 -- 172.21.15.151:0/3030298816 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e500095f0 con 0x7f2e5c068670 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e5affd700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c068670 0x7f2e5c19e7e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00d8d0 tx=0x7f2e4c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e43fff700 1 -- 172.21.15.151:0/3030298816 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e4c0098a0 con 0x7f2e5c068670 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5c198b40 con 0x7f2e5c068670 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e43fff700 1 -- 172.21.15.151:0/3030298816 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e4c010460 con 0x7f2e5c068670 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e43fff700 1 -- 172.21.15.151:0/3030298816 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e4c00f5f0 con 0x7f2e5c068670 2024-04-16T02:53:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.993+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5c199090 con 0x7f2e5c068670 2024-04-16T02:53:46.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.994+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5c04ea90 con 0x7f2e5c068670 2024-04-16T02:53:46.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.995+0000 7f2e43fff700 1 -- 172.21.15.151:0/3030298816 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2e4c009a00 con 0x7f2e5c068670 2024-04-16T02:53:46.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.995+0000 7f2e43fff700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e4406ecc0 0x7f2e44071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:46.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.995+0000 7f2e43fff700 1 -- 172.21.15.151:0/3030298816 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2e4c08ed30 con 0x7f2e5c068670 2024-04-16T02:53:46.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.995+0000 7f2e5a7fc700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e4406ecc0 0x7f2e44071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:46.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:46.996+0000 7f2e5a7fc700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e4406ecc0 0x7f2e44071180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0fe9d0 tx=0x7f2e5001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:47.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.000+0000 7f2e43fff700 1 -- 172.21.15.151:0/3030298816 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2e4c05b610 con 0x7f2e5c068670 2024-04-16T02:53:47.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:46 smithi153 ceph-mon[87858]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:46 smithi153 ceph-mon[87858]: from='client.34878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:47.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:46 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4010219761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:53:47.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.220+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e5c199390 con 0x7f2e4406ecc0 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.221+0000 7f2e43fff700 1 -- 172.21.15.151:0/3030298816 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2e5c199390 con 0x7f2e4406ecc0 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:53:47.224 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:53:47.225 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:53:47.225 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:53:47.225 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:53:47.225 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:53:47.225 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:53:47.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e4406ecc0 msgr2=0x7f2e44071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:47.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e4406ecc0 0x7f2e44071180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0fe9d0 tx=0x7f2e5001a040 comp rx=0 tx=0).stop 2024-04-16T02:53:47.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c068670 msgr2=0x7f2e5c19e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:47.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c068670 0x7f2e5c19e7e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00d8d0 tx=0x7f2e4c00dbe0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 shutdown_connections 2024-04-16T02:53:47.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e4406ecc0 0x7f2e44071180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e5c068670 0x7f2e5c19e7e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 --2- 172.21.15.151:0/3030298816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e5c069020 0x7f2e5c19ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 >> 172.21.15.151:0/3030298816 conn(0x7f2e5c0755a0 msgr2=0x7f2e5c102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:47.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 shutdown_connections 2024-04-16T02:53:47.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.225+0000 7f2e61267700 1 -- 172.21.15.151:0/3030298816 wait complete. 2024-04-16T02:53:47.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.329+0000 7f8107093700 1 -- 172.21.15.151:0/1747254130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 msgr2=0x7f8100068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:47.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.329+0000 7f8107093700 1 --2- 172.21.15.151:0/1747254130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 0x7f8100068a50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009910 tx=0x7f80f4009c20 comp rx=0 tx=0).stop 2024-04-16T02:53:47.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.329+0000 7f8107093700 1 -- 172.21.15.151:0/1747254130 shutdown_connections 2024-04-16T02:53:47.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.329+0000 7f8107093700 1 --2- 172.21.15.151:0/1747254130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8100069020 0x7f8100105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 --2- 172.21.15.151:0/1747254130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 0x7f8100068a50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 -- 172.21.15.151:0/1747254130 >> 172.21.15.151:0/1747254130 conn(0x7f81000755a0 msgr2=0x7f81000759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:47.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 -- 172.21.15.151:0/1747254130 shutdown_connections 2024-04-16T02:53:47.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 -- 172.21.15.151:0/1747254130 wait complete. 2024-04-16T02:53:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 Processor -- start 2024-04-16T02:53:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 -- start start 2024-04-16T02:53:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 0x7f81000ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8100069020 0x7f81001001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100100720 con 0x7f8100068670 2024-04-16T02:53:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8107093700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100100860 con 0x7f8100069020 2024-04-16T02:53:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8104e2f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 0x7f81000ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.330+0000 7f8104e2f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 0x7f81000ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55340/0 (socket says 172.21.15.151:55340) 2024-04-16T02:53:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f8104e2f700 1 -- 172.21.15.151:0/1215508427 learned_addr learned my addr 172.21.15.151:0/1215508427 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f80fffff700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8100069020 0x7f81001001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f8104e2f700 1 -- 172.21.15.151:0/1215508427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8100069020 msgr2=0x7f81001001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f8104e2f700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8100069020 0x7f81001001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f8104e2f700 1 -- 172.21.15.151:0/1215508427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f40095f0 con 0x7f8100068670 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f8104e2f700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 0x7f81000ffca0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f80f400f690 tx=0x7f80f400f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f80fdffb700 1 -- 172.21.15.151:0/1215508427 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80f401d070 con 0x7f8100068670 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8100100ae0 con 0x7f8100068670 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f80fdffb700 1 -- 172.21.15.151:0/1215508427 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80f40050d0 con 0x7f8100068670 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f80fdffb700 1 -- 172.21.15.151:0/1215508427 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80f4022470 con 0x7f8100068670 2024-04-16T02:53:47.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.331+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81001a2c40 con 0x7f8100068670 2024-04-16T02:53:47.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.332+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8100109130 con 0x7f8100068670 2024-04-16T02:53:47.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.333+0000 7f80fdffb700 1 -- 172.21.15.151:0/1215508427 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f80f4003680 con 0x7f8100068670 2024-04-16T02:53:47.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.333+0000 7f80fdffb700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80f006ecc0 0x7f80f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:53:47.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.333+0000 7f80fdffb700 1 -- 172.21.15.151:0/1215508427 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f80f4090240 con 0x7f8100068670 2024-04-16T02:53:47.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.333+0000 7f80fffff700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80f006ecc0 0x7f80f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:53:47.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.334+0000 7f80fffff700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80f006ecc0 0x7f80f0071180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8100101490 tx=0x7f80ec009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:53:47.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.338+0000 7f80fdffb700 1 -- 172.21.15.151:0/1215508427 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f80f4059070 con 0x7f8100068670 2024-04-16T02:53:47.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.628+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f810002cd30 con 0x7f8100068670 2024-04-16T02:53:47.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.628+0000 7f80fdffb700 1 -- 172.21.15.151:0/1215508427 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f80f405c730 con 0x7f8100068670 2024-04-16T02:53:47.631 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:53:47.631 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:53:47.631 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:53:47.631 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:53:47.631 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80f006ecc0 msgr2=0x7f80f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80f006ecc0 0x7f80f0071180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8100101490 tx=0x7f80ec009260 comp rx=0 tx=0).stop 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 msgr2=0x7f81000ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 0x7f81000ffca0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f80f400f690 tx=0x7f80f400f770 comp rx=0 tx=0).stop 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 shutdown_connections 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80f006ecc0 0x7f80f0071180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8100068670 0x7f81000ffca0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 --2- 172.21.15.151:0/1215508427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8100069020 0x7f81001001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 >> 172.21.15.151:0/1215508427 conn(0x7f81000755a0 msgr2=0x7f81000fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:53:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.630+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 shutdown_connections 2024-04-16T02:53:47.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:53:47.631+0000 7f8107093700 1 -- 172.21.15.151:0/1215508427 wait complete. 2024-04-16T02:53:47.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:47 smithi151 ceph-mon[112233]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:47.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:47 smithi151 ceph-mon[112233]: pgmap v454: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:47.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:47 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/192812198' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:53:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:47 smithi153 ceph-mon[87858]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:47 smithi153 ceph-mon[87858]: pgmap v454: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:47 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/192812198' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:53:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:48 smithi151 ceph-mon[112233]: from='client.34894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:48.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:48 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1215508427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:53:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:48 smithi153 ceph-mon[87858]: from='client.34894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:53:49.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:48 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1215508427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:53:49.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:49 smithi153 ceph-mon[87858]: pgmap v455: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:49 smithi151 ceph-mon[112233]: pgmap v455: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:51.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:51 smithi151 ceph-mon[112233]: pgmap v456: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:52.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:51 smithi153 ceph-mon[87858]: pgmap v456: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:53.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:53 smithi151 ceph-mon[112233]: pgmap v457: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:54.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:53 smithi153 ceph-mon[87858]: pgmap v457: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:54.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:53:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:53:55.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:55 smithi151 ceph-mon[112233]: pgmap v458: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:55 smithi153 ceph-mon[87858]: pgmap v458: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:57 smithi151 ceph-mon[112233]: pgmap v459: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:58.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:57 smithi153 ceph-mon[87858]: pgmap v459: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:53:59.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:53:59 smithi151 ceph-mon[112233]: pgmap v460: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:00.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:53:59 smithi153 ceph-mon[87858]: pgmap v460: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:01.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:01 smithi151 ceph-mon[112233]: pgmap v461: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:01 smithi153 ceph-mon[87858]: pgmap v461: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:03.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:03 smithi151 ceph-mon[112233]: pgmap v462: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:04.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:03 smithi153 ceph-mon[87858]: pgmap v462: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:05.774 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:05 smithi153 ceph-mon[87858]: pgmap v463: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:05.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:05 smithi151 ceph-mon[112233]: pgmap v463: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:07.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:07 smithi151 ceph-mon[112233]: pgmap v464: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:08.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:07 smithi153 ceph-mon[87858]: pgmap v464: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:09 smithi151 ceph-mon[112233]: pgmap v465: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:54:10.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:09 smithi153 ceph-mon[87858]: pgmap v465: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:10.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:54:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:11 smithi151 ceph-mon[112233]: pgmap v466: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:12.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:11 smithi153 ceph-mon[87858]: pgmap v466: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:13.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:13 smithi151 ceph-mon[112233]: pgmap v467: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:14.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:13 smithi153 ceph-mon[87858]: pgmap v467: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:15.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:15 smithi153 ceph-mon[87858]: pgmap v468: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:15.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:15 smithi151 ceph-mon[112233]: pgmap v468: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:17.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.749+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/4075873027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 msgr2=0x7fa1c81057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:17.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.749+0000 7fa1cdbd0700 1 --2- 172.21.15.151:0/4075873027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 0x7fa1c81057c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc009910 tx=0x7fa1bc009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:17.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.749+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/4075873027 shutdown_connections 2024-04-16T02:54:17.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.749+0000 7fa1cdbd0700 1 --2- 172.21.15.151:0/4075873027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 0x7fa1c81057c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:17.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.749+0000 7fa1cdbd0700 1 --2- 172.21.15.151:0/4075873027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1c8100dc0 0x7fa1c81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:17.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.749+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/4075873027 >> 172.21.15.151:0/4075873027 conn(0x7fa1c80fc650 msgr2=0x7fa1c80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:17.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.749+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/4075873027 shutdown_connections 2024-04-16T02:54:17.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.749+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/4075873027 wait complete. 2024-04-16T02:54:17.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.750+0000 7fa1cdbd0700 1 Processor -- start 2024-04-16T02:54:17.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.750+0000 7fa1cdbd0700 1 -- start start 2024-04-16T02:54:17.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.750+0000 7fa1cdbd0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1c8100dc0 0x7fa1c8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:17.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.750+0000 7fa1cdbd0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 0x7fa1c80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:17.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.750+0000 7fa1cdbd0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c8073b10 con 0x7fa1c8101770 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.750+0000 7fa1cdbd0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c8073c50 con 0x7fa1c8100dc0 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.750+0000 7fa1c6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 0x7fa1c80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 0x7fa1c80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59060/0 (socket says 172.21.15.151:59060) 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c6ffd700 1 -- 172.21.15.151:0/3656510133 learned_addr learned my addr 172.21.15.151:0/3656510133 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c77fe700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1c8100dc0 0x7fa1c8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c6ffd700 1 -- 172.21.15.151:0/3656510133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1c8100dc0 msgr2=0x7fa1c8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c6ffd700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1c8100dc0 0x7fa1c8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c6ffd700 1 -- 172.21.15.151:0/3656510133 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc0095f0 con 0x7fa1c8101770 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c6ffd700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 0x7fa1c80735d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0092e0 tx=0x7fa1bc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c4ff9700 1 -- 172.21.15.151:0/3656510133 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1bc01d070 con 0x7fa1c8101770 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c8073ed0 con 0x7fa1c8101770 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c4ff9700 1 -- 172.21.15.151:0/3656510133 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1bc00f460 con 0x7fa1c8101770 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1c4ff9700 1 -- 172.21.15.151:0/3656510133 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1bc0175d0 con 0x7fa1c8101770 2024-04-16T02:54:17.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.751+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c81a2c60 con 0x7fa1c8101770 2024-04-16T02:54:17.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.752+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c81091c0 con 0x7fa1c8101770 2024-04-16T02:54:17.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.753+0000 7fa1c4ff9700 1 -- 172.21.15.151:0/3656510133 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa1bc00f5d0 con 0x7fa1c8101770 2024-04-16T02:54:17.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.753+0000 7fa1c4ff9700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa1b006ecc0 0x7fa1b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:17.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.753+0000 7fa1c4ff9700 1 -- 172.21.15.151:0/3656510133 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa1bc090390 con 0x7fa1c8101770 2024-04-16T02:54:17.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.753+0000 7fa1c77fe700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa1b006ecc0 0x7fa1b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:17.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.754+0000 7fa1c77fe700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa1b006ecc0 0x7fa1b0071180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8005950 tx=0x7fa1b800a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:17.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.758+0000 7fa1c4ff9700 1 -- 172.21.15.151:0/3656510133 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa1bc0590a0 con 0x7fa1c8101770 2024-04-16T02:54:17.987 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.984+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1c802cce0 con 0x7fa1b006ecc0 2024-04-16T02:54:17.987 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:17 smithi151 ceph-mon[112233]: pgmap v469: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:17.988 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.986+0000 7fa1c4ff9700 1 -- 172.21.15.151:0/3656510133 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa1c802cce0 con 0x7fa1b006ecc0 2024-04-16T02:54:17.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa1b006ecc0 msgr2=0x7fa1b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:17.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa1b006ecc0 0x7fa1b0071180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8005950 tx=0x7fa1b800a300 comp rx=0 tx=0).stop 2024-04-16T02:54:17.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 msgr2=0x7fa1c80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:17.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 0x7fa1c80735d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0092e0 tx=0x7fa1bc00bd70 comp rx=0 tx=0).stop 2024-04-16T02:54:17.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 shutdown_connections 2024-04-16T02:54:17.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa1b006ecc0 0x7fa1b0071180 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:17.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1c8100dc0 0x7fa1c8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:17.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 --2- 172.21.15.151:0/3656510133 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c8101770 0x7fa1c80735d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:17.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 >> 172.21.15.151:0/3656510133 conn(0x7fa1c80fc650 msgr2=0x7fa1c80fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:17.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 shutdown_connections 2024-04-16T02:54:17.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:17.988+0000 7fa1cdbd0700 1 -- 172.21.15.151:0/3656510133 wait complete. 2024-04-16T02:54:18.001 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:54:18.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:17 smithi153 ceph-mon[87858]: pgmap v469: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:18.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.095+0000 7fa9ec087700 1 -- 172.21.15.151:0/3461678021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 msgr2=0x7fa9e4111630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.095+0000 7fa9ec087700 1 --2- 172.21.15.151:0/3461678021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 0x7fa9e4111630 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009910 tx=0x7fa9d4009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:18.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.095+0000 7fa9ec087700 1 -- 172.21.15.151:0/3461678021 shutdown_connections 2024-04-16T02:54:18.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.095+0000 7fa9ec087700 1 --2- 172.21.15.151:0/3461678021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 0x7fa9e4111630 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.097 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.095+0000 7fa9ec087700 1 --2- 172.21.15.151:0/3461678021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e4073130 0x7fa9e4073510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.095+0000 7fa9ec087700 1 -- 172.21.15.151:0/3461678021 >> 172.21.15.151:0/3461678021 conn(0x7fa9e40fc650 msgr2=0x7fa9e40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:18.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.095+0000 7fa9ec087700 1 -- 172.21.15.151:0/3461678021 shutdown_connections 2024-04-16T02:54:18.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.095+0000 7fa9ec087700 1 -- 172.21.15.151:0/3461678021 wait complete. 2024-04-16T02:54:18.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.095+0000 7fa9ec087700 1 Processor -- start 2024-04-16T02:54:18.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9ec087700 1 -- start start 2024-04-16T02:54:18.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9ec087700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e4073130 0x7fa9e41a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:18.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9ec087700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 0x7fa9e41a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:18.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9ec087700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e41a3680 con 0x7fa9e4073a50 2024-04-16T02:54:18.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9ec087700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e419cb30 con 0x7fa9e4073130 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9e9622700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 0x7fa9e41a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9e9622700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 0x7fa9e41a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59068/0 (socket says 172.21.15.151:59068) 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9e9622700 1 -- 172.21.15.151:0/2854487765 learned_addr learned my addr 172.21.15.151:0/2854487765 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9e9e23700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e4073130 0x7fa9e41a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9e9622700 1 -- 172.21.15.151:0/2854487765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e4073130 msgr2=0x7fa9e41a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9e9622700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e4073130 0x7fa9e41a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9e9622700 1 -- 172.21.15.151:0/2854487765 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d40095f0 con 0x7fa9e4073a50 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9e9622700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 0x7fa9e41a2ff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4006010 tx=0x7fa9d400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.096+0000 7fa9daffd700 1 -- 172.21.15.151:0/2854487765 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9d401d070 con 0x7fa9e4073a50 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.097+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e419ce10 con 0x7fa9e4073a50 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.097+0000 7fa9daffd700 1 -- 172.21.15.151:0/2854487765 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9d4005dc0 con 0x7fa9e4073a50 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.097+0000 7fa9daffd700 1 -- 172.21.15.151:0/2854487765 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9d4017400 con 0x7fa9e4073a50 2024-04-16T02:54:18.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.097+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e419d360 con 0x7fa9e4073a50 2024-04-16T02:54:18.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.097+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e410edb0 con 0x7fa9e4073a50 2024-04-16T02:54:18.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.098+0000 7fa9daffd700 1 -- 172.21.15.151:0/2854487765 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa9d4003a60 con 0x7fa9e4073a50 2024-04-16T02:54:18.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.098+0000 7fa9daffd700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa9d006ecc0 0x7fa9d0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:18.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.099+0000 7fa9daffd700 1 -- 172.21.15.151:0/2854487765 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa9d40902a0 con 0x7fa9e4073a50 2024-04-16T02:54:18.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.099+0000 7fa9e9e23700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa9d006ecc0 0x7fa9d0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:18.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.099+0000 7fa9e9e23700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa9d006ecc0 0x7fa9d0071180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0005950 tx=0x7fa9e000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:18.105 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.103+0000 7fa9daffd700 1 -- 172.21.15.151:0/2854487765 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa9d405cb80 con 0x7fa9e4073a50 2024-04-16T02:54:18.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.320+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9e419d640 con 0x7fa9d006ecc0 2024-04-16T02:54:18.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.321+0000 7fa9daffd700 1 -- 172.21.15.151:0/2854487765 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa9e419d640 con 0x7fa9d006ecc0 2024-04-16T02:54:18.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa9d006ecc0 msgr2=0x7fa9d0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa9d006ecc0 0x7fa9d0071180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0005950 tx=0x7fa9e000b410 comp rx=0 tx=0).stop 2024-04-16T02:54:18.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 msgr2=0x7fa9e41a2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 0x7fa9e41a2ff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4006010 tx=0x7fa9d400bd70 comp rx=0 tx=0).stop 2024-04-16T02:54:18.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 shutdown_connections 2024-04-16T02:54:18.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa9d006ecc0 0x7fa9d0071180 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e4073130 0x7fa9e41a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 --2- 172.21.15.151:0/2854487765 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9e4073a50 0x7fa9e41a2ff0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 >> 172.21.15.151:0/2854487765 conn(0x7fa9e40fc650 msgr2=0x7fa9e4103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:18.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 shutdown_connections 2024-04-16T02:54:18.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.324+0000 7fa9ec087700 1 -- 172.21.15.151:0/2854487765 wait complete. 2024-04-16T02:54:18.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.428+0000 7f44433eb700 1 -- 172.21.15.151:0/1217008756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 msgr2=0x7f443c103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.428+0000 7f44433eb700 1 --2- 172.21.15.151:0/1217008756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 0x7f443c103eb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f442c009910 tx=0x7f442c009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:18.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.428+0000 7f44433eb700 1 -- 172.21.15.151:0/1217008756 shutdown_connections 2024-04-16T02:54:18.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.428+0000 7f44433eb700 1 --2- 172.21.15.151:0/1217008756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 0x7f443c103eb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.428+0000 7f44433eb700 1 --2- 172.21.15.151:0/1217008756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f443c0ff530 0x7f443c0ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.428+0000 7f44433eb700 1 -- 172.21.15.151:0/1217008756 >> 172.21.15.151:0/1217008756 conn(0x7f443c074df0 msgr2=0x7f443c075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:18.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.429+0000 7f44433eb700 1 -- 172.21.15.151:0/1217008756 shutdown_connections 2024-04-16T02:54:18.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.429+0000 7f44433eb700 1 -- 172.21.15.151:0/1217008756 wait complete. 2024-04-16T02:54:18.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.429+0000 7f44433eb700 1 Processor -- start 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.429+0000 7f44433eb700 1 -- start start 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.429+0000 7f44433eb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f443c0ff530 0x7f443c19c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.429+0000 7f44433eb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 0x7f443c19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.429+0000 7f44433eb700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c19d230 con 0x7f443c0ffee0 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.429+0000 7f44433eb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1966b0 con 0x7f443c0ff530 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f4440986700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 0x7f443c19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f4440986700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 0x7f443c19cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59088/0 (socket says 172.21.15.151:59088) 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f4441187700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f443c0ff530 0x7f443c19c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f4440986700 1 -- 172.21.15.151:0/2939233788 learned_addr learned my addr 172.21.15.151:0/2939233788 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f4440986700 1 -- 172.21.15.151:0/2939233788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f443c0ff530 msgr2=0x7f443c19c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f4440986700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f443c0ff530 0x7f443c19c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f4440986700 1 -- 172.21.15.151:0/2939233788 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c0095f0 con 0x7f443c0ffee0 2024-04-16T02:54:18.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f4440986700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 0x7f443c19cb50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f442c00bda0 tx=0x7f442c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f44327fc700 1 -- 172.21.15.151:0/2939233788 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f442c01d070 con 0x7f443c0ffee0 2024-04-16T02:54:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c196930 con 0x7f443c0ffee0 2024-04-16T02:54:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f44327fc700 1 -- 172.21.15.151:0/2939233788 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f442c005dc0 con 0x7f443c0ffee0 2024-04-16T02:54:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f44327fc700 1 -- 172.21.15.151:0/2939233788 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f442c017400 con 0x7f443c0ffee0 2024-04-16T02:54:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.430+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c196e20 con 0x7f443c0ffee0 2024-04-16T02:54:18.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.431+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c04ea90 con 0x7f443c0ffee0 2024-04-16T02:54:18.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.432+0000 7f44327fc700 1 -- 172.21.15.151:0/2939233788 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f442c00f460 con 0x7f443c0ffee0 2024-04-16T02:54:18.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.432+0000 7f44327fc700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f442806ecc0 0x7f4428071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:18.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.432+0000 7f44327fc700 1 -- 172.21.15.151:0/2939233788 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f442c090200 con 0x7f443c0ffee0 2024-04-16T02:54:18.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.432+0000 7f4441187700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f442806ecc0 0x7f4428071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:18.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.432+0000 7f4441187700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f442806ecc0 0x7f4428071180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4438005950 tx=0x7f443800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:18.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.437+0000 7f44327fc700 1 -- 172.21.15.151:0/2939233788 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f442c05cae0 con 0x7f443c0ffee0 2024-04-16T02:54:18.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.646+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f443c197210 con 0x7f442806ecc0 2024-04-16T02:54:18.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.657+0000 7f44327fc700 1 -- 172.21.15.151:0/2939233788 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f443c197210 con 0x7f442806ecc0 2024-04-16T02:54:18.660 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:54:18.660 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (15m) 9m ago 27m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:54:18.660 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (27m) 9m ago 27m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:54:18.660 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (26m) 9m ago 26m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:54:18.660 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (13m) 9m ago 27m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (13m) 9m ago 26m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (15m) 9m ago 26m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (9m) 9m ago 21m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (9m) 9m ago 21m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (21m) 9m ago 21m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (9m) 9m ago 21m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (17m) 9m ago 28m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (16m) 9m ago 26m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (14m) 9m ago 28m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (14m) 9m ago 26m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (16m) 9m ago 27m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (16m) 9m ago 26m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (12m) 9m ago 25m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (12m) 9m ago 24m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (11m) 9m ago 24m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (11m) 9m ago 23m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:54:18.661 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (10m) 9m ago 23m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:54:18.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (10m) 9m ago 23m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:54:18.662 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (16m) 9m ago 26m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f442806ecc0 msgr2=0x7f4428071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f442806ecc0 0x7f4428071180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4438005950 tx=0x7f443800b410 comp rx=0 tx=0).stop 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 msgr2=0x7f443c19cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 0x7f443c19cb50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f442c00bda0 tx=0x7f442c00bdd0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 shutdown_connections 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f442806ecc0 0x7f4428071180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f443c0ff530 0x7f443c19c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 --2- 172.21.15.151:0/2939233788 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f443c0ffee0 0x7f443c19cb50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 >> 172.21.15.151:0/2939233788 conn(0x7f443c074df0 msgr2=0x7f443c103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 shutdown_connections 2024-04-16T02:54:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.660+0000 7f44433eb700 1 -- 172.21.15.151:0/2939233788 wait complete. 2024-04-16T02:54:18.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.764+0000 7fb7b42d0700 1 -- 172.21.15.151:0/2514882863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac068670 msgr2=0x7fb7ac068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.764+0000 7fb7b42d0700 1 --2- 172.21.15.151:0/2514882863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac068670 0x7fb7ac068a50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009910 tx=0x7fb79c009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:18.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.764+0000 7fb7b42d0700 1 -- 172.21.15.151:0/2514882863 shutdown_connections 2024-04-16T02:54:18.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.764+0000 7fb7b42d0700 1 --2- 172.21.15.151:0/2514882863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac069020 0x7fb7ac105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.764+0000 7fb7b42d0700 1 --2- 172.21.15.151:0/2514882863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac068670 0x7fb7ac068a50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.764+0000 7fb7b42d0700 1 -- 172.21.15.151:0/2514882863 >> 172.21.15.151:0/2514882863 conn(0x7fb7ac0755a0 msgr2=0x7fb7ac0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:18.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.764+0000 7fb7b42d0700 1 -- 172.21.15.151:0/2514882863 shutdown_connections 2024-04-16T02:54:18.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.764+0000 7fb7b42d0700 1 -- 172.21.15.151:0/2514882863 wait complete. 2024-04-16T02:54:18.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b42d0700 1 Processor -- start 2024-04-16T02:54:18.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b42d0700 1 -- start start 2024-04-16T02:54:18.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b42d0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac068670 0x7fb7ac19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:18.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b42d0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac069020 0x7fb7ac19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:18.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b42d0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac19d0e0 con 0x7fb7ac069020 2024-04-16T02:54:18.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b42d0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac196590 con 0x7fb7ac068670 2024-04-16T02:54:18.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b186b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac069020 0x7fb7ac19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:18.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b186b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac069020 0x7fb7ac19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59102/0 (socket says 172.21.15.151:59102) 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b186b700 1 -- 172.21.15.151:0/3153949037 learned_addr learned my addr 172.21.15.151:0/3153949037 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.765+0000 7fb7b206c700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac068670 0x7fb7ac19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.766+0000 7fb7b186b700 1 -- 172.21.15.151:0/3153949037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac068670 msgr2=0x7fb7ac19c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.766+0000 7fb7b186b700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac068670 0x7fb7ac19c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.766+0000 7fb7b186b700 1 -- 172.21.15.151:0/3153949037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c0095f0 con 0x7fb7ac069020 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.766+0000 7fb7b186b700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac069020 0x7fb7ac19ca50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800d900 tx=0x7fb7a800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.766+0000 7fb7a37fe700 1 -- 172.21.15.151:0/3153949037 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7a8009900 con 0x7fb7ac069020 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.766+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ac196870 con 0x7fb7ac069020 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.766+0000 7fb7a37fe700 1 -- 172.21.15.151:0/3153949037 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7a8010460 con 0x7fb7ac069020 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.766+0000 7fb7a37fe700 1 -- 172.21.15.151:0/3153949037 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7a800f5f0 con 0x7fb7ac069020 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.766+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac196dc0 con 0x7fb7ac069020 2024-04-16T02:54:18.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.767+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac109130 con 0x7fb7ac069020 2024-04-16T02:54:18.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.767+0000 7fb7a37fe700 1 -- 172.21.15.151:0/3153949037 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb7a80105d0 con 0x7fb7ac069020 2024-04-16T02:54:18.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.768+0000 7fb7a37fe700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb79806ecc0 0x7fb798071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:18.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.768+0000 7fb7a37fe700 1 -- 172.21.15.151:0/3153949037 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb7a808ea30 con 0x7fb7ac069020 2024-04-16T02:54:18.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.768+0000 7fb7b206c700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb79806ecc0 0x7fb798071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:18.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.768+0000 7fb7b206c700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb79806ecc0 0x7fb798071180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb79c006010 tx=0x7fb79c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:18.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:18.773+0000 7fb7a37fe700 1 -- 172.21.15.151:0/3153949037 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb7a805b310 con 0x7fb7ac069020 2024-04-16T02:54:19.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.055+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb7ac04ea90 con 0x7fb7ac069020 2024-04-16T02:54:19.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.055+0000 7fb7a37fe700 1 -- 172.21.15.151:0/3153949037 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fb7a805aea0 con 0x7fb7ac069020 2024-04-16T02:54:19.058 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:54:19.058 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:54:19.058 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:54:19.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:54:19.058 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:54:19.058 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:54:19.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:54:19.059 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:54:19.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.057+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb79806ecc0 msgr2=0x7fb798071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.057+0000 7fb7b42d0700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb79806ecc0 0x7fb798071180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb79c006010 tx=0x7fb79c01a040 comp rx=0 tx=0).stop 2024-04-16T02:54:19.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.057+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac069020 msgr2=0x7fb7ac19ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.057+0000 7fb7b42d0700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac069020 0x7fb7ac19ca50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800d900 tx=0x7fb7a800dc10 comp rx=0 tx=0).stop 2024-04-16T02:54:19.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.058+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 shutdown_connections 2024-04-16T02:54:19.060 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.058+0000 7fb7b42d0700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb79806ecc0 0x7fb798071180 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.058+0000 7fb7b42d0700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac068670 0x7fb7ac19c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.058+0000 7fb7b42d0700 1 --2- 172.21.15.151:0/3153949037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7ac069020 0x7fb7ac19ca50 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.058+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 >> 172.21.15.151:0/3153949037 conn(0x7fb7ac0755a0 msgr2=0x7fb7ac0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:19.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.058+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 shutdown_connections 2024-04-16T02:54:19.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.058+0000 7fb7b42d0700 1 -- 172.21.15.151:0/3153949037 wait complete. 2024-04-16T02:54:19.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.162+0000 7ffbbb17b700 1 -- 172.21.15.151:0/423409553 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb40ffa40 msgr2=0x7ffbb40ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.162+0000 7ffbbb17b700 1 --2- 172.21.15.151:0/423409553 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb40ffa40 0x7ffbb40ffec0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009910 tx=0x7ffba0009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:19.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.162+0000 7ffbbb17b700 1 -- 172.21.15.151:0/423409553 shutdown_connections 2024-04-16T02:54:19.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.162+0000 7ffbbb17b700 1 --2- 172.21.15.151:0/423409553 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb40ffa40 0x7ffbb40ffec0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.162+0000 7ffbbb17b700 1 --2- 172.21.15.151:0/423409553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbb4105a40 0x7ffbb4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.162+0000 7ffbbb17b700 1 -- 172.21.15.151:0/423409553 >> 172.21.15.151:0/423409553 conn(0x7ffbb40fb570 msgr2=0x7ffbb40fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:19.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.162+0000 7ffbbb17b700 1 -- 172.21.15.151:0/423409553 shutdown_connections 2024-04-16T02:54:19.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.162+0000 7ffbbb17b700 1 -- 172.21.15.151:0/423409553 wait complete. 2024-04-16T02:54:19.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbbb17b700 1 Processor -- start 2024-04-16T02:54:19.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbbb17b700 1 -- start start 2024-04-16T02:54:19.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbbb17b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbb40ffa40 0x7ffbb41976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:19.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbbb17b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb4105a40 0x7ffbb4197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:19.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbbb17b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb4198200 con 0x7ffbb4105a40 2024-04-16T02:54:19.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbbb17b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb4198340 con 0x7ffbb40ffa40 2024-04-16T02:54:19.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbb3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb4105a40 0x7ffbb4197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:19.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbb3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb4105a40 0x7ffbb4197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59118/0 (socket says 172.21.15.151:59118) 2024-04-16T02:54:19.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbb3fff700 1 -- 172.21.15.151:0/2538100749 learned_addr learned my addr 172.21.15.151:0/2538100749 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.163+0000 7ffbb8f17700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbb40ffa40 0x7ffbb41976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.164+0000 7ffbb3fff700 1 -- 172.21.15.151:0/2538100749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbb40ffa40 msgr2=0x7ffbb41976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.164+0000 7ffbb3fff700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbb40ffa40 0x7ffbb41976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.164+0000 7ffbb3fff700 1 -- 172.21.15.151:0/2538100749 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba00095f0 con 0x7ffbb4105a40 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.164+0000 7ffbb3fff700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb4105a40 0x7ffbb4197be0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ffba0006010 tx=0x7ffba000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.164+0000 7ffbb1ffb700 1 -- 172.21.15.151:0/2538100749 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba001d070 con 0x7ffbb4105a40 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.164+0000 7ffbb1ffb700 1 -- 172.21.15.151:0/2538100749 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffba0004fb0 con 0x7ffbb4105a40 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.164+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb41a0710 con 0x7ffbb4105a40 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.164+0000 7ffbb1ffb700 1 -- 172.21.15.151:0/2538100749 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba0022ea0 con 0x7ffbb4105a40 2024-04-16T02:54:19.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.164+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb41a0b20 con 0x7ffbb4105a40 2024-04-16T02:54:19.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.165+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb4108060 con 0x7ffbb4105a40 2024-04-16T02:54:19.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.166+0000 7ffbb1ffb700 1 -- 172.21.15.151:0/2538100749 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ffba0022470 con 0x7ffbb4105a40 2024-04-16T02:54:19.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.166+0000 7ffbb1ffb700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffba406ecc0 0x7ffba4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:19.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.166+0000 7ffbb1ffb700 1 -- 172.21.15.151:0/2538100749 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffba00903f0 con 0x7ffbb4105a40 2024-04-16T02:54:19.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.166+0000 7ffbb8f17700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffba406ecc0 0x7ffba4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:19.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.167+0000 7ffbb8f17700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffba406ecc0 0x7ffba4071180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffba8006fd0 tx=0x7ffba8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:19.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.171+0000 7ffbb1ffb700 1 -- 172.21.15.151:0/2538100749 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ffba0059250 con 0x7ffbb4105a40 2024-04-16T02:54:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.410+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffbb404ea90 con 0x7ffbb4105a40 2024-04-16T02:54:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.411+0000 7ffbb1ffb700 1 -- 172.21.15.151:0/2538100749 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7ffba0027770 con 0x7ffbb4105a40 2024-04-16T02:54:19.413 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:54:19.414 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:54:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:54:19.416 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:54:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:54:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:54:19.416 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:54:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:54:19.416 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:54:19.416 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:54:19.416 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:54:19.416 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffba406ecc0 msgr2=0x7ffba4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffba406ecc0 0x7ffba4071180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffba8006fd0 tx=0x7ffba8008040 comp rx=0 tx=0).stop 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb4105a40 msgr2=0x7ffbb4197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb4105a40 0x7ffbb4197be0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ffba0006010 tx=0x7ffba000be20 comp rx=0 tx=0).stop 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 shutdown_connections 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffba406ecc0 0x7ffba4071180 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbb40ffa40 0x7ffbb41976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 --2- 172.21.15.151:0/2538100749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb4105a40 0x7ffbb4197be0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 >> 172.21.15.151:0/2538100749 conn(0x7ffbb40fb570 msgr2=0x7ffbb40fc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 shutdown_connections 2024-04-16T02:54:19.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.414+0000 7ffbbb17b700 1 -- 172.21.15.151:0/2538100749 wait complete. 2024-04-16T02:54:19.418 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:54:19.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.520+0000 7ff651327700 1 -- 172.21.15.151:0/715254103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 msgr2=0x7ff64c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.520+0000 7ff651327700 1 --2- 172.21.15.151:0/715254103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 0x7ff64c105e20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009910 tx=0x7ff63c009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:19.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.520+0000 7ff651327700 1 -- 172.21.15.151:0/715254103 shutdown_connections 2024-04-16T02:54:19.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.520+0000 7ff651327700 1 --2- 172.21.15.151:0/715254103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff64c0ffa40 0x7ff64c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.520+0000 7ff651327700 1 --2- 172.21.15.151:0/715254103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 0x7ff64c105e20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.520+0000 7ff651327700 1 -- 172.21.15.151:0/715254103 >> 172.21.15.151:0/715254103 conn(0x7ff64c0fb550 msgr2=0x7ff64c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:19.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.520+0000 7ff651327700 1 -- 172.21.15.151:0/715254103 shutdown_connections 2024-04-16T02:54:19.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.520+0000 7ff651327700 1 -- 172.21.15.151:0/715254103 wait complete. 2024-04-16T02:54:19.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.520+0000 7ff651327700 1 Processor -- start 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff651327700 1 -- start start 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff651327700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff64c0ffa40 0x7ff64c0fbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff651327700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 0x7ff64c0fc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff651327700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0fc940 con 0x7ff64c105a40 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff651327700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0fca80 con 0x7ff64c0ffa40 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff64a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 0x7ff64c0fc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff64a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 0x7ff64c0fc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59134/0 (socket says 172.21.15.151:59134) 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff64a7fc700 1 -- 172.21.15.151:0/891211188 learned_addr learned my addr 172.21.15.151:0/891211188 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff64affd700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff64c0ffa40 0x7ff64c0fbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff64a7fc700 1 -- 172.21.15.151:0/891211188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff64c0ffa40 msgr2=0x7ff64c0fbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff64a7fc700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff64c0ffa40 0x7ff64c0fbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff64a7fc700 1 -- 172.21.15.151:0/891211188 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c0095f0 con 0x7ff64c105a40 2024-04-16T02:54:19.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff64affd700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff64c0ffa40 0x7ff64c0fbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:54:19.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.521+0000 7ff64a7fc700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 0x7ff64c0fc400 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff64000b4b0 tx=0x7ff64000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:19.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.522+0000 7ff633fff700 1 -- 172.21.15.151:0/891211188 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff640011800 con 0x7ff64c105a40 2024-04-16T02:54:19.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.522+0000 7ff633fff700 1 -- 172.21.15.151:0/891211188 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff640011e20 con 0x7ff64c105a40 2024-04-16T02:54:19.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.522+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c1a29f0 con 0x7ff64c105a40 2024-04-16T02:54:19.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.522+0000 7ff633fff700 1 -- 172.21.15.151:0/891211188 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff64000f2e0 con 0x7ff64c105a40 2024-04-16T02:54:19.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.522+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c1a2e00 con 0x7ff64c105a40 2024-04-16T02:54:19.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.523+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c109050 con 0x7ff64c105a40 2024-04-16T02:54:19.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.523+0000 7ff633fff700 1 -- 172.21.15.151:0/891211188 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff640010380 con 0x7ff64c105a40 2024-04-16T02:54:19.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.524+0000 7ff633fff700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff63406ecc0 0x7ff634071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:19.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.524+0000 7ff633fff700 1 -- 172.21.15.151:0/891211188 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff64008ea00 con 0x7ff64c105a40 2024-04-16T02:54:19.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.524+0000 7ff64affd700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff63406ecc0 0x7ff634071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:19.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.524+0000 7ff64affd700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff63406ecc0 0x7ff634071180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff63c006010 tx=0x7ff63c00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:19.531 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.528+0000 7ff633fff700 1 -- 172.21.15.151:0/891211188 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff6400577b0 con 0x7ff64c105a40 2024-04-16T02:54:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:19 smithi151 ceph-mon[112233]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:19 smithi151 ceph-mon[112233]: pgmap v470: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:19 smithi151 ceph-mon[112233]: from='client.34906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:19 smithi151 ceph-mon[112233]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:19 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3153949037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:54:19.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:19 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2538100749' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:54:19.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.746+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff64c02cd80 con 0x7ff63406ecc0 2024-04-16T02:54:19.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.747+0000 7ff633fff700 1 -- 172.21.15.151:0/891211188 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff64c02cd80 con 0x7ff63406ecc0 2024-04-16T02:54:19.749 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:54:19.749 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:54:19.749 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:54:19.750 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff63406ecc0 msgr2=0x7ff634071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff63406ecc0 0x7ff634071180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff63c006010 tx=0x7ff63c00b540 comp rx=0 tx=0).stop 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 msgr2=0x7ff64c0fc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 0x7ff64c0fc400 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff64000b4b0 tx=0x7ff64000b870 comp rx=0 tx=0).stop 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 shutdown_connections 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff63406ecc0 0x7ff634071180 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff64c0ffa40 0x7ff64c0fbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 --2- 172.21.15.151:0/891211188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff64c105a40 0x7ff64c0fc400 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 >> 172.21.15.151:0/891211188 conn(0x7ff64c0fb550 msgr2=0x7ff64c069090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 shutdown_connections 2024-04-16T02:54:19.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.749+0000 7ff651327700 1 -- 172.21.15.151:0/891211188 wait complete. 2024-04-16T02:54:19.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.854+0000 7f23b4f78700 1 -- 172.21.15.151:0/890685636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 msgr2=0x7f23b0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.854+0000 7f23b4f78700 1 --2- 172.21.15.151:0/890685636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 0x7f23b0105e20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009910 tx=0x7f23a0009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:19.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.854+0000 7f23b4f78700 1 -- 172.21.15.151:0/890685636 shutdown_connections 2024-04-16T02:54:19.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.854+0000 7f23b4f78700 1 --2- 172.21.15.151:0/890685636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23b00ffbd0 0x7f23b0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.854+0000 7f23b4f78700 1 --2- 172.21.15.151:0/890685636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 0x7f23b0105e20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.854+0000 7f23b4f78700 1 -- 172.21.15.151:0/890685636 >> 172.21.15.151:0/890685636 conn(0x7f23b00755a0 msgr2=0x7f23b00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:19.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.854+0000 7f23b4f78700 1 -- 172.21.15.151:0/890685636 shutdown_connections 2024-04-16T02:54:19.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.854+0000 7f23b4f78700 1 -- 172.21.15.151:0/890685636 wait complete. 2024-04-16T02:54:19.858 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23b4f78700 1 Processor -- start 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23b4f78700 1 -- start start 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23b4f78700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23b00ffbd0 0x7f23b019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23b4f78700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 0x7f23b019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23b4f78700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b019d040 con 0x7f23b0105a40 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23b4f78700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0196500 con 0x7f23b00ffbd0 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23add9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 0x7f23b019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23add9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 0x7f23b019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59152/0 (socket says 172.21.15.151:59152) 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23add9b700 1 -- 172.21.15.151:0/924784106 learned_addr learned my addr 172.21.15.151:0/924784106 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.855+0000 7f23ae59c700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23b00ffbd0 0x7f23b019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.856+0000 7f23add9b700 1 -- 172.21.15.151:0/924784106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23b00ffbd0 msgr2=0x7f23b019c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.856+0000 7f23add9b700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23b00ffbd0 0x7f23b019c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.856+0000 7f23add9b700 1 -- 172.21.15.151:0/924784106 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a00095f0 con 0x7f23b0105a40 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.856+0000 7f23add9b700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 0x7f23b019c960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f23a400d900 tx=0x7f23a400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:19.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.856+0000 7f239f7fe700 1 -- 172.21.15.151:0/924784106 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23a4004a00 con 0x7f23b0105a40 2024-04-16T02:54:19.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.856+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b01967e0 con 0x7f23b0105a40 2024-04-16T02:54:19.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.856+0000 7f239f7fe700 1 -- 172.21.15.151:0/924784106 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23a40054e0 con 0x7f23b0105a40 2024-04-16T02:54:19.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.856+0000 7f239f7fe700 1 -- 172.21.15.151:0/924784106 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23a4009d00 con 0x7f23b0105a40 2024-04-16T02:54:19.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.856+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b0196d30 con 0x7f23b0105a40 2024-04-16T02:54:19.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.857+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b0109050 con 0x7f23b0105a40 2024-04-16T02:54:19.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.858+0000 7f239f7fe700 1 -- 172.21.15.151:0/924784106 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f23a4005020 con 0x7f23b0105a40 2024-04-16T02:54:19.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.858+0000 7f239f7fe700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f239806ecc0 0x7f2398071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:19.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.858+0000 7f23ae59c700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f239806ecc0 0x7f2398071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:19.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.858+0000 7f239f7fe700 1 -- 172.21.15.151:0/924784106 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f23a408ede0 con 0x7f23b0105a40 2024-04-16T02:54:19.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.859+0000 7f23ae59c700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f239806ecc0 0x7f2398071180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f23a0006010 tx=0x7f23a001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:19.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:19.863+0000 7f239f7fe700 1 -- 172.21.15.151:0/924784106 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f23a405b6c0 con 0x7f23b0105a40 2024-04-16T02:54:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:19 smithi153 ceph-mon[87858]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:19 smithi153 ceph-mon[87858]: pgmap v470: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:19 smithi153 ceph-mon[87858]: from='client.34906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:19 smithi153 ceph-mon[87858]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:19 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3153949037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:54:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:19 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2538100749' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:54:20.150 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.147+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f23b004ea90 con 0x7f23b0105a40 2024-04-16T02:54:20.150 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.147+0000 7f239f7fe700 1 -- 172.21.15.151:0/924784106 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f23a4009e60 con 0x7f23b0105a40 2024-04-16T02:54:20.150 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:54:20.150 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:54:20.150 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:54:20.150 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:54:20.150 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:54:20.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.149+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f239806ecc0 msgr2=0x7f2398071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:20.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.149+0000 7f23b4f78700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f239806ecc0 0x7f2398071180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f23a0006010 tx=0x7f23a001a040 comp rx=0 tx=0).stop 2024-04-16T02:54:20.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.150+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 msgr2=0x7f23b019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:20.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.150+0000 7f23b4f78700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 0x7f23b019c960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f23a400d900 tx=0x7f23a400dc10 comp rx=0 tx=0).stop 2024-04-16T02:54:20.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.150+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 shutdown_connections 2024-04-16T02:54:20.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.150+0000 7f23b4f78700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f239806ecc0 0x7f2398071180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:20.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.150+0000 7f23b4f78700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23b00ffbd0 0x7f23b019c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:20.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.150+0000 7f23b4f78700 1 --2- 172.21.15.151:0/924784106 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23b0105a40 0x7f23b019c960 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:20.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.150+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 >> 172.21.15.151:0/924784106 conn(0x7f23b00755a0 msgr2=0x7f23b00685b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:20.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.150+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 shutdown_connections 2024-04-16T02:54:20.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:20.150+0000 7f23b4f78700 1 -- 172.21.15.151:0/924784106 wait complete. 2024-04-16T02:54:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:20 smithi151 ceph-mon[112233]: from='client.34922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:20.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:20 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/924784106' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:54:21.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:20 smithi153 ceph-mon[87858]: from='client.34922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:21.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:20 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/924784106' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:54:21.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:21 smithi151 ceph-mon[112233]: pgmap v471: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:22.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:21 smithi153 ceph-mon[87858]: pgmap v471: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:23.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:23 smithi151 ceph-mon[112233]: pgmap v472: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:24.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:23 smithi153 ceph-mon[87858]: pgmap v472: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:54:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:54:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:25 smithi151 ceph-mon[112233]: pgmap v473: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:25 smithi153 ceph-mon[87858]: pgmap v473: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:27.907 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:27 smithi151 ceph-mon[112233]: pgmap v474: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:27 smithi153 ceph-mon[87858]: pgmap v474: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:29.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:29 smithi151 ceph-mon[112233]: pgmap v475: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:29 smithi153 ceph-mon[87858]: pgmap v475: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:31.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:31 smithi151 ceph-mon[112233]: pgmap v476: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:31 smithi153 ceph-mon[87858]: pgmap v476: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:33.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:33 smithi151 ceph-mon[112233]: pgmap v477: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:34.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:33 smithi153 ceph-mon[87858]: pgmap v477: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:35.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:35 smithi151 ceph-mon[112233]: pgmap v478: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:35 smithi153 ceph-mon[87858]: pgmap v478: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:37.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:37 smithi151 ceph-mon[112233]: pgmap v479: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:38.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:37 smithi153 ceph-mon[87858]: pgmap v479: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:39.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:39 smithi151 ceph-mon[112233]: pgmap v480: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:39.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:54:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:39 smithi153 ceph-mon[87858]: pgmap v480: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:54:41.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:41 smithi151 ceph-mon[112233]: pgmap v481: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:42.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:41 smithi153 ceph-mon[87858]: pgmap v481: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:43.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:43 smithi151 ceph-mon[112233]: pgmap v482: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:44.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:43 smithi153 ceph-mon[87858]: pgmap v482: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:45.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:45 smithi151 ceph-mon[112233]: pgmap v483: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:45 smithi153 ceph-mon[87858]: pgmap v483: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:47.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:47 smithi151 ceph-mon[112233]: pgmap v484: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:48.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:47 smithi153 ceph-mon[87858]: pgmap v484: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:49.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:49 smithi151 ceph-mon[112233]: pgmap v485: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:49 smithi153 ceph-mon[87858]: pgmap v485: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:50.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.269+0000 7f910ce57700 1 -- 172.21.15.151:0/3655272727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ff510 msgr2=0x7f91080ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.269+0000 7f910ce57700 1 --2- 172.21.15.151:0/3655272727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ff510 0x7f91080ff8f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009910 tx=0x7f90f8009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.269+0000 7f910ce57700 1 -- 172.21.15.151:0/3655272727 shutdown_connections 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.269+0000 7f910ce57700 1 --2- 172.21.15.151:0/3655272727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91080ffec0 0x7f9108105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.269+0000 7f910ce57700 1 --2- 172.21.15.151:0/3655272727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ff510 0x7f91080ff8f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.269+0000 7f910ce57700 1 -- 172.21.15.151:0/3655272727 >> 172.21.15.151:0/3655272727 conn(0x7f9108074d80 msgr2=0x7f9108075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.269+0000 7f910ce57700 1 -- 172.21.15.151:0/3655272727 shutdown_connections 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.269+0000 7f910ce57700 1 -- 172.21.15.151:0/3655272727 wait complete. 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f910ce57700 1 Processor -- start 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f910ce57700 1 -- start start 2024-04-16T02:54:50.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f910ce57700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91080ff510 0x7f910819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f910ce57700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ffec0 0x7f910819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f910ce57700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910819d0d0 con 0x7f91080ffec0 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f910ce57700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9108196580 con 0x7f91080ff510 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f9105d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ffec0 0x7f910819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f9105d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ffec0 0x7f910819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45732/0 (socket says 172.21.15.151:45732) 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f9105d9b700 1 -- 172.21.15.151:0/1632791198 learned_addr learned my addr 172.21.15.151:0/1632791198 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f910659c700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91080ff510 0x7f910819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f9105d9b700 1 -- 172.21.15.151:0/1632791198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91080ff510 msgr2=0x7f910819c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f9105d9b700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91080ff510 0x7f910819c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.270+0000 7f9105d9b700 1 -- 172.21.15.151:0/1632791198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f80095f0 con 0x7f91080ffec0 2024-04-16T02:54:50.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.271+0000 7f9105d9b700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ffec0 0x7f910819ca40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00d8d0 tx=0x7f90fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:50.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.271+0000 7f90f77fe700 1 -- 172.21.15.151:0/1632791198 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90fc009900 con 0x7f91080ffec0 2024-04-16T02:54:50.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.271+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9108196860 con 0x7f91080ffec0 2024-04-16T02:54:50.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.271+0000 7f90f77fe700 1 -- 172.21.15.151:0/1632791198 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90fc010460 con 0x7f91080ffec0 2024-04-16T02:54:50.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.271+0000 7f90f77fe700 1 -- 172.21.15.151:0/1632791198 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90fc00f5f0 con 0x7f91080ffec0 2024-04-16T02:54:50.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.271+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9108196db0 con 0x7f91080ffec0 2024-04-16T02:54:50.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.272+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91081091a0 con 0x7f91080ffec0 2024-04-16T02:54:50.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.272+0000 7f90f77fe700 1 -- 172.21.15.151:0/1632791198 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f90fc0105d0 con 0x7f91080ffec0 2024-04-16T02:54:50.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.273+0000 7f90f77fe700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f90f006ecc0 0x7f90f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:50.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.273+0000 7f90f77fe700 1 -- 172.21.15.151:0/1632791198 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f90fc08eae0 con 0x7f91080ffec0 2024-04-16T02:54:50.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.273+0000 7f910659c700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f90f006ecc0 0x7f90f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:50.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.273+0000 7f910659c700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f90f006ecc0 0x7f90f0071180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f90f800b5c0 tx=0x7f90f801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:50.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.278+0000 7f90f77fe700 1 -- 172.21.15.151:0/1632791198 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f90fc05b3c0 con 0x7f91080ffec0 2024-04-16T02:54:50.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.498+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9108196fd0 con 0x7f90f006ecc0 2024-04-16T02:54:50.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.499+0000 7f90f77fe700 1 -- 172.21.15.151:0/1632791198 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9108196fd0 con 0x7f90f006ecc0 2024-04-16T02:54:50.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f90f006ecc0 msgr2=0x7f90f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f90f006ecc0 0x7f90f0071180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f90f800b5c0 tx=0x7f90f801a040 comp rx=0 tx=0).stop 2024-04-16T02:54:50.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ffec0 msgr2=0x7f910819ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ffec0 0x7f910819ca40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00d8d0 tx=0x7f90fc00dc90 comp rx=0 tx=0).stop 2024-04-16T02:54:50.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 shutdown_connections 2024-04-16T02:54:50.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f90f006ecc0 0x7f90f0071180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91080ff510 0x7f910819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 --2- 172.21.15.151:0/1632791198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91080ffec0 0x7f910819ca40 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 >> 172.21.15.151:0/1632791198 conn(0x7f9108074d80 msgr2=0x7f91080feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:50.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 shutdown_connections 2024-04-16T02:54:50.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.502+0000 7f910ce57700 1 -- 172.21.15.151:0/1632791198 wait complete. 2024-04-16T02:54:50.515 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:54:50.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.607+0000 7f6386f63700 1 -- 172.21.15.151:0/1257811141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6380105a40 msgr2=0x7f6380105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.607+0000 7f6386f63700 1 --2- 172.21.15.151:0/1257811141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6380105a40 0x7f6380105e20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6374009910 tx=0x7f6374009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:50.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.607+0000 7f6386f63700 1 -- 172.21.15.151:0/1257811141 shutdown_connections 2024-04-16T02:54:50.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.607+0000 7f6386f63700 1 --2- 172.21.15.151:0/1257811141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63800ffa40 0x7f63800ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.607+0000 7f6386f63700 1 --2- 172.21.15.151:0/1257811141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6380105a40 0x7f6380105e20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.607+0000 7f6386f63700 1 -- 172.21.15.151:0/1257811141 >> 172.21.15.151:0/1257811141 conn(0x7f63800fb570 msgr2=0x7f63800fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:50.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.607+0000 7f6386f63700 1 -- 172.21.15.151:0/1257811141 shutdown_connections 2024-04-16T02:54:50.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.607+0000 7f6386f63700 1 -- 172.21.15.151:0/1257811141 wait complete. 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.608+0000 7f6386f63700 1 Processor -- start 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.608+0000 7f6386f63700 1 -- start start 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.608+0000 7f6386f63700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63800ffa40 0x7f63800fbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.608+0000 7f6386f63700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6380105a40 0x7f63800fc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.608+0000 7f6386f63700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63800fc900 con 0x7f63800ffa40 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.608+0000 7f6386f63700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63800fca70 con 0x7f6380105a40 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.608+0000 7f6384cff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63800ffa40 0x7f63800fbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.608+0000 7f6384cff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63800ffa40 0x7f63800fbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45738/0 (socket says 172.21.15.151:45738) 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.608+0000 7f6384cff700 1 -- 172.21.15.151:0/4175989672 learned_addr learned my addr 172.21.15.151:0/4175989672 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f637ffff700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6380105a40 0x7f63800fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:50.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f6384cff700 1 -- 172.21.15.151:0/4175989672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6380105a40 msgr2=0x7f63800fc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f6384cff700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6380105a40 0x7f63800fc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f6384cff700 1 -- 172.21.15.151:0/4175989672 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63740095f0 con 0x7f63800ffa40 2024-04-16T02:54:50.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f637ffff700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6380105a40 0x7f63800fc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:54:50.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f6384cff700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63800ffa40 0x7f63800fbdf0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6374006010 tx=0x7f637400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:50.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f637dffb700 1 -- 172.21.15.151:0/4175989672 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f637401d070 con 0x7f63800ffa40 2024-04-16T02:54:50.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f637dffb700 1 -- 172.21.15.151:0/4175989672 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f637400f460 con 0x7f63800ffa40 2024-04-16T02:54:50.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63801a28e0 con 0x7f63800ffa40 2024-04-16T02:54:50.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f637dffb700 1 -- 172.21.15.151:0/4175989672 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6374017630 con 0x7f63800ffa40 2024-04-16T02:54:50.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.609+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63801a2c90 con 0x7f63800ffa40 2024-04-16T02:54:50.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.610+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6380109050 con 0x7f63800ffa40 2024-04-16T02:54:50.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.611+0000 7f637dffb700 1 -- 172.21.15.151:0/4175989672 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f637400f5d0 con 0x7f63800ffa40 2024-04-16T02:54:50.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.611+0000 7f637dffb700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f637006ecc0 0x7f6370071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:50.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.611+0000 7f637dffb700 1 -- 172.21.15.151:0/4175989672 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f63740902b0 con 0x7f63800ffa40 2024-04-16T02:54:50.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.612+0000 7f637ffff700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f637006ecc0 0x7f6370071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:50.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.612+0000 7f637ffff700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f637006ecc0 0x7f6370071180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f636c005fd0 tx=0x7f636c005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:50.618 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.616+0000 7f637dffb700 1 -- 172.21.15.151:0/4175989672 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f63740590e0 con 0x7f63800ffa40 2024-04-16T02:54:50.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.835+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f638002cd50 con 0x7f637006ecc0 2024-04-16T02:54:50.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.837+0000 7f637dffb700 1 -- 172.21.15.151:0/4175989672 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f638002cd50 con 0x7f637006ecc0 2024-04-16T02:54:50.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f637006ecc0 msgr2=0x7f6370071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f637006ecc0 0x7f6370071180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f636c005fd0 tx=0x7f636c005ee0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63800ffa40 msgr2=0x7f63800fbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63800ffa40 0x7f63800fbdf0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6374006010 tx=0x7f637400be20 comp rx=0 tx=0).stop 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 shutdown_connections 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f637006ecc0 0x7f6370071180 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63800ffa40 0x7f63800fbdf0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 --2- 172.21.15.151:0/4175989672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6380105a40 0x7f63800fc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 >> 172.21.15.151:0/4175989672 conn(0x7f63800fb570 msgr2=0x7f6380068fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 shutdown_connections 2024-04-16T02:54:50.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.839+0000 7f6386f63700 1 -- 172.21.15.151:0/4175989672 wait complete. 2024-04-16T02:54:50.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.945+0000 7fe258631700 1 -- 172.21.15.151:0/3420234217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 msgr2=0x7fe250105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.945+0000 7fe258631700 1 --2- 172.21.15.151:0/3420234217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 0x7fe250105850 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe244009910 tx=0x7fe244009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:50.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.945+0000 7fe258631700 1 -- 172.21.15.151:0/3420234217 shutdown_connections 2024-04-16T02:54:50.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.945+0000 7fe258631700 1 --2- 172.21.15.151:0/3420234217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 0x7fe250105850 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.945+0000 7fe258631700 1 --2- 172.21.15.151:0/3420234217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2500ff510 0x7fe2500ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.945+0000 7fe258631700 1 -- 172.21.15.151:0/3420234217 >> 172.21.15.151:0/3420234217 conn(0x7fe250074d80 msgr2=0x7fe250075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:50.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.945+0000 7fe258631700 1 -- 172.21.15.151:0/3420234217 shutdown_connections 2024-04-16T02:54:50.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.945+0000 7fe258631700 1 -- 172.21.15.151:0/3420234217 wait complete. 2024-04-16T02:54:50.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe258631700 1 Processor -- start 2024-04-16T02:54:50.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe258631700 1 -- start start 2024-04-16T02:54:50.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe258631700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2500ff510 0x7fe25019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe258631700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 0x7fe25019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe258631700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25019d0e0 con 0x7fe2500ffec0 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe258631700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250196590 con 0x7fe2500ff510 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe255bcc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 0x7fe25019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe255bcc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 0x7fe25019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45756/0 (socket says 172.21.15.151:45756) 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe255bcc700 1 -- 172.21.15.151:0/1065883973 learned_addr learned my addr 172.21.15.151:0/1065883973 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe2563cd700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2500ff510 0x7fe25019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe255bcc700 1 -- 172.21.15.151:0/1065883973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2500ff510 msgr2=0x7fe25019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe255bcc700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2500ff510 0x7fe25019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe255bcc700 1 -- 172.21.15.151:0/1065883973 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2440095f0 con 0x7fe2500ffec0 2024-04-16T02:54:50.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.946+0000 7fe2563cd700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2500ff510 0x7fe25019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:54:50.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.947+0000 7fe255bcc700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 0x7fe25019ca50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe24400bd70 tx=0x7fe24400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:50.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.947+0000 7fe24f7fe700 1 -- 172.21.15.151:0/1065883973 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe24401d070 con 0x7fe2500ffec0 2024-04-16T02:54:50.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.947+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe250196810 con 0x7fe2500ffec0 2024-04-16T02:54:50.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.947+0000 7fe24f7fe700 1 -- 172.21.15.151:0/1065883973 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe244005dc0 con 0x7fe2500ffec0 2024-04-16T02:54:50.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.947+0000 7fe24f7fe700 1 -- 172.21.15.151:0/1065883973 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe244017400 con 0x7fe2500ffec0 2024-04-16T02:54:50.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.947+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe250196d00 con 0x7fe2500ffec0 2024-04-16T02:54:50.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.948+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe250109180 con 0x7fe2500ffec0 2024-04-16T02:54:50.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.948+0000 7fe24f7fe700 1 -- 172.21.15.151:0/1065883973 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe24400f7b0 con 0x7fe2500ffec0 2024-04-16T02:54:50.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.949+0000 7fe24f7fe700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe23806ecc0 0x7fe238071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:50.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.949+0000 7fe24f7fe700 1 -- 172.21.15.151:0/1065883973 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe244090340 con 0x7fe2500ffec0 2024-04-16T02:54:50.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.949+0000 7fe2563cd700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe23806ecc0 0x7fe238071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:50.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.949+0000 7fe2563cd700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe23806ecc0 0x7fe238071180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe248005950 tx=0x7fe2480058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:50.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:50.954+0000 7fe24f7fe700 1 -- 172.21.15.151:0/1065883973 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe24405cc20 con 0x7fe2500ffec0 2024-04-16T02:54:51.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.173+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe25002ccc0 con 0x7fe23806ecc0 2024-04-16T02:54:51.186 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.184+0000 7fe24f7fe700 1 -- 172.21.15.151:0/1065883973 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe25002ccc0 con 0x7fe23806ecc0 2024-04-16T02:54:51.187 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:54:51.187 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (16m) 9m ago 27m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:54:51.187 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (28m) 9m ago 28m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:54:51.187 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (26m) 9m ago 26m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:54:51.187 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (13m) 9m ago 27m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:54:51.187 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (13m) 9m ago 26m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:54:51.187 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (16m) 9m ago 27m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:54:51.187 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (10m) 9m ago 22m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (9m) 9m ago 22m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (22m) 9m ago 22m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (9m) 9m ago 22m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (18m) 9m ago 29m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (17m) 9m ago 26m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (15m) 9m ago 29m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (14m) 9m ago 26m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (17m) 9m ago 27m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (17m) 9m ago 26m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (13m) 9m ago 25m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:54:51.188 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (12m) 9m ago 25m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:54:51.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (12m) 9m ago 24m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:54:51.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (11m) 9m ago 24m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:54:51.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (11m) 9m ago 24m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:54:51.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (10m) 9m ago 23m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:54:51.189 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (16m) 9m ago 27m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:54:51.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe23806ecc0 msgr2=0x7fe238071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe23806ecc0 0x7fe238071180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe248005950 tx=0x7fe2480058e0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 msgr2=0x7fe25019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 0x7fe25019ca50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe24400bd70 tx=0x7fe24400be50 comp rx=0 tx=0).stop 2024-04-16T02:54:51.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 shutdown_connections 2024-04-16T02:54:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe23806ecc0 0x7fe238071180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2500ff510 0x7fe25019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 --2- 172.21.15.151:0/1065883973 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2500ffec0 0x7fe25019ca50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 >> 172.21.15.151:0/1065883973 conn(0x7fe250074d80 msgr2=0x7fe2500feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 shutdown_connections 2024-04-16T02:54:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.186+0000 7fe258631700 1 -- 172.21.15.151:0/1065883973 wait complete. 2024-04-16T02:54:51.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.292+0000 7f2fce662700 1 -- 172.21.15.151:0/3571927664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 msgr2=0x7f2fc8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.293+0000 7f2fce662700 1 --2- 172.21.15.151:0/3571927664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 0x7f2fc8100050 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009910 tx=0x7f2fbc009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:51.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.293+0000 7f2fce662700 1 -- 172.21.15.151:0/3571927664 shutdown_connections 2024-04-16T02:54:51.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.293+0000 7f2fce662700 1 --2- 172.21.15.151:0/3571927664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 0x7f2fc8100050 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.293+0000 7f2fce662700 1 --2- 172.21.15.151:0/3571927664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fc8105a40 0x7f2fc8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.293+0000 7f2fce662700 1 -- 172.21.15.151:0/3571927664 >> 172.21.15.151:0/3571927664 conn(0x7f2fc80755a0 msgr2=0x7f2fc80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:51.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.293+0000 7f2fce662700 1 -- 172.21.15.151:0/3571927664 shutdown_connections 2024-04-16T02:54:51.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.293+0000 7f2fce662700 1 -- 172.21.15.151:0/3571927664 wait complete. 2024-04-16T02:54:51.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.293+0000 7f2fce662700 1 Processor -- start 2024-04-16T02:54:51.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.293+0000 7f2fce662700 1 -- start start 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fce662700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 0x7f2fc819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fce662700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fc8105a40 0x7f2fc819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fce662700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc819d0d0 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fce662700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc8196580 con 0x7f2fc8105a40 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fc7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 0x7f2fc819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fc7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 0x7f2fc819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45766/0 (socket says 172.21.15.151:45766) 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fc7fff700 1 -- 172.21.15.151:0/696616279 learned_addr learned my addr 172.21.15.151:0/696616279 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fc77fe700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fc8105a40 0x7f2fc819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fc7fff700 1 -- 172.21.15.151:0/696616279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fc8105a40 msgr2=0x7f2fc819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fc7fff700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fc8105a40 0x7f2fc819ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fc7fff700 1 -- 172.21.15.151:0/696616279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc0095f0 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.294+0000 7f2fc7fff700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 0x7f2fc819c500 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800d900 tx=0x7f2fb800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:51.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.295+0000 7f2fc57fa700 1 -- 172.21.15.151:0/696616279 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fb80041f0 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.295+0000 7f2fc57fa700 1 -- 172.21.15.151:0/696616279 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fb8004cf0 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.295+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc8196860 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.295+0000 7f2fc57fa700 1 -- 172.21.15.151:0/696616279 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fb800b750 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.295+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc8196db0 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.296+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc8108fe0 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.296+0000 7f2fc57fa700 1 -- 172.21.15.151:0/696616279 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2fb8004750 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.297+0000 7f2fc57fa700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fb006ecc0 0x7f2fb0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:51.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.297+0000 7f2fc57fa700 1 -- 172.21.15.151:0/696616279 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2fb808ed60 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.297+0000 7f2fc77fe700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fb006ecc0 0x7f2fb0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:51.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.297+0000 7f2fc77fe700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fb006ecc0 0x7f2fb0071180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8104d90 tx=0x7f2fbc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:51.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.301+0000 7f2fc57fa700 1 -- 172.21.15.151:0/696616279 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2fb805b640 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.580+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2fc8197020 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.581+0000 7f2fc57fa700 1 -- 172.21.15.151:0/696616279 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f2fb805b1d0 con 0x7f2fc80ffbd0 2024-04-16T02:54:51.583 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:54:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:54:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:54:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:54:51.585 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fb006ecc0 msgr2=0x7f2fb0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fb006ecc0 0x7f2fb0071180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8104d90 tx=0x7f2fbc01a040 comp rx=0 tx=0).stop 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 msgr2=0x7f2fc819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 0x7f2fc819c500 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800d900 tx=0x7f2fb800dc10 comp rx=0 tx=0).stop 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 shutdown_connections 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2fb006ecc0 0x7f2fb0071180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fc80ffbd0 0x7f2fc819c500 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 --2- 172.21.15.151:0/696616279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fc8105a40 0x7f2fc819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 >> 172.21.15.151:0/696616279 conn(0x7f2fc80755a0 msgr2=0x7f2fc8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.583+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 shutdown_connections 2024-04-16T02:54:51.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.584+0000 7f2fce662700 1 -- 172.21.15.151:0/696616279 wait complete. 2024-04-16T02:54:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.689+0000 7f44ff44b700 1 -- 172.21.15.151:0/4174702870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 msgr2=0x7f44f810ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.689+0000 7f44ff44b700 1 --2- 172.21.15.151:0/4174702870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 0x7f44f810ab70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009940 tx=0x7f44ec009c50 comp rx=0 tx=0).stop 2024-04-16T02:54:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.689+0000 7f44ff44b700 1 -- 172.21.15.151:0/4174702870 shutdown_connections 2024-04-16T02:54:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.689+0000 7f44ff44b700 1 --2- 172.21.15.151:0/4174702870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 0x7f44f810ab70 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.689+0000 7f44ff44b700 1 --2- 172.21.15.151:0/4174702870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44f8101d60 0x7f44f8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.689+0000 7f44ff44b700 1 -- 172.21.15.151:0/4174702870 >> 172.21.15.151:0/4174702870 conn(0x7f44f80fb570 msgr2=0x7f44f80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.689+0000 7f44ff44b700 1 -- 172.21.15.151:0/4174702870 shutdown_connections 2024-04-16T02:54:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.689+0000 7f44ff44b700 1 -- 172.21.15.151:0/4174702870 wait complete. 2024-04-16T02:54:51.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44ff44b700 1 Processor -- start 2024-04-16T02:54:51.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44ff44b700 1 -- start start 2024-04-16T02:54:51.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44ff44b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44f8101d60 0x7f44f819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44ff44b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 0x7f44f819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44ff44b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f819f2b0 con 0x7f44f8102680 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44ff44b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f8198760 con 0x7f44f8101d60 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44fc9e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 0x7f44f819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44fc9e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 0x7f44f819ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45796/0 (socket says 172.21.15.151:45796) 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44fc9e6700 1 -- 172.21.15.151:0/1762654751 learned_addr learned my addr 172.21.15.151:0/1762654751 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44fd1e7700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44f8101d60 0x7f44f819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44fc9e6700 1 -- 172.21.15.151:0/1762654751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44f8101d60 msgr2=0x7f44f819e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44fc9e6700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44f8101d60 0x7f44f819e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.690+0000 7f44fc9e6700 1 -- 172.21.15.151:0/1762654751 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec0095f0 con 0x7f44f8102680 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.691+0000 7f44fc9e6700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 0x7f44f819ec20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f44ec00bb40 tx=0x7f44ec00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.691+0000 7f44f67fc700 1 -- 172.21.15.151:0/1762654751 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44ec01d070 con 0x7f44f8102680 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.691+0000 7f44f67fc700 1 -- 172.21.15.151:0/1762654751 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44ec003d10 con 0x7f44f8102680 2024-04-16T02:54:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.691+0000 7f44f67fc700 1 -- 172.21.15.151:0/1762654751 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44ec021410 con 0x7f44f8102680 2024-04-16T02:54:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.691+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f81989e0 con 0x7f44f8102680 2024-04-16T02:54:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.691+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f8198ed0 con 0x7f44f8102680 2024-04-16T02:54:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.692+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f810a6d0 con 0x7f44f8102680 2024-04-16T02:54:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.693+0000 7f44f67fc700 1 -- 172.21.15.151:0/1762654751 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f44ec00f810 con 0x7f44f8102680 2024-04-16T02:54:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.693+0000 7f44f67fc700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f44e006ecc0 0x7f44e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.693+0000 7f44f67fc700 1 -- 172.21.15.151:0/1762654751 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f44ec08ffb0 con 0x7f44f8102680 2024-04-16T02:54:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.693+0000 7f44fd1e7700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f44e006ecc0 0x7f44e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:51.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.694+0000 7f44fd1e7700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f44e006ecc0 0x7f44e0071180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009300 tx=0x7f44f000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:51.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.698+0000 7f44f67fc700 1 -- 172.21.15.151:0/1762654751 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f44ec05c890 con 0x7f44f8102680 2024-04-16T02:54:51.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.942+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f44f804ea90 con 0x7f44f8102680 2024-04-16T02:54:51.945 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:51 smithi151 ceph-mon[112233]: pgmap v486: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:51.945 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:51 smithi151 ceph-mon[112233]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:51.945 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:51 smithi151 ceph-mon[112233]: from='client.34934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:51.945 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:51 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/696616279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:54:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.943+0000 7f44f67fc700 1 -- 172.21.15.151:0/1762654751 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f44ec05c420 con 0x7f44f8102680 2024-04-16T02:54:51.945 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:54:51.945 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:54:51.945 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:54:51.945 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:54:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:54:51.946 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:54:51.947 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:54:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:54:51.948 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:54:51.948 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:54:51.948 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:54:51.948 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:54:51.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f44e006ecc0 msgr2=0x7f44e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f44e006ecc0 0x7f44e0071180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009300 tx=0x7f44f000b040 comp rx=0 tx=0).stop 2024-04-16T02:54:51.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 msgr2=0x7f44f819ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 0x7f44f819ec20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f44ec00bb40 tx=0x7f44ec00bc20 comp rx=0 tx=0).stop 2024-04-16T02:54:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 shutdown_connections 2024-04-16T02:54:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f44e006ecc0 0x7f44e0071180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44f8101d60 0x7f44f819e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 --2- 172.21.15.151:0/1762654751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44f8102680 0x7f44f819ec20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 >> 172.21.15.151:0/1762654751 conn(0x7f44f80fb570 msgr2=0x7f44f8100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 shutdown_connections 2024-04-16T02:54:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:51.946+0000 7f44ff44b700 1 -- 172.21.15.151:0/1762654751 wait complete. 2024-04-16T02:54:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:54:52.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:51 smithi153 ceph-mon[87858]: pgmap v486: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:52.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:51 smithi153 ceph-mon[87858]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:52.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:51 smithi153 ceph-mon[87858]: from='client.34934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:52.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:51 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/696616279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.052+0000 7ff08034a700 1 -- 172.21.15.151:0/1988757222 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0780ffa60 msgr2=0x7ff0780ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.052+0000 7ff08034a700 1 --2- 172.21.15.151:0/1988757222 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0780ffa60 0x7ff0780ffee0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff068009910 tx=0x7ff068009c20 comp rx=0 tx=0).stop 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.052+0000 7ff08034a700 1 -- 172.21.15.151:0/1988757222 shutdown_connections 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.052+0000 7ff08034a700 1 --2- 172.21.15.151:0/1988757222 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0780ffa60 0x7ff0780ffee0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.052+0000 7ff08034a700 1 --2- 172.21.15.151:0/1988757222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff078105a60 0x7ff078105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.052+0000 7ff08034a700 1 -- 172.21.15.151:0/1988757222 >> 172.21.15.151:0/1988757222 conn(0x7ff0780fb550 msgr2=0x7ff0780fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.052+0000 7ff08034a700 1 -- 172.21.15.151:0/1988757222 shutdown_connections 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.052+0000 7ff08034a700 1 -- 172.21.15.151:0/1988757222 wait complete. 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff08034a700 1 Processor -- start 2024-04-16T02:54:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff08034a700 1 -- start start 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff08034a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0780ffa60 0x7ff07819c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff08034a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff078105a60 0x7ff07819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff08034a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07819d120 con 0x7ff078105a60 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff08034a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0781965d0 con 0x7ff0780ffa60 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff07d8e5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff078105a60 0x7ff07819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff07d8e5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff078105a60 0x7ff07819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45816/0 (socket says 172.21.15.151:45816) 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff07d8e5700 1 -- 172.21.15.151:0/398037562 learned_addr learned my addr 172.21.15.151:0/398037562 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.053+0000 7ff07e0e6700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0780ffa60 0x7ff07819c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.054+0000 7ff07d8e5700 1 -- 172.21.15.151:0/398037562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0780ffa60 msgr2=0x7ff07819c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.054+0000 7ff07d8e5700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0780ffa60 0x7ff07819c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.054+0000 7ff07d8e5700 1 -- 172.21.15.151:0/398037562 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0680095f0 con 0x7ff078105a60 2024-04-16T02:54:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.054+0000 7ff07d8e5700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff078105a60 0x7ff07819ca90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff068006010 tx=0x7ff06800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.054+0000 7ff06f7fe700 1 -- 172.21.15.151:0/398037562 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff06801d070 con 0x7ff078105a60 2024-04-16T02:54:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.054+0000 7ff06f7fe700 1 -- 172.21.15.151:0/398037562 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff06800f460 con 0x7ff078105a60 2024-04-16T02:54:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.054+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff078196850 con 0x7ff078105a60 2024-04-16T02:54:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.054+0000 7ff06f7fe700 1 -- 172.21.15.151:0/398037562 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff068017630 con 0x7ff078105a60 2024-04-16T02:54:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.054+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff078196d40 con 0x7ff078105a60 2024-04-16T02:54:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.055+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff078109000 con 0x7ff078105a60 2024-04-16T02:54:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.056+0000 7ff06f7fe700 1 -- 172.21.15.151:0/398037562 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff06800f5d0 con 0x7ff078105a60 2024-04-16T02:54:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.056+0000 7ff06f7fe700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff06406ecc0 0x7ff064071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.056+0000 7ff07e0e6700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff06406ecc0 0x7ff064071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.056+0000 7ff06f7fe700 1 -- 172.21.15.151:0/398037562 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff0680902b0 con 0x7ff078105a60 2024-04-16T02:54:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.056+0000 7ff07e0e6700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff06406ecc0 0x7ff064071180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff074005950 tx=0x7ff0740058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:52.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.061+0000 7ff06f7fe700 1 -- 172.21.15.151:0/398037562 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff06805cc10 con 0x7ff078105a60 2024-04-16T02:54:52.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.279+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff078197020 con 0x7ff06406ecc0 2024-04-16T02:54:52.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.281+0000 7ff06f7fe700 1 -- 172.21.15.151:0/398037562 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff078197020 con 0x7ff06406ecc0 2024-04-16T02:54:52.283 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:54:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:54:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:54:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:54:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:54:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:54:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:54:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:54:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:54:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:54:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:54:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:54:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:54:52.284 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:54:52.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff06406ecc0 msgr2=0x7ff064071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:52.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff06406ecc0 0x7ff064071180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff074005950 tx=0x7ff0740058e0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff078105a60 msgr2=0x7ff07819ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:52.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff078105a60 0x7ff07819ca90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff068006010 tx=0x7ff06800be20 comp rx=0 tx=0).stop 2024-04-16T02:54:52.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 shutdown_connections 2024-04-16T02:54:52.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff06406ecc0 0x7ff064071180 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0780ffa60 0x7ff07819c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 --2- 172.21.15.151:0/398037562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff078105a60 0x7ff07819ca90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 >> 172.21.15.151:0/398037562 conn(0x7ff0780fb550 msgr2=0x7ff078069050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:52.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 shutdown_connections 2024-04-16T02:54:52.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.283+0000 7ff08034a700 1 -- 172.21.15.151:0/398037562 wait complete. 2024-04-16T02:54:52.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.387+0000 7f1d572f6700 1 -- 172.21.15.151:0/1519969010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50102680 msgr2=0x7f1d5010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:52.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.387+0000 7f1d572f6700 1 --2- 172.21.15.151:0/1519969010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50102680 0x7f1d5010ab70 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009940 tx=0x7f1d40009c50 comp rx=0 tx=0).stop 2024-04-16T02:54:52.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 -- 172.21.15.151:0/1519969010 shutdown_connections 2024-04-16T02:54:52.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 --2- 172.21.15.151:0/1519969010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50102680 0x7f1d5010ab70 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 --2- 172.21.15.151:0/1519969010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50101d60 0x7f1d50102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 -- 172.21.15.151:0/1519969010 >> 172.21.15.151:0/1519969010 conn(0x7f1d500fb570 msgr2=0x7f1d500fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:52.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 -- 172.21.15.151:0/1519969010 shutdown_connections 2024-04-16T02:54:52.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 -- 172.21.15.151:0/1519969010 wait complete. 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 Processor -- start 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 -- start start 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50101d60 0x7f1d5019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.388+0000 7f1d572f6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102680 0x7f1d5019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d572f6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5019d100 con 0x7f1d50101d60 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d572f6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d501965b0 con 0x7f1d50102680 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d55092700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50101d60 0x7f1d5019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d55092700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50101d60 0x7f1d5019c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45832/0 (socket says 172.21.15.151:45832) 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d55092700 1 -- 172.21.15.151:0/2134349107 learned_addr learned my addr 172.21.15.151:0/2134349107 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d54891700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102680 0x7f1d5019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d55092700 1 -- 172.21.15.151:0/2134349107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102680 msgr2=0x7f1d5019ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d55092700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102680 0x7f1d5019ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d55092700 1 -- 172.21.15.151:0/2134349107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d400095f0 con 0x7f1d50101d60 2024-04-16T02:54:52.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d54891700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102680 0x7f1d5019ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:54:52.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d55092700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50101d60 0x7f1d5019c530 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00d8d0 tx=0x7f1d4c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:52.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d467fc700 1 -- 172.21.15.151:0/2134349107 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d4c009940 con 0x7f1d50101d60 2024-04-16T02:54:52.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d467fc700 1 -- 172.21.15.151:0/2134349107 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d4c010460 con 0x7f1d50101d60 2024-04-16T02:54:52.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d467fc700 1 -- 172.21.15.151:0/2134349107 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d4c00f5f0 con 0x7f1d50101d60 2024-04-16T02:54:52.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d50196890 con 0x7f1d50101d60 2024-04-16T02:54:52.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.389+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d50196db0 con 0x7f1d50101d60 2024-04-16T02:54:52.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.390+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d50108270 con 0x7f1d50101d60 2024-04-16T02:54:52.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.391+0000 7f1d467fc700 1 -- 172.21.15.151:0/2134349107 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1d4c0105d0 con 0x7f1d50101d60 2024-04-16T02:54:52.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.392+0000 7f1d467fc700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3c06ecc0 0x7f1d3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:54:52.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.392+0000 7f1d54891700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3c06ecc0 0x7f1d3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:54:52.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.392+0000 7f1d467fc700 1 -- 172.21.15.151:0/2134349107 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1d4c08ee80 con 0x7f1d50101d60 2024-04-16T02:54:52.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.392+0000 7f1d54891700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3c06ecc0 0x7f1d3c071180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009910 tx=0x7f1d40005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:54:52.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.396+0000 7f1d467fc700 1 -- 172.21.15.151:0/2134349107 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1d4c05b760 con 0x7f1d50101d60 2024-04-16T02:54:52.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.683+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1d50197000 con 0x7f1d50101d60 2024-04-16T02:54:52.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.683+0000 7f1d467fc700 1 -- 172.21.15.151:0/2134349107 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f1d4c05b2f0 con 0x7f1d50101d60 2024-04-16T02:54:52.686 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:54:52.686 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:54:52.686 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:54:52.686 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:54:52.686 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:54:52.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3c06ecc0 msgr2=0x7f1d3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:52.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3c06ecc0 0x7f1d3c071180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009910 tx=0x7f1d40005a90 comp rx=0 tx=0).stop 2024-04-16T02:54:52.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50101d60 msgr2=0x7f1d5019c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:54:52.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50101d60 0x7f1d5019c530 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00d8d0 tx=0x7f1d4c00dc90 comp rx=0 tx=0).stop 2024-04-16T02:54:52.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 shutdown_connections 2024-04-16T02:54:52.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3c06ecc0 0x7f1d3c071180 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d50101d60 0x7f1d5019c530 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 --2- 172.21.15.151:0/2134349107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102680 0x7f1d5019ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:54:52.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 >> 172.21.15.151:0/2134349107 conn(0x7f1d500fb570 msgr2=0x7f1d501053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:54:52.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 shutdown_connections 2024-04-16T02:54:52.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:54:52.686+0000 7f1d572f6700 1 -- 172.21.15.151:0/2134349107 wait complete. 2024-04-16T02:54:52.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:52 smithi151 ceph-mon[112233]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:52.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:52 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1762654751' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:54:53.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:52 smithi153 ceph-mon[87858]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:53.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:52 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1762654751' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:54:53.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:53 smithi151 ceph-mon[112233]: pgmap v487: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:53.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:53 smithi151 ceph-mon[112233]: from='client.34950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:53.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:53 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2134349107' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:54:54.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:53 smithi153 ceph-mon[87858]: pgmap v487: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:54.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:53 smithi153 ceph-mon[87858]: from='client.34950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:54:54.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:53 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2134349107' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:54:54.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:54:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:54:55.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:55 smithi151 ceph-mon[112233]: pgmap v488: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:56.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:55 smithi153 ceph-mon[87858]: pgmap v488: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:57 smithi151 ceph-mon[112233]: pgmap v489: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:58.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:57 smithi153 ceph-mon[87858]: pgmap v489: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:54:59.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:54:59 smithi151 ceph-mon[112233]: pgmap v490: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:00.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:54:59 smithi153 ceph-mon[87858]: pgmap v490: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:01.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:01 smithi151 ceph-mon[112233]: pgmap v491: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:01 smithi153 ceph-mon[87858]: pgmap v491: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:03.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:03 smithi151 ceph-mon[112233]: pgmap v492: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:04.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:03 smithi153 ceph-mon[87858]: pgmap v492: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:05.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:05 smithi151 ceph-mon[112233]: pgmap v493: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:06.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:05 smithi153 ceph-mon[87858]: pgmap v493: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:07.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:07 smithi151 ceph-mon[112233]: pgmap v494: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:08.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:07 smithi153 ceph-mon[87858]: pgmap v494: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:09 smithi151 ceph-mon[112233]: pgmap v495: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:55:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:09 smithi153 ceph-mon[87858]: pgmap v495: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:55:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:11 smithi151 ceph-mon[112233]: pgmap v496: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:12.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:11 smithi153 ceph-mon[87858]: pgmap v496: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:13.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:13 smithi151 ceph-mon[112233]: pgmap v497: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:14.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:13 smithi153 ceph-mon[87858]: pgmap v497: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:15.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:15 smithi151 ceph-mon[112233]: pgmap v498: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:15 smithi153 ceph-mon[87858]: pgmap v498: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:17.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:17 smithi151 ceph-mon[112233]: pgmap v499: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:18.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:17 smithi153 ceph-mon[87858]: pgmap v499: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:19.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:19 smithi151 ceph-mon[112233]: pgmap v500: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:20.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:19 smithi153 ceph-mon[87858]: pgmap v500: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:21.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:21 smithi151 ceph-mon[112233]: pgmap v501: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:22.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:21 smithi153 ceph-mon[87858]: pgmap v501: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:22.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.801+0000 7f70089f9700 1 -- 172.21.15.151:0/305896277 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7004105a40 msgr2=0x7f7004105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.801+0000 7f70089f9700 1 --2- 172.21.15.151:0/305896277 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7004105a40 0x7f7004105e20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009910 tx=0x7f6ff4009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.801+0000 7f70089f9700 1 -- 172.21.15.151:0/305896277 shutdown_connections 2024-04-16T02:55:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.801+0000 7f70089f9700 1 --2- 172.21.15.151:0/305896277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70040ffa40 0x7f70040ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.801+0000 7f70089f9700 1 --2- 172.21.15.151:0/305896277 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7004105a40 0x7f7004105e20 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.801+0000 7f70089f9700 1 -- 172.21.15.151:0/305896277 >> 172.21.15.151:0/305896277 conn(0x7f70040fb550 msgr2=0x7f70040fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.801+0000 7f70089f9700 1 -- 172.21.15.151:0/305896277 shutdown_connections 2024-04-16T02:55:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.801+0000 7f70089f9700 1 -- 172.21.15.151:0/305896277 wait complete. 2024-04-16T02:55:22.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f70089f9700 1 Processor -- start 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f70089f9700 1 -- start start 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f70089f9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70040ffa40 0x7f7004197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f70089f9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7004105a40 0x7f7004197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f70089f9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70041980d0 con 0x7f70040ffa40 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f70089f9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7004198240 con 0x7f7004105a40 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f700259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70040ffa40 0x7f7004197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f700259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70040ffa40 0x7f7004197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49596/0 (socket says 172.21.15.151:49596) 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f700259c700 1 -- 172.21.15.151:0/4237449858 learned_addr learned my addr 172.21.15.151:0/4237449858 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f7001d9b700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7004105a40 0x7f7004197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f700259c700 1 -- 172.21.15.151:0/4237449858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7004105a40 msgr2=0x7f7004197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f700259c700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7004105a40 0x7f7004197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:22.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.802+0000 7f700259c700 1 -- 172.21.15.151:0/4237449858 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40095f0 con 0x7f70040ffa40 2024-04-16T02:55:22.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.803+0000 7f700259c700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70040ffa40 0x7f7004197570 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4006010 tx=0x7f6ff400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:22.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.803+0000 7f6ff37fe700 1 -- 172.21.15.151:0/4237449858 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff401d070 con 0x7f70040ffa40 2024-04-16T02:55:22.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.803+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70041a05e0 con 0x7f70040ffa40 2024-04-16T02:55:22.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.803+0000 7f6ff37fe700 1 -- 172.21.15.151:0/4237449858 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ff400f460 con 0x7f70040ffa40 2024-04-16T02:55:22.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.803+0000 7f6ff37fe700 1 -- 172.21.15.151:0/4237449858 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff4017630 con 0x7f70040ffa40 2024-04-16T02:55:22.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.803+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70041a0a50 con 0x7f70040ffa40 2024-04-16T02:55:22.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.804+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7004108040 con 0x7f70040ffa40 2024-04-16T02:55:22.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.804+0000 7f6ff37fe700 1 -- 172.21.15.151:0/4237449858 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6ff4003a60 con 0x7f70040ffa40 2024-04-16T02:55:22.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.805+0000 7f6ff37fe700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6fec06ecc0 0x7f6fec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:22.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.805+0000 7f6ff37fe700 1 -- 172.21.15.151:0/4237449858 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6ff4090330 con 0x7f70040ffa40 2024-04-16T02:55:22.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.805+0000 7f7001d9b700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6fec06ecc0 0x7f6fec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:22.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.805+0000 7f7001d9b700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6fec06ecc0 0x7f6fec071180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7004069680 tx=0x7f6ff800b2c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:22.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:22.810+0000 7f6ff37fe700 1 -- 172.21.15.151:0/4237449858 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6ff40590e0 con 0x7f70040ffa40 2024-04-16T02:55:23.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.027+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f70040611d0 con 0x7f6fec06ecc0 2024-04-16T02:55:23.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.028+0000 7f6ff37fe700 1 -- 172.21.15.151:0/4237449858 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f70040611d0 con 0x7f6fec06ecc0 2024-04-16T02:55:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6fec06ecc0 msgr2=0x7f6fec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6fec06ecc0 0x7f6fec071180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7004069680 tx=0x7f6ff800b2c0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70040ffa40 msgr2=0x7f7004197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70040ffa40 0x7f7004197570 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4006010 tx=0x7f6ff400be20 comp rx=0 tx=0).stop 2024-04-16T02:55:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 shutdown_connections 2024-04-16T02:55:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6fec06ecc0 0x7f6fec071180 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70040ffa40 0x7f7004197570 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 --2- 172.21.15.151:0/4237449858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7004105a40 0x7f7004197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 >> 172.21.15.151:0/4237449858 conn(0x7f70040fb550 msgr2=0x7f70040fbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:23.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 shutdown_connections 2024-04-16T02:55:23.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.030+0000 7f70089f9700 1 -- 172.21.15.151:0/4237449858 wait complete. 2024-04-16T02:55:23.043 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:55:23.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.137+0000 7f49fcca3700 1 -- 172.21.15.151:0/2744195598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8101770 msgr2=0x7f49f81057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.137+0000 7f49fcca3700 1 --2- 172.21.15.151:0/2744195598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8101770 0x7f49f81057c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009910 tx=0x7f49e0009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:23.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.137+0000 7f49fcca3700 1 -- 172.21.15.151:0/2744195598 shutdown_connections 2024-04-16T02:55:23.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.137+0000 7f49fcca3700 1 --2- 172.21.15.151:0/2744195598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8101770 0x7f49f81057c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.137+0000 7f49fcca3700 1 --2- 172.21.15.151:0/2744195598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f8100dc0 0x7f49f81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.137+0000 7f49fcca3700 1 -- 172.21.15.151:0/2744195598 >> 172.21.15.151:0/2744195598 conn(0x7f49f80fc650 msgr2=0x7f49f80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:23.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.137+0000 7f49fcca3700 1 -- 172.21.15.151:0/2744195598 shutdown_connections 2024-04-16T02:55:23.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.137+0000 7f49fcca3700 1 -- 172.21.15.151:0/2744195598 wait complete. 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49fcca3700 1 Processor -- start 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49fcca3700 1 -- start start 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49fcca3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8100dc0 0x7f49f8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49fcca3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f8101770 0x7f49f80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49fcca3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f8073b10 con 0x7f49f8100dc0 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49fcca3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f8073c50 con 0x7f49f8101770 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49f659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8100dc0 0x7f49f8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49f659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8100dc0 0x7f49f8073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49610/0 (socket says 172.21.15.151:49610) 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49f659c700 1 -- 172.21.15.151:0/3562465032 learned_addr learned my addr 172.21.15.151:0/3562465032 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.138+0000 7f49edd9b700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f8101770 0x7f49f80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:23.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.139+0000 7f49f659c700 1 -- 172.21.15.151:0/3562465032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f8101770 msgr2=0x7f49f80735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.139+0000 7f49f659c700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f8101770 0x7f49f80735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.139+0000 7f49f659c700 1 -- 172.21.15.151:0/3562465032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e00095f0 con 0x7f49f8100dc0 2024-04-16T02:55:23.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.139+0000 7f49f659c700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8100dc0 0x7f49f8073090 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f49e800e920 tx=0x7f49e800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:23.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.139+0000 7f49effff700 1 -- 172.21.15.151:0/3562465032 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49e800c9d0 con 0x7f49f8100dc0 2024-04-16T02:55:23.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.139+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f81a2960 con 0x7f49f8100dc0 2024-04-16T02:55:23.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.139+0000 7f49effff700 1 -- 172.21.15.151:0/3562465032 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49e8010430 con 0x7f49f8100dc0 2024-04-16T02:55:23.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.139+0000 7f49effff700 1 -- 172.21.15.151:0/3562465032 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49e8018630 con 0x7f49f8100dc0 2024-04-16T02:55:23.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.139+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f81a2d10 con 0x7f49f8100dc0 2024-04-16T02:55:23.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.140+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f81091c0 con 0x7f49f8100dc0 2024-04-16T02:55:23.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.141+0000 7f49effff700 1 -- 172.21.15.151:0/3562465032 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f49e800cb30 con 0x7f49f8100dc0 2024-04-16T02:55:23.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.141+0000 7f49effff700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f49e406ecc0 0x7f49e4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:23.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.141+0000 7f49effff700 1 -- 172.21.15.151:0/3562465032 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f49e8014070 con 0x7f49f8100dc0 2024-04-16T02:55:23.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.141+0000 7f49edd9b700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f49e406ecc0 0x7f49e4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:23.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.142+0000 7f49edd9b700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f49e406ecc0 0x7f49e4071180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f49e0005b40 tx=0x7f49e0005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:23.148 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.146+0000 7f49effff700 1 -- 172.21.15.151:0/3562465032 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f49e8053ff0 con 0x7f49f8100dc0 2024-04-16T02:55:23.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.364+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49f802cce0 con 0x7f49e406ecc0 2024-04-16T02:55:23.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.365+0000 7f49effff700 1 -- 172.21.15.151:0/3562465032 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f49f802cce0 con 0x7f49e406ecc0 2024-04-16T02:55:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.367+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f49e406ecc0 msgr2=0x7f49e4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.367+0000 7f49fcca3700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f49e406ecc0 0x7f49e4071180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f49e0005b40 tx=0x7f49e0005a90 comp rx=0 tx=0).stop 2024-04-16T02:55:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.367+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8100dc0 msgr2=0x7f49f8073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.367+0000 7f49fcca3700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8100dc0 0x7f49f8073090 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f49e800e920 tx=0x7f49e800ec30 comp rx=0 tx=0).stop 2024-04-16T02:55:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.367+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 shutdown_connections 2024-04-16T02:55:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.367+0000 7f49fcca3700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f49e406ecc0 0x7f49e4071180 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.367+0000 7f49fcca3700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49f8100dc0 0x7f49f8073090 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.367+0000 7f49fcca3700 1 --2- 172.21.15.151:0/3562465032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f8101770 0x7f49f80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.367+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 >> 172.21.15.151:0/3562465032 conn(0x7f49f80fc650 msgr2=0x7f49f80fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:23.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.368+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 shutdown_connections 2024-04-16T02:55:23.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.368+0000 7f49fcca3700 1 -- 172.21.15.151:0/3562465032 wait complete. 2024-04-16T02:55:23.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.473+0000 7f274374f700 1 -- 172.21.15.151:0/3092925488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c100dc0 msgr2=0x7f273c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.473+0000 7f274374f700 1 --2- 172.21.15.151:0/3092925488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c100dc0 0x7f273c1011a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f272c009910 tx=0x7f272c009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:23.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.473+0000 7f274374f700 1 -- 172.21.15.151:0/3092925488 shutdown_connections 2024-04-16T02:55:23.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.473+0000 7f274374f700 1 --2- 172.21.15.151:0/3092925488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f273c101770 0x7f273c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.473+0000 7f274374f700 1 --2- 172.21.15.151:0/3092925488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c100dc0 0x7f273c1011a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.473+0000 7f274374f700 1 -- 172.21.15.151:0/3092925488 >> 172.21.15.151:0/3092925488 conn(0x7f273c0fc650 msgr2=0x7f273c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:23.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.473+0000 7f274374f700 1 -- 172.21.15.151:0/3092925488 shutdown_connections 2024-04-16T02:55:23.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.473+0000 7f274374f700 1 -- 172.21.15.151:0/3092925488 wait complete. 2024-04-16T02:55:23.477 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f274374f700 1 Processor -- start 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f274374f700 1 -- start start 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f274374f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f273c100dc0 0x7f273c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f274374f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c101770 0x7f273c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f274374f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c073b10 con 0x7f273c101770 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f274374f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c073c50 con 0x7f273c100dc0 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f2740cea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c101770 0x7f273c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f2740cea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c101770 0x7f273c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49636/0 (socket says 172.21.15.151:49636) 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f2740cea700 1 -- 172.21.15.151:0/1184102188 learned_addr learned my addr 172.21.15.151:0/1184102188 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.474+0000 7f27414eb700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f273c100dc0 0x7f273c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.475+0000 7f2740cea700 1 -- 172.21.15.151:0/1184102188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f273c100dc0 msgr2=0x7f273c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.475+0000 7f2740cea700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f273c100dc0 0x7f273c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.475+0000 7f2740cea700 1 -- 172.21.15.151:0/1184102188 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c0095f0 con 0x7f273c101770 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.475+0000 7f2740cea700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c101770 0x7f273c0735d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f273800b480 tx=0x7f273800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:23.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.475+0000 7f27327fc700 1 -- 172.21.15.151:0/1184102188 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2738011800 con 0x7f273c101770 2024-04-16T02:55:23.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.475+0000 7f27327fc700 1 -- 172.21.15.151:0/1184102188 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2738011e20 con 0x7f273c101770 2024-04-16T02:55:23.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.475+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273c1a2a40 con 0x7f273c101770 2024-04-16T02:55:23.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.475+0000 7f27327fc700 1 -- 172.21.15.151:0/1184102188 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2738010ea0 con 0x7f273c101770 2024-04-16T02:55:23.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.475+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273c1a2e50 con 0x7f273c101770 2024-04-16T02:55:23.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.476+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273c109160 con 0x7f273c101770 2024-04-16T02:55:23.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.477+0000 7f27327fc700 1 -- 172.21.15.151:0/1184102188 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2738010380 con 0x7f273c101770 2024-04-16T02:55:23.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.477+0000 7f27327fc700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f272806ecc0 0x7f2728071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:23.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.477+0000 7f27327fc700 1 -- 172.21.15.151:0/1184102188 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f273808e920 con 0x7f273c101770 2024-04-16T02:55:23.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.477+0000 7f27414eb700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f272806ecc0 0x7f2728071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:23.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.478+0000 7f27414eb700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f272806ecc0 0x7f2728071180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f272c006010 tx=0x7f272c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:23.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.482+0000 7f27327fc700 1 -- 172.21.15.151:0/1184102188 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f27380576d0 con 0x7f273c101770 2024-04-16T02:55:23.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.699+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f273c02cda0 con 0x7f272806ecc0 2024-04-16T02:55:23.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.710+0000 7f27327fc700 1 -- 172.21.15.151:0/1184102188 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f273c02cda0 con 0x7f272806ecc0 2024-04-16T02:55:23.712 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:55:23.712 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (16m) 10m ago 28m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (28m) 10m ago 28m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (27m) 10m ago 27m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (14m) 10m ago 28m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (14m) 10m ago 27m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (16m) 10m ago 27m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (10m) 10m ago 22m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (10m) 10m ago 22m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (22m) 10m ago 22m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (10m) 10m ago 22m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (18m) 10m ago 29m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (17m) 10m ago 27m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (15m) 10m ago 29m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (15m) 10m ago 27m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (17m) 10m ago 28m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (17m) 10m ago 27m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:55:23.713 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (14m) 10m ago 26m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:55:23.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (13m) 10m ago 25m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:55:23.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (12m) 10m ago 25m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:55:23.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (12m) 10m ago 25m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:55:23.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (11m) 10m ago 24m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:55:23.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (11m) 10m ago 24m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:55:23.714 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (17m) 10m ago 27m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f272806ecc0 msgr2=0x7f2728071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f272806ecc0 0x7f2728071180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f272c006010 tx=0x7f272c01a040 comp rx=0 tx=0).stop 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c101770 msgr2=0x7f273c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c101770 0x7f273c0735d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f273800b480 tx=0x7f273800b790 comp rx=0 tx=0).stop 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 shutdown_connections 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f272806ecc0 0x7f2728071180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f273c100dc0 0x7f273c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 --2- 172.21.15.151:0/1184102188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f273c101770 0x7f273c0735d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 >> 172.21.15.151:0/1184102188 conn(0x7f273c0fc650 msgr2=0x7f273c0fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 shutdown_connections 2024-04-16T02:55:23.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.712+0000 7f274374f700 1 -- 172.21.15.151:0/1184102188 wait complete. 2024-04-16T02:55:23.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:23 smithi151 ceph-mon[112233]: pgmap v502: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:23.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.818+0000 7f47c1c71700 1 -- 172.21.15.151:0/1833280310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 msgr2=0x7f47bc068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.818+0000 7f47c1c71700 1 --2- 172.21.15.151:0/1833280310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 0x7f47bc068a50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009910 tx=0x7f47ac009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:23.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.818+0000 7f47c1c71700 1 -- 172.21.15.151:0/1833280310 shutdown_connections 2024-04-16T02:55:23.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.818+0000 7f47c1c71700 1 --2- 172.21.15.151:0/1833280310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc069020 0x7f47bc105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.818+0000 7f47c1c71700 1 --2- 172.21.15.151:0/1833280310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 0x7f47bc068a50 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.818+0000 7f47c1c71700 1 -- 172.21.15.151:0/1833280310 >> 172.21.15.151:0/1833280310 conn(0x7f47bc0755a0 msgr2=0x7f47bc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:23.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.818+0000 7f47c1c71700 1 -- 172.21.15.151:0/1833280310 shutdown_connections 2024-04-16T02:55:23.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.818+0000 7f47c1c71700 1 -- 172.21.15.151:0/1833280310 wait complete. 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47c1c71700 1 Processor -- start 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47c1c71700 1 -- start start 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47c1c71700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 0x7f47bc19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47c1c71700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc069020 0x7f47bc19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47c1c71700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc19d290 con 0x7f47bc068670 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47c1c71700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc196740 con 0x7f47bc069020 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47bb7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 0x7f47bc19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47bb7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 0x7f47bc19c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49652/0 (socket says 172.21.15.151:49652) 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47bb7fe700 1 -- 172.21.15.151:0/1718308519 learned_addr learned my addr 172.21.15.151:0/1718308519 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47baffd700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc069020 0x7f47bc19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.819+0000 7f47bb7fe700 1 -- 172.21.15.151:0/1718308519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc069020 msgr2=0x7f47bc19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.820+0000 7f47bb7fe700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc069020 0x7f47bc19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.820+0000 7f47bb7fe700 1 -- 172.21.15.151:0/1718308519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac0095f0 con 0x7f47bc068670 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.820+0000 7f47bb7fe700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 0x7f47bc19c6c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f47ac006010 tx=0x7f47ac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.820+0000 7f47b8ff9700 1 -- 172.21.15.151:0/1718308519 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47ac01d070 con 0x7f47bc068670 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.820+0000 7f47b8ff9700 1 -- 172.21.15.151:0/1718308519 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47ac004fb0 con 0x7f47bc068670 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.820+0000 7f47b8ff9700 1 -- 172.21.15.151:0/1718308519 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47ac022ea0 con 0x7f47bc068670 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.820+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc1969c0 con 0x7f47bc068670 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.820+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc196eb0 con 0x7f47bc068670 2024-04-16T02:55:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.821+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc109130 con 0x7f47bc068670 2024-04-16T02:55:23.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.821+0000 7f47b8ff9700 1 -- 172.21.15.151:0/1718308519 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f47ac022470 con 0x7f47bc068670 2024-04-16T02:55:23.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.822+0000 7f47b8ff9700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47a806ecc0 0x7f47a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:23.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.822+0000 7f47baffd700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47a806ecc0 0x7f47a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:23.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.822+0000 7f47b8ff9700 1 -- 172.21.15.151:0/1718308519 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47ac090350 con 0x7f47bc068670 2024-04-16T02:55:23.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.822+0000 7f47baffd700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47a806ecc0 0x7f47a8071180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f47a4006fd0 tx=0x7f47a4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:23.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:23.827+0000 7f47b8ff9700 1 -- 172.21.15.151:0/1718308519 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f47ac05cd60 con 0x7f47bc068670 2024-04-16T02:55:24.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:23 smithi153 ceph-mon[87858]: pgmap v502: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:24.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.119+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f47bc197d20 con 0x7f47bc068670 2024-04-16T02:55:24.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.120+0000 7f47b8ff9700 1 -- 172.21.15.151:0/1718308519 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f47ac022720 con 0x7f47bc068670 2024-04-16T02:55:24.122 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:55:24.122 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:55:24.122 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:55:24.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:55:24.123 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:55:24.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.122+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47a806ecc0 msgr2=0x7f47a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.122+0000 7f47c1c71700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47a806ecc0 0x7f47a8071180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f47a4006fd0 tx=0x7f47a4008040 comp rx=0 tx=0).stop 2024-04-16T02:55:24.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.122+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 msgr2=0x7f47bc19c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.122+0000 7f47c1c71700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 0x7f47bc19c6c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f47ac006010 tx=0x7f47ac00be20 comp rx=0 tx=0).stop 2024-04-16T02:55:24.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.123+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 shutdown_connections 2024-04-16T02:55:24.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.123+0000 7f47c1c71700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47a806ecc0 0x7f47a8071180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.123+0000 7f47c1c71700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47bc068670 0x7f47bc19c6c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.123+0000 7f47c1c71700 1 --2- 172.21.15.151:0/1718308519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47bc069020 0x7f47bc19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.123+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 >> 172.21.15.151:0/1718308519 conn(0x7f47bc0755a0 msgr2=0x7f47bc0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:24.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.123+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 shutdown_connections 2024-04-16T02:55:24.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.123+0000 7f47c1c71700 1 -- 172.21.15.151:0/1718308519 wait complete. 2024-04-16T02:55:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.228+0000 7fa8047d0700 1 -- 172.21.15.151:0/736525269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc101d60 msgr2=0x7fa7fc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.228+0000 7fa8047d0700 1 --2- 172.21.15.151:0/736525269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc101d60 0x7fa7fc102140 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8009940 tx=0x7fa7f8009c50 comp rx=0 tx=0).stop 2024-04-16T02:55:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.228+0000 7fa8047d0700 1 -- 172.21.15.151:0/736525269 shutdown_connections 2024-04-16T02:55:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.228+0000 7fa8047d0700 1 --2- 172.21.15.151:0/736525269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc102680 0x7fa7fc10ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.228+0000 7fa8047d0700 1 --2- 172.21.15.151:0/736525269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc101d60 0x7fa7fc102140 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.228+0000 7fa8047d0700 1 -- 172.21.15.151:0/736525269 >> 172.21.15.151:0/736525269 conn(0x7fa7fc0fb530 msgr2=0x7fa7fc0fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.228+0000 7fa8047d0700 1 -- 172.21.15.151:0/736525269 shutdown_connections 2024-04-16T02:55:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.228+0000 7fa8047d0700 1 -- 172.21.15.151:0/736525269 wait complete. 2024-04-16T02:55:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa8047d0700 1 Processor -- start 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa8047d0700 1 -- start start 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa8047d0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc101d60 0x7fa7fc19c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa8047d0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc102680 0x7fa7fc19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa8047d0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc19d0f0 con 0x7fa7fc102680 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa8047d0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc1965a0 con 0x7fa7fc101d60 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa801d6b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc102680 0x7fa7fc19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa801d6b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc102680 0x7fa7fc19ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49680/0 (socket says 172.21.15.151:49680) 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa801d6b700 1 -- 172.21.15.151:0/2978732951 learned_addr learned my addr 172.21.15.151:0/2978732951 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa80256c700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc101d60 0x7fa7fc19c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa801d6b700 1 -- 172.21.15.151:0/2978732951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc101d60 msgr2=0x7fa7fc19c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa801d6b700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc101d60 0x7fa7fc19c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa801d6b700 1 -- 172.21.15.151:0/2978732951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f80095f0 con 0x7fa7fc102680 2024-04-16T02:55:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.229+0000 7fa801d6b700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc102680 0x7fa7fc19ca60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00d8d0 tx=0x7fa7ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:24.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.230+0000 7fa7f37fe700 1 -- 172.21.15.151:0/2978732951 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7ec00b920 con 0x7fa7fc102680 2024-04-16T02:55:24.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.230+0000 7fa7f37fe700 1 -- 172.21.15.151:0/2978732951 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7ec007620 con 0x7fa7fc102680 2024-04-16T02:55:24.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.230+0000 7fa7f37fe700 1 -- 172.21.15.151:0/2978732951 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7ec00f5f0 con 0x7fa7fc102680 2024-04-16T02:55:24.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.230+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc196880 con 0x7fa7fc102680 2024-04-16T02:55:24.234 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.230+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc196dd0 con 0x7fa7fc102680 2024-04-16T02:55:24.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.231+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc108270 con 0x7fa7fc102680 2024-04-16T02:55:24.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.231+0000 7fa7f37fe700 1 -- 172.21.15.151:0/2978732951 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa7ec010460 con 0x7fa7fc102680 2024-04-16T02:55:24.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.231+0000 7fa7f37fe700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa7e806ecc0 0x7fa7e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:24.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.232+0000 7fa7f37fe700 1 -- 172.21.15.151:0/2978732951 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa7ec08eda0 con 0x7fa7fc102680 2024-04-16T02:55:24.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.232+0000 7fa80256c700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa7e806ecc0 0x7fa7e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:24.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.232+0000 7fa80256c700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa7e806ecc0 0x7fa7e8071180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8005b40 tx=0x7fa7f8005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:24.238 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.236+0000 7fa7f37fe700 1 -- 172.21.15.151:0/2978732951 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa7ec05b680 con 0x7fa7fc102680 2024-04-16T02:55:24.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.478+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa7fc04ea90 con 0x7fa7fc102680 2024-04-16T02:55:24.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.479+0000 7fa7f37fe700 1 -- 172.21.15.151:0/2978732951 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7fa7ec05b210 con 0x7fa7fc102680 2024-04-16T02:55:24.481 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:55:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:55:24.483 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:55:24.484 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa7e806ecc0 msgr2=0x7fa7e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa7e806ecc0 0x7fa7e8071180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8005b40 tx=0x7fa7f8005a90 comp rx=0 tx=0).stop 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc102680 msgr2=0x7fa7fc19ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc102680 0x7fa7fc19ca60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00d8d0 tx=0x7fa7ec00dc90 comp rx=0 tx=0).stop 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 shutdown_connections 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa7e806ecc0 0x7fa7e8071180 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc101d60 0x7fa7fc19c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 --2- 172.21.15.151:0/2978732951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7fc102680 0x7fa7fc19ca60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 >> 172.21.15.151:0/2978732951 conn(0x7fa7fc0fb530 msgr2=0x7fa7fc1053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 shutdown_connections 2024-04-16T02:55:24.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.482+0000 7fa8047d0700 1 -- 172.21.15.151:0/2978732951 wait complete. 2024-04-16T02:55:24.486 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:55:24.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 -- 172.21.15.151:0/3444658890 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ffec0 msgr2=0x7fd710105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 --2- 172.21.15.151:0/3444658890 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ffec0 0x7fd710105850 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd700009910 tx=0x7fd700009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 -- 172.21.15.151:0/3444658890 shutdown_connections 2024-04-16T02:55:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 --2- 172.21.15.151:0/3444658890 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ffec0 0x7fd710105850 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 --2- 172.21.15.151:0/3444658890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7100ff510 0x7fd7100ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 -- 172.21.15.151:0/3444658890 >> 172.21.15.151:0/3444658890 conn(0x7fd710074d80 msgr2=0x7fd710075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 -- 172.21.15.151:0/3444658890 shutdown_connections 2024-04-16T02:55:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 -- 172.21.15.151:0/3444658890 wait complete. 2024-04-16T02:55:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 Processor -- start 2024-04-16T02:55:24.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.589+0000 7fd717d86700 1 -- start start 2024-04-16T02:55:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd717d86700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ff510 0x7fd710100490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd717d86700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7100ffec0 0x7fd7101009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd717d86700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd710100f10 con 0x7fd7100ff510 2024-04-16T02:55:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd717d86700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd710101050 con 0x7fd7100ffec0 2024-04-16T02:55:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd715b22700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ff510 0x7fd710100490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd715b22700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ff510 0x7fd710100490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49698/0 (socket says 172.21.15.151:49698) 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd715b22700 1 -- 172.21.15.151:0/2402555270 learned_addr learned my addr 172.21.15.151:0/2402555270 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd715321700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7100ffec0 0x7fd7101009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd715b22700 1 -- 172.21.15.151:0/2402555270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7100ffec0 msgr2=0x7fd7101009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd715b22700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7100ffec0 0x7fd7101009d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd715b22700 1 -- 172.21.15.151:0/2402555270 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7000095f0 con 0x7fd7100ff510 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd715b22700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ff510 0x7fd710100490 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00c930 tx=0x7fd70c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd706ffd700 1 -- 172.21.15.151:0/2402555270 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd70c007ad0 con 0x7fd7100ff510 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd706ffd700 1 -- 172.21.15.151:0/2402555270 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd70c007c30 con 0x7fd7100ff510 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7101a2910 con 0x7fd7100ff510 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd706ffd700 1 -- 172.21.15.151:0/2402555270 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd70c018610 con 0x7fd7100ff510 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.590+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7101a2c70 con 0x7fd7100ff510 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.591+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd710109180 con 0x7fd7100ff510 2024-04-16T02:55:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.592+0000 7fd706ffd700 1 -- 172.21.15.151:0/2402555270 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd70c00f450 con 0x7fd7100ff510 2024-04-16T02:55:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.592+0000 7fd706ffd700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd6fc06ecc0 0x7fd6fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.592+0000 7fd706ffd700 1 -- 172.21.15.151:0/2402555270 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd70c08f0a0 con 0x7fd7100ff510 2024-04-16T02:55:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.592+0000 7fd715321700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd6fc06ecc0 0x7fd6fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.593+0000 7fd715321700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd6fc06ecc0 0x7fd6fc071180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd70000b5c0 tx=0x7fd700005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:24.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.597+0000 7fd706ffd700 1 -- 172.21.15.151:0/2402555270 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd70c057ed0 con 0x7fd7100ff510 2024-04-16T02:55:24.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:24 smithi151 ceph-mon[112233]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:24.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:24 smithi151 ceph-mon[112233]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:24.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:24 smithi151 ceph-mon[112233]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:24.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:55:24.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:24 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1718308519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:55:24.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:24 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2978732951' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:55:24.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.817+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd71002cd50 con 0x7fd6fc06ecc0 2024-04-16T02:55:24.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.818+0000 7fd706ffd700 1 -- 172.21.15.151:0/2402555270 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd71002cd50 con 0x7fd6fc06ecc0 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:55:24.821 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:55:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.820+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd6fc06ecc0 msgr2=0x7fd6fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.820+0000 7fd717d86700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd6fc06ecc0 0x7fd6fc071180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd70000b5c0 tx=0x7fd700005fd0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.820+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ff510 msgr2=0x7fd710100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.820+0000 7fd717d86700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ff510 0x7fd710100490 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00c930 tx=0x7fd70c00ccf0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.820+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 shutdown_connections 2024-04-16T02:55:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.820+0000 7fd717d86700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd6fc06ecc0 0x7fd6fc071180 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.820+0000 7fd717d86700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7100ff510 0x7fd710100490 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.820+0000 7fd717d86700 1 --2- 172.21.15.151:0/2402555270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7100ffec0 0x7fd7101009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.820+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 >> 172.21.15.151:0/2402555270 conn(0x7fd710074d80 msgr2=0x7fd7100feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:24.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.821+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 shutdown_connections 2024-04-16T02:55:24.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.821+0000 7fd717d86700 1 -- 172.21.15.151:0/2402555270 wait complete. 2024-04-16T02:55:24.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.925+0000 7f1c346c6700 1 -- 172.21.15.151:0/1926499910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 msgr2=0x7f1c2c101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.925+0000 7f1c346c6700 1 --2- 172.21.15.151:0/1926499910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 0x7f1c2c101180 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009910 tx=0x7f1c28009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:24.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.925+0000 7f1c346c6700 1 -- 172.21.15.151:0/1926499910 shutdown_connections 2024-04-16T02:55:24.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.925+0000 7f1c346c6700 1 --2- 172.21.15.151:0/1926499910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c2c101750 0x7f1c2c1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.925+0000 7f1c346c6700 1 --2- 172.21.15.151:0/1926499910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 0x7f1c2c101180 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.925+0000 7f1c346c6700 1 -- 172.21.15.151:0/1926499910 >> 172.21.15.151:0/1926499910 conn(0x7f1c2c0fc630 msgr2=0x7f1c2c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:24.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.925+0000 7f1c346c6700 1 -- 172.21.15.151:0/1926499910 shutdown_connections 2024-04-16T02:55:24.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.925+0000 7f1c346c6700 1 -- 172.21.15.151:0/1926499910 wait complete. 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c346c6700 1 Processor -- start 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c346c6700 1 -- start start 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c346c6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 0x7f1c2c19c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c346c6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c2c101750 0x7f1c2c19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c346c6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c19d090 con 0x7f1c2c100da0 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c346c6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c196620 con 0x7f1c2c101750 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c32462700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 0x7f1c2c19c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c32462700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 0x7f1c2c19c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49730/0 (socket says 172.21.15.151:49730) 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c32462700 1 -- 172.21.15.151:0/3635779744 learned_addr learned my addr 172.21.15.151:0/3635779744 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c31c61700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c2c101750 0x7f1c2c19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c32462700 1 -- 172.21.15.151:0/3635779744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c2c101750 msgr2=0x7f1c2c19ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c32462700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c2c101750 0x7f1c2c19ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c32462700 1 -- 172.21.15.151:0/3635779744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280095f0 con 0x7f1c2c100da0 2024-04-16T02:55:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.926+0000 7f1c31c61700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c2c101750 0x7f1c2c19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:55:24.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.927+0000 7f1c32462700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 0x7f1c2c19c530 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006010 tx=0x7f1c2800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:24.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.927+0000 7f1c237fe700 1 -- 172.21.15.151:0/3635779744 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c2801d070 con 0x7f1c2c100da0 2024-04-16T02:55:24.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.927+0000 7f1c237fe700 1 -- 172.21.15.151:0/3635779744 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c2800f460 con 0x7f1c2c100da0 2024-04-16T02:55:24.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.927+0000 7f1c237fe700 1 -- 172.21.15.151:0/3635779744 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c28017630 con 0x7f1c2c100da0 2024-04-16T02:55:24.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.927+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c1968a0 con 0x7f1c2c100da0 2024-04-16T02:55:24.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.927+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c196d90 con 0x7f1c2c100da0 2024-04-16T02:55:24.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.928+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c109140 con 0x7f1c2c100da0 2024-04-16T02:55:24.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.928+0000 7f1c237fe700 1 -- 172.21.15.151:0/3635779744 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1c2800f5d0 con 0x7f1c2c100da0 2024-04-16T02:55:24.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.929+0000 7f1c237fe700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c1806ecc0 0x7f1c18071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:24.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.929+0000 7f1c237fe700 1 -- 172.21.15.151:0/3635779744 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1c28090310 con 0x7f1c2c100da0 2024-04-16T02:55:24.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.929+0000 7f1c31c61700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c1806ecc0 0x7f1c18071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:24.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.929+0000 7f1c31c61700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c1806ecc0 0x7f1c18071180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c005950 tx=0x7f1c1c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:24.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:24.934+0000 7f1c237fe700 1 -- 172.21.15.151:0/3635779744 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1c2805cbf0 con 0x7f1c2c100da0 2024-04-16T02:55:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:24 smithi153 ceph-mon[87858]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:24 smithi153 ceph-mon[87858]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:24 smithi153 ceph-mon[87858]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:55:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:24 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1718308519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:55:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:24 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2978732951' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:55:25.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.213+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1c2c04ea90 con 0x7f1c2c100da0 2024-04-16T02:55:25.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.213+0000 7f1c237fe700 1 -- 172.21.15.151:0/3635779744 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f1c2805c780 con 0x7f1c2c100da0 2024-04-16T02:55:25.216 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:55:25.216 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:55:25.216 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:55:25.216 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:55:25.216 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.215+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c1806ecc0 msgr2=0x7f1c18071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.215+0000 7f1c346c6700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c1806ecc0 0x7f1c18071180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c005950 tx=0x7f1c1c0058e0 comp rx=0 tx=0).stop 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.215+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 msgr2=0x7f1c2c19c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.215+0000 7f1c346c6700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 0x7f1c2c19c530 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006010 tx=0x7f1c2800be20 comp rx=0 tx=0).stop 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.215+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 shutdown_connections 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.215+0000 7f1c346c6700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c1806ecc0 0x7f1c18071180 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.215+0000 7f1c346c6700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c2c100da0 0x7f1c2c19c530 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.215+0000 7f1c346c6700 1 --2- 172.21.15.151:0/3635779744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c2c101750 0x7f1c2c19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.215+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 >> 172.21.15.151:0/3635779744 conn(0x7f1c2c0fc630 msgr2=0x7f1c2c0fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.216+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 shutdown_connections 2024-04-16T02:55:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:25.216+0000 7f1c346c6700 1 -- 172.21.15.151:0/3635779744 wait complete. 2024-04-16T02:55:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:25 smithi151 ceph-mon[112233]: pgmap v503: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:25 smithi151 ceph-mon[112233]: from='client.34978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:25 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3635779744' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:55:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:25 smithi153 ceph-mon[87858]: pgmap v503: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:25 smithi153 ceph-mon[87858]: from='client.34978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:25 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3635779744' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:55:27.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:27 smithi151 ceph-mon[112233]: pgmap v504: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:27 smithi153 ceph-mon[87858]: pgmap v504: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:29.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:29 smithi151 ceph-mon[112233]: pgmap v505: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:29 smithi153 ceph-mon[87858]: pgmap v505: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:31.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:31 smithi151 ceph-mon[112233]: pgmap v506: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:32.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:31 smithi153 ceph-mon[87858]: pgmap v506: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:33.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:33 smithi151 ceph-mon[112233]: pgmap v507: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:34.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:33 smithi153 ceph-mon[87858]: pgmap v507: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:35.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:35 smithi151 ceph-mon[112233]: pgmap v508: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:35 smithi153 ceph-mon[87858]: pgmap v508: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:37.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:37 smithi151 ceph-mon[112233]: pgmap v509: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:38.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:37 smithi153 ceph-mon[87858]: pgmap v509: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:39.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:39 smithi151 ceph-mon[112233]: pgmap v510: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:39.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:55:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:39 smithi153 ceph-mon[87858]: pgmap v510: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:55:42.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:41 smithi153 ceph-mon[87858]: pgmap v511: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:42.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:41 smithi151 ceph-mon[112233]: pgmap v511: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:43.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:43 smithi151 ceph-mon[112233]: pgmap v512: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:44.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:43 smithi153 ceph-mon[87858]: pgmap v512: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:46.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:45 smithi153 ceph-mon[87858]: pgmap v513: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:46.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:45 smithi151 ceph-mon[112233]: pgmap v513: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:48.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:47 smithi153 ceph-mon[87858]: pgmap v514: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:48.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:47 smithi151 ceph-mon[112233]: pgmap v514: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:50.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:49 smithi153 ceph-mon[87858]: pgmap v515: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:49 smithi151 ceph-mon[112233]: pgmap v515: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:52.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:51 smithi153 ceph-mon[87858]: pgmap v516: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:52.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:51 smithi151 ceph-mon[112233]: pgmap v516: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:54.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:53 smithi153 ceph-mon[87858]: pgmap v517: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:53 smithi151 ceph-mon[112233]: pgmap v517: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:55.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:55:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:55:55.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.335+0000 7f0643f13700 1 -- 172.21.15.151:0/2174866923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 msgr2=0x7f063c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:55.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.335+0000 7f0643f13700 1 --2- 172.21.15.151:0/2174866923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 0x7f063c1057c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0638009910 tx=0x7f0638009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:55.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.335+0000 7f0643f13700 1 -- 172.21.15.151:0/2174866923 shutdown_connections 2024-04-16T02:55:55.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.335+0000 7f0643f13700 1 --2- 172.21.15.151:0/2174866923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 0x7f063c1057c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.335+0000 7f0643f13700 1 --2- 172.21.15.151:0/2174866923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f063c100dc0 0x7f063c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.335+0000 7f0643f13700 1 -- 172.21.15.151:0/2174866923 >> 172.21.15.151:0/2174866923 conn(0x7f063c0fc650 msgr2=0x7f063c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:55.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.335+0000 7f0643f13700 1 -- 172.21.15.151:0/2174866923 shutdown_connections 2024-04-16T02:55:55.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.335+0000 7f0643f13700 1 -- 172.21.15.151:0/2174866923 wait complete. 2024-04-16T02:55:55.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f0643f13700 1 Processor -- start 2024-04-16T02:55:55.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f0643f13700 1 -- start start 2024-04-16T02:55:55.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f0643f13700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f063c100dc0 0x7f063c19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f0643f13700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 0x7f063c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f0643f13700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063c19d130 con 0x7f063c101770 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f0643f13700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063c1965e0 con 0x7f063c100dc0 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f06414ae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 0x7f063c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f06414ae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 0x7f063c19caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58362/0 (socket says 172.21.15.151:58362) 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f06414ae700 1 -- 172.21.15.151:0/1612020829 learned_addr learned my addr 172.21.15.151:0/1612020829 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f0641caf700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f063c100dc0 0x7f063c19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f06414ae700 1 -- 172.21.15.151:0/1612020829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f063c100dc0 msgr2=0x7f063c19c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f06414ae700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f063c100dc0 0x7f063c19c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f06414ae700 1 -- 172.21.15.151:0/1612020829 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06380095f0 con 0x7f063c101770 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.336+0000 7f06414ae700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 0x7f063c19caa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f06380092e0 tx=0x7f063800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.337+0000 7f062effd700 1 -- 172.21.15.151:0/1612020829 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f063801d070 con 0x7f063c101770 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.337+0000 7f062effd700 1 -- 172.21.15.151:0/1612020829 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0638005dc0 con 0x7f063c101770 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.337+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063c196860 con 0x7f063c101770 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.337+0000 7f062effd700 1 -- 172.21.15.151:0/1612020829 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0638017400 con 0x7f063c101770 2024-04-16T02:55:55.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.337+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063c196d50 con 0x7f063c101770 2024-04-16T02:55:55.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.338+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063c1091c0 con 0x7f063c101770 2024-04-16T02:55:55.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.338+0000 7f062effd700 1 -- 172.21.15.151:0/1612020829 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f063800f6f0 con 0x7f063c101770 2024-04-16T02:55:55.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.339+0000 7f062effd700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f062806ecc0 0x7f0628071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:55.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.339+0000 7f062effd700 1 -- 172.21.15.151:0/1612020829 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0638090250 con 0x7f063c101770 2024-04-16T02:55:55.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.339+0000 7f0641caf700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f062806ecc0 0x7f0628071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:55.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.339+0000 7f0641caf700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f062806ecc0 0x7f0628071180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0630005950 tx=0x7f063000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:55.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.344+0000 7f062effd700 1 -- 172.21.15.151:0/1612020829 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f063805cb30 con 0x7f063c101770 2024-04-16T02:55:55.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.563+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f063c197030 con 0x7f062806ecc0 2024-04-16T02:55:55.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.564+0000 7f062effd700 1 -- 172.21.15.151:0/1612020829 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f063c197030 con 0x7f062806ecc0 2024-04-16T02:55:55.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.566+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f062806ecc0 msgr2=0x7f0628071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:55.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.566+0000 7f0643f13700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f062806ecc0 0x7f0628071180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0630005950 tx=0x7f063000b410 comp rx=0 tx=0).stop 2024-04-16T02:55:55.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.566+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 msgr2=0x7f063c19caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:55.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.566+0000 7f0643f13700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 0x7f063c19caa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f06380092e0 tx=0x7f063800bd70 comp rx=0 tx=0).stop 2024-04-16T02:55:55.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.567+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 shutdown_connections 2024-04-16T02:55:55.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.567+0000 7f0643f13700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f062806ecc0 0x7f0628071180 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.567+0000 7f0643f13700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f063c100dc0 0x7f063c19c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.567+0000 7f0643f13700 1 --2- 172.21.15.151:0/1612020829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f063c101770 0x7f063c19caa0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.567+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 >> 172.21.15.151:0/1612020829 conn(0x7f063c0fc650 msgr2=0x7f063c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:55.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.567+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 shutdown_connections 2024-04-16T02:55:55.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.567+0000 7f0643f13700 1 -- 172.21.15.151:0/1612020829 wait complete. 2024-04-16T02:55:55.580 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:55:55.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 -- 172.21.15.151:0/3956515446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c0ffa40 msgr2=0x7f7b0c0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:55.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 --2- 172.21.15.151:0/3956515446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c0ffa40 0x7f7b0c0ffec0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009910 tx=0x7f7afc009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:55.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 -- 172.21.15.151:0/3956515446 shutdown_connections 2024-04-16T02:55:55.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 --2- 172.21.15.151:0/3956515446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c0ffa40 0x7f7b0c0ffec0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 --2- 172.21.15.151:0/3956515446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c105a40 0x7f7b0c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 -- 172.21.15.151:0/3956515446 >> 172.21.15.151:0/3956515446 conn(0x7f7b0c0fb550 msgr2=0x7f7b0c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:55.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 -- 172.21.15.151:0/3956515446 shutdown_connections 2024-04-16T02:55:55.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 -- 172.21.15.151:0/3956515446 wait complete. 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 Processor -- start 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.673+0000 7f7b13ede700 1 -- start start 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b13ede700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c0ffa40 0x7f7b0c19c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b13ede700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c105a40 0x7f7b0c19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b13ede700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c19d0f0 con 0x7f7b0c105a40 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b13ede700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c1965a0 con 0x7f7b0c0ffa40 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b11479700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c105a40 0x7f7b0c19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b11479700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c105a40 0x7f7b0c19ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58378/0 (socket says 172.21.15.151:58378) 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b11479700 1 -- 172.21.15.151:0/2143375868 learned_addr learned my addr 172.21.15.151:0/2143375868 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b11c7a700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c0ffa40 0x7f7b0c19c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b11479700 1 -- 172.21.15.151:0/2143375868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c0ffa40 msgr2=0x7f7b0c19c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b11479700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c0ffa40 0x7f7b0c19c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b11479700 1 -- 172.21.15.151:0/2143375868 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc0095f0 con 0x7f7b0c105a40 2024-04-16T02:55:55.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b11479700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c105a40 0x7f7b0c19ca60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7afc006010 tx=0x7f7afc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:55.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.674+0000 7f7b02ffd700 1 -- 172.21.15.151:0/2143375868 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7afc01d070 con 0x7f7b0c105a40 2024-04-16T02:55:55.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.675+0000 7f7b02ffd700 1 -- 172.21.15.151:0/2143375868 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7afc005dc0 con 0x7f7b0c105a40 2024-04-16T02:55:55.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.675+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0c196820 con 0x7f7b0c105a40 2024-04-16T02:55:55.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.675+0000 7f7b02ffd700 1 -- 172.21.15.151:0/2143375868 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7afc017400 con 0x7f7b0c105a40 2024-04-16T02:55:55.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.675+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c196d10 con 0x7f7b0c105a40 2024-04-16T02:55:55.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.676+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c108fe0 con 0x7f7b0c105a40 2024-04-16T02:55:55.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.676+0000 7f7b02ffd700 1 -- 172.21.15.151:0/2143375868 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7afc003a60 con 0x7f7b0c105a40 2024-04-16T02:55:55.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.676+0000 7f7b02ffd700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7af806ecc0 0x7f7af8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:55.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.677+0000 7f7b02ffd700 1 -- 172.21.15.151:0/2143375868 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7afc0902f0 con 0x7f7b0c105a40 2024-04-16T02:55:55.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.677+0000 7f7b11c7a700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7af806ecc0 0x7f7af8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:55.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.677+0000 7f7b11c7a700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7af806ecc0 0x7f7af8071180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b08005950 tx=0x7f7b0800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:55.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.681+0000 7f7b02ffd700 1 -- 172.21.15.151:0/2143375868 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7afc05cbd0 con 0x7f7b0c105a40 2024-04-16T02:55:55.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.900+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b0c196ff0 con 0x7f7af806ecc0 2024-04-16T02:55:55.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.901+0000 7f7b02ffd700 1 -- 172.21.15.151:0/2143375868 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7b0c196ff0 con 0x7f7af806ecc0 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7af806ecc0 msgr2=0x7f7af8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7af806ecc0 0x7f7af8071180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b08005950 tx=0x7f7b0800b410 comp rx=0 tx=0).stop 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c105a40 msgr2=0x7f7b0c19ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c105a40 0x7f7b0c19ca60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7afc006010 tx=0x7f7afc00bd70 comp rx=0 tx=0).stop 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 shutdown_connections 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7af806ecc0 0x7f7af8071180 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c0ffa40 0x7f7b0c19c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 --2- 172.21.15.151:0/2143375868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b0c105a40 0x7f7b0c19ca60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 >> 172.21.15.151:0/2143375868 conn(0x7f7b0c0fb550 msgr2=0x7f7b0c069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 shutdown_connections 2024-04-16T02:55:55.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:55.903+0000 7f7b13ede700 1 -- 172.21.15.151:0/2143375868 wait complete. 2024-04-16T02:55:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:55 smithi151 ceph-mon[112233]: pgmap v518: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:56.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.008+0000 7f2b85b5c700 1 -- 172.21.15.151:0/3196215549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 msgr2=0x7f2b800ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.008+0000 7f2b85b5c700 1 --2- 172.21.15.151:0/3196215549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 0x7f2b800ffec0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2b74009910 tx=0x7f2b74009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:56.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.008+0000 7f2b85b5c700 1 -- 172.21.15.151:0/3196215549 shutdown_connections 2024-04-16T02:55:56.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.008+0000 7f2b85b5c700 1 --2- 172.21.15.151:0/3196215549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 0x7f2b800ffec0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.008+0000 7f2b85b5c700 1 --2- 172.21.15.151:0/3196215549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b80105a40 0x7f2b80105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.008+0000 7f2b85b5c700 1 -- 172.21.15.151:0/3196215549 >> 172.21.15.151:0/3196215549 conn(0x7f2b800fb570 msgr2=0x7f2b800fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:56.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.008+0000 7f2b85b5c700 1 -- 172.21.15.151:0/3196215549 shutdown_connections 2024-04-16T02:55:56.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.008+0000 7f2b85b5c700 1 -- 172.21.15.151:0/3196215549 wait complete. 2024-04-16T02:55:56.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b85b5c700 1 Processor -- start 2024-04-16T02:55:56.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b85b5c700 1 -- start start 2024-04-16T02:55:56.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b85b5c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 0x7f2b800fbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:56.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b85b5c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b80105a40 0x7f2b800fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:56.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b85b5c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b800fc960 con 0x7f2b800ffa40 2024-04-16T02:55:56.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b85b5c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b800fcaa0 con 0x7f2b80105a40 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b7f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 0x7f2b800fbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b7f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 0x7f2b800fbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58394/0 (socket says 172.21.15.151:58394) 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b7f7fe700 1 -- 172.21.15.151:0/1372275889 learned_addr learned my addr 172.21.15.151:0/1372275889 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.009+0000 7f2b7effd700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b80105a40 0x7f2b800fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.010+0000 7f2b7f7fe700 1 -- 172.21.15.151:0/1372275889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b80105a40 msgr2=0x7f2b800fc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.010+0000 7f2b7f7fe700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b80105a40 0x7f2b800fc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.010+0000 7f2b7f7fe700 1 -- 172.21.15.151:0/1372275889 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b740095f0 con 0x7f2b800ffa40 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.010+0000 7f2b7f7fe700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 0x7f2b800fbee0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000b4b0 tx=0x7f2b7000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.010+0000 7f2b7cff9700 1 -- 172.21.15.151:0/1372275889 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b70011800 con 0x7f2b800ffa40 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.010+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b801a29e0 con 0x7f2b800ffa40 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.010+0000 7f2b7cff9700 1 -- 172.21.15.151:0/1372275889 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b70011e20 con 0x7f2b800ffa40 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.010+0000 7f2b7cff9700 1 -- 172.21.15.151:0/1372275889 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b7000f2e0 con 0x7f2b800ffa40 2024-04-16T02:55:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.010+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b801a2df0 con 0x7f2b800ffa40 2024-04-16T02:55:56.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.011+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b80109070 con 0x7f2b800ffa40 2024-04-16T02:55:56.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.011+0000 7f2b7cff9700 1 -- 172.21.15.151:0/1372275889 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2b70011960 con 0x7f2b800ffa40 2024-04-16T02:55:56.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.012+0000 7f2b7cff9700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b6806ecc0 0x7f2b68071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:56.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.012+0000 7f2b7cff9700 1 -- 172.21.15.151:0/1372275889 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2b7008e960 con 0x7f2b800ffa40 2024-04-16T02:55:56.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.012+0000 7f2b7effd700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b6806ecc0 0x7f2b68071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:56.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.012+0000 7f2b7effd700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b6806ecc0 0x7f2b68071180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2b800fd6d0 tx=0x7f2b7401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:56.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.017+0000 7f2b7cff9700 1 -- 172.21.15.151:0/1372275889 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2b70057710 con 0x7f2b800ffa40 2024-04-16T02:55:56.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.223+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b800611d0 con 0x7f2b6806ecc0 2024-04-16T02:55:56.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.234+0000 7f2b7cff9700 1 -- 172.21.15.151:0/1372275889 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2b800611d0 con 0x7f2b6806ecc0 2024-04-16T02:55:56.236 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:55:56.236 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (17m) 10m ago 28m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:55:56.236 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (29m) 10m ago 29m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:55:56.236 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (28m) 10m ago 28m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (15m) 10m ago 29m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (14m) 10m ago 27m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (17m) 10m ago 28m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (11m) 10m ago 23m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (11m) 10m ago 23m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (23m) 10m ago 23m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (10m) 10m ago 23m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (19m) 10m ago 30m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (18m) 10m ago 27m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (16m) 10m ago 30m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (15m) 10m ago 27m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (18m) 10m ago 28m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (18m) 10m ago 27m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (14m) 10m ago 26m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (13m) 10m ago 26m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (13m) 10m ago 26m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:55:56.237 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (12m) 10m ago 25m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:55:56.238 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (12m) 10m ago 25m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:55:56.238 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (11m) 10m ago 24m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:55:56.238 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (17m) 10m ago 28m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:55:56.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b6806ecc0 msgr2=0x7f2b68071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b6806ecc0 0x7f2b68071180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2b800fd6d0 tx=0x7f2b7401a040 comp rx=0 tx=0).stop 2024-04-16T02:55:56.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 msgr2=0x7f2b800fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 0x7f2b800fbee0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000b4b0 tx=0x7f2b7000b7c0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 shutdown_connections 2024-04-16T02:55:56.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b6806ecc0 0x7f2b68071180 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b800ffa40 0x7f2b800fbee0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 --2- 172.21.15.151:0/1372275889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b80105a40 0x7f2b800fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 >> 172.21.15.151:0/1372275889 conn(0x7f2b800fb570 msgr2=0x7f2b80068f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:56.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 shutdown_connections 2024-04-16T02:55:56.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.237+0000 7f2b85b5c700 1 -- 172.21.15.151:0/1372275889 wait complete. 2024-04-16T02:55:56.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:55 smithi153 ceph-mon[87858]: pgmap v518: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:56.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.341+0000 7fe480e82700 1 -- 172.21.15.151:0/500666929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 msgr2=0x7fe47c10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.341+0000 7fe480e82700 1 --2- 172.21.15.151:0/500666929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 0x7fe47c10cf00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe464009910 tx=0x7fe464009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:56.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.341+0000 7fe480e82700 1 -- 172.21.15.151:0/500666929 shutdown_connections 2024-04-16T02:55:56.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.341+0000 7fe480e82700 1 --2- 172.21.15.151:0/500666929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 0x7fe47c10cf00 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.341+0000 7fe480e82700 1 --2- 172.21.15.151:0/500666929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0730b0 0x7fe47c073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.341+0000 7fe480e82700 1 -- 172.21.15.151:0/500666929 >> 172.21.15.151:0/500666929 conn(0x7fe47c0fc670 msgr2=0x7fe47c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:56.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.341+0000 7fe480e82700 1 -- 172.21.15.151:0/500666929 shutdown_connections 2024-04-16T02:55:56.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.341+0000 7fe480e82700 1 -- 172.21.15.151:0/500666929 wait complete. 2024-04-16T02:55:56.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe480e82700 1 Processor -- start 2024-04-16T02:55:56.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe480e82700 1 -- start start 2024-04-16T02:55:56.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe480e82700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0730b0 0x7fe47c19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:56.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe480e82700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 0x7fe47c19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:56.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe480e82700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c19f250 con 0x7fe47c0739d0 2024-04-16T02:55:56.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe480e82700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c198790 con 0x7fe47c0730b0 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe479d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 0x7fe47c19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe479d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 0x7fe47c19ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58414/0 (socket says 172.21.15.151:58414) 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe479d9b700 1 -- 172.21.15.151:0/1333519313 learned_addr learned my addr 172.21.15.151:0/1333519313 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.342+0000 7fe47a59c700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0730b0 0x7fe47c19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.343+0000 7fe479d9b700 1 -- 172.21.15.151:0/1333519313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0730b0 msgr2=0x7fe47c19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.343+0000 7fe479d9b700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0730b0 0x7fe47c19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.343+0000 7fe479d9b700 1 -- 172.21.15.151:0/1333519313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640095f0 con 0x7fe47c0739d0 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.343+0000 7fe479d9b700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 0x7fe47c19ec50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe46400b5c0 tx=0x7fe46400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.343+0000 7fe4737fe700 1 -- 172.21.15.151:0/1333519313 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe46401d070 con 0x7fe47c0739d0 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.343+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47c198a10 con 0x7fe47c0739d0 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.343+0000 7fe4737fe700 1 -- 172.21.15.151:0/1333519313 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe46400f460 con 0x7fe47c0739d0 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.343+0000 7fe4737fe700 1 -- 172.21.15.151:0/1333519313 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe464017630 con 0x7fe47c0739d0 2024-04-16T02:55:56.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.343+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47c198f00 con 0x7fe47c0739d0 2024-04-16T02:55:56.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.344+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47c10a600 con 0x7fe47c0739d0 2024-04-16T02:55:56.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.345+0000 7fe4737fe700 1 -- 172.21.15.151:0/1333519313 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe46400f5d0 con 0x7fe47c0739d0 2024-04-16T02:55:56.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.345+0000 7fe4737fe700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe46806ecc0 0x7fe468071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:56.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.346+0000 7fe4737fe700 1 -- 172.21.15.151:0/1333519313 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe4640902d0 con 0x7fe47c0739d0 2024-04-16T02:55:56.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.346+0000 7fe47a59c700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe46806ecc0 0x7fe468071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:56.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.346+0000 7fe47a59c700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe46806ecc0 0x7fe468071180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe47400f440 tx=0x7fe4740079e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:56.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.350+0000 7fe4737fe700 1 -- 172.21.15.151:0/1333519313 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe46405cc30 con 0x7fe47c0739d0 2024-04-16T02:55:56.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.631+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe47c199d70 con 0x7fe47c0739d0 2024-04-16T02:55:56.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.632+0000 7fe4737fe700 1 -- 172.21.15.151:0/1333519313 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fe46405c7c0 con 0x7fe47c0739d0 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:55:56.634 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:55:56.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:55:56.635 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:55:56.635 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:55:56.635 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:55:56.635 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:55:56.635 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:55:56.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe46806ecc0 msgr2=0x7fe468071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe46806ecc0 0x7fe468071180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe47400f440 tx=0x7fe4740079e0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 msgr2=0x7fe47c19ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 0x7fe47c19ec50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe46400b5c0 tx=0x7fe46400bd50 comp rx=0 tx=0).stop 2024-04-16T02:55:56.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 shutdown_connections 2024-04-16T02:55:56.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe46806ecc0 0x7fe468071180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0730b0 0x7fe47c19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 --2- 172.21.15.151:0/1333519313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe47c0739d0 0x7fe47c19ec50 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 >> 172.21.15.151:0/1333519313 conn(0x7fe47c0fc670 msgr2=0x7fe47c107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:56.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 shutdown_connections 2024-04-16T02:55:56.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.634+0000 7fe480e82700 1 -- 172.21.15.151:0/1333519313 wait complete. 2024-04-16T02:55:56.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.739+0000 7f4188337700 1 -- 172.21.15.151:0/1493062438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180100dc0 msgr2=0x7f41801011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.739+0000 7f4188337700 1 --2- 172.21.15.151:0/1493062438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180100dc0 0x7f41801011a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f417c00b0a0 tx=0x7f417c00b3b0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.739+0000 7f4188337700 1 -- 172.21.15.151:0/1493062438 shutdown_connections 2024-04-16T02:55:56.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.739+0000 7f4188337700 1 --2- 172.21.15.151:0/1493062438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4180101770 0x7f41801057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.739+0000 7f4188337700 1 --2- 172.21.15.151:0/1493062438 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180100dc0 0x7f41801011a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.739+0000 7f4188337700 1 -- 172.21.15.151:0/1493062438 >> 172.21.15.151:0/1493062438 conn(0x7f41800fc650 msgr2=0x7f41800fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:56.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.739+0000 7f4188337700 1 -- 172.21.15.151:0/1493062438 shutdown_connections 2024-04-16T02:55:56.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.739+0000 7f4188337700 1 -- 172.21.15.151:0/1493062438 wait complete. 2024-04-16T02:55:56.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f4188337700 1 Processor -- start 2024-04-16T02:55:56.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f4188337700 1 -- start start 2024-04-16T02:55:56.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f4188337700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4180100dc0 0x7f418019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:56.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f4188337700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180101770 0x7f418019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:56.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f4188337700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418019d210 con 0x7f4180101770 2024-04-16T02:55:56.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f4188337700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41801966c0 con 0x7f4180100dc0 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f41858d2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180101770 0x7f418019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f41858d2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180101770 0x7f418019cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43064/0 (socket says 172.21.15.151:43064) 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f41858d2700 1 -- 172.21.15.151:0/2647964531 learned_addr learned my addr 172.21.15.151:0/2647964531 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f41860d3700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4180100dc0 0x7f418019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f41858d2700 1 -- 172.21.15.151:0/2647964531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4180100dc0 msgr2=0x7f418019c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f41858d2700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4180100dc0 0x7f418019c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.740+0000 7f41858d2700 1 -- 172.21.15.151:0/2647964531 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c009d00 con 0x7f4180101770 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.741+0000 7f41858d2700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180101770 0x7f418019cb80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f417000b480 tx=0x7f417000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.741+0000 7f41777fe700 1 -- 172.21.15.151:0/2647964531 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4170011800 con 0x7f4180101770 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.741+0000 7f41777fe700 1 -- 172.21.15.151:0/2647964531 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4170011e20 con 0x7f4180101770 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.741+0000 7f41777fe700 1 -- 172.21.15.151:0/2647964531 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4170010ea0 con 0x7f4180101770 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.741+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41801969a0 con 0x7f4180101770 2024-04-16T02:55:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.741+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4180196ef0 con 0x7f4180101770 2024-04-16T02:55:56.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.742+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4180109160 con 0x7f4180101770 2024-04-16T02:55:56.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.742+0000 7f41777fe700 1 -- 172.21.15.151:0/2647964531 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4170010380 con 0x7f4180101770 2024-04-16T02:55:56.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.743+0000 7f41777fe700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416c06ecc0 0x7f416c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:56.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.743+0000 7f41777fe700 1 -- 172.21.15.151:0/2647964531 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f417008e910 con 0x7f4180101770 2024-04-16T02:55:56.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.743+0000 7f41860d3700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416c06ecc0 0x7f416c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:56.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.743+0000 7f41860d3700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416c06ecc0 0x7f416c071180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f417c000f80 tx=0x7f417c0060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:56.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.748+0000 7f41777fe700 1 -- 172.21.15.151:0/2647964531 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f417005b1f0 con 0x7f4180101770 2024-04-16T02:55:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:56 smithi151 ceph-mon[112233]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:56 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1333519313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:55:56.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.996+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f418004ea90 con 0x7f4180101770 2024-04-16T02:55:56.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.997+0000 7f41777fe700 1 -- 172.21.15.151:0/2647964531 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f417005ad80 con 0x7f4180101770 2024-04-16T02:55:56.999 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:55:56.999 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:55:56.999 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:55:56.999 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:55:56.999 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:55:56.999 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:55:56.999 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:55:56.999 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:55:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:55:57.001 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:55:57.002 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:55:57.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416c06ecc0 msgr2=0x7f416c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416c06ecc0 0x7f416c071180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f417c000f80 tx=0x7f417c0060c0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180101770 msgr2=0x7f418019cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180101770 0x7f418019cb80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f417000b480 tx=0x7f417000b790 comp rx=0 tx=0).stop 2024-04-16T02:55:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 shutdown_connections 2024-04-16T02:55:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416c06ecc0 0x7f416c071180 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4180100dc0 0x7f418019c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 --2- 172.21.15.151:0/2647964531 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4180101770 0x7f418019cb80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 >> 172.21.15.151:0/2647964531 conn(0x7f41800fc650 msgr2=0x7f41800fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 shutdown_connections 2024-04-16T02:55:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:56.999+0000 7f4188337700 1 -- 172.21.15.151:0/2647964531 wait complete. 2024-04-16T02:55:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:55:57.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:56 smithi153 ceph-mon[87858]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:57.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:56 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1333519313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:55:57.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 -- 172.21.15.151:0/2223289624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0100dd0 msgr2=0x7fdfe01011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 --2- 172.21.15.151:0/2223289624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0100dd0 0x7fdfe01011b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009910 tx=0x7fdfd0009c20 comp rx=0 tx=0).stop 2024-04-16T02:55:57.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 -- 172.21.15.151:0/2223289624 shutdown_connections 2024-04-16T02:55:57.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 --2- 172.21.15.151:0/2223289624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfe0101780 0x7fdfe01057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 --2- 172.21.15.151:0/2223289624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0100dd0 0x7fdfe01011b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 -- 172.21.15.151:0/2223289624 >> 172.21.15.151:0/2223289624 conn(0x7fdfe00fc640 msgr2=0x7fdfe00fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:57.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 -- 172.21.15.151:0/2223289624 shutdown_connections 2024-04-16T02:55:57.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 -- 172.21.15.151:0/2223289624 wait complete. 2024-04-16T02:55:57.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 Processor -- start 2024-04-16T02:55:57.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.107+0000 7fdfe6675700 1 -- start start 2024-04-16T02:55:57.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfe6675700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfe0100dd0 0x7fdfe019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:57.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfe6675700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0101780 0x7fdfe019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfe6675700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe019d050 con 0x7fdfe0101780 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfe6675700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe0196590 con 0x7fdfe0100dd0 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfd7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0101780 0x7fdfe019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfd7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0101780 0x7fdfe019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43078/0 (socket says 172.21.15.151:43078) 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfd7fff700 1 -- 172.21.15.151:0/1045564940 learned_addr learned my addr 172.21.15.151:0/1045564940 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfdffff700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfe0100dd0 0x7fdfe019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfd7fff700 1 -- 172.21.15.151:0/1045564940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfe0100dd0 msgr2=0x7fdfe019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfd7fff700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfe0100dd0 0x7fdfe019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfd7fff700 1 -- 172.21.15.151:0/1045564940 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd00095f0 con 0x7fdfe0101780 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfdffff700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfe0100dd0 0x7fdfe019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfd7fff700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0101780 0x7fdfe019ca50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800c960 tx=0x7fdfc800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfddffb700 1 -- 172.21.15.151:0/1045564940 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc8007a70 con 0x7fdfe0101780 2024-04-16T02:55:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfddffb700 1 -- 172.21.15.151:0/1045564940 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfc8007bd0 con 0x7fdfe0101780 2024-04-16T02:55:57.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfddffb700 1 -- 172.21.15.151:0/1045564940 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc80186c0 con 0x7fdfe0101780 2024-04-16T02:55:57.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.108+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe0196870 con 0x7fdfe0101780 2024-04-16T02:55:57.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.109+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe0196dc0 con 0x7fdfe0101780 2024-04-16T02:55:57.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.109+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe0109170 con 0x7fdfe0101780 2024-04-16T02:55:57.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.110+0000 7fdfddffb700 1 -- 172.21.15.151:0/1045564940 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fdfc800faa0 con 0x7fdfe0101780 2024-04-16T02:55:57.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.111+0000 7fdfddffb700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdfcc06ecc0 0x7fdfcc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:57.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.111+0000 7fdfdffff700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdfcc06ecc0 0x7fdfcc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:57.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.111+0000 7fdfddffb700 1 -- 172.21.15.151:0/1045564940 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdfc808eed0 con 0x7fdfe0101780 2024-04-16T02:55:57.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.111+0000 7fdfdffff700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdfcc06ecc0 0x7fdfcc071180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0005b40 tx=0x7fdfd001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:57.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.115+0000 7fdfddffb700 1 -- 172.21.15.151:0/1045564940 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdfc805b4c0 con 0x7fdfe0101780 2024-04-16T02:55:57.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.334+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfe0197040 con 0x7fdfcc06ecc0 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.336+0000 7fdfddffb700 1 -- 172.21.15.151:0/1045564940 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdfe0197040 con 0x7fdfcc06ecc0 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:55:57.338 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:55:57.339 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:55:57.339 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:55:57.339 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:55:57.339 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:55:57.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdfcc06ecc0 msgr2=0x7fdfcc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdfcc06ecc0 0x7fdfcc071180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0005b40 tx=0x7fdfd001a040 comp rx=0 tx=0).stop 2024-04-16T02:55:57.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0101780 msgr2=0x7fdfe019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0101780 0x7fdfe019ca50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800c960 tx=0x7fdfc800cd20 comp rx=0 tx=0).stop 2024-04-16T02:55:57.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 shutdown_connections 2024-04-16T02:55:57.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdfcc06ecc0 0x7fdfcc071180 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfe0100dd0 0x7fdfe019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 --2- 172.21.15.151:0/1045564940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0101780 0x7fdfe019ca50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 >> 172.21.15.151:0/1045564940 conn(0x7fdfe00fc640 msgr2=0x7fdfe00fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:57.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 shutdown_connections 2024-04-16T02:55:57.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.338+0000 7fdfe6675700 1 -- 172.21.15.151:0/1045564940 wait complete. 2024-04-16T02:55:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.444+0000 7f56fde6e700 1 -- 172.21.15.151:0/3838325436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8100dc0 msgr2=0x7f56f81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.444+0000 7f56fde6e700 1 --2- 172.21.15.151:0/3838325436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8100dc0 0x7f56f81011a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f56e800b0a0 tx=0x7f56e800b3b0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.444+0000 7f56fde6e700 1 -- 172.21.15.151:0/3838325436 shutdown_connections 2024-04-16T02:55:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.444+0000 7f56fde6e700 1 --2- 172.21.15.151:0/3838325436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f8101770 0x7f56f81057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.444+0000 7f56fde6e700 1 --2- 172.21.15.151:0/3838325436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8100dc0 0x7f56f81011a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.444+0000 7f56fde6e700 1 -- 172.21.15.151:0/3838325436 >> 172.21.15.151:0/3838325436 conn(0x7f56f80fc650 msgr2=0x7f56f80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.444+0000 7f56fde6e700 1 -- 172.21.15.151:0/3838325436 shutdown_connections 2024-04-16T02:55:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.444+0000 7f56fde6e700 1 -- 172.21.15.151:0/3838325436 wait complete. 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.444+0000 7f56fde6e700 1 Processor -- start 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56fde6e700 1 -- start start 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56fde6e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f8100dc0 0x7f56f8198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56fde6e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8101770 0x7f56f8198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56fde6e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f8199540 con 0x7f56f8101770 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56fde6e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f819d350 con 0x7f56f8100dc0 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56f6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8101770 0x7f56f8198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56f6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8101770 0x7f56f8198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43100/0 (socket says 172.21.15.151:43100) 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56f6ffd700 1 -- 172.21.15.151:0/1531861958 learned_addr learned my addr 172.21.15.151:0/1531861958 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56f77fe700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f8100dc0 0x7f56f8198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:57.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56f6ffd700 1 -- 172.21.15.151:0/1531861958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f8100dc0 msgr2=0x7f56f8198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56f6ffd700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f8100dc0 0x7f56f8198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56f6ffd700 1 -- 172.21.15.151:0/1531861958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e8009d00 con 0x7f56f8101770 2024-04-16T02:55:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.445+0000 7f56f6ffd700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8101770 0x7f56f8198eb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f56e000e8f0 tx=0x7f56e000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.446+0000 7f56f4ff9700 1 -- 172.21.15.151:0/1531861958 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56e0009790 con 0x7f56f8101770 2024-04-16T02:55:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.446+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f819d630 con 0x7f56f8101770 2024-04-16T02:55:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.446+0000 7f56f4ff9700 1 -- 172.21.15.151:0/1531861958 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56e000c810 con 0x7f56f8101770 2024-04-16T02:55:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.446+0000 7f56f4ff9700 1 -- 172.21.15.151:0/1531861958 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56e0005580 con 0x7f56f8101770 2024-04-16T02:55:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.446+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f819db80 con 0x7f56f8101770 2024-04-16T02:55:57.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.447+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f8104f40 con 0x7f56f8101770 2024-04-16T02:55:57.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.447+0000 7f56f4ff9700 1 -- 172.21.15.151:0/1531861958 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f56e000c980 con 0x7f56f8101770 2024-04-16T02:55:57.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.448+0000 7f56f4ff9700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56e406ecc0 0x7f56e4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:55:57.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.448+0000 7f56f77fe700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56e406ecc0 0x7f56e4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:55:57.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.448+0000 7f56f4ff9700 1 -- 172.21.15.151:0/1531861958 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f56e0014070 con 0x7f56f8101770 2024-04-16T02:55:57.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.448+0000 7f56f77fe700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56e406ecc0 0x7f56e4071180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f56e8000f80 tx=0x7f56e80060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:55:57.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.452+0000 7f56f4ff9700 1 -- 172.21.15.151:0/1531861958 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f56e005ba00 con 0x7f56f8101770 2024-04-16T02:55:57.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.732+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f56f804ea90 con 0x7f56f8101770 2024-04-16T02:55:57.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.732+0000 7f56f4ff9700 1 -- 172.21.15.151:0/1531861958 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f56e0062030 con 0x7f56f8101770 2024-04-16T02:55:57.735 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:55:57.735 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:55:57.735 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:55:57.735 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:55:57.735 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:55:57.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56e406ecc0 msgr2=0x7f56e4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56e406ecc0 0x7f56e4071180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f56e8000f80 tx=0x7f56e80060c0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8101770 msgr2=0x7f56f8198eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:55:57.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8101770 0x7f56f8198eb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f56e000e8f0 tx=0x7f56e000ec00 comp rx=0 tx=0).stop 2024-04-16T02:55:57.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 shutdown_connections 2024-04-16T02:55:57.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f56e406ecc0 0x7f56e4071180 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f8100dc0 0x7f56f8198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 --2- 172.21.15.151:0/1531861958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56f8101770 0x7f56f8198eb0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:55:57.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 >> 172.21.15.151:0/1531861958 conn(0x7f56f80fc650 msgr2=0x7f56f80fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:55:57.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 shutdown_connections 2024-04-16T02:55:57.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:55:57.736+0000 7f56fde6e700 1 -- 172.21.15.151:0/1531861958 wait complete. 2024-04-16T02:55:57.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:57 smithi151 ceph-mon[112233]: from='client.34990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:57.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:57 smithi151 ceph-mon[112233]: pgmap v519: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:57.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:57 smithi151 ceph-mon[112233]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:57 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2647964531' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:55:57.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:57 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1531861958' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:55:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:57 smithi153 ceph-mon[87858]: from='client.34990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:57 smithi153 ceph-mon[87858]: pgmap v519: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:55:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:57 smithi153 ceph-mon[87858]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:57 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2647964531' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:55:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:57 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1531861958' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:55:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:58 smithi151 ceph-mon[112233]: from='client.35006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:55:59.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:58 smithi153 ceph-mon[87858]: from='client.35006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:56:00.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:55:59 smithi151 ceph-mon[112233]: pgmap v520: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:55:59 smithi153 ceph-mon[87858]: pgmap v520: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:01 smithi153 ceph-mon[87858]: pgmap v521: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:02.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:01 smithi151 ceph-mon[112233]: pgmap v521: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:04.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:03 smithi151 ceph-mon[112233]: pgmap v522: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:03 smithi153 ceph-mon[87858]: pgmap v522: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:06.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:05 smithi151 ceph-mon[112233]: pgmap v523: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:05 smithi153 ceph-mon[87858]: pgmap v523: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:08.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:07 smithi151 ceph-mon[112233]: pgmap v524: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:07 smithi153 ceph-mon[87858]: pgmap v524: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:56:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:56:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:09 smithi151 ceph-mon[112233]: pgmap v525: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:10.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:09 smithi153 ceph-mon[87858]: pgmap v525: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:12.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:11 smithi153 ceph-mon[87858]: pgmap v526: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:12.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:11 smithi151 ceph-mon[112233]: pgmap v526: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:14.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:13 smithi151 ceph-mon[112233]: pgmap v527: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:14.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:13 smithi153 ceph-mon[87858]: pgmap v527: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:16.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:15 smithi151 ceph-mon[112233]: pgmap v528: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:16.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:15 smithi153 ceph-mon[87858]: pgmap v528: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:18.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:17 smithi151 ceph-mon[112233]: pgmap v529: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:17 smithi153 ceph-mon[87858]: pgmap v529: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:20.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:19 smithi153 ceph-mon[87858]: pgmap v530: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:20.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:19 smithi151 ceph-mon[112233]: pgmap v530: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:22.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:21 smithi151 ceph-mon[112233]: pgmap v531: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:22.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:21 smithi153 ceph-mon[87858]: pgmap v531: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:24.141 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:23 smithi151 ceph-mon[112233]: pgmap v532: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:24.141 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:56:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:23 smithi153 ceph-mon[87858]: pgmap v532: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:56:26.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:25 smithi151 ceph-mon[112233]: pgmap v533: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:26.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:25 smithi153 ceph-mon[87858]: pgmap v533: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.855+0000 7f64d3658700 1 -- 172.21.15.151:0/2559962758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 msgr2=0x7f64cc1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.855+0000 7f64d3658700 1 --2- 172.21.15.151:0/2559962758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 0x7f64cc1011a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f64c8009910 tx=0x7f64c8009c20 comp rx=0 tx=0).stop 2024-04-16T02:56:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.855+0000 7f64d3658700 1 -- 172.21.15.151:0/2559962758 shutdown_connections 2024-04-16T02:56:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.855+0000 7f64d3658700 1 --2- 172.21.15.151:0/2559962758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc101770 0x7f64cc1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.855+0000 7f64d3658700 1 --2- 172.21.15.151:0/2559962758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 0x7f64cc1011a0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.855+0000 7f64d3658700 1 -- 172.21.15.151:0/2559962758 >> 172.21.15.151:0/2559962758 conn(0x7f64cc0fc650 msgr2=0x7f64cc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.855+0000 7f64d3658700 1 -- 172.21.15.151:0/2559962758 shutdown_connections 2024-04-16T02:56:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.855+0000 7f64d3658700 1 -- 172.21.15.151:0/2559962758 wait complete. 2024-04-16T02:56:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d3658700 1 Processor -- start 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d3658700 1 -- start start 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d3658700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 0x7f64cc19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d3658700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc101770 0x7f64cc19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d3658700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc19d130 con 0x7f64cc100dc0 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d3658700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc1965e0 con 0x7f64cc101770 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d13f4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 0x7f64cc19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d13f4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 0x7f64cc19c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45006/0 (socket says 172.21.15.151:45006) 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d13f4700 1 -- 172.21.15.151:0/594076911 learned_addr learned my addr 172.21.15.151:0/594076911 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d13f4700 1 -- 172.21.15.151:0/594076911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc101770 msgr2=0x7f64cc19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d0bf3700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc101770 0x7f64cc19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d13f4700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc101770 0x7f64cc19caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d13f4700 1 -- 172.21.15.151:0/594076911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c80095f0 con 0x7f64cc100dc0 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.856+0000 7f64d13f4700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 0x7f64cc19c560 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f64c8006010 tx=0x7f64c800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.857+0000 7f64c27fc700 1 -- 172.21.15.151:0/594076911 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c801d070 con 0x7f64cc100dc0 2024-04-16T02:56:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.857+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc196860 con 0x7f64cc100dc0 2024-04-16T02:56:27.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.857+0000 7f64c27fc700 1 -- 172.21.15.151:0/594076911 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64c8004fb0 con 0x7f64cc100dc0 2024-04-16T02:56:27.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.857+0000 7f64c27fc700 1 -- 172.21.15.151:0/594076911 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c8022ea0 con 0x7f64cc100dc0 2024-04-16T02:56:27.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.857+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc196d50 con 0x7f64cc100dc0 2024-04-16T02:56:27.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.858+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc109160 con 0x7f64cc100dc0 2024-04-16T02:56:27.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.858+0000 7f64c27fc700 1 -- 172.21.15.151:0/594076911 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f64c8022470 con 0x7f64cc100dc0 2024-04-16T02:56:27.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.859+0000 7f64c27fc700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f64b806ecc0 0x7f64b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:27.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.859+0000 7f64c27fc700 1 -- 172.21.15.151:0/594076911 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f64c80903f0 con 0x7f64cc100dc0 2024-04-16T02:56:27.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.859+0000 7f64d0bf3700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f64b806ecc0 0x7f64b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:27.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.859+0000 7f64d0bf3700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f64b806ecc0 0x7f64b8071180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f64cc1059b0 tx=0x7f64bc005cf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:27.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:27.864+0000 7f64c27fc700 1 -- 172.21.15.151:0/594076911 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f64c805cd80 con 0x7f64cc100dc0 2024-04-16T02:56:28.088 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.085+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f64cc02cd00 con 0x7f64b806ecc0 2024-04-16T02:56:28.089 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:27 smithi151 ceph-mon[112233]: pgmap v534: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:28.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.087+0000 7f64c27fc700 1 -- 172.21.15.151:0/594076911 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f64cc02cd00 con 0x7f64b806ecc0 2024-04-16T02:56:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f64b806ecc0 msgr2=0x7f64b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f64b806ecc0 0x7f64b8071180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f64cc1059b0 tx=0x7f64bc005cf0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 msgr2=0x7f64cc19c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 0x7f64cc19c560 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f64c8006010 tx=0x7f64c800be20 comp rx=0 tx=0).stop 2024-04-16T02:56:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 shutdown_connections 2024-04-16T02:56:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f64b806ecc0 0x7f64b8071180 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64cc100dc0 0x7f64cc19c560 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 --2- 172.21.15.151:0/594076911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc101770 0x7f64cc19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 >> 172.21.15.151:0/594076911 conn(0x7f64cc0fc650 msgr2=0x7f64cc0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:28.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 shutdown_connections 2024-04-16T02:56:28.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.090+0000 7f64d3658700 1 -- 172.21.15.151:0/594076911 wait complete. 2024-04-16T02:56:28.102 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:56:28.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.195+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3075714095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 msgr2=0x7fdf981011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.195+0000 7fdf9dff2700 1 --2- 172.21.15.151:0/3075714095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 0x7fdf981011a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009910 tx=0x7fdf88009c20 comp rx=0 tx=0).stop 2024-04-16T02:56:28.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.195+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3075714095 shutdown_connections 2024-04-16T02:56:28.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.195+0000 7fdf9dff2700 1 --2- 172.21.15.151:0/3075714095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98101770 0x7fdf981057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.195+0000 7fdf9dff2700 1 --2- 172.21.15.151:0/3075714095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 0x7fdf981011a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.195+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3075714095 >> 172.21.15.151:0/3075714095 conn(0x7fdf980fc650 msgr2=0x7fdf980fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:28.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.195+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3075714095 shutdown_connections 2024-04-16T02:56:28.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.196+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3075714095 wait complete. 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.196+0000 7fdf9dff2700 1 Processor -- start 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.196+0000 7fdf9dff2700 1 -- start start 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.196+0000 7fdf9dff2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 0x7fdf9819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.196+0000 7fdf9dff2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98101770 0x7fdf9819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.196+0000 7fdf9dff2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9819d210 con 0x7fdf98100dc0 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.196+0000 7fdf9dff2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf981966c0 con 0x7fdf98101770 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf977fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 0x7fdf9819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf977fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 0x7fdf9819c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45034/0 (socket says 172.21.15.151:45034) 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf977fe700 1 -- 172.21.15.151:0/3277351882 learned_addr learned my addr 172.21.15.151:0/3277351882 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf96ffd700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98101770 0x7fdf9819cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf977fe700 1 -- 172.21.15.151:0/3277351882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98101770 msgr2=0x7fdf9819cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf977fe700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98101770 0x7fdf9819cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf977fe700 1 -- 172.21.15.151:0/3277351882 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf880095f0 con 0x7fdf98100dc0 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf96ffd700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98101770 0x7fdf9819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:56:28.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf977fe700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 0x7fdf9819c640 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fdf88006010 tx=0x7fdf8800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:28.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf94ff9700 1 -- 172.21.15.151:0/3277351882 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf8801d070 con 0x7fdf98100dc0 2024-04-16T02:56:28.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf94ff9700 1 -- 172.21.15.151:0/3277351882 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf88004fb0 con 0x7fdf98100dc0 2024-04-16T02:56:28.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf98196940 con 0x7fdf98100dc0 2024-04-16T02:56:28.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf94ff9700 1 -- 172.21.15.151:0/3277351882 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf88022ea0 con 0x7fdf98100dc0 2024-04-16T02:56:28.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.197+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf98196e30 con 0x7fdf98100dc0 2024-04-16T02:56:28.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.198+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf98109160 con 0x7fdf98100dc0 2024-04-16T02:56:28.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.199+0000 7fdf94ff9700 1 -- 172.21.15.151:0/3277351882 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fdf88022470 con 0x7fdf98100dc0 2024-04-16T02:56:28.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.199+0000 7fdf94ff9700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf8006ecc0 0x7fdf80071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:28.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.199+0000 7fdf94ff9700 1 -- 172.21.15.151:0/3277351882 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdf88090350 con 0x7fdf98100dc0 2024-04-16T02:56:28.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.199+0000 7fdf96ffd700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf8006ecc0 0x7fdf80071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:28.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.200+0000 7fdf96ffd700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf8006ecc0 0x7fdf80071180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdf98197b30 tx=0x7fdf8c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:28.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.204+0000 7fdf94ff9700 1 -- 172.21.15.151:0/3277351882 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdf8805cd60 con 0x7fdf98100dc0 2024-04-16T02:56:28.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:27 smithi153 ceph-mon[87858]: pgmap v534: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:28.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.422+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf981970a0 con 0x7fdf8006ecc0 2024-04-16T02:56:28.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.423+0000 7fdf94ff9700 1 -- 172.21.15.151:0/3277351882 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdf981970a0 con 0x7fdf8006ecc0 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.425+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf8006ecc0 msgr2=0x7fdf80071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.425+0000 7fdf9dff2700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf8006ecc0 0x7fdf80071180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdf98197b30 tx=0x7fdf8c008040 comp rx=0 tx=0).stop 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.425+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 msgr2=0x7fdf9819c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.425+0000 7fdf9dff2700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 0x7fdf9819c640 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fdf88006010 tx=0x7fdf8800be20 comp rx=0 tx=0).stop 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.425+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 shutdown_connections 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.425+0000 7fdf9dff2700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf8006ecc0 0x7fdf80071180 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.425+0000 7fdf9dff2700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf98100dc0 0x7fdf9819c640 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.425+0000 7fdf9dff2700 1 --2- 172.21.15.151:0/3277351882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98101770 0x7fdf9819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.425+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 >> 172.21.15.151:0/3277351882 conn(0x7fdf980fc650 msgr2=0x7fdf980fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.426+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 shutdown_connections 2024-04-16T02:56:28.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.426+0000 7fdf9dff2700 1 -- 172.21.15.151:0/3277351882 wait complete. 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.530+0000 7f48df392700 1 -- 172.21.15.151:0/2382698897 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d8105a80 msgr2=0x7f48d8105e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.530+0000 7f48df392700 1 --2- 172.21.15.151:0/2382698897 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d8105a80 0x7f48d8105e60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009910 tx=0x7f48d4009c20 comp rx=0 tx=0).stop 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.530+0000 7f48df392700 1 -- 172.21.15.151:0/2382698897 shutdown_connections 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.530+0000 7f48df392700 1 --2- 172.21.15.151:0/2382698897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48d80ffa80 0x7f48d80fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.530+0000 7f48df392700 1 --2- 172.21.15.151:0/2382698897 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d8105a80 0x7f48d8105e60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.530+0000 7f48df392700 1 -- 172.21.15.151:0/2382698897 >> 172.21.15.151:0/2382698897 conn(0x7f48d80fb590 msgr2=0x7f48d80fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.530+0000 7f48df392700 1 -- 172.21.15.151:0/2382698897 shutdown_connections 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.530+0000 7f48df392700 1 -- 172.21.15.151:0/2382698897 wait complete. 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48df392700 1 Processor -- start 2024-04-16T02:56:28.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48df392700 1 -- start start 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48df392700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d80ffa80 0x7f48d8197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48df392700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48d8105a80 0x7f48d8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48df392700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8198180 con 0x7f48d80ffa80 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48df392700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d81982c0 con 0x7f48d8105a80 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48dd12e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d80ffa80 0x7f48d8197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48dd12e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d80ffa80 0x7f48d8197560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45060/0 (socket says 172.21.15.151:45060) 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48dd12e700 1 -- 172.21.15.151:0/3209670562 learned_addr learned my addr 172.21.15.151:0/3209670562 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.531+0000 7f48dc92d700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48d8105a80 0x7f48d8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.532+0000 7f48dd12e700 1 -- 172.21.15.151:0/3209670562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48d8105a80 msgr2=0x7f48d8197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.532+0000 7f48dd12e700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48d8105a80 0x7f48d8197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.532+0000 7f48dd12e700 1 -- 172.21.15.151:0/3209670562 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d40095f0 con 0x7f48d80ffa80 2024-04-16T02:56:28.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.532+0000 7f48dd12e700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d80ffa80 0x7f48d8197560 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f48d4006010 tx=0x7f48d400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:28.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.532+0000 7f48ca7fc700 1 -- 172.21.15.151:0/3209670562 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48d401d070 con 0x7f48d80ffa80 2024-04-16T02:56:28.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.532+0000 7f48ca7fc700 1 -- 172.21.15.151:0/3209670562 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48d4005dc0 con 0x7f48d80ffa80 2024-04-16T02:56:28.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.532+0000 7f48ca7fc700 1 -- 172.21.15.151:0/3209670562 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48d4017400 con 0x7f48d80ffa80 2024-04-16T02:56:28.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.532+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d81a05d0 con 0x7f48d80ffa80 2024-04-16T02:56:28.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.532+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d81a09e0 con 0x7f48d80ffa80 2024-04-16T02:56:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.533+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d804ea90 con 0x7f48d80ffa80 2024-04-16T02:56:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.534+0000 7f48ca7fc700 1 -- 172.21.15.151:0/3209670562 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f48d4003a60 con 0x7f48d80ffa80 2024-04-16T02:56:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.534+0000 7f48ca7fc700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f48c406ecc0 0x7f48c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.534+0000 7f48dc92d700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f48c406ecc0 0x7f48c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.534+0000 7f48ca7fc700 1 -- 172.21.15.151:0/3209670562 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f48d40902f0 con 0x7f48d80ffa80 2024-04-16T02:56:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.535+0000 7f48dc92d700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f48c406ecc0 0x7f48c4071180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f48d8069e70 tx=0x7f48cc00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:28.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.540+0000 7f48ca7fc700 1 -- 172.21.15.151:0/3209670562 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f48d40590a0 con 0x7f48d80ffa80 2024-04-16T02:56:28.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.749+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f48d80fc5a0 con 0x7f48c406ecc0 2024-04-16T02:56:28.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.760+0000 7f48ca7fc700 1 -- 172.21.15.151:0/3209670562 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f48d80fc5a0 con 0x7f48c406ecc0 2024-04-16T02:56:28.763 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:56:28.763 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (18m) 11m ago 29m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (29m) 11m ago 29m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (28m) 11m ago 28m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (15m) 11m ago 29m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (15m) 11m ago 28m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (17m) 11m ago 29m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (11m) 11m ago 23m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (11m) 11m ago 23m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (23m) 11m ago 23m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (11m) 11m ago 23m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (20m) 11m ago 30m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (18m) 11m ago 28m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (16m) 11m ago 30m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:56:28.764 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (16m) 11m ago 28m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:56:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (18m) 11m ago 29m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:56:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (18m) 11m ago 28m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:56:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (15m) 11m ago 27m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:56:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (14m) 11m ago 27m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:56:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (13m) 11m ago 26m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:56:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (13m) 11m ago 26m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:56:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (12m) 11m ago 25m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:56:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (12m) 11m ago 25m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:56:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (18m) 11m ago 28m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:56:28.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f48c406ecc0 msgr2=0x7f48c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f48c406ecc0 0x7f48c4071180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f48d8069e70 tx=0x7f48cc00b410 comp rx=0 tx=0).stop 2024-04-16T02:56:28.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d80ffa80 msgr2=0x7f48d8197560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d80ffa80 0x7f48d8197560 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f48d4006010 tx=0x7f48d400bd70 comp rx=0 tx=0).stop 2024-04-16T02:56:28.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 shutdown_connections 2024-04-16T02:56:28.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f48c406ecc0 0x7f48c4071180 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d80ffa80 0x7f48d8197560 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 --2- 172.21.15.151:0/3209670562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48d8105a80 0x7f48d8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 >> 172.21.15.151:0/3209670562 conn(0x7f48d80fb590 msgr2=0x7f48d80fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:28.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 shutdown_connections 2024-04-16T02:56:28.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.764+0000 7f48df392700 1 -- 172.21.15.151:0/3209670562 wait complete. 2024-04-16T02:56:28.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.867+0000 7f5d98d87700 1 -- 172.21.15.151:0/2900335046 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 msgr2=0x7f5d941011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.867+0000 7f5d98d87700 1 --2- 172.21.15.151:0/2900335046 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 0x7f5d941011b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009910 tx=0x7f5d8c009c20 comp rx=0 tx=0).stop 2024-04-16T02:56:28.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.867+0000 7f5d98d87700 1 -- 172.21.15.151:0/2900335046 shutdown_connections 2024-04-16T02:56:28.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.867+0000 7f5d98d87700 1 --2- 172.21.15.151:0/2900335046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d94101780 0x7f5d941057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.867+0000 7f5d98d87700 1 --2- 172.21.15.151:0/2900335046 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 0x7f5d941011b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.867+0000 7f5d98d87700 1 -- 172.21.15.151:0/2900335046 >> 172.21.15.151:0/2900335046 conn(0x7f5d940fc640 msgr2=0x7f5d940fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:28.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.867+0000 7f5d98d87700 1 -- 172.21.15.151:0/2900335046 shutdown_connections 2024-04-16T02:56:28.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.867+0000 7f5d98d87700 1 -- 172.21.15.151:0/2900335046 wait complete. 2024-04-16T02:56:28.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d98d87700 1 Processor -- start 2024-04-16T02:56:28.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d98d87700 1 -- start start 2024-04-16T02:56:28.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d98d87700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 0x7f5d94196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:28.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d98d87700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d94101780 0x7f5d94196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:28.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d98d87700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d94197310 con 0x7f5d94100dd0 2024-04-16T02:56:28.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d98d87700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d94199fd0 con 0x7f5d94101780 2024-04-16T02:56:28.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d9259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 0x7f5d94196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d9259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 0x7f5d94196740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45078/0 (socket says 172.21.15.151:45078) 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d9259c700 1 -- 172.21.15.151:0/2996095905 learned_addr learned my addr 172.21.15.151:0/2996095905 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d91d9b700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d94101780 0x7f5d94196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d9259c700 1 -- 172.21.15.151:0/2996095905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d94101780 msgr2=0x7f5d94196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d9259c700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d94101780 0x7f5d94196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.868+0000 7f5d9259c700 1 -- 172.21.15.151:0/2996095905 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c0095f0 con 0x7f5d94100dd0 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.869+0000 7f5d9259c700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 0x7f5d94196740 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c006010 tx=0x7f5d8c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.869+0000 7f5d8b7fe700 1 -- 172.21.15.151:0/2996095905 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d8c01d070 con 0x7f5d94100dd0 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.869+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9419a250 con 0x7f5d94100dd0 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.869+0000 7f5d8b7fe700 1 -- 172.21.15.151:0/2996095905 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d8c005dc0 con 0x7f5d94100dd0 2024-04-16T02:56:28.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.869+0000 7f5d8b7fe700 1 -- 172.21.15.151:0/2996095905 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d8c017400 con 0x7f5d94100dd0 2024-04-16T02:56:28.875 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.869+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d9419a740 con 0x7f5d94100dd0 2024-04-16T02:56:28.875 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.870+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d94109170 con 0x7f5d94100dd0 2024-04-16T02:56:28.875 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.870+0000 7f5d8b7fe700 1 -- 172.21.15.151:0/2996095905 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5d8c003a60 con 0x7f5d94100dd0 2024-04-16T02:56:28.875 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.871+0000 7f5d8b7fe700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d8006ecc0 0x7f5d80071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:28.875 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.871+0000 7f5d8b7fe700 1 -- 172.21.15.151:0/2996095905 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5d8c0902f0 con 0x7f5d94100dd0 2024-04-16T02:56:28.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.871+0000 7f5d91d9b700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d8006ecc0 0x7f5d80071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:28.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.871+0000 7f5d91d9b700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d8006ecc0 0x7f5d80071180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5d94197cf0 tx=0x7f5d7c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:28.878 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:28.876+0000 7f5d8b7fe700 1 -- 172.21.15.151:0/2996095905 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5d8c0590a0 con 0x7f5d94100dd0 2024-04-16T02:56:29.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.162+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5d94108b50 con 0x7f5d94100dd0 2024-04-16T02:56:29.164 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.162+0000 7f5d8b7fe700 1 -- 172.21.15.151:0/2996095905 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f5d8c05c760 con 0x7f5d94100dd0 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:56:29.165 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:56:29.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:56:29.166 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:56:29.166 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:56:29.166 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:56:29.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:56:29.166 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:56:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.164+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d8006ecc0 msgr2=0x7f5d80071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.164+0000 7f5d98d87700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d8006ecc0 0x7f5d80071180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5d94197cf0 tx=0x7f5d7c00b410 comp rx=0 tx=0).stop 2024-04-16T02:56:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.164+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 msgr2=0x7f5d94196740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.164+0000 7f5d98d87700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 0x7f5d94196740 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c006010 tx=0x7f5d8c00bd70 comp rx=0 tx=0).stop 2024-04-16T02:56:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.165+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 shutdown_connections 2024-04-16T02:56:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.165+0000 7f5d98d87700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d8006ecc0 0x7f5d80071180 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.165+0000 7f5d98d87700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d94100dd0 0x7f5d94196740 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.165+0000 7f5d98d87700 1 --2- 172.21.15.151:0/2996095905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d94101780 0x7f5d94196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.165+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 >> 172.21.15.151:0/2996095905 conn(0x7f5d940fc640 msgr2=0x7f5d940fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:29.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.165+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 shutdown_connections 2024-04-16T02:56:29.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.165+0000 7f5d98d87700 1 -- 172.21.15.151:0/2996095905 wait complete. 2024-04-16T02:56:29.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.273+0000 7efca23bf700 1 -- 172.21.15.151:0/32468683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ff530 msgr2=0x7efc9c0ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.273+0000 7efca23bf700 1 --2- 172.21.15.151:0/32468683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ff530 0x7efc9c0ff910 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7efc8c009910 tx=0x7efc8c009c20 comp rx=0 tx=0).stop 2024-04-16T02:56:29.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.273+0000 7efca23bf700 1 -- 172.21.15.151:0/32468683 shutdown_connections 2024-04-16T02:56:29.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.273+0000 7efca23bf700 1 --2- 172.21.15.151:0/32468683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc9c0ffee0 0x7efc9c105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.273+0000 7efca23bf700 1 --2- 172.21.15.151:0/32468683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ff530 0x7efc9c0ff910 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.273+0000 7efca23bf700 1 -- 172.21.15.151:0/32468683 >> 172.21.15.151:0/32468683 conn(0x7efc9c074d20 msgr2=0x7efc9c075130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.273+0000 7efca23bf700 1 -- 172.21.15.151:0/32468683 shutdown_connections 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.273+0000 7efca23bf700 1 -- 172.21.15.151:0/32468683 wait complete. 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efca23bf700 1 Processor -- start 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efca23bf700 1 -- start start 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efca23bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc9c0ff530 0x7efc9c19c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efca23bf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ffee0 0x7efc9c19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efca23bf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c19d150 con 0x7efc9c0ffee0 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efca23bf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c196600 con 0x7efc9c0ff530 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efc93fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ffee0 0x7efc9c19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efc93fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ffee0 0x7efc9c19cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45094/0 (socket says 172.21.15.151:45094) 2024-04-16T02:56:29.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efc93fff700 1 -- 172.21.15.151:0/2807455339 learned_addr learned my addr 172.21.15.151:0/2807455339 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efc9bfff700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc9c0ff530 0x7efc9c19c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efc93fff700 1 -- 172.21.15.151:0/2807455339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc9c0ff530 msgr2=0x7efc9c19c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efc93fff700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc9c0ff530 0x7efc9c19c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efc93fff700 1 -- 172.21.15.151:0/2807455339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0095f0 con 0x7efc9c0ffee0 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.274+0000 7efc9bfff700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc9c0ff530 0x7efc9c19c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.275+0000 7efc93fff700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ffee0 0x7efc9c19cac0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7efc8400e8f0 tx=0x7efc8400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.275+0000 7efc99ffb700 1 -- 172.21.15.151:0/2807455339 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc84009790 con 0x7efc9c0ffee0 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.275+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c1968e0 con 0x7efc9c0ffee0 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.275+0000 7efc99ffb700 1 -- 172.21.15.151:0/2807455339 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc84004cf0 con 0x7efc9c0ffee0 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.275+0000 7efc99ffb700 1 -- 172.21.15.151:0/2807455339 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc84005690 con 0x7efc9c0ffee0 2024-04-16T02:56:29.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.275+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c196e30 con 0x7efc9c0ffee0 2024-04-16T02:56:29.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.276+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c1091c0 con 0x7efc9c0ffee0 2024-04-16T02:56:29.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.276+0000 7efc99ffb700 1 -- 172.21.15.151:0/2807455339 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efc84004750 con 0x7efc9c0ffee0 2024-04-16T02:56:29.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.277+0000 7efc99ffb700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc8806ecc0 0x7efc88071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:29.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.277+0000 7efc99ffb700 1 -- 172.21.15.151:0/2807455339 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efc84014070 con 0x7efc9c0ffee0 2024-04-16T02:56:29.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.277+0000 7efc9bfff700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc8806ecc0 0x7efc88071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:29.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.277+0000 7efc9bfff700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc8806ecc0 0x7efc88071180 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00b760 tx=0x7efc8c009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:29.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.281+0000 7efc99ffb700 1 -- 172.21.15.151:0/2807455339 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efc8405bb60 con 0x7efc9c0ffee0 2024-04-16T02:56:29.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.532+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efc9c1970b0 con 0x7efc9c0ffee0 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.533+0000 7efc99ffb700 1 -- 172.21.15.151:0/2807455339 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7efc8405b6f0 con 0x7efc9c0ffee0 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:56:29.535 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:56:29.536 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:56:29.537 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:56:29.538 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:56:29.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc8806ecc0 msgr2=0x7efc88071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc8806ecc0 0x7efc88071180 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00b760 tx=0x7efc8c009f70 comp rx=0 tx=0).stop 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ffee0 msgr2=0x7efc9c19cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ffee0 0x7efc9c19cac0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7efc8400e8f0 tx=0x7efc8400ecb0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 shutdown_connections 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc8806ecc0 0x7efc88071180 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc9c0ff530 0x7efc9c19c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 --2- 172.21.15.151:0/2807455339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc9c0ffee0 0x7efc9c19cac0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 >> 172.21.15.151:0/2807455339 conn(0x7efc9c074d20 msgr2=0x7efc9c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 shutdown_connections 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.535+0000 7efca23bf700 1 -- 172.21.15.151:0/2807455339 wait complete. 2024-04-16T02:56:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:56:29.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.641+0000 7f04c0e2a700 1 -- 172.21.15.151:0/306504907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 msgr2=0x7f04bc1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.641+0000 7f04c0e2a700 1 --2- 172.21.15.151:0/306504907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 0x7f04bc1011a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009910 tx=0x7f04ac009c20 comp rx=0 tx=0).stop 2024-04-16T02:56:29.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.641+0000 7f04c0e2a700 1 -- 172.21.15.151:0/306504907 shutdown_connections 2024-04-16T02:56:29.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.641+0000 7f04c0e2a700 1 --2- 172.21.15.151:0/306504907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04bc101770 0x7f04bc1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.641+0000 7f04c0e2a700 1 --2- 172.21.15.151:0/306504907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 0x7f04bc1011a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.641+0000 7f04c0e2a700 1 -- 172.21.15.151:0/306504907 >> 172.21.15.151:0/306504907 conn(0x7f04bc0fc650 msgr2=0x7f04bc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:29.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.641+0000 7f04c0e2a700 1 -- 172.21.15.151:0/306504907 shutdown_connections 2024-04-16T02:56:29.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.641+0000 7f04c0e2a700 1 -- 172.21.15.151:0/306504907 wait complete. 2024-04-16T02:56:29.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04c0e2a700 1 Processor -- start 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04c0e2a700 1 -- start start 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04c0e2a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 0x7f04bc19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04c0e2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04bc101770 0x7f04bc19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04c0e2a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc19d110 con 0x7f04bc100dc0 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04c0e2a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc1965c0 con 0x7f04bc101770 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04ba59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 0x7f04bc19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04ba59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 0x7f04bc19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45106/0 (socket says 172.21.15.151:45106) 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04ba59c700 1 -- 172.21.15.151:0/2637091611 learned_addr learned my addr 172.21.15.151:0/2637091611 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04b9d9b700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04bc101770 0x7f04bc19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04ba59c700 1 -- 172.21.15.151:0/2637091611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04bc101770 msgr2=0x7f04bc19ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04ba59c700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04bc101770 0x7f04bc19ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04ba59c700 1 -- 172.21.15.151:0/2637091611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac0095f0 con 0x7f04bc100dc0 2024-04-16T02:56:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04ba59c700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 0x7f04bc19c540 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f04ac006010 tx=0x7f04ac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:29.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04ab7fe700 1 -- 172.21.15.151:0/2637091611 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04ac01d070 con 0x7f04bc100dc0 2024-04-16T02:56:29.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.642+0000 7f04ab7fe700 1 -- 172.21.15.151:0/2637091611 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04ac00f460 con 0x7f04bc100dc0 2024-04-16T02:56:29.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.643+0000 7f04ab7fe700 1 -- 172.21.15.151:0/2637091611 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04ac017630 con 0x7f04bc100dc0 2024-04-16T02:56:29.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.643+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc196840 con 0x7f04bc100dc0 2024-04-16T02:56:29.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.643+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc196d30 con 0x7f04bc100dc0 2024-04-16T02:56:29.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.644+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc109160 con 0x7f04bc100dc0 2024-04-16T02:56:29.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.644+0000 7f04ab7fe700 1 -- 172.21.15.151:0/2637091611 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f04ac00f5d0 con 0x7f04bc100dc0 2024-04-16T02:56:29.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.645+0000 7f04ab7fe700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f04a406ecc0 0x7f04a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:29.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.645+0000 7f04b9d9b700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f04a406ecc0 0x7f04a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:29.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.645+0000 7f04ab7fe700 1 -- 172.21.15.151:0/2637091611 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f04ac090330 con 0x7f04bc100dc0 2024-04-16T02:56:29.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.645+0000 7f04b9d9b700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f04a406ecc0 0x7f04a4071180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f04b0005950 tx=0x7f04b00058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:29.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.650+0000 7f04ab7fe700 1 -- 172.21.15.151:0/2637091611 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f04ac05cc10 con 0x7f04bc100dc0 2024-04-16T02:56:29.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.868+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f04bc197010 con 0x7f04a406ecc0 2024-04-16T02:56:29.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.869+0000 7f04ab7fe700 1 -- 172.21.15.151:0/2637091611 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f04bc197010 con 0x7f04a406ecc0 2024-04-16T02:56:29.871 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:56:29.871 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:56:29.872 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:56:29.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.871+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f04a406ecc0 msgr2=0x7f04a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.871+0000 7f04c0e2a700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f04a406ecc0 0x7f04a4071180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f04b0005950 tx=0x7f04b00058e0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.871+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 msgr2=0x7f04bc19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.871+0000 7f04c0e2a700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 0x7f04bc19c540 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f04ac006010 tx=0x7f04ac00be20 comp rx=0 tx=0).stop 2024-04-16T02:56:29.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.872+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 shutdown_connections 2024-04-16T02:56:29.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.872+0000 7f04c0e2a700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f04a406ecc0 0x7f04a4071180 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.872+0000 7f04c0e2a700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04bc100dc0 0x7f04bc19c540 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.875 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.872+0000 7f04c0e2a700 1 --2- 172.21.15.151:0/2637091611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04bc101770 0x7f04bc19ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.875 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.872+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 >> 172.21.15.151:0/2637091611 conn(0x7f04bc0fc650 msgr2=0x7f04bc0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:29.875 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.872+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 shutdown_connections 2024-04-16T02:56:29.875 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.872+0000 7f04c0e2a700 1 -- 172.21.15.151:0/2637091611 wait complete. 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.977+0000 7fca6f18e700 1 -- 172.21.15.151:0/4144509812 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca68101dc0 msgr2=0x7fca681021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.977+0000 7fca6f18e700 1 --2- 172.21.15.151:0/4144509812 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca68101dc0 0x7fca681021a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009910 tx=0x7fca5c009c20 comp rx=0 tx=0).stop 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.977+0000 7fca6f18e700 1 -- 172.21.15.151:0/4144509812 shutdown_connections 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.977+0000 7fca6f18e700 1 --2- 172.21.15.151:0/4144509812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca681026e0 0x7fca6810abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.977+0000 7fca6f18e700 1 --2- 172.21.15.151:0/4144509812 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca68101dc0 0x7fca681021a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.977+0000 7fca6f18e700 1 -- 172.21.15.151:0/4144509812 >> 172.21.15.151:0/4144509812 conn(0x7fca680fb550 msgr2=0x7fca680fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.977+0000 7fca6f18e700 1 -- 172.21.15.151:0/4144509812 shutdown_connections 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.977+0000 7fca6f18e700 1 -- 172.21.15.151:0/4144509812 wait complete. 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca6f18e700 1 Processor -- start 2024-04-16T02:56:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca6f18e700 1 -- start start 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca6f18e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca68101dc0 0x7fca680ffa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca6f18e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca681026e0 0x7fca680fff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca6f18e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca681004a0 con 0x7fca681026e0 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca6f18e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca681005e0 con 0x7fca68101dc0 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca67fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca681026e0 0x7fca680fff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca67fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca681026e0 0x7fca680fff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45118/0 (socket says 172.21.15.151:45118) 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca67fff700 1 -- 172.21.15.151:0/3848063091 learned_addr learned my addr 172.21.15.151:0/3848063091 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.978+0000 7fca6cf2a700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca68101dc0 0x7fca680ffa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.979+0000 7fca67fff700 1 -- 172.21.15.151:0/3848063091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca68101dc0 msgr2=0x7fca680ffa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.979+0000 7fca67fff700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca68101dc0 0x7fca680ffa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.979+0000 7fca67fff700 1 -- 172.21.15.151:0/3848063091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c0095f0 con 0x7fca681026e0 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.979+0000 7fca67fff700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca681026e0 0x7fca680fff60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fca5400b480 tx=0x7fca5400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:29.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.979+0000 7fca65ffb700 1 -- 172.21.15.151:0/3848063091 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca54011800 con 0x7fca681026e0 2024-04-16T02:56:29.984 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.979+0000 7fca65ffb700 1 -- 172.21.15.151:0/3848063091 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca54011e20 con 0x7fca681026e0 2024-04-16T02:56:29.984 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.979+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca681a2a00 con 0x7fca681026e0 2024-04-16T02:56:29.984 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.979+0000 7fca65ffb700 1 -- 172.21.15.151:0/3848063091 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca54010ea0 con 0x7fca681026e0 2024-04-16T02:56:29.984 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.979+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca681a2e10 con 0x7fca681026e0 2024-04-16T02:56:29.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.980+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca681082d0 con 0x7fca681026e0 2024-04-16T02:56:29.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.981+0000 7fca65ffb700 1 -- 172.21.15.151:0/3848063091 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fca54010380 con 0x7fca681026e0 2024-04-16T02:56:29.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.981+0000 7fca65ffb700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fca5806ecc0 0x7fca58071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:56:29.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.981+0000 7fca65ffb700 1 -- 172.21.15.151:0/3848063091 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fca5408e890 con 0x7fca681026e0 2024-04-16T02:56:29.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.981+0000 7fca6cf2a700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fca5806ecc0 0x7fca58071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:56:29.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.982+0000 7fca6cf2a700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fca5806ecc0 0x7fca58071180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fca5c005f50 tx=0x7fca5c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:56:29.988 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:29.986+0000 7fca65ffb700 1 -- 172.21.15.151:0/3848063091 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fca540576c0 con 0x7fca681026e0 2024-04-16T02:56:30.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:29 smithi151 ceph-mon[112233]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:56:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:29 smithi151 ceph-mon[112233]: pgmap v535: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:29 smithi151 ceph-mon[112233]: from='client.35018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:56:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:29 smithi151 ceph-mon[112233]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:56:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:29 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2996095905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:56:30.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:29 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2807455339' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:56:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:29 smithi153 ceph-mon[87858]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:56:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:29 smithi153 ceph-mon[87858]: pgmap v535: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:29 smithi153 ceph-mon[87858]: from='client.35018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:56:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:29 smithi153 ceph-mon[87858]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:56:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:29 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2996095905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:56:30.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:29 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2807455339' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:56:30.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.275+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fca6804ea90 con 0x7fca681026e0 2024-04-16T02:56:30.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.275+0000 7fca65ffb700 1 -- 172.21.15.151:0/3848063091 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fca5405ad80 con 0x7fca681026e0 2024-04-16T02:56:30.277 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:56:30.277 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:56:30.277 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:56:30.277 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:56:30.278 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:56:30.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fca5806ecc0 msgr2=0x7fca58071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fca5806ecc0 0x7fca58071180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fca5c005f50 tx=0x7fca5c01a040 comp rx=0 tx=0).stop 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca681026e0 msgr2=0x7fca680fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca681026e0 0x7fca680fff60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fca5400b480 tx=0x7fca5400b790 comp rx=0 tx=0).stop 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 shutdown_connections 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fca5806ecc0 0x7fca58071180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca68101dc0 0x7fca680ffa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 --2- 172.21.15.151:0/3848063091 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca681026e0 0x7fca680fff60 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 >> 172.21.15.151:0/3848063091 conn(0x7fca680fb550 msgr2=0x7fca68105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 shutdown_connections 2024-04-16T02:56:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:56:30.277+0000 7fca6f18e700 1 -- 172.21.15.151:0/3848063091 wait complete. 2024-04-16T02:56:31.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:30 smithi151 ceph-mon[112233]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:56:31.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:30 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3848063091' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:56:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:30 smithi153 ceph-mon[87858]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:56:31.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:30 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3848063091' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:56:32.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:31 smithi151 ceph-mon[112233]: pgmap v536: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:32.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:31 smithi153 ceph-mon[87858]: pgmap v536: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:34.243 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:33 smithi151 ceph-mon[112233]: pgmap v537: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:33 smithi153 ceph-mon[87858]: pgmap v537: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:36.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:35 smithi151 ceph-mon[112233]: pgmap v538: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:36.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:35 smithi153 ceph-mon[87858]: pgmap v538: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:38.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:37 smithi151 ceph-mon[112233]: pgmap v539: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:38.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:37 smithi153 ceph-mon[87858]: pgmap v539: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:39.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:56:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:56:40.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:39 smithi151 ceph-mon[112233]: pgmap v540: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:40.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:39 smithi153 ceph-mon[87858]: pgmap v540: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:42.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:41 smithi151 ceph-mon[112233]: pgmap v541: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:42.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:41 smithi153 ceph-mon[87858]: pgmap v541: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:44.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:43 smithi151 ceph-mon[112233]: pgmap v542: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:44.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:43 smithi153 ceph-mon[87858]: pgmap v542: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:46.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:45 smithi151 ceph-mon[112233]: pgmap v543: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:46.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:45 smithi153 ceph-mon[87858]: pgmap v543: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:48.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:47 smithi151 ceph-mon[112233]: pgmap v544: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:48.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:47 smithi153 ceph-mon[87858]: pgmap v544: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:50.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:49 smithi151 ceph-mon[112233]: pgmap v545: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:50.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:49 smithi153 ceph-mon[87858]: pgmap v545: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:52.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:51 smithi153 ceph-mon[87858]: pgmap v546: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:52.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:51 smithi151 ceph-mon[112233]: pgmap v546: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:54.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:53 smithi151 ceph-mon[112233]: pgmap v547: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:54.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:56:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:53 smithi153 ceph-mon[87858]: pgmap v547: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:56:56.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:55 smithi151 ceph-mon[112233]: pgmap v548: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:56.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:55 smithi153 ceph-mon[87858]: pgmap v548: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:58.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:57 smithi151 ceph-mon[112233]: pgmap v549: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:56:58.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:57 smithi153 ceph-mon[87858]: pgmap v549: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:00.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:56:59 smithi151 ceph-mon[112233]: pgmap v550: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:56:59 smithi153 ceph-mon[87858]: pgmap v550: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:00.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.397+0000 7ff6b49c1700 1 -- 172.21.15.151:0/3151994814 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 msgr2=0x7ff6b0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:00.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.397+0000 7ff6b49c1700 1 --2- 172.21.15.151:0/3151994814 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 0x7ff6b0105e20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009910 tx=0x7ff6a0009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:00.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.397+0000 7ff6b49c1700 1 -- 172.21.15.151:0/3151994814 shutdown_connections 2024-04-16T02:57:00.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.397+0000 7ff6b49c1700 1 --2- 172.21.15.151:0/3151994814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b00ffa40 0x7ff6b00ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.397+0000 7ff6b49c1700 1 --2- 172.21.15.151:0/3151994814 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 0x7ff6b0105e20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.397+0000 7ff6b49c1700 1 -- 172.21.15.151:0/3151994814 >> 172.21.15.151:0/3151994814 conn(0x7ff6b00fb530 msgr2=0x7ff6b00fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:00.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.397+0000 7ff6b49c1700 1 -- 172.21.15.151:0/3151994814 shutdown_connections 2024-04-16T02:57:00.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.397+0000 7ff6b49c1700 1 -- 172.21.15.151:0/3151994814 wait complete. 2024-04-16T02:57:00.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6b49c1700 1 Processor -- start 2024-04-16T02:57:00.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6b49c1700 1 -- start start 2024-04-16T02:57:00.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6b49c1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b00ffa40 0x7ff6b0197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:00.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6b49c1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 0x7ff6b0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6b49c1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0198120 con 0x7ff6b0105a40 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6b49c1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0198260 con 0x7ff6b00ffa40 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6add9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 0x7ff6b0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6add9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 0x7ff6b0197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42900/0 (socket says 172.21.15.151:42900) 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6add9b700 1 -- 172.21.15.151:0/2708396101 learned_addr learned my addr 172.21.15.151:0/2708396101 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6ae59c700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b00ffa40 0x7ff6b0197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6add9b700 1 -- 172.21.15.151:0/2708396101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b00ffa40 msgr2=0x7ff6b0197500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6add9b700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b00ffa40 0x7ff6b0197500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6add9b700 1 -- 172.21.15.151:0/2708396101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a00095f0 con 0x7ff6b0105a40 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.398+0000 7ff6ae59c700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b00ffa40 0x7ff6b0197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:57:00.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.399+0000 7ff6add9b700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 0x7ff6b0197a40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400c8a0 tx=0x7ff6a400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:00.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.399+0000 7ff69f7fe700 1 -- 172.21.15.151:0/2708396101 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6a400cea0 con 0x7ff6b0105a40 2024-04-16T02:57:00.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.399+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b01a0570 con 0x7ff6b0105a40 2024-04-16T02:57:00.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.399+0000 7ff69f7fe700 1 -- 172.21.15.151:0/2708396101 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6a4004830 con 0x7ff6b0105a40 2024-04-16T02:57:00.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.399+0000 7ff69f7fe700 1 -- 172.21.15.151:0/2708396101 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6a40055b0 con 0x7ff6b0105a40 2024-04-16T02:57:00.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.399+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b01a0a90 con 0x7ff6b0105a40 2024-04-16T02:57:00.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.400+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b0108040 con 0x7ff6b0105a40 2024-04-16T02:57:00.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.401+0000 7ff69f7fe700 1 -- 172.21.15.151:0/2708396101 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff6a40049a0 con 0x7ff6b0105a40 2024-04-16T02:57:00.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.401+0000 7ff69f7fe700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff69806ecc0 0x7ff698071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:00.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.401+0000 7ff69f7fe700 1 -- 172.21.15.151:0/2708396101 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff6a408eb50 con 0x7ff6b0105a40 2024-04-16T02:57:00.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.401+0000 7ff6ae59c700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff69806ecc0 0x7ff698071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:00.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.402+0000 7ff6ae59c700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff69806ecc0 0x7ff698071180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff6a000b5c0 tx=0x7ff6a0005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:00.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.406+0000 7ff69f7fe700 1 -- 172.21.15.151:0/2708396101 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff6a4057900 con 0x7ff6b0105a40 2024-04-16T02:57:00.625 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.623+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6b002cd80 con 0x7ff69806ecc0 2024-04-16T02:57:00.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.625+0000 7ff69f7fe700 1 -- 172.21.15.151:0/2708396101 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff6b002cd80 con 0x7ff69806ecc0 2024-04-16T02:57:00.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff69806ecc0 msgr2=0x7ff698071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:00.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff69806ecc0 0x7ff698071180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff6a000b5c0 tx=0x7ff6a0005960 comp rx=0 tx=0).stop 2024-04-16T02:57:00.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 msgr2=0x7ff6b0197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:00.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 0x7ff6b0197a40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400c8a0 tx=0x7ff6a400cc60 comp rx=0 tx=0).stop 2024-04-16T02:57:00.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 shutdown_connections 2024-04-16T02:57:00.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff69806ecc0 0x7ff698071180 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b00ffa40 0x7ff6b0197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 --2- 172.21.15.151:0/2708396101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b0105a40 0x7ff6b0197a40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 >> 172.21.15.151:0/2708396101 conn(0x7ff6b00fb530 msgr2=0x7ff6b00fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:00.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 shutdown_connections 2024-04-16T02:57:00.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.627+0000 7ff6b49c1700 1 -- 172.21.15.151:0/2708396101 wait complete. 2024-04-16T02:57:00.640 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:57:00.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.734+0000 7f6ae3864700 1 -- 172.21.15.151:0/3977702205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 msgr2=0x7f6adc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:00.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.734+0000 7f6ae3864700 1 --2- 172.21.15.151:0/3977702205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 0x7f6adc100050 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009910 tx=0x7f6acc009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:00.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.734+0000 7f6ae3864700 1 -- 172.21.15.151:0/3977702205 shutdown_connections 2024-04-16T02:57:00.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.734+0000 7f6ae3864700 1 --2- 172.21.15.151:0/3977702205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 0x7f6adc100050 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.734+0000 7f6ae3864700 1 --2- 172.21.15.151:0/3977702205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc105a40 0x7f6adc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.735+0000 7f6ae3864700 1 -- 172.21.15.151:0/3977702205 >> 172.21.15.151:0/3977702205 conn(0x7f6adc0755a0 msgr2=0x7f6adc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:00.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.735+0000 7f6ae3864700 1 -- 172.21.15.151:0/3977702205 shutdown_connections 2024-04-16T02:57:00.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.735+0000 7f6ae3864700 1 -- 172.21.15.151:0/3977702205 wait complete. 2024-04-16T02:57:00.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.735+0000 7f6ae3864700 1 Processor -- start 2024-04-16T02:57:00.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.735+0000 7f6ae3864700 1 -- start start 2024-04-16T02:57:00.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.735+0000 7f6ae3864700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 0x7f6adc101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:00.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.735+0000 7f6ae3864700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc105a40 0x7f6adc101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:00.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.735+0000 7f6ae3864700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc102090 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.735+0000 7f6ae3864700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc1021d0 con 0x7f6adc105a40 2024-04-16T02:57:00.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae1600700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 0x7f6adc101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:00.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae1600700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 0x7f6adc101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42920/0 (socket says 172.21.15.151:42920) 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae1600700 1 -- 172.21.15.151:0/2539394744 learned_addr learned my addr 172.21.15.151:0/2539394744 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae0dff700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc105a40 0x7f6adc101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae1600700 1 -- 172.21.15.151:0/2539394744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc105a40 msgr2=0x7f6adc101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae1600700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc105a40 0x7f6adc101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae1600700 1 -- 172.21.15.151:0/2539394744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc0095f0 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae1600700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 0x7f6adc101610 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800c930 tx=0x7f6ad800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ad27fc700 1 -- 172.21.15.151:0/2539394744 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad8007ad0 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ad27fc700 1 -- 172.21.15.151:0/2539394744 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ad8007c30 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc1a2900 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ad27fc700 1 -- 172.21.15.151:0/2539394744 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad8018610 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.736+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc1a2c60 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.737+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc108fe0 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.738+0000 7f6ad27fc700 1 -- 172.21.15.151:0/2539394744 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6ad800f450 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.739+0000 7f6ad27fc700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ac806ecc0 0x7f6ac8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:00.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.739+0000 7f6ad27fc700 1 -- 172.21.15.151:0/2539394744 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6ad808f1a0 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.739+0000 7f6ae0dff700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ac806ecc0 0x7f6ac8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:00.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.739+0000 7f6ae0dff700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ac806ecc0 0x7f6ac8071180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6acc000c00 tx=0x7f6acc005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:00.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.743+0000 7f6ad27fc700 1 -- 172.21.15.151:0/2539394744 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6ad8058070 con 0x7f6adc0ffbd0 2024-04-16T02:57:00.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.959+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6adc02cd50 con 0x7f6ac806ecc0 2024-04-16T02:57:00.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.960+0000 7f6ad27fc700 1 -- 172.21.15.151:0/2539394744 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6adc02cd50 con 0x7f6ac806ecc0 2024-04-16T02:57:00.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ac806ecc0 msgr2=0x7f6ac8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:00.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ac806ecc0 0x7f6ac8071180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6acc000c00 tx=0x7f6acc005fd0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 msgr2=0x7f6adc101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:00.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 0x7f6adc101610 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800c930 tx=0x7f6ad800ccf0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 shutdown_connections 2024-04-16T02:57:00.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ac806ecc0 0x7f6ac8071180 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6adc0ffbd0 0x7f6adc101610 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 --2- 172.21.15.151:0/2539394744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc105a40 0x7f6adc101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:00.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 >> 172.21.15.151:0/2539394744 conn(0x7f6adc0755a0 msgr2=0x7f6adc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:00.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 shutdown_connections 2024-04-16T02:57:00.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:00.963+0000 7f6ae3864700 1 -- 172.21.15.151:0/2539394744 wait complete. 2024-04-16T02:57:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.068+0000 7f4a73806700 1 -- 172.21.15.151:0/3130693281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 msgr2=0x7f4a6c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.068+0000 7f4a73806700 1 --2- 172.21.15.151:0/3130693281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 0x7f4a6c1011a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009910 tx=0x7f4a5c009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.068+0000 7f4a73806700 1 -- 172.21.15.151:0/3130693281 shutdown_connections 2024-04-16T02:57:01.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.068+0000 7f4a73806700 1 --2- 172.21.15.151:0/3130693281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c101770 0x7f4a6c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.068+0000 7f4a73806700 1 --2- 172.21.15.151:0/3130693281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 0x7f4a6c1011a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.068+0000 7f4a73806700 1 -- 172.21.15.151:0/3130693281 >> 172.21.15.151:0/3130693281 conn(0x7f4a6c0fc650 msgr2=0x7f4a6c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:01.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.068+0000 7f4a73806700 1 -- 172.21.15.151:0/3130693281 shutdown_connections 2024-04-16T02:57:01.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.068+0000 7f4a73806700 1 -- 172.21.15.151:0/3130693281 wait complete. 2024-04-16T02:57:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a73806700 1 Processor -- start 2024-04-16T02:57:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a73806700 1 -- start start 2024-04-16T02:57:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a73806700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 0x7f4a6c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a73806700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c101770 0x7f4a6c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a73806700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c19d110 con 0x7f4a6c100dc0 2024-04-16T02:57:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a73806700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c1965c0 con 0x7f4a6c101770 2024-04-16T02:57:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a715a2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 0x7f4a6c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a715a2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 0x7f4a6c19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42928/0 (socket says 172.21.15.151:42928) 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a715a2700 1 -- 172.21.15.151:0/2039574450 learned_addr learned my addr 172.21.15.151:0/2039574450 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a70da1700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c101770 0x7f4a6c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a715a2700 1 -- 172.21.15.151:0/2039574450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c101770 msgr2=0x7f4a6c19ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a715a2700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c101770 0x7f4a6c19ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.069+0000 7f4a715a2700 1 -- 172.21.15.151:0/2039574450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c0095f0 con 0x7f4a6c100dc0 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.070+0000 7f4a715a2700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 0x7f4a6c19c540 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c006010 tx=0x7f4a5c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.070+0000 7f4a627fc700 1 -- 172.21.15.151:0/2039574450 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5c01d070 con 0x7f4a6c100dc0 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.070+0000 7f4a627fc700 1 -- 172.21.15.151:0/2039574450 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a5c005dc0 con 0x7f4a6c100dc0 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.070+0000 7f4a627fc700 1 -- 172.21.15.151:0/2039574450 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5c017400 con 0x7f4a6c100dc0 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.070+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c196840 con 0x7f4a6c100dc0 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.070+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c196d30 con 0x7f4a6c100dc0 2024-04-16T02:57:01.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.071+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c109160 con 0x7f4a6c100dc0 2024-04-16T02:57:01.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.071+0000 7f4a627fc700 1 -- 172.21.15.151:0/2039574450 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4a5c003a60 con 0x7f4a6c100dc0 2024-04-16T02:57:01.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.072+0000 7f4a627fc700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a5806ecc0 0x7f4a58071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:01.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.072+0000 7f4a627fc700 1 -- 172.21.15.151:0/2039574450 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4a5c090200 con 0x7f4a6c100dc0 2024-04-16T02:57:01.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.072+0000 7f4a70da1700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a5806ecc0 0x7f4a58071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:01.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.072+0000 7f4a70da1700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a5806ecc0 0x7f4a58071180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c1059b0 tx=0x7f4a6800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:01.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.077+0000 7f4a627fc700 1 -- 172.21.15.151:0/2039574450 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4a5c05cb60 con 0x7f4a6c100dc0 2024-04-16T02:57:01.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.287+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a6c02ccc0 con 0x7f4a5806ecc0 2024-04-16T02:57:01.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.298+0000 7f4a627fc700 1 -- 172.21.15.151:0/2039574450 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4a6c02ccc0 con 0x7f4a5806ecc0 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (18m) 12m ago 29m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (30m) 12m ago 30m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (29m) 11m ago 29m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (16m) 12m ago 30m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (16m) 11m ago 29m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (18m) 12m ago 29m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (12m) 12m ago 24m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (12m) 12m ago 24m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (24m) 11m ago 24m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (11m) 11m ago 24m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (20m) 12m ago 31m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:57:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (19m) 11m ago 28m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (17m) 12m ago 31m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (16m) 11m ago 28m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (19m) 12m ago 30m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (19m) 11m ago 28m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (15m) 12m ago 28m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (15m) 12m ago 27m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (14m) 12m ago 27m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (13m) 11m ago 26m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (13m) 11m ago 26m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:57:01.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (12m) 11m ago 25m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:57:01.303 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (18m) 12m ago 29m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:57:01.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.300+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a5806ecc0 msgr2=0x7f4a58071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.300+0000 7f4a73806700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a5806ecc0 0x7f4a58071180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c1059b0 tx=0x7f4a6800b410 comp rx=0 tx=0).stop 2024-04-16T02:57:01.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.300+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 msgr2=0x7f4a6c19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.300+0000 7f4a73806700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 0x7f4a6c19c540 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c006010 tx=0x7f4a5c00bd70 comp rx=0 tx=0).stop 2024-04-16T02:57:01.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.300+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 shutdown_connections 2024-04-16T02:57:01.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.300+0000 7f4a73806700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a5806ecc0 0x7f4a58071180 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.300+0000 7f4a73806700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c100dc0 0x7f4a6c19c540 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.300+0000 7f4a73806700 1 --2- 172.21.15.151:0/2039574450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c101770 0x7f4a6c19ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.300+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 >> 172.21.15.151:0/2039574450 conn(0x7f4a6c0fc650 msgr2=0x7f4a6c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:01.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.301+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 shutdown_connections 2024-04-16T02:57:01.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.301+0000 7f4a73806700 1 -- 172.21.15.151:0/2039574450 wait complete. 2024-04-16T02:57:01.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 -- 172.21.15.151:0/2013101785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c00ffa70 msgr2=0x7f53c00ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 --2- 172.21.15.151:0/2013101785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c00ffa70 0x7f53c00ffef0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009910 tx=0x7f53b4009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:01.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 -- 172.21.15.151:0/2013101785 shutdown_connections 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 --2- 172.21.15.151:0/2013101785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c00ffa70 0x7f53c00ffef0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 --2- 172.21.15.151:0/2013101785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c0105a70 0x7f53c0105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 -- 172.21.15.151:0/2013101785 >> 172.21.15.151:0/2013101785 conn(0x7f53c00fb560 msgr2=0x7f53c00fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 -- 172.21.15.151:0/2013101785 shutdown_connections 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 -- 172.21.15.151:0/2013101785 wait complete. 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 Processor -- start 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.405+0000 7f53c60e6700 1 -- start start 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53c60e6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c00ffa70 0x7f53c00fbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53c60e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c0105a70 0x7f53c00fc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53c60e6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00fc9d0 con 0x7f53c0105a70 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53c60e6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00fcb40 con 0x7f53c00ffa70 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53beffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c0105a70 0x7f53c00fc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53beffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c0105a70 0x7f53c00fc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42952/0 (socket says 172.21.15.151:42952) 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53beffd700 1 -- 172.21.15.151:0/1287428359 learned_addr learned my addr 172.21.15.151:0/1287428359 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53bf7fe700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c00ffa70 0x7f53c00fbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53beffd700 1 -- 172.21.15.151:0/1287428359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c00ffa70 msgr2=0x7f53c00fbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53beffd700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c00ffa70 0x7f53c00fbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53beffd700 1 -- 172.21.15.151:0/1287428359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40095f0 con 0x7f53c0105a70 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53beffd700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c0105a70 0x7f53c00fc400 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f53b4006010 tx=0x7f53b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.406+0000 7f53bcff9700 1 -- 172.21.15.151:0/1287428359 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53b401d070 con 0x7f53c0105a70 2024-04-16T02:57:01.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.407+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c01a2910 con 0x7f53c0105a70 2024-04-16T02:57:01.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.407+0000 7f53bcff9700 1 -- 172.21.15.151:0/1287428359 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53b400f460 con 0x7f53c0105a70 2024-04-16T02:57:01.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.407+0000 7f53bcff9700 1 -- 172.21.15.151:0/1287428359 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53b4017630 con 0x7f53c0105a70 2024-04-16T02:57:01.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.407+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c01a2cc0 con 0x7f53c0105a70 2024-04-16T02:57:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.408+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c0109010 con 0x7f53c0105a70 2024-04-16T02:57:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.408+0000 7f53bcff9700 1 -- 172.21.15.151:0/1287428359 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f53b400f5d0 con 0x7f53c0105a70 2024-04-16T02:57:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.409+0000 7f53bcff9700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53a806ecc0 0x7f53a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.409+0000 7f53bcff9700 1 -- 172.21.15.151:0/1287428359 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f53b4090310 con 0x7f53c0105a70 2024-04-16T02:57:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.409+0000 7f53bf7fe700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53a806ecc0 0x7f53a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.409+0000 7f53bf7fe700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53a806ecc0 0x7f53a8071180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009d10 tx=0x7f53b0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.414+0000 7f53bcff9700 1 -- 172.21.15.151:0/1287428359 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f53b40590c0 con 0x7f53c0105a70 2024-04-16T02:57:01.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.698+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f53c004ea90 con 0x7f53c0105a70 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.698+0000 7f53bcff9700 1 -- 172.21.15.151:0/1287428359 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f53b405c780 con 0x7f53c0105a70 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:57:01.701 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:57:01.702 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:57:01.702 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:57:01.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:57:01.702 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:57:01.702 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:57:01.702 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:57:01.702 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:57:01.702 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:57:01.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53a806ecc0 msgr2=0x7f53a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53a806ecc0 0x7f53a8071180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009d10 tx=0x7f53b0009040 comp rx=0 tx=0).stop 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c0105a70 msgr2=0x7f53c00fc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c0105a70 0x7f53c00fc400 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f53b4006010 tx=0x7f53b400be20 comp rx=0 tx=0).stop 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 shutdown_connections 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53a806ecc0 0x7f53a8071180 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c00ffa70 0x7f53c00fbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 --2- 172.21.15.151:0/1287428359 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53c0105a70 0x7f53c00fc400 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 >> 172.21.15.151:0/1287428359 conn(0x7f53c00fb560 msgr2=0x7f53c00690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 shutdown_connections 2024-04-16T02:57:01.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.701+0000 7f53c60e6700 1 -- 172.21.15.151:0/1287428359 wait complete. 2024-04-16T02:57:01.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.805+0000 7f9bd68fc700 1 -- 172.21.15.151:0/1015521112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0101d30 msgr2=0x7f9bd0102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.805+0000 7f9bd68fc700 1 --2- 172.21.15.151:0/1015521112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0101d30 0x7f9bd0102110 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009910 tx=0x7f9bc0009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:01.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 -- 172.21.15.151:0/1015521112 shutdown_connections 2024-04-16T02:57:01.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 --2- 172.21.15.151:0/1015521112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd0102650 0x7f9bd010ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 --2- 172.21.15.151:0/1015521112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0101d30 0x7f9bd0102110 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 -- 172.21.15.151:0/1015521112 >> 172.21.15.151:0/1015521112 conn(0x7f9bd00fb560 msgr2=0x7f9bd00fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:01.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 -- 172.21.15.151:0/1015521112 shutdown_connections 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 -- 172.21.15.151:0/1015521112 wait complete. 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 Processor -- start 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 -- start start 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd0101d30 0x7f9bd019c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0102650 0x7f9bd019ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd019d120 con 0x7f9bd0102650 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.806+0000 7f9bd68fc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd01965d0 con 0x7f9bd0101d30 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bc7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0102650 0x7f9bd019ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bc7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0102650 0x7f9bd019ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42956/0 (socket says 172.21.15.151:42956) 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bc7fff700 1 -- 172.21.15.151:0/538141821 learned_addr learned my addr 172.21.15.151:0/538141821 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:01.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bcffff700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd0101d30 0x7f9bd019c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bc7fff700 1 -- 172.21.15.151:0/538141821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd0101d30 msgr2=0x7f9bd019c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bc7fff700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd0101d30 0x7f9bd019c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bc7fff700 1 -- 172.21.15.151:0/538141821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc00095f0 con 0x7f9bd0102650 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bcffff700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd0101d30 0x7f9bd019c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bc7fff700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0102650 0x7f9bd019ca90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800d8d0 tx=0x7f9bb800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bcdffb700 1 -- 172.21.15.151:0/538141821 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb80041f0 con 0x7f9bd0102650 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bcdffb700 1 -- 172.21.15.151:0/538141821 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bb8004350 con 0x7f9bd0102650 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd01968b0 con 0x7f9bd0102650 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bcdffb700 1 -- 172.21.15.151:0/538141821 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb8003d60 con 0x7f9bd0102650 2024-04-16T02:57:01.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.807+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd0196e00 con 0x7f9bd0102650 2024-04-16T02:57:01.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.808+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd0108240 con 0x7f9bd0102650 2024-04-16T02:57:01.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.809+0000 7f9bcdffb700 1 -- 172.21.15.151:0/538141821 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9bb8009730 con 0x7f9bd0102650 2024-04-16T02:57:01.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.809+0000 7f9bcdffb700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bbc06ecc0 0x7f9bbc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:01.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.809+0000 7f9bcdffb700 1 -- 172.21.15.151:0/538141821 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9bb808ea00 con 0x7f9bd0102650 2024-04-16T02:57:01.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.809+0000 7f9bcffff700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bbc06ecc0 0x7f9bbc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:01.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.810+0000 7f9bcffff700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bbc06ecc0 0x7f9bbc071180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000b5c0 tx=0x7f9bc00053a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:01.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:01.814+0000 7f9bcdffb700 1 -- 172.21.15.151:0/538141821 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9bb805b360 con 0x7f9bd0102650 2024-04-16T02:57:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:01 smithi151 ceph-mon[112233]: pgmap v551: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:01 smithi151 ceph-mon[112233]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:01 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1287428359' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:57:02.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.065+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9bd004ea90 con 0x7f9bd0102650 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.065+0000 7f9bcdffb700 1 -- 172.21.15.151:0/538141821 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f9bb8017070 con 0x7f9bd0102650 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:57:02.068 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:57:02.069 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:57:02.070 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:57:02.071 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:57:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:57:02.071 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:57:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:57:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:57:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:57:02.071 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:57:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.067+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bbc06ecc0 msgr2=0x7f9bbc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.067+0000 7f9bd68fc700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bbc06ecc0 0x7f9bbc071180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000b5c0 tx=0x7f9bc00053a0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.067+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0102650 msgr2=0x7f9bd019ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.067+0000 7f9bd68fc700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0102650 0x7f9bd019ca90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800d8d0 tx=0x7f9bb800dc90 comp rx=0 tx=0).stop 2024-04-16T02:57:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.068+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 shutdown_connections 2024-04-16T02:57:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.068+0000 7f9bd68fc700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9bbc06ecc0 0x7f9bbc071180 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.068+0000 7f9bd68fc700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd0101d30 0x7f9bd019c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.068+0000 7f9bd68fc700 1 --2- 172.21.15.151:0/538141821 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9bd0102650 0x7f9bd019ca90 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.068+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 >> 172.21.15.151:0/538141821 conn(0x7f9bd00fb560 msgr2=0x7f9bd0105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:02.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.068+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 shutdown_connections 2024-04-16T02:57:02.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.068+0000 7f9bd68fc700 1 -- 172.21.15.151:0/538141821 wait complete. 2024-04-16T02:57:02.073 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:57:02.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.174+0000 7fe43a740700 1 -- 172.21.15.151:0/3540382787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 msgr2=0x7fe434105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.174+0000 7fe43a740700 1 --2- 172.21.15.151:0/3540382787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 0x7fe434105870 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009910 tx=0x7fe41c009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:02.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.174+0000 7fe43a740700 1 -- 172.21.15.151:0/3540382787 shutdown_connections 2024-04-16T02:57:02.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.174+0000 7fe43a740700 1 --2- 172.21.15.151:0/3540382787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 0x7fe434105870 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.174+0000 7fe43a740700 1 --2- 172.21.15.151:0/3540382787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4340ff530 0x7fe4340ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.174+0000 7fe43a740700 1 -- 172.21.15.151:0/3540382787 >> 172.21.15.151:0/3540382787 conn(0x7fe434074df0 msgr2=0x7fe434075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:02.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.174+0000 7fe43a740700 1 -- 172.21.15.151:0/3540382787 shutdown_connections 2024-04-16T02:57:02.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.174+0000 7fe43a740700 1 -- 172.21.15.151:0/3540382787 wait complete. 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.175+0000 7fe43a740700 1 Processor -- start 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.175+0000 7fe43a740700 1 -- start start 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.175+0000 7fe43a740700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4340ff530 0x7fe43419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.175+0000 7fe43a740700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 0x7fe43419cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.175+0000 7fe43a740700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43419d170 con 0x7fe4340ffee0 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.175+0000 7fe43a740700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe434196620 con 0x7fe4340ff530 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.175+0000 7fe42bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 0x7fe43419cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe42bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 0x7fe43419cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42980/0 (socket says 172.21.15.151:42980) 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe42bfff700 1 -- 172.21.15.151:0/1999850083 learned_addr learned my addr 172.21.15.151:0/1999850083 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe433fff700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4340ff530 0x7fe43419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe42bfff700 1 -- 172.21.15.151:0/1999850083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4340ff530 msgr2=0x7fe43419c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe42bfff700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4340ff530 0x7fe43419c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe42bfff700 1 -- 172.21.15.151:0/1999850083 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0095f0 con 0x7fe4340ffee0 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe433fff700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4340ff530 0x7fe43419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe42bfff700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 0x7fe43419cae0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0092e0 tx=0x7fe41c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe431ffb700 1 -- 172.21.15.151:0/1999850083 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe41c01d070 con 0x7fe4340ffee0 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe431ffb700 1 -- 172.21.15.151:0/1999850083 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe41c005dc0 con 0x7fe4340ffee0 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4341968a0 con 0x7fe4340ffee0 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe431ffb700 1 -- 172.21.15.151:0/1999850083 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe41c017400 con 0x7fe4340ffee0 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.176+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe434196d90 con 0x7fe4340ffee0 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.177+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe434109150 con 0x7fe4340ffee0 2024-04-16T02:57:02.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.178+0000 7fe431ffb700 1 -- 172.21.15.151:0/1999850083 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe41c00f460 con 0x7fe4340ffee0 2024-04-16T02:57:02.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.178+0000 7fe431ffb700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe42006ecc0 0x7fe420071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:02.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.178+0000 7fe431ffb700 1 -- 172.21.15.151:0/1999850083 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe41c090420 con 0x7fe4340ffee0 2024-04-16T02:57:02.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.178+0000 7fe433fff700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe42006ecc0 0x7fe420071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:02.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.179+0000 7fe433fff700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe42006ecc0 0x7fe420071180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe424005950 tx=0x7fe4240058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:02.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.183+0000 7fe431ffb700 1 -- 172.21.15.151:0/1999850083 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe41c05cc60 con 0x7fe4340ffee0 2024-04-16T02:57:02.257 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:01 smithi153 ceph-mon[87858]: pgmap v551: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:02.258 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:01 smithi153 ceph-mon[87858]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:02.258 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:01 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1287428359' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:57:02.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.401+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe434196fb0 con 0x7fe42006ecc0 2024-04-16T02:57:02.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.403+0000 7fe431ffb700 1 -- 172.21.15.151:0/1999850083 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe434196fb0 con 0x7fe42006ecc0 2024-04-16T02:57:02.405 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:57:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:57:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:57:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:57:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:57:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:57:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:57:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:57:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:57:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:57:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:57:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:57:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:57:02.406 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:57:02.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe42006ecc0 msgr2=0x7fe420071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe42006ecc0 0x7fe420071180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe424005950 tx=0x7fe4240058e0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 msgr2=0x7fe43419cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 0x7fe43419cae0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0092e0 tx=0x7fe41c00be20 comp rx=0 tx=0).stop 2024-04-16T02:57:02.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 shutdown_connections 2024-04-16T02:57:02.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe42006ecc0 0x7fe420071180 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4340ff530 0x7fe43419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 --2- 172.21.15.151:0/1999850083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4340ffee0 0x7fe43419cae0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 >> 172.21.15.151:0/1999850083 conn(0x7fe434074df0 msgr2=0x7fe4341050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:02.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 shutdown_connections 2024-04-16T02:57:02.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.405+0000 7fe43a740700 1 -- 172.21.15.151:0/1999850083 wait complete. 2024-04-16T02:57:02.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.511+0000 7f3ec27cc700 1 -- 172.21.15.151:0/304506739 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 msgr2=0x7f3ebc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.511+0000 7f3ec27cc700 1 --2- 172.21.15.151:0/304506739 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 0x7f3ebc100050 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009910 tx=0x7f3eb0009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:02.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.511+0000 7f3ec27cc700 1 -- 172.21.15.151:0/304506739 shutdown_connections 2024-04-16T02:57:02.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.511+0000 7f3ec27cc700 1 --2- 172.21.15.151:0/304506739 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 0x7f3ebc100050 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.511+0000 7f3ec27cc700 1 --2- 172.21.15.151:0/304506739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ebc105a40 0x7f3ebc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.511+0000 7f3ec27cc700 1 -- 172.21.15.151:0/304506739 >> 172.21.15.151:0/304506739 conn(0x7f3ebc0755a0 msgr2=0x7f3ebc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:02.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.511+0000 7f3ec27cc700 1 -- 172.21.15.151:0/304506739 shutdown_connections 2024-04-16T02:57:02.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.511+0000 7f3ec27cc700 1 -- 172.21.15.151:0/304506739 wait complete. 2024-04-16T02:57:02.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ec27cc700 1 Processor -- start 2024-04-16T02:57:02.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ec27cc700 1 -- start start 2024-04-16T02:57:02.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ec27cc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 0x7f3ebc101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:02.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ec27cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ebc105a40 0x7f3ebc101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:02.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ec27cc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc102090 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ec27cc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc1021d0 con 0x7f3ebc105a40 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ebbfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 0x7f3ebc101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ebbfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 0x7f3ebc101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42992/0 (socket says 172.21.15.151:42992) 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ebbfff700 1 -- 172.21.15.151:0/1883447458 learned_addr learned my addr 172.21.15.151:0/1883447458 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ebb7fe700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ebc105a40 0x7f3ebc101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ebbfff700 1 -- 172.21.15.151:0/1883447458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ebc105a40 msgr2=0x7f3ebc101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ebbfff700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ebc105a40 0x7f3ebc101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.512+0000 7f3ebbfff700 1 -- 172.21.15.151:0/1883447458 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb00095f0 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.513+0000 7f3ebbfff700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 0x7f3ebc101610 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00b4b0 tx=0x7f3eac00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.513+0000 7f3eb97fa700 1 -- 172.21.15.151:0/1883447458 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3eac011800 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.513+0000 7f3eb97fa700 1 -- 172.21.15.151:0/1883447458 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3eac011e20 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.513+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ebc1a2a30 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.513+0000 7f3eb97fa700 1 -- 172.21.15.151:0/1883447458 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3eac00f2e0 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.513+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ebc1a2e40 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.514+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ebc109070 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.514+0000 7f3eb97fa700 1 -- 172.21.15.151:0/1883447458 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3eac010380 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.515+0000 7f3eb97fa700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3ea406ecc0 0x7f3ea4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:02.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.515+0000 7f3eb97fa700 1 -- 172.21.15.151:0/1883447458 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3eac08e8d0 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.515+0000 7f3ebb7fe700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3ea406ecc0 0x7f3ea4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:02.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.515+0000 7f3ebb7fe700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3ea406ecc0 0x7f3ea4071180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc102e00 tx=0x7f3eb001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:02.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.520+0000 7f3eb97fa700 1 -- 172.21.15.151:0/1883447458 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3eac057700 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.801+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3ebc04ea90 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.801+0000 7f3eb97fa700 1 -- 172.21.15.151:0/1883447458 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f3eac05adc0 con 0x7f3ebc0ffbd0 2024-04-16T02:57:02.804 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:57:02.804 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:57:02.804 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:57:02.804 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:57:02.804 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:57:02.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.803+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3ea406ecc0 msgr2=0x7f3ea4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.803+0000 7f3ec27cc700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3ea406ecc0 0x7f3ea4071180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc102e00 tx=0x7f3eb001a040 comp rx=0 tx=0).stop 2024-04-16T02:57:02.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.803+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 msgr2=0x7f3ebc101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:02.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.803+0000 7f3ec27cc700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 0x7f3ebc101610 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00b4b0 tx=0x7f3eac00b7c0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.804+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 shutdown_connections 2024-04-16T02:57:02.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.804+0000 7f3ec27cc700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3ea406ecc0 0x7f3ea4071180 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.804+0000 7f3ec27cc700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ebc0ffbd0 0x7f3ebc101610 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.804+0000 7f3ec27cc700 1 --2- 172.21.15.151:0/1883447458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ebc105a40 0x7f3ebc101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:02.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.804+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 >> 172.21.15.151:0/1883447458 conn(0x7f3ebc0755a0 msgr2=0x7f3ebc068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:02.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.804+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 shutdown_connections 2024-04-16T02:57:02.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:02.804+0000 7f3ec27cc700 1 -- 172.21.15.151:0/1883447458 wait complete. 2024-04-16T02:57:03.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:02 smithi151 ceph-mon[112233]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:02 smithi151 ceph-mon[112233]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:02 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/538141821' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:57:03.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:02 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1883447458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:57:03.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:02 smithi153 ceph-mon[87858]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:03.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:02 smithi153 ceph-mon[87858]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:03.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:02 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/538141821' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:57:03.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:02 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1883447458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:57:04.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:03 smithi151 ceph-mon[112233]: pgmap v552: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:04.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:03 smithi151 ceph-mon[112233]: from='client.35062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:03 smithi153 ceph-mon[87858]: pgmap v552: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:03 smithi153 ceph-mon[87858]: from='client.35062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:06.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:05 smithi151 ceph-mon[112233]: pgmap v553: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:06.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:05 smithi153 ceph-mon[87858]: pgmap v553: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:08.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:07 smithi151 ceph-mon[112233]: pgmap v554: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:08.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:07 smithi153 ceph-mon[87858]: pgmap v554: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:57:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:57:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:09 smithi151 ceph-mon[112233]: pgmap v555: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:10.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:09 smithi153 ceph-mon[87858]: pgmap v555: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:11.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:10 smithi151 ceph-mon[112233]: pgmap v556: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:10 smithi153 ceph-mon[87858]: pgmap v556: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:13.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:13 smithi153 ceph-mon[87858]: pgmap v557: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:13.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:13 smithi151 ceph-mon[112233]: pgmap v557: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:15.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:15 smithi153 ceph-mon[87858]: pgmap v558: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:15.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:15 smithi151 ceph-mon[112233]: pgmap v558: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:17.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:17 smithi153 ceph-mon[87858]: pgmap v559: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:17.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:17 smithi151 ceph-mon[112233]: pgmap v559: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:19.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:19 smithi153 ceph-mon[87858]: pgmap v560: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:19.727 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:19 smithi151 ceph-mon[112233]: pgmap v560: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:21.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:21 smithi151 ceph-mon[112233]: pgmap v561: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:21.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:21 smithi153 ceph-mon[87858]: pgmap v561: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:23.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:23 smithi153 ceph-mon[87858]: pgmap v562: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:23.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:23 smithi151 ceph-mon[112233]: pgmap v562: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:24.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:57:24.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:57:25.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:25 smithi151 ceph-mon[112233]: pgmap v563: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:25.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:25 smithi153 ceph-mon[87858]: pgmap v563: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:27.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:27 smithi153 ceph-mon[87858]: pgmap v564: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:27.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:27 smithi151 ceph-mon[112233]: pgmap v564: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:29.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:29 smithi151 ceph-mon[112233]: pgmap v565: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:29.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:29 smithi153 ceph-mon[87858]: pgmap v565: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:31.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:31 smithi151 ceph-mon[112233]: pgmap v566: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:31.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:31 smithi153 ceph-mon[87858]: pgmap v566: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:32.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.922+0000 7ff3ce714700 1 -- 172.21.15.151:0/1253464741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8100dc0 msgr2=0x7ff3c81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.922+0000 7ff3ce714700 1 --2- 172.21.15.151:0/1253464741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8100dc0 0x7ff3c81011a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009910 tx=0x7ff3b8009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.922+0000 7ff3ce714700 1 -- 172.21.15.151:0/1253464741 shutdown_connections 2024-04-16T02:57:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.922+0000 7ff3ce714700 1 --2- 172.21.15.151:0/1253464741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8101770 0x7ff3c81057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.922+0000 7ff3ce714700 1 --2- 172.21.15.151:0/1253464741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8100dc0 0x7ff3c81011a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.922+0000 7ff3ce714700 1 -- 172.21.15.151:0/1253464741 >> 172.21.15.151:0/1253464741 conn(0x7ff3c80fc650 msgr2=0x7ff3c80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.922+0000 7ff3ce714700 1 -- 172.21.15.151:0/1253464741 shutdown_connections 2024-04-16T02:57:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.922+0000 7ff3ce714700 1 -- 172.21.15.151:0/1253464741 wait complete. 2024-04-16T02:57:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.922+0000 7ff3ce714700 1 Processor -- start 2024-04-16T02:57:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3ce714700 1 -- start start 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3ce714700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8100dc0 0x7ff3c8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3ce714700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8101770 0x7ff3c80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3ce714700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c8073b10 con 0x7ff3c8101770 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3ce714700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c8073c50 con 0x7ff3c8100dc0 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3c77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8101770 0x7ff3c80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3c77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8101770 0x7ff3c80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56480/0 (socket says 172.21.15.151:56480) 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3c77fe700 1 -- 172.21.15.151:0/2612634862 learned_addr learned my addr 172.21.15.151:0/2612634862 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3c7fff700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8100dc0 0x7ff3c8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3c77fe700 1 -- 172.21.15.151:0/2612634862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8100dc0 msgr2=0x7ff3c8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3c77fe700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8100dc0 0x7ff3c8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3c77fe700 1 -- 172.21.15.151:0/2612634862 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b80095f0 con 0x7ff3c8101770 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.923+0000 7ff3c77fe700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8101770 0x7ff3c80735d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000d900 tx=0x7ff3b000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:32.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.924+0000 7ff3c57fa700 1 -- 172.21.15.151:0/2612634862 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b0009900 con 0x7ff3c8101770 2024-04-16T02:57:32.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.924+0000 7ff3c57fa700 1 -- 172.21.15.151:0/2612634862 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3b0010460 con 0x7ff3c8101770 2024-04-16T02:57:32.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.924+0000 7ff3c57fa700 1 -- 172.21.15.151:0/2612634862 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b000f5f0 con 0x7ff3c8101770 2024-04-16T02:57:32.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.924+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c81a2940 con 0x7ff3c8101770 2024-04-16T02:57:32.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.924+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c81a2d50 con 0x7ff3c8101770 2024-04-16T02:57:32.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.925+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c8109160 con 0x7ff3c8101770 2024-04-16T02:57:32.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.925+0000 7ff3c57fa700 1 -- 172.21.15.151:0/2612634862 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff3b00105d0 con 0x7ff3c8101770 2024-04-16T02:57:32.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.926+0000 7ff3c57fa700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3b406ecc0 0x7ff3b4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:32.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.926+0000 7ff3c57fa700 1 -- 172.21.15.151:0/2612634862 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff3b008e910 con 0x7ff3c8101770 2024-04-16T02:57:32.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.926+0000 7ff3c7fff700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3b406ecc0 0x7ff3b4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:32.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.926+0000 7ff3c7fff700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3b406ecc0 0x7ff3b4071180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800b5c0 tx=0x7ff3b801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:32.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:32.931+0000 7ff3c57fa700 1 -- 172.21.15.151:0/2612634862 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff3b0057740 con 0x7ff3c8101770 2024-04-16T02:57:33.150 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.148+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3c802cda0 con 0x7ff3b406ecc0 2024-04-16T02:57:33.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.150+0000 7ff3c57fa700 1 -- 172.21.15.151:0/2612634862 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff3c802cda0 con 0x7ff3b406ecc0 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3b406ecc0 msgr2=0x7ff3b4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3b406ecc0 0x7ff3b4071180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800b5c0 tx=0x7ff3b801a040 comp rx=0 tx=0).stop 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8101770 msgr2=0x7ff3c80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8101770 0x7ff3c80735d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000d900 tx=0x7ff3b000dc10 comp rx=0 tx=0).stop 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 shutdown_connections 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff3b406ecc0 0x7ff3b4071180 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8100dc0 0x7ff3c8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 --2- 172.21.15.151:0/2612634862 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3c8101770 0x7ff3c80735d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 >> 172.21.15.151:0/2612634862 conn(0x7ff3c80fc650 msgr2=0x7ff3c80fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:33.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 shutdown_connections 2024-04-16T02:57:33.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.152+0000 7ff3ce714700 1 -- 172.21.15.151:0/2612634862 wait complete. 2024-04-16T02:57:33.165 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:57:33.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.258+0000 7efcb4712700 1 -- 172.21.15.151:0/3507463926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac101770 msgr2=0x7efcac1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.258+0000 7efcb4712700 1 --2- 172.21.15.151:0/3507463926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac101770 0x7efcac1057c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7efca0009910 tx=0x7efca0009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.258+0000 7efcb4712700 1 -- 172.21.15.151:0/3507463926 shutdown_connections 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.258+0000 7efcb4712700 1 --2- 172.21.15.151:0/3507463926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac101770 0x7efcac1057c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.258+0000 7efcb4712700 1 --2- 172.21.15.151:0/3507463926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcac100dc0 0x7efcac1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.258+0000 7efcb4712700 1 -- 172.21.15.151:0/3507463926 >> 172.21.15.151:0/3507463926 conn(0x7efcac0fc670 msgr2=0x7efcac0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.259+0000 7efcb4712700 1 -- 172.21.15.151:0/3507463926 shutdown_connections 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.259+0000 7efcb4712700 1 -- 172.21.15.151:0/3507463926 wait complete. 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.259+0000 7efcb4712700 1 Processor -- start 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.259+0000 7efcb4712700 1 -- start start 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.259+0000 7efcb4712700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac100dc0 0x7efcac19c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.259+0000 7efcb4712700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcac101770 0x7efcac19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.259+0000 7efcb4712700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac19d150 con 0x7efcac100dc0 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.259+0000 7efcb4712700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac196600 con 0x7efcac101770 2024-04-16T02:57:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.259+0000 7efcb24ae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac100dc0 0x7efcac19c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcb24ae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac100dc0 0x7efcac19c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56506/0 (socket says 172.21.15.151:56506) 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcb24ae700 1 -- 172.21.15.151:0/3347081659 learned_addr learned my addr 172.21.15.151:0/3347081659 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcb24ae700 1 -- 172.21.15.151:0/3347081659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcac101770 msgr2=0x7efcac19cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcb1cad700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcac101770 0x7efcac19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcb24ae700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcac101770 0x7efcac19cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcb24ae700 1 -- 172.21.15.151:0/3347081659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca00095f0 con 0x7efcac100dc0 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcb24ae700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac100dc0 0x7efcac19c580 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0060b0 tx=0x7efc9c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcab7fe700 1 -- 172.21.15.151:0/3347081659 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc9c00dda0 con 0x7efcac100dc0 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcab7fe700 1 -- 172.21.15.151:0/3347081659 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc9c00f040 con 0x7efcac100dc0 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcac1968e0 con 0x7efcac100dc0 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcab7fe700 1 -- 172.21.15.151:0/3347081659 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc9c014790 con 0x7efcac100dc0 2024-04-16T02:57:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.260+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcac196e30 con 0x7efcac100dc0 2024-04-16T02:57:33.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.261+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcac1091c0 con 0x7efcac100dc0 2024-04-16T02:57:33.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.262+0000 7efcab7fe700 1 -- 172.21.15.151:0/3347081659 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efc9c0148f0 con 0x7efcac100dc0 2024-04-16T02:57:33.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.262+0000 7efcab7fe700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc9406ecc0 0x7efc94071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:33.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.262+0000 7efcab7fe700 1 -- 172.21.15.151:0/3347081659 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efc9c08e830 con 0x7efcac100dc0 2024-04-16T02:57:33.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.262+0000 7efcb1cad700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc9406ecc0 0x7efc94071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:33.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.263+0000 7efcb1cad700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc9406ecc0 0x7efc94071180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efca0006010 tx=0x7efca000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:33.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.267+0000 7efcab7fe700 1 -- 172.21.15.151:0/3347081659 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efc9c05b240 con 0x7efcac100dc0 2024-04-16T02:57:33.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.487+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efcac197150 con 0x7efc9406ecc0 2024-04-16T02:57:33.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.488+0000 7efcab7fe700 1 -- 172.21.15.151:0/3347081659 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efcac197150 con 0x7efc9406ecc0 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc9406ecc0 msgr2=0x7efc94071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc9406ecc0 0x7efc94071180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efca0006010 tx=0x7efca000b540 comp rx=0 tx=0).stop 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac100dc0 msgr2=0x7efcac19c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac100dc0 0x7efcac19c580 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0060b0 tx=0x7efc9c00d3b0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 shutdown_connections 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efc9406ecc0 0x7efc94071180 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efcac100dc0 0x7efcac19c580 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 --2- 172.21.15.151:0/3347081659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcac101770 0x7efcac19cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 >> 172.21.15.151:0/3347081659 conn(0x7efcac0fc670 msgr2=0x7efcac0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 shutdown_connections 2024-04-16T02:57:33.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.491+0000 7efcb4712700 1 -- 172.21.15.151:0/3347081659 wait complete. 2024-04-16T02:57:33.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.596+0000 7f757a37a700 1 -- 172.21.15.151:0/3843786085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 msgr2=0x7f7574102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.596+0000 7f757a37a700 1 --2- 172.21.15.151:0/3843786085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 0x7f7574102110 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7564009910 tx=0x7f7564009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:33.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.596+0000 7f757a37a700 1 -- 172.21.15.151:0/3843786085 shutdown_connections 2024-04-16T02:57:33.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.596+0000 7f757a37a700 1 --2- 172.21.15.151:0/3843786085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7574102650 0x7f757410ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.596+0000 7f757a37a700 1 --2- 172.21.15.151:0/3843786085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 0x7f7574102110 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.596+0000 7f757a37a700 1 -- 172.21.15.151:0/3843786085 >> 172.21.15.151:0/3843786085 conn(0x7f75740fb560 msgr2=0x7f75740fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:33.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.596+0000 7f757a37a700 1 -- 172.21.15.151:0/3843786085 shutdown_connections 2024-04-16T02:57:33.599 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:33 smithi151 ceph-mon[112233]: pgmap v567: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:33.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.596+0000 7f757a37a700 1 -- 172.21.15.151:0/3843786085 wait complete. 2024-04-16T02:57:33.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.597+0000 7f757a37a700 1 Processor -- start 2024-04-16T02:57:33.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.597+0000 7f757a37a700 1 -- start start 2024-04-16T02:57:33.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.597+0000 7f757a37a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 0x7f75740ff990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:33.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.597+0000 7f757a37a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7574102650 0x7f75740ffed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.597+0000 7f757a37a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7574100410 con 0x7f7574101d30 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.597+0000 7f757a37a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7574100550 con 0x7f7574102650 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f7573fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 0x7f75740ff990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f7573fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 0x7f75740ff990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56528/0 (socket says 172.21.15.151:56528) 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f7573fff700 1 -- 172.21.15.151:0/237266346 learned_addr learned my addr 172.21.15.151:0/237266346 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f75737fe700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7574102650 0x7f75740ffed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f7573fff700 1 -- 172.21.15.151:0/237266346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7574102650 msgr2=0x7f75740ffed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f7573fff700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7574102650 0x7f75740ffed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f7573fff700 1 -- 172.21.15.151:0/237266346 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75640095f0 con 0x7f7574101d30 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f75737fe700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7574102650 0x7f75740ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f7573fff700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 0x7f75740ff990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7564006010 tx=0x7f756400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f75717fa700 1 -- 172.21.15.151:0/237266346 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f756401d070 con 0x7f7574101d30 2024-04-16T02:57:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f75717fa700 1 -- 172.21.15.151:0/237266346 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7564004fb0 con 0x7f7574101d30 2024-04-16T02:57:33.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75741007d0 con 0x7f7574101d30 2024-04-16T02:57:33.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f75717fa700 1 -- 172.21.15.151:0/237266346 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f756400f690 con 0x7f7574101d30 2024-04-16T02:57:33.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.598+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75741a2c80 con 0x7f7574101d30 2024-04-16T02:57:33.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.599+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7574108240 con 0x7f7574101d30 2024-04-16T02:57:33.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.600+0000 7f75717fa700 1 -- 172.21.15.151:0/237266346 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f75640228a0 con 0x7f7574101d30 2024-04-16T02:57:33.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.601+0000 7f75717fa700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f755c06ecc0 0x7f755c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:33.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.601+0000 7f75717fa700 1 -- 172.21.15.151:0/237266346 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7564090d30 con 0x7f7574101d30 2024-04-16T02:57:33.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.601+0000 7f75737fe700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f755c06ecc0 0x7f755c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:33.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.601+0000 7f75737fe700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f755c06ecc0 0x7f755c071180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7568005fd0 tx=0x7f7568005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:33.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.605+0000 7f75717fa700 1 -- 172.21.15.151:0/237266346 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7564059c00 con 0x7f7574101d30 2024-04-16T02:57:33.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:33 smithi153 ceph-mon[87858]: pgmap v567: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:33.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.813+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f757402cd80 con 0x7f755c06ecc0 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.824+0000 7f75717fa700 1 -- 172.21.15.151:0/237266346 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f757402cd80 con 0x7f755c06ecc0 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (19m) 12m ago 30m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (30m) 12m ago 30m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (29m) 12m ago 29m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (16m) 12m ago 30m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (16m) 12m ago 29m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (18m) 12m ago 30m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (12m) 12m ago 24m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (12m) 12m ago 24m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:57:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (24m) 12m ago 24m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (12m) 12m ago 24m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (21m) 12m ago 31m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (20m) 12m ago 29m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (17m) 12m ago 31m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (17m) 12m ago 29m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (19m) 12m ago 30m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (19m) 12m ago 29m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (16m) 12m ago 28m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (15m) 12m ago 28m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (15m) 12m ago 27m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (14m) 12m ago 27m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (13m) 12m ago 26m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (13m) 12m ago 26m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:57:33.827 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (19m) 12m ago 29m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:57:33.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f755c06ecc0 msgr2=0x7f755c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f755c06ecc0 0x7f755c071180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7568005fd0 tx=0x7f7568005dc0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 msgr2=0x7f75740ff990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 0x7f75740ff990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7564006010 tx=0x7f756400be20 comp rx=0 tx=0).stop 2024-04-16T02:57:33.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 shutdown_connections 2024-04-16T02:57:33.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f755c06ecc0 0x7f755c071180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7574101d30 0x7f75740ff990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 --2- 172.21.15.151:0/237266346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7574102650 0x7f75740ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 >> 172.21.15.151:0/237266346 conn(0x7f75740fb560 msgr2=0x7f7574105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:33.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 shutdown_connections 2024-04-16T02:57:33.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.826+0000 7f757a37a700 1 -- 172.21.15.151:0/237266346 wait complete. 2024-04-16T02:57:33.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.930+0000 7f778ef05700 1 -- 172.21.15.151:0/707126188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 msgr2=0x7f7788068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.930+0000 7f778ef05700 1 --2- 172.21.15.151:0/707126188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 0x7f7788068a50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f777c009910 tx=0x7f777c009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:33.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.930+0000 7f778ef05700 1 -- 172.21.15.151:0/707126188 shutdown_connections 2024-04-16T02:57:33.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.930+0000 7f778ef05700 1 --2- 172.21.15.151:0/707126188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7788069020 0x7f7788105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.930+0000 7f778ef05700 1 --2- 172.21.15.151:0/707126188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 0x7f7788068a50 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.930+0000 7f778ef05700 1 -- 172.21.15.151:0/707126188 >> 172.21.15.151:0/707126188 conn(0x7f77880755a0 msgr2=0x7f77880759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:33.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.931+0000 7f778ef05700 1 -- 172.21.15.151:0/707126188 shutdown_connections 2024-04-16T02:57:33.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.931+0000 7f778ef05700 1 -- 172.21.15.151:0/707126188 wait complete. 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.931+0000 7f778ef05700 1 Processor -- start 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.931+0000 7f778ef05700 1 -- start start 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.931+0000 7f778ef05700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 0x7f77880ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.931+0000 7f778ef05700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7788069020 0x7f77881001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.931+0000 7f778ef05700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7788100720 con 0x7f7788068670 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.931+0000 7f778ef05700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7788100860 con 0x7f7788069020 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f778cca1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 0x7f77880ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f778cca1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 0x7f77880ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56552/0 (socket says 172.21.15.151:56552) 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f778cca1700 1 -- 172.21.15.151:0/726405731 learned_addr learned my addr 172.21.15.151:0/726405731 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:33.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f7787fff700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7788069020 0x7f77881001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f778cca1700 1 -- 172.21.15.151:0/726405731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7788069020 msgr2=0x7f77881001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f778cca1700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7788069020 0x7f77881001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f778cca1700 1 -- 172.21.15.151:0/726405731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0095f0 con 0x7f7788068670 2024-04-16T02:57:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f778cca1700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 0x7f77880ffca0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f777c006010 tx=0x7f777c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f7785ffb700 1 -- 172.21.15.151:0/726405731 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f777c01d070 con 0x7f7788068670 2024-04-16T02:57:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f7785ffb700 1 -- 172.21.15.151:0/726405731 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f777c004fb0 con 0x7f7788068670 2024-04-16T02:57:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f7785ffb700 1 -- 172.21.15.151:0/726405731 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f777c022ea0 con 0x7f7788068670 2024-04-16T02:57:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.932+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7788100ae0 con 0x7f7788068670 2024-04-16T02:57:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.933+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77881a2ce0 con 0x7f7788068670 2024-04-16T02:57:33.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.933+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7788109130 con 0x7f7788068670 2024-04-16T02:57:33.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.934+0000 7f7785ffb700 1 -- 172.21.15.151:0/726405731 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f777c022470 con 0x7f7788068670 2024-04-16T02:57:33.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.935+0000 7f7785ffb700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f777806ecc0 0x7f7778071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:33.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.935+0000 7f7785ffb700 1 -- 172.21.15.151:0/726405731 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f777c090300 con 0x7f7788068670 2024-04-16T02:57:33.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.935+0000 7f7787fff700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f777806ecc0 0x7f7778071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:33.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.935+0000 7f7787fff700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f777806ecc0 0x7f7778071180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7788101490 tx=0x7f7774005eb0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:33.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:33.939+0000 7f7785ffb700 1 -- 172.21.15.151:0/726405731 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f777c059300 con 0x7f7788068670 2024-04-16T02:57:34.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.224+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f778804ea90 con 0x7f7788068670 2024-04-16T02:57:34.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.224+0000 7f7785ffb700 1 -- 172.21.15.151:0/726405731 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f777c027340 con 0x7f7788068670 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:57:34.227 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:57:34.228 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:57:34.228 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:57:34.228 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:57:34.228 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:57:34.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f777806ecc0 msgr2=0x7f7778071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f777806ecc0 0x7f7778071180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7788101490 tx=0x7f7774005eb0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 msgr2=0x7f77880ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 0x7f77880ffca0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f777c006010 tx=0x7f777c00be20 comp rx=0 tx=0).stop 2024-04-16T02:57:34.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 shutdown_connections 2024-04-16T02:57:34.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f777806ecc0 0x7f7778071180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7788068670 0x7f77880ffca0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 --2- 172.21.15.151:0/726405731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7788069020 0x7f77881001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 >> 172.21.15.151:0/726405731 conn(0x7f77880755a0 msgr2=0x7f77880fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:34.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 shutdown_connections 2024-04-16T02:57:34.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.227+0000 7f778ef05700 1 -- 172.21.15.151:0/726405731 wait complete. 2024-04-16T02:57:34.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.333+0000 7fe3b10f8700 1 -- 172.21.15.151:0/4291553021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac105a40 msgr2=0x7fe3ac105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.333+0000 7fe3b10f8700 1 --2- 172.21.15.151:0/4291553021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac105a40 0x7fe3ac105e20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009910 tx=0x7fe3a0009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:34.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.333+0000 7fe3b10f8700 1 -- 172.21.15.151:0/4291553021 shutdown_connections 2024-04-16T02:57:34.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.333+0000 7fe3b10f8700 1 --2- 172.21.15.151:0/4291553021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ac0ffa40 0x7fe3ac0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.333+0000 7fe3b10f8700 1 --2- 172.21.15.151:0/4291553021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac105a40 0x7fe3ac105e20 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.333+0000 7fe3b10f8700 1 -- 172.21.15.151:0/4291553021 >> 172.21.15.151:0/4291553021 conn(0x7fe3ac0fb570 msgr2=0x7fe3ac0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:34.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.333+0000 7fe3b10f8700 1 -- 172.21.15.151:0/4291553021 shutdown_connections 2024-04-16T02:57:34.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.333+0000 7fe3b10f8700 1 -- 172.21.15.151:0/4291553021 wait complete. 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3b10f8700 1 Processor -- start 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3b10f8700 1 -- start start 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3b10f8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac0ffa40 0x7fe3ac0fbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3b10f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ac105a40 0x7fe3ac0fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3b10f8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac0fc960 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3b10f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac0fcaa0 con 0x7fe3ac105a40 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3aad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac0ffa40 0x7fe3ac0fbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3aad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac0ffa40 0x7fe3ac0fbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56570/0 (socket says 172.21.15.151:56570) 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3aad9d700 1 -- 172.21.15.151:0/2276521746 learned_addr learned my addr 172.21.15.151:0/2276521746 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.334+0000 7fe3aa59c700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ac105a40 0x7fe3ac0fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe3aad9d700 1 -- 172.21.15.151:0/2276521746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ac105a40 msgr2=0x7fe3ac0fc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe3aad9d700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ac105a40 0x7fe3ac0fc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe3aad9d700 1 -- 172.21.15.151:0/2276521746 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a00095f0 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe3aa59c700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ac105a40 0x7fe3ac0fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe3aad9d700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac0ffa40 0x7fe3ac0fbee0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0006010 tx=0x7fe3a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe393fff700 1 -- 172.21.15.151:0/2276521746 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3a001d070 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ac0fcd20 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe393fff700 1 -- 172.21.15.151:0/2276521746 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3a0004fb0 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe393fff700 1 -- 172.21.15.151:0/2276521746 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3a0022ea0 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.335+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac1a2d30 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.336+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ac109050 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.336+0000 7fe393fff700 1 -- 172.21.15.151:0/2276521746 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe3a0022470 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.337+0000 7fe393fff700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe39406ecc0 0x7fe394071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.337+0000 7fe393fff700 1 -- 172.21.15.151:0/2276521746 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe3a00903d0 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.337+0000 7fe3aa59c700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe39406ecc0 0x7fe394071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.338+0000 7fe3aa59c700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe39406ecc0 0x7fe394071180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0fd6d0 tx=0x7fe39c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:34.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.342+0000 7fe393fff700 1 -- 172.21.15.151:0/2276521746 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe3a0059230 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:34 smithi151 ceph-mon[112233]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:34.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:34 smithi151 ceph-mon[112233]: from='client.35074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:34.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:34 smithi151 ceph-mon[112233]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:34.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:34 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/726405731' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:57:34.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.585+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe3ac04ea90 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.585+0000 7fe393fff700 1 -- 172.21.15.151:0/2276521746 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7fe3a0022720 con 0x7fe3ac0ffa40 2024-04-16T02:57:34.587 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:57:34.587 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:57:34.588 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:57:34.589 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:57:34.590 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:57:34.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.587+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe39406ecc0 msgr2=0x7fe394071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.587+0000 7fe3b10f8700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe39406ecc0 0x7fe394071180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0fd6d0 tx=0x7fe39c008040 comp rx=0 tx=0).stop 2024-04-16T02:57:34.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.587+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac0ffa40 msgr2=0x7fe3ac0fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.587+0000 7fe3b10f8700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac0ffa40 0x7fe3ac0fbee0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0006010 tx=0x7fe3a000be20 comp rx=0 tx=0).stop 2024-04-16T02:57:34.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.587+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 shutdown_connections 2024-04-16T02:57:34.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.587+0000 7fe3b10f8700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe39406ecc0 0x7fe394071180 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.587+0000 7fe3b10f8700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3ac0ffa40 0x7fe3ac0fbee0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.587+0000 7fe3b10f8700 1 --2- 172.21.15.151:0/2276521746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ac105a40 0x7fe3ac0fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.587+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 >> 172.21.15.151:0/2276521746 conn(0x7fe3ac0fb570 msgr2=0x7fe3ac068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:34.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.588+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 shutdown_connections 2024-04-16T02:57:34.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.588+0000 7fe3b10f8700 1 -- 172.21.15.151:0/2276521746 wait complete. 2024-04-16T02:57:34.592 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:57:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 -- 172.21.15.151:0/194137542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 msgr2=0x7fdc601011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 --2- 172.21.15.151:0/194137542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 0x7fdc601011a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdc54009910 tx=0x7fdc54009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 -- 172.21.15.151:0/194137542 shutdown_connections 2024-04-16T02:57:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 --2- 172.21.15.151:0/194137542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc60101770 0x7fdc601057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 --2- 172.21.15.151:0/194137542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 0x7fdc601011a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 -- 172.21.15.151:0/194137542 >> 172.21.15.151:0/194137542 conn(0x7fdc600fc650 msgr2=0x7fdc600fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 -- 172.21.15.151:0/194137542 shutdown_connections 2024-04-16T02:57:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 -- 172.21.15.151:0/194137542 wait complete. 2024-04-16T02:57:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 Processor -- start 2024-04-16T02:57:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.694+0000 7fdc66c5f700 1 -- start start 2024-04-16T02:57:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc66c5f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 0x7fdc60073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc66c5f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc60101770 0x7fdc600735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc66c5f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc60073b10 con 0x7fdc60100dc0 2024-04-16T02:57:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc66c5f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc60073c50 con 0x7fdc60101770 2024-04-16T02:57:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc649fb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 0x7fdc60073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc649fb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 0x7fdc60073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56594/0 (socket says 172.21.15.151:56594) 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc649fb700 1 -- 172.21.15.151:0/2999093494 learned_addr learned my addr 172.21.15.151:0/2999093494 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc5ffff700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc60101770 0x7fdc600735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc649fb700 1 -- 172.21.15.151:0/2999093494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc60101770 msgr2=0x7fdc600735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc649fb700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc60101770 0x7fdc600735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc649fb700 1 -- 172.21.15.151:0/2999093494 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc540095f0 con 0x7fdc60100dc0 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc649fb700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 0x7fdc60073090 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdc54006010 tx=0x7fdc5400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc5dffb700 1 -- 172.21.15.151:0/2999093494 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc5401d070 con 0x7fdc60100dc0 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.695+0000 7fdc5dffb700 1 -- 172.21.15.151:0/2999093494 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc54005dc0 con 0x7fdc60100dc0 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.696+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc60073ed0 con 0x7fdc60100dc0 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.696+0000 7fdc5dffb700 1 -- 172.21.15.151:0/2999093494 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc54017400 con 0x7fdc60100dc0 2024-04-16T02:57:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.696+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc601a2be0 con 0x7fdc60100dc0 2024-04-16T02:57:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.697+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc60109160 con 0x7fdc60100dc0 2024-04-16T02:57:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.697+0000 7fdc5dffb700 1 -- 172.21.15.151:0/2999093494 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fdc54003a60 con 0x7fdc60100dc0 2024-04-16T02:57:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.698+0000 7fdc5dffb700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdc5006ecc0 0x7fdc50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.698+0000 7fdc5dffb700 1 -- 172.21.15.151:0/2999093494 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdc54090200 con 0x7fdc60100dc0 2024-04-16T02:57:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.698+0000 7fdc5ffff700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdc5006ecc0 0x7fdc50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.698+0000 7fdc5ffff700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdc5006ecc0 0x7fdc50071180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdc60074810 tx=0x7fdc4c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.702+0000 7fdc5dffb700 1 -- 172.21.15.151:0/2999093494 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdc54059030 con 0x7fdc60100dc0 2024-04-16T02:57:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:34 smithi153 ceph-mon[87858]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:34 smithi153 ceph-mon[87858]: from='client.35074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:34 smithi153 ceph-mon[87858]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:34.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:34 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/726405731' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:57:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.926+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc600611d0 con 0x7fdc5006ecc0 2024-04-16T02:57:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.927+0000 7fdc5dffb700 1 -- 172.21.15.151:0/2999093494 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdc600611d0 con 0x7fdc5006ecc0 2024-04-16T02:57:34.929 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:57:34.929 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:57:34.929 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:57:34.929 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:57:34.929 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:57:34.929 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:57:34.929 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:57:34.930 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:57:34.930 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:57:34.930 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:57:34.930 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:57:34.930 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:57:34.930 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:57:34.930 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:57:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.929+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdc5006ecc0 msgr2=0x7fdc50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.929+0000 7fdc66c5f700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdc5006ecc0 0x7fdc50071180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdc60074810 tx=0x7fdc4c00b410 comp rx=0 tx=0).stop 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.929+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 msgr2=0x7fdc60073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.929+0000 7fdc66c5f700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 0x7fdc60073090 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdc54006010 tx=0x7fdc5400bd70 comp rx=0 tx=0).stop 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.929+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 shutdown_connections 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.929+0000 7fdc66c5f700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdc5006ecc0 0x7fdc50071180 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.929+0000 7fdc66c5f700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc60100dc0 0x7fdc60073090 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.929+0000 7fdc66c5f700 1 --2- 172.21.15.151:0/2999093494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc60101770 0x7fdc600735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.929+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 >> 172.21.15.151:0/2999093494 conn(0x7fdc600fc650 msgr2=0x7fdc600fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.930+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 shutdown_connections 2024-04-16T02:57:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:34.930+0000 7fdc66c5f700 1 -- 172.21.15.151:0/2999093494 wait complete. 2024-04-16T02:57:35.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.035+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3584044146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac101770 msgr2=0x7f0eac1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:35.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.035+0000 7f0eb0f03700 1 --2- 172.21.15.151:0/3584044146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac101770 0x7f0eac1057c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009910 tx=0x7f0e94009c20 comp rx=0 tx=0).stop 2024-04-16T02:57:35.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.035+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3584044146 shutdown_connections 2024-04-16T02:57:35.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.035+0000 7f0eb0f03700 1 --2- 172.21.15.151:0/3584044146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac101770 0x7f0eac1057c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:35.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.035+0000 7f0eb0f03700 1 --2- 172.21.15.151:0/3584044146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac100dc0 0x7f0eac1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:35.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.035+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3584044146 >> 172.21.15.151:0/3584044146 conn(0x7f0eac0fc650 msgr2=0x7f0eac0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:35.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.035+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3584044146 shutdown_connections 2024-04-16T02:57:35.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.035+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3584044146 wait complete. 2024-04-16T02:57:35.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0eb0f03700 1 Processor -- start 2024-04-16T02:57:35.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0eb0f03700 1 -- start start 2024-04-16T02:57:35.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0eb0f03700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac100dc0 0x7f0eac19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:35.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0eb0f03700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac101770 0x7f0eac19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0eb0f03700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac19d130 con 0x7f0eac100dc0 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0eb0f03700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac1965e0 con 0x7f0eac101770 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0eaa59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac100dc0 0x7f0eac19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0eaa59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac100dc0 0x7f0eac19c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56604/0 (socket says 172.21.15.151:56604) 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0eaa59c700 1 -- 172.21.15.151:0/3084148853 learned_addr learned my addr 172.21.15.151:0/3084148853 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.036+0000 7f0ea1d9b700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac101770 0x7f0eac19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.037+0000 7f0eaa59c700 1 -- 172.21.15.151:0/3084148853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac101770 msgr2=0x7f0eac19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.037+0000 7f0eaa59c700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac101770 0x7f0eac19caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.037+0000 7f0eaa59c700 1 -- 172.21.15.151:0/3084148853 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e940095f0 con 0x7f0eac100dc0 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.037+0000 7f0eaa59c700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac100dc0 0x7f0eac19c560 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00ada0 tx=0x7f0e9c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.037+0000 7f0ea3fff700 1 -- 172.21.15.151:0/3084148853 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e9c00f070 con 0x7f0eac100dc0 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.037+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac1968c0 con 0x7f0eac100dc0 2024-04-16T02:57:35.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.037+0000 7f0ea3fff700 1 -- 172.21.15.151:0/3084148853 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e9c00ae90 con 0x7f0eac100dc0 2024-04-16T02:57:35.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.037+0000 7f0ea3fff700 1 -- 172.21.15.151:0/3084148853 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e9c005690 con 0x7f0eac100dc0 2024-04-16T02:57:35.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.037+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac196e10 con 0x7f0eac100dc0 2024-04-16T02:57:35.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.038+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eac1091c0 con 0x7f0eac100dc0 2024-04-16T02:57:35.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.038+0000 7f0ea3fff700 1 -- 172.21.15.151:0/3084148853 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f0e9c005900 con 0x7f0eac100dc0 2024-04-16T02:57:35.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.039+0000 7f0ea3fff700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0e9806ecc0 0x7f0e98071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:57:35.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.039+0000 7f0ea3fff700 1 -- 172.21.15.151:0/3084148853 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0e9c08e7e0 con 0x7f0eac100dc0 2024-04-16T02:57:35.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.039+0000 7f0ea1d9b700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0e9806ecc0 0x7f0e98071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:57:35.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.039+0000 7f0ea1d9b700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0e9806ecc0 0x7f0e98071180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0fcea0 tx=0x7f0e9401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:57:35.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.044+0000 7f0ea3fff700 1 -- 172.21.15.151:0/3084148853 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0e9c05b0c0 con 0x7f0eac100dc0 2024-04-16T02:57:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.323+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0eac1970b0 con 0x7f0eac100dc0 2024-04-16T02:57:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.324+0000 7f0ea3fff700 1 -- 172.21.15.151:0/3084148853 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f0e9c055e60 con 0x7f0eac100dc0 2024-04-16T02:57:35.326 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:57:35.326 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:57:35.326 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:57:35.326 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:57:35.326 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:57:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0e9806ecc0 msgr2=0x7f0e98071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0e9806ecc0 0x7f0e98071180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0fcea0 tx=0x7f0e9401a040 comp rx=0 tx=0).stop 2024-04-16T02:57:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac100dc0 msgr2=0x7f0eac19c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:57:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac100dc0 0x7f0eac19c560 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00ada0 tx=0x7f0e9c00c330 comp rx=0 tx=0).stop 2024-04-16T02:57:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 shutdown_connections 2024-04-16T02:57:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0e9806ecc0 0x7f0e98071180 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0eac100dc0 0x7f0eac19c560 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 --2- 172.21.15.151:0/3084148853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac101770 0x7f0eac19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:57:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 >> 172.21.15.151:0/3084148853 conn(0x7f0eac0fc650 msgr2=0x7f0eac0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:57:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 shutdown_connections 2024-04-16T02:57:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:57:35.326+0000 7f0eb0f03700 1 -- 172.21.15.151:0/3084148853 wait complete. 2024-04-16T02:57:35.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:35 smithi151 ceph-mon[112233]: pgmap v568: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:35.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:35 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2276521746' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:57:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:35 smithi153 ceph-mon[87858]: pgmap v568: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:35.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:35 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2276521746' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:57:36.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:36 smithi151 ceph-mon[112233]: from='client.35090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:36.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:36 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3084148853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:57:36.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:36 smithi153 ceph-mon[87858]: from='client.35090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:57:36.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:36 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3084148853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:57:37.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:37 smithi151 ceph-mon[112233]: pgmap v569: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:37.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:37 smithi153 ceph-mon[87858]: pgmap v569: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:39.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:39 smithi151 ceph-mon[112233]: pgmap v570: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:39.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:57:39.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:39 smithi153 ceph-mon[87858]: pgmap v570: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:39.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:57:41.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:41 smithi151 ceph-mon[112233]: pgmap v571: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:41.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:41 smithi153 ceph-mon[87858]: pgmap v571: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:43.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:43 smithi151 ceph-mon[112233]: pgmap v572: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:43.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:43 smithi153 ceph-mon[87858]: pgmap v572: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:45.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:45 smithi151 ceph-mon[112233]: pgmap v573: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:45.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:45 smithi153 ceph-mon[87858]: pgmap v573: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:47.742 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:47 smithi153 ceph-mon[87858]: pgmap v574: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:47.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:47 smithi151 ceph-mon[112233]: pgmap v574: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:49.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:49 smithi151 ceph-mon[112233]: pgmap v575: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:49.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:49 smithi153 ceph-mon[87858]: pgmap v575: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:51.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:51 smithi151 ceph-mon[112233]: pgmap v576: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:51.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:51 smithi153 ceph-mon[87858]: pgmap v576: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:53.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:53 smithi151 ceph-mon[112233]: pgmap v577: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:53.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:53 smithi153 ceph-mon[87858]: pgmap v577: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:54.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:57:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:57:55.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:55 smithi153 ceph-mon[87858]: pgmap v578: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:55 smithi151 ceph-mon[112233]: pgmap v578: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:57.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:57 smithi153 ceph-mon[87858]: pgmap v579: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:57.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:57 smithi151 ceph-mon[112233]: pgmap v579: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:59.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:57:59 smithi153 ceph-mon[87858]: pgmap v580: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:57:59.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:57:59 smithi151 ceph-mon[112233]: pgmap v580: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:01.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:01 smithi153 ceph-mon[87858]: pgmap v581: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:01 smithi151 ceph-mon[112233]: pgmap v581: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:03.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:03 smithi153 ceph-mon[87858]: pgmap v582: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:03.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:03 smithi151 ceph-mon[112233]: pgmap v582: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:05.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.441+0000 7fcd81985700 1 -- 172.21.15.151:0/816499324 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 msgr2=0x7fcd7c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.441+0000 7fcd81985700 1 --2- 172.21.15.151:0/816499324 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 0x7fcd7c100050 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009910 tx=0x7fcd6c009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.441+0000 7fcd81985700 1 -- 172.21.15.151:0/816499324 shutdown_connections 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.441+0000 7fcd81985700 1 --2- 172.21.15.151:0/816499324 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 0x7fcd7c100050 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.441+0000 7fcd81985700 1 --2- 172.21.15.151:0/816499324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c105a40 0x7fcd7c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.441+0000 7fcd81985700 1 -- 172.21.15.151:0/816499324 >> 172.21.15.151:0/816499324 conn(0x7fcd7c0755a0 msgr2=0x7fcd7c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.441+0000 7fcd81985700 1 -- 172.21.15.151:0/816499324 shutdown_connections 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.441+0000 7fcd81985700 1 -- 172.21.15.151:0/816499324 wait complete. 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.442+0000 7fcd81985700 1 Processor -- start 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.442+0000 7fcd81985700 1 -- start start 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.442+0000 7fcd81985700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 0x7fcd7c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.442+0000 7fcd81985700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c105a40 0x7fcd7c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.442+0000 7fcd81985700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c19d0d0 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.442+0000 7fcd81985700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c196580 con 0x7fcd7c105a40 2024-04-16T02:58:05.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.442+0000 7fcd7affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 0x7fcd7c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:05.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.443+0000 7fcd7affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 0x7fcd7c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43530/0 (socket says 172.21.15.151:43530) 2024-04-16T02:58:05.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.443+0000 7fcd7affd700 1 -- 172.21.15.151:0/2158264480 learned_addr learned my addr 172.21.15.151:0/2158264480 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:05.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.443+0000 7fcd7a7fc700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c105a40 0x7fcd7c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.443+0000 7fcd7affd700 1 -- 172.21.15.151:0/2158264480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c105a40 msgr2=0x7fcd7c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.443+0000 7fcd7affd700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c105a40 0x7fcd7c19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.443+0000 7fcd7affd700 1 -- 172.21.15.151:0/2158264480 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c0095f0 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.443+0000 7fcd7affd700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 0x7fcd7c19c500 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d8d0 tx=0x7fcd7000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.443+0000 7fcd80983700 1 -- 172.21.15.151:0/2158264480 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd700098a0 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.444+0000 7fcd80983700 1 -- 172.21.15.151:0/2158264480 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd70010460 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.444+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c196860 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.444+0000 7fcd80983700 1 -- 172.21.15.151:0/2158264480 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd7000f5f0 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.444+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c196db0 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.445+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c108fe0 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.445+0000 7fcd80983700 1 -- 172.21.15.151:0/2158264480 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fcd70009a00 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.446+0000 7fcd80983700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcd6406ecc0 0x7fcd64071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:05.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.446+0000 7fcd80983700 1 -- 172.21.15.151:0/2158264480 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcd7008ed30 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.446+0000 7fcd7a7fc700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcd6406ecc0 0x7fcd64071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:05.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.446+0000 7fcd7a7fc700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcd6406ecc0 0x7fcd64071180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c104d90 tx=0x7fcd6c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:05.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.451+0000 7fcd80983700 1 -- 172.21.15.151:0/2158264480 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcd7005b610 con 0x7fcd7c0ffbd0 2024-04-16T02:58:05.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.675+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd7c02cd00 con 0x7fcd6406ecc0 2024-04-16T02:58:05.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.676+0000 7fcd80983700 1 -- 172.21.15.151:0/2158264480 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcd7c02cd00 con 0x7fcd6406ecc0 2024-04-16T02:58:05.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcd6406ecc0 msgr2=0x7fcd64071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:05.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcd6406ecc0 0x7fcd64071180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c104d90 tx=0x7fcd6c01a040 comp rx=0 tx=0).stop 2024-04-16T02:58:05.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 msgr2=0x7fcd7c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:05.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 0x7fcd7c19c500 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d8d0 tx=0x7fcd7000dbe0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 shutdown_connections 2024-04-16T02:58:05.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcd6406ecc0 0x7fcd64071180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0ffbd0 0x7fcd7c19c500 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 --2- 172.21.15.151:0/2158264480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c105a40 0x7fcd7c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 >> 172.21.15.151:0/2158264480 conn(0x7fcd7c0755a0 msgr2=0x7fcd7c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:05.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 shutdown_connections 2024-04-16T02:58:05.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.678+0000 7fcd81985700 1 -- 172.21.15.151:0/2158264480 wait complete. 2024-04-16T02:58:05.691 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:58:05.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:05 smithi153 ceph-mon[87858]: pgmap v583: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:05.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.784+0000 7f6229ac8700 1 -- 172.21.15.151:0/4219852878 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 msgr2=0x7f622410ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:05.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.784+0000 7f6229ac8700 1 --2- 172.21.15.151:0/4219852878 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 0x7f622410ab10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6218009910 tx=0x7f6218009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:05.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.784+0000 7f6229ac8700 1 -- 172.21.15.151:0/4219852878 shutdown_connections 2024-04-16T02:58:05.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.784+0000 7f6229ac8700 1 --2- 172.21.15.151:0/4219852878 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 0x7f622410ab10 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.784+0000 7f6229ac8700 1 --2- 172.21.15.151:0/4219852878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c70 0x7f6224102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.786 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:05 smithi151 ceph-mon[112233]: pgmap v583: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.784+0000 7f6229ac8700 1 -- 172.21.15.151:0/4219852878 >> 172.21.15.151:0/4219852878 conn(0x7f62240fb550 msgr2=0x7f62240fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.784+0000 7f6229ac8700 1 -- 172.21.15.151:0/4219852878 shutdown_connections 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.784+0000 7f6229ac8700 1 -- 172.21.15.151:0/4219852878 wait complete. 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.784+0000 7f6229ac8700 1 Processor -- start 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6229ac8700 1 -- start start 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6229ac8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c70 0x7f622419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6229ac8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 0x7f622419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6229ac8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622419d130 con 0x7f6224102620 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6229ac8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62241965e0 con 0x7f6224101c70 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6222ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 0x7f622419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6222ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 0x7f622419caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43534/0 (socket says 172.21.15.151:43534) 2024-04-16T02:58:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6222ffd700 1 -- 172.21.15.151:0/2641431260 learned_addr learned my addr 172.21.15.151:0/2641431260 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f62237fe700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c70 0x7f622419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6222ffd700 1 -- 172.21.15.151:0/2641431260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c70 msgr2=0x7f622419c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6222ffd700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c70 0x7f622419c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f6222ffd700 1 -- 172.21.15.151:0/2641431260 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180095f0 con 0x7f6224102620 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.785+0000 7f62237fe700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c70 0x7f622419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.786+0000 7f6222ffd700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 0x7f622419caa0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6218009fd0 tx=0x7f621800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.786+0000 7f6220ff9700 1 -- 172.21.15.151:0/2641431260 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f621801d070 con 0x7f6224102620 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.786+0000 7f6220ff9700 1 -- 172.21.15.151:0/2641431260 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6218022470 con 0x7f6224102620 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.786+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6224196860 con 0x7f6224102620 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.786+0000 7f6220ff9700 1 -- 172.21.15.151:0/2641431260 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6218017610 con 0x7f6224102620 2024-04-16T02:58:05.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.786+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6224196cf0 con 0x7f6224102620 2024-04-16T02:58:05.790 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.787+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6224108210 con 0x7f6224102620 2024-04-16T02:58:05.790 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.788+0000 7f6220ff9700 1 -- 172.21.15.151:0/2641431260 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6218005080 con 0x7f6224102620 2024-04-16T02:58:05.790 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.788+0000 7f6220ff9700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f620c06ecc0 0x7f620c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:05.790 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.788+0000 7f6220ff9700 1 -- 172.21.15.151:0/2641431260 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6218090500 con 0x7f6224102620 2024-04-16T02:58:05.790 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.788+0000 7f62237fe700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f620c06ecc0 0x7f620c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:05.791 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.789+0000 7f62237fe700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f620c06ecc0 0x7f620c071180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f62140093a0 tx=0x7f621400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:05.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:05.793+0000 7f6220ff9700 1 -- 172.21.15.151:0/2641431260 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f621805cde0 con 0x7f6224102620 2024-04-16T02:58:06.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.012+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f622402ccf0 con 0x7f620c06ecc0 2024-04-16T02:58:06.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.013+0000 7f6220ff9700 1 -- 172.21.15.151:0/2641431260 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f622402ccf0 con 0x7f620c06ecc0 2024-04-16T02:58:06.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f620c06ecc0 msgr2=0x7f620c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f620c06ecc0 0x7f620c071180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f62140093a0 tx=0x7f621400b040 comp rx=0 tx=0).stop 2024-04-16T02:58:06.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 msgr2=0x7f622419caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 0x7f622419caa0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6218009fd0 tx=0x7f621800f690 comp rx=0 tx=0).stop 2024-04-16T02:58:06.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 shutdown_connections 2024-04-16T02:58:06.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f620c06ecc0 0x7f620c071180 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c70 0x7f622419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 --2- 172.21.15.151:0/2641431260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6224102620 0x7f622419caa0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 >> 172.21.15.151:0/2641431260 conn(0x7f62240fb550 msgr2=0x7f6224105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:06.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 shutdown_connections 2024-04-16T02:58:06.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.015+0000 7f6229ac8700 1 -- 172.21.15.151:0/2641431260 wait complete. 2024-04-16T02:58:06.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.124+0000 7f96736fd700 1 -- 172.21.15.151:0/1052108030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c069020 msgr2=0x7f966c105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.124+0000 7f96736fd700 1 --2- 172.21.15.151:0/1052108030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c069020 0x7f966c105790 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f965c009910 tx=0x7f965c009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:06.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.124+0000 7f96736fd700 1 -- 172.21.15.151:0/1052108030 shutdown_connections 2024-04-16T02:58:06.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.124+0000 7f96736fd700 1 --2- 172.21.15.151:0/1052108030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c069020 0x7f966c105790 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.124+0000 7f96736fd700 1 --2- 172.21.15.151:0/1052108030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f966c068670 0x7f966c068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.124+0000 7f96736fd700 1 -- 172.21.15.151:0/1052108030 >> 172.21.15.151:0/1052108030 conn(0x7f966c0755a0 msgr2=0x7f966c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:06.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.124+0000 7f96736fd700 1 -- 172.21.15.151:0/1052108030 shutdown_connections 2024-04-16T02:58:06.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.124+0000 7f96736fd700 1 -- 172.21.15.151:0/1052108030 wait complete. 2024-04-16T02:58:06.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f96736fd700 1 Processor -- start 2024-04-16T02:58:06.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f96736fd700 1 -- start start 2024-04-16T02:58:06.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f96736fd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c068670 0x7f966c0ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:06.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f96736fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f966c069020 0x7f966c1001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f96736fd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c100720 con 0x7f966c068670 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f96736fd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c100860 con 0x7f966c069020 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f9671499700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c068670 0x7f966c0ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f9671499700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c068670 0x7f966c0ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43550/0 (socket says 172.21.15.151:43550) 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f9671499700 1 -- 172.21.15.151:0/3132104376 learned_addr learned my addr 172.21.15.151:0/3132104376 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.125+0000 7f9670c98700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f966c069020 0x7f966c1001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.126+0000 7f9671499700 1 -- 172.21.15.151:0/3132104376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f966c069020 msgr2=0x7f966c1001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.126+0000 7f9671499700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f966c069020 0x7f966c1001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.126+0000 7f9671499700 1 -- 172.21.15.151:0/3132104376 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c0095f0 con 0x7f966c068670 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.126+0000 7f9671499700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c068670 0x7f966c0ffca0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f966800cfd0 tx=0x7f966800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.126+0000 7f96627fc700 1 -- 172.21.15.151:0/3132104376 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f966800ad70 con 0x7f966c068670 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.126+0000 7f96627fc700 1 -- 172.21.15.151:0/3132104376 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9668007e70 con 0x7f966c068670 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.126+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c1a28d0 con 0x7f966c068670 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.126+0000 7f96627fc700 1 -- 172.21.15.151:0/3132104376 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9668011810 con 0x7f966c068670 2024-04-16T02:58:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.126+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c1a2c30 con 0x7f966c068670 2024-04-16T02:58:06.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.127+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966c1090a0 con 0x7f966c068670 2024-04-16T02:58:06.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.128+0000 7f96627fc700 1 -- 172.21.15.151:0/3132104376 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9668011a70 con 0x7f966c068670 2024-04-16T02:58:06.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.128+0000 7f96627fc700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f965806ecc0 0x7f9658071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:06.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.128+0000 7f96627fc700 1 -- 172.21.15.151:0/3132104376 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f966808f180 con 0x7f966c068670 2024-04-16T02:58:06.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.128+0000 7f9670c98700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f965806ecc0 0x7f9658071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:06.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.128+0000 7f9670c98700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f965806ecc0 0x7f9658071180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f966c101490 tx=0x7f965c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:06.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.133+0000 7f96627fc700 1 -- 172.21.15.151:0/3132104376 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9668058fe0 con 0x7f966c068670 2024-04-16T02:58:06.343 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.341+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f966c0611d0 con 0x7f965806ecc0 2024-04-16T02:58:06.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.352+0000 7f96627fc700 1 -- 172.21.15.151:0/3132104376 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f966c0611d0 con 0x7f965806ecc0 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (19m) 13m ago 30m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (31m) 13m ago 31m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (30m) 12m ago 30m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (17m) 13m ago 31m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (17m) 12m ago 30m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (19m) 13m ago 30m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (13m) 13m ago 25m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (13m) 13m ago 25m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (25m) 12m ago 25m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:58:06.354 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (12m) 12m ago 25m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (21m) 13m ago 32m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (20m) 12m ago 29m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (18m) 13m ago 32m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (17m) 12m ago 29m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (20m) 13m ago 31m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (20m) 12m ago 30m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (16m) 13m ago 29m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (16m) 13m ago 28m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (15m) 13m ago 28m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (14m) 12m ago 27m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (14m) 12m ago 27m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (13m) 12m ago 26m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:58:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (19m) 13m ago 30m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:58:06.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f965806ecc0 msgr2=0x7f9658071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f965806ecc0 0x7f9658071180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f966c101490 tx=0x7f965c01a040 comp rx=0 tx=0).stop 2024-04-16T02:58:06.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c068670 msgr2=0x7f966c0ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c068670 0x7f966c0ffca0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f966800cfd0 tx=0x7f966800a340 comp rx=0 tx=0).stop 2024-04-16T02:58:06.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 shutdown_connections 2024-04-16T02:58:06.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f965806ecc0 0x7f9658071180 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f966c068670 0x7f966c0ffca0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 --2- 172.21.15.151:0/3132104376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f966c069020 0x7f966c1001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 >> 172.21.15.151:0/3132104376 conn(0x7f966c0755a0 msgr2=0x7f966c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:06.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 shutdown_connections 2024-04-16T02:58:06.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.354+0000 7f96736fd700 1 -- 172.21.15.151:0/3132104376 wait complete. 2024-04-16T02:58:06.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.457+0000 7fb2f1098700 1 -- 172.21.15.151:0/978155177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec068670 msgr2=0x7fb2ec068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.457+0000 7fb2f1098700 1 --2- 172.21.15.151:0/978155177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec068670 0x7fb2ec068a50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009910 tx=0x7fb2dc009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:06.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.457+0000 7fb2f1098700 1 -- 172.21.15.151:0/978155177 shutdown_connections 2024-04-16T02:58:06.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.457+0000 7fb2f1098700 1 --2- 172.21.15.151:0/978155177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2ec069020 0x7fb2ec105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.457+0000 7fb2f1098700 1 --2- 172.21.15.151:0/978155177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec068670 0x7fb2ec068a50 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.457+0000 7fb2f1098700 1 -- 172.21.15.151:0/978155177 >> 172.21.15.151:0/978155177 conn(0x7fb2ec0755a0 msgr2=0x7fb2ec0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:06.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.457+0000 7fb2f1098700 1 -- 172.21.15.151:0/978155177 shutdown_connections 2024-04-16T02:58:06.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.457+0000 7fb2f1098700 1 -- 172.21.15.151:0/978155177 wait complete. 2024-04-16T02:58:06.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2f1098700 1 Processor -- start 2024-04-16T02:58:06.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2f1098700 1 -- start start 2024-04-16T02:58:06.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2f1098700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2ec068670 0x7fb2ec19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:06.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2f1098700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec069020 0x7fb2ec19caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:06.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2f1098700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec19d180 con 0x7fb2ec069020 2024-04-16T02:58:06.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2f1098700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec196630 con 0x7fb2ec068670 2024-04-16T02:58:06.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2ea59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec069020 0x7fb2ec19caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:06.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2ea59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec069020 0x7fb2ec19caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56354/0 (socket says 172.21.15.151:56354) 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2ea59c700 1 -- 172.21.15.151:0/3021528379 learned_addr learned my addr 172.21.15.151:0/3021528379 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.458+0000 7fb2ead9d700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2ec068670 0x7fb2ec19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.459+0000 7fb2ea59c700 1 -- 172.21.15.151:0/3021528379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2ec068670 msgr2=0x7fb2ec19c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.459+0000 7fb2ea59c700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2ec068670 0x7fb2ec19c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.459+0000 7fb2ea59c700 1 -- 172.21.15.151:0/3021528379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0095f0 con 0x7fb2ec069020 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.459+0000 7fb2ea59c700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec069020 0x7fb2ec19caf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000e970 tx=0x7fb2e000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.459+0000 7fb2d3fff700 1 -- 172.21.15.151:0/3021528379 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2e000c9d0 con 0x7fb2ec069020 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.459+0000 7fb2d3fff700 1 -- 172.21.15.151:0/3021528379 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2e0004830 con 0x7fb2ec069020 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.459+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec196910 con 0x7fb2ec069020 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.459+0000 7fb2d3fff700 1 -- 172.21.15.151:0/3021528379 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2e0005660 con 0x7fb2ec069020 2024-04-16T02:58:06.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.459+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec196e60 con 0x7fb2ec069020 2024-04-16T02:58:06.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.460+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ec109130 con 0x7fb2ec069020 2024-04-16T02:58:06.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.461+0000 7fb2d3fff700 1 -- 172.21.15.151:0/3021528379 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb2e000cb30 con 0x7fb2ec069020 2024-04-16T02:58:06.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.461+0000 7fb2d3fff700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb2d406ecc0 0x7fb2d4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:06.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.462+0000 7fb2d3fff700 1 -- 172.21.15.151:0/3021528379 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb2e0014070 con 0x7fb2ec069020 2024-04-16T02:58:06.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.462+0000 7fb2ead9d700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb2d406ecc0 0x7fb2d4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:06.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.462+0000 7fb2ead9d700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb2d406ecc0 0x7fb2d4071180 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0059d0 tx=0x7fb2dc005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:06.468 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.466+0000 7fb2d3fff700 1 -- 172.21.15.151:0/3021528379 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb2e005ba50 con 0x7fb2ec069020 2024-04-16T02:58:06.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:06 smithi151 ceph-mon[112233]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:06.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.746+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb2ec1970e0 con 0x7fb2ec069020 2024-04-16T02:58:06.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.747+0000 7fb2d3fff700 1 -- 172.21.15.151:0/3021528379 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fb2e005b5e0 con 0x7fb2ec069020 2024-04-16T02:58:06.749 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:58:06.749 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:58:06.749 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:58:06.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:58:06.749 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:58:06.749 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:58:06.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:58:06.750 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:58:06.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb2d406ecc0 msgr2=0x7fb2d4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb2d406ecc0 0x7fb2d4071180 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0059d0 tx=0x7fb2dc005940 comp rx=0 tx=0).stop 2024-04-16T02:58:06.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec069020 msgr2=0x7fb2ec19caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec069020 0x7fb2ec19caf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000e970 tx=0x7fb2e000ec80 comp rx=0 tx=0).stop 2024-04-16T02:58:06.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 shutdown_connections 2024-04-16T02:58:06.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb2d406ecc0 0x7fb2d4071180 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2ec068670 0x7fb2ec19c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 --2- 172.21.15.151:0/3021528379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2ec069020 0x7fb2ec19caf0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 >> 172.21.15.151:0/3021528379 conn(0x7fb2ec0755a0 msgr2=0x7fb2ec0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:06.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 shutdown_connections 2024-04-16T02:58:06.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.749+0000 7fb2f1098700 1 -- 172.21.15.151:0/3021528379 wait complete. 2024-04-16T02:58:06.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:06 smithi153 ceph-mon[87858]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:06.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.857+0000 7f1e9088b700 1 -- 172.21.15.151:0/2360503894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 msgr2=0x7f1e8c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.857+0000 7f1e9088b700 1 --2- 172.21.15.151:0/2360503894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 0x7f1e8c1057c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009910 tx=0x7f1e74009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:06.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.857+0000 7f1e9088b700 1 -- 172.21.15.151:0/2360503894 shutdown_connections 2024-04-16T02:58:06.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.857+0000 7f1e9088b700 1 --2- 172.21.15.151:0/2360503894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 0x7f1e8c1057c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.857+0000 7f1e9088b700 1 --2- 172.21.15.151:0/2360503894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e8c100dc0 0x7f1e8c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.857+0000 7f1e9088b700 1 -- 172.21.15.151:0/2360503894 >> 172.21.15.151:0/2360503894 conn(0x7f1e8c0fc650 msgr2=0x7f1e8c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:06.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.857+0000 7f1e9088b700 1 -- 172.21.15.151:0/2360503894 shutdown_connections 2024-04-16T02:58:06.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.857+0000 7f1e9088b700 1 -- 172.21.15.151:0/2360503894 wait complete. 2024-04-16T02:58:06.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e9088b700 1 Processor -- start 2024-04-16T02:58:06.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e9088b700 1 -- start start 2024-04-16T02:58:06.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e9088b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e8c100dc0 0x7f1e8c19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e9088b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 0x7f1e8c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e9088b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19d130 con 0x7f1e8c101770 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e9088b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c1965e0 con 0x7f1e8c100dc0 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e83fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 0x7f1e8c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e83fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 0x7f1e8c19caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56372/0 (socket says 172.21.15.151:56372) 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e83fff700 1 -- 172.21.15.151:0/1951517649 learned_addr learned my addr 172.21.15.151:0/1951517649 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e8a59c700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e8c100dc0 0x7f1e8c19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e83fff700 1 -- 172.21.15.151:0/1951517649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e8c100dc0 msgr2=0x7f1e8c19c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e83fff700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e8c100dc0 0x7f1e8c19c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e83fff700 1 -- 172.21.15.151:0/1951517649 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e740095f0 con 0x7f1e8c101770 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.858+0000 7f1e8a59c700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e8c100dc0 0x7f1e8c19c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.859+0000 7f1e83fff700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 0x7f1e8c19caa0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1e740092e0 tx=0x7f1e7400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:06.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.859+0000 7f1e837fe700 1 -- 172.21.15.151:0/1951517649 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e7401d070 con 0x7f1e8c101770 2024-04-16T02:58:06.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.859+0000 7f1e837fe700 1 -- 172.21.15.151:0/1951517649 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e74005dc0 con 0x7f1e8c101770 2024-04-16T02:58:06.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.859+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c196860 con 0x7f1e8c101770 2024-04-16T02:58:06.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.859+0000 7f1e837fe700 1 -- 172.21.15.151:0/1951517649 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e74017400 con 0x7f1e8c101770 2024-04-16T02:58:06.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.859+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c196d50 con 0x7f1e8c101770 2024-04-16T02:58:06.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.860+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c1091c0 con 0x7f1e8c101770 2024-04-16T02:58:06.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.860+0000 7f1e837fe700 1 -- 172.21.15.151:0/1951517649 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1e7400f460 con 0x7f1e8c101770 2024-04-16T02:58:06.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.861+0000 7f1e837fe700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e7806ecc0 0x7f1e78071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:06.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.861+0000 7f1e837fe700 1 -- 172.21.15.151:0/1951517649 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1e74090380 con 0x7f1e8c101770 2024-04-16T02:58:06.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.861+0000 7f1e8a59c700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e7806ecc0 0x7f1e78071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:06.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.861+0000 7f1e8a59c700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e7806ecc0 0x7f1e78071180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c005fd0 tx=0x7f1e7c005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:06.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:06.865+0000 7f1e837fe700 1 -- 172.21.15.151:0/1951517649 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1e7405cc60 con 0x7f1e8c101770 2024-04-16T02:58:07.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.110+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1e8c04ea90 con 0x7f1e8c101770 2024-04-16T02:58:07.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.110+0000 7f1e837fe700 1 -- 172.21.15.151:0/1951517649 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f1e7405c7f0 con 0x7f1e8c101770 2024-04-16T02:58:07.112 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:58:07.112 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:58:07.113 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:58:07.114 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:58:07.115 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:58:07.115 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:58:07.115 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:58:07.115 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:58:07.115 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:58:07.115 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:58:07.115 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:58:07.115 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:58:07.115 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:58:07.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.113+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e7806ecc0 msgr2=0x7f1e78071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.113+0000 7f1e9088b700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e7806ecc0 0x7f1e78071180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c005fd0 tx=0x7f1e7c005ee0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.113+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 msgr2=0x7f1e8c19caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.113+0000 7f1e9088b700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 0x7f1e8c19caa0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1e740092e0 tx=0x7f1e7400be20 comp rx=0 tx=0).stop 2024-04-16T02:58:07.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.114+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 shutdown_connections 2024-04-16T02:58:07.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.114+0000 7f1e9088b700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1e7806ecc0 0x7f1e78071180 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.114+0000 7f1e9088b700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e8c100dc0 0x7f1e8c19c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.114+0000 7f1e9088b700 1 --2- 172.21.15.151:0/1951517649 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e8c101770 0x7f1e8c19caa0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.114+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 >> 172.21.15.151:0/1951517649 conn(0x7f1e8c0fc650 msgr2=0x7f1e8c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:07.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.114+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 shutdown_connections 2024-04-16T02:58:07.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.114+0000 7f1e9088b700 1 -- 172.21.15.151:0/1951517649 wait complete. 2024-04-16T02:58:07.117 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:58:07.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.219+0000 7f32c1998700 1 -- 172.21.15.151:0/1944928673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc101770 msgr2=0x7f32bc1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.219+0000 7f32c1998700 1 --2- 172.21.15.151:0/1944928673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc101770 0x7f32bc1057c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009910 tx=0x7f32b0009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:07.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.219+0000 7f32c1998700 1 -- 172.21.15.151:0/1944928673 shutdown_connections 2024-04-16T02:58:07.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.219+0000 7f32c1998700 1 --2- 172.21.15.151:0/1944928673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc101770 0x7f32bc1057c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.219+0000 7f32c1998700 1 --2- 172.21.15.151:0/1944928673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc100dc0 0x7f32bc1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.219+0000 7f32c1998700 1 -- 172.21.15.151:0/1944928673 >> 172.21.15.151:0/1944928673 conn(0x7f32bc0fc650 msgr2=0x7f32bc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:07.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.219+0000 7f32c1998700 1 -- 172.21.15.151:0/1944928673 shutdown_connections 2024-04-16T02:58:07.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.219+0000 7f32c1998700 1 -- 172.21.15.151:0/1944928673 wait complete. 2024-04-16T02:58:07.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.220+0000 7f32c1998700 1 Processor -- start 2024-04-16T02:58:07.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.220+0000 7f32c1998700 1 -- start start 2024-04-16T02:58:07.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.220+0000 7f32c1998700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc100dc0 0x7f32bc19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:07.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.220+0000 7f32c1998700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc101770 0x7f32bc19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:07.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.220+0000 7f32c1998700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc19d250 con 0x7f32bc100dc0 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.220+0000 7f32c1998700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc196700 con 0x7f32bc101770 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32baffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc100dc0 0x7f32bc19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32baffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc100dc0 0x7f32bc19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56390/0 (socket says 172.21.15.151:56390) 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32baffd700 1 -- 172.21.15.151:0/2260162549 learned_addr learned my addr 172.21.15.151:0/2260162549 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32ba7fc700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc101770 0x7f32bc19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32baffd700 1 -- 172.21.15.151:0/2260162549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc101770 msgr2=0x7f32bc19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32baffd700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc101770 0x7f32bc19cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32baffd700 1 -- 172.21.15.151:0/2260162549 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b00095f0 con 0x7f32bc100dc0 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32ba7fc700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc101770 0x7f32bc19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32baffd700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc100dc0 0x7f32bc19c680 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00b4b0 tx=0x7f32ac00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32c0996700 1 -- 172.21.15.151:0/2260162549 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32ac011800 con 0x7f32bc100dc0 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32bc1969e0 con 0x7f32bc100dc0 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32c0996700 1 -- 172.21.15.151:0/2260162549 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32ac011e20 con 0x7f32bc100dc0 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32c0996700 1 -- 172.21.15.151:0/2260162549 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32ac00f2e0 con 0x7f32bc100dc0 2024-04-16T02:58:07.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.221+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32bc196f30 con 0x7f32bc100dc0 2024-04-16T02:58:07.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.222+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32bc1090f0 con 0x7f32bc100dc0 2024-04-16T02:58:07.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.223+0000 7f32c0996700 1 -- 172.21.15.151:0/2260162549 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f32ac011960 con 0x7f32bc100dc0 2024-04-16T02:58:07.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.223+0000 7f32c0996700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32a406ecc0 0x7f32a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:07.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.223+0000 7f32c0996700 1 -- 172.21.15.151:0/2260162549 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f32ac08ea30 con 0x7f32bc100dc0 2024-04-16T02:58:07.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.223+0000 7f32ba7fc700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32a406ecc0 0x7f32a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:07.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.224+0000 7f32ba7fc700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32a406ecc0 0x7f32a4071180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f32b000b5c0 tx=0x7f32b0005480 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:07.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.228+0000 7f32c0996700 1 -- 172.21.15.151:0/2260162549 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f32ac05b310 con 0x7f32bc100dc0 2024-04-16T02:58:07.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.443+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f32bc197220 con 0x7f32a406ecc0 2024-04-16T02:58:07.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.444+0000 7f32c0996700 1 -- 172.21.15.151:0/2260162549 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f32bc197220 con 0x7f32a406ecc0 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:58:07.447 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:58:07.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32a406ecc0 msgr2=0x7f32a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32a406ecc0 0x7f32a4071180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f32b000b5c0 tx=0x7f32b0005480 comp rx=0 tx=0).stop 2024-04-16T02:58:07.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc100dc0 msgr2=0x7f32bc19c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc100dc0 0x7f32bc19c680 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00b4b0 tx=0x7f32ac00b870 comp rx=0 tx=0).stop 2024-04-16T02:58:07.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 shutdown_connections 2024-04-16T02:58:07.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f32a406ecc0 0x7f32a4071180 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32bc100dc0 0x7f32bc19c680 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 --2- 172.21.15.151:0/2260162549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc101770 0x7f32bc19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 >> 172.21.15.151:0/2260162549 conn(0x7f32bc0fc650 msgr2=0x7f32bc0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:07.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 shutdown_connections 2024-04-16T02:58:07.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.447+0000 7f32c1998700 1 -- 172.21.15.151:0/2260162549 wait complete. 2024-04-16T02:58:07.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.554+0000 7f9f459ac700 1 -- 172.21.15.151:0/990344855 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 msgr2=0x7f9f401011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.554+0000 7f9f459ac700 1 --2- 172.21.15.151:0/990344855 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 0x7f9f401011b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000b0a0 tx=0x7f9f3000b3b0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.554+0000 7f9f459ac700 1 -- 172.21.15.151:0/990344855 shutdown_connections 2024-04-16T02:58:07.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.554+0000 7f9f459ac700 1 --2- 172.21.15.151:0/990344855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f40101780 0x7f9f401057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.554+0000 7f9f459ac700 1 --2- 172.21.15.151:0/990344855 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 0x7f9f401011b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.554+0000 7f9f459ac700 1 -- 172.21.15.151:0/990344855 >> 172.21.15.151:0/990344855 conn(0x7f9f400fc640 msgr2=0x7f9f400fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:07.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.554+0000 7f9f459ac700 1 -- 172.21.15.151:0/990344855 shutdown_connections 2024-04-16T02:58:07.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.554+0000 7f9f459ac700 1 -- 172.21.15.151:0/990344855 wait complete. 2024-04-16T02:58:07.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.554+0000 7f9f459ac700 1 Processor -- start 2024-04-16T02:58:07.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f459ac700 1 -- start start 2024-04-16T02:58:07.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f459ac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 0x7f9f40073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f459ac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f40101780 0x7f9f400735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f459ac700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f40073b10 con 0x7f9f40100dd0 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f459ac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f40073c50 con 0x7f9f40101780 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f3effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 0x7f9f40073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f3effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 0x7f9f40073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56400/0 (socket says 172.21.15.151:56400) 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f3effd700 1 -- 172.21.15.151:0/3678288450 learned_addr learned my addr 172.21.15.151:0/3678288450 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f3e7fc700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f40101780 0x7f9f400735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f3effd700 1 -- 172.21.15.151:0/3678288450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f40101780 msgr2=0x7f9f400735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f3effd700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f40101780 0x7f9f400735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f3effd700 1 -- 172.21.15.151:0/3678288450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f30009d00 con 0x7f9f40100dd0 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f3e7fc700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f40101780 0x7f9f400735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:58:07.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.555+0000 7f9f3effd700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 0x7f9f40073090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9f30000f80 tx=0x7f9f3000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:07.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.556+0000 7f9f449aa700 1 -- 172.21.15.151:0/3678288450 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f3000e040 con 0x7f9f40100dd0 2024-04-16T02:58:07.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.556+0000 7f9f449aa700 1 -- 172.21.15.151:0/3678288450 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f30003e60 con 0x7f9f40100dd0 2024-04-16T02:58:07.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.556+0000 7f9f449aa700 1 -- 172.21.15.151:0/3678288450 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f3001d910 con 0x7f9f40100dd0 2024-04-16T02:58:07.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.556+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f40073ed0 con 0x7f9f40100dd0 2024-04-16T02:58:07.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.556+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f401a2c40 con 0x7f9f40100dd0 2024-04-16T02:58:07.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.557+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f40109170 con 0x7f9f40100dd0 2024-04-16T02:58:07.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.557+0000 7f9f449aa700 1 -- 172.21.15.151:0/3678288450 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9f30004820 con 0x7f9f40100dd0 2024-04-16T02:58:07.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.558+0000 7f9f449aa700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ecc0 0x7f9f2c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:07.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.558+0000 7f9f3e7fc700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ecc0 0x7f9f2c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:07.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.558+0000 7f9f449aa700 1 -- 172.21.15.151:0/3678288450 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9f30090650 con 0x7f9f40100dd0 2024-04-16T02:58:07.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.558+0000 7f9f3e7fc700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ecc0 0x7f9f2c071180 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9f40074880 tx=0x7f9f28005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:07.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.562+0000 7f9f449aa700 1 -- 172.21.15.151:0/3678288450 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9f30059480 con 0x7f9f40100dd0 2024-04-16T02:58:07.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:07 smithi151 ceph-mon[112233]: from='client.35102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:07.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:07 smithi151 ceph-mon[112233]: pgmap v584: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:07.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:07 smithi151 ceph-mon[112233]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:07.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:07 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3021528379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:58:07.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:07 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1951517649' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:58:07.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.845+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9f4004ea90 con 0x7f9f40100dd0 2024-04-16T02:58:07.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.846+0000 7f9f449aa700 1 -- 172.21.15.151:0/3678288450 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f9f3005cb40 con 0x7f9f40100dd0 2024-04-16T02:58:07.848 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:58:07.848 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:58:07.848 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:58:07.848 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:58:07.848 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:58:07.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ecc0 msgr2=0x7f9f2c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ecc0 0x7f9f2c071180 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9f40074880 tx=0x7f9f28005f50 comp rx=0 tx=0).stop 2024-04-16T02:58:07.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 msgr2=0x7f9f40073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:07.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 0x7f9f40073090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9f30000f80 tx=0x7f9f3000cdd0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 shutdown_connections 2024-04-16T02:58:07.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9f2c06ecc0 0x7f9f2c071180 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f40100dd0 0x7f9f40073090 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 --2- 172.21.15.151:0/3678288450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f40101780 0x7f9f400735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:07.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 >> 172.21.15.151:0/3678288450 conn(0x7f9f400fc640 msgr2=0x7f9f400fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:07.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 shutdown_connections 2024-04-16T02:58:07.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:07.848+0000 7f9f459ac700 1 -- 172.21.15.151:0/3678288450 wait complete. 2024-04-16T02:58:07.958 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:07 smithi153 ceph-mon[87858]: from='client.35102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:07.958 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:07 smithi153 ceph-mon[87858]: pgmap v584: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:07.958 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:07 smithi153 ceph-mon[87858]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:07.958 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:07 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3021528379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:58:07.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:07 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1951517649' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:58:08.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:08 smithi151 ceph-mon[112233]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:08.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:08 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3678288450' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:58:09.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:08 smithi153 ceph-mon[87858]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:09.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:08 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3678288450' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:58:09.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:09 smithi151 ceph-mon[112233]: pgmap v585: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:09.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:58:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:09 smithi153 ceph-mon[87858]: pgmap v585: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:58:11.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:11 smithi151 ceph-mon[112233]: pgmap v586: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:12.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:11 smithi153 ceph-mon[87858]: pgmap v586: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:13.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:13 smithi151 ceph-mon[112233]: pgmap v587: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:14.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:13 smithi153 ceph-mon[87858]: pgmap v587: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:15.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:15 smithi151 ceph-mon[112233]: pgmap v588: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:15 smithi153 ceph-mon[87858]: pgmap v588: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:17.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:17 smithi151 ceph-mon[112233]: pgmap v589: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:18.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:17 smithi153 ceph-mon[87858]: pgmap v589: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:19.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:19 smithi151 ceph-mon[112233]: pgmap v590: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:19 smithi153 ceph-mon[87858]: pgmap v590: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:21.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:21 smithi153 ceph-mon[87858]: pgmap v591: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:21.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:21 smithi151 ceph-mon[112233]: pgmap v591: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:23.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:23 smithi151 ceph-mon[112233]: pgmap v592: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:24.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:23 smithi153 ceph-mon[87858]: pgmap v592: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:58:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:58:25.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:25 smithi151 ceph-mon[112233]: pgmap v593: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:25 smithi153 ceph-mon[87858]: pgmap v593: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:27.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:27 smithi151 ceph-mon[112233]: pgmap v594: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:27 smithi153 ceph-mon[87858]: pgmap v594: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:29.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:29 smithi151 ceph-mon[112233]: pgmap v595: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:29 smithi153 ceph-mon[87858]: pgmap v595: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:31.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:31 smithi151 ceph-mon[112233]: pgmap v596: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:32.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:31 smithi153 ceph-mon[87858]: pgmap v596: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:33.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:33 smithi151 ceph-mon[112233]: pgmap v597: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:34.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:33 smithi153 ceph-mon[87858]: pgmap v597: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:35.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:35 smithi151 ceph-mon[112233]: pgmap v598: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:35 smithi153 ceph-mon[87858]: pgmap v598: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:37.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.968+0000 7fc49ec6b700 1 -- 172.21.15.151:0/2376281302 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 msgr2=0x7fc498103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:37.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.968+0000 7fc49ec6b700 1 --2- 172.21.15.151:0/2376281302 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 0x7fc498103e80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc48400aa20 tx=0x7fc48400ad30 comp rx=0 tx=0).stop 2024-04-16T02:58:37.971 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:37 smithi151 ceph-mon[112233]: pgmap v599: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:37.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.968+0000 7fc49ec6b700 1 -- 172.21.15.151:0/2376281302 shutdown_connections 2024-04-16T02:58:37.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.968+0000 7fc49ec6b700 1 --2- 172.21.15.151:0/2376281302 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 0x7fc498103e80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:37.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.968+0000 7fc49ec6b700 1 --2- 172.21.15.151:0/2376281302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4980ff500 0x7fc4980ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:37.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.968+0000 7fc49ec6b700 1 -- 172.21.15.151:0/2376281302 >> 172.21.15.151:0/2376281302 conn(0x7fc498074d80 msgr2=0x7fc498075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:37.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.968+0000 7fc49ec6b700 1 -- 172.21.15.151:0/2376281302 shutdown_connections 2024-04-16T02:58:37.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.968+0000 7fc49ec6b700 1 -- 172.21.15.151:0/2376281302 wait complete. 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc49ec6b700 1 Processor -- start 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc49ec6b700 1 -- start start 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc49ec6b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4980ff500 0x7fc498194570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc49ec6b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 0x7fc498194ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc49ec6b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc498195190 con 0x7fc4980ffeb0 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc49ec6b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc498198f20 con 0x7fc4980ff500 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc497fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 0x7fc498194ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc497fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 0x7fc498194ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48142/0 (socket says 172.21.15.151:48142) 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc497fff700 1 -- 172.21.15.151:0/1147842466 learned_addr learned my addr 172.21.15.151:0/1147842466 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:37.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc49ca07700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4980ff500 0x7fc498194570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc497fff700 1 -- 172.21.15.151:0/1147842466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4980ff500 msgr2=0x7fc498194570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc497fff700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4980ff500 0x7fc498194570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.969+0000 7fc497fff700 1 -- 172.21.15.151:0/1147842466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48400a700 con 0x7fc4980ffeb0 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.970+0000 7fc497fff700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 0x7fc498194ab0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc484006010 tx=0x7fc4840048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.970+0000 7fc495ffb700 1 -- 172.21.15.151:0/1147842466 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc484004d30 con 0x7fc4980ffeb0 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.970+0000 7fc495ffb700 1 -- 172.21.15.151:0/1147842466 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc48400e910 con 0x7fc4980ffeb0 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.970+0000 7fc495ffb700 1 -- 172.21.15.151:0/1147842466 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc484016830 con 0x7fc4980ffeb0 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.970+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4981991a0 con 0x7fc4980ffeb0 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.970+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc498199640 con 0x7fc4980ffeb0 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.971+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc498107880 con 0x7fc4980ffeb0 2024-04-16T02:58:37.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.972+0000 7fc495ffb700 1 -- 172.21.15.151:0/1147842466 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc48400e450 con 0x7fc4980ffeb0 2024-04-16T02:58:37.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.972+0000 7fc495ffb700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc48806ecc0 0x7fc488071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:37.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.972+0000 7fc495ffb700 1 -- 172.21.15.151:0/1147842466 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc484012070 con 0x7fc4980ffeb0 2024-04-16T02:58:37.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.972+0000 7fc49ca07700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc48806ecc0 0x7fc488071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:37.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.973+0000 7fc49ca07700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc48806ecc0 0x7fc488071180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc48c005950 tx=0x7fc48c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:37.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:37.977+0000 7fc495ffb700 1 -- 172.21.15.151:0/1147842466 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc484059b40 con 0x7fc4980ffeb0 2024-04-16T02:58:38.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:37 smithi153 ceph-mon[87858]: pgmap v599: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:38.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.196+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc498199a30 con 0x7fc48806ecc0 2024-04-16T02:58:38.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.197+0000 7fc495ffb700 1 -- 172.21.15.151:0/1147842466 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc498199a30 con 0x7fc48806ecc0 2024-04-16T02:58:38.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc48806ecc0 msgr2=0x7fc488071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc48806ecc0 0x7fc488071180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc48c005950 tx=0x7fc48c00b410 comp rx=0 tx=0).stop 2024-04-16T02:58:38.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 msgr2=0x7fc498194ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 0x7fc498194ab0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc484006010 tx=0x7fc4840048c0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 shutdown_connections 2024-04-16T02:58:38.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc48806ecc0 0x7fc488071180 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4980ff500 0x7fc498194570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 --2- 172.21.15.151:0/1147842466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4980ffeb0 0x7fc498194ab0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 >> 172.21.15.151:0/1147842466 conn(0x7fc498074d80 msgr2=0x7fc4980fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:38.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 shutdown_connections 2024-04-16T02:58:38.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.200+0000 7fc49ec6b700 1 -- 172.21.15.151:0/1147842466 wait complete. 2024-04-16T02:58:38.212 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:58:38.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 -- 172.21.15.151:0/1152986180 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 msgr2=0x7f97ac0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 --2- 172.21.15.151:0/1152986180 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 0x7f97ac0ffec0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9798009910 tx=0x7f9798009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:38.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 -- 172.21.15.151:0/1152986180 shutdown_connections 2024-04-16T02:58:38.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 --2- 172.21.15.151:0/1152986180 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 0x7f97ac0ffec0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 --2- 172.21.15.151:0/1152986180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97ac105a40 0x7f97ac105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 -- 172.21.15.151:0/1152986180 >> 172.21.15.151:0/1152986180 conn(0x7f97ac0fb550 msgr2=0x7f97ac0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:38.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 -- 172.21.15.151:0/1152986180 shutdown_connections 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 -- 172.21.15.151:0/1152986180 wait complete. 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 Processor -- start 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 -- start start 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 0x7f97ac19c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97ac105a40 0x7f97ac19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac19d0f0 con 0x7f97ac0ffa40 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.307+0000 7f97b2bbf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac1965a0 con 0x7f97ac105a40 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97b095b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 0x7f97ac19c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97b095b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 0x7f97ac19c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48150/0 (socket says 172.21.15.151:48150) 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97b095b700 1 -- 172.21.15.151:0/3107078279 learned_addr learned my addr 172.21.15.151:0/3107078279 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97abfff700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97ac105a40 0x7f97ac19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:38.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97b095b700 1 -- 172.21.15.151:0/3107078279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97ac105a40 msgr2=0x7f97ac19ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97b095b700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97ac105a40 0x7f97ac19ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97b095b700 1 -- 172.21.15.151:0/3107078279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980095f0 con 0x7f97ac0ffa40 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97b095b700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 0x7f97ac19c520 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f97a000d900 tx=0x7f97a000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97a9ffb700 1 -- 172.21.15.151:0/3107078279 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97a00041f0 con 0x7f97ac0ffa40 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97a9ffb700 1 -- 172.21.15.151:0/3107078279 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97a0004cf0 con 0x7f97ac0ffa40 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97a9ffb700 1 -- 172.21.15.151:0/3107078279 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97a000b750 con 0x7f97ac0ffa40 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.308+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ac196880 con 0x7f97ac0ffa40 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.309+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ac196dd0 con 0x7f97ac0ffa40 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.309+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ac108fe0 con 0x7f97ac0ffa40 2024-04-16T02:58:38.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.310+0000 7f97a9ffb700 1 -- 172.21.15.151:0/3107078279 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f97a0004750 con 0x7f97ac0ffa40 2024-04-16T02:58:38.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.310+0000 7f97a9ffb700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f979c06ecc0 0x7f979c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:38.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.310+0000 7f97a9ffb700 1 -- 172.21.15.151:0/3107078279 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f97a008ed60 con 0x7f97ac0ffa40 2024-04-16T02:58:38.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.310+0000 7f97abfff700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f979c06ecc0 0x7f979c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:38.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.311+0000 7f97abfff700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f979c06ecc0 0x7f979c071180 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f97ac105310 tx=0x7f979801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:38.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.315+0000 7f97a9ffb700 1 -- 172.21.15.151:0/3107078279 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f97a005b640 con 0x7f97ac0ffa40 2024-04-16T02:58:38.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.531+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97ac197070 con 0x7f979c06ecc0 2024-04-16T02:58:38.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.533+0000 7f97a9ffb700 1 -- 172.21.15.151:0/3107078279 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f97ac197070 con 0x7f979c06ecc0 2024-04-16T02:58:38.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f979c06ecc0 msgr2=0x7f979c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f979c06ecc0 0x7f979c071180 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f97ac105310 tx=0x7f979801a040 comp rx=0 tx=0).stop 2024-04-16T02:58:38.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 msgr2=0x7f97ac19c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 0x7f97ac19c520 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f97a000d900 tx=0x7f97a000dc10 comp rx=0 tx=0).stop 2024-04-16T02:58:38.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 shutdown_connections 2024-04-16T02:58:38.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f979c06ecc0 0x7f979c071180 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97ac0ffa40 0x7f97ac19c520 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 --2- 172.21.15.151:0/3107078279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97ac105a40 0x7f97ac19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 >> 172.21.15.151:0/3107078279 conn(0x7f97ac0fb550 msgr2=0x7f97ac069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:38.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 shutdown_connections 2024-04-16T02:58:38.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.535+0000 7f97b2bbf700 1 -- 172.21.15.151:0/3107078279 wait complete. 2024-04-16T02:58:38.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.640+0000 7fd16b32b700 1 -- 172.21.15.151:0/2252515802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 msgr2=0x7fd1641007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.640+0000 7fd16b32b700 1 --2- 172.21.15.151:0/2252515802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 0x7fd1641007a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009910 tx=0x7fd15c009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:38.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.640+0000 7fd16b32b700 1 -- 172.21.15.151:0/2252515802 shutdown_connections 2024-04-16T02:58:38.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.640+0000 7fd16b32b700 1 --2- 172.21.15.151:0/2252515802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd164100d70 0x7fd1641048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.640+0000 7fd16b32b700 1 --2- 172.21.15.151:0/2252515802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 0x7fd1641007a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.640+0000 7fd16b32b700 1 -- 172.21.15.151:0/2252515802 >> 172.21.15.151:0/2252515802 conn(0x7fd164075e30 msgr2=0x7fd1640fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:38.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.640+0000 7fd16b32b700 1 -- 172.21.15.151:0/2252515802 shutdown_connections 2024-04-16T02:58:38.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.640+0000 7fd16b32b700 1 -- 172.21.15.151:0/2252515802 wait complete. 2024-04-16T02:58:38.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd16b32b700 1 Processor -- start 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd16b32b700 1 -- start start 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd16b32b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 0x7fd16419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd16b32b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd164100d70 0x7fd16419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd16b32b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16419d3c0 con 0x7fd1641003c0 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd16b32b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd164196870 con 0x7fd164100d70 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd1690c7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 0x7fd16419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd1690c7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 0x7fd16419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48166/0 (socket says 172.21.15.151:48166) 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd1690c7700 1 -- 172.21.15.151:0/1639733583 learned_addr learned my addr 172.21.15.151:0/1639733583 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd1688c6700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd164100d70 0x7fd16419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd1690c7700 1 -- 172.21.15.151:0/1639733583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd164100d70 msgr2=0x7fd16419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.641+0000 7fd1690c7700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd164100d70 0x7fd16419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.642+0000 7fd1690c7700 1 -- 172.21.15.151:0/1639733583 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c0095f0 con 0x7fd1641003c0 2024-04-16T02:58:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.642+0000 7fd1690c7700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 0x7fd16419c7f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006010 tx=0x7fd15c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:38.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.642+0000 7fd1627fc700 1 -- 172.21.15.151:0/1639733583 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd15c01d070 con 0x7fd1641003c0 2024-04-16T02:58:38.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.642+0000 7fd1627fc700 1 -- 172.21.15.151:0/1639733583 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd15c004fb0 con 0x7fd1641003c0 2024-04-16T02:58:38.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.642+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd164196af0 con 0x7fd1641003c0 2024-04-16T02:58:38.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.642+0000 7fd1627fc700 1 -- 172.21.15.151:0/1639733583 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd15c022ea0 con 0x7fd1641003c0 2024-04-16T02:58:38.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.642+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd164196fe0 con 0x7fd1641003c0 2024-04-16T02:58:38.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.643+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16404ea90 con 0x7fd1641003c0 2024-04-16T02:58:38.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.643+0000 7fd1627fc700 1 -- 172.21.15.151:0/1639733583 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd15c022470 con 0x7fd1641003c0 2024-04-16T02:58:38.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.644+0000 7fd1627fc700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd14c06ecc0 0x7fd14c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:38.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.644+0000 7fd1627fc700 1 -- 172.21.15.151:0/1639733583 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd15c0903f0 con 0x7fd1641003c0 2024-04-16T02:58:38.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.644+0000 7fd1688c6700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd14c06ecc0 0x7fd14c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:38.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.645+0000 7fd1688c6700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd14c06ecc0 0x7fd14c071180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd158006fd0 tx=0x7fd158008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:38.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.649+0000 7fd1627fc700 1 -- 172.21.15.151:0/1639733583 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd15c05cd80 con 0x7fd1641003c0 2024-04-16T02:58:38.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.855+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1641972c0 con 0x7fd14c06ecc0 2024-04-16T02:58:38.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.866+0000 7fd1627fc700 1 -- 172.21.15.151:0/1639733583 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd1641972c0 con 0x7fd14c06ecc0 2024-04-16T02:58:38.868 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:58:38.868 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (20m) 13m ago 31m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:58:38.868 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (31m) 13m ago 31m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:58:38.868 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (30m) 13m ago 30m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (17m) 13m ago 31m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (17m) 13m ago 30m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (19m) 13m ago 31m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (14m) 13m ago 25m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (13m) 13m ago 25m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (25m) 13m ago 25m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (13m) 13m ago 25m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (22m) 13m ago 32m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (21m) 13m ago 30m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (18m) 13m ago 33m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (18m) 13m ago 30m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (20m) 13m ago 31m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (20m) 13m ago 30m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (17m) 13m ago 29m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (16m) 13m ago 29m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:58:38.869 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (16m) 13m ago 28m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:58:38.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (15m) 13m ago 28m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:58:38.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (14m) 13m ago 27m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:58:38.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (14m) 13m ago 27m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:58:38.870 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (20m) 13m ago 31m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.868+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd14c06ecc0 msgr2=0x7fd14c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.868+0000 7fd16b32b700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd14c06ecc0 0x7fd14c071180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd158006fd0 tx=0x7fd158008040 comp rx=0 tx=0).stop 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.868+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 msgr2=0x7fd16419c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.868+0000 7fd16b32b700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 0x7fd16419c7f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006010 tx=0x7fd15c00be20 comp rx=0 tx=0).stop 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.868+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 shutdown_connections 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.868+0000 7fd16b32b700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd14c06ecc0 0x7fd14c071180 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.868+0000 7fd16b32b700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1641003c0 0x7fd16419c7f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.868+0000 7fd16b32b700 1 --2- 172.21.15.151:0/1639733583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd164100d70 0x7fd16419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.868+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 >> 172.21.15.151:0/1639733583 conn(0x7fd164075e30 msgr2=0x7fd164103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.869+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 shutdown_connections 2024-04-16T02:58:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.869+0000 7fd16b32b700 1 -- 172.21.15.151:0/1639733583 wait complete. 2024-04-16T02:58:38.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.972+0000 7fa781c1a700 1 -- 172.21.15.151:0/1264196892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 msgr2=0x7fa77c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.972+0000 7fa781c1a700 1 --2- 172.21.15.151:0/1264196892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 0x7fa77c1057e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa764009910 tx=0x7fa764009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:38.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.972+0000 7fa781c1a700 1 -- 172.21.15.151:0/1264196892 shutdown_connections 2024-04-16T02:58:38.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.972+0000 7fa781c1a700 1 --2- 172.21.15.151:0/1264196892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 0x7fa77c1057e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.972+0000 7fa781c1a700 1 --2- 172.21.15.151:0/1264196892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa77c100de0 0x7fa77c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.972+0000 7fa781c1a700 1 -- 172.21.15.151:0/1264196892 >> 172.21.15.151:0/1264196892 conn(0x7fa77c0fc6b0 msgr2=0x7fa77c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.972+0000 7fa781c1a700 1 -- 172.21.15.151:0/1264196892 shutdown_connections 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.972+0000 7fa781c1a700 1 -- 172.21.15.151:0/1264196892 wait complete. 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.973+0000 7fa781c1a700 1 Processor -- start 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.973+0000 7fa781c1a700 1 -- start start 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.973+0000 7fa781c1a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa77c100de0 0x7fa77c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.973+0000 7fa781c1a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 0x7fa77c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.973+0000 7fa781c1a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c073b10 con 0x7fa77c101790 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.973+0000 7fa781c1a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c073c50 con 0x7fa77c100de0 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.973+0000 7fa773fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 0x7fa77c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa773fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 0x7fa77c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48196/0 (socket says 172.21.15.151:48196) 2024-04-16T02:58:38.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa773fff700 1 -- 172.21.15.151:0/2404307693 learned_addr learned my addr 172.21.15.151:0/2404307693 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa77b7fe700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa77c100de0 0x7fa77c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa773fff700 1 -- 172.21.15.151:0/2404307693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa77c100de0 msgr2=0x7fa77c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa773fff700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa77c100de0 0x7fa77c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa773fff700 1 -- 172.21.15.151:0/2404307693 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7640095f0 con 0x7fa77c101790 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa773fff700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 0x7fa77c0735d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa76400bd70 tx=0x7fa76400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa77b7fe700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa77c100de0 0x7fa77c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa7797fa700 1 -- 172.21.15.151:0/2404307693 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa76401d070 con 0x7fa77c101790 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa7797fa700 1 -- 172.21.15.151:0/2404307693 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa764005dc0 con 0x7fa77c101790 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa7797fa700 1 -- 172.21.15.151:0/2404307693 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa764017400 con 0x7fa77c101790 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c073ed0 con 0x7fa77c101790 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.974+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c1a2c50 con 0x7fa77c101790 2024-04-16T02:58:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.975+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c1091e0 con 0x7fa77c101790 2024-04-16T02:58:38.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.976+0000 7fa7797fa700 1 -- 172.21.15.151:0/2404307693 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa76400f7d0 con 0x7fa77c101790 2024-04-16T02:58:38.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.977+0000 7fa7797fa700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa76806ecc0 0x7fa768071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:38.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.977+0000 7fa7797fa700 1 -- 172.21.15.151:0/2404307693 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa764090280 con 0x7fa77c101790 2024-04-16T02:58:38.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.977+0000 7fa77b7fe700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa76806ecc0 0x7fa768071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:38.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.977+0000 7fa77b7fe700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa76806ecc0 0x7fa768071180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa76c005fd0 tx=0x7fa76c005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:38.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:38.981+0000 7fa7797fa700 1 -- 172.21.15.151:0/2404307693 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa7640590b0 con 0x7fa77c101790 2024-04-16T02:58:39.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.265+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa77c04ea90 con 0x7fa77c101790 2024-04-16T02:58:39.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.266+0000 7fa7797fa700 1 -- 172.21.15.151:0/2404307693 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fa76400f460 con 0x7fa77c101790 2024-04-16T02:58:39.268 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:58:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:58:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:58:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:58:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:58:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:58:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:58:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:58:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:58:39.269 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:58:39.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa76806ecc0 msgr2=0x7fa768071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa76806ecc0 0x7fa768071180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa76c005fd0 tx=0x7fa76c005ee0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 msgr2=0x7fa77c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 0x7fa77c0735d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa76400bd70 tx=0x7fa76400be50 comp rx=0 tx=0).stop 2024-04-16T02:58:39.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 shutdown_connections 2024-04-16T02:58:39.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa76806ecc0 0x7fa768071180 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa77c100de0 0x7fa77c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 --2- 172.21.15.151:0/2404307693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa77c101790 0x7fa77c0735d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 >> 172.21.15.151:0/2404307693 conn(0x7fa77c0fc6b0 msgr2=0x7fa77c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:39.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 shutdown_connections 2024-04-16T02:58:39.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.268+0000 7fa781c1a700 1 -- 172.21.15.151:0/2404307693 wait complete. 2024-04-16T02:58:39.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.373+0000 7fb4b774d700 1 -- 172.21.15.151:0/3080130901 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 msgr2=0x7fb4b0105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.373+0000 7fb4b774d700 1 --2- 172.21.15.151:0/3080130901 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 0x7fb4b0105790 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009910 tx=0x7fb4ac009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:39.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.373+0000 7fb4b774d700 1 -- 172.21.15.151:0/3080130901 shutdown_connections 2024-04-16T02:58:39.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.373+0000 7fb4b774d700 1 --2- 172.21.15.151:0/3080130901 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 0x7fb4b0105790 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.373+0000 7fb4b774d700 1 --2- 172.21.15.151:0/3080130901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4b0068670 0x7fb4b0068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.373+0000 7fb4b774d700 1 -- 172.21.15.151:0/3080130901 >> 172.21.15.151:0/3080130901 conn(0x7fb4b00755a0 msgr2=0x7fb4b00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.373+0000 7fb4b774d700 1 -- 172.21.15.151:0/3080130901 shutdown_connections 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.373+0000 7fb4b774d700 1 -- 172.21.15.151:0/3080130901 wait complete. 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b774d700 1 Processor -- start 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b774d700 1 -- start start 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b774d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4b0068670 0x7fb4b0196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b774d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 0x7fb4b0196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b774d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b01972d0 con 0x7fb4b0069020 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b774d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b019a000 con 0x7fb4b0068670 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b4ce8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 0x7fb4b0196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:39.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b4ce8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 0x7fb4b0196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48214/0 (socket says 172.21.15.151:48214) 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b4ce8700 1 -- 172.21.15.151:0/3406267764 learned_addr learned my addr 172.21.15.151:0/3406267764 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b54e9700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4b0068670 0x7fb4b0196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b4ce8700 1 -- 172.21.15.151:0/3406267764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4b0068670 msgr2=0x7fb4b0196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b4ce8700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4b0068670 0x7fb4b0196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.374+0000 7fb4b4ce8700 1 -- 172.21.15.151:0/3406267764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac0095f0 con 0x7fb4b0069020 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.375+0000 7fb4b4ce8700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 0x7fb4b0196c40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0092e0 tx=0x7fb4ac00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.375+0000 7fb4a67fc700 1 -- 172.21.15.151:0/3406267764 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4ac01d070 con 0x7fb4b0069020 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.375+0000 7fb4a67fc700 1 -- 172.21.15.151:0/3406267764 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4ac005dc0 con 0x7fb4b0069020 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.375+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b019a280 con 0x7fb4b0069020 2024-04-16T02:58:39.379 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.375+0000 7fb4a67fc700 1 -- 172.21.15.151:0/3406267764 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4ac017400 con 0x7fb4b0069020 2024-04-16T02:58:39.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.375+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b019a770 con 0x7fb4b0069020 2024-04-16T02:58:39.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.376+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b01090a0 con 0x7fb4b0069020 2024-04-16T02:58:39.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.376+0000 7fb4a67fc700 1 -- 172.21.15.151:0/3406267764 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb4ac00f6f0 con 0x7fb4b0069020 2024-04-16T02:58:39.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.377+0000 7fb4a67fc700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb49c06ecc0 0x7fb49c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:39.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.377+0000 7fb4a67fc700 1 -- 172.21.15.151:0/3406267764 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb4ac090170 con 0x7fb4b0069020 2024-04-16T02:58:39.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.377+0000 7fb4b54e9700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb49c06ecc0 0x7fb49c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:39.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.377+0000 7fb4b54e9700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb49c06ecc0 0x7fb49c071180 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005950 tx=0x7fb4a000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:39.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.382+0000 7fb4a67fc700 1 -- 172.21.15.151:0/3406267764 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb4ac058fa0 con 0x7fb4b0069020 2024-04-16T02:58:39.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.625+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb4b004ea90 con 0x7fb4b0069020 2024-04-16T02:58:39.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.625+0000 7fb4a67fc700 1 -- 172.21.15.151:0/3406267764 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7fb4ac05c660 con 0x7fb4b0069020 2024-04-16T02:58:39.627 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:58:39.627 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:58:39.628 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:58:39.629 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:58:39.630 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:58:39.630 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:58:39.630 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:58:39.630 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:58:39.630 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:58:39.630 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:58:39.630 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:58:39.630 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:58:39.630 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb49c06ecc0 msgr2=0x7fb49c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb49c06ecc0 0x7fb49c071180 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005950 tx=0x7fb4a000b410 comp rx=0 tx=0).stop 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 msgr2=0x7fb4b0196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 0x7fb4b0196c40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0092e0 tx=0x7fb4ac00bd70 comp rx=0 tx=0).stop 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 shutdown_connections 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb49c06ecc0 0x7fb49c071180 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4b0068670 0x7fb4b0196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 --2- 172.21.15.151:0/3406267764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4b0069020 0x7fb4b0196c40 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 >> 172.21.15.151:0/3406267764 conn(0x7fb4b00755a0 msgr2=0x7fb4b0105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:39.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 shutdown_connections 2024-04-16T02:58:39.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.628+0000 7fb4b774d700 1 -- 172.21.15.151:0/3406267764 wait complete. 2024-04-16T02:58:39.632 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:58:39.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.734+0000 7fceac179700 1 -- 172.21.15.151:0/661251151 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea40ffbd0 msgr2=0x7fcea4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.734+0000 7fceac179700 1 --2- 172.21.15.151:0/661251151 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea40ffbd0 0x7fcea4100050 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fce94009910 tx=0x7fce94009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:39.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.734+0000 7fceac179700 1 -- 172.21.15.151:0/661251151 shutdown_connections 2024-04-16T02:58:39.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.734+0000 7fceac179700 1 --2- 172.21.15.151:0/661251151 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea40ffbd0 0x7fcea4100050 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.734+0000 7fceac179700 1 --2- 172.21.15.151:0/661251151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea4105a40 0x7fcea4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.734+0000 7fceac179700 1 -- 172.21.15.151:0/661251151 >> 172.21.15.151:0/661251151 conn(0x7fcea40755a0 msgr2=0x7fcea40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:39.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.734+0000 7fceac179700 1 -- 172.21.15.151:0/661251151 shutdown_connections 2024-04-16T02:58:39.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.734+0000 7fceac179700 1 -- 172.21.15.151:0/661251151 wait complete. 2024-04-16T02:58:39.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fceac179700 1 Processor -- start 2024-04-16T02:58:39.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fceac179700 1 -- start start 2024-04-16T02:58:39.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fceac179700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea40ffbd0 0x7fcea4101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:39.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fceac179700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea4105a40 0x7fcea4101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:39.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fceac179700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4102090 con 0x7fcea4105a40 2024-04-16T02:58:39.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fceac179700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea41021d0 con 0x7fcea40ffbd0 2024-04-16T02:58:39.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fcea9714700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea4105a40 0x7fcea4101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fcea9714700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea4105a40 0x7fcea4101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48232/0 (socket says 172.21.15.151:48232) 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fcea9714700 1 -- 172.21.15.151:0/2679171064 learned_addr learned my addr 172.21.15.151:0/2679171064 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fcea9f15700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea40ffbd0 0x7fcea4101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fcea9714700 1 -- 172.21.15.151:0/2679171064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea40ffbd0 msgr2=0x7fcea4101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fcea9714700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea40ffbd0 0x7fcea4101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fcea9714700 1 -- 172.21.15.151:0/2679171064 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce940095f0 con 0x7fcea4105a40 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.735+0000 7fcea9714700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea4105a40 0x7fcea4101b50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fce94006010 tx=0x7fce9400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.736+0000 7fce9affd700 1 -- 172.21.15.151:0/2679171064 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce9401d070 con 0x7fcea4105a40 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.736+0000 7fce9affd700 1 -- 172.21.15.151:0/2679171064 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce94005dc0 con 0x7fcea4105a40 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.736+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea4102450 con 0x7fcea4105a40 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.736+0000 7fce9affd700 1 -- 172.21.15.151:0/2679171064 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce94017400 con 0x7fcea4105a40 2024-04-16T02:58:39.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.736+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea41a2c30 con 0x7fcea4105a40 2024-04-16T02:58:39.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.737+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea4108fe0 con 0x7fcea4105a40 2024-04-16T02:58:39.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.737+0000 7fce9affd700 1 -- 172.21.15.151:0/2679171064 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fce94003a60 con 0x7fcea4105a40 2024-04-16T02:58:39.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.738+0000 7fce9affd700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce9006ecc0 0x7fce90071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:39.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.738+0000 7fce9affd700 1 -- 172.21.15.151:0/2679171064 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fce940902e0 con 0x7fcea4105a40 2024-04-16T02:58:39.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.738+0000 7fcea9f15700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce9006ecc0 0x7fce90071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:39.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.738+0000 7fcea9f15700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce9006ecc0 0x7fce90071180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcea0005950 tx=0x7fcea000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:39.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.742+0000 7fce9affd700 1 -- 172.21.15.151:0/2679171064 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fce94059090 con 0x7fcea4105a40 2024-04-16T02:58:39.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.963+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcea40611d0 con 0x7fce9006ecc0 2024-04-16T02:58:39.965 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:39 smithi151 ceph-mon[112233]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:39.966 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:39 smithi151 ceph-mon[112233]: pgmap v600: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:39.966 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:39 smithi151 ceph-mon[112233]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:39.966 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:58:39.966 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:39 smithi151 ceph-mon[112233]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:39.966 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:39 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2404307693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:58:39.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.964+0000 7fce9affd700 1 -- 172.21.15.151:0/2679171064 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcea40611d0 con 0x7fce9006ecc0 2024-04-16T02:58:39.966 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:58:39.967 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce9006ecc0 msgr2=0x7fce90071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce9006ecc0 0x7fce90071180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcea0005950 tx=0x7fcea000b410 comp rx=0 tx=0).stop 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea4105a40 msgr2=0x7fcea4101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea4105a40 0x7fcea4101b50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fce94006010 tx=0x7fce9400bd70 comp rx=0 tx=0).stop 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 shutdown_connections 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce9006ecc0 0x7fce90071180 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea40ffbd0 0x7fcea4101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 --2- 172.21.15.151:0/2679171064 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea4105a40 0x7fcea4101b50 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 >> 172.21.15.151:0/2679171064 conn(0x7fcea40755a0 msgr2=0x7fcea4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 shutdown_connections 2024-04-16T02:58:39.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:39.967+0000 7fceac179700 1 -- 172.21.15.151:0/2679171064 wait complete. 2024-04-16T02:58:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:39 smithi153 ceph-mon[87858]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:39 smithi153 ceph-mon[87858]: pgmap v600: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:39 smithi153 ceph-mon[87858]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:58:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:39 smithi153 ceph-mon[87858]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:40.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:39 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2404307693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:58:40.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.073+0000 7f865739e700 1 -- 172.21.15.151:0/3814601838 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8650105880 msgr2=0x7f8650105c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:40.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.073+0000 7f865739e700 1 --2- 172.21.15.151:0/3814601838 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8650105880 0x7f8650105c60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f864c009910 tx=0x7f864c009c20 comp rx=0 tx=0).stop 2024-04-16T02:58:40.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.073+0000 7f865739e700 1 -- 172.21.15.151:0/3814601838 shutdown_connections 2024-04-16T02:58:40.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.073+0000 7f865739e700 1 --2- 172.21.15.151:0/3814601838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86500ffa10 0x7f86500ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:40.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.073+0000 7f865739e700 1 --2- 172.21.15.151:0/3814601838 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8650105880 0x7f8650105c60 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:40.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.073+0000 7f865739e700 1 -- 172.21.15.151:0/3814601838 >> 172.21.15.151:0/3814601838 conn(0x7f8650075330 msgr2=0x7f8650075740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:40.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.073+0000 7f865739e700 1 -- 172.21.15.151:0/3814601838 shutdown_connections 2024-04-16T02:58:40.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.073+0000 7f865739e700 1 -- 172.21.15.151:0/3814601838 wait complete. 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f865739e700 1 Processor -- start 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f865739e700 1 -- start start 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f865739e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ffa10 0x7f8650196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f865739e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8650105880 0x7f8650196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f865739e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8650197100 con 0x7f86500ffa10 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f865739e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8650197240 con 0x7f8650105880 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f865639c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ffa10 0x7f8650196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f865639c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ffa10 0x7f8650196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48254/0 (socket says 172.21.15.151:48254) 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f865639c700 1 -- 172.21.15.151:0/3136059201 learned_addr learned my addr 172.21.15.151:0/3136059201 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.074+0000 7f8655b9b700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8650105880 0x7f8650196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f865639c700 1 -- 172.21.15.151:0/3136059201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8650105880 msgr2=0x7f8650196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:40.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f865639c700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8650105880 0x7f8650196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:40.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f865639c700 1 -- 172.21.15.151:0/3136059201 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c0095f0 con 0x7f86500ffa10 2024-04-16T02:58:40.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f8655b9b700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8650105880 0x7f8650196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:58:40.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f865639c700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ffa10 0x7f8650196570 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f864c006010 tx=0x7f864c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:40.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f86477fe700 1 -- 172.21.15.151:0/3136059201 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f864c01d070 con 0x7f86500ffa10 2024-04-16T02:58:40.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865019b030 con 0x7f86500ffa10 2024-04-16T02:58:40.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f86477fe700 1 -- 172.21.15.151:0/3136059201 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f864c004fb0 con 0x7f86500ffa10 2024-04-16T02:58:40.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f86477fe700 1 -- 172.21.15.151:0/3136059201 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f864c022ea0 con 0x7f86500ffa10 2024-04-16T02:58:40.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.075+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865019b520 con 0x7f86500ffa10 2024-04-16T02:58:40.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.076+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8650108e90 con 0x7f86500ffa10 2024-04-16T02:58:40.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.077+0000 7f86477fe700 1 -- 172.21.15.151:0/3136059201 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f864c022470 con 0x7f86500ffa10 2024-04-16T02:58:40.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.077+0000 7f86477fe700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:58:40.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.077+0000 7f86477fe700 1 -- 172.21.15.151:0/3136059201 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f864c090400 con 0x7f86500ffa10 2024-04-16T02:58:40.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.077+0000 7f8655b9b700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:58:40.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.077+0000 7f8655b9b700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8640005fd0 tx=0x7f8640005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:58:40.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.082+0000 7f86477fe700 1 -- 172.21.15.151:0/3136059201 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f864c059260 con 0x7f86500ffa10 2024-04-16T02:58:40.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.366+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f865004f2e0 con 0x7f86500ffa10 2024-04-16T02:58:40.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.367+0000 7f86477fe700 1 -- 172.21.15.151:0/3136059201 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f864c022720 con 0x7f86500ffa10 2024-04-16T02:58:40.369 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:58:40.369 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:58:40.369 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:58:40.369 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:58:40.369 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:58:40.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 msgr2=0x7f863c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:40.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8640005fd0 tx=0x7f8640005e20 comp rx=0 tx=0).stop 2024-04-16T02:58:40.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ffa10 msgr2=0x7f8650196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:58:40.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ffa10 0x7f8650196570 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f864c006010 tx=0x7f864c00be20 comp rx=0 tx=0).stop 2024-04-16T02:58:40.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 shutdown_connections 2024-04-16T02:58:40.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f863c06ecc0 0x7f863c071180 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:40.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500ffa10 0x7f8650196570 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:40.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 --2- 172.21.15.151:0/3136059201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8650105880 0x7f8650196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:58:40.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 >> 172.21.15.151:0/3136059201 conn(0x7f8650075330 msgr2=0x7f86500685b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:58:40.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 shutdown_connections 2024-04-16T02:58:40.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:58:40.369+0000 7f865739e700 1 -- 172.21.15.151:0/3136059201 wait complete. 2024-04-16T02:58:40.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:40 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3406267764' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:58:40.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:40 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3136059201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:58:41.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:40 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3406267764' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:58:41.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:40 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3136059201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:58:41.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:41 smithi151 ceph-mon[112233]: from='client.35146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:41.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:41 smithi151 ceph-mon[112233]: pgmap v601: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:42.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:41 smithi153 ceph-mon[87858]: from='client.35146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:58:42.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:41 smithi153 ceph-mon[87858]: pgmap v601: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:43.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:43 smithi151 ceph-mon[112233]: pgmap v602: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:44.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:43 smithi153 ceph-mon[87858]: pgmap v602: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:45.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:45 smithi151 ceph-mon[112233]: pgmap v603: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:45 smithi153 ceph-mon[87858]: pgmap v603: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:47.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:47 smithi151 ceph-mon[112233]: pgmap v604: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:48.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:47 smithi153 ceph-mon[87858]: pgmap v604: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:49.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:49 smithi151 ceph-mon[112233]: pgmap v605: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:49 smithi153 ceph-mon[87858]: pgmap v605: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:51.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:51 smithi151 ceph-mon[112233]: pgmap v606: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:52.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:51 smithi153 ceph-mon[87858]: pgmap v606: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:53.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:53 smithi153 ceph-mon[87858]: pgmap v607: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:53.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:53 smithi151 ceph-mon[112233]: pgmap v607: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:58:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:58:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:55 smithi151 ceph-mon[112233]: pgmap v608: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:55 smithi153 ceph-mon[87858]: pgmap v608: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:57.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:57 smithi151 ceph-mon[112233]: pgmap v609: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:58.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:57 smithi153 ceph-mon[87858]: pgmap v609: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:58:59.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:58:59 smithi151 ceph-mon[112233]: pgmap v610: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:00.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:58:59 smithi153 ceph-mon[87858]: pgmap v610: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:01 smithi151 ceph-mon[112233]: pgmap v611: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:01 smithi153 ceph-mon[87858]: pgmap v611: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:03.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:03 smithi151 ceph-mon[112233]: pgmap v612: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:04.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:03 smithi153 ceph-mon[87858]: pgmap v612: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:05.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:05 smithi151 ceph-mon[112233]: pgmap v613: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:06.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:05 smithi153 ceph-mon[87858]: pgmap v613: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:07.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:07 smithi151 ceph-mon[112233]: pgmap v614: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:08.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:07 smithi153 ceph-mon[87858]: pgmap v614: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:09.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:09 smithi151 ceph-mon[112233]: pgmap v615: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:09.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:59:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:09 smithi153 ceph-mon[87858]: pgmap v615: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:59:10.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.487+0000 7fcecfede700 1 -- 172.21.15.151:0/3703644127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8102610 msgr2=0x7fcec810ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.487+0000 7fcecfede700 1 --2- 172.21.15.151:0/3703644127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8102610 0x7fcec810ab00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009910 tx=0x7fcec4009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.487+0000 7fcecfede700 1 -- 172.21.15.151:0/3703644127 shutdown_connections 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.487+0000 7fcecfede700 1 --2- 172.21.15.151:0/3703644127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8102610 0x7fcec810ab00 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.487+0000 7fcecfede700 1 --2- 172.21.15.151:0/3703644127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcec8101c60 0x7fcec8102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.487+0000 7fcecfede700 1 -- 172.21.15.151:0/3703644127 >> 172.21.15.151:0/3703644127 conn(0x7fcec80fb540 msgr2=0x7fcec80fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.488+0000 7fcecfede700 1 -- 172.21.15.151:0/3703644127 shutdown_connections 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.488+0000 7fcecfede700 1 -- 172.21.15.151:0/3703644127 wait complete. 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.488+0000 7fcecfede700 1 Processor -- start 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.488+0000 7fcecfede700 1 -- start start 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.488+0000 7fcecfede700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8101c60 0x7fcec819c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.488+0000 7fcecfede700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcec8102610 0x7fcec819caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.488+0000 7fcecfede700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec819d180 con 0x7fcec8101c60 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.488+0000 7fcecfede700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec8196630 con 0x7fcec8102610 2024-04-16T02:59:10.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecdc7a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8101c60 0x7fcec819c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecdc7a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8101c60 0x7fcec819c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49754/0 (socket says 172.21.15.151:49754) 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecdc7a700 1 -- 172.21.15.151:0/2837951921 learned_addr learned my addr 172.21.15.151:0/2837951921 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecd479700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcec8102610 0x7fcec819caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecdc7a700 1 -- 172.21.15.151:0/2837951921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcec8102610 msgr2=0x7fcec819caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecdc7a700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcec8102610 0x7fcec819caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecdc7a700 1 -- 172.21.15.151:0/2837951921 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec40095f0 con 0x7fcec8101c60 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecdc7a700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8101c60 0x7fcec819c5b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fceb800d8d0 tx=0x7fceb800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcebeffd700 1 -- 172.21.15.151:0/2837951921 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceb80098a0 con 0x7fcec8101c60 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcebeffd700 1 -- 172.21.15.151:0/2837951921 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fceb8010460 con 0x7fcec8101c60 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec8196910 con 0x7fcec8101c60 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcebeffd700 1 -- 172.21.15.151:0/2837951921 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceb800f5f0 con 0x7fcec8101c60 2024-04-16T02:59:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.489+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec8196e60 con 0x7fcec8101c60 2024-04-16T02:59:10.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.490+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec8108200 con 0x7fcec8101c60 2024-04-16T02:59:10.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.491+0000 7fcebeffd700 1 -- 172.21.15.151:0/2837951921 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fceb8009a00 con 0x7fcec8101c60 2024-04-16T02:59:10.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.492+0000 7fcebeffd700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fceb406ecc0 0x7fceb4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:10.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.492+0000 7fcecd479700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fceb406ecc0 0x7fceb4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:10.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.492+0000 7fcebeffd700 1 -- 172.21.15.151:0/2837951921 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fceb808ed30 con 0x7fcec8101c60 2024-04-16T02:59:10.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.492+0000 7fcecd479700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fceb406ecc0 0x7fceb4071180 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcec80fcd70 tx=0x7fcec401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:10.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.497+0000 7fcebeffd700 1 -- 172.21.15.151:0/2837951921 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fceb805b610 con 0x7fcec8101c60 2024-04-16T02:59:10.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.715+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcec81970b0 con 0x7fceb406ecc0 2024-04-16T02:59:10.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.716+0000 7fcebeffd700 1 -- 172.21.15.151:0/2837951921 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcec81970b0 con 0x7fceb406ecc0 2024-04-16T02:59:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fceb406ecc0 msgr2=0x7fceb4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fceb406ecc0 0x7fceb4071180 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcec80fcd70 tx=0x7fcec401a040 comp rx=0 tx=0).stop 2024-04-16T02:59:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8101c60 msgr2=0x7fcec819c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8101c60 0x7fcec819c5b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fceb800d8d0 tx=0x7fceb800dbe0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 shutdown_connections 2024-04-16T02:59:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fceb406ecc0 0x7fceb4071180 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcec8101c60 0x7fcec819c5b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 --2- 172.21.15.151:0/2837951921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcec8102610 0x7fcec819caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 >> 172.21.15.151:0/2837951921 conn(0x7fcec80fb540 msgr2=0x7fcec8105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.718+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 shutdown_connections 2024-04-16T02:59:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.719+0000 7fcecfede700 1 -- 172.21.15.151:0/2837951921 wait complete. 2024-04-16T02:59:10.731 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:59:10.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.824+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/1184525388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a64100410 msgr2=0x7f7a641048e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:10.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.824+0000 7f7a6b3d2700 1 --2- 172.21.15.151:0/1184525388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a64100410 0x7f7a641048e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009910 tx=0x7f7a58009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:10.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.824+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/1184525388 shutdown_connections 2024-04-16T02:59:10.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.824+0000 7f7a6b3d2700 1 --2- 172.21.15.151:0/1184525388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a64100410 0x7f7a641048e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.824+0000 7f7a6b3d2700 1 --2- 172.21.15.151:0/1184525388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a640ffa60 0x7f7a640ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.824+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/1184525388 >> 172.21.15.151:0/1184525388 conn(0x7f7a640fb570 msgr2=0x7f7a640fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:10.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/1184525388 shutdown_connections 2024-04-16T02:59:10.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/1184525388 wait complete. 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6b3d2700 1 Processor -- start 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6b3d2700 1 -- start start 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6b3d2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a640ffa60 0x7f7a6419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6b3d2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a64100410 0x7f7a6419d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6b3d2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6419d950 con 0x7f7a640ffa60 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6b3d2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641a16e0 con 0x7f7a64100410 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6916e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a640ffa60 0x7f7a6419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.825+0000 7f7a6916e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a640ffa60 0x7f7a6419cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49772/0 (socket says 172.21.15.151:49772) 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a6916e700 1 -- 172.21.15.151:0/3614996196 learned_addr learned my addr 172.21.15.151:0/3614996196 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a6896d700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a64100410 0x7f7a6419d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a6916e700 1 -- 172.21.15.151:0/3614996196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a64100410 msgr2=0x7f7a6419d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a6916e700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a64100410 0x7f7a6419d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:10.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a6916e700 1 -- 172.21.15.151:0/3614996196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a580095f0 con 0x7f7a640ffa60 2024-04-16T02:59:10.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a6916e700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a640ffa60 0x7f7a6419cd30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7a540060b0 tx=0x7f7a5400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:10.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a627fc700 1 -- 172.21.15.151:0/3614996196 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a5400dda0 con 0x7f7a640ffa60 2024-04-16T02:59:10.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a627fc700 1 -- 172.21.15.151:0/3614996196 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a5400f040 con 0x7f7a640ffa60 2024-04-16T02:59:10.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a627fc700 1 -- 172.21.15.151:0/3614996196 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a540146f0 con 0x7f7a640ffa60 2024-04-16T02:59:10.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a641a19c0 con 0x7f7a640ffa60 2024-04-16T02:59:10.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.826+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a641a1ee0 con 0x7f7a640ffa60 2024-04-16T02:59:10.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.827+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a64102040 con 0x7f7a640ffa60 2024-04-16T02:59:10.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.828+0000 7f7a627fc700 1 -- 172.21.15.151:0/3614996196 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7a54015400 con 0x7f7a640ffa60 2024-04-16T02:59:10.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.828+0000 7f7a627fc700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a4c06ecc0 0x7f7a4c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:10.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.828+0000 7f7a6896d700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a4c06ecc0 0x7f7a4c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:10.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.828+0000 7f7a627fc700 1 -- 172.21.15.151:0/3614996196 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7a5408e780 con 0x7f7a640ffa60 2024-04-16T02:59:10.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.829+0000 7f7a6896d700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a4c06ecc0 0x7f7a4c071180 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7a58006010 tx=0x7f7a5801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:10.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:10.833+0000 7f7a627fc700 1 -- 172.21.15.151:0/3614996196 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7a54057660 con 0x7f7a640ffa60 2024-04-16T02:59:11.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.053+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a641a22d0 con 0x7f7a4c06ecc0 2024-04-16T02:59:11.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.054+0000 7f7a627fc700 1 -- 172.21.15.151:0/3614996196 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7a641a22d0 con 0x7f7a4c06ecc0 2024-04-16T02:59:11.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a4c06ecc0 msgr2=0x7f7a4c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a4c06ecc0 0x7f7a4c071180 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7a58006010 tx=0x7f7a5801a040 comp rx=0 tx=0).stop 2024-04-16T02:59:11.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a640ffa60 msgr2=0x7f7a6419cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a640ffa60 0x7f7a6419cd30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7a540060b0 tx=0x7f7a5400d3b0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 shutdown_connections 2024-04-16T02:59:11.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a4c06ecc0 0x7f7a4c071180 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a640ffa60 0x7f7a6419cd30 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 --2- 172.21.15.151:0/3614996196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a64100410 0x7f7a6419d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 >> 172.21.15.151:0/3614996196 conn(0x7f7a640fb570 msgr2=0x7f7a64110f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:11.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 shutdown_connections 2024-04-16T02:59:11.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.056+0000 7f7a6b3d2700 1 -- 172.21.15.151:0/3614996196 wait complete. 2024-04-16T02:59:11.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.164+0000 7fe05a834700 1 -- 172.21.15.151:0/3636635157 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 msgr2=0x7fe054100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.164+0000 7fe05a834700 1 --2- 172.21.15.151:0/3636635157 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 0x7fe054100050 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009910 tx=0x7fe03c009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:11.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.164+0000 7fe05a834700 1 -- 172.21.15.151:0/3636635157 shutdown_connections 2024-04-16T02:59:11.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.164+0000 7fe05a834700 1 --2- 172.21.15.151:0/3636635157 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 0x7fe054100050 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.164+0000 7fe05a834700 1 --2- 172.21.15.151:0/3636635157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe054105a40 0x7fe054105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.164+0000 7fe05a834700 1 -- 172.21.15.151:0/3636635157 >> 172.21.15.151:0/3636635157 conn(0x7fe0540755a0 msgr2=0x7fe0540759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:11.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.164+0000 7fe05a834700 1 -- 172.21.15.151:0/3636635157 shutdown_connections 2024-04-16T02:59:11.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.164+0000 7fe05a834700 1 -- 172.21.15.151:0/3636635157 wait complete. 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe05a834700 1 Processor -- start 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe05a834700 1 -- start start 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe05a834700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 0x7fe05419c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe05a834700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe054105a40 0x7fe05419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe05a834700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe054196570 con 0x7fe0540ffbd0 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe05a834700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0541966e0 con 0x7fe054105a40 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe053fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 0x7fe05419c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe053fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 0x7fe05419c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49786/0 (socket says 172.21.15.151:49786) 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe053fff700 1 -- 172.21.15.151:0/170406544 learned_addr learned my addr 172.21.15.151:0/170406544 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:11.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe04bfff700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe054105a40 0x7fe05419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe053fff700 1 -- 172.21.15.151:0/170406544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe054105a40 msgr2=0x7fe05419cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe053fff700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe054105a40 0x7fe05419cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe053fff700 1 -- 172.21.15.151:0/170406544 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c0095f0 con 0x7fe0540ffbd0 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.165+0000 7fe053fff700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 0x7fe05419c580 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe04400d8d0 tx=0x7fe04400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.166+0000 7fe051ffb700 1 -- 172.21.15.151:0/170406544 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0440098a0 con 0x7fe0540ffbd0 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.166+0000 7fe051ffb700 1 -- 172.21.15.151:0/170406544 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe044010460 con 0x7fe0540ffbd0 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.166+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0541969c0 con 0x7fe0540ffbd0 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.166+0000 7fe051ffb700 1 -- 172.21.15.151:0/170406544 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe04400f5f0 con 0x7fe0540ffbd0 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.166+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe054196ee0 con 0x7fe0540ffbd0 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.166+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe054107fd0 con 0x7fe0540ffbd0 2024-04-16T02:59:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.167+0000 7fe051ffb700 1 -- 172.21.15.151:0/170406544 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe044009a00 con 0x7fe0540ffbd0 2024-04-16T02:59:11.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.168+0000 7fe051ffb700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe04006ecc0 0x7fe040071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:11.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.168+0000 7fe051ffb700 1 -- 172.21.15.151:0/170406544 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe04408ec30 con 0x7fe0540ffbd0 2024-04-16T02:59:11.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.168+0000 7fe04bfff700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe04006ecc0 0x7fe040071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:11.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.168+0000 7fe04bfff700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe04006ecc0 0x7fe040071180 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe03c00b5c0 tx=0x7fe03c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:11.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.172+0000 7fe051ffb700 1 -- 172.21.15.151:0/170406544 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe044057a60 con 0x7fe0540ffbd0 2024-04-16T02:59:11.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.383+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe05410dd60 con 0x7fe04006ecc0 2024-04-16T02:59:11.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.394+0000 7fe051ffb700 1 -- 172.21.15.151:0/170406544 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe05410dd60 con 0x7fe04006ecc0 2024-04-16T02:59:11.396 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (20m) 14m ago 32m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (32m) 14m ago 32m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (31m) 13m ago 31m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (18m) 14m ago 32m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (18m) 13m ago 31m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (20m) 14m ago 31m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (14m) 14m ago 26m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (14m) 14m ago 26m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (26m) 13m ago 26m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (13m) 13m ago 26m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:59:11.397 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (22m) 14m ago 33m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (21m) 13m ago 31m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (19m) 14m ago 33m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (19m) 13m ago 30m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (21m) 14m ago 32m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (21m) 13m ago 31m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (17m) 14m ago 30m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (17m) 14m ago 29m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (16m) 14m ago 29m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (16m) 13m ago 28m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (15m) 13m ago 28m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:59:11.398 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (14m) 13m ago 28m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:59:11.399 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (21m) 14m ago 31m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.397+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe04006ecc0 msgr2=0x7fe040071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.397+0000 7fe05a834700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe04006ecc0 0x7fe040071180 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe03c00b5c0 tx=0x7fe03c01a040 comp rx=0 tx=0).stop 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.397+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 msgr2=0x7fe05419c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.397+0000 7fe05a834700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 0x7fe05419c580 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe04400d8d0 tx=0x7fe04400dbe0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.397+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 shutdown_connections 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.397+0000 7fe05a834700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe04006ecc0 0x7fe040071180 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.397+0000 7fe05a834700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0540ffbd0 0x7fe05419c580 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.397+0000 7fe05a834700 1 --2- 172.21.15.151:0/170406544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe054105a40 0x7fe05419cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.397+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 >> 172.21.15.151:0/170406544 conn(0x7fe0540755a0 msgr2=0x7fe0540fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:11.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.398+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 shutdown_connections 2024-04-16T02:59:11.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.398+0000 7fe05a834700 1 -- 172.21.15.151:0/170406544 wait complete. 2024-04-16T02:59:11.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.501+0000 7f5deeba0700 1 -- 172.21.15.151:0/3016173910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de80ffbd0 msgr2=0x7f5de8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.501+0000 7f5deeba0700 1 --2- 172.21.15.151:0/3016173910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de80ffbd0 0x7f5de8100050 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800b0a0 tx=0x7f5dd800b3b0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.501+0000 7f5deeba0700 1 -- 172.21.15.151:0/3016173910 shutdown_connections 2024-04-16T02:59:11.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.501+0000 7f5deeba0700 1 --2- 172.21.15.151:0/3016173910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de80ffbd0 0x7f5de8100050 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.501+0000 7f5deeba0700 1 --2- 172.21.15.151:0/3016173910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de8105a40 0x7f5de8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.501+0000 7f5deeba0700 1 -- 172.21.15.151:0/3016173910 >> 172.21.15.151:0/3016173910 conn(0x7f5de80755a0 msgr2=0x7f5de80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:11.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5deeba0700 1 -- 172.21.15.151:0/3016173910 shutdown_connections 2024-04-16T02:59:11.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5deeba0700 1 -- 172.21.15.151:0/3016173910 wait complete. 2024-04-16T02:59:11.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5deeba0700 1 Processor -- start 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5deeba0700 1 -- start start 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5deeba0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80ffbd0 0x7f5de819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5deeba0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de8105a40 0x7f5de819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5deeba0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de819d0d0 con 0x7f5de8105a40 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5deeba0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de8196580 con 0x7f5de80ffbd0 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5de7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de8105a40 0x7f5de819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5de7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de8105a40 0x7f5de819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49790/0 (socket says 172.21.15.151:49790) 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.502+0000 7f5de7fff700 1 -- 172.21.15.151:0/3758406935 learned_addr learned my addr 172.21.15.151:0/3758406935 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5dec93c700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80ffbd0 0x7f5de819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5de7fff700 1 -- 172.21.15.151:0/3758406935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80ffbd0 msgr2=0x7f5de819c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5de7fff700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80ffbd0 0x7f5de819c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5de7fff700 1 -- 172.21.15.151:0/3758406935 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd8009d00 con 0x7f5de8105a40 2024-04-16T02:59:11.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5de7fff700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de8105a40 0x7f5de819ca40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8000f80 tx=0x7f5dd800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:11.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5de5ffb700 1 -- 172.21.15.151:0/3758406935 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd800e040 con 0x7f5de8105a40 2024-04-16T02:59:11.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5de5ffb700 1 -- 172.21.15.151:0/3758406935 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5dd8003c10 con 0x7f5de8105a40 2024-04-16T02:59:11.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de8196800 con 0x7f5de8105a40 2024-04-16T02:59:11.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5de5ffb700 1 -- 172.21.15.151:0/3758406935 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd801d6b0 con 0x7f5de8105a40 2024-04-16T02:59:11.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.503+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de8196cf0 con 0x7f5de8105a40 2024-04-16T02:59:11.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.504+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de8108fe0 con 0x7f5de8105a40 2024-04-16T02:59:11.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.505+0000 7f5de5ffb700 1 -- 172.21.15.151:0/3758406935 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5dd8004040 con 0x7f5de8105a40 2024-04-16T02:59:11.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.505+0000 7f5de5ffb700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5dd406ecc0 0x7f5dd4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:11.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.505+0000 7f5de5ffb700 1 -- 172.21.15.151:0/3758406935 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5dd8090590 con 0x7f5de8105a40 2024-04-16T02:59:11.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.505+0000 7f5dec93c700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5dd406ecc0 0x7f5dd4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:11.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.506+0000 7f5dec93c700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5dd406ecc0 0x7f5dd4071180 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5de0005950 tx=0x7f5de000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:11.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.510+0000 7f5de5ffb700 1 -- 172.21.15.151:0/3758406935 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5dd805cef0 con 0x7f5de8105a40 2024-04-16T02:59:11.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:11 smithi151 ceph-mon[112233]: pgmap v616: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:11.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:11 smithi151 ceph-mon[112233]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:11.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.791+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5de804ea90 con 0x7f5de8105a40 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.792+0000 7f5de5ffb700 1 -- 172.21.15.151:0/3758406935 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f5dd805ca80 con 0x7f5de8105a40 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:59:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:59:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:59:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:59:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:59:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:59:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:59:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:59:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:59:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:59:11.795 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:59:11.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.794+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5dd406ecc0 msgr2=0x7f5dd4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.794+0000 7f5deeba0700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5dd406ecc0 0x7f5dd4071180 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5de0005950 tx=0x7f5de000b410 comp rx=0 tx=0).stop 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.794+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de8105a40 msgr2=0x7f5de819ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.794+0000 7f5deeba0700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de8105a40 0x7f5de819ca40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8000f80 tx=0x7f5dd800cd20 comp rx=0 tx=0).stop 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.795+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 shutdown_connections 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.795+0000 7f5deeba0700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5dd406ecc0 0x7f5dd4071180 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.795+0000 7f5deeba0700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80ffbd0 0x7f5de819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.795+0000 7f5deeba0700 1 --2- 172.21.15.151:0/3758406935 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de8105a40 0x7f5de819ca40 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.795+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 >> 172.21.15.151:0/3758406935 conn(0x7f5de80755a0 msgr2=0x7f5de8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.795+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 shutdown_connections 2024-04-16T02:59:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.795+0000 7f5deeba0700 1 -- 172.21.15.151:0/3758406935 wait complete. 2024-04-16T02:59:11.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.899+0000 7f3274e51700 1 -- 172.21.15.151:0/1216346538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3270105a40 msgr2=0x7f3270105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.899+0000 7f3274e51700 1 --2- 172.21.15.151:0/1216346538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3270105a40 0x7f3270105e20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3260009910 tx=0x7f3260009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:11.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.900+0000 7f3274e51700 1 -- 172.21.15.151:0/1216346538 shutdown_connections 2024-04-16T02:59:11.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.900+0000 7f3274e51700 1 --2- 172.21.15.151:0/1216346538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32700ffa40 0x7f32700ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.900+0000 7f3274e51700 1 --2- 172.21.15.151:0/1216346538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3270105a40 0x7f3270105e20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.900+0000 7f3274e51700 1 -- 172.21.15.151:0/1216346538 >> 172.21.15.151:0/1216346538 conn(0x7f32700fb550 msgr2=0x7f32700fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:11.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.900+0000 7f3274e51700 1 -- 172.21.15.151:0/1216346538 shutdown_connections 2024-04-16T02:59:11.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.900+0000 7f3274e51700 1 -- 172.21.15.151:0/1216346538 wait complete. 2024-04-16T02:59:11.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.900+0000 7f3274e51700 1 Processor -- start 2024-04-16T02:59:11.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.900+0000 7f3274e51700 1 -- start start 2024-04-16T02:59:11.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f3274e51700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32700ffa40 0x7f32701976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:11.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f3274e51700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3270105a40 0x7f3270197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:11.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f3274e51700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32701982f0 con 0x7f32700ffa40 2024-04-16T02:59:11.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f3274e51700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3270198460 con 0x7f3270105a40 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f326e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32700ffa40 0x7f32701976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f326e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32700ffa40 0x7f32701976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49816/0 (socket says 172.21.15.151:49816) 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f326e59c700 1 -- 172.21.15.151:0/933392363 learned_addr learned my addr 172.21.15.151:0/933392363 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f326dd9b700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3270105a40 0x7f3270197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f326e59c700 1 -- 172.21.15.151:0/933392363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3270105a40 msgr2=0x7f3270197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f326e59c700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3270105a40 0x7f3270197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f326e59c700 1 -- 172.21.15.151:0/933392363 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32600095f0 con 0x7f32700ffa40 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f326e59c700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32700ffa40 0x7f32701976d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3260006010 tx=0x7f326000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f325f7fe700 1 -- 172.21.15.151:0/933392363 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f326001d070 con 0x7f32700ffa40 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f325f7fe700 1 -- 172.21.15.151:0/933392363 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3260004fb0 con 0x7f32700ffa40 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32701a0740 con 0x7f32700ffa40 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.901+0000 7f325f7fe700 1 -- 172.21.15.151:0/933392363 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3260022ea0 con 0x7f32700ffa40 2024-04-16T02:59:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.902+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32701a0bb0 con 0x7f32700ffa40 2024-04-16T02:59:11.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.902+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3270108040 con 0x7f32700ffa40 2024-04-16T02:59:11.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.903+0000 7f325f7fe700 1 -- 172.21.15.151:0/933392363 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3260022470 con 0x7f32700ffa40 2024-04-16T02:59:11.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.904+0000 7f325f7fe700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:11.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.904+0000 7f326dd9b700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:11.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.904+0000 7f325f7fe700 1 -- 172.21.15.151:0/933392363 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f32600903f0 con 0x7f32700ffa40 2024-04-16T02:59:11.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.904+0000 7f326dd9b700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3264006fd0 tx=0x7f3264008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:11.910 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:11.908+0000 7f325f7fe700 1 -- 172.21.15.151:0/933392363 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3260059250 con 0x7f32700ffa40 2024-04-16T02:59:12.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:11 smithi153 ceph-mon[87858]: pgmap v616: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:12.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:11 smithi153 ceph-mon[87858]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:12.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.151+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3270107a20 con 0x7f32700ffa40 2024-04-16T02:59:12.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.152+0000 7f325f7fe700 1 -- 172.21.15.151:0/933392363 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f3260027770 con 0x7f32700ffa40 2024-04-16T02:59:12.154 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:59:12.154 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:59:12.154 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:59:12.154 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:59:12.154 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:59:12.154 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:59:12.154 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:59:12.155 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:59:12.156 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:59:12.157 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:59:12.157 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:59:12.157 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:59:12.157 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:59:12.157 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:59:12.157 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:59:12.157 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:59:12.157 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:59:12.157 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:59:12.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 msgr2=0x7f3258071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3264006fd0 tx=0x7f3264008040 comp rx=0 tx=0).stop 2024-04-16T02:59:12.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32700ffa40 msgr2=0x7f32701976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32700ffa40 0x7f32701976d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3260006010 tx=0x7f326000be20 comp rx=0 tx=0).stop 2024-04-16T02:59:12.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 shutdown_connections 2024-04-16T02:59:12.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f325806ecc0 0x7f3258071180 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32700ffa40 0x7f32701976d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 --2- 172.21.15.151:0/933392363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3270105a40 0x7f3270197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 >> 172.21.15.151:0/933392363 conn(0x7f32700fb550 msgr2=0x7f32700fc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:12.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 shutdown_connections 2024-04-16T02:59:12.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.154+0000 7f3274e51700 1 -- 172.21.15.151:0/933392363 wait complete. 2024-04-16T02:59:12.159 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.259+0000 7efbf81c8700 1 -- 172.21.15.151:0/830910724 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0100de0 msgr2=0x7efbf01011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.259+0000 7efbf81c8700 1 --2- 172.21.15.151:0/830910724 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0100de0 0x7efbf01011c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7efbec00b0a0 tx=0x7efbec00b3b0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.259+0000 7efbf81c8700 1 -- 172.21.15.151:0/830910724 shutdown_connections 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.259+0000 7efbf81c8700 1 --2- 172.21.15.151:0/830910724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf0101790 0x7efbf01057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.259+0000 7efbf81c8700 1 --2- 172.21.15.151:0/830910724 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0100de0 0x7efbf01011c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.259+0000 7efbf81c8700 1 -- 172.21.15.151:0/830910724 >> 172.21.15.151:0/830910724 conn(0x7efbf00fc670 msgr2=0x7efbf00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.259+0000 7efbf81c8700 1 -- 172.21.15.151:0/830910724 shutdown_connections 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.259+0000 7efbf81c8700 1 -- 172.21.15.151:0/830910724 wait complete. 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf81c8700 1 Processor -- start 2024-04-16T02:59:12.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf81c8700 1 -- start start 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf81c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf0100de0 0x7efbf019c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf81c8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0101790 0x7efbf019caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf81c8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf019d110 con 0x7efbf0101790 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf81c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf01966a0 con 0x7efbf0100de0 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf5763700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0101790 0x7efbf019caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf5763700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0101790 0x7efbf019caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49838/0 (socket says 172.21.15.151:49838) 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf5763700 1 -- 172.21.15.151:0/1074727820 learned_addr learned my addr 172.21.15.151:0/1074727820 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf5f64700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf0100de0 0x7efbf019c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf5763700 1 -- 172.21.15.151:0/1074727820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf0100de0 msgr2=0x7efbf019c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf5763700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf0100de0 0x7efbf019c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.260+0000 7efbf5763700 1 -- 172.21.15.151:0/1074727820 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbec009d00 con 0x7efbf0101790 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.261+0000 7efbf5763700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0101790 0x7efbf019caf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7efbe000b480 tx=0x7efbe000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:12.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.261+0000 7efbe6ffd700 1 -- 172.21.15.151:0/1074727820 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbe0011800 con 0x7efbf0101790 2024-04-16T02:59:12.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.261+0000 7efbe6ffd700 1 -- 172.21.15.151:0/1074727820 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbe0011e20 con 0x7efbf0101790 2024-04-16T02:59:12.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.261+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf0196980 con 0x7efbf0101790 2024-04-16T02:59:12.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.261+0000 7efbe6ffd700 1 -- 172.21.15.151:0/1074727820 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbe0010ea0 con 0x7efbf0101790 2024-04-16T02:59:12.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.261+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf0196ed0 con 0x7efbf0101790 2024-04-16T02:59:12.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.262+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf0109180 con 0x7efbf0101790 2024-04-16T02:59:12.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.262+0000 7efbe6ffd700 1 -- 172.21.15.151:0/1074727820 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efbe001a400 con 0x7efbf0101790 2024-04-16T02:59:12.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.263+0000 7efbe6ffd700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efbdc06ecc0 0x7efbdc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:12.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.263+0000 7efbe6ffd700 1 -- 172.21.15.151:0/1074727820 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efbe008e940 con 0x7efbf0101790 2024-04-16T02:59:12.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.263+0000 7efbf5f64700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efbdc06ecc0 0x7efbdc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:12.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.263+0000 7efbf5f64700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efbdc06ecc0 0x7efbdc071180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efbec000f80 tx=0x7efbec0060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:12.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.267+0000 7efbe6ffd700 1 -- 172.21.15.151:0/1074727820 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efbe005b300 con 0x7efbf0101790 2024-04-16T02:59:12.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.489+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efbf01971a0 con 0x7efbdc06ecc0 2024-04-16T02:59:12.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.490+0000 7efbe6ffd700 1 -- 172.21.15.151:0/1074727820 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efbf01971a0 con 0x7efbdc06ecc0 2024-04-16T02:59:12.492 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:59:12.492 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:59:12.492 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:59:12.492 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:59:12.493 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:59:12.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.492+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efbdc06ecc0 msgr2=0x7efbdc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efbdc06ecc0 0x7efbdc071180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efbec000f80 tx=0x7efbec0060c0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0101790 msgr2=0x7efbf019caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0101790 0x7efbf019caf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7efbe000b480 tx=0x7efbe000b790 comp rx=0 tx=0).stop 2024-04-16T02:59:12.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 shutdown_connections 2024-04-16T02:59:12.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efbdc06ecc0 0x7efbdc071180 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf0100de0 0x7efbf019c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 --2- 172.21.15.151:0/1074727820 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efbf0101790 0x7efbf019caf0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 >> 172.21.15.151:0/1074727820 conn(0x7efbf00fc670 msgr2=0x7efbf00fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:12.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 shutdown_connections 2024-04-16T02:59:12.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.493+0000 7efbf81c8700 1 -- 172.21.15.151:0/1074727820 wait complete. 2024-04-16T02:59:12.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.597+0000 7f64a455f700 1 -- 172.21.15.151:0/386292472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c101750 msgr2=0x7f649c1057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.597+0000 7f64a455f700 1 --2- 172.21.15.151:0/386292472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c101750 0x7f649c1057a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6490009910 tx=0x7f6490009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:12.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.597+0000 7f64a455f700 1 -- 172.21.15.151:0/386292472 shutdown_connections 2024-04-16T02:59:12.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.597+0000 7f64a455f700 1 --2- 172.21.15.151:0/386292472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c101750 0x7f649c1057a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.597+0000 7f64a455f700 1 --2- 172.21.15.151:0/386292472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f649c100da0 0x7f649c101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.597+0000 7f64a455f700 1 -- 172.21.15.151:0/386292472 >> 172.21.15.151:0/386292472 conn(0x7f649c0fc630 msgr2=0x7f649c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.597+0000 7f64a455f700 1 -- 172.21.15.151:0/386292472 shutdown_connections 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.597+0000 7f64a455f700 1 -- 172.21.15.151:0/386292472 wait complete. 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a455f700 1 Processor -- start 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a455f700 1 -- start start 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a455f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c100da0 0x7f649c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a455f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f649c101750 0x7f649c198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a455f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c199520 con 0x7f649c100da0 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a455f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c19d330 con 0x7f649c101750 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a22fb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c100da0 0x7f649c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a22fb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c100da0 0x7f649c198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49868/0 (socket says 172.21.15.151:49868) 2024-04-16T02:59:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a22fb700 1 -- 172.21.15.151:0/2696950002 learned_addr learned my addr 172.21.15.151:0/2696950002 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a1afa700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f649c101750 0x7f649c198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a22fb700 1 -- 172.21.15.151:0/2696950002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f649c101750 msgr2=0x7f649c198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a22fb700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f649c101750 0x7f649c198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.598+0000 7f64a22fb700 1 -- 172.21.15.151:0/2696950002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64900095f0 con 0x7f649c100da0 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.599+0000 7f64a22fb700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c100da0 0x7f649c198950 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6494007e90 tx=0x7f649400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.599+0000 7f649b7fe700 1 -- 172.21.15.151:0/2696950002 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f649400de60 con 0x7f649c100da0 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.599+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c19d610 con 0x7f649c100da0 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.599+0000 7f649b7fe700 1 -- 172.21.15.151:0/2696950002 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f649400f040 con 0x7f649c100da0 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.599+0000 7f649b7fe700 1 -- 172.21.15.151:0/2696950002 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6494004520 con 0x7f649c100da0 2024-04-16T02:59:12.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.599+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c19db30 con 0x7f649c100da0 2024-04-16T02:59:12.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.600+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c104f20 con 0x7f649c100da0 2024-04-16T02:59:12.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.600+0000 7f649b7fe700 1 -- 172.21.15.151:0/2696950002 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6494004750 con 0x7f649c100da0 2024-04-16T02:59:12.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.601+0000 7f649b7fe700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f648406ecc0 0x7f6484071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:12.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.601+0000 7f649b7fe700 1 -- 172.21.15.151:0/2696950002 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f649408ea80 con 0x7f649c100da0 2024-04-16T02:59:12.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.601+0000 7f64a1afa700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f648406ecc0 0x7f6484071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:12.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.601+0000 7f64a1afa700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f648406ecc0 0x7f6484071180 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f649c1998c0 tx=0x7f649001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:12.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.606+0000 7f649b7fe700 1 -- 172.21.15.151:0/2696950002 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f649405b360 con 0x7f649c100da0 2024-04-16T02:59:12.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:12 smithi151 ceph-mon[112233]: from='client.35158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:12.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:12 smithi151 ceph-mon[112233]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:12.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:12 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3758406935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:59:12.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:12 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/933392363' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:59:12.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.889+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f649c04ea90 con 0x7f649c100da0 2024-04-16T02:59:12.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.890+0000 7f649b7fe700 1 -- 172.21.15.151:0/2696950002 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6494019090 con 0x7f649c100da0 2024-04-16T02:59:12.892 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:59:12.892 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:59:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:59:12.892 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:59:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:59:12.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f648406ecc0 msgr2=0x7f6484071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f648406ecc0 0x7f6484071180 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f649c1998c0 tx=0x7f649001a040 comp rx=0 tx=0).stop 2024-04-16T02:59:12.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c100da0 msgr2=0x7f649c198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:12.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c100da0 0x7f649c198950 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6494007e90 tx=0x7f649400d3b0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 shutdown_connections 2024-04-16T02:59:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f648406ecc0 0x7f6484071180 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f649c100da0 0x7f649c198950 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 --2- 172.21.15.151:0/2696950002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f649c101750 0x7f649c198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 >> 172.21.15.151:0/2696950002 conn(0x7f649c0fc630 msgr2=0x7f649c0fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 shutdown_connections 2024-04-16T02:59:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:12.892+0000 7f64a455f700 1 -- 172.21.15.151:0/2696950002 wait complete. 2024-04-16T02:59:13.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:12 smithi153 ceph-mon[87858]: from='client.35158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:13.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:12 smithi153 ceph-mon[87858]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:13.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:12 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3758406935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:59:13.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:12 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/933392363' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:59:13.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:13 smithi151 ceph-mon[112233]: pgmap v617: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:13.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:13 smithi151 ceph-mon[112233]: from='client.35174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:13.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:13 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2696950002' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:59:14.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:13 smithi153 ceph-mon[87858]: pgmap v617: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:14.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:13 smithi153 ceph-mon[87858]: from='client.35174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:14.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:13 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2696950002' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:59:15.958 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:15 smithi151 ceph-mon[112233]: pgmap v618: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:15 smithi153 ceph-mon[87858]: pgmap v618: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:17.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:17 smithi151 ceph-mon[112233]: pgmap v619: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:18.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:17 smithi153 ceph-mon[87858]: pgmap v619: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:19.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:19 smithi151 ceph-mon[112233]: pgmap v620: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:20.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:19 smithi153 ceph-mon[87858]: pgmap v620: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:21.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:21 smithi151 ceph-mon[112233]: pgmap v621: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:22.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:21 smithi153 ceph-mon[87858]: pgmap v621: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:23.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:23 smithi151 ceph-mon[112233]: pgmap v622: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:24.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:23 smithi153 ceph-mon[87858]: pgmap v622: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:59:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:59:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:25 smithi151 ceph-mon[112233]: pgmap v623: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:25 smithi153 ceph-mon[87858]: pgmap v623: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:27.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:27 smithi151 ceph-mon[112233]: pgmap v624: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:27 smithi153 ceph-mon[87858]: pgmap v624: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:29.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:29 smithi151 ceph-mon[112233]: pgmap v625: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:29 smithi153 ceph-mon[87858]: pgmap v625: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:31.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:31 smithi151 ceph-mon[112233]: pgmap v626: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:32.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:31 smithi153 ceph-mon[87858]: pgmap v626: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:33.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:33 smithi151 ceph-mon[112233]: pgmap v627: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:34.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:33 smithi153 ceph-mon[87858]: pgmap v627: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:35.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:35 smithi151 ceph-mon[112233]: pgmap v628: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:35 smithi153 ceph-mon[87858]: pgmap v628: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:37.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:37 smithi151 ceph-mon[112233]: pgmap v629: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:38.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:37 smithi153 ceph-mon[87858]: pgmap v629: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:39.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:39 smithi151 ceph-mon[112233]: pgmap v630: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:39.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:59:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:39 smithi153 ceph-mon[87858]: pgmap v630: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:59:41.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:41 smithi151 ceph-mon[112233]: pgmap v631: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:42.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:41 smithi153 ceph-mon[87858]: pgmap v631: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:43.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.011+0000 7f80b64c7700 1 -- 172.21.15.151:0/3118424225 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 msgr2=0x7f80b0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.011+0000 7f80b64c7700 1 --2- 172.21.15.151:0/3118424225 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 0x7f80b0100050 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8098009910 tx=0x7f8098009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 -- 172.21.15.151:0/3118424225 shutdown_connections 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 --2- 172.21.15.151:0/3118424225 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 0x7f80b0100050 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 --2- 172.21.15.151:0/3118424225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80b0105a40 0x7f80b0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 -- 172.21.15.151:0/3118424225 >> 172.21.15.151:0/3118424225 conn(0x7f80b00755a0 msgr2=0x7f80b00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 -- 172.21.15.151:0/3118424225 shutdown_connections 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 -- 172.21.15.151:0/3118424225 wait complete. 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 Processor -- start 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 -- start start 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 0x7f80b019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80b0105a40 0x7f80b019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b019d0c0 con 0x7f80b00ffbd0 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.012+0000 7f80b64c7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b0196570 con 0x7f80b0105a40 2024-04-16T02:59:43.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80affff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 0x7f80b019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80affff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 0x7f80b019c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46754/0 (socket says 172.21.15.151:46754) 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80affff700 1 -- 172.21.15.151:0/10964675 learned_addr learned my addr 172.21.15.151:0/10964675 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80a7fff700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80b0105a40 0x7f80b019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80affff700 1 -- 172.21.15.151:0/10964675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80b0105a40 msgr2=0x7f80b019ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80affff700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80b0105a40 0x7f80b019ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80affff700 1 -- 172.21.15.151:0/10964675 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80980095f0 con 0x7f80b00ffbd0 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80a7fff700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80b0105a40 0x7f80b019ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80affff700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 0x7f80b019c4f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f80a000d8d0 tx=0x7f80a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80adffb700 1 -- 172.21.15.151:0/10964675 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80a0009940 con 0x7f80b00ffbd0 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80adffb700 1 -- 172.21.15.151:0/10964675 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80a0010460 con 0x7f80b00ffbd0 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80adffb700 1 -- 172.21.15.151:0/10964675 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80a000f5f0 con 0x7f80b00ffbd0 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b0196850 con 0x7f80b00ffbd0 2024-04-16T02:59:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.013+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b0196da0 con 0x7f80b00ffbd0 2024-04-16T02:59:43.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.014+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b0108fe0 con 0x7f80b00ffbd0 2024-04-16T02:59:43.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.015+0000 7f80adffb700 1 -- 172.21.15.151:0/10964675 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f80a00105d0 con 0x7f80b00ffbd0 2024-04-16T02:59:43.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.015+0000 7f80adffb700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f809c06ecc0 0x7f809c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:43.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.015+0000 7f80adffb700 1 -- 172.21.15.151:0/10964675 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f80a008ee80 con 0x7f80b00ffbd0 2024-04-16T02:59:43.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.015+0000 7f80a7fff700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f809c06ecc0 0x7f809c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:43.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.016+0000 7f80a7fff700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f809c06ecc0 0x7f809c071180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f809800b5c0 tx=0x7f80980053a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:43.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.020+0000 7f80adffb700 1 -- 172.21.15.151:0/10964675 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f80a005b760 con 0x7f80b00ffbd0 2024-04-16T02:59:43.237 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.235+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f80b0197040 con 0x7f809c06ecc0 2024-04-16T02:59:43.238 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.237+0000 7f80adffb700 1 -- 172.21.15.151:0/10964675 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f80b0197040 con 0x7f809c06ecc0 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f809c06ecc0 msgr2=0x7f809c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f809c06ecc0 0x7f809c071180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f809800b5c0 tx=0x7f80980053a0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 msgr2=0x7f80b019c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 0x7f80b019c4f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f80a000d8d0 tx=0x7f80a000dc90 comp rx=0 tx=0).stop 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 shutdown_connections 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f809c06ecc0 0x7f809c071180 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80b00ffbd0 0x7f80b019c4f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 --2- 172.21.15.151:0/10964675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80b0105a40 0x7f80b019ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 >> 172.21.15.151:0/10964675 conn(0x7f80b00755a0 msgr2=0x7f80b0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 shutdown_connections 2024-04-16T02:59:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.239+0000 7f80b64c7700 1 -- 172.21.15.151:0/10964675 wait complete. 2024-04-16T02:59:43.251 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T02:59:43.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.344+0000 7f9237d66700 1 -- 172.21.15.151:0/753309826 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 msgr2=0x7f92301057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.344+0000 7f9237d66700 1 --2- 172.21.15.151:0/753309826 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 0x7f92301057a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f922c009910 tx=0x7f922c009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:43.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.344+0000 7f9237d66700 1 -- 172.21.15.151:0/753309826 shutdown_connections 2024-04-16T02:59:43.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.344+0000 7f9237d66700 1 --2- 172.21.15.151:0/753309826 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 0x7f92301057a0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.344+0000 7f9237d66700 1 --2- 172.21.15.151:0/753309826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9230100da0 0x7f9230101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.344+0000 7f9237d66700 1 -- 172.21.15.151:0/753309826 >> 172.21.15.151:0/753309826 conn(0x7f92300fc630 msgr2=0x7f92300fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:43.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.344+0000 7f9237d66700 1 -- 172.21.15.151:0/753309826 shutdown_connections 2024-04-16T02:59:43.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.344+0000 7f9237d66700 1 -- 172.21.15.151:0/753309826 wait complete. 2024-04-16T02:59:43.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9237d66700 1 Processor -- start 2024-04-16T02:59:43.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9237d66700 1 -- start start 2024-04-16T02:59:43.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9237d66700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9230100da0 0x7f923019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:43.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9237d66700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 0x7f923019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9237d66700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923019f350 con 0x7f9230101750 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9237d66700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9230198800 con 0x7f9230100da0 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9235301700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 0x7f923019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9235301700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 0x7f923019ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46764/0 (socket says 172.21.15.151:46764) 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9235301700 1 -- 172.21.15.151:0/448695040 learned_addr learned my addr 172.21.15.151:0/448695040 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.345+0000 7f9235b02700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9230100da0 0x7f923019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9235301700 1 -- 172.21.15.151:0/448695040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9230100da0 msgr2=0x7f923019e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9235301700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9230100da0 0x7f923019e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9235301700 1 -- 172.21.15.151:0/448695040 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c0095f0 con 0x7f9230101750 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9235b02700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9230100da0 0x7f923019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9235301700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 0x7f923019ecc0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f922c0092e0 tx=0x7f922c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9226ffd700 1 -- 172.21.15.151:0/448695040 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922c01d070 con 0x7f9230101750 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9226ffd700 1 -- 172.21.15.151:0/448695040 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f922c00f460 con 0x7f9230101750 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9230198a80 con 0x7f9230101750 2024-04-16T02:59:43.349 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9226ffd700 1 -- 172.21.15.151:0/448695040 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922c0175d0 con 0x7f9230101750 2024-04-16T02:59:43.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.346+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9230198f70 con 0x7f9230101750 2024-04-16T02:59:43.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.347+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9230104f20 con 0x7f9230101750 2024-04-16T02:59:43.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.348+0000 7f9226ffd700 1 -- 172.21.15.151:0/448695040 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f922c00f5d0 con 0x7f9230101750 2024-04-16T02:59:43.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.348+0000 7f9226ffd700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f921c06ecc0 0x7f921c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:43.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.348+0000 7f9226ffd700 1 -- 172.21.15.151:0/448695040 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f922c0902c0 con 0x7f9230101750 2024-04-16T02:59:43.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.348+0000 7f9235b02700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f921c06ecc0 0x7f921c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:43.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.349+0000 7f9235b02700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f921c06ecc0 0x7f921c071180 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9220005950 tx=0x7f922000a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:43.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.353+0000 7f9226ffd700 1 -- 172.21.15.151:0/448695040 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f922c05cba0 con 0x7f9230101750 2024-04-16T02:59:43.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.572+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9230199250 con 0x7f921c06ecc0 2024-04-16T02:59:43.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.573+0000 7f9226ffd700 1 -- 172.21.15.151:0/448695040 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9230199250 con 0x7f921c06ecc0 2024-04-16T02:59:43.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.575+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f921c06ecc0 msgr2=0x7f921c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.575+0000 7f9237d66700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f921c06ecc0 0x7f921c071180 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9220005950 tx=0x7f922000a300 comp rx=0 tx=0).stop 2024-04-16T02:59:43.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.575+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 msgr2=0x7f923019ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.577 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.575+0000 7f9237d66700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 0x7f923019ecc0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f922c0092e0 tx=0x7f922c00bd70 comp rx=0 tx=0).stop 2024-04-16T02:59:43.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.575+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 shutdown_connections 2024-04-16T02:59:43.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.575+0000 7f9237d66700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f921c06ecc0 0x7f921c071180 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.575+0000 7f9237d66700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9230100da0 0x7f923019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.575+0000 7f9237d66700 1 --2- 172.21.15.151:0/448695040 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9230101750 0x7f923019ecc0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.575+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 >> 172.21.15.151:0/448695040 conn(0x7f92300fc630 msgr2=0x7f92300fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:43.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.576+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 shutdown_connections 2024-04-16T02:59:43.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.576+0000 7f9237d66700 1 -- 172.21.15.151:0/448695040 wait complete. 2024-04-16T02:59:43.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.679+0000 7ff7269ca700 1 -- 172.21.15.151:0/1390478176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 msgr2=0x7ff720105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.679+0000 7ff7269ca700 1 --2- 172.21.15.151:0/1390478176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 0x7ff720105e20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff708009910 tx=0x7ff708009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:43.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.679+0000 7ff7269ca700 1 -- 172.21.15.151:0/1390478176 shutdown_connections 2024-04-16T02:59:43.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.679+0000 7ff7269ca700 1 --2- 172.21.15.151:0/1390478176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7200ffa40 0x7ff7200ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.679+0000 7ff7269ca700 1 --2- 172.21.15.151:0/1390478176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 0x7ff720105e20 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.680+0000 7ff7269ca700 1 -- 172.21.15.151:0/1390478176 >> 172.21.15.151:0/1390478176 conn(0x7ff7200fb550 msgr2=0x7ff7200fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:43.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.680+0000 7ff7269ca700 1 -- 172.21.15.151:0/1390478176 shutdown_connections 2024-04-16T02:59:43.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.680+0000 7ff7269ca700 1 -- 172.21.15.151:0/1390478176 wait complete. 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.680+0000 7ff7269ca700 1 Processor -- start 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.680+0000 7ff7269ca700 1 -- start start 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff7269ca700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7200ffa40 0x7ff72019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff7269ca700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 0x7ff72019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff7269ca700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72019f2b0 con 0x7ff720105a40 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff7269ca700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff720198760 con 0x7ff7200ffa40 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 0x7ff72019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 0x7ff72019ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46770/0 (socket says 172.21.15.151:46770) 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71f7fe700 1 -- 172.21.15.151:0/3963386873 learned_addr learned my addr 172.21.15.151:0/3963386873 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:43.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71ffff700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7200ffa40 0x7ff72019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71f7fe700 1 -- 172.21.15.151:0/3963386873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7200ffa40 msgr2=0x7ff72019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71f7fe700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7200ffa40 0x7ff72019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71f7fe700 1 -- 172.21.15.151:0/3963386873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7080095f0 con 0x7ff720105a40 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71ffff700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7200ffa40 0x7ff72019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71f7fe700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 0x7ff72019ec20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff71000d900 tx=0x7ff71000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.681+0000 7ff71d7fa700 1 -- 172.21.15.151:0/3963386873 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7100041f0 con 0x7ff720105a40 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.682+0000 7ff71d7fa700 1 -- 172.21.15.151:0/3963386873 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff710004cf0 con 0x7ff720105a40 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.682+0000 7ff71d7fa700 1 -- 172.21.15.151:0/3963386873 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff71000b750 con 0x7ff720105a40 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.682+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff720198a40 con 0x7ff720105a40 2024-04-16T02:59:43.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.682+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff720198f90 con 0x7ff720105a40 2024-04-16T02:59:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.683+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72004ea90 con 0x7ff720105a40 2024-04-16T02:59:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.683+0000 7ff71d7fa700 1 -- 172.21.15.151:0/3963386873 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff710004350 con 0x7ff720105a40 2024-04-16T02:59:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.684+0000 7ff71d7fa700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff70c06ecc0 0x7ff70c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.684+0000 7ff71ffff700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff70c06ecc0 0x7ff70c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:43.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.684+0000 7ff71d7fa700 1 -- 172.21.15.151:0/3963386873 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff71008efb0 con 0x7ff720105a40 2024-04-16T02:59:43.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.684+0000 7ff71ffff700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff70c06ecc0 0x7ff70c071180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff708005b40 tx=0x7ff70801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:43.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.689+0000 7ff71d7fa700 1 -- 172.21.15.151:0/3963386873 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff7100c3ff0 con 0x7ff720105a40 2024-04-16T02:59:43.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.897+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7201993b0 con 0x7ff70c06ecc0 2024-04-16T02:59:43.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.907+0000 7ff71d7fa700 1 -- 172.21.15.151:0/3963386873 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff7201993b0 con 0x7ff70c06ecc0 2024-04-16T02:59:43.909 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T02:59:43.909 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (21m) 14m ago 32m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T02:59:43.909 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (32m) 14m ago 32m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T02:59:43.909 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (31m) 14m ago 31m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T02:59:43.909 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (18m) 14m ago 32m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (18m) 14m ago 31m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (20m) 14m ago 32m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (15m) 14m ago 27m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (14m) 14m ago 26m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (26m) 14m ago 26m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (14m) 14m ago 27m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (23m) 14m ago 34m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (22m) 14m ago 31m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (19m) 14m ago 34m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (19m) 14m ago 31m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (22m) 14m ago 32m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (21m) 14m ago 31m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (18m) 14m ago 30m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (17m) 14m ago 30m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T02:59:43.911 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (17m) 14m ago 29m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T02:59:43.911 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (16m) 14m ago 29m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T02:59:43.911 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (16m) 14m ago 28m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T02:59:43.911 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (15m) 14m ago 28m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T02:59:43.911 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (21m) 14m ago 32m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff70c06ecc0 msgr2=0x7ff70c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff70c06ecc0 0x7ff70c071180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff708005b40 tx=0x7ff70801a040 comp rx=0 tx=0).stop 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 msgr2=0x7ff72019ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 0x7ff72019ec20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff71000d900 tx=0x7ff71000dcc0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 shutdown_connections 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff70c06ecc0 0x7ff70c071180 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7200ffa40 0x7ff72019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 --2- 172.21.15.151:0/3963386873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff720105a40 0x7ff72019ec20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 >> 172.21.15.151:0/3963386873 conn(0x7ff7200fb550 msgr2=0x7ff720068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:43.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 shutdown_connections 2024-04-16T02:59:43.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:43.910+0000 7ff7269ca700 1 -- 172.21.15.151:0/3963386873 wait complete. 2024-04-16T02:59:43.976 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:43 smithi153 ceph-mon[87858]: pgmap v632: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:43.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:43 smithi151 ceph-mon[112233]: pgmap v632: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.014+0000 7fa095fa0700 1 -- 172.21.15.151:0/991963910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa090105a40 msgr2=0x7fa090105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.014+0000 7fa095fa0700 1 --2- 172.21.15.151:0/991963910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa090105a40 0x7fa090105e20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa088009910 tx=0x7fa088009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.014+0000 7fa095fa0700 1 -- 172.21.15.151:0/991963910 shutdown_connections 2024-04-16T02:59:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.014+0000 7fa095fa0700 1 --2- 172.21.15.151:0/991963910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0900ffbd0 0x7fa090100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.014+0000 7fa095fa0700 1 --2- 172.21.15.151:0/991963910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa090105a40 0x7fa090105e20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.014+0000 7fa095fa0700 1 -- 172.21.15.151:0/991963910 >> 172.21.15.151:0/991963910 conn(0x7fa0900755a0 msgr2=0x7fa0900759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:44.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.015+0000 7fa095fa0700 1 -- 172.21.15.151:0/991963910 shutdown_connections 2024-04-16T02:59:44.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.015+0000 7fa095fa0700 1 -- 172.21.15.151:0/991963910 wait complete. 2024-04-16T02:59:44.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.015+0000 7fa095fa0700 1 Processor -- start 2024-04-16T02:59:44.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.015+0000 7fa095fa0700 1 -- start start 2024-04-16T02:59:44.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.015+0000 7fa095fa0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0900ffbd0 0x7fa090101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:44.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.015+0000 7fa095fa0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa090105a40 0x7fa090101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.015+0000 7fa095fa0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa090102120 con 0x7fa0900ffbd0 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.015+0000 7fa095fa0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa090102290 con 0x7fa090105a40 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0900ffbd0 0x7fa090101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0900ffbd0 0x7fa090101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46788/0 (socket says 172.21.15.151:46788) 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08f7fe700 1 -- 172.21.15.151:0/3338432932 learned_addr learned my addr 172.21.15.151:0/3338432932 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08effd700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa090105a40 0x7fa090101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08f7fe700 1 -- 172.21.15.151:0/3338432932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa090105a40 msgr2=0x7fa090101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08f7fe700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa090105a40 0x7fa090101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08f7fe700 1 -- 172.21.15.151:0/3338432932 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0880095f0 con 0x7fa0900ffbd0 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08f7fe700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0900ffbd0 0x7fa090101610 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa088006010 tx=0x7fa08800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08cff9700 1 -- 172.21.15.151:0/3338432932 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa08801d070 con 0x7fa0900ffbd0 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0901a28c0 con 0x7fa0900ffbd0 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08cff9700 1 -- 172.21.15.151:0/3338432932 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa088005dc0 con 0x7fa0900ffbd0 2024-04-16T02:59:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa08cff9700 1 -- 172.21.15.151:0/3338432932 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa088017400 con 0x7fa0900ffbd0 2024-04-16T02:59:44.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.016+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0901a2c70 con 0x7fa0900ffbd0 2024-04-16T02:59:44.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.017+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa090109050 con 0x7fa0900ffbd0 2024-04-16T02:59:44.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.018+0000 7fa08cff9700 1 -- 172.21.15.151:0/3338432932 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa088003a60 con 0x7fa0900ffbd0 2024-04-16T02:59:44.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.018+0000 7fa08cff9700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa07c06ecc0 0x7fa07c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:44.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.018+0000 7fa08cff9700 1 -- 172.21.15.151:0/3338432932 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa0880902a0 con 0x7fa0900ffbd0 2024-04-16T02:59:44.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.018+0000 7fa08effd700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa07c06ecc0 0x7fa07c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:44.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.019+0000 7fa08effd700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa07c06ecc0 0x7fa07c071180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa090102e00 tx=0x7fa07800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:44.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.023+0000 7fa08cff9700 1 -- 172.21.15.151:0/3338432932 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa088059050 con 0x7fa0900ffbd0 2024-04-16T02:59:44.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.304+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa09004ea90 con 0x7fa0900ffbd0 2024-04-16T02:59:44.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.304+0000 7fa08cff9700 1 -- 172.21.15.151:0/3338432932 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fa08805c710 con 0x7fa0900ffbd0 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T02:59:44.307 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T02:59:44.308 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T02:59:44.308 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T02:59:44.308 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T02:59:44.308 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:59:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa07c06ecc0 msgr2=0x7fa07c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa07c06ecc0 0x7fa07c071180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa090102e00 tx=0x7fa07800b410 comp rx=0 tx=0).stop 2024-04-16T02:59:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0900ffbd0 msgr2=0x7fa090101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0900ffbd0 0x7fa090101610 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa088006010 tx=0x7fa08800bd70 comp rx=0 tx=0).stop 2024-04-16T02:59:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 shutdown_connections 2024-04-16T02:59:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa07c06ecc0 0x7fa07c071180 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0900ffbd0 0x7fa090101610 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 --2- 172.21.15.151:0/3338432932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa090105a40 0x7fa090101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 >> 172.21.15.151:0/3338432932 conn(0x7fa0900755a0 msgr2=0x7fa0900685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:44.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 shutdown_connections 2024-04-16T02:59:44.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.307+0000 7fa095fa0700 1 -- 172.21.15.151:0/3338432932 wait complete. 2024-04-16T02:59:44.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.414+0000 7fc8d64f7700 1 -- 172.21.15.151:0/1976568579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0101770 msgr2=0x7fc8d01057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.414+0000 7fc8d64f7700 1 --2- 172.21.15.151:0/1976568579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0101770 0x7fc8d01057c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009910 tx=0x7fc8c4009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:44.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.414+0000 7fc8d64f7700 1 -- 172.21.15.151:0/1976568579 shutdown_connections 2024-04-16T02:59:44.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.414+0000 7fc8d64f7700 1 --2- 172.21.15.151:0/1976568579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0101770 0x7fc8d01057c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.414+0000 7fc8d64f7700 1 --2- 172.21.15.151:0/1976568579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8d0100dc0 0x7fc8d01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.414+0000 7fc8d64f7700 1 -- 172.21.15.151:0/1976568579 >> 172.21.15.151:0/1976568579 conn(0x7fc8d00fc650 msgr2=0x7fc8d00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:44.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.414+0000 7fc8d64f7700 1 -- 172.21.15.151:0/1976568579 shutdown_connections 2024-04-16T02:59:44.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.414+0000 7fc8d64f7700 1 -- 172.21.15.151:0/1976568579 wait complete. 2024-04-16T02:59:44.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.414+0000 7fc8d64f7700 1 Processor -- start 2024-04-16T02:59:44.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8d64f7700 1 -- start start 2024-04-16T02:59:44.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8d64f7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0100dc0 0x7fc8d019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:44.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8d64f7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8d0101770 0x7fc8d019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:44.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8d64f7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d019d240 con 0x7fc8d0100dc0 2024-04-16T02:59:44.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8d64f7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d01966f0 con 0x7fc8d0101770 2024-04-16T02:59:44.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8cffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0100dc0 0x7fc8d019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:44.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8cffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0100dc0 0x7fc8d019c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46802/0 (socket says 172.21.15.151:46802) 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8cffff700 1 -- 172.21.15.151:0/3525726107 learned_addr learned my addr 172.21.15.151:0/3525726107 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8cf7fe700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8d0101770 0x7fc8d019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8cffff700 1 -- 172.21.15.151:0/3525726107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8d0101770 msgr2=0x7fc8d019cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8cffff700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8d0101770 0x7fc8d019cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8cffff700 1 -- 172.21.15.151:0/3525726107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40095f0 con 0x7fc8d0100dc0 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.415+0000 7fc8cf7fe700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8d0101770 0x7fc8d019cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.416+0000 7fc8cffff700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0100dc0 0x7fc8d019c670 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000b4b0 tx=0x7fc8c000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.416+0000 7fc8cd7fa700 1 -- 172.21.15.151:0/3525726107 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8c0011800 con 0x7fc8d0100dc0 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.416+0000 7fc8cd7fa700 1 -- 172.21.15.151:0/3525726107 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8c0011e20 con 0x7fc8d0100dc0 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.416+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d01969d0 con 0x7fc8d0100dc0 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.416+0000 7fc8cd7fa700 1 -- 172.21.15.151:0/3525726107 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8c000f2e0 con 0x7fc8d0100dc0 2024-04-16T02:59:44.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.416+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d0196f20 con 0x7fc8d0100dc0 2024-04-16T02:59:44.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.417+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d01090f0 con 0x7fc8d0100dc0 2024-04-16T02:59:44.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.417+0000 7fc8cd7fa700 1 -- 172.21.15.151:0/3525726107 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc8c0010380 con 0x7fc8d0100dc0 2024-04-16T02:59:44.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.418+0000 7fc8cd7fa700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8b806ecc0 0x7fc8b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:44.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.418+0000 7fc8cd7fa700 1 -- 172.21.15.151:0/3525726107 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc8c008e990 con 0x7fc8d0100dc0 2024-04-16T02:59:44.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.418+0000 7fc8cf7fe700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8b806ecc0 0x7fc8b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:44.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.418+0000 7fc8cf7fe700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8b806ecc0 0x7fc8b8071180 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0197b60 tx=0x7fc8c401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:44.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.423+0000 7fc8cd7fa700 1 -- 172.21.15.151:0/3525726107 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc8c005b3a0 con 0x7fc8d0100dc0 2024-04-16T02:59:44.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.666+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc8d0197250 con 0x7fc8d0100dc0 2024-04-16T02:59:44.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.666+0000 7fc8cd7fa700 1 -- 172.21.15.151:0/3525726107 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7fc8c005af30 con 0x7fc8d0100dc0 2024-04-16T02:59:44.668 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T02:59:44.668 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T02:59:44.668 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:59:44.668 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T02:59:44.668 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T02:59:44.669 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:59:44.670 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T02:59:44.671 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T02:59:44.671 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:59:44.671 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:59:44.671 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T02:59:44.671 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T02:59:44.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.669+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8b806ecc0 msgr2=0x7fc8b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.669+0000 7fc8d64f7700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8b806ecc0 0x7fc8b8071180 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0197b60 tx=0x7fc8c401a040 comp rx=0 tx=0).stop 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.669+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0100dc0 msgr2=0x7fc8d019c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.669+0000 7fc8d64f7700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0100dc0 0x7fc8d019c670 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000b4b0 tx=0x7fc8c000b870 comp rx=0 tx=0).stop 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.669+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 shutdown_connections 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.669+0000 7fc8d64f7700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8b806ecc0 0x7fc8b8071180 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.669+0000 7fc8d64f7700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8d0100dc0 0x7fc8d019c670 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.669+0000 7fc8d64f7700 1 --2- 172.21.15.151:0/3525726107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8d0101770 0x7fc8d019cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.669+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 >> 172.21.15.151:0/3525726107 conn(0x7fc8d00fc650 msgr2=0x7fc8d00fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.670+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 shutdown_connections 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.670+0000 7fc8d64f7700 1 -- 172.21.15.151:0/3525726107 wait complete. 2024-04-16T02:59:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.777+0000 7fb4a6745700 1 -- 172.21.15.151:0/1144524852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 msgr2=0x7fb4a0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.777+0000 7fb4a6745700 1 --2- 172.21.15.151:0/1144524852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 0x7fb4a0105e20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb490009910 tx=0x7fb490009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.777+0000 7fb4a6745700 1 -- 172.21.15.151:0/1144524852 shutdown_connections 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.777+0000 7fb4a6745700 1 --2- 172.21.15.151:0/1144524852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a00ffbd0 0x7fb4a0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.777+0000 7fb4a6745700 1 --2- 172.21.15.151:0/1144524852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 0x7fb4a0105e20 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.777+0000 7fb4a6745700 1 -- 172.21.15.151:0/1144524852 >> 172.21.15.151:0/1144524852 conn(0x7fb4a00755a0 msgr2=0x7fb4a00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.777+0000 7fb4a6745700 1 -- 172.21.15.151:0/1144524852 shutdown_connections 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.777+0000 7fb4a6745700 1 -- 172.21.15.151:0/1144524852 wait complete. 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb4a6745700 1 Processor -- start 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb4a6745700 1 -- start start 2024-04-16T02:59:44.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb4a6745700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a00ffbd0 0x7fb4a019c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb4a6745700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 0x7fb4a019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb4a6745700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a019d110 con 0x7fb4a0105a40 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb4a6745700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a01965c0 con 0x7fb4a00ffbd0 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb497fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 0x7fb4a019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb497fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 0x7fb4a019ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46824/0 (socket says 172.21.15.151:46824) 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb497fff700 1 -- 172.21.15.151:0/2719670940 learned_addr learned my addr 172.21.15.151:0/2719670940 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb49ffff700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a00ffbd0 0x7fb4a019c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb497fff700 1 -- 172.21.15.151:0/2719670940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a00ffbd0 msgr2=0x7fb4a019c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb497fff700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a00ffbd0 0x7fb4a019c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.778+0000 7fb497fff700 1 -- 172.21.15.151:0/2719670940 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4900095f0 con 0x7fb4a0105a40 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.779+0000 7fb497fff700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 0x7fb4a019ca80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb48800d8d0 tx=0x7fb48800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.779+0000 7fb49dffb700 1 -- 172.21.15.151:0/2719670940 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4880098a0 con 0x7fb4a0105a40 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.779+0000 7fb49dffb700 1 -- 172.21.15.151:0/2719670940 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb488010460 con 0x7fb4a0105a40 2024-04-16T02:59:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.779+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a01968a0 con 0x7fb4a0105a40 2024-04-16T02:59:44.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.779+0000 7fb49dffb700 1 -- 172.21.15.151:0/2719670940 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb48800f5f0 con 0x7fb4a0105a40 2024-04-16T02:59:44.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.779+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a0196df0 con 0x7fb4a0105a40 2024-04-16T02:59:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.780+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a0109050 con 0x7fb4a0105a40 2024-04-16T02:59:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.780+0000 7fb49dffb700 1 -- 172.21.15.151:0/2719670940 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb488009a00 con 0x7fb4a0105a40 2024-04-16T02:59:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.781+0000 7fb49dffb700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb48c06ecc0 0x7fb48c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.781+0000 7fb49dffb700 1 -- 172.21.15.151:0/2719670940 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb48808ed10 con 0x7fb4a0105a40 2024-04-16T02:59:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.781+0000 7fb49ffff700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb48c06ecc0 0x7fb48c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.781+0000 7fb49ffff700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb48c06ecc0 0x7fb48c071180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb490006010 tx=0x7fb49001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:44.787 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:44.785+0000 7fb49dffb700 1 -- 172.21.15.151:0/2719670940 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb48805b5f0 con 0x7fb4a0105a40 2024-04-16T02:59:44.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:44 smithi151 ceph-mon[112233]: from='client.35182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:44.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:44 smithi151 ceph-mon[112233]: from='client.35186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:44.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:44 smithi151 ceph-mon[112233]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:44.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:44 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3338432932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:59:44.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:44 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3525726107' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:59:45.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.003+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4a0197010 con 0x7fb48c06ecc0 2024-04-16T02:59:45.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.005+0000 7fb49dffb700 1 -- 172.21.15.151:0/2719670940 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb4a0197010 con 0x7fb48c06ecc0 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T02:59:45.007 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T02:59:45.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb48c06ecc0 msgr2=0x7fb48c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:45.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb48c06ecc0 0x7fb48c071180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb490006010 tx=0x7fb49001a040 comp rx=0 tx=0).stop 2024-04-16T02:59:45.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 msgr2=0x7fb4a019ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:45.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 0x7fb4a019ca80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb48800d8d0 tx=0x7fb48800dbe0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 shutdown_connections 2024-04-16T02:59:45.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb48c06ecc0 0x7fb48c071180 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a00ffbd0 0x7fb4a019c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 --2- 172.21.15.151:0/2719670940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4a0105a40 0x7fb4a019ca80 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 >> 172.21.15.151:0/2719670940 conn(0x7fb4a00755a0 msgr2=0x7fb4a00685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:45.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 shutdown_connections 2024-04-16T02:59:45.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.007+0000 7fb4a6745700 1 -- 172.21.15.151:0/2719670940 wait complete. 2024-04-16T02:59:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:44 smithi153 ceph-mon[87858]: from='client.35182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:44 smithi153 ceph-mon[87858]: from='client.35186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:44 smithi153 ceph-mon[87858]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:44 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3338432932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T02:59:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:44 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3525726107' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T02:59:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.115+0000 7f1964018700 1 -- 172.21.15.151:0/605929155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c0ffbd0 msgr2=0x7f195c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.115+0000 7f1964018700 1 --2- 172.21.15.151:0/605929155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c0ffbd0 0x7f195c100050 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1950009910 tx=0x7f1950009c20 comp rx=0 tx=0).stop 2024-04-16T02:59:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.115+0000 7f1964018700 1 -- 172.21.15.151:0/605929155 shutdown_connections 2024-04-16T02:59:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.115+0000 7f1964018700 1 --2- 172.21.15.151:0/605929155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c0ffbd0 0x7f195c100050 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.115+0000 7f1964018700 1 --2- 172.21.15.151:0/605929155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f195c105a40 0x7f195c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.115+0000 7f1964018700 1 -- 172.21.15.151:0/605929155 >> 172.21.15.151:0/605929155 conn(0x7f195c0755a0 msgr2=0x7f195c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.115+0000 7f1964018700 1 -- 172.21.15.151:0/605929155 shutdown_connections 2024-04-16T02:59:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.115+0000 7f1964018700 1 -- 172.21.15.151:0/605929155 wait complete. 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f1964018700 1 Processor -- start 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f1964018700 1 -- start start 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f1964018700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f195c0ffbd0 0x7f195c101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f1964018700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c105a40 0x7f195c101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f1964018700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c102090 con 0x7f195c105a40 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f1964018700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c1021d0 con 0x7f195c0ffbd0 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f19615b3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c105a40 0x7f195c101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f19615b3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c105a40 0x7f195c101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46832/0 (socket says 172.21.15.151:46832) 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f19615b3700 1 -- 172.21.15.151:0/1889632255 learned_addr learned my addr 172.21.15.151:0/1889632255 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f1961db4700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f195c0ffbd0 0x7f195c101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f19615b3700 1 -- 172.21.15.151:0/1889632255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f195c0ffbd0 msgr2=0x7f195c101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.116+0000 7f19615b3700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f195c0ffbd0 0x7f195c101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.117+0000 7f19615b3700 1 -- 172.21.15.151:0/1889632255 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500095f0 con 0x7f195c105a40 2024-04-16T02:59:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.117+0000 7f19615b3700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c105a40 0x7f195c101b50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1950006010 tx=0x7f195000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:45.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.117+0000 7f194effd700 1 -- 172.21.15.151:0/1889632255 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f195001d070 con 0x7f195c105a40 2024-04-16T02:59:45.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.117+0000 7f194effd700 1 -- 172.21.15.151:0/1889632255 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f195000f460 con 0x7f195c105a40 2024-04-16T02:59:45.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.117+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c102450 con 0x7f195c105a40 2024-04-16T02:59:45.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.117+0000 7f194effd700 1 -- 172.21.15.151:0/1889632255 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1950017630 con 0x7f195c105a40 2024-04-16T02:59:45.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.117+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c1a2be0 con 0x7f195c105a40 2024-04-16T02:59:45.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.118+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195c108fe0 con 0x7f195c105a40 2024-04-16T02:59:45.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.118+0000 7f194effd700 1 -- 172.21.15.151:0/1889632255 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f195000f5d0 con 0x7f195c105a40 2024-04-16T02:59:45.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.119+0000 7f194effd700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f194806ecc0 0x7f1948071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T02:59:45.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.119+0000 7f194effd700 1 -- 172.21.15.151:0/1889632255 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f19500902d0 con 0x7f195c105a40 2024-04-16T02:59:45.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.119+0000 7f1961db4700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f194806ecc0 0x7f1948071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T02:59:45.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.119+0000 7f1961db4700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f194806ecc0 0x7f1948071180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f195800ac50 tx=0x7f195800a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T02:59:45.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.124+0000 7f194effd700 1 -- 172.21.15.151:0/1889632255 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1950059100 con 0x7f195c105a40 2024-04-16T02:59:45.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.404+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f195c02cca0 con 0x7f195c105a40 2024-04-16T02:59:45.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.404+0000 7f194effd700 1 -- 172.21.15.151:0/1889632255 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f195005c7c0 con 0x7f195c105a40 2024-04-16T02:59:45.406 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T02:59:45.406 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T02:59:45.407 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T02:59:45.407 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T02:59:45.407 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T02:59:45.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.406+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f194806ecc0 msgr2=0x7f1948071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.406+0000 7f1964018700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f194806ecc0 0x7f1948071180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f195800ac50 tx=0x7f195800a380 comp rx=0 tx=0).stop 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.407+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c105a40 msgr2=0x7f195c101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.407+0000 7f1964018700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c105a40 0x7f195c101b50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1950006010 tx=0x7f195000bd70 comp rx=0 tx=0).stop 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.407+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 shutdown_connections 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.407+0000 7f1964018700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f194806ecc0 0x7f1948071180 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.407+0000 7f1964018700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f195c0ffbd0 0x7f195c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.407+0000 7f1964018700 1 --2- 172.21.15.151:0/1889632255 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f195c105a40 0x7f195c101b50 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.407+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 >> 172.21.15.151:0/1889632255 conn(0x7f195c0755a0 msgr2=0x7f195c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.407+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 shutdown_connections 2024-04-16T02:59:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T02:59:45.407+0000 7f1964018700 1 -- 172.21.15.151:0/1889632255 wait complete. 2024-04-16T02:59:45.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:45 smithi151 ceph-mon[112233]: pgmap v633: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:45.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:45 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1889632255' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:59:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:45 smithi153 ceph-mon[87858]: pgmap v633: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:46.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:45 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1889632255' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T02:59:46.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:46 smithi151 ceph-mon[112233]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:47.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:46 smithi153 ceph-mon[87858]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T02:59:47.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:47 smithi151 ceph-mon[112233]: pgmap v634: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:48.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:47 smithi153 ceph-mon[87858]: pgmap v634: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:49.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:49 smithi151 ceph-mon[112233]: pgmap v635: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:49 smithi153 ceph-mon[87858]: pgmap v635: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:51.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:51 smithi151 ceph-mon[112233]: pgmap v636: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:52.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:51 smithi153 ceph-mon[87858]: pgmap v636: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:53.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:53 smithi151 ceph-mon[112233]: pgmap v637: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:54.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:53 smithi153 ceph-mon[87858]: pgmap v637: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:59:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T02:59:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:55 smithi151 ceph-mon[112233]: pgmap v638: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:55 smithi153 ceph-mon[87858]: pgmap v638: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:57.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:57 smithi151 ceph-mon[112233]: pgmap v639: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:58.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:57 smithi153 ceph-mon[87858]: pgmap v639: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T02:59:59.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 02:59:59 smithi151 ceph-mon[112233]: pgmap v640: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:00.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 02:59:59 smithi153 ceph-mon[87858]: pgmap v640: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:00.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T02:59:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:00:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:00 smithi151 ceph-mon[112233]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:00:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:00 smithi153 ceph-mon[87858]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:00:01.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:01 smithi151 ceph-mon[112233]: pgmap v641: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:02.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:01 smithi153 ceph-mon[87858]: pgmap v641: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:03.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:03 smithi151 ceph-mon[112233]: pgmap v642: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:04.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:03 smithi153 ceph-mon[87858]: pgmap v642: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:05.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:05 smithi151 ceph-mon[112233]: pgmap v643: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:06.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:05 smithi153 ceph-mon[87858]: pgmap v643: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:07.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:07 smithi151 ceph-mon[112233]: pgmap v644: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:08.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:07 smithi153 ceph-mon[87858]: pgmap v644: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:09 smithi153 ceph-mon[87858]: pgmap v645: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:00:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:09 smithi151 ceph-mon[112233]: pgmap v645: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:00:11.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:11 smithi151 ceph-mon[112233]: pgmap v646: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:12.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:11 smithi153 ceph-mon[87858]: pgmap v646: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:14.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:13 smithi153 ceph-mon[87858]: pgmap v647: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:14.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:13 smithi151 ceph-mon[112233]: pgmap v647: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:15.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.519+0000 7f8538119700 1 -- 172.21.15.151:0/3382518909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 msgr2=0x7f8530105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:15.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.519+0000 7f8538119700 1 --2- 172.21.15.151:0/3382518909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 0x7f8530105e20 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f852c009910 tx=0x7f852c009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:15.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.519+0000 7f8538119700 1 -- 172.21.15.151:0/3382518909 shutdown_connections 2024-04-16T03:00:15.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.519+0000 7f8538119700 1 --2- 172.21.15.151:0/3382518909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85300ffa40 0x7f85300ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.519+0000 7f8538119700 1 --2- 172.21.15.151:0/3382518909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 0x7f8530105e20 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.519+0000 7f8538119700 1 -- 172.21.15.151:0/3382518909 >> 172.21.15.151:0/3382518909 conn(0x7f85300fb570 msgr2=0x7f85300fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:15.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.519+0000 7f8538119700 1 -- 172.21.15.151:0/3382518909 shutdown_connections 2024-04-16T03:00:15.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.519+0000 7f8538119700 1 -- 172.21.15.151:0/3382518909 wait complete. 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f8538119700 1 Processor -- start 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f8538119700 1 -- start start 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f8538119700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85300ffa40 0x7f85301975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f8538119700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 0x7f8530197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f8538119700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8530198100 con 0x7f8530105a40 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f8538119700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8530198270 con 0x7f85300ffa40 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f85356b4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 0x7f8530197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f8535eb5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85300ffa40 0x7f85301975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f85356b4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 0x7f8530197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60528/0 (socket says 172.21.15.151:60528) 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.520+0000 7f85356b4700 1 -- 172.21.15.151:0/3338320533 learned_addr learned my addr 172.21.15.151:0/3338320533 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f85356b4700 1 -- 172.21.15.151:0/3338320533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85300ffa40 msgr2=0x7f85301975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:15.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f85356b4700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85300ffa40 0x7f85301975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f85356b4700 1 -- 172.21.15.151:0/3338320533 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c0095f0 con 0x7f8530105a40 2024-04-16T03:00:15.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f8535eb5700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85300ffa40 0x7f85301975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T03:00:15.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f85356b4700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 0x7f8530197ae0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f852000d8d0 tx=0x7f852000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:15.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f8526ffd700 1 -- 172.21.15.151:0/3338320533 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85200098a0 con 0x7f8530105a40 2024-04-16T03:00:15.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85301a0610 con 0x7f8530105a40 2024-04-16T03:00:15.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f8526ffd700 1 -- 172.21.15.151:0/3338320533 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8520010460 con 0x7f8530105a40 2024-04-16T03:00:15.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f8526ffd700 1 -- 172.21.15.151:0/3338320533 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f852000f5f0 con 0x7f8530105a40 2024-04-16T03:00:15.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.521+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85301a0b00 con 0x7f8530105a40 2024-04-16T03:00:15.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.522+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8530108040 con 0x7f8530105a40 2024-04-16T03:00:15.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.522+0000 7f8526ffd700 1 -- 172.21.15.151:0/3338320533 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8520009a00 con 0x7f8530105a40 2024-04-16T03:00:15.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.523+0000 7f8526ffd700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851c06ecc0 0x7f851c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:15.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.523+0000 7f8526ffd700 1 -- 172.21.15.151:0/3338320533 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f852008ed30 con 0x7f8530105a40 2024-04-16T03:00:15.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.523+0000 7f8535eb5700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851c06ecc0 0x7f851c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:15.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.523+0000 7f8535eb5700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851c06ecc0 0x7f851c071180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f852c006010 tx=0x7f852c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:15.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.528+0000 7f8526ffd700 1 -- 172.21.15.151:0/3338320533 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8520057ae0 con 0x7f8530105a40 2024-04-16T03:00:15.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.746+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f853002cd60 con 0x7f851c06ecc0 2024-04-16T03:00:15.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.747+0000 7f8526ffd700 1 -- 172.21.15.151:0/3338320533 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f853002cd60 con 0x7f851c06ecc0 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.749+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851c06ecc0 msgr2=0x7f851c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.749+0000 7f8538119700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851c06ecc0 0x7f851c071180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f852c006010 tx=0x7f852c01a040 comp rx=0 tx=0).stop 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.749+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 msgr2=0x7f8530197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.749+0000 7f8538119700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 0x7f8530197ae0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f852000d8d0 tx=0x7f852000dbe0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.750+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 shutdown_connections 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.750+0000 7f8538119700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851c06ecc0 0x7f851c071180 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.750+0000 7f8538119700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85300ffa40 0x7f85301975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.750+0000 7f8538119700 1 --2- 172.21.15.151:0/3338320533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530105a40 0x7f8530197ae0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.750+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 >> 172.21.15.151:0/3338320533 conn(0x7f85300fb570 msgr2=0x7f85300fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.750+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 shutdown_connections 2024-04-16T03:00:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.750+0000 7f8538119700 1 -- 172.21.15.151:0/3338320533 wait complete. 2024-04-16T03:00:15.763 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:00:15.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.857+0000 7ff1b744a700 1 -- 172.21.15.151:0/3769281230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 msgr2=0x7ff1b01011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:15.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.857+0000 7ff1b744a700 1 --2- 172.21.15.151:0/3769281230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 0x7ff1b01011d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009910 tx=0x7ff1a0009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:15.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.858+0000 7ff1b744a700 1 -- 172.21.15.151:0/3769281230 shutdown_connections 2024-04-16T03:00:15.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.858+0000 7ff1b744a700 1 --2- 172.21.15.151:0/3769281230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b01017a0 0x7ff1b01057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.858+0000 7ff1b744a700 1 --2- 172.21.15.151:0/3769281230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 0x7ff1b01011d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.858+0000 7ff1b744a700 1 -- 172.21.15.151:0/3769281230 >> 172.21.15.151:0/3769281230 conn(0x7ff1b00fc6a0 msgr2=0x7ff1b00feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:15.860 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:15 smithi151 ceph-mon[112233]: pgmap v648: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:15.860 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:00:15.861 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:00:15.861 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:15.861 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:00:15.861 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:15.861 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:15.861 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:15.861 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:15 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.858+0000 7ff1b744a700 1 -- 172.21.15.151:0/3769281230 shutdown_connections 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.858+0000 7ff1b744a700 1 -- 172.21.15.151:0/3769281230 wait complete. 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.858+0000 7ff1b744a700 1 Processor -- start 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.858+0000 7ff1b744a700 1 -- start start 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b744a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 0x7ff1b0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b744a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b01017a0 0x7ff1b00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b744a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0073b10 con 0x7ff1b0100df0 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b744a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0073c50 con 0x7ff1b01017a0 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b51e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 0x7ff1b0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b51e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 0x7ff1b0073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60542/0 (socket says 172.21.15.151:60542) 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b51e6700 1 -- 172.21.15.151:0/4136445637 learned_addr learned my addr 172.21.15.151:0/4136445637 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b49e5700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b01017a0 0x7ff1b00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b51e6700 1 -- 172.21.15.151:0/4136445637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b01017a0 msgr2=0x7ff1b00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:15.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b51e6700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b01017a0 0x7ff1b00735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:15.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b51e6700 1 -- 172.21.15.151:0/4136445637 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a00095f0 con 0x7ff1b0100df0 2024-04-16T03:00:15.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.859+0000 7ff1b51e6700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 0x7ff1b0073090 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0006010 tx=0x7ff1a000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:15.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.860+0000 7ff1a67fc700 1 -- 172.21.15.151:0/4136445637 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a001d070 con 0x7ff1b0100df0 2024-04-16T03:00:15.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.860+0000 7ff1a67fc700 1 -- 172.21.15.151:0/4136445637 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1a0005dc0 con 0x7ff1b0100df0 2024-04-16T03:00:15.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.860+0000 7ff1a67fc700 1 -- 172.21.15.151:0/4136445637 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a0017400 con 0x7ff1b0100df0 2024-04-16T03:00:15.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.860+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b0073ed0 con 0x7ff1b0100df0 2024-04-16T03:00:15.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.860+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b01a2c60 con 0x7ff1b0100df0 2024-04-16T03:00:15.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.861+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b0109190 con 0x7ff1b0100df0 2024-04-16T03:00:15.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.861+0000 7ff1a67fc700 1 -- 172.21.15.151:0/4136445637 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff1a0003a60 con 0x7ff1b0100df0 2024-04-16T03:00:15.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.862+0000 7ff1a67fc700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff19c06ecc0 0x7ff19c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:15.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.862+0000 7ff1a67fc700 1 -- 172.21.15.151:0/4136445637 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff1a0090380 con 0x7ff1b0100df0 2024-04-16T03:00:15.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.862+0000 7ff1b49e5700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff19c06ecc0 0x7ff19c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:15.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.862+0000 7ff1b49e5700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff19c06ecc0 0x7ff19c071180 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0074880 tx=0x7ff1ac0073d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:15.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:15.866+0000 7ff1a67fc700 1 -- 172.21.15.151:0/4136445637 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff1a0059090 con 0x7ff1b0100df0 2024-04-16T03:00:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:15 smithi153 ceph-mon[87858]: pgmap v648: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:00:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:00:16.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:16.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:00:16.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:16.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:16.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:16.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:15 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:16.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.086+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1b00611d0 con 0x7ff19c06ecc0 2024-04-16T03:00:16.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.087+0000 7ff1a67fc700 1 -- 172.21.15.151:0/4136445637 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff1b00611d0 con 0x7ff19c06ecc0 2024-04-16T03:00:16.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.089+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff19c06ecc0 msgr2=0x7ff19c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.089+0000 7ff1b744a700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff19c06ecc0 0x7ff19c071180 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0074880 tx=0x7ff1ac0073d0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.089+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 msgr2=0x7ff1b0073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.089+0000 7ff1b744a700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 0x7ff1b0073090 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0006010 tx=0x7ff1a000bd70 comp rx=0 tx=0).stop 2024-04-16T03:00:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.089+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 shutdown_connections 2024-04-16T03:00:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.089+0000 7ff1b744a700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff19c06ecc0 0x7ff19c071180 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.089+0000 7ff1b744a700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1b0100df0 0x7ff1b0073090 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.089+0000 7ff1b744a700 1 --2- 172.21.15.151:0/4136445637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b01017a0 0x7ff1b00735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.089+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 >> 172.21.15.151:0/4136445637 conn(0x7ff1b00fc6a0 msgr2=0x7ff1b00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.090+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 shutdown_connections 2024-04-16T03:00:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.090+0000 7ff1b744a700 1 -- 172.21.15.151:0/4136445637 wait complete. 2024-04-16T03:00:16.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.194+0000 7fcdce2ef700 1 -- 172.21.15.151:0/3244670709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 msgr2=0x7fcdc80ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.194+0000 7fcdce2ef700 1 --2- 172.21.15.151:0/3244670709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 0x7fcdc80ff910 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8009910 tx=0x7fcdb8009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:16.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.194+0000 7fcdce2ef700 1 -- 172.21.15.151:0/3244670709 shutdown_connections 2024-04-16T03:00:16.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.194+0000 7fcdce2ef700 1 --2- 172.21.15.151:0/3244670709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc80ffee0 0x7fcdc8105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.194+0000 7fcdce2ef700 1 --2- 172.21.15.151:0/3244670709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 0x7fcdc80ff910 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.194+0000 7fcdce2ef700 1 -- 172.21.15.151:0/3244670709 >> 172.21.15.151:0/3244670709 conn(0x7fcdc8074df0 msgr2=0x7fcdc8075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:16.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.194+0000 7fcdce2ef700 1 -- 172.21.15.151:0/3244670709 shutdown_connections 2024-04-16T03:00:16.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.194+0000 7fcdce2ef700 1 -- 172.21.15.151:0/3244670709 wait complete. 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdce2ef700 1 Processor -- start 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdce2ef700 1 -- start start 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdce2ef700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 0x7fcdc819e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdce2ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc80ffee0 0x7fcdc819ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdce2ef700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc819f3a0 con 0x7fcdc80ff530 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdce2ef700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc8198850 con 0x7fcdc80ffee0 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdc7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 0x7fcdc819e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdc7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 0x7fcdc819e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60562/0 (socket says 172.21.15.151:60562) 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdc7fff700 1 -- 172.21.15.151:0/2483891402 learned_addr learned my addr 172.21.15.151:0/2483891402 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:16.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdbffff700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc80ffee0 0x7fcdc819ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:16.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdc7fff700 1 -- 172.21.15.151:0/2483891402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc80ffee0 msgr2=0x7fcdc819ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdc7fff700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc80ffee0 0x7fcdc819ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdc7fff700 1 -- 172.21.15.151:0/2483891402 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb80095f0 con 0x7fcdc80ff530 2024-04-16T03:00:16.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.195+0000 7fcdc7fff700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 0x7fcdc819e7d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800f690 tx=0x7fcdb800f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:16.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.196+0000 7fcdc5ffb700 1 -- 172.21.15.151:0/2483891402 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb801d070 con 0x7fcdc80ff530 2024-04-16T03:00:16.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.196+0000 7fcdc5ffb700 1 -- 172.21.15.151:0/2483891402 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdb80050d0 con 0x7fcdc80ff530 2024-04-16T03:00:16.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.196+0000 7fcdc5ffb700 1 -- 172.21.15.151:0/2483891402 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb8022470 con 0x7fcdc80ff530 2024-04-16T03:00:16.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.196+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc8198ad0 con 0x7fcdc80ff530 2024-04-16T03:00:16.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.196+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc8198fc0 con 0x7fcdc80ff530 2024-04-16T03:00:16.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.197+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc8104ff0 con 0x7fcdc80ff530 2024-04-16T03:00:16.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.197+0000 7fcdc5ffb700 1 -- 172.21.15.151:0/2483891402 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fcdb8003780 con 0x7fcdc80ff530 2024-04-16T03:00:16.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.198+0000 7fcdc5ffb700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcdb406ecc0 0x7fcdb4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:16.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.198+0000 7fcdc5ffb700 1 -- 172.21.15.151:0/2483891402 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcdb8090230 con 0x7fcdc80ff530 2024-04-16T03:00:16.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.198+0000 7fcdbffff700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcdb406ecc0 0x7fcdb4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:16.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.198+0000 7fcdbffff700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcdb406ecc0 0x7fcdb4071180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8107110 tx=0x7fcdb0009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:16.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.203+0000 7fcdc5ffb700 1 -- 172.21.15.151:0/2483891402 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcdb805cb90 con 0x7fcdc80ff530 2024-04-16T03:00:16.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.413+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdc81992f0 con 0x7fcdb406ecc0 2024-04-16T03:00:16.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.423+0000 7fcdc5ffb700 1 -- 172.21.15.151:0/2483891402 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fcdc81992f0 con 0x7fcdb406ecc0 2024-04-16T03:00:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:00:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (21m) 15m ago 33m 26.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (33m) 15m ago 33m 9101k - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (32m) 14m ago 32m 8980k - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (19m) 15m ago 33m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (19m) 14m ago 32m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (21m) 15m ago 32m 80.0M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (15m) 15m ago 27m 15.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (15m) 15m ago 27m 11.6M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (27m) 14m ago 27m 12.0M - 18.2.1 5be31c24972a 8eefd56690dc 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (15m) 14m ago 27m 11.3M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (23m) 15m ago 34m 510M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (22m) 14m ago 32m 453M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (20m) 15m ago 34m 54.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (20m) 14m ago 32m 47.9M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:00:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (22m) 15m ago 33m 15.8M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:00:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (22m) 14m ago 32m 15.7M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:00:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (18m) 15m ago 31m 93.0M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:00:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (18m) 15m ago 30m 99.6M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:00:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (17m) 15m ago 30m 83.5M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:00:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (17m) 14m ago 29m 110M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:00:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (16m) 14m ago 29m 90.4M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:00:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (16m) 14m ago 29m 100M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:00:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (22m) 15m ago 32m 75.3M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcdb406ecc0 msgr2=0x7fcdb4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcdb406ecc0 0x7fcdb4071180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8107110 tx=0x7fcdb0009260 comp rx=0 tx=0).stop 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 msgr2=0x7fcdc819e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 0x7fcdc819e7d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800f690 tx=0x7fcdb800f6c0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 shutdown_connections 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcdb406ecc0 0x7fcdb4071180 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdc80ff530 0x7fcdc819e7d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 --2- 172.21.15.151:0/2483891402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc80ffee0 0x7fcdc819ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 >> 172.21.15.151:0/2483891402 conn(0x7fcdc8074df0 msgr2=0x7fcdc80fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 shutdown_connections 2024-04-16T03:00:16.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.426+0000 7fcdce2ef700 1 -- 172.21.15.151:0/2483891402 wait complete. 2024-04-16T03:00:16.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.531+0000 7f288956e700 1 -- 172.21.15.151:0/3974079610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 msgr2=0x7f28840ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.531+0000 7f288956e700 1 --2- 172.21.15.151:0/3974079610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 0x7f28840ffcf0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2874009910 tx=0x7f2874009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:16.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.531+0000 7f288956e700 1 -- 172.21.15.151:0/3974079610 shutdown_connections 2024-04-16T03:00:16.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.531+0000 7f288956e700 1 --2- 172.21.15.151:0/3974079610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884100230 0x7f288410cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.531+0000 7f288956e700 1 --2- 172.21.15.151:0/3974079610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 0x7f28840ffcf0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.533 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.531+0000 7f288956e700 1 -- 172.21.15.151:0/3974079610 >> 172.21.15.151:0/3974079610 conn(0x7f28840fb550 msgr2=0x7f28840fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:16.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.531+0000 7f288956e700 1 -- 172.21.15.151:0/3974079610 shutdown_connections 2024-04-16T03:00:16.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.531+0000 7f288956e700 1 -- 172.21.15.151:0/3974079610 wait complete. 2024-04-16T03:00:16.534 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f288956e700 1 Processor -- start 2024-04-16T03:00:16.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f288956e700 1 -- start start 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f288956e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 0x7f288419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f288956e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884100230 0x7f288419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f288956e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288419f2b0 con 0x7f28840ff910 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f288956e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28841987d0 con 0x7f2884100230 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f2882ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 0x7f288419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f2882ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 0x7f288419e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54654/0 (socket says 172.21.15.151:54654) 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f2882ffd700 1 -- 172.21.15.151:0/656165442 learned_addr learned my addr 172.21.15.151:0/656165442 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f287a7fc700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884100230 0x7f288419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f2882ffd700 1 -- 172.21.15.151:0/656165442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884100230 msgr2=0x7f288419ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f2882ffd700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884100230 0x7f288419ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.532+0000 7f2882ffd700 1 -- 172.21.15.151:0/656165442 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28740095f0 con 0x7f28840ff910 2024-04-16T03:00:16.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.533+0000 7f2882ffd700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 0x7f288419e6e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2874006010 tx=0x7f287400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:16.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.533+0000 7f2880ff9700 1 -- 172.21.15.151:0/656165442 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f287401d070 con 0x7f28840ff910 2024-04-16T03:00:16.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.533+0000 7f2880ff9700 1 -- 172.21.15.151:0/656165442 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2874004fb0 con 0x7f28840ff910 2024-04-16T03:00:16.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.533+0000 7f2880ff9700 1 -- 172.21.15.151:0/656165442 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2874022ea0 con 0x7f28840ff910 2024-04-16T03:00:16.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.533+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2884198a50 con 0x7f28840ff910 2024-04-16T03:00:16.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.533+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2884198f40 con 0x7f28840ff910 2024-04-16T03:00:16.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.534+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288410a6f0 con 0x7f28840ff910 2024-04-16T03:00:16.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.535+0000 7f2880ff9700 1 -- 172.21.15.151:0/656165442 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2874022470 con 0x7f28840ff910 2024-04-16T03:00:16.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.535+0000 7f2880ff9700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f287006ecc0 0x7f2870071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:16.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.535+0000 7f2880ff9700 1 -- 172.21.15.151:0/656165442 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2874090350 con 0x7f28840ff910 2024-04-16T03:00:16.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.535+0000 7f287a7fc700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f287006ecc0 0x7f2870071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:16.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.536+0000 7f287a7fc700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f287006ecc0 0x7f2870071180 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f286c005fd0 tx=0x7f286c005dd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:16.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.540+0000 7f2880ff9700 1 -- 172.21.15.151:0/656165442 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f287405cd60 con 0x7f28840ff910 2024-04-16T03:00:16.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.830+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f288404ea90 con 0x7f28840ff910 2024-04-16T03:00:16.833 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:16 smithi151 ceph-mon[112233]: pgmap v649: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:16.833 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:16 smithi151 ceph-mon[112233]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:16.833 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:16 smithi151 ceph-mon[112233]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-16T03:00:16.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.831+0000 7f2880ff9700 1 -- 172.21.15.151:0/656165442 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f2874022720 con 0x7f28840ff910 2024-04-16T03:00:16.833 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:00:16.833 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:00:16.833 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:00:16.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 3 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 13 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:00:16.834 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:00:16.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f287006ecc0 msgr2=0x7f2870071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f287006ecc0 0x7f2870071180 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f286c005fd0 tx=0x7f286c005dd0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 msgr2=0x7f288419e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 0x7f288419e6e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2874006010 tx=0x7f287400be20 comp rx=0 tx=0).stop 2024-04-16T03:00:16.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 shutdown_connections 2024-04-16T03:00:16.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f287006ecc0 0x7f2870071180 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28840ff910 0x7f288419e6e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 --2- 172.21.15.151:0/656165442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884100230 0x7f288419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 >> 172.21.15.151:0/656165442 conn(0x7f28840fb550 msgr2=0x7f28840fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:16.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.834+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 shutdown_connections 2024-04-16T03:00:16.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.835+0000 7f288956e700 1 -- 172.21.15.151:0/656165442 wait complete. 2024-04-16T03:00:16.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.939+0000 7f1912b86700 1 -- 172.21.15.151:0/4114071785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 msgr2=0x7f190c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.939+0000 7f1912b86700 1 --2- 172.21.15.151:0/4114071785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 0x7f190c100050 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009910 tx=0x7f18f8009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:16.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.939+0000 7f1912b86700 1 -- 172.21.15.151:0/4114071785 shutdown_connections 2024-04-16T03:00:16.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.939+0000 7f1912b86700 1 --2- 172.21.15.151:0/4114071785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 0x7f190c100050 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.939+0000 7f1912b86700 1 --2- 172.21.15.151:0/4114071785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f190c105a40 0x7f190c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.939+0000 7f1912b86700 1 -- 172.21.15.151:0/4114071785 >> 172.21.15.151:0/4114071785 conn(0x7f190c0755a0 msgr2=0x7f190c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:16.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.939+0000 7f1912b86700 1 -- 172.21.15.151:0/4114071785 shutdown_connections 2024-04-16T03:00:16.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.939+0000 7f1912b86700 1 -- 172.21.15.151:0/4114071785 wait complete. 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f1912b86700 1 Processor -- start 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f1912b86700 1 -- start start 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f1912b86700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 0x7f190c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f1912b86700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f190c105a40 0x7f190c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f1912b86700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c19d040 con 0x7f190c0ffbd0 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f1912b86700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c196580 con 0x7f190c105a40 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f1910922700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 0x7f190c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f1910922700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 0x7f190c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54666/0 (socket says 172.21.15.151:54666) 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f1910922700 1 -- 172.21.15.151:0/3325883771 learned_addr learned my addr 172.21.15.151:0/3325883771 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.940+0000 7f190bfff700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f190c105a40 0x7f190c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f1910922700 1 -- 172.21.15.151:0/3325883771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f190c105a40 msgr2=0x7f190c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f1910922700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f190c105a40 0x7f190c19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:16.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f1910922700 1 -- 172.21.15.151:0/3325883771 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f80095f0 con 0x7f190c0ffbd0 2024-04-16T03:00:16.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f190bfff700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f190c105a40 0x7f190c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:00:16.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f1910922700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 0x7f190c19c500 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f190000c930 tx=0x7f190000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:16.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f1909ffb700 1 -- 172.21.15.151:0/3325883771 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1900007ad0 con 0x7f190c0ffbd0 2024-04-16T03:00:16.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f1909ffb700 1 -- 172.21.15.151:0/3325883771 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1900007c30 con 0x7f190c0ffbd0 2024-04-16T03:00:16.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190c196860 con 0x7f190c0ffbd0 2024-04-16T03:00:16.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f1909ffb700 1 -- 172.21.15.151:0/3325883771 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1900018610 con 0x7f190c0ffbd0 2024-04-16T03:00:16.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.941+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190c196db0 con 0x7f190c0ffbd0 2024-04-16T03:00:16.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.942+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190c108fe0 con 0x7f190c0ffbd0 2024-04-16T03:00:16.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.943+0000 7f1909ffb700 1 -- 172.21.15.151:0/3325883771 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f190000f450 con 0x7f190c0ffbd0 2024-04-16T03:00:16.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.943+0000 7f1909ffb700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f18fc06ecc0 0x7f18fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:16.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.943+0000 7f1909ffb700 1 -- 172.21.15.151:0/3325883771 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f190008efb0 con 0x7f190c0ffbd0 2024-04-16T03:00:16.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.943+0000 7f190bfff700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f18fc06ecc0 0x7f18fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:16.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.944+0000 7f190bfff700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f18fc06ecc0 0x7f18fc071180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f18f8005b40 tx=0x7f18f801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:16.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:16.948+0000 7f1909ffb700 1 -- 172.21.15.151:0/3325883771 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f19000c3ff0 con 0x7f190c0ffbd0 2024-04-16T03:00:16.982 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:16 smithi153 ceph-mon[87858]: pgmap v649: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:16.982 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:16 smithi153 ceph-mon[87858]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:16.982 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:16 smithi153 ceph-mon[87858]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-16T03:00:17.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.197+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f190c197030 con 0x7f190c0ffbd0 2024-04-16T03:00:17.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.198+0000 7f1909ffb700 1 -- 172.21.15.151:0/3325883771 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f190005b910 con 0x7f190c0ffbd0 2024-04-16T03:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:e31 2024-04-16T03:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44247} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/341639119,v1:172.21.15.153:6827/341639119] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T03:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:00:17.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f18fc06ecc0 msgr2=0x7f18fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f18fc06ecc0 0x7f18fc071180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f18f8005b40 tx=0x7f18f801a040 comp rx=0 tx=0).stop 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 msgr2=0x7f190c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 0x7f190c19c500 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f190000c930 tx=0x7f190000ccf0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 shutdown_connections 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f18fc06ecc0 0x7f18fc071180 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f190c0ffbd0 0x7f190c19c500 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 --2- 172.21.15.151:0/3325883771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f190c105a40 0x7f190c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 >> 172.21.15.151:0/3325883771 conn(0x7f190c0755a0 msgr2=0x7f190c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 shutdown_connections 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.200+0000 7f1912b86700 1 -- 172.21.15.151:0/3325883771 wait complete. 2024-04-16T03:00:17.204 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T03:00:17.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.307+0000 7f4046c46700 1 -- 172.21.15.151:0/2746942130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 msgr2=0x7f4040105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.307+0000 7f4046c46700 1 --2- 172.21.15.151:0/2746942130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 0x7f4040105e20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4034009910 tx=0x7f4034009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:17.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.307+0000 7f4046c46700 1 -- 172.21.15.151:0/2746942130 shutdown_connections 2024-04-16T03:00:17.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.307+0000 7f4046c46700 1 --2- 172.21.15.151:0/2746942130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40400ffbd0 0x7f4040100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.307+0000 7f4046c46700 1 --2- 172.21.15.151:0/2746942130 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 0x7f4040105e20 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.307+0000 7f4046c46700 1 -- 172.21.15.151:0/2746942130 >> 172.21.15.151:0/2746942130 conn(0x7f40400755a0 msgr2=0x7f40400759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:17.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.307+0000 7f4046c46700 1 -- 172.21.15.151:0/2746942130 shutdown_connections 2024-04-16T03:00:17.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.307+0000 7f4046c46700 1 -- 172.21.15.151:0/2746942130 wait complete. 2024-04-16T03:00:17.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.308+0000 7f4046c46700 1 Processor -- start 2024-04-16T03:00:17.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.308+0000 7f4046c46700 1 -- start start 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.308+0000 7f4046c46700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40400ffbd0 0x7f404010b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.308+0000 7f4046c46700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 0x7f404010bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.308+0000 7f4046c46700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404010c1b0 con 0x7f4040105a40 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.308+0000 7f4046c46700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404010c2f0 con 0x7f40400ffbd0 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.308+0000 7f403ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 0x7f404010bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.308+0000 7f403ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 0x7f404010bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54680/0 (socket says 172.21.15.151:54680) 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.308+0000 7f403ffff700 1 -- 172.21.15.151:0/4081343349 learned_addr learned my addr 172.21.15.151:0/4081343349 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.309+0000 7f403ffff700 1 -- 172.21.15.151:0/4081343349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40400ffbd0 msgr2=0x7f404010b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.309+0000 7f403ffff700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40400ffbd0 0x7f404010b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.309+0000 7f403ffff700 1 -- 172.21.15.151:0/4081343349 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340095f0 con 0x7f4040105a40 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.309+0000 7f403ffff700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 0x7f404010bc70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f402c00b4d0 tx=0x7f402c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.309+0000 7f403dffb700 1 -- 172.21.15.151:0/4081343349 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f402c011800 con 0x7f4040105a40 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.309+0000 7f403dffb700 1 -- 172.21.15.151:0/4081343349 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f402c011e20 con 0x7f4040105a40 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.309+0000 7f403dffb700 1 -- 172.21.15.151:0/4081343349 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f402c00f2e0 con 0x7f4040105a40 2024-04-16T03:00:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.309+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40401a2a30 con 0x7f4040105a40 2024-04-16T03:00:17.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.309+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40401a2e40 con 0x7f4040105a40 2024-04-16T03:00:17.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.310+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4040108040 con 0x7f4040105a40 2024-04-16T03:00:17.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.311+0000 7f403dffb700 1 -- 172.21.15.151:0/4081343349 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f402c010380 con 0x7f4040105a40 2024-04-16T03:00:17.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.311+0000 7f403dffb700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f403006ecc0 0x7f4030071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:17.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.311+0000 7f403dffb700 1 -- 172.21.15.151:0/4081343349 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f402c08e930 con 0x7f4040105a40 2024-04-16T03:00:17.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.311+0000 7f40449e2700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f403006ecc0 0x7f4030071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:17.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.312+0000 7f40449e2700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f403006ecc0 0x7f4030071180 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4034009fd0 tx=0x7f4034005fb0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:17.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.316+0000 7f403dffb700 1 -- 172.21.15.151:0/4081343349 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f402c057760 con 0x7f4040105a40 2024-04-16T03:00:17.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.533+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f404002cce0 con 0x7f403006ecc0 2024-04-16T03:00:17.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.534+0000 7f403dffb700 1 -- 172.21.15.151:0/4081343349 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f404002cce0 con 0x7f403006ecc0 2024-04-16T03:00:17.536 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:00:17.537 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:00:17.537 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:00:17.537 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:00:17.537 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:00:17.537 INFO:teuthology.orchestra.run.smithi151.stdout: "osd", 2024-04-16T03:00:17.537 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:00:17.537 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:00:17.538 INFO:teuthology.orchestra.run.smithi151.stdout: "mon" 2024-04-16T03:00:17.538 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:00:17.538 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "15/23 daemons upgraded", 2024-04-16T03:00:17.538 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:00:17.538 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:00:17.538 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:00:17.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.536+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f403006ecc0 msgr2=0x7f4030071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.536+0000 7f4046c46700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f403006ecc0 0x7f4030071180 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4034009fd0 tx=0x7f4034005fb0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.536+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 msgr2=0x7f404010bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.536+0000 7f4046c46700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 0x7f404010bc70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f402c00b4d0 tx=0x7f402c00b890 comp rx=0 tx=0).stop 2024-04-16T03:00:17.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.536+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 shutdown_connections 2024-04-16T03:00:17.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.536+0000 7f4046c46700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f403006ecc0 0x7f4030071180 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.536+0000 7f4046c46700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40400ffbd0 0x7f404010b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.536+0000 7f4046c46700 1 --2- 172.21.15.151:0/4081343349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4040105a40 0x7f404010bc70 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.536+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 >> 172.21.15.151:0/4081343349 conn(0x7f40400755a0 msgr2=0x7f40400fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.537+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 shutdown_connections 2024-04-16T03:00:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.537+0000 7f4046c46700 1 -- 172.21.15.151:0/4081343349 wait complete. 2024-04-16T03:00:17.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 -- 172.21.15.151:0/3947171900 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 msgr2=0x7f1d8c105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 --2- 172.21.15.151:0/3947171900 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 0x7f1d8c105790 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c009910 tx=0x7f1d7c009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:17.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 -- 172.21.15.151:0/3947171900 shutdown_connections 2024-04-16T03:00:17.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 --2- 172.21.15.151:0/3947171900 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 0x7f1d8c105790 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 --2- 172.21.15.151:0/3947171900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c068670 0x7f1d8c068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 -- 172.21.15.151:0/3947171900 >> 172.21.15.151:0/3947171900 conn(0x7f1d8c0755a0 msgr2=0x7f1d8c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:17.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 -- 172.21.15.151:0/3947171900 shutdown_connections 2024-04-16T03:00:17.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 -- 172.21.15.151:0/3947171900 wait complete. 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 Processor -- start 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.641+0000 7f1d938db700 1 -- start start 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d938db700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c068670 0x7f1d8c196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d938db700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 0x7f1d8c196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d938db700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c1972d0 con 0x7f1d8c069020 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d938db700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c19a000 con 0x7f1d8c068670 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d90e76700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 0x7f1d8c196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d90e76700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 0x7f1d8c196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54700/0 (socket says 172.21.15.151:54700) 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d90e76700 1 -- 172.21.15.151:0/2654848545 learned_addr learned my addr 172.21.15.151:0/2654848545 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d91677700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c068670 0x7f1d8c196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:17.645 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d90e76700 1 -- 172.21.15.151:0/2654848545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c068670 msgr2=0x7f1d8c196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d90e76700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c068670 0x7f1d8c196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d90e76700 1 -- 172.21.15.151:0/2654848545 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0095f0 con 0x7f1d8c069020 2024-04-16T03:00:17.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d90e76700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 0x7f1d8c196c40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0092e0 tx=0x7f1d7c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:17.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d827fc700 1 -- 172.21.15.151:0/2654848545 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7c01d070 con 0x7f1d8c069020 2024-04-16T03:00:17.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c19a280 con 0x7f1d8c069020 2024-04-16T03:00:17.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d827fc700 1 -- 172.21.15.151:0/2654848545 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d7c005dc0 con 0x7f1d8c069020 2024-04-16T03:00:17.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d827fc700 1 -- 172.21.15.151:0/2654848545 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7c017400 con 0x7f1d8c069020 2024-04-16T03:00:17.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.642+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8c19a770 con 0x7f1d8c069020 2024-04-16T03:00:17.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.643+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d8c1090a0 con 0x7f1d8c069020 2024-04-16T03:00:17.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.644+0000 7f1d827fc700 1 -- 172.21.15.151:0/2654848545 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1d7c00f6f0 con 0x7f1d8c069020 2024-04-16T03:00:17.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.645+0000 7f1d827fc700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ecc0 0x7f1d78071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:17.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.645+0000 7f1d91677700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ecc0 0x7f1d78071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:17.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.645+0000 7f1d827fc700 1 -- 172.21.15.151:0/2654848545 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1d7c0902c0 con 0x7f1d8c069020 2024-04-16T03:00:17.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.645+0000 7f1d91677700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ecc0 0x7f1d78071180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d88005950 tx=0x7f1d8800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:17.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.649+0000 7f1d827fc700 1 -- 172.21.15.151:0/2654848545 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1d7c059070 con 0x7f1d8c069020 2024-04-16T03:00:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.931+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1d8c19aa50 con 0x7f1d8c069020 2024-04-16T03:00:17.933 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:17 smithi151 ceph-mon[112233]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:17.933 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:17 smithi151 ceph-mon[112233]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:17.933 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:17 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/656165442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:17.933 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:17 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:17.933 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:17 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.mqbgav", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T03:00:17.933 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:17 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:00:17.934 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:17 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3325883771' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:00:17.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.931+0000 7f1d827fc700 1 -- 172.21.15.151:0/2654848545 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f1d7c05c730 con 0x7f1d8c069020 2024-04-16T03:00:17.934 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:00:17.934 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:00:17.934 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:00:17.934 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:00:17.934 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:00:17.934 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:00:17.934 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:00:17.935 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.934+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ecc0 msgr2=0x7f1d78071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.934+0000 7f1d938db700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ecc0 0x7f1d78071180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d88005950 tx=0x7f1d8800b410 comp rx=0 tx=0).stop 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.934+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 msgr2=0x7f1d8c196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.934+0000 7f1d938db700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 0x7f1d8c196c40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0092e0 tx=0x7f1d7c00bd70 comp rx=0 tx=0).stop 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.935+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 shutdown_connections 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.935+0000 7f1d938db700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d7806ecc0 0x7f1d78071180 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.935+0000 7f1d938db700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d8c068670 0x7f1d8c196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.935+0000 7f1d938db700 1 --2- 172.21.15.151:0/2654848545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d8c069020 0x7f1d8c196c40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.935+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 >> 172.21.15.151:0/2654848545 conn(0x7f1d8c0755a0 msgr2=0x7f1d8c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.935+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 shutdown_connections 2024-04-16T03:00:17.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:17.935+0000 7f1d938db700 1 -- 172.21.15.151:0/2654848545 wait complete. 2024-04-16T03:00:17.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:17 smithi153 ceph-mon[87858]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:17.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:17 smithi153 ceph-mon[87858]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:17.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:17 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/656165442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:17.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:17 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:17.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:17 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.mqbgav", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T03:00:17.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:17 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:00:17.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:17 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3325883771' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:00:18.898 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:18 smithi153 ceph-mon[87858]: Upgrade: Updating mds.cephfs.smithi153.mqbgav 2024-04-16T03:00:18.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:18 smithi153 ceph-mon[87858]: Deploying daemon mds.cephfs.smithi153.mqbgav on smithi153 2024-04-16T03:00:18.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:18 smithi153 ceph-mon[87858]: pgmap v650: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:18.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:18 smithi153 ceph-mon[87858]: from='client.35230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:18.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:18 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2654848545' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:00:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:18 smithi151 ceph-mon[112233]: Upgrade: Updating mds.cephfs.smithi153.mqbgav 2024-04-16T03:00:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:18 smithi151 ceph-mon[112233]: Deploying daemon mds.cephfs.smithi153.mqbgav on smithi153 2024-04-16T03:00:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:18 smithi151 ceph-mon[112233]: pgmap v650: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:18 smithi151 ceph-mon[112233]: from='client.35230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:18 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2654848545' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:00:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:20 smithi153 ceph-mon[87858]: pgmap v651: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:20 smithi153 ceph-mon[87858]: osdmap e86: 6 total, 6 up, 6 in 2024-04-16T03:00:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:20 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-16T03:00:21.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:20 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T03:00:21.047 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:20 smithi151 ceph-mon[112233]: pgmap v651: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:21.047 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:20 smithi151 ceph-mon[112233]: osdmap e86: 6 total, 6 up, 6 in 2024-04-16T03:00:21.047 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:20 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-16T03:00:21.047 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:20 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T03:00:23.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:22 smithi153 ceph-mon[87858]: pgmap v653: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:23.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:22 smithi151 ceph-mon[112233]: pgmap v653: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:00:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: pgmap v654: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] up:boot 2024-04-16T03:00:24.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T03:00:24.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T03:00:24.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:00:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: pgmap v654: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] up:boot 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:24.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:00:26.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:26 smithi151 ceph-mon[112233]: pgmap v655: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:26.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:26 smithi153 ceph-mon[87858]: pgmap v655: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:27.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:27 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:27.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:27 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:27.830 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:27 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:27.830 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:27 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:28.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:28 smithi153 ceph-mon[87858]: pgmap v656: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:28.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:28.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:28.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:28.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:28 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:28 smithi151 ceph-mon[112233]: pgmap v656: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:28 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:00:30.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:30 smithi153 ceph-mon[87858]: pgmap v657: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:30.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:30 smithi151 ceph-mon[112233]: pgmap v657: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:32 smithi151 ceph-mon[112233]: pgmap v658: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:33.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:32 smithi153 ceph-mon[87858]: pgmap v658: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:34.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:34 smithi153 ceph-mon[87858]: pgmap v659: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:34.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:34 smithi151 ceph-mon[112233]: pgmap v659: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:36 smithi151 ceph-mon[112233]: pgmap v660: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:36 smithi153 ceph-mon[87858]: pgmap v660: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:38 smithi151 ceph-mon[112233]: pgmap v661: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:39.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:38 smithi153 ceph-mon[87858]: pgmap v661: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:39.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:00:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:00:40.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:40 smithi151 ceph-mon[112233]: pgmap v662: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:41.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:40 smithi153 ceph-mon[87858]: pgmap v662: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:42.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:42 smithi151 ceph-mon[112233]: pgmap v663: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:43.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:42 smithi153 ceph-mon[87858]: pgmap v663: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:44.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:44 smithi151 ceph-mon[112233]: pgmap v664: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:45.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:44 smithi153 ceph-mon[87858]: pgmap v664: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:46.927 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:46 smithi151 ceph-mon[112233]: pgmap v665: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:47.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:46 smithi153 ceph-mon[87858]: pgmap v665: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:48.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.059+0000 7f2b6eef9700 1 -- 172.21.15.151:0/3071300085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68100de0 msgr2=0x7f2b681011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.061 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.059+0000 7f2b6eef9700 1 --2- 172.21.15.151:0/3071300085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68100de0 0x7f2b681011c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009910 tx=0x7f2b5c009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:48.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.059+0000 7f2b6eef9700 1 -- 172.21.15.151:0/3071300085 shutdown_connections 2024-04-16T03:00:48.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.059+0000 7f2b6eef9700 1 --2- 172.21.15.151:0/3071300085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68101790 0x7f2b681057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.059+0000 7f2b6eef9700 1 --2- 172.21.15.151:0/3071300085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68100de0 0x7f2b681011c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.059+0000 7f2b6eef9700 1 -- 172.21.15.151:0/3071300085 >> 172.21.15.151:0/3071300085 conn(0x7f2b680fc6b0 msgr2=0x7f2b680fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:48.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.059+0000 7f2b6eef9700 1 -- 172.21.15.151:0/3071300085 shutdown_connections 2024-04-16T03:00:48.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.059+0000 7f2b6eef9700 1 -- 172.21.15.151:0/3071300085 wait complete. 2024-04-16T03:00:48.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.060+0000 7f2b6eef9700 1 Processor -- start 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.060+0000 7f2b6eef9700 1 -- start start 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.060+0000 7f2b6eef9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68100de0 0x7f2b6819c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.060+0000 7f2b6eef9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68101790 0x7f2b6819cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.060+0000 7f2b6eef9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6819d190 con 0x7f2b68101790 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.060+0000 7f2b6eef9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68196640 con 0x7f2b68100de0 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68101790 0x7f2b6819cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68101790 0x7f2b6819cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38504/0 (socket says 172.21.15.151:38504) 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b67fff700 1 -- 172.21.15.151:0/4283844796 learned_addr learned my addr 172.21.15.151:0/4283844796 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b6cc95700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68100de0 0x7f2b6819c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b67fff700 1 -- 172.21.15.151:0/4283844796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68100de0 msgr2=0x7f2b6819c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b67fff700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68100de0 0x7f2b6819c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b67fff700 1 -- 172.21.15.151:0/4283844796 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0095f0 con 0x7f2b68101790 2024-04-16T03:00:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b67fff700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68101790 0x7f2b6819cb00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400e8f0 tx=0x7f2b5400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:48.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b65ffb700 1 -- 172.21.15.151:0/4283844796 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b54009790 con 0x7f2b68101790 2024-04-16T03:00:48.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b65ffb700 1 -- 172.21.15.151:0/4283844796 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b5400c810 con 0x7f2b68101790 2024-04-16T03:00:48.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b65ffb700 1 -- 172.21.15.151:0/4283844796 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b54005580 con 0x7f2b68101790 2024-04-16T03:00:48.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.061+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b68196920 con 0x7f2b68101790 2024-04-16T03:00:48.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.062+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b68196e70 con 0x7f2b68101790 2024-04-16T03:00:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.063+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b68109180 con 0x7f2b68101790 2024-04-16T03:00:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.063+0000 7f2b65ffb700 1 -- 172.21.15.151:0/4283844796 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2b5400c980 con 0x7f2b68101790 2024-04-16T03:00:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.063+0000 7f2b65ffb700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b5806ecc0 0x7f2b58071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.064+0000 7f2b65ffb700 1 -- 172.21.15.151:0/4283844796 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2b54014070 con 0x7f2b68101790 2024-04-16T03:00:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.064+0000 7f2b6cc95700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b5806ecc0 0x7f2b58071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.064+0000 7f2b6cc95700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b5806ecc0 0x7f2b58071180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0059d0 tx=0x7f2b5c005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:48.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.068+0000 7f2b65ffb700 1 -- 172.21.15.151:0/4283844796 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2b5405ba00 con 0x7f2b68101790 2024-04-16T03:00:48.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.286+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b681970f0 con 0x7f2b5806ecc0 2024-04-16T03:00:48.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.288+0000 7f2b65ffb700 1 -- 172.21.15.151:0/4283844796 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2b681970f0 con 0x7f2b5806ecc0 2024-04-16T03:00:48.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b5806ecc0 msgr2=0x7f2b58071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b5806ecc0 0x7f2b58071180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0059d0 tx=0x7f2b5c005940 comp rx=0 tx=0).stop 2024-04-16T03:00:48.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68101790 msgr2=0x7f2b6819cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68101790 0x7f2b6819cb00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400e8f0 tx=0x7f2b5400ec00 comp rx=0 tx=0).stop 2024-04-16T03:00:48.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 shutdown_connections 2024-04-16T03:00:48.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2b5806ecc0 0x7f2b58071180 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68100de0 0x7f2b6819c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 --2- 172.21.15.151:0/4283844796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b68101790 0x7f2b6819cb00 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 >> 172.21.15.151:0/4283844796 conn(0x7f2b680fc6b0 msgr2=0x7f2b680fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:48.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 shutdown_connections 2024-04-16T03:00:48.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.290+0000 7f2b6eef9700 1 -- 172.21.15.151:0/4283844796 wait complete. 2024-04-16T03:00:48.302 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:00:48.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.398+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/257092894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 msgr2=0x7ffba41057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.398+0000 7ffbaaf4d700 1 --2- 172.21.15.151:0/257092894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 0x7ffba41057d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400b0a0 tx=0x7ffb9400b3b0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.398+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/257092894 shutdown_connections 2024-04-16T03:00:48.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.398+0000 7ffbaaf4d700 1 --2- 172.21.15.151:0/257092894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 0x7ffba41057d0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.398+0000 7ffbaaf4d700 1 --2- 172.21.15.151:0/257092894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba4100dd0 0x7ffba41011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.398+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/257092894 >> 172.21.15.151:0/257092894 conn(0x7ffba40fc640 msgr2=0x7ffba40fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:48.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.398+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/257092894 shutdown_connections 2024-04-16T03:00:48.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.398+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/257092894 wait complete. 2024-04-16T03:00:48.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffbaaf4d700 1 Processor -- start 2024-04-16T03:00:48.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffbaaf4d700 1 -- start start 2024-04-16T03:00:48.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffbaaf4d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba4100dd0 0x7ffba4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:48.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffbaaf4d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 0x7ffba40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:48.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffbaaf4d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba4073b10 con 0x7ffba4101780 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffbaaf4d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba4073c50 con 0x7ffba4100dd0 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffba3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 0x7ffba40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffba3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 0x7ffba40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38524/0 (socket says 172.21.15.151:38524) 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffba3fff700 1 -- 172.21.15.151:0/535406777 learned_addr learned my addr 172.21.15.151:0/535406777 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.399+0000 7ffba8ce9700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba4100dd0 0x7ffba4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.400+0000 7ffba3fff700 1 -- 172.21.15.151:0/535406777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba4100dd0 msgr2=0x7ffba4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.400+0000 7ffba3fff700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba4100dd0 0x7ffba4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.400+0000 7ffba3fff700 1 -- 172.21.15.151:0/535406777 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94009d00 con 0x7ffba4101780 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.400+0000 7ffba3fff700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 0x7ffba40735d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ffb94006130 tx=0x7ffb9400cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.400+0000 7ffba1ffb700 1 -- 172.21.15.151:0/535406777 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb9400e040 con 0x7ffba4101780 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.400+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba4073ed0 con 0x7ffba4101780 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.400+0000 7ffba1ffb700 1 -- 172.21.15.151:0/535406777 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb94003c10 con 0x7ffba4101780 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.400+0000 7ffba1ffb700 1 -- 172.21.15.151:0/535406777 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb9401d6b0 con 0x7ffba4101780 2024-04-16T03:00:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.400+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba41a2cc0 con 0x7ffba4101780 2024-04-16T03:00:48.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.401+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba41091d0 con 0x7ffba4101780 2024-04-16T03:00:48.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.402+0000 7ffba1ffb700 1 -- 172.21.15.151:0/535406777 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ffb94004600 con 0x7ffba4101780 2024-04-16T03:00:48.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.402+0000 7ffba1ffb700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffb9006ecc0 0x7ffb90071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:48.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.402+0000 7ffba1ffb700 1 -- 172.21.15.151:0/535406777 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffb94090630 con 0x7ffba4101780 2024-04-16T03:00:48.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.402+0000 7ffba8ce9700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffb9006ecc0 0x7ffb90071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:48.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.403+0000 7ffba8ce9700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffb9006ecc0 0x7ffb90071180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c005950 tx=0x7ffb8c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:48.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.407+0000 7ffba1ffb700 1 -- 172.21.15.151:0/535406777 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ffb940593e0 con 0x7ffba4101780 2024-04-16T03:00:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.626+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffba40611d0 con 0x7ffb9006ecc0 2024-04-16T03:00:48.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.628+0000 7ffba1ffb700 1 -- 172.21.15.151:0/535406777 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ffba40611d0 con 0x7ffb9006ecc0 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffb9006ecc0 msgr2=0x7ffb90071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffb9006ecc0 0x7ffb90071180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c005950 tx=0x7ffb8c00b410 comp rx=0 tx=0).stop 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 msgr2=0x7ffba40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 0x7ffba40735d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ffb94006130 tx=0x7ffb9400cd00 comp rx=0 tx=0).stop 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 shutdown_connections 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ffb9006ecc0 0x7ffb90071180 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba4100dd0 0x7ffba4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 --2- 172.21.15.151:0/535406777 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffba4101780 0x7ffba40735d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 >> 172.21.15.151:0/535406777 conn(0x7ffba40fc640 msgr2=0x7ffba40fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 shutdown_connections 2024-04-16T03:00:48.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.630+0000 7ffbaaf4d700 1 -- 172.21.15.151:0/535406777 wait complete. 2024-04-16T03:00:48.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.735+0000 7f4ff7b47700 1 -- 172.21.15.151:0/3203075869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0101790 msgr2=0x7f4ff01057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.735+0000 7f4ff7b47700 1 --2- 172.21.15.151:0/3203075869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0101790 0x7f4ff01057e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009910 tx=0x7f4fe0009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:48.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.735+0000 7f4ff7b47700 1 -- 172.21.15.151:0/3203075869 shutdown_connections 2024-04-16T03:00:48.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.735+0000 7f4ff7b47700 1 --2- 172.21.15.151:0/3203075869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0101790 0x7f4ff01057e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.735+0000 7f4ff7b47700 1 --2- 172.21.15.151:0/3203075869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ff0100de0 0x7f4ff01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.735+0000 7f4ff7b47700 1 -- 172.21.15.151:0/3203075869 >> 172.21.15.151:0/3203075869 conn(0x7f4ff00fc670 msgr2=0x7f4ff00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:48.738 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:48 smithi151 ceph-mon[112233]: pgmap v666: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:48.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.735+0000 7f4ff7b47700 1 -- 172.21.15.151:0/3203075869 shutdown_connections 2024-04-16T03:00:48.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.735+0000 7f4ff7b47700 1 -- 172.21.15.151:0/3203075869 wait complete. 2024-04-16T03:00:48.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff7b47700 1 Processor -- start 2024-04-16T03:00:48.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff7b47700 1 -- start start 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff7b47700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0100de0 0x7f4ff0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff7b47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ff0101790 0x7f4ff00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff7b47700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff0073b10 con 0x7f4ff0100de0 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff7b47700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff0073c50 con 0x7f4ff0101790 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff58e3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0100de0 0x7f4ff0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff58e3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0100de0 0x7f4ff0073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38530/0 (socket says 172.21.15.151:38530) 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff58e3700 1 -- 172.21.15.151:0/1453419542 learned_addr learned my addr 172.21.15.151:0/1453419542 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.736+0000 7f4ff50e2700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ff0101790 0x7f4ff00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.737+0000 7f4ff58e3700 1 -- 172.21.15.151:0/1453419542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ff0101790 msgr2=0x7f4ff00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.737+0000 7f4ff58e3700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ff0101790 0x7f4ff00735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.737+0000 7f4ff58e3700 1 -- 172.21.15.151:0/1453419542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe00095f0 con 0x7f4ff0100de0 2024-04-16T03:00:48.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.737+0000 7f4ff58e3700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0100de0 0x7f4ff0073090 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00e920 tx=0x7f4fec00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:48.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.737+0000 7f4fe6ffd700 1 -- 172.21.15.151:0/1453419542 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fec00c9d0 con 0x7f4ff0100de0 2024-04-16T03:00:48.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.737+0000 7f4fe6ffd700 1 -- 172.21.15.151:0/1453419542 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fec010430 con 0x7f4ff0100de0 2024-04-16T03:00:48.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.737+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff01a2960 con 0x7f4ff0100de0 2024-04-16T03:00:48.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.737+0000 7f4fe6ffd700 1 -- 172.21.15.151:0/1453419542 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fec018630 con 0x7f4ff0100de0 2024-04-16T03:00:48.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.737+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff01a2d10 con 0x7f4ff0100de0 2024-04-16T03:00:48.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.738+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff01091e0 con 0x7f4ff0100de0 2024-04-16T03:00:48.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.739+0000 7f4fe6ffd700 1 -- 172.21.15.151:0/1453419542 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4fec0105a0 con 0x7f4ff0100de0 2024-04-16T03:00:48.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.739+0000 7f4fe6ffd700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fdc06ecc0 0x7f4fdc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:48.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.740+0000 7f4fe6ffd700 1 -- 172.21.15.151:0/1453419542 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4fec014070 con 0x7f4ff0100de0 2024-04-16T03:00:48.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.740+0000 7f4ff50e2700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fdc06ecc0 0x7f4fdc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:48.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.740+0000 7f4ff50e2700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fdc06ecc0 0x7f4fdc071180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0005b40 tx=0x7f4fe0005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:48.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.744+0000 7f4fe6ffd700 1 -- 172.21.15.151:0/1453419542 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4fec053ff0 con 0x7f4ff0100de0 2024-04-16T03:00:48.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.949+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ff002ccc0 con 0x7f4fdc06ecc0 2024-04-16T03:00:48.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.959+0000 7f4fe6ffd700 1 -- 172.21.15.151:0/1453419542 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4ff002ccc0 con 0x7f4fdc06ecc0 2024-04-16T03:00:48.961 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:00:48.961 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (22m) 20s ago 33m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:00:48.961 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (33m) 20s ago 33m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (32m) 22s ago 32m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (19m) 20s ago 33m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (19m) 22s ago 32m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (22m) 20s ago 33m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (16m) 20s ago 28m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (15m) 20s ago 28m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (26s) 22s ago 28m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (15m) 22s ago 28m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (24m) 20s ago 35m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (23m) 22s ago 32m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (21m) 20s ago 35m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (20m) 22s ago 32m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (23m) 20s ago 33m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:00:48.962 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (23m) 22s ago 32m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:00:48.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (19m) 20s ago 31m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:00:48.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (18m) 20s ago 31m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:00:48.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (18m) 20s ago 30m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:00:48.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (17m) 22s ago 30m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:00:48.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (17m) 22s ago 30m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:00:48.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (16m) 22s ago 29m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:00:48.963 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (22m) 20s ago 33m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.961+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fdc06ecc0 msgr2=0x7f4fdc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.961+0000 7f4ff7b47700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fdc06ecc0 0x7f4fdc071180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0005b40 tx=0x7f4fe0005a90 comp rx=0 tx=0).stop 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.962+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0100de0 msgr2=0x7f4ff0073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.962+0000 7f4ff7b47700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0100de0 0x7f4ff0073090 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00e920 tx=0x7f4fec00ec30 comp rx=0 tx=0).stop 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.962+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 shutdown_connections 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.962+0000 7f4ff7b47700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fdc06ecc0 0x7f4fdc071180 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.962+0000 7f4ff7b47700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ff0100de0 0x7f4ff0073090 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.962+0000 7f4ff7b47700 1 --2- 172.21.15.151:0/1453419542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ff0101790 0x7f4ff00735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.962+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 >> 172.21.15.151:0/1453419542 conn(0x7f4ff00fc670 msgr2=0x7f4ff00fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.962+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 shutdown_connections 2024-04-16T03:00:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:48.962+0000 7f4ff7b47700 1 -- 172.21.15.151:0/1453419542 wait complete. 2024-04-16T03:00:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:48 smithi153 ceph-mon[87858]: pgmap v666: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:49.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.068+0000 7f366003d700 1 -- 172.21.15.151:0/3701019648 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 msgr2=0x7f36581011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.068+0000 7f366003d700 1 --2- 172.21.15.151:0/3701019648 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 0x7f36581011b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3654009910 tx=0x7f3654009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:49.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.068+0000 7f366003d700 1 -- 172.21.15.151:0/3701019648 shutdown_connections 2024-04-16T03:00:49.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.068+0000 7f366003d700 1 --2- 172.21.15.151:0/3701019648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3658101780 0x7f36581057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.068+0000 7f366003d700 1 --2- 172.21.15.151:0/3701019648 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 0x7f36581011b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.068+0000 7f366003d700 1 -- 172.21.15.151:0/3701019648 >> 172.21.15.151:0/3701019648 conn(0x7f36580fc640 msgr2=0x7f36580fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:49.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.068+0000 7f366003d700 1 -- 172.21.15.151:0/3701019648 shutdown_connections 2024-04-16T03:00:49.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.068+0000 7f366003d700 1 -- 172.21.15.151:0/3701019648 wait complete. 2024-04-16T03:00:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f366003d700 1 Processor -- start 2024-04-16T03:00:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f366003d700 1 -- start start 2024-04-16T03:00:49.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f366003d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 0x7f365819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f366003d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3658101780 0x7f365819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f366003d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365819d0e0 con 0x7f3658100dd0 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f366003d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3658196590 con 0x7f3658101780 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f365ddd9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 0x7f365819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f365ddd9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 0x7f365819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38550/0 (socket says 172.21.15.151:38550) 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f365ddd9700 1 -- 172.21.15.151:0/1095054744 learned_addr learned my addr 172.21.15.151:0/1095054744 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f365d5d8700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3658101780 0x7f365819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f365ddd9700 1 -- 172.21.15.151:0/1095054744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3658101780 msgr2=0x7f365819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f365ddd9700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3658101780 0x7f365819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f365ddd9700 1 -- 172.21.15.151:0/1095054744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36540095f0 con 0x7f3658100dd0 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f365d5d8700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3658101780 0x7f365819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:00:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.069+0000 7f365ddd9700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 0x7f365819c510 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3654006010 tx=0x7f365400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.070+0000 7f364effd700 1 -- 172.21.15.151:0/1095054744 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f365401d070 con 0x7f3658100dd0 2024-04-16T03:00:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.070+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3658196810 con 0x7f3658100dd0 2024-04-16T03:00:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.070+0000 7f364effd700 1 -- 172.21.15.151:0/1095054744 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3654004fb0 con 0x7f3658100dd0 2024-04-16T03:00:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.070+0000 7f364effd700 1 -- 172.21.15.151:0/1095054744 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3654022ea0 con 0x7f3658100dd0 2024-04-16T03:00:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.070+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3658196d00 con 0x7f3658100dd0 2024-04-16T03:00:49.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.071+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3658109170 con 0x7f3658100dd0 2024-04-16T03:00:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.071+0000 7f364effd700 1 -- 172.21.15.151:0/1095054744 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3654022470 con 0x7f3658100dd0 2024-04-16T03:00:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.072+0000 7f364effd700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f364406ecc0 0x7f3644071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.072+0000 7f365d5d8700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f364406ecc0 0x7f3644071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.072+0000 7f364effd700 1 -- 172.21.15.151:0/1095054744 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f36540903f0 con 0x7f3658100dd0 2024-04-16T03:00:49.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.072+0000 7f365d5d8700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f364406ecc0 0x7f3644071180 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f36580fce60 tx=0x7f3648005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:49.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.077+0000 7f364effd700 1 -- 172.21.15.151:0/1095054744 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f365405cd80 con 0x7f3658100dd0 2024-04-16T03:00:49.368 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.366+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f365804ea90 con 0x7f3658100dd0 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.367+0000 7f364effd700 1 -- 172.21.15.151:0/1095054744 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f3654022720 con 0x7f3658100dd0 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:00:49.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:00:49.370 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:00:49.370 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:00:49.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:00:49.370 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:00:49.370 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:00:49.370 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:00:49.370 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:00:49.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f364406ecc0 msgr2=0x7f3644071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f364406ecc0 0x7f3644071180 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f36580fce60 tx=0x7f3648005f50 comp rx=0 tx=0).stop 2024-04-16T03:00:49.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 msgr2=0x7f365819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 0x7f365819c510 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3654006010 tx=0x7f365400be20 comp rx=0 tx=0).stop 2024-04-16T03:00:49.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 shutdown_connections 2024-04-16T03:00:49.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f364406ecc0 0x7f3644071180 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3658100dd0 0x7f365819c510 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 --2- 172.21.15.151:0/1095054744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3658101780 0x7f365819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 >> 172.21.15.151:0/1095054744 conn(0x7f36580fc640 msgr2=0x7f36580fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:49.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 shutdown_connections 2024-04-16T03:00:49.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.370+0000 7f366003d700 1 -- 172.21.15.151:0/1095054744 wait complete. 2024-04-16T03:00:49.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.476+0000 7f8c7c247700 1 -- 172.21.15.151:0/3442046740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c740ffa30 msgr2=0x7f8c740ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.476+0000 7f8c7c247700 1 --2- 172.21.15.151:0/3442046740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c740ffa30 0x7f8c740ffeb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009910 tx=0x7f8c64009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:49.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.476+0000 7f8c7c247700 1 -- 172.21.15.151:0/3442046740 shutdown_connections 2024-04-16T03:00:49.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.476+0000 7f8c7c247700 1 --2- 172.21.15.151:0/3442046740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c740ffa30 0x7f8c740ffeb0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.476+0000 7f8c7c247700 1 --2- 172.21.15.151:0/3442046740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c74105a30 0x7f8c74105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.476+0000 7f8c7c247700 1 -- 172.21.15.151:0/3442046740 >> 172.21.15.151:0/3442046740 conn(0x7f8c740fb520 msgr2=0x7f8c740fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:49.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.476+0000 7f8c7c247700 1 -- 172.21.15.151:0/3442046740 shutdown_connections 2024-04-16T03:00:49.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.476+0000 7f8c7c247700 1 -- 172.21.15.151:0/3442046740 wait complete. 2024-04-16T03:00:49.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c7c247700 1 Processor -- start 2024-04-16T03:00:49.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c7c247700 1 -- start start 2024-04-16T03:00:49.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c7c247700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c740ffa30 0x7f8c74197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:49.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c7c247700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c74105a30 0x7f8c74197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c7c247700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c741981a0 con 0x7f8c74105a30 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c7c247700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c74198310 con 0x7f8c740ffa30 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c797e2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c74105a30 0x7f8c74197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c797e2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c74105a30 0x7f8c74197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38572/0 (socket says 172.21.15.151:38572) 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c797e2700 1 -- 172.21.15.151:0/3696351513 learned_addr learned my addr 172.21.15.151:0/3696351513 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c79fe3700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c740ffa30 0x7f8c74197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c797e2700 1 -- 172.21.15.151:0/3696351513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c740ffa30 msgr2=0x7f8c74197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c797e2700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c740ffa30 0x7f8c74197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c797e2700 1 -- 172.21.15.151:0/3696351513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c640095f0 con 0x7f8c74105a30 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c79fe3700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c740ffa30 0x7f8c74197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.477+0000 7f8c797e2700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c74105a30 0x7f8c74197ac0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8c64006010 tx=0x7f8c6400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.478+0000 7f8c6affd700 1 -- 172.21.15.151:0/3696351513 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c6401d070 con 0x7f8c74105a30 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.478+0000 7f8c6affd700 1 -- 172.21.15.151:0/3696351513 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c64004fb0 con 0x7f8c74105a30 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.478+0000 7f8c6affd700 1 -- 172.21.15.151:0/3696351513 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c64022ea0 con 0x7f8c74105a30 2024-04-16T03:00:49.481 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.478+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c741a05f0 con 0x7f8c74105a30 2024-04-16T03:00:49.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.478+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c741a0a90 con 0x7f8c74105a30 2024-04-16T03:00:49.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.479+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c74107fc0 con 0x7f8c74105a30 2024-04-16T03:00:49.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.479+0000 7f8c6affd700 1 -- 172.21.15.151:0/3696351513 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8c64022470 con 0x7f8c74105a30 2024-04-16T03:00:49.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.480+0000 7f8c6affd700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c6006ecc0 0x7f8c60071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:49.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.480+0000 7f8c79fe3700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c6006ecc0 0x7f8c60071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:49.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.480+0000 7f8c6affd700 1 -- 172.21.15.151:0/3696351513 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8c64090350 con 0x7f8c74105a30 2024-04-16T03:00:49.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.480+0000 7f8c79fe3700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c6006ecc0 0x7f8c60071180 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8c70005950 tx=0x7f8c70005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:49.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.485+0000 7f8c6affd700 1 -- 172.21.15.151:0/3696351513 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8c64059230 con 0x7f8c74105a30 2024-04-16T03:00:49.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.726+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8c741079a0 con 0x7f8c74105a30 2024-04-16T03:00:49.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.726+0000 7f8c6affd700 1 -- 172.21.15.151:0/3696351513 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f8c64022720 con 0x7f8c74105a30 2024-04-16T03:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:00:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:00:49.731 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:00:49.731 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:00:49.731 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:00:49.731 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:00:49.731 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:00:49.731 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:00:49.731 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:00:49.731 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:00:49.731 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.728+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c6006ecc0 msgr2=0x7f8c60071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.728+0000 7f8c7c247700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c6006ecc0 0x7f8c60071180 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8c70005950 tx=0x7f8c70005f50 comp rx=0 tx=0).stop 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.729+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c74105a30 msgr2=0x7f8c74197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.729+0000 7f8c7c247700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c74105a30 0x7f8c74197ac0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8c64006010 tx=0x7f8c6400be20 comp rx=0 tx=0).stop 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.729+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 shutdown_connections 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.729+0000 7f8c7c247700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8c6006ecc0 0x7f8c60071180 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.729+0000 7f8c7c247700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c740ffa30 0x7f8c74197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.729+0000 7f8c7c247700 1 --2- 172.21.15.151:0/3696351513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c74105a30 0x7f8c74197ac0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.729+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 >> 172.21.15.151:0/3696351513 conn(0x7f8c740fb520 msgr2=0x7f8c740fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.729+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 shutdown_connections 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.729+0000 7f8c7c247700 1 -- 172.21.15.151:0/3696351513 wait complete. 2024-04-16T03:00:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:00:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.835+0000 7f755c6c6700 1 -- 172.21.15.151:0/767535751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 msgr2=0x7f75540ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.835+0000 7f755c6c6700 1 --2- 172.21.15.151:0/767535751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 0x7f75540ffec0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7548009910 tx=0x7f7548009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.835+0000 7f755c6c6700 1 -- 172.21.15.151:0/767535751 shutdown_connections 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.835+0000 7f755c6c6700 1 --2- 172.21.15.151:0/767535751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 0x7f75540ffec0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.835+0000 7f755c6c6700 1 --2- 172.21.15.151:0/767535751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7554105a40 0x7f7554105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.835+0000 7f755c6c6700 1 -- 172.21.15.151:0/767535751 >> 172.21.15.151:0/767535751 conn(0x7f75540fb550 msgr2=0x7f75540fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.835+0000 7f755c6c6700 1 -- 172.21.15.151:0/767535751 shutdown_connections 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.835+0000 7f755c6c6700 1 -- 172.21.15.151:0/767535751 wait complete. 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.835+0000 7f755c6c6700 1 Processor -- start 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755c6c6700 1 -- start start 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755c6c6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 0x7f7554195640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755c6c6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7554105a40 0x7f7554195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755c6c6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7554196260 con 0x7f75540ffa40 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755c6c6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7554199ff0 con 0x7f7554105a40 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755a462700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 0x7f7554195640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755a462700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 0x7f7554195640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38584/0 (socket says 172.21.15.151:38584) 2024-04-16T03:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755a462700 1 -- 172.21.15.151:0/2172197777 learned_addr learned my addr 172.21.15.151:0/2172197777 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755a462700 1 -- 172.21.15.151:0/2172197777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7554105a40 msgr2=0x7f7554195b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f7559c61700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7554105a40 0x7f7554195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755a462700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7554105a40 0x7f7554195b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755a462700 1 -- 172.21.15.151:0/2172197777 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75480095f0 con 0x7f75540ffa40 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f755a462700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 0x7f7554195640 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f754c00cfb0 tx=0x7f754c00e330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.836+0000 7f75537fe700 1 -- 172.21.15.151:0/2172197777 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f754c011070 con 0x7f75540ffa40 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.837+0000 7f75537fe700 1 -- 172.21.15.151:0/2172197777 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f754c00c4b0 con 0x7f75540ffa40 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.837+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755419a270 con 0x7f75540ffa40 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.837+0000 7f75537fe700 1 -- 172.21.15.151:0/2172197777 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f754c015620 con 0x7f75540ffa40 2024-04-16T03:00:49.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.837+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755419a7c0 con 0x7f75540ffa40 2024-04-16T03:00:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.837+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755404ea90 con 0x7f75540ffa40 2024-04-16T03:00:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.838+0000 7f75537fe700 1 -- 172.21.15.151:0/2172197777 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f754c00a600 con 0x7f75540ffa40 2024-04-16T03:00:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.839+0000 7f75537fe700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f753c06ecc0 0x7f753c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.839+0000 7f75537fe700 1 -- 172.21.15.151:0/2172197777 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f754c090c20 con 0x7f75540ffa40 2024-04-16T03:00:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.839+0000 7f7559c61700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f753c06ecc0 0x7f753c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:49.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.839+0000 7f7559c61700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f753c06ecc0 0x7f753c071180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f754800b5c0 tx=0x7f7548005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:49.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:49.843+0000 7f75537fe700 1 -- 172.21.15.151:0/2172197777 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f754c0599d0 con 0x7f75540ffa40 2024-04-16T03:00:49.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:49 smithi151 ceph-mon[112233]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:49.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:49 smithi151 ceph-mon[112233]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:49.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:49 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1095054744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:49 smithi153 ceph-mon[87858]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:49 smithi153 ceph-mon[87858]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:50.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:49 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1095054744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:00:50.069 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.068+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f755419ad80 con 0x7f753c06ecc0 2024-04-16T03:00:50.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.069+0000 7f75537fe700 1 -- 172.21.15.151:0/2172197777 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f755419ad80 con 0x7f753c06ecc0 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:00:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:00:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:00:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:00:50.072 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:00:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f753c06ecc0 msgr2=0x7f753c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f753c06ecc0 0x7f753c071180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f754800b5c0 tx=0x7f7548005960 comp rx=0 tx=0).stop 2024-04-16T03:00:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 msgr2=0x7f7554195640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 0x7f7554195640 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f754c00cfb0 tx=0x7f754c00e330 comp rx=0 tx=0).stop 2024-04-16T03:00:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 shutdown_connections 2024-04-16T03:00:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f753c06ecc0 0x7f753c071180 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75540ffa40 0x7f7554195640 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 --2- 172.21.15.151:0/2172197777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7554105a40 0x7f7554195b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 >> 172.21.15.151:0/2172197777 conn(0x7f75540fb550 msgr2=0x7f75540fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:50.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 shutdown_connections 2024-04-16T03:00:50.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.071+0000 7f755c6c6700 1 -- 172.21.15.151:0/2172197777 wait complete. 2024-04-16T03:00:50.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 -- 172.21.15.151:0/2571166347 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048101d00 msgr2=0x7fa0481020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:50.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 --2- 172.21.15.151:0/2571166347 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048101d00 0x7fa0481020e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa044009910 tx=0x7fa044009c20 comp rx=0 tx=0).stop 2024-04-16T03:00:50.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 -- 172.21.15.151:0/2571166347 shutdown_connections 2024-04-16T03:00:50.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 --2- 172.21.15.151:0/2571166347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048102620 0x7fa04810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 --2- 172.21.15.151:0/2571166347 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048101d00 0x7fa0481020e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 -- 172.21.15.151:0/2571166347 >> 172.21.15.151:0/2571166347 conn(0x7fa0480fb550 msgr2=0x7fa0480fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 -- 172.21.15.151:0/2571166347 shutdown_connections 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 -- 172.21.15.151:0/2571166347 wait complete. 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 Processor -- start 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.178+0000 7fa04fce1700 1 -- start start 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04fce1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048101d00 0x7fa04819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04fce1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048102620 0x7fa04819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04fce1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04819d0e0 con 0x7fa048102620 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04fce1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa048196590 con 0x7fa048101d00 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04d27c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048102620 0x7fa04819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04d27c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048102620 0x7fa04819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38610/0 (socket says 172.21.15.151:38610) 2024-04-16T03:00:50.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04d27c700 1 -- 172.21.15.151:0/1043480311 learned_addr learned my addr 172.21.15.151:0/1043480311 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04da7d700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048101d00 0x7fa04819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04d27c700 1 -- 172.21.15.151:0/1043480311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048101d00 msgr2=0x7fa04819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04d27c700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048101d00 0x7fa04819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04d27c700 1 -- 172.21.15.151:0/1043480311 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0440095f0 con 0x7fa048102620 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04da7d700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048101d00 0x7fa04819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04d27c700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048102620 0x7fa04819ca50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa03800d8d0 tx=0x7fa03800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa03effd700 1 -- 172.21.15.151:0/1043480311 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0380041f0 con 0x7fa048102620 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa03effd700 1 -- 172.21.15.151:0/1043480311 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa038004350 con 0x7fa048102620 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa048196870 con 0x7fa048102620 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa03effd700 1 -- 172.21.15.151:0/1043480311 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa038003d60 con 0x7fa048102620 2024-04-16T03:00:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.179+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa048196dc0 con 0x7fa048102620 2024-04-16T03:00:50.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.180+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa048108210 con 0x7fa048102620 2024-04-16T03:00:50.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.181+0000 7fa03effd700 1 -- 172.21.15.151:0/1043480311 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa038009730 con 0x7fa048102620 2024-04-16T03:00:50.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.181+0000 7fa03effd700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:00:50.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.182+0000 7fa03effd700 1 -- 172.21.15.151:0/1043480311 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa03808f310 con 0x7fa048102620 2024-04-16T03:00:50.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.182+0000 7fa04da7d700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:00:50.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.182+0000 7fa04da7d700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa04400b5c0 tx=0x7fa0440053a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:00:50.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.186+0000 7fa03effd700 1 -- 172.21.15.151:0/1043480311 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa03805bc50 con 0x7fa048102620 2024-04-16T03:00:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.465+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa04804ea90 con 0x7fa048102620 2024-04-16T03:00:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.465+0000 7fa03effd700 1 -- 172.21.15.151:0/1043480311 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7fa03805b7e0 con 0x7fa048102620 2024-04-16T03:00:50.467 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:00:50.467 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:00:50.468 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:00:50.469 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:00:50.469 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:00:50.469 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:00:50.469 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:00:50.469 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:00:50.469 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:00:50.469 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.467+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 msgr2=0x7fa034071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.467+0000 7fa04fce1700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa04400b5c0 tx=0x7fa0440053a0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.467+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048102620 msgr2=0x7fa04819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.467+0000 7fa04fce1700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048102620 0x7fa04819ca50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa03800d8d0 tx=0x7fa03800dc90 comp rx=0 tx=0).stop 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.468+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 shutdown_connections 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.468+0000 7fa04fce1700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.468+0000 7fa04fce1700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048101d00 0x7fa04819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.468+0000 7fa04fce1700 1 --2- 172.21.15.151:0/1043480311 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048102620 0x7fa04819ca50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.468+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 >> 172.21.15.151:0/1043480311 conn(0x7fa0480fb550 msgr2=0x7fa048105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.468+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 shutdown_connections 2024-04-16T03:00:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:00:50.468+0000 7fa04fce1700 1 -- 172.21.15.151:0/1043480311 wait complete. 2024-04-16T03:00:50.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:50 smithi151 ceph-mon[112233]: from='client.35248 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:50.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:50 smithi151 ceph-mon[112233]: pgmap v667: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:50.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:50 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3696351513' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:00:50.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:50 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1043480311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:00:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:50 smithi153 ceph-mon[87858]: from='client.35248 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:50 smithi153 ceph-mon[87858]: pgmap v667: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:50 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3696351513' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:00:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:50 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1043480311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:00:51.767 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:51 smithi151 ceph-mon[112233]: from='client.35260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:51.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:51 smithi153 ceph-mon[87858]: from='client.35260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:00:52.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:52 smithi151 ceph-mon[112233]: pgmap v668: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:53.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:52 smithi153 ceph-mon[87858]: pgmap v668: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:54 smithi151 ceph-mon[112233]: pgmap v669: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:00:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:54 smithi153 ceph-mon[87858]: pgmap v669: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:00:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:56 smithi151 ceph-mon[112233]: pgmap v670: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:57.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:56 smithi153 ceph-mon[87858]: pgmap v670: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:00:58 smithi151 ceph-mon[112233]: pgmap v671: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:00:59.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:00:58 smithi153 ceph-mon[87858]: pgmap v671: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:00 smithi151 ceph-mon[112233]: pgmap v672: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:00 smithi153 ceph-mon[87858]: pgmap v672: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:02 smithi151 ceph-mon[112233]: pgmap v673: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:03.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:02 smithi153 ceph-mon[87858]: pgmap v673: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:04.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:04 smithi151 ceph-mon[112233]: pgmap v674: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:05.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:04 smithi153 ceph-mon[87858]: pgmap v674: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:06.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:06 smithi151 ceph-mon[112233]: pgmap v675: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:07.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:06 smithi153 ceph-mon[87858]: pgmap v675: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:08.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:08 smithi151 ceph-mon[112233]: pgmap v676: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:09.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:08 smithi153 ceph-mon[87858]: pgmap v676: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:09.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:01:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:01:10.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:10 smithi151 ceph-mon[112233]: pgmap v677: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:11.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:10 smithi153 ceph-mon[87858]: pgmap v677: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:12.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:12 smithi151 ceph-mon[112233]: pgmap v678: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:13.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:12 smithi153 ceph-mon[87858]: pgmap v678: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:14.939 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:14 smithi153 ceph-mon[87858]: pgmap v679: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:14.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:14 smithi151 ceph-mon[112233]: pgmap v679: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:16 smithi151 ceph-mon[112233]: pgmap v680: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:17.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:16 smithi153 ceph-mon[87858]: pgmap v680: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:18.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:18 smithi151 ceph-mon[112233]: pgmap v681: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:19.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:18 smithi153 ceph-mon[87858]: pgmap v681: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.589+0000 7f01704e8700 1 -- 172.21.15.151:0/1520905640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168102620 msgr2=0x7f016810ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.589+0000 7f01704e8700 1 --2- 172.21.15.151:0/1520905640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168102620 0x7f016810ab10 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0158009910 tx=0x7f0158009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.589+0000 7f01704e8700 1 -- 172.21.15.151:0/1520905640 shutdown_connections 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.589+0000 7f01704e8700 1 --2- 172.21.15.151:0/1520905640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168102620 0x7f016810ab10 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.589+0000 7f01704e8700 1 --2- 172.21.15.151:0/1520905640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0168101c70 0x7f0168102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.589+0000 7f01704e8700 1 -- 172.21.15.151:0/1520905640 >> 172.21.15.151:0/1520905640 conn(0x7f01680fb550 msgr2=0x7f01680fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.589+0000 7f01704e8700 1 -- 172.21.15.151:0/1520905640 shutdown_connections 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.589+0000 7f01704e8700 1 -- 172.21.15.151:0/1520905640 wait complete. 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f01704e8700 1 Processor -- start 2024-04-16T03:01:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f01704e8700 1 -- start start 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f01704e8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168101c70 0x7f016819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f01704e8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0168102620 0x7f016819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f01704e8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016819d130 con 0x7f0168101c70 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f01704e8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01681965e0 con 0x7f0168102620 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f016e284700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168101c70 0x7f016819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f016e284700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168101c70 0x7f016819c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45194/0 (socket says 172.21.15.151:45194) 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f016e284700 1 -- 172.21.15.151:0/3498750319 learned_addr learned my addr 172.21.15.151:0/3498750319 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.590+0000 7f016da83700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0168102620 0x7f016819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.591+0000 7f016e284700 1 -- 172.21.15.151:0/3498750319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0168102620 msgr2=0x7f016819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.591+0000 7f016e284700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0168102620 0x7f016819caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.591+0000 7f016e284700 1 -- 172.21.15.151:0/3498750319 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01580095f0 con 0x7f0168101c70 2024-04-16T03:01:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.591+0000 7f016e284700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168101c70 0x7f016819c560 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f016400d900 tx=0x7f016400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:20.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.591+0000 7f015f7fe700 1 -- 172.21.15.151:0/3498750319 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0164004a00 con 0x7f0168101c70 2024-04-16T03:01:20.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.591+0000 7f015f7fe700 1 -- 172.21.15.151:0/3498750319 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01640054e0 con 0x7f0168101c70 2024-04-16T03:01:20.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.591+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01681968c0 con 0x7f0168101c70 2024-04-16T03:01:20.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.591+0000 7f015f7fe700 1 -- 172.21.15.151:0/3498750319 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0164009d60 con 0x7f0168101c70 2024-04-16T03:01:20.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.591+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0168196e10 con 0x7f0168101c70 2024-04-16T03:01:20.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.592+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0168108210 con 0x7f0168101c70 2024-04-16T03:01:20.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.593+0000 7f015f7fe700 1 -- 172.21.15.151:0/3498750319 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f0164005020 con 0x7f0168101c70 2024-04-16T03:01:20.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.593+0000 7f015f7fe700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f015406ecc0 0x7f0154071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:20.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.593+0000 7f015f7fe700 1 -- 172.21.15.151:0/3498750319 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f016408efa0 con 0x7f0168101c70 2024-04-16T03:01:20.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.593+0000 7f016da83700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f015406ecc0 0x7f0154071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:20.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.594+0000 7f016da83700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f015406ecc0 0x7f0154071180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f01680fcc90 tx=0x7f015801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:20.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.598+0000 7f015f7fe700 1 -- 172.21.15.151:0/3498750319 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f016405b900 con 0x7f0168101c70 2024-04-16T03:01:20.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.816+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f01681970d0 con 0x7f015406ecc0 2024-04-16T03:01:20.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.818+0000 7f015f7fe700 1 -- 172.21.15.151:0/3498750319 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f01681970d0 con 0x7f015406ecc0 2024-04-16T03:01:20.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f015406ecc0 msgr2=0x7f0154071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f015406ecc0 0x7f0154071180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f01680fcc90 tx=0x7f015801a040 comp rx=0 tx=0).stop 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168101c70 msgr2=0x7f016819c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168101c70 0x7f016819c560 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f016400d900 tx=0x7f016400dcc0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 shutdown_connections 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f015406ecc0 0x7f0154071180 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0168101c70 0x7f016819c560 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 --2- 172.21.15.151:0/3498750319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0168102620 0x7f016819caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 >> 172.21.15.151:0/3498750319 conn(0x7f01680fb550 msgr2=0x7f0168105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 shutdown_connections 2024-04-16T03:01:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.820+0000 7f01704e8700 1 -- 172.21.15.151:0/3498750319 wait complete. 2024-04-16T03:01:20.832 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:01:20.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.925+0000 7fa8ba00b700 1 -- 172.21.15.151:0/4061790575 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 msgr2=0x7fa8b4102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:20.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.925+0000 7fa8ba00b700 1 --2- 172.21.15.151:0/4061790575 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 0x7fa8b4102110 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4009910 tx=0x7fa8a4009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:20.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.925+0000 7fa8ba00b700 1 -- 172.21.15.151:0/4061790575 shutdown_connections 2024-04-16T03:01:20.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.925+0000 7fa8ba00b700 1 --2- 172.21.15.151:0/4061790575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4102650 0x7fa8b410ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.925+0000 7fa8ba00b700 1 --2- 172.21.15.151:0/4061790575 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 0x7fa8b4102110 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.925+0000 7fa8ba00b700 1 -- 172.21.15.151:0/4061790575 >> 172.21.15.151:0/4061790575 conn(0x7fa8b40fb560 msgr2=0x7fa8b40fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:20.928 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:20 smithi151 ceph-mon[112233]: pgmap v682: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:20.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.925+0000 7fa8ba00b700 1 -- 172.21.15.151:0/4061790575 shutdown_connections 2024-04-16T03:01:20.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.926+0000 7fa8ba00b700 1 -- 172.21.15.151:0/4061790575 wait complete. 2024-04-16T03:01:20.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.926+0000 7fa8ba00b700 1 Processor -- start 2024-04-16T03:01:20.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.926+0000 7fa8ba00b700 1 -- start start 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8ba00b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 0x7fa8b40ff990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8ba00b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4102650 0x7fa8b40ffed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8ba00b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b4100410 con 0x7fa8b4101d30 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8ba00b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b4100550 con 0x7fa8b4102650 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 0x7fa8b40ff990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 0x7fa8b40ff990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45216/0 (socket says 172.21.15.151:45216) 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b37fe700 1 -- 172.21.15.151:0/3865396977 learned_addr learned my addr 172.21.15.151:0/3865396977 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b2ffd700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4102650 0x7fa8b40ffed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b37fe700 1 -- 172.21.15.151:0/3865396977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4102650 msgr2=0x7fa8b40ffed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b37fe700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4102650 0x7fa8b40ffed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b37fe700 1 -- 172.21.15.151:0/3865396977 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a40095f0 con 0x7fa8b4101d30 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b37fe700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 0x7fa8b40ff990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4006010 tx=0x7fa8a400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b0ff9700 1 -- 172.21.15.151:0/3865396977 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8a401d070 con 0x7fa8b4101d30 2024-04-16T03:01:20.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b0ff9700 1 -- 172.21.15.151:0/3865396977 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8a4005dc0 con 0x7fa8b4101d30 2024-04-16T03:01:20.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.927+0000 7fa8b0ff9700 1 -- 172.21.15.151:0/3865396977 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8a4017400 con 0x7fa8b4101d30 2024-04-16T03:01:20.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.928+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b41007d0 con 0x7fa8b4101d30 2024-04-16T03:01:20.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.928+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b41a2c80 con 0x7fa8b4101d30 2024-04-16T03:01:20.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.929+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b4108240 con 0x7fa8b4101d30 2024-04-16T03:01:20.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.929+0000 7fa8b0ff9700 1 -- 172.21.15.151:0/3865396977 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa8a4003a60 con 0x7fa8b4101d30 2024-04-16T03:01:20.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.930+0000 7fa8b0ff9700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa89c06ecc0 0x7fa89c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:20.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.930+0000 7fa8b0ff9700 1 -- 172.21.15.151:0/3865396977 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa8a40901b0 con 0x7fa8b4101d30 2024-04-16T03:01:20.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.930+0000 7fa8b2ffd700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa89c06ecc0 0x7fa89c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:20.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.930+0000 7fa8b2ffd700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa89c06ecc0 0x7fa89c071180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4101180 tx=0x7fa8a8007400 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:20.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:20.934+0000 7fa8b0ff9700 1 -- 172.21.15.151:0/3865396977 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa8a4058fe0 con 0x7fa8b4101d30 2024-04-16T03:01:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:20 smithi153 ceph-mon[87858]: pgmap v682: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:21.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.155+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8b40611d0 con 0x7fa89c06ecc0 2024-04-16T03:01:21.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.157+0000 7fa8b0ff9700 1 -- 172.21.15.151:0/3865396977 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa8b40611d0 con 0x7fa89c06ecc0 2024-04-16T03:01:21.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa89c06ecc0 msgr2=0x7fa89c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa89c06ecc0 0x7fa89c071180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4101180 tx=0x7fa8a8007400 comp rx=0 tx=0).stop 2024-04-16T03:01:21.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 msgr2=0x7fa8b40ff990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 0x7fa8b40ff990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4006010 tx=0x7fa8a400bd70 comp rx=0 tx=0).stop 2024-04-16T03:01:21.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 shutdown_connections 2024-04-16T03:01:21.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa89c06ecc0 0x7fa89c071180 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8b4101d30 0x7fa8b40ff990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 --2- 172.21.15.151:0/3865396977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4102650 0x7fa8b40ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 >> 172.21.15.151:0/3865396977 conn(0x7fa8b40fb560 msgr2=0x7fa8b4105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:21.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 shutdown_connections 2024-04-16T03:01:21.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.159+0000 7fa8ba00b700 1 -- 172.21.15.151:0/3865396977 wait complete. 2024-04-16T03:01:21.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.268+0000 7ff5779ae700 1 -- 172.21.15.151:0/893389054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 msgr2=0x7ff5700ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.268+0000 7ff5779ae700 1 --2- 172.21.15.151:0/893389054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 0x7ff5700ffec0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff560009910 tx=0x7ff560009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:21.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.268+0000 7ff5779ae700 1 -- 172.21.15.151:0/893389054 shutdown_connections 2024-04-16T03:01:21.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.268+0000 7ff5779ae700 1 --2- 172.21.15.151:0/893389054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 0x7ff5700ffec0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.268+0000 7ff5779ae700 1 --2- 172.21.15.151:0/893389054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff570105a40 0x7ff570105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.268+0000 7ff5779ae700 1 -- 172.21.15.151:0/893389054 >> 172.21.15.151:0/893389054 conn(0x7ff5700fb570 msgr2=0x7ff5700fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:21.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.268+0000 7ff5779ae700 1 -- 172.21.15.151:0/893389054 shutdown_connections 2024-04-16T03:01:21.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.268+0000 7ff5779ae700 1 -- 172.21.15.151:0/893389054 wait complete. 2024-04-16T03:01:21.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.269+0000 7ff5779ae700 1 Processor -- start 2024-04-16T03:01:21.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.269+0000 7ff5779ae700 1 -- start start 2024-04-16T03:01:21.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.269+0000 7ff5779ae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 0x7ff570195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:21.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.269+0000 7ff5779ae700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff570105a40 0x7ff570195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.269+0000 7ff5779ae700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff570196240 con 0x7ff5700ffa40 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.269+0000 7ff5779ae700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff570199fd0 con 0x7ff570105a40 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.269+0000 7ff57574a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 0x7ff570195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.269+0000 7ff57574a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 0x7ff570195620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45242/0 (socket says 172.21.15.151:45242) 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.269+0000 7ff57574a700 1 -- 172.21.15.151:0/3038069664 learned_addr learned my addr 172.21.15.151:0/3038069664 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff574f49700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff570105a40 0x7ff570195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff57574a700 1 -- 172.21.15.151:0/3038069664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff570105a40 msgr2=0x7ff570195b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff57574a700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff570105a40 0x7ff570195b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff57574a700 1 -- 172.21.15.151:0/3038069664 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5600095f0 con 0x7ff5700ffa40 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff57574a700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 0x7ff570195620 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00d8d0 tx=0x7ff56c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff5667fc700 1 -- 172.21.15.151:0/3038069664 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff56c009940 con 0x7ff5700ffa40 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57019a2b0 con 0x7ff5700ffa40 2024-04-16T03:01:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff5667fc700 1 -- 172.21.15.151:0/3038069664 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff56c010460 con 0x7ff5700ffa40 2024-04-16T03:01:21.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff5667fc700 1 -- 172.21.15.151:0/3038069664 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff56c00f5f0 con 0x7ff5700ffa40 2024-04-16T03:01:21.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.270+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57019a800 con 0x7ff5700ffa40 2024-04-16T03:01:21.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.271+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff570107fd0 con 0x7ff5700ffa40 2024-04-16T03:01:21.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.272+0000 7ff5667fc700 1 -- 172.21.15.151:0/3038069664 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff56c0105d0 con 0x7ff5700ffa40 2024-04-16T03:01:21.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.272+0000 7ff5667fc700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff55c06ecc0 0x7ff55c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:21.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.272+0000 7ff5667fc700 1 -- 172.21.15.151:0/3038069664 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff56c08ee80 con 0x7ff5700ffa40 2024-04-16T03:01:21.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.272+0000 7ff574f49700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff55c06ecc0 0x7ff55c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:21.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.273+0000 7ff574f49700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff55c06ecc0 0x7ff55c071180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff560005b40 tx=0x7ff56001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:21.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.277+0000 7ff5667fc700 1 -- 172.21.15.151:0/3038069664 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff56c057c30 con 0x7ff5700ffa40 2024-04-16T03:01:21.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.490+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff57019aae0 con 0x7ff55c06ecc0 2024-04-16T03:01:21.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.511+0000 7ff5667fc700 1 -- 172.21.15.151:0/3038069664 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff57019aae0 con 0x7ff55c06ecc0 2024-04-16T03:01:21.513 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:01:21.513 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (22m) 52s ago 34m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (34m) 52s ago 34m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (33m) 55s ago 33m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (20m) 52s ago 34m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (20m) 55s ago 33m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (22m) 52s ago 33m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (16m) 52s ago 28m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (16m) 52s ago 28m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (58s) 55s ago 28m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (16m) 55s ago 28m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:01:21.514 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (24m) 52s ago 35m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (23m) 55s ago 33m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (21m) 52s ago 35m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (21m) 55s ago 33m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (23m) 52s ago 34m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (23m) 55s ago 33m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (20m) 52s ago 32m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (19m) 52s ago 31m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (18m) 52s ago 31m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (18m) 55s ago 31m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (17m) 55s ago 30m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (17m) 55s ago 30m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:01:21.515 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (23m) 52s ago 33m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:01:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff55c06ecc0 msgr2=0x7ff55c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff55c06ecc0 0x7ff55c071180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff560005b40 tx=0x7ff56001a040 comp rx=0 tx=0).stop 2024-04-16T03:01:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 msgr2=0x7ff570195620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 0x7ff570195620 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00d8d0 tx=0x7ff56c00dc90 comp rx=0 tx=0).stop 2024-04-16T03:01:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 shutdown_connections 2024-04-16T03:01:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff55c06ecc0 0x7ff55c071180 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff5700ffa40 0x7ff570195620 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 --2- 172.21.15.151:0/3038069664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff570105a40 0x7ff570195b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 >> 172.21.15.151:0/3038069664 conn(0x7ff5700fb570 msgr2=0x7ff5700fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:21.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 shutdown_connections 2024-04-16T03:01:21.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.515+0000 7ff5779ae700 1 -- 172.21.15.151:0/3038069664 wait complete. 2024-04-16T03:01:21.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.623+0000 7f20e72a3700 1 -- 172.21.15.151:0/1844023484 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 msgr2=0x7f20e01057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.623+0000 7f20e72a3700 1 --2- 172.21.15.151:0/1844023484 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 0x7f20e01057a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009910 tx=0x7f20d0009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.623+0000 7f20e72a3700 1 -- 172.21.15.151:0/1844023484 shutdown_connections 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.623+0000 7f20e72a3700 1 --2- 172.21.15.151:0/1844023484 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 0x7f20e01057a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.623+0000 7f20e72a3700 1 --2- 172.21.15.151:0/1844023484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20e0100da0 0x7f20e0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.623+0000 7f20e72a3700 1 -- 172.21.15.151:0/1844023484 >> 172.21.15.151:0/1844023484 conn(0x7f20e00fc630 msgr2=0x7f20e00fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.623+0000 7f20e72a3700 1 -- 172.21.15.151:0/1844023484 shutdown_connections 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.623+0000 7f20e72a3700 1 -- 172.21.15.151:0/1844023484 wait complete. 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.624+0000 7f20e72a3700 1 Processor -- start 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.624+0000 7f20e72a3700 1 -- start start 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.624+0000 7f20e72a3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20e0100da0 0x7f20e0196730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.624+0000 7f20e72a3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 0x7f20e0196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:21.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.624+0000 7f20e72a3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0197300 con 0x7f20e0101750 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.624+0000 7f20e72a3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0199fc0 con 0x7f20e0100da0 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.624+0000 7f20e483e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 0x7f20e0196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20e483e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 0x7f20e0196c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45256/0 (socket says 172.21.15.151:45256) 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20e483e700 1 -- 172.21.15.151:0/540773009 learned_addr learned my addr 172.21.15.151:0/540773009 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20e503f700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20e0100da0 0x7f20e0196730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20e483e700 1 -- 172.21.15.151:0/540773009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20e0100da0 msgr2=0x7f20e0196730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20e483e700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20e0100da0 0x7f20e0196730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20e483e700 1 -- 172.21.15.151:0/540773009 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d00095f0 con 0x7f20e0101750 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20e483e700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 0x7f20e0196c70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f20d0005f50 tx=0x7f20d000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20d67fc700 1 -- 172.21.15.151:0/540773009 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d001d070 con 0x7f20e0101750 2024-04-16T03:01:21.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e019a240 con 0x7f20e0101750 2024-04-16T03:01:21.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20d67fc700 1 -- 172.21.15.151:0/540773009 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20d000f460 con 0x7f20e0101750 2024-04-16T03:01:21.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e019a730 con 0x7f20e0101750 2024-04-16T03:01:21.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.625+0000 7f20d67fc700 1 -- 172.21.15.151:0/540773009 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d0017630 con 0x7f20e0101750 2024-04-16T03:01:21.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.626+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e01091a0 con 0x7f20e0101750 2024-04-16T03:01:21.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.627+0000 7f20d67fc700 1 -- 172.21.15.151:0/540773009 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f20d000f5d0 con 0x7f20e0101750 2024-04-16T03:01:21.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.628+0000 7f20d67fc700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f20cc06ecc0 0x7f20cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:21.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.628+0000 7f20e503f700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f20cc06ecc0 0x7f20cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:21.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.628+0000 7f20d67fc700 1 -- 172.21.15.151:0/540773009 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f20d0090460 con 0x7f20e0101750 2024-04-16T03:01:21.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.629+0000 7f20e503f700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f20cc06ecc0 0x7f20cc071180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f20dc005fd0 tx=0x7f20dc00a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:21.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.633+0000 7f20d67fc700 1 -- 172.21.15.151:0/540773009 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f20d0059210 con 0x7f20e0101750 2024-04-16T03:01:21.925 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:21 smithi151 ceph-mon[112233]: from='client.35268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:21.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.923+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f20e004ea90 con 0x7f20e0101750 2024-04-16T03:01:21.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.924+0000 7f20d67fc700 1 -- 172.21.15.151:0/540773009 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f20d005c8d0 con 0x7f20e0101750 2024-04-16T03:01:21.926 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:01:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:01:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:01:21.927 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:01:21.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.926+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f20cc06ecc0 msgr2=0x7f20cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.926+0000 7f20e72a3700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f20cc06ecc0 0x7f20cc071180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f20dc005fd0 tx=0x7f20dc00a560 comp rx=0 tx=0).stop 2024-04-16T03:01:21.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.926+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 msgr2=0x7f20e0196c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:21.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.926+0000 7f20e72a3700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 0x7f20e0196c70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f20d0005f50 tx=0x7f20d000be20 comp rx=0 tx=0).stop 2024-04-16T03:01:21.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.926+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 shutdown_connections 2024-04-16T03:01:21.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.926+0000 7f20e72a3700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f20cc06ecc0 0x7f20cc071180 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.926+0000 7f20e72a3700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20e0100da0 0x7f20e0196730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.926+0000 7f20e72a3700 1 --2- 172.21.15.151:0/540773009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20e0101750 0x7f20e0196c70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:21.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.926+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 >> 172.21.15.151:0/540773009 conn(0x7f20e00fc630 msgr2=0x7f20e00fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:21.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.927+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 shutdown_connections 2024-04-16T03:01:21.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:21.927+0000 7f20e72a3700 1 -- 172.21.15.151:0/540773009 wait complete. 2024-04-16T03:01:22.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:21 smithi153 ceph-mon[87858]: from='client.35268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:22.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.034+0000 7f7cbcf55700 1 -- 172.21.15.151:0/1181136884 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 msgr2=0x7f7cb8068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.034+0000 7f7cbcf55700 1 --2- 172.21.15.151:0/1181136884 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 0x7f7cb8068a50 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009910 tx=0x7f7ca8009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:22.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.034+0000 7f7cbcf55700 1 -- 172.21.15.151:0/1181136884 shutdown_connections 2024-04-16T03:01:22.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.034+0000 7f7cbcf55700 1 --2- 172.21.15.151:0/1181136884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8069020 0x7f7cb8105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.034+0000 7f7cbcf55700 1 --2- 172.21.15.151:0/1181136884 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 0x7f7cb8068a50 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.034+0000 7f7cbcf55700 1 -- 172.21.15.151:0/1181136884 >> 172.21.15.151:0/1181136884 conn(0x7f7cb80755a0 msgr2=0x7f7cb80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:22.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.034+0000 7f7cbcf55700 1 -- 172.21.15.151:0/1181136884 shutdown_connections 2024-04-16T03:01:22.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.034+0000 7f7cbcf55700 1 -- 172.21.15.151:0/1181136884 wait complete. 2024-04-16T03:01:22.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cbcf55700 1 Processor -- start 2024-04-16T03:01:22.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cbcf55700 1 -- start start 2024-04-16T03:01:22.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cbcf55700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 0x7f7cb819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cbcf55700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8069020 0x7f7cb819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cbcf55700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb819d0e0 con 0x7f7cb8068670 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cbcf55700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8196590 con 0x7f7cb8069020 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cb659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 0x7f7cb819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cb659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 0x7f7cb819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45282/0 (socket says 172.21.15.151:45282) 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cb659c700 1 -- 172.21.15.151:0/76562660 learned_addr learned my addr 172.21.15.151:0/76562660 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cb5d9b700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8069020 0x7f7cb819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cb659c700 1 -- 172.21.15.151:0/76562660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8069020 msgr2=0x7f7cb819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cb659c700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8069020 0x7f7cb819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.035+0000 7f7cb659c700 1 -- 172.21.15.151:0/76562660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca80095f0 con 0x7f7cb8068670 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.036+0000 7f7cb659c700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 0x7f7cb819c510 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8005f50 tx=0x7f7ca800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.036+0000 7f7ca77fe700 1 -- 172.21.15.151:0/76562660 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ca801d070 con 0x7f7cb8068670 2024-04-16T03:01:22.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.036+0000 7f7ca77fe700 1 -- 172.21.15.151:0/76562660 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ca800b610 con 0x7f7cb8068670 2024-04-16T03:01:22.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.036+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb8196810 con 0x7f7cb8068670 2024-04-16T03:01:22.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.036+0000 7f7ca77fe700 1 -- 172.21.15.151:0/76562660 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ca8022470 con 0x7f7cb8068670 2024-04-16T03:01:22.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.036+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb8196d00 con 0x7f7cb8068670 2024-04-16T03:01:22.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.037+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb8109130 con 0x7f7cb8068670 2024-04-16T03:01:22.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.037+0000 7f7ca77fe700 1 -- 172.21.15.151:0/76562660 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7ca800faf0 con 0x7f7cb8068670 2024-04-16T03:01:22.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.038+0000 7f7ca77fe700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca006ecc0 0x7f7ca0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:22.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.038+0000 7f7ca77fe700 1 -- 172.21.15.151:0/76562660 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7ca8090170 con 0x7f7cb8068670 2024-04-16T03:01:22.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.038+0000 7f7cb5d9b700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca006ecc0 0x7f7ca0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:22.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.038+0000 7f7cb5d9b700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca006ecc0 0x7f7ca0071180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8107750 tx=0x7f7cac009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:22.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.043+0000 7f7ca77fe700 1 -- 172.21.15.151:0/76562660 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7ca805cad0 con 0x7f7cb8068670 2024-04-16T03:01:22.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.293+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7cb804ea90 con 0x7f7cb8068670 2024-04-16T03:01:22.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.294+0000 7f7ca77fe700 1 -- 172.21.15.151:0/76562660 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f7ca805c660 con 0x7f7cb8068670 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:01:22.296 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:01:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:01:22.298 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:01:22.299 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:01:22.299 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:01:22.299 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca006ecc0 msgr2=0x7f7ca0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca006ecc0 0x7f7ca0071180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8107750 tx=0x7f7cac009260 comp rx=0 tx=0).stop 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 msgr2=0x7f7cb819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 0x7f7cb819c510 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8005f50 tx=0x7f7ca800f690 comp rx=0 tx=0).stop 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 shutdown_connections 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca006ecc0 0x7f7ca0071180 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8068670 0x7f7cb819c510 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 --2- 172.21.15.151:0/76562660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8069020 0x7f7cb819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 >> 172.21.15.151:0/76562660 conn(0x7f7cb80755a0 msgr2=0x7f7cb80fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 shutdown_connections 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.296+0000 7f7cbcf55700 1 -- 172.21.15.151:0/76562660 wait complete. 2024-04-16T03:01:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:01:22.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.406+0000 7fa91b127700 1 -- 172.21.15.151:0/3651805060 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 msgr2=0x7fa9141057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.406+0000 7fa91b127700 1 --2- 172.21.15.151:0/3651805060 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 0x7fa9141057f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa90400b0a0 tx=0x7fa90400b3b0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.406+0000 7fa91b127700 1 -- 172.21.15.151:0/3651805060 shutdown_connections 2024-04-16T03:01:22.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.406+0000 7fa91b127700 1 --2- 172.21.15.151:0/3651805060 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 0x7fa9141057f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.406+0000 7fa91b127700 1 --2- 172.21.15.151:0/3651805060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914100df0 0x7fa9141011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.406+0000 7fa91b127700 1 -- 172.21.15.151:0/3651805060 >> 172.21.15.151:0/3651805060 conn(0x7fa9140fc6a0 msgr2=0x7fa9140feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:22.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.406+0000 7fa91b127700 1 -- 172.21.15.151:0/3651805060 shutdown_connections 2024-04-16T03:01:22.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.406+0000 7fa91b127700 1 -- 172.21.15.151:0/3651805060 wait complete. 2024-04-16T03:01:22.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa91b127700 1 Processor -- start 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa91b127700 1 -- start start 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa91b127700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914100df0 0x7fa91419c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa91b127700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 0x7fa91419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa91b127700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91419d140 con 0x7fa9141017a0 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa91b127700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9141965f0 con 0x7fa914100df0 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa913fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 0x7fa91419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa913fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 0x7fa91419cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45298/0 (socket says 172.21.15.151:45298) 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa913fff700 1 -- 172.21.15.151:0/1477332776 learned_addr learned my addr 172.21.15.151:0/1477332776 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa918ec3700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914100df0 0x7fa91419c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa913fff700 1 -- 172.21.15.151:0/1477332776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914100df0 msgr2=0x7fa91419c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa913fff700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914100df0 0x7fa91419c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa913fff700 1 -- 172.21.15.151:0/1477332776 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa904009d00 con 0x7fa9141017a0 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.407+0000 7fa913fff700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 0x7fa91419cab0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa904006130 tx=0x7fa90400cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.408+0000 7fa911ffb700 1 -- 172.21.15.151:0/1477332776 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa90400e040 con 0x7fa9141017a0 2024-04-16T03:01:22.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.408+0000 7fa911ffb700 1 -- 172.21.15.151:0/1477332776 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa904003de0 con 0x7fa9141017a0 2024-04-16T03:01:22.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.408+0000 7fa911ffb700 1 -- 172.21.15.151:0/1477332776 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa90401d8e0 con 0x7fa9141017a0 2024-04-16T03:01:22.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.408+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa914196870 con 0x7fa9141017a0 2024-04-16T03:01:22.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.408+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa914196d60 con 0x7fa9141017a0 2024-04-16T03:01:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.409+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9141091f0 con 0x7fa9141017a0 2024-04-16T03:01:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.409+0000 7fa911ffb700 1 -- 172.21.15.151:0/1477332776 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa904004820 con 0x7fa9141017a0 2024-04-16T03:01:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.410+0000 7fa911ffb700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.410+0000 7fa911ffb700 1 -- 172.21.15.151:0/1477332776 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa904090830 con 0x7fa9141017a0 2024-04-16T03:01:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.410+0000 7fa918ec3700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.410+0000 7fa918ec3700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa90c005950 tx=0x7fa90c005cf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:22.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.415+0000 7fa911ffb700 1 -- 172.21.15.151:0/1477332776 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa90405d070 con 0x7fa9141017a0 2024-04-16T03:01:22.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.635+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa914197040 con 0x7fa90006ecc0 2024-04-16T03:01:22.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.636+0000 7fa911ffb700 1 -- 172.21.15.151:0/1477332776 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa914197040 con 0x7fa90006ecc0 2024-04-16T03:01:22.638 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:01:22.638 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:01:22.638 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:01:22.638 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:01:22.638 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:01:22.638 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:01:22.638 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:01:22.639 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:01:22.639 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:01:22.639 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:01:22.639 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:01:22.639 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:01:22.639 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:01:22.639 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:01:22.639 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:01:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.638+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 msgr2=0x7fa900071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.638+0000 7fa91b127700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa90c005950 tx=0x7fa90c005cf0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.638+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 msgr2=0x7fa91419cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.638+0000 7fa91b127700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 0x7fa91419cab0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa904006130 tx=0x7fa90400cdd0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.639+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 shutdown_connections 2024-04-16T03:01:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.639+0000 7fa91b127700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.639+0000 7fa91b127700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914100df0 0x7fa91419c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.639+0000 7fa91b127700 1 --2- 172.21.15.151:0/1477332776 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9141017a0 0x7fa91419cab0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.639+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 >> 172.21.15.151:0/1477332776 conn(0x7fa9140fc6a0 msgr2=0x7fa9140fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:22.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.639+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 shutdown_connections 2024-04-16T03:01:22.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.639+0000 7fa91b127700 1 -- 172.21.15.151:0/1477332776 wait complete. 2024-04-16T03:01:22.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.744+0000 7fab086e1700 1 -- 172.21.15.151:0/997690763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 msgr2=0x7fab000ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.744+0000 7fab086e1700 1 --2- 172.21.15.151:0/997690763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 0x7fab000ffec0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7faafc009910 tx=0x7faafc009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:22.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.744+0000 7fab086e1700 1 -- 172.21.15.151:0/997690763 shutdown_connections 2024-04-16T03:01:22.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.744+0000 7fab086e1700 1 --2- 172.21.15.151:0/997690763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 0x7fab000ffec0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.744+0000 7fab086e1700 1 --2- 172.21.15.151:0/997690763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab00105a40 0x7fab00105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.744+0000 7fab086e1700 1 -- 172.21.15.151:0/997690763 >> 172.21.15.151:0/997690763 conn(0x7fab000fb570 msgr2=0x7fab000fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.744+0000 7fab086e1700 1 -- 172.21.15.151:0/997690763 shutdown_connections 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.744+0000 7fab086e1700 1 -- 172.21.15.151:0/997690763 wait complete. 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab086e1700 1 Processor -- start 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab086e1700 1 -- start start 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab086e1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 0x7fab00197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab086e1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab00105a40 0x7fab00197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab086e1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab00198150 con 0x7fab000ffa40 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab086e1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab00198290 con 0x7fab00105a40 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab0647d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 0x7fab00197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab0647d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 0x7fab00197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45314/0 (socket says 172.21.15.151:45314) 2024-04-16T03:01:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab0647d700 1 -- 172.21.15.151:0/1919832361 learned_addr learned my addr 172.21.15.151:0/1919832361 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.745+0000 7fab05c7c700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab00105a40 0x7fab00197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.746+0000 7fab0647d700 1 -- 172.21.15.151:0/1919832361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab00105a40 msgr2=0x7fab00197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.746+0000 7fab0647d700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab00105a40 0x7fab00197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.746+0000 7fab0647d700 1 -- 172.21.15.151:0/1919832361 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc0095f0 con 0x7fab000ffa40 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.746+0000 7fab0647d700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 0x7fab00197530 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faaf000c8a0 tx=0x7faaf000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.746+0000 7faaf77fe700 1 -- 172.21.15.151:0/1919832361 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaf00078e0 con 0x7fab000ffa40 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.746+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab001a05a0 con 0x7fab000ffa40 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.746+0000 7faaf77fe700 1 -- 172.21.15.151:0/1919832361 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faaf000f450 con 0x7fab000ffa40 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.746+0000 7faaf77fe700 1 -- 172.21.15.151:0/1919832361 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaf000e5e0 con 0x7fab000ffa40 2024-04-16T03:01:22.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.746+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab001a0ac0 con 0x7fab000ffa40 2024-04-16T03:01:22.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.747+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab00107fd0 con 0x7fab000ffa40 2024-04-16T03:01:22.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.748+0000 7faaf77fe700 1 -- 172.21.15.151:0/1919832361 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7faaf000f5c0 con 0x7fab000ffa40 2024-04-16T03:01:22.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.748+0000 7faaf77fe700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faaec06ecc0 0x7faaec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:22.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.748+0000 7faaf77fe700 1 -- 172.21.15.151:0/1919832361 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faaf008ee80 con 0x7fab000ffa40 2024-04-16T03:01:22.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.748+0000 7fab05c7c700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faaec06ecc0 0x7faaec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:22.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.749+0000 7fab05c7c700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faaec06ecc0 0x7faaec071180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fab00069d80 tx=0x7faafc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:22.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:22.753+0000 7faaf77fe700 1 -- 172.21.15.151:0/1919832361 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7faaf0057c30 con 0x7fab000ffa40 2024-04-16T03:01:22.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:22 smithi151 ceph-mon[112233]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:22.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:22 smithi151 ceph-mon[112233]: pgmap v683: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:22.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:22 smithi151 ceph-mon[112233]: from='client.35276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:22.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:22 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/540773009' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:01:22.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:22 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/76562660' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:01:23.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:22 smithi153 ceph-mon[87858]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:23.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:22 smithi153 ceph-mon[87858]: pgmap v683: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:23.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:22 smithi153 ceph-mon[87858]: from='client.35276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:23.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:22 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/540773009' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:01:23.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:22 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/76562660' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:01:23.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.033+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fab0004ea90 con 0x7fab000ffa40 2024-04-16T03:01:23.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.033+0000 7faaf77fe700 1 -- 172.21.15.151:0/1919832361 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7faaf005b2f0 con 0x7fab000ffa40 2024-04-16T03:01:23.035 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:01:23.035 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:01:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:01:23.035 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:01:23.035 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:01:23.035 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:01:23.036 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:01:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:01:23.037 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:01:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:01:23.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faaec06ecc0 msgr2=0x7faaec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faaec06ecc0 0x7faaec071180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fab00069d80 tx=0x7faafc01a040 comp rx=0 tx=0).stop 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 msgr2=0x7fab00197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 0x7fab00197530 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faaf000c8a0 tx=0x7faaf000cbb0 comp rx=0 tx=0).stop 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 shutdown_connections 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faaec06ecc0 0x7faaec071180 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab000ffa40 0x7fab00197530 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 --2- 172.21.15.151:0/1919832361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab00105a40 0x7fab00197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 >> 172.21.15.151:0/1919832361 conn(0x7fab000fb570 msgr2=0x7fab000fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.035+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 shutdown_connections 2024-04-16T03:01:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:23.036+0000 7fab086e1700 1 -- 172.21.15.151:0/1919832361 wait complete. 2024-04-16T03:01:23.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:23 smithi151 ceph-mon[112233]: from='client.35288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:23.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:23 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1919832361' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:01:24.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:23 smithi153 ceph-mon[87858]: from='client.35288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:24.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:23 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1919832361' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:01:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:24 smithi151 ceph-mon[112233]: pgmap v684: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:01:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:24 smithi153 ceph-mon[87858]: pgmap v684: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:01:26.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:26 smithi151 ceph-mon[112233]: pgmap v685: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:27.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:26 smithi153 ceph-mon[87858]: pgmap v685: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:28 smithi151 ceph-mon[112233]: pgmap v686: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:29.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:28 smithi153 ceph-mon[87858]: pgmap v686: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:30.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:30 smithi151 ceph-mon[112233]: pgmap v687: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:31.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:30 smithi153 ceph-mon[87858]: pgmap v687: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:32 smithi151 ceph-mon[112233]: pgmap v688: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:33.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:32 smithi153 ceph-mon[87858]: pgmap v688: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:34.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:34 smithi151 ceph-mon[112233]: pgmap v689: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:35.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:34 smithi153 ceph-mon[87858]: pgmap v689: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:36 smithi151 ceph-mon[112233]: pgmap v690: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:36 smithi153 ceph-mon[87858]: pgmap v690: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:38 smithi151 ceph-mon[112233]: pgmap v691: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:39.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:38 smithi153 ceph-mon[87858]: pgmap v691: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:39.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:01:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:01:40.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:40 smithi151 ceph-mon[112233]: pgmap v692: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:41.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:40 smithi153 ceph-mon[87858]: pgmap v692: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:42.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:42 smithi151 ceph-mon[112233]: pgmap v693: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:43.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:42 smithi153 ceph-mon[87858]: pgmap v693: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:44.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:44 smithi151 ceph-mon[112233]: pgmap v694: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:44 smithi153 ceph-mon[87858]: pgmap v694: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:46.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:46 smithi151 ceph-mon[112233]: pgmap v695: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:47.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:46 smithi153 ceph-mon[87858]: pgmap v695: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:48.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:48 smithi151 ceph-mon[112233]: pgmap v696: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:48 smithi153 ceph-mon[87858]: pgmap v696: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:50.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:50 smithi151 ceph-mon[112233]: pgmap v697: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:50 smithi153 ceph-mon[87858]: pgmap v697: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:52.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:52 smithi151 ceph-mon[112233]: pgmap v698: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:53.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:52 smithi153 ceph-mon[87858]: pgmap v698: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:53.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.154+0000 7fb23398a700 1 -- 172.21.15.151:0/2960636759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073130 msgr2=0x7fb22c073510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.154+0000 7fb23398a700 1 --2- 172.21.15.151:0/2960636759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073130 0x7fb22c073510 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb228009910 tx=0x7fb228009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.154+0000 7fb23398a700 1 -- 172.21.15.151:0/2960636759 shutdown_connections 2024-04-16T03:01:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.154+0000 7fb23398a700 1 --2- 172.21.15.151:0/2960636759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c073a50 0x7fb22c111630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.154+0000 7fb23398a700 1 --2- 172.21.15.151:0/2960636759 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073130 0x7fb22c073510 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.154+0000 7fb23398a700 1 -- 172.21.15.151:0/2960636759 >> 172.21.15.151:0/2960636759 conn(0x7fb22c0fc650 msgr2=0x7fb22c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.154+0000 7fb23398a700 1 -- 172.21.15.151:0/2960636759 shutdown_connections 2024-04-16T03:01:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.154+0000 7fb23398a700 1 -- 172.21.15.151:0/2960636759 wait complete. 2024-04-16T03:01:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb23398a700 1 Processor -- start 2024-04-16T03:01:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb23398a700 1 -- start start 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb23398a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c073130 0x7fb22c1a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb23398a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073a50 0x7fb22c1a3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb23398a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c1a3610 con 0x7fb22c073a50 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb23398a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c19cba0 con 0x7fb22c073130 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb230f25700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073a50 0x7fb22c1a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb230f25700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073a50 0x7fb22c1a3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44354/0 (socket says 172.21.15.151:44354) 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb230f25700 1 -- 172.21.15.151:0/4074671313 learned_addr learned my addr 172.21.15.151:0/4074671313 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb231726700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c073130 0x7fb22c1a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb230f25700 1 -- 172.21.15.151:0/4074671313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c073130 msgr2=0x7fb22c1a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.155+0000 7fb230f25700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c073130 0x7fb22c1a2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.156+0000 7fb230f25700 1 -- 172.21.15.151:0/4074671313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2280095f0 con 0x7fb22c073a50 2024-04-16T03:01:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.156+0000 7fb230f25700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073a50 0x7fb22c1a3010 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00c930 tx=0x7fb21c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.156+0000 7fb2227fc700 1 -- 172.21.15.151:0/4074671313 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb21c007ad0 con 0x7fb22c073a50 2024-04-16T03:01:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.156+0000 7fb2227fc700 1 -- 172.21.15.151:0/4074671313 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb21c00ce80 con 0x7fb22c073a50 2024-04-16T03:01:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.156+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c19ce80 con 0x7fb22c073a50 2024-04-16T03:01:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.156+0000 7fb2227fc700 1 -- 172.21.15.151:0/4074671313 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb21c018630 con 0x7fb22c073a50 2024-04-16T03:01:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.156+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c19d3d0 con 0x7fb22c073a50 2024-04-16T03:01:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.157+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22c10edb0 con 0x7fb22c073a50 2024-04-16T03:01:53.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.158+0000 7fb2227fc700 1 -- 172.21.15.151:0/4074671313 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb21c007c30 con 0x7fb22c073a50 2024-04-16T03:01:53.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.158+0000 7fb2227fc700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb21806ecc0 0x7fb218071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:53.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.158+0000 7fb2227fc700 1 -- 172.21.15.151:0/4074671313 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb21c08eda0 con 0x7fb22c073a50 2024-04-16T03:01:53.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.158+0000 7fb231726700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb21806ecc0 0x7fb218071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:53.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.159+0000 7fb231726700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb21806ecc0 0x7fb218071180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb228006010 tx=0x7fb22801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:53.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.163+0000 7fb2227fc700 1 -- 172.21.15.151:0/4074671313 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb21c05b700 con 0x7fb22c073a50 2024-04-16T03:01:53.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.384+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb22c19d710 con 0x7fb21806ecc0 2024-04-16T03:01:53.387 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.386+0000 7fb2227fc700 1 -- 172.21.15.151:0/4074671313 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb22c19d710 con 0x7fb21806ecc0 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb21806ecc0 msgr2=0x7fb218071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb21806ecc0 0x7fb218071180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb228006010 tx=0x7fb22801a040 comp rx=0 tx=0).stop 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073a50 msgr2=0x7fb22c1a3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073a50 0x7fb22c1a3010 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00c930 tx=0x7fb21c00cc40 comp rx=0 tx=0).stop 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 shutdown_connections 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb21806ecc0 0x7fb218071180 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c073130 0x7fb22c1a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 --2- 172.21.15.151:0/4074671313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb22c073a50 0x7fb22c1a3010 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 >> 172.21.15.151:0/4074671313 conn(0x7fb22c0fc650 msgr2=0x7fb22c103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:53.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 shutdown_connections 2024-04-16T03:01:53.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.388+0000 7fb23398a700 1 -- 172.21.15.151:0/4074671313 wait complete. 2024-04-16T03:01:53.400 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.493+0000 7f22e82dd700 1 -- 172.21.15.151:0/797038732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 msgr2=0x7f22e0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.493+0000 7f22e82dd700 1 --2- 172.21.15.151:0/797038732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 0x7f22e0100050 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009910 tx=0x7f22dc009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.493+0000 7f22e82dd700 1 -- 172.21.15.151:0/797038732 shutdown_connections 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.493+0000 7f22e82dd700 1 --2- 172.21.15.151:0/797038732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 0x7f22e0100050 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.493+0000 7f22e82dd700 1 --2- 172.21.15.151:0/797038732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e0105a40 0x7f22e0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.493+0000 7f22e82dd700 1 -- 172.21.15.151:0/797038732 >> 172.21.15.151:0/797038732 conn(0x7f22e00755a0 msgr2=0x7f22e00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.493+0000 7f22e82dd700 1 -- 172.21.15.151:0/797038732 shutdown_connections 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.493+0000 7f22e82dd700 1 -- 172.21.15.151:0/797038732 wait complete. 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e82dd700 1 Processor -- start 2024-04-16T03:01:53.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e82dd700 1 -- start start 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e82dd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 0x7f22e0101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e82dd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e0105a40 0x7f22e0101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e82dd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0102120 con 0x7f22e00ffbd0 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e82dd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0102290 con 0x7f22e0105a40 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e6079700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 0x7f22e0101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e6079700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 0x7f22e0101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44368/0 (socket says 172.21.15.151:44368) 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e6079700 1 -- 172.21.15.151:0/202578245 learned_addr learned my addr 172.21.15.151:0/202578245 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.494+0000 7f22e5878700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e0105a40 0x7f22e0101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.495+0000 7f22e6079700 1 -- 172.21.15.151:0/202578245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e0105a40 msgr2=0x7f22e0101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.495+0000 7f22e6079700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e0105a40 0x7f22e0101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.495+0000 7f22e6079700 1 -- 172.21.15.151:0/202578245 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc0095f0 con 0x7f22e00ffbd0 2024-04-16T03:01:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.495+0000 7f22e6079700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 0x7f22e0101610 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f22d400e920 tx=0x7f22d400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.495+0000 7f22d37fe700 1 -- 172.21.15.151:0/202578245 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22d400c9d0 con 0x7f22e00ffbd0 2024-04-16T03:01:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.495+0000 7f22d37fe700 1 -- 172.21.15.151:0/202578245 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22d4010430 con 0x7f22e00ffbd0 2024-04-16T03:01:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.495+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e01a28f0 con 0x7f22e00ffbd0 2024-04-16T03:01:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.495+0000 7f22d37fe700 1 -- 172.21.15.151:0/202578245 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22d4018630 con 0x7f22e00ffbd0 2024-04-16T03:01:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.495+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e01a2d00 con 0x7f22e00ffbd0 2024-04-16T03:01:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.496+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e0102450 con 0x7f22e00ffbd0 2024-04-16T03:01:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.496+0000 7f22d37fe700 1 -- 172.21.15.151:0/202578245 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f22d40105a0 con 0x7f22e00ffbd0 2024-04-16T03:01:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.497+0000 7f22d37fe700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f22cc06ecc0 0x7f22cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.497+0000 7f22d37fe700 1 -- 172.21.15.151:0/202578245 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f22d4014070 con 0x7f22e00ffbd0 2024-04-16T03:01:53.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.497+0000 7f22e5878700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f22cc06ecc0 0x7f22cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:53.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.497+0000 7f22e5878700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f22cc06ecc0 0x7f22cc071180 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f22e0103180 tx=0x7f22dc005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:53.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.502+0000 7f22d37fe700 1 -- 172.21.15.151:0/202578245 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f22d4053ff0 con 0x7f22e00ffbd0 2024-04-16T03:01:53.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.722+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f22e00fda40 con 0x7f22cc06ecc0 2024-04-16T03:01:53.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.723+0000 7f22d37fe700 1 -- 172.21.15.151:0/202578245 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f22e00fda40 con 0x7f22cc06ecc0 2024-04-16T03:01:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.725+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f22cc06ecc0 msgr2=0x7f22cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.725+0000 7f22e82dd700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f22cc06ecc0 0x7f22cc071180 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f22e0103180 tx=0x7f22dc005940 comp rx=0 tx=0).stop 2024-04-16T03:01:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.725+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 msgr2=0x7f22e0101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.725+0000 7f22e82dd700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 0x7f22e0101610 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f22d400e920 tx=0x7f22d400ec30 comp rx=0 tx=0).stop 2024-04-16T03:01:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.726+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 shutdown_connections 2024-04-16T03:01:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.726+0000 7f22e82dd700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f22cc06ecc0 0x7f22cc071180 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.726+0000 7f22e82dd700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22e00ffbd0 0x7f22e0101610 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.726+0000 7f22e82dd700 1 --2- 172.21.15.151:0/202578245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e0105a40 0x7f22e0101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.726+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 >> 172.21.15.151:0/202578245 conn(0x7f22e00755a0 msgr2=0x7f22e0103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:53.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.726+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 shutdown_connections 2024-04-16T03:01:53.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.726+0000 7f22e82dd700 1 -- 172.21.15.151:0/202578245 wait complete. 2024-04-16T03:01:53.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.833+0000 7f2af8c9c700 1 -- 172.21.15.151:0/3715365186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 msgr2=0x7f2af4105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.833+0000 7f2af8c9c700 1 --2- 172.21.15.151:0/3715365186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 0x7f2af4105870 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8009910 tx=0x7f2ae8009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:53.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.833+0000 7f2af8c9c700 1 -- 172.21.15.151:0/3715365186 shutdown_connections 2024-04-16T03:01:53.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.833+0000 7f2af8c9c700 1 --2- 172.21.15.151:0/3715365186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 0x7f2af4105870 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.833+0000 7f2af8c9c700 1 --2- 172.21.15.151:0/3715365186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af40ff530 0x7f2af40ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.833+0000 7f2af8c9c700 1 -- 172.21.15.151:0/3715365186 >> 172.21.15.151:0/3715365186 conn(0x7f2af4074df0 msgr2=0x7f2af4075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:53.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.833+0000 7f2af8c9c700 1 -- 172.21.15.151:0/3715365186 shutdown_connections 2024-04-16T03:01:53.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.833+0000 7f2af8c9c700 1 -- 172.21.15.151:0/3715365186 wait complete. 2024-04-16T03:01:53.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.834+0000 7f2af8c9c700 1 Processor -- start 2024-04-16T03:01:53.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.834+0000 7f2af8c9c700 1 -- start start 2024-04-16T03:01:53.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.834+0000 7f2af8c9c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af40ff530 0x7f2af419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:53.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.834+0000 7f2af8c9c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 0x7f2af419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:53.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.834+0000 7f2af8c9c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af419d090 con 0x7f2af40ffee0 2024-04-16T03:01:53.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.834+0000 7f2af8c9c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af4196620 con 0x7f2af40ff530 2024-04-16T03:01:53.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.834+0000 7f2af1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 0x7f2af419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:53.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.834+0000 7f2af1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 0x7f2af419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44390/0 (socket says 172.21.15.151:44390) 2024-04-16T03:01:53.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.834+0000 7f2af1d9b700 1 -- 172.21.15.151:0/226902537 learned_addr learned my addr 172.21.15.151:0/226902537 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2af259c700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af40ff530 0x7f2af419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2af1d9b700 1 -- 172.21.15.151:0/226902537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af40ff530 msgr2=0x7f2af419c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2af1d9b700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af40ff530 0x7f2af419c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2af1d9b700 1 -- 172.21.15.151:0/226902537 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae80095f0 con 0x7f2af40ffee0 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2af1d9b700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 0x7f2af419ca70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80059d0 tx=0x7f2ae800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2ae37fe700 1 -- 172.21.15.151:0/226902537 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ae801d070 con 0x7f2af40ffee0 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2ae37fe700 1 -- 172.21.15.151:0/226902537 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ae800b610 con 0x7f2af40ffee0 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af41968a0 con 0x7f2af40ffee0 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2ae37fe700 1 -- 172.21.15.151:0/226902537 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ae8022470 con 0x7f2af40ffee0 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.835+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af4196d40 con 0x7f2af40ffee0 2024-04-16T03:01:53.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.836+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af4109150 con 0x7f2af40ffee0 2024-04-16T03:01:53.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.837+0000 7f2ae37fe700 1 -- 172.21.15.151:0/226902537 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2ae800fc70 con 0x7f2af40ffee0 2024-04-16T03:01:53.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.837+0000 7f2ae37fe700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2adc06ecc0 0x7f2adc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:53.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.837+0000 7f2af259c700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2adc06ecc0 0x7f2adc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:53.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.838+0000 7f2ae37fe700 1 -- 172.21.15.151:0/226902537 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2ae80902b0 con 0x7f2af40ffee0 2024-04-16T03:01:53.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.838+0000 7f2af259c700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2adc06ecc0 0x7f2adc071180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009bf0 tx=0x7f2ae4009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:53.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:53.842+0000 7f2ae37fe700 1 -- 172.21.15.151:0/226902537 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2ae805cb90 con 0x7f2af40ffee0 2024-04-16T03:01:54.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.051+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2af4196fb0 con 0x7f2adc06ecc0 2024-04-16T03:01:54.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.063+0000 7f2ae37fe700 1 -- 172.21.15.151:0/226902537 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2af4196fb0 con 0x7f2adc06ecc0 2024-04-16T03:01:54.064 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (23m) 85s ago 34m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (35m) 85s ago 35m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (33m) 87s ago 33m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (21m) 85s ago 34m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (20m) 87s ago 33m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (23m) 85s ago 34m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (17m) 85s ago 29m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (17m) 85s ago 29m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (91s) 87s ago 29m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (16m) 87s ago 29m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (25m) 85s ago 36m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:01:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (24m) 87s ago 33m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (22m) 85s ago 36m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (21m) 87s ago 33m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (24m) 85s ago 34m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (24m) 87s ago 33m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (20m) 85s ago 32m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (19m) 85s ago 32m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (19m) 85s ago 31m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (18m) 87s ago 31m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (18m) 87s ago 31m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (17m) 87s ago 30m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:01:54.066 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (23m) 85s ago 34m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.065+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2adc06ecc0 msgr2=0x7f2adc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.065+0000 7f2af8c9c700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2adc06ecc0 0x7f2adc071180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009bf0 tx=0x7f2ae4009260 comp rx=0 tx=0).stop 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.065+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 msgr2=0x7f2af419ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.065+0000 7f2af8c9c700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 0x7f2af419ca70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80059d0 tx=0x7f2ae800f740 comp rx=0 tx=0).stop 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.066+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 shutdown_connections 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.066+0000 7f2af8c9c700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2adc06ecc0 0x7f2adc071180 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.066+0000 7f2af8c9c700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af40ff530 0x7f2af419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.066+0000 7f2af8c9c700 1 --2- 172.21.15.151:0/226902537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2af40ffee0 0x7f2af419ca70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.066+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 >> 172.21.15.151:0/226902537 conn(0x7f2af4074df0 msgr2=0x7f2af41050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.066+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 shutdown_connections 2024-04-16T03:01:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.066+0000 7f2af8c9c700 1 -- 172.21.15.151:0/226902537 wait complete. 2024-04-16T03:01:54.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.170+0000 7f1d50972700 1 -- 172.21.15.151:0/1287839903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 msgr2=0x7f1d4c10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.170+0000 7f1d50972700 1 --2- 172.21.15.151:0/1287839903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 0x7f1d4c10ab10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009910 tx=0x7f1d3c009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:54.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.170+0000 7f1d50972700 1 -- 172.21.15.151:0/1287839903 shutdown_connections 2024-04-16T03:01:54.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.170+0000 7f1d50972700 1 --2- 172.21.15.151:0/1287839903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 0x7f1d4c10ab10 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.170+0000 7f1d50972700 1 --2- 172.21.15.151:0/1287839903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d4c101c70 0x7f1d4c102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.170+0000 7f1d50972700 1 -- 172.21.15.151:0/1287839903 >> 172.21.15.151:0/1287839903 conn(0x7f1d4c0fb570 msgr2=0x7f1d4c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:54.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.170+0000 7f1d50972700 1 -- 172.21.15.151:0/1287839903 shutdown_connections 2024-04-16T03:01:54.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.170+0000 7f1d50972700 1 -- 172.21.15.151:0/1287839903 wait complete. 2024-04-16T03:01:54.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.171+0000 7f1d50972700 1 Processor -- start 2024-04-16T03:01:54.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.171+0000 7f1d50972700 1 -- start start 2024-04-16T03:01:54.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.171+0000 7f1d50972700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d4c101c70 0x7f1d4c0ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:54.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.171+0000 7f1d50972700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 0x7f1d4c0ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:54.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.171+0000 7f1d50972700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c1003e0 con 0x7f1d4c102620 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.171+0000 7f1d50972700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c100520 con 0x7f1d4c101c70 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d49d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 0x7f1d4c0ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d49d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 0x7f1d4c0ffea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44414/0 (socket says 172.21.15.151:44414) 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d49d9b700 1 -- 172.21.15.151:0/2761410819 learned_addr learned my addr 172.21.15.151:0/2761410819 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d4a59c700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d4c101c70 0x7f1d4c0ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d49d9b700 1 -- 172.21.15.151:0/2761410819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d4c101c70 msgr2=0x7f1d4c0ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d49d9b700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d4c101c70 0x7f1d4c0ff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d49d9b700 1 -- 172.21.15.151:0/2761410819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0095f0 con 0x7f1d4c102620 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d49d9b700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 0x7f1d4c0ffea0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0092e0 tx=0x7f1d3c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d3b7fe700 1 -- 172.21.15.151:0/2761410819 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d3c01d070 con 0x7f1d4c102620 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4c1007a0 con 0x7f1d4c102620 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d3b7fe700 1 -- 172.21.15.151:0/2761410819 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d3c00f460 con 0x7f1d4c102620 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d3b7fe700 1 -- 172.21.15.151:0/2761410819 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d3c0175d0 con 0x7f1d4c102620 2024-04-16T03:01:54.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.172+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4c1a2c40 con 0x7f1d4c102620 2024-04-16T03:01:54.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.174+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4c108210 con 0x7f1d4c102620 2024-04-16T03:01:54.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.174+0000 7f1d3b7fe700 1 -- 172.21.15.151:0/2761410819 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1d3c00f5d0 con 0x7f1d4c102620 2024-04-16T03:01:54.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.174+0000 7f1d3b7fe700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3406ecc0 0x7f1d34071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:54.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.175+0000 7f1d3b7fe700 1 -- 172.21.15.151:0/2761410819 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1d3c090360 con 0x7f1d4c102620 2024-04-16T03:01:54.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.175+0000 7f1d4a59c700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3406ecc0 0x7f1d34071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:54.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.175+0000 7f1d4a59c700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3406ecc0 0x7f1d34071180 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1d40005950 tx=0x7f1d4000a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:54.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.179+0000 7f1d3b7fe700 1 -- 172.21.15.151:0/2761410819 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1d3c0590b0 con 0x7f1d4c102620 2024-04-16T03:01:54.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.461+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1d4c04ea90 con 0x7f1d4c102620 2024-04-16T03:01:54.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.462+0000 7f1d3b7fe700 1 -- 172.21.15.151:0/2761410819 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f1d3c05c770 con 0x7f1d4c102620 2024-04-16T03:01:54.464 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:01:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:01:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:01:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:01:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:01:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:01:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:01:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:01:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:01:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:01:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:01:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:01:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:01:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:01:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:01:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:01:54.465 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:01:54.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3406ecc0 msgr2=0x7f1d34071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3406ecc0 0x7f1d34071180 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1d40005950 tx=0x7f1d4000a300 comp rx=0 tx=0).stop 2024-04-16T03:01:54.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 msgr2=0x7f1d4c0ffea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 0x7f1d4c0ffea0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0092e0 tx=0x7f1d3c00bd70 comp rx=0 tx=0).stop 2024-04-16T03:01:54.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 shutdown_connections 2024-04-16T03:01:54.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d3406ecc0 0x7f1d34071180 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d4c101c70 0x7f1d4c0ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 --2- 172.21.15.151:0/2761410819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d4c102620 0x7f1d4c0ffea0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 >> 172.21.15.151:0/2761410819 conn(0x7f1d4c0fb570 msgr2=0x7f1d4c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:54.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 shutdown_connections 2024-04-16T03:01:54.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.464+0000 7f1d50972700 1 -- 172.21.15.151:0/2761410819 wait complete. 2024-04-16T03:01:54.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.569+0000 7f29c61cf700 1 -- 172.21.15.151:0/4118992773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c00ffbd0 msgr2=0x7f29c0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.569+0000 7f29c61cf700 1 --2- 172.21.15.151:0/4118992773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c00ffbd0 0x7f29c0100050 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009910 tx=0x7f29b4009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:54.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.569+0000 7f29c61cf700 1 -- 172.21.15.151:0/4118992773 shutdown_connections 2024-04-16T03:01:54.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.569+0000 7f29c61cf700 1 --2- 172.21.15.151:0/4118992773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c00ffbd0 0x7f29c0100050 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.569+0000 7f29c61cf700 1 --2- 172.21.15.151:0/4118992773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c0105a40 0x7f29c0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.569+0000 7f29c61cf700 1 -- 172.21.15.151:0/4118992773 >> 172.21.15.151:0/4118992773 conn(0x7f29c00755a0 msgr2=0x7f29c00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:54.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.569+0000 7f29c61cf700 1 -- 172.21.15.151:0/4118992773 shutdown_connections 2024-04-16T03:01:54.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.569+0000 7f29c61cf700 1 -- 172.21.15.151:0/4118992773 wait complete. 2024-04-16T03:01:54.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29c61cf700 1 Processor -- start 2024-04-16T03:01:54.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29c61cf700 1 -- start start 2024-04-16T03:01:54.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29c61cf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c00ffbd0 0x7f29c0101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:54.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29c61cf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c0105a40 0x7f29c0101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:54.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29c61cf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0102090 con 0x7f29c0105a40 2024-04-16T03:01:54.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29c61cf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c01021d0 con 0x7f29c00ffbd0 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29beffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c0105a40 0x7f29c0101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29beffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c0105a40 0x7f29c0101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44424/0 (socket says 172.21.15.151:44424) 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29beffd700 1 -- 172.21.15.151:0/2133736604 learned_addr learned my addr 172.21.15.151:0/2133736604 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29bf7fe700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c00ffbd0 0x7f29c0101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29beffd700 1 -- 172.21.15.151:0/2133736604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c00ffbd0 msgr2=0x7f29c0101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29beffd700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c00ffbd0 0x7f29c0101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.570+0000 7f29beffd700 1 -- 172.21.15.151:0/2133736604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b40095f0 con 0x7f29c0105a40 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.571+0000 7f29beffd700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c0105a40 0x7f29c0101b50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f29b4006010 tx=0x7f29b400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.571+0000 7f29bcff9700 1 -- 172.21.15.151:0/2133736604 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29b401d070 con 0x7f29c0105a40 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.571+0000 7f29bcff9700 1 -- 172.21.15.151:0/2133736604 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29b4005dc0 con 0x7f29c0105a40 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.571+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c0102450 con 0x7f29c0105a40 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.571+0000 7f29bcff9700 1 -- 172.21.15.151:0/2133736604 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29b4017400 con 0x7f29c0105a40 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.571+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c01a2c30 con 0x7f29c0105a40 2024-04-16T03:01:54.574 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.572+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c0108fe0 con 0x7f29c0105a40 2024-04-16T03:01:54.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.572+0000 7f29bcff9700 1 -- 172.21.15.151:0/2133736604 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f29b4003a60 con 0x7f29c0105a40 2024-04-16T03:01:54.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.573+0000 7f29bcff9700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f29a806ecc0 0x7f29a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:54.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.573+0000 7f29bcff9700 1 -- 172.21.15.151:0/2133736604 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f29b4090200 con 0x7f29c0105a40 2024-04-16T03:01:54.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.573+0000 7f29bf7fe700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f29a806ecc0 0x7f29a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:54.575 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.573+0000 7f29bf7fe700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f29a806ecc0 0x7f29a8071180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f29b0005950 tx=0x7f29b000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:54.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.577+0000 7f29bcff9700 1 -- 172.21.15.151:0/2133736604 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f29b4059030 con 0x7f29c0105a40 2024-04-16T03:01:54.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.823+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f29c004ea90 con 0x7f29c0105a40 2024-04-16T03:01:54.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.823+0000 7f29bcff9700 1 -- 172.21.15.151:0/2133736604 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f29b405c6f0 con 0x7f29c0105a40 2024-04-16T03:01:54.825 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:01:54.825 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:01:54.825 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:01:54.825 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:01:54.826 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:01:54.827 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:01:54.828 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:01:54.828 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:01:54.828 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:01:54.828 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:01:54.828 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:01:54.828 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:01:54.828 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:01:54.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.825+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f29a806ecc0 msgr2=0x7f29a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.825+0000 7f29c61cf700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f29a806ecc0 0x7f29a8071180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f29b0005950 tx=0x7f29b000b410 comp rx=0 tx=0).stop 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.825+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c0105a40 msgr2=0x7f29c0101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.825+0000 7f29c61cf700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c0105a40 0x7f29c0101b50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f29b4006010 tx=0x7f29b400bd70 comp rx=0 tx=0).stop 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.825+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 shutdown_connections 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.825+0000 7f29c61cf700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f29a806ecc0 0x7f29a8071180 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.825+0000 7f29c61cf700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c00ffbd0 0x7f29c0101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.826+0000 7f29c61cf700 1 --2- 172.21.15.151:0/2133736604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29c0105a40 0x7f29c0101b50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.826+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 >> 172.21.15.151:0/2133736604 conn(0x7f29c00755a0 msgr2=0x7f29c0068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.826+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 shutdown_connections 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.826+0000 7f29c61cf700 1 -- 172.21.15.151:0/2133736604 wait complete. 2024-04-16T03:01:54.829 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:01:54.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.930+0000 7f6d12a9e700 1 -- 172.21.15.151:0/883723959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c100dc0 msgr2=0x7f6d0c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.930+0000 7f6d12a9e700 1 --2- 172.21.15.151:0/883723959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c100dc0 0x7f6d0c1011a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009910 tx=0x7f6d00009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:54.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.930+0000 7f6d12a9e700 1 -- 172.21.15.151:0/883723959 shutdown_connections 2024-04-16T03:01:54.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.930+0000 7f6d12a9e700 1 --2- 172.21.15.151:0/883723959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d0c101770 0x7f6d0c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.930+0000 7f6d12a9e700 1 --2- 172.21.15.151:0/883723959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c100dc0 0x7f6d0c1011a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.930+0000 7f6d12a9e700 1 -- 172.21.15.151:0/883723959 >> 172.21.15.151:0/883723959 conn(0x7f6d0c0fc650 msgr2=0x7f6d0c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:54.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.930+0000 7f6d12a9e700 1 -- 172.21.15.151:0/883723959 shutdown_connections 2024-04-16T03:01:54.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.930+0000 7f6d12a9e700 1 -- 172.21.15.151:0/883723959 wait complete. 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d12a9e700 1 Processor -- start 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d12a9e700 1 -- start start 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d12a9e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d0c100dc0 0x7f6d0c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d12a9e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c101770 0x7f6d0c198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d12a9e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0c199520 con 0x7f6d0c101770 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d12a9e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0c19d330 con 0x7f6d0c100dc0 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d0bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c101770 0x7f6d0c198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d0bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c101770 0x7f6d0c198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44442/0 (socket says 172.21.15.151:44442) 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d0bfff700 1 -- 172.21.15.151:0/2706805839 learned_addr learned my addr 172.21.15.151:0/2706805839 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.931+0000 7f6d1083a700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d0c100dc0 0x7f6d0c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.932+0000 7f6d0bfff700 1 -- 172.21.15.151:0/2706805839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d0c100dc0 msgr2=0x7f6d0c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:54.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.932+0000 7f6d0bfff700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d0c100dc0 0x7f6d0c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.932+0000 7f6d0bfff700 1 -- 172.21.15.151:0/2706805839 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d000095f0 con 0x7f6d0c101770 2024-04-16T03:01:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.932+0000 7f6d0bfff700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c101770 0x7f6d0c198e90 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800d900 tx=0x7f6cf800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.932+0000 7f6d09ffb700 1 -- 172.21.15.151:0/2706805839 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cf8009900 con 0x7f6d0c101770 2024-04-16T03:01:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.932+0000 7f6d09ffb700 1 -- 172.21.15.151:0/2706805839 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cf8010460 con 0x7f6d0c101770 2024-04-16T03:01:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.932+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0c19d610 con 0x7f6d0c101770 2024-04-16T03:01:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.932+0000 7f6d09ffb700 1 -- 172.21.15.151:0/2706805839 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cf800f5f0 con 0x7f6d0c101770 2024-04-16T03:01:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.932+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d0c19db60 con 0x7f6d0c101770 2024-04-16T03:01:54.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.933+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0c104f40 con 0x7f6d0c101770 2024-04-16T03:01:54.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.934+0000 7f6d09ffb700 1 -- 172.21.15.151:0/2706805839 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6cf80105d0 con 0x7f6d0c101770 2024-04-16T03:01:54.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.934+0000 7f6d09ffb700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6cfc06ecc0 0x7f6cfc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:54.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.934+0000 7f6d09ffb700 1 -- 172.21.15.151:0/2706805839 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6cf808ea30 con 0x7f6d0c101770 2024-04-16T03:01:54.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.934+0000 7f6d1083a700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6cfc06ecc0 0x7f6cfc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:54.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.935+0000 7f6d1083a700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6cfc06ecc0 0x7f6cfc071180 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6d00006010 tx=0x7f6d0001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:54.940 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:54.939+0000 7f6d09ffb700 1 -- 172.21.15.151:0/2706805839 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6cf805b310 con 0x7f6d0c101770 2024-04-16T03:01:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:54 smithi153 ceph-mon[87858]: pgmap v699: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:54 smithi153 ceph-mon[87858]: from='client.35296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:54 smithi153 ceph-mon[87858]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:01:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:54 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2761410819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:01:55.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.156+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d0c19dd80 con 0x7f6cfc06ecc0 2024-04-16T03:01:55.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:54 smithi151 ceph-mon[112233]: pgmap v699: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:55.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:54 smithi151 ceph-mon[112233]: from='client.35296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:55.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:54 smithi151 ceph-mon[112233]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:55.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:01:55.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:54 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2761410819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:01:55.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.158+0000 7f6d09ffb700 1 -- 172.21.15.151:0/2706805839 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6d0c19dd80 con 0x7f6cfc06ecc0 2024-04-16T03:01:55.159 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:01:55.159 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:01:55.159 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:01:55.159 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:01:55.159 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:01:55.160 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6cfc06ecc0 msgr2=0x7f6cfc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6cfc06ecc0 0x7f6cfc071180 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6d00006010 tx=0x7f6d0001a040 comp rx=0 tx=0).stop 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c101770 msgr2=0x7f6d0c198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c101770 0x7f6d0c198e90 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800d900 tx=0x7f6cf800dc10 comp rx=0 tx=0).stop 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 shutdown_connections 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6cfc06ecc0 0x7f6cfc071180 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d0c100dc0 0x7f6d0c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 --2- 172.21.15.151:0/2706805839 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d0c101770 0x7f6d0c198e90 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 >> 172.21.15.151:0/2706805839 conn(0x7f6d0c0fc650 msgr2=0x7f6d0c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 shutdown_connections 2024-04-16T03:01:55.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.161+0000 7f6d12a9e700 1 -- 172.21.15.151:0/2706805839 wait complete. 2024-04-16T03:01:55.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.265+0000 7f3518301700 1 -- 172.21.15.151:0/2631382019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510101770 msgr2=0x7f35101057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:55.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.265+0000 7f3518301700 1 --2- 172.21.15.151:0/2631382019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510101770 0x7f35101057c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f350c009910 tx=0x7f350c009c20 comp rx=0 tx=0).stop 2024-04-16T03:01:55.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.265+0000 7f3518301700 1 -- 172.21.15.151:0/2631382019 shutdown_connections 2024-04-16T03:01:55.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.265+0000 7f3518301700 1 --2- 172.21.15.151:0/2631382019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510101770 0x7f35101057c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:55.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.265+0000 7f3518301700 1 --2- 172.21.15.151:0/2631382019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3510100dc0 0x7f35101011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:55.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.265+0000 7f3518301700 1 -- 172.21.15.151:0/2631382019 >> 172.21.15.151:0/2631382019 conn(0x7f35100fc650 msgr2=0x7f35100fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:55.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f3518301700 1 -- 172.21.15.151:0/2631382019 shutdown_connections 2024-04-16T03:01:55.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f3518301700 1 -- 172.21.15.151:0/2631382019 wait complete. 2024-04-16T03:01:55.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f3518301700 1 Processor -- start 2024-04-16T03:01:55.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f3518301700 1 -- start start 2024-04-16T03:01:55.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f3518301700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510100dc0 0x7f351019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:55.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f3518301700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3510101770 0x7f351019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:55.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f3518301700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351019d130 con 0x7f3510100dc0 2024-04-16T03:01:55.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f3518301700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35101965e0 con 0x7f3510101770 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f351609d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510100dc0 0x7f351019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.266+0000 7f351609d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510100dc0 0x7f351019c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44460/0 (socket says 172.21.15.151:44460) 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f351609d700 1 -- 172.21.15.151:0/4237100131 learned_addr learned my addr 172.21.15.151:0/4237100131 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f351589c700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3510101770 0x7f351019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f351609d700 1 -- 172.21.15.151:0/4237100131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3510101770 msgr2=0x7f351019caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f351609d700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3510101770 0x7f351019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f351609d700 1 -- 172.21.15.151:0/4237100131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c0095f0 con 0x7f3510100dc0 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f351609d700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510100dc0 0x7f351019c560 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f350000d8d0 tx=0x7f350000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f35077fe700 1 -- 172.21.15.151:0/4237100131 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3500009940 con 0x7f3510100dc0 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f35077fe700 1 -- 172.21.15.151:0/4237100131 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3500010460 con 0x7f3510100dc0 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35101968c0 con 0x7f3510100dc0 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f35077fe700 1 -- 172.21.15.151:0/4237100131 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f350000f5f0 con 0x7f3510100dc0 2024-04-16T03:01:55.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.267+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3510196e10 con 0x7f3510100dc0 2024-04-16T03:01:55.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.268+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35101091c0 con 0x7f3510100dc0 2024-04-16T03:01:55.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.269+0000 7f35077fe700 1 -- 172.21.15.151:0/4237100131 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f35000105d0 con 0x7f3510100dc0 2024-04-16T03:01:55.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.269+0000 7f35077fe700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34fc06ecc0 0x7f34fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:01:55.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.269+0000 7f35077fe700 1 -- 172.21.15.151:0/4237100131 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f350008ed70 con 0x7f3510100dc0 2024-04-16T03:01:55.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.269+0000 7f351589c700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34fc06ecc0 0x7f34fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:01:55.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.270+0000 7f351589c700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34fc06ecc0 0x7f34fc071180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f350c0059d0 tx=0x7f350c005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:01:55.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.274+0000 7f35077fe700 1 -- 172.21.15.151:0/4237100131 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f350005b6d0 con 0x7f3510100dc0 2024-04-16T03:01:55.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.555+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f35101970b0 con 0x7f3510100dc0 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.556+0000 7f35077fe700 1 -- 172.21.15.151:0/4237100131 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f350005b260 con 0x7f3510100dc0 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:01:55.558 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:01:55.559 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34fc06ecc0 msgr2=0x7f34fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34fc06ecc0 0x7f34fc071180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f350c0059d0 tx=0x7f350c005960 comp rx=0 tx=0).stop 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510100dc0 msgr2=0x7f351019c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510100dc0 0x7f351019c560 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f350000d8d0 tx=0x7f350000dc90 comp rx=0 tx=0).stop 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 shutdown_connections 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f34fc06ecc0 0x7f34fc071180 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3510100dc0 0x7f351019c560 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 --2- 172.21.15.151:0/4237100131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3510101770 0x7f351019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 >> 172.21.15.151:0/4237100131 conn(0x7f35100fc650 msgr2=0x7f35100fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 shutdown_connections 2024-04-16T03:01:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:01:55.559+0000 7f3518301700 1 -- 172.21.15.151:0/4237100131 wait complete. 2024-04-16T03:01:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:55 smithi151 ceph-mon[112233]: from='client.35304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:55 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2133736604' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:01:55.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:55 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4237100131' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:01:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:55 smithi153 ceph-mon[87858]: from='client.35304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:55 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2133736604' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:01:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:55 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4237100131' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:01:57.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:56 smithi153 ceph-mon[87858]: from='client.35316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:57.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:56 smithi153 ceph-mon[87858]: pgmap v700: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:57.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:56 smithi151 ceph-mon[112233]: from='client.35316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:01:57.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:56 smithi151 ceph-mon[112233]: pgmap v700: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:59.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:01:58 smithi153 ceph-mon[87858]: pgmap v701: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:01:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:01:58 smithi151 ceph-mon[112233]: pgmap v701: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:00 smithi153 ceph-mon[87858]: pgmap v702: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:01.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:00 smithi151 ceph-mon[112233]: pgmap v702: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:02 smithi151 ceph-mon[112233]: pgmap v703: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:03.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:02 smithi153 ceph-mon[87858]: pgmap v703: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:05.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:04 smithi153 ceph-mon[87858]: pgmap v704: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:05.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:04 smithi151 ceph-mon[112233]: pgmap v704: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:07.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:06 smithi151 ceph-mon[112233]: pgmap v705: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:07.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:06 smithi153 ceph-mon[87858]: pgmap v705: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:08 smithi151 ceph-mon[112233]: pgmap v706: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:08 smithi153 ceph-mon[87858]: pgmap v706: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:02:10.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:02:11.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:10 smithi151 ceph-mon[112233]: pgmap v707: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:10 smithi153 ceph-mon[87858]: pgmap v707: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:12 smithi151 ceph-mon[112233]: pgmap v708: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:12 smithi153 ceph-mon[87858]: pgmap v708: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:15.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:14 smithi151 ceph-mon[112233]: pgmap v709: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:15.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:14 smithi153 ceph-mon[87858]: pgmap v709: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:17.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:16 smithi151 ceph-mon[112233]: pgmap v710: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:17.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:16 smithi153 ceph-mon[87858]: pgmap v710: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:18 smithi151 ceph-mon[112233]: pgmap v711: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:19.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:18 smithi153 ceph-mon[87858]: pgmap v711: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:20 smithi153 ceph-mon[87858]: pgmap v712: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:21.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:20 smithi151 ceph-mon[112233]: pgmap v712: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:23.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:22 smithi151 ceph-mon[112233]: pgmap v713: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:23.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:22 smithi153 ceph-mon[87858]: pgmap v713: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:24.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:02:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:02:25.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:24 smithi151 ceph-mon[112233]: pgmap v714: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:24 smithi153 ceph-mon[87858]: pgmap v714: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:25.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.677+0000 7fbade7a9700 1 -- 172.21.15.151:0/2239122464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 msgr2=0x7fbad8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:25.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.677+0000 7fbade7a9700 1 --2- 172.21.15.151:0/2239122464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 0x7fbad8105e20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009910 tx=0x7fbac8009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:25.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.677+0000 7fbade7a9700 1 -- 172.21.15.151:0/2239122464 shutdown_connections 2024-04-16T03:02:25.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.677+0000 7fbade7a9700 1 --2- 172.21.15.151:0/2239122464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbad80ffa40 0x7fbad80ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:25.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.677+0000 7fbade7a9700 1 --2- 172.21.15.151:0/2239122464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 0x7fbad8105e20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:25.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.677+0000 7fbade7a9700 1 -- 172.21.15.151:0/2239122464 >> 172.21.15.151:0/2239122464 conn(0x7fbad80fb570 msgr2=0x7fbad80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:25.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.677+0000 7fbade7a9700 1 -- 172.21.15.151:0/2239122464 shutdown_connections 2024-04-16T03:02:25.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.677+0000 7fbade7a9700 1 -- 172.21.15.151:0/2239122464 wait complete. 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbade7a9700 1 Processor -- start 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbade7a9700 1 -- start start 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbade7a9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbad80ffa40 0x7fbad81975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbade7a9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 0x7fbad8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbade7a9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8198110 con 0x7fbad8105a40 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbade7a9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8198250 con 0x7fbad80ffa40 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbacffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 0x7fbad8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbacffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 0x7fbad8197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59642/0 (socket says 172.21.15.151:59642) 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbacffff700 1 -- 172.21.15.151:0/1241333008 learned_addr learned my addr 172.21.15.151:0/1241333008 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbad7fff700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbad80ffa40 0x7fbad81975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbacffff700 1 -- 172.21.15.151:0/1241333008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbad80ffa40 msgr2=0x7fbad81975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbacffff700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbad80ffa40 0x7fbad81975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.678+0000 7fbacffff700 1 -- 172.21.15.151:0/1241333008 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac80095f0 con 0x7fbad8105a40 2024-04-16T03:02:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.679+0000 7fbacffff700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 0x7fbad8197af0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fbac000c8a0 tx=0x7fbac000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.679+0000 7fbad5ffb700 1 -- 172.21.15.151:0/1241333008 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbac00078e0 con 0x7fbad8105a40 2024-04-16T03:02:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.679+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad81a0620 con 0x7fbad8105a40 2024-04-16T03:02:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.679+0000 7fbad5ffb700 1 -- 172.21.15.151:0/1241333008 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbac000f450 con 0x7fbad8105a40 2024-04-16T03:02:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.679+0000 7fbad5ffb700 1 -- 172.21.15.151:0/1241333008 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbac000e5e0 con 0x7fbad8105a40 2024-04-16T03:02:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.679+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad81a0ae0 con 0x7fbad8105a40 2024-04-16T03:02:25.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.680+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad8108040 con 0x7fbad8105a40 2024-04-16T03:02:25.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.680+0000 7fbad5ffb700 1 -- 172.21.15.151:0/1241333008 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fbac000f5c0 con 0x7fbad8105a40 2024-04-16T03:02:25.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.681+0000 7fbad5ffb700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbac406ecc0 0x7fbac4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:25.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.681+0000 7fbad5ffb700 1 -- 172.21.15.151:0/1241333008 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbac008ee40 con 0x7fbad8105a40 2024-04-16T03:02:25.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.681+0000 7fbad7fff700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbac406ecc0 0x7fbac4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:25.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.682+0000 7fbad7fff700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbac406ecc0 0x7fbac4071180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbac8006010 tx=0x7fbac801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:25.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.686+0000 7fbad5ffb700 1 -- 172.21.15.151:0/1241333008 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbac0057bf0 con 0x7fbad8105a40 2024-04-16T03:02:25.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.905+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbad802cd80 con 0x7fbac406ecc0 2024-04-16T03:02:25.908 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.906+0000 7fbad5ffb700 1 -- 172.21.15.151:0/1241333008 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbad802cd80 con 0x7fbac406ecc0 2024-04-16T03:02:25.910 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.908+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbac406ecc0 msgr2=0x7fbac4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.908+0000 7fbade7a9700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbac406ecc0 0x7fbac4071180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbac8006010 tx=0x7fbac801a040 comp rx=0 tx=0).stop 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.908+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 msgr2=0x7fbad8197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.908+0000 7fbade7a9700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 0x7fbad8197af0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fbac000c8a0 tx=0x7fbac000cbb0 comp rx=0 tx=0).stop 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.908+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 shutdown_connections 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.908+0000 7fbade7a9700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbac406ecc0 0x7fbac4071180 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.908+0000 7fbade7a9700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbad80ffa40 0x7fbad81975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.908+0000 7fbade7a9700 1 --2- 172.21.15.151:0/1241333008 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad8105a40 0x7fbad8197af0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.908+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 >> 172.21.15.151:0/1241333008 conn(0x7fbad80fb570 msgr2=0x7fbad80fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.909+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 shutdown_connections 2024-04-16T03:02:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:25.909+0000 7fbade7a9700 1 -- 172.21.15.151:0/1241333008 wait complete. 2024-04-16T03:02:25.921 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:02:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.016+0000 7ff461982700 1 -- 172.21.15.151:0/2665058002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 msgr2=0x7ff45c105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.016+0000 7ff461982700 1 --2- 172.21.15.151:0/2665058002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 0x7ff45c105790 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ff444009910 tx=0x7ff444009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.016+0000 7ff461982700 1 -- 172.21.15.151:0/2665058002 shutdown_connections 2024-04-16T03:02:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.016+0000 7ff461982700 1 --2- 172.21.15.151:0/2665058002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 0x7ff45c105790 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.016+0000 7ff461982700 1 --2- 172.21.15.151:0/2665058002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff45c068670 0x7ff45c068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.016+0000 7ff461982700 1 -- 172.21.15.151:0/2665058002 >> 172.21.15.151:0/2665058002 conn(0x7ff45c0755a0 msgr2=0x7ff45c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.016+0000 7ff461982700 1 -- 172.21.15.151:0/2665058002 shutdown_connections 2024-04-16T03:02:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.016+0000 7ff461982700 1 -- 172.21.15.151:0/2665058002 wait complete. 2024-04-16T03:02:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff461982700 1 Processor -- start 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff461982700 1 -- start start 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff461982700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff45c068670 0x7ff45c196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff461982700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 0x7ff45c196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff461982700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c1972d0 con 0x7ff45c069020 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff461982700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c19a000 con 0x7ff45c068670 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff4527fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 0x7ff45c196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff4527fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 0x7ff45c196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59658/0 (socket says 172.21.15.151:59658) 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff4527fc700 1 -- 172.21.15.151:0/3631796425 learned_addr learned my addr 172.21.15.151:0/3631796425 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.017+0000 7ff45affd700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff45c068670 0x7ff45c196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.018+0000 7ff4527fc700 1 -- 172.21.15.151:0/3631796425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff45c068670 msgr2=0x7ff45c196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.018+0000 7ff4527fc700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff45c068670 0x7ff45c196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.018+0000 7ff4527fc700 1 -- 172.21.15.151:0/3631796425 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4440095f0 con 0x7ff45c069020 2024-04-16T03:02:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.018+0000 7ff4527fc700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 0x7ff45c196c40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff4440092e0 tx=0x7ff44400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.018+0000 7ff460980700 1 -- 172.21.15.151:0/3631796425 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff44401d070 con 0x7ff45c069020 2024-04-16T03:02:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.018+0000 7ff460980700 1 -- 172.21.15.151:0/3631796425 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff44400f460 con 0x7ff45c069020 2024-04-16T03:02:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.018+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c19a280 con 0x7ff45c069020 2024-04-16T03:02:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.018+0000 7ff460980700 1 -- 172.21.15.151:0/3631796425 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4440175d0 con 0x7ff45c069020 2024-04-16T03:02:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.018+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c19a770 con 0x7ff45c069020 2024-04-16T03:02:26.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.019+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c1090a0 con 0x7ff45c069020 2024-04-16T03:02:26.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.020+0000 7ff460980700 1 -- 172.21.15.151:0/3631796425 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff44400f5d0 con 0x7ff45c069020 2024-04-16T03:02:26.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.020+0000 7ff460980700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44806ecc0 0x7ff448071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:26.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.020+0000 7ff45affd700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44806ecc0 0x7ff448071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:26.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.020+0000 7ff460980700 1 -- 172.21.15.151:0/3631796425 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff4440902c0 con 0x7ff45c069020 2024-04-16T03:02:26.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.021+0000 7ff45affd700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44806ecc0 0x7ff448071180 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff44c005950 tx=0x7ff44c00a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:26.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.025+0000 7ff460980700 1 -- 172.21.15.151:0/3631796425 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff444059070 con 0x7ff45c069020 2024-04-16T03:02:26.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.243+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff45c19aa50 con 0x7ff44806ecc0 2024-04-16T03:02:26.246 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.245+0000 7ff460980700 1 -- 172.21.15.151:0/3631796425 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff45c19aa50 con 0x7ff44806ecc0 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44806ecc0 msgr2=0x7ff448071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44806ecc0 0x7ff448071180 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff44c005950 tx=0x7ff44c00a300 comp rx=0 tx=0).stop 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 msgr2=0x7ff45c196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 0x7ff45c196c40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff4440092e0 tx=0x7ff44400bd70 comp rx=0 tx=0).stop 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 shutdown_connections 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff44806ecc0 0x7ff448071180 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff45c068670 0x7ff45c196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 --2- 172.21.15.151:0/3631796425 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff45c069020 0x7ff45c196c40 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 >> 172.21.15.151:0/3631796425 conn(0x7ff45c0755a0 msgr2=0x7ff45c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 shutdown_connections 2024-04-16T03:02:26.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.247+0000 7ff461982700 1 -- 172.21.15.151:0/3631796425 wait complete. 2024-04-16T03:02:26.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.352+0000 7f1db2c1b700 1 -- 172.21.15.151:0/902434622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac101770 msgr2=0x7f1dac1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.352+0000 7f1db2c1b700 1 --2- 172.21.15.151:0/902434622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac101770 0x7f1dac1057c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009910 tx=0x7f1d98009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:26.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.352+0000 7f1db2c1b700 1 -- 172.21.15.151:0/902434622 shutdown_connections 2024-04-16T03:02:26.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.352+0000 7f1db2c1b700 1 --2- 172.21.15.151:0/902434622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac101770 0x7f1dac1057c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.352+0000 7f1db2c1b700 1 --2- 172.21.15.151:0/902434622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dac100dc0 0x7f1dac1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.352+0000 7f1db2c1b700 1 -- 172.21.15.151:0/902434622 >> 172.21.15.151:0/902434622 conn(0x7f1dac0fc670 msgr2=0x7f1dac0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:26.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.352+0000 7f1db2c1b700 1 -- 172.21.15.151:0/902434622 shutdown_connections 2024-04-16T03:02:26.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.352+0000 7f1db2c1b700 1 -- 172.21.15.151:0/902434622 wait complete. 2024-04-16T03:02:26.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db2c1b700 1 Processor -- start 2024-04-16T03:02:26.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db2c1b700 1 -- start start 2024-04-16T03:02:26.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db2c1b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac100dc0 0x7f1dac19c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:26.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db2c1b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dac101770 0x7f1dac19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:26.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db2c1b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac19d0c0 con 0x7f1dac100dc0 2024-04-16T03:02:26.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db2c1b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac196600 con 0x7f1dac101770 2024-04-16T03:02:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db09b7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac100dc0 0x7f1dac19c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db09b7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac100dc0 0x7f1dac19c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59680/0 (socket says 172.21.15.151:59680) 2024-04-16T03:02:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db09b7700 1 -- 172.21.15.151:0/3256716149 learned_addr learned my addr 172.21.15.151:0/3256716149 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1dabfff700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dac101770 0x7f1dac19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db09b7700 1 -- 172.21.15.151:0/3256716149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dac101770 msgr2=0x7f1dac19cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db09b7700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dac101770 0x7f1dac19cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.353+0000 7f1db09b7700 1 -- 172.21.15.151:0/3256716149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d980095f0 con 0x7f1dac100dc0 2024-04-16T03:02:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.354+0000 7f1db09b7700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac100dc0 0x7f1dac19c580 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c930 tx=0x7f1da000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.354+0000 7f1da9ffb700 1 -- 172.21.15.151:0/3256716149 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1da0007ad0 con 0x7f1dac100dc0 2024-04-16T03:02:26.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.354+0000 7f1da9ffb700 1 -- 172.21.15.151:0/3256716149 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1da000ce80 con 0x7f1dac100dc0 2024-04-16T03:02:26.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.354+0000 7f1da9ffb700 1 -- 172.21.15.151:0/3256716149 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1da0018630 con 0x7f1dac100dc0 2024-04-16T03:02:26.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.354+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac1968e0 con 0x7f1dac100dc0 2024-04-16T03:02:26.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.354+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac196e30 con 0x7f1dac100dc0 2024-04-16T03:02:26.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.355+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac1091c0 con 0x7f1dac100dc0 2024-04-16T03:02:26.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.355+0000 7f1da9ffb700 1 -- 172.21.15.151:0/3256716149 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1da0007c30 con 0x7f1dac100dc0 2024-04-16T03:02:26.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.356+0000 7f1da9ffb700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d9c06ecc0 0x7f1d9c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:26.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.356+0000 7f1da9ffb700 1 -- 172.21.15.151:0/3256716149 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1da008edd0 con 0x7f1dac100dc0 2024-04-16T03:02:26.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.356+0000 7f1dabfff700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d9c06ecc0 0x7f1d9c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:26.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.356+0000 7f1dabfff700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d9c06ecc0 0x7f1d9c071180 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0fce40 tx=0x7f1d9801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:26.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.361+0000 7f1da9ffb700 1 -- 172.21.15.151:0/3256716149 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1da005b730 con 0x7f1dac100dc0 2024-04-16T03:02:26.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.570+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dac197100 con 0x7f1d9c06ecc0 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.581+0000 7f1da9ffb700 1 -- 172.21.15.151:0/3256716149 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1dac197100 con 0x7f1d9c06ecc0 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (24m) 118s ago 35m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (35m) 118s ago 35m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (34m) 2m ago 34m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (21m) 118s ago 35m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (21m) 2m ago 34m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (23m) 118s ago 35m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (17m) 118s ago 29m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (17m) 118s ago 29m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (2m) 2m ago 29m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (17m) 2m ago 29m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:02:26.583 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (26m) 118s ago 36m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (24m) 2m ago 34m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (22m) 118s ago 36m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (22m) 2m ago 34m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (24m) 118s ago 35m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (24m) 2m ago 34m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (21m) 118s ago 33m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (20m) 118s ago 33m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (19m) 118s ago 32m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (19m) 2m ago 32m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (18m) 2m ago 31m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (18m) 2m ago 31m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:02:26.584 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (24m) 118s ago 34m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:02:26.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d9c06ecc0 msgr2=0x7f1d9c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d9c06ecc0 0x7f1d9c071180 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0fce40 tx=0x7f1d9801a040 comp rx=0 tx=0).stop 2024-04-16T03:02:26.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac100dc0 msgr2=0x7f1dac19c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac100dc0 0x7f1dac19c580 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c930 tx=0x7f1da000cc40 comp rx=0 tx=0).stop 2024-04-16T03:02:26.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 shutdown_connections 2024-04-16T03:02:26.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1d9c06ecc0 0x7f1d9c071180 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1dac100dc0 0x7f1dac19c580 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 --2- 172.21.15.151:0/3256716149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dac101770 0x7f1dac19cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 >> 172.21.15.151:0/3256716149 conn(0x7f1dac0fc670 msgr2=0x7f1dac0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:26.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 shutdown_connections 2024-04-16T03:02:26.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.583+0000 7f1db2c1b700 1 -- 172.21.15.151:0/3256716149 wait complete. 2024-04-16T03:02:26.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.687+0000 7fa3b93d9700 1 -- 172.21.15.151:0/4279611013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4100de0 msgr2=0x7fa3b41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.687+0000 7fa3b93d9700 1 --2- 172.21.15.151:0/4279611013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4100de0 0x7fa3b41011c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009910 tx=0x7fa3a4009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:26.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.687+0000 7fa3b93d9700 1 -- 172.21.15.151:0/4279611013 shutdown_connections 2024-04-16T03:02:26.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.687+0000 7fa3b93d9700 1 --2- 172.21.15.151:0/4279611013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4101790 0x7fa3b41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.687+0000 7fa3b93d9700 1 --2- 172.21.15.151:0/4279611013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4100de0 0x7fa3b41011c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.687+0000 7fa3b93d9700 1 -- 172.21.15.151:0/4279611013 >> 172.21.15.151:0/4279611013 conn(0x7fa3b40fc670 msgr2=0x7fa3b40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:26.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.687+0000 7fa3b93d9700 1 -- 172.21.15.151:0/4279611013 shutdown_connections 2024-04-16T03:02:26.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.687+0000 7fa3b93d9700 1 -- 172.21.15.151:0/4279611013 wait complete. 2024-04-16T03:02:26.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.688+0000 7fa3b93d9700 1 Processor -- start 2024-04-16T03:02:26.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.688+0000 7fa3b93d9700 1 -- start start 2024-04-16T03:02:26.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.688+0000 7fa3b93d9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4100de0 0x7fa3b4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.688+0000 7fa3b93d9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4101790 0x7fa3b40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.688+0000 7fa3b93d9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b4073b10 con 0x7fa3b4101790 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.688+0000 7fa3b93d9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b4073c50 con 0x7fa3b4100de0 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.688+0000 7fa3b27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4101790 0x7fa3b40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa3b27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4101790 0x7fa3b40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59902/0 (socket says 172.21.15.151:59902) 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa3b27fc700 1 -- 172.21.15.151:0/3724306604 learned_addr learned my addr 172.21.15.151:0/3724306604 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa3b2ffd700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4100de0 0x7fa3b4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa3b27fc700 1 -- 172.21.15.151:0/3724306604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4100de0 msgr2=0x7fa3b4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa3b27fc700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4100de0 0x7fa3b4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa3b27fc700 1 -- 172.21.15.151:0/3724306604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a40095f0 con 0x7fa3b4101790 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa3b27fc700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4101790 0x7fa3b40735d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800d900 tx=0x7fa3a800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa39bfff700 1 -- 172.21.15.151:0/3724306604 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a8009900 con 0x7fa3b4101790 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa39bfff700 1 -- 172.21.15.151:0/3724306604 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3a8010460 con 0x7fa3b4101790 2024-04-16T03:02:26.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b41a2930 con 0x7fa3b4101790 2024-04-16T03:02:26.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa39bfff700 1 -- 172.21.15.151:0/3724306604 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a800f5f0 con 0x7fa3b4101790 2024-04-16T03:02:26.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.689+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b41a2d40 con 0x7fa3b4101790 2024-04-16T03:02:26.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.690+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b4109180 con 0x7fa3b4101790 2024-04-16T03:02:26.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.691+0000 7fa39bfff700 1 -- 172.21.15.151:0/3724306604 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa3a80105d0 con 0x7fa3b4101790 2024-04-16T03:02:26.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.692+0000 7fa39bfff700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa39c06ecc0 0x7fa39c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:26.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.692+0000 7fa39bfff700 1 -- 172.21.15.151:0/3724306604 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa3a808ea00 con 0x7fa3b4101790 2024-04-16T03:02:26.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.692+0000 7fa3b2ffd700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa39c06ecc0 0x7fa39c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:26.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.693+0000 7fa3b2ffd700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa39c06ecc0 0x7fa39c071180 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4006010 tx=0x7fa3a401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:26.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.697+0000 7fa39bfff700 1 -- 172.21.15.151:0/3724306604 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa3a80577b0 con 0x7fa3b4101790 2024-04-16T03:02:26.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.980+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa3b404ea90 con 0x7fa3b4101790 2024-04-16T03:02:26.982 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:26 smithi151 ceph-mon[112233]: pgmap v715: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:26.982 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:26 smithi151 ceph-mon[112233]: from='client.35324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:02:26.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.980+0000 7fa39bfff700 1 -- 172.21.15.151:0/3724306604 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fa3a805ae70 con 0x7fa3b4101790 2024-04-16T03:02:26.982 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:02:26.982 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:02:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:02:26.984 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.982+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa39c06ecc0 msgr2=0x7fa39c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.982+0000 7fa3b93d9700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa39c06ecc0 0x7fa39c071180 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4006010 tx=0x7fa3a401a040 comp rx=0 tx=0).stop 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.982+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4101790 msgr2=0x7fa3b40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.982+0000 7fa3b93d9700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4101790 0x7fa3b40735d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800d900 tx=0x7fa3a800dc10 comp rx=0 tx=0).stop 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.983+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 shutdown_connections 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.983+0000 7fa3b93d9700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa39c06ecc0 0x7fa39c071180 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.983+0000 7fa3b93d9700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b4100de0 0x7fa3b4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.983+0000 7fa3b93d9700 1 --2- 172.21.15.151:0/3724306604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3b4101790 0x7fa3b40735d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.983+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 >> 172.21.15.151:0/3724306604 conn(0x7fa3b40fc670 msgr2=0x7fa3b40fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.983+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 shutdown_connections 2024-04-16T03:02:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:26.983+0000 7fa3b93d9700 1 -- 172.21.15.151:0/3724306604 wait complete. 2024-04-16T03:02:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.090+0000 7f4cb07f4700 1 -- 172.21.15.151:0/512198559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 msgr2=0x7f4ca80ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.090+0000 7f4cb07f4700 1 --2- 172.21.15.151:0/512198559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 0x7f4ca80ffef0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009910 tx=0x7f4c98009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.090+0000 7f4cb07f4700 1 -- 172.21.15.151:0/512198559 shutdown_connections 2024-04-16T03:02:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.090+0000 7f4cb07f4700 1 --2- 172.21.15.151:0/512198559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 0x7f4ca80ffef0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.090+0000 7f4cb07f4700 1 --2- 172.21.15.151:0/512198559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8105a70 0x7f4ca8105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.090+0000 7f4cb07f4700 1 -- 172.21.15.151:0/512198559 >> 172.21.15.151:0/512198559 conn(0x7f4ca80fb560 msgr2=0x7f4ca80fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.090+0000 7f4cb07f4700 1 -- 172.21.15.151:0/512198559 shutdown_connections 2024-04-16T03:02:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.090+0000 7f4cb07f4700 1 -- 172.21.15.151:0/512198559 wait complete. 2024-04-16T03:02:27.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cb07f4700 1 Processor -- start 2024-04-16T03:02:27.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cb07f4700 1 -- start start 2024-04-16T03:02:27.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cb07f4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 0x7f4ca80fbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:27.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cb07f4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8105a70 0x7f4ca80fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:27.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cb07f4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80fc900 con 0x7f4ca80ffa70 2024-04-16T03:02:27.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cb07f4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80fca70 con 0x7f4ca8105a70 2024-04-16T03:02:27.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cae590700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 0x7f4ca80fbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:27.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cae590700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 0x7f4ca80fbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59920/0 (socket says 172.21.15.151:59920) 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cae590700 1 -- 172.21.15.151:0/1481368778 learned_addr learned my addr 172.21.15.151:0/1481368778 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cadd8f700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8105a70 0x7f4ca80fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cae590700 1 -- 172.21.15.151:0/1481368778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8105a70 msgr2=0x7f4ca80fc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cae590700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8105a70 0x7f4ca80fc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.091+0000 7f4cae590700 1 -- 172.21.15.151:0/1481368778 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c980095f0 con 0x7f4ca80ffa70 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.092+0000 7f4cae590700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 0x7f4ca80fbe80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c930 tx=0x7f4ca400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.092+0000 7f4c9f7fe700 1 -- 172.21.15.151:0/1481368778 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca4007ad0 con 0x7f4ca80ffa70 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.092+0000 7f4c9f7fe700 1 -- 172.21.15.151:0/1481368778 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ca400ce80 con 0x7f4ca80ffa70 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.092+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca80fccf0 con 0x7f4ca80ffa70 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.092+0000 7f4c9f7fe700 1 -- 172.21.15.151:0/1481368778 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca4018630 con 0x7f4ca80ffa70 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.092+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca81a2cd0 con 0x7f4ca80ffa70 2024-04-16T03:02:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.093+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca8109010 con 0x7f4ca80ffa70 2024-04-16T03:02:27.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.093+0000 7f4c9f7fe700 1 -- 172.21.15.151:0/1481368778 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4ca4007c30 con 0x7f4ca80ffa70 2024-04-16T03:02:27.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.094+0000 7f4c9f7fe700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4c9406ecc0 0x7f4c94071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:27.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.094+0000 7f4c9f7fe700 1 -- 172.21.15.151:0/1481368778 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4ca408ebd0 con 0x7f4ca80ffa70 2024-04-16T03:02:27.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.094+0000 7f4cadd8f700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4c9406ecc0 0x7f4c94071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:27.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.094+0000 7f4cadd8f700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4c9406ecc0 0x7f4c94071180 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80fd670 tx=0x7f4c9801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:27.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.099+0000 7f4c9f7fe700 1 -- 172.21.15.151:0/1481368778 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4ca4057a00 con 0x7f4ca80ffa70 2024-04-16T03:02:27.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:26 smithi153 ceph-mon[87858]: pgmap v715: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:27.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:26 smithi153 ceph-mon[87858]: from='client.35324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:02:27.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.340+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4ca80fcc30 con 0x7f4ca80ffa70 2024-04-16T03:02:27.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.341+0000 7f4c9f7fe700 1 -- 172.21.15.151:0/1481368778 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f4ca405b0c0 con 0x7f4ca80ffa70 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:02:27.343 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:02:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:02:27.345 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:02:27.346 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:02:27.346 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:02:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4c9406ecc0 msgr2=0x7f4c94071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4c9406ecc0 0x7f4c94071180 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80fd670 tx=0x7f4c9801a040 comp rx=0 tx=0).stop 2024-04-16T03:02:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 msgr2=0x7f4ca80fbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 0x7f4ca80fbe80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c930 tx=0x7f4ca400cc40 comp rx=0 tx=0).stop 2024-04-16T03:02:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 shutdown_connections 2024-04-16T03:02:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4c9406ecc0 0x7f4c94071180 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ca80ffa70 0x7f4ca80fbe80 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 --2- 172.21.15.151:0/1481368778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8105a70 0x7f4ca80fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 >> 172.21.15.151:0/1481368778 conn(0x7f4ca80fb560 msgr2=0x7f4ca80690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:27.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 shutdown_connections 2024-04-16T03:02:27.348 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.343+0000 7f4cb07f4700 1 -- 172.21.15.151:0/1481368778 wait complete. 2024-04-16T03:02:27.348 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.448+0000 7fb76db30700 1 -- 172.21.15.151:0/2167139931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768101780 msgr2=0x7fb7681057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.448+0000 7fb76db30700 1 --2- 172.21.15.151:0/2167139931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768101780 0x7fb7681057d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb750009910 tx=0x7fb750009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.449+0000 7fb76db30700 1 -- 172.21.15.151:0/2167139931 shutdown_connections 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.449+0000 7fb76db30700 1 --2- 172.21.15.151:0/2167139931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768101780 0x7fb7681057d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.449+0000 7fb76db30700 1 --2- 172.21.15.151:0/2167139931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb768100dd0 0x7fb7681011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.449+0000 7fb76db30700 1 -- 172.21.15.151:0/2167139931 >> 172.21.15.151:0/2167139931 conn(0x7fb7680fc640 msgr2=0x7fb7680fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.449+0000 7fb76db30700 1 -- 172.21.15.151:0/2167139931 shutdown_connections 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.449+0000 7fb76db30700 1 -- 172.21.15.151:0/2167139931 wait complete. 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.449+0000 7fb76db30700 1 Processor -- start 2024-04-16T03:02:27.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.449+0000 7fb76db30700 1 -- start start 2024-04-16T03:02:27.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb76db30700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768100dd0 0x7fb76819c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:27.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb76db30700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb768101780 0x7fb76819cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:27.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb76db30700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76819d160 con 0x7fb768100dd0 2024-04-16T03:02:27.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb76db30700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb768196610 con 0x7fb768101780 2024-04-16T03:02:27.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7677fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768100dd0 0x7fb76819c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:27.453 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7677fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768100dd0 0x7fb76819c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59932/0 (socket says 172.21.15.151:59932) 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7677fe700 1 -- 172.21.15.151:0/848467276 learned_addr learned my addr 172.21.15.151:0/848467276 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7677fe700 1 -- 172.21.15.151:0/848467276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb768101780 msgr2=0x7fb76819cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7677fe700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb768101780 0x7fb76819cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7677fe700 1 -- 172.21.15.151:0/848467276 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7500095f0 con 0x7fb768100dd0 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7677fe700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768100dd0 0x7fb76819c590 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb75800d900 tx=0x7fb75800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7657fa700 1 -- 172.21.15.151:0/848467276 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7580041f0 con 0x7fb768100dd0 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7657fa700 1 -- 172.21.15.151:0/848467276 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb758004cf0 con 0x7fb768100dd0 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7681968f0 con 0x7fb768100dd0 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb7657fa700 1 -- 172.21.15.151:0/848467276 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb75800b750 con 0x7fb768100dd0 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.450+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb768196e40 con 0x7fb768100dd0 2024-04-16T03:02:27.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.451+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7681091d0 con 0x7fb768100dd0 2024-04-16T03:02:27.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.452+0000 7fb7657fa700 1 -- 172.21.15.151:0/848467276 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb758004350 con 0x7fb768100dd0 2024-04-16T03:02:27.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.452+0000 7fb7657fa700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb75406ecc0 0x7fb754071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:27.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.453+0000 7fb7657fa700 1 -- 172.21.15.151:0/848467276 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb75808ef10 con 0x7fb768100dd0 2024-04-16T03:02:27.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.453+0000 7fb75ffff700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb75406ecc0 0x7fb754071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:27.456 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.453+0000 7fb75ffff700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb75406ecc0 0x7fb754071180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb75000b5c0 tx=0x7fb750005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:27.459 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.457+0000 7fb7657fa700 1 -- 172.21.15.151:0/848467276 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb75805b870 con 0x7fb768100dd0 2024-04-16T03:02:27.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.672+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7681970b0 con 0x7fb75406ecc0 2024-04-16T03:02:27.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.674+0000 7fb7657fa700 1 -- 172.21.15.151:0/848467276 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb7681970b0 con 0x7fb75406ecc0 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:02:27.676 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:02:27.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb75406ecc0 msgr2=0x7fb754071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb75406ecc0 0x7fb754071180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb75000b5c0 tx=0x7fb750005960 comp rx=0 tx=0).stop 2024-04-16T03:02:27.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768100dd0 msgr2=0x7fb76819c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768100dd0 0x7fb76819c590 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb75800d900 tx=0x7fb75800dcc0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 shutdown_connections 2024-04-16T03:02:27.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb75406ecc0 0x7fb754071180 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb768100dd0 0x7fb76819c590 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 --2- 172.21.15.151:0/848467276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb768101780 0x7fb76819cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 >> 172.21.15.151:0/848467276 conn(0x7fb7680fc640 msgr2=0x7fb7680fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:27.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 shutdown_connections 2024-04-16T03:02:27.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.676+0000 7fb76db30700 1 -- 172.21.15.151:0/848467276 wait complete. 2024-04-16T03:02:27.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.780+0000 7f707fb22700 1 -- 172.21.15.151:0/1070953521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7078105a50 msgr2=0x7f7078105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.780+0000 7f707fb22700 1 --2- 172.21.15.151:0/1070953521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7078105a50 0x7f7078105e30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7074009910 tx=0x7f7074009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:27.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.780+0000 7f707fb22700 1 -- 172.21.15.151:0/1070953521 shutdown_connections 2024-04-16T03:02:27.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.780+0000 7f707fb22700 1 --2- 172.21.15.151:0/1070953521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70780ffa50 0x7f70780ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.780+0000 7f707fb22700 1 --2- 172.21.15.151:0/1070953521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7078105a50 0x7f7078105e30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.780+0000 7f707fb22700 1 -- 172.21.15.151:0/1070953521 >> 172.21.15.151:0/1070953521 conn(0x7f70780fb540 msgr2=0x7f70780fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:27.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.780+0000 7f707fb22700 1 -- 172.21.15.151:0/1070953521 shutdown_connections 2024-04-16T03:02:27.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.780+0000 7f707fb22700 1 -- 172.21.15.151:0/1070953521 wait complete. 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707fb22700 1 Processor -- start 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707fb22700 1 -- start start 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707fb22700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70780ffa50 0x7f70781975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707fb22700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7078105a50 0x7f7078197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707fb22700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7078198110 con 0x7f70780ffa50 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707fb22700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7078198250 con 0x7f7078105a50 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707d8be700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70780ffa50 0x7f70781975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707d8be700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70780ffa50 0x7f70781975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59946/0 (socket says 172.21.15.151:59946) 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707d8be700 1 -- 172.21.15.151:0/2199655309 learned_addr learned my addr 172.21.15.151:0/2199655309 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.781+0000 7f707d0bd700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7078105a50 0x7f7078197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.782+0000 7f707d8be700 1 -- 172.21.15.151:0/2199655309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7078105a50 msgr2=0x7f7078197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:27.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.782+0000 7f707d8be700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7078105a50 0x7f7078197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:27.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.782+0000 7f707d8be700 1 -- 172.21.15.151:0/2199655309 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70740095f0 con 0x7f70780ffa50 2024-04-16T03:02:27.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.782+0000 7f707d8be700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70780ffa50 0x7f70781975b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7074006010 tx=0x7f707400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:27.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.782+0000 7f706effd700 1 -- 172.21.15.151:0/2199655309 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f707401d070 con 0x7f70780ffa50 2024-04-16T03:02:27.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.782+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70781a0620 con 0x7f70780ffa50 2024-04-16T03:02:27.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.782+0000 7f706effd700 1 -- 172.21.15.151:0/2199655309 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7074005dc0 con 0x7f70780ffa50 2024-04-16T03:02:27.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.782+0000 7f706effd700 1 -- 172.21.15.151:0/2199655309 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7074017400 con 0x7f70780ffa50 2024-04-16T03:02:27.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.782+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70781a0a30 con 0x7f70780ffa50 2024-04-16T03:02:27.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.783+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7078108050 con 0x7f70780ffa50 2024-04-16T03:02:27.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.783+0000 7f706effd700 1 -- 172.21.15.151:0/2199655309 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7074003a60 con 0x7f70780ffa50 2024-04-16T03:02:27.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.784+0000 7f706effd700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f706406ecc0 0x7f7064071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:27.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.784+0000 7f707d0bd700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f706406ecc0 0x7f7064071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:27.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.784+0000 7f706effd700 1 -- 172.21.15.151:0/2199655309 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f70740902f0 con 0x7f70780ffa50 2024-04-16T03:02:27.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.785+0000 7f707d0bd700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f706406ecc0 0x7f7064071180 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f70780696a0 tx=0x7f706800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:27.790 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:27.789+0000 7f706effd700 1 -- 172.21.15.151:0/2199655309 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f70740590a0 con 0x7f70780ffa50 2024-04-16T03:02:27.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:27 smithi151 ceph-mon[112233]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:02:27.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:27 smithi151 ceph-mon[112233]: from='client.35332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:02:27.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:27 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3724306604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:02:27.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:27 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1481368778' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:02:28.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.078+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7078198410 con 0x7f70780ffa50 2024-04-16T03:02:28.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.078+0000 7f706effd700 1 -- 172.21.15.151:0/2199655309 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f707405c760 con 0x7f70780ffa50 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:02:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:02:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:02:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:02:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:02:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:02:28.081 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:02:28.081 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:02:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:02:28.081 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:02:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:02:28.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f706406ecc0 msgr2=0x7f7064071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:28.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f706406ecc0 0x7f7064071180 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f70780696a0 tx=0x7f706800b410 comp rx=0 tx=0).stop 2024-04-16T03:02:28.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70780ffa50 msgr2=0x7f70781975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:28.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70780ffa50 0x7f70781975b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7074006010 tx=0x7f707400bd70 comp rx=0 tx=0).stop 2024-04-16T03:02:28.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 shutdown_connections 2024-04-16T03:02:28.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f706406ecc0 0x7f7064071180 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:28.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70780ffa50 0x7f70781975b0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:28.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 --2- 172.21.15.151:0/2199655309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7078105a50 0x7f7078197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:28.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 >> 172.21.15.151:0/2199655309 conn(0x7f70780fb540 msgr2=0x7f70780fbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:28.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 shutdown_connections 2024-04-16T03:02:28.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:28.080+0000 7f707fb22700 1 -- 172.21.15.151:0/2199655309 wait complete. 2024-04-16T03:02:28.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:27 smithi153 ceph-mon[87858]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:02:28.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:27 smithi153 ceph-mon[87858]: from='client.35332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:02:28.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:27 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3724306604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:02:28.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:27 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1481368778' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:02:29.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:28 smithi151 ceph-mon[112233]: pgmap v716: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:29.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:28 smithi151 ceph-mon[112233]: from='client.35344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:02:29.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:28 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2199655309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:02:29.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:28 smithi153 ceph-mon[87858]: pgmap v716: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:29.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:28 smithi153 ceph-mon[87858]: from='client.35344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:02:29.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:28 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2199655309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:02:31.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:30 smithi153 ceph-mon[87858]: pgmap v717: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:31.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:30 smithi151 ceph-mon[112233]: pgmap v717: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:33.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:32 smithi151 ceph-mon[112233]: pgmap v718: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:33.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:32 smithi153 ceph-mon[87858]: pgmap v718: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:34 smithi151 ceph-mon[112233]: pgmap v719: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:35.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:34 smithi153 ceph-mon[87858]: pgmap v719: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:37.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:36 smithi151 ceph-mon[112233]: pgmap v720: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:37.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:36 smithi153 ceph-mon[87858]: pgmap v720: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:38 smithi151 ceph-mon[112233]: pgmap v721: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:02:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:38 smithi153 ceph-mon[87858]: pgmap v721: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:02:41.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:40 smithi151 ceph-mon[112233]: pgmap v722: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:40 smithi153 ceph-mon[87858]: pgmap v722: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:43.159 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:42 smithi151 ceph-mon[112233]: pgmap v723: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:43.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:42 smithi153 ceph-mon[87858]: pgmap v723: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:45.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:44 smithi151 ceph-mon[112233]: pgmap v724: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:45.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:44 smithi153 ceph-mon[87858]: pgmap v724: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:47.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:46 smithi151 ceph-mon[112233]: pgmap v725: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:47.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:46 smithi153 ceph-mon[87858]: pgmap v725: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:48 smithi151 ceph-mon[112233]: pgmap v726: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:49.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:48 smithi153 ceph-mon[87858]: pgmap v726: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:51.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:50 smithi151 ceph-mon[112233]: pgmap v727: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:51.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:50 smithi153 ceph-mon[87858]: pgmap v727: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:52 smithi151 ceph-mon[112233]: pgmap v728: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:52 smithi153 ceph-mon[87858]: pgmap v728: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:02:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:02:55.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:54 smithi151 ceph-mon[112233]: pgmap v729: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:54 smithi153 ceph-mon[87858]: pgmap v729: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:57.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:56 smithi151 ceph-mon[112233]: pgmap v730: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:57.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:56 smithi153 ceph-mon[87858]: pgmap v730: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.198+0000 7f3e0600e700 1 -- 172.21.15.151:0/1124555164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00100dc0 msgr2=0x7f3e001011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.198+0000 7f3e0600e700 1 --2- 172.21.15.151:0/1124555164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00100dc0 0x7f3e001011a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009910 tx=0x7f3df0009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.198+0000 7f3e0600e700 1 -- 172.21.15.151:0/1124555164 shutdown_connections 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.198+0000 7f3e0600e700 1 --2- 172.21.15.151:0/1124555164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e00101770 0x7f3e001057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.198+0000 7f3e0600e700 1 --2- 172.21.15.151:0/1124555164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00100dc0 0x7f3e001011a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.198+0000 7f3e0600e700 1 -- 172.21.15.151:0/1124555164 >> 172.21.15.151:0/1124555164 conn(0x7f3e000fc650 msgr2=0x7f3e000fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.199+0000 7f3e0600e700 1 -- 172.21.15.151:0/1124555164 shutdown_connections 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.199+0000 7f3e0600e700 1 -- 172.21.15.151:0/1124555164 wait complete. 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.199+0000 7f3e0600e700 1 Processor -- start 2024-04-16T03:02:58.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.199+0000 7f3e0600e700 1 -- start start 2024-04-16T03:02:58.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3e0600e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e00100dc0 0x7f3e00073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:58.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3e0600e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00101770 0x7f3e000735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:58.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3e0600e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00073b10 con 0x7f3e00101770 2024-04-16T03:02:58.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3e0600e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00073c50 con 0x7f3e00100dc0 2024-04-16T03:02:58.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00101770 0x7f3e000735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:58.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00101770 0x7f3e000735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60068/0 (socket says 172.21.15.151:60068) 2024-04-16T03:02:58.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfeffd700 1 -- 172.21.15.151:0/568445610 learned_addr learned my addr 172.21.15.151:0/568445610 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:58.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dff7fe700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e00100dc0 0x7f3e00073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfeffd700 1 -- 172.21.15.151:0/568445610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e00100dc0 msgr2=0x7f3e00073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfeffd700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e00100dc0 0x7f3e00073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfeffd700 1 -- 172.21.15.151:0/568445610 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df00095f0 con 0x7f3e00101770 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dff7fe700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e00100dc0 0x7f3e00073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfeffd700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00101770 0x7f3e000735d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3df400d900 tx=0x7f3df400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfcff9700 1 -- 172.21.15.151:0/568445610 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df4009900 con 0x7f3e00101770 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e001a2940 con 0x7f3e00101770 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfcff9700 1 -- 172.21.15.151:0/568445610 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3df4010460 con 0x7f3e00101770 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3dfcff9700 1 -- 172.21.15.151:0/568445610 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df400f5f0 con 0x7f3e00101770 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.200+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e001a2d50 con 0x7f3e00101770 2024-04-16T03:02:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.201+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e00109160 con 0x7f3e00101770 2024-04-16T03:02:58.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.202+0000 7f3dfcff9700 1 -- 172.21.15.151:0/568445610 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3df40105d0 con 0x7f3e00101770 2024-04-16T03:02:58.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.202+0000 7f3dfcff9700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3de806ecc0 0x7f3de8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:58.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.203+0000 7f3dfcff9700 1 -- 172.21.15.151:0/568445610 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3df408ecc0 con 0x7f3e00101770 2024-04-16T03:02:58.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.203+0000 7f3dff7fe700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3de806ecc0 0x7f3de8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:58.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.203+0000 7f3dff7fe700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3de806ecc0 0x7f3de8071180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006010 tx=0x7f3df000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:58.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.207+0000 7f3dfcff9700 1 -- 172.21.15.151:0/568445610 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3df4057a70 con 0x7f3e00101770 2024-04-16T03:02:58.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.429+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e0002cda0 con 0x7f3de806ecc0 2024-04-16T03:02:58.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.430+0000 7f3dfcff9700 1 -- 172.21.15.151:0/568445610 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3e0002cda0 con 0x7f3de806ecc0 2024-04-16T03:02:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3de806ecc0 msgr2=0x7f3de8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3de806ecc0 0x7f3de8071180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006010 tx=0x7f3df000b540 comp rx=0 tx=0).stop 2024-04-16T03:02:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00101770 msgr2=0x7f3e000735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00101770 0x7f3e000735d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3df400d900 tx=0x7f3df400dcc0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 shutdown_connections 2024-04-16T03:02:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3de806ecc0 0x7f3de8071180 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e00100dc0 0x7f3e00073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 --2- 172.21.15.151:0/568445610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e00101770 0x7f3e000735d0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 >> 172.21.15.151:0/568445610 conn(0x7f3e000fc650 msgr2=0x7f3e000fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 shutdown_connections 2024-04-16T03:02:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.432+0000 7f3e0600e700 1 -- 172.21.15.151:0/568445610 wait complete. 2024-04-16T03:02:58.445 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:02:58.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.539+0000 7fe45e87c700 1 -- 172.21.15.151:0/4011621441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 msgr2=0x7fe458101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.539+0000 7fe45e87c700 1 --2- 172.21.15.151:0/4011621441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 0x7fe458101180 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe448009910 tx=0x7fe448009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:58.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.539+0000 7fe45e87c700 1 -- 172.21.15.151:0/4011621441 shutdown_connections 2024-04-16T03:02:58.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.539+0000 7fe45e87c700 1 --2- 172.21.15.151:0/4011621441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe458101750 0x7fe4581057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.539+0000 7fe45e87c700 1 --2- 172.21.15.151:0/4011621441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 0x7fe458101180 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.539+0000 7fe45e87c700 1 -- 172.21.15.151:0/4011621441 >> 172.21.15.151:0/4011621441 conn(0x7fe4580fc650 msgr2=0x7fe4580fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.539+0000 7fe45e87c700 1 -- 172.21.15.151:0/4011621441 shutdown_connections 2024-04-16T03:02:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.540+0000 7fe45e87c700 1 -- 172.21.15.151:0/4011621441 wait complete. 2024-04-16T03:02:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.540+0000 7fe45e87c700 1 Processor -- start 2024-04-16T03:02:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.540+0000 7fe45e87c700 1 -- start start 2024-04-16T03:02:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.540+0000 7fe45e87c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 0x7fe45819e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.540+0000 7fe45e87c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe458101750 0x7fe45819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.540+0000 7fe45e87c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45819f360 con 0x7fe458100da0 2024-04-16T03:02:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.540+0000 7fe45e87c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe458198810 con 0x7fe458101750 2024-04-16T03:02:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe457fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 0x7fe45819e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe457fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 0x7fe45819e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60076/0 (socket says 172.21.15.151:60076) 2024-04-16T03:02:58.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe457fff700 1 -- 172.21.15.151:0/299895959 learned_addr learned my addr 172.21.15.151:0/299895959 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe4577fe700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe458101750 0x7fe45819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe457fff700 1 -- 172.21.15.151:0/299895959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe458101750 msgr2=0x7fe45819ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe457fff700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe458101750 0x7fe45819ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe457fff700 1 -- 172.21.15.151:0/299895959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4480095f0 con 0x7fe458100da0 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe457fff700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 0x7fe45819e790 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe448006010 tx=0x7fe44800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe4557fa700 1 -- 172.21.15.151:0/299895959 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe44801d070 con 0x7fe458100da0 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe4557fa700 1 -- 172.21.15.151:0/299895959 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe44800f460 con 0x7fe458100da0 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe4557fa700 1 -- 172.21.15.151:0/299895959 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe448017630 con 0x7fe458100da0 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe458198a90 con 0x7fe458100da0 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.541+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe458198f30 con 0x7fe458100da0 2024-04-16T03:02:58.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.542+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe458104f20 con 0x7fe458100da0 2024-04-16T03:02:58.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.543+0000 7fe4557fa700 1 -- 172.21.15.151:0/299895959 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe44800f5d0 con 0x7fe458100da0 2024-04-16T03:02:58.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.543+0000 7fe4557fa700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe44006ecc0 0x7fe440071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:58.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.543+0000 7fe4557fa700 1 -- 172.21.15.151:0/299895959 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe448090310 con 0x7fe458100da0 2024-04-16T03:02:58.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.543+0000 7fe4577fe700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe44006ecc0 0x7fe440071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:58.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.544+0000 7fe4577fe700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe44006ecc0 0x7fe440071180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe44c005950 tx=0x7fe44c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:58.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.548+0000 7fe4557fa700 1 -- 172.21.15.151:0/299895959 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe44805cbf0 con 0x7fe458100da0 2024-04-16T03:02:58.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.769+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4581991a0 con 0x7fe44006ecc0 2024-04-16T03:02:58.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.771+0000 7fe4557fa700 1 -- 172.21.15.151:0/299895959 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe4581991a0 con 0x7fe44006ecc0 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe44006ecc0 msgr2=0x7fe440071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe44006ecc0 0x7fe440071180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe44c005950 tx=0x7fe44c0058e0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 msgr2=0x7fe45819e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 0x7fe45819e790 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe448006010 tx=0x7fe44800be20 comp rx=0 tx=0).stop 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 shutdown_connections 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe44006ecc0 0x7fe440071180 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe458100da0 0x7fe45819e790 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 --2- 172.21.15.151:0/299895959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe458101750 0x7fe45819ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 >> 172.21.15.151:0/299895959 conn(0x7fe4580fc650 msgr2=0x7fe4580fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 shutdown_connections 2024-04-16T03:02:58.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.773+0000 7fe45e87c700 1 -- 172.21.15.151:0/299895959 wait complete. 2024-04-16T03:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.879+0000 7f47ef263700 1 -- 172.21.15.151:0/1208666958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ff530 msgr2=0x7f47e80ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.879+0000 7f47ef263700 1 --2- 172.21.15.151:0/1208666958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ff530 0x7f47e80ff910 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009910 tx=0x7f47dc009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.879+0000 7f47ef263700 1 -- 172.21.15.151:0/1208666958 shutdown_connections 2024-04-16T03:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.879+0000 7f47ef263700 1 --2- 172.21.15.151:0/1208666958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47e80ffee0 0x7f47e8105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.879+0000 7f47ef263700 1 --2- 172.21.15.151:0/1208666958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ff530 0x7f47e80ff910 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.879+0000 7f47ef263700 1 -- 172.21.15.151:0/1208666958 >> 172.21.15.151:0/1208666958 conn(0x7f47e8074df0 msgr2=0x7f47e8075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.879+0000 7f47ef263700 1 -- 172.21.15.151:0/1208666958 shutdown_connections 2024-04-16T03:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.879+0000 7f47ef263700 1 -- 172.21.15.151:0/1208666958 wait complete. 2024-04-16T03:02:58.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47ef263700 1 Processor -- start 2024-04-16T03:02:58.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47ef263700 1 -- start start 2024-04-16T03:02:58.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47ef263700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47e80ff530 0x7f47e81004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:58.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47ef263700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ffee0 0x7f47e81009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:58.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47ef263700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e8100f30 con 0x7f47e80ffee0 2024-04-16T03:02:58.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47ef263700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e8101070 con 0x7f47e80ff530 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47e7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ffee0 0x7f47e81009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47e7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ffee0 0x7f47e81009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60094/0 (socket says 172.21.15.151:60094) 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47e7fff700 1 -- 172.21.15.151:0/3634506039 learned_addr learned my addr 172.21.15.151:0/3634506039 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47ecfff700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47e80ff530 0x7f47e81004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47e7fff700 1 -- 172.21.15.151:0/3634506039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47e80ff530 msgr2=0x7f47e81004b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47e7fff700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47e80ff530 0x7f47e81004b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47e7fff700 1 -- 172.21.15.151:0/3634506039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc0095f0 con 0x7f47e80ffee0 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.880+0000 7f47e7fff700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ffee0 0x7f47e81009f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f47d400d900 tx=0x7f47d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.881+0000 7f47e5ffb700 1 -- 172.21.15.151:0/3634506039 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47d4009900 con 0x7f47e80ffee0 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.881+0000 7f47e5ffb700 1 -- 172.21.15.151:0/3634506039 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47d4010460 con 0x7f47e80ffee0 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.881+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e81a2990 con 0x7f47e80ffee0 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.881+0000 7f47e5ffb700 1 -- 172.21.15.151:0/3634506039 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47d400f5f0 con 0x7f47e80ffee0 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.881+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e81a2da0 con 0x7f47e80ffee0 2024-04-16T03:02:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.882+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e81091e0 con 0x7f47e80ffee0 2024-04-16T03:02:58.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.882+0000 7f47e5ffb700 1 -- 172.21.15.151:0/3634506039 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f47d40105d0 con 0x7f47e80ffee0 2024-04-16T03:02:58.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.882+0000 7f47e5ffb700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47d806ecc0 0x7f47d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:58.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.883+0000 7f47e5ffb700 1 -- 172.21.15.151:0/3634506039 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47d408ecc0 con 0x7f47e80ffee0 2024-04-16T03:02:58.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.883+0000 7f47ecfff700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47d806ecc0 0x7f47d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:58.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.883+0000 7f47ecfff700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47d806ecc0 0x7f47d8071180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b5c0 tx=0x7f47dc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:58.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:58.887+0000 7f47e5ffb700 1 -- 172.21.15.151:0/3634506039 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f47d4057a70 con 0x7f47e80ffee0 2024-04-16T03:02:59.102 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.100+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f47e802cd30 con 0x7f47d806ecc0 2024-04-16T03:02:59.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.110+0000 7f47e5ffb700 1 -- 172.21.15.151:0/3634506039 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f47e802cd30 con 0x7f47d806ecc0 2024-04-16T03:02:59.113 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:02:59.113 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (24m) 2m ago 35m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:02:59.113 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (36m) 2m ago 36m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:02:59.113 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (35m) 2m ago 35m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:02:59.113 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (22m) 2m ago 36m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:02:59.113 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (22m) 2m ago 35m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (24m) 2m ago 35m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (18m) 2m ago 30m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (18m) 2m ago 30m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (2m) 2m ago 30m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (17m) 2m ago 30m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (26m) 2m ago 37m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (25m) 2m ago 34m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (23m) 2m ago 37m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (22m) 2m ago 34m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (25m) 2m ago 36m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (25m) 2m ago 34m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (21m) 2m ago 34m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (21m) 2m ago 33m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:02:59.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (20m) 2m ago 33m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:02:59.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (19m) 2m ago 32m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:02:59.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (19m) 2m ago 32m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:02:59.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (18m) 2m ago 31m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:02:59.115 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (24m) 2m ago 35m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47d806ecc0 msgr2=0x7f47d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47d806ecc0 0x7f47d8071180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b5c0 tx=0x7f47dc01a040 comp rx=0 tx=0).stop 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ffee0 msgr2=0x7f47e81009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ffee0 0x7f47e81009f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f47d400d900 tx=0x7f47d400dcc0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 shutdown_connections 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f47d806ecc0 0x7f47d8071180 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47e80ff530 0x7f47e81004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 --2- 172.21.15.151:0/3634506039 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47e80ffee0 0x7f47e81009f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 >> 172.21.15.151:0/3634506039 conn(0x7f47e8074df0 msgr2=0x7f47e80feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 shutdown_connections 2024-04-16T03:02:59.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.113+0000 7f47ef263700 1 -- 172.21.15.151:0/3634506039 wait complete. 2024-04-16T03:02:59.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.216+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2396116861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 msgr2=0x7f2e18102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.216+0000 7f2e1cb60700 1 --2- 172.21.15.151:0/2396116861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 0x7f2e18102110 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2e08009910 tx=0x7f2e08009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:59.219 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:58 smithi151 ceph-mon[112233]: pgmap v731: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:59.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.217+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2396116861 shutdown_connections 2024-04-16T03:02:59.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.217+0000 7f2e1cb60700 1 --2- 172.21.15.151:0/2396116861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e18102650 0x7f2e1810ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.217+0000 7f2e1cb60700 1 --2- 172.21.15.151:0/2396116861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 0x7f2e18102110 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.217+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2396116861 >> 172.21.15.151:0/2396116861 conn(0x7f2e180fb5a0 msgr2=0x7f2e180fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:59.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.217+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2396116861 shutdown_connections 2024-04-16T03:02:59.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.217+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2396116861 wait complete. 2024-04-16T03:02:59.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.218+0000 7f2e1cb60700 1 Processor -- start 2024-04-16T03:02:59.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.218+0000 7f2e1cb60700 1 -- start start 2024-04-16T03:02:59.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.218+0000 7f2e1cb60700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 0x7f2e1819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:59.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.218+0000 7f2e1cb60700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e18102650 0x7f2e1819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:59.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.218+0000 7f2e1cb60700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1819d0c0 con 0x7f2e18101d30 2024-04-16T03:02:59.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.218+0000 7f2e1cb60700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18196570 con 0x7f2e18102650 2024-04-16T03:02:59.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.218+0000 7f2e1659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 0x7f2e1819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:59.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.218+0000 7f2e1659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 0x7f2e1819c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60108/0 (socket says 172.21.15.151:60108) 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.218+0000 7f2e1659c700 1 -- 172.21.15.151:0/2564916114 learned_addr learned my addr 172.21.15.151:0/2564916114 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e15d9b700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e18102650 0x7f2e1819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e1659c700 1 -- 172.21.15.151:0/2564916114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e18102650 msgr2=0x7f2e1819ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e1659c700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e18102650 0x7f2e1819ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e1659c700 1 -- 172.21.15.151:0/2564916114 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e080095f0 con 0x7f2e18101d30 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e1659c700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 0x7f2e1819c4f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2e08006010 tx=0x7f2e0800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e077fe700 1 -- 172.21.15.151:0/2564916114 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0801d070 con 0x7f2e18101d30 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e181967f0 con 0x7f2e18101d30 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e077fe700 1 -- 172.21.15.151:0/2564916114 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e08004fb0 con 0x7f2e18101d30 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e077fe700 1 -- 172.21.15.151:0/2564916114 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0800f690 con 0x7f2e18101d30 2024-04-16T03:02:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.219+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e18196ce0 con 0x7f2e18101d30 2024-04-16T03:02:59.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.220+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e18108240 con 0x7f2e18101d30 2024-04-16T03:02:59.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.221+0000 7f2e077fe700 1 -- 172.21.15.151:0/2564916114 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2e080228a0 con 0x7f2e18101d30 2024-04-16T03:02:59.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.221+0000 7f2e077fe700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e0006ecc0 0x7f2e00071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:59.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.221+0000 7f2e077fe700 1 -- 172.21.15.151:0/2564916114 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2e080906d0 con 0x7f2e18101d30 2024-04-16T03:02:59.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.221+0000 7f2e15d9b700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e0006ecc0 0x7f2e00071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:59.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.222+0000 7f2e15d9b700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e0006ecc0 0x7f2e00071180 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c005fd0 tx=0x7f2e0c005dd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:59.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.226+0000 7f2e077fe700 1 -- 172.21.15.151:0/2564916114 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2e0805cfb0 con 0x7f2e18101d30 2024-04-16T03:02:59.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:58 smithi153 ceph-mon[87858]: pgmap v731: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:02:59.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.508+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2e1804ea90 con 0x7f2e18101d30 2024-04-16T03:02:59.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.508+0000 7f2e077fe700 1 -- 172.21.15.151:0/2564916114 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2e0805cb40 con 0x7f2e18101d30 2024-04-16T03:02:59.511 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:02:59.511 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:02:59.511 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:02:59.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:02:59.511 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:02:59.511 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:02:59.512 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:02:59.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.510+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e0006ecc0 msgr2=0x7f2e00071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.510+0000 7f2e1cb60700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e0006ecc0 0x7f2e00071180 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c005fd0 tx=0x7f2e0c005dd0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.510+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 msgr2=0x7f2e1819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.510+0000 7f2e1cb60700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 0x7f2e1819c4f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2e08006010 tx=0x7f2e0800be20 comp rx=0 tx=0).stop 2024-04-16T03:02:59.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.511+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 shutdown_connections 2024-04-16T03:02:59.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.511+0000 7f2e1cb60700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e0006ecc0 0x7f2e00071180 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.511+0000 7f2e1cb60700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e18101d30 0x7f2e1819c4f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.511+0000 7f2e1cb60700 1 --2- 172.21.15.151:0/2564916114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e18102650 0x7f2e1819ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.511+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 >> 172.21.15.151:0/2564916114 conn(0x7f2e180fb5a0 msgr2=0x7f2e18105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:59.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.511+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 shutdown_connections 2024-04-16T03:02:59.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.511+0000 7f2e1cb60700 1 -- 172.21.15.151:0/2564916114 wait complete. 2024-04-16T03:02:59.618 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.615+0000 7f974509f700 1 -- 172.21.15.151:0/391073303 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 msgr2=0x7f9740103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.618 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.615+0000 7f974509f700 1 --2- 172.21.15.151:0/391073303 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 0x7f9740103e80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9734009910 tx=0x7f9734009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:59.618 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.615+0000 7f974509f700 1 -- 172.21.15.151:0/391073303 shutdown_connections 2024-04-16T03:02:59.618 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.615+0000 7f974509f700 1 --2- 172.21.15.151:0/391073303 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 0x7f9740103e80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.618 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.615+0000 7f974509f700 1 --2- 172.21.15.151:0/391073303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97400ff500 0x7f97400ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.618 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.615+0000 7f974509f700 1 -- 172.21.15.151:0/391073303 >> 172.21.15.151:0/391073303 conn(0x7f9740074d80 msgr2=0x7f9740075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:59.618 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.615+0000 7f974509f700 1 -- 172.21.15.151:0/391073303 shutdown_connections 2024-04-16T03:02:59.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.615+0000 7f974509f700 1 -- 172.21.15.151:0/391073303 wait complete. 2024-04-16T03:02:59.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.616+0000 7f974509f700 1 Processor -- start 2024-04-16T03:02:59.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.616+0000 7f974509f700 1 -- start start 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f974509f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97400ff500 0x7f97400fccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f974509f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 0x7f97400fd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f974509f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97400fd750 con 0x7f97400ffeb0 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f974509f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97400fd890 con 0x7f97400ff500 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f973e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 0x7f97400fd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f973e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 0x7f97400fd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60138/0 (socket says 172.21.15.151:60138) 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f973e59c700 1 -- 172.21.15.151:0/2919687704 learned_addr learned my addr 172.21.15.151:0/2919687704 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f973ed9d700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97400ff500 0x7f97400fccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f973e59c700 1 -- 172.21.15.151:0/2919687704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97400ff500 msgr2=0x7f97400fccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f973e59c700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97400ff500 0x7f97400fccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f973e59c700 1 -- 172.21.15.151:0/2919687704 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97340095f0 con 0x7f97400ffeb0 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f973ed9d700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97400ff500 0x7f97400fccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:02:59.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f973e59c700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 0x7f97400fd210 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f973400bd70 tx=0x7f973400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:59.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f9727fff700 1 -- 172.21.15.151:0/2919687704 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f973401d070 con 0x7f97400ffeb0 2024-04-16T03:02:59.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f9727fff700 1 -- 172.21.15.151:0/2919687704 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9734005dc0 con 0x7f97400ffeb0 2024-04-16T03:02:59.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97400fdb10 con 0x7f97400ffeb0 2024-04-16T03:02:59.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.617+0000 7f9727fff700 1 -- 172.21.15.151:0/2919687704 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9734017400 con 0x7f97400ffeb0 2024-04-16T03:02:59.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.618+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97401a2d40 con 0x7f97400ffeb0 2024-04-16T03:02:59.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.618+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974004ea90 con 0x7f97400ffeb0 2024-04-16T03:02:59.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.619+0000 7f9727fff700 1 -- 172.21.15.151:0/2919687704 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f973400f7f0 con 0x7f97400ffeb0 2024-04-16T03:02:59.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.619+0000 7f9727fff700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f972806ecc0 0x7f9728071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:59.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.620+0000 7f9727fff700 1 -- 172.21.15.151:0/2919687704 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f97340902b0 con 0x7f97400ffeb0 2024-04-16T03:02:59.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.620+0000 7f973ed9d700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f972806ecc0 0x7f9728071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:59.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.620+0000 7f973ed9d700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f972806ecc0 0x7f9728071180 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9730005fd0 tx=0x7f9730005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:59.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.624+0000 7f9727fff700 1 -- 172.21.15.151:0/2919687704 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f97340590e0 con 0x7f97400ffeb0 2024-04-16T03:02:59.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.866+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f97400fe200 con 0x7f97400ffeb0 2024-04-16T03:02:59.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.867+0000 7f9727fff700 1 -- 172.21.15.151:0/2919687704 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f973405c7a0 con 0x7f97400ffeb0 2024-04-16T03:02:59.869 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:02:59.869 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:02:59.870 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:02:59.871 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:02:59.872 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:02:59.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.869+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f972806ecc0 msgr2=0x7f9728071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.869+0000 7f974509f700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f972806ecc0 0x7f9728071180 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9730005fd0 tx=0x7f9730005ee0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.869+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 msgr2=0x7f97400fd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.869+0000 7f974509f700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 0x7f97400fd210 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f973400bd70 tx=0x7f973400be50 comp rx=0 tx=0).stop 2024-04-16T03:02:59.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.870+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 shutdown_connections 2024-04-16T03:02:59.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.870+0000 7f974509f700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f972806ecc0 0x7f9728071180 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.870+0000 7f974509f700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97400ff500 0x7f97400fccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.870+0000 7f974509f700 1 --2- 172.21.15.151:0/2919687704 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97400ffeb0 0x7f97400fd210 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.870+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 >> 172.21.15.151:0/2919687704 conn(0x7f9740074d80 msgr2=0x7f9740102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:59.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.870+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 shutdown_connections 2024-04-16T03:02:59.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.870+0000 7f974509f700 1 -- 172.21.15.151:0/2919687704 wait complete. 2024-04-16T03:02:59.874 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:02:59.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 -- 172.21.15.151:0/138829808 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448101770 msgr2=0x7f24481057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 --2- 172.21.15.151:0/138829808 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448101770 0x7f24481057c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f243c009910 tx=0x7f243c009c20 comp rx=0 tx=0).stop 2024-04-16T03:02:59.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 -- 172.21.15.151:0/138829808 shutdown_connections 2024-04-16T03:02:59.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 --2- 172.21.15.151:0/138829808 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448101770 0x7f24481057c0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 --2- 172.21.15.151:0/138829808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2448100dc0 0x7f24481011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 -- 172.21.15.151:0/138829808 >> 172.21.15.151:0/138829808 conn(0x7f24480fc650 msgr2=0x7f24480fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:02:59.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 -- 172.21.15.151:0/138829808 shutdown_connections 2024-04-16T03:02:59.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 -- 172.21.15.151:0/138829808 wait complete. 2024-04-16T03:02:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 Processor -- start 2024-04-16T03:02:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.976+0000 7f244e29f700 1 -- start start 2024-04-16T03:02:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f244e29f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448100dc0 0x7f244819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f244e29f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2448101770 0x7f244819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f244e29f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244819d130 con 0x7f2448100dc0 2024-04-16T03:02:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f244e29f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24481965e0 con 0x7f2448101770 2024-04-16T03:02:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f2447fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448100dc0 0x7f244819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f2447fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448100dc0 0x7f244819c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60168/0 (socket says 172.21.15.151:60168) 2024-04-16T03:02:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f2447fff700 1 -- 172.21.15.151:0/2728340757 learned_addr learned my addr 172.21.15.151:0/2728340757 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f24477fe700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2448101770 0x7f244819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f2447fff700 1 -- 172.21.15.151:0/2728340757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2448101770 msgr2=0x7f244819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f2447fff700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2448101770 0x7f244819caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f2447fff700 1 -- 172.21.15.151:0/2728340757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243c0095f0 con 0x7f2448100dc0 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f2447fff700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448100dc0 0x7f244819c560 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f243800d8d0 tx=0x7f243800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f24457fa700 1 -- 172.21.15.151:0/2728340757 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2438009940 con 0x7f2448100dc0 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f24457fa700 1 -- 172.21.15.151:0/2728340757 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2438010460 con 0x7f2448100dc0 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24481968c0 con 0x7f2448100dc0 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.977+0000 7f24457fa700 1 -- 172.21.15.151:0/2728340757 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f243800f5f0 con 0x7f2448100dc0 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.978+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2448196e10 con 0x7f2448100dc0 2024-04-16T03:02:59.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.978+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24481091c0 con 0x7f2448100dc0 2024-04-16T03:02:59.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.979+0000 7f24457fa700 1 -- 172.21.15.151:0/2728340757 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f24380105d0 con 0x7f2448100dc0 2024-04-16T03:02:59.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.979+0000 7f24457fa700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f243006ecc0 0x7f2430071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:02:59.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.980+0000 7f24457fa700 1 -- 172.21.15.151:0/2728340757 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f243808ed70 con 0x7f2448100dc0 2024-04-16T03:02:59.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.980+0000 7f24477fe700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f243006ecc0 0x7f2430071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:02:59.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.980+0000 7f24477fe700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f243006ecc0 0x7f2430071180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f24480fcea0 tx=0x7f243c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:02:59.987 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:02:59.984+0000 7f24457fa700 1 -- 172.21.15.151:0/2728340757 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f243805b6d0 con 0x7f2448100dc0 2024-04-16T03:03:00.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.204+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2448197100 con 0x7f243006ecc0 2024-04-16T03:03:00.207 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:59 smithi151 ceph-mon[112233]: from='client.35352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:00.207 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:59 smithi151 ceph-mon[112233]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:00.207 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:59 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2564916114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:03:00.207 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:02:59 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2919687704' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:03:00.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.206+0000 7f24457fa700 1 -- 172.21.15.151:0/2728340757 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2448197100 con 0x7f243006ecc0 2024-04-16T03:03:00.208 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:03:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:03:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:03:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:03:00.209 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:03:00.211 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f243006ecc0 msgr2=0x7f2430071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f243006ecc0 0x7f2430071180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f24480fcea0 tx=0x7f243c01a040 comp rx=0 tx=0).stop 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448100dc0 msgr2=0x7f244819c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448100dc0 0x7f244819c560 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f243800d8d0 tx=0x7f243800dc90 comp rx=0 tx=0).stop 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 shutdown_connections 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f243006ecc0 0x7f2430071180 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2448100dc0 0x7f244819c560 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 --2- 172.21.15.151:0/2728340757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2448101770 0x7f244819caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 >> 172.21.15.151:0/2728340757 conn(0x7f24480fc650 msgr2=0x7f24480fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 shutdown_connections 2024-04-16T03:03:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.209+0000 7f244e29f700 1 -- 172.21.15.151:0/2728340757 wait complete. 2024-04-16T03:03:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:59 smithi153 ceph-mon[87858]: from='client.35352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:59 smithi153 ceph-mon[87858]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:59 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2564916114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:03:00.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:02:59 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2919687704' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:03:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.313+0000 7f6de90d1700 1 -- 172.21.15.151:0/1583605806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 msgr2=0x7f6de4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.313+0000 7f6de90d1700 1 --2- 172.21.15.151:0/1583605806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 0x7f6de4105e20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009910 tx=0x7f6dd4009c20 comp rx=0 tx=0).stop 2024-04-16T03:03:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.313+0000 7f6de90d1700 1 -- 172.21.15.151:0/1583605806 shutdown_connections 2024-04-16T03:03:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.313+0000 7f6de90d1700 1 --2- 172.21.15.151:0/1583605806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6de40ffa40 0x7f6de40ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.314+0000 7f6de90d1700 1 --2- 172.21.15.151:0/1583605806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 0x7f6de4105e20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.314+0000 7f6de90d1700 1 -- 172.21.15.151:0/1583605806 >> 172.21.15.151:0/1583605806 conn(0x7f6de40fb550 msgr2=0x7f6de40fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.314+0000 7f6de90d1700 1 -- 172.21.15.151:0/1583605806 shutdown_connections 2024-04-16T03:03:00.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.314+0000 7f6de90d1700 1 -- 172.21.15.151:0/1583605806 wait complete. 2024-04-16T03:03:00.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.314+0000 7f6de90d1700 1 Processor -- start 2024-04-16T03:03:00.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de90d1700 1 -- start start 2024-04-16T03:03:00.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de90d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6de40ffa40 0x7f6de419c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de90d1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 0x7f6de419cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de90d1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de419d0c0 con 0x7f6de4105a40 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de90d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de4196670 con 0x7f6de40ffa40 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 0x7f6de419cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 0x7f6de419cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60194/0 (socket says 172.21.15.151:60194) 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de259c700 1 -- 172.21.15.151:0/1634975175 learned_addr learned my addr 172.21.15.151:0/1634975175 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de2d9d700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6de40ffa40 0x7f6de419c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de259c700 1 -- 172.21.15.151:0/1634975175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6de40ffa40 msgr2=0x7f6de419c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de259c700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6de40ffa40 0x7f6de419c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de259c700 1 -- 172.21.15.151:0/1634975175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd40095f0 con 0x7f6de4105a40 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6de259c700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 0x7f6de419cb30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800c930 tx=0x7f6dd800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.315+0000 7f6dcbfff700 1 -- 172.21.15.151:0/1634975175 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd8007ad0 con 0x7f6de4105a40 2024-04-16T03:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.316+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de4196950 con 0x7f6de4105a40 2024-04-16T03:03:00.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.316+0000 7f6dcbfff700 1 -- 172.21.15.151:0/1634975175 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dd8007c30 con 0x7f6de4105a40 2024-04-16T03:03:00.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.316+0000 7f6dcbfff700 1 -- 172.21.15.151:0/1634975175 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd8018610 con 0x7f6de4105a40 2024-04-16T03:03:00.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.316+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de4196ea0 con 0x7f6de4105a40 2024-04-16T03:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.316+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de4109050 con 0x7f6de4105a40 2024-04-16T03:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.317+0000 7f6dcbfff700 1 -- 172.21.15.151:0/1634975175 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6dd800f450 con 0x7f6de4105a40 2024-04-16T03:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.317+0000 7f6dcbfff700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ecc0 0x7f6dcc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.317+0000 7f6dcbfff700 1 -- 172.21.15.151:0/1634975175 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6dd808f020 con 0x7f6de4105a40 2024-04-16T03:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.318+0000 7f6de2d9d700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ecc0 0x7f6dcc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.318+0000 7f6de2d9d700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ecc0 0x7f6dcc071180 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400b5c0 tx=0x7f6dd401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:00.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.322+0000 7f6dcbfff700 1 -- 172.21.15.151:0/1634975175 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6dd805b900 con 0x7f6de4105a40 2024-04-16T03:03:00.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.605+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6de404ea90 con 0x7f6de4105a40 2024-04-16T03:03:00.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.606+0000 7f6dcbfff700 1 -- 172.21.15.151:0/1634975175 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f6dd805ce60 con 0x7f6de4105a40 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:03:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:03:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:03:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:03:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:03:00.610 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:03:00.610 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:03:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:03:00.610 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:03:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.608+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ecc0 msgr2=0x7f6dcc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.608+0000 7f6de90d1700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ecc0 0x7f6dcc071180 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400b5c0 tx=0x7f6dd401a040 comp rx=0 tx=0).stop 2024-04-16T03:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.608+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 msgr2=0x7f6de419cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.608+0000 7f6de90d1700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 0x7f6de419cb30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800c930 tx=0x7f6dd800ccf0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.608+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 shutdown_connections 2024-04-16T03:03:00.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.608+0000 7f6de90d1700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ecc0 0x7f6dcc071180 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.608+0000 7f6de90d1700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6de40ffa40 0x7f6de419c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.608+0000 7f6de90d1700 1 --2- 172.21.15.151:0/1634975175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6de4105a40 0x7f6de419cb30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:00.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.608+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 >> 172.21.15.151:0/1634975175 conn(0x7f6de40fb550 msgr2=0x7f6de4069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:00.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.609+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 shutdown_connections 2024-04-16T03:03:00.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:00.609+0000 7f6de90d1700 1 -- 172.21.15.151:0/1634975175 wait complete. 2024-04-16T03:03:01.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:00 smithi151 ceph-mon[112233]: from='client.35360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:01.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:00 smithi151 ceph-mon[112233]: pgmap v732: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:01.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:00 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1634975175' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:03:01.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:00 smithi153 ceph-mon[87858]: from='client.35360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:01.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:00 smithi153 ceph-mon[87858]: pgmap v732: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:01.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:00 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1634975175' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:03:02.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:01 smithi151 ceph-mon[112233]: from='client.35372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:02.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:01 smithi153 ceph-mon[87858]: from='client.35372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:03.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:02 smithi151 ceph-mon[112233]: pgmap v733: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:03.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:02 smithi153 ceph-mon[87858]: pgmap v733: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:05.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:04 smithi151 ceph-mon[112233]: pgmap v734: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:05.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:04 smithi153 ceph-mon[87858]: pgmap v734: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:07.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:06 smithi151 ceph-mon[112233]: pgmap v735: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:07.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:06 smithi153 ceph-mon[87858]: pgmap v735: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:08 smithi151 ceph-mon[112233]: pgmap v736: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:03:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:08 smithi153 ceph-mon[87858]: pgmap v736: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:03:11.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:10 smithi153 ceph-mon[87858]: pgmap v737: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:11.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:10 smithi151 ceph-mon[112233]: pgmap v737: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:12 smithi151 ceph-mon[112233]: pgmap v738: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:12 smithi153 ceph-mon[87858]: pgmap v738: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:15.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:14 smithi151 ceph-mon[112233]: pgmap v739: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:15.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:14 smithi153 ceph-mon[87858]: pgmap v739: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:17.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:16 smithi151 ceph-mon[112233]: pgmap v740: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:17.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:16 smithi153 ceph-mon[87858]: pgmap v740: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:18 smithi151 ceph-mon[112233]: pgmap v741: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:19.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:18 smithi153 ceph-mon[87858]: pgmap v741: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:21.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:20 smithi151 ceph-mon[112233]: pgmap v742: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:20 smithi153 ceph-mon[87858]: pgmap v742: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:23.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:22 smithi151 ceph-mon[112233]: pgmap v743: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:23.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:22 smithi153 ceph-mon[87858]: pgmap v743: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:24.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:03:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:03:25.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:24 smithi151 ceph-mon[112233]: pgmap v744: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:24 smithi153 ceph-mon[87858]: pgmap v744: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:26 smithi151 ceph-mon[112233]: pgmap v745: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:27.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:26 smithi153 ceph-mon[87858]: pgmap v745: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:29.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:28 smithi151 ceph-mon[112233]: pgmap v746: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:29.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:28 smithi153 ceph-mon[87858]: pgmap v746: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:30.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.729+0000 7fa4be448700 1 -- 172.21.15.151:0/310736639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8100dc0 msgr2=0x7fa4b81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:30.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.729+0000 7fa4be448700 1 --2- 172.21.15.151:0/310736639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8100dc0 0x7fa4b81011a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009910 tx=0x7fa4a8009c20 comp rx=0 tx=0).stop 2024-04-16T03:03:30.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.729+0000 7fa4be448700 1 -- 172.21.15.151:0/310736639 shutdown_connections 2024-04-16T03:03:30.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.729+0000 7fa4be448700 1 --2- 172.21.15.151:0/310736639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4b8101770 0x7fa4b81057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:30.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.729+0000 7fa4be448700 1 --2- 172.21.15.151:0/310736639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8100dc0 0x7fa4b81011a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:30.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.729+0000 7fa4be448700 1 -- 172.21.15.151:0/310736639 >> 172.21.15.151:0/310736639 conn(0x7fa4b80fc650 msgr2=0x7fa4b80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:30.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.729+0000 7fa4be448700 1 -- 172.21.15.151:0/310736639 shutdown_connections 2024-04-16T03:03:30.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.729+0000 7fa4be448700 1 -- 172.21.15.151:0/310736639 wait complete. 2024-04-16T03:03:30.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4be448700 1 Processor -- start 2024-04-16T03:03:30.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4be448700 1 -- start start 2024-04-16T03:03:30.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4be448700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4b8100dc0 0x7fa4b819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:30.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4be448700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8101770 0x7fa4b819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:30.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4be448700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b819d130 con 0x7fa4b8101770 2024-04-16T03:03:30.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4be448700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b81965e0 con 0x7fa4b8100dc0 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4b77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8101770 0x7fa4b819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4b77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8101770 0x7fa4b819caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33500/0 (socket says 172.21.15.151:33500) 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4b77fe700 1 -- 172.21.15.151:0/1017459301 learned_addr learned my addr 172.21.15.151:0/1017459301 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.730+0000 7fa4b7fff700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4b8100dc0 0x7fa4b819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.731+0000 7fa4b77fe700 1 -- 172.21.15.151:0/1017459301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4b8100dc0 msgr2=0x7fa4b819c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.731+0000 7fa4b77fe700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4b8100dc0 0x7fa4b819c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.731+0000 7fa4b77fe700 1 -- 172.21.15.151:0/1017459301 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a80095f0 con 0x7fa4b8101770 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.731+0000 7fa4b77fe700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8101770 0x7fa4b819caa0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00e970 tx=0x7fa4ac00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.731+0000 7fa4b57fa700 1 -- 172.21.15.151:0/1017459301 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4ac00c9d0 con 0x7fa4b8101770 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.731+0000 7fa4b57fa700 1 -- 172.21.15.151:0/1017459301 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4ac004830 con 0x7fa4b8101770 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.731+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b81968c0 con 0x7fa4b8101770 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.731+0000 7fa4b57fa700 1 -- 172.21.15.151:0/1017459301 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4ac005660 con 0x7fa4b8101770 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.731+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b8196e10 con 0x7fa4b8101770 2024-04-16T03:03:30.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.732+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b8109160 con 0x7fa4b8101770 2024-04-16T03:03:30.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.732+0000 7fa4b57fa700 1 -- 172.21.15.151:0/1017459301 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa4ac00cb30 con 0x7fa4b8101770 2024-04-16T03:03:30.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.733+0000 7fa4b57fa700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4a006ecc0 0x7fa4a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:30.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.733+0000 7fa4b57fa700 1 -- 172.21.15.151:0/1017459301 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa4ac014070 con 0x7fa4b8101770 2024-04-16T03:03:30.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.733+0000 7fa4b7fff700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4a006ecc0 0x7fa4a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:30.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.734+0000 7fa4b7fff700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4a006ecc0 0x7fa4a0071180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80059d0 tx=0x7fa4a8005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:30.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.738+0000 7fa4b57fa700 1 -- 172.21.15.151:0/1017459301 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa4ac05b9f0 con 0x7fa4b8101770 2024-04-16T03:03:30.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.955+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4b8197160 con 0x7fa4a006ecc0 2024-04-16T03:03:30.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.957+0000 7fa4b57fa700 1 -- 172.21.15.151:0/1017459301 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa4b8197160 con 0x7fa4a006ecc0 2024-04-16T03:03:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4a006ecc0 msgr2=0x7fa4a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4a006ecc0 0x7fa4a0071180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80059d0 tx=0x7fa4a8005940 comp rx=0 tx=0).stop 2024-04-16T03:03:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8101770 msgr2=0x7fa4b819caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8101770 0x7fa4b819caa0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00e970 tx=0x7fa4ac00ec80 comp rx=0 tx=0).stop 2024-04-16T03:03:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 shutdown_connections 2024-04-16T03:03:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa4a006ecc0 0x7fa4a0071180 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4b8100dc0 0x7fa4b819c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 --2- 172.21.15.151:0/1017459301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4b8101770 0x7fa4b819caa0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:30.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 >> 172.21.15.151:0/1017459301 conn(0x7fa4b80fc650 msgr2=0x7fa4b80fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:30.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 shutdown_connections 2024-04-16T03:03:30.962 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:30.959+0000 7fa4be448700 1 -- 172.21.15.151:0/1017459301 wait complete. 2024-04-16T03:03:30.972 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:03:30.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:30 smithi151 ceph-mon[112233]: pgmap v747: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.066+0000 7fde0b035700 1 -- 172.21.15.151:0/3460384281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 msgr2=0x7fde04105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.066+0000 7fde0b035700 1 --2- 172.21.15.151:0/3460384281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 0x7fde04105e20 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009910 tx=0x7fddf8009c20 comp rx=0 tx=0).stop 2024-04-16T03:03:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.066+0000 7fde0b035700 1 -- 172.21.15.151:0/3460384281 shutdown_connections 2024-04-16T03:03:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.066+0000 7fde0b035700 1 --2- 172.21.15.151:0/3460384281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde040ffbd0 0x7fde04100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.066+0000 7fde0b035700 1 --2- 172.21.15.151:0/3460384281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 0x7fde04105e20 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.066+0000 7fde0b035700 1 -- 172.21.15.151:0/3460384281 >> 172.21.15.151:0/3460384281 conn(0x7fde040755a0 msgr2=0x7fde040759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.066+0000 7fde0b035700 1 -- 172.21.15.151:0/3460384281 shutdown_connections 2024-04-16T03:03:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.066+0000 7fde0b035700 1 -- 172.21.15.151:0/3460384281 wait complete. 2024-04-16T03:03:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde0b035700 1 Processor -- start 2024-04-16T03:03:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde0b035700 1 -- start start 2024-04-16T03:03:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde0b035700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde040ffbd0 0x7fde04101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde0b035700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 0x7fde04101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde0b035700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde04102090 con 0x7fde04105a40 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde0b035700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde04102200 con 0x7fde040ffbd0 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde03fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 0x7fde04101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde03fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 0x7fde04101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33528/0 (socket says 172.21.15.151:33528) 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde03fff700 1 -- 172.21.15.151:0/1017796103 learned_addr learned my addr 172.21.15.151:0/1017796103 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde08dd1700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde040ffbd0 0x7fde04101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde03fff700 1 -- 172.21.15.151:0/1017796103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde040ffbd0 msgr2=0x7fde04101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde03fff700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde040ffbd0 0x7fde04101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.067+0000 7fde03fff700 1 -- 172.21.15.151:0/1017796103 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf80095f0 con 0x7fde04105a40 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.068+0000 7fde03fff700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 0x7fde04101b50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c930 tx=0x7fddf000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:31.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.068+0000 7fde01ffb700 1 -- 172.21.15.151:0/1017796103 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddf0007ad0 con 0x7fde04105a40 2024-04-16T03:03:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.068+0000 7fde01ffb700 1 -- 172.21.15.151:0/1017796103 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddf0007c30 con 0x7fde04105a40 2024-04-16T03:03:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.068+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde04102480 con 0x7fde04105a40 2024-04-16T03:03:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.068+0000 7fde01ffb700 1 -- 172.21.15.151:0/1017796103 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddf0018610 con 0x7fde04105a40 2024-04-16T03:03:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.068+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde041a2c80 con 0x7fde04105a40 2024-04-16T03:03:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.069+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde04109050 con 0x7fde04105a40 2024-04-16T03:03:31.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.069+0000 7fde01ffb700 1 -- 172.21.15.151:0/1017796103 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fddf000f450 con 0x7fde04105a40 2024-04-16T03:03:31.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.070+0000 7fde01ffb700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddf406ecc0 0x7fddf4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:31.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.070+0000 7fde01ffb700 1 -- 172.21.15.151:0/1017796103 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fddf008eeb0 con 0x7fde04105a40 2024-04-16T03:03:31.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.070+0000 7fde08dd1700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddf406ecc0 0x7fddf4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:31.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.070+0000 7fde08dd1700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddf406ecc0 0x7fddf4071180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fddf8000c00 tx=0x7fddf8005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:31.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.075+0000 7fde01ffb700 1 -- 172.21.15.151:0/1017796103 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fddf0057c60 con 0x7fde04105a40 2024-04-16T03:03:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:30 smithi153 ceph-mon[87858]: pgmap v747: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:31.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.294+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde04102450 con 0x7fddf406ecc0 2024-04-16T03:03:31.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.295+0000 7fde01ffb700 1 -- 172.21.15.151:0/1017796103 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fde04102450 con 0x7fddf406ecc0 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddf406ecc0 msgr2=0x7fddf4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddf406ecc0 0x7fddf4071180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fddf8000c00 tx=0x7fddf8005fd0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 msgr2=0x7fde04101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 0x7fde04101b50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c930 tx=0x7fddf000ccf0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 shutdown_connections 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fddf406ecc0 0x7fddf4071180 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde040ffbd0 0x7fde04101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 --2- 172.21.15.151:0/1017796103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde04105a40 0x7fde04101b50 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 >> 172.21.15.151:0/1017796103 conn(0x7fde040755a0 msgr2=0x7fde040685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 shutdown_connections 2024-04-16T03:03:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.298+0000 7fde0b035700 1 -- 172.21.15.151:0/1017796103 wait complete. 2024-04-16T03:03:31.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 -- 172.21.15.151:0/4088858907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388101770 msgr2=0x7f03881057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 --2- 172.21.15.151:0/4088858907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388101770 0x7f03881057c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0374009910 tx=0x7f0374009c20 comp rx=0 tx=0).stop 2024-04-16T03:03:31.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 -- 172.21.15.151:0/4088858907 shutdown_connections 2024-04-16T03:03:31.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 --2- 172.21.15.151:0/4088858907 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388101770 0x7f03881057c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 --2- 172.21.15.151:0/4088858907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388100dc0 0x7f03881011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 -- 172.21.15.151:0/4088858907 >> 172.21.15.151:0/4088858907 conn(0x7f03880fc650 msgr2=0x7f03880fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:31.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 -- 172.21.15.151:0/4088858907 shutdown_connections 2024-04-16T03:03:31.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 -- 172.21.15.151:0/4088858907 wait complete. 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 Processor -- start 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.403+0000 7f038ec43700 1 -- start start 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038ec43700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388100dc0 0x7f038819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038ec43700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388101770 0x7f038819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038ec43700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038819d130 con 0x7f0388100dc0 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038ec43700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03881965e0 con 0x7f0388101770 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038c9df700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388100dc0 0x7f038819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038c9df700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388100dc0 0x7f038819c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33554/0 (socket says 172.21.15.151:33554) 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038c9df700 1 -- 172.21.15.151:0/3316851087 learned_addr learned my addr 172.21.15.151:0/3316851087 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f0387fff700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388101770 0x7f038819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038c9df700 1 -- 172.21.15.151:0/3316851087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388101770 msgr2=0x7f038819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038c9df700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388101770 0x7f038819caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038c9df700 1 -- 172.21.15.151:0/3316851087 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740095f0 con 0x7f0388100dc0 2024-04-16T03:03:31.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038c9df700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388100dc0 0x7f038819c560 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f037c00e920 tx=0x7f037c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:31.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f0385ffb700 1 -- 172.21.15.151:0/3316851087 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f037c00c9d0 con 0x7f0388100dc0 2024-04-16T03:03:31.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03881968c0 con 0x7f0388100dc0 2024-04-16T03:03:31.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.404+0000 7f0385ffb700 1 -- 172.21.15.151:0/3316851087 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f037c010430 con 0x7f0388100dc0 2024-04-16T03:03:31.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.405+0000 7f0385ffb700 1 -- 172.21.15.151:0/3316851087 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f037c018630 con 0x7f0388100dc0 2024-04-16T03:03:31.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.405+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0388196de0 con 0x7f0388100dc0 2024-04-16T03:03:31.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.406+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03881091c0 con 0x7f0388100dc0 2024-04-16T03:03:31.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.406+0000 7f0385ffb700 1 -- 172.21.15.151:0/3316851087 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f037c0105a0 con 0x7f0388100dc0 2024-04-16T03:03:31.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.407+0000 7f0385ffb700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f037806ecc0 0x7f0378071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:31.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.407+0000 7f0385ffb700 1 -- 172.21.15.151:0/3316851087 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f037c014070 con 0x7f0388100dc0 2024-04-16T03:03:31.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.407+0000 7f0387fff700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f037806ecc0 0x7f0378071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:31.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.407+0000 7f0387fff700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f037806ecc0 0x7f0378071180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f03740059d0 tx=0x7f0374005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:31.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.411+0000 7f0385ffb700 1 -- 172.21.15.151:0/3316851087 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f037c053ff0 con 0x7f0388100dc0 2024-04-16T03:03:31.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.617+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0388197030 con 0x7f037806ecc0 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.628+0000 7f0385ffb700 1 -- 172.21.15.151:0/3316851087 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0388197030 con 0x7f037806ecc0 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (25m) 3m ago 36m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (36m) 3m ago 36m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (35m) 3m ago 35m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (22m) 3m ago 36m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (22m) 3m ago 35m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (24m) 3m ago 36m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (18m) 3m ago 30m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (18m) 3m ago 30m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:03:31.630 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (3m) 3m ago 30m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (18m) 3m ago 30m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (27m) 3m ago 37m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (26m) 3m ago 35m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (23m) 3m ago 37m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (23m) 3m ago 35m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (25m) 3m ago 36m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (25m) 3m ago 35m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (22m) 3m ago 34m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (21m) 3m ago 34m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (20m) 3m ago 33m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (20m) 3m ago 33m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (19m) 3m ago 32m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (19m) 3m ago 32m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:03:31.631 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (25m) 3m ago 35m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:03:31.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.630+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f037806ecc0 msgr2=0x7f0378071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.630+0000 7f038ec43700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f037806ecc0 0x7f0378071180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f03740059d0 tx=0x7f0374005940 comp rx=0 tx=0).stop 2024-04-16T03:03:31.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.630+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388100dc0 msgr2=0x7f038819c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.630+0000 7f038ec43700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388100dc0 0x7f038819c560 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f037c00e920 tx=0x7f037c00ec30 comp rx=0 tx=0).stop 2024-04-16T03:03:31.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.630+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 shutdown_connections 2024-04-16T03:03:31.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.630+0000 7f038ec43700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f037806ecc0 0x7f0378071180 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.630+0000 7f038ec43700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0388100dc0 0x7f038819c560 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.630+0000 7f038ec43700 1 --2- 172.21.15.151:0/3316851087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388101770 0x7f038819caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.630+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 >> 172.21.15.151:0/3316851087 conn(0x7f03880fc650 msgr2=0x7f03880fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:31.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.631+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 shutdown_connections 2024-04-16T03:03:31.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.631+0000 7f038ec43700 1 -- 172.21.15.151:0/3316851087 wait complete. 2024-04-16T03:03:31.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.733+0000 7f67e212b700 1 -- 172.21.15.151:0/1296615547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 msgr2=0x7f67dc105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.733+0000 7f67e212b700 1 --2- 172.21.15.151:0/1296615547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 0x7f67dc105e20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009910 tx=0x7f67cc009c20 comp rx=0 tx=0).stop 2024-04-16T03:03:31.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.733+0000 7f67e212b700 1 -- 172.21.15.151:0/1296615547 shutdown_connections 2024-04-16T03:03:31.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.733+0000 7f67e212b700 1 --2- 172.21.15.151:0/1296615547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67dc0ffa40 0x7f67dc0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.733+0000 7f67e212b700 1 --2- 172.21.15.151:0/1296615547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 0x7f67dc105e20 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.733+0000 7f67e212b700 1 -- 172.21.15.151:0/1296615547 >> 172.21.15.151:0/1296615547 conn(0x7f67dc0fb550 msgr2=0x7f67dc0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:31.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.734+0000 7f67e212b700 1 -- 172.21.15.151:0/1296615547 shutdown_connections 2024-04-16T03:03:31.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.734+0000 7f67e212b700 1 -- 172.21.15.151:0/1296615547 wait complete. 2024-04-16T03:03:31.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.734+0000 7f67e212b700 1 Processor -- start 2024-04-16T03:03:31.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.734+0000 7f67e212b700 1 -- start start 2024-04-16T03:03:31.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67e212b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67dc0ffa40 0x7f67dc10a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:31.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67e212b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 0x7f67dc10af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:31.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67e212b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc10b450 con 0x7f67dc105a40 2024-04-16T03:03:31.737 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67e212b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc10b590 con 0x7f67dc0ffa40 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67daffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 0x7f67dc10af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67daffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 0x7f67dc10af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33562/0 (socket says 172.21.15.151:33562) 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67daffd700 1 -- 172.21.15.151:0/3383467305 learned_addr learned my addr 172.21.15.151:0/3383467305 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67db7fe700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67dc0ffa40 0x7f67dc10a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67daffd700 1 -- 172.21.15.151:0/3383467305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67dc0ffa40 msgr2=0x7f67dc10a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67daffd700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67dc0ffa40 0x7f67dc10a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67daffd700 1 -- 172.21.15.151:0/3383467305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc0095f0 con 0x7f67dc105a40 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67db7fe700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67dc0ffa40 0x7f67dc10a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.735+0000 7f67daffd700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 0x7f67dc10af10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f67d000c930 tx=0x7f67d000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.736+0000 7f67d8ff9700 1 -- 172.21.15.151:0/3383467305 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67d0007ad0 con 0x7f67dc105a40 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.736+0000 7f67d8ff9700 1 -- 172.21.15.151:0/3383467305 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67d0007c30 con 0x7f67dc105a40 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.736+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc1a2910 con 0x7f67dc105a40 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.736+0000 7f67d8ff9700 1 -- 172.21.15.151:0/3383467305 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67d0018610 con 0x7f67dc105a40 2024-04-16T03:03:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.736+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67dc1a2c70 con 0x7f67dc105a40 2024-04-16T03:03:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.737+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67dc108040 con 0x7f67dc105a40 2024-04-16T03:03:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.737+0000 7f67d8ff9700 1 -- 172.21.15.151:0/3383467305 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f67d000f450 con 0x7f67dc105a40 2024-04-16T03:03:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.738+0000 7f67d8ff9700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f67c406ecc0 0x7f67c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.738+0000 7f67d8ff9700 1 -- 172.21.15.151:0/3383467305 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f67d008ef60 con 0x7f67dc105a40 2024-04-16T03:03:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.738+0000 7f67db7fe700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f67c406ecc0 0x7f67c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:31.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.738+0000 7f67db7fe700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f67c406ecc0 0x7f67c4071180 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f67cc005b40 tx=0x7f67cc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:31.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:31.743+0000 7f67d8ff9700 1 -- 172.21.15.151:0/3383467305 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f67d00c3ff0 con 0x7f67dc105a40 2024-04-16T03:03:31.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:31 smithi151 ceph-mon[112233]: from='client.35380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:32.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.025+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f67dc04ea90 con 0x7f67dc105a40 2024-04-16T03:03:32.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.026+0000 7f67d8ff9700 1 -- 172.21.15.151:0/3383467305 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f67d0057d90 con 0x7f67dc105a40 2024-04-16T03:03:32.027 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:03:32.027 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:03:32.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:03:32.029 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f67c406ecc0 msgr2=0x7f67c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f67c406ecc0 0x7f67c4071180 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f67cc005b40 tx=0x7f67cc01a040 comp rx=0 tx=0).stop 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 msgr2=0x7f67dc10af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 0x7f67dc10af10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f67d000c930 tx=0x7f67d000ccf0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 shutdown_connections 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f67c406ecc0 0x7f67c4071180 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67dc0ffa40 0x7f67dc10a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 --2- 172.21.15.151:0/3383467305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f67dc105a40 0x7f67dc10af10 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 >> 172.21.15.151:0/3383467305 conn(0x7f67dc0fb550 msgr2=0x7f67dc0fbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 shutdown_connections 2024-04-16T03:03:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.028+0000 7f67e212b700 1 -- 172.21.15.151:0/3383467305 wait complete. 2024-04-16T03:03:32.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.133+0000 7fe925fac700 1 -- 172.21.15.151:0/1828005661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 msgr2=0x7fe92010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.133+0000 7fe925fac700 1 --2- 172.21.15.151:0/1828005661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 0x7fe92010ab30 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe914009910 tx=0x7fe914009c20 comp rx=0 tx=0).stop 2024-04-16T03:03:32.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.133+0000 7fe925fac700 1 -- 172.21.15.151:0/1828005661 shutdown_connections 2024-04-16T03:03:32.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.133+0000 7fe925fac700 1 --2- 172.21.15.151:0/1828005661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 0x7fe92010ab30 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.133+0000 7fe925fac700 1 --2- 172.21.15.151:0/1828005661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe920101c90 0x7fe920102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.133+0000 7fe925fac700 1 -- 172.21.15.151:0/1828005661 >> 172.21.15.151:0/1828005661 conn(0x7fe9200fb550 msgr2=0x7fe9200fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:32.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.133+0000 7fe925fac700 1 -- 172.21.15.151:0/1828005661 shutdown_connections 2024-04-16T03:03:32.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.133+0000 7fe925fac700 1 -- 172.21.15.151:0/1828005661 wait complete. 2024-04-16T03:03:32.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.134+0000 7fe925fac700 1 Processor -- start 2024-04-16T03:03:32.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.134+0000 7fe925fac700 1 -- start start 2024-04-16T03:03:32.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.134+0000 7fe925fac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe920101c90 0x7fe92019c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:32.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.134+0000 7fe925fac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 0x7fe92019ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:32.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.134+0000 7fe925fac700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92019d120 con 0x7fe920102640 2024-04-16T03:03:32.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.134+0000 7fe925fac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9201965d0 con 0x7fe920101c90 2024-04-16T03:03:32.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 0x7fe92019ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:32.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 0x7fe92019ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33584/0 (socket says 172.21.15.151:33584) 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91effd700 1 -- 172.21.15.151:0/2549994500 learned_addr learned my addr 172.21.15.151:0/2549994500 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91f7fe700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe920101c90 0x7fe92019c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91effd700 1 -- 172.21.15.151:0/2549994500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe920101c90 msgr2=0x7fe92019c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91effd700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe920101c90 0x7fe92019c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91effd700 1 -- 172.21.15.151:0/2549994500 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9140095f0 con 0x7fe920102640 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91effd700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 0x7fe92019ca90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe9140092e0 tx=0x7fe91400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91cff9700 1 -- 172.21.15.151:0/2549994500 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe91401d070 con 0x7fe920102640 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe920196850 con 0x7fe920102640 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91cff9700 1 -- 172.21.15.151:0/2549994500 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe914005dc0 con 0x7fe920102640 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe91cff9700 1 -- 172.21.15.151:0/2549994500 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe914017400 con 0x7fe920102640 2024-04-16T03:03:32.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.135+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe920196d40 con 0x7fe920102640 2024-04-16T03:03:32.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.136+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe920108230 con 0x7fe920102640 2024-04-16T03:03:32.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.137+0000 7fe91cff9700 1 -- 172.21.15.151:0/2549994500 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe91400f6f0 con 0x7fe920102640 2024-04-16T03:03:32.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.137+0000 7fe91cff9700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90806ecc0 0x7fe908071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:32.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.137+0000 7fe91cff9700 1 -- 172.21.15.151:0/2549994500 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe914090250 con 0x7fe920102640 2024-04-16T03:03:32.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.137+0000 7fe91f7fe700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90806ecc0 0x7fe908071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:32.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.138+0000 7fe91f7fe700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90806ecc0 0x7fe908071180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe910005950 tx=0x7fe91000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:32.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.142+0000 7fe91cff9700 1 -- 172.21.15.151:0/2549994500 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe91405cb30 con 0x7fe920102640 2024-04-16T03:03:32.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:31 smithi153 ceph-mon[87858]: from='client.35380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:32.387 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.385+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe92004ea90 con 0x7fe920102640 2024-04-16T03:03:32.387 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.386+0000 7fe91cff9700 1 -- 172.21.15.151:0/2549994500 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fe91405c6c0 con 0x7fe920102640 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:03:32.388 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:03:32.389 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:03:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:03:32.391 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:03:32.391 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:03:32.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90806ecc0 msgr2=0x7fe908071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90806ecc0 0x7fe908071180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe910005950 tx=0x7fe91000b410 comp rx=0 tx=0).stop 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 msgr2=0x7fe92019ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 0x7fe92019ca90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe9140092e0 tx=0x7fe91400bd70 comp rx=0 tx=0).stop 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 shutdown_connections 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90806ecc0 0x7fe908071180 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe920101c90 0x7fe92019c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 --2- 172.21.15.151:0/2549994500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe920102640 0x7fe92019ca90 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 >> 172.21.15.151:0/2549994500 conn(0x7fe9200fb550 msgr2=0x7fe920105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 shutdown_connections 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.388+0000 7fe925fac700 1 -- 172.21.15.151:0/2549994500 wait complete. 2024-04-16T03:03:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:03:32.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.493+0000 7fc11bd9b700 1 -- 172.21.15.151:0/906793754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114100dd0 msgr2=0x7fc1141011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.493+0000 7fc11bd9b700 1 --2- 172.21.15.151:0/906793754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114100dd0 0x7fc1141011b0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc110009910 tx=0x7fc110009c20 comp rx=0 tx=0).stop 2024-04-16T03:03:32.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.493+0000 7fc11bd9b700 1 -- 172.21.15.151:0/906793754 shutdown_connections 2024-04-16T03:03:32.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.493+0000 7fc11bd9b700 1 --2- 172.21.15.151:0/906793754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc114101780 0x7fc1141057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.493+0000 7fc11bd9b700 1 --2- 172.21.15.151:0/906793754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114100dd0 0x7fc1141011b0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.493+0000 7fc11bd9b700 1 -- 172.21.15.151:0/906793754 >> 172.21.15.151:0/906793754 conn(0x7fc1140fc640 msgr2=0x7fc1140fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:32.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.494+0000 7fc11bd9b700 1 -- 172.21.15.151:0/906793754 shutdown_connections 2024-04-16T03:03:32.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.494+0000 7fc11bd9b700 1 -- 172.21.15.151:0/906793754 wait complete. 2024-04-16T03:03:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.494+0000 7fc11bd9b700 1 Processor -- start 2024-04-16T03:03:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.494+0000 7fc11bd9b700 1 -- start start 2024-04-16T03:03:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc11bd9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc114100dd0 0x7fc11419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc11bd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114101780 0x7fc11419cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc11bd9b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11419d220 con 0x7fc114101780 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc11bd9b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1141966d0 con 0x7fc114100dd0 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc119336700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114101780 0x7fc11419cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc119336700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114101780 0x7fc11419cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33606/0 (socket says 172.21.15.151:33606) 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc119336700 1 -- 172.21.15.151:0/690488732 learned_addr learned my addr 172.21.15.151:0/690488732 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc119b37700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc114100dd0 0x7fc11419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc119336700 1 -- 172.21.15.151:0/690488732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc114100dd0 msgr2=0x7fc11419c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc119336700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc114100dd0 0x7fc11419c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc119336700 1 -- 172.21.15.151:0/690488732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1100095f0 con 0x7fc114101780 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc119336700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114101780 0x7fc11419cb90 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc10400b480 tx=0x7fc10400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc10affd700 1 -- 172.21.15.151:0/690488732 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc104011800 con 0x7fc114101780 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1141969b0 con 0x7fc114101780 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc10affd700 1 -- 172.21.15.151:0/690488732 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc104011e20 con 0x7fc114101780 2024-04-16T03:03:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc10affd700 1 -- 172.21.15.151:0/690488732 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc104010ea0 con 0x7fc114101780 2024-04-16T03:03:32.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.495+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc114196f00 con 0x7fc114101780 2024-04-16T03:03:32.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.496+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc114109170 con 0x7fc114101780 2024-04-16T03:03:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.497+0000 7fc10affd700 1 -- 172.21.15.151:0/690488732 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc104010380 con 0x7fc114101780 2024-04-16T03:03:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.497+0000 7fc10affd700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc10006ecc0 0x7fc100071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.497+0000 7fc10affd700 1 -- 172.21.15.151:0/690488732 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc10408e8d0 con 0x7fc114101780 2024-04-16T03:03:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.497+0000 7fc119b37700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc10006ecc0 0x7fc100071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.498+0000 7fc119b37700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc10006ecc0 0x7fc100071180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc110006010 tx=0x7fc11001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:32.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.502+0000 7fc10affd700 1 -- 172.21.15.151:0/690488732 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc10405b230 con 0x7fc114101780 2024-04-16T03:03:32.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.718+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1141971f0 con 0x7fc10006ecc0 2024-04-16T03:03:32.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.720+0000 7fc10affd700 1 -- 172.21.15.151:0/690488732 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc1141971f0 con 0x7fc10006ecc0 2024-04-16T03:03:32.721 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:03:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:03:32.722 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc10006ecc0 msgr2=0x7fc100071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc10006ecc0 0x7fc100071180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc110006010 tx=0x7fc11001a040 comp rx=0 tx=0).stop 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114101780 msgr2=0x7fc11419cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114101780 0x7fc11419cb90 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc10400b480 tx=0x7fc10400b790 comp rx=0 tx=0).stop 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 shutdown_connections 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc10006ecc0 0x7fc100071180 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc114100dd0 0x7fc11419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 --2- 172.21.15.151:0/690488732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc114101780 0x7fc11419cb90 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 >> 172.21.15.151:0/690488732 conn(0x7fc1140fc640 msgr2=0x7fc1140fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 shutdown_connections 2024-04-16T03:03:32.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.722+0000 7fc11bd9b700 1 -- 172.21.15.151:0/690488732 wait complete. 2024-04-16T03:03:32.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.826+0000 7f9c7ce62700 1 -- 172.21.15.151:0/4258099512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 msgr2=0x7f9c781057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.826+0000 7f9c7ce62700 1 --2- 172.21.15.151:0/4258099512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 0x7f9c781057a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009910 tx=0x7f9c6c009c20 comp rx=0 tx=0).stop 2024-04-16T03:03:32.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.827+0000 7f9c7ce62700 1 -- 172.21.15.151:0/4258099512 shutdown_connections 2024-04-16T03:03:32.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.827+0000 7f9c7ce62700 1 --2- 172.21.15.151:0/4258099512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 0x7f9c781057a0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.827+0000 7f9c7ce62700 1 --2- 172.21.15.151:0/4258099512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c78100da0 0x7f9c78101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.827+0000 7f9c7ce62700 1 -- 172.21.15.151:0/4258099512 >> 172.21.15.151:0/4258099512 conn(0x7f9c780fc650 msgr2=0x7f9c780fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:32.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.827+0000 7f9c7ce62700 1 -- 172.21.15.151:0/4258099512 shutdown_connections 2024-04-16T03:03:32.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.827+0000 7f9c7ce62700 1 -- 172.21.15.151:0/4258099512 wait complete. 2024-04-16T03:03:32.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.827+0000 7f9c7ce62700 1 Processor -- start 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.827+0000 7f9c7ce62700 1 -- start start 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c7ce62700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c78100da0 0x7f9c78073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c7ce62700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 0x7f9c780735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c7ce62700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c78073b10 con 0x7f9c78101750 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c7ce62700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c78073c50 con 0x7f9c78100da0 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c75d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 0x7f9c780735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c75d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 0x7f9c780735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33632/0 (socket says 172.21.15.151:33632) 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c75d9b700 1 -- 172.21.15.151:0/3419689049 learned_addr learned my addr 172.21.15.151:0/3419689049 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c7659c700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c78100da0 0x7f9c78073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c75d9b700 1 -- 172.21.15.151:0/3419689049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c78100da0 msgr2=0x7f9c78073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c75d9b700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c78100da0 0x7f9c78073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c75d9b700 1 -- 172.21.15.151:0/3419689049 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c0095f0 con 0x7f9c78101750 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c75d9b700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 0x7f9c780735d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0092e0 tx=0x7f9c6c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c677fe700 1 -- 172.21.15.151:0/3419689049 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c6c01d070 con 0x7f9c78101750 2024-04-16T03:03:32.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c78073ed0 con 0x7f9c78101750 2024-04-16T03:03:32.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c677fe700 1 -- 172.21.15.151:0/3419689049 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c6c00f460 con 0x7f9c78101750 2024-04-16T03:03:32.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c677fe700 1 -- 172.21.15.151:0/3419689049 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c6c0175f0 con 0x7f9c78101750 2024-04-16T03:03:32.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.828+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c781a2d50 con 0x7f9c78101750 2024-04-16T03:03:32.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.829+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c781090f0 con 0x7f9c78101750 2024-04-16T03:03:32.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.830+0000 7f9c677fe700 1 -- 172.21.15.151:0/3419689049 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9c6c00f5d0 con 0x7f9c78101750 2024-04-16T03:03:32.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.830+0000 7f9c677fe700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c6006ecc0 0x7f9c60071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:03:32.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.830+0000 7f9c677fe700 1 -- 172.21.15.151:0/3419689049 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9c6c0903b0 con 0x7f9c78101750 2024-04-16T03:03:32.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.830+0000 7f9c7659c700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c6006ecc0 0x7f9c60071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:03:32.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.831+0000 7f9c7659c700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c6006ecc0 0x7f9c60071180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9c68006fd0 tx=0x7f9c68008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:03:32.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:32.835+0000 7f9c677fe700 1 -- 172.21.15.151:0/3419689049 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9c6c059160 con 0x7f9c78101750 2024-04-16T03:03:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:32 smithi151 ceph-mon[112233]: pgmap v748: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:32 smithi151 ceph-mon[112233]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:32 smithi151 ceph-mon[112233]: from='client.35388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:32 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3383467305' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:03:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:32 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2549994500' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:03:33.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.117+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9c7804ea90 con 0x7f9c78101750 2024-04-16T03:03:33.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.118+0000 7f9c677fe700 1 -- 172.21.15.151:0/3419689049 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f9c6c05c820 con 0x7f9c78101750 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:03:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:03:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:03:33.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c6006ecc0 msgr2=0x7f9c60071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:33.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c6006ecc0 0x7f9c60071180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9c68006fd0 tx=0x7f9c68008040 comp rx=0 tx=0).stop 2024-04-16T03:03:33.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 msgr2=0x7f9c780735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:03:33.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 0x7f9c780735d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0092e0 tx=0x7f9c6c00be20 comp rx=0 tx=0).stop 2024-04-16T03:03:33.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 shutdown_connections 2024-04-16T03:03:33.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c6006ecc0 0x7f9c60071180 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:33.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c78100da0 0x7f9c78073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:33.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 --2- 172.21.15.151:0/3419689049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c78101750 0x7f9c780735d0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:03:33.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 >> 172.21.15.151:0/3419689049 conn(0x7f9c780fc650 msgr2=0x7f9c780fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:03:33.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 shutdown_connections 2024-04-16T03:03:33.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:03:33.120+0000 7f9c7ce62700 1 -- 172.21.15.151:0/3419689049 wait complete. 2024-04-16T03:03:33.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:32 smithi153 ceph-mon[87858]: pgmap v748: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:33.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:32 smithi153 ceph-mon[87858]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:33.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:32 smithi153 ceph-mon[87858]: from='client.35388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:33.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:32 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3383467305' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:03:33.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:32 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2549994500' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:03:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:33 smithi151 ceph-mon[112233]: from='client.35400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:34.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:33 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3419689049' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:03:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:33 smithi153 ceph-mon[87858]: from='client.35400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:03:34.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:33 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3419689049' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:03:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:34 smithi151 ceph-mon[112233]: pgmap v749: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:35.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:34 smithi153 ceph-mon[87858]: pgmap v749: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:37.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:36 smithi151 ceph-mon[112233]: pgmap v750: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:37.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:36 smithi153 ceph-mon[87858]: pgmap v750: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:38 smithi151 ceph-mon[112233]: pgmap v751: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:03:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:38 smithi153 ceph-mon[87858]: pgmap v751: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:03:41.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:40 smithi151 ceph-mon[112233]: pgmap v752: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:40 smithi153 ceph-mon[87858]: pgmap v752: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:42 smithi151 ceph-mon[112233]: pgmap v753: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:43.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:42 smithi153 ceph-mon[87858]: pgmap v753: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:45.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:44 smithi151 ceph-mon[112233]: pgmap v754: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:45.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:44 smithi153 ceph-mon[87858]: pgmap v754: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:47.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:46 smithi151 ceph-mon[112233]: pgmap v755: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:47.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:46 smithi153 ceph-mon[87858]: pgmap v755: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:48 smithi151 ceph-mon[112233]: pgmap v756: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:49.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:48 smithi153 ceph-mon[87858]: pgmap v756: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:51.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:50 smithi151 ceph-mon[112233]: pgmap v757: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:51.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:50 smithi153 ceph-mon[87858]: pgmap v757: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:52 smithi151 ceph-mon[112233]: pgmap v758: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:52 smithi153 ceph-mon[87858]: pgmap v758: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:53 smithi151 ceph-mon[112233]: pgmap v759: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:03:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:53 smithi153 ceph-mon[87858]: pgmap v759: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:03:56.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:56 smithi153 ceph-mon[87858]: pgmap v760: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:56.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:56 smithi151 ceph-mon[112233]: pgmap v760: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:58.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:03:58 smithi153 ceph-mon[87858]: pgmap v761: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:03:58.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:03:58 smithi151 ceph-mon[112233]: pgmap v761: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:00 smithi153 ceph-mon[87858]: pgmap v762: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:00.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:00 smithi151 ceph-mon[112233]: pgmap v762: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:02.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:02 smithi153 ceph-mon[87858]: pgmap v763: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:02.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:02 smithi151 ceph-mon[112233]: pgmap v763: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:03.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.239+0000 7fbefe44e700 1 -- 172.21.15.151:0/842324115 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef8105a60 msgr2=0x7fbef8105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.239+0000 7fbefe44e700 1 --2- 172.21.15.151:0/842324115 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef8105a60 0x7fbef8105e40 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009910 tx=0x7fbee8009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:03.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.239+0000 7fbefe44e700 1 -- 172.21.15.151:0/842324115 shutdown_connections 2024-04-16T03:04:03.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.239+0000 7fbefe44e700 1 --2- 172.21.15.151:0/842324115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbef80ffa60 0x7fbef80ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.239+0000 7fbefe44e700 1 --2- 172.21.15.151:0/842324115 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef8105a60 0x7fbef8105e40 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.239+0000 7fbefe44e700 1 -- 172.21.15.151:0/842324115 >> 172.21.15.151:0/842324115 conn(0x7fbef80fb550 msgr2=0x7fbef80fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:03.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.239+0000 7fbefe44e700 1 -- 172.21.15.151:0/842324115 shutdown_connections 2024-04-16T03:04:03.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.239+0000 7fbefe44e700 1 -- 172.21.15.151:0/842324115 wait complete. 2024-04-16T03:04:03.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbefe44e700 1 Processor -- start 2024-04-16T03:04:03.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbefe44e700 1 -- start start 2024-04-16T03:04:03.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbefe44e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef80ffa60 0x7fbef819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:03.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbefe44e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbef8105a60 0x7fbef819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbefe44e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef819d0e0 con 0x7fbef80ffa60 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbefe44e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef8196590 con 0x7fbef8105a60 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbef7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef80ffa60 0x7fbef819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbef7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef80ffa60 0x7fbef819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54334/0 (socket says 172.21.15.151:54334) 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbef7fff700 1 -- 172.21.15.151:0/2186103933 learned_addr learned my addr 172.21.15.151:0/2186103933 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.240+0000 7fbeeffff700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbef8105a60 0x7fbef819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.241+0000 7fbef7fff700 1 -- 172.21.15.151:0/2186103933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbef8105a60 msgr2=0x7fbef819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.241+0000 7fbef7fff700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbef8105a60 0x7fbef819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.241+0000 7fbef7fff700 1 -- 172.21.15.151:0/2186103933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee80095f0 con 0x7fbef80ffa60 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.241+0000 7fbef7fff700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef80ffa60 0x7fbef819c510 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000c00 tx=0x7fbee800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.241+0000 7fbef5ffb700 1 -- 172.21.15.151:0/2186103933 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbee801d070 con 0x7fbef80ffa60 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.241+0000 7fbef5ffb700 1 -- 172.21.15.151:0/2186103933 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbee8005dc0 con 0x7fbef80ffa60 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.241+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef8196810 con 0x7fbef80ffa60 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.241+0000 7fbef5ffb700 1 -- 172.21.15.151:0/2186103933 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbee8017400 con 0x7fbef80ffa60 2024-04-16T03:04:03.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.241+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef8196d00 con 0x7fbef80ffa60 2024-04-16T03:04:03.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.242+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef8109070 con 0x7fbef80ffa60 2024-04-16T03:04:03.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.243+0000 7fbef5ffb700 1 -- 172.21.15.151:0/2186103933 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fbee800f460 con 0x7fbef80ffa60 2024-04-16T03:04:03.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.243+0000 7fbef5ffb700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbee406ecc0 0x7fbee4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:03.246 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.243+0000 7fbef5ffb700 1 -- 172.21.15.151:0/2186103933 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbee8090280 con 0x7fbef80ffa60 2024-04-16T03:04:03.246 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.243+0000 7fbeeffff700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbee406ecc0 0x7fbee4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:03.246 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.244+0000 7fbeeffff700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbee406ecc0 0x7fbee4071180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbef81051e0 tx=0x7fbee000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:03.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.248+0000 7fbef5ffb700 1 -- 172.21.15.151:0/2186103933 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbee805cb60 con 0x7fbef80ffa60 2024-04-16T03:04:03.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.465+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbef802ccc0 con 0x7fbee406ecc0 2024-04-16T03:04:03.468 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.467+0000 7fbef5ffb700 1 -- 172.21.15.151:0/2186103933 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbef802ccc0 con 0x7fbee406ecc0 2024-04-16T03:04:03.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbee406ecc0 msgr2=0x7fbee4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.470 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbee406ecc0 0x7fbee4071180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbef81051e0 tx=0x7fbee000b410 comp rx=0 tx=0).stop 2024-04-16T03:04:03.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef80ffa60 msgr2=0x7fbef819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef80ffa60 0x7fbef819c510 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000c00 tx=0x7fbee800bd70 comp rx=0 tx=0).stop 2024-04-16T03:04:03.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 shutdown_connections 2024-04-16T03:04:03.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbee406ecc0 0x7fbee4071180 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbef80ffa60 0x7fbef819c510 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 --2- 172.21.15.151:0/2186103933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbef8105a60 0x7fbef819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 >> 172.21.15.151:0/2186103933 conn(0x7fbef80fb550 msgr2=0x7fbef8068fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:03.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 shutdown_connections 2024-04-16T03:04:03.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.469+0000 7fbefe44e700 1 -- 172.21.15.151:0/2186103933 wait complete. 2024-04-16T03:04:03.481 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.576+0000 7f949752d700 1 -- 172.21.15.151:0/1892720367 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 msgr2=0x7f94900ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.576+0000 7f949752d700 1 --2- 172.21.15.151:0/1892720367 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 0x7f94900ffec0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9480009910 tx=0x7f9480009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.576+0000 7f949752d700 1 -- 172.21.15.151:0/1892720367 shutdown_connections 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.576+0000 7f949752d700 1 --2- 172.21.15.151:0/1892720367 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 0x7f94900ffec0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.576+0000 7f949752d700 1 --2- 172.21.15.151:0/1892720367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9490105a40 0x7f9490105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.576+0000 7f949752d700 1 -- 172.21.15.151:0/1892720367 >> 172.21.15.151:0/1892720367 conn(0x7f94900fb550 msgr2=0x7f94900fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.576+0000 7f949752d700 1 -- 172.21.15.151:0/1892720367 shutdown_connections 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.576+0000 7f949752d700 1 -- 172.21.15.151:0/1892720367 wait complete. 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.576+0000 7f949752d700 1 Processor -- start 2024-04-16T03:04:03.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f949752d700 1 -- start start 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f949752d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 0x7f949019c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f949752d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9490105a40 0x7f949019ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f949752d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949019d0f0 con 0x7f94900ffa40 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f949752d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94901965a0 con 0x7f9490105a40 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f94952c9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 0x7f949019c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f94952c9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 0x7f949019c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54356/0 (socket says 172.21.15.151:54356) 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f94952c9700 1 -- 172.21.15.151:0/4160844105 learned_addr learned my addr 172.21.15.151:0/4160844105 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f9494ac8700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9490105a40 0x7f949019ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f94952c9700 1 -- 172.21.15.151:0/4160844105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9490105a40 msgr2=0x7f949019ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f94952c9700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9490105a40 0x7f949019ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f94952c9700 1 -- 172.21.15.151:0/4160844105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94800095f0 con 0x7f94900ffa40 2024-04-16T03:04:03.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f9494ac8700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9490105a40 0x7f949019ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:04:03.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f94952c9700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 0x7f949019c520 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f948c00d900 tx=0x7f948c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:03.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f94867fc700 1 -- 172.21.15.151:0/4160844105 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f948c0041f0 con 0x7f94900ffa40 2024-04-16T03:04:03.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9490196880 con 0x7f94900ffa40 2024-04-16T03:04:03.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.577+0000 7f94867fc700 1 -- 172.21.15.151:0/4160844105 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f948c004cf0 con 0x7f94900ffa40 2024-04-16T03:04:03.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.578+0000 7f94867fc700 1 -- 172.21.15.151:0/4160844105 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f948c00b750 con 0x7f94900ffa40 2024-04-16T03:04:03.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.578+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9490196dd0 con 0x7f94900ffa40 2024-04-16T03:04:03.581 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.578+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9490108fe0 con 0x7f94900ffa40 2024-04-16T03:04:03.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.579+0000 7f94867fc700 1 -- 172.21.15.151:0/4160844105 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f948c004350 con 0x7f94900ffa40 2024-04-16T03:04:03.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.579+0000 7f94867fc700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f947c06ecc0 0x7f947c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:03.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.580+0000 7f94867fc700 1 -- 172.21.15.151:0/4160844105 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f948c08ef10 con 0x7f94900ffa40 2024-04-16T03:04:03.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.580+0000 7f9494ac8700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f947c06ecc0 0x7f947c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:03.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.580+0000 7f9494ac8700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f947c06ecc0 0x7f947c071180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9480005b40 tx=0x7f948001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:03.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.584+0000 7f94867fc700 1 -- 172.21.15.151:0/4160844105 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f948c05b870 con 0x7f94900ffa40 2024-04-16T03:04:03.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.803+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f94901970c0 con 0x7f947c06ecc0 2024-04-16T03:04:03.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.804+0000 7f94867fc700 1 -- 172.21.15.151:0/4160844105 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f94901970c0 con 0x7f947c06ecc0 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.806+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f947c06ecc0 msgr2=0x7f947c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.806+0000 7f949752d700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f947c06ecc0 0x7f947c071180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9480005b40 tx=0x7f948001a040 comp rx=0 tx=0).stop 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.806+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 msgr2=0x7f949019c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.806+0000 7f949752d700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 0x7f949019c520 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f948c00d900 tx=0x7f948c00dcc0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.806+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 shutdown_connections 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.806+0000 7f949752d700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f947c06ecc0 0x7f947c071180 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.806+0000 7f949752d700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94900ffa40 0x7f949019c520 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.806+0000 7f949752d700 1 --2- 172.21.15.151:0/4160844105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9490105a40 0x7f949019ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.806+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 >> 172.21.15.151:0/4160844105 conn(0x7f94900fb550 msgr2=0x7f9490069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.807+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 shutdown_connections 2024-04-16T03:04:03.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.807+0000 7f949752d700 1 -- 172.21.15.151:0/4160844105 wait complete. 2024-04-16T03:04:03.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.911+0000 7faa953fa700 1 -- 172.21.15.151:0/3984796087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 msgr2=0x7faa90068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.911+0000 7faa953fa700 1 --2- 172.21.15.151:0/3984796087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 0x7faa90068a50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7faa78009910 tx=0x7faa78009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:03.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.911+0000 7faa953fa700 1 -- 172.21.15.151:0/3984796087 shutdown_connections 2024-04-16T03:04:03.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.911+0000 7faa953fa700 1 --2- 172.21.15.151:0/3984796087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa90069020 0x7faa90105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.911+0000 7faa953fa700 1 --2- 172.21.15.151:0/3984796087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 0x7faa90068a50 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.911+0000 7faa953fa700 1 -- 172.21.15.151:0/3984796087 >> 172.21.15.151:0/3984796087 conn(0x7faa900755a0 msgr2=0x7faa900759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:03.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.911+0000 7faa953fa700 1 -- 172.21.15.151:0/3984796087 shutdown_connections 2024-04-16T03:04:03.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.911+0000 7faa953fa700 1 -- 172.21.15.151:0/3984796087 wait complete. 2024-04-16T03:04:03.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa953fa700 1 Processor -- start 2024-04-16T03:04:03.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa953fa700 1 -- start start 2024-04-16T03:04:03.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa953fa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 0x7faa9019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:03.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa953fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa90069020 0x7faa9019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:03.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa953fa700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9019d0e0 con 0x7faa90068670 2024-04-16T03:04:03.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa953fa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa90196590 con 0x7faa90069020 2024-04-16T03:04:03.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa8effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 0x7faa9019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:03.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa8effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 0x7faa9019c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54370/0 (socket says 172.21.15.151:54370) 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa8effd700 1 -- 172.21.15.151:0/1416639174 learned_addr learned my addr 172.21.15.151:0/1416639174 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.912+0000 7faa8e7fc700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa90069020 0x7faa9019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.913+0000 7faa8effd700 1 -- 172.21.15.151:0/1416639174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa90069020 msgr2=0x7faa9019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.913+0000 7faa8effd700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa90069020 0x7faa9019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.913+0000 7faa8effd700 1 -- 172.21.15.151:0/1416639174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780095f0 con 0x7faa90068670 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.913+0000 7faa8effd700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 0x7faa9019c510 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7faa78006010 tx=0x7faa7800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.913+0000 7faa87fff700 1 -- 172.21.15.151:0/1416639174 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa7801d070 con 0x7faa90068670 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.913+0000 7faa87fff700 1 -- 172.21.15.151:0/1416639174 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa78005dc0 con 0x7faa90068670 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.913+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa90196810 con 0x7faa90068670 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.913+0000 7faa87fff700 1 -- 172.21.15.151:0/1416639174 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa78017400 con 0x7faa90068670 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.913+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa90196d00 con 0x7faa90068670 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.914+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa90109130 con 0x7faa90068670 2024-04-16T03:04:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.915+0000 7faa87fff700 1 -- 172.21.15.151:0/1416639174 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7faa78003a60 con 0x7faa90068670 2024-04-16T03:04:03.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.915+0000 7faa87fff700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa7c06ecc0 0x7faa7c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:03.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.915+0000 7faa87fff700 1 -- 172.21.15.151:0/1416639174 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faa780901b0 con 0x7faa90068670 2024-04-16T03:04:03.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.915+0000 7faa8e7fc700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa7c06ecc0 0x7faa7c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:03.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.916+0000 7faa8e7fc700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa7c06ecc0 0x7faa7c071180 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faa90107750 tx=0x7faa800073d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:03.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:03.920+0000 7faa87fff700 1 -- 172.21.15.151:0/1416639174 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7faa7805cb10 con 0x7faa90068670 2024-04-16T03:04:04.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.140+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faa9002ccc0 con 0x7faa7c06ecc0 2024-04-16T03:04:04.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.150+0000 7faa87fff700 1 -- 172.21.15.151:0/1416639174 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7faa9002ccc0 con 0x7faa7c06ecc0 2024-04-16T03:04:04.152 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:04:04.152 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (25m) 3m ago 36m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:04:04.152 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (37m) 3m ago 37m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:04:04.152 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (36m) 3m ago 36m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (23m) 3m ago 37m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (23m) 3m ago 36m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (25m) 3m ago 36m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (19m) 3m ago 31m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (19m) 3m ago 31m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (3m) 3m ago 31m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (18m) 3m ago 31m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (27m) 3m ago 38m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (26m) 3m ago 35m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (24m) 3m ago 38m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (23m) 3m ago 35m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (26m) 3m ago 37m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (26m) 3m ago 36m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:04:04.153 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (22m) 3m ago 35m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:04:04.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (22m) 3m ago 34m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:04:04.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (21m) 3m ago 34m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:04:04.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (20m) 3m ago 33m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:04:04.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (20m) 3m ago 33m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:04:04.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (19m) 3m ago 32m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:04:04.154 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (25m) 3m ago 36m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:04:04.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa7c06ecc0 msgr2=0x7faa7c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa7c06ecc0 0x7faa7c071180 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faa90107750 tx=0x7faa800073d0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 msgr2=0x7faa9019c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 0x7faa9019c510 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7faa78006010 tx=0x7faa7800bd70 comp rx=0 tx=0).stop 2024-04-16T03:04:04.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 shutdown_connections 2024-04-16T03:04:04.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa7c06ecc0 0x7faa7c071180 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa90068670 0x7faa9019c510 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 --2- 172.21.15.151:0/1416639174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa90069020 0x7faa9019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 >> 172.21.15.151:0/1416639174 conn(0x7faa900755a0 msgr2=0x7faa900fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:04.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 shutdown_connections 2024-04-16T03:04:04.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.153+0000 7faa953fa700 1 -- 172.21.15.151:0/1416639174 wait complete. 2024-04-16T03:04:04.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.257+0000 7f57d84ef700 1 -- 172.21.15.151:0/311699608 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d00ffbd0 msgr2=0x7f57d0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.257+0000 7f57d84ef700 1 --2- 172.21.15.151:0/311699608 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d00ffbd0 0x7f57d0100050 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009910 tx=0x7f57cc009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.257+0000 7f57d84ef700 1 -- 172.21.15.151:0/311699608 shutdown_connections 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.257+0000 7f57d84ef700 1 --2- 172.21.15.151:0/311699608 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d00ffbd0 0x7f57d0100050 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.257+0000 7f57d84ef700 1 --2- 172.21.15.151:0/311699608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d0105a40 0x7f57d0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.257+0000 7f57d84ef700 1 -- 172.21.15.151:0/311699608 >> 172.21.15.151:0/311699608 conn(0x7f57d00755a0 msgr2=0x7f57d00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.257+0000 7f57d84ef700 1 -- 172.21.15.151:0/311699608 shutdown_connections 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.257+0000 7f57d84ef700 1 -- 172.21.15.151:0/311699608 wait complete. 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d84ef700 1 Processor -- start 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d84ef700 1 -- start start 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d84ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d00ffbd0 0x7f57d0197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d84ef700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d0105a40 0x7f57d0197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d84ef700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0198150 con 0x7f57d0105a40 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d84ef700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0198290 con 0x7f57d00ffbd0 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d5a8a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d0105a40 0x7f57d0197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d5a8a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d0105a40 0x7f57d0197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54380/0 (socket says 172.21.15.151:54380) 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d5a8a700 1 -- 172.21.15.151:0/2474826835 learned_addr learned my addr 172.21.15.151:0/2474826835 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d628b700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d00ffbd0 0x7f57d0197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d5a8a700 1 -- 172.21.15.151:0/2474826835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d00ffbd0 msgr2=0x7f57d0197530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d5a8a700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d00ffbd0 0x7f57d0197530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.258+0000 7f57d5a8a700 1 -- 172.21.15.151:0/2474826835 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc0095f0 con 0x7f57d0105a40 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.259+0000 7f57d5a8a700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d0105a40 0x7f57d0197a70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f57cc006010 tx=0x7f57cc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.259+0000 7f57c77fe700 1 -- 172.21.15.151:0/2474826835 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57cc01d070 con 0x7f57d0105a40 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.259+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d01a05a0 con 0x7f57d0105a40 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.259+0000 7f57c77fe700 1 -- 172.21.15.151:0/2474826835 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57cc005dc0 con 0x7f57d0105a40 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.259+0000 7f57c77fe700 1 -- 172.21.15.151:0/2474826835 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57cc017400 con 0x7f57d0105a40 2024-04-16T03:04:04.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.259+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d01a0a60 con 0x7f57d0105a40 2024-04-16T03:04:04.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.260+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d0107fd0 con 0x7f57d0105a40 2024-04-16T03:04:04.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.260+0000 7f57c77fe700 1 -- 172.21.15.151:0/2474826835 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f57cc003a60 con 0x7f57d0105a40 2024-04-16T03:04:04.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.261+0000 7f57c77fe700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f57bc06ecc0 0x7f57bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:04.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.261+0000 7f57c77fe700 1 -- 172.21.15.151:0/2474826835 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f57cc0902f0 con 0x7f57d0105a40 2024-04-16T03:04:04.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.261+0000 7f57d628b700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f57bc06ecc0 0x7f57bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:04.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.261+0000 7f57d628b700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f57bc06ecc0 0x7f57bc071180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57c0005950 tx=0x7f57c000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:04.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.265+0000 7f57c77fe700 1 -- 172.21.15.151:0/2474826835 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f57cc0590a0 con 0x7f57d0105a40 2024-04-16T03:04:04.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.548+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f57d004ea90 con 0x7f57d0105a40 2024-04-16T03:04:04.550 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:04 smithi151 ceph-mon[112233]: pgmap v764: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:04.550 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:04 smithi151 ceph-mon[112233]: from='client.35408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:04.550 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:04 smithi151 ceph-mon[112233]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:04.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.549+0000 7f57c77fe700 1 -- 172.21.15.151:0/2474826835 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f57cc05c760 con 0x7f57d0105a40 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:04:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:04:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:04:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:04:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:04:04.552 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:04:04.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f57bc06ecc0 msgr2=0x7f57bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f57bc06ecc0 0x7f57bc071180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57c0005950 tx=0x7f57c000b410 comp rx=0 tx=0).stop 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d0105a40 msgr2=0x7f57d0197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d0105a40 0x7f57d0197a70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f57cc006010 tx=0x7f57cc00bd70 comp rx=0 tx=0).stop 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 shutdown_connections 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f57bc06ecc0 0x7f57bc071180 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d00ffbd0 0x7f57d0197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 --2- 172.21.15.151:0/2474826835 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f57d0105a40 0x7f57d0197a70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 >> 172.21.15.151:0/2474826835 conn(0x7f57d00755a0 msgr2=0x7f57d00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 shutdown_connections 2024-04-16T03:04:04.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.551+0000 7f57d84ef700 1 -- 172.21.15.151:0/2474826835 wait complete. 2024-04-16T03:04:04.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.655+0000 7f9536ca3700 1 -- 172.21.15.151:0/4087593322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530101770 msgr2=0x7f95301057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.655+0000 7f9536ca3700 1 --2- 172.21.15.151:0/4087593322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530101770 0x7f95301057c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f951c009910 tx=0x7f951c009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:04.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 -- 172.21.15.151:0/4087593322 shutdown_connections 2024-04-16T03:04:04.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 --2- 172.21.15.151:0/4087593322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530101770 0x7f95301057c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 --2- 172.21.15.151:0/4087593322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9530100dc0 0x7f95301011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 -- 172.21.15.151:0/4087593322 >> 172.21.15.151:0/4087593322 conn(0x7f95300fc650 msgr2=0x7f95300fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:04.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 -- 172.21.15.151:0/4087593322 shutdown_connections 2024-04-16T03:04:04.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 -- 172.21.15.151:0/4087593322 wait complete. 2024-04-16T03:04:04.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 Processor -- start 2024-04-16T03:04:04.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 -- start start 2024-04-16T03:04:04.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530100dc0 0x7f9530073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:04.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9530101770 0x7f95300735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9530073b10 con 0x7f9530100dc0 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.656+0000 7f9536ca3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9530073c50 con 0x7f9530101770 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f9534a3f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530100dc0 0x7f9530073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f9534a3f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530100dc0 0x7f9530073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54402/0 (socket says 172.21.15.151:54402) 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f9534a3f700 1 -- 172.21.15.151:0/3343407911 learned_addr learned my addr 172.21.15.151:0/3343407911 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f952ffff700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9530101770 0x7f95300735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f9534a3f700 1 -- 172.21.15.151:0/3343407911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9530101770 msgr2=0x7f95300735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f9534a3f700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9530101770 0x7f95300735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f9534a3f700 1 -- 172.21.15.151:0/3343407911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c0095f0 con 0x7f9530100dc0 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f9534a3f700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530100dc0 0x7f9530073090 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f952400b4b0 tx=0x7f952400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f952dffb700 1 -- 172.21.15.151:0/3343407911 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9524011800 con 0x7f9530100dc0 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95301a2a70 con 0x7f9530100dc0 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f952dffb700 1 -- 172.21.15.151:0/3343407911 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9524011e20 con 0x7f9530100dc0 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f952dffb700 1 -- 172.21.15.151:0/3343407911 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f952400f2e0 con 0x7f9530100dc0 2024-04-16T03:04:04.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.657+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95301a2e80 con 0x7f9530100dc0 2024-04-16T03:04:04.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.658+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95301090f0 con 0x7f9530100dc0 2024-04-16T03:04:04.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.659+0000 7f952dffb700 1 -- 172.21.15.151:0/3343407911 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9524010380 con 0x7f9530100dc0 2024-04-16T03:04:04.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.659+0000 7f952dffb700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f952006ecc0 0x7f9520071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:04.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.659+0000 7f952dffb700 1 -- 172.21.15.151:0/3343407911 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f952408ea40 con 0x7f9530100dc0 2024-04-16T03:04:04.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.659+0000 7f952ffff700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f952006ecc0 0x7f9520071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:04.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.660+0000 7f952ffff700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f952006ecc0 0x7f9520071180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9530074880 tx=0x7f951c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:04.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.664+0000 7f952dffb700 1 -- 172.21.15.151:0/3343407911 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f95240577f0 con 0x7f9530100dc0 2024-04-16T03:04:04.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:04 smithi153 ceph-mon[87858]: pgmap v764: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:04.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:04 smithi153 ceph-mon[87858]: from='client.35408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:04.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:04 smithi153 ceph-mon[87858]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:04.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.908+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f953002cdd0 con 0x7f9530100dc0 2024-04-16T03:04:04.910 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.908+0000 7f952dffb700 1 -- 172.21.15.151:0/3343407911 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f952405aeb0 con 0x7f9530100dc0 2024-04-16T03:04:04.910 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:04:04.910 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:04:04.910 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:04:04.910 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:04:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:04:04.910 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:04:04.911 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:04:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:04:04.913 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:04:04.913 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:04:04.913 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:04:04.913 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f952006ecc0 msgr2=0x7f9520071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f952006ecc0 0x7f9520071180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9530074880 tx=0x7f951c01a040 comp rx=0 tx=0).stop 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530100dc0 msgr2=0x7f9530073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530100dc0 0x7f9530073090 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f952400b4b0 tx=0x7f952400b870 comp rx=0 tx=0).stop 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 shutdown_connections 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f952006ecc0 0x7f9520071180 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9530100dc0 0x7f9530073090 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 --2- 172.21.15.151:0/3343407911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9530101770 0x7f95300735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 >> 172.21.15.151:0/3343407911 conn(0x7f95300fc650 msgr2=0x7f95300fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.911+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 shutdown_connections 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:04.912+0000 7f9536ca3700 1 -- 172.21.15.151:0/3343407911 wait complete. 2024-04-16T03:04:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:04:05.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.017+0000 7f00059ef700 1 -- 172.21.15.151:0/396306299 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 msgr2=0x7f0000102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:05.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.017+0000 7f00059ef700 1 --2- 172.21.15.151:0/396306299 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 0x7f0000102180 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7efff0009910 tx=0x7efff0009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:05.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.017+0000 7f00059ef700 1 -- 172.21.15.151:0/396306299 shutdown_connections 2024-04-16T03:04:05.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.017+0000 7f00059ef700 1 --2- 172.21.15.151:0/396306299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00001026c0 0x7f000010abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.017+0000 7f00059ef700 1 --2- 172.21.15.151:0/396306299 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 0x7f0000102180 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.017+0000 7f00059ef700 1 -- 172.21.15.151:0/396306299 >> 172.21.15.151:0/396306299 conn(0x7f00000fb570 msgr2=0x7f00000fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:05.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.018+0000 7f00059ef700 1 -- 172.21.15.151:0/396306299 shutdown_connections 2024-04-16T03:04:05.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.018+0000 7f00059ef700 1 -- 172.21.15.151:0/396306299 wait complete. 2024-04-16T03:04:05.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.018+0000 7f00059ef700 1 Processor -- start 2024-04-16T03:04:05.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.018+0000 7f00059ef700 1 -- start start 2024-04-16T03:04:05.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.018+0000 7f00059ef700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 0x7f00000ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:05.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7f00059ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00001026c0 0x7f00000fff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7f00059ef700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0000100480 con 0x7f0000101da0 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7f00059ef700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00001005c0 con 0x7f00001026c0 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7efffeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 0x7f00000ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7efffeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 0x7f00000ffa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54424/0 (socket says 172.21.15.151:54424) 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7efffeffd700 1 -- 172.21.15.151:0/2025867957 learned_addr learned my addr 172.21.15.151:0/2025867957 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7efffe7fc700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00001026c0 0x7f00000fff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7efffeffd700 1 -- 172.21.15.151:0/2025867957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00001026c0 msgr2=0x7f00000fff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7efffeffd700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00001026c0 0x7f00000fff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7efffeffd700 1 -- 172.21.15.151:0/2025867957 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff00095f0 con 0x7f0000101da0 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7efffeffd700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 0x7f00000ffa00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7efff000bd70 tx=0x7efff000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7f00049ed700 1 -- 172.21.15.151:0/2025867957 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff001d070 con 0x7f0000101da0 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0000100840 con 0x7f0000101da0 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7f00049ed700 1 -- 172.21.15.151:0/2025867957 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efff000f460 con 0x7f0000101da0 2024-04-16T03:04:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7f00049ed700 1 -- 172.21.15.151:0/2025867957 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff0017630 con 0x7f0000101da0 2024-04-16T03:04:05.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.019+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00001a2c40 con 0x7f0000101da0 2024-04-16T03:04:05.023 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.021+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00001082b0 con 0x7f0000101da0 2024-04-16T03:04:05.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.021+0000 7f00049ed700 1 -- 172.21.15.151:0/2025867957 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efff0003a60 con 0x7f0000101da0 2024-04-16T03:04:05.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.021+0000 7f00049ed700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7effe806ecc0 0x7effe8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:05.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.021+0000 7f00049ed700 1 -- 172.21.15.151:0/2025867957 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efff00903c0 con 0x7f0000101da0 2024-04-16T03:04:05.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.021+0000 7efffe7fc700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7effe806ecc0 0x7effe8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:05.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.022+0000 7efffe7fc700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7effe806ecc0 0x7effe8071180 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efff4005fd0 tx=0x7efff4005f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:05.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.026+0000 7f00049ed700 1 -- 172.21.15.151:0/2025867957 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efff0059230 con 0x7f0000101da0 2024-04-16T03:04:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.243+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f000002cce0 con 0x7effe806ecc0 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.244+0000 7f00049ed700 1 -- 172.21.15.151:0/2025867957 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f000002cce0 con 0x7effe806ecc0 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:04:05.246 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:04:05.247 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:04:05.247 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:04:05.247 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:04:05.247 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:04:05.247 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:04:05.247 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:04:05.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.247+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7effe806ecc0 msgr2=0x7effe8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:05.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.247+0000 7f00059ef700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7effe806ecc0 0x7effe8071180 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efff4005fd0 tx=0x7efff4005f60 comp rx=0 tx=0).stop 2024-04-16T03:04:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.247+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 msgr2=0x7f00000ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.247+0000 7f00059ef700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 0x7f00000ffa00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7efff000bd70 tx=0x7efff000be50 comp rx=0 tx=0).stop 2024-04-16T03:04:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.247+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 shutdown_connections 2024-04-16T03:04:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.247+0000 7f00059ef700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7effe806ecc0 0x7effe8071180 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.247+0000 7f00059ef700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000101da0 0x7f00000ffa00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.247+0000 7f00059ef700 1 --2- 172.21.15.151:0/2025867957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00001026c0 0x7f00000fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.248+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 >> 172.21.15.151:0/2025867957 conn(0x7f00000fb570 msgr2=0x7f00001053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.248+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 shutdown_connections 2024-04-16T03:04:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.248+0000 7f00059ef700 1 -- 172.21.15.151:0/2025867957 wait complete. 2024-04-16T03:04:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.355+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2661209804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 msgr2=0x7fc3cc0ffce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.355+0000 7fc3d19e9700 1 --2- 172.21.15.151:0/2661209804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 0x7fc3cc0ffce0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc009910 tx=0x7fc3bc009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.355+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2661209804 shutdown_connections 2024-04-16T03:04:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.355+0000 7fc3d19e9700 1 --2- 172.21.15.151:0/2661209804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3cc100220 0x7fc3cc10cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.355+0000 7fc3d19e9700 1 --2- 172.21.15.151:0/2661209804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 0x7fc3cc0ffce0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.355+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2661209804 >> 172.21.15.151:0/2661209804 conn(0x7fc3cc0fb520 msgr2=0x7fc3cc0fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.355+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2661209804 shutdown_connections 2024-04-16T03:04:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.355+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2661209804 wait complete. 2024-04-16T03:04:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3d19e9700 1 Processor -- start 2024-04-16T03:04:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3d19e9700 1 -- start start 2024-04-16T03:04:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3d19e9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 0x7fc3cc19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3d19e9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3cc100220 0x7fc3cc19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3d19e9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc19f230 con 0x7fc3cc0ff900 2024-04-16T03:04:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3d19e9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc198770 con 0x7fc3cc100220 2024-04-16T03:04:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3caffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 0x7fc3cc19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3caffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 0x7fc3cc19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54448/0 (socket says 172.21.15.151:54448) 2024-04-16T03:04:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3caffd700 1 -- 172.21.15.151:0/2975823603 learned_addr learned my addr 172.21.15.151:0/2975823603 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.356+0000 7fc3ca7fc700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3cc100220 0x7fc3cc19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.357+0000 7fc3caffd700 1 -- 172.21.15.151:0/2975823603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3cc100220 msgr2=0x7fc3cc19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.357+0000 7fc3caffd700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3cc100220 0x7fc3cc19ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.357+0000 7fc3caffd700 1 -- 172.21.15.151:0/2975823603 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc0095f0 con 0x7fc3cc0ff900 2024-04-16T03:04:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.357+0000 7fc3caffd700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 0x7fc3cc19e6f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc006010 tx=0x7fc3bc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.357+0000 7fc3d09e7700 1 -- 172.21.15.151:0/2975823603 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3bc01d070 con 0x7fc3cc0ff900 2024-04-16T03:04:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.357+0000 7fc3d09e7700 1 -- 172.21.15.151:0/2975823603 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3bc00f460 con 0x7fc3cc0ff900 2024-04-16T03:04:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.357+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3cc1989f0 con 0x7fc3cc0ff900 2024-04-16T03:04:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.357+0000 7fc3d09e7700 1 -- 172.21.15.151:0/2975823603 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3bc017630 con 0x7fc3cc0ff900 2024-04-16T03:04:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.357+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3cc198ee0 con 0x7fc3cc0ff900 2024-04-16T03:04:05.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.358+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3cc10a6e0 con 0x7fc3cc0ff900 2024-04-16T03:04:05.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.359+0000 7fc3d09e7700 1 -- 172.21.15.151:0/2975823603 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc3bc00f5d0 con 0x7fc3cc0ff900 2024-04-16T03:04:05.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.359+0000 7fc3d09e7700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc3b406ecc0 0x7fc3b4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:05.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.359+0000 7fc3ca7fc700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc3b406ecc0 0x7fc3b4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:05.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.359+0000 7fc3d09e7700 1 -- 172.21.15.151:0/2975823603 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc3bc0903b0 con 0x7fc3cc0ff900 2024-04-16T03:04:05.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.360+0000 7fc3ca7fc700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc3b406ecc0 0x7fc3b4071180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc199be0 tx=0x7fc3c000a3b0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:05.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.364+0000 7fc3d09e7700 1 -- 172.21.15.151:0/2975823603 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc3bc05cc90 con 0x7fc3cc0ff900 2024-04-16T03:04:05.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.647+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc3cc04ea90 con 0x7fc3cc0ff900 2024-04-16T03:04:05.648 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:05 smithi151 ceph-mon[112233]: from='client.35416 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:05.649 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:05 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2474826835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:04:05.649 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:05 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3343407911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:04:05.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.647+0000 7fc3d09e7700 1 -- 172.21.15.151:0/2975823603 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7fc3bc05c820 con 0x7fc3cc0ff900 2024-04-16T03:04:05.649 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:04:05.649 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:04:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:04:05.649 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:04:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:04:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:04:05.651 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:04:05.651 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:04:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:04:05.651 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:04:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc3b406ecc0 msgr2=0x7fc3b4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc3b406ecc0 0x7fc3b4071180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc199be0 tx=0x7fc3c000a3b0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 msgr2=0x7fc3cc19e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 0x7fc3cc19e6f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc006010 tx=0x7fc3bc00bd70 comp rx=0 tx=0).stop 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 shutdown_connections 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc3b406ecc0 0x7fc3b4071180 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3cc0ff900 0x7fc3cc19e6f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 --2- 172.21.15.151:0/2975823603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3cc100220 0x7fc3cc19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 >> 172.21.15.151:0/2975823603 conn(0x7fc3cc0fb520 msgr2=0x7fc3cc0fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:05.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.649+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 shutdown_connections 2024-04-16T03:04:05.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:05.650+0000 7fc3d19e9700 1 -- 172.21.15.151:0/2975823603 wait complete. 2024-04-16T03:04:05.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:05 smithi153 ceph-mon[87858]: from='client.35416 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:05.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:05 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2474826835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:04:05.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:05 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3343407911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:04:06.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:06 smithi151 ceph-mon[112233]: pgmap v765: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:06.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:06 smithi151 ceph-mon[112233]: from='client.35428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:06.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:06 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2975823603' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:04:06.768 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:06 smithi153 ceph-mon[87858]: pgmap v765: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:06.768 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:06 smithi153 ceph-mon[87858]: from='client.35428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:06.768 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:06 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2975823603' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:04:08.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:08 smithi151 ceph-mon[112233]: pgmap v766: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:08.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:08 smithi153 ceph-mon[87858]: pgmap v766: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:09.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:04:09.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:04:10.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:10 smithi151 ceph-mon[112233]: pgmap v767: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:10.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:10 smithi153 ceph-mon[87858]: pgmap v767: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:12.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:12 smithi151 ceph-mon[112233]: pgmap v768: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:12.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:12 smithi153 ceph-mon[87858]: pgmap v768: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:14.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:14 smithi151 ceph-mon[112233]: pgmap v769: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:14.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:14 smithi153 ceph-mon[87858]: pgmap v769: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:16.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:16 smithi151 ceph-mon[112233]: pgmap v770: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:16.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:16 smithi153 ceph-mon[87858]: pgmap v770: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:18.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:18 smithi151 ceph-mon[112233]: pgmap v771: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:18.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:18 smithi153 ceph-mon[87858]: pgmap v771: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:20.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:20 smithi151 ceph-mon[112233]: pgmap v772: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:20.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:20 smithi153 ceph-mon[87858]: pgmap v772: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:22.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:22 smithi151 ceph-mon[112233]: pgmap v773: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:22.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:22 smithi153 ceph-mon[87858]: pgmap v773: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:24.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:24 smithi151 ceph-mon[112233]: pgmap v774: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:24.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:04:24.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:24 smithi153 ceph-mon[87858]: pgmap v774: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:24.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:04:26.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:26 smithi151 ceph-mon[112233]: pgmap v775: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:26.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:26 smithi153 ceph-mon[87858]: pgmap v775: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:28.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:28 smithi151 ceph-mon[112233]: pgmap v776: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:28.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:28 smithi153 ceph-mon[87858]: pgmap v776: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:30.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:30 smithi151 ceph-mon[112233]: pgmap v777: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:30.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:30 smithi153 ceph-mon[87858]: pgmap v777: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:32.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:32 smithi151 ceph-mon[112233]: pgmap v778: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:32.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:32 smithi153 ceph-mon[87858]: pgmap v778: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:34.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:34 smithi151 ceph-mon[112233]: pgmap v779: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:34.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:34 smithi153 ceph-mon[87858]: pgmap v779: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:35.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.767+0000 7fdf5a363700 1 -- 172.21.15.151:0/2712298264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54100dc0 msgr2=0x7fdf541011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:35.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.767+0000 7fdf5a363700 1 --2- 172.21.15.151:0/2712298264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54100dc0 0x7fdf541011a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009910 tx=0x7fdf44009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.767+0000 7fdf5a363700 1 -- 172.21.15.151:0/2712298264 shutdown_connections 2024-04-16T03:04:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.767+0000 7fdf5a363700 1 --2- 172.21.15.151:0/2712298264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf54101770 0x7fdf541057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.767+0000 7fdf5a363700 1 --2- 172.21.15.151:0/2712298264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54100dc0 0x7fdf541011a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.767+0000 7fdf5a363700 1 -- 172.21.15.151:0/2712298264 >> 172.21.15.151:0/2712298264 conn(0x7fdf540fc650 msgr2=0x7fdf540fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.767+0000 7fdf5a363700 1 -- 172.21.15.151:0/2712298264 shutdown_connections 2024-04-16T03:04:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.767+0000 7fdf5a363700 1 -- 172.21.15.151:0/2712298264 wait complete. 2024-04-16T03:04:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf5a363700 1 Processor -- start 2024-04-16T03:04:35.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf5a363700 1 -- start start 2024-04-16T03:04:35.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf5a363700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf54100dc0 0x7fdf54073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:35.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf5a363700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54101770 0x7fdf540735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:35.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf5a363700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf54073b10 con 0x7fdf54101770 2024-04-16T03:04:35.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf5a363700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf54073c50 con 0x7fdf54100dc0 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf537fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54101770 0x7fdf540735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf537fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54101770 0x7fdf540735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56784/0 (socket says 172.21.15.151:56784) 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf537fe700 1 -- 172.21.15.151:0/172388781 learned_addr learned my addr 172.21.15.151:0/172388781 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.768+0000 7fdf53fff700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf54100dc0 0x7fdf54073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.769+0000 7fdf537fe700 1 -- 172.21.15.151:0/172388781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf54100dc0 msgr2=0x7fdf54073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.769+0000 7fdf537fe700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf54100dc0 0x7fdf54073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.769+0000 7fdf537fe700 1 -- 172.21.15.151:0/172388781 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf440095f0 con 0x7fdf54101770 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.769+0000 7fdf537fe700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54101770 0x7fdf540735d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800d900 tx=0x7fdf4800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.769+0000 7fdf517fa700 1 -- 172.21.15.151:0/172388781 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf48009900 con 0x7fdf54101770 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.769+0000 7fdf517fa700 1 -- 172.21.15.151:0/172388781 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf48010460 con 0x7fdf54101770 2024-04-16T03:04:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.769+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf541a2940 con 0x7fdf54101770 2024-04-16T03:04:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.769+0000 7fdf517fa700 1 -- 172.21.15.151:0/172388781 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf4800f5f0 con 0x7fdf54101770 2024-04-16T03:04:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.769+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf541a2d50 con 0x7fdf54101770 2024-04-16T03:04:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.770+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf54109160 con 0x7fdf54101770 2024-04-16T03:04:35.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.771+0000 7fdf517fa700 1 -- 172.21.15.151:0/172388781 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fdf480105d0 con 0x7fdf54101770 2024-04-16T03:04:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.771+0000 7fdf517fa700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf3c06ecc0 0x7fdf3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.771+0000 7fdf517fa700 1 -- 172.21.15.151:0/172388781 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdf4808ea00 con 0x7fdf54101770 2024-04-16T03:04:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.771+0000 7fdf53fff700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf3c06ecc0 0x7fdf3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.772+0000 7fdf53fff700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf3c06ecc0 0x7fdf3c071180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdf44006010 tx=0x7fdf4401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:35.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.776+0000 7fdf517fa700 1 -- 172.21.15.151:0/172388781 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdf480577b0 con 0x7fdf54101770 2024-04-16T03:04:35.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.993+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf5402cd50 con 0x7fdf3c06ecc0 2024-04-16T03:04:35.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.994+0000 7fdf517fa700 1 -- 172.21.15.151:0/172388781 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdf5402cd50 con 0x7fdf3c06ecc0 2024-04-16T03:04:35.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.996+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf3c06ecc0 msgr2=0x7fdf3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:35.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.996+0000 7fdf5a363700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf3c06ecc0 0x7fdf3c071180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdf44006010 tx=0x7fdf4401a040 comp rx=0 tx=0).stop 2024-04-16T03:04:35.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.996+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54101770 msgr2=0x7fdf540735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:35.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.996+0000 7fdf5a363700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54101770 0x7fdf540735d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800d900 tx=0x7fdf4800dc10 comp rx=0 tx=0).stop 2024-04-16T03:04:35.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.996+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 shutdown_connections 2024-04-16T03:04:35.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.996+0000 7fdf5a363700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdf3c06ecc0 0x7fdf3c071180 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:35.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.996+0000 7fdf5a363700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf54100dc0 0x7fdf54073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:35.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.996+0000 7fdf5a363700 1 --2- 172.21.15.151:0/172388781 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf54101770 0x7fdf540735d0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:35.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.997+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 >> 172.21.15.151:0/172388781 conn(0x7fdf540fc650 msgr2=0x7fdf540fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:35.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.997+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 shutdown_connections 2024-04-16T03:04:35.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:35.997+0000 7fdf5a363700 1 -- 172.21.15.151:0/172388781 wait complete. 2024-04-16T03:04:36.008 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:04:36.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.106+0000 7f94ffaa8700 1 -- 172.21.15.151:0/1790216139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 msgr2=0x7f94f8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.106+0000 7f94ffaa8700 1 --2- 172.21.15.151:0/1790216139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 0x7f94f8100050 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009910 tx=0x7f94ec009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:36.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.106+0000 7f94ffaa8700 1 -- 172.21.15.151:0/1790216139 shutdown_connections 2024-04-16T03:04:36.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.106+0000 7f94ffaa8700 1 --2- 172.21.15.151:0/1790216139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 0x7f94f8100050 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.106+0000 7f94ffaa8700 1 --2- 172.21.15.151:0/1790216139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94f8105a40 0x7f94f8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.106+0000 7f94ffaa8700 1 -- 172.21.15.151:0/1790216139 >> 172.21.15.151:0/1790216139 conn(0x7f94f80755a0 msgr2=0x7f94f80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:36.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.106+0000 7f94ffaa8700 1 -- 172.21.15.151:0/1790216139 shutdown_connections 2024-04-16T03:04:36.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.106+0000 7f94ffaa8700 1 -- 172.21.15.151:0/1790216139 wait complete. 2024-04-16T03:04:36.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94ffaa8700 1 Processor -- start 2024-04-16T03:04:36.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94ffaa8700 1 -- start start 2024-04-16T03:04:36.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94ffaa8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 0x7f94f8101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:36.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94ffaa8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94f8105a40 0x7f94f8101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:36.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94ffaa8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f8102090 con 0x7f94f80ffbd0 2024-04-16T03:04:36.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94ffaa8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f81021d0 con 0x7f94f8105a40 2024-04-16T03:04:36.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94fd844700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 0x7f94f8101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:36.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94fd844700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 0x7f94f8101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:56816/0 (socket says 172.21.15.151:56816) 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94fd844700 1 -- 172.21.15.151:0/2568787395 learned_addr learned my addr 172.21.15.151:0/2568787395 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94fd043700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94f8105a40 0x7f94f8101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94fd844700 1 -- 172.21.15.151:0/2568787395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94f8105a40 msgr2=0x7f94f8101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.107+0000 7f94fd844700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94f8105a40 0x7f94f8101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.108+0000 7f94fd844700 1 -- 172.21.15.151:0/2568787395 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec0095f0 con 0x7f94f80ffbd0 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.108+0000 7f94fd043700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94f8105a40 0x7f94f8101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.108+0000 7f94fd844700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 0x7f94f8101610 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f94f400d900 tx=0x7f94f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.108+0000 7f94eaffd700 1 -- 172.21.15.151:0/2568787395 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94f4004a00 con 0x7f94f80ffbd0 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.108+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f81a2920 con 0x7f94f80ffbd0 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.108+0000 7f94eaffd700 1 -- 172.21.15.151:0/2568787395 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94f40054e0 con 0x7f94f80ffbd0 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.108+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f81a2d30 con 0x7f94f80ffbd0 2024-04-16T03:04:36.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.108+0000 7f94eaffd700 1 -- 172.21.15.151:0/2568787395 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94f4009d60 con 0x7f94f80ffbd0 2024-04-16T03:04:36.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.109+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f8108fe0 con 0x7f94f80ffbd0 2024-04-16T03:04:36.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.110+0000 7f94eaffd700 1 -- 172.21.15.151:0/2568787395 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f94f4005020 con 0x7f94f80ffbd0 2024-04-16T03:04:36.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.110+0000 7f94eaffd700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f94e406ecc0 0x7f94e4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:36.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.110+0000 7f94eaffd700 1 -- 172.21.15.151:0/2568787395 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f94f408f030 con 0x7f94f80ffbd0 2024-04-16T03:04:36.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.110+0000 7f94fd043700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f94e406ecc0 0x7f94e4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:36.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.111+0000 7f94fd043700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f94e406ecc0 0x7f94e4071180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f94ec005b40 tx=0x7f94ec01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:36.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.115+0000 7f94eaffd700 1 -- 172.21.15.151:0/2568787395 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f94f4057de0 con 0x7f94f80ffbd0 2024-04-16T03:04:36.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.332+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f94f802cdd0 con 0x7f94e406ecc0 2024-04-16T03:04:36.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.333+0000 7f94eaffd700 1 -- 172.21.15.151:0/2568787395 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f94f802cdd0 con 0x7f94e406ecc0 2024-04-16T03:04:36.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.335+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f94e406ecc0 msgr2=0x7f94e4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.335+0000 7f94ffaa8700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f94e406ecc0 0x7f94e4071180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f94ec005b40 tx=0x7f94ec01a040 comp rx=0 tx=0).stop 2024-04-16T03:04:36.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.335+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 msgr2=0x7f94f8101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.335+0000 7f94ffaa8700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 0x7f94f8101610 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f94f400d900 tx=0x7f94f400dcc0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.335+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 shutdown_connections 2024-04-16T03:04:36.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.335+0000 7f94ffaa8700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f94e406ecc0 0x7f94e4071180 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.335+0000 7f94ffaa8700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f80ffbd0 0x7f94f8101610 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.335+0000 7f94ffaa8700 1 --2- 172.21.15.151:0/2568787395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94f8105a40 0x7f94f8101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.336+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 >> 172.21.15.151:0/2568787395 conn(0x7f94f80755a0 msgr2=0x7f94f8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:36.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.336+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 shutdown_connections 2024-04-16T03:04:36.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.336+0000 7f94ffaa8700 1 -- 172.21.15.151:0/2568787395 wait complete. 2024-04-16T03:04:36.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.440+0000 7f200c60c700 1 -- 172.21.15.151:0/3357387333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2004105a40 msgr2=0x7f2004105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.440+0000 7f200c60c700 1 --2- 172.21.15.151:0/3357387333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2004105a40 0x7f2004105e20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009910 tx=0x7f1ff4009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:36.442 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:36 smithi151 ceph-mon[112233]: pgmap v780: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:36.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.440+0000 7f200c60c700 1 -- 172.21.15.151:0/3357387333 shutdown_connections 2024-04-16T03:04:36.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.440+0000 7f200c60c700 1 --2- 172.21.15.151:0/3357387333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20040ffa40 0x7f20040ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.440+0000 7f200c60c700 1 --2- 172.21.15.151:0/3357387333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2004105a40 0x7f2004105e20 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.441+0000 7f200c60c700 1 -- 172.21.15.151:0/3357387333 >> 172.21.15.151:0/3357387333 conn(0x7f20040fb550 msgr2=0x7f20040fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.441+0000 7f200c60c700 1 -- 172.21.15.151:0/3357387333 shutdown_connections 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.441+0000 7f200c60c700 1 -- 172.21.15.151:0/3357387333 wait complete. 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.441+0000 7f200c60c700 1 Processor -- start 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.441+0000 7f200c60c700 1 -- start start 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.441+0000 7f200c60c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20040ffa40 0x7f2004197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.441+0000 7f200c60c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2004105a40 0x7f2004197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.441+0000 7f200c60c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20041980f0 con 0x7f20040ffa40 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.441+0000 7f200c60c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2004198230 con 0x7f2004105a40 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f200a3a8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20040ffa40 0x7f2004197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f200a3a8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20040ffa40 0x7f2004197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49310/0 (socket says 172.21.15.151:49310) 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f200a3a8700 1 -- 172.21.15.151:0/324073495 learned_addr learned my addr 172.21.15.151:0/324073495 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f2009ba7700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2004105a40 0x7f2004197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f200a3a8700 1 -- 172.21.15.151:0/324073495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2004105a40 msgr2=0x7f2004197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f200a3a8700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2004105a40 0x7f2004197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f200a3a8700 1 -- 172.21.15.151:0/324073495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff40095f0 con 0x7f20040ffa40 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f2009ba7700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2004105a40 0x7f2004197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f200a3a8700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20040ffa40 0x7f2004197590 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4006010 tx=0x7f1ff400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f1ffb7fe700 1 -- 172.21.15.151:0/324073495 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff401d070 con 0x7f20040ffa40 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f1ffb7fe700 1 -- 172.21.15.151:0/324073495 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ff4004fb0 con 0x7f20040ffa40 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20041a0600 con 0x7f20040ffa40 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f1ffb7fe700 1 -- 172.21.15.151:0/324073495 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff4022ea0 con 0x7f20040ffa40 2024-04-16T03:04:36.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.442+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20041a0a10 con 0x7f20040ffa40 2024-04-16T03:04:36.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.443+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2004108040 con 0x7f20040ffa40 2024-04-16T03:04:36.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.444+0000 7f1ffb7fe700 1 -- 172.21.15.151:0/324073495 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1ff4022470 con 0x7f20040ffa40 2024-04-16T03:04:36.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.444+0000 7f1ffb7fe700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ff006ecc0 0x7f1ff0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:36.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.445+0000 7f2009ba7700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ff006ecc0 0x7f1ff0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:36.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.445+0000 7f1ffb7fe700 1 -- 172.21.15.151:0/324073495 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1ff4090350 con 0x7f20040ffa40 2024-04-16T03:04:36.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.445+0000 7f2009ba7700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ff006ecc0 0x7f1ff0071180 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2000005fd0 tx=0x7f2000005de0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:36.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.449+0000 7f1ffb7fe700 1 -- 172.21.15.151:0/324073495 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1ff4059230 con 0x7f20040ffa40 2024-04-16T03:04:36.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.659+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f20040611d0 con 0x7f1ff006ecc0 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.669+0000 7f1ffb7fe700 1 -- 172.21.15.151:0/324073495 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f20040611d0 con 0x7f1ff006ecc0 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (26m) 4m ago 37m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (37m) 4m ago 37m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (36m) 4m ago 36m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (23m) 4m ago 37m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (23m) 4m ago 36m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (25m) 4m ago 37m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (20m) 4m ago 31m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:04:36.671 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (19m) 4m ago 31m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (4m) 4m ago 31m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (19m) 4m ago 31m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (28m) 4m ago 38m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (27m) 4m ago 36m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (24m) 4m ago 39m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (24m) 4m ago 36m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (26m) 4m ago 37m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (26m) 4m ago 36m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (23m) 4m ago 35m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (22m) 4m ago 35m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (22m) 4m ago 34m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (21m) 4m ago 34m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (20m) 4m ago 33m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (20m) 4m ago 33m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:04:36.672 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (26m) 4m ago 37m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:04:36.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.671+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ff006ecc0 msgr2=0x7f1ff0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.671+0000 7f200c60c700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ff006ecc0 0x7f1ff0071180 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2000005fd0 tx=0x7f2000005de0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.671+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20040ffa40 msgr2=0x7f2004197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.671+0000 7f200c60c700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20040ffa40 0x7f2004197590 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4006010 tx=0x7f1ff400be20 comp rx=0 tx=0).stop 2024-04-16T03:04:36.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.671+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 shutdown_connections 2024-04-16T03:04:36.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.671+0000 7f200c60c700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ff006ecc0 0x7f1ff0071180 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.673 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.671+0000 7f200c60c700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f20040ffa40 0x7f2004197590 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.671+0000 7f200c60c700 1 --2- 172.21.15.151:0/324073495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2004105a40 0x7f2004197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.671+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 >> 172.21.15.151:0/324073495 conn(0x7f20040fb550 msgr2=0x7f20040fbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:36.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.672+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 shutdown_connections 2024-04-16T03:04:36.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.672+0000 7f200c60c700 1 -- 172.21.15.151:0/324073495 wait complete. 2024-04-16T03:04:36.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:36 smithi153 ceph-mon[87858]: pgmap v780: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:36.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.775+0000 7fdcc8016700 1 -- 172.21.15.151:0/729812873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc00ffa70 msgr2=0x7fdcc00ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.775+0000 7fdcc8016700 1 --2- 172.21.15.151:0/729812873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc00ffa70 0x7fdcc00ffef0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009910 tx=0x7fdcb0009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:36.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.776+0000 7fdcc8016700 1 -- 172.21.15.151:0/729812873 shutdown_connections 2024-04-16T03:04:36.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.776+0000 7fdcc8016700 1 --2- 172.21.15.151:0/729812873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc00ffa70 0x7fdcc00ffef0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.776+0000 7fdcc8016700 1 --2- 172.21.15.151:0/729812873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc0105a70 0x7fdcc0105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.777 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.776+0000 7fdcc8016700 1 -- 172.21.15.151:0/729812873 >> 172.21.15.151:0/729812873 conn(0x7fdcc00fb5a0 msgr2=0x7fdcc00fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:36.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.776+0000 7fdcc8016700 1 -- 172.21.15.151:0/729812873 shutdown_connections 2024-04-16T03:04:36.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.776+0000 7fdcc8016700 1 -- 172.21.15.151:0/729812873 wait complete. 2024-04-16T03:04:36.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.776+0000 7fdcc8016700 1 Processor -- start 2024-04-16T03:04:36.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.776+0000 7fdcc8016700 1 -- start start 2024-04-16T03:04:36.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc8016700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc00ffa70 0x7fdcc019c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:36.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc8016700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc0105a70 0x7fdcc019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:36.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc8016700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc019d110 con 0x7fdcc0105a70 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc8016700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc01965c0 con 0x7fdcc00ffa70 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc55b1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc0105a70 0x7fdcc019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc55b1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc0105a70 0x7fdcc019ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49326/0 (socket says 172.21.15.151:49326) 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc55b1700 1 -- 172.21.15.151:0/1146836210 learned_addr learned my addr 172.21.15.151:0/1146836210 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc5db2700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc00ffa70 0x7fdcc019c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc55b1700 1 -- 172.21.15.151:0/1146836210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc00ffa70 msgr2=0x7fdcc019c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc55b1700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc00ffa70 0x7fdcc019c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc55b1700 1 -- 172.21.15.151:0/1146836210 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb00095f0 con 0x7fdcc0105a70 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc55b1700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc0105a70 0x7fdcc019ca80 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005f50 tx=0x7fdcb000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcb6ffd700 1 -- 172.21.15.151:0/1146836210 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcb001d070 con 0x7fdcc0105a70 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcb6ffd700 1 -- 172.21.15.151:0/1146836210 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcb000b610 con 0x7fdcc0105a70 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc0196840 con 0x7fdcc0105a70 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcb6ffd700 1 -- 172.21.15.151:0/1146836210 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcb0022470 con 0x7fdcc0105a70 2024-04-16T03:04:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.777+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc0196d30 con 0x7fdcc0105a70 2024-04-16T03:04:36.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.778+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc004ea90 con 0x7fdcc0105a70 2024-04-16T03:04:36.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.779+0000 7fdcb6ffd700 1 -- 172.21.15.151:0/1146836210 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fdcb0003780 con 0x7fdcc0105a70 2024-04-16T03:04:36.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.780+0000 7fdcb6ffd700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdcac06ecc0 0x7fdcac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:36.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.780+0000 7fdcc5db2700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdcac06ecc0 0x7fdcac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:36.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.780+0000 7fdcb6ffd700 1 -- 172.21.15.151:0/1146836210 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdcb0090140 con 0x7fdcc0105a70 2024-04-16T03:04:36.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.780+0000 7fdcc5db2700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdcac06ecc0 0x7fdcac071180 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009c60 tx=0x7fdcbc009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:36.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:36.784+0000 7fdcb6ffd700 1 -- 172.21.15.151:0/1146836210 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdcb005caa0 con 0x7fdcc0105a70 2024-04-16T03:04:37.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.065+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdcc0196fa0 con 0x7fdcc0105a70 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.065+0000 7fdcb6ffd700 1 -- 172.21.15.151:0/1146836210 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fdcb005c630 con 0x7fdcc0105a70 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:04:37.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:04:37.068 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:04:37.068 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:04:37.068 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:04:37.068 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdcac06ecc0 msgr2=0x7fdcac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdcac06ecc0 0x7fdcac071180 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009c60 tx=0x7fdcbc009260 comp rx=0 tx=0).stop 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc0105a70 msgr2=0x7fdcc019ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc0105a70 0x7fdcc019ca80 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005f50 tx=0x7fdcb000f690 comp rx=0 tx=0).stop 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 shutdown_connections 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdcac06ecc0 0x7fdcac071180 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc00ffa70 0x7fdcc019c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 --2- 172.21.15.151:0/1146836210 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcc0105a70 0x7fdcc019ca80 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 >> 172.21.15.151:0/1146836210 conn(0x7fdcc00fb5a0 msgr2=0x7fdcc0103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 shutdown_connections 2024-04-16T03:04:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.068+0000 7fdcc8016700 1 -- 172.21.15.151:0/1146836210 wait complete. 2024-04-16T03:04:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.173+0000 7face4dfd700 1 -- 172.21.15.151:0/500426895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 msgr2=0x7face01057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.173+0000 7face4dfd700 1 --2- 172.21.15.151:0/500426895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 0x7face01057d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7facd4009910 tx=0x7facd4009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.173+0000 7face4dfd700 1 -- 172.21.15.151:0/500426895 shutdown_connections 2024-04-16T03:04:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.173+0000 7face4dfd700 1 --2- 172.21.15.151:0/500426895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 0x7face01057d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.173+0000 7face4dfd700 1 --2- 172.21.15.151:0/500426895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7face0100dd0 0x7face01011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.173+0000 7face4dfd700 1 -- 172.21.15.151:0/500426895 >> 172.21.15.151:0/500426895 conn(0x7face00fc640 msgr2=0x7face00fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.174+0000 7face4dfd700 1 -- 172.21.15.151:0/500426895 shutdown_connections 2024-04-16T03:04:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.174+0000 7face4dfd700 1 -- 172.21.15.151:0/500426895 wait complete. 2024-04-16T03:04:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.174+0000 7face4dfd700 1 Processor -- start 2024-04-16T03:04:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.174+0000 7face4dfd700 1 -- start start 2024-04-16T03:04:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7face4dfd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7face0100dd0 0x7face019c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7face4dfd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 0x7face019cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7face4dfd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face019d0d0 con 0x7face0101780 2024-04-16T03:04:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7face4dfd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face0196610 con 0x7face0100dd0 2024-04-16T03:04:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7facddd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 0x7face019cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7facddd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 0x7face019cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49340/0 (socket says 172.21.15.151:49340) 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7facddd9b700 1 -- 172.21.15.151:0/3598069857 learned_addr learned my addr 172.21.15.151:0/3598069857 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7facde59c700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7face0100dd0 0x7face019c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7facddd9b700 1 -- 172.21.15.151:0/3598069857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7face0100dd0 msgr2=0x7face019c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7facddd9b700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7face0100dd0 0x7face019c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7facddd9b700 1 -- 172.21.15.151:0/3598069857 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40095f0 con 0x7face0101780 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7facddd9b700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 0x7face019cad0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7facd400bda0 tx=0x7facd400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7faccf7fe700 1 -- 172.21.15.151:0/3598069857 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd401d070 con 0x7face0101780 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7faccf7fe700 1 -- 172.21.15.151:0/3598069857 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facd4004fb0 con 0x7face0101780 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7faccf7fe700 1 -- 172.21.15.151:0/3598069857 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd400f480 con 0x7face0101780 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.175+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face0196890 con 0x7face0101780 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.176+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face0196d80 con 0x7face0101780 2024-04-16T03:04:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.177+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face01091d0 con 0x7face0101780 2024-04-16T03:04:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.177+0000 7faccf7fe700 1 -- 172.21.15.151:0/3598069857 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7facd4022870 con 0x7face0101780 2024-04-16T03:04:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.177+0000 7faccf7fe700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7facc806ecc0 0x7facc8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.178+0000 7faccf7fe700 1 -- 172.21.15.151:0/3598069857 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7facd40902d0 con 0x7face0101780 2024-04-16T03:04:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.178+0000 7facde59c700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7facc806ecc0 0x7facc8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.178+0000 7facde59c700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7facc806ecc0 0x7facc8071180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7facd000ba60 tx=0x7facd0005d50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:37.183 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.182+0000 7faccf7fe700 1 -- 172.21.15.151:0/3598069857 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7facd405cce0 con 0x7face0101780 2024-04-16T03:04:37.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.426+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7face0197bc0 con 0x7face0101780 2024-04-16T03:04:37.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.427+0000 7faccf7fe700 1 -- 172.21.15.151:0/3598069857 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7facd405c870 con 0x7face0101780 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:04:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:04:37.430 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:04:37.431 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7facc806ecc0 msgr2=0x7facc8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7facc806ecc0 0x7facc8071180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7facd000ba60 tx=0x7facd0005d50 comp rx=0 tx=0).stop 2024-04-16T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 msgr2=0x7face019cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 0x7face019cad0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7facd400bda0 tx=0x7facd400be80 comp rx=0 tx=0).stop 2024-04-16T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 shutdown_connections 2024-04-16T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7facc806ecc0 0x7facc8071180 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7face0100dd0 0x7face019c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 --2- 172.21.15.151:0/3598069857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7face0101780 0x7face019cad0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 >> 172.21.15.151:0/3598069857 conn(0x7face00fc640 msgr2=0x7face00fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:37.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 shutdown_connections 2024-04-16T03:04:37.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.429+0000 7face4dfd700 1 -- 172.21.15.151:0/3598069857 wait complete. 2024-04-16T03:04:37.433 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:04:37.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:37 smithi153 ceph-mon[87858]: from='client.35436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:37.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:37 smithi153 ceph-mon[87858]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:37.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:37 smithi153 ceph-mon[87858]: from='client.35444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:37.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:37 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1146836210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:04:37.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/4102712745 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4102610 msgr2=0x7fe9f410ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 --2- 172.21.15.151:0/4102712745 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4102610 0x7fe9f410ab00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0009910 tx=0x7fe9f0009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:37.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/4102712745 shutdown_connections 2024-04-16T03:04:37.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 --2- 172.21.15.151:0/4102712745 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4102610 0x7fe9f410ab00 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 --2- 172.21.15.151:0/4102712745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f4101c60 0x7fe9f4102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/4102712745 >> 172.21.15.151:0/4102712745 conn(0x7fe9f40fb540 msgr2=0x7fe9f40fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:37.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/4102712745 shutdown_connections 2024-04-16T03:04:37.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/4102712745 wait complete. 2024-04-16T03:04:37.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 Processor -- start 2024-04-16T03:04:37.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.537+0000 7fe9fc0f1700 1 -- start start 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9fc0f1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4101c60 0x7fe9f419c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9fc0f1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f4102610 0x7fe9f419caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9fc0f1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f419d0f0 con 0x7fe9f4101c60 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9fc0f1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f4196630 con 0x7fe9f4102610 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9f9e8d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4101c60 0x7fe9f419c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9f9e8d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4101c60 0x7fe9f419c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49350/0 (socket says 172.21.15.151:49350) 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9f9e8d700 1 -- 172.21.15.151:0/580157615 learned_addr learned my addr 172.21.15.151:0/580157615 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9f968c700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f4102610 0x7fe9f419caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9f9e8d700 1 -- 172.21.15.151:0/580157615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f4102610 msgr2=0x7fe9f419caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9f9e8d700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f4102610 0x7fe9f419caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9f9e8d700 1 -- 172.21.15.151:0/580157615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00095f0 con 0x7fe9f4101c60 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9f968c700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f4102610 0x7fe9f419caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:04:37.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9f9e8d700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4101c60 0x7fe9f419c5b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400c930 tx=0x7fe9e400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:37.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9eaffd700 1 -- 172.21.15.151:0/580157615 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e4007ad0 con 0x7fe9f4101c60 2024-04-16T03:04:37.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9eaffd700 1 -- 172.21.15.151:0/580157615 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9e4007c30 con 0x7fe9f4101c60 2024-04-16T03:04:37.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9eaffd700 1 -- 172.21.15.151:0/580157615 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e4018610 con 0x7fe9f4101c60 2024-04-16T03:04:37.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f4196910 con 0x7fe9f4101c60 2024-04-16T03:04:37.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.538+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f4196e10 con 0x7fe9f4101c60 2024-04-16T03:04:37.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.539+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f4108200 con 0x7fe9f4101c60 2024-04-16T03:04:37.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.540+0000 7fe9eaffd700 1 -- 172.21.15.151:0/580157615 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe9e400f450 con 0x7fe9f4101c60 2024-04-16T03:04:37.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.541+0000 7fe9eaffd700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9e006ecc0 0x7fe9e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:37.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.541+0000 7fe9f968c700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9e006ecc0 0x7fe9e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:37.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.541+0000 7fe9eaffd700 1 -- 172.21.15.151:0/580157615 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe9e408f080 con 0x7fe9f4101c60 2024-04-16T03:04:37.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.541+0000 7fe9f968c700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9e006ecc0 0x7fe9e0071180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40fcd70 tx=0x7fe9f001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:37.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.545+0000 7fe9eaffd700 1 -- 172.21.15.151:0/580157615 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe9e405b960 con 0x7fe9f4101c60 2024-04-16T03:04:37.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:37 smithi151 ceph-mon[112233]: from='client.35436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:37.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:37 smithi151 ceph-mon[112233]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:37.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:37 smithi151 ceph-mon[112233]: from='client.35444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:37.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:37 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1146836210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:04:37.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.767+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9f41970e0 con 0x7fe9e006ecc0 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.768+0000 7fe9eaffd700 1 -- 172.21.15.151:0/580157615 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe9f41970e0 con 0x7fe9e006ecc0 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:04:37.770 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:04:37.771 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:04:37.771 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:04:37.771 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:04:37.771 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:04:37.771 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:04:37.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9e006ecc0 msgr2=0x7fe9e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9e006ecc0 0x7fe9e0071180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40fcd70 tx=0x7fe9f001a040 comp rx=0 tx=0).stop 2024-04-16T03:04:37.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4101c60 msgr2=0x7fe9f419c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4101c60 0x7fe9f419c5b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400c930 tx=0x7fe9e400ccf0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 shutdown_connections 2024-04-16T03:04:37.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe9e006ecc0 0x7fe9e0071180 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9f4101c60 0x7fe9f419c5b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 --2- 172.21.15.151:0/580157615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f4102610 0x7fe9f419caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 >> 172.21.15.151:0/580157615 conn(0x7fe9f40fb540 msgr2=0x7fe9f4105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:37.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.770+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 shutdown_connections 2024-04-16T03:04:37.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.771+0000 7fe9fc0f1700 1 -- 172.21.15.151:0/580157615 wait complete. 2024-04-16T03:04:37.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.874+0000 7f75c7cca700 1 -- 172.21.15.151:0/860471757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c0105a40 msgr2=0x7f75c0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 --2- 172.21.15.151:0/860471757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c0105a40 0x7f75c0105e20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f75bc009910 tx=0x7f75bc009c20 comp rx=0 tx=0).stop 2024-04-16T03:04:37.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 -- 172.21.15.151:0/860471757 shutdown_connections 2024-04-16T03:04:37.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 --2- 172.21.15.151:0/860471757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c00ffa40 0x7f75c00ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 --2- 172.21.15.151:0/860471757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c0105a40 0x7f75c0105e20 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 -- 172.21.15.151:0/860471757 >> 172.21.15.151:0/860471757 conn(0x7f75c00fb570 msgr2=0x7f75c00fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:37.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 -- 172.21.15.151:0/860471757 shutdown_connections 2024-04-16T03:04:37.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 -- 172.21.15.151:0/860471757 wait complete. 2024-04-16T03:04:37.878 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 Processor -- start 2024-04-16T03:04:37.878 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 -- start start 2024-04-16T03:04:37.878 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c00ffa40 0x7f75c01975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:37.878 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.875+0000 7f75c7cca700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c0105a40 0x7f75c0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:37.878 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c7cca700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c0198110 con 0x7f75c00ffa40 2024-04-16T03:04:37.878 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c7cca700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c0198250 con 0x7f75c0105a40 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c5a66700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c00ffa40 0x7f75c01975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c5a66700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c00ffa40 0x7f75c01975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49368/0 (socket says 172.21.15.151:49368) 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c5a66700 1 -- 172.21.15.151:0/927940616 learned_addr learned my addr 172.21.15.151:0/927940616 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c5265700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c0105a40 0x7f75c0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c5a66700 1 -- 172.21.15.151:0/927940616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c0105a40 msgr2=0x7f75c0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c5a66700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c0105a40 0x7f75c0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c5a66700 1 -- 172.21.15.151:0/927940616 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc0095f0 con 0x7f75c00ffa40 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c5a66700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c00ffa40 0x7f75c01975b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005f50 tx=0x7f75bc00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75b6ffd700 1 -- 172.21.15.151:0/927940616 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75bc01d070 con 0x7f75c00ffa40 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75b6ffd700 1 -- 172.21.15.151:0/927940616 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75bc00b610 con 0x7f75c00ffa40 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c01a0620 con 0x7f75c00ffa40 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75b6ffd700 1 -- 172.21.15.151:0/927940616 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75bc022470 con 0x7f75c00ffa40 2024-04-16T03:04:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.876+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c01a0a30 con 0x7f75c00ffa40 2024-04-16T03:04:37.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.877+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c0108040 con 0x7f75c00ffa40 2024-04-16T03:04:37.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.878+0000 7f75b6ffd700 1 -- 172.21.15.151:0/927940616 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f75bc00fca0 con 0x7f75c00ffa40 2024-04-16T03:04:37.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.878+0000 7f75b6ffd700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f75ac06ecc0 0x7f75ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:04:37.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.878+0000 7f75b6ffd700 1 -- 172.21.15.151:0/927940616 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f75bc0902d0 con 0x7f75c00ffa40 2024-04-16T03:04:37.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.878+0000 7f75c5265700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f75ac06ecc0 0x7f75ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:04:37.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.879+0000 7f75c5265700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f75ac06ecc0 0x7f75ac071180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f75c0069640 tx=0x7f75b0009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:04:37.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:37.883+0000 7f75b6ffd700 1 -- 172.21.15.151:0/927940616 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f75bc059080 con 0x7f75c00ffa40 2024-04-16T03:04:38.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.164+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f75c004ea90 con 0x7f75c00ffa40 2024-04-16T03:04:38.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.164+0000 7f75b6ffd700 1 -- 172.21.15.151:0/927940616 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f75bc05c740 con 0x7f75c00ffa40 2024-04-16T03:04:38.166 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:04:38.166 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:04:38.166 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:04:38.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:04:38.166 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:04:38.166 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:04:38.166 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:04:38.167 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:04:38.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f75ac06ecc0 msgr2=0x7f75ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:38.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f75ac06ecc0 0x7f75ac071180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f75c0069640 tx=0x7f75b0009260 comp rx=0 tx=0).stop 2024-04-16T03:04:38.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c00ffa40 msgr2=0x7f75c01975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:04:38.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c00ffa40 0x7f75c01975b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005f50 tx=0x7f75bc00f740 comp rx=0 tx=0).stop 2024-04-16T03:04:38.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 shutdown_connections 2024-04-16T03:04:38.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f75ac06ecc0 0x7f75ac071180 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:38.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75c00ffa40 0x7f75c01975b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:38.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 --2- 172.21.15.151:0/927940616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c0105a40 0x7f75c0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:04:38.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 >> 172.21.15.151:0/927940616 conn(0x7f75c00fb570 msgr2=0x7f75c00fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:04:38.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 shutdown_connections 2024-04-16T03:04:38.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:04:38.167+0000 7f75c7cca700 1 -- 172.21.15.151:0/927940616 wait complete. 2024-04-16T03:04:38.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:38 smithi151 ceph-mon[112233]: pgmap v781: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:38.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:38 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3598069857' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:04:38.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:38 smithi151 ceph-mon[112233]: from='client.35456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:38.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:38 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/927940616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:04:38.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:38 smithi153 ceph-mon[87858]: pgmap v781: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:38.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:38 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3598069857' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:04:38.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:38 smithi153 ceph-mon[87858]: from='client.35456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:04:38.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:38 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/927940616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:04:39.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:04:39.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:04:40.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:40 smithi151 ceph-mon[112233]: pgmap v782: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:40.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:40 smithi153 ceph-mon[87858]: pgmap v782: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:42.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:42 smithi153 ceph-mon[87858]: pgmap v783: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:42.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:42 smithi151 ceph-mon[112233]: pgmap v783: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:44.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:44 smithi151 ceph-mon[112233]: pgmap v784: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:44.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:44 smithi153 ceph-mon[87858]: pgmap v784: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:46.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:46 smithi151 ceph-mon[112233]: pgmap v785: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:46.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:46 smithi153 ceph-mon[87858]: pgmap v785: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:48.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:48 smithi151 ceph-mon[112233]: pgmap v786: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:48.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:48 smithi153 ceph-mon[87858]: pgmap v786: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:50.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:50 smithi151 ceph-mon[112233]: pgmap v787: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:50.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:50 smithi153 ceph-mon[87858]: pgmap v787: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:52.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:52 smithi153 ceph-mon[87858]: pgmap v788: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:52.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:52 smithi151 ceph-mon[112233]: pgmap v788: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:54.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:54 smithi151 ceph-mon[112233]: pgmap v789: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:54.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:04:54.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:54 smithi153 ceph-mon[87858]: pgmap v789: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:54.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:04:56.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:56 smithi151 ceph-mon[112233]: pgmap v790: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:56.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:56 smithi153 ceph-mon[87858]: pgmap v790: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:58.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:04:58 smithi151 ceph-mon[112233]: pgmap v791: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:04:58.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:04:58 smithi153 ceph-mon[87858]: pgmap v791: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:00.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:00 smithi151 ceph-mon[112233]: pgmap v792: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:00.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:00 smithi153 ceph-mon[87858]: pgmap v792: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:02.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:02 smithi151 ceph-mon[112233]: pgmap v793: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:02.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:02 smithi153 ceph-mon[87858]: pgmap v793: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:04.661 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:04 smithi151 ceph-mon[112233]: pgmap v794: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:04.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:04 smithi153 ceph-mon[87858]: pgmap v794: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:06.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:06 smithi151 ceph-mon[112233]: pgmap v795: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:07.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:06 smithi153 ceph-mon[87858]: pgmap v795: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:08.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.285+0000 7fc77f788700 1 -- 172.21.15.151:0/505340457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 msgr2=0x7fc7781057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.285+0000 7fc77f788700 1 --2- 172.21.15.151:0/505340457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 0x7fc7781057d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc768009910 tx=0x7fc768009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:08.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.285+0000 7fc77f788700 1 -- 172.21.15.151:0/505340457 shutdown_connections 2024-04-16T03:05:08.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.285+0000 7fc77f788700 1 --2- 172.21.15.151:0/505340457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 0x7fc7781057d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.285+0000 7fc77f788700 1 --2- 172.21.15.151:0/505340457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc778100dd0 0x7fc7781011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.285+0000 7fc77f788700 1 -- 172.21.15.151:0/505340457 >> 172.21.15.151:0/505340457 conn(0x7fc7780fc640 msgr2=0x7fc7780fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:08.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.285+0000 7fc77f788700 1 -- 172.21.15.151:0/505340457 shutdown_connections 2024-04-16T03:05:08.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.285+0000 7fc77f788700 1 -- 172.21.15.151:0/505340457 wait complete. 2024-04-16T03:05:08.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77f788700 1 Processor -- start 2024-04-16T03:05:08.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77f788700 1 -- start start 2024-04-16T03:05:08.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77f788700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc778100dd0 0x7fc77819c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:08.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77f788700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 0x7fc77819cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:08.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77f788700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77819d160 con 0x7fc778101780 2024-04-16T03:05:08.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77f788700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc778196610 con 0x7fc778100dd0 2024-04-16T03:05:08.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77cd23700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 0x7fc77819cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:08.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77cd23700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 0x7fc77819cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41224/0 (socket says 172.21.15.151:41224) 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77cd23700 1 -- 172.21.15.151:0/683886978 learned_addr learned my addr 172.21.15.151:0/683886978 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.286+0000 7fc77d524700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc778100dd0 0x7fc77819c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.287+0000 7fc77cd23700 1 -- 172.21.15.151:0/683886978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc778100dd0 msgr2=0x7fc77819c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.287+0000 7fc77cd23700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc778100dd0 0x7fc77819c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.287+0000 7fc77cd23700 1 -- 172.21.15.151:0/683886978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7680095f0 con 0x7fc778101780 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.287+0000 7fc77cd23700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 0x7fc77819cad0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc768000c00 tx=0x7fc76800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.287+0000 7fc76e7fc700 1 -- 172.21.15.151:0/683886978 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc76801d070 con 0x7fc778101780 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.287+0000 7fc76e7fc700 1 -- 172.21.15.151:0/683886978 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc768005dc0 con 0x7fc778101780 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.287+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc778196890 con 0x7fc778101780 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.287+0000 7fc76e7fc700 1 -- 172.21.15.151:0/683886978 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc768017400 con 0x7fc778101780 2024-04-16T03:05:08.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.287+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc778196d80 con 0x7fc778101780 2024-04-16T03:05:08.291 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.288+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7781091d0 con 0x7fc778101780 2024-04-16T03:05:08.291 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.289+0000 7fc76e7fc700 1 -- 172.21.15.151:0/683886978 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc76800f460 con 0x7fc778101780 2024-04-16T03:05:08.291 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.289+0000 7fc76e7fc700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc76406ecc0 0x7fc764071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:08.291 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.289+0000 7fc76e7fc700 1 -- 172.21.15.151:0/683886978 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc7680902f0 con 0x7fc778101780 2024-04-16T03:05:08.291 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.289+0000 7fc77d524700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc76406ecc0 0x7fc764071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:08.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.290+0000 7fc77d524700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc76406ecc0 0x7fc764071180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc774007be0 tx=0x7fc7740073d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:08.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.294+0000 7fc76e7fc700 1 -- 172.21.15.151:0/683886978 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc76805cbd0 con 0x7fc778101780 2024-04-16T03:05:08.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.512+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc778197060 con 0x7fc76406ecc0 2024-04-16T03:05:08.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.513+0000 7fc76e7fc700 1 -- 172.21.15.151:0/683886978 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc778197060 con 0x7fc76406ecc0 2024-04-16T03:05:08.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc76406ecc0 msgr2=0x7fc764071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc76406ecc0 0x7fc764071180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc774007be0 tx=0x7fc7740073d0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 msgr2=0x7fc77819cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 0x7fc77819cad0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc768000c00 tx=0x7fc76800bd70 comp rx=0 tx=0).stop 2024-04-16T03:05:08.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 shutdown_connections 2024-04-16T03:05:08.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc76406ecc0 0x7fc764071180 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc778100dd0 0x7fc77819c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 --2- 172.21.15.151:0/683886978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc778101780 0x7fc77819cad0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 >> 172.21.15.151:0/683886978 conn(0x7fc7780fc640 msgr2=0x7fc7780fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:08.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.515+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 shutdown_connections 2024-04-16T03:05:08.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.516+0000 7fc77f788700 1 -- 172.21.15.151:0/683886978 wait complete. 2024-04-16T03:05:08.527 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:05:08.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.622+0000 7f80f74b6700 1 -- 172.21.15.151:0/2417897167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f00ffa40 msgr2=0x7f80f00ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.622+0000 7f80f74b6700 1 --2- 172.21.15.151:0/2417897167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f00ffa40 0x7f80f00ffec0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009910 tx=0x7f80ec009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:08.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.622+0000 7f80f74b6700 1 -- 172.21.15.151:0/2417897167 shutdown_connections 2024-04-16T03:05:08.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.622+0000 7f80f74b6700 1 --2- 172.21.15.151:0/2417897167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f00ffa40 0x7f80f00ffec0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.622+0000 7f80f74b6700 1 --2- 172.21.15.151:0/2417897167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80f0105a40 0x7f80f0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.622+0000 7f80f74b6700 1 -- 172.21.15.151:0/2417897167 >> 172.21.15.151:0/2417897167 conn(0x7f80f00fb550 msgr2=0x7f80f00fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:08.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.622+0000 7f80f74b6700 1 -- 172.21.15.151:0/2417897167 shutdown_connections 2024-04-16T03:05:08.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.622+0000 7f80f74b6700 1 -- 172.21.15.151:0/2417897167 wait complete. 2024-04-16T03:05:08.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f74b6700 1 Processor -- start 2024-04-16T03:05:08.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f74b6700 1 -- start start 2024-04-16T03:05:08.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f74b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80f00ffa40 0x7f80f019c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:08.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f74b6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f0105a40 0x7f80f019cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:08.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f74b6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f019d1f0 con 0x7f80f0105a40 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f74b6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01966a0 con 0x7f80f00ffa40 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f4a51700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f0105a40 0x7f80f019cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f4a51700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f0105a40 0x7f80f019cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41234/0 (socket says 172.21.15.151:41234) 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f4a51700 1 -- 172.21.15.151:0/2748628120 learned_addr learned my addr 172.21.15.151:0/2748628120 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.623+0000 7f80f5252700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80f00ffa40 0x7f80f019c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80f4a51700 1 -- 172.21.15.151:0/2748628120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80f00ffa40 msgr2=0x7f80f019c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80f4a51700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80f00ffa40 0x7f80f019c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80f4a51700 1 -- 172.21.15.151:0/2748628120 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec0095f0 con 0x7f80f0105a40 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80f5252700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80f00ffa40 0x7f80f019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80f4a51700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f0105a40 0x7f80f019cb60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f80ec006010 tx=0x7f80ec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80e67fc700 1 -- 172.21.15.151:0/2748628120 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80ec01d070 con 0x7f80f0105a40 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80e67fc700 1 -- 172.21.15.151:0/2748628120 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80ec004fb0 con 0x7f80f0105a40 2024-04-16T03:05:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f0196920 con 0x7f80f0105a40 2024-04-16T03:05:08.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80e67fc700 1 -- 172.21.15.151:0/2748628120 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80ec022ea0 con 0x7f80f0105a40 2024-04-16T03:05:08.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.624+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f0196e10 con 0x7f80f0105a40 2024-04-16T03:05:08.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.625+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f0109070 con 0x7f80f0105a40 2024-04-16T03:05:08.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.625+0000 7f80e67fc700 1 -- 172.21.15.151:0/2748628120 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f80ec022470 con 0x7f80f0105a40 2024-04-16T03:05:08.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.626+0000 7f80e67fc700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80dc06ecc0 0x7f80dc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:08.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.626+0000 7f80e67fc700 1 -- 172.21.15.151:0/2748628120 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f80ec090490 con 0x7f80f0105a40 2024-04-16T03:05:08.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.626+0000 7f80f5252700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80dc06ecc0 0x7f80dc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:08.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.626+0000 7f80f5252700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80dc06ecc0 0x7f80dc071180 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f80e0007900 tx=0x7f80e0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:08.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.630+0000 7f80e67fc700 1 -- 172.21.15.151:0/2748628120 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f80ec05cd80 con 0x7f80f0105a40 2024-04-16T03:05:08.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.849+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f80f002cce0 con 0x7f80dc06ecc0 2024-04-16T03:05:08.851 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:08 smithi151 ceph-mon[112233]: pgmap v796: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:08.852 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.851+0000 7f80e67fc700 1 -- 172.21.15.151:0/2748628120 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f80f002cce0 con 0x7f80dc06ecc0 2024-04-16T03:05:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80dc06ecc0 msgr2=0x7f80dc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80dc06ecc0 0x7f80dc071180 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f80e0007900 tx=0x7f80e0008040 comp rx=0 tx=0).stop 2024-04-16T03:05:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f0105a40 msgr2=0x7f80f019cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f0105a40 0x7f80f019cb60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f80ec006010 tx=0x7f80ec00be20 comp rx=0 tx=0).stop 2024-04-16T03:05:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 shutdown_connections 2024-04-16T03:05:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f80dc06ecc0 0x7f80dc071180 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80f00ffa40 0x7f80f019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 --2- 172.21.15.151:0/2748628120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f0105a40 0x7f80f019cb60 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 >> 172.21.15.151:0/2748628120 conn(0x7f80f00fb550 msgr2=0x7f80f0069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:08.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 shutdown_connections 2024-04-16T03:05:08.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.853+0000 7f80f74b6700 1 -- 172.21.15.151:0/2748628120 wait complete. 2024-04-16T03:05:08.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.959+0000 7f795f3a2700 1 -- 172.21.15.151:0/2276480003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 msgr2=0x7f7958100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.959+0000 7f795f3a2700 1 --2- 172.21.15.151:0/2276480003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 0x7f7958100e40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7948009910 tx=0x7f7948009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:08.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.959+0000 7f795f3a2700 1 -- 172.21.15.151:0/2276480003 shutdown_connections 2024-04-16T03:05:08.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.959+0000 7f795f3a2700 1 --2- 172.21.15.151:0/2276480003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 0x7f7958100e40 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.959+0000 7f795f3a2700 1 --2- 172.21.15.151:0/2276480003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79581069c0 0x7f7958106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.959+0000 7f795f3a2700 1 -- 172.21.15.151:0/2276480003 >> 172.21.15.151:0/2276480003 conn(0x7f79580fc4d0 msgr2=0x7f79580fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.959+0000 7f795f3a2700 1 -- 172.21.15.151:0/2276480003 shutdown_connections 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.959+0000 7f795f3a2700 1 -- 172.21.15.151:0/2276480003 wait complete. 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795f3a2700 1 Processor -- start 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795f3a2700 1 -- start start 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795f3a2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 0x7f79581965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795f3a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79581069c0 0x7f7958196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795f3a2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79581971d0 con 0x7f79581009c0 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795f3a2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795819af60 con 0x7f79581069c0 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795e3a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 0x7f79581965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795e3a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 0x7f79581965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41238/0 (socket says 172.21.15.151:41238) 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795e3a0700 1 -- 172.21.15.151:0/1773191571 learned_addr learned my addr 172.21.15.151:0/1773191571 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:08.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795db9f700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79581069c0 0x7f7958196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795e3a0700 1 -- 172.21.15.151:0/1773191571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79581069c0 msgr2=0x7f7958196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795e3a0700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79581069c0 0x7f7958196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795e3a0700 1 -- 172.21.15.151:0/1773191571 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79480095f0 con 0x7f79581009c0 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.960+0000 7f795e3a0700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 0x7f79581965b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f795400d8d0 tx=0x7f795400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.961+0000 7f794f7fe700 1 -- 172.21.15.151:0/1773191571 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7954009940 con 0x7f79581009c0 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.961+0000 7f794f7fe700 1 -- 172.21.15.151:0/1773191571 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7954010460 con 0x7f79581009c0 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.961+0000 7f794f7fe700 1 -- 172.21.15.151:0/1773191571 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f795400f5f0 con 0x7f79581009c0 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.961+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795819b240 con 0x7f79581009c0 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.961+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795819b790 con 0x7f79581009c0 2024-04-16T03:05:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.962+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7958108f50 con 0x7f79581009c0 2024-04-16T03:05:08.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.962+0000 7f794f7fe700 1 -- 172.21.15.151:0/1773191571 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f79540105d0 con 0x7f79581009c0 2024-04-16T03:05:08.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.962+0000 7f794f7fe700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f794406ecc0 0x7f7944071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:08.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.963+0000 7f794f7fe700 1 -- 172.21.15.151:0/1773191571 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f795408ed70 con 0x7f79581009c0 2024-04-16T03:05:08.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.963+0000 7f795db9f700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f794406ecc0 0x7f7944071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:08.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.963+0000 7f795db9f700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f794406ecc0 0x7f7944071180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7948005b40 tx=0x7f794801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:08.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:08.967+0000 7f794f7fe700 1 -- 172.21.15.151:0/1773191571 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7954057ba0 con 0x7f79581009c0 2024-04-16T03:05:09.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:08 smithi153 ceph-mon[87858]: pgmap v796: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:09.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.187+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f795819bb10 con 0x7f794406ecc0 2024-04-16T03:05:09.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.199+0000 7f794f7fe700 1 -- 172.21.15.151:0/1773191571 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f795819bb10 con 0x7f794406ecc0 2024-04-16T03:05:09.201 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:05:09.202 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (26m) 4m ago 38m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:05:09.202 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (38m) 4m ago 38m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:05:09.202 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (37m) 4m ago 37m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:05:09.202 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (24m) 4m ago 38m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:05:09.202 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (24m) 4m ago 37m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:05:09.202 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (26m) 4m ago 37m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:05:09.202 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (20m) 4m ago 32m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:05:09.202 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (20m) 4m ago 32m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:05:09.202 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (4m) 4m ago 32m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (19m) 4m ago 32m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (28m) 4m ago 39m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (27m) 4m ago 37m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (25m) 4m ago 39m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (24m) 4m ago 36m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (27m) 4m ago 38m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (27m) 4m ago 37m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (23m) 4m ago 36m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (23m) 4m ago 35m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (22m) 4m ago 35m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:05:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (22m) 4m ago 34m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:05:09.204 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (21m) 4m ago 34m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:05:09.204 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (20m) 4m ago 33m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:05:09.204 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (27m) 4m ago 37m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:05:09.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.201+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f794406ecc0 msgr2=0x7f7944071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f794406ecc0 0x7f7944071180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7948005b40 tx=0x7f794801a040 comp rx=0 tx=0).stop 2024-04-16T03:05:09.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 msgr2=0x7f79581965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 0x7f79581965b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f795400d8d0 tx=0x7f795400dc90 comp rx=0 tx=0).stop 2024-04-16T03:05:09.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 shutdown_connections 2024-04-16T03:05:09.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f794406ecc0 0x7f7944071180 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79581009c0 0x7f79581965b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 --2- 172.21.15.151:0/1773191571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79581069c0 0x7f7958196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 >> 172.21.15.151:0/1773191571 conn(0x7f79580fc4d0 msgr2=0x7f79580fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:09.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 shutdown_connections 2024-04-16T03:05:09.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.202+0000 7f795f3a2700 1 -- 172.21.15.151:0/1773191571 wait complete. 2024-04-16T03:05:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.306+0000 7f4d237fc700 1 -- 172.21.15.151:0/2890502315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c105c30 msgr2=0x7f4d1c106010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.306+0000 7f4d237fc700 1 --2- 172.21.15.151:0/2890502315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c105c30 0x7f4d1c106010 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00b0a0 tx=0x7f4d0c00b3b0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.306+0000 7f4d237fc700 1 -- 172.21.15.151:0/2890502315 shutdown_connections 2024-04-16T03:05:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.306+0000 7f4d237fc700 1 --2- 172.21.15.151:0/2890502315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d1c1003b0 0x7f4d1c100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.306+0000 7f4d237fc700 1 --2- 172.21.15.151:0/2890502315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c105c30 0x7f4d1c106010 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.306+0000 7f4d237fc700 1 -- 172.21.15.151:0/2890502315 >> 172.21.15.151:0/2890502315 conn(0x7f4d1c074d80 msgr2=0x7f4d1c075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.306+0000 7f4d237fc700 1 -- 172.21.15.151:0/2890502315 shutdown_connections 2024-04-16T03:05:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.306+0000 7f4d237fc700 1 -- 172.21.15.151:0/2890502315 wait complete. 2024-04-16T03:05:09.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d237fc700 1 Processor -- start 2024-04-16T03:05:09.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d237fc700 1 -- start start 2024-04-16T03:05:09.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d237fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c1003b0 0x7f4d1c1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:09.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d237fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d1c105c30 0x7f4d1c101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:09.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d237fc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c102230 con 0x7f4d1c1003b0 2024-04-16T03:05:09.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d237fc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c102370 con 0x7f4d1c105c30 2024-04-16T03:05:09.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d21598700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c1003b0 0x7f4d1c1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:09.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d21598700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c1003b0 0x7f4d1c1017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41248/0 (socket says 172.21.15.151:41248) 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d21598700 1 -- 172.21.15.151:0/3216346464 learned_addr learned my addr 172.21.15.151:0/3216346464 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d20d97700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d1c105c30 0x7f4d1c101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d21598700 1 -- 172.21.15.151:0/3216346464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d1c105c30 msgr2=0x7f4d1c101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.307+0000 7f4d21598700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d1c105c30 0x7f4d1c101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.308+0000 7f4d21598700 1 -- 172.21.15.151:0/3216346464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0c009d00 con 0x7f4d1c1003b0 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.308+0000 7f4d21598700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c1003b0 0x7f4d1c1017b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c000f80 tx=0x7f4d0c00cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.308+0000 7f4d1a7fc700 1 -- 172.21.15.151:0/3216346464 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d0c00e040 con 0x7f4d1c1003b0 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.308+0000 7f4d1a7fc700 1 -- 172.21.15.151:0/3216346464 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d0c003de0 con 0x7f4d1c1003b0 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.308+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c1025f0 con 0x7f4d1c1003b0 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.308+0000 7f4d1a7fc700 1 -- 172.21.15.151:0/3216346464 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d0c01d790 con 0x7f4d1c1003b0 2024-04-16T03:05:09.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.308+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c1a2d50 con 0x7f4d1c1003b0 2024-04-16T03:05:09.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.309+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1c04ea90 con 0x7f4d1c1003b0 2024-04-16T03:05:09.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.310+0000 7f4d1a7fc700 1 -- 172.21.15.151:0/3216346464 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4d0c004040 con 0x7f4d1c1003b0 2024-04-16T03:05:09.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.310+0000 7f4d1a7fc700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0406ecc0 0x7f4d04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:09.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.310+0000 7f4d1a7fc700 1 -- 172.21.15.151:0/3216346464 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4d0c090570 con 0x7f4d1c1003b0 2024-04-16T03:05:09.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.310+0000 7f4d20d97700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0406ecc0 0x7f4d04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:09.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.311+0000 7f4d20d97700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0406ecc0 0x7f4d04071180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4d10005950 tx=0x7f4d100058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:09.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.315+0000 7f4d1a7fc700 1 -- 172.21.15.151:0/3216346464 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4d0c0593a0 con 0x7f4d1c1003b0 2024-04-16T03:05:09.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.607+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4d1c1a3120 con 0x7f4d1c1003b0 2024-04-16T03:05:09.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.607+0000 7f4d1a7fc700 1 -- 172.21.15.151:0/3216346464 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4d0c05ca60 con 0x7f4d1c1003b0 2024-04-16T03:05:09.609 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:05:09.609 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:05:09.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:05:09.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:05:09.610 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:05:09.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0406ecc0 msgr2=0x7f4d04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0406ecc0 0x7f4d04071180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4d10005950 tx=0x7f4d100058e0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c1003b0 msgr2=0x7f4d1c1017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c1003b0 0x7f4d1c1017b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c000f80 tx=0x7f4d0c00cdd0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 shutdown_connections 2024-04-16T03:05:09.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0406ecc0 0x7f4d04071180 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d1c1003b0 0x7f4d1c1017b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 --2- 172.21.15.151:0/3216346464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d1c105c30 0x7f4d1c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 >> 172.21.15.151:0/3216346464 conn(0x7f4d1c074d80 msgr2=0x7f4d1c0ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:09.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 shutdown_connections 2024-04-16T03:05:09.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.610+0000 7f4d237fc700 1 -- 172.21.15.151:0/3216346464 wait complete. 2024-04-16T03:05:09.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 -- 172.21.15.151:0/146291609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ff530 msgr2=0x7fe22c0ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 --2- 172.21.15.151:0/146291609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ff530 0x7fe22c0ff910 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fe220009910 tx=0x7fe220009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:09.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 -- 172.21.15.151:0/146291609 shutdown_connections 2024-04-16T03:05:09.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 --2- 172.21.15.151:0/146291609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c0ffee0 0x7fe22c105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 --2- 172.21.15.151:0/146291609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ff530 0x7fe22c0ff910 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 -- 172.21.15.151:0/146291609 >> 172.21.15.151:0/146291609 conn(0x7fe22c074df0 msgr2=0x7fe22c075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:09.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 -- 172.21.15.151:0/146291609 shutdown_connections 2024-04-16T03:05:09.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 -- 172.21.15.151:0/146291609 wait complete. 2024-04-16T03:05:09.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 Processor -- start 2024-04-16T03:05:09.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.715+0000 7fe232a6e700 1 -- start start 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe232a6e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c0ff530 0x7fe22c19c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe232a6e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ffee0 0x7fe22c19cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe232a6e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c19d160 con 0x7fe22c0ffee0 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe232a6e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c196610 con 0x7fe22c0ff530 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe22bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ffee0 0x7fe22c19cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe22bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ffee0 0x7fe22c19cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41260/0 (socket says 172.21.15.151:41260) 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe22bfff700 1 -- 172.21.15.151:0/1759310719 learned_addr learned my addr 172.21.15.151:0/1759310719 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe23080a700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c0ff530 0x7fe22c19c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe22bfff700 1 -- 172.21.15.151:0/1759310719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c0ff530 msgr2=0x7fe22c19c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe22bfff700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c0ff530 0x7fe22c19c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe22bfff700 1 -- 172.21.15.151:0/1759310719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2200095f0 con 0x7fe22c0ffee0 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe23080a700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c0ff530 0x7fe22c19c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe22bfff700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ffee0 0x7fe22c19cad0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe21800d900 tx=0x7fe21800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:09.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe229ffb700 1 -- 172.21.15.151:0/1759310719 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe218009900 con 0x7fe22c0ffee0 2024-04-16T03:05:09.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.716+0000 7fe229ffb700 1 -- 172.21.15.151:0/1759310719 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe218010460 con 0x7fe22c0ffee0 2024-04-16T03:05:09.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.717+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c1968f0 con 0x7fe22c0ffee0 2024-04-16T03:05:09.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.717+0000 7fe229ffb700 1 -- 172.21.15.151:0/1759310719 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe21800f5f0 con 0x7fe22c0ffee0 2024-04-16T03:05:09.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.717+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c196e40 con 0x7fe22c0ffee0 2024-04-16T03:05:09.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.718+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c1091e0 con 0x7fe22c0ffee0 2024-04-16T03:05:09.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.719+0000 7fe229ffb700 1 -- 172.21.15.151:0/1759310719 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe2180105d0 con 0x7fe22c0ffee0 2024-04-16T03:05:09.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.719+0000 7fe229ffb700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe21c06ecc0 0x7fe21c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:09.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.719+0000 7fe229ffb700 1 -- 172.21.15.151:0/1759310719 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe21808ec00 con 0x7fe22c0ffee0 2024-04-16T03:05:09.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.719+0000 7fe23080a700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe21c06ecc0 0x7fe21c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:09.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.720+0000 7fe23080a700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe21c06ecc0 0x7fe21c071180 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe22000b5c0 tx=0x7fe2200053a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:09.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.726+0000 7fe229ffb700 1 -- 172.21.15.151:0/1759310719 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe218094020 con 0x7fe22c0ffee0 2024-04-16T03:05:09.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.974+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe22c04ea90 con 0x7fe22c0ffee0 2024-04-16T03:05:09.976 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:09 smithi151 ceph-mon[112233]: from='client.35464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:09.976 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:05:09.976 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:09 smithi151 ceph-mon[112233]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:09.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.974+0000 7fe229ffb700 1 -- 172.21.15.151:0/1759310719 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fe218017770 con 0x7fe22c0ffee0 2024-04-16T03:05:09.976 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:05:09.977 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:05:09.978 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:05:09.979 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:05:09.980 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:05:09.980 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:05:09.980 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:05:09.980 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:05:09.980 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe21c06ecc0 msgr2=0x7fe21c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe21c06ecc0 0x7fe21c071180 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe22000b5c0 tx=0x7fe2200053a0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ffee0 msgr2=0x7fe22c19cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ffee0 0x7fe22c19cad0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe21800d900 tx=0x7fe21800dcc0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 shutdown_connections 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe21c06ecc0 0x7fe21c071180 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c0ff530 0x7fe22c19c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 --2- 172.21.15.151:0/1759310719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe22c0ffee0 0x7fe22c19cad0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 >> 172.21.15.151:0/1759310719 conn(0x7fe22c074df0 msgr2=0x7fe22c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 shutdown_connections 2024-04-16T03:05:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:09.977+0000 7fe232a6e700 1 -- 172.21.15.151:0/1759310719 wait complete. 2024-04-16T03:05:09.982 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:05:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:09 smithi153 ceph-mon[87858]: from='client.35464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:05:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:09 smithi153 ceph-mon[87858]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:10.088 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.084+0000 7f4181a6d700 1 -- 172.21.15.151:0/2480169524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c0ffbd0 msgr2=0x7f417c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:10.088 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.084+0000 7f4181a6d700 1 --2- 172.21.15.151:0/2480169524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c0ffbd0 0x7f417c100050 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4164009910 tx=0x7f4164009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.084+0000 7f4181a6d700 1 -- 172.21.15.151:0/2480169524 shutdown_connections 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.084+0000 7f4181a6d700 1 --2- 172.21.15.151:0/2480169524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c0ffbd0 0x7f417c100050 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.084+0000 7f4181a6d700 1 --2- 172.21.15.151:0/2480169524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f417c105a40 0x7f417c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.084+0000 7f4181a6d700 1 -- 172.21.15.151:0/2480169524 >> 172.21.15.151:0/2480169524 conn(0x7f417c0755a0 msgr2=0x7f417c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.085+0000 7f4181a6d700 1 -- 172.21.15.151:0/2480169524 shutdown_connections 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.085+0000 7f4181a6d700 1 -- 172.21.15.151:0/2480169524 wait complete. 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.085+0000 7f4181a6d700 1 Processor -- start 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.085+0000 7f4181a6d700 1 -- start start 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.085+0000 7f4181a6d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f417c0ffbd0 0x7f417c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:10.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.085+0000 7f4181a6d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c105a40 0x7f417c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.085+0000 7f4181a6d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c19d0d0 con 0x7f417c105a40 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.085+0000 7f4181a6d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c196580 con 0x7f417c0ffbd0 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f4173fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c105a40 0x7f417c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f4173fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c105a40 0x7f417c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41284/0 (socket says 172.21.15.151:41284) 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f4173fff700 1 -- 172.21.15.151:0/1555355780 learned_addr learned my addr 172.21.15.151:0/1555355780 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f417b7fe700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f417c0ffbd0 0x7f417c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f4173fff700 1 -- 172.21.15.151:0/1555355780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f417c0ffbd0 msgr2=0x7f417c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f4173fff700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f417c0ffbd0 0x7f417c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f4173fff700 1 -- 172.21.15.151:0/1555355780 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41640095f0 con 0x7f417c105a40 2024-04-16T03:05:10.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f4173fff700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c105a40 0x7f417c19ca40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4164006010 tx=0x7f416400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f41797fa700 1 -- 172.21.15.151:0/1555355780 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f416401d070 con 0x7f417c105a40 2024-04-16T03:05:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417c196800 con 0x7f417c105a40 2024-04-16T03:05:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f41797fa700 1 -- 172.21.15.151:0/1555355780 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4164004fb0 con 0x7f417c105a40 2024-04-16T03:05:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f41797fa700 1 -- 172.21.15.151:0/1555355780 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f416400f690 con 0x7f417c105a40 2024-04-16T03:05:10.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.086+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417c196cf0 con 0x7f417c105a40 2024-04-16T03:05:10.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.087+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417c108fe0 con 0x7f417c105a40 2024-04-16T03:05:10.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.088+0000 7f41797fa700 1 -- 172.21.15.151:0/1555355780 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f41640228a0 con 0x7f417c105a40 2024-04-16T03:05:10.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.088+0000 7f41797fa700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416806ecc0 0x7f4168071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:10.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.088+0000 7f41797fa700 1 -- 172.21.15.151:0/1555355780 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4164090770 con 0x7f417c105a40 2024-04-16T03:05:10.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.088+0000 7f417b7fe700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416806ecc0 0x7f4168071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:10.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.089+0000 7f417b7fe700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416806ecc0 0x7f4168071180 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f416c005950 tx=0x7f416c005cf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:10.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.093+0000 7f41797fa700 1 -- 172.21.15.151:0/1555355780 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f416405cfb0 con 0x7f417c105a40 2024-04-16T03:05:10.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.310+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f417c02cca0 con 0x7f416806ecc0 2024-04-16T03:05:10.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.312+0000 7f41797fa700 1 -- 172.21.15.151:0/1555355780 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f417c02cca0 con 0x7f416806ecc0 2024-04-16T03:05:10.313 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:05:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:05:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:05:10.314 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416806ecc0 msgr2=0x7f4168071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416806ecc0 0x7f4168071180 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f416c005950 tx=0x7f416c005cf0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c105a40 msgr2=0x7f417c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c105a40 0x7f417c19ca40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4164006010 tx=0x7f416400be20 comp rx=0 tx=0).stop 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 shutdown_connections 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f416806ecc0 0x7f4168071180 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f417c0ffbd0 0x7f417c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 --2- 172.21.15.151:0/1555355780 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f417c105a40 0x7f417c19ca40 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 >> 172.21.15.151:0/1555355780 conn(0x7f417c0755a0 msgr2=0x7f417c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 shutdown_connections 2024-04-16T03:05:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.314+0000 7f4181a6d700 1 -- 172.21.15.151:0/1555355780 wait complete. 2024-04-16T03:05:10.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.419+0000 7f847dde8700 1 -- 172.21.15.151:0/3534358843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 msgr2=0x7f8478100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:10.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.419+0000 7f847dde8700 1 --2- 172.21.15.151:0/3534358843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 0x7f8478100050 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8460009910 tx=0x7f8460009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:10.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.419+0000 7f847dde8700 1 -- 172.21.15.151:0/3534358843 shutdown_connections 2024-04-16T03:05:10.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.419+0000 7f847dde8700 1 --2- 172.21.15.151:0/3534358843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 0x7f8478100050 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.419+0000 7f847dde8700 1 --2- 172.21.15.151:0/3534358843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8478105a40 0x7f8478105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.419+0000 7f847dde8700 1 -- 172.21.15.151:0/3534358843 >> 172.21.15.151:0/3534358843 conn(0x7f84780755a0 msgr2=0x7f84780759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:10.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.419+0000 7f847dde8700 1 -- 172.21.15.151:0/3534358843 shutdown_connections 2024-04-16T03:05:10.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.419+0000 7f847dde8700 1 -- 172.21.15.151:0/3534358843 wait complete. 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f847dde8700 1 Processor -- start 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f847dde8700 1 -- start start 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f847dde8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 0x7f8478197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f847dde8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8478105a40 0x7f8478197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f847dde8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781980e0 con 0x7f84780ffbd0 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f847dde8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8478198220 con 0x7f8478105a40 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f84777fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 0x7f8478197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f84777fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 0x7f8478197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41300/0 (socket says 172.21.15.151:41300) 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f84777fe700 1 -- 172.21.15.151:0/4260178115 learned_addr learned my addr 172.21.15.151:0/4260178115 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.420+0000 7f8476ffd700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8478105a40 0x7f8478197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.421+0000 7f84777fe700 1 -- 172.21.15.151:0/4260178115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8478105a40 msgr2=0x7f8478197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.421+0000 7f84777fe700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8478105a40 0x7f8478197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.421+0000 7f84777fe700 1 -- 172.21.15.151:0/4260178115 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84600095f0 con 0x7f84780ffbd0 2024-04-16T03:05:10.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.421+0000 7f84777fe700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 0x7f8478197580 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f847000ef30 tx=0x7f847000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:10.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.421+0000 7f8474ff9700 1 -- 172.21.15.151:0/4260178115 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8470018070 con 0x7f84780ffbd0 2024-04-16T03:05:10.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.421+0000 7f8474ff9700 1 -- 172.21.15.151:0/4260178115 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f847000f040 con 0x7f84780ffbd0 2024-04-16T03:05:10.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.421+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84781a05f0 con 0x7f84780ffbd0 2024-04-16T03:05:10.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.421+0000 7f8474ff9700 1 -- 172.21.15.151:0/4260178115 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8470004020 con 0x7f84780ffbd0 2024-04-16T03:05:10.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.421+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84781a0a60 con 0x7f84780ffbd0 2024-04-16T03:05:10.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.422+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8478107fd0 con 0x7f84780ffbd0 2024-04-16T03:05:10.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.423+0000 7f8474ff9700 1 -- 172.21.15.151:0/4260178115 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8470004ad0 con 0x7f84780ffbd0 2024-04-16T03:05:10.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.423+0000 7f8474ff9700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f846406ecc0 0x7f8464071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:10.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.423+0000 7f8474ff9700 1 -- 172.21.15.151:0/4260178115 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f847008ec10 con 0x7f84780ffbd0 2024-04-16T03:05:10.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.423+0000 7f8476ffd700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f846406ecc0 0x7f8464071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:10.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.424+0000 7f8476ffd700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f846406ecc0 0x7f8464071180 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8478104d40 tx=0x7f8460005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:10.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.428+0000 7f8474ff9700 1 -- 172.21.15.151:0/4260178115 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8470057a40 con 0x7f84780ffbd0 2024-04-16T03:05:10.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.709+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f847804ea90 con 0x7f84780ffbd0 2024-04-16T03:05:10.711 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:10 smithi151 ceph-mon[112233]: from='client.35472 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:10.711 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:10 smithi151 ceph-mon[112233]: pgmap v797: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:10.711 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:10 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3216346464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:05:10.711 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:10 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1759310719' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.710+0000 7f8474ff9700 1 -- 172.21.15.151:0/4260178115 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f847005b100 con 0x7f84780ffbd0 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:05:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:05:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:05:10.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f846406ecc0 msgr2=0x7f8464071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:10.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f846406ecc0 0x7f8464071180 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8478104d40 tx=0x7f8460005940 comp rx=0 tx=0).stop 2024-04-16T03:05:10.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 msgr2=0x7f8478197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:10.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 0x7f8478197580 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f847000ef30 tx=0x7f847000c3a0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 shutdown_connections 2024-04-16T03:05:10.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f846406ecc0 0x7f8464071180 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84780ffbd0 0x7f8478197580 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 --2- 172.21.15.151:0/4260178115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8478105a40 0x7f8478197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:10.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 >> 172.21.15.151:0/4260178115 conn(0x7f84780755a0 msgr2=0x7f84780fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:10.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 shutdown_connections 2024-04-16T03:05:10.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:10.712+0000 7f847dde8700 1 -- 172.21.15.151:0/4260178115 wait complete. 2024-04-16T03:05:11.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:10 smithi153 ceph-mon[87858]: from='client.35472 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:11.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:10 smithi153 ceph-mon[87858]: pgmap v797: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:11.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:10 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3216346464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:05:11.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:10 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1759310719' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:05:11.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:11 smithi151 ceph-mon[112233]: from='client.35484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:11.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:11 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4260178115' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:05:12.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:11 smithi153 ceph-mon[87858]: from='client.35484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:12.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:11 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4260178115' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:05:12.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:12 smithi151 ceph-mon[112233]: pgmap v798: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:12.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:12 smithi153 ceph-mon[87858]: pgmap v798: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:14.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:14 smithi151 ceph-mon[112233]: pgmap v799: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:15.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:14 smithi153 ceph-mon[87858]: pgmap v799: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:16 smithi151 ceph-mon[112233]: pgmap v800: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:17.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:16 smithi153 ceph-mon[87858]: pgmap v800: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:18.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:18 smithi151 ceph-mon[112233]: pgmap v801: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:19.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:18 smithi153 ceph-mon[87858]: pgmap v801: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:20.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:20 smithi151 ceph-mon[112233]: pgmap v802: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:20 smithi153 ceph-mon[87858]: pgmap v802: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:22 smithi153 ceph-mon[87858]: pgmap v803: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:22.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:22 smithi151 ceph-mon[112233]: pgmap v803: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:24.882 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:24 smithi151 ceph-mon[112233]: pgmap v804: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:24.882 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:05:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:24 smithi153 ceph-mon[87858]: pgmap v804: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:05:26.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:26 smithi151 ceph-mon[112233]: pgmap v805: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:27.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:26 smithi153 ceph-mon[87858]: pgmap v805: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:28 smithi151 ceph-mon[112233]: pgmap v806: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:29.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:28 smithi153 ceph-mon[87858]: pgmap v806: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:30.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:30 smithi151 ceph-mon[112233]: pgmap v807: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:31.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:30 smithi153 ceph-mon[87858]: pgmap v807: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:32 smithi151 ceph-mon[112233]: pgmap v808: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:33.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:32 smithi153 ceph-mon[87858]: pgmap v808: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:34.989 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:34 smithi151 ceph-mon[112233]: pgmap v809: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:35.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:34 smithi153 ceph-mon[87858]: pgmap v809: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:36 smithi151 ceph-mon[112233]: pgmap v810: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:36 smithi153 ceph-mon[87858]: pgmap v810: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:38 smithi151 ceph-mon[112233]: pgmap v811: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:39.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:38 smithi153 ceph-mon[87858]: pgmap v811: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:39.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:05:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:05:40.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.830+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/3706451191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 msgr2=0x7fb5a01011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:40.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.830+0000 7fb5a5bbf700 1 --2- 172.21.15.151:0/3706451191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 0x7fb5a01011c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb590009910 tx=0x7fb590009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:40.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.830+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/3706451191 shutdown_connections 2024-04-16T03:05:40.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.830+0000 7fb5a5bbf700 1 --2- 172.21.15.151:0/3706451191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a0101790 0x7fb5a01057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:40.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.830+0000 7fb5a5bbf700 1 --2- 172.21.15.151:0/3706451191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 0x7fb5a01011c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:40.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.830+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/3706451191 >> 172.21.15.151:0/3706451191 conn(0x7fb5a00fc6b0 msgr2=0x7fb5a00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:40.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.830+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/3706451191 shutdown_connections 2024-04-16T03:05:40.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.830+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/3706451191 wait complete. 2024-04-16T03:05:40.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb5a5bbf700 1 Processor -- start 2024-04-16T03:05:40.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb5a5bbf700 1 -- start start 2024-04-16T03:05:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb5a5bbf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 0x7fb5a0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb5a5bbf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a0101790 0x7fb5a00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb5a5bbf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a0073b10 con 0x7fb5a0100de0 2024-04-16T03:05:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb5a5bbf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a0073c50 con 0x7fb5a0101790 2024-04-16T03:05:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb59f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 0x7fb5a0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb59f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 0x7fb5a0073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40130/0 (socket says 172.21.15.151:40130) 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb59f7fe700 1 -- 172.21.15.151:0/2809887385 learned_addr learned my addr 172.21.15.151:0/2809887385 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.831+0000 7fb597fff700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a0101790 0x7fb5a00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.832+0000 7fb59f7fe700 1 -- 172.21.15.151:0/2809887385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a0101790 msgr2=0x7fb5a00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.832+0000 7fb59f7fe700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a0101790 0x7fb5a00735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.832+0000 7fb59f7fe700 1 -- 172.21.15.151:0/2809887385 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5900095f0 con 0x7fb5a0100de0 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.832+0000 7fb59f7fe700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 0x7fb5a0073090 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb590006010 tx=0x7fb59000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.832+0000 7fb59d7fa700 1 -- 172.21.15.151:0/2809887385 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb59001d070 con 0x7fb5a0100de0 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.832+0000 7fb59d7fa700 1 -- 172.21.15.151:0/2809887385 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb59000f460 con 0x7fb5a0100de0 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.832+0000 7fb59d7fa700 1 -- 172.21.15.151:0/2809887385 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb590017630 con 0x7fb5a0100de0 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.832+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a0073ed0 con 0x7fb5a0100de0 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.832+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a01a2ca0 con 0x7fb5a0100de0 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.833+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a0109180 con 0x7fb5a0100de0 2024-04-16T03:05:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.834+0000 7fb59d7fa700 1 -- 172.21.15.151:0/2809887385 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb59000f5d0 con 0x7fb5a0100de0 2024-04-16T03:05:40.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.834+0000 7fb59d7fa700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58c06ecc0 0x7fb58c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:40.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.834+0000 7fb597fff700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58c06ecc0 0x7fb58c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:40.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.834+0000 7fb59d7fa700 1 -- 172.21.15.151:0/2809887385 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb5900902d0 con 0x7fb5a0100de0 2024-04-16T03:05:40.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.835+0000 7fb597fff700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58c06ecc0 0x7fb58c071180 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0074880 tx=0x7fb58800a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:40.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:40.840+0000 7fb59d7fa700 1 -- 172.21.15.151:0/2809887385 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb590059100 con 0x7fb5a0100de0 2024-04-16T03:05:40.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:40 smithi151 ceph-mon[112233]: pgmap v812: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:41.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:40 smithi153 ceph-mon[87858]: pgmap v812: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:41.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.055+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5a00611d0 con 0x7fb58c06ecc0 2024-04-16T03:05:41.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.057+0000 7fb59d7fa700 1 -- 172.21.15.151:0/2809887385 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb5a00611d0 con 0x7fb58c06ecc0 2024-04-16T03:05:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58c06ecc0 msgr2=0x7fb58c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58c06ecc0 0x7fb58c071180 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0074880 tx=0x7fb58800a380 comp rx=0 tx=0).stop 2024-04-16T03:05:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 msgr2=0x7fb5a0073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 0x7fb5a0073090 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb590006010 tx=0x7fb59000bd70 comp rx=0 tx=0).stop 2024-04-16T03:05:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 shutdown_connections 2024-04-16T03:05:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58c06ecc0 0x7fb58c071180 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5a0100de0 0x7fb5a0073090 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 --2- 172.21.15.151:0/2809887385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a0101790 0x7fb5a00735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 >> 172.21.15.151:0/2809887385 conn(0x7fb5a00fc6b0 msgr2=0x7fb5a00fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:41.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 shutdown_connections 2024-04-16T03:05:41.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.059+0000 7fb5a5bbf700 1 -- 172.21.15.151:0/2809887385 wait complete. 2024-04-16T03:05:41.072 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:05:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.165+0000 7fe3c4596700 1 -- 172.21.15.151:0/3125644196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 msgr2=0x7fe3bc0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.165+0000 7fe3c4596700 1 --2- 172.21.15.151:0/3125644196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 0x7fe3bc0ffec0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009910 tx=0x7fe3b0009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.165+0000 7fe3c4596700 1 -- 172.21.15.151:0/3125644196 shutdown_connections 2024-04-16T03:05:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.165+0000 7fe3c4596700 1 --2- 172.21.15.151:0/3125644196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 0x7fe3bc0ffec0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.165+0000 7fe3c4596700 1 --2- 172.21.15.151:0/3125644196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc105a40 0x7fe3bc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.165+0000 7fe3c4596700 1 -- 172.21.15.151:0/3125644196 >> 172.21.15.151:0/3125644196 conn(0x7fe3bc0fb570 msgr2=0x7fe3bc0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.165+0000 7fe3c4596700 1 -- 172.21.15.151:0/3125644196 shutdown_connections 2024-04-16T03:05:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.165+0000 7fe3c4596700 1 -- 172.21.15.151:0/3125644196 wait complete. 2024-04-16T03:05:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c4596700 1 Processor -- start 2024-04-16T03:05:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c4596700 1 -- start start 2024-04-16T03:05:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c4596700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 0x7fe3bc197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c4596700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc105a40 0x7fe3bc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c4596700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc1980f0 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c4596700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc198230 con 0x7fe3bc105a40 2024-04-16T03:05:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c2332700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 0x7fe3bc197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c2332700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 0x7fe3bc197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40152/0 (socket says 172.21.15.151:40152) 2024-04-16T03:05:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c2332700 1 -- 172.21.15.151:0/3786430933 learned_addr learned my addr 172.21.15.151:0/3786430933 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c1b31700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc105a40 0x7fe3bc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c2332700 1 -- 172.21.15.151:0/3786430933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc105a40 msgr2=0x7fe3bc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.166+0000 7fe3c2332700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc105a40 0x7fe3bc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.167+0000 7fe3c2332700 1 -- 172.21.15.151:0/3786430933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b00095f0 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.167+0000 7fe3c2332700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 0x7fe3bc197590 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4007e90 tx=0x7fe3b400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.167+0000 7fe3bb7fe700 1 -- 172.21.15.151:0/3786430933 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3b400de60 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.167+0000 7fe3bb7fe700 1 -- 172.21.15.151:0/3786430933 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3b400f040 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.167+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc1a0600 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.167+0000 7fe3bb7fe700 1 -- 172.21.15.151:0/3786430933 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3b4004520 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.167+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc1a0ac0 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.168+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc107fd0 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.168+0000 7fe3bb7fe700 1 -- 172.21.15.151:0/3786430933 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe3b4004750 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.169+0000 7fe3bb7fe700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3a406ecc0 0x7fe3a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.169+0000 7fe3bb7fe700 1 -- 172.21.15.151:0/3786430933 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe3b408ea80 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.169+0000 7fe3c1b31700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3a406ecc0 0x7fe3a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:41.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.169+0000 7fe3c1b31700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3a406ecc0 0x7fe3a4071180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc069550 tx=0x7fe3b001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:41.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.175+0000 7fe3bb7fe700 1 -- 172.21.15.151:0/3786430933 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe3b4057830 con 0x7fe3bc0ffa40 2024-04-16T03:05:41.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.391+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3bc02cce0 con 0x7fe3a406ecc0 2024-04-16T03:05:41.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.392+0000 7fe3bb7fe700 1 -- 172.21.15.151:0/3786430933 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe3bc02cce0 con 0x7fe3a406ecc0 2024-04-16T03:05:41.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3a406ecc0 msgr2=0x7fe3a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3a406ecc0 0x7fe3a4071180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc069550 tx=0x7fe3b001a040 comp rx=0 tx=0).stop 2024-04-16T03:05:41.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 msgr2=0x7fe3bc197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 0x7fe3bc197590 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4007e90 tx=0x7fe3b400d3b0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 shutdown_connections 2024-04-16T03:05:41.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe3a406ecc0 0x7fe3a4071180 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe3bc0ffa40 0x7fe3bc197590 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 --2- 172.21.15.151:0/3786430933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc105a40 0x7fe3bc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 >> 172.21.15.151:0/3786430933 conn(0x7fe3bc0fb570 msgr2=0x7fe3bc0fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:41.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 shutdown_connections 2024-04-16T03:05:41.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.394+0000 7fe3c4596700 1 -- 172.21.15.151:0/3786430933 wait complete. 2024-04-16T03:05:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.500+0000 7f481d564700 1 -- 172.21.15.151:0/3459205307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818068670 msgr2=0x7f4818068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.500+0000 7f481d564700 1 --2- 172.21.15.151:0/3459205307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818068670 0x7f4818068a50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4808009910 tx=0x7f4808009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.500+0000 7f481d564700 1 -- 172.21.15.151:0/3459205307 shutdown_connections 2024-04-16T03:05:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.500+0000 7f481d564700 1 --2- 172.21.15.151:0/3459205307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4818069020 0x7f4818105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.500+0000 7f481d564700 1 --2- 172.21.15.151:0/3459205307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818068670 0x7f4818068a50 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.500+0000 7f481d564700 1 -- 172.21.15.151:0/3459205307 >> 172.21.15.151:0/3459205307 conn(0x7f48180755a0 msgr2=0x7f48180759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.500+0000 7f481d564700 1 -- 172.21.15.151:0/3459205307 shutdown_connections 2024-04-16T03:05:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.500+0000 7f481d564700 1 -- 172.21.15.151:0/3459205307 wait complete. 2024-04-16T03:05:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.500+0000 7f481d564700 1 Processor -- start 2024-04-16T03:05:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f481d564700 1 -- start start 2024-04-16T03:05:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f481d564700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4818068670 0x7f481819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f481d564700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818069020 0x7f481819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f481d564700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481819f2e0 con 0x7f4818069020 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f481d564700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4818198790 con 0x7f4818068670 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f480e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818069020 0x7f481819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f480e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818069020 0x7f481819ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40172/0 (socket says 172.21.15.151:40172) 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f480e7fc700 1 -- 172.21.15.151:0/2626040261 learned_addr learned my addr 172.21.15.151:0/2626040261 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f4816ffd700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4818068670 0x7f481819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f480e7fc700 1 -- 172.21.15.151:0/2626040261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4818068670 msgr2=0x7f481819e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f480e7fc700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4818068670 0x7f481819e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.501+0000 7f480e7fc700 1 -- 172.21.15.151:0/2626040261 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48080095f0 con 0x7f4818069020 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.502+0000 7f480e7fc700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818069020 0x7f481819ec50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f480000d900 tx=0x7f480000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.502+0000 7f4814ff9700 1 -- 172.21.15.151:0/2626040261 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4800009900 con 0x7f4818069020 2024-04-16T03:05:41.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.502+0000 7f4814ff9700 1 -- 172.21.15.151:0/2626040261 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4800010460 con 0x7f4818069020 2024-04-16T03:05:41.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.502+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4818198a70 con 0x7f4818069020 2024-04-16T03:05:41.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.502+0000 7f4814ff9700 1 -- 172.21.15.151:0/2626040261 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f480000f5f0 con 0x7f4818069020 2024-04-16T03:05:41.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.502+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4818198fc0 con 0x7f4818069020 2024-04-16T03:05:41.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.503+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481804ea90 con 0x7f4818069020 2024-04-16T03:05:41.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.503+0000 7f4814ff9700 1 -- 172.21.15.151:0/2626040261 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f48000105d0 con 0x7f4818069020 2024-04-16T03:05:41.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.504+0000 7f4814ff9700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f480406ecc0 0x7f4804071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:41.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.504+0000 7f4814ff9700 1 -- 172.21.15.151:0/2626040261 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f480008ecc0 con 0x7f4818069020 2024-04-16T03:05:41.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.504+0000 7f4816ffd700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f480406ecc0 0x7f4804071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:41.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.504+0000 7f4816ffd700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f480406ecc0 0x7f4804071180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f480800b5c0 tx=0x7f480801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:41.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.509+0000 7f4814ff9700 1 -- 172.21.15.151:0/2626040261 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f480005b5a0 con 0x7f4818069020 2024-04-16T03:05:41.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.716+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4818199460 con 0x7f480406ecc0 2024-04-16T03:05:41.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.728+0000 7f4814ff9700 1 -- 172.21.15.151:0/2626040261 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4818199460 con 0x7f480406ecc0 2024-04-16T03:05:41.730 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:05:41.730 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (27m) 5m ago 38m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:05:41.730 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (38m) 5m ago 38m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:05:41.730 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (37m) 5m ago 37m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:05:41.730 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (24m) 5m ago 38m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:05:41.730 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (24m) 5m ago 37m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (26m) 5m ago 38m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (21m) 5m ago 33m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (20m) 5m ago 32m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (5m) 5m ago 32m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (20m) 5m ago 32m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (29m) 5m ago 40m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (28m) 5m ago 37m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (25m) 5m ago 40m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (25m) 5m ago 37m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (27m) 5m ago 38m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (27m) 5m ago 37m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (24m) 5m ago 36m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:05:41.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (23m) 5m ago 36m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:05:41.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (23m) 5m ago 35m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:05:41.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (22m) 5m ago 35m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:05:41.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (22m) 5m ago 34m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:05:41.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (21m) 5m ago 34m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:05:41.732 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (27m) 5m ago 38m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.729+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f480406ecc0 msgr2=0x7f4804071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.729+0000 7f481d564700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f480406ecc0 0x7f4804071180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f480800b5c0 tx=0x7f480801a040 comp rx=0 tx=0).stop 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.729+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818069020 msgr2=0x7f481819ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.729+0000 7f481d564700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818069020 0x7f481819ec50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f480000d900 tx=0x7f480000dcc0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.729+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 shutdown_connections 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.729+0000 7f481d564700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f480406ecc0 0x7f4804071180 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.729+0000 7f481d564700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4818068670 0x7f481819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.729+0000 7f481d564700 1 --2- 172.21.15.151:0/2626040261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4818069020 0x7f481819ec50 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.729+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 >> 172.21.15.151:0/2626040261 conn(0x7f48180755a0 msgr2=0x7f4818102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.730+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 shutdown_connections 2024-04-16T03:05:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.730+0000 7f481d564700 1 -- 172.21.15.151:0/2626040261 wait complete. 2024-04-16T03:05:41.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.833+0000 7fdef451d700 1 -- 172.21.15.151:0/2945809861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec0ffbd0 msgr2=0x7fdeec100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.833+0000 7fdef451d700 1 --2- 172.21.15.151:0/2945809861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec0ffbd0 0x7fdeec100050 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009910 tx=0x7fdee8009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.833+0000 7fdef451d700 1 -- 172.21.15.151:0/2945809861 shutdown_connections 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.833+0000 7fdef451d700 1 --2- 172.21.15.151:0/2945809861 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec0ffbd0 0x7fdeec100050 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.833+0000 7fdef451d700 1 --2- 172.21.15.151:0/2945809861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeec105a40 0x7fdeec105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.833+0000 7fdef451d700 1 -- 172.21.15.151:0/2945809861 >> 172.21.15.151:0/2945809861 conn(0x7fdeec0755a0 msgr2=0x7fdeec0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.833+0000 7fdef451d700 1 -- 172.21.15.151:0/2945809861 shutdown_connections 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.833+0000 7fdef451d700 1 -- 172.21.15.151:0/2945809861 wait complete. 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef451d700 1 Processor -- start 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef451d700 1 -- start start 2024-04-16T03:05:41.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef451d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeec0ffbd0 0x7fdeec19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef451d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec105a40 0x7fdeec19cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef451d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec19d200 con 0x7fdeec105a40 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef451d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec1966b0 con 0x7fdeec0ffbd0 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef1ab8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec105a40 0x7fdeec19cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef1ab8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec105a40 0x7fdeec19cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40198/0 (socket says 172.21.15.151:40198) 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef1ab8700 1 -- 172.21.15.151:0/2365170782 learned_addr learned my addr 172.21.15.151:0/2365170782 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef22b9700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeec0ffbd0 0x7fdeec19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef1ab8700 1 -- 172.21.15.151:0/2365170782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeec0ffbd0 msgr2=0x7fdeec19c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef1ab8700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeec0ffbd0 0x7fdeec19c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.834+0000 7fdef1ab8700 1 -- 172.21.15.151:0/2365170782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee80095f0 con 0x7fdeec105a40 2024-04-16T03:05:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.835+0000 7fdef1ab8700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec105a40 0x7fdeec19cb70 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006010 tx=0x7fdee800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:41.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.835+0000 7fdedf7fe700 1 -- 172.21.15.151:0/2365170782 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdee801d070 con 0x7fdeec105a40 2024-04-16T03:05:41.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.835+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec196930 con 0x7fdeec105a40 2024-04-16T03:05:41.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.835+0000 7fdedf7fe700 1 -- 172.21.15.151:0/2365170782 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdee8004fb0 con 0x7fdeec105a40 2024-04-16T03:05:41.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.835+0000 7fdedf7fe700 1 -- 172.21.15.151:0/2365170782 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdee8022ea0 con 0x7fdeec105a40 2024-04-16T03:05:41.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.835+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec196e20 con 0x7fdeec105a40 2024-04-16T03:05:41.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.836+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec109070 con 0x7fdeec105a40 2024-04-16T03:05:41.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.836+0000 7fdedf7fe700 1 -- 172.21.15.151:0/2365170782 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fdee8022470 con 0x7fdeec105a40 2024-04-16T03:05:41.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.837+0000 7fdedf7fe700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fded806ecc0 0x7fded8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:41.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.837+0000 7fdedf7fe700 1 -- 172.21.15.151:0/2365170782 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdee8090350 con 0x7fdeec105a40 2024-04-16T03:05:41.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.837+0000 7fdef22b9700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fded806ecc0 0x7fded8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:41.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.837+0000 7fdef22b9700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fded806ecc0 0x7fded8071180 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdee0006fd0 tx=0x7fdee0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:41.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:41.842+0000 7fdedf7fe700 1 -- 172.21.15.151:0/2365170782 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdee805cd60 con 0x7fdeec105a40 2024-04-16T03:05:42.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.133+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdeec04ea90 con 0x7fdeec105a40 2024-04-16T03:05:42.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.133+0000 7fdedf7fe700 1 -- 172.21.15.151:0/2365170782 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fdee8022720 con 0x7fdeec105a40 2024-04-16T03:05:42.136 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:05:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:05:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:05:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:05:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:05:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:05:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:05:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:05:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:05:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:05:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:05:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:05:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:05:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:05:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:05:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:05:42.138 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:05:42.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fded806ecc0 msgr2=0x7fded8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fded806ecc0 0x7fded8071180 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdee0006fd0 tx=0x7fdee0008040 comp rx=0 tx=0).stop 2024-04-16T03:05:42.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec105a40 msgr2=0x7fdeec19cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec105a40 0x7fdeec19cb70 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006010 tx=0x7fdee800be20 comp rx=0 tx=0).stop 2024-04-16T03:05:42.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 shutdown_connections 2024-04-16T03:05:42.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fded806ecc0 0x7fded8071180 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeec0ffbd0 0x7fdeec19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 --2- 172.21.15.151:0/2365170782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdeec105a40 0x7fdeec19cb70 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 >> 172.21.15.151:0/2365170782 conn(0x7fdeec0755a0 msgr2=0x7fdeec068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:42.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 shutdown_connections 2024-04-16T03:05:42.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.137+0000 7fdef451d700 1 -- 172.21.15.151:0/2365170782 wait complete. 2024-04-16T03:05:42.247 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.242+0000 7f14049db700 1 -- 172.21.15.151:0/240077549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ffee0 msgr2=0x7f1400105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.242+0000 7f14049db700 1 --2- 172.21.15.151:0/240077549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ffee0 0x7f1400105870 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009910 tx=0x7f13e8009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:42.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.242+0000 7f14049db700 1 -- 172.21.15.151:0/240077549 shutdown_connections 2024-04-16T03:05:42.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.242+0000 7f14049db700 1 --2- 172.21.15.151:0/240077549 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ffee0 0x7f1400105870 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.242+0000 7f14049db700 1 --2- 172.21.15.151:0/240077549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14000ff530 0x7f14000ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.242+0000 7f14049db700 1 -- 172.21.15.151:0/240077549 >> 172.21.15.151:0/240077549 conn(0x7f1400074df0 msgr2=0x7f1400075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:42.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.242+0000 7f14049db700 1 -- 172.21.15.151:0/240077549 shutdown_connections 2024-04-16T03:05:42.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.242+0000 7f14049db700 1 -- 172.21.15.151:0/240077549 wait complete. 2024-04-16T03:05:42.248 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f14049db700 1 Processor -- start 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f14049db700 1 -- start start 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f14049db700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ff530 0x7f14001004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f14049db700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14000ffee0 0x7f14001009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f14049db700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1400100f30 con 0x7f14000ff530 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f14049db700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1400101070 con 0x7f14000ffee0 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f13fe59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ff530 0x7f14001004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f13fe59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ff530 0x7f14001004b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40228/0 (socket says 172.21.15.151:40228) 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f13fe59c700 1 -- 172.21.15.151:0/17896183 learned_addr learned my addr 172.21.15.151:0/17896183 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f13f7fff700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14000ffee0 0x7f14001009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f13fe59c700 1 -- 172.21.15.151:0/17896183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14000ffee0 msgr2=0x7f14001009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f13fe59c700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14000ffee0 0x7f14001009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.243+0000 7f13fe59c700 1 -- 172.21.15.151:0/17896183 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e80095f0 con 0x7f14000ff530 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.244+0000 7f13fe59c700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ff530 0x7f14001004b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c8a0 tx=0x7f13f000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.244+0000 7f13f77fe700 1 -- 172.21.15.151:0/17896183 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f00078e0 con 0x7f14000ff530 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.244+0000 7f13f77fe700 1 -- 172.21.15.151:0/17896183 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13f000f450 con 0x7f14000ff530 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.244+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14001a29a0 con 0x7f14000ff530 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.244+0000 7f13f77fe700 1 -- 172.21.15.151:0/17896183 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f000e5e0 con 0x7f14000ff530 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.244+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14001a2db0 con 0x7f14000ff530 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.245+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1400109150 con 0x7f14000ff530 2024-04-16T03:05:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.245+0000 7f13f77fe700 1 -- 172.21.15.151:0/17896183 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f13f000f5c0 con 0x7f14000ff530 2024-04-16T03:05:42.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.246+0000 7f13f77fe700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13ec06ecc0 0x7f13ec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:42.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.246+0000 7f13f77fe700 1 -- 172.21.15.151:0/17896183 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f13f008ee80 con 0x7f14000ff530 2024-04-16T03:05:42.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.246+0000 7f13f7fff700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13ec06ecc0 0x7f13ec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:42.252 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.246+0000 7f13f7fff700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13ec06ecc0 0x7f13ec071180 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1400101ca0 tx=0x7f13e801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:42.253 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.250+0000 7f13f77fe700 1 -- 172.21.15.151:0/17896183 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f13f0057c30 con 0x7f14000ff530 2024-04-16T03:05:42.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.492+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f140004ea90 con 0x7f14000ff530 2024-04-16T03:05:42.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.493+0000 7f13f77fe700 1 -- 172.21.15.151:0/17896183 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f13f005b2f0 con 0x7f14000ff530 2024-04-16T03:05:42.495 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:05:42.496 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:05:42.497 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:05:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:05:42.499 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:05:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:05:42.499 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:05:42.499 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:05:42.499 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:05:42.499 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:05:42.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13ec06ecc0 msgr2=0x7f13ec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13ec06ecc0 0x7f13ec071180 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1400101ca0 tx=0x7f13e801a040 comp rx=0 tx=0).stop 2024-04-16T03:05:42.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ff530 msgr2=0x7f14001004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ff530 0x7f14001004b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c8a0 tx=0x7f13f000cbb0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 shutdown_connections 2024-04-16T03:05:42.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13ec06ecc0 0x7f13ec071180 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14000ff530 0x7f14001004b0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 --2- 172.21.15.151:0/17896183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14000ffee0 0x7f14001009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 >> 172.21.15.151:0/17896183 conn(0x7f1400074df0 msgr2=0x7f14001050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:42.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 shutdown_connections 2024-04-16T03:05:42.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.495+0000 7f14049db700 1 -- 172.21.15.151:0/17896183 wait complete. 2024-04-16T03:05:42.501 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:05:42.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.602+0000 7f53100d9700 1 -- 172.21.15.151:0/39973496 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308101780 msgr2=0x7f53081057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.602+0000 7f53100d9700 1 --2- 172.21.15.151:0/39973496 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308101780 0x7f53081057d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009910 tx=0x7f52f8009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:42.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.602+0000 7f53100d9700 1 -- 172.21.15.151:0/39973496 shutdown_connections 2024-04-16T03:05:42.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.602+0000 7f53100d9700 1 --2- 172.21.15.151:0/39973496 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308101780 0x7f53081057d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.602+0000 7f53100d9700 1 --2- 172.21.15.151:0/39973496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5308100dd0 0x7f53081011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.602+0000 7f53100d9700 1 -- 172.21.15.151:0/39973496 >> 172.21.15.151:0/39973496 conn(0x7f53080fc640 msgr2=0x7f53080fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:42.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.603+0000 7f53100d9700 1 -- 172.21.15.151:0/39973496 shutdown_connections 2024-04-16T03:05:42.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.603+0000 7f53100d9700 1 -- 172.21.15.151:0/39973496 wait complete. 2024-04-16T03:05:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.603+0000 7f53100d9700 1 Processor -- start 2024-04-16T03:05:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.603+0000 7f53100d9700 1 -- start start 2024-04-16T03:05:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.603+0000 7f53100d9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308100dd0 0x7f5308073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.603+0000 7f53100d9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5308101780 0x7f53080735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.603+0000 7f53100d9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5308073b10 con 0x7f5308100dd0 2024-04-16T03:05:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.603+0000 7f53100d9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5308073c50 con 0x7f5308101780 2024-04-16T03:05:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.604+0000 7f530de75700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308100dd0 0x7f5308073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:42.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.604+0000 7f530de75700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308100dd0 0x7f5308073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40246/0 (socket says 172.21.15.151:40246) 2024-04-16T03:05:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.604+0000 7f530de75700 1 -- 172.21.15.151:0/3320686742 learned_addr learned my addr 172.21.15.151:0/3320686742 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.604+0000 7f530d674700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5308101780 0x7f53080735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.604+0000 7f530de75700 1 -- 172.21.15.151:0/3320686742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5308101780 msgr2=0x7f53080735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.604+0000 7f530de75700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5308101780 0x7f53080735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.604+0000 7f530de75700 1 -- 172.21.15.151:0/3320686742 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f80095f0 con 0x7f5308100dd0 2024-04-16T03:05:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.604+0000 7f530de75700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308100dd0 0x7f5308073090 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f530400d8d0 tx=0x7f530400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:42.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.605+0000 7f52feffd700 1 -- 172.21.15.151:0/3320686742 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53040098a0 con 0x7f5308100dd0 2024-04-16T03:05:42.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.605+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53081a2990 con 0x7f5308100dd0 2024-04-16T03:05:42.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.605+0000 7f52feffd700 1 -- 172.21.15.151:0/3320686742 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5304010460 con 0x7f5308100dd0 2024-04-16T03:05:42.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.605+0000 7f52feffd700 1 -- 172.21.15.151:0/3320686742 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f530400f5f0 con 0x7f5308100dd0 2024-04-16T03:05:42.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.606+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53081a2d40 con 0x7f5308100dd0 2024-04-16T03:05:42.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.606+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53081091d0 con 0x7f5308100dd0 2024-04-16T03:05:42.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.607+0000 7f52feffd700 1 -- 172.21.15.151:0/3320686742 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5304009a00 con 0x7f5308100dd0 2024-04-16T03:05:42.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.608+0000 7f52feffd700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f52f406ecc0 0x7f52f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:42.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.608+0000 7f530d674700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f52f406ecc0 0x7f52f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:42.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.608+0000 7f52feffd700 1 -- 172.21.15.151:0/3320686742 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f530408ed30 con 0x7f5308100dd0 2024-04-16T03:05:42.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.608+0000 7f530d674700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f52f406ecc0 0x7f52f4071180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f52f8006010 tx=0x7f52f801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:42.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.612+0000 7f52feffd700 1 -- 172.21.15.151:0/3320686742 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5304057ae0 con 0x7f5308100dd0 2024-04-16T03:05:42.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.829+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f53080611d0 con 0x7f52f406ecc0 2024-04-16T03:05:42.832 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:42 smithi151 ceph-mon[112233]: from='client.35492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:42.832 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:42 smithi151 ceph-mon[112233]: pgmap v813: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:42.832 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:42 smithi151 ceph-mon[112233]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:42.833 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:42 smithi151 ceph-mon[112233]: from='client.35500 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:42.833 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:42 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2365170782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:05:42.833 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:42 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/17896183' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:05:42.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.832+0000 7f52feffd700 1 -- 172.21.15.151:0/3320686742 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f53080611d0 con 0x7f52f406ecc0 2024-04-16T03:05:42.833 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:05:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:05:42.834 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:05:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f52f406ecc0 msgr2=0x7f52f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f52f406ecc0 0x7f52f4071180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f52f8006010 tx=0x7f52f801a040 comp rx=0 tx=0).stop 2024-04-16T03:05:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308100dd0 msgr2=0x7f5308073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308100dd0 0x7f5308073090 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f530400d8d0 tx=0x7f530400dbe0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 shutdown_connections 2024-04-16T03:05:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f52f406ecc0 0x7f52f4071180 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5308100dd0 0x7f5308073090 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 --2- 172.21.15.151:0/3320686742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5308101780 0x7f53080735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 >> 172.21.15.151:0/3320686742 conn(0x7f53080fc640 msgr2=0x7f53080fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:42.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 shutdown_connections 2024-04-16T03:05:42.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.834+0000 7f53100d9700 1 -- 172.21.15.151:0/3320686742 wait complete. 2024-04-16T03:05:42.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 -- 172.21.15.151:0/3673962510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 msgr2=0x7fee6010ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 --2- 172.21.15.151:0/3673962510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 0x7fee6010ab10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fee4c009910 tx=0x7fee4c009c20 comp rx=0 tx=0).stop 2024-04-16T03:05:42.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 -- 172.21.15.151:0/3673962510 shutdown_connections 2024-04-16T03:05:42.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 --2- 172.21.15.151:0/3673962510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 0x7fee6010ab10 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 --2- 172.21.15.151:0/3673962510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60101d00 0x7fee601020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 -- 172.21.15.151:0/3673962510 >> 172.21.15.151:0/3673962510 conn(0x7fee600fb570 msgr2=0x7fee600fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 -- 172.21.15.151:0/3673962510 shutdown_connections 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 -- 172.21.15.151:0/3673962510 wait complete. 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 Processor -- start 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 -- start start 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60101d00 0x7fee600ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 0x7fee600ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee601003e0 con 0x7fee60102620 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.940+0000 7fee66e88700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee60100520 con 0x7fee60101d00 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 0x7fee600ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 0x7fee600ffea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40268/0 (socket says 172.21.15.151:40268) 2024-04-16T03:05:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5ffff700 1 -- 172.21.15.151:0/4166221002 learned_addr learned my addr 172.21.15.151:0/4166221002 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee64c24700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60101d00 0x7fee600ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5ffff700 1 -- 172.21.15.151:0/4166221002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60101d00 msgr2=0x7fee600ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5ffff700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60101d00 0x7fee600ff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5ffff700 1 -- 172.21.15.151:0/4166221002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0095f0 con 0x7fee60102620 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5ffff700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 0x7fee600ffea0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00bda0 tx=0x7fee4c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5dffb700 1 -- 172.21.15.151:0/4166221002 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee4c01d070 con 0x7fee60102620 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5dffb700 1 -- 172.21.15.151:0/4166221002 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee4c00f460 con 0x7fee60102620 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee601007a0 con 0x7fee60102620 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee5dffb700 1 -- 172.21.15.151:0/4166221002 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee4c017630 con 0x7fee60102620 2024-04-16T03:05:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.941+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee601a2c60 con 0x7fee60102620 2024-04-16T03:05:42.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.942+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee60108210 con 0x7fee60102620 2024-04-16T03:05:42.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.943+0000 7fee5dffb700 1 -- 172.21.15.151:0/4166221002 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fee4c00fab0 con 0x7fee60102620 2024-04-16T03:05:42.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.943+0000 7fee5dffb700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fee5006ecc0 0x7fee50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:05:42.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.943+0000 7fee5dffb700 1 -- 172.21.15.151:0/4166221002 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fee4c090240 con 0x7fee60102620 2024-04-16T03:05:42.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.943+0000 7fee64c24700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fee5006ecc0 0x7fee50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:05:42.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.944+0000 7fee64c24700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fee5006ecc0 0x7fee50071180 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fee5400ac50 tx=0x7fee5400a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:05:42.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:42.948+0000 7fee5dffb700 1 -- 172.21.15.151:0/4166221002 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fee4c058ff0 con 0x7fee60102620 2024-04-16T03:05:43.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:42 smithi153 ceph-mon[87858]: from='client.35492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:43.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:42 smithi153 ceph-mon[87858]: pgmap v813: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:43.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:42 smithi153 ceph-mon[87858]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:43.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:42 smithi153 ceph-mon[87858]: from='client.35500 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:43.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:42 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2365170782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:05:43.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:42 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/17896183' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:05:43.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.226+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fee6004ea90 con 0x7fee60102620 2024-04-16T03:05:43.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.227+0000 7fee5dffb700 1 -- 172.21.15.151:0/4166221002 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7fee4c05c6b0 con 0x7fee60102620 2024-04-16T03:05:43.228 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:05:43.228 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:05:43.229 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:05:43.230 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:05:43.230 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:05:43.230 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:05:43.230 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:05:43.230 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:05:43.230 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:05:43.230 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:05:43.230 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fee5006ecc0 msgr2=0x7fee50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fee5006ecc0 0x7fee50071180 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fee5400ac50 tx=0x7fee5400a380 comp rx=0 tx=0).stop 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 msgr2=0x7fee600ffea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 0x7fee600ffea0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00bda0 tx=0x7fee4c00bdd0 comp rx=0 tx=0).stop 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 shutdown_connections 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fee5006ecc0 0x7fee50071180 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60101d00 0x7fee600ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 --2- 172.21.15.151:0/4166221002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee60102620 0x7fee600ffea0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 >> 172.21.15.151:0/4166221002 conn(0x7fee600fb570 msgr2=0x7fee60105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:05:43.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 shutdown_connections 2024-04-16T03:05:43.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:05:43.230+0000 7fee66e88700 1 -- 172.21.15.151:0/4166221002 wait complete. 2024-04-16T03:05:43.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:43 smithi151 ceph-mon[112233]: from='client.35512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:43.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:43 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4166221002' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:05:44.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:43 smithi153 ceph-mon[87858]: from='client.35512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:05:44.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:43 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4166221002' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:05:44.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:44 smithi151 ceph-mon[112233]: pgmap v814: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:44 smithi153 ceph-mon[87858]: pgmap v814: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:46.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:46 smithi151 ceph-mon[112233]: pgmap v815: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:47.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:46 smithi153 ceph-mon[87858]: pgmap v815: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:48.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:48 smithi151 ceph-mon[112233]: pgmap v816: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:49.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:48 smithi153 ceph-mon[87858]: pgmap v816: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:50.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:50 smithi151 ceph-mon[112233]: pgmap v817: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:50 smithi153 ceph-mon[87858]: pgmap v817: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:52.897 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:52 smithi153 ceph-mon[87858]: pgmap v818: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:52.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:52 smithi151 ceph-mon[112233]: pgmap v818: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:54 smithi151 ceph-mon[112233]: pgmap v819: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:05:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:54 smithi153 ceph-mon[87858]: pgmap v819: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:05:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:56 smithi151 ceph-mon[112233]: pgmap v820: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:57.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:56 smithi153 ceph-mon[87858]: pgmap v820: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:05:58 smithi151 ceph-mon[112233]: pgmap v821: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:05:59.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:05:58 smithi153 ceph-mon[87858]: pgmap v821: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:00 smithi151 ceph-mon[112233]: pgmap v822: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:00 smithi153 ceph-mon[87858]: pgmap v822: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:02 smithi151 ceph-mon[112233]: pgmap v823: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:03.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:02 smithi153 ceph-mon[87858]: pgmap v823: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:04.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:04 smithi151 ceph-mon[112233]: pgmap v824: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:05.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:04 smithi153 ceph-mon[87858]: pgmap v824: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:06.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:06 smithi151 ceph-mon[112233]: pgmap v825: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:07.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:06 smithi153 ceph-mon[87858]: pgmap v825: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:08.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:08 smithi151 ceph-mon[112233]: pgmap v826: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:09.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:08 smithi153 ceph-mon[87858]: pgmap v826: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:09.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:06:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:06:10.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:10 smithi151 ceph-mon[112233]: pgmap v827: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:11.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:10 smithi153 ceph-mon[87858]: pgmap v827: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:12.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:12 smithi151 ceph-mon[112233]: pgmap v828: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:13.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:12 smithi153 ceph-mon[87858]: pgmap v828: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:13.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.349+0000 7f5ab997d700 1 -- 172.21.15.151:0/3780221683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4100dc0 msgr2=0x7f5ab41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:13.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.349+0000 7f5ab997d700 1 --2- 172.21.15.151:0/3780221683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4100dc0 0x7f5ab41011a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009910 tx=0x7f5aa4009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:13.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.349+0000 7f5ab997d700 1 -- 172.21.15.151:0/3780221683 shutdown_connections 2024-04-16T03:06:13.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.349+0000 7f5ab997d700 1 --2- 172.21.15.151:0/3780221683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab4101770 0x7f5ab41057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.349+0000 7f5ab997d700 1 --2- 172.21.15.151:0/3780221683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4100dc0 0x7f5ab41011a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.349+0000 7f5ab997d700 1 -- 172.21.15.151:0/3780221683 >> 172.21.15.151:0/3780221683 conn(0x7f5ab40fc650 msgr2=0x7f5ab40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:13.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.349+0000 7f5ab997d700 1 -- 172.21.15.151:0/3780221683 shutdown_connections 2024-04-16T03:06:13.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.349+0000 7f5ab997d700 1 -- 172.21.15.151:0/3780221683 wait complete. 2024-04-16T03:06:13.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab997d700 1 Processor -- start 2024-04-16T03:06:13.352 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab997d700 1 -- start start 2024-04-16T03:06:13.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab997d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab4100dc0 0x7f5ab419c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:13.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab997d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4101770 0x7f5ab419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:13.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab997d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab419d080 con 0x7f5ab4101770 2024-04-16T03:06:13.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab997d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab41965c0 con 0x7f5ab4100dc0 2024-04-16T03:06:13.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4101770 0x7f5ab419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab27fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4101770 0x7f5ab419ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52558/0 (socket says 172.21.15.151:52558) 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab27fc700 1 -- 172.21.15.151:0/3569505683 learned_addr learned my addr 172.21.15.151:0/3569505683 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.350+0000 7f5ab2ffd700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab4100dc0 0x7f5ab419c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.351+0000 7f5ab27fc700 1 -- 172.21.15.151:0/3569505683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab4100dc0 msgr2=0x7f5ab419c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.351+0000 7f5ab27fc700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab4100dc0 0x7f5ab419c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.351+0000 7f5ab27fc700 1 -- 172.21.15.151:0/3569505683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa40095f0 con 0x7f5ab4101770 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.351+0000 7f5ab27fc700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4101770 0x7f5ab419ca80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800c960 tx=0x7f5aa800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.351+0000 7f5ab897b700 1 -- 172.21.15.151:0/3569505683 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa8007a70 con 0x7f5ab4101770 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.351+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab41968a0 con 0x7f5ab4101770 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.351+0000 7f5ab897b700 1 -- 172.21.15.151:0/3569505683 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5aa8007bd0 con 0x7f5ab4101770 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.351+0000 7f5ab897b700 1 -- 172.21.15.151:0/3569505683 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa8018630 con 0x7f5ab4101770 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.351+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab4196df0 con 0x7f5ab4101770 2024-04-16T03:06:13.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.352+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab4109160 con 0x7f5ab4101770 2024-04-16T03:06:13.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.353+0000 7f5ab897b700 1 -- 172.21.15.151:0/3569505683 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5aa800f450 con 0x7f5ab4101770 2024-04-16T03:06:13.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.353+0000 7f5ab897b700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5a9c06ecc0 0x7f5a9c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:13.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.353+0000 7f5ab897b700 1 -- 172.21.15.151:0/3569505683 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5aa808edf0 con 0x7f5ab4101770 2024-04-16T03:06:13.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.353+0000 7f5ab2ffd700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5a9c06ecc0 0x7f5a9c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:13.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.354+0000 7f5ab2ffd700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5a9c06ecc0 0x7f5a9c071180 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4006010 tx=0x7f5aa401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:13.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.358+0000 7f5ab897b700 1 -- 172.21.15.151:0/3569505683 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5aa805b6d0 con 0x7f5ab4101770 2024-04-16T03:06:13.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.577+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ab41970c0 con 0x7f5a9c06ecc0 2024-04-16T03:06:13.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.578+0000 7f5ab897b700 1 -- 172.21.15.151:0/3569505683 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5ab41970c0 con 0x7f5a9c06ecc0 2024-04-16T03:06:13.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5a9c06ecc0 msgr2=0x7f5a9c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:13.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5a9c06ecc0 0x7f5a9c071180 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4006010 tx=0x7f5aa401a040 comp rx=0 tx=0).stop 2024-04-16T03:06:13.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4101770 msgr2=0x7f5ab419ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:13.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4101770 0x7f5ab419ca80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800c960 tx=0x7f5aa800cc70 comp rx=0 tx=0).stop 2024-04-16T03:06:13.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 shutdown_connections 2024-04-16T03:06:13.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5a9c06ecc0 0x7f5a9c071180 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab4100dc0 0x7f5ab419c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 --2- 172.21.15.151:0/3569505683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab4101770 0x7f5ab419ca80 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 >> 172.21.15.151:0/3569505683 conn(0x7f5ab40fc650 msgr2=0x7f5ab40fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:13.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 shutdown_connections 2024-04-16T03:06:13.583 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.581+0000 7f5ab997d700 1 -- 172.21.15.151:0/3569505683 wait complete. 2024-04-16T03:06:13.593 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:06:13.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.687+0000 7fa685613700 1 -- 172.21.15.151:0/3067609288 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 msgr2=0x7fa680105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:13.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.687+0000 7fa685613700 1 --2- 172.21.15.151:0/3067609288 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 0x7fa680105e20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fa668009910 tx=0x7fa668009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:13.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.687+0000 7fa685613700 1 -- 172.21.15.151:0/3067609288 shutdown_connections 2024-04-16T03:06:13.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.687+0000 7fa685613700 1 --2- 172.21.15.151:0/3067609288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6800ffa40 0x7fa6800ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.687+0000 7fa685613700 1 --2- 172.21.15.151:0/3067609288 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 0x7fa680105e20 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.687+0000 7fa685613700 1 -- 172.21.15.151:0/3067609288 >> 172.21.15.151:0/3067609288 conn(0x7fa6800fb570 msgr2=0x7fa6800fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:13.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.687+0000 7fa685613700 1 -- 172.21.15.151:0/3067609288 shutdown_connections 2024-04-16T03:06:13.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.687+0000 7fa685613700 1 -- 172.21.15.151:0/3067609288 wait complete. 2024-04-16T03:06:13.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.688+0000 7fa685613700 1 Processor -- start 2024-04-16T03:06:13.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.688+0000 7fa685613700 1 -- start start 2024-04-16T03:06:13.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.688+0000 7fa685613700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6800ffa40 0x7fa6800fbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:13.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.688+0000 7fa685613700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 0x7fa6800fc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:13.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.688+0000 7fa685613700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800fc900 con 0x7fa680105a40 2024-04-16T03:06:13.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.688+0000 7fa685613700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800fca70 con 0x7fa6800ffa40 2024-04-16T03:06:13.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa67e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 0x7fa6800fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa67e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 0x7fa6800fc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52578/0 (socket says 172.21.15.151:52578) 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa67e7fc700 1 -- 172.21.15.151:0/2493014865 learned_addr learned my addr 172.21.15.151:0/2493014865 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa67effd700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6800ffa40 0x7fa6800fbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa67e7fc700 1 -- 172.21.15.151:0/2493014865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6800ffa40 msgr2=0x7fa6800fbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa67e7fc700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6800ffa40 0x7fa6800fbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa67e7fc700 1 -- 172.21.15.151:0/2493014865 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6680095f0 con 0x7fa680105a40 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa67e7fc700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 0x7fa6800fc330 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa67000c8a0 tx=0x7fa67000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa677fff700 1 -- 172.21.15.151:0/2493014865 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6700078e0 con 0x7fa680105a40 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6801a28e0 con 0x7fa680105a40 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa677fff700 1 -- 172.21.15.151:0/2493014865 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa67000f450 con 0x7fa680105a40 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa677fff700 1 -- 172.21.15.151:0/2493014865 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa67000e5e0 con 0x7fa680105a40 2024-04-16T03:06:13.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.689+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6801a2d50 con 0x7fa680105a40 2024-04-16T03:06:13.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.691+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa680109050 con 0x7fa680105a40 2024-04-16T03:06:13.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.691+0000 7fa677fff700 1 -- 172.21.15.151:0/2493014865 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa67000f5c0 con 0x7fa680105a40 2024-04-16T03:06:13.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.692+0000 7fa677fff700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa66c06ecc0 0x7fa66c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:13.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.692+0000 7fa67effd700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa66c06ecc0 0x7fa66c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:13.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.692+0000 7fa677fff700 1 -- 172.21.15.151:0/2493014865 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa67008ee80 con 0x7fa680105a40 2024-04-16T03:06:13.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.692+0000 7fa67effd700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa66c06ecc0 0x7fa66c071180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa668006010 tx=0x7fa66801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:13.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.696+0000 7fa677fff700 1 -- 172.21.15.151:0/2493014865 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa670057c30 con 0x7fa680105a40 2024-04-16T03:06:13.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.916+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa68002cce0 con 0x7fa66c06ecc0 2024-04-16T03:06:13.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.917+0000 7fa677fff700 1 -- 172.21.15.151:0/2493014865 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa68002cce0 con 0x7fa66c06ecc0 2024-04-16T03:06:13.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.919+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa66c06ecc0 msgr2=0x7fa66c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:13.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.919+0000 7fa685613700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa66c06ecc0 0x7fa66c071180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa668006010 tx=0x7fa66801a040 comp rx=0 tx=0).stop 2024-04-16T03:06:13.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.919+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 msgr2=0x7fa6800fc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:13.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.919+0000 7fa685613700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 0x7fa6800fc330 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa67000c8a0 tx=0x7fa67000cbb0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.919+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 shutdown_connections 2024-04-16T03:06:13.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.919+0000 7fa685613700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa66c06ecc0 0x7fa66c071180 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.919+0000 7fa685613700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6800ffa40 0x7fa6800fbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.919+0000 7fa685613700 1 --2- 172.21.15.151:0/2493014865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa680105a40 0x7fa6800fc330 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:13.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.919+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 >> 172.21.15.151:0/2493014865 conn(0x7fa6800fb570 msgr2=0x7fa680068fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:13.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.920+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 shutdown_connections 2024-04-16T03:06:13.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:13.920+0000 7fa685613700 1 -- 172.21.15.151:0/2493014865 wait complete. 2024-04-16T03:06:14.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.024+0000 7f742c39a700 1 -- 172.21.15.151:0/2313999387 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 msgr2=0x7f7424102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.024+0000 7f742c39a700 1 --2- 172.21.15.151:0/2313999387 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 0x7f7424102160 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7420009910 tx=0x7f7420009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:14.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 -- 172.21.15.151:0/2313999387 shutdown_connections 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 --2- 172.21.15.151:0/2313999387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74241026a0 0x7f742410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 --2- 172.21.15.151:0/2313999387 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 0x7f7424102160 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 -- 172.21.15.151:0/2313999387 >> 172.21.15.151:0/2313999387 conn(0x7f74240fb570 msgr2=0x7f74240fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 -- 172.21.15.151:0/2313999387 shutdown_connections 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 -- 172.21.15.151:0/2313999387 wait complete. 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 Processor -- start 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 -- start start 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 0x7f742419c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74241026a0 0x7f742419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742419d150 con 0x7f7424101d80 2024-04-16T03:06:14.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.025+0000 7f742c39a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7424196600 con 0x7f74241026a0 2024-04-16T03:06:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f742a136700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 0x7f742419c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f742a136700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 0x7f742419c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52602/0 (socket says 172.21.15.151:52602) 2024-04-16T03:06:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f742a136700 1 -- 172.21.15.151:0/3998687010 learned_addr learned my addr 172.21.15.151:0/3998687010 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f7429935700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74241026a0 0x7f742419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f742a136700 1 -- 172.21.15.151:0/3998687010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74241026a0 msgr2=0x7f742419cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f742a136700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74241026a0 0x7f742419cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f742a136700 1 -- 172.21.15.151:0/3998687010 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74200095f0 con 0x7f7424101d80 2024-04-16T03:06:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f742a136700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 0x7f742419c580 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7420006010 tx=0x7f742000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f741b7fe700 1 -- 172.21.15.151:0/3998687010 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f742001d070 con 0x7f7424101d80 2024-04-16T03:06:14.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f741b7fe700 1 -- 172.21.15.151:0/3998687010 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7420004fb0 con 0x7f7424101d80 2024-04-16T03:06:14.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7424196880 con 0x7f7424101d80 2024-04-16T03:06:14.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f741b7fe700 1 -- 172.21.15.151:0/3998687010 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7420022ea0 con 0x7f7424101d80 2024-04-16T03:06:14.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.026+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7424196d70 con 0x7f7424101d80 2024-04-16T03:06:14.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.028+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74241081e0 con 0x7f7424101d80 2024-04-16T03:06:14.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.028+0000 7f741b7fe700 1 -- 172.21.15.151:0/3998687010 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7420022470 con 0x7f7424101d80 2024-04-16T03:06:14.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.029+0000 7f741b7fe700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f741006ecc0 0x7f7410071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:14.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.029+0000 7f741b7fe700 1 -- 172.21.15.151:0/3998687010 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7420090350 con 0x7f7424101d80 2024-04-16T03:06:14.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.029+0000 7f7429935700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f741006ecc0 0x7f7410071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:14.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.029+0000 7f7429935700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f741006ecc0 0x7f7410071180 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7414005fd0 tx=0x7f741400a560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:14.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.034+0000 7f741b7fe700 1 -- 172.21.15.151:0/3998687010 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f742005cd60 con 0x7f7424101d80 2024-04-16T03:06:14.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.243+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f74241970a0 con 0x7f741006ecc0 2024-04-16T03:06:14.256 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.254+0000 7f741b7fe700 1 -- 172.21.15.151:0/3998687010 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f74241970a0 con 0x7f741006ecc0 2024-04-16T03:06:14.256 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:06:14.256 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (27m) 5m ago 39m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:06:14.256 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (39m) 5m ago 39m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:06:14.256 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (38m) 5m ago 38m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:06:14.256 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (25m) 5m ago 39m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (25m) 5m ago 38m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (27m) 5m ago 38m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (21m) 5m ago 33m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (21m) 5m ago 33m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (5m) 5m ago 33m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (21m) 5m ago 33m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (29m) 5m ago 40m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (28m) 5m ago 38m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (26m) 5m ago 40m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (26m) 5m ago 38m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (28m) 5m ago 39m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:06:14.257 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (28m) 5m ago 38m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:06:14.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (24m) 5m ago 37m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:06:14.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (24m) 5m ago 36m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:06:14.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (23m) 5m ago 36m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:06:14.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (23m) 5m ago 35m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:06:14.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (22m) 5m ago 35m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:06:14.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (21m) 5m ago 35m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:06:14.258 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (28m) 5m ago 38m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:06:14.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f741006ecc0 msgr2=0x7f7410071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f741006ecc0 0x7f7410071180 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7414005fd0 tx=0x7f741400a560 comp rx=0 tx=0).stop 2024-04-16T03:06:14.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 msgr2=0x7f742419c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 0x7f742419c580 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7420006010 tx=0x7f742000be20 comp rx=0 tx=0).stop 2024-04-16T03:06:14.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 shutdown_connections 2024-04-16T03:06:14.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f741006ecc0 0x7f7410071180 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7424101d80 0x7f742419c580 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 --2- 172.21.15.151:0/3998687010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74241026a0 0x7f742419cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 >> 172.21.15.151:0/3998687010 conn(0x7f74240fb570 msgr2=0x7f7424105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:14.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 shutdown_connections 2024-04-16T03:06:14.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.257+0000 7f742c39a700 1 -- 172.21.15.151:0/3998687010 wait complete. 2024-04-16T03:06:14.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.360+0000 7f959e80e700 1 -- 172.21.15.151:0/1357237155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9598105a70 msgr2=0x7f9598105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.360+0000 7f959e80e700 1 --2- 172.21.15.151:0/1357237155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9598105a70 0x7f9598105e50 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f9588009910 tx=0x7f9588009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:14.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.360+0000 7f959e80e700 1 -- 172.21.15.151:0/1357237155 shutdown_connections 2024-04-16T03:06:14.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.360+0000 7f959e80e700 1 --2- 172.21.15.151:0/1357237155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95980ffa70 0x7f95980ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.360+0000 7f959e80e700 1 --2- 172.21.15.151:0/1357237155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9598105a70 0x7f9598105e50 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.360+0000 7f959e80e700 1 -- 172.21.15.151:0/1357237155 >> 172.21.15.151:0/1357237155 conn(0x7f95980fb560 msgr2=0x7f95980fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:14.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.360+0000 7f959e80e700 1 -- 172.21.15.151:0/1357237155 shutdown_connections 2024-04-16T03:06:14.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.360+0000 7f959e80e700 1 -- 172.21.15.151:0/1357237155 wait complete. 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.361+0000 7f959e80e700 1 Processor -- start 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.361+0000 7f959e80e700 1 -- start start 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.361+0000 7f959e80e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95980ffa70 0x7f959819e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.361+0000 7f959e80e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9598105a70 0x7f959819ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.361+0000 7f959e80e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959819f3e0 con 0x7f95980ffa70 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.361+0000 7f959e80e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9598198890 con 0x7f9598105a70 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.361+0000 7f9597fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95980ffa70 0x7f959819e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.361+0000 7f9597fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95980ffa70 0x7f959819e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52616/0 (socket says 172.21.15.151:52616) 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.361+0000 7f9597fff700 1 -- 172.21.15.151:0/617690623 learned_addr learned my addr 172.21.15.151:0/617690623 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f95977fe700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9598105a70 0x7f959819ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f9597fff700 1 -- 172.21.15.151:0/617690623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9598105a70 msgr2=0x7f959819ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f9597fff700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9598105a70 0x7f959819ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f9597fff700 1 -- 172.21.15.151:0/617690623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95880095f0 con 0x7f95980ffa70 2024-04-16T03:06:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f9597fff700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95980ffa70 0x7f959819e810 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9588006010 tx=0x7f958800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:14.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f95957fa700 1 -- 172.21.15.151:0/617690623 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f958801d070 con 0x7f95980ffa70 2024-04-16T03:06:14.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9598198b10 con 0x7f95980ffa70 2024-04-16T03:06:14.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f95957fa700 1 -- 172.21.15.151:0/617690623 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9588004fb0 con 0x7f95980ffa70 2024-04-16T03:06:14.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f95957fa700 1 -- 172.21.15.151:0/617690623 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9588022ea0 con 0x7f95980ffa70 2024-04-16T03:06:14.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.362+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9598199000 con 0x7f95980ffa70 2024-04-16T03:06:14.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.363+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959804ea90 con 0x7f95980ffa70 2024-04-16T03:06:14.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.364+0000 7f95957fa700 1 -- 172.21.15.151:0/617690623 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9588022470 con 0x7f95980ffa70 2024-04-16T03:06:14.366 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.365+0000 7f95957fa700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f958406ecc0 0x7f9584071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:14.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.365+0000 7f95957fa700 1 -- 172.21.15.151:0/617690623 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f95880903f0 con 0x7f95980ffa70 2024-04-16T03:06:14.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.365+0000 7f95977fe700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f958406ecc0 0x7f9584071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:14.367 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.365+0000 7f95977fe700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f958406ecc0 0x7f9584071180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9580006fd0 tx=0x7f9580008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:14.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.369+0000 7f95957fa700 1 -- 172.21.15.151:0/617690623 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f958805cd60 con 0x7f95980ffa70 2024-04-16T03:06:14.656 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.655+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9598199da0 con 0x7f95980ffa70 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.655+0000 7f95957fa700 1 -- 172.21.15.151:0/617690623 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9588022720 con 0x7f95980ffa70 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:06:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:06:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:06:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:06:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:06:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:06:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:06:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:06:14.658 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.657+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f958406ecc0 msgr2=0x7f9584071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.657+0000 7f959e80e700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f958406ecc0 0x7f9584071180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9580006fd0 tx=0x7f9580008040 comp rx=0 tx=0).stop 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.657+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95980ffa70 msgr2=0x7f959819e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.657+0000 7f959e80e700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95980ffa70 0x7f959819e810 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9588006010 tx=0x7f958800be20 comp rx=0 tx=0).stop 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.657+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 shutdown_connections 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.657+0000 7f959e80e700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f958406ecc0 0x7f9584071180 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.657+0000 7f959e80e700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95980ffa70 0x7f959819e810 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.657+0000 7f959e80e700 1 --2- 172.21.15.151:0/617690623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9598105a70 0x7f959819ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.657+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 >> 172.21.15.151:0/617690623 conn(0x7f95980fb560 msgr2=0x7f9598068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:14.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.658+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 shutdown_connections 2024-04-16T03:06:14.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.658+0000 7f959e80e700 1 -- 172.21.15.151:0/617690623 wait complete. 2024-04-16T03:06:14.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.763+0000 7f3d7d008700 1 -- 172.21.15.151:0/3353975110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78069020 msgr2=0x7f3d78105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.763+0000 7f3d7d008700 1 --2- 172.21.15.151:0/3353975110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78069020 0x7f3d78105790 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009910 tx=0x7f3d68009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:14.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.763+0000 7f3d7d008700 1 -- 172.21.15.151:0/3353975110 shutdown_connections 2024-04-16T03:06:14.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.763+0000 7f3d7d008700 1 --2- 172.21.15.151:0/3353975110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78069020 0x7f3d78105790 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.763+0000 7f3d7d008700 1 --2- 172.21.15.151:0/3353975110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d78068670 0x7f3d78068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.763+0000 7f3d7d008700 1 -- 172.21.15.151:0/3353975110 >> 172.21.15.151:0/3353975110 conn(0x7f3d780755a0 msgr2=0x7f3d780759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:14.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.763+0000 7f3d7d008700 1 -- 172.21.15.151:0/3353975110 shutdown_connections 2024-04-16T03:06:14.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.763+0000 7f3d7d008700 1 -- 172.21.15.151:0/3353975110 wait complete. 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.763+0000 7f3d7d008700 1 Processor -- start 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d7d008700 1 -- start start 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d7d008700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78068670 0x7f3d7819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d7d008700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d78069020 0x7f3d7819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d7d008700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7819d0f0 con 0x7f3d78068670 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d7d008700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d781965a0 con 0x7f3d78069020 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d76d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78068670 0x7f3d7819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d76d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78068670 0x7f3d7819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52646/0 (socket says 172.21.15.151:52646) 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d76d9d700 1 -- 172.21.15.151:0/2869971786 learned_addr learned my addr 172.21.15.151:0/2869971786 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d7659c700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d78069020 0x7f3d7819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d76d9d700 1 -- 172.21.15.151:0/2869971786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d78069020 msgr2=0x7f3d7819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d76d9d700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d78069020 0x7f3d7819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d76d9d700 1 -- 172.21.15.151:0/2869971786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d680095f0 con 0x7f3d78068670 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.764+0000 7f3d7659c700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d78069020 0x7f3d7819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:06:14.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.765+0000 7f3d76d9d700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78068670 0x7f3d7819c520 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000d900 tx=0x7f3d6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:14.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.765+0000 7f3d6ffff700 1 -- 172.21.15.151:0/2869971786 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d60004a00 con 0x7f3d78068670 2024-04-16T03:06:14.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.765+0000 7f3d6ffff700 1 -- 172.21.15.151:0/2869971786 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d600054e0 con 0x7f3d78068670 2024-04-16T03:06:14.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.765+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d78196880 con 0x7f3d78068670 2024-04-16T03:06:14.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.765+0000 7f3d6ffff700 1 -- 172.21.15.151:0/2869971786 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d60009d60 con 0x7f3d78068670 2024-04-16T03:06:14.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.765+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d78196dd0 con 0x7f3d78068670 2024-04-16T03:06:14.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.766+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d781090a0 con 0x7f3d78068670 2024-04-16T03:06:14.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.766+0000 7f3d6ffff700 1 -- 172.21.15.151:0/2869971786 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3d60005020 con 0x7f3d78068670 2024-04-16T03:06:14.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.767+0000 7f3d6ffff700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3d6406ecc0 0x7f3d64071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:14.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.767+0000 7f3d6ffff700 1 -- 172.21.15.151:0/2869971786 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3d6008f030 con 0x7f3d78068670 2024-04-16T03:06:14.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.767+0000 7f3d7659c700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3d6406ecc0 0x7f3d64071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:14.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.767+0000 7f3d7659c700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3d6406ecc0 0x7f3d64071180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800b5c0 tx=0x7f3d680053a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:14.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:14.772+0000 7f3d6ffff700 1 -- 172.21.15.151:0/2869971786 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3d6005b910 con 0x7f3d78068670 2024-04-16T03:06:14.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:14 smithi151 ceph-mon[112233]: pgmap v829: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:14.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:14 smithi151 ceph-mon[112233]: from='client.35520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:14.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:14 smithi151 ceph-mon[112233]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:14.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:14 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/617690623' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:06:15.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.013+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3d78197070 con 0x7f3d78068670 2024-04-16T03:06:15.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.014+0000 7f3d6ffff700 1 -- 172.21.15.151:0/2869971786 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f3d6005b4a0 con 0x7f3d78068670 2024-04-16T03:06:15.015 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:06:15.015 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:06:15.015 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:06:15.015 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:06:15.016 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:06:15.017 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:06:15.018 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:06:15.018 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:06:15.018 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:06:15.018 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:06:15.018 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:06:15.018 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:06:15.018 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:06:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3d6406ecc0 msgr2=0x7f3d64071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3d6406ecc0 0x7f3d64071180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800b5c0 tx=0x7f3d680053a0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78068670 msgr2=0x7f3d7819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78068670 0x7f3d7819c520 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000d900 tx=0x7f3d6000dcc0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 shutdown_connections 2024-04-16T03:06:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3d6406ecc0 0x7f3d64071180 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3d78068670 0x7f3d7819c520 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 --2- 172.21.15.151:0/2869971786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d78069020 0x7f3d7819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 >> 172.21.15.151:0/2869971786 conn(0x7f3d780755a0 msgr2=0x7f3d78105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:15.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 shutdown_connections 2024-04-16T03:06:15.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.017+0000 7f3d7d008700 1 -- 172.21.15.151:0/2869971786 wait complete. 2024-04-16T03:06:15.020 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:06:15.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:14 smithi153 ceph-mon[87858]: pgmap v829: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:15.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:14 smithi153 ceph-mon[87858]: from='client.35520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:15.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:14 smithi153 ceph-mon[87858]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:15.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:14 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/617690623' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:06:15.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.125+0000 7ff0d584d700 1 -- 172.21.15.151:0/2053888721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ff500 msgr2=0x7ff0d00ff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.125+0000 7ff0d584d700 1 --2- 172.21.15.151:0/2053888721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ff500 0x7ff0d00ff8e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009910 tx=0x7ff0c0009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:15.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.125+0000 7ff0d584d700 1 -- 172.21.15.151:0/2053888721 shutdown_connections 2024-04-16T03:06:15.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.125+0000 7ff0d584d700 1 --2- 172.21.15.151:0/2053888721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0d00ffeb0 0x7ff0d01111f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.125+0000 7ff0d584d700 1 --2- 172.21.15.151:0/2053888721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ff500 0x7ff0d00ff8e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.125+0000 7ff0d584d700 1 -- 172.21.15.151:0/2053888721 >> 172.21.15.151:0/2053888721 conn(0x7ff0d0074d80 msgr2=0x7ff0d0075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:15.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.126+0000 7ff0d584d700 1 -- 172.21.15.151:0/2053888721 shutdown_connections 2024-04-16T03:06:15.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.126+0000 7ff0d584d700 1 -- 172.21.15.151:0/2053888721 wait complete. 2024-04-16T03:06:15.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.126+0000 7ff0d584d700 1 Processor -- start 2024-04-16T03:06:15.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.126+0000 7ff0d584d700 1 -- start start 2024-04-16T03:06:15.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0d584d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0d00ff500 0x7ff0d019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:15.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0d584d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ffeb0 0x7ff0d019d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:15.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0d584d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d019d980 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0d584d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d01a16c0 con 0x7ff0d00ff500 2024-04-16T03:06:15.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0ce7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ffeb0 0x7ff0d019d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:15.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0ce7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ffeb0 0x7ff0d019d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52666/0 (socket says 172.21.15.151:52666) 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0ce7fc700 1 -- 172.21.15.151:0/2101836692 learned_addr learned my addr 172.21.15.151:0/2101836692 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0ceffd700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0d00ff500 0x7ff0d019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0ce7fc700 1 -- 172.21.15.151:0/2101836692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0d00ff500 msgr2=0x7ff0d019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0ce7fc700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0d00ff500 0x7ff0d019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0ce7fc700 1 -- 172.21.15.151:0/2101836692 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c00095f0 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0ce7fc700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ffeb0 0x7ff0d019d2a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400d8d0 tx=0x7ff0c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0d484b700 1 -- 172.21.15.151:0/2101836692 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0c4009900 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.127+0000 7ff0d484b700 1 -- 172.21.15.151:0/2101836692 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0c4010460 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.128+0000 7ff0d484b700 1 -- 172.21.15.151:0/2101836692 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0c400f5f0 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.128+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d01a19a0 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.128+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d01a1f70 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.128+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d0103f50 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.129+0000 7ff0d484b700 1 -- 172.21.15.151:0/2101836692 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff0c40105d0 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.129+0000 7ff0d484b700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff0b806ecc0 0x7ff0b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:15.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.130+0000 7ff0d484b700 1 -- 172.21.15.151:0/2101836692 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff0c408ea30 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.130+0000 7ff0ceffd700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff0b806ecc0 0x7ff0b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:15.132 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.130+0000 7ff0ceffd700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff0b806ecc0 0x7ff0b8071180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000b5c0 tx=0x7ff0c0005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:15.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.134+0000 7ff0d484b700 1 -- 172.21.15.151:0/2101836692 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff0c4057860 con 0x7ff0d00ffeb0 2024-04-16T03:06:15.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.354+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0d01a22d0 con 0x7ff0b806ecc0 2024-04-16T03:06:15.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.356+0000 7ff0d484b700 1 -- 172.21.15.151:0/2101836692 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff0d01a22d0 con 0x7ff0b806ecc0 2024-04-16T03:06:15.357 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:06:15.358 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:06:15.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff0b806ecc0 msgr2=0x7ff0b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff0b806ecc0 0x7ff0b8071180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000b5c0 tx=0x7ff0c0005960 comp rx=0 tx=0).stop 2024-04-16T03:06:15.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ffeb0 msgr2=0x7ff0d019d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ffeb0 0x7ff0d019d2a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400d8d0 tx=0x7ff0c400dc90 comp rx=0 tx=0).stop 2024-04-16T03:06:15.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 shutdown_connections 2024-04-16T03:06:15.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff0b806ecc0 0x7ff0b8071180 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0d00ff500 0x7ff0d019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 --2- 172.21.15.151:0/2101836692 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0d00ffeb0 0x7ff0d019d2a0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 >> 172.21.15.151:0/2101836692 conn(0x7ff0d0074d80 msgr2=0x7ff0d00fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:15.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 shutdown_connections 2024-04-16T03:06:15.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.358+0000 7ff0d584d700 1 -- 172.21.15.151:0/2101836692 wait complete. 2024-04-16T03:06:15.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.468+0000 7f5c63b05700 1 -- 172.21.15.151:0/549474334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c0ffbd0 msgr2=0x7f5c5c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.468+0000 7f5c63b05700 1 --2- 172.21.15.151:0/549474334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c0ffbd0 0x7f5c5c100050 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009910 tx=0x7f5c4c009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.468+0000 7f5c63b05700 1 -- 172.21.15.151:0/549474334 shutdown_connections 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.468+0000 7f5c63b05700 1 --2- 172.21.15.151:0/549474334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c0ffbd0 0x7f5c5c100050 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.468+0000 7f5c63b05700 1 --2- 172.21.15.151:0/549474334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c105a40 0x7f5c5c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.468+0000 7f5c63b05700 1 -- 172.21.15.151:0/549474334 >> 172.21.15.151:0/549474334 conn(0x7f5c5c0755a0 msgr2=0x7f5c5c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.468+0000 7f5c63b05700 1 -- 172.21.15.151:0/549474334 shutdown_connections 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.468+0000 7f5c63b05700 1 -- 172.21.15.151:0/549474334 wait complete. 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c63b05700 1 Processor -- start 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c63b05700 1 -- start start 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c63b05700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c0ffbd0 0x7f5c5c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c63b05700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c105a40 0x7f5c5c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c63b05700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c19d0d0 con 0x7f5c5c105a40 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c63b05700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c196580 con 0x7f5c5c0ffbd0 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c610a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c105a40 0x7f5c5c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c610a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c105a40 0x7f5c5c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52680/0 (socket says 172.21.15.151:52680) 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c610a0700 1 -- 172.21.15.151:0/4108469810 learned_addr learned my addr 172.21.15.151:0/4108469810 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.469+0000 7f5c618a1700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c0ffbd0 0x7f5c5c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.470+0000 7f5c610a0700 1 -- 172.21.15.151:0/4108469810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c0ffbd0 msgr2=0x7f5c5c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.470+0000 7f5c610a0700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c0ffbd0 0x7f5c5c19c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.470+0000 7f5c610a0700 1 -- 172.21.15.151:0/4108469810 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c0095f0 con 0x7f5c5c105a40 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.470+0000 7f5c610a0700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c105a40 0x7f5c5c19ca40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c006010 tx=0x7f5c4c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.470+0000 7f5c52ffd700 1 -- 172.21.15.151:0/4108469810 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c4c01d070 con 0x7f5c5c105a40 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.470+0000 7f5c52ffd700 1 -- 172.21.15.151:0/4108469810 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c4c005dc0 con 0x7f5c5c105a40 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.470+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c196800 con 0x7f5c5c105a40 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.470+0000 7f5c52ffd700 1 -- 172.21.15.151:0/4108469810 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c4c017400 con 0x7f5c5c105a40 2024-04-16T03:06:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.470+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c196cf0 con 0x7f5c5c105a40 2024-04-16T03:06:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.471+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c108fe0 con 0x7f5c5c105a40 2024-04-16T03:06:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.472+0000 7f5c52ffd700 1 -- 172.21.15.151:0/4108469810 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5c4c003a60 con 0x7f5c5c105a40 2024-04-16T03:06:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.472+0000 7f5c52ffd700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4806ecc0 0x7f5c48071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.472+0000 7f5c52ffd700 1 -- 172.21.15.151:0/4108469810 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5c4c090350 con 0x7f5c5c105a40 2024-04-16T03:06:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.472+0000 7f5c618a1700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4806ecc0 0x7f5c48071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.473+0000 7f5c618a1700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4806ecc0 0x7f5c48071180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5c58005950 tx=0x7f5c5800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:15.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.477+0000 7f5c52ffd700 1 -- 172.21.15.151:0/4108469810 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5c4c05cbd0 con 0x7f5c5c105a40 2024-04-16T03:06:15.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.760+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5c5c04ea90 con 0x7f5c5c105a40 2024-04-16T03:06:15.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.761+0000 7f5c52ffd700 1 -- 172.21.15.151:0/4108469810 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f5c4c05c760 con 0x7f5c5c105a40 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:06:15.763 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:06:15.764 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:06:15.765 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:06:15.765 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:06:15.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4806ecc0 msgr2=0x7f5c48071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4806ecc0 0x7f5c48071180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5c58005950 tx=0x7f5c5800b410 comp rx=0 tx=0).stop 2024-04-16T03:06:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c105a40 msgr2=0x7f5c5c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c105a40 0x7f5c5c19ca40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c006010 tx=0x7f5c4c00bd70 comp rx=0 tx=0).stop 2024-04-16T03:06:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 shutdown_connections 2024-04-16T03:06:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5c4806ecc0 0x7f5c48071180 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c0ffbd0 0x7f5c5c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 --2- 172.21.15.151:0/4108469810 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c5c105a40 0x7f5c5c19ca40 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 >> 172.21.15.151:0/4108469810 conn(0x7f5c5c0755a0 msgr2=0x7f5c5c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 shutdown_connections 2024-04-16T03:06:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:15.763+0000 7f5c63b05700 1 -- 172.21.15.151:0/4108469810 wait complete. 2024-04-16T03:06:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:15 smithi151 ceph-mon[112233]: from='client.35528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:15.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:15 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2869971786' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:06:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:15 smithi153 ceph-mon[87858]: from='client.35528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:16.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:15 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2869971786' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:06:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:16 smithi151 ceph-mon[112233]: pgmap v830: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:16 smithi151 ceph-mon[112233]: from='client.35540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:16 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4108469810' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:06:17.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:16 smithi153 ceph-mon[87858]: pgmap v830: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:17.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:16 smithi153 ceph-mon[87858]: from='client.35540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:17.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:16 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4108469810' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:06:18.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:18 smithi151 ceph-mon[112233]: pgmap v831: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:19.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:18 smithi153 ceph-mon[87858]: pgmap v831: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:20.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:20 smithi151 ceph-mon[112233]: pgmap v832: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:21.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:20 smithi153 ceph-mon[87858]: pgmap v832: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:22.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:22 smithi151 ceph-mon[112233]: pgmap v833: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:23.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:22 smithi153 ceph-mon[87858]: pgmap v833: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:24 smithi151 ceph-mon[112233]: pgmap v834: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:06:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:24 smithi153 ceph-mon[87858]: pgmap v834: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:06:26.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:26 smithi151 ceph-mon[112233]: pgmap v835: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:27.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:26 smithi153 ceph-mon[87858]: pgmap v835: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:28 smithi151 ceph-mon[112233]: pgmap v836: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:29.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:28 smithi153 ceph-mon[87858]: pgmap v836: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:30.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:30 smithi151 ceph-mon[112233]: pgmap v837: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:31.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:30 smithi153 ceph-mon[87858]: pgmap v837: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:32 smithi151 ceph-mon[112233]: pgmap v838: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:33.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:32 smithi153 ceph-mon[87858]: pgmap v838: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:34.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:34 smithi151 ceph-mon[112233]: pgmap v839: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:35.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:34 smithi153 ceph-mon[87858]: pgmap v839: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:36 smithi151 ceph-mon[112233]: pgmap v840: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:36 smithi153 ceph-mon[87858]: pgmap v840: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:38 smithi151 ceph-mon[112233]: pgmap v841: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:39.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:38 smithi153 ceph-mon[87858]: pgmap v841: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:39.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:06:40.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:06:40.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:40 smithi151 ceph-mon[112233]: pgmap v842: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:41.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:40 smithi153 ceph-mon[87858]: pgmap v842: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:42.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:42 smithi151 ceph-mon[112233]: pgmap v843: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:43.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:42 smithi153 ceph-mon[87858]: pgmap v843: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:44.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:44 smithi151 ceph-mon[112233]: pgmap v844: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:44 smithi153 ceph-mon[87858]: pgmap v844: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:45.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.876+0000 7f2392ba4700 1 -- 172.21.15.151:0/608613913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 msgr2=0x7f238c106010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:45.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.876+0000 7f2392ba4700 1 --2- 172.21.15.151:0/608613913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 0x7f238c106010 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2384009910 tx=0x7f2384009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:45.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.876+0000 7f2392ba4700 1 -- 172.21.15.151:0/608613913 shutdown_connections 2024-04-16T03:06:45.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.876+0000 7f2392ba4700 1 --2- 172.21.15.151:0/608613913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f238c1003b0 0x7f238c100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:45.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.876+0000 7f2392ba4700 1 --2- 172.21.15.151:0/608613913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 0x7f238c106010 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:45.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.877+0000 7f2392ba4700 1 -- 172.21.15.151:0/608613913 >> 172.21.15.151:0/608613913 conn(0x7f238c074d80 msgr2=0x7f238c075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:45.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.877+0000 7f2392ba4700 1 -- 172.21.15.151:0/608613913 shutdown_connections 2024-04-16T03:06:45.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.877+0000 7f2392ba4700 1 -- 172.21.15.151:0/608613913 wait complete. 2024-04-16T03:06:45.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.877+0000 7f2392ba4700 1 Processor -- start 2024-04-16T03:06:45.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.877+0000 7f2392ba4700 1 -- start start 2024-04-16T03:06:45.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.877+0000 7f2392ba4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f238c1003b0 0x7f238c195510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:45.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.877+0000 7f2392ba4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 0x7f238c195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:45.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.877+0000 7f2392ba4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c195f90 con 0x7f238c105c30 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.877+0000 7f2392ba4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c196100 con 0x7f238c1003b0 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f238bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 0x7f238c195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f238bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 0x7f238c195a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55216/0 (socket says 172.21.15.151:55216) 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f238bfff700 1 -- 172.21.15.151:0/2794428182 learned_addr learned my addr 172.21.15.151:0/2794428182 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f2390940700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f238c1003b0 0x7f238c195510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f238bfff700 1 -- 172.21.15.151:0/2794428182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f238c1003b0 msgr2=0x7f238c195510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f238bfff700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f238c1003b0 0x7f238c195510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f238bfff700 1 -- 172.21.15.151:0/2794428182 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23840095f0 con 0x7f238c105c30 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f238bfff700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 0x7f238c195a50 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f237c0060b0 tx=0x7f237c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f2389ffb700 1 -- 172.21.15.151:0/2794428182 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f237c00dda0 con 0x7f238c105c30 2024-04-16T03:06:45.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f2389ffb700 1 -- 172.21.15.151:0/2794428182 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f237c00f040 con 0x7f238c105c30 2024-04-16T03:06:45.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c1a2b80 con 0x7f238c105c30 2024-04-16T03:06:45.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.878+0000 7f2389ffb700 1 -- 172.21.15.151:0/2794428182 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f237c0146f0 con 0x7f238c105c30 2024-04-16T03:06:45.883 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.879+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c1a2ee0 con 0x7f238c105c30 2024-04-16T03:06:45.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.879+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c04ea90 con 0x7f238c105c30 2024-04-16T03:06:45.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.880+0000 7f2389ffb700 1 -- 172.21.15.151:0/2794428182 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f237c015400 con 0x7f238c105c30 2024-04-16T03:06:45.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.880+0000 7f2389ffb700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f237806ecc0 0x7f2378071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:45.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.880+0000 7f2389ffb700 1 -- 172.21.15.151:0/2794428182 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f237c08e7c0 con 0x7f238c105c30 2024-04-16T03:06:45.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.880+0000 7f2390940700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f237806ecc0 0x7f2378071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:45.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.881+0000 7f2390940700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f237806ecc0 0x7f2378071180 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2384006010 tx=0x7f238401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:45.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:45.885+0000 7f2389ffb700 1 -- 172.21.15.151:0/2794428182 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f237c05b150 con 0x7f238c105c30 2024-04-16T03:06:46.108 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.106+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f238c0fd960 con 0x7f237806ecc0 2024-04-16T03:06:46.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.108+0000 7f2389ffb700 1 -- 172.21.15.151:0/2794428182 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f238c0fd960 con 0x7f237806ecc0 2024-04-16T03:06:46.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f237806ecc0 msgr2=0x7f2378071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f237806ecc0 0x7f2378071180 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2384006010 tx=0x7f238401a040 comp rx=0 tx=0).stop 2024-04-16T03:06:46.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 msgr2=0x7f238c195a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 0x7f238c195a50 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f237c0060b0 tx=0x7f237c00d3b0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 shutdown_connections 2024-04-16T03:06:46.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f237806ecc0 0x7f2378071180 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f238c1003b0 0x7f238c195510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 --2- 172.21.15.151:0/2794428182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f238c105c30 0x7f238c195a50 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 >> 172.21.15.151:0/2794428182 conn(0x7f238c074d80 msgr2=0x7f238c0fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:46.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 shutdown_connections 2024-04-16T03:06:46.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.110+0000 7f2392ba4700 1 -- 172.21.15.151:0/2794428182 wait complete. 2024-04-16T03:06:46.122 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:06:46.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.215+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/1026193669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f68101bc0 msgr2=0x7f4f68101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.215+0000 7f4f6d1a2700 1 --2- 172.21.15.151:0/1026193669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f68101bc0 0x7f4f68101fa0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4f58009940 tx=0x7f4f58009c50 comp rx=0 tx=0).stop 2024-04-16T03:06:46.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.215+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/1026193669 shutdown_connections 2024-04-16T03:06:46.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.215+0000 7f4f6d1a2700 1 --2- 172.21.15.151:0/1026193669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f681024e0 0x7f4f6810a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.215+0000 7f4f6d1a2700 1 --2- 172.21.15.151:0/1026193669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f68101bc0 0x7f4f68101fa0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.215+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/1026193669 >> 172.21.15.151:0/1026193669 conn(0x7f4f680fb390 msgr2=0x7f4f680fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.215+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/1026193669 shutdown_connections 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.215+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/1026193669 wait complete. 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.216+0000 7f4f6d1a2700 1 Processor -- start 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.216+0000 7f4f6d1a2700 1 -- start start 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.216+0000 7f4f6d1a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f68101bc0 0x7f4f68196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.216+0000 7f4f6d1a2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f681024e0 0x7f4f68196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.216+0000 7f4f6d1a2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68197190 con 0x7f4f681024e0 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.216+0000 7f4f6d1a2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6819af20 con 0x7f4f68101bc0 2024-04-16T03:06:46.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.216+0000 7f4f677fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f681024e0 0x7f4f68196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.216+0000 7f4f677fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f681024e0 0x7f4f68196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55248/0 (socket says 172.21.15.151:55248) 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.216+0000 7f4f677fe700 1 -- 172.21.15.151:0/2884489431 learned_addr learned my addr 172.21.15.151:0/2884489431 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f67fff700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f68101bc0 0x7f4f68196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f677fe700 1 -- 172.21.15.151:0/2884489431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f68101bc0 msgr2=0x7f4f68196570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f677fe700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f68101bc0 0x7f4f68196570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f677fe700 1 -- 172.21.15.151:0/2884489431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f580095f0 con 0x7f4f681024e0 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f677fe700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f681024e0 0x7f4f68196ab0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00c8f0 tx=0x7f4f5c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f657fa700 1 -- 172.21.15.151:0/2884489431 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f5c007a50 con 0x7f4f681024e0 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6819b200 con 0x7f4f681024e0 2024-04-16T03:06:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f657fa700 1 -- 172.21.15.151:0/2884489431 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f5c007bb0 con 0x7f4f681024e0 2024-04-16T03:06:46.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f657fa700 1 -- 172.21.15.151:0/2884489431 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f5c018630 con 0x7f4f681024e0 2024-04-16T03:06:46.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.217+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6819b750 con 0x7f4f681024e0 2024-04-16T03:06:46.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.218+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f681080d0 con 0x7f4f681024e0 2024-04-16T03:06:46.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.219+0000 7f4f657fa700 1 -- 172.21.15.151:0/2884489431 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4f5c007d20 con 0x7f4f681024e0 2024-04-16T03:06:46.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.219+0000 7f4f657fa700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4f5006ecc0 0x7f4f50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:46.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.220+0000 7f4f67fff700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4f5006ecc0 0x7f4f50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:46.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.220+0000 7f4f657fa700 1 -- 172.21.15.151:0/2884489431 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4f5c08ee50 con 0x7f4f681024e0 2024-04-16T03:06:46.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.220+0000 7f4f67fff700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4f5006ecc0 0x7f4f50071180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4f58006010 tx=0x7f4f5800b810 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:46.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.224+0000 7f4f657fa700 1 -- 172.21.15.151:0/2884489431 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4f5c057c00 con 0x7f4f681024e0 2024-04-16T03:06:46.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.442+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f6819bac0 con 0x7f4f5006ecc0 2024-04-16T03:06:46.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.443+0000 7f4f657fa700 1 -- 172.21.15.151:0/2884489431 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4f6819bac0 con 0x7f4f5006ecc0 2024-04-16T03:06:46.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4f5006ecc0 msgr2=0x7f4f50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4f5006ecc0 0x7f4f50071180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4f58006010 tx=0x7f4f5800b810 comp rx=0 tx=0).stop 2024-04-16T03:06:46.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f681024e0 msgr2=0x7f4f68196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f681024e0 0x7f4f68196ab0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00c8f0 tx=0x7f4f5c00ccb0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 shutdown_connections 2024-04-16T03:06:46.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4f5006ecc0 0x7f4f50071180 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f68101bc0 0x7f4f68196570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 --2- 172.21.15.151:0/2884489431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f681024e0 0x7f4f68196ab0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 >> 172.21.15.151:0/2884489431 conn(0x7f4f680fb390 msgr2=0x7f4f68105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 shutdown_connections 2024-04-16T03:06:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.446+0000 7f4f6d1a2700 1 -- 172.21.15.151:0/2884489431 wait complete. 2024-04-16T03:06:46.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.553+0000 7f6f0cef7700 1 -- 172.21.15.151:0/511033996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f08105a40 msgr2=0x7f6f08105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.553+0000 7f6f0cef7700 1 --2- 172.21.15.151:0/511033996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f08105a40 0x7f6f08105e20 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009910 tx=0x7f6efc009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:46.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.553+0000 7f6f0cef7700 1 -- 172.21.15.151:0/511033996 shutdown_connections 2024-04-16T03:06:46.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.553+0000 7f6f0cef7700 1 --2- 172.21.15.151:0/511033996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f080ffa40 0x7f6f080ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.553+0000 7f6f0cef7700 1 --2- 172.21.15.151:0/511033996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f08105a40 0x7f6f08105e20 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.553+0000 7f6f0cef7700 1 -- 172.21.15.151:0/511033996 >> 172.21.15.151:0/511033996 conn(0x7f6f080fb570 msgr2=0x7f6f080fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:46.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.553+0000 7f6f0cef7700 1 -- 172.21.15.151:0/511033996 shutdown_connections 2024-04-16T03:06:46.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.553+0000 7f6f0cef7700 1 -- 172.21.15.151:0/511033996 wait complete. 2024-04-16T03:06:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.554+0000 7f6f0cef7700 1 Processor -- start 2024-04-16T03:06:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.554+0000 7f6f0cef7700 1 -- start start 2024-04-16T03:06:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.554+0000 7f6f0cef7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f080ffa40 0x7f6f08195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.554+0000 7f6f0cef7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f08105a40 0x7f6f08195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.554+0000 7f6f0cef7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f08196270 con 0x7f6f080ffa40 2024-04-16T03:06:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.554+0000 7f6f0cef7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0819a000 con 0x7f6f08105a40 2024-04-16T03:06:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.554+0000 7f6f0659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f080ffa40 0x7f6f08195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.554+0000 7f6f0659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f080ffa40 0x7f6f08195650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51568/0 (socket says 172.21.15.151:51568) 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.554+0000 7f6f0659c700 1 -- 172.21.15.151:0/3578917630 learned_addr learned my addr 172.21.15.151:0/3578917630 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6f05d9b700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f08105a40 0x7f6f08195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6f0659c700 1 -- 172.21.15.151:0/3578917630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f08105a40 msgr2=0x7f6f08195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6f0659c700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f08105a40 0x7f6f08195b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6f0659c700 1 -- 172.21.15.151:0/3578917630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc0095f0 con 0x7f6f080ffa40 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6f0659c700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f080ffa40 0x7f6f08195650 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6efc006010 tx=0x7f6efc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6ef77fe700 1 -- 172.21.15.151:0/3578917630 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6efc01d070 con 0x7f6f080ffa40 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6ef77fe700 1 -- 172.21.15.151:0/3578917630 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6efc00f460 con 0x7f6f080ffa40 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6ef77fe700 1 -- 172.21.15.151:0/3578917630 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6efc017630 con 0x7f6f080ffa40 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0819a280 con 0x7f6f080ffa40 2024-04-16T03:06:46.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.555+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0819a770 con 0x7f6f080ffa40 2024-04-16T03:06:46.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.556+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f08108040 con 0x7f6f080ffa40 2024-04-16T03:06:46.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.557+0000 7f6ef77fe700 1 -- 172.21.15.151:0/3578917630 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6efc00f5d0 con 0x7f6f080ffa40 2024-04-16T03:06:46.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.557+0000 7f6ef77fe700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ef006ecc0 0x7f6ef0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:46.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.557+0000 7f6ef77fe700 1 -- 172.21.15.151:0/3578917630 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6efc090310 con 0x7f6f080ffa40 2024-04-16T03:06:46.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.557+0000 7f6f05d9b700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ef006ecc0 0x7f6ef0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:46.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.558+0000 7f6f05d9b700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ef006ecc0 0x7f6ef0071180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8005950 tx=0x7f6ef80058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:46.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.562+0000 7f6ef77fe700 1 -- 172.21.15.151:0/3578917630 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6efc0590c0 con 0x7f6f080ffa40 2024-04-16T03:06:46.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.772+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f0802cd30 con 0x7f6ef006ecc0 2024-04-16T03:06:46.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.783+0000 7f6ef77fe700 1 -- 172.21.15.151:0/3578917630 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6f0802cd30 con 0x7f6ef006ecc0 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (28m) 6m ago 39m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (39m) 6m ago 39m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (38m) 6m ago 38m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (25m) 6m ago 39m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (25m) 6m ago 38m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (27m) 6m ago 39m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (22m) 6m ago 34m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (21m) 6m ago 34m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (6m) 6m ago 33m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:06:46.785 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (21m) 6m ago 34m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (30m) 6m ago 41m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (29m) 6m ago 38m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (27m) 6m ago 41m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (26m) 6m ago 38m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (29m) 6m ago 39m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (28m) 6m ago 38m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (25m) 6m ago 37m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (24m) 6m ago 37m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (24m) 6m ago 36m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (23m) 6m ago 36m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (23m) 6m ago 36m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:06:46.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (22m) 6m ago 35m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:06:46.787 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (28m) 6m ago 39m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:06:46.787 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.785+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ef006ecc0 msgr2=0x7f6ef0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.787 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.785+0000 7f6f0cef7700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ef006ecc0 0x7f6ef0071180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8005950 tx=0x7f6ef80058e0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.785+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f080ffa40 msgr2=0x7f6f08195650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.785+0000 7f6f0cef7700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f080ffa40 0x7f6f08195650 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6efc006010 tx=0x7f6efc00be20 comp rx=0 tx=0).stop 2024-04-16T03:06:46.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.786+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 shutdown_connections 2024-04-16T03:06:46.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.786+0000 7f6f0cef7700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6ef006ecc0 0x7f6ef0071180 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.786+0000 7f6f0cef7700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f080ffa40 0x7f6f08195650 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.786+0000 7f6f0cef7700 1 --2- 172.21.15.151:0/3578917630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f08105a40 0x7f6f08195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.786+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 >> 172.21.15.151:0/3578917630 conn(0x7f6f080fb570 msgr2=0x7f6f080fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:46.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.786+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 shutdown_connections 2024-04-16T03:06:46.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.786+0000 7f6f0cef7700 1 -- 172.21.15.151:0/3578917630 wait complete. 2024-04-16T03:06:46.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.890+0000 7f6a87662700 1 -- 172.21.15.151:0/3892603030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 msgr2=0x7f6a800fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.890+0000 7f6a87662700 1 --2- 172.21.15.151:0/3892603030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 0x7f6a800fff00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009910 tx=0x7f6a70009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:46.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.890+0000 7f6a87662700 1 -- 172.21.15.151:0/3892603030 shutdown_connections 2024-04-16T03:06:46.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.890+0000 7f6a87662700 1 --2- 172.21.15.151:0/3892603030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 0x7f6a800fff00 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.890+0000 7f6a87662700 1 --2- 172.21.15.151:0/3892603030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80105a80 0x7f6a80105e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.890+0000 7f6a87662700 1 -- 172.21.15.151:0/3892603030 >> 172.21.15.151:0/3892603030 conn(0x7f6a800fb590 msgr2=0x7f6a800fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:46.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.890+0000 7f6a87662700 1 -- 172.21.15.151:0/3892603030 shutdown_connections 2024-04-16T03:06:46.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.890+0000 7f6a87662700 1 -- 172.21.15.151:0/3892603030 wait complete. 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a87662700 1 Processor -- start 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a87662700 1 -- start start 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a87662700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 0x7f6a8019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a87662700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80105a80 0x7f6a8019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a87662700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8019d0c0 con 0x7f6a800ffa80 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a87662700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a80196570 con 0x7f6a80105a80 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a853fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 0x7f6a8019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a853fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 0x7f6a8019c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51588/0 (socket says 172.21.15.151:51588) 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a853fe700 1 -- 172.21.15.151:0/1273991949 learned_addr learned my addr 172.21.15.151:0/1273991949 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a84bfd700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80105a80 0x7f6a8019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a853fe700 1 -- 172.21.15.151:0/1273991949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80105a80 msgr2=0x7f6a8019ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:46.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a853fe700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80105a80 0x7f6a8019ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:46.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.891+0000 7f6a853fe700 1 -- 172.21.15.151:0/1273991949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a700095f0 con 0x7f6a800ffa80 2024-04-16T03:06:46.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.892+0000 7f6a853fe700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 0x7f6a8019c4f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00d8d0 tx=0x7f6a7c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:46.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.892+0000 7f6a767fc700 1 -- 172.21.15.151:0/1273991949 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a7c0098a0 con 0x7f6a800ffa80 2024-04-16T03:06:46.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.892+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a80196850 con 0x7f6a800ffa80 2024-04-16T03:06:46.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.892+0000 7f6a767fc700 1 -- 172.21.15.151:0/1273991949 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a7c010460 con 0x7f6a800ffa80 2024-04-16T03:06:46.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.892+0000 7f6a767fc700 1 -- 172.21.15.151:0/1273991949 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a7c00f5f0 con 0x7f6a800ffa80 2024-04-16T03:06:46.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.892+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a80196da0 con 0x7f6a800ffa80 2024-04-16T03:06:46.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.893+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a80109020 con 0x7f6a800ffa80 2024-04-16T03:06:46.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.893+0000 7f6a767fc700 1 -- 172.21.15.151:0/1273991949 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6a7c009a00 con 0x7f6a800ffa80 2024-04-16T03:06:46.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.894+0000 7f6a767fc700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a6c06ecc0 0x7f6a6c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:46.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.894+0000 7f6a767fc700 1 -- 172.21.15.151:0/1273991949 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6a7c08ed30 con 0x7f6a800ffa80 2024-04-16T03:06:46.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.894+0000 7f6a84bfd700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a6c06ecc0 0x7f6a6c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:46.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.894+0000 7f6a84bfd700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a6c06ecc0 0x7f6a6c071180 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6a80105240 tx=0x7f6a7001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:46.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:46.899+0000 7f6a767fc700 1 -- 172.21.15.151:0/1273991949 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6a7c05b610 con 0x7f6a800ffa80 2024-04-16T03:06:46.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:46 smithi151 ceph-mon[112233]: pgmap v845: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:47.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:46 smithi153 ceph-mon[87858]: pgmap v845: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:47.181 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.180+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6a80197040 con 0x7f6a800ffa80 2024-04-16T03:06:47.182 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.180+0000 7f6a767fc700 1 -- 172.21.15.151:0/1273991949 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6a7c05b1a0 con 0x7f6a800ffa80 2024-04-16T03:06:47.182 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:06:47.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:06:47.182 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:06:47.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:06:47.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:06:47.182 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:06:47.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:06:47.183 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:06:47.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a6c06ecc0 msgr2=0x7f6a6c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a6c06ecc0 0x7f6a6c071180 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6a80105240 tx=0x7f6a7001a040 comp rx=0 tx=0).stop 2024-04-16T03:06:47.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 msgr2=0x7f6a8019c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 0x7f6a8019c4f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00d8d0 tx=0x7f6a7c00dbe0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 shutdown_connections 2024-04-16T03:06:47.184 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a6c06ecc0 0x7f6a6c071180 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a800ffa80 0x7f6a8019c4f0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 --2- 172.21.15.151:0/1273991949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80105a80 0x7f6a8019ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 >> 172.21.15.151:0/1273991949 conn(0x7f6a800fb590 msgr2=0x7f6a80069010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:47.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 shutdown_connections 2024-04-16T03:06:47.185 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.183+0000 7f6a87662700 1 -- 172.21.15.151:0/1273991949 wait complete. 2024-04-16T03:06:47.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.288+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/2250035286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 msgr2=0x7f0b541011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.288+0000 7f0b5c0ee700 1 --2- 172.21.15.151:0/2250035286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 0x7f0b541011b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009910 tx=0x7f0b50009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:47.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.288+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/2250035286 shutdown_connections 2024-04-16T03:06:47.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.288+0000 7f0b5c0ee700 1 --2- 172.21.15.151:0/2250035286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b54101780 0x7f0b541057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.288+0000 7f0b5c0ee700 1 --2- 172.21.15.151:0/2250035286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 0x7f0b541011b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.288+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/2250035286 >> 172.21.15.151:0/2250035286 conn(0x7f0b540fc640 msgr2=0x7f0b540fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:47.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.288+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/2250035286 shutdown_connections 2024-04-16T03:06:47.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.288+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/2250035286 wait complete. 2024-04-16T03:06:47.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.288+0000 7f0b5c0ee700 1 Processor -- start 2024-04-16T03:06:47.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b5c0ee700 1 -- start start 2024-04-16T03:06:47.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b5c0ee700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 0x7f0b54073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:47.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b5c0ee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b54101780 0x7f0b540735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:47.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b5c0ee700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b54073b10 con 0x7f0b54100dd0 2024-04-16T03:06:47.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b5c0ee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b54073c50 con 0x7f0b54101780 2024-04-16T03:06:47.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b59e8a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 0x7f0b54073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:47.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b59e8a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 0x7f0b54073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51618/0 (socket says 172.21.15.151:51618) 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b59e8a700 1 -- 172.21.15.151:0/1491624542 learned_addr learned my addr 172.21.15.151:0/1491624542 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b59689700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b54101780 0x7f0b540735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b59e8a700 1 -- 172.21.15.151:0/1491624542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b54101780 msgr2=0x7f0b540735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b59e8a700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b54101780 0x7f0b540735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b59e8a700 1 -- 172.21.15.151:0/1491624542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b500095f0 con 0x7f0b54100dd0 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b59689700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b54101780 0x7f0b540735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b59e8a700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 0x7f0b54073090 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000f690 tx=0x7f0b5000f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b4affd700 1 -- 172.21.15.151:0/1491624542 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b5001d070 con 0x7f0b54100dd0 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b4affd700 1 -- 172.21.15.151:0/1491624542 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b500050d0 con 0x7f0b54100dd0 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.289+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b54073ed0 con 0x7f0b54100dd0 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.290+0000 7f0b4affd700 1 -- 172.21.15.151:0/1491624542 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b50022470 con 0x7f0b54100dd0 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.290+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b541a2c40 con 0x7f0b54100dd0 2024-04-16T03:06:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.291+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b54109170 con 0x7f0b54100dd0 2024-04-16T03:06:47.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.291+0000 7f0b4affd700 1 -- 172.21.15.151:0/1491624542 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f0b50003680 con 0x7f0b54100dd0 2024-04-16T03:06:47.294 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.292+0000 7f0b4affd700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b4006ecc0 0x7f0b40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:47.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.292+0000 7f0b4affd700 1 -- 172.21.15.151:0/1491624542 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0b50090220 con 0x7f0b54100dd0 2024-04-16T03:06:47.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.292+0000 7f0b59689700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b4006ecc0 0x7f0b40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:47.295 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.292+0000 7f0b59689700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b4006ecc0 0x7f0b40071180 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0b54074880 tx=0x7f0b44009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:47.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.296+0000 7f0b4affd700 1 -- 172.21.15.151:0/1491624542 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0b50059100 con 0x7f0b54100dd0 2024-04-16T03:06:47.540 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.539+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0b5404ea90 con 0x7f0b54100dd0 2024-04-16T03:06:47.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.539+0000 7f0b4affd700 1 -- 172.21.15.151:0/1491624542 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f0b5005c7c0 con 0x7f0b54100dd0 2024-04-16T03:06:47.541 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:06:47.541 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:06:47.541 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:06:47.541 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:06:47.541 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:06:47.541 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:06:47.541 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:06:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:06:47.543 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:06:47.544 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:06:47.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b4006ecc0 msgr2=0x7f0b40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b4006ecc0 0x7f0b40071180 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0b54074880 tx=0x7f0b44009260 comp rx=0 tx=0).stop 2024-04-16T03:06:47.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 msgr2=0x7f0b54073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 0x7f0b54073090 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000f690 tx=0x7f0b5000f770 comp rx=0 tx=0).stop 2024-04-16T03:06:47.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 shutdown_connections 2024-04-16T03:06:47.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b4006ecc0 0x7f0b40071180 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b54100dd0 0x7f0b54073090 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 --2- 172.21.15.151:0/1491624542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b54101780 0x7f0b540735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 >> 172.21.15.151:0/1491624542 conn(0x7f0b540fc640 msgr2=0x7f0b540fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:47.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 shutdown_connections 2024-04-16T03:06:47.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.541+0000 7f0b5c0ee700 1 -- 172.21.15.151:0/1491624542 wait complete. 2024-04-16T03:06:47.546 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:06:47.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.647+0000 7fcff1ac3700 1 -- 172.21.15.151:0/3801551710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 msgr2=0x7fcfec111630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.647+0000 7fcff1ac3700 1 --2- 172.21.15.151:0/3801551710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 0x7fcfec111630 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009910 tx=0x7fcfd4009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:47.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.647+0000 7fcff1ac3700 1 -- 172.21.15.151:0/3801551710 shutdown_connections 2024-04-16T03:06:47.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.647+0000 7fcff1ac3700 1 --2- 172.21.15.151:0/3801551710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 0x7fcfec111630 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.647+0000 7fcff1ac3700 1 --2- 172.21.15.151:0/3801551710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfec073130 0x7fcfec073510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.647+0000 7fcff1ac3700 1 -- 172.21.15.151:0/3801551710 >> 172.21.15.151:0/3801551710 conn(0x7fcfec0fc650 msgr2=0x7fcfec0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:47.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.647+0000 7fcff1ac3700 1 -- 172.21.15.151:0/3801551710 shutdown_connections 2024-04-16T03:06:47.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.647+0000 7fcff1ac3700 1 -- 172.21.15.151:0/3801551710 wait complete. 2024-04-16T03:06:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcff1ac3700 1 Processor -- start 2024-04-16T03:06:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcff1ac3700 1 -- start start 2024-04-16T03:06:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcff1ac3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfec073130 0x7fcfec1a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcff1ac3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 0x7fcfec1a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcff1ac3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec1a3680 con 0x7fcfec073a50 2024-04-16T03:06:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcff1ac3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec19cb30 con 0x7fcfec073130 2024-04-16T03:06:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcfe3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 0x7fcfec1a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcfe3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 0x7fcfec1a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51636/0 (socket says 172.21.15.151:51636) 2024-04-16T03:06:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcfe3fff700 1 -- 172.21.15.151:0/161790586 learned_addr learned my addr 172.21.15.151:0/161790586 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcfeb7fe700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfec073130 0x7fcfec1a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcfe3fff700 1 -- 172.21.15.151:0/161790586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfec073130 msgr2=0x7fcfec1a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcfe3fff700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfec073130 0x7fcfec1a2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcfe3fff700 1 -- 172.21.15.151:0/161790586 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd40095f0 con 0x7fcfec073a50 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcfe3fff700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 0x7fcfec1a2ff0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4006010 tx=0x7fcfd400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.648+0000 7fcfe97fa700 1 -- 172.21.15.151:0/161790586 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd401d070 con 0x7fcfec073a50 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.649+0000 7fcfe97fa700 1 -- 172.21.15.151:0/161790586 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfd400f460 con 0x7fcfec073a50 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.649+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec19ce10 con 0x7fcfec073a50 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.649+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec19d360 con 0x7fcfec073a50 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.649+0000 7fcfe97fa700 1 -- 172.21.15.151:0/161790586 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd4017630 con 0x7fcfec073a50 2024-04-16T03:06:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.649+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec10edb0 con 0x7fcfec073a50 2024-04-16T03:06:47.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.650+0000 7fcfe97fa700 1 -- 172.21.15.151:0/161790586 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fcfd400f5d0 con 0x7fcfec073a50 2024-04-16T03:06:47.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.650+0000 7fcfe97fa700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcfd806ecc0 0x7fcfd8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:47.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.651+0000 7fcfe97fa700 1 -- 172.21.15.151:0/161790586 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcfd40902b0 con 0x7fcfec073a50 2024-04-16T03:06:47.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.651+0000 7fcfeb7fe700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcfd806ecc0 0x7fcfd8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:47.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.651+0000 7fcfeb7fe700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcfd806ecc0 0x7fcfd8071180 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005950 tx=0x7fcfdc0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:47.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.655+0000 7fcfe97fa700 1 -- 172.21.15.151:0/161790586 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcfd405cb90 con 0x7fcfec073a50 2024-04-16T03:06:47.876 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.874+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfec19d690 con 0x7fcfd806ecc0 2024-04-16T03:06:47.876 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:47 smithi151 ceph-mon[112233]: from='client.35548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:47.876 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:47 smithi151 ceph-mon[112233]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:47.876 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:47 smithi151 ceph-mon[112233]: from='client.35556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:47.876 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:47 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1273991949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:06:47.876 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:47 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1491624542' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:06:47.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.876+0000 7fcfe97fa700 1 -- 172.21.15.151:0/161790586 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcfec19d690 con 0x7fcfd806ecc0 2024-04-16T03:06:47.877 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:06:47.877 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:06:47.877 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:06:47.877 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:06:47.877 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:06:47.878 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcfd806ecc0 msgr2=0x7fcfd8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcfd806ecc0 0x7fcfd8071180 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005950 tx=0x7fcfdc0058e0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 msgr2=0x7fcfec1a2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 0x7fcfec1a2ff0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4006010 tx=0x7fcfd400be20 comp rx=0 tx=0).stop 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 shutdown_connections 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fcfd806ecc0 0x7fcfd8071180 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfec073130 0x7fcfec1a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 --2- 172.21.15.151:0/161790586 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfec073a50 0x7fcfec1a2ff0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 >> 172.21.15.151:0/161790586 conn(0x7fcfec0fc650 msgr2=0x7fcfec103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 shutdown_connections 2024-04-16T03:06:47.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.878+0000 7fcff1ac3700 1 -- 172.21.15.151:0/161790586 wait complete. 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 -- 172.21.15.151:0/579222275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ff500 msgr2=0x7f79dc0ff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 --2- 172.21.15.151:0/579222275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ff500 0x7f79dc0ff8e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009910 tx=0x7f79cc009c20 comp rx=0 tx=0).stop 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 -- 172.21.15.151:0/579222275 shutdown_connections 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 --2- 172.21.15.151:0/579222275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79dc0ffeb0 0x7f79dc105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 --2- 172.21.15.151:0/579222275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ff500 0x7f79dc0ff8e0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 -- 172.21.15.151:0/579222275 >> 172.21.15.151:0/579222275 conn(0x7f79dc074d80 msgr2=0x7f79dc075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 -- 172.21.15.151:0/579222275 shutdown_connections 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 -- 172.21.15.151:0/579222275 wait complete. 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 Processor -- start 2024-04-16T03:06:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.984+0000 7f79e2964700 1 -- start start 2024-04-16T03:06:47.988 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79e2964700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79dc0ff500 0x7f79dc19c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:47.988 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79e2964700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ffeb0 0x7f79dc19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:47.988 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79e2964700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc19d1e0 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79e2964700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc196690 con 0x7f79dc0ff500 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79db7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ffeb0 0x7f79dc19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79db7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ffeb0 0x7f79dc19cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:51658/0 (socket says 172.21.15.151:51658) 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79db7fe700 1 -- 172.21.15.151:0/981613004 learned_addr learned my addr 172.21.15.151:0/981613004 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79dbfff700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79dc0ff500 0x7f79dc19c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79db7fe700 1 -- 172.21.15.151:0/981613004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79dc0ff500 msgr2=0x7f79dc19c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79db7fe700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79dc0ff500 0x7f79dc19c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79db7fe700 1 -- 172.21.15.151:0/981613004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79cc0095f0 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79db7fe700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ffeb0 0x7f79dc19cb50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f79d000d900 tx=0x7f79d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79d97fa700 1 -- 172.21.15.151:0/981613004 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79d0009900 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79d97fa700 1 -- 172.21.15.151:0/981613004 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79d0010460 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc196970 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.985+0000 7f79d97fa700 1 -- 172.21.15.151:0/981613004 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79d000f5f0 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.986+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc196e70 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.986+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc109190 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.987+0000 7f79d97fa700 1 -- 172.21.15.151:0/981613004 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f79d00105d0 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.988+0000 7f79d97fa700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f79c406ecc0 0x7f79c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:06:47.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.988+0000 7f79d97fa700 1 -- 172.21.15.151:0/981613004 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f79d008ecb0 con 0x7f79dc0ffeb0 2024-04-16T03:06:47.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.988+0000 7f79dbfff700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f79c406ecc0 0x7f79c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:06:47.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.988+0000 7f79dbfff700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f79c406ecc0 0x7f79c4071180 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00b5c0 tx=0x7f79cc005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:06:47.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:47.992+0000 7f79d97fa700 1 -- 172.21.15.151:0/981613004 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f79d005b590 con 0x7f79dc0ffeb0 2024-04-16T03:06:48.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:47 smithi153 ceph-mon[87858]: from='client.35548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:48.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:47 smithi153 ceph-mon[87858]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:48.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:47 smithi153 ceph-mon[87858]: from='client.35556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:48.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:47 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1273991949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:06:48.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:47 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1491624542' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:06:48.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.276+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f79dc04ea90 con 0x7f79dc0ffeb0 2024-04-16T03:06:48.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.276+0000 7f79d97fa700 1 -- 172.21.15.151:0/981613004 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f79d005b120 con 0x7f79dc0ffeb0 2024-04-16T03:06:48.278 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:06:48.278 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:06:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:06:48.279 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:06:48.279 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:06:48.279 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:06:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:06:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:06:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:06:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:06:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:06:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:06:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:06:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:06:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:06:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:06:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:06:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:06:48.280 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:06:48.280 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:06:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:06:48.281 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:06:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:06:48.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f79c406ecc0 msgr2=0x7f79c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f79c406ecc0 0x7f79c4071180 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00b5c0 tx=0x7f79cc005960 comp rx=0 tx=0).stop 2024-04-16T03:06:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ffeb0 msgr2=0x7f79dc19cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:06:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ffeb0 0x7f79dc19cb50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f79d000d900 tx=0x7f79d000dcc0 comp rx=0 tx=0).stop 2024-04-16T03:06:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 shutdown_connections 2024-04-16T03:06:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f79c406ecc0 0x7f79c4071180 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79dc0ff500 0x7f79dc19c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 --2- 172.21.15.151:0/981613004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79dc0ffeb0 0x7f79dc19cb50 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:06:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 >> 172.21.15.151:0/981613004 conn(0x7f79dc074d80 msgr2=0x7f79dc0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:06:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 shutdown_connections 2024-04-16T03:06:48.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:06:48.279+0000 7f79e2964700 1 -- 172.21.15.151:0/981613004 wait complete. 2024-04-16T03:06:48.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:48 smithi153 ceph-mon[87858]: pgmap v846: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:48.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:48 smithi153 ceph-mon[87858]: from='client.35568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:48.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:48 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/981613004' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:06:48.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:48 smithi151 ceph-mon[112233]: pgmap v846: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:48.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:48 smithi151 ceph-mon[112233]: from='client.35568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:06:48.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:48 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/981613004' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:06:50.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:50 smithi151 ceph-mon[112233]: pgmap v847: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:51.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:50 smithi153 ceph-mon[87858]: pgmap v847: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:52.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:52 smithi151 ceph-mon[112233]: pgmap v848: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:53.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:52 smithi153 ceph-mon[87858]: pgmap v848: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:54 smithi151 ceph-mon[112233]: pgmap v849: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:06:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:54 smithi153 ceph-mon[87858]: pgmap v849: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:06:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:56 smithi151 ceph-mon[112233]: pgmap v850: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:57.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:56 smithi153 ceph-mon[87858]: pgmap v850: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:06:58 smithi151 ceph-mon[112233]: pgmap v851: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:06:59.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:06:58 smithi153 ceph-mon[87858]: pgmap v851: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:00 smithi151 ceph-mon[112233]: pgmap v852: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:00 smithi153 ceph-mon[87858]: pgmap v852: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:02 smithi151 ceph-mon[112233]: pgmap v853: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:03.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:02 smithi153 ceph-mon[87858]: pgmap v853: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:05.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:04 smithi153 ceph-mon[87858]: pgmap v854: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:05.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:04 smithi151 ceph-mon[112233]: pgmap v854: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:07.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:06 smithi153 ceph-mon[87858]: pgmap v855: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:07.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:06 smithi151 ceph-mon[112233]: pgmap v855: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:09.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:08 smithi153 ceph-mon[87858]: pgmap v856: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:08 smithi151 ceph-mon[112233]: pgmap v856: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:07:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:07:11.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:10 smithi153 ceph-mon[87858]: pgmap v857: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:11.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:10 smithi151 ceph-mon[112233]: pgmap v857: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:13.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:12 smithi153 ceph-mon[87858]: pgmap v858: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:12 smithi151 ceph-mon[112233]: pgmap v858: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:15.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:14 smithi153 ceph-mon[87858]: pgmap v859: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:15.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:14 smithi151 ceph-mon[112233]: pgmap v859: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:17.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:16 smithi153 ceph-mon[87858]: pgmap v860: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:17.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:16 smithi151 ceph-mon[112233]: pgmap v860: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.398+0000 7f3522652700 1 -- 172.21.15.151:0/1963660491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c102620 msgr2=0x7f351c10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.398+0000 7f3522652700 1 --2- 172.21.15.151:0/1963660491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c102620 0x7f351c10ab10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3510009910 tx=0x7f3510009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.398+0000 7f3522652700 1 -- 172.21.15.151:0/1963660491 shutdown_connections 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.398+0000 7f3522652700 1 --2- 172.21.15.151:0/1963660491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c102620 0x7f351c10ab10 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.398+0000 7f3522652700 1 --2- 172.21.15.151:0/1963660491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c101d00 0x7f351c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.398+0000 7f3522652700 1 -- 172.21.15.151:0/1963660491 >> 172.21.15.151:0/1963660491 conn(0x7f351c0fb550 msgr2=0x7f351c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.398+0000 7f3522652700 1 -- 172.21.15.151:0/1963660491 shutdown_connections 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.398+0000 7f3522652700 1 -- 172.21.15.151:0/1963660491 wait complete. 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.399+0000 7f3522652700 1 Processor -- start 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.399+0000 7f3522652700 1 -- start start 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.399+0000 7f3522652700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c101d00 0x7f351c19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:18.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.399+0000 7f3522652700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102620 0x7f351c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.399+0000 7f3522652700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c19d130 con 0x7f351c101d00 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.399+0000 7f3522652700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c1965e0 con 0x7f351c102620 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f351bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c101d00 0x7f351c19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f351bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c101d00 0x7f351c19c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53906/0 (socket says 172.21.15.151:53906) 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f351bfff700 1 -- 172.21.15.151:0/3687006482 learned_addr learned my addr 172.21.15.151:0/3687006482 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f351b7fe700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102620 0x7f351c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f351bfff700 1 -- 172.21.15.151:0/3687006482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102620 msgr2=0x7f351c19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f351bfff700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102620 0x7f351c19caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f351bfff700 1 -- 172.21.15.151:0/3687006482 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35100095f0 con 0x7f351c101d00 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f351bfff700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c101d00 0x7f351c19c560 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f350c00d8d0 tx=0x7f350c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f35197fa700 1 -- 172.21.15.151:0/3687006482 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f350c009940 con 0x7f351c101d00 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f35197fa700 1 -- 172.21.15.151:0/3687006482 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f350c010460 con 0x7f351c101d00 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c1968c0 con 0x7f351c101d00 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f35197fa700 1 -- 172.21.15.151:0/3687006482 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f350c00f5f0 con 0x7f351c101d00 2024-04-16T03:07:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.400+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c196de0 con 0x7f351c101d00 2024-04-16T03:07:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.401+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c108210 con 0x7f351c101d00 2024-04-16T03:07:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.402+0000 7f35197fa700 1 -- 172.21.15.151:0/3687006482 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f350c0105d0 con 0x7f351c101d00 2024-04-16T03:07:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.402+0000 7f35197fa700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.402+0000 7f35197fa700 1 -- 172.21.15.151:0/3687006482 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f350c08ed80 con 0x7f351c101d00 2024-04-16T03:07:18.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.403+0000 7f351b7fe700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:18.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.403+0000 7f351b7fe700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3510006010 tx=0x7f351001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:18.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.407+0000 7f35197fa700 1 -- 172.21.15.151:0/3687006482 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f350c05b6e0 con 0x7f351c101d00 2024-04-16T03:07:18.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.624+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f351c197110 con 0x7f350406ecc0 2024-04-16T03:07:18.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.626+0000 7f35197fa700 1 -- 172.21.15.151:0/3687006482 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f351c197110 con 0x7f350406ecc0 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 msgr2=0x7f3504071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3510006010 tx=0x7f351001a040 comp rx=0 tx=0).stop 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c101d00 msgr2=0x7f351c19c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c101d00 0x7f351c19c560 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f350c00d8d0 tx=0x7f350c00dc90 comp rx=0 tx=0).stop 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 shutdown_connections 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f350406ecc0 0x7f3504071180 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f351c101d00 0x7f351c19c560 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 --2- 172.21.15.151:0/3687006482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102620 0x7f351c19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 >> 172.21.15.151:0/3687006482 conn(0x7f351c0fb550 msgr2=0x7f351c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 shutdown_connections 2024-04-16T03:07:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.628+0000 7f3522652700 1 -- 172.21.15.151:0/3687006482 wait complete. 2024-04-16T03:07:18.640 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:07:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.734+0000 7f8535342700 1 -- 172.21.15.151:0/1699937027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 msgr2=0x7f85301011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.734+0000 7f8535342700 1 --2- 172.21.15.151:0/1699937027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 0x7f85301011d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8520009910 tx=0x7f8520009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.734+0000 7f8535342700 1 -- 172.21.15.151:0/1699937027 shutdown_connections 2024-04-16T03:07:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.735+0000 7f8535342700 1 --2- 172.21.15.151:0/1699937027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85301017a0 0x7f85301057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.735+0000 7f8535342700 1 --2- 172.21.15.151:0/1699937027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 0x7f85301011d0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.735+0000 7f8535342700 1 -- 172.21.15.151:0/1699937027 >> 172.21.15.151:0/1699937027 conn(0x7f85300fc6a0 msgr2=0x7f85300feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.735+0000 7f8535342700 1 -- 172.21.15.151:0/1699937027 shutdown_connections 2024-04-16T03:07:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.735+0000 7f8535342700 1 -- 172.21.15.151:0/1699937027 wait complete. 2024-04-16T03:07:18.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.735+0000 7f8535342700 1 Processor -- start 2024-04-16T03:07:18.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.735+0000 7f8535342700 1 -- start start 2024-04-16T03:07:18.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f8535342700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 0x7f8530073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:18.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f8535342700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85301017a0 0x7f85300735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:18.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f8535342700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8530073b10 con 0x7f8530100df0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f8535342700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8530073c50 con 0x7f85301017a0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f852effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 0x7f8530073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f852effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 0x7f8530073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53924/0 (socket says 172.21.15.151:53924) 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f852effd700 1 -- 172.21.15.151:0/3560393796 learned_addr learned my addr 172.21.15.151:0/3560393796 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f852e7fc700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85301017a0 0x7f85300735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f852effd700 1 -- 172.21.15.151:0/3560393796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85301017a0 msgr2=0x7f85300735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f852effd700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85301017a0 0x7f85300735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f852effd700 1 -- 172.21.15.151:0/3560393796 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85200095f0 con 0x7f8530100df0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f852e7fc700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85301017a0 0x7f85300735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.736+0000 7f852effd700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 0x7f8530073090 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8520006010 tx=0x7f852000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.737+0000 7f8517fff700 1 -- 172.21.15.151:0/3560393796 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f852001d070 con 0x7f8530100df0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.737+0000 7f8517fff700 1 -- 172.21.15.151:0/3560393796 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f852000f460 con 0x7f8530100df0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.737+0000 7f8517fff700 1 -- 172.21.15.151:0/3560393796 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8520017630 con 0x7f8530100df0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.737+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8530073ed0 con 0x7f8530100df0 2024-04-16T03:07:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.737+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85301a2c60 con 0x7f8530100df0 2024-04-16T03:07:18.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.738+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8530109190 con 0x7f8530100df0 2024-04-16T03:07:18.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.738+0000 7f8517fff700 1 -- 172.21.15.151:0/3560393796 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f852000f5d0 con 0x7f8530100df0 2024-04-16T03:07:18.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.739+0000 7f8517fff700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851806ecc0 0x7f8518071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:18.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.739+0000 7f8517fff700 1 -- 172.21.15.151:0/3560393796 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8520090280 con 0x7f8530100df0 2024-04-16T03:07:18.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.739+0000 7f852e7fc700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851806ecc0 0x7f8518071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:18.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.739+0000 7f852e7fc700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851806ecc0 0x7f8518071180 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8524005950 tx=0x7f85240058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:18.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.743+0000 7f8517fff700 1 -- 172.21.15.151:0/3560393796 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f85200590b0 con 0x7f8530100df0 2024-04-16T03:07:18.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.963+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f853002cce0 con 0x7f851806ecc0 2024-04-16T03:07:18.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.965+0000 7f8517fff700 1 -- 172.21.15.151:0/3560393796 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f853002cce0 con 0x7f851806ecc0 2024-04-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851806ecc0 msgr2=0x7f8518071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:18.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851806ecc0 0x7f8518071180 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8524005950 tx=0x7f85240058e0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 msgr2=0x7f8530073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 0x7f8530073090 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8520006010 tx=0x7f852000be20 comp rx=0 tx=0).stop 2024-04-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 shutdown_connections 2024-04-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f851806ecc0 0x7f8518071180 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8530100df0 0x7f8530073090 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 --2- 172.21.15.151:0/3560393796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85301017a0 0x7f85300735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 >> 172.21.15.151:0/3560393796 conn(0x7f85300fc6a0 msgr2=0x7f85300fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 shutdown_connections 2024-04-16T03:07:18.969 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:18.967+0000 7f8535342700 1 -- 172.21.15.151:0/3560393796 wait complete. 2024-04-16T03:07:18.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:18 smithi151 ceph-mon[112233]: pgmap v861: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:19.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.073+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/171069253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0100dc0 msgr2=0x7f9cd01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.073+0000 7f9cd5a2d700 1 --2- 172.21.15.151:0/171069253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0100dc0 0x7f9cd01011a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009910 tx=0x7f9cc4009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:19.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.073+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/171069253 shutdown_connections 2024-04-16T03:07:19.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.073+0000 7f9cd5a2d700 1 --2- 172.21.15.151:0/171069253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd0101770 0x7f9cd01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.073+0000 7f9cd5a2d700 1 --2- 172.21.15.151:0/171069253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0100dc0 0x7f9cd01011a0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.073+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/171069253 >> 172.21.15.151:0/171069253 conn(0x7f9cd00fc650 msgr2=0x7f9cd00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:19.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.073+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/171069253 shutdown_connections 2024-04-16T03:07:19.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.073+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/171069253 wait complete. 2024-04-16T03:07:19.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.074+0000 7f9cd5a2d700 1 Processor -- start 2024-04-16T03:07:19.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.074+0000 7f9cd5a2d700 1 -- start start 2024-04-16T03:07:19.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.074+0000 7f9cd5a2d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd0100dc0 0x7f9cd0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.074+0000 7f9cd5a2d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0101770 0x7f9cd00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.074+0000 7f9cd5a2d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd0073b10 con 0x7f9cd0101770 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.074+0000 7f9cd5a2d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd0073c50 con 0x7f9cd0100dc0 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.074+0000 7f9cce7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0101770 0x7f9cd00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.074+0000 7f9cce7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0101770 0x7f9cd00735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53952/0 (socket says 172.21.15.151:53952) 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.074+0000 7f9cce7fc700 1 -- 172.21.15.151:0/1067507090 learned_addr learned my addr 172.21.15.151:0/1067507090 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cceffd700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd0100dc0 0x7f9cd0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cce7fc700 1 -- 172.21.15.151:0/1067507090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd0100dc0 msgr2=0x7f9cd0073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cce7fc700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd0100dc0 0x7f9cd0073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cce7fc700 1 -- 172.21.15.151:0/1067507090 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc40095f0 con 0x7f9cd0101770 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cce7fc700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0101770 0x7f9cd00735d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000c960 tx=0x7f9cc000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cd4a2b700 1 -- 172.21.15.151:0/1067507090 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cc0007a70 con 0x7f9cd0101770 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cd4a2b700 1 -- 172.21.15.151:0/1067507090 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cc0007bd0 con 0x7f9cd0101770 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd01a2940 con 0x7f9cd0101770 2024-04-16T03:07:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cd4a2b700 1 -- 172.21.15.151:0/1067507090 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cc0018630 con 0x7f9cd0101770 2024-04-16T03:07:19.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.075+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd01a2ca0 con 0x7f9cd0101770 2024-04-16T03:07:19.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.076+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd0109160 con 0x7f9cd0101770 2024-04-16T03:07:19.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.077+0000 7f9cd4a2b700 1 -- 172.21.15.151:0/1067507090 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9cc000f450 con 0x7f9cd0101770 2024-04-16T03:07:19.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.077+0000 7f9cd4a2b700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9cb806ecc0 0x7f9cb8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:19.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.077+0000 7f9cd4a2b700 1 -- 172.21.15.151:0/1067507090 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9cc008edf0 con 0x7f9cd0101770 2024-04-16T03:07:19.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.077+0000 7f9cceffd700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9cb806ecc0 0x7f9cb8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:19.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.078+0000 7f9cceffd700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9cb806ecc0 0x7f9cb8071180 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4006010 tx=0x7f9cc401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:19.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.082+0000 7f9cd4a2b700 1 -- 172.21.15.151:0/1067507090 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9cc0057ba0 con 0x7f9cd0101770 2024-04-16T03:07:19.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:18 smithi153 ceph-mon[87858]: pgmap v861: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:19.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.292+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cd002cd60 con 0x7f9cb806ecc0 2024-04-16T03:07:19.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.303+0000 7f9cd4a2b700 1 -- 172.21.15.151:0/1067507090 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9cd002cd60 con 0x7f9cb806ecc0 2024-04-16T03:07:19.304 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:07:19.304 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (28m) 6m ago 40m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:07:19.304 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (40m) 6m ago 40m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:07:19.304 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (39m) 6m ago 39m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (26m) 6m ago 40m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (26m) 6m ago 39m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (28m) 6m ago 39m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (22m) 6m ago 34m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (22m) 6m ago 34m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (6m) 6m ago 34m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (22m) 6m ago 34m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (30m) 6m ago 41m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (29m) 6m ago 39m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (27m) 6m ago 41m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (27m) 6m ago 39m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (29m) 6m ago 40m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (29m) 6m ago 39m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (25m) 6m ago 38m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:07:19.305 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (25m) 6m ago 37m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:07:19.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (24m) 6m ago 37m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:07:19.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (24m) 6m ago 36m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:07:19.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (23m) 6m ago 36m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:07:19.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (23m) 6m ago 36m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:07:19.306 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (29m) 6m ago 39m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9cb806ecc0 msgr2=0x7f9cb8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9cb806ecc0 0x7f9cb8071180 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4006010 tx=0x7f9cc401a040 comp rx=0 tx=0).stop 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0101770 msgr2=0x7f9cd00735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0101770 0x7f9cd00735d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000c960 tx=0x7f9cc000cc70 comp rx=0 tx=0).stop 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 shutdown_connections 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9cb806ecc0 0x7f9cb8071180 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd0100dc0 0x7f9cd0073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 --2- 172.21.15.151:0/1067507090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cd0101770 0x7f9cd00735d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 >> 172.21.15.151:0/1067507090 conn(0x7f9cd00fc650 msgr2=0x7f9cd00fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 shutdown_connections 2024-04-16T03:07:19.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.305+0000 7f9cd5a2d700 1 -- 172.21.15.151:0/1067507090 wait complete. 2024-04-16T03:07:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 -- 172.21.15.151:0/4181920473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0069020 msgr2=0x7f7aa0105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 --2- 172.21.15.151:0/4181920473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0069020 0x7f7aa0105790 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f7a90009910 tx=0x7f7a90009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 -- 172.21.15.151:0/4181920473 shutdown_connections 2024-04-16T03:07:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 --2- 172.21.15.151:0/4181920473 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0069020 0x7f7aa0105790 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 --2- 172.21.15.151:0/4181920473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aa0068670 0x7f7aa0068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 -- 172.21.15.151:0/4181920473 >> 172.21.15.151:0/4181920473 conn(0x7f7aa00755a0 msgr2=0x7f7aa00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 -- 172.21.15.151:0/4181920473 shutdown_connections 2024-04-16T03:07:19.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 -- 172.21.15.151:0/4181920473 wait complete. 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 Processor -- start 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.410+0000 7f7aa832c700 1 -- start start 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa832c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0068670 0x7f7aa00ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa832c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aa0069020 0x7f7aa01001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa832c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa0100720 con 0x7f7aa0068670 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa832c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa0100860 con 0x7f7aa0069020 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa60c8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0068670 0x7f7aa00ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa60c8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0068670 0x7f7aa00ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53972/0 (socket says 172.21.15.151:53972) 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa60c8700 1 -- 172.21.15.151:0/227490456 learned_addr learned my addr 172.21.15.151:0/227490456 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa58c7700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aa0069020 0x7f7aa01001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa60c8700 1 -- 172.21.15.151:0/227490456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aa0069020 msgr2=0x7f7aa01001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa60c8700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aa0069020 0x7f7aa01001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa60c8700 1 -- 172.21.15.151:0/227490456 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a900095f0 con 0x7f7aa0068670 2024-04-16T03:07:19.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7aa60c8700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0068670 0x7f7aa00ffca0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c00c930 tx=0x7f7a9c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:19.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.411+0000 7f7a977fe700 1 -- 172.21.15.151:0/227490456 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a9c007ad0 con 0x7f7aa0068670 2024-04-16T03:07:19.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.412+0000 7f7a977fe700 1 -- 172.21.15.151:0/227490456 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a9c00ce80 con 0x7f7aa0068670 2024-04-16T03:07:19.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.412+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa01a28d0 con 0x7f7aa0068670 2024-04-16T03:07:19.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.412+0000 7f7a977fe700 1 -- 172.21.15.151:0/227490456 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a9c018630 con 0x7f7aa0068670 2024-04-16T03:07:19.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.412+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa01a2c30 con 0x7f7aa0068670 2024-04-16T03:07:19.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.413+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa01090a0 con 0x7f7aa0068670 2024-04-16T03:07:19.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.413+0000 7f7a977fe700 1 -- 172.21.15.151:0/227490456 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7a9c007c30 con 0x7f7aa0068670 2024-04-16T03:07:19.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.414+0000 7f7a977fe700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a8c06ecc0 0x7f7a8c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:19.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.414+0000 7f7a977fe700 1 -- 172.21.15.151:0/227490456 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7a9c08eec0 con 0x7f7aa0068670 2024-04-16T03:07:19.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.414+0000 7f7aa58c7700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a8c06ecc0 0x7f7a8c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:19.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.414+0000 7f7aa58c7700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a8c06ecc0 0x7f7a8c071180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0101490 tx=0x7f7a9001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:19.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.419+0000 7f7a977fe700 1 -- 172.21.15.151:0/227490456 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7a9c057c70 con 0x7f7aa0068670 2024-04-16T03:07:19.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.701+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7aa002cc70 con 0x7f7aa0068670 2024-04-16T03:07:19.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.702+0000 7f7a977fe700 1 -- 172.21.15.151:0/227490456 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7a9c05b330 con 0x7f7aa0068670 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:07:19.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:07:19.705 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:07:19.705 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:07:19.705 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:07:19.705 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:07:19.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.704+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a8c06ecc0 msgr2=0x7f7a8c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.704+0000 7f7aa832c700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a8c06ecc0 0x7f7a8c071180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0101490 tx=0x7f7a9001a040 comp rx=0 tx=0).stop 2024-04-16T03:07:19.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.704+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0068670 msgr2=0x7f7aa00ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.704+0000 7f7aa832c700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0068670 0x7f7aa00ffca0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c00c930 tx=0x7f7a9c00cc40 comp rx=0 tx=0).stop 2024-04-16T03:07:19.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.704+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 shutdown_connections 2024-04-16T03:07:19.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.704+0000 7f7aa832c700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a8c06ecc0 0x7f7a8c071180 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.704+0000 7f7aa832c700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aa0068670 0x7f7aa00ffca0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.704+0000 7f7aa832c700 1 --2- 172.21.15.151:0/227490456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aa0069020 0x7f7aa01001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.704+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 >> 172.21.15.151:0/227490456 conn(0x7f7aa00755a0 msgr2=0x7f7aa0105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:19.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.705+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 shutdown_connections 2024-04-16T03:07:19.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.705+0000 7f7aa832c700 1 -- 172.21.15.151:0/227490456 wait complete. 2024-04-16T03:07:19.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.809+0000 7f819307b700 1 -- 172.21.15.151:0/1793704370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 msgr2=0x7f818c105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.809+0000 7f819307b700 1 --2- 172.21.15.151:0/1793704370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 0x7f818c105e30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8180009910 tx=0x7f8180009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:19.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.809+0000 7f819307b700 1 -- 172.21.15.151:0/1793704370 shutdown_connections 2024-04-16T03:07:19.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.809+0000 7f819307b700 1 --2- 172.21.15.151:0/1793704370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c0ffa50 0x7f818c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.809+0000 7f819307b700 1 --2- 172.21.15.151:0/1793704370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 0x7f818c105e30 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.809+0000 7f819307b700 1 -- 172.21.15.151:0/1793704370 >> 172.21.15.151:0/1793704370 conn(0x7f818c0fb540 msgr2=0x7f818c0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:19.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.809+0000 7f819307b700 1 -- 172.21.15.151:0/1793704370 shutdown_connections 2024-04-16T03:07:19.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.809+0000 7f819307b700 1 -- 172.21.15.151:0/1793704370 wait complete. 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f819307b700 1 Processor -- start 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f819307b700 1 -- start start 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f819307b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c0ffa50 0x7f818c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f819307b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 0x7f818c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f819307b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c19d110 con 0x7f818c105a50 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f819307b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c1965c0 con 0x7f818c0ffa50 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f818bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 0x7f818c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f818bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 0x7f818c19ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53992/0 (socket says 172.21.15.151:53992) 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f818bfff700 1 -- 172.21.15.151:0/3947106271 learned_addr learned my addr 172.21.15.151:0/3947106271 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f8190e17700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c0ffa50 0x7f818c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:19.813 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f818bfff700 1 -- 172.21.15.151:0/3947106271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c0ffa50 msgr2=0x7f818c19c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:19.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f818bfff700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c0ffa50 0x7f818c19c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:19.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.810+0000 7f818bfff700 1 -- 172.21.15.151:0/3947106271 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81800095f0 con 0x7f818c105a50 2024-04-16T03:07:19.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.811+0000 7f818bfff700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 0x7f818c19ca80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f817800d8d0 tx=0x7f817800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:19.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.811+0000 7f8189ffb700 1 -- 172.21.15.151:0/3947106271 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8178009940 con 0x7f818c105a50 2024-04-16T03:07:19.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.811+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818c1968a0 con 0x7f818c105a50 2024-04-16T03:07:19.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.811+0000 7f8189ffb700 1 -- 172.21.15.151:0/3947106271 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8178010460 con 0x7f818c105a50 2024-04-16T03:07:19.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.811+0000 7f8189ffb700 1 -- 172.21.15.151:0/3947106271 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f817800f5f0 con 0x7f818c105a50 2024-04-16T03:07:19.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.811+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818c196df0 con 0x7f818c105a50 2024-04-16T03:07:19.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.812+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f818c04ea90 con 0x7f818c105a50 2024-04-16T03:07:19.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.813+0000 7f8189ffb700 1 -- 172.21.15.151:0/3947106271 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f81780105d0 con 0x7f818c105a50 2024-04-16T03:07:19.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.813+0000 7f8189ffb700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f817c06ecc0 0x7f817c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:19.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.813+0000 7f8189ffb700 1 -- 172.21.15.151:0/3947106271 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f817808ed70 con 0x7f818c105a50 2024-04-16T03:07:19.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.813+0000 7f8190e17700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f817c06ecc0 0x7f817c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:19.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.814+0000 7f8190e17700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f817c06ecc0 0x7f817c071180 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f818000b5c0 tx=0x7f818001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:19.819 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:19.818+0000 7f8189ffb700 1 -- 172.21.15.151:0/3947106271 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f817805b6d0 con 0x7f818c105a50 2024-04-16T03:07:19.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:19 smithi151 ceph-mon[112233]: from='client.35576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:19.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:19 smithi151 ceph-mon[112233]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:19.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:19 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/227490456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:07:20.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.061+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f818c197ad0 con 0x7f818c105a50 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.061+0000 7f8189ffb700 1 -- 172.21.15.151:0/3947106271 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f8178016770 con 0x7f818c105a50 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:07:20.063 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:07:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:07:20.065 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:07:20.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f817c06ecc0 msgr2=0x7f817c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f817c06ecc0 0x7f817c071180 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f818000b5c0 tx=0x7f818001a040 comp rx=0 tx=0).stop 2024-04-16T03:07:20.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 msgr2=0x7f818c19ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 0x7f818c19ca80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f817800d8d0 tx=0x7f817800dc90 comp rx=0 tx=0).stop 2024-04-16T03:07:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 shutdown_connections 2024-04-16T03:07:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f817c06ecc0 0x7f817c071180 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c0ffa50 0x7f818c19c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 --2- 172.21.15.151:0/3947106271 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f818c105a50 0x7f818c19ca80 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 >> 172.21.15.151:0/3947106271 conn(0x7f818c0fb540 msgr2=0x7f818c103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 shutdown_connections 2024-04-16T03:07:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.063+0000 7f819307b700 1 -- 172.21.15.151:0/3947106271 wait complete. 2024-04-16T03:07:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:07:20.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.169+0000 7f55c16ec700 1 -- 172.21.15.151:0/2309114254 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 msgr2=0x7f55bc0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.169+0000 7f55c16ec700 1 --2- 172.21.15.151:0/2309114254 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 0x7f55bc0ffed0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009910 tx=0x7f55b0009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:20.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.169+0000 7f55c16ec700 1 -- 172.21.15.151:0/2309114254 shutdown_connections 2024-04-16T03:07:20.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.169+0000 7f55c16ec700 1 --2- 172.21.15.151:0/2309114254 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 0x7f55bc0ffed0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.169+0000 7f55c16ec700 1 --2- 172.21.15.151:0/2309114254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55bc105a50 0x7f55bc105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.169+0000 7f55c16ec700 1 -- 172.21.15.151:0/2309114254 >> 172.21.15.151:0/2309114254 conn(0x7f55bc0fb540 msgr2=0x7f55bc0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:20.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.169+0000 7f55c16ec700 1 -- 172.21.15.151:0/2309114254 shutdown_connections 2024-04-16T03:07:20.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.169+0000 7f55c16ec700 1 -- 172.21.15.151:0/2309114254 wait complete. 2024-04-16T03:07:20.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.170+0000 7f55c16ec700 1 Processor -- start 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.170+0000 7f55c16ec700 1 -- start start 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.170+0000 7f55c16ec700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 0x7f55bc197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.170+0000 7f55c16ec700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55bc105a50 0x7f55bc197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.170+0000 7f55c16ec700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc1981c0 con 0x7f55bc0ffa50 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.170+0000 7f55c16ec700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc198300 con 0x7f55bc105a50 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.170+0000 7f55baffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 0x7f55bc197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55baffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 0x7f55bc197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54014/0 (socket says 172.21.15.151:54014) 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55baffd700 1 -- 172.21.15.151:0/741272400 learned_addr learned my addr 172.21.15.151:0/741272400 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55ba7fc700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55bc105a50 0x7f55bc197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55baffd700 1 -- 172.21.15.151:0/741272400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55bc105a50 msgr2=0x7f55bc197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55baffd700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55bc105a50 0x7f55bc197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55baffd700 1 -- 172.21.15.151:0/741272400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b00095f0 con 0x7f55bc0ffa50 2024-04-16T03:07:20.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55baffd700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 0x7f55bc197660 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00b4b0 tx=0x7f55ac00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:20.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55a3fff700 1 -- 172.21.15.151:0/741272400 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55ac011800 con 0x7f55bc0ffa50 2024-04-16T03:07:20.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc1a06d0 con 0x7f55bc0ffa50 2024-04-16T03:07:20.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55a3fff700 1 -- 172.21.15.151:0/741272400 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55ac011e20 con 0x7f55bc0ffa50 2024-04-16T03:07:20.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55a3fff700 1 -- 172.21.15.151:0/741272400 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55ac00f2e0 con 0x7f55bc0ffa50 2024-04-16T03:07:20.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.171+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc1a0b90 con 0x7f55bc0ffa50 2024-04-16T03:07:20.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.172+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55bc108070 con 0x7f55bc0ffa50 2024-04-16T03:07:20.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.173+0000 7f55a3fff700 1 -- 172.21.15.151:0/741272400 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f55ac010380 con 0x7f55bc0ffa50 2024-04-16T03:07:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.173+0000 7f55a3fff700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55a406ecc0 0x7f55a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.173+0000 7f55a3fff700 1 -- 172.21.15.151:0/741272400 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f55ac08e9a0 con 0x7f55bc0ffa50 2024-04-16T03:07:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.173+0000 7f55ba7fc700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55a406ecc0 0x7f55a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:20.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.174+0000 7f55ba7fc700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55a406ecc0 0x7f55a4071180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f55bc0695e0 tx=0x7f55b001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.178+0000 7f55a3fff700 1 -- 172.21.15.151:0/741272400 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f55ac057750 con 0x7f55bc0ffa50 2024-04-16T03:07:20.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:19 smithi153 ceph-mon[87858]: from='client.35576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:20.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:19 smithi153 ceph-mon[87858]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:20.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:19 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/227490456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:07:20.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.403+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f55bc02cce0 con 0x7f55a406ecc0 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.404+0000 7f55a3fff700 1 -- 172.21.15.151:0/741272400 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f55bc02cce0 con 0x7f55a406ecc0 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:07:20.406 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:07:20.407 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:07:20.407 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:07:20.407 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:07:20.407 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:07:20.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55a406ecc0 msgr2=0x7f55a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55a406ecc0 0x7f55a4071180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f55bc0695e0 tx=0x7f55b001a040 comp rx=0 tx=0).stop 2024-04-16T03:07:20.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 msgr2=0x7f55bc197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 0x7f55bc197660 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00b4b0 tx=0x7f55ac00b7c0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 shutdown_connections 2024-04-16T03:07:20.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f55a406ecc0 0x7f55a4071180 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55bc0ffa50 0x7f55bc197660 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 --2- 172.21.15.151:0/741272400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55bc105a50 0x7f55bc197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 >> 172.21.15.151:0/741272400 conn(0x7f55bc0fb540 msgr2=0x7f55bc0fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:20.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 shutdown_connections 2024-04-16T03:07:20.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.407+0000 7f55c16ec700 1 -- 172.21.15.151:0/741272400 wait complete. 2024-04-16T03:07:20.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 -- 172.21.15.151:0/3672285804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 msgr2=0x7f32ac101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 --2- 172.21.15.151:0/3672285804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 0x7f32ac101180 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f329c009910 tx=0x7f329c009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:20.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 -- 172.21.15.151:0/3672285804 shutdown_connections 2024-04-16T03:07:20.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 --2- 172.21.15.151:0/3672285804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac101750 0x7f32ac1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 --2- 172.21.15.151:0/3672285804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 0x7f32ac101180 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 -- 172.21.15.151:0/3672285804 >> 172.21.15.151:0/3672285804 conn(0x7f32ac0fc650 msgr2=0x7f32ac0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:20.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 -- 172.21.15.151:0/3672285804 shutdown_connections 2024-04-16T03:07:20.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 -- 172.21.15.151:0/3672285804 wait complete. 2024-04-16T03:07:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 Processor -- start 2024-04-16T03:07:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 -- start start 2024-04-16T03:07:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.511+0000 7f32b3a15700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 0x7f32ac196750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b3a15700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac101750 0x7f32ac196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b3a15700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac197320 con 0x7f32ac100da0 2024-04-16T03:07:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b3a15700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac199fe0 con 0x7f32ac101750 2024-04-16T03:07:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b17b1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 0x7f32ac196750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b17b1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 0x7f32ac196750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54028/0 (socket says 172.21.15.151:54028) 2024-04-16T03:07:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b17b1700 1 -- 172.21.15.151:0/3414573533 learned_addr learned my addr 172.21.15.151:0/3414573533 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b0fb0700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac101750 0x7f32ac196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b17b1700 1 -- 172.21.15.151:0/3414573533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac101750 msgr2=0x7f32ac196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b17b1700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac101750 0x7f32ac196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b17b1700 1 -- 172.21.15.151:0/3414573533 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c0095f0 con 0x7f32ac100da0 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b17b1700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 0x7f32ac196750 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f329c006010 tx=0x7f329c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32a27fc700 1 -- 172.21.15.151:0/3414573533 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329c01d070 con 0x7f32ac100da0 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32a27fc700 1 -- 172.21.15.151:0/3414573533 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f329c005dc0 con 0x7f32ac100da0 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ac19a260 con 0x7f32ac100da0 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32a27fc700 1 -- 172.21.15.151:0/3414573533 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329c017400 con 0x7f32ac100da0 2024-04-16T03:07:20.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.512+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ac19a750 con 0x7f32ac100da0 2024-04-16T03:07:20.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.513+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ac109140 con 0x7f32ac100da0 2024-04-16T03:07:20.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.514+0000 7f32a27fc700 1 -- 172.21.15.151:0/3414573533 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f329c003a60 con 0x7f32ac100da0 2024-04-16T03:07:20.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.514+0000 7f32a27fc700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f329806ecc0 0x7f3298071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:20.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.514+0000 7f32a27fc700 1 -- 172.21.15.151:0/3414573533 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f329c0902f0 con 0x7f32ac100da0 2024-04-16T03:07:20.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.514+0000 7f32b0fb0700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f329806ecc0 0x7f3298071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:20.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.515+0000 7f32b0fb0700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f329806ecc0 0x7f3298071180 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f32ac197d00 tx=0x7f32a800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:20.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.519+0000 7f32a27fc700 1 -- 172.21.15.151:0/3414573533 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f329c0590a0 con 0x7f32ac100da0 2024-04-16T03:07:20.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.801+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f32ac108b20 con 0x7f32ac100da0 2024-04-16T03:07:20.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.802+0000 7f32a27fc700 1 -- 172.21.15.151:0/3414573533 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f329c05c760 con 0x7f32ac100da0 2024-04-16T03:07:20.803 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:07:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:07:20.805 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:07:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:07:20.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f329806ecc0 msgr2=0x7f3298071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f329806ecc0 0x7f3298071180 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f32ac197d00 tx=0x7f32a800b410 comp rx=0 tx=0).stop 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 msgr2=0x7f32ac196750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 0x7f32ac196750 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f329c006010 tx=0x7f329c00bd70 comp rx=0 tx=0).stop 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 shutdown_connections 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f329806ecc0 0x7f3298071180 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac100da0 0x7f32ac196750 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 --2- 172.21.15.151:0/3414573533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac101750 0x7f32ac196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 >> 172.21.15.151:0/3414573533 conn(0x7f32ac0fc650 msgr2=0x7f32ac0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 shutdown_connections 2024-04-16T03:07:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:20.804+0000 7f32b3a15700 1 -- 172.21.15.151:0/3414573533 wait complete. 2024-04-16T03:07:21.116 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:20 smithi151 ceph-mon[112233]: pgmap v862: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:21.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:20 smithi151 ceph-mon[112233]: from='client.35584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:21.117 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:20 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3947106271' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:07:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:20 smithi153 ceph-mon[87858]: pgmap v862: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:20 smithi153 ceph-mon[87858]: from='client.35584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:20 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3947106271' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:07:22.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:21 smithi153 ceph-mon[87858]: from='client.35596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:22.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:21 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3414573533' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:07:22.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:21 smithi151 ceph-mon[112233]: from='client.35596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:22.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:21 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3414573533' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:07:23.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:22 smithi151 ceph-mon[112233]: pgmap v863: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:23.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:22 smithi153 ceph-mon[87858]: pgmap v863: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:25.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:24 smithi151 ceph-mon[112233]: pgmap v864: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:25.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:07:25.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:24 smithi153 ceph-mon[87858]: pgmap v864: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:07:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:26 smithi151 ceph-mon[112233]: pgmap v865: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:27.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:26 smithi153 ceph-mon[87858]: pgmap v865: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:29.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:28 smithi151 ceph-mon[112233]: pgmap v866: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:29.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:28 smithi153 ceph-mon[87858]: pgmap v866: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:31.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:30 smithi151 ceph-mon[112233]: pgmap v867: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:31.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:30 smithi153 ceph-mon[87858]: pgmap v867: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:33.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:32 smithi151 ceph-mon[112233]: pgmap v868: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:33.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:32 smithi153 ceph-mon[87858]: pgmap v868: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:34 smithi151 ceph-mon[112233]: pgmap v869: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:35.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:34 smithi153 ceph-mon[87858]: pgmap v869: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:37.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:36 smithi151 ceph-mon[112233]: pgmap v870: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:37.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:36 smithi153 ceph-mon[87858]: pgmap v870: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:38 smithi151 ceph-mon[112233]: pgmap v871: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:07:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:38 smithi153 ceph-mon[87858]: pgmap v871: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:07:41.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:40 smithi151 ceph-mon[112233]: pgmap v872: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:40 smithi153 ceph-mon[87858]: pgmap v872: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:42 smithi151 ceph-mon[112233]: pgmap v873: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:43.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:42 smithi153 ceph-mon[87858]: pgmap v873: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:45.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:44 smithi151 ceph-mon[112233]: pgmap v874: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:45.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:44 smithi153 ceph-mon[87858]: pgmap v874: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:47.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:46 smithi151 ceph-mon[112233]: pgmap v875: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:47.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:46 smithi153 ceph-mon[87858]: pgmap v875: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:49.137 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:48 smithi153 ceph-mon[87858]: pgmap v876: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:49.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:48 smithi151 ceph-mon[112233]: pgmap v876: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:50.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/4009579036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 msgr2=0x7f2f241057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:50.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 --2- 172.21.15.151:0/4009579036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 0x7f2f241057c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009910 tx=0x7f2f14009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/4009579036 shutdown_connections 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 --2- 172.21.15.151:0/4009579036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 0x7f2f241057c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 --2- 172.21.15.151:0/4009579036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f24100dc0 0x7f2f241011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/4009579036 >> 172.21.15.151:0/4009579036 conn(0x7f2f240fc650 msgr2=0x7f2f240fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/4009579036 shutdown_connections 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/4009579036 wait complete. 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 Processor -- start 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.924+0000 7f2f2b7ea700 1 -- start start 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f2b7ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f24100dc0 0x7f2f24073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:50.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f2b7ea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 0x7f2f240735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f2b7ea700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f24073b10 con 0x7f2f24101770 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f2b7ea700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f24073c50 con 0x7f2f24100dc0 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f28d85700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 0x7f2f240735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f28d85700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 0x7f2f240735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58706/0 (socket says 172.21.15.151:58706) 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f28d85700 1 -- 172.21.15.151:0/2734918310 learned_addr learned my addr 172.21.15.151:0/2734918310 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f29586700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f24100dc0 0x7f2f24073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f28d85700 1 -- 172.21.15.151:0/2734918310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f24100dc0 msgr2=0x7f2f24073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f28d85700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f24100dc0 0x7f2f24073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f28d85700 1 -- 172.21.15.151:0/2734918310 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f140095f0 con 0x7f2f24101770 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f28d85700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 0x7f2f240735d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2f140092e0 tx=0x7f2f1400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f1a7fc700 1 -- 172.21.15.151:0/2734918310 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f1401d070 con 0x7f2f24101770 2024-04-16T03:07:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f1a7fc700 1 -- 172.21.15.151:0/2734918310 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f14005dc0 con 0x7f2f24101770 2024-04-16T03:07:50.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.925+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f24073ed0 con 0x7f2f24101770 2024-04-16T03:07:50.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.926+0000 7f2f1a7fc700 1 -- 172.21.15.151:0/2734918310 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f14017400 con 0x7f2f24101770 2024-04-16T03:07:50.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.926+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f241a2c90 con 0x7f2f24101770 2024-04-16T03:07:50.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.927+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f241091c0 con 0x7f2f24101770 2024-04-16T03:07:50.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.927+0000 7f2f1a7fc700 1 -- 172.21.15.151:0/2734918310 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2f1400f6f0 con 0x7f2f24101770 2024-04-16T03:07:50.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.928+0000 7f2f1a7fc700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f1006ecc0 0x7f2f10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:50.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.928+0000 7f2f1a7fc700 1 -- 172.21.15.151:0/2734918310 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2f14090190 con 0x7f2f24101770 2024-04-16T03:07:50.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.928+0000 7f2f29586700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f1006ecc0 0x7f2f10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:50.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.928+0000 7f2f29586700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f1006ecc0 0x7f2f10071180 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2f20005950 tx=0x7f2f2000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:50.934 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:50.933+0000 7f2f1a7fc700 1 -- 172.21.15.151:0/2734918310 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2f14058fc0 con 0x7f2f24101770 2024-04-16T03:07:51.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:50 smithi151 ceph-mon[112233]: pgmap v877: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:51.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.150+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f2402cce0 con 0x7f2f1006ecc0 2024-04-16T03:07:51.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.152+0000 7f2f1a7fc700 1 -- 172.21.15.151:0/2734918310 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2f2402cce0 con 0x7f2f1006ecc0 2024-04-16T03:07:51.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f1006ecc0 msgr2=0x7f2f10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.155 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f1006ecc0 0x7f2f10071180 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2f20005950 tx=0x7f2f2000b410 comp rx=0 tx=0).stop 2024-04-16T03:07:51.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 msgr2=0x7f2f240735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 0x7f2f240735d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2f140092e0 tx=0x7f2f1400bd70 comp rx=0 tx=0).stop 2024-04-16T03:07:51.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 shutdown_connections 2024-04-16T03:07:51.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f1006ecc0 0x7f2f10071180 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f24100dc0 0x7f2f24073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 --2- 172.21.15.151:0/2734918310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f24101770 0x7f2f240735d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 >> 172.21.15.151:0/2734918310 conn(0x7f2f240fc650 msgr2=0x7f2f240fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:51.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 shutdown_connections 2024-04-16T03:07:51.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.154+0000 7f2f2b7ea700 1 -- 172.21.15.151:0/2734918310 wait complete. 2024-04-16T03:07:51.166 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:07:51.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.259+0000 7f388c5fe700 1 -- 172.21.15.151:0/3841562061 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38840ffbd0 msgr2=0x7f3884100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.259+0000 7f388c5fe700 1 --2- 172.21.15.151:0/3841562061 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38840ffbd0 0x7f3884100050 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3874009910 tx=0x7f3874009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:51.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.259+0000 7f388c5fe700 1 -- 172.21.15.151:0/3841562061 shutdown_connections 2024-04-16T03:07:51.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.259+0000 7f388c5fe700 1 --2- 172.21.15.151:0/3841562061 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38840ffbd0 0x7f3884100050 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.259+0000 7f388c5fe700 1 --2- 172.21.15.151:0/3841562061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3884105a40 0x7f3884105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.259+0000 7f388c5fe700 1 -- 172.21.15.151:0/3841562061 >> 172.21.15.151:0/3841562061 conn(0x7f38840755a0 msgr2=0x7f38840759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:51.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.259+0000 7f388c5fe700 1 -- 172.21.15.151:0/3841562061 shutdown_connections 2024-04-16T03:07:51.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.259+0000 7f388c5fe700 1 -- 172.21.15.151:0/3841562061 wait complete. 2024-04-16T03:07:51.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f388c5fe700 1 Processor -- start 2024-04-16T03:07:51.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f388c5fe700 1 -- start start 2024-04-16T03:07:51.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f388c5fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38840ffbd0 0x7f3884197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f388c5fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3884105a40 0x7f3884197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f388c5fe700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38841981b0 con 0x7f3884105a40 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f388c5fe700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3884198320 con 0x7f38840ffbd0 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f3889b99700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3884105a40 0x7f3884197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f3889b99700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3884105a40 0x7f3884197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58718/0 (socket says 172.21.15.151:58718) 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f3889b99700 1 -- 172.21.15.151:0/3842445926 learned_addr learned my addr 172.21.15.151:0/3842445926 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.260+0000 7f388a39a700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38840ffbd0 0x7f3884197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.261+0000 7f3889b99700 1 -- 172.21.15.151:0/3842445926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38840ffbd0 msgr2=0x7f3884197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.261+0000 7f3889b99700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38840ffbd0 0x7f3884197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.261+0000 7f3889b99700 1 -- 172.21.15.151:0/3842445926 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38740095f0 con 0x7f3884105a40 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.261+0000 7f3889b99700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3884105a40 0x7f3884197ad0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3874006010 tx=0x7f387400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.261+0000 7f387b7fe700 1 -- 172.21.15.151:0/3842445926 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f387401d070 con 0x7f3884105a40 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.261+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38841a0600 con 0x7f3884105a40 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.261+0000 7f387b7fe700 1 -- 172.21.15.151:0/3842445926 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3874005dc0 con 0x7f3884105a40 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.261+0000 7f387b7fe700 1 -- 172.21.15.151:0/3842445926 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3874017400 con 0x7f3884105a40 2024-04-16T03:07:51.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.261+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38841a0a70 con 0x7f3884105a40 2024-04-16T03:07:51.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.262+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3884107fd0 con 0x7f3884105a40 2024-04-16T03:07:51.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.263+0000 7f387b7fe700 1 -- 172.21.15.151:0/3842445926 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3874003a60 con 0x7f3884105a40 2024-04-16T03:07:51.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.263+0000 7f387b7fe700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f387006ecc0 0x7f3870071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:51.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.263+0000 7f387b7fe700 1 -- 172.21.15.151:0/3842445926 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f38740902f0 con 0x7f3884105a40 2024-04-16T03:07:51.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.263+0000 7f388a39a700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f387006ecc0 0x7f3870071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:51.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.264+0000 7f388a39a700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f387006ecc0 0x7f3870071180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3880005950 tx=0x7f388000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:51.270 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.269+0000 7f387b7fe700 1 -- 172.21.15.151:0/3842445926 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f38740590a0 con 0x7f3884105a40 2024-04-16T03:07:51.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:50 smithi153 ceph-mon[87858]: pgmap v877: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:51.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.490+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f388402cce0 con 0x7f387006ecc0 2024-04-16T03:07:51.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.492+0000 7f387b7fe700 1 -- 172.21.15.151:0/3842445926 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f388402cce0 con 0x7f387006ecc0 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f387006ecc0 msgr2=0x7f3870071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f387006ecc0 0x7f3870071180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3880005950 tx=0x7f388000b410 comp rx=0 tx=0).stop 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3884105a40 msgr2=0x7f3884197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3884105a40 0x7f3884197ad0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3874006010 tx=0x7f387400bd70 comp rx=0 tx=0).stop 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 shutdown_connections 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f387006ecc0 0x7f3870071180 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38840ffbd0 0x7f3884197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 --2- 172.21.15.151:0/3842445926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3884105a40 0x7f3884197ad0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 >> 172.21.15.151:0/3842445926 conn(0x7f38840755a0 msgr2=0x7f38840fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 shutdown_connections 2024-04-16T03:07:51.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.494+0000 7f388c5fe700 1 -- 172.21.15.151:0/3842445926 wait complete. 2024-04-16T03:07:51.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 -- 172.21.15.151:0/3074395454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 msgr2=0x7f2f4c102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 --2- 172.21.15.151:0/3074395454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 0x7f2f4c102110 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009910 tx=0x7f2f3c009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 -- 172.21.15.151:0/3074395454 shutdown_connections 2024-04-16T03:07:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 --2- 172.21.15.151:0/3074395454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f4c102650 0x7f2f4c10ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 --2- 172.21.15.151:0/3074395454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 0x7f2f4c102110 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 -- 172.21.15.151:0/3074395454 >> 172.21.15.151:0/3074395454 conn(0x7f2f4c0fb560 msgr2=0x7f2f4c0fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 -- 172.21.15.151:0/3074395454 shutdown_connections 2024-04-16T03:07:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 -- 172.21.15.151:0/3074395454 wait complete. 2024-04-16T03:07:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 Processor -- start 2024-04-16T03:07:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.599+0000 7f2f52941700 1 -- start start 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f52941700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 0x7f2f4c0ff990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f52941700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f4c102650 0x7f2f4c0ffed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f52941700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c100410 con 0x7f2f4c101d30 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f52941700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c100550 con 0x7f2f4c102650 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f4bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 0x7f2f4c0ff990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f4bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 0x7f2f4c0ff990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58730/0 (socket says 172.21.15.151:58730) 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f4bfff700 1 -- 172.21.15.151:0/516546329 learned_addr learned my addr 172.21.15.151:0/516546329 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f4b7fe700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f4c102650 0x7f2f4c0ffed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f4bfff700 1 -- 172.21.15.151:0/516546329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f4c102650 msgr2=0x7f2f4c0ffed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f4bfff700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f4c102650 0x7f2f4c0ffed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f4bfff700 1 -- 172.21.15.151:0/516546329 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c0095f0 con 0x7f2f4c101d30 2024-04-16T03:07:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f4bfff700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 0x7f2f4c0ff990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c005f50 tx=0x7f2f3c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f497fa700 1 -- 172.21.15.151:0/516546329 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3c01d070 con 0x7f2f4c101d30 2024-04-16T03:07:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4c1007d0 con 0x7f2f4c101d30 2024-04-16T03:07:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.600+0000 7f2f497fa700 1 -- 172.21.15.151:0/516546329 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f3c00b610 con 0x7f2f4c101d30 2024-04-16T03:07:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.601+0000 7f2f497fa700 1 -- 172.21.15.151:0/516546329 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3c022470 con 0x7f2f4c101d30 2024-04-16T03:07:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.601+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4c1a2c80 con 0x7f2f4c101d30 2024-04-16T03:07:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.602+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4c108240 con 0x7f2f4c101d30 2024-04-16T03:07:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.602+0000 7f2f497fa700 1 -- 172.21.15.151:0/516546329 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2f3c00fca0 con 0x7f2f4c101d30 2024-04-16T03:07:51.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.603+0000 7f2f497fa700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f3406ecc0 0x7f2f34071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:51.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.603+0000 7f2f4b7fe700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f3406ecc0 0x7f2f34071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:51.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.603+0000 7f2f497fa700 1 -- 172.21.15.151:0/516546329 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2f3c0902c0 con 0x7f2f4c101d30 2024-04-16T03:07:51.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.603+0000 7f2f4b7fe700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f3406ecc0 0x7f2f34071180 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c101180 tx=0x7f2f40009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:51.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.608+0000 7f2f497fa700 1 -- 172.21.15.151:0/516546329 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2f3c059070 con 0x7f2f4c101d30 2024-04-16T03:07:51.816 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.815+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f4c02cd80 con 0x7f2f3406ecc0 2024-04-16T03:07:51.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.826+0000 7f2f497fa700 1 -- 172.21.15.151:0/516546329 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2f4c02cd80 con 0x7f2f3406ecc0 2024-04-16T03:07:51.827 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:07:51.827 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (29m) 7m ago 40m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:07:51.827 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (41m) 7m ago 41m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (39m) 7m ago 39m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (26m) 7m ago 40m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (26m) 7m ago 39m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (29m) 7m ago 40m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (23m) 7m ago 35m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (22m) 7m ago 35m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (7m) 7m ago 35m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (22m) 7m ago 35m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (31m) 7m ago 42m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (30m) 7m ago 39m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (28m) 7m ago 42m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (27m) 7m ago 39m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (30m) 7m ago 40m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (30m) 7m ago 39m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (26m) 7m ago 38m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:07:51.828 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (25m) 7m ago 38m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:07:51.829 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (25m) 7m ago 37m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:07:51.829 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (24m) 7m ago 37m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:07:51.829 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (24m) 7m ago 37m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:07:51.829 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (23m) 7m ago 36m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:07:51.829 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (29m) 7m ago 40m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:07:51.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f3406ecc0 msgr2=0x7f2f34071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f3406ecc0 0x7f2f34071180 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c101180 tx=0x7f2f40009260 comp rx=0 tx=0).stop 2024-04-16T03:07:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 msgr2=0x7f2f4c0ff990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 0x7f2f4c0ff990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c005f50 tx=0x7f2f3c00f740 comp rx=0 tx=0).stop 2024-04-16T03:07:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 shutdown_connections 2024-04-16T03:07:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f3406ecc0 0x7f2f34071180 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c101d30 0x7f2f4c0ff990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 --2- 172.21.15.151:0/516546329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f4c102650 0x7f2f4c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 >> 172.21.15.151:0/516546329 conn(0x7f2f4c0fb560 msgr2=0x7f2f4c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 shutdown_connections 2024-04-16T03:07:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.828+0000 7f2f52941700 1 -- 172.21.15.151:0/516546329 wait complete. 2024-04-16T03:07:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.933+0000 7fe4763d7700 1 -- 172.21.15.151:0/3439748351 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 msgr2=0x7fe470100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.933+0000 7fe4763d7700 1 --2- 172.21.15.151:0/3439748351 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 0x7fe470100050 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe458009910 tx=0x7fe458009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.933+0000 7fe4763d7700 1 -- 172.21.15.151:0/3439748351 shutdown_connections 2024-04-16T03:07:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.933+0000 7fe4763d7700 1 --2- 172.21.15.151:0/3439748351 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 0x7fe470100050 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.933+0000 7fe4763d7700 1 --2- 172.21.15.151:0/3439748351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe470105a40 0x7fe470105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.933+0000 7fe4763d7700 1 -- 172.21.15.151:0/3439748351 >> 172.21.15.151:0/3439748351 conn(0x7fe4700755a0 msgr2=0x7fe4700759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.933+0000 7fe4763d7700 1 -- 172.21.15.151:0/3439748351 shutdown_connections 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.933+0000 7fe4763d7700 1 -- 172.21.15.151:0/3439748351 wait complete. 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe4763d7700 1 Processor -- start 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe4763d7700 1 -- start start 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe4763d7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 0x7fe47019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe4763d7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe470105a40 0x7fe47019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe4763d7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47019d0d0 con 0x7fe4700ffbd0 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe4763d7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470196580 con 0x7fe470105a40 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe46ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 0x7fe47019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe46ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 0x7fe47019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58750/0 (socket says 172.21.15.151:58750) 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe46ffff700 1 -- 172.21.15.151:0/963607552 learned_addr learned my addr 172.21.15.151:0/963607552 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.934+0000 7fe467fff700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe470105a40 0x7fe47019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.935+0000 7fe46ffff700 1 -- 172.21.15.151:0/963607552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe470105a40 msgr2=0x7fe47019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.935+0000 7fe46ffff700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe470105a40 0x7fe47019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.935+0000 7fe46ffff700 1 -- 172.21.15.151:0/963607552 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4580095f0 con 0x7fe4700ffbd0 2024-04-16T03:07:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.935+0000 7fe46ffff700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 0x7fe47019c500 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe46000c8a0 tx=0x7fe46000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.935+0000 7fe46dffb700 1 -- 172.21.15.151:0/963607552 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4600078e0 con 0x7fe4700ffbd0 2024-04-16T03:07:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.935+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe470196860 con 0x7fe4700ffbd0 2024-04-16T03:07:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.935+0000 7fe46dffb700 1 -- 172.21.15.151:0/963607552 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe46000f450 con 0x7fe4700ffbd0 2024-04-16T03:07:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.935+0000 7fe46dffb700 1 -- 172.21.15.151:0/963607552 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe46000e5e0 con 0x7fe4700ffbd0 2024-04-16T03:07:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.935+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe470196db0 con 0x7fe4700ffbd0 2024-04-16T03:07:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.936+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe470108fe0 con 0x7fe4700ffbd0 2024-04-16T03:07:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.936+0000 7fe46dffb700 1 -- 172.21.15.151:0/963607552 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe46000f5c0 con 0x7fe4700ffbd0 2024-04-16T03:07:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.937+0000 7fe46dffb700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe45c06ecc0 0x7fe45c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.937+0000 7fe467fff700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe45c06ecc0 0x7fe45c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.937+0000 7fe46dffb700 1 -- 172.21.15.151:0/963607552 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe46008ee40 con 0x7fe4700ffbd0 2024-04-16T03:07:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.938+0000 7fe467fff700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe45c06ecc0 0x7fe45c071180 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe470104d90 tx=0x7fe45801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:51.942+0000 7fe46dffb700 1 -- 172.21.15.151:0/963607552 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe46005b720 con 0x7fe4700ffbd0 2024-04-16T03:07:52.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.219+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe470197090 con 0x7fe4700ffbd0 2024-04-16T03:07:52.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.220+0000 7fe46dffb700 1 -- 172.21.15.151:0/963607552 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe46005b2b0 con 0x7fe4700ffbd0 2024-04-16T03:07:52.221 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:07:52.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:07:52.221 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:07:52.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:07:52.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:07:52.221 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:07:52.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:07:52.222 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe45c06ecc0 msgr2=0x7fe45c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe45c06ecc0 0x7fe45c071180 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe470104d90 tx=0x7fe45801a040 comp rx=0 tx=0).stop 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 msgr2=0x7fe47019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 0x7fe47019c500 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe46000c8a0 tx=0x7fe46000cbb0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 shutdown_connections 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe45c06ecc0 0x7fe45c071180 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4700ffbd0 0x7fe47019c500 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 --2- 172.21.15.151:0/963607552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe470105a40 0x7fe47019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 >> 172.21.15.151:0/963607552 conn(0x7fe4700755a0 msgr2=0x7fe470068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:52.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 shutdown_connections 2024-04-16T03:07:52.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.222+0000 7fe4763d7700 1 -- 172.21.15.151:0/963607552 wait complete. 2024-04-16T03:07:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 -- 172.21.15.151:0/3566731946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 msgr2=0x7f7b6c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 --2- 172.21.15.151:0/3566731946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 0x7f7b6c1057c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009910 tx=0x7f7b5c009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 -- 172.21.15.151:0/3566731946 shutdown_connections 2024-04-16T03:07:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 --2- 172.21.15.151:0/3566731946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 0x7f7b6c1057c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 --2- 172.21.15.151:0/3566731946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b6c100dc0 0x7f7b6c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 -- 172.21.15.151:0/3566731946 >> 172.21.15.151:0/3566731946 conn(0x7f7b6c0fc650 msgr2=0x7f7b6c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 -- 172.21.15.151:0/3566731946 shutdown_connections 2024-04-16T03:07:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 -- 172.21.15.151:0/3566731946 wait complete. 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 Processor -- start 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.328+0000 7f7b73f9f700 1 -- start start 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b73f9f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b6c100dc0 0x7f7b6c19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b73f9f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 0x7f7b6c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b73f9f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c19d130 con 0x7f7b6c101770 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b73f9f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c1965e0 con 0x7f7b6c100dc0 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b7153a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 0x7f7b6c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b7153a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 0x7f7b6c19caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58762/0 (socket says 172.21.15.151:58762) 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b7153a700 1 -- 172.21.15.151:0/468637985 learned_addr learned my addr 172.21.15.151:0/468637985 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b71d3b700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b6c100dc0 0x7f7b6c19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b7153a700 1 -- 172.21.15.151:0/468637985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b6c100dc0 msgr2=0x7f7b6c19c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b7153a700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b6c100dc0 0x7f7b6c19c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b7153a700 1 -- 172.21.15.151:0/468637985 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c0095f0 con 0x7f7b6c101770 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b7153a700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 0x7f7b6c19caa0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0092e0 tx=0x7f7b5c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b62ffd700 1 -- 172.21.15.151:0/468637985 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5c01d070 con 0x7f7b6c101770 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b62ffd700 1 -- 172.21.15.151:0/468637985 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b5c005dc0 con 0x7f7b6c101770 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c196860 con 0x7f7b6c101770 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b62ffd700 1 -- 172.21.15.151:0/468637985 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5c017400 con 0x7f7b6c101770 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.329+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c196d50 con 0x7f7b6c101770 2024-04-16T03:07:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.330+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c1091c0 con 0x7f7b6c101770 2024-04-16T03:07:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.331+0000 7f7b62ffd700 1 -- 172.21.15.151:0/468637985 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7b5c00f460 con 0x7f7b6c101770 2024-04-16T03:07:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.332+0000 7f7b62ffd700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7b5806ed10 0x7f7b580711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.332+0000 7f7b62ffd700 1 -- 172.21.15.151:0/468637985 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7b5c090420 con 0x7f7b6c101770 2024-04-16T03:07:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.332+0000 7f7b71d3b700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7b5806ed10 0x7f7b580711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:52.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.333+0000 7f7b71d3b700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7b5806ed10 0x7f7b580711d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7b68005950 tx=0x7f7b680058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:52.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.336+0000 7f7b62ffd700 1 -- 172.21.15.151:0/468637985 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7b5c05cc60 con 0x7f7b6c101770 2024-04-16T03:07:52.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.580+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7b6c04ea90 con 0x7f7b6c101770 2024-04-16T03:07:52.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.581+0000 7f7b62ffd700 1 -- 172.21.15.151:0/468637985 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f7b5c05c7f0 con 0x7f7b6c101770 2024-04-16T03:07:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:07:52.582 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:07:52.583 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:07:52.584 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:07:52.585 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:07:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:07:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:07:52.585 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:07:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:07:52.585 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:07:52.585 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:07:52.585 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:07:52.585 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:07:52.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7b5806ed10 msgr2=0x7f7b580711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7b5806ed10 0x7f7b580711d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7b68005950 tx=0x7f7b680058e0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 msgr2=0x7f7b6c19caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 0x7f7b6c19caa0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0092e0 tx=0x7f7b5c00be20 comp rx=0 tx=0).stop 2024-04-16T03:07:52.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 shutdown_connections 2024-04-16T03:07:52.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7b5806ed10 0x7f7b580711d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b6c100dc0 0x7f7b6c19c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 --2- 172.21.15.151:0/468637985 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b6c101770 0x7f7b6c19caa0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 >> 172.21.15.151:0/468637985 conn(0x7f7b6c0fc650 msgr2=0x7f7b6c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:52.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 shutdown_connections 2024-04-16T03:07:52.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.583+0000 7f7b73f9f700 1 -- 172.21.15.151:0/468637985 wait complete. 2024-04-16T03:07:52.587 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:07:52.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3039643529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c100e00 msgr2=0x7f3b2c1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 --2- 172.21.15.151:0/3039643529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c100e00 0x7f3b2c1011e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009910 tx=0x7f3b20009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:52.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3039643529 shutdown_connections 2024-04-16T03:07:52.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 --2- 172.21.15.151:0/3039643529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c1017b0 0x7f3b2c105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 --2- 172.21.15.151:0/3039643529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c100e00 0x7f3b2c1011e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3039643529 >> 172.21.15.151:0/3039643529 conn(0x7f3b2c0fc6b0 msgr2=0x7f3b2c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:52.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3039643529 shutdown_connections 2024-04-16T03:07:52.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3039643529 wait complete. 2024-04-16T03:07:52.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 Processor -- start 2024-04-16T03:07:52.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 -- start start 2024-04-16T03:07:52.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.690+0000 7f3b32c8f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100e00 0x7f3b2c19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b32c8f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c1017b0 0x7f3b2c19caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b32c8f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c19d180 con 0x7f3b2c1017b0 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b32c8f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c196630 con 0x7f3b2c100e00 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b2bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c1017b0 0x7f3b2c19caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b2bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c1017b0 0x7f3b2c19caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58776/0 (socket says 172.21.15.151:58776) 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b2bfff700 1 -- 172.21.15.151:0/3911162282 learned_addr learned my addr 172.21.15.151:0/3911162282 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b30a2b700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100e00 0x7f3b2c19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b2bfff700 1 -- 172.21.15.151:0/3911162282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100e00 msgr2=0x7f3b2c19c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b2bfff700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100e00 0x7f3b2c19c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b2bfff700 1 -- 172.21.15.151:0/3911162282 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b200095f0 con 0x7f3b2c1017b0 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b2bfff700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c1017b0 0x7f3b2c19caf0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800e8f0 tx=0x7f3b1800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b29ffb700 1 -- 172.21.15.151:0/3911162282 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b18009790 con 0x7f3b2c1017b0 2024-04-16T03:07:52.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b29ffb700 1 -- 172.21.15.151:0/3911162282 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b1800c810 con 0x7f3b2c1017b0 2024-04-16T03:07:52.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c196910 con 0x7f3b2c1017b0 2024-04-16T03:07:52.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.691+0000 7f3b29ffb700 1 -- 172.21.15.151:0/3911162282 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b18005580 con 0x7f3b2c1017b0 2024-04-16T03:07:52.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.692+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c196e60 con 0x7f3b2c1017b0 2024-04-16T03:07:52.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.692+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c1091a0 con 0x7f3b2c1017b0 2024-04-16T03:07:52.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.693+0000 7f3b29ffb700 1 -- 172.21.15.151:0/3911162282 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3b1800c980 con 0x7f3b2c1017b0 2024-04-16T03:07:52.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.693+0000 7f3b29ffb700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b1c06ecc0 0x7f3b1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:52.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.693+0000 7f3b29ffb700 1 -- 172.21.15.151:0/3911162282 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3b18014070 con 0x7f3b2c1017b0 2024-04-16T03:07:52.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.694+0000 7f3b30a2b700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b1c06ecc0 0x7f3b1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:52.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.694+0000 7f3b30a2b700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b1c06ecc0 0x7f3b1c071180 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3b200059d0 tx=0x7f3b20005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:52.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.698+0000 7f3b29ffb700 1 -- 172.21.15.151:0/3911162282 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3b1805ba00 con 0x7f3b2c1017b0 2024-04-16T03:07:52.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.919+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b2c1970e0 con 0x7f3b1c06ecc0 2024-04-16T03:07:52.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.920+0000 7f3b29ffb700 1 -- 172.21.15.151:0/3911162282 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3b2c1970e0 con 0x7f3b1c06ecc0 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:07:52.922 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:07:52.923 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:07:52.923 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:07:52.923 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:07:52.923 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:07:52.923 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b1c06ecc0 msgr2=0x7f3b1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b1c06ecc0 0x7f3b1c071180 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3b200059d0 tx=0x7f3b20005940 comp rx=0 tx=0).stop 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c1017b0 msgr2=0x7f3b2c19caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c1017b0 0x7f3b2c19caf0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800e8f0 tx=0x7f3b1800ec00 comp rx=0 tx=0).stop 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 shutdown_connections 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3b1c06ecc0 0x7f3b1c071180 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100e00 0x7f3b2c19c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 --2- 172.21.15.151:0/3911162282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b2c1017b0 0x7f3b2c19caf0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 >> 172.21.15.151:0/3911162282 conn(0x7f3b2c0fc6b0 msgr2=0x7f3b2c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:52.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 shutdown_connections 2024-04-16T03:07:52.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:52.922+0000 7f3b32c8f700 1 -- 172.21.15.151:0/3911162282 wait complete. 2024-04-16T03:07:53.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.027+0000 7fb93b5af700 1 -- 172.21.15.151:0/1659975357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb934105a60 msgr2=0x7fb934105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:53.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.027+0000 7fb93b5af700 1 --2- 172.21.15.151:0/1659975357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb934105a60 0x7fb934105e40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb930009910 tx=0x7fb930009c20 comp rx=0 tx=0).stop 2024-04-16T03:07:53.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.027+0000 7fb93b5af700 1 -- 172.21.15.151:0/1659975357 shutdown_connections 2024-04-16T03:07:53.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.027+0000 7fb93b5af700 1 --2- 172.21.15.151:0/1659975357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9340ffa60 0x7fb9340ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:53.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.027+0000 7fb93b5af700 1 --2- 172.21.15.151:0/1659975357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb934105a60 0x7fb934105e40 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:53.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.027+0000 7fb93b5af700 1 -- 172.21.15.151:0/1659975357 >> 172.21.15.151:0/1659975357 conn(0x7fb9340fb550 msgr2=0x7fb9340fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:53.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.027+0000 7fb93b5af700 1 -- 172.21.15.151:0/1659975357 shutdown_connections 2024-04-16T03:07:53.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.027+0000 7fb93b5af700 1 -- 172.21.15.151:0/1659975357 wait complete. 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.028+0000 7fb93b5af700 1 Processor -- start 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.028+0000 7fb93b5af700 1 -- start start 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.028+0000 7fb93b5af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9340ffa60 0x7fb934197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.028+0000 7fb93b5af700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb934105a60 0x7fb934197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.028+0000 7fb93b5af700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9341981a0 con 0x7fb9340ffa60 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.028+0000 7fb93b5af700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb934198310 con 0x7fb934105a60 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.028+0000 7fb93934b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9340ffa60 0x7fb934197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.028+0000 7fb93934b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9340ffa60 0x7fb934197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58792/0 (socket says 172.21.15.151:58792) 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.028+0000 7fb93934b700 1 -- 172.21.15.151:0/274277630 learned_addr learned my addr 172.21.15.151:0/274277630 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:07:53.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb938b4a700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb934105a60 0x7fb934197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb93934b700 1 -- 172.21.15.151:0/274277630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb934105a60 msgr2=0x7fb934197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb93934b700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb934105a60 0x7fb934197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb93934b700 1 -- 172.21.15.151:0/274277630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9300095f0 con 0x7fb9340ffa60 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb93934b700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9340ffa60 0x7fb934197580 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb930005f50 tx=0x7fb93000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb9267fc700 1 -- 172.21.15.151:0/274277630 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb93001d070 con 0x7fb9340ffa60 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb9267fc700 1 -- 172.21.15.151:0/274277630 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb93000b610 con 0x7fb9340ffa60 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9341a05f0 con 0x7fb9340ffa60 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb9267fc700 1 -- 172.21.15.151:0/274277630 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb930022470 con 0x7fb9340ffa60 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.029+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9341a0a60 con 0x7fb9340ffa60 2024-04-16T03:07:53.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.030+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb934108060 con 0x7fb9340ffa60 2024-04-16T03:07:53.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.031+0000 7fb9267fc700 1 -- 172.21.15.151:0/274277630 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb930003780 con 0x7fb9340ffa60 2024-04-16T03:07:53.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.031+0000 7fb9267fc700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb92006ecc0 0x7fb920071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:07:53.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.031+0000 7fb9267fc700 1 -- 172.21.15.151:0/274277630 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb9300901f0 con 0x7fb9340ffa60 2024-04-16T03:07:53.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.031+0000 7fb938b4a700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb92006ecc0 0x7fb920071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:07:53.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.032+0000 7fb938b4a700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb92006ecc0 0x7fb920071180 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb934069640 tx=0x7fb928009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:07:53.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.036+0000 7fb9267fc700 1 -- 172.21.15.151:0/274277630 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb930058fa0 con 0x7fb9340ffa60 2024-04-16T03:07:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:52 smithi151 ceph-mon[112233]: from='client.35604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:52 smithi151 ceph-mon[112233]: pgmap v878: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:52 smithi151 ceph-mon[112233]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:52 smithi151 ceph-mon[112233]: from='client.35612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:52 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/963607552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:07:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:52 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/468637985' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:07:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:52 smithi153 ceph-mon[87858]: from='client.35604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:52 smithi153 ceph-mon[87858]: pgmap v878: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:52 smithi153 ceph-mon[87858]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:52 smithi153 ceph-mon[87858]: from='client.35612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:52 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/963607552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:07:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:52 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/468637985' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:07:53.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.316+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb93404ea90 con 0x7fb9340ffa60 2024-04-16T03:07:53.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.317+0000 7fb9267fc700 1 -- 172.21.15.151:0/274277630 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7fb93005c660 con 0x7fb9340ffa60 2024-04-16T03:07:53.319 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:07:53.319 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:07:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:07:53.319 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:07:53.319 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:07:53.319 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:07:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:07:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:07:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:07:53.321 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:07:53.321 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:07:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:07:53.321 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:07:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:07:53.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.319+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb92006ecc0 msgr2=0x7fb920071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:53.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.319+0000 7fb93b5af700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb92006ecc0 0x7fb920071180 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb934069640 tx=0x7fb928009260 comp rx=0 tx=0).stop 2024-04-16T03:07:53.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.319+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9340ffa60 msgr2=0x7fb934197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:07:53.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.319+0000 7fb93b5af700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9340ffa60 0x7fb934197580 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb930005f50 tx=0x7fb93000f690 comp rx=0 tx=0).stop 2024-04-16T03:07:53.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.320+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 shutdown_connections 2024-04-16T03:07:53.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.320+0000 7fb93b5af700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb92006ecc0 0x7fb920071180 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:53.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.320+0000 7fb93b5af700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9340ffa60 0x7fb934197580 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:53.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.320+0000 7fb93b5af700 1 --2- 172.21.15.151:0/274277630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb934105a60 0x7fb934197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:07:53.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.320+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 >> 172.21.15.151:0/274277630 conn(0x7fb9340fb550 msgr2=0x7fb9340fbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:07:53.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.320+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 shutdown_connections 2024-04-16T03:07:53.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:07:53.320+0000 7fb93b5af700 1 -- 172.21.15.151:0/274277630 wait complete. 2024-04-16T03:07:54.191 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:53 smithi153 ceph-mon[87858]: from='client.35624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:54.191 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:53 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/274277630' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:07:54.191 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:07:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:53 smithi151 ceph-mon[112233]: from='client.35624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:07:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:53 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/274277630' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:07:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:07:55.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:54 smithi151 ceph-mon[112233]: pgmap v879: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:55.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:54 smithi153 ceph-mon[87858]: pgmap v879: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:57.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:56 smithi151 ceph-mon[112233]: pgmap v880: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:57.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:56 smithi153 ceph-mon[87858]: pgmap v880: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:59.245 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:07:58 smithi153 ceph-mon[87858]: pgmap v881: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:07:59.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:07:58 smithi151 ceph-mon[112233]: pgmap v881: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:01.145 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:00 smithi151 ceph-mon[112233]: pgmap v882: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:01.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:00 smithi153 ceph-mon[87858]: pgmap v882: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:03.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:02 smithi151 ceph-mon[112233]: pgmap v883: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:03.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:02 smithi153 ceph-mon[87858]: pgmap v883: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:05.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:04 smithi151 ceph-mon[112233]: pgmap v884: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:05.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:04 smithi153 ceph-mon[87858]: pgmap v884: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:07.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:06 smithi151 ceph-mon[112233]: pgmap v885: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:07.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:06 smithi153 ceph-mon[87858]: pgmap v885: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:08 smithi151 ceph-mon[112233]: pgmap v886: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:09.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:08:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:08 smithi153 ceph-mon[87858]: pgmap v886: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:08:11.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:10 smithi151 ceph-mon[112233]: pgmap v887: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:10 smithi153 ceph-mon[87858]: pgmap v887: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:12 smithi151 ceph-mon[112233]: pgmap v888: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:12 smithi153 ceph-mon[87858]: pgmap v888: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:15.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:14 smithi151 ceph-mon[112233]: pgmap v889: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:15.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:14 smithi153 ceph-mon[87858]: pgmap v889: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:16.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:16 smithi151 ceph-mon[112233]: pgmap v890: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:17.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:16 smithi153 ceph-mon[87858]: pgmap v890: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:18 smithi151 ceph-mon[112233]: pgmap v891: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:19.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:18 smithi153 ceph-mon[87858]: pgmap v891: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:21.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:20 smithi151 ceph-mon[112233]: pgmap v892: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:20 smithi153 ceph-mon[87858]: pgmap v892: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:23.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:22 smithi151 ceph-mon[112233]: pgmap v893: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:23.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:22 smithi153 ceph-mon[87858]: pgmap v893: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:23.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.437+0000 7efeac25f700 1 -- 172.21.15.151:0/1443847301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4100dc0 msgr2=0x7efea41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:23.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.437+0000 7efeac25f700 1 --2- 172.21.15.151:0/1443847301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4100dc0 0x7efea41011a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7efea0009910 tx=0x7efea0009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:23.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.437+0000 7efeac25f700 1 -- 172.21.15.151:0/1443847301 shutdown_connections 2024-04-16T03:08:23.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.437+0000 7efeac25f700 1 --2- 172.21.15.151:0/1443847301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efea4101770 0x7efea41057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.437+0000 7efeac25f700 1 --2- 172.21.15.151:0/1443847301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4100dc0 0x7efea41011a0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.437+0000 7efeac25f700 1 -- 172.21.15.151:0/1443847301 >> 172.21.15.151:0/1443847301 conn(0x7efea40fc650 msgr2=0x7efea40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.438+0000 7efeac25f700 1 -- 172.21.15.151:0/1443847301 shutdown_connections 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.438+0000 7efeac25f700 1 -- 172.21.15.151:0/1443847301 wait complete. 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.438+0000 7efeac25f700 1 Processor -- start 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.438+0000 7efeac25f700 1 -- start start 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.438+0000 7efeac25f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efea4100dc0 0x7efea419c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.438+0000 7efeac25f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4101770 0x7efea419cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.438+0000 7efeac25f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea419d1e0 con 0x7efea4101770 2024-04-16T03:08:23.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.438+0000 7efeac25f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea4196690 con 0x7efea4100dc0 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efea97fa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4101770 0x7efea419cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efea97fa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4101770 0x7efea419cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44684/0 (socket says 172.21.15.151:44684) 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efea97fa700 1 -- 172.21.15.151:0/3352916643 learned_addr learned my addr 172.21.15.151:0/3352916643 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efea9ffb700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efea4100dc0 0x7efea419c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efea97fa700 1 -- 172.21.15.151:0/3352916643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efea4100dc0 msgr2=0x7efea419c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efea97fa700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efea4100dc0 0x7efea419c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efea97fa700 1 -- 172.21.15.151:0/3352916643 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea00095f0 con 0x7efea4101770 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efea97fa700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4101770 0x7efea419cb50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7efe9400b480 tx=0x7efe9400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efe9affd700 1 -- 172.21.15.151:0/3352916643 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe94011800 con 0x7efea4101770 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efe9affd700 1 -- 172.21.15.151:0/3352916643 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe94011e20 con 0x7efea4101770 2024-04-16T03:08:23.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efe9affd700 1 -- 172.21.15.151:0/3352916643 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe94010ea0 con 0x7efea4101770 2024-04-16T03:08:23.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea4196970 con 0x7efea4101770 2024-04-16T03:08:23.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.439+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea4196ec0 con 0x7efea4101770 2024-04-16T03:08:23.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.440+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea4109160 con 0x7efea4101770 2024-04-16T03:08:23.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.441+0000 7efe9affd700 1 -- 172.21.15.151:0/3352916643 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efe94010380 con 0x7efea4101770 2024-04-16T03:08:23.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.441+0000 7efe9affd700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe9006ecc0 0x7efe90071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:23.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.441+0000 7efe9affd700 1 -- 172.21.15.151:0/3352916643 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efe9408e890 con 0x7efea4101770 2024-04-16T03:08:23.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.441+0000 7efea9ffb700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe9006ecc0 0x7efe90071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:23.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.442+0000 7efea9ffb700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe9006ecc0 0x7efe90071180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efea0006010 tx=0x7efea001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:23.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.446+0000 7efe9affd700 1 -- 172.21.15.151:0/3352916643 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efe9405b1f0 con 0x7efea4101770 2024-04-16T03:08:23.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.662+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efea41971b0 con 0x7efe9006ecc0 2024-04-16T03:08:23.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.664+0000 7efe9affd700 1 -- 172.21.15.151:0/3352916643 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7efea41971b0 con 0x7efe9006ecc0 2024-04-16T03:08:23.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe9006ecc0 msgr2=0x7efe90071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe9006ecc0 0x7efe90071180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efea0006010 tx=0x7efea001a040 comp rx=0 tx=0).stop 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4101770 msgr2=0x7efea419cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4101770 0x7efea419cb50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7efe9400b480 tx=0x7efe9400b790 comp rx=0 tx=0).stop 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 shutdown_connections 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe9006ecc0 0x7efe90071180 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efea4100dc0 0x7efea419c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 --2- 172.21.15.151:0/3352916643 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efea4101770 0x7efea419cb50 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 >> 172.21.15.151:0/3352916643 conn(0x7efea40fc650 msgr2=0x7efea40fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 shutdown_connections 2024-04-16T03:08:23.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.666+0000 7efeac25f700 1 -- 172.21.15.151:0/3352916643 wait complete. 2024-04-16T03:08:23.678 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:08:23.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.773+0000 7f613b474700 1 -- 172.21.15.151:0/2515087501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 msgr2=0x7f61341057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:23.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.773+0000 7f613b474700 1 --2- 172.21.15.151:0/2515087501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 0x7f61341057c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6124009910 tx=0x7f6124009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:23.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.773+0000 7f613b474700 1 -- 172.21.15.151:0/2515087501 shutdown_connections 2024-04-16T03:08:23.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.773+0000 7f613b474700 1 --2- 172.21.15.151:0/2515087501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 0x7f61341057c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:23.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.773+0000 7f613b474700 1 --2- 172.21.15.151:0/2515087501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6134100dc0 0x7f61341011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:23.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.773+0000 7f613b474700 1 -- 172.21.15.151:0/2515087501 >> 172.21.15.151:0/2515087501 conn(0x7f61340fc650 msgr2=0x7f61340fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:23.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.774+0000 7f613b474700 1 -- 172.21.15.151:0/2515087501 shutdown_connections 2024-04-16T03:08:23.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.774+0000 7f613b474700 1 -- 172.21.15.151:0/2515087501 wait complete. 2024-04-16T03:08:23.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.774+0000 7f613b474700 1 Processor -- start 2024-04-16T03:08:23.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.774+0000 7f613b474700 1 -- start start 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.774+0000 7f613b474700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6134100dc0 0x7f613419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.774+0000 7f613b474700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 0x7f613419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.774+0000 7f613b474700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613419d130 con 0x7f6134101770 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.774+0000 7f613b474700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61341965e0 con 0x7f6134100dc0 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.774+0000 7f6138a0f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 0x7f613419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f6138a0f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 0x7f613419caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44704/0 (socket says 172.21.15.151:44704) 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f6138a0f700 1 -- 172.21.15.151:0/2641721452 learned_addr learned my addr 172.21.15.151:0/2641721452 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f6139210700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6134100dc0 0x7f613419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f6138a0f700 1 -- 172.21.15.151:0/2641721452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6134100dc0 msgr2=0x7f613419c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f6138a0f700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6134100dc0 0x7f613419c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:23.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f6138a0f700 1 -- 172.21.15.151:0/2641721452 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61240095f0 con 0x7f6134101770 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f6138a0f700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 0x7f613419caa0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f61240059d0 tx=0x7f612400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f612a7fc700 1 -- 172.21.15.151:0/2641721452 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f612401d070 con 0x7f6134101770 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f612a7fc700 1 -- 172.21.15.151:0/2641721452 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f612400b610 con 0x7f6134101770 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6134196860 con 0x7f6134101770 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f612a7fc700 1 -- 172.21.15.151:0/2641721452 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6124022470 con 0x7f6134101770 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.775+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6134196d50 con 0x7f6134101770 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.776+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61341091c0 con 0x7f6134101770 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.777+0000 7f612a7fc700 1 -- 172.21.15.151:0/2641721452 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6124003710 con 0x7f6134101770 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.777+0000 7f612a7fc700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f612006ecc0 0x7f6120071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.777+0000 7f612a7fc700 1 -- 172.21.15.151:0/2641721452 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6124090350 con 0x7f6134101770 2024-04-16T03:08:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.777+0000 7f6139210700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f612006ecc0 0x7f6120071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:23.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.778+0000 7f6139210700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f612006ecc0 0x7f6120071180 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6130009c60 tx=0x7f6130009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:23.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:23.782+0000 7f612a7fc700 1 -- 172.21.15.151:0/2641721452 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f612405cbd0 con 0x7f6134101770 2024-04-16T03:08:24.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.012+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6134197030 con 0x7f612006ecc0 2024-04-16T03:08:24.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.013+0000 7f612a7fc700 1 -- 172.21.15.151:0/2641721452 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6134197030 con 0x7f612006ecc0 2024-04-16T03:08:24.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f612006ecc0 msgr2=0x7f6120071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f612006ecc0 0x7f6120071180 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6130009c60 tx=0x7f6130009260 comp rx=0 tx=0).stop 2024-04-16T03:08:24.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 msgr2=0x7f613419caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 0x7f613419caa0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f61240059d0 tx=0x7f612400f690 comp rx=0 tx=0).stop 2024-04-16T03:08:24.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 shutdown_connections 2024-04-16T03:08:24.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f612006ecc0 0x7f6120071180 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6134100dc0 0x7f613419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 --2- 172.21.15.151:0/2641721452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6134101770 0x7f613419caa0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 >> 172.21.15.151:0/2641721452 conn(0x7f61340fc650 msgr2=0x7f61340fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:24.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 shutdown_connections 2024-04-16T03:08:24.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.016+0000 7f613b474700 1 -- 172.21.15.151:0/2641721452 wait complete. 2024-04-16T03:08:24.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.122+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2468265002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a58106c10 msgr2=0x7f2a58106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.122+0000 7f2a5d0cf700 1 --2- 172.21.15.151:0/2468265002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a58106c10 0x7f2a58106ff0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2a40009910 tx=0x7f2a40009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:24.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.122+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2468265002 shutdown_connections 2024-04-16T03:08:24.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.122+0000 7f2a5d0cf700 1 --2- 172.21.15.151:0/2468265002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a580ff310 0x7f2a580ff770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.122+0000 7f2a5d0cf700 1 --2- 172.21.15.151:0/2468265002 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a58106c10 0x7f2a58106ff0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.122+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2468265002 >> 172.21.15.151:0/2468265002 conn(0x7f2a58074bf0 msgr2=0x7f2a58075000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:24.125 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:08:24.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.122+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2468265002 shutdown_connections 2024-04-16T03:08:24.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.122+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2468265002 wait complete. 2024-04-16T03:08:24.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.123+0000 7f2a5d0cf700 1 Processor -- start 2024-04-16T03:08:24.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.123+0000 7f2a5d0cf700 1 -- start start 2024-04-16T03:08:24.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a5d0cf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a580ff310 0x7f2a581004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:24.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a5d0cf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a58106c10 0x7f2a58100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:24.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a5d0cf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a58100f70 con 0x7f2a580ff310 2024-04-16T03:08:24.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a5d0cf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a581010b0 con 0x7f2a58106c10 2024-04-16T03:08:24.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a56d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a580ff310 0x7f2a581004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a56d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a580ff310 0x7f2a581004f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44722/0 (socket says 172.21.15.151:44722) 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a56d9d700 1 -- 172.21.15.151:0/2123882735 learned_addr learned my addr 172.21.15.151:0/2123882735 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a5659c700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a58106c10 0x7f2a58100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a56d9d700 1 -- 172.21.15.151:0/2123882735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a58106c10 msgr2=0x7f2a58100a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a56d9d700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a58106c10 0x7f2a58100a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a56d9d700 1 -- 172.21.15.151:0/2123882735 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a400095f0 con 0x7f2a580ff310 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a56d9d700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a580ff310 0x7f2a581004f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2a40006010 tx=0x7f2a4000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a4ffff700 1 -- 172.21.15.151:0/2123882735 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a4001d070 con 0x7f2a580ff310 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a58101330 con 0x7f2a580ff310 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a4ffff700 1 -- 172.21.15.151:0/2123882735 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a4000f460 con 0x7f2a580ff310 2024-04-16T03:08:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.124+0000 7f2a4ffff700 1 -- 172.21.15.151:0/2123882735 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a40017630 con 0x7f2a580ff310 2024-04-16T03:08:24.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.125+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a581a2bf0 con 0x7f2a580ff310 2024-04-16T03:08:24.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.126+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a58109130 con 0x7f2a580ff310 2024-04-16T03:08:24.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.126+0000 7f2a4ffff700 1 -- 172.21.15.151:0/2123882735 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2a4000f5d0 con 0x7f2a580ff310 2024-04-16T03:08:24.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.126+0000 7f2a4ffff700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2a4406ecc0 0x7f2a44071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:24.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.127+0000 7f2a4ffff700 1 -- 172.21.15.151:0/2123882735 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2a40090300 con 0x7f2a580ff310 2024-04-16T03:08:24.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.127+0000 7f2a5659c700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2a4406ecc0 0x7f2a44071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:24.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.127+0000 7f2a5659c700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2a4406ecc0 0x7f2a44071180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2a48005950 tx=0x7f2a480058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:24.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.131+0000 7f2a4ffff700 1 -- 172.21.15.151:0/2123882735 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2a400590b0 con 0x7f2a580ff310 2024-04-16T03:08:24.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:08:24.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.340+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a5802cce0 con 0x7f2a4406ecc0 2024-04-16T03:08:24.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.351+0000 7f2a4ffff700 1 -- 172.21.15.151:0/2123882735 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2a5802cce0 con 0x7f2a4406ecc0 2024-04-16T03:08:24.353 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:08:24.353 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (30m) 7m ago 41m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:08:24.353 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (41m) 7m ago 41m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:08:24.353 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (40m) 7m ago 40m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:08:24.353 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (27m) 7m ago 41m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:08:24.353 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (27m) 7m ago 40m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (29m) 7m ago 40m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (23m) 7m ago 35m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (23m) 7m ago 35m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (8m) 7m ago 35m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (23m) 7m ago 35m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (31m) 7m ago 42m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (30m) 7m ago 40m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (28m) 7m ago 42m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (28m) 7m ago 40m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:08:24.354 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (30m) 7m ago 41m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:08:24.355 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (30m) 7m ago 40m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:08:24.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (27m) 7m ago 39m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:08:24.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (26m) 7m ago 38m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:08:24.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (25m) 7m ago 38m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:08:24.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (25m) 7m ago 38m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:08:24.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (24m) 7m ago 37m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:08:24.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (24m) 7m ago 37m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:08:24.355 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (30m) 7m ago 40m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:08:24.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2a4406ecc0 msgr2=0x7f2a44071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2a4406ecc0 0x7f2a44071180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2a48005950 tx=0x7f2a480058e0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a580ff310 msgr2=0x7f2a581004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a580ff310 0x7f2a581004f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2a40006010 tx=0x7f2a4000be20 comp rx=0 tx=0).stop 2024-04-16T03:08:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 shutdown_connections 2024-04-16T03:08:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2a4406ecc0 0x7f2a44071180 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a580ff310 0x7f2a581004f0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 --2- 172.21.15.151:0/2123882735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a58106c10 0x7f2a58100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 >> 172.21.15.151:0/2123882735 conn(0x7f2a58074bf0 msgr2=0x7f2a580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 shutdown_connections 2024-04-16T03:08:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.354+0000 7f2a5d0cf700 1 -- 172.21.15.151:0/2123882735 wait complete. 2024-04-16T03:08:24.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.458+0000 7f7056581700 1 -- 172.21.15.151:0/3255749559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 msgr2=0x7f70501011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.458+0000 7f7056581700 1 --2- 172.21.15.151:0/3255749559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 0x7f70501011a0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7040009910 tx=0x7f7040009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:24.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.458+0000 7f7056581700 1 -- 172.21.15.151:0/3255749559 shutdown_connections 2024-04-16T03:08:24.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.458+0000 7f7056581700 1 --2- 172.21.15.151:0/3255749559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7050101770 0x7f70501057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.458+0000 7f7056581700 1 --2- 172.21.15.151:0/3255749559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 0x7f70501011a0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.458+0000 7f7056581700 1 -- 172.21.15.151:0/3255749559 >> 172.21.15.151:0/3255749559 conn(0x7f70500fc650 msgr2=0x7f70500fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.458+0000 7f7056581700 1 -- 172.21.15.151:0/3255749559 shutdown_connections 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.458+0000 7f7056581700 1 -- 172.21.15.151:0/3255749559 wait complete. 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f7056581700 1 Processor -- start 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f7056581700 1 -- start start 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f7056581700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 0x7f705019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f7056581700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7050101770 0x7f705019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f7056581700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705019d210 con 0x7f7050100dc0 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f7056581700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70501966c0 con 0x7f7050101770 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f704ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 0x7f705019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f704ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 0x7f705019c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44752/0 (socket says 172.21.15.151:44752) 2024-04-16T03:08:24.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f704ffff700 1 -- 172.21.15.151:0/2583151811 learned_addr learned my addr 172.21.15.151:0/2583151811 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f7047fff700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7050101770 0x7f705019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f704ffff700 1 -- 172.21.15.151:0/2583151811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7050101770 msgr2=0x7f705019cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f704ffff700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7050101770 0x7f705019cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f704ffff700 1 -- 172.21.15.151:0/2583151811 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70400095f0 con 0x7f7050100dc0 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.459+0000 7f7047fff700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7050101770 0x7f705019cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.460+0000 7f704ffff700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 0x7f705019c640 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f704000b5c0 tx=0x7f704000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.460+0000 7f704dffb700 1 -- 172.21.15.151:0/2583151811 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f704001d070 con 0x7f7050100dc0 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.460+0000 7f704dffb700 1 -- 172.21.15.151:0/2583151811 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f704000f460 con 0x7f7050100dc0 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.460+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7050196940 con 0x7f7050100dc0 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.460+0000 7f704dffb700 1 -- 172.21.15.151:0/2583151811 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7040017610 con 0x7f7050100dc0 2024-04-16T03:08:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.460+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7050196e30 con 0x7f7050100dc0 2024-04-16T03:08:24.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.461+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7050109160 con 0x7f7050100dc0 2024-04-16T03:08:24.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.461+0000 7f704dffb700 1 -- 172.21.15.151:0/2583151811 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f704000f5d0 con 0x7f7050100dc0 2024-04-16T03:08:24.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.462+0000 7f704dffb700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f703c06ecc0 0x7f703c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:24.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.462+0000 7f704dffb700 1 -- 172.21.15.151:0/2583151811 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f70400903b0 con 0x7f7050100dc0 2024-04-16T03:08:24.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.462+0000 7f7047fff700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f703c06ecc0 0x7f703c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:24.466 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.462+0000 7f7047fff700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f703c06ecc0 0x7f703c071180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7050197b30 tx=0x7f7038008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:24.468 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.467+0000 7f704dffb700 1 -- 172.21.15.151:0/2583151811 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f704005cd10 con 0x7f7050100dc0 2024-04-16T03:08:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.751+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f705004ea90 con 0x7f7050100dc0 2024-04-16T03:08:24.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.752+0000 7f704dffb700 1 -- 172.21.15.151:0/2583151811 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f704005c8a0 con 0x7f7050100dc0 2024-04-16T03:08:24.753 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:08:24.753 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:08:24.754 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:08:24.755 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:08:24.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f703c06ecc0 msgr2=0x7f703c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f703c06ecc0 0x7f703c071180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7050197b30 tx=0x7f7038008040 comp rx=0 tx=0).stop 2024-04-16T03:08:24.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 msgr2=0x7f705019c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 0x7f705019c640 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f704000b5c0 tx=0x7f704000be00 comp rx=0 tx=0).stop 2024-04-16T03:08:24.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 shutdown_connections 2024-04-16T03:08:24.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f703c06ecc0 0x7f703c071180 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7050100dc0 0x7f705019c640 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 --2- 172.21.15.151:0/2583151811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7050101770 0x7f705019cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 >> 172.21.15.151:0/2583151811 conn(0x7f70500fc650 msgr2=0x7f70500fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:24.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 shutdown_connections 2024-04-16T03:08:24.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.754+0000 7f7056581700 1 -- 172.21.15.151:0/2583151811 wait complete. 2024-04-16T03:08:24.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.859+0000 7eff4d7cf700 1 -- 172.21.15.151:0/2879256185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 msgr2=0x7eff481057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.859+0000 7eff4d7cf700 1 --2- 172.21.15.151:0/2879256185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 0x7eff481057c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7eff30009910 tx=0x7eff30009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:24.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.859+0000 7eff4d7cf700 1 -- 172.21.15.151:0/2879256185 shutdown_connections 2024-04-16T03:08:24.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.859+0000 7eff4d7cf700 1 --2- 172.21.15.151:0/2879256185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 0x7eff481057c0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.859+0000 7eff4d7cf700 1 --2- 172.21.15.151:0/2879256185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff48100dc0 0x7eff481011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.859+0000 7eff4d7cf700 1 -- 172.21.15.151:0/2879256185 >> 172.21.15.151:0/2879256185 conn(0x7eff480fc650 msgr2=0x7eff480fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:24.862 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.859+0000 7eff4d7cf700 1 -- 172.21.15.151:0/2879256185 shutdown_connections 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.859+0000 7eff4d7cf700 1 -- 172.21.15.151:0/2879256185 wait complete. 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff4d7cf700 1 Processor -- start 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff4d7cf700 1 -- start start 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff4d7cf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff48100dc0 0x7eff4819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff4d7cf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 0x7eff4819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff4d7cf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4819d130 con 0x7eff48101770 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff4d7cf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff481965e0 con 0x7eff48100dc0 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff3ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 0x7eff4819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff3ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 0x7eff4819caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44762/0 (socket says 172.21.15.151:44762) 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff3ffff700 1 -- 172.21.15.151:0/176418435 learned_addr learned my addr 172.21.15.151:0/176418435 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff3ffff700 1 -- 172.21.15.151:0/176418435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff48100dc0 msgr2=0x7eff4819c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff3ffff700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff48100dc0 0x7eff4819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:24.863 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff3ffff700 1 -- 172.21.15.151:0/176418435 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff300095f0 con 0x7eff48101770 2024-04-16T03:08:24.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff3ffff700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 0x7eff4819caa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7eff300092e0 tx=0x7eff3000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:24.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff44ff9700 1 -- 172.21.15.151:0/176418435 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff3001d070 con 0x7eff48101770 2024-04-16T03:08:24.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.860+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff48196860 con 0x7eff48101770 2024-04-16T03:08:24.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.861+0000 7eff44ff9700 1 -- 172.21.15.151:0/176418435 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff30005dc0 con 0x7eff48101770 2024-04-16T03:08:24.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.861+0000 7eff44ff9700 1 -- 172.21.15.151:0/176418435 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff30017400 con 0x7eff48101770 2024-04-16T03:08:24.864 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.861+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff48196d50 con 0x7eff48101770 2024-04-16T03:08:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.861+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff481091c0 con 0x7eff48101770 2024-04-16T03:08:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.862+0000 7eff44ff9700 1 -- 172.21.15.151:0/176418435 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7eff3000f460 con 0x7eff48101770 2024-04-16T03:08:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.862+0000 7eff44ff9700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7eff3406ecc0 0x7eff34071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.863+0000 7eff44ff9700 1 -- 172.21.15.151:0/176418435 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7eff30090380 con 0x7eff48101770 2024-04-16T03:08:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.863+0000 7eff46ffd700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7eff3406ecc0 0x7eff34071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.863+0000 7eff46ffd700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7eff3406ecc0 0x7eff34071180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7eff38005fd0 tx=0x7eff38005f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:24.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:24.867+0000 7eff44ff9700 1 -- 172.21.15.151:0/176418435 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7eff3005cc60 con 0x7eff48101770 2024-04-16T03:08:25.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.110+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7eff48197080 con 0x7eff48101770 2024-04-16T03:08:25.112 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:24 smithi151 ceph-mon[112233]: pgmap v894: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:25.112 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:24 smithi151 ceph-mon[112233]: from='client.35632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:25.112 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:24 smithi151 ceph-mon[112233]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:25.112 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:24 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2583151811' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:08:25.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.110+0000 7eff44ff9700 1 -- 172.21.15.151:0/176418435 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7eff3005c7f0 con 0x7eff48101770 2024-04-16T03:08:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:08:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:08:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:08:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:08:25.113 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:08:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:08:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:08:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:08:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:08:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:08:25.115 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:08:25.116 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:08:25.116 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:08:25.116 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:08:25.116 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:08:25.116 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:08:25.116 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:08:25.116 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:08:25.116 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:08:25.116 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:08:25.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.112+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7eff3406ecc0 msgr2=0x7eff34071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.112+0000 7eff4d7cf700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7eff3406ecc0 0x7eff34071180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7eff38005fd0 tx=0x7eff38005f60 comp rx=0 tx=0).stop 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.112+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 msgr2=0x7eff4819caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.112+0000 7eff4d7cf700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 0x7eff4819caa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7eff300092e0 tx=0x7eff3000be20 comp rx=0 tx=0).stop 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.113+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 shutdown_connections 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.113+0000 7eff4d7cf700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7eff3406ecc0 0x7eff34071180 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.113+0000 7eff4d7cf700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff48100dc0 0x7eff4819c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.113+0000 7eff4d7cf700 1 --2- 172.21.15.151:0/176418435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7eff48101770 0x7eff4819caa0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.113+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 >> 172.21.15.151:0/176418435 conn(0x7eff480fc650 msgr2=0x7eff480fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.113+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 shutdown_connections 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.113+0000 7eff4d7cf700 1 -- 172.21.15.151:0/176418435 wait complete. 2024-04-16T03:08:25.118 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.220+0000 7f42a9ecb700 1 -- 172.21.15.151:0/927937481 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 msgr2=0x7f42a41057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.220+0000 7f42a9ecb700 1 --2- 172.21.15.151:0/927937481 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 0x7f42a41057e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f428c009910 tx=0x7f428c009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.220+0000 7f42a9ecb700 1 -- 172.21.15.151:0/927937481 shutdown_connections 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.220+0000 7f42a9ecb700 1 --2- 172.21.15.151:0/927937481 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 0x7f42a41057e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.220+0000 7f42a9ecb700 1 --2- 172.21.15.151:0/927937481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42a4100de0 0x7f42a41011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.220+0000 7f42a9ecb700 1 -- 172.21.15.151:0/927937481 >> 172.21.15.151:0/927937481 conn(0x7f42a40fc670 msgr2=0x7f42a40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.220+0000 7f42a9ecb700 1 -- 172.21.15.151:0/927937481 shutdown_connections 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.220+0000 7f42a9ecb700 1 -- 172.21.15.151:0/927937481 wait complete. 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f42a9ecb700 1 Processor -- start 2024-04-16T03:08:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f42a9ecb700 1 -- start start 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f42a9ecb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42a4100de0 0x7f42a4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f42a9ecb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 0x7f42a40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f42a9ecb700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4073b10 con 0x7f42a4101790 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f42a9ecb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4073c50 con 0x7f42a4100de0 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f429bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 0x7f42a40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f429bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 0x7f42a40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44780/0 (socket says 172.21.15.151:44780) 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f429bfff700 1 -- 172.21.15.151:0/1244984498 learned_addr learned my addr 172.21.15.151:0/1244984498 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.221+0000 7f42a37fe700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42a4100de0 0x7f42a4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.222+0000 7f429bfff700 1 -- 172.21.15.151:0/1244984498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42a4100de0 msgr2=0x7f42a4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.222+0000 7f429bfff700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42a4100de0 0x7f42a4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.222+0000 7f429bfff700 1 -- 172.21.15.151:0/1244984498 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c0095f0 con 0x7f42a4101790 2024-04-16T03:08:25.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.222+0000 7f429bfff700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 0x7f42a40735d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f428c00bd70 tx=0x7f428c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:25.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.222+0000 7f42a17fa700 1 -- 172.21.15.151:0/1244984498 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f428c01d070 con 0x7f42a4101790 2024-04-16T03:08:25.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.222+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a4073ed0 con 0x7f42a4101790 2024-04-16T03:08:25.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.222+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a41a2c90 con 0x7f42a4101790 2024-04-16T03:08:25.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.222+0000 7f42a17fa700 1 -- 172.21.15.151:0/1244984498 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f428c005dc0 con 0x7f42a4101790 2024-04-16T03:08:25.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.222+0000 7f42a17fa700 1 -- 172.21.15.151:0/1244984498 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f428c017400 con 0x7f42a4101790 2024-04-16T03:08:25.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.223+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a41091e0 con 0x7f42a4101790 2024-04-16T03:08:25.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.223+0000 7f42a17fa700 1 -- 172.21.15.151:0/1244984498 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f428c00f460 con 0x7f42a4101790 2024-04-16T03:08:25.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.224+0000 7f42a17fa700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f429006ecc0 0x7f4290071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:25.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.224+0000 7f42a17fa700 1 -- 172.21.15.151:0/1244984498 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f428c0901c0 con 0x7f42a4101790 2024-04-16T03:08:25.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.224+0000 7f42a37fe700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f429006ecc0 0x7f4290071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:25.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.224+0000 7f42a37fe700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f429006ecc0 0x7f4290071180 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4294005950 tx=0x7f429400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:25.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.229+0000 7f42a17fa700 1 -- 172.21.15.151:0/1244984498 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f428c058ff0 con 0x7f42a4101790 2024-04-16T03:08:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:24 smithi153 ceph-mon[87858]: pgmap v894: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:24 smithi153 ceph-mon[87858]: from='client.35632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:24 smithi153 ceph-mon[87858]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:24 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2583151811' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:08:25.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.450+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f42a402cce0 con 0x7f429006ecc0 2024-04-16T03:08:25.452 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.451+0000 7f42a17fa700 1 -- 172.21.15.151:0/1244984498 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f42a402cce0 con 0x7f429006ecc0 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:08:25.453 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:08:25.454 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:08:25.454 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:08:25.454 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:08:25.454 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:08:25.454 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:08:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f429006ecc0 msgr2=0x7f4290071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f429006ecc0 0x7f4290071180 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4294005950 tx=0x7f429400b410 comp rx=0 tx=0).stop 2024-04-16T03:08:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 msgr2=0x7f42a40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 0x7f42a40735d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f428c00bd70 tx=0x7f428c00bda0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 shutdown_connections 2024-04-16T03:08:25.457 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f429006ecc0 0x7f4290071180 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.457 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42a4100de0 0x7f42a4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.457 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 --2- 172.21.15.151:0/1244984498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f42a4101790 0x7f42a40735d0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.457 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 >> 172.21.15.151:0/1244984498 conn(0x7f42a40fc670 msgr2=0x7f42a40fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:25.457 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 shutdown_connections 2024-04-16T03:08:25.457 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.454+0000 7f42a9ecb700 1 -- 172.21.15.151:0/1244984498 wait complete. 2024-04-16T03:08:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.558+0000 7f33d5acd700 1 -- 172.21.15.151:0/481804580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 msgr2=0x7f33d01057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.558+0000 7f33d5acd700 1 --2- 172.21.15.151:0/481804580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 0x7f33d01057d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009910 tx=0x7f33c4009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.558+0000 7f33d5acd700 1 -- 172.21.15.151:0/481804580 shutdown_connections 2024-04-16T03:08:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.558+0000 7f33d5acd700 1 --2- 172.21.15.151:0/481804580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 0x7f33d01057d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.558+0000 7f33d5acd700 1 --2- 172.21.15.151:0/481804580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d0100dd0 0x7f33d01011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.558+0000 7f33d5acd700 1 -- 172.21.15.151:0/481804580 >> 172.21.15.151:0/481804580 conn(0x7f33d00fc640 msgr2=0x7f33d00fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:25.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.558+0000 7f33d5acd700 1 -- 172.21.15.151:0/481804580 shutdown_connections 2024-04-16T03:08:25.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.558+0000 7f33d5acd700 1 -- 172.21.15.151:0/481804580 wait complete. 2024-04-16T03:08:25.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33d5acd700 1 Processor -- start 2024-04-16T03:08:25.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33d5acd700 1 -- start start 2024-04-16T03:08:25.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33d5acd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d0100dd0 0x7f33d019c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:25.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33d5acd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 0x7f33d019cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:25.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33d5acd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d019d140 con 0x7f33d0101780 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33d5acd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d01965f0 con 0x7f33d0100dd0 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33ceffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 0x7f33d019cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33ceffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 0x7f33d019cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44792/0 (socket says 172.21.15.151:44792) 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33ceffd700 1 -- 172.21.15.151:0/2947014851 learned_addr learned my addr 172.21.15.151:0/2947014851 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33cf7fe700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d0100dd0 0x7f33d019c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33ceffd700 1 -- 172.21.15.151:0/2947014851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d0100dd0 msgr2=0x7f33d019c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33ceffd700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d0100dd0 0x7f33d019c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.559+0000 7f33ceffd700 1 -- 172.21.15.151:0/2947014851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c40095f0 con 0x7f33d0101780 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.560+0000 7f33ceffd700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 0x7f33d019cab0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f33c40092e0 tx=0x7f33c400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.560+0000 7f33ccff9700 1 -- 172.21.15.151:0/2947014851 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33c401d070 con 0x7f33d0101780 2024-04-16T03:08:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.560+0000 7f33ccff9700 1 -- 172.21.15.151:0/2947014851 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33c4005dc0 con 0x7f33d0101780 2024-04-16T03:08:25.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.560+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d0196870 con 0x7f33d0101780 2024-04-16T03:08:25.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.560+0000 7f33ccff9700 1 -- 172.21.15.151:0/2947014851 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33c4017400 con 0x7f33d0101780 2024-04-16T03:08:25.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.560+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d0196d60 con 0x7f33d0101780 2024-04-16T03:08:25.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.561+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d01091d0 con 0x7f33d0101780 2024-04-16T03:08:25.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.561+0000 7f33ccff9700 1 -- 172.21.15.151:0/2947014851 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f33c400f6f0 con 0x7f33d0101780 2024-04-16T03:08:25.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.562+0000 7f33ccff9700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f33b806ecc0 0x7f33b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:25.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.562+0000 7f33ccff9700 1 -- 172.21.15.151:0/2947014851 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f33c4090170 con 0x7f33d0101780 2024-04-16T03:08:25.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.562+0000 7f33cf7fe700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f33b806ecc0 0x7f33b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:25.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.562+0000 7f33cf7fe700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f33b806ecc0 0x7f33b8071180 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009550 tx=0x7f33c00091b0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:25.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.567+0000 7f33ccff9700 1 -- 172.21.15.151:0/2947014851 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f33c405cad0 con 0x7f33d0101780 2024-04-16T03:08:25.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.845+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f33d004ea90 con 0x7f33d0101780 2024-04-16T03:08:25.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.846+0000 7f33ccff9700 1 -- 172.21.15.151:0/2947014851 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f33c405c660 con 0x7f33d0101780 2024-04-16T03:08:25.847 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:08:25.847 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:08:25.847 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:08:25.847 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:08:25.848 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:08:25.849 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:08:25.849 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:08:25.849 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:08:25.849 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:08:25.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.848+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f33b806ecc0 msgr2=0x7f33b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.848+0000 7f33d5acd700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f33b806ecc0 0x7f33b8071180 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009550 tx=0x7f33c00091b0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.848+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 msgr2=0x7f33d019cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:25.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.848+0000 7f33d5acd700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 0x7f33d019cab0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f33c40092e0 tx=0x7f33c400bd70 comp rx=0 tx=0).stop 2024-04-16T03:08:25.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.848+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 shutdown_connections 2024-04-16T03:08:25.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.848+0000 7f33d5acd700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f33b806ecc0 0x7f33b8071180 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.848+0000 7f33d5acd700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d0100dd0 0x7f33d019c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.848+0000 7f33d5acd700 1 --2- 172.21.15.151:0/2947014851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33d0101780 0x7f33d019cab0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:25.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.848+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 >> 172.21.15.151:0/2947014851 conn(0x7f33d00fc640 msgr2=0x7f33d00fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:25.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.849+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 shutdown_connections 2024-04-16T03:08:25.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:25.849+0000 7f33d5acd700 1 -- 172.21.15.151:0/2947014851 wait complete. 2024-04-16T03:08:26.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:25 smithi151 ceph-mon[112233]: from='client.35640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:26.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:25 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/176418435' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:08:26.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:25 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2947014851' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:08:26.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:25 smithi153 ceph-mon[87858]: from='client.35640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:26.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:25 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/176418435' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:08:26.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:25 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2947014851' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:08:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:26 smithi151 ceph-mon[112233]: pgmap v895: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:26 smithi151 ceph-mon[112233]: from='client.35652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:27.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:26 smithi153 ceph-mon[87858]: pgmap v895: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:27.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:26 smithi153 ceph-mon[87858]: from='client.35652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:29.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:28 smithi151 ceph-mon[112233]: pgmap v896: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:29.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:28 smithi153 ceph-mon[87858]: pgmap v896: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:31.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:30 smithi151 ceph-mon[112233]: pgmap v897: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:30 smithi153 ceph-mon[87858]: pgmap v897: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:33.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:32 smithi151 ceph-mon[112233]: pgmap v898: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:33.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:32 smithi153 ceph-mon[87858]: pgmap v898: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:35.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:34 smithi151 ceph-mon[112233]: pgmap v899: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:35.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:34 smithi153 ceph-mon[87858]: pgmap v899: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:37.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:36 smithi151 ceph-mon[112233]: pgmap v900: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:37.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:36 smithi153 ceph-mon[87858]: pgmap v900: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:38 smithi151 ceph-mon[112233]: pgmap v901: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:08:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:38 smithi153 ceph-mon[87858]: pgmap v901: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:08:41.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:40 smithi151 ceph-mon[112233]: pgmap v902: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:40 smithi153 ceph-mon[87858]: pgmap v902: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:42 smithi151 ceph-mon[112233]: pgmap v903: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:43.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:42 smithi153 ceph-mon[87858]: pgmap v903: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:44 smithi153 ceph-mon[87858]: pgmap v904: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:45.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:44 smithi151 ceph-mon[112233]: pgmap v904: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:47.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:46 smithi151 ceph-mon[112233]: pgmap v905: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:47.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:46 smithi153 ceph-mon[87858]: pgmap v905: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:48 smithi151 ceph-mon[112233]: pgmap v906: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:49.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:48 smithi153 ceph-mon[87858]: pgmap v906: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:51.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:50 smithi151 ceph-mon[112233]: pgmap v907: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:51.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:50 smithi153 ceph-mon[87858]: pgmap v907: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:52 smithi151 ceph-mon[112233]: pgmap v908: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:53.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:52 smithi153 ceph-mon[87858]: pgmap v908: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:54.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:08:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:08:55.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:54 smithi151 ceph-mon[112233]: pgmap v909: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:55.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:54 smithi153 ceph-mon[87858]: pgmap v909: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:55.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.967+0000 7f14d2940700 1 -- 172.21.15.151:0/617303301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ffee0 msgr2=0x7f14cc105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:55.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.967+0000 7f14d2940700 1 --2- 172.21.15.151:0/617303301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ffee0 0x7f14cc105870 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009910 tx=0x7f14b4009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:55.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.968+0000 7f14d2940700 1 -- 172.21.15.151:0/617303301 shutdown_connections 2024-04-16T03:08:55.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.968+0000 7f14d2940700 1 --2- 172.21.15.151:0/617303301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ffee0 0x7f14cc105870 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:55.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.968+0000 7f14d2940700 1 --2- 172.21.15.151:0/617303301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14cc0ff530 0x7f14cc0ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:55.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.968+0000 7f14d2940700 1 -- 172.21.15.151:0/617303301 >> 172.21.15.151:0/617303301 conn(0x7f14cc074df0 msgr2=0x7f14cc075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:55.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.968+0000 7f14d2940700 1 -- 172.21.15.151:0/617303301 shutdown_connections 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.968+0000 7f14d2940700 1 -- 172.21.15.151:0/617303301 wait complete. 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.968+0000 7f14d2940700 1 Processor -- start 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.968+0000 7f14d2940700 1 -- start start 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14d2940700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ff530 0x7f14cc1004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14d2940700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14cc0ffee0 0x7f14cc1009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14d2940700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc100f30 con 0x7f14cc0ff530 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14d2940700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc101070 con 0x7f14cc0ffee0 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14cbfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ff530 0x7f14cc1004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14cbfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ff530 0x7f14cc1004b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41220/0 (socket says 172.21.15.151:41220) 2024-04-16T03:08:55.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14cbfff700 1 -- 172.21.15.151:0/189213405 learned_addr learned my addr 172.21.15.151:0/189213405 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14cb7fe700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14cc0ffee0 0x7f14cc1009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14cbfff700 1 -- 172.21.15.151:0/189213405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14cc0ffee0 msgr2=0x7f14cc1009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14cbfff700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14cc0ffee0 0x7f14cc1009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14cbfff700 1 -- 172.21.15.151:0/189213405 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b40095f0 con 0x7f14cc0ff530 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.969+0000 7f14cbfff700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ff530 0x7f14cc1004b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00e920 tx=0x7f14bc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.970+0000 7f14c97fa700 1 -- 172.21.15.151:0/189213405 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14bc00c9d0 con 0x7f14cc0ff530 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.970+0000 7f14c97fa700 1 -- 172.21.15.151:0/189213405 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14bc010430 con 0x7f14cc0ff530 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.970+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14cc1a29a0 con 0x7f14cc0ff530 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.970+0000 7f14c97fa700 1 -- 172.21.15.151:0/189213405 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14bc018630 con 0x7f14cc0ff530 2024-04-16T03:08:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.970+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14cc1a2d50 con 0x7f14cc0ff530 2024-04-16T03:08:55.975 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.971+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14cc109150 con 0x7f14cc0ff530 2024-04-16T03:08:55.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.971+0000 7f14c97fa700 1 -- 172.21.15.151:0/189213405 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f14bc0105a0 con 0x7f14cc0ff530 2024-04-16T03:08:55.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.972+0000 7f14c97fa700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14b806ecc0 0x7f14b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:55.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.972+0000 7f14c97fa700 1 -- 172.21.15.151:0/189213405 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f14bc014070 con 0x7f14cc0ff530 2024-04-16T03:08:55.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.972+0000 7f14cb7fe700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14b806ecc0 0x7f14b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:55.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.972+0000 7f14cb7fe700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14b806ecc0 0x7f14b8071180 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f14b40059d0 tx=0x7f14b4005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:55.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:55.977+0000 7f14c97fa700 1 -- 172.21.15.151:0/189213405 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f14bc053ff0 con 0x7f14cc0ff530 2024-04-16T03:08:56.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.197+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f14cc02cd60 con 0x7f14b806ecc0 2024-04-16T03:08:56.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.198+0000 7f14c97fa700 1 -- 172.21.15.151:0/189213405 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f14cc02cd60 con 0x7f14b806ecc0 2024-04-16T03:08:56.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.200+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14b806ecc0 msgr2=0x7f14b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.200+0000 7f14d2940700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14b806ecc0 0x7f14b8071180 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f14b40059d0 tx=0x7f14b4005940 comp rx=0 tx=0).stop 2024-04-16T03:08:56.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.200+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ff530 msgr2=0x7f14cc1004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.200+0000 7f14d2940700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ff530 0x7f14cc1004b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00e920 tx=0x7f14bc00ec30 comp rx=0 tx=0).stop 2024-04-16T03:08:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.201+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 shutdown_connections 2024-04-16T03:08:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.201+0000 7f14d2940700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f14b806ecc0 0x7f14b8071180 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.201+0000 7f14d2940700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14cc0ff530 0x7f14cc1004b0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.201+0000 7f14d2940700 1 --2- 172.21.15.151:0/189213405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14cc0ffee0 0x7f14cc1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.201+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 >> 172.21.15.151:0/189213405 conn(0x7f14cc074df0 msgr2=0x7f14cc1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.201+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 shutdown_connections 2024-04-16T03:08:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.201+0000 7f14d2940700 1 -- 172.21.15.151:0/189213405 wait complete. 2024-04-16T03:08:56.212 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:08:56.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.307+0000 7f5d42b21700 1 -- 172.21.15.151:0/621949773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 msgr2=0x7f5d3c0ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.307+0000 7f5d42b21700 1 --2- 172.21.15.151:0/621949773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 0x7f5d3c0ff910 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009910 tx=0x7f5d30009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:56.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.307+0000 7f5d42b21700 1 -- 172.21.15.151:0/621949773 shutdown_connections 2024-04-16T03:08:56.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.307+0000 7f5d42b21700 1 --2- 172.21.15.151:0/621949773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d3c0ffee0 0x7f5d3c105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.307+0000 7f5d42b21700 1 --2- 172.21.15.151:0/621949773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 0x7f5d3c0ff910 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.307+0000 7f5d42b21700 1 -- 172.21.15.151:0/621949773 >> 172.21.15.151:0/621949773 conn(0x7f5d3c074d50 msgr2=0x7f5d3c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:56.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.307+0000 7f5d42b21700 1 -- 172.21.15.151:0/621949773 shutdown_connections 2024-04-16T03:08:56.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.307+0000 7f5d42b21700 1 -- 172.21.15.151:0/621949773 wait complete. 2024-04-16T03:08:56.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d42b21700 1 Processor -- start 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d42b21700 1 -- start start 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d42b21700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 0x7f5d3c19c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d42b21700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d3c0ffee0 0x7f5d3c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d42b21700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c19d190 con 0x7f5d3c0ff530 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d42b21700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c196640 con 0x7f5d3c0ffee0 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d408bd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 0x7f5d3c19c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d408bd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 0x7f5d3c19c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41242/0 (socket says 172.21.15.151:41242) 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d408bd700 1 -- 172.21.15.151:0/705291938 learned_addr learned my addr 172.21.15.151:0/705291938 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.308+0000 7f5d3bfff700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d3c0ffee0 0x7f5d3c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.309+0000 7f5d408bd700 1 -- 172.21.15.151:0/705291938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d3c0ffee0 msgr2=0x7f5d3c19cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.309+0000 7f5d408bd700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d3c0ffee0 0x7f5d3c19cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.309+0000 7f5d408bd700 1 -- 172.21.15.151:0/705291938 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d300095f0 con 0x7f5d3c0ff530 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.309+0000 7f5d408bd700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 0x7f5d3c19c5c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000b5c0 tx=0x7f5d3000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:56.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.309+0000 7f5d39ffb700 1 -- 172.21.15.151:0/705291938 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d3001d070 con 0x7f5d3c0ff530 2024-04-16T03:08:56.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.309+0000 7f5d39ffb700 1 -- 172.21.15.151:0/705291938 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d30005e00 con 0x7f5d3c0ff530 2024-04-16T03:08:56.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.309+0000 7f5d39ffb700 1 -- 172.21.15.151:0/705291938 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d30017400 con 0x7f5d3c0ff530 2024-04-16T03:08:56.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.309+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c1968c0 con 0x7f5d3c0ff530 2024-04-16T03:08:56.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.309+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c196db0 con 0x7f5d3c0ff530 2024-04-16T03:08:56.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.310+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3c109180 con 0x7f5d3c0ff530 2024-04-16T03:08:56.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.311+0000 7f5d39ffb700 1 -- 172.21.15.151:0/705291938 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5d3000f6f0 con 0x7f5d3c0ff530 2024-04-16T03:08:56.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.311+0000 7f5d39ffb700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d2c06ecc0 0x7f5d2c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:56.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.311+0000 7f5d39ffb700 1 -- 172.21.15.151:0/705291938 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5d300902c0 con 0x7f5d3c0ff530 2024-04-16T03:08:56.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.311+0000 7f5d3bfff700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d2c06ecc0 0x7f5d2c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:56.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.312+0000 7f5d3bfff700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d2c06ecc0 0x7f5d2c071180 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c1077d0 tx=0x7f5d2800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:56.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.316+0000 7f5d39ffb700 1 -- 172.21.15.151:0/705291938 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5d3005cba0 con 0x7f5d3c0ff530 2024-04-16T03:08:56.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.536+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d3c02ccc0 con 0x7f5d2c06ecc0 2024-04-16T03:08:56.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.538+0000 7f5d39ffb700 1 -- 172.21.15.151:0/705291938 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5d3c02ccc0 con 0x7f5d2c06ecc0 2024-04-16T03:08:56.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d2c06ecc0 msgr2=0x7f5d2c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d2c06ecc0 0x7f5d2c071180 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c1077d0 tx=0x7f5d2800b410 comp rx=0 tx=0).stop 2024-04-16T03:08:56.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 msgr2=0x7f5d3c19c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 0x7f5d3c19c5c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000b5c0 tx=0x7f5d3000bd70 comp rx=0 tx=0).stop 2024-04-16T03:08:56.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 shutdown_connections 2024-04-16T03:08:56.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5d2c06ecc0 0x7f5d2c071180 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d3c0ff530 0x7f5d3c19c5c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 --2- 172.21.15.151:0/705291938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d3c0ffee0 0x7f5d3c19cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 >> 172.21.15.151:0/705291938 conn(0x7f5d3c074d50 msgr2=0x7f5d3c0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:56.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 shutdown_connections 2024-04-16T03:08:56.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.540+0000 7f5d42b21700 1 -- 172.21.15.151:0/705291938 wait complete. 2024-04-16T03:08:56.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.644+0000 7feba4f92700 1 -- 172.21.15.151:0/4138258373 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 msgr2=0x7feba01057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.644+0000 7feba4f92700 1 --2- 172.21.15.151:0/4138258373 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 0x7feba01057e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7feb90009910 tx=0x7feb90009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:56.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.644+0000 7feba4f92700 1 -- 172.21.15.151:0/4138258373 shutdown_connections 2024-04-16T03:08:56.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.644+0000 7feba4f92700 1 --2- 172.21.15.151:0/4138258373 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 0x7feba01057e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.644+0000 7feba4f92700 1 --2- 172.21.15.151:0/4138258373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba0100de0 0x7feba01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.644+0000 7feba4f92700 1 -- 172.21.15.151:0/4138258373 >> 172.21.15.151:0/4138258373 conn(0x7feba00fc670 msgr2=0x7feba00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:56.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.644+0000 7feba4f92700 1 -- 172.21.15.151:0/4138258373 shutdown_connections 2024-04-16T03:08:56.646 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.644+0000 7feba4f92700 1 -- 172.21.15.151:0/4138258373 wait complete. 2024-04-16T03:08:56.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.645+0000 7feba4f92700 1 Processor -- start 2024-04-16T03:08:56.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.645+0000 7feba4f92700 1 -- start start 2024-04-16T03:08:56.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.645+0000 7feba4f92700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba0100de0 0x7feba019c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:56.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.645+0000 7feba4f92700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 0x7feba019cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:56.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.645+0000 7feba4f92700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba019d1f0 con 0x7feba0101790 2024-04-16T03:08:56.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.645+0000 7feba4f92700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba01966a0 con 0x7feba0100de0 2024-04-16T03:08:56.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb9dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 0x7feba019cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb9dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 0x7feba019cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44744/0 (socket says 172.21.15.151:44744) 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb9dd9b700 1 -- 172.21.15.151:0/2078166618 learned_addr learned my addr 172.21.15.151:0/2078166618 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb9e59c700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba0100de0 0x7feba019c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb9dd9b700 1 -- 172.21.15.151:0/2078166618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba0100de0 msgr2=0x7feba019c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb9dd9b700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba0100de0 0x7feba019c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb9dd9b700 1 -- 172.21.15.151:0/2078166618 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb900095f0 con 0x7feba0101790 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb9dd9b700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 0x7feba019cb60 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7feb900092e0 tx=0x7feb9000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb8f7fe700 1 -- 172.21.15.151:0/2078166618 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb9001d070 con 0x7feba0101790 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba0196920 con 0x7feba0101790 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb8f7fe700 1 -- 172.21.15.151:0/2078166618 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb9000f460 con 0x7feba0101790 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feb8f7fe700 1 -- 172.21.15.151:0/2078166618 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb900175f0 con 0x7feba0101790 2024-04-16T03:08:56.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.646+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba0196e10 con 0x7feba0101790 2024-04-16T03:08:56.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.647+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba0109130 con 0x7feba0101790 2024-04-16T03:08:56.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.648+0000 7feb8f7fe700 1 -- 172.21.15.151:0/2078166618 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7feb9000f5d0 con 0x7feba0101790 2024-04-16T03:08:56.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.648+0000 7feb8f7fe700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7feb8806ecc0 0x7feb88071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:56.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.648+0000 7feb8f7fe700 1 -- 172.21.15.151:0/2078166618 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7feb90090390 con 0x7feba0101790 2024-04-16T03:08:56.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.649+0000 7feb9e59c700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7feb8806ecc0 0x7feb88071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:56.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.649+0000 7feb9e59c700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7feb8806ecc0 0x7feb88071180 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7feb94006fd0 tx=0x7feb94008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:56.654 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.653+0000 7feb8f7fe700 1 -- 172.21.15.151:0/2078166618 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7feb9005cc70 con 0x7feba0101790 2024-04-16T03:08:56.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.858+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feba01970f0 con 0x7feb8806ecc0 2024-04-16T03:08:56.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.868+0000 7feb8f7fe700 1 -- 172.21.15.151:0/2078166618 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7feba01970f0 con 0x7feb8806ecc0 2024-04-16T03:08:56.870 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:08:56.870 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (30m) 8m ago 41m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:08:56.870 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (42m) 8m ago 42m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:08:56.870 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (41m) 8m ago 41m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:08:56.870 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (28m) 8m ago 42m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (27m) 8m ago 40m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (30m) 8m ago 41m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (24m) 8m ago 36m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (24m) 8m ago 36m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (8m) 8m ago 36m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (23m) 8m ago 36m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (32m) 8m ago 43m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (31m) 8m ago 40m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (29m) 8m ago 43m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (28m) 8m ago 40m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (31m) 8m ago 41m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (31m) 8m ago 40m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (27m) 8m ago 39m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:08:56.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (27m) 8m ago 39m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:08:56.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (26m) 8m ago 39m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:08:56.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (25m) 8m ago 38m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:08:56.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (25m) 8m ago 38m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:08:56.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (24m) 8m ago 37m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:08:56.872 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (30m) 8m ago 41m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7feb8806ecc0 msgr2=0x7feb88071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7feb8806ecc0 0x7feb88071180 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7feb94006fd0 tx=0x7feb94008040 comp rx=0 tx=0).stop 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 msgr2=0x7feba019cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 0x7feba019cb60 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7feb900092e0 tx=0x7feb9000be20 comp rx=0 tx=0).stop 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 shutdown_connections 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7feb8806ecc0 0x7feb88071180 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba0100de0 0x7feba019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 --2- 172.21.15.151:0/2078166618 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feba0101790 0x7feba019cb60 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 >> 172.21.15.151:0/2078166618 conn(0x7feba00fc670 msgr2=0x7feba00fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:56.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 shutdown_connections 2024-04-16T03:08:56.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.871+0000 7feba4f92700 1 -- 172.21.15.151:0/2078166618 wait complete. 2024-04-16T03:08:56.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.974+0000 7f685f2ad700 1 -- 172.21.15.151:0/3491555376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6858106c10 msgr2=0x7f6858106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.974+0000 7f685f2ad700 1 --2- 172.21.15.151:0/3491555376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6858106c10 0x7f6858106ff0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6854009910 tx=0x7f6854009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.974+0000 7f685f2ad700 1 -- 172.21.15.151:0/3491555376 shutdown_connections 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.974+0000 7f685f2ad700 1 --2- 172.21.15.151:0/3491555376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68580ff310 0x7f68580ff770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.974+0000 7f685f2ad700 1 --2- 172.21.15.151:0/3491555376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6858106c10 0x7f6858106ff0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.974+0000 7f685f2ad700 1 -- 172.21.15.151:0/3491555376 >> 172.21.15.151:0/3491555376 conn(0x7f6858074bf0 msgr2=0x7f6858075000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.974+0000 7f685f2ad700 1 -- 172.21.15.151:0/3491555376 shutdown_connections 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.974+0000 7f685f2ad700 1 -- 172.21.15.151:0/3491555376 wait complete. 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.975+0000 7f685f2ad700 1 Processor -- start 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.975+0000 7f685f2ad700 1 -- start start 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.975+0000 7f685f2ad700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68580ff310 0x7f685819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:56.981 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.975+0000 7f685f2ad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6858106c10 0x7f685819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.975+0000 7f685f2ad700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685819d0e0 con 0x7f68580ff310 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.975+0000 7f685f2ad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6858196590 con 0x7f6858106c10 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.975+0000 7f685d049700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68580ff310 0x7f685819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.975+0000 7f685d049700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68580ff310 0x7f685819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44770/0 (socket says 172.21.15.151:44770) 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.975+0000 7f685d049700 1 -- 172.21.15.151:0/3767964579 learned_addr learned my addr 172.21.15.151:0/3767964579 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f685c848700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6858106c10 0x7f685819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f685d049700 1 -- 172.21.15.151:0/3767964579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6858106c10 msgr2=0x7f685819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f685d049700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6858106c10 0x7f685819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f685d049700 1 -- 172.21.15.151:0/3767964579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68540095f0 con 0x7f68580ff310 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f685c848700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6858106c10 0x7f685819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f685d049700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68580ff310 0x7f685819c510 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f685400b5c0 tx=0x7f685400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f684e7fc700 1 -- 172.21.15.151:0/3767964579 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f685401d070 con 0x7f68580ff310 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f684e7fc700 1 -- 172.21.15.151:0/3767964579 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f685400f460 con 0x7f68580ff310 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6858196810 con 0x7f68580ff310 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f684e7fc700 1 -- 172.21.15.151:0/3767964579 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6854017610 con 0x7f68580ff310 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.976+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6858196d00 con 0x7f68580ff310 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.977+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6858109130 con 0x7f68580ff310 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.977+0000 7f684e7fc700 1 -- 172.21.15.151:0/3767964579 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f685400f5d0 con 0x7f68580ff310 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.978+0000 7f684e7fc700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f684406ecc0 0x7f6844071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.978+0000 7f684e7fc700 1 -- 172.21.15.151:0/3767964579 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6854090470 con 0x7f68580ff310 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.978+0000 7f685c848700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f684406ecc0 0x7f6844071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.978+0000 7f685c848700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f684406ecc0 0x7f6844071180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6848005fd0 tx=0x7f6848005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:56.982+0000 7f684e7fc700 1 -- 172.21.15.151:0/3767964579 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f685405cd50 con 0x7f68580ff310 2024-04-16T03:08:57.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:56 smithi151 ceph-mon[112233]: pgmap v910: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:57.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:56 smithi153 ceph-mon[87858]: pgmap v910: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:57.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.277+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f685804ea90 con 0x7f68580ff310 2024-04-16T03:08:57.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.278+0000 7f684e7fc700 1 -- 172.21.15.151:0/3767964579 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f685405c8e0 con 0x7f68580ff310 2024-04-16T03:08:57.279 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:08:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:08:57.281 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:08:57.281 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f684406ecc0 msgr2=0x7f6844071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f684406ecc0 0x7f6844071180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6848005fd0 tx=0x7f6848005dc0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68580ff310 msgr2=0x7f685819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68580ff310 0x7f685819c510 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f685400b5c0 tx=0x7f685400be00 comp rx=0 tx=0).stop 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 shutdown_connections 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f684406ecc0 0x7f6844071180 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68580ff310 0x7f685819c510 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 --2- 172.21.15.151:0/3767964579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6858106c10 0x7f685819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 >> 172.21.15.151:0/3767964579 conn(0x7f6858074bf0 msgr2=0x7f68580fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 shutdown_connections 2024-04-16T03:08:57.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.280+0000 7f685f2ad700 1 -- 172.21.15.151:0/3767964579 wait complete. 2024-04-16T03:08:57.387 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.386+0000 7f3456e26700 1 -- 172.21.15.151:0/1333404140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 msgr2=0x7f3450101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.386+0000 7f3456e26700 1 --2- 172.21.15.151:0/1333404140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 0x7f3450101180 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3444009910 tx=0x7f3444009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:57.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.386+0000 7f3456e26700 1 -- 172.21.15.151:0/1333404140 shutdown_connections 2024-04-16T03:08:57.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.386+0000 7f3456e26700 1 --2- 172.21.15.151:0/1333404140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3450101750 0x7f34501057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.386+0000 7f3456e26700 1 --2- 172.21.15.151:0/1333404140 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 0x7f3450101180 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.386+0000 7f3456e26700 1 -- 172.21.15.151:0/1333404140 >> 172.21.15.151:0/1333404140 conn(0x7f34500fc630 msgr2=0x7f34500fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:57.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.386+0000 7f3456e26700 1 -- 172.21.15.151:0/1333404140 shutdown_connections 2024-04-16T03:08:57.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.386+0000 7f3456e26700 1 -- 172.21.15.151:0/1333404140 wait complete. 2024-04-16T03:08:57.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3456e26700 1 Processor -- start 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3456e26700 1 -- start start 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3456e26700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 0x7f345019c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3456e26700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3450101750 0x7f345019cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3456e26700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345019d170 con 0x7f3450100da0 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3456e26700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3450196620 con 0x7f3450101750 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3454bc2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 0x7f345019c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3454bc2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 0x7f345019c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44786/0 (socket says 172.21.15.151:44786) 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3454bc2700 1 -- 172.21.15.151:0/4290719591 learned_addr learned my addr 172.21.15.151:0/4290719591 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f344ffff700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3450101750 0x7f345019cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3454bc2700 1 -- 172.21.15.151:0/4290719591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3450101750 msgr2=0x7f345019cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3454bc2700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3450101750 0x7f345019cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3454bc2700 1 -- 172.21.15.151:0/4290719591 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34440095f0 con 0x7f3450100da0 2024-04-16T03:08:57.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f344ffff700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3450101750 0x7f345019cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:08:57.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.387+0000 7f3454bc2700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 0x7f345019c5a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3444006010 tx=0x7f344400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:57.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.388+0000 7f344dffb700 1 -- 172.21.15.151:0/4290719591 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f344401d070 con 0x7f3450100da0 2024-04-16T03:08:57.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.388+0000 7f344dffb700 1 -- 172.21.15.151:0/4290719591 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f344400f460 con 0x7f3450100da0 2024-04-16T03:08:57.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.388+0000 7f344dffb700 1 -- 172.21.15.151:0/4290719591 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3444017630 con 0x7f3450100da0 2024-04-16T03:08:57.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.388+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34501968a0 con 0x7f3450100da0 2024-04-16T03:08:57.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.388+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3450196d90 con 0x7f3450100da0 2024-04-16T03:08:57.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.389+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3450109140 con 0x7f3450100da0 2024-04-16T03:08:57.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.390+0000 7f344dffb700 1 -- 172.21.15.151:0/4290719591 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f344400f5d0 con 0x7f3450100da0 2024-04-16T03:08:57.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.390+0000 7f344dffb700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f344006ecc0 0x7f3440071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:57.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.390+0000 7f344dffb700 1 -- 172.21.15.151:0/4290719591 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3444090290 con 0x7f3450100da0 2024-04-16T03:08:57.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.390+0000 7f344ffff700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f344006ecc0 0x7f3440071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:57.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.390+0000 7f344ffff700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f344006ecc0 0x7f3440071180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f343c005fd0 tx=0x7f343c005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:57.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.395+0000 7f344dffb700 1 -- 172.21.15.151:0/4290719591 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f344405cbf0 con 0x7f3450100da0 2024-04-16T03:08:57.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.636+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f345004ea90 con 0x7f3450100da0 2024-04-16T03:08:57.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.637+0000 7f344dffb700 1 -- 172.21.15.151:0/4290719591 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f344405c780 con 0x7f3450100da0 2024-04-16T03:08:57.639 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:08:57.639 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:08:57.639 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:08:57.639 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:08:57.639 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:08:57.640 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:08:57.641 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:08:57.642 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:08:57.643 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:08:57.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f344006ecc0 msgr2=0x7f3440071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f344006ecc0 0x7f3440071180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f343c005fd0 tx=0x7f343c005ee0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 msgr2=0x7f345019c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 0x7f345019c5a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3444006010 tx=0x7f344400be20 comp rx=0 tx=0).stop 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 shutdown_connections 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f344006ecc0 0x7f3440071180 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3450100da0 0x7f345019c5a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 --2- 172.21.15.151:0/4290719591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3450101750 0x7f345019cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 >> 172.21.15.151:0/4290719591 conn(0x7f34500fc630 msgr2=0x7f34500fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 shutdown_connections 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.639+0000 7f3456e26700 1 -- 172.21.15.151:0/4290719591 wait complete. 2024-04-16T03:08:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:08:57.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.745+0000 7f3058d3b700 1 -- 172.21.15.151:0/3712654474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30540ffa40 msgr2=0x7f30540ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.745+0000 7f3058d3b700 1 --2- 172.21.15.151:0/3712654474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30540ffa40 0x7f30540ffec0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f303c009910 tx=0x7f303c009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:57.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.745+0000 7f3058d3b700 1 -- 172.21.15.151:0/3712654474 shutdown_connections 2024-04-16T03:08:57.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.745+0000 7f3058d3b700 1 --2- 172.21.15.151:0/3712654474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30540ffa40 0x7f30540ffec0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.745+0000 7f3058d3b700 1 --2- 172.21.15.151:0/3712654474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3054105a40 0x7f3054105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.745+0000 7f3058d3b700 1 -- 172.21.15.151:0/3712654474 >> 172.21.15.151:0/3712654474 conn(0x7f30540fb570 msgr2=0x7f30540fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:57.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.745+0000 7f3058d3b700 1 -- 172.21.15.151:0/3712654474 shutdown_connections 2024-04-16T03:08:57.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.745+0000 7f3058d3b700 1 -- 172.21.15.151:0/3712654474 wait complete. 2024-04-16T03:08:57.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3058d3b700 1 Processor -- start 2024-04-16T03:08:57.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3058d3b700 1 -- start start 2024-04-16T03:08:57.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3058d3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30540ffa40 0x7f3054195380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:57.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3058d3b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3054105a40 0x7f30541958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3058d3b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3054195e00 con 0x7f3054105a40 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3058d3b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3054195f70 con 0x7f30540ffa40 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3049d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3054105a40 0x7f30541958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3049d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3054105a40 0x7f30541958c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44814/0 (socket says 172.21.15.151:44814) 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3049d9b700 1 -- 172.21.15.151:0/2264810228 learned_addr learned my addr 172.21.15.151:0/2264810228 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f305259c700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30540ffa40 0x7f3054195380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3049d9b700 1 -- 172.21.15.151:0/2264810228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30540ffa40 msgr2=0x7f3054195380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3049d9b700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30540ffa40 0x7f3054195380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.746+0000 7f3049d9b700 1 -- 172.21.15.151:0/2264810228 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c0095f0 con 0x7f3054105a40 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.747+0000 7f3049d9b700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3054105a40 0x7f30541958c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f303c006010 tx=0x7f303c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:57.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.747+0000 7f304bfff700 1 -- 172.21.15.151:0/2264810228 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303c01d070 con 0x7f3054105a40 2024-04-16T03:08:57.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.747+0000 7f304bfff700 1 -- 172.21.15.151:0/2264810228 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f303c005dc0 con 0x7f3054105a40 2024-04-16T03:08:57.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.747+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30541961f0 con 0x7f3054105a40 2024-04-16T03:08:57.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.747+0000 7f304bfff700 1 -- 172.21.15.151:0/2264810228 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303c017400 con 0x7f3054105a40 2024-04-16T03:08:57.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.747+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30541a2c70 con 0x7f3054105a40 2024-04-16T03:08:57.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.748+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3054107fd0 con 0x7f3054105a40 2024-04-16T03:08:57.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.748+0000 7f304bfff700 1 -- 172.21.15.151:0/2264810228 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f303c003a60 con 0x7f3054105a40 2024-04-16T03:08:57.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.749+0000 7f304bfff700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f304006ecc0 0x7f3040071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:57.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.749+0000 7f304bfff700 1 -- 172.21.15.151:0/2264810228 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f303c090160 con 0x7f3054105a40 2024-04-16T03:08:57.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.749+0000 7f305259c700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f304006ecc0 0x7f3040071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:57.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.749+0000 7f305259c700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f304006ecc0 0x7f3040071180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3044005950 tx=0x7f304400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:57.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.754+0000 7f304bfff700 1 -- 172.21.15.151:0/2264810228 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f303c05ca40 con 0x7f3054105a40 2024-04-16T03:08:57.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.973+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30541961f0 con 0x7f304006ecc0 2024-04-16T03:08:57.976 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.974+0000 7f304bfff700 1 -- 172.21.15.151:0/2264810228 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f30541961f0 con 0x7f304006ecc0 2024-04-16T03:08:57.976 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:08:57.976 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:08:57.976 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:08:57.976 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:08:57.976 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:08:57.976 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:08:57.977 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:08:57.977 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:08:57.977 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:08:57.977 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:08:57.977 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:08:57.977 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:08:57.977 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:08:57.977 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:08:57.977 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:08:57.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.976+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f304006ecc0 msgr2=0x7f3040071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.976+0000 7f3058d3b700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f304006ecc0 0x7f3040071180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3044005950 tx=0x7f304400b410 comp rx=0 tx=0).stop 2024-04-16T03:08:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.976+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3054105a40 msgr2=0x7f30541958c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.976+0000 7f3058d3b700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3054105a40 0x7f30541958c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f303c006010 tx=0x7f303c00bd70 comp rx=0 tx=0).stop 2024-04-16T03:08:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.976+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 shutdown_connections 2024-04-16T03:08:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.976+0000 7f3058d3b700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f304006ecc0 0x7f3040071180 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.976+0000 7f3058d3b700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30540ffa40 0x7f3054195380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.976+0000 7f3058d3b700 1 --2- 172.21.15.151:0/2264810228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3054105a40 0x7f30541958c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.976+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 >> 172.21.15.151:0/2264810228 conn(0x7f30540fb570 msgr2=0x7f30540fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.977+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 shutdown_connections 2024-04-16T03:08:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:57.977+0000 7f3058d3b700 1 -- 172.21.15.151:0/2264810228 wait complete. 2024-04-16T03:08:58.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.081+0000 7fb5997c4700 1 -- 172.21.15.151:0/723792627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb594105a40 msgr2=0x7fb594105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:58.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.081+0000 7fb5997c4700 1 --2- 172.21.15.151:0/723792627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb594105a40 0x7fb594105e20 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb584009910 tx=0x7fb584009c20 comp rx=0 tx=0).stop 2024-04-16T03:08:58.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.081+0000 7fb5997c4700 1 -- 172.21.15.151:0/723792627 shutdown_connections 2024-04-16T03:08:58.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.081+0000 7fb5997c4700 1 --2- 172.21.15.151:0/723792627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5940ffbd0 0x7fb594100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:58.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.081+0000 7fb5997c4700 1 --2- 172.21.15.151:0/723792627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb594105a40 0x7fb594105e20 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.081+0000 7fb5997c4700 1 -- 172.21.15.151:0/723792627 >> 172.21.15.151:0/723792627 conn(0x7fb5940755a0 msgr2=0x7fb5940759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.081+0000 7fb5997c4700 1 -- 172.21.15.151:0/723792627 shutdown_connections 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.081+0000 7fb5997c4700 1 -- 172.21.15.151:0/723792627 wait complete. 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb5997c4700 1 Processor -- start 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb5997c4700 1 -- start start 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb5997c4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5940ffbd0 0x7fb5941953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb5997c4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb594105a40 0x7fb5941958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb5997c4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940693d0 con 0x7fb5940ffbd0 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb5997c4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb594195e20 con 0x7fb594105a40 2024-04-16T03:08:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb592ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5940ffbd0 0x7fb5941953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb592ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5940ffbd0 0x7fb5941953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44834/0 (socket says 172.21.15.151:44834) 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb592ffd700 1 -- 172.21.15.151:0/3372990904 learned_addr learned my addr 172.21.15.151:0/3372990904 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.082+0000 7fb58a7fc700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb594105a40 0x7fb5941958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.083+0000 7fb592ffd700 1 -- 172.21.15.151:0/3372990904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb594105a40 msgr2=0x7fb5941958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.083+0000 7fb592ffd700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb594105a40 0x7fb5941958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.083+0000 7fb592ffd700 1 -- 172.21.15.151:0/3372990904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5840095f0 con 0x7fb5940ffbd0 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.083+0000 7fb592ffd700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5940ffbd0 0x7fb5941953a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb584006010 tx=0x7fb58400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.083+0000 7fb590ff9700 1 -- 172.21.15.151:0/3372990904 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb58401d070 con 0x7fb5940ffbd0 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.083+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5941960a0 con 0x7fb5940ffbd0 2024-04-16T03:08:58.087 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.083+0000 7fb590ff9700 1 -- 172.21.15.151:0/3372990904 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb58400f460 con 0x7fb5940ffbd0 2024-04-16T03:08:58.088 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.083+0000 7fb590ff9700 1 -- 172.21.15.151:0/3372990904 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb584017630 con 0x7fb5940ffbd0 2024-04-16T03:08:58.088 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.083+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5941a2c30 con 0x7fb5940ffbd0 2024-04-16T03:08:58.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.084+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb594108040 con 0x7fb5940ffbd0 2024-04-16T03:08:58.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.085+0000 7fb590ff9700 1 -- 172.21.15.151:0/3372990904 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb58400f5d0 con 0x7fb5940ffbd0 2024-04-16T03:08:58.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.085+0000 7fb590ff9700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58006ecc0 0x7fb580071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:08:58.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.085+0000 7fb58a7fc700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58006ecc0 0x7fb580071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:08:58.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.085+0000 7fb590ff9700 1 -- 172.21.15.151:0/3372990904 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb5840903a0 con 0x7fb5940ffbd0 2024-04-16T03:08:58.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.085+0000 7fb58a7fc700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58006ecc0 0x7fb580071180 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb594068d70 tx=0x7fb57c00a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:08:58.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.090+0000 7fb590ff9700 1 -- 172.21.15.151:0/3372990904 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb58405cc80 con 0x7fb5940ffbd0 2024-04-16T03:08:58.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:57 smithi151 ceph-mon[112233]: from='client.35660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:57 smithi151 ceph-mon[112233]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:57 smithi151 ceph-mon[112233]: from='client.35668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:58.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:57 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3767964579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:08:58.247 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:57 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4290719591' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:08:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:57 smithi153 ceph-mon[87858]: from='client.35660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:57 smithi153 ceph-mon[87858]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:57 smithi153 ceph-mon[87858]: from='client.35668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:58.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:57 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3767964579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:08:58.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:57 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4290719591' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:08:58.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.376+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb59404ea90 con 0x7fb5940ffbd0 2024-04-16T03:08:58.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.376+0000 7fb590ff9700 1 -- 172.21.15.151:0/3372990904 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7fb58405c810 con 0x7fb5940ffbd0 2024-04-16T03:08:58.377 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:08:58.378 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:08:58.379 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:08:58.379 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:08:58.379 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:08:58.379 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:08:58.379 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:08:58.379 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:08:58.379 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:08:58.379 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58006ecc0 msgr2=0x7fb580071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58006ecc0 0x7fb580071180 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb594068d70 tx=0x7fb57c00a380 comp rx=0 tx=0).stop 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5940ffbd0 msgr2=0x7fb5941953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5940ffbd0 0x7fb5941953a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb584006010 tx=0x7fb58400bd70 comp rx=0 tx=0).stop 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 shutdown_connections 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb58006ecc0 0x7fb580071180 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5940ffbd0 0x7fb5941953a0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 --2- 172.21.15.151:0/3372990904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb594105a40 0x7fb5941958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 >> 172.21.15.151:0/3372990904 conn(0x7fb5940755a0 msgr2=0x7fb5940fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:08:58.380 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 shutdown_connections 2024-04-16T03:08:58.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:08:58.378+0000 7fb5997c4700 1 -- 172.21.15.151:0/3372990904 wait complete. 2024-04-16T03:08:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:58 smithi151 ceph-mon[112233]: pgmap v911: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:58 smithi151 ceph-mon[112233]: from='client.35680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:08:58 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3372990904' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:08:59.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:58 smithi153 ceph-mon[87858]: pgmap v911: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:08:59.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:58 smithi153 ceph-mon[87858]: from='client.35680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:08:59.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:08:58 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3372990904' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:09:01.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:00 smithi151 ceph-mon[112233]: pgmap v912: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:01.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:00 smithi153 ceph-mon[87858]: pgmap v912: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:03.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:02 smithi151 ceph-mon[112233]: pgmap v913: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:03.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:02 smithi153 ceph-mon[87858]: pgmap v913: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:05.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:04 smithi151 ceph-mon[112233]: pgmap v914: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:05.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:04 smithi153 ceph-mon[87858]: pgmap v914: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:07.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:06 smithi151 ceph-mon[112233]: pgmap v915: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:07.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:06 smithi153 ceph-mon[87858]: pgmap v915: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:08 smithi151 ceph-mon[112233]: pgmap v916: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:09:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:08 smithi153 ceph-mon[87858]: pgmap v916: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:09:11.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:10 smithi151 ceph-mon[112233]: pgmap v917: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:10 smithi153 ceph-mon[87858]: pgmap v917: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:13.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:12 smithi151 ceph-mon[112233]: pgmap v918: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:12 smithi153 ceph-mon[87858]: pgmap v918: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:15.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:14 smithi151 ceph-mon[112233]: pgmap v919: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:15.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:14 smithi153 ceph-mon[87858]: pgmap v919: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:17.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:16 smithi151 ceph-mon[112233]: pgmap v920: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:17.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:16 smithi153 ceph-mon[87858]: pgmap v920: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:18 smithi151 ceph-mon[112233]: pgmap v921: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:19.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:18 smithi153 ceph-mon[87858]: pgmap v921: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:21.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:20 smithi153 ceph-mon[87858]: pgmap v922: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:21.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:20 smithi151 ceph-mon[112233]: pgmap v922: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:23.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:22 smithi153 ceph-mon[87858]: pgmap v923: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:23.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:22 smithi151 ceph-mon[112233]: pgmap v923: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:24.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:09:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:09:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:25 smithi153 ceph-mon[87858]: pgmap v924: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:25.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:25 smithi151 ceph-mon[112233]: pgmap v924: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:27.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:27 smithi153 ceph-mon[87858]: pgmap v925: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:27.459 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:27 smithi151 ceph-mon[112233]: pgmap v925: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:28.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.497+0000 7f6913657700 1 -- 172.21.15.151:0/142526533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 msgr2=0x7f690c105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:28.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.497+0000 7f6913657700 1 --2- 172.21.15.151:0/142526533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 0x7f690c105e30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f690800b0a0 tx=0x7f690800b3b0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.498+0000 7f6913657700 1 -- 172.21.15.151:0/142526533 shutdown_connections 2024-04-16T03:09:28.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.498+0000 7f6913657700 1 --2- 172.21.15.151:0/142526533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f690c0ffa50 0x7f690c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.498+0000 7f6913657700 1 --2- 172.21.15.151:0/142526533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 0x7f690c105e30 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.498+0000 7f6913657700 1 -- 172.21.15.151:0/142526533 >> 172.21.15.151:0/142526533 conn(0x7f690c0fb540 msgr2=0x7f690c0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:28.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.498+0000 7f6913657700 1 -- 172.21.15.151:0/142526533 shutdown_connections 2024-04-16T03:09:28.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.498+0000 7f6913657700 1 -- 172.21.15.151:0/142526533 wait complete. 2024-04-16T03:09:28.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.498+0000 7f6913657700 1 Processor -- start 2024-04-16T03:09:28.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6913657700 1 -- start start 2024-04-16T03:09:28.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6913657700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f690c0ffa50 0x7f690c195680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:28.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6913657700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 0x7f690c195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:28.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6913657700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c196250 con 0x7f690c105a50 2024-04-16T03:09:28.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6913657700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c198f80 con 0x7f690c0ffa50 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6910bf2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 0x7f690c195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6910bf2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 0x7f690c195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54660/0 (socket says 172.21.15.151:54660) 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6910bf2700 1 -- 172.21.15.151:0/3675583912 learned_addr learned my addr 172.21.15.151:0/3675583912 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f69113f3700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f690c0ffa50 0x7f690c195680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6910bf2700 1 -- 172.21.15.151:0/3675583912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f690c0ffa50 msgr2=0x7f690c195680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6910bf2700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f690c0ffa50 0x7f690c195680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6910bf2700 1 -- 172.21.15.151:0/3675583912 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6908009d00 con 0x7f690c105a50 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.499+0000 7f6910bf2700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 0x7f690c195bc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00d900 tx=0x7f68fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.500+0000 7f69027fc700 1 -- 172.21.15.151:0/3675583912 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68fc0041f0 con 0x7f690c105a50 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.500+0000 7f69027fc700 1 -- 172.21.15.151:0/3675583912 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68fc004cf0 con 0x7f690c105a50 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.500+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c199260 con 0x7f690c105a50 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.500+0000 7f69027fc700 1 -- 172.21.15.151:0/3675583912 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68fc00b750 con 0x7f690c105a50 2024-04-16T03:09:28.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.500+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c1997b0 con 0x7f690c105a50 2024-04-16T03:09:28.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.501+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690c108050 con 0x7f690c105a50 2024-04-16T03:09:28.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.501+0000 7f69027fc700 1 -- 172.21.15.151:0/3675583912 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f68fc004750 con 0x7f690c105a50 2024-04-16T03:09:28.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.502+0000 7f69027fc700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f68f806ecc0 0x7f68f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:28.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.502+0000 7f69113f3700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f68f806ecc0 0x7f68f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:28.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.502+0000 7f69027fc700 1 -- 172.21.15.151:0/3675583912 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f68fc08ed60 con 0x7f690c105a50 2024-04-16T03:09:28.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.503+0000 7f69113f3700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f68f806ecc0 0x7f68f8071180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6908000f80 tx=0x7f69080061f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:28.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.507+0000 7f69027fc700 1 -- 172.21.15.151:0/3675583912 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f68fc057b10 con 0x7f690c105a50 2024-04-16T03:09:28.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.723+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f690c199a40 con 0x7f68f806ecc0 2024-04-16T03:09:28.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.724+0000 7f69027fc700 1 -- 172.21.15.151:0/3675583912 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f690c199a40 con 0x7f68f806ecc0 2024-04-16T03:09:28.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f68f806ecc0 msgr2=0x7f68f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:28.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f68f806ecc0 0x7f68f8071180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6908000f80 tx=0x7f69080061f0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 msgr2=0x7f690c195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:28.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 0x7f690c195bc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00d900 tx=0x7f68fc00dc10 comp rx=0 tx=0).stop 2024-04-16T03:09:28.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 shutdown_connections 2024-04-16T03:09:28.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f68f806ecc0 0x7f68f8071180 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f690c0ffa50 0x7f690c195680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 --2- 172.21.15.151:0/3675583912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f690c105a50 0x7f690c195bc0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 >> 172.21.15.151:0/3675583912 conn(0x7f690c0fb540 msgr2=0x7f690c0fbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:28.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 shutdown_connections 2024-04-16T03:09:28.729 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.726+0000 7f6913657700 1 -- 172.21.15.151:0/3675583912 wait complete. 2024-04-16T03:09:28.738 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:09:28.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.833+0000 7f13f8132700 1 -- 172.21.15.151:0/379937647 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 msgr2=0x7f13f00ff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:28.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.833+0000 7f13f8132700 1 --2- 172.21.15.151:0/379937647 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 0x7f13f00ff8e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f13e0009910 tx=0x7f13e0009c20 comp rx=0 tx=0).stop 2024-04-16T03:09:28.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.833+0000 7f13f8132700 1 -- 172.21.15.151:0/379937647 shutdown_connections 2024-04-16T03:09:28.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.833+0000 7f13f8132700 1 --2- 172.21.15.151:0/379937647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13f00ffeb0 0x7f13f0105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.833+0000 7f13f8132700 1 --2- 172.21.15.151:0/379937647 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 0x7f13f00ff8e0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.833+0000 7f13f8132700 1 -- 172.21.15.151:0/379937647 >> 172.21.15.151:0/379937647 conn(0x7f13f0074d80 msgr2=0x7f13f0075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:28.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.833+0000 7f13f8132700 1 -- 172.21.15.151:0/379937647 shutdown_connections 2024-04-16T03:09:28.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.833+0000 7f13f8132700 1 -- 172.21.15.151:0/379937647 wait complete. 2024-04-16T03:09:28.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.834+0000 7f13f8132700 1 Processor -- start 2024-04-16T03:09:28.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.834+0000 7f13f8132700 1 -- start start 2024-04-16T03:09:28.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.834+0000 7f13f8132700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 0x7f13f0100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:28.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.834+0000 7f13f8132700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13f00ffeb0 0x7f13f01009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:28.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.834+0000 7f13f8132700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f0100f00 con 0x7f13f00ff500 2024-04-16T03:09:28.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.834+0000 7f13f8132700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f0101040 con 0x7f13f00ffeb0 2024-04-16T03:09:28.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.834+0000 7f13f5ece700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 0x7f13f0100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.834+0000 7f13f5ece700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 0x7f13f0100480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54682/0 (socket says 172.21.15.151:54682) 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.834+0000 7f13f5ece700 1 -- 172.21.15.151:0/3049109992 learned_addr learned my addr 172.21.15.151:0/3049109992 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13f56cd700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13f00ffeb0 0x7f13f01009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13f5ece700 1 -- 172.21.15.151:0/3049109992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13f00ffeb0 msgr2=0x7f13f01009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13f5ece700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13f00ffeb0 0x7f13f01009c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13f5ece700 1 -- 172.21.15.151:0/3049109992 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00095f0 con 0x7f13f00ff500 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13f56cd700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13f00ffeb0 0x7f13f01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13f5ece700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 0x7f13f0100480 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f13e000b5c0 tx=0x7f13e000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13e6ffd700 1 -- 172.21.15.151:0/3049109992 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13e001d070 con 0x7f13f00ff500 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f01012c0 con 0x7f13f00ff500 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13e6ffd700 1 -- 172.21.15.151:0/3049109992 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13e000f460 con 0x7f13f00ff500 2024-04-16T03:09:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13e6ffd700 1 -- 172.21.15.151:0/3049109992 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13e0017610 con 0x7f13f00ff500 2024-04-16T03:09:28.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.835+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f01a2db0 con 0x7f13f00ff500 2024-04-16T03:09:28.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.836+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f0109190 con 0x7f13f00ff500 2024-04-16T03:09:28.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.837+0000 7f13e6ffd700 1 -- 172.21.15.151:0/3049109992 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f13e000f5d0 con 0x7f13f00ff500 2024-04-16T03:09:28.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.837+0000 7f13e6ffd700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13dc06ecc0 0x7f13dc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:28.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.837+0000 7f13e6ffd700 1 -- 172.21.15.151:0/3049109992 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f13e0090380 con 0x7f13f00ff500 2024-04-16T03:09:28.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.837+0000 7f13f56cd700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13dc06ecc0 0x7f13dc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:28.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.838+0000 7f13f56cd700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13dc06ecc0 0x7f13dc071180 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f13f0101c70 tx=0x7f13ec008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:28.843 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:28.842+0000 7f13e6ffd700 1 -- 172.21.15.151:0/3049109992 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f13e00591b0 con 0x7f13f00ff500 2024-04-16T03:09:29.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.062+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13f002cce0 con 0x7f13dc06ecc0 2024-04-16T03:09:29.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.063+0000 7f13e6ffd700 1 -- 172.21.15.151:0/3049109992 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f13f002cce0 con 0x7f13dc06ecc0 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13dc06ecc0 msgr2=0x7f13dc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13dc06ecc0 0x7f13dc071180 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f13f0101c70 tx=0x7f13ec008040 comp rx=0 tx=0).stop 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 msgr2=0x7f13f0100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 0x7f13f0100480 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f13e000b5c0 tx=0x7f13e000be00 comp rx=0 tx=0).stop 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 shutdown_connections 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f13dc06ecc0 0x7f13dc071180 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13f00ff500 0x7f13f0100480 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 --2- 172.21.15.151:0/3049109992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13f00ffeb0 0x7f13f01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 >> 172.21.15.151:0/3049109992 conn(0x7f13f0074d80 msgr2=0x7f13f00fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 shutdown_connections 2024-04-16T03:09:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.065+0000 7f13f8132700 1 -- 172.21.15.151:0/3049109992 wait complete. 2024-04-16T03:09:29.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.170+0000 7fdec8957700 1 -- 172.21.15.151:0/2578936770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ff510 msgr2=0x7fdec40ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.170+0000 7fdec8957700 1 --2- 172.21.15.151:0/2578936770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ff510 0x7fdec40ff8f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009910 tx=0x7fdeb4009c20 comp rx=0 tx=0).stop 2024-04-16T03:09:29.172 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:29 smithi151 ceph-mon[112233]: pgmap v926: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.170+0000 7fdec8957700 1 -- 172.21.15.151:0/2578936770 shutdown_connections 2024-04-16T03:09:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.170+0000 7fdec8957700 1 --2- 172.21.15.151:0/2578936770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdec40ffec0 0x7fdec41057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.170+0000 7fdec8957700 1 --2- 172.21.15.151:0/2578936770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ff510 0x7fdec40ff8f0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.170+0000 7fdec8957700 1 -- 172.21.15.151:0/2578936770 >> 172.21.15.151:0/2578936770 conn(0x7fdec4074e00 msgr2=0x7fdec4075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.170+0000 7fdec8957700 1 -- 172.21.15.151:0/2578936770 shutdown_connections 2024-04-16T03:09:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.170+0000 7fdec8957700 1 -- 172.21.15.151:0/2578936770 wait complete. 2024-04-16T03:09:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.171+0000 7fdec8957700 1 Processor -- start 2024-04-16T03:09:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.171+0000 7fdec8957700 1 -- start start 2024-04-16T03:09:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.171+0000 7fdec8957700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdec40ff510 0x7fdec4100490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.171+0000 7fdec8957700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ffec0 0x7fdec41009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.171+0000 7fdec8957700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec4100f10 con 0x7fdec40ffec0 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.171+0000 7fdec8957700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec4101050 con 0x7fdec40ff510 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ffec0 0x7fdec41009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ffec0 0x7fdec41009d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54690/0 (socket says 172.21.15.151:54690) 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec1d9b700 1 -- 172.21.15.151:0/3757349779 learned_addr learned my addr 172.21.15.151:0/3757349779 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec259c700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdec40ff510 0x7fdec4100490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec1d9b700 1 -- 172.21.15.151:0/3757349779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdec40ff510 msgr2=0x7fdec4100490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec1d9b700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdec40ff510 0x7fdec4100490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec1d9b700 1 -- 172.21.15.151:0/3757349779 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb40095f0 con 0x7fdec40ffec0 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec1d9b700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ffec0 0x7fdec41009d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00d900 tx=0x7fdeac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3757349779 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdeac009900 con 0x7fdec40ffec0 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3757349779 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdeac010460 con 0x7fdec40ffec0 2024-04-16T03:09:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3757349779 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdeac00f5f0 con 0x7fdec40ffec0 2024-04-16T03:09:29.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec41a2970 con 0x7fdec40ffec0 2024-04-16T03:09:29.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.172+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec41a2d20 con 0x7fdec40ffec0 2024-04-16T03:09:29.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.173+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec41091e0 con 0x7fdec40ffec0 2024-04-16T03:09:29.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.174+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3757349779 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fdeac0105d0 con 0x7fdec40ffec0 2024-04-16T03:09:29.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.174+0000 7fdebb7fe700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdeb006ecc0 0x7fdeb0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:29.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.175+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3757349779 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdeac08eac0 con 0x7fdec40ffec0 2024-04-16T03:09:29.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.175+0000 7fdec259c700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdeb006ecc0 0x7fdeb0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:29.177 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.175+0000 7fdec259c700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdeb006ecc0 0x7fdeb0071180 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400b5c0 tx=0x7fdeb401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:29.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.179+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3757349779 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdeac057870 con 0x7fdec40ffec0 2024-04-16T03:09:29.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:29 smithi153 ceph-mon[87858]: pgmap v926: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:29.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.386+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdec40611d0 con 0x7fdeb006ecc0 2024-04-16T03:09:29.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.397+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3757349779 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fdec40611d0 con 0x7fdeb006ecc0 2024-04-16T03:09:29.398 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (31m) 9m ago 42m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (42m) 9m ago 42m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (41m) 9m ago 41m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (28m) 9m ago 42m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (28m) 9m ago 41m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (30m) 9m ago 42m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (24m) 9m ago 36m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (24m) 9m ago 36m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (9m) 9m ago 36m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (24m) 9m ago 36m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (33m) 9m ago 43m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (31m) 9m ago 41m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (29m) 9m ago 43m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (29m) 9m ago 41m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (31m) 9m ago 42m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:09:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (31m) 9m ago 41m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:09:29.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (28m) 9m ago 40m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:09:29.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (27m) 9m ago 40m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:09:29.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (26m) 9m ago 39m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:09:29.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (26m) 9m ago 39m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:09:29.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (25m) 9m ago 38m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:09:29.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (25m) 9m ago 38m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:09:29.400 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (31m) 9m ago 41m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdeb006ecc0 msgr2=0x7fdeb0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdeb006ecc0 0x7fdeb0071180 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400b5c0 tx=0x7fdeb401a040 comp rx=0 tx=0).stop 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ffec0 msgr2=0x7fdec41009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ffec0 0x7fdec41009d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00d900 tx=0x7fdeac00dcc0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 shutdown_connections 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdeb006ecc0 0x7fdeb0071180 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdec40ff510 0x7fdec4100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 --2- 172.21.15.151:0/3757349779 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdec40ffec0 0x7fdec41009d0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 >> 172.21.15.151:0/3757349779 conn(0x7fdec4074e00 msgr2=0x7fdec40feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 shutdown_connections 2024-04-16T03:09:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.399+0000 7fdec8957700 1 -- 172.21.15.151:0/3757349779 wait complete. 2024-04-16T03:09:29.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.502+0000 7f763ef34700 1 -- 172.21.15.151:0/102545019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638101c70 msgr2=0x7f7638102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.506 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.502+0000 7f763ef34700 1 --2- 172.21.15.151:0/102545019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638101c70 0x7f7638102050 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f762c009910 tx=0x7f762c009c20 comp rx=0 tx=0).stop 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.502+0000 7f763ef34700 1 -- 172.21.15.151:0/102545019 shutdown_connections 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.502+0000 7f763ef34700 1 --2- 172.21.15.151:0/102545019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7638102620 0x7f763810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 --2- 172.21.15.151:0/102545019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638101c70 0x7f7638102050 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 -- 172.21.15.151:0/102545019 >> 172.21.15.151:0/102545019 conn(0x7f76380fb550 msgr2=0x7f76380fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 -- 172.21.15.151:0/102545019 shutdown_connections 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 -- 172.21.15.151:0/102545019 wait complete. 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 Processor -- start 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 -- start start 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7638101c70 0x7f763819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638102620 0x7f763819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763819d0c0 con 0x7f7638102620 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.503+0000 7f763ef34700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7638196570 con 0x7f7638101c70 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7637fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638102620 0x7f763819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:29.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7637fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638102620 0x7f763819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54710/0 (socket says 172.21.15.151:54710) 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7637fff700 1 -- 172.21.15.151:0/3933660806 learned_addr learned my addr 172.21.15.151:0/3933660806 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f763ccd0700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7638101c70 0x7f763819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7637fff700 1 -- 172.21.15.151:0/3933660806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7638101c70 msgr2=0x7f763819c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7637fff700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7638101c70 0x7f763819c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7637fff700 1 -- 172.21.15.151:0/3933660806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c0095f0 con 0x7f7638102620 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7637fff700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638102620 0x7f763819ca30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f762400d900 tx=0x7f762400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7635ffb700 1 -- 172.21.15.151:0/3933660806 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7624009900 con 0x7f7638102620 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7638196850 con 0x7f7638102620 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7635ffb700 1 -- 172.21.15.151:0/3933660806 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7624010460 con 0x7f7638102620 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f7635ffb700 1 -- 172.21.15.151:0/3933660806 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f762400f5f0 con 0x7f7638102620 2024-04-16T03:09:29.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.504+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7638196da0 con 0x7f7638102620 2024-04-16T03:09:29.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.505+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7638108210 con 0x7f7638102620 2024-04-16T03:09:29.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.506+0000 7f7635ffb700 1 -- 172.21.15.151:0/3933660806 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f76240105d0 con 0x7f7638102620 2024-04-16T03:09:29.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.506+0000 7f7635ffb700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f762806ecc0 0x7f7628071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:29.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.506+0000 7f7635ffb700 1 -- 172.21.15.151:0/3933660806 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f762408ea60 con 0x7f7638102620 2024-04-16T03:09:29.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.506+0000 7f763ccd0700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f762806ecc0 0x7f7628071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:29.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.507+0000 7f763ccd0700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f762806ecc0 0x7f7628071180 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f762c006010 tx=0x7f762c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:29.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.511+0000 7f7635ffb700 1 -- 172.21.15.151:0/3933660806 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f762405b2e0 con 0x7f7638102620 2024-04-16T03:09:29.792 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.790+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f763804ea90 con 0x7f7638102620 2024-04-16T03:09:29.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.791+0000 7f7635ffb700 1 -- 172.21.15.151:0/3933660806 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f762405ae70 con 0x7f7638102620 2024-04-16T03:09:29.793 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:09:29.793 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:09:29.793 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:09:29.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:09:29.793 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:09:29.794 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:09:29.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f762806ecc0 msgr2=0x7f7628071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f762806ecc0 0x7f7628071180 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f762c006010 tx=0x7f762c01a040 comp rx=0 tx=0).stop 2024-04-16T03:09:29.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638102620 msgr2=0x7f763819ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638102620 0x7f763819ca30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f762400d900 tx=0x7f762400dc10 comp rx=0 tx=0).stop 2024-04-16T03:09:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 shutdown_connections 2024-04-16T03:09:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f762806ecc0 0x7f7628071180 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7638101c70 0x7f763819c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 --2- 172.21.15.151:0/3933660806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7638102620 0x7f763819ca30 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 >> 172.21.15.151:0/3933660806 conn(0x7f76380fb550 msgr2=0x7f7638105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 shutdown_connections 2024-04-16T03:09:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.793+0000 7f763ef34700 1 -- 172.21.15.151:0/3933660806 wait complete. 2024-04-16T03:09:29.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.897+0000 7f5000e09700 1 -- 172.21.15.151:0/2446668412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 msgr2=0x7f4ffc1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.897+0000 7f5000e09700 1 --2- 172.21.15.151:0/2446668412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 0x7f4ffc1011a0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009910 tx=0x7f4ff0009c20 comp rx=0 tx=0).stop 2024-04-16T03:09:29.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.897+0000 7f5000e09700 1 -- 172.21.15.151:0/2446668412 shutdown_connections 2024-04-16T03:09:29.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.897+0000 7f5000e09700 1 --2- 172.21.15.151:0/2446668412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ffc101770 0x7f4ffc1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.897+0000 7f5000e09700 1 --2- 172.21.15.151:0/2446668412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 0x7f4ffc1011a0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.897+0000 7f5000e09700 1 -- 172.21.15.151:0/2446668412 >> 172.21.15.151:0/2446668412 conn(0x7f4ffc0fc650 msgr2=0x7f4ffc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:29.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.897+0000 7f5000e09700 1 -- 172.21.15.151:0/2446668412 shutdown_connections 2024-04-16T03:09:29.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.897+0000 7f5000e09700 1 -- 172.21.15.151:0/2446668412 wait complete. 2024-04-16T03:09:29.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.898+0000 7f5000e09700 1 Processor -- start 2024-04-16T03:09:29.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.898+0000 7f5000e09700 1 -- start start 2024-04-16T03:09:29.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.898+0000 7f5000e09700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 0x7f4ffc19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:29.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.898+0000 7f5000e09700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ffc101770 0x7f4ffc19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:29.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.898+0000 7f5000e09700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc19d110 con 0x7f4ffc100dc0 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.898+0000 7f5000e09700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc1965c0 con 0x7f4ffc101770 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.898+0000 7f4ffa59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 0x7f4ffc19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.898+0000 7f4ffa59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 0x7f4ffc19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54736/0 (socket says 172.21.15.151:54736) 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.898+0000 7f4ffa59c700 1 -- 172.21.15.151:0/2523713743 learned_addr learned my addr 172.21.15.151:0/2523713743 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f4ff9d9b700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ffc101770 0x7f4ffc19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f4ffa59c700 1 -- 172.21.15.151:0/2523713743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ffc101770 msgr2=0x7f4ffc19ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f4ffa59c700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ffc101770 0x7f4ffc19ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f4ffa59c700 1 -- 172.21.15.151:0/2523713743 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00095f0 con 0x7f4ffc100dc0 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f4ffa59c700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 0x7f4ffc19c540 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00092e0 tx=0x7f4ff000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f4feb7fe700 1 -- 172.21.15.151:0/2523713743 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ff001d070 con 0x7f4ffc100dc0 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc196840 con 0x7f4ffc100dc0 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f4feb7fe700 1 -- 172.21.15.151:0/2523713743 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ff0005e00 con 0x7f4ffc100dc0 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f4feb7fe700 1 -- 172.21.15.151:0/2523713743 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ff0017400 con 0x7f4ffc100dc0 2024-04-16T03:09:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.899+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc196d30 con 0x7f4ffc100dc0 2024-04-16T03:09:29.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.900+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc109160 con 0x7f4ffc100dc0 2024-04-16T03:09:29.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.900+0000 7f4feb7fe700 1 -- 172.21.15.151:0/2523713743 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4ff000f460 con 0x7f4ffc100dc0 2024-04-16T03:09:29.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.901+0000 7f4feb7fe700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fe406ecc0 0x7f4fe4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:29.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.901+0000 7f4feb7fe700 1 -- 172.21.15.151:0/2523713743 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4ff0090390 con 0x7f4ffc100dc0 2024-04-16T03:09:29.904 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.901+0000 7f4ff9d9b700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fe406ecc0 0x7f4fe4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:29.905 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.901+0000 7f4ff9d9b700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fe406ecc0 0x7f4fe4071180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4fec005fd0 tx=0x7f4fec005f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:29.908 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:29.906+0000 7f4feb7fe700 1 -- 172.21.15.151:0/2523713743 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4ff005cc70 con 0x7f4ffc100dc0 2024-04-16T03:09:30.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.149+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4ffc04ea90 con 0x7f4ffc100dc0 2024-04-16T03:09:30.152 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:30 smithi151 ceph-mon[112233]: from='client.35688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:09:30.152 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:30 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3933660806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:09:30.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.149+0000 7f4feb7fe700 1 -- 172.21.15.151:0/2523713743 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f4ff005c800 con 0x7f4ffc100dc0 2024-04-16T03:09:30.152 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:09:30.152 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:09:30.152 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:09:30.152 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:09:30.153 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:09:30.154 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:09:30.155 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:09:30.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fe406ecc0 msgr2=0x7f4fe4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fe406ecc0 0x7f4fe4071180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4fec005fd0 tx=0x7f4fec005f60 comp rx=0 tx=0).stop 2024-04-16T03:09:30.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 msgr2=0x7f4ffc19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 0x7f4ffc19c540 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00092e0 tx=0x7f4ff000be20 comp rx=0 tx=0).stop 2024-04-16T03:09:30.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 shutdown_connections 2024-04-16T03:09:30.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4fe406ecc0 0x7f4fe4071180 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ffc100dc0 0x7f4ffc19c540 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 --2- 172.21.15.151:0/2523713743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ffc101770 0x7f4ffc19ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 >> 172.21.15.151:0/2523713743 conn(0x7f4ffc0fc650 msgr2=0x7f4ffc0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:30.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 shutdown_connections 2024-04-16T03:09:30.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.152+0000 7f5000e09700 1 -- 172.21.15.151:0/2523713743 wait complete. 2024-04-16T03:09:30.157 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:09:30.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.258+0000 7fa68d24b700 1 -- 172.21.15.151:0/1147028282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688101c70 msgr2=0x7fa688102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.258+0000 7fa68d24b700 1 --2- 172.21.15.151:0/1147028282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688101c70 0x7fa688102050 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa678009910 tx=0x7fa678009c20 comp rx=0 tx=0).stop 2024-04-16T03:09:30.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.258+0000 7fa68d24b700 1 -- 172.21.15.151:0/1147028282 shutdown_connections 2024-04-16T03:09:30.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.258+0000 7fa68d24b700 1 --2- 172.21.15.151:0/1147028282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688102620 0x7fa68810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.258+0000 7fa68d24b700 1 --2- 172.21.15.151:0/1147028282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688101c70 0x7fa688102050 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.258+0000 7fa68d24b700 1 -- 172.21.15.151:0/1147028282 >> 172.21.15.151:0/1147028282 conn(0x7fa6880fb550 msgr2=0x7fa6880fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:30.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.259+0000 7fa68d24b700 1 -- 172.21.15.151:0/1147028282 shutdown_connections 2024-04-16T03:09:30.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.259+0000 7fa68d24b700 1 -- 172.21.15.151:0/1147028282 wait complete. 2024-04-16T03:09:30.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.259+0000 7fa68d24b700 1 Processor -- start 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.259+0000 7fa68d24b700 1 -- start start 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68d24b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688101c70 0x7fa68819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68d24b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688102620 0x7fa68819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68d24b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68819d0c0 con 0x7fa688102620 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68d24b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa688196570 con 0x7fa688101c70 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688102620 0x7fa68819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688102620 0x7fa68819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54766/0 (socket says 172.21.15.151:54766) 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68659c700 1 -- 172.21.15.151:0/4228893769 learned_addr learned my addr 172.21.15.151:0/4228893769 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa686d9d700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688101c70 0x7fa68819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68659c700 1 -- 172.21.15.151:0/4228893769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688101c70 msgr2=0x7fa68819c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68659c700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688101c70 0x7fa68819c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68659c700 1 -- 172.21.15.151:0/4228893769 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6780095f0 con 0x7fa688102620 2024-04-16T03:09:30.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa68659c700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688102620 0x7fa68819ca30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa67c00d8d0 tx=0x7fa67c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:30.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.260+0000 7fa66ffff700 1 -- 172.21.15.151:0/4228893769 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa67c0041f0 con 0x7fa688102620 2024-04-16T03:09:30.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.261+0000 7fa66ffff700 1 -- 172.21.15.151:0/4228893769 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa67c007620 con 0x7fa688102620 2024-04-16T03:09:30.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.261+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa688196850 con 0x7fa688102620 2024-04-16T03:09:30.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.261+0000 7fa66ffff700 1 -- 172.21.15.151:0/4228893769 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa67c0056b0 con 0x7fa688102620 2024-04-16T03:09:30.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.261+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa688196da0 con 0x7fa688102620 2024-04-16T03:09:30.265 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.261+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa688108210 con 0x7fa688102620 2024-04-16T03:09:30.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.262+0000 7fa66ffff700 1 -- 172.21.15.151:0/4228893769 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa67c00f770 con 0x7fa688102620 2024-04-16T03:09:30.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.262+0000 7fa66ffff700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa67006ecc0 0x7fa670071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:30.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.263+0000 7fa66ffff700 1 -- 172.21.15.151:0/4228893769 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa67c08eca0 con 0x7fa688102620 2024-04-16T03:09:30.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.263+0000 7fa686d9d700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa67006ecc0 0x7fa670071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:30.266 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.263+0000 7fa686d9d700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa67006ecc0 0x7fa670071180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa67800b5c0 tx=0x7fa67801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:30.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.267+0000 7fa66ffff700 1 -- 172.21.15.151:0/4228893769 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa67c093070 con 0x7fa688102620 2024-04-16T03:09:30.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:30 smithi153 ceph-mon[87858]: from='client.35688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:09:30.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:30 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3933660806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:09:30.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.484+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa688197010 con 0x7fa67006ecc0 2024-04-16T03:09:30.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.485+0000 7fa66ffff700 1 -- 172.21.15.151:0/4228893769 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa688197010 con 0x7fa67006ecc0 2024-04-16T03:09:30.487 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:09:30.488 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.487+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa67006ecc0 msgr2=0x7fa670071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.487+0000 7fa68d24b700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa67006ecc0 0x7fa670071180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa67800b5c0 tx=0x7fa67801a040 comp rx=0 tx=0).stop 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.487+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688102620 msgr2=0x7fa68819ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.487+0000 7fa68d24b700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688102620 0x7fa68819ca30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa67c00d8d0 tx=0x7fa67c00dbe0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.487+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 shutdown_connections 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.487+0000 7fa68d24b700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa67006ecc0 0x7fa670071180 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.487+0000 7fa68d24b700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688101c70 0x7fa68819c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.487+0000 7fa68d24b700 1 --2- 172.21.15.151:0/4228893769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa688102620 0x7fa68819ca30 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.487+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 >> 172.21.15.151:0/4228893769 conn(0x7fa6880fb550 msgr2=0x7fa688105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.488+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 shutdown_connections 2024-04-16T03:09:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.488+0000 7fa68d24b700 1 -- 172.21.15.151:0/4228893769 wait complete. 2024-04-16T03:09:30.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.592+0000 7faa508ab700 1 -- 172.21.15.151:0/251206118 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 msgr2=0x7faa4c0ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.592+0000 7faa508ab700 1 --2- 172.21.15.151:0/251206118 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 0x7faa4c0ff3f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7faa34009910 tx=0x7faa34009c20 comp rx=0 tx=0).stop 2024-04-16T03:09:30.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.592+0000 7faa508ab700 1 -- 172.21.15.151:0/251206118 shutdown_connections 2024-04-16T03:09:30.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.592+0000 7faa508ab700 1 --2- 172.21.15.151:0/251206118 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 0x7faa4c0ff3f0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.592+0000 7faa508ab700 1 --2- 172.21.15.151:0/251206118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c105b50 0x7faa4c105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.592+0000 7faa508ab700 1 -- 172.21.15.151:0/251206118 >> 172.21.15.151:0/251206118 conn(0x7faa4c075e30 msgr2=0x7faa4c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:30.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.592+0000 7faa508ab700 1 -- 172.21.15.151:0/251206118 shutdown_connections 2024-04-16T03:09:30.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.592+0000 7faa508ab700 1 -- 172.21.15.151:0/251206118 wait complete. 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.593+0000 7faa508ab700 1 Processor -- start 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.593+0000 7faa508ab700 1 -- start start 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.593+0000 7faa508ab700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 0x7faa4c19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.593+0000 7faa508ab700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c105b50 0x7faa4c19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.593+0000 7faa508ab700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c19d140 con 0x7faa4c0fef70 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.593+0000 7faa508ab700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c1965f0 con 0x7faa4c105b50 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.593+0000 7faa4a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 0x7faa4c19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.593+0000 7faa4a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 0x7faa4c19c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54776/0 (socket says 172.21.15.151:54776) 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.593+0000 7faa4a59c700 1 -- 172.21.15.151:0/370871113 learned_addr learned my addr 172.21.15.151:0/370871113 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa43fff700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c105b50 0x7faa4c19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa4a59c700 1 -- 172.21.15.151:0/370871113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c105b50 msgr2=0x7faa4c19cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa4a59c700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c105b50 0x7faa4c19cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa4a59c700 1 -- 172.21.15.151:0/370871113 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa340095f0 con 0x7faa4c0fef70 2024-04-16T03:09:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa4a59c700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 0x7faa4c19c570 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00d8d0 tx=0x7faa3c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:30.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa437fe700 1 -- 172.21.15.151:0/370871113 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa3c009940 con 0x7faa4c0fef70 2024-04-16T03:09:30.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4c1968d0 con 0x7faa4c0fef70 2024-04-16T03:09:30.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa437fe700 1 -- 172.21.15.151:0/370871113 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa3c010460 con 0x7faa4c0fef70 2024-04-16T03:09:30.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa437fe700 1 -- 172.21.15.151:0/370871113 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa3c00f5f0 con 0x7faa4c0fef70 2024-04-16T03:09:30.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.594+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4c196e20 con 0x7faa4c0fef70 2024-04-16T03:09:30.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.595+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4c1090d0 con 0x7faa4c0fef70 2024-04-16T03:09:30.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.596+0000 7faa437fe700 1 -- 172.21.15.151:0/370871113 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7faa3c0105d0 con 0x7faa4c0fef70 2024-04-16T03:09:30.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.596+0000 7faa437fe700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa3806ecc0 0x7faa38071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:09:30.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.596+0000 7faa437fe700 1 -- 172.21.15.151:0/370871113 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faa3c08ee80 con 0x7faa4c0fef70 2024-04-16T03:09:30.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.596+0000 7faa43fff700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa3806ecc0 0x7faa38071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:09:30.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.597+0000 7faa43fff700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa3806ecc0 0x7faa38071180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7faa34005b40 tx=0x7faa3401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:09:30.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.601+0000 7faa437fe700 1 -- 172.21.15.151:0/370871113 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7faa3c05b760 con 0x7faa4c0fef70 2024-04-16T03:09:30.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.882+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faa4c197040 con 0x7faa4c0fef70 2024-04-16T03:09:30.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.882+0000 7faa437fe700 1 -- 172.21.15.151:0/370871113 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7faa3c05b2f0 con 0x7faa4c0fef70 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:09:30.885 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:09:30.886 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:09:30.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.884+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa3806ecc0 msgr2=0x7faa38071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.884+0000 7faa508ab700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa3806ecc0 0x7faa38071180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7faa34005b40 tx=0x7faa3401a040 comp rx=0 tx=0).stop 2024-04-16T03:09:30.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.884+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 msgr2=0x7faa4c19c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:09:30.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.884+0000 7faa508ab700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 0x7faa4c19c570 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00d8d0 tx=0x7faa3c00dc90 comp rx=0 tx=0).stop 2024-04-16T03:09:30.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.884+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 shutdown_connections 2024-04-16T03:09:30.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.884+0000 7faa508ab700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7faa3806ecc0 0x7faa38071180 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.884+0000 7faa508ab700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faa4c0fef70 0x7faa4c19c570 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.884+0000 7faa508ab700 1 --2- 172.21.15.151:0/370871113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c105b50 0x7faa4c19cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:09:30.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.884+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 >> 172.21.15.151:0/370871113 conn(0x7faa4c075e30 msgr2=0x7faa4c104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:09:30.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.885+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 shutdown_connections 2024-04-16T03:09:30.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:09:30.885+0000 7faa508ab700 1 -- 172.21.15.151:0/370871113 wait complete. 2024-04-16T03:09:31.143 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:31 smithi151 ceph-mon[112233]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:09:31.143 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:31 smithi151 ceph-mon[112233]: pgmap v927: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:31.143 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:31 smithi151 ceph-mon[112233]: from='client.35696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:09:31.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:31 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2523713743' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:09:31.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:31 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/370871113' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:09:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:31 smithi153 ceph-mon[87858]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:09:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:31 smithi153 ceph-mon[87858]: pgmap v927: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:31 smithi153 ceph-mon[87858]: from='client.35696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:09:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:31 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2523713743' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:09:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:31 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/370871113' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:09:32.315 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:32 smithi151 ceph-mon[112233]: from='client.35708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:09:32.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:32 smithi153 ceph-mon[87858]: from='client.35708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:09:33.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:33 smithi151 ceph-mon[112233]: pgmap v928: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:33.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:33 smithi153 ceph-mon[87858]: pgmap v928: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:35.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:35 smithi151 ceph-mon[112233]: pgmap v929: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:35.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:35 smithi153 ceph-mon[87858]: pgmap v929: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:36 smithi151 ceph-mon[112233]: pgmap v930: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:36 smithi153 ceph-mon[87858]: pgmap v930: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:38 smithi151 ceph-mon[112233]: pgmap v931: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:39.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:38 smithi153 ceph-mon[87858]: pgmap v931: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:39.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:09:40.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:09:40.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:40 smithi151 ceph-mon[112233]: pgmap v932: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:41.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:40 smithi153 ceph-mon[87858]: pgmap v932: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:42.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:42 smithi151 ceph-mon[112233]: pgmap v933: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:43.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:42 smithi153 ceph-mon[87858]: pgmap v933: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:44.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:44 smithi151 ceph-mon[112233]: pgmap v934: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:45.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:44 smithi153 ceph-mon[87858]: pgmap v934: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:46.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:46 smithi151 ceph-mon[112233]: pgmap v935: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:47.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:46 smithi153 ceph-mon[87858]: pgmap v935: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:48.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:48 smithi151 ceph-mon[112233]: pgmap v936: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:49.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:48 smithi153 ceph-mon[87858]: pgmap v936: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:50.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:50 smithi153 ceph-mon[87858]: pgmap v937: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:50.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:50 smithi151 ceph-mon[112233]: pgmap v937: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:52.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:52 smithi151 ceph-mon[112233]: pgmap v938: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:53.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:52 smithi153 ceph-mon[87858]: pgmap v938: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:54 smithi151 ceph-mon[112233]: pgmap v939: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:09:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:54 smithi153 ceph-mon[87858]: pgmap v939: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:09:56.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:56 smithi151 ceph-mon[112233]: pgmap v940: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:57.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:56 smithi153 ceph-mon[87858]: pgmap v940: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:09:58 smithi151 ceph-mon[112233]: pgmap v941: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:09:59.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:09:58 smithi153 ceph-mon[87858]: pgmap v941: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:00.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:10:00.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:10:00.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : return _run_code(code, main_globals, None, 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:10:00.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:10:00.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:10:00.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:10:00.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:00.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:10:00.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-16T03:10:00.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:10:00.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:09:59.999+0000 7f7be3ae3700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-16T03:10:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: pgmap v942: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:10:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:10:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: /usr/bin/podman: timeout after 895 seconds 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: Traceback (most recent call last): 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: return _run_code(code, main_globals, None, 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: exec(code, run_globals) 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:10:00.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:10:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:10:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:10:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:10:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:10:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:10:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: fs cephfs is degraded 2024-04-16T03:10:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:10:00.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:00 smithi151 ceph-mon[112233]: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:10:01.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.003+0000 7fa4467dc700 1 -- 172.21.15.151:0/2610596984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4400ffbd0 msgr2=0x7fa440100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.003+0000 7fa4467dc700 1 --2- 172.21.15.151:0/2610596984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4400ffbd0 0x7fa440100050 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa434009910 tx=0x7fa434009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:01.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.003+0000 7fa4467dc700 1 -- 172.21.15.151:0/2610596984 shutdown_connections 2024-04-16T03:10:01.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.003+0000 7fa4467dc700 1 --2- 172.21.15.151:0/2610596984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa4400ffbd0 0x7fa440100050 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.003+0000 7fa4467dc700 1 --2- 172.21.15.151:0/2610596984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440105a40 0x7fa440105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.003+0000 7fa4467dc700 1 -- 172.21.15.151:0/2610596984 >> 172.21.15.151:0/2610596984 conn(0x7fa4400755a0 msgr2=0x7fa4400759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:01.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.003+0000 7fa4467dc700 1 -- 172.21.15.151:0/2610596984 shutdown_connections 2024-04-16T03:10:01.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.003+0000 7fa4467dc700 1 -- 172.21.15.151:0/2610596984 wait complete. 2024-04-16T03:10:01.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.004+0000 7fa4467dc700 1 Processor -- start 2024-04-16T03:10:01.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.004+0000 7fa4467dc700 1 -- start start 2024-04-16T03:10:01.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.004+0000 7fa4467dc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400ffbd0 0x7fa440195380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:01.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.004+0000 7fa4467dc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa440105a40 0x7fa4401958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:01.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.004+0000 7fa4467dc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa440195e00 con 0x7fa440105a40 2024-04-16T03:10:01.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.004+0000 7fa4467dc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa440195f70 con 0x7fa4400ffbd0 2024-04-16T03:10:01.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.004+0000 7fa43f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa440105a40 0x7fa4401958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:01.008 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.004+0000 7fa43f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa440105a40 0x7fa4401958c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49074/0 (socket says 172.21.15.151:49074) 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa43f7fe700 1 -- 172.21.15.151:0/1466881806 learned_addr learned my addr 172.21.15.151:0/1466881806 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa43ffff700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400ffbd0 0x7fa440195380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa43f7fe700 1 -- 172.21.15.151:0/1466881806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400ffbd0 msgr2=0x7fa440195380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa43f7fe700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400ffbd0 0x7fa440195380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa43f7fe700 1 -- 172.21.15.151:0/1466881806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4340095f0 con 0x7fa440105a40 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa43f7fe700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa440105a40 0x7fa4401958c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa434006010 tx=0x7fa43400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa43d7fa700 1 -- 172.21.15.151:0/1466881806 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa43401d070 con 0x7fa440105a40 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa43d7fa700 1 -- 172.21.15.151:0/1466881806 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa43400f460 con 0x7fa440105a40 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa43d7fa700 1 -- 172.21.15.151:0/1466881806 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa434017630 con 0x7fa440105a40 2024-04-16T03:10:01.009 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4401961f0 con 0x7fa440105a40 2024-04-16T03:10:01.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.005+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4401a2c20 con 0x7fa440105a40 2024-04-16T03:10:01.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.006+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa440107fd0 con 0x7fa440105a40 2024-04-16T03:10:01.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.007+0000 7fa43d7fa700 1 -- 172.21.15.151:0/1466881806 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa43400f5d0 con 0x7fa440105a40 2024-04-16T03:10:01.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.007+0000 7fa43d7fa700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa42806ecc0 0x7fa428071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:01.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.008+0000 7fa43ffff700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa42806ecc0 0x7fa428071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:01.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.008+0000 7fa43d7fa700 1 -- 172.21.15.151:0/1466881806 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa434090270 con 0x7fa440105a40 2024-04-16T03:10:01.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.008+0000 7fa43ffff700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa42806ecc0 0x7fa428071180 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa430005950 tx=0x7fa4300058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:01.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.012+0000 7fa43d7fa700 1 -- 172.21.15.151:0/1466881806 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa434058f80 con 0x7fa440105a40 2024-04-16T03:10:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: pgmap v942: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:01.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: /usr/bin/podman: timeout after 895 seconds 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: Traceback (most recent call last): 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: return _run_code(code, main_globals, None, 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: exec(code, run_globals) 2024-04-16T03:10:01.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:10:01.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: fs cephfs is degraded 2024-04-16T03:10:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:10:01.024 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:00 smithi153 ceph-mon[87858]: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:10:01.231 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.230+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4401961f0 con 0x7fa42806ecc0 2024-04-16T03:10:01.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.231+0000 7fa43d7fa700 1 -- 172.21.15.151:0/1466881806 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa4401961f0 con 0x7fa42806ecc0 2024-04-16T03:10:01.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.233+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa42806ecc0 msgr2=0x7fa428071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.233+0000 7fa4467dc700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa42806ecc0 0x7fa428071180 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa430005950 tx=0x7fa4300058e0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.233+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa440105a40 msgr2=0x7fa4401958c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.233+0000 7fa4467dc700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa440105a40 0x7fa4401958c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa434006010 tx=0x7fa43400be20 comp rx=0 tx=0).stop 2024-04-16T03:10:01.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.233+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 shutdown_connections 2024-04-16T03:10:01.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.233+0000 7fa4467dc700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa42806ecc0 0x7fa428071180 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.233+0000 7fa4467dc700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400ffbd0 0x7fa440195380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.233+0000 7fa4467dc700 1 --2- 172.21.15.151:0/1466881806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa440105a40 0x7fa4401958c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.233+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 >> 172.21.15.151:0/1466881806 conn(0x7fa4400755a0 msgr2=0x7fa4400fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:01.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.234+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 shutdown_connections 2024-04-16T03:10:01.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.234+0000 7fa4467dc700 1 -- 172.21.15.151:0/1466881806 wait complete. 2024-04-16T03:10:01.245 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:10:01.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.340+0000 7f408acaf700 1 -- 172.21.15.151:0/3596555906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 msgr2=0x7f40841011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.340+0000 7f408acaf700 1 --2- 172.21.15.151:0/3596555906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 0x7f40841011a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f406c009910 tx=0x7f406c009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:01.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.340+0000 7f408acaf700 1 -- 172.21.15.151:0/3596555906 shutdown_connections 2024-04-16T03:10:01.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.340+0000 7f408acaf700 1 --2- 172.21.15.151:0/3596555906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4084101770 0x7f40841057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.340+0000 7f408acaf700 1 --2- 172.21.15.151:0/3596555906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 0x7f40841011a0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.340+0000 7f408acaf700 1 -- 172.21.15.151:0/3596555906 >> 172.21.15.151:0/3596555906 conn(0x7f40840fc650 msgr2=0x7f40840fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:01.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.340+0000 7f408acaf700 1 -- 172.21.15.151:0/3596555906 shutdown_connections 2024-04-16T03:10:01.342 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.340+0000 7f408acaf700 1 -- 172.21.15.151:0/3596555906 wait complete. 2024-04-16T03:10:01.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.341+0000 7f408acaf700 1 Processor -- start 2024-04-16T03:10:01.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.341+0000 7f408acaf700 1 -- start start 2024-04-16T03:10:01.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f408acaf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 0x7f408419c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:01.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f408acaf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4084101770 0x7f408419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:01.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f408acaf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408419d110 con 0x7f4084100dc0 2024-04-16T03:10:01.344 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f408acaf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40841965c0 con 0x7f4084101770 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4088a4b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 0x7f408419c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4088a4b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 0x7f408419c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49092/0 (socket says 172.21.15.151:49092) 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4088a4b700 1 -- 172.21.15.151:0/264480545 learned_addr learned my addr 172.21.15.151:0/264480545 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4083fff700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4084101770 0x7f408419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4088a4b700 1 -- 172.21.15.151:0/264480545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4084101770 msgr2=0x7f408419ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4088a4b700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4084101770 0x7f408419ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4088a4b700 1 -- 172.21.15.151:0/264480545 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0095f0 con 0x7f4084100dc0 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4088a4b700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 0x7f408419c540 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f406c006010 tx=0x7f406c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4081ffb700 1 -- 172.21.15.151:0/264480545 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f406c01d070 con 0x7f4084100dc0 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4081ffb700 1 -- 172.21.15.151:0/264480545 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f406c005dc0 con 0x7f4084100dc0 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f4081ffb700 1 -- 172.21.15.151:0/264480545 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f406c017400 con 0x7f4084100dc0 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.342+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4084196840 con 0x7f4084100dc0 2024-04-16T03:10:01.345 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.343+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4084196d30 con 0x7f4084100dc0 2024-04-16T03:10:01.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.344+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4084109160 con 0x7f4084100dc0 2024-04-16T03:10:01.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.344+0000 7f4081ffb700 1 -- 172.21.15.151:0/264480545 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f406c003a60 con 0x7f4084100dc0 2024-04-16T03:10:01.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.345+0000 7f4081ffb700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f40700730d0 0x7f4070075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:01.346 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.345+0000 7f4081ffb700 1 -- 172.21.15.151:0/264480545 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f406c090200 con 0x7f4084100dc0 2024-04-16T03:10:01.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.345+0000 7f4083fff700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f40700730d0 0x7f4070075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:01.347 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.346+0000 7f4083fff700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f40700730d0 0x7f4070075590 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4074009550 tx=0x7f40740091f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:01.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.349+0000 7f4081ffb700 1 -- 172.21.15.151:0/264480545 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f406c05cb60 con 0x7f4084100dc0 2024-04-16T03:10:01.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.565+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f408402ccc0 con 0x7f40700730d0 2024-04-16T03:10:01.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.566+0000 7f4081ffb700 1 -- 172.21.15.151:0/264480545 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f408402ccc0 con 0x7f40700730d0 2024-04-16T03:10:01.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f40700730d0 msgr2=0x7f4070075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f40700730d0 0x7f4070075590 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4074009550 tx=0x7f40740091f0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 msgr2=0x7f408419c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 0x7f408419c540 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f406c006010 tx=0x7f406c00bd70 comp rx=0 tx=0).stop 2024-04-16T03:10:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 shutdown_connections 2024-04-16T03:10:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f40700730d0 0x7f4070075590 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4084100dc0 0x7f408419c540 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 --2- 172.21.15.151:0/264480545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4084101770 0x7f408419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 >> 172.21.15.151:0/264480545 conn(0x7f40840fc650 msgr2=0x7f40840fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 shutdown_connections 2024-04-16T03:10:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.569+0000 7f408acaf700 1 -- 172.21.15.151:0/264480545 wait complete. 2024-04-16T03:10:01.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.674+0000 7fa19508e700 1 -- 172.21.15.151:0/1295804667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 msgr2=0x7fa1901057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.674+0000 7fa19508e700 1 --2- 172.21.15.151:0/1295804667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 0x7fa1901057c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fa184009910 tx=0x7fa184009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:01.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.674+0000 7fa19508e700 1 -- 172.21.15.151:0/1295804667 shutdown_connections 2024-04-16T03:10:01.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.674+0000 7fa19508e700 1 --2- 172.21.15.151:0/1295804667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 0x7fa1901057c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.674+0000 7fa19508e700 1 --2- 172.21.15.151:0/1295804667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa190100dc0 0x7fa1901011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.674+0000 7fa19508e700 1 -- 172.21.15.151:0/1295804667 >> 172.21.15.151:0/1295804667 conn(0x7fa1900fc650 msgr2=0x7fa1900fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:01.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.674+0000 7fa19508e700 1 -- 172.21.15.151:0/1295804667 shutdown_connections 2024-04-16T03:10:01.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.674+0000 7fa19508e700 1 -- 172.21.15.151:0/1295804667 wait complete. 2024-04-16T03:10:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.675+0000 7fa19508e700 1 Processor -- start 2024-04-16T03:10:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.675+0000 7fa19508e700 1 -- start start 2024-04-16T03:10:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.675+0000 7fa19508e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa190100dc0 0x7fa19019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.675+0000 7fa19508e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 0x7fa19019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.675+0000 7fa19508e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19019d240 con 0x7fa190101770 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.675+0000 7fa19508e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901966f0 con 0x7fa190100dc0 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.675+0000 7fa18e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 0x7fa19019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa18e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 0x7fa19019cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49120/0 (socket says 172.21.15.151:49120) 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa18e59c700 1 -- 172.21.15.151:0/2477907863 learned_addr learned my addr 172.21.15.151:0/2477907863 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa18ed9d700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa190100dc0 0x7fa19019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa18e59c700 1 -- 172.21.15.151:0/2477907863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa190100dc0 msgr2=0x7fa19019c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa18e59c700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa190100dc0 0x7fa19019c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa18e59c700 1 -- 172.21.15.151:0/2477907863 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1840095f0 con 0x7fa190101770 2024-04-16T03:10:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa18e59c700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 0x7fa19019cbb0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa184005f50 tx=0x7fa18400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa177fff700 1 -- 172.21.15.151:0/2477907863 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa18401d070 con 0x7fa190101770 2024-04-16T03:10:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa190196970 con 0x7fa190101770 2024-04-16T03:10:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa177fff700 1 -- 172.21.15.151:0/2477907863 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa18400f460 con 0x7fa190101770 2024-04-16T03:10:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa177fff700 1 -- 172.21.15.151:0/2477907863 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa184017630 con 0x7fa190101770 2024-04-16T03:10:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.676+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa190196e60 con 0x7fa190101770 2024-04-16T03:10:01.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.677+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa190109110 con 0x7fa190101770 2024-04-16T03:10:01.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.678+0000 7fa177fff700 1 -- 172.21.15.151:0/2477907863 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa18400f5d0 con 0x7fa190101770 2024-04-16T03:10:01.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.679+0000 7fa177fff700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa17806ecc0 0x7fa178071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:01.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.679+0000 7fa177fff700 1 -- 172.21.15.151:0/2477907863 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa1840903c0 con 0x7fa190101770 2024-04-16T03:10:01.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.679+0000 7fa18ed9d700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa17806ecc0 0x7fa178071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:01.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.679+0000 7fa18ed9d700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa17806ecc0 0x7fa178071180 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa180009a10 tx=0x7fa180009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:01.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.683+0000 7fa177fff700 1 -- 172.21.15.151:0/2477907863 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa18405cc80 con 0x7fa190101770 2024-04-16T03:10:01.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.891+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa19002cce0 con 0x7fa17806ecc0 2024-04-16T03:10:01.903 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.901+0000 7fa177fff700 1 -- 172.21.15.151:0/2477907863 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa19002cce0 con 0x7fa17806ecc0 2024-04-16T03:10:01.903 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:10:01.903 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (31m) 9m ago 42m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:10:01.903 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (43m) 9m ago 43m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:10:01.903 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (42m) 9m ago 42m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:10:01.903 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (29m) 9m ago 43m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:10:01.903 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (29m) 9m ago 42m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (31m) 9m ago 42m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (25m) 9m ago 37m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (25m) 9m ago 37m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (9m) 9m ago 37m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (24m) 9m ago 37m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (33m) 9m ago 44m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (32m) 9m ago 41m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (30m) 9m ago 44m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (29m) 9m ago 41m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (32m) 9m ago 43m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (32m) 9m ago 41m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:10:01.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (28m) 9m ago 41m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:10:01.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (28m) 9m ago 40m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:10:01.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (27m) 9m ago 40m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:10:01.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (26m) 9m ago 39m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:10:01.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (26m) 9m ago 39m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:10:01.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (25m) 9m ago 38m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:10:01.905 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (31m) 9m ago 42m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:10:01.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa17806ecc0 msgr2=0x7fa178071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa17806ecc0 0x7fa178071180 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa180009a10 tx=0x7fa180009190 comp rx=0 tx=0).stop 2024-04-16T03:10:01.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 msgr2=0x7fa19019cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:01.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 0x7fa19019cbb0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa184005f50 tx=0x7fa18400bd70 comp rx=0 tx=0).stop 2024-04-16T03:10:01.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 shutdown_connections 2024-04-16T03:10:01.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa17806ecc0 0x7fa178071180 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa190100dc0 0x7fa19019c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 --2- 172.21.15.151:0/2477907863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa190101770 0x7fa19019cbb0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:01.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 >> 172.21.15.151:0/2477907863 conn(0x7fa1900fc650 msgr2=0x7fa1900fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:01.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 shutdown_connections 2024-04-16T03:10:01.907 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:01.904+0000 7fa19508e700 1 -- 172.21.15.151:0/2477907863 wait complete. 2024-04-16T03:10:02.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.007+0000 7f6a70745700 1 -- 172.21.15.151:0/140622447 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 msgr2=0x7f6a68105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.007+0000 7f6a70745700 1 --2- 172.21.15.151:0/140622447 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 0x7f6a68105790 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009910 tx=0x7f6a5c009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:02.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.007+0000 7f6a70745700 1 -- 172.21.15.151:0/140622447 shutdown_connections 2024-04-16T03:10:02.010 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.007+0000 7f6a70745700 1 --2- 172.21.15.151:0/140622447 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 0x7f6a68105790 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.007+0000 7f6a70745700 1 --2- 172.21.15.151:0/140622447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68068670 0x7f6a68068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.007+0000 7f6a70745700 1 -- 172.21.15.151:0/140622447 >> 172.21.15.151:0/140622447 conn(0x7f6a680755a0 msgr2=0x7f6a680759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.007+0000 7f6a70745700 1 -- 172.21.15.151:0/140622447 shutdown_connections 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.007+0000 7f6a70745700 1 -- 172.21.15.151:0/140622447 wait complete. 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a70745700 1 Processor -- start 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a70745700 1 -- start start 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a70745700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68068670 0x7f6a680ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a70745700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 0x7f6a681001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a70745700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68100720 con 0x7f6a68069020 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a70745700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68100860 con 0x7f6a68068670 2024-04-16T03:10:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a6dce0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 0x7f6a681001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a6dce0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 0x7f6a681001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49146/0 (socket says 172.21.15.151:49146) 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a6dce0700 1 -- 172.21.15.151:0/3902484084 learned_addr learned my addr 172.21.15.151:0/3902484084 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a6e4e1700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68068670 0x7f6a680ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a6dce0700 1 -- 172.21.15.151:0/3902484084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68068670 msgr2=0x7f6a680ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a6dce0700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68068670 0x7f6a680ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a6dce0700 1 -- 172.21.15.151:0/3902484084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c0095f0 con 0x7f6a68069020 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.008+0000 7f6a6dce0700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 0x7f6a681001e0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0059d0 tx=0x7f6a5c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.009+0000 7f6a677fe700 1 -- 172.21.15.151:0/3902484084 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a5c01d070 con 0x7f6a68069020 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.009+0000 7f6a677fe700 1 -- 172.21.15.151:0/3902484084 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a5c00b610 con 0x7f6a68069020 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.009+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a68100ae0 con 0x7f6a68069020 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.009+0000 7f6a677fe700 1 -- 172.21.15.151:0/3902484084 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a5c022470 con 0x7f6a68069020 2024-04-16T03:10:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.009+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a681a2c00 con 0x7f6a68069020 2024-04-16T03:10:02.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.010+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a681090a0 con 0x7f6a68069020 2024-04-16T03:10:02.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.010+0000 7f6a677fe700 1 -- 172.21.15.151:0/3902484084 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6a5c003710 con 0x7f6a68069020 2024-04-16T03:10:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.011+0000 7f6a677fe700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a5006ecc0 0x7f6a50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.011+0000 7f6a677fe700 1 -- 172.21.15.151:0/3902484084 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6a5c0901f0 con 0x7f6a68069020 2024-04-16T03:10:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.011+0000 7f6a6e4e1700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a5006ecc0 0x7f6a50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.011+0000 7f6a6e4e1700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a5006ecc0 0x7f6a50071180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009c60 tx=0x7f6a60009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:02.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.015+0000 7f6a677fe700 1 -- 172.21.15.151:0/3902484084 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6a5c059020 con 0x7f6a68069020 2024-04-16T03:10:02.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.296+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6a6804ea90 con 0x7f6a68069020 2024-04-16T03:10:02.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.297+0000 7f6a677fe700 1 -- 172.21.15.151:0/3902484084 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6a5c05c6e0 con 0x7f6a68069020 2024-04-16T03:10:02.298 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:10:02.299 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:10:02.300 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:10:02.300 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:10:02.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a5006ecc0 msgr2=0x7f6a50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a5006ecc0 0x7f6a50071180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009c60 tx=0x7f6a60009260 comp rx=0 tx=0).stop 2024-04-16T03:10:02.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 msgr2=0x7f6a681001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 0x7f6a681001e0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0059d0 tx=0x7f6a5c00f690 comp rx=0 tx=0).stop 2024-04-16T03:10:02.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 shutdown_connections 2024-04-16T03:10:02.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6a5006ecc0 0x7f6a50071180 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68068670 0x7f6a680ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 --2- 172.21.15.151:0/3902484084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a68069020 0x7f6a681001e0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 >> 172.21.15.151:0/3902484084 conn(0x7f6a680755a0 msgr2=0x7f6a68105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:02.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 shutdown_connections 2024-04-16T03:10:02.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.300+0000 7f6a70745700 1 -- 172.21.15.151:0/3902484084 wait complete. 2024-04-16T03:10:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.404+0000 7fefd32c8700 1 -- 172.21.15.151:0/2626974013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 msgr2=0x7fefcc100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.404+0000 7fefd32c8700 1 --2- 172.21.15.151:0/2626974013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 0x7fefcc100860 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fefc8009910 tx=0x7fefc8009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.404+0000 7fefd32c8700 1 -- 172.21.15.151:0/2626974013 shutdown_connections 2024-04-16T03:10:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.404+0000 7fefd32c8700 1 --2- 172.21.15.151:0/2626974013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 0x7fefcc100860 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.404+0000 7fefd32c8700 1 --2- 172.21.15.151:0/2626974013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefcc104ae0 0x7fefcc104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.404+0000 7fefd32c8700 1 -- 172.21.15.151:0/2626974013 >> 172.21.15.151:0/2626974013 conn(0x7fefcc074df0 msgr2=0x7fefcc075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.404+0000 7fefd32c8700 1 -- 172.21.15.151:0/2626974013 shutdown_connections 2024-04-16T03:10:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.404+0000 7fefd32c8700 1 -- 172.21.15.151:0/2626974013 wait complete. 2024-04-16T03:10:02.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd32c8700 1 Processor -- start 2024-04-16T03:10:02.408 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd32c8700 1 -- start start 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd32c8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 0x7fefcc197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd32c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefcc104ae0 0x7fefcc197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd32c8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc198170 con 0x7fefcc1003e0 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd32c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc1982e0 con 0x7fefcc104ae0 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd1064700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 0x7fefcc197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd1064700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 0x7fefcc197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49170/0 (socket says 172.21.15.151:49170) 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd1064700 1 -- 172.21.15.151:0/2628581070 learned_addr learned my addr 172.21.15.151:0/2628581070 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.405+0000 7fefd0863700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefcc104ae0 0x7fefcc197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.406+0000 7fefd1064700 1 -- 172.21.15.151:0/2628581070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefcc104ae0 msgr2=0x7fefcc197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.406+0000 7fefd1064700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefcc104ae0 0x7fefcc197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.406+0000 7fefd1064700 1 -- 172.21.15.151:0/2628581070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc80095f0 con 0x7fefcc1003e0 2024-04-16T03:10:02.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.406+0000 7fefd1064700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 0x7fefcc197610 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fefc000d8d0 tx=0x7fefc000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:02.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.406+0000 7fefbe7fc700 1 -- 172.21.15.151:0/2628581070 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefc00098a0 con 0x7fefcc1003e0 2024-04-16T03:10:02.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.406+0000 7fefbe7fc700 1 -- 172.21.15.151:0/2628581070 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefc0010460 con 0x7fefcc1003e0 2024-04-16T03:10:02.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.406+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc1a0680 con 0x7fefcc1003e0 2024-04-16T03:10:02.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.406+0000 7fefbe7fc700 1 -- 172.21.15.151:0/2628581070 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefc000f5f0 con 0x7fefcc1003e0 2024-04-16T03:10:02.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.406+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc1a0b70 con 0x7fefcc1003e0 2024-04-16T03:10:02.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.407+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc04ea90 con 0x7fefcc1003e0 2024-04-16T03:10:02.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.407+0000 7fefbe7fc700 1 -- 172.21.15.151:0/2628581070 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fefc0009a00 con 0x7fefcc1003e0 2024-04-16T03:10:02.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.408+0000 7fefbe7fc700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefb806ecc0 0x7fefb8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:02.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.408+0000 7fefbe7fc700 1 -- 172.21.15.151:0/2628581070 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fefc008ec30 con 0x7fefcc1003e0 2024-04-16T03:10:02.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.408+0000 7fefd0863700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefb806ecc0 0x7fefb8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:02.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.408+0000 7fefd0863700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefb806ecc0 0x7fefb8071180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fefcc0ffba0 tx=0x7fefc801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:02.414 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.413+0000 7fefbe7fc700 1 -- 172.21.15.151:0/2628581070 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fefc0057a60 con 0x7fefcc1003e0 2024-04-16T03:10:02.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.658+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fefcc1a0f70 con 0x7fefcc1003e0 2024-04-16T03:10:02.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.659+0000 7fefbe7fc700 1 -- 172.21.15.151:0/2628581070 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fefc0016770 con 0x7fefcc1003e0 2024-04-16T03:10:02.660 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:10:02.660 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:10:02.661 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:10:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:10:02.663 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:10:02.664 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:10:02.664 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:10:02.664 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:10:02.664 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:10:02.664 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:10:02.664 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:10:02.664 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:10:02.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefb806ecc0 msgr2=0x7fefb8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefb806ecc0 0x7fefb8071180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fefcc0ffba0 tx=0x7fefc801a040 comp rx=0 tx=0).stop 2024-04-16T03:10:02.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 msgr2=0x7fefcc197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 0x7fefcc197610 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fefc000d8d0 tx=0x7fefc000dbe0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 shutdown_connections 2024-04-16T03:10:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fefb806ecc0 0x7fefb8071180 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefcc1003e0 0x7fefcc197610 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 --2- 172.21.15.151:0/2628581070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefcc104ae0 0x7fefcc197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 >> 172.21.15.151:0/2628581070 conn(0x7fefcc074df0 msgr2=0x7fefcc0fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 shutdown_connections 2024-04-16T03:10:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.661+0000 7fefd32c8700 1 -- 172.21.15.151:0/2628581070 wait complete. 2024-04-16T03:10:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:10:02.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.766+0000 7f973b022700 1 -- 172.21.15.151:0/1352103464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073a50 msgr2=0x7f9734111630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.766+0000 7f973b022700 1 --2- 172.21.15.151:0/1352103464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073a50 0x7f9734111630 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f9724009910 tx=0x7f9724009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:02.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.766+0000 7f973b022700 1 -- 172.21.15.151:0/1352103464 shutdown_connections 2024-04-16T03:10:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.766+0000 7f973b022700 1 --2- 172.21.15.151:0/1352103464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073a50 0x7f9734111630 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.766+0000 7f973b022700 1 --2- 172.21.15.151:0/1352103464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9734073130 0x7f9734073510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.766+0000 7f973b022700 1 -- 172.21.15.151:0/1352103464 >> 172.21.15.151:0/1352103464 conn(0x7f97340fc650 msgr2=0x7f97340fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.766+0000 7f973b022700 1 -- 172.21.15.151:0/1352103464 shutdown_connections 2024-04-16T03:10:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.766+0000 7f973b022700 1 -- 172.21.15.151:0/1352103464 wait complete. 2024-04-16T03:10:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f973b022700 1 Processor -- start 2024-04-16T03:10:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f973b022700 1 -- start start 2024-04-16T03:10:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f973b022700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073130 0x7f973419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f973b022700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9734073a50 0x7f973419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f973b022700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973419da20 con 0x7f9734073130 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f973b022700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97341a17b0 con 0x7f9734073a50 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f9738dbe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073130 0x7f973419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f9738dbe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073130 0x7f973419ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49180/0 (socket says 172.21.15.151:49180) 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f9738dbe700 1 -- 172.21.15.151:0/1475953 learned_addr learned my addr 172.21.15.151:0/1475953 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f972bfff700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9734073a50 0x7f973419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f9738dbe700 1 -- 172.21.15.151:0/1475953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9734073a50 msgr2=0x7f973419d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f9738dbe700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9734073a50 0x7f973419d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.767+0000 7f9738dbe700 1 -- 172.21.15.151:0/1475953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97240095f0 con 0x7f9734073130 2024-04-16T03:10:02.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.768+0000 7f9738dbe700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073130 0x7f973419ce00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f973000ba50 tx=0x7f973000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:02.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.768+0000 7f9729ffb700 1 -- 172.21.15.151:0/1475953 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f973000ca00 con 0x7f9734073130 2024-04-16T03:10:02.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.768+0000 7f9729ffb700 1 -- 172.21.15.151:0/1475953 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f973000cb60 con 0x7f9734073130 2024-04-16T03:10:02.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.768+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97341a1a90 con 0x7f9734073130 2024-04-16T03:10:02.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.768+0000 7f9729ffb700 1 -- 172.21.15.151:0/1475953 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97300075a0 con 0x7f9734073130 2024-04-16T03:10:02.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.768+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97341a1fe0 con 0x7f9734073130 2024-04-16T03:10:02.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.769+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973410edb0 con 0x7f9734073130 2024-04-16T03:10:02.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.770+0000 7f9729ffb700 1 -- 172.21.15.151:0/1475953 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f973000ccd0 con 0x7f9734073130 2024-04-16T03:10:02.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.770+0000 7f9729ffb700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f971c06ecc0 0x7f971c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:02.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.770+0000 7f9729ffb700 1 -- 172.21.15.151:0/1475953 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9730006e40 con 0x7f9734073130 2024-04-16T03:10:02.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.770+0000 7f972bfff700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f971c06ecc0 0x7f971c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:02.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.771+0000 7f972bfff700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f971c06ecc0 0x7f971c071180 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f97240059d0 tx=0x7f972401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:02.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.775+0000 7f9729ffb700 1 -- 172.21.15.151:0/1475953 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9730057410 con 0x7f9734073130 2024-04-16T03:10:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:02 smithi151 ceph-mon[112233]: from='client.35716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:02 smithi151 ceph-mon[112233]: pgmap v943: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:02 smithi151 ceph-mon[112233]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:02 smithi151 ceph-mon[112233]: from='client.35724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:02 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3902484084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:10:02.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.997+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97341a2360 con 0x7f971c06ecc0 2024-04-16T03:10:03.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:02.998+0000 7f9729ffb700 1 -- 172.21.15.151:0/1475953 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f97341a2360 con 0x7f971c06ecc0 2024-04-16T03:10:03.000 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:10:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:10:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:10:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:10:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:10:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:10:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:10:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:10:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:10:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:10:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:10:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:10:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:10:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:10:03.002 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:10:03.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f971c06ecc0 msgr2=0x7f971c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:03.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f971c06ecc0 0x7f971c071180 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f97240059d0 tx=0x7f972401a040 comp rx=0 tx=0).stop 2024-04-16T03:10:03.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073130 msgr2=0x7f973419ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:03.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073130 0x7f973419ce00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f973000ba50 tx=0x7f973000be10 comp rx=0 tx=0).stop 2024-04-16T03:10:03.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 shutdown_connections 2024-04-16T03:10:03.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f971c06ecc0 0x7f971c071180 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:03.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9734073130 0x7f973419ce00 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:03.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 --2- 172.21.15.151:0/1475953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9734073a50 0x7f973419d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:03.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 >> 172.21.15.151:0/1475953 conn(0x7f97340fc650 msgr2=0x7f9734103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:03.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 shutdown_connections 2024-04-16T03:10:03.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.001+0000 7f973b022700 1 -- 172.21.15.151:0/1475953 wait complete. 2024-04-16T03:10:03.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:02 smithi153 ceph-mon[87858]: from='client.35716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:03.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:02 smithi153 ceph-mon[87858]: pgmap v943: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:03.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:02 smithi153 ceph-mon[87858]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:03.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:02 smithi153 ceph-mon[87858]: from='client.35724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:03.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:02 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3902484084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:10:03.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.107+0000 7f153c3cb700 1 -- 172.21.15.151:0/2792268640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 msgr2=0x7f15340ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:03.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.107+0000 7f153c3cb700 1 --2- 172.21.15.151:0/2792268640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 0x7f15340ffec0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1524009910 tx=0x7f1524009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:03.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.107+0000 7f153c3cb700 1 -- 172.21.15.151:0/2792268640 shutdown_connections 2024-04-16T03:10:03.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.107+0000 7f153c3cb700 1 --2- 172.21.15.151:0/2792268640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 0x7f15340ffec0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:03.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.107+0000 7f153c3cb700 1 --2- 172.21.15.151:0/2792268640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1534105a40 0x7f1534105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:03.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.107+0000 7f153c3cb700 1 -- 172.21.15.151:0/2792268640 >> 172.21.15.151:0/2792268640 conn(0x7f15340fb570 msgr2=0x7f15340fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:03.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.107+0000 7f153c3cb700 1 -- 172.21.15.151:0/2792268640 shutdown_connections 2024-04-16T03:10:03.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.107+0000 7f153c3cb700 1 -- 172.21.15.151:0/2792268640 wait complete. 2024-04-16T03:10:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153c3cb700 1 Processor -- start 2024-04-16T03:10:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153c3cb700 1 -- start start 2024-04-16T03:10:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153c3cb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 0x7f1534197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153c3cb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1534105a40 0x7f1534197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153c3cb700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15341980d0 con 0x7f15340ffa40 2024-04-16T03:10:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153c3cb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1534198240 con 0x7f1534105a40 2024-04-16T03:10:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153a167700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 0x7f1534197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153a167700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 0x7f1534197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49188/0 (socket says 172.21.15.151:49188) 2024-04-16T03:10:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153a167700 1 -- 172.21.15.151:0/1726174714 learned_addr learned my addr 172.21.15.151:0/1726174714 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:03.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f1539966700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1534105a40 0x7f1534197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:03.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153a167700 1 -- 172.21.15.151:0/1726174714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1534105a40 msgr2=0x7f1534197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:03.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153a167700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1534105a40 0x7f1534197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:03.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153a167700 1 -- 172.21.15.151:0/1726174714 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15240095f0 con 0x7f15340ffa40 2024-04-16T03:10:03.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.108+0000 7f153a167700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 0x7f1534197570 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f153000ada0 tx=0x7f153000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:03.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.109+0000 7f152b7fe700 1 -- 172.21.15.151:0/1726174714 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f153000f070 con 0x7f15340ffa40 2024-04-16T03:10:03.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.109+0000 7f152b7fe700 1 -- 172.21.15.151:0/1726174714 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f153000ae90 con 0x7f15340ffa40 2024-04-16T03:10:03.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.109+0000 7f152b7fe700 1 -- 172.21.15.151:0/1726174714 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1530005690 con 0x7f15340ffa40 2024-04-16T03:10:03.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.109+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15341a05e0 con 0x7f15340ffa40 2024-04-16T03:10:03.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.109+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15341a0ad0 con 0x7f15340ffa40 2024-04-16T03:10:03.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.110+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1534107fd0 con 0x7f15340ffa40 2024-04-16T03:10:03.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.110+0000 7f152b7fe700 1 -- 172.21.15.151:0/1726174714 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f15300057f0 con 0x7f15340ffa40 2024-04-16T03:10:03.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.111+0000 7f152b7fe700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f152006ecc0 0x7f1520071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:03.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.111+0000 7f152b7fe700 1 -- 172.21.15.151:0/1726174714 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f153008e6c0 con 0x7f15340ffa40 2024-04-16T03:10:03.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.111+0000 7f1539966700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f152006ecc0 0x7f1520071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:03.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.111+0000 7f1539966700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f152006ecc0 0x7f1520071180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1534069570 tx=0x7f152401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:03.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.115+0000 7f152b7fe700 1 -- 172.21.15.151:0/1726174714 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f15300575a0 con 0x7f15340ffa40 2024-04-16T03:10:03.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.395+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f153404ea90 con 0x7f15340ffa40 2024-04-16T03:10:03.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.396+0000 7f152b7fe700 1 -- 172.21.15.151:0/1726174714 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f153005ac60 con 0x7f15340ffa40 2024-04-16T03:10:03.397 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:10:03.398 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:10:03.399 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:10:03.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f152006ecc0 msgr2=0x7f1520071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f152006ecc0 0x7f1520071180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1534069570 tx=0x7f152401a040 comp rx=0 tx=0).stop 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 msgr2=0x7f1534197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 0x7f1534197570 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f153000ada0 tx=0x7f153000c330 comp rx=0 tx=0).stop 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 shutdown_connections 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f152006ecc0 0x7f1520071180 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15340ffa40 0x7f1534197570 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 --2- 172.21.15.151:0/1726174714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1534105a40 0x7f1534197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 >> 172.21.15.151:0/1726174714 conn(0x7f15340fb570 msgr2=0x7f15340fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 shutdown_connections 2024-04-16T03:10:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:03.398+0000 7f153c3cb700 1 -- 172.21.15.151:0/1726174714 wait complete. 2024-04-16T03:10:03.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:03 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2628581070' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:10:03.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:03 smithi151 ceph-mon[112233]: from='client.35736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:03.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:03 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1726174714' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:10:04.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:03 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2628581070' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:10:04.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:03 smithi153 ceph-mon[87858]: from='client.35736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:04.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:03 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1726174714' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:10:04.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:04 smithi151 ceph-mon[112233]: pgmap v944: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:05.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:04 smithi153 ceph-mon[87858]: pgmap v944: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:06.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:06 smithi151 ceph-mon[112233]: pgmap v945: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:07.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:06 smithi153 ceph-mon[87858]: pgmap v945: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:08.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:08 smithi151 ceph-mon[112233]: pgmap v946: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:09.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:08 smithi153 ceph-mon[87858]: pgmap v946: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:09.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:10:10.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:10:10.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:10 smithi151 ceph-mon[112233]: pgmap v947: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:11.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:10 smithi153 ceph-mon[87858]: pgmap v947: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:12.939 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:12 smithi151 ceph-mon[112233]: pgmap v948: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:13.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:12 smithi153 ceph-mon[87858]: pgmap v948: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:14.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:14 smithi151 ceph-mon[112233]: pgmap v949: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:15.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:14 smithi153 ceph-mon[87858]: pgmap v949: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:16.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:16 smithi151 ceph-mon[112233]: pgmap v950: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:17.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:16 smithi153 ceph-mon[87858]: pgmap v950: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:18.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:18 smithi151 ceph-mon[112233]: pgmap v951: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:19.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:18 smithi153 ceph-mon[87858]: pgmap v951: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:20.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:20 smithi151 ceph-mon[112233]: pgmap v952: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:21.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:20 smithi153 ceph-mon[87858]: pgmap v952: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:22.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:22 smithi151 ceph-mon[112233]: pgmap v953: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:23.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:22 smithi153 ceph-mon[87858]: pgmap v953: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:24 smithi151 ceph-mon[112233]: pgmap v954: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:10:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:24 smithi153 ceph-mon[87858]: pgmap v954: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:10:26.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:26 smithi151 ceph-mon[112233]: pgmap v955: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:27.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:26 smithi153 ceph-mon[87858]: pgmap v955: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:28.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:28 smithi151 ceph-mon[112233]: pgmap v956: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:29.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:28 smithi153 ceph-mon[87858]: pgmap v956: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:30.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:30 smithi151 ceph-mon[112233]: pgmap v957: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:31.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:30 smithi153 ceph-mon[87858]: pgmap v957: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:32.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:32 smithi151 ceph-mon[112233]: pgmap v958: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:33.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:32 smithi153 ceph-mon[87858]: pgmap v958: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:33.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.518+0000 7f1b910c1700 1 -- 172.21.15.151:0/2874078363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 msgr2=0x7f1b8c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:33.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.518+0000 7f1b910c1700 1 --2- 172.21.15.151:0/2874078363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 0x7f1b8c105e20 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009910 tx=0x7f1b7c009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:33.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.518+0000 7f1b910c1700 1 -- 172.21.15.151:0/2874078363 shutdown_connections 2024-04-16T03:10:33.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.518+0000 7f1b910c1700 1 --2- 172.21.15.151:0/2874078363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b8c0ffa40 0x7f1b8c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.518+0000 7f1b910c1700 1 --2- 172.21.15.151:0/2874078363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 0x7f1b8c105e20 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.518+0000 7f1b910c1700 1 -- 172.21.15.151:0/2874078363 >> 172.21.15.151:0/2874078363 conn(0x7f1b8c0fb570 msgr2=0x7f1b8c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.518+0000 7f1b910c1700 1 -- 172.21.15.151:0/2874078363 shutdown_connections 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.518+0000 7f1b910c1700 1 -- 172.21.15.151:0/2874078363 wait complete. 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b910c1700 1 Processor -- start 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b910c1700 1 -- start start 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b910c1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b8c0ffa40 0x7f1b8c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b910c1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 0x7f1b8c198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b910c1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c199530 con 0x7f1b8c105a40 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b910c1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c19d340 con 0x7f1b8c0ffa40 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b83fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 0x7f1b8c198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b83fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 0x7f1b8c198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53164/0 (socket says 172.21.15.151:53164) 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b83fff700 1 -- 172.21.15.151:0/2575158043 learned_addr learned my addr 172.21.15.151:0/2575158043 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b8ad9d700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b8c0ffa40 0x7f1b8c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b83fff700 1 -- 172.21.15.151:0/2575158043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b8c0ffa40 msgr2=0x7f1b8c198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b83fff700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b8c0ffa40 0x7f1b8c198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b83fff700 1 -- 172.21.15.151:0/2575158043 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c0095f0 con 0x7f1b8c105a40 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.519+0000 7f1b83fff700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 0x7f1b8c198ea0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400d8d0 tx=0x7f1b7400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.520+0000 7f1b88d99700 1 -- 172.21.15.151:0/2575158043 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b74009940 con 0x7f1b8c105a40 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.520+0000 7f1b88d99700 1 -- 172.21.15.151:0/2575158043 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b74010460 con 0x7f1b8c105a40 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.520+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c19d620 con 0x7f1b8c105a40 2024-04-16T03:10:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.520+0000 7f1b88d99700 1 -- 172.21.15.151:0/2575158043 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b7400f5f0 con 0x7f1b8c105a40 2024-04-16T03:10:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.520+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c19db70 con 0x7f1b8c105a40 2024-04-16T03:10:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.521+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c04ea90 con 0x7f1b8c105a40 2024-04-16T03:10:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.521+0000 7f1b88d99700 1 -- 172.21.15.151:0/2575158043 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1b740105d0 con 0x7f1b8c105a40 2024-04-16T03:10:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.522+0000 7f1b88d99700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b7806ecc0 0x7f1b78071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.522+0000 7f1b88d99700 1 -- 172.21.15.151:0/2575158043 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1b7408ee80 con 0x7f1b8c105a40 2024-04-16T03:10:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.522+0000 7f1b8ad9d700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b7806ecc0 0x7f1b78071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.522+0000 7f1b8ad9d700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b7806ecc0 0x7f1b78071180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00b5c0 tx=0x7f1b7c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:33.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.527+0000 7f1b88d99700 1 -- 172.21.15.151:0/2575158043 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1b7405b760 con 0x7f1b8c105a40 2024-04-16T03:10:33.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.745+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b8c19df00 con 0x7f1b7806ecc0 2024-04-16T03:10:33.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.747+0000 7f1b88d99700 1 -- 172.21.15.151:0/2575158043 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1b8c19df00 con 0x7f1b7806ecc0 2024-04-16T03:10:33.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b7806ecc0 msgr2=0x7f1b78071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b7806ecc0 0x7f1b78071180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00b5c0 tx=0x7f1b7c01a040 comp rx=0 tx=0).stop 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 msgr2=0x7f1b8c198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 0x7f1b8c198ea0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400d8d0 tx=0x7f1b7400dc90 comp rx=0 tx=0).stop 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 shutdown_connections 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1b7806ecc0 0x7f1b78071180 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b8c0ffa40 0x7f1b8c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 --2- 172.21.15.151:0/2575158043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b8c105a40 0x7f1b8c198ea0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 >> 172.21.15.151:0/2575158043 conn(0x7f1b8c0fb570 msgr2=0x7f1b8c068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 shutdown_connections 2024-04-16T03:10:33.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.749+0000 7f1b910c1700 1 -- 172.21.15.151:0/2575158043 wait complete. 2024-04-16T03:10:33.761 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:10:33.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.855+0000 7f65cf754700 1 -- 172.21.15.151:0/2879795721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c8105a40 msgr2=0x7f65c8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:33.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.855+0000 7f65cf754700 1 --2- 172.21.15.151:0/2879795721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c8105a40 0x7f65c8105e20 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f65c400b0a0 tx=0x7f65c400b3b0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.856 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.855+0000 7f65cf754700 1 -- 172.21.15.151:0/2879795721 shutdown_connections 2024-04-16T03:10:33.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.855+0000 7f65cf754700 1 --2- 172.21.15.151:0/2879795721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c80ffbd0 0x7f65c8100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.855+0000 7f65cf754700 1 --2- 172.21.15.151:0/2879795721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c8105a40 0x7f65c8105e20 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.855+0000 7f65cf754700 1 -- 172.21.15.151:0/2879795721 >> 172.21.15.151:0/2879795721 conn(0x7f65c80755a0 msgr2=0x7f65c80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:33.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.855+0000 7f65cf754700 1 -- 172.21.15.151:0/2879795721 shutdown_connections 2024-04-16T03:10:33.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.855+0000 7f65cf754700 1 -- 172.21.15.151:0/2879795721 wait complete. 2024-04-16T03:10:33.858 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cf754700 1 Processor -- start 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cf754700 1 -- start start 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cf754700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c80ffbd0 0x7f65c8101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cf754700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8105a40 0x7f65c8101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cf754700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c8102090 con 0x7f65c80ffbd0 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cf754700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c8102200 con 0x7f65c8105a40 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cd4f0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c80ffbd0 0x7f65c8101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cd4f0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c80ffbd0 0x7f65c8101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53176/0 (socket says 172.21.15.151:53176) 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cd4f0700 1 -- 172.21.15.151:0/1845086540 learned_addr learned my addr 172.21.15.151:0/1845086540 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cccef700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8105a40 0x7f65c8101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cd4f0700 1 -- 172.21.15.151:0/1845086540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8105a40 msgr2=0x7f65c8101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cd4f0700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8105a40 0x7f65c8101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cd4f0700 1 -- 172.21.15.151:0/1845086540 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c4009d00 con 0x7f65c80ffbd0 2024-04-16T03:10:33.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.856+0000 7f65cccef700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8105a40 0x7f65c8101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:10:33.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.857+0000 7f65cd4f0700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c80ffbd0 0x7f65c8101610 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f65c4000f80 tx=0x7f65c400cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:33.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.857+0000 7f65be7fc700 1 -- 172.21.15.151:0/1845086540 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c400e040 con 0x7f65c80ffbd0 2024-04-16T03:10:33.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.857+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c8102420 con 0x7f65c80ffbd0 2024-04-16T03:10:33.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.857+0000 7f65be7fc700 1 -- 172.21.15.151:0/1845086540 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65c4003e60 con 0x7f65c80ffbd0 2024-04-16T03:10:33.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.857+0000 7f65be7fc700 1 -- 172.21.15.151:0/1845086540 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c401d910 con 0x7f65c80ffbd0 2024-04-16T03:10:33.860 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.857+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c81a2bd0 con 0x7f65c80ffbd0 2024-04-16T03:10:33.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.858+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c8109050 con 0x7f65c80ffbd0 2024-04-16T03:10:33.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.858+0000 7f65be7fc700 1 -- 172.21.15.151:0/1845086540 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f65c4004820 con 0x7f65c80ffbd0 2024-04-16T03:10:33.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.859+0000 7f65be7fc700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65b406ecc0 0x7f65b4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:33.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.859+0000 7f65be7fc700 1 -- 172.21.15.151:0/1845086540 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f65c40907d0 con 0x7f65c80ffbd0 2024-04-16T03:10:33.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.859+0000 7f65cccef700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65b406ecc0 0x7f65b4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:33.861 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.860+0000 7f65cccef700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65b406ecc0 0x7f65b4071180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f65b8005fd0 tx=0x7f65b8005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:33.865 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:33.864+0000 7f65be7fc700 1 -- 172.21.15.151:0/1845086540 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f65c4059580 con 0x7f65c80ffbd0 2024-04-16T03:10:34.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.080+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f65c802ccc0 con 0x7f65b406ecc0 2024-04-16T03:10:34.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.081+0000 7f65be7fc700 1 -- 172.21.15.151:0/1845086540 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f65c802ccc0 con 0x7f65b406ecc0 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65b406ecc0 msgr2=0x7f65b4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65b406ecc0 0x7f65b4071180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f65b8005fd0 tx=0x7f65b8005e30 comp rx=0 tx=0).stop 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c80ffbd0 msgr2=0x7f65c8101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c80ffbd0 0x7f65c8101610 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f65c4000f80 tx=0x7f65c400cdd0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 shutdown_connections 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65b406ecc0 0x7f65b4071180 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65c80ffbd0 0x7f65c8101610 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 --2- 172.21.15.151:0/1845086540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8105a40 0x7f65c8101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 >> 172.21.15.151:0/1845086540 conn(0x7f65c80755a0 msgr2=0x7f65c80685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 shutdown_connections 2024-04-16T03:10:34.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.083+0000 7f65cf754700 1 -- 172.21.15.151:0/1845086540 wait complete. 2024-04-16T03:10:34.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.188+0000 7f344779a700 1 -- 172.21.15.151:0/3814486330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 msgr2=0x7f34401057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.188+0000 7f344779a700 1 --2- 172.21.15.151:0/3814486330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 0x7f34401057e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3430009910 tx=0x7f3430009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:34.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.188+0000 7f344779a700 1 -- 172.21.15.151:0/3814486330 shutdown_connections 2024-04-16T03:10:34.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.188+0000 7f344779a700 1 --2- 172.21.15.151:0/3814486330 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 0x7f34401057e0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.188+0000 7f344779a700 1 --2- 172.21.15.151:0/3814486330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3440100de0 0x7f34401011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.188+0000 7f344779a700 1 -- 172.21.15.151:0/3814486330 >> 172.21.15.151:0/3814486330 conn(0x7f34400fc670 msgr2=0x7f34400fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:34.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.188+0000 7f344779a700 1 -- 172.21.15.151:0/3814486330 shutdown_connections 2024-04-16T03:10:34.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.188+0000 7f344779a700 1 -- 172.21.15.151:0/3814486330 wait complete. 2024-04-16T03:10:34.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f344779a700 1 Processor -- start 2024-04-16T03:10:34.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f344779a700 1 -- start start 2024-04-16T03:10:34.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f344779a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3440100de0 0x7f3440073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f344779a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 0x7f34400735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f344779a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440073b10 con 0x7f3440101790 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f344779a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440073c50 con 0x7f3440100de0 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f3444d35700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 0x7f34400735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f3444d35700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 0x7f34400735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53190/0 (socket says 172.21.15.151:53190) 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f3444d35700 1 -- 172.21.15.151:0/260669895 learned_addr learned my addr 172.21.15.151:0/260669895 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f3445536700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3440100de0 0x7f3440073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f3444d35700 1 -- 172.21.15.151:0/260669895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3440100de0 msgr2=0x7f3440073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f3444d35700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3440100de0 0x7f3440073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.189+0000 7f3444d35700 1 -- 172.21.15.151:0/260669895 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300095f0 con 0x7f3440101790 2024-04-16T03:10:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.190+0000 7f3444d35700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 0x7f34400735d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f343000bd70 tx=0x7f343000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:34.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.190+0000 7f34367fc700 1 -- 172.21.15.151:0/260669895 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f343001d070 con 0x7f3440101790 2024-04-16T03:10:34.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.190+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3440073ed0 con 0x7f3440101790 2024-04-16T03:10:34.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.190+0000 7f34367fc700 1 -- 172.21.15.151:0/260669895 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3430005dc0 con 0x7f3440101790 2024-04-16T03:10:34.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.190+0000 7f34367fc700 1 -- 172.21.15.151:0/260669895 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3430017400 con 0x7f3440101790 2024-04-16T03:10:34.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.190+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34401a2c90 con 0x7f3440101790 2024-04-16T03:10:34.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.191+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34401091e0 con 0x7f3440101790 2024-04-16T03:10:34.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.191+0000 7f34367fc700 1 -- 172.21.15.151:0/260669895 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f343000f460 con 0x7f3440101790 2024-04-16T03:10:34.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.192+0000 7f34367fc700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f342c06ecc0 0x7f342c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:34.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.192+0000 7f34367fc700 1 -- 172.21.15.151:0/260669895 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f34300902d0 con 0x7f3440101790 2024-04-16T03:10:34.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.192+0000 7f3445536700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f342c06ecc0 0x7f342c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:34.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.192+0000 7f3445536700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f342c06ecc0 0x7f342c071180 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f343c005950 tx=0x7f343c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:34.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.197+0000 7f34367fc700 1 -- 172.21.15.151:0/260669895 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3430059080 con 0x7f3440101790 2024-04-16T03:10:34.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.403+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f34400611d0 con 0x7f342c06ecc0 2024-04-16T03:10:34.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.414+0000 7f34367fc700 1 -- 172.21.15.151:0/260669895 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f34400611d0 con 0x7f342c06ecc0 2024-04-16T03:10:34.415 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (32m) 10m ago 43m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (43m) 10m ago 43m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (42m) 10m ago 42m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (29m) 10m ago 43m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (29m) 10m ago 42m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (31m) 10m ago 43m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (25m) 10m ago 37m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (25m) 10m ago 37m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (10m) 10m ago 37m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:10:34.416 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (25m) 10m ago 37m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (34m) 10m ago 44m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (33m) 10m ago 42m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (30m) 10m ago 44m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (30m) 10m ago 42m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (32m) 10m ago 43m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (32m) 10m ago 42m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (29m) 10m ago 41m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (28m) 10m ago 41m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (28m) 10m ago 40m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:10:34.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (27m) 10m ago 40m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:10:34.418 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (26m) 10m ago 39m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:10:34.418 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (26m) 10m ago 39m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:10:34.418 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (32m) 10m ago 42m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:10:34.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f342c06ecc0 msgr2=0x7f342c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f342c06ecc0 0x7f342c071180 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f343c005950 tx=0x7f343c00b410 comp rx=0 tx=0).stop 2024-04-16T03:10:34.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 msgr2=0x7f34400735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.419 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 0x7f34400735d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f343000bd70 tx=0x7f343000bda0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 shutdown_connections 2024-04-16T03:10:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f342c06ecc0 0x7f342c071180 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3440100de0 0x7f3440073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 --2- 172.21.15.151:0/260669895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3440101790 0x7f34400735d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 >> 172.21.15.151:0/260669895 conn(0x7f34400fc670 msgr2=0x7f34400fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 shutdown_connections 2024-04-16T03:10:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.416+0000 7f344779a700 1 -- 172.21.15.151:0/260669895 wait complete. 2024-04-16T03:10:34.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.521+0000 7f4a7d824700 1 -- 172.21.15.151:0/1704083099 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78101c70 msgr2=0x7f4a78102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.521+0000 7f4a7d824700 1 --2- 172.21.15.151:0/1704083099 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78101c70 0x7f4a78102050 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009910 tx=0x7f4a68009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:34.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.521+0000 7f4a7d824700 1 -- 172.21.15.151:0/1704083099 shutdown_connections 2024-04-16T03:10:34.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.521+0000 7f4a7d824700 1 --2- 172.21.15.151:0/1704083099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a78102620 0x7f4a7810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.521+0000 7f4a7d824700 1 --2- 172.21.15.151:0/1704083099 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78101c70 0x7f4a78102050 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.521+0000 7f4a7d824700 1 -- 172.21.15.151:0/1704083099 >> 172.21.15.151:0/1704083099 conn(0x7f4a780fb550 msgr2=0x7f4a780fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:34.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.521+0000 7f4a7d824700 1 -- 172.21.15.151:0/1704083099 shutdown_connections 2024-04-16T03:10:34.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.521+0000 7f4a7d824700 1 -- 172.21.15.151:0/1704083099 wait complete. 2024-04-16T03:10:34.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.522+0000 7f4a7d824700 1 Processor -- start 2024-04-16T03:10:34.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.522+0000 7f4a7d824700 1 -- start start 2024-04-16T03:10:34.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a7d824700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a78101c70 0x7f4a7819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:34.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a7d824700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78102620 0x7f4a7819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:34.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a7d824700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7819d0c0 con 0x7f4a78102620 2024-04-16T03:10:34.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a7d824700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a78196570 con 0x7f4a78101c70 2024-04-16T03:10:34.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a767fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78102620 0x7f4a7819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:34.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a767fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78102620 0x7f4a7819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53210/0 (socket says 172.21.15.151:53210) 2024-04-16T03:10:34.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a767fc700 1 -- 172.21.15.151:0/1671589770 learned_addr learned my addr 172.21.15.151:0/1671589770 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:34.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a76ffd700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a78101c70 0x7f4a7819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:34.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a767fc700 1 -- 172.21.15.151:0/1671589770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a78101c70 msgr2=0x7f4a7819c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a767fc700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a78101c70 0x7f4a7819c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.529 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a767fc700 1 -- 172.21.15.151:0/1671589770 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a680095f0 con 0x7f4a78102620 2024-04-16T03:10:34.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a76ffd700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a78101c70 0x7f4a7819c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:10:34.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a767fc700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78102620 0x7f4a7819ca30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00c8f0 tx=0x7f4a6c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:34.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a7c822700 1 -- 172.21.15.151:0/1671589770 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a6c007a50 con 0x7f4a78102620 2024-04-16T03:10:34.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a78196850 con 0x7f4a78102620 2024-04-16T03:10:34.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a7c822700 1 -- 172.21.15.151:0/1671589770 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a6c007bb0 con 0x7f4a78102620 2024-04-16T03:10:34.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.523+0000 7f4a7c822700 1 -- 172.21.15.151:0/1671589770 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a6c018630 con 0x7f4a78102620 2024-04-16T03:10:34.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.524+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a78196da0 con 0x7f4a78102620 2024-04-16T03:10:34.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.524+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a78108210 con 0x7f4a78102620 2024-04-16T03:10:34.530 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.525+0000 7f4a7c822700 1 -- 172.21.15.151:0/1671589770 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4a6c007d20 con 0x7f4a78102620 2024-04-16T03:10:34.531 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.525+0000 7f4a7c822700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a6006ecc0 0x7f4a60071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:34.531 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.526+0000 7f4a7c822700 1 -- 172.21.15.151:0/1671589770 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4a6c08ee50 con 0x7f4a78102620 2024-04-16T03:10:34.531 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.526+0000 7f4a76ffd700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a6006ecc0 0x7f4a60071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:34.531 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.526+0000 7f4a76ffd700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a6006ecc0 0x7f4a60071180 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800b5c0 tx=0x7f4a68005480 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:34.531 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.530+0000 7f4a7c822700 1 -- 172.21.15.151:0/1671589770 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4a6c05b730 con 0x7f4a78102620 2024-04-16T03:10:34.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.810+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4a7804ea90 con 0x7f4a78102620 2024-04-16T03:10:34.812 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:34 smithi151 ceph-mon[112233]: pgmap v959: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:34.812 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:34 smithi151 ceph-mon[112233]: from='client.35744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:34.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.811+0000 7f4a7c822700 1 -- 172.21.15.151:0/1671589770 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4a6c05b2c0 con 0x7f4a78102620 2024-04-16T03:10:34.812 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:10:34.813 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:10:34.814 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:10:34.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a6006ecc0 msgr2=0x7f4a60071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a6006ecc0 0x7f4a60071180 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800b5c0 tx=0x7f4a68005480 comp rx=0 tx=0).stop 2024-04-16T03:10:34.814 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78102620 msgr2=0x7f4a7819ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78102620 0x7f4a7819ca30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00c8f0 tx=0x7f4a6c00ccb0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 shutdown_connections 2024-04-16T03:10:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a6006ecc0 0x7f4a60071180 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a78101c70 0x7f4a7819c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 --2- 172.21.15.151:0/1671589770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a78102620 0x7f4a7819ca30 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 >> 172.21.15.151:0/1671589770 conn(0x7f4a780fb550 msgr2=0x7f4a78105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 shutdown_connections 2024-04-16T03:10:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.813+0000 7f4a7d824700 1 -- 172.21.15.151:0/1671589770 wait complete. 2024-04-16T03:10:34.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.918+0000 7efe9c2f1700 1 -- 172.21.15.151:0/2998458120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 msgr2=0x7efe94102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.918+0000 7efe9c2f1700 1 --2- 172.21.15.151:0/2998458120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 0x7efe94102050 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7efe84009910 tx=0x7efe84009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:34.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.918+0000 7efe9c2f1700 1 -- 172.21.15.151:0/2998458120 shutdown_connections 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.918+0000 7efe9c2f1700 1 --2- 172.21.15.151:0/2998458120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94102620 0x7efe9410ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.918+0000 7efe9c2f1700 1 --2- 172.21.15.151:0/2998458120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 0x7efe94102050 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.918+0000 7efe9c2f1700 1 -- 172.21.15.151:0/2998458120 >> 172.21.15.151:0/2998458120 conn(0x7efe940fb550 msgr2=0x7efe940fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.918+0000 7efe9c2f1700 1 -- 172.21.15.151:0/2998458120 shutdown_connections 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.918+0000 7efe9c2f1700 1 -- 172.21.15.151:0/2998458120 wait complete. 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9c2f1700 1 Processor -- start 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9c2f1700 1 -- start start 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9c2f1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 0x7efe9419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9c2f1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94102620 0x7efe9419ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9c2f1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9419d0c0 con 0x7efe94101c70 2024-04-16T03:10:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9c2f1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe94196570 con 0x7efe94102620 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9a08d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 0x7efe9419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9a08d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 0x7efe9419c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53230/0 (socket says 172.21.15.151:53230) 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9a08d700 1 -- 172.21.15.151:0/1254340127 learned_addr learned my addr 172.21.15.151:0/1254340127 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9988c700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94102620 0x7efe9419ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9a08d700 1 -- 172.21.15.151:0/1254340127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94102620 msgr2=0x7efe9419ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9a08d700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94102620 0x7efe9419ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.919+0000 7efe9a08d700 1 -- 172.21.15.151:0/1254340127 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe840095f0 con 0x7efe94101c70 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.920+0000 7efe9a08d700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 0x7efe9419c4f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7efe84006010 tx=0x7efe8400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.920+0000 7efe8b7fe700 1 -- 172.21.15.151:0/1254340127 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe8401d070 con 0x7efe94101c70 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.920+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe941967f0 con 0x7efe94101c70 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.920+0000 7efe8b7fe700 1 -- 172.21.15.151:0/1254340127 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe84005dc0 con 0x7efe94101c70 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.920+0000 7efe8b7fe700 1 -- 172.21.15.151:0/1254340127 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe84017400 con 0x7efe94101c70 2024-04-16T03:10:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.920+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe94196ce0 con 0x7efe94101c70 2024-04-16T03:10:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.921+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe94108210 con 0x7efe94101c70 2024-04-16T03:10:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.921+0000 7efe8b7fe700 1 -- 172.21.15.151:0/1254340127 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efe84003a60 con 0x7efe94101c70 2024-04-16T03:10:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.922+0000 7efe8b7fe700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe8006ecc0 0x7efe80071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.922+0000 7efe8b7fe700 1 -- 172.21.15.151:0/1254340127 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efe840902a0 con 0x7efe94101c70 2024-04-16T03:10:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.922+0000 7efe9988c700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe8006ecc0 0x7efe80071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.922+0000 7efe9988c700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe8006ecc0 0x7efe80071180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efe940fcc30 tx=0x7efe9000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:34.926+0000 7efe8b7fe700 1 -- 172.21.15.151:0/1254340127 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efe8405cb80 con 0x7efe94101c70 2024-04-16T03:10:35.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:34 smithi153 ceph-mon[87858]: pgmap v959: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:35.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:34 smithi153 ceph-mon[87858]: from='client.35744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:35.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.173+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efe9404ea90 con 0x7efe94101c70 2024-04-16T03:10:35.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.173+0000 7efe8b7fe700 1 -- 172.21.15.151:0/1254340127 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7efe8405c710 con 0x7efe94101c70 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:10:35.175 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:10:35.176 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:10:35.177 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:10:35.178 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:10:35.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe8006ecc0 msgr2=0x7efe80071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe8006ecc0 0x7efe80071180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efe940fcc30 tx=0x7efe9000b410 comp rx=0 tx=0).stop 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 msgr2=0x7efe9419c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 0x7efe9419c4f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7efe84006010 tx=0x7efe8400bd70 comp rx=0 tx=0).stop 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 shutdown_connections 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efe8006ecc0 0x7efe80071180 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe94101c70 0x7efe9419c4f0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 --2- 172.21.15.151:0/1254340127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94102620 0x7efe9419ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 >> 172.21.15.151:0/1254340127 conn(0x7efe940fb550 msgr2=0x7efe94105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 shutdown_connections 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.176+0000 7efe9c2f1700 1 -- 172.21.15.151:0/1254340127 wait complete. 2024-04-16T03:10:35.179 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:10:35.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 -- 172.21.15.151:0/2518423668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 msgr2=0x7f39dc1057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 --2- 172.21.15.151:0/2518423668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 0x7f39dc1057d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009910 tx=0x7f39d0009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:35.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 -- 172.21.15.151:0/2518423668 shutdown_connections 2024-04-16T03:10:35.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 --2- 172.21.15.151:0/2518423668 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 0x7f39dc1057d0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 --2- 172.21.15.151:0/2518423668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39dc100dd0 0x7f39dc1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 -- 172.21.15.151:0/2518423668 >> 172.21.15.151:0/2518423668 conn(0x7f39dc0fc640 msgr2=0x7f39dc0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:35.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 -- 172.21.15.151:0/2518423668 shutdown_connections 2024-04-16T03:10:35.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 -- 172.21.15.151:0/2518423668 wait complete. 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 Processor -- start 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.283+0000 7f39e37d7700 1 -- start start 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e37d7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39dc100dd0 0x7f39dc19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e37d7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 0x7f39dc19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e37d7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc19d140 con 0x7f39dc101780 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e37d7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc1965f0 con 0x7f39dc100dd0 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e0d72700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 0x7f39dc19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e0d72700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 0x7f39dc19cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53248/0 (socket says 172.21.15.151:53248) 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e0d72700 1 -- 172.21.15.151:0/2227324122 learned_addr learned my addr 172.21.15.151:0/2227324122 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e1573700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39dc100dd0 0x7f39dc19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e0d72700 1 -- 172.21.15.151:0/2227324122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39dc100dd0 msgr2=0x7f39dc19c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e0d72700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39dc100dd0 0x7f39dc19c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e0d72700 1 -- 172.21.15.151:0/2227324122 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d00095f0 con 0x7f39dc101780 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e0d72700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 0x7f39dc19cab0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f39d00092e0 tx=0x7f39d000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39ce7fc700 1 -- 172.21.15.151:0/2227324122 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39d001d070 con 0x7f39dc101780 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39ce7fc700 1 -- 172.21.15.151:0/2227324122 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39d0005dc0 con 0x7f39dc101780 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.284+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc196870 con 0x7f39dc101780 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.285+0000 7f39ce7fc700 1 -- 172.21.15.151:0/2227324122 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39d0017400 con 0x7f39dc101780 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.285+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc196d60 con 0x7f39dc101780 2024-04-16T03:10:35.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.286+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc1091d0 con 0x7f39dc101780 2024-04-16T03:10:35.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.286+0000 7f39ce7fc700 1 -- 172.21.15.151:0/2227324122 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f39d000f6f0 con 0x7f39dc101780 2024-04-16T03:10:35.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.287+0000 7f39ce7fc700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f39c806ecc0 0x7f39c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:35.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.287+0000 7f39ce7fc700 1 -- 172.21.15.151:0/2227324122 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f39d0090170 con 0x7f39dc101780 2024-04-16T03:10:35.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.287+0000 7f39e1573700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f39c806ecc0 0x7f39c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:35.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.287+0000 7f39e1573700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f39c806ecc0 0x7f39c8071180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f39d8005950 tx=0x7f39d800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:35.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.291+0000 7f39ce7fc700 1 -- 172.21.15.151:0/2227324122 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f39d005cad0 con 0x7f39dc101780 2024-04-16T03:10:35.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.508+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39dc02cca0 con 0x7f39c806ecc0 2024-04-16T03:10:35.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.510+0000 7f39ce7fc700 1 -- 172.21.15.151:0/2227324122 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f39dc02cca0 con 0x7f39c806ecc0 2024-04-16T03:10:35.511 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:10:35.511 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:10:35.511 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:10:35.512 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:10:35.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f39c806ecc0 msgr2=0x7f39c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f39c806ecc0 0x7f39c8071180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f39d8005950 tx=0x7f39d800b410 comp rx=0 tx=0).stop 2024-04-16T03:10:35.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 msgr2=0x7f39dc19cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 0x7f39dc19cab0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f39d00092e0 tx=0x7f39d000bd70 comp rx=0 tx=0).stop 2024-04-16T03:10:35.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 shutdown_connections 2024-04-16T03:10:35.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f39c806ecc0 0x7f39c8071180 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39dc100dd0 0x7f39dc19c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 --2- 172.21.15.151:0/2227324122 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39dc101780 0x7f39dc19cab0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 >> 172.21.15.151:0/2227324122 conn(0x7f39dc0fc640 msgr2=0x7f39dc0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:35.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 shutdown_connections 2024-04-16T03:10:35.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.512+0000 7f39e37d7700 1 -- 172.21.15.151:0/2227324122 wait complete. 2024-04-16T03:10:35.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.617+0000 7f5449527700 1 -- 172.21.15.151:0/617058974 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444101770 msgr2=0x7f54441057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.617+0000 7f5449527700 1 --2- 172.21.15.151:0/617058974 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444101770 0x7f54441057c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f542c009910 tx=0x7f542c009c20 comp rx=0 tx=0).stop 2024-04-16T03:10:35.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.617+0000 7f5449527700 1 -- 172.21.15.151:0/617058974 shutdown_connections 2024-04-16T03:10:35.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.617+0000 7f5449527700 1 --2- 172.21.15.151:0/617058974 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444101770 0x7f54441057c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.617+0000 7f5449527700 1 --2- 172.21.15.151:0/617058974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444100dc0 0x7f54441011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.617+0000 7f5449527700 1 -- 172.21.15.151:0/617058974 >> 172.21.15.151:0/617058974 conn(0x7f54440fc650 msgr2=0x7f54440fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:35.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.617+0000 7f5449527700 1 -- 172.21.15.151:0/617058974 shutdown_connections 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.617+0000 7f5449527700 1 -- 172.21.15.151:0/617058974 wait complete. 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.617+0000 7f5449527700 1 Processor -- start 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5449527700 1 -- start start 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5449527700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444100dc0 0x7f5444073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5449527700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444101770 0x7f54440735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5449527700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5444073b10 con 0x7f5444100dc0 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5449527700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5444073c50 con 0x7f5444101770 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5442ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444100dc0 0x7f5444073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5442ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444100dc0 0x7f5444073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53256/0 (socket says 172.21.15.151:53256) 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5442ffd700 1 -- 172.21.15.151:0/2902100797 learned_addr learned my addr 172.21.15.151:0/2902100797 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:10:35.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f543bfff700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444101770 0x7f54440735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5442ffd700 1 -- 172.21.15.151:0/2902100797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444101770 msgr2=0x7f54440735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5442ffd700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444101770 0x7f54440735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5442ffd700 1 -- 172.21.15.151:0/2902100797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c0095f0 con 0x7f5444100dc0 2024-04-16T03:10:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5442ffd700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444100dc0 0x7f5444073090 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f543400b4b0 tx=0x7f543400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.618+0000 7f5440ff9700 1 -- 172.21.15.151:0/2902100797 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5434011800 con 0x7f5444100dc0 2024-04-16T03:10:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.619+0000 7f5440ff9700 1 -- 172.21.15.151:0/2902100797 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5434011e20 con 0x7f5444100dc0 2024-04-16T03:10:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.619+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54441a2a70 con 0x7f5444100dc0 2024-04-16T03:10:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.619+0000 7f5440ff9700 1 -- 172.21.15.151:0/2902100797 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f543400f2e0 con 0x7f5444100dc0 2024-04-16T03:10:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.619+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54441a2e80 con 0x7f5444100dc0 2024-04-16T03:10:35.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.620+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54441090f0 con 0x7f5444100dc0 2024-04-16T03:10:35.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.620+0000 7f5440ff9700 1 -- 172.21.15.151:0/2902100797 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5434010380 con 0x7f5444100dc0 2024-04-16T03:10:35.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.620+0000 7f5440ff9700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f543006ecc0 0x7f5430071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:10:35.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.621+0000 7f5440ff9700 1 -- 172.21.15.151:0/2902100797 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f543408e990 con 0x7f5444100dc0 2024-04-16T03:10:35.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.621+0000 7f543bfff700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f543006ecc0 0x7f5430071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:10:35.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.621+0000 7f543bfff700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f543006ecc0 0x7f5430071180 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5444074880 tx=0x7f542c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:10:35.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.625+0000 7f5440ff9700 1 -- 172.21.15.151:0/2902100797 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5434057870 con 0x7f5444100dc0 2024-04-16T03:10:35.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.907+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f544404ea90 con 0x7f5444100dc0 2024-04-16T03:10:35.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:35 smithi151 ceph-mon[112233]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:35.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:35 smithi151 ceph-mon[112233]: from='client.35752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:35.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:35 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1671589770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:10:35.909 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:35 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1254340127' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:10:35.909 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.908+0000 7f5440ff9700 1 -- 172.21.15.151:0/2902100797 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f543405af30 con 0x7f5444100dc0 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:10:35.910 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:10:35.911 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f543006ecc0 msgr2=0x7f5430071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f543006ecc0 0x7f5430071180 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5444074880 tx=0x7f542c01a040 comp rx=0 tx=0).stop 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444100dc0 msgr2=0x7f5444073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444100dc0 0x7f5444073090 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f543400b4b0 tx=0x7f543400b870 comp rx=0 tx=0).stop 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 shutdown_connections 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f543006ecc0 0x7f5430071180 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5444100dc0 0x7f5444073090 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 --2- 172.21.15.151:0/2902100797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444101770 0x7f54440735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 >> 172.21.15.151:0/2902100797 conn(0x7f54440fc650 msgr2=0x7f54440fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 shutdown_connections 2024-04-16T03:10:35.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:10:35.910+0000 7f5449527700 1 -- 172.21.15.151:0/2902100797 wait complete. 2024-04-16T03:10:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:35 smithi153 ceph-mon[87858]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:35 smithi153 ceph-mon[87858]: from='client.35752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:35 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1671589770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:10:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:35 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1254340127' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:10:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:36 smithi151 ceph-mon[112233]: pgmap v960: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:36 smithi151 ceph-mon[112233]: from='client.35764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:36 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2902100797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:10:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:36 smithi153 ceph-mon[87858]: pgmap v960: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:36 smithi153 ceph-mon[87858]: from='client.35764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:10:37.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:36 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2902100797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:10:38.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:38 smithi151 ceph-mon[112233]: pgmap v961: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:39.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:38 smithi153 ceph-mon[87858]: pgmap v961: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:39.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:10:40.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:10:40.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:40 smithi153 ceph-mon[87858]: pgmap v962: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:40.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:40 smithi151 ceph-mon[112233]: pgmap v962: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:42.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:42 smithi151 ceph-mon[112233]: pgmap v963: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:43.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:42 smithi153 ceph-mon[87858]: pgmap v963: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:44.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:44 smithi151 ceph-mon[112233]: pgmap v964: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:45.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:44 smithi153 ceph-mon[87858]: pgmap v964: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:46.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:46 smithi151 ceph-mon[112233]: pgmap v965: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:47.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:46 smithi153 ceph-mon[87858]: pgmap v965: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:48.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:48 smithi151 ceph-mon[112233]: pgmap v966: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:49.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:48 smithi153 ceph-mon[87858]: pgmap v966: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:50.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:50 smithi151 ceph-mon[112233]: pgmap v967: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:51.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:50 smithi153 ceph-mon[87858]: pgmap v967: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:52.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:52 smithi151 ceph-mon[112233]: pgmap v968: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:53.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:52 smithi153 ceph-mon[87858]: pgmap v968: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:54 smithi151 ceph-mon[112233]: pgmap v969: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:54.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:10:55.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:54 smithi153 ceph-mon[87858]: pgmap v969: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:10:56.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:56 smithi151 ceph-mon[112233]: pgmap v970: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:57.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:56 smithi153 ceph-mon[87858]: pgmap v970: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:58.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:10:58 smithi151 ceph-mon[112233]: pgmap v971: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:10:59.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:10:58 smithi153 ceph-mon[87858]: pgmap v971: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:00 smithi151 ceph-mon[112233]: pgmap v972: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:01.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:00 smithi153 ceph-mon[87858]: pgmap v972: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:02.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:02 smithi151 ceph-mon[112233]: pgmap v973: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:03.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:02 smithi153 ceph-mon[87858]: pgmap v973: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:04.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:04 smithi151 ceph-mon[112233]: pgmap v974: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:05.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:04 smithi153 ceph-mon[87858]: pgmap v974: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:06.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.028+0000 7f0b3dc14700 1 -- 172.21.15.151:0/1190817188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 msgr2=0x7f0b38102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.028+0000 7f0b3dc14700 1 --2- 172.21.15.151:0/1190817188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 0x7f0b38102090 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800b0a0 tx=0x7f0b2800b3b0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.028+0000 7f0b3dc14700 1 -- 172.21.15.151:0/1190817188 shutdown_connections 2024-04-16T03:11:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.028+0000 7f0b3dc14700 1 --2- 172.21.15.151:0/1190817188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b38102660 0x7f0b3810ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.028+0000 7f0b3dc14700 1 --2- 172.21.15.151:0/1190817188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 0x7f0b38102090 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.028+0000 7f0b3dc14700 1 -- 172.21.15.151:0/1190817188 >> 172.21.15.151:0/1190817188 conn(0x7f0b380fb590 msgr2=0x7f0b380fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.028+0000 7f0b3dc14700 1 -- 172.21.15.151:0/1190817188 shutdown_connections 2024-04-16T03:11:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.028+0000 7f0b3dc14700 1 -- 172.21.15.151:0/1190817188 wait complete. 2024-04-16T03:11:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.028+0000 7f0b3dc14700 1 Processor -- start 2024-04-16T03:11:06.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b3dc14700 1 -- start start 2024-04-16T03:11:06.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b3dc14700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 0x7f0b38196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:06.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b3dc14700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b38102660 0x7f0b38196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:06.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b3dc14700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b381972d0 con 0x7f0b38101cb0 2024-04-16T03:11:06.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b3dc14700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3819a000 con 0x7f0b38102660 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b377fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 0x7f0b38196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b377fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 0x7f0b38196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44620/0 (socket says 172.21.15.151:44620) 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b377fe700 1 -- 172.21.15.151:0/962401840 learned_addr learned my addr 172.21.15.151:0/962401840 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b36ffd700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b38102660 0x7f0b38196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b377fe700 1 -- 172.21.15.151:0/962401840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b38102660 msgr2=0x7f0b38196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b377fe700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b38102660 0x7f0b38196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.029+0000 7f0b377fe700 1 -- 172.21.15.151:0/962401840 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b28009d00 con 0x7f0b38101cb0 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.030+0000 7f0b377fe700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 0x7f0b38196700 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0b28000f80 tx=0x7f0b2800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.030+0000 7f0b34ff9700 1 -- 172.21.15.151:0/962401840 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b2800e040 con 0x7f0b38101cb0 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.030+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3819a280 con 0x7f0b38101cb0 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.030+0000 7f0b34ff9700 1 -- 172.21.15.151:0/962401840 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b28003c10 con 0x7f0b38101cb0 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.030+0000 7f0b34ff9700 1 -- 172.21.15.151:0/962401840 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b2801d6b0 con 0x7f0b38101cb0 2024-04-16T03:11:06.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.030+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3819a770 con 0x7f0b38101cb0 2024-04-16T03:11:06.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.031+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b38108250 con 0x7f0b38101cb0 2024-04-16T03:11:06.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.031+0000 7f0b34ff9700 1 -- 172.21.15.151:0/962401840 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f0b28004040 con 0x7f0b38101cb0 2024-04-16T03:11:06.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.032+0000 7f0b34ff9700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b2006ecc0 0x7f0b20071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:06.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.032+0000 7f0b34ff9700 1 -- 172.21.15.151:0/962401840 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0b280905c0 con 0x7f0b38101cb0 2024-04-16T03:11:06.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.032+0000 7f0b36ffd700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b2006ecc0 0x7f0b20071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:06.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.032+0000 7f0b36ffd700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b2006ecc0 0x7f0b20071180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0b38197d20 tx=0x7f0b2c0073d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:06.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.037+0000 7f0b34ff9700 1 -- 172.21.15.151:0/962401840 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0b28059370 con 0x7f0b38101cb0 2024-04-16T03:11:06.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.259+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b3819a990 con 0x7f0b2006ecc0 2024-04-16T03:11:06.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.260+0000 7f0b34ff9700 1 -- 172.21.15.151:0/962401840 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0b3819a990 con 0x7f0b2006ecc0 2024-04-16T03:11:06.263 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b2006ecc0 msgr2=0x7f0b20071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b2006ecc0 0x7f0b20071180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0b38197d20 tx=0x7f0b2c0073d0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 msgr2=0x7f0b38196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 0x7f0b38196700 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0b28000f80 tx=0x7f0b2800cd20 comp rx=0 tx=0).stop 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 shutdown_connections 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0b2006ecc0 0x7f0b20071180 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b38101cb0 0x7f0b38196700 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 --2- 172.21.15.151:0/962401840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b38102660 0x7f0b38196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 >> 172.21.15.151:0/962401840 conn(0x7f0b380fb590 msgr2=0x7f0b38105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 shutdown_connections 2024-04-16T03:11:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.262+0000 7f0b3dc14700 1 -- 172.21.15.151:0/962401840 wait complete. 2024-04-16T03:11:06.274 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:11:06.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.368+0000 7f937531a700 1 -- 172.21.15.151:0/3092425555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f93700ffa40 msgr2=0x7f93700ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.368+0000 7f937531a700 1 --2- 172.21.15.151:0/3092425555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f93700ffa40 0x7f93700ffec0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9358009910 tx=0x7f9358009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:06.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.368+0000 7f937531a700 1 -- 172.21.15.151:0/3092425555 shutdown_connections 2024-04-16T03:11:06.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.368+0000 7f937531a700 1 --2- 172.21.15.151:0/3092425555 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f93700ffa40 0x7f93700ffec0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.368+0000 7f937531a700 1 --2- 172.21.15.151:0/3092425555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9370105a40 0x7f9370105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.368+0000 7f937531a700 1 -- 172.21.15.151:0/3092425555 >> 172.21.15.151:0/3092425555 conn(0x7f93700fb570 msgr2=0x7f93700fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:06.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.368+0000 7f937531a700 1 -- 172.21.15.151:0/3092425555 shutdown_connections 2024-04-16T03:11:06.370 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.368+0000 7f937531a700 1 -- 172.21.15.151:0/3092425555 wait complete. 2024-04-16T03:11:06.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f937531a700 1 Processor -- start 2024-04-16T03:11:06.371 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f937531a700 1 -- start start 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f937531a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93700ffa40 0x7f93700fbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f937531a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9370105a40 0x7f93700fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f937531a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93700fc960 con 0x7f9370105a40 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f937531a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93700fcaa0 con 0x7f93700ffa40 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f93667fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9370105a40 0x7f93700fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f93667fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9370105a40 0x7f93700fc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44636/0 (socket says 172.21.15.151:44636) 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f93667fc700 1 -- 172.21.15.151:0/727393093 learned_addr learned my addr 172.21.15.151:0/727393093 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.369+0000 7f936effd700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93700ffa40 0x7f93700fbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.370+0000 7f93667fc700 1 -- 172.21.15.151:0/727393093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93700ffa40 msgr2=0x7f93700fbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.370+0000 7f93667fc700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93700ffa40 0x7f93700fbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.370+0000 7f93667fc700 1 -- 172.21.15.151:0/727393093 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93580095f0 con 0x7f9370105a40 2024-04-16T03:11:06.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.370+0000 7f93667fc700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9370105a40 0x7f93700fc420 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9358006010 tx=0x7f935800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:06.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.370+0000 7f936cff9700 1 -- 172.21.15.151:0/727393093 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f935801d070 con 0x7f9370105a40 2024-04-16T03:11:06.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.370+0000 7f936cff9700 1 -- 172.21.15.151:0/727393093 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f935800f460 con 0x7f9370105a40 2024-04-16T03:11:06.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.370+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93700fcd20 con 0x7f9370105a40 2024-04-16T03:11:06.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.370+0000 7f936cff9700 1 -- 172.21.15.151:0/727393093 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9358017630 con 0x7f9370105a40 2024-04-16T03:11:06.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.370+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93701a2d10 con 0x7f9370105a40 2024-04-16T03:11:06.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.371+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9370109070 con 0x7f9370105a40 2024-04-16T03:11:06.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.372+0000 7f936cff9700 1 -- 172.21.15.151:0/727393093 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f935800f5d0 con 0x7f9370105a40 2024-04-16T03:11:06.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.373+0000 7f936cff9700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f935c06ecc0 0x7f935c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:06.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.373+0000 7f936cff9700 1 -- 172.21.15.151:0/727393093 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9358090390 con 0x7f9370105a40 2024-04-16T03:11:06.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.373+0000 7f936effd700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f935c06ecc0 0x7f935c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:06.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.373+0000 7f936effd700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f935c06ecc0 0x7f935c071180 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9360009a10 tx=0x7f9360009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:06.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.377+0000 7f936cff9700 1 -- 172.21.15.151:0/727393093 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9358059140 con 0x7f9370105a40 2024-04-16T03:11:06.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.595+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f937002cd30 con 0x7f935c06ecc0 2024-04-16T03:11:06.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.596+0000 7f936cff9700 1 -- 172.21.15.151:0/727393093 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f937002cd30 con 0x7f935c06ecc0 2024-04-16T03:11:06.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.598+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f935c06ecc0 msgr2=0x7f935c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.598+0000 7f937531a700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f935c06ecc0 0x7f935c071180 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9360009a10 tx=0x7f9360009190 comp rx=0 tx=0).stop 2024-04-16T03:11:06.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.598+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9370105a40 msgr2=0x7f93700fc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.598+0000 7f937531a700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9370105a40 0x7f93700fc420 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9358006010 tx=0x7f935800bd70 comp rx=0 tx=0).stop 2024-04-16T03:11:06.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.598+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 shutdown_connections 2024-04-16T03:11:06.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.599+0000 7f937531a700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f935c06ecc0 0x7f935c071180 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.599+0000 7f937531a700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93700ffa40 0x7f93700fbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.599+0000 7f937531a700 1 --2- 172.21.15.151:0/727393093 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9370105a40 0x7f93700fc420 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.599+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 >> 172.21.15.151:0/727393093 conn(0x7f93700fb570 msgr2=0x7f9370068f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:06.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.599+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 shutdown_connections 2024-04-16T03:11:06.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.599+0000 7f937531a700 1 -- 172.21.15.151:0/727393093 wait complete. 2024-04-16T03:11:06.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.702+0000 7f58fbf3a700 1 -- 172.21.15.151:0/2917817488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 msgr2=0x7f58f4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.702+0000 7f58fbf3a700 1 --2- 172.21.15.151:0/2917817488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 0x7f58f4105e20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009910 tx=0x7f58e8009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:06.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.703+0000 7f58fbf3a700 1 -- 172.21.15.151:0/2917817488 shutdown_connections 2024-04-16T03:11:06.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.703+0000 7f58fbf3a700 1 --2- 172.21.15.151:0/2917817488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40ffa40 0x7f58f40ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.703+0000 7f58fbf3a700 1 --2- 172.21.15.151:0/2917817488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 0x7f58f4105e20 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.703+0000 7f58fbf3a700 1 -- 172.21.15.151:0/2917817488 >> 172.21.15.151:0/2917817488 conn(0x7f58f40fb570 msgr2=0x7f58f40fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:06.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.703+0000 7f58fbf3a700 1 -- 172.21.15.151:0/2917817488 shutdown_connections 2024-04-16T03:11:06.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.703+0000 7f58fbf3a700 1 -- 172.21.15.151:0/2917817488 wait complete. 2024-04-16T03:11:06.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58fbf3a700 1 Processor -- start 2024-04-16T03:11:06.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58fbf3a700 1 -- start start 2024-04-16T03:11:06.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58fbf3a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40ffa40 0x7f58f4198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58fbf3a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 0x7f58f4198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58fbf3a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f4199530 con 0x7f58f4105a40 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58fbf3a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f419d340 con 0x7f58f40ffa40 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58f94d5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 0x7f58f4198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58f94d5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 0x7f58f4198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57766/0 (socket says 172.21.15.151:57766) 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58f94d5700 1 -- 172.21.15.151:0/79150541 learned_addr learned my addr 172.21.15.151:0/79150541 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58f9cd6700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40ffa40 0x7f58f4198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58f94d5700 1 -- 172.21.15.151:0/79150541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40ffa40 msgr2=0x7f58f4198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58f94d5700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40ffa40 0x7f58f4198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.704+0000 7f58f94d5700 1 -- 172.21.15.151:0/79150541 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80095f0 con 0x7f58f4105a40 2024-04-16T03:11:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.705+0000 7f58f94d5700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 0x7f58f4198ea0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f58f000e940 tx=0x7f58f000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:06.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.705+0000 7f58e6ffd700 1 -- 172.21.15.151:0/79150541 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58f000cba0 con 0x7f58f4105a40 2024-04-16T03:11:06.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.705+0000 7f58e6ffd700 1 -- 172.21.15.151:0/79150541 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58f000ee90 con 0x7f58f4105a40 2024-04-16T03:11:06.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.705+0000 7f58e6ffd700 1 -- 172.21.15.151:0/79150541 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58f00055d0 con 0x7f58f4105a40 2024-04-16T03:11:06.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.705+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f419d620 con 0x7f58f4105a40 2024-04-16T03:11:06.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.705+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f419db70 con 0x7f58f4105a40 2024-04-16T03:11:06.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.706+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f404ea90 con 0x7f58f4105a40 2024-04-16T03:11:06.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.706+0000 7f58e6ffd700 1 -- 172.21.15.151:0/79150541 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f58f0004750 con 0x7f58f4105a40 2024-04-16T03:11:06.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.707+0000 7f58e6ffd700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58e006ecc0 0x7f58e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:06.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.707+0000 7f58e6ffd700 1 -- 172.21.15.151:0/79150541 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f58f0014070 con 0x7f58f4105a40 2024-04-16T03:11:06.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.707+0000 7f58f9cd6700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58e006ecc0 0x7f58e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:06.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.707+0000 7f58f9cd6700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58e006ecc0 0x7f58e0071180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f58e80059d0 tx=0x7f58e8005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:06.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.712+0000 7f58e6ffd700 1 -- 172.21.15.151:0/79150541 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f58f005baa0 con 0x7f58f4105a40 2024-04-16T03:11:06.922 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.921+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f58f419dec0 con 0x7f58e006ecc0 2024-04-16T03:11:06.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.932+0000 7f58e6ffd700 1 -- 172.21.15.151:0/79150541 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f58f419dec0 con 0x7f58e006ecc0 2024-04-16T03:11:06.933 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:11:06.933 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (32m) 10m ago 43m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:11:06.933 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (44m) 10m ago 44m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:11:06.933 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (43m) 10m ago 43m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:11:06.933 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (30m) 10m ago 44m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (30m) 10m ago 43m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (32m) 10m ago 43m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (26m) 10m ago 38m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (26m) 10m ago 38m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (10m) 10m ago 38m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (25m) 10m ago 38m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (34m) 10m ago 45m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (33m) 10m ago 42m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (31m) 10m ago 45m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (30m) 10m ago 42m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (33m) 10m ago 44m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (33m) 10m ago 43m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:11:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (29m) 10m ago 42m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:11:06.935 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (29m) 10m ago 41m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:11:06.935 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (28m) 10m ago 41m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:11:06.935 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (27m) 10m ago 40m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:11:06.935 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (27m) 10m ago 40m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:11:06.935 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (26m) 10m ago 39m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:11:06.935 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (33m) 10m ago 43m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:11:06.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58e006ecc0 msgr2=0x7f58e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58e006ecc0 0x7f58e0071180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f58e80059d0 tx=0x7f58e8005940 comp rx=0 tx=0).stop 2024-04-16T03:11:06.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 msgr2=0x7f58f4198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:06.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 0x7f58f4198ea0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f58f000e940 tx=0x7f58f000ec50 comp rx=0 tx=0).stop 2024-04-16T03:11:06.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 shutdown_connections 2024-04-16T03:11:06.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f58e006ecc0 0x7f58e0071180 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40ffa40 0x7f58f4198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 --2- 172.21.15.151:0/79150541 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f58f4105a40 0x7f58f4198ea0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:06.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 >> 172.21.15.151:0/79150541 conn(0x7f58f40fb570 msgr2=0x7f58f4068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:06.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 shutdown_connections 2024-04-16T03:11:06.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:06.934+0000 7f58fbf3a700 1 -- 172.21.15.151:0/79150541 wait complete. 2024-04-16T03:11:06.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:06 smithi151 ceph-mon[112233]: pgmap v975: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:07.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:06 smithi153 ceph-mon[87858]: pgmap v975: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:07.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.038+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1199930393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e0105a40 msgr2=0x7fa0e0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.038+0000 7fa0e7a85700 1 --2- 172.21.15.151:0/1199930393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e0105a40 0x7fa0e0105e20 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0009910 tx=0x7fa0d0009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:07.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.038+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1199930393 shutdown_connections 2024-04-16T03:11:07.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.038+0000 7fa0e7a85700 1 --2- 172.21.15.151:0/1199930393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e00ffbd0 0x7fa0e0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.038+0000 7fa0e7a85700 1 --2- 172.21.15.151:0/1199930393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e0105a40 0x7fa0e0105e20 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.038+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1199930393 >> 172.21.15.151:0/1199930393 conn(0x7fa0e00755a0 msgr2=0x7fa0e00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:07.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.038+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1199930393 shutdown_connections 2024-04-16T03:11:07.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.038+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1199930393 wait complete. 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e7a85700 1 Processor -- start 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e7a85700 1 -- start start 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e7a85700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e00ffbd0 0x7fa0e0101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e7a85700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e0105a40 0x7fa0e0101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e7a85700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e0102090 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e7a85700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e01021d0 con 0x7fa0e0105a40 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e5821700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e00ffbd0 0x7fa0e0101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e5821700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e00ffbd0 0x7fa0e0101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57772/0 (socket says 172.21.15.151:57772) 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e5821700 1 -- 172.21.15.151:0/1073686086 learned_addr learned my addr 172.21.15.151:0/1073686086 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e5020700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e0105a40 0x7fa0e0101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e5821700 1 -- 172.21.15.151:0/1073686086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e0105a40 msgr2=0x7fa0e0101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e5821700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e0105a40 0x7fa0e0101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e5821700 1 -- 172.21.15.151:0/1073686086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d00095f0 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.039+0000 7fa0e5821700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e00ffbd0 0x7fa0e0101610 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0006010 tx=0x7fa0d000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.040+0000 7fa0d6ffd700 1 -- 172.21.15.151:0/1073686086 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d001d070 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.040+0000 7fa0d6ffd700 1 -- 172.21.15.151:0/1073686086 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0d0004fb0 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.040+0000 7fa0d6ffd700 1 -- 172.21.15.151:0/1073686086 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d0022ea0 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.040+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e0102450 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.040+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e01a2cd0 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.041+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e0109050 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.041+0000 7fa0d6ffd700 1 -- 172.21.15.151:0/1073686086 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa0d0022470 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.042+0000 7fa0d6ffd700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa0cc06ecc0 0x7fa0cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:07.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.042+0000 7fa0d6ffd700 1 -- 172.21.15.151:0/1073686086 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa0d00903d0 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.042+0000 7fa0e5020700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa0cc06ecc0 0x7fa0cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:07.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.042+0000 7fa0e5020700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa0cc06ecc0 0x7fa0cc071180 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc006fd0 tx=0x7fa0dc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:07.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.047+0000 7fa0d6ffd700 1 -- 172.21.15.151:0/1073686086 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa0d0059230 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.328+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa0e004ea90 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.329+0000 7fa0d6ffd700 1 -- 172.21.15.151:0/1073686086 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fa0d0022720 con 0x7fa0e00ffbd0 2024-04-16T03:11:07.330 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:11:07.330 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:11:07.330 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:11:07.331 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa0cc06ecc0 msgr2=0x7fa0cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa0cc06ecc0 0x7fa0cc071180 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc006fd0 tx=0x7fa0dc008040 comp rx=0 tx=0).stop 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e00ffbd0 msgr2=0x7fa0e0101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e00ffbd0 0x7fa0e0101610 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0006010 tx=0x7fa0d000be20 comp rx=0 tx=0).stop 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 shutdown_connections 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa0cc06ecc0 0x7fa0cc071180 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0e00ffbd0 0x7fa0e0101610 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 --2- 172.21.15.151:0/1073686086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e0105a40 0x7fa0e0101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 >> 172.21.15.151:0/1073686086 conn(0x7fa0e00755a0 msgr2=0x7fa0e00686b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 shutdown_connections 2024-04-16T03:11:07.333 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.331+0000 7fa0e7a85700 1 -- 172.21.15.151:0/1073686086 wait complete. 2024-04-16T03:11:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.436+0000 7fc223451700 1 -- 172.21.15.151:0/687800548 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 msgr2=0x7fc21c1057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.436+0000 7fc223451700 1 --2- 172.21.15.151:0/687800548 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 0x7fc21c1057d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc218009910 tx=0x7fc218009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.437+0000 7fc223451700 1 -- 172.21.15.151:0/687800548 shutdown_connections 2024-04-16T03:11:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.437+0000 7fc223451700 1 --2- 172.21.15.151:0/687800548 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 0x7fc21c1057d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.437+0000 7fc223451700 1 --2- 172.21.15.151:0/687800548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc21c100dd0 0x7fc21c1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.437+0000 7fc223451700 1 -- 172.21.15.151:0/687800548 >> 172.21.15.151:0/687800548 conn(0x7fc21c0fc640 msgr2=0x7fc21c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.437+0000 7fc223451700 1 -- 172.21.15.151:0/687800548 shutdown_connections 2024-04-16T03:11:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.437+0000 7fc223451700 1 -- 172.21.15.151:0/687800548 wait complete. 2024-04-16T03:11:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.437+0000 7fc223451700 1 Processor -- start 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.437+0000 7fc223451700 1 -- start start 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.437+0000 7fc223451700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc21c100dd0 0x7fc21c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc223451700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 0x7fc21c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc223451700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c073b10 con 0x7fc21c101780 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc223451700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c073c50 con 0x7fc21c100dd0 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2209ec700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 0x7fc21c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2209ec700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 0x7fc21c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57794/0 (socket says 172.21.15.151:57794) 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2209ec700 1 -- 172.21.15.151:0/357018672 learned_addr learned my addr 172.21.15.151:0/357018672 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2209ec700 1 -- 172.21.15.151:0/357018672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc21c100dd0 msgr2=0x7fc21c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2211ed700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc21c100dd0 0x7fc21c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2209ec700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc21c100dd0 0x7fc21c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2209ec700 1 -- 172.21.15.151:0/357018672 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2180095f0 con 0x7fc21c101780 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2209ec700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 0x7fc21c0735d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc2180092e0 tx=0x7fc21800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2127fc700 1 -- 172.21.15.151:0/357018672 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc21801d070 con 0x7fc21c101780 2024-04-16T03:11:07.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2127fc700 1 -- 172.21.15.151:0/357018672 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc21800f460 con 0x7fc21c101780 2024-04-16T03:11:07.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc2127fc700 1 -- 172.21.15.151:0/357018672 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2180175f0 con 0x7fc21c101780 2024-04-16T03:11:07.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21c073ed0 con 0x7fc21c101780 2024-04-16T03:11:07.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.438+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21c1a2da0 con 0x7fc21c101780 2024-04-16T03:11:07.442 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.439+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21c109120 con 0x7fc21c101780 2024-04-16T03:11:07.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.440+0000 7fc2127fc700 1 -- 172.21.15.151:0/357018672 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc21800f5d0 con 0x7fc21c101780 2024-04-16T03:11:07.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.440+0000 7fc2127fc700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc20806ecc0 0x7fc208071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:07.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.441+0000 7fc2211ed700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc20806ecc0 0x7fc208071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:07.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.441+0000 7fc2127fc700 1 -- 172.21.15.151:0/357018672 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc2180902f0 con 0x7fc21c101780 2024-04-16T03:11:07.443 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.441+0000 7fc2211ed700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc20806ecc0 0x7fc208071180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc20c006fd0 tx=0x7fc20c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:07.446 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.445+0000 7fc2127fc700 1 -- 172.21.15.151:0/357018672 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc218059120 con 0x7fc21c101780 2024-04-16T03:11:07.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.685+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc21c04ea90 con 0x7fc21c101780 2024-04-16T03:11:07.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.686+0000 7fc2127fc700 1 -- 172.21.15.151:0/357018672 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fc21805c7e0 con 0x7fc21c101780 2024-04-16T03:11:07.687 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:11:07.687 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:11:07.687 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:11:07.687 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:11:07.687 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:11:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:11:07.689 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:11:07.690 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:11:07.690 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:11:07.690 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:11:07.690 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:11:07.690 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:11:07.690 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:11:07.690 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:11:07.690 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:11:07.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc20806ecc0 msgr2=0x7fc208071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc20806ecc0 0x7fc208071180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc20c006fd0 tx=0x7fc20c008040 comp rx=0 tx=0).stop 2024-04-16T03:11:07.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 msgr2=0x7fc21c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 0x7fc21c0735d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc2180092e0 tx=0x7fc21800be20 comp rx=0 tx=0).stop 2024-04-16T03:11:07.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 shutdown_connections 2024-04-16T03:11:07.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc20806ecc0 0x7fc208071180 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc21c100dd0 0x7fc21c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 --2- 172.21.15.151:0/357018672 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc21c101780 0x7fc21c0735d0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 >> 172.21.15.151:0/357018672 conn(0x7fc21c0fc640 msgr2=0x7fc21c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:07.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 shutdown_connections 2024-04-16T03:11:07.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.689+0000 7fc223451700 1 -- 172.21.15.151:0/357018672 wait complete. 2024-04-16T03:11:07.692 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:11:07.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 -- 172.21.15.151:0/2492133956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 msgr2=0x7fc904104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 --2- 172.21.15.151:0/2492133956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 0x7fc904104570 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009910 tx=0x7fc8f4009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:07.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 -- 172.21.15.151:0/2492133956 shutdown_connections 2024-04-16T03:11:07.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 --2- 172.21.15.151:0/2492133956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9040ff260 0x7fc9040ff6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 --2- 172.21.15.151:0/2492133956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 0x7fc904104570 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 -- 172.21.15.151:0/2492133956 >> 172.21.15.151:0/2492133956 conn(0x7fc904074ce0 msgr2=0x7fc9040750f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:07.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 -- 172.21.15.151:0/2492133956 shutdown_connections 2024-04-16T03:11:07.798 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 -- 172.21.15.151:0/2492133956 wait complete. 2024-04-16T03:11:07.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 Processor -- start 2024-04-16T03:11:07.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.796+0000 7fc90a37a700 1 -- start start 2024-04-16T03:11:07.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc90a37a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9040ff260 0x7fc9041975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:07.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc90a37a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 0x7fc904197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:07.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc90a37a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc904198100 con 0x7fc904104190 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc90a37a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc904198240 con 0x7fc9040ff260 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc9037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 0x7fc904197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc9037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 0x7fc904197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57814/0 (socket says 172.21.15.151:57814) 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc9037fe700 1 -- 172.21.15.151:0/2669173631 learned_addr learned my addr 172.21.15.151:0/2669173631 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc903fff700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9040ff260 0x7fc9041975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc9037fe700 1 -- 172.21.15.151:0/2669173631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9040ff260 msgr2=0x7fc9041975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc9037fe700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9040ff260 0x7fc9041975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc9037fe700 1 -- 172.21.15.151:0/2669173631 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f40095f0 con 0x7fc904104190 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc9037fe700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 0x7fc904197ae0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800c8a0 tx=0x7fc8f800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.797+0000 7fc9017fa700 1 -- 172.21.15.151:0/2669173631 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8f800cea0 con 0x7fc904104190 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.798+0000 7fc9017fa700 1 -- 172.21.15.151:0/2669173631 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8f8004830 con 0x7fc904104190 2024-04-16T03:11:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.798+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9041a0610 con 0x7fc904104190 2024-04-16T03:11:07.802 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.798+0000 7fc9017fa700 1 -- 172.21.15.151:0/2669173631 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8f80055b0 con 0x7fc904104190 2024-04-16T03:11:07.802 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.798+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9041a0ad0 con 0x7fc904104190 2024-04-16T03:11:07.802 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.799+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90404ea90 con 0x7fc904104190 2024-04-16T03:11:07.802 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.799+0000 7fc9017fa700 1 -- 172.21.15.151:0/2669173631 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc8f800f450 con 0x7fc904104190 2024-04-16T03:11:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.799+0000 7fc9017fa700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8ec06ecc0 0x7fc8ec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.800+0000 7fc9017fa700 1 -- 172.21.15.151:0/2669173631 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc8f808eb50 con 0x7fc904104190 2024-04-16T03:11:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.800+0000 7fc903fff700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8ec06ecc0 0x7fc8ec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.800+0000 7fc903fff700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8ec06ecc0 0x7fc8ec071180 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400b5c0 tx=0x7fc8f401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:07.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:07.804+0000 7fc9017fa700 1 -- 172.21.15.151:0/2669173631 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc8f8057900 con 0x7fc904104190 2024-04-16T03:11:07.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:07 smithi151 ceph-mon[112233]: from='client.35772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:07.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:07 smithi151 ceph-mon[112233]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:07.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:07 smithi151 ceph-mon[112233]: from='client.35780 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:07.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:07 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1073686086' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:11:07.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:07 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/357018672' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:11:08.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:07 smithi153 ceph-mon[87858]: from='client.35772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:08.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:07 smithi153 ceph-mon[87858]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:08.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:07 smithi153 ceph-mon[87858]: from='client.35780 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:08.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:07 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1073686086' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:11:08.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:07 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/357018672' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:11:08.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.024+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc904103500 con 0x7fc8ec06ecc0 2024-04-16T03:11:08.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.026+0000 7fc9017fa700 1 -- 172.21.15.151:0/2669173631 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc904103500 con 0x7fc8ec06ecc0 2024-04-16T03:11:08.027 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:11:08.027 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:11:08.027 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:11:08.027 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:11:08.027 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:11:08.028 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:11:08.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8ec06ecc0 msgr2=0x7fc8ec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:08.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8ec06ecc0 0x7fc8ec071180 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400b5c0 tx=0x7fc8f401a040 comp rx=0 tx=0).stop 2024-04-16T03:11:08.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 msgr2=0x7fc904197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:08.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 0x7fc904197ae0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800c8a0 tx=0x7fc8f800cc60 comp rx=0 tx=0).stop 2024-04-16T03:11:08.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 shutdown_connections 2024-04-16T03:11:08.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc8ec06ecc0 0x7fc8ec071180 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:08.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9040ff260 0x7fc9041975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:08.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 --2- 172.21.15.151:0/2669173631 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc904104190 0x7fc904197ae0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:08.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 >> 172.21.15.151:0/2669173631 conn(0x7fc904074ce0 msgr2=0x7fc904102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:08.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 shutdown_connections 2024-04-16T03:11:08.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.028+0000 7fc90a37a700 1 -- 172.21.15.151:0/2669173631 wait complete. 2024-04-16T03:11:08.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.133+0000 7f1ef0262700 1 -- 172.21.15.151:0/927618708 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 msgr2=0x7f1ee8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:08.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.133+0000 7f1ef0262700 1 --2- 172.21.15.151:0/927618708 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 0x7f1ee8100050 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8009910 tx=0x7f1ed8009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:08.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.133+0000 7f1ef0262700 1 -- 172.21.15.151:0/927618708 shutdown_connections 2024-04-16T03:11:08.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.133+0000 7f1ef0262700 1 --2- 172.21.15.151:0/927618708 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 0x7f1ee8100050 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:08.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.133+0000 7f1ef0262700 1 --2- 172.21.15.151:0/927618708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee8105a40 0x7f1ee8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:08.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.133+0000 7f1ef0262700 1 -- 172.21.15.151:0/927618708 >> 172.21.15.151:0/927618708 conn(0x7f1ee80755a0 msgr2=0x7f1ee80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:08.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.133+0000 7f1ef0262700 1 -- 172.21.15.151:0/927618708 shutdown_connections 2024-04-16T03:11:08.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.133+0000 7f1ef0262700 1 -- 172.21.15.151:0/927618708 wait complete. 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.134+0000 7f1ef0262700 1 Processor -- start 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.134+0000 7f1ef0262700 1 -- start start 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.134+0000 7f1ef0262700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 0x7f1ee8195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.134+0000 7f1ef0262700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee8105a40 0x7f1ee81959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.134+0000 7f1ef0262700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee8195f10 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.134+0000 7f1ef0262700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee8196080 con 0x7f1ee8105a40 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.134+0000 7f1eedffe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 0x7f1ee8195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.134+0000 7f1eedffe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 0x7f1ee8195490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57840/0 (socket says 172.21.15.151:57840) 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.134+0000 7f1eedffe700 1 -- 172.21.15.151:0/427023939 learned_addr learned my addr 172.21.15.151:0/427023939 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1eed7fd700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee8105a40 0x7f1ee81959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1eedffe700 1 -- 172.21.15.151:0/427023939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee8105a40 msgr2=0x7f1ee81959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1eedffe700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee8105a40 0x7f1ee81959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1eedffe700 1 -- 172.21.15.151:0/427023939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed80095f0 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1eedffe700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 0x7f1ee8195490 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400b4b0 tx=0x7f1ee400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1edeffd700 1 -- 172.21.15.151:0/427023939 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ee4011800 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee81a2b00 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1edeffd700 1 -- 172.21.15.151:0/427023939 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ee4011e20 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1edeffd700 1 -- 172.21.15.151:0/427023939 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ee400f2e0 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.135+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee81a2e60 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.136+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee8108060 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.137+0000 7f1edeffd700 1 -- 172.21.15.151:0/427023939 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1ee4010380 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.137+0000 7f1edeffd700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ed406ecc0 0x7f1ed4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:08.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.137+0000 7f1edeffd700 1 -- 172.21.15.151:0/427023939 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1ee408e900 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.137+0000 7f1eed7fd700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ed406ecc0 0x7f1ed4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:08.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.137+0000 7f1eed7fd700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ed406ecc0 0x7f1ed4071180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8068d70 tx=0x7f1ed801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:08.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.142+0000 7f1edeffd700 1 -- 172.21.15.151:0/427023939 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1ee405b310 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.420+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1ee8196240 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.420+0000 7f1edeffd700 1 -- 172.21.15.151:0/427023939 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f1ee405aea0 con 0x7f1ee80ffbd0 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:11:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:11:08.423 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:11:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:11:08.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ed406ecc0 msgr2=0x7f1ed4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ed406ecc0 0x7f1ed4071180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8068d70 tx=0x7f1ed801a040 comp rx=0 tx=0).stop 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 msgr2=0x7f1ee8195490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 0x7f1ee8195490 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400b4b0 tx=0x7f1ee400b870 comp rx=0 tx=0).stop 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 shutdown_connections 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1ed406ecc0 0x7f1ed4071180 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ee80ffbd0 0x7f1ee8195490 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 --2- 172.21.15.151:0/427023939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee8105a40 0x7f1ee81959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 >> 172.21.15.151:0/427023939 conn(0x7f1ee80755a0 msgr2=0x7f1ee80fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 shutdown_connections 2024-04-16T03:11:08.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:08.422+0000 7f1ef0262700 1 -- 172.21.15.151:0/427023939 wait complete. 2024-04-16T03:11:09.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:08 smithi153 ceph-mon[87858]: pgmap v976: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:09.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:08 smithi153 ceph-mon[87858]: from='client.35792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:09.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:08 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/427023939' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:11:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:08 smithi151 ceph-mon[112233]: pgmap v976: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:08 smithi151 ceph-mon[112233]: from='client.35792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:08 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/427023939' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:11:10.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:11:10.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:11:11.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:10 smithi153 ceph-mon[87858]: pgmap v977: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:11.143 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:10 smithi151 ceph-mon[112233]: pgmap v977: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:13.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:12 smithi153 ceph-mon[87858]: pgmap v978: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:13.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:12 smithi151 ceph-mon[112233]: pgmap v978: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:15.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:14 smithi151 ceph-mon[112233]: pgmap v979: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:15.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:14 smithi153 ceph-mon[87858]: pgmap v979: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:17.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:16 smithi151 ceph-mon[112233]: pgmap v980: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:17.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:16 smithi153 ceph-mon[87858]: pgmap v980: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:18.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:18 smithi151 ceph-mon[112233]: pgmap v981: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:19.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:18 smithi153 ceph-mon[87858]: pgmap v981: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:21.143 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:20 smithi151 ceph-mon[112233]: pgmap v982: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:21.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:20 smithi153 ceph-mon[87858]: pgmap v982: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:23.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:22 smithi151 ceph-mon[112233]: pgmap v983: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:23.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:22 smithi153 ceph-mon[87858]: pgmap v983: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:24 smithi153 ceph-mon[87858]: pgmap v984: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:11:25.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:24 smithi151 ceph-mon[112233]: pgmap v984: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:25.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:11:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:26 smithi151 ceph-mon[112233]: pgmap v985: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:27.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:26 smithi153 ceph-mon[87858]: pgmap v985: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:28.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:28 smithi151 ceph-mon[112233]: pgmap v986: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:29.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:28 smithi153 ceph-mon[87858]: pgmap v986: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:31.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:30 smithi151 ceph-mon[112233]: pgmap v987: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:31.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:30 smithi153 ceph-mon[87858]: pgmap v987: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:33.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:32 smithi151 ceph-mon[112233]: pgmap v988: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:33.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:32 smithi153 ceph-mon[87858]: pgmap v988: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:34 smithi151 ceph-mon[112233]: pgmap v989: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:35.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:34 smithi153 ceph-mon[87858]: pgmap v989: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:37.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:36 smithi151 ceph-mon[112233]: pgmap v990: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:37.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:36 smithi153 ceph-mon[87858]: pgmap v990: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:38.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.541+0000 7ff649507700 1 -- 172.21.15.151:0/142111836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 msgr2=0x7ff644105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.541+0000 7ff649507700 1 --2- 172.21.15.151:0/142111836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 0x7ff644105790 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff638009910 tx=0x7ff638009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.542+0000 7ff649507700 1 -- 172.21.15.151:0/142111836 shutdown_connections 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.542+0000 7ff649507700 1 --2- 172.21.15.151:0/142111836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 0x7ff644105790 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.542+0000 7ff649507700 1 --2- 172.21.15.151:0/142111836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff644068670 0x7ff644068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.542+0000 7ff649507700 1 -- 172.21.15.151:0/142111836 >> 172.21.15.151:0/142111836 conn(0x7ff6440755a0 msgr2=0x7ff6440759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.542+0000 7ff649507700 1 -- 172.21.15.151:0/142111836 shutdown_connections 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.542+0000 7ff649507700 1 -- 172.21.15.151:0/142111836 wait complete. 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.542+0000 7ff649507700 1 Processor -- start 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.542+0000 7ff649507700 1 -- start start 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff649507700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff644068670 0x7ff6440ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff649507700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 0x7ff6441001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:38.545 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff649507700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff644100720 con 0x7ff644069020 2024-04-16T03:11:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff649507700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff644100860 con 0x7ff644068670 2024-04-16T03:11:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff6427fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 0x7ff6441001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff6427fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 0x7ff6441001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43584/0 (socket says 172.21.15.151:43584) 2024-04-16T03:11:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff6427fc700 1 -- 172.21.15.151:0/251893558 learned_addr learned my addr 172.21.15.151:0/251893558 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff642ffd700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff644068670 0x7ff6440ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff6427fc700 1 -- 172.21.15.151:0/251893558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff644068670 msgr2=0x7ff6440ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff6427fc700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff644068670 0x7ff6440ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff6427fc700 1 -- 172.21.15.151:0/251893558 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6380095f0 con 0x7ff644069020 2024-04-16T03:11:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.543+0000 7ff6427fc700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 0x7ff6441001e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff638000c00 tx=0x7ff63800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.544+0000 7ff62bfff700 1 -- 172.21.15.151:0/251893558 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63801d070 con 0x7ff644069020 2024-04-16T03:11:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.544+0000 7ff62bfff700 1 -- 172.21.15.151:0/251893558 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff638005dc0 con 0x7ff644069020 2024-04-16T03:11:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.544+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff644100ae0 con 0x7ff644069020 2024-04-16T03:11:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.544+0000 7ff62bfff700 1 -- 172.21.15.151:0/251893558 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff638017400 con 0x7ff644069020 2024-04-16T03:11:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.544+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6441a2c00 con 0x7ff644069020 2024-04-16T03:11:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.545+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6441090a0 con 0x7ff644069020 2024-04-16T03:11:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.545+0000 7ff62bfff700 1 -- 172.21.15.151:0/251893558 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff63800f460 con 0x7ff644069020 2024-04-16T03:11:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.546+0000 7ff62bfff700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff62c06ecc0 0x7ff62c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.546+0000 7ff62bfff700 1 -- 172.21.15.151:0/251893558 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff6380902b0 con 0x7ff644069020 2024-04-16T03:11:38.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.546+0000 7ff642ffd700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff62c06ecc0 0x7ff62c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:38.549 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.546+0000 7ff642ffd700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff62c06ecc0 0x7ff62c071180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff634005950 tx=0x7ff63400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:38.552 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.551+0000 7ff62bfff700 1 -- 172.21.15.151:0/251893558 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff638059060 con 0x7ff644069020 2024-04-16T03:11:38.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.769+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff64402cce0 con 0x7ff62c06ecc0 2024-04-16T03:11:38.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.771+0000 7ff62bfff700 1 -- 172.21.15.151:0/251893558 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff64402cce0 con 0x7ff62c06ecc0 2024-04-16T03:11:38.774 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff62c06ecc0 msgr2=0x7ff62c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff62c06ecc0 0x7ff62c071180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff634005950 tx=0x7ff63400b410 comp rx=0 tx=0).stop 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 msgr2=0x7ff6441001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 0x7ff6441001e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff638000c00 tx=0x7ff63800bd70 comp rx=0 tx=0).stop 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 shutdown_connections 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff62c06ecc0 0x7ff62c071180 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff644068670 0x7ff6440ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 --2- 172.21.15.151:0/251893558 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff644069020 0x7ff6441001e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 >> 172.21.15.151:0/251893558 conn(0x7ff6440755a0 msgr2=0x7ff644105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 shutdown_connections 2024-04-16T03:11:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.773+0000 7ff649507700 1 -- 172.21.15.151:0/251893558 wait complete. 2024-04-16T03:11:38.785 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:11:38.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.884+0000 7fa91c026700 1 -- 172.21.15.151:0/1257350685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 msgr2=0x7fa914105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:38.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.884+0000 7fa91c026700 1 --2- 172.21.15.151:0/1257350685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 0x7fa914105790 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fa910009910 tx=0x7fa910009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:38.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.884+0000 7fa91c026700 1 -- 172.21.15.151:0/1257350685 shutdown_connections 2024-04-16T03:11:38.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.884+0000 7fa91c026700 1 --2- 172.21.15.151:0/1257350685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 0x7fa914105790 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:38.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.884+0000 7fa91c026700 1 --2- 172.21.15.151:0/1257350685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914068670 0x7fa914068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:38.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.884+0000 7fa91c026700 1 -- 172.21.15.151:0/1257350685 >> 172.21.15.151:0/1257350685 conn(0x7fa9140755a0 msgr2=0x7fa9140759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:38.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.884+0000 7fa91c026700 1 -- 172.21.15.151:0/1257350685 shutdown_connections 2024-04-16T03:11:38.886 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.884+0000 7fa91c026700 1 -- 172.21.15.151:0/1257350685 wait complete. 2024-04-16T03:11:38.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.885+0000 7fa91c026700 1 Processor -- start 2024-04-16T03:11:38.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.885+0000 7fa91c026700 1 -- start start 2024-04-16T03:11:38.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.885+0000 7fa91c026700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914068670 0x7fa91419e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:38.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.885+0000 7fa91c026700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 0x7fa91419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.885+0000 7fa91c026700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91419f320 con 0x7fa914069020 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.885+0000 7fa91c026700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914198860 con 0x7fa914068670 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.885+0000 7fa9195c1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 0x7fa91419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.885+0000 7fa9195c1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 0x7fa91419ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43604/0 (socket says 172.21.15.151:43604) 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.885+0000 7fa9195c1700 1 -- 172.21.15.151:0/252309797 learned_addr learned my addr 172.21.15.151:0/252309797 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa919dc2700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914068670 0x7fa91419e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa9195c1700 1 -- 172.21.15.151:0/252309797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914068670 msgr2=0x7fa91419e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa9195c1700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914068670 0x7fa91419e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa9195c1700 1 -- 172.21.15.151:0/252309797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9100095f0 con 0x7fa914069020 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa9195c1700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 0x7fa91419ed20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa91000bda0 tx=0x7fa91000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa90affd700 1 -- 172.21.15.151:0/252309797 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa91001d070 con 0x7fa914069020 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa914198ae0 con 0x7fa914069020 2024-04-16T03:11:38.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa90affd700 1 -- 172.21.15.151:0/252309797 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa910004fb0 con 0x7fa914069020 2024-04-16T03:11:38.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa90affd700 1 -- 172.21.15.151:0/252309797 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa91000f480 con 0x7fa914069020 2024-04-16T03:11:38.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.886+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa914198fd0 con 0x7fa914069020 2024-04-16T03:11:38.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.887+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91404ea90 con 0x7fa914069020 2024-04-16T03:11:38.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.889+0000 7fa90affd700 1 -- 172.21.15.151:0/252309797 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa910022870 con 0x7fa914069020 2024-04-16T03:11:38.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.889+0000 7fa90affd700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:38.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.889+0000 7fa90affd700 1 -- 172.21.15.151:0/252309797 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa9100903f0 con 0x7fa914069020 2024-04-16T03:11:38.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.889+0000 7fa919dc2700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:38.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.890+0000 7fa919dc2700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa904005ea0 tx=0x7fa904005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:38.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:38.894+0000 7fa90affd700 1 -- 172.21.15.151:0/252309797 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa91005cd80 con 0x7fa914069020 2024-04-16T03:11:39.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.114+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa914199300 con 0x7fa90006ecc0 2024-04-16T03:11:39.116 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:38 smithi151 ceph-mon[112233]: pgmap v991: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:39.116 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:11:39.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.115+0000 7fa90affd700 1 -- 172.21.15.151:0/252309797 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa914199300 con 0x7fa90006ecc0 2024-04-16T03:11:39.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 msgr2=0x7fa900071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa904005ea0 tx=0x7fa904005e30 comp rx=0 tx=0).stop 2024-04-16T03:11:39.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 msgr2=0x7fa91419ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 0x7fa91419ed20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa91000bda0 tx=0x7fa91000be80 comp rx=0 tx=0).stop 2024-04-16T03:11:39.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 shutdown_connections 2024-04-16T03:11:39.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa90006ecc0 0x7fa900071180 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914068670 0x7fa91419e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 --2- 172.21.15.151:0/252309797 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa914069020 0x7fa91419ed20 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 >> 172.21.15.151:0/252309797 conn(0x7fa9140755a0 msgr2=0x7fa914102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:39.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 shutdown_connections 2024-04-16T03:11:39.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.118+0000 7fa91c026700 1 -- 172.21.15.151:0/252309797 wait complete. 2024-04-16T03:11:39.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.222+0000 7f386fc3b700 1 -- 172.21.15.151:0/648345089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 msgr2=0x7f3868105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.222+0000 7f386fc3b700 1 --2- 172.21.15.151:0/648345089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 0x7f3868105e20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3858009910 tx=0x7f3858009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:39.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.222+0000 7f386fc3b700 1 -- 172.21.15.151:0/648345089 shutdown_connections 2024-04-16T03:11:39.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.222+0000 7f386fc3b700 1 --2- 172.21.15.151:0/648345089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38680ffbd0 0x7f3868100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.222+0000 7f386fc3b700 1 --2- 172.21.15.151:0/648345089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 0x7f3868105e20 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.222+0000 7f386fc3b700 1 -- 172.21.15.151:0/648345089 >> 172.21.15.151:0/648345089 conn(0x7f38680755a0 msgr2=0x7f38680759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.222+0000 7f386fc3b700 1 -- 172.21.15.151:0/648345089 shutdown_connections 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.222+0000 7f386fc3b700 1 -- 172.21.15.151:0/648345089 wait complete. 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386fc3b700 1 Processor -- start 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386fc3b700 1 -- start start 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386fc3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38680ffbd0 0x7f38681975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386fc3b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 0x7f3868197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386fc3b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3868198110 con 0x7f3868105a40 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386fc3b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3868198250 con 0x7f38680ffbd0 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386d1d6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 0x7f3868197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386d1d6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 0x7f3868197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43626/0 (socket says 172.21.15.151:43626) 2024-04-16T03:11:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386d1d6700 1 -- 172.21.15.151:0/2387587729 learned_addr learned my addr 172.21.15.151:0/2387587729 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386d9d7700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38680ffbd0 0x7f38681975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386d1d6700 1 -- 172.21.15.151:0/2387587729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38680ffbd0 msgr2=0x7f38681975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386d1d6700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38680ffbd0 0x7f38681975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386d1d6700 1 -- 172.21.15.151:0/2387587729 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38580095f0 con 0x7f3868105a40 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.223+0000 7f386d1d6700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 0x7f3868197af0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f386400d8d0 tx=0x7f386400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.224+0000 7f385effd700 1 -- 172.21.15.151:0/2387587729 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3864009940 con 0x7f3868105a40 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.224+0000 7f385effd700 1 -- 172.21.15.151:0/2387587729 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3864010460 con 0x7f3868105a40 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.224+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38681a0620 con 0x7f3868105a40 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.224+0000 7f385effd700 1 -- 172.21.15.151:0/2387587729 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f386400f5f0 con 0x7f3868105a40 2024-04-16T03:11:39.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.224+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38681a0ae0 con 0x7f3868105a40 2024-04-16T03:11:39.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.225+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3868108040 con 0x7f3868105a40 2024-04-16T03:11:39.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.225+0000 7f385effd700 1 -- 172.21.15.151:0/2387587729 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f38640105d0 con 0x7f3868105a40 2024-04-16T03:11:39.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.226+0000 7f385effd700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f385406ecc0 0x7f3854071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:39.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.226+0000 7f385effd700 1 -- 172.21.15.151:0/2387587729 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f386408ee80 con 0x7f3868105a40 2024-04-16T03:11:39.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.226+0000 7f386d9d7700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f385406ecc0 0x7f3854071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:39.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.226+0000 7f386d9d7700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f385406ecc0 0x7f3854071180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f385800b5c0 tx=0x7f3858005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:39.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.230+0000 7f385effd700 1 -- 172.21.15.151:0/2387587729 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3864057c30 con 0x7f3868105a40 2024-04-16T03:11:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:38 smithi153 ceph-mon[87858]: pgmap v991: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:11:39.440 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.438+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f386802cd50 con 0x7f385406ecc0 2024-04-16T03:11:39.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.450+0000 7f385effd700 1 -- 172.21.15.151:0/2387587729 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f386802cd50 con 0x7f385406ecc0 2024-04-16T03:11:39.451 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:11:39.451 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (33m) 11m ago 44m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:11:39.451 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (44m) 11m ago 44m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:11:39.451 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (43m) 11m ago 43m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (30m) 11m ago 44m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (30m) 11m ago 43m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (32m) 11m ago 44m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (27m) 11m ago 39m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (26m) 11m ago 38m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (11m) 11m ago 38m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (26m) 11m ago 38m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (35m) 11m ago 45m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (34m) 11m ago 43m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (31m) 11m ago 46m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:11:39.452 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (31m) 11m ago 43m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:11:39.453 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (33m) 11m ago 44m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:11:39.453 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (33m) 11m ago 43m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:11:39.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (30m) 11m ago 42m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:11:39.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (29m) 11m ago 42m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:11:39.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (29m) 11m ago 41m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:11:39.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (28m) 11m ago 41m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:11:39.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (27m) 11m ago 40m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:11:39.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (27m) 11m ago 40m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:11:39.453 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (33m) 11m ago 44m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:11:39.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f385406ecc0 msgr2=0x7f3854071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.454 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f385406ecc0 0x7f3854071180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f385800b5c0 tx=0x7f3858005960 comp rx=0 tx=0).stop 2024-04-16T03:11:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 msgr2=0x7f3868197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 0x7f3868197af0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f386400d8d0 tx=0x7f386400dc90 comp rx=0 tx=0).stop 2024-04-16T03:11:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 shutdown_connections 2024-04-16T03:11:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f385406ecc0 0x7f3854071180 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38680ffbd0 0x7f38681975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 --2- 172.21.15.151:0/2387587729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3868105a40 0x7f3868197af0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 >> 172.21.15.151:0/2387587729 conn(0x7f38680755a0 msgr2=0x7f38680fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 shutdown_connections 2024-04-16T03:11:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.452+0000 7f386fc3b700 1 -- 172.21.15.151:0/2387587729 wait complete. 2024-04-16T03:11:39.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.556+0000 7f2fa9337700 1 -- 172.21.15.151:0/4275214963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 msgr2=0x7f2fa41057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.556+0000 7f2fa9337700 1 --2- 172.21.15.151:0/4275214963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 0x7f2fa41057c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009910 tx=0x7f2f8c009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.556+0000 7f2fa9337700 1 -- 172.21.15.151:0/4275214963 shutdown_connections 2024-04-16T03:11:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.556+0000 7f2fa9337700 1 --2- 172.21.15.151:0/4275214963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 0x7f2fa41057c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.556+0000 7f2fa9337700 1 --2- 172.21.15.151:0/4275214963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa4100dc0 0x7f2fa41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.556+0000 7f2fa9337700 1 -- 172.21.15.151:0/4275214963 >> 172.21.15.151:0/4275214963 conn(0x7f2fa40fc650 msgr2=0x7f2fa40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.556+0000 7f2fa9337700 1 -- 172.21.15.151:0/4275214963 shutdown_connections 2024-04-16T03:11:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.556+0000 7f2fa9337700 1 -- 172.21.15.151:0/4275214963 wait complete. 2024-04-16T03:11:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.556+0000 7f2fa9337700 1 Processor -- start 2024-04-16T03:11:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2fa9337700 1 -- start start 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2fa9337700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa4100dc0 0x7f2fa419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2fa9337700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 0x7f2fa419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2fa9337700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa419d130 con 0x7f2fa4101770 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2fa9337700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa41965e0 con 0x7f2fa4100dc0 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2f9a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 0x7f2fa419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2f9a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 0x7f2fa419caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43650/0 (socket says 172.21.15.151:43650) 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2f9a7fc700 1 -- 172.21.15.151:0/2751795864 learned_addr learned my addr 172.21.15.151:0/2751795864 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2fa2ffd700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa4100dc0 0x7f2fa419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2f9a7fc700 1 -- 172.21.15.151:0/2751795864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa4100dc0 msgr2=0x7f2fa419c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2f9a7fc700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa4100dc0 0x7f2fa419c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2f9a7fc700 1 -- 172.21.15.151:0/2751795864 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c0095f0 con 0x7f2fa4101770 2024-04-16T03:11:39.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2fa2ffd700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa4100dc0 0x7f2fa419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:11:39.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2f9a7fc700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 0x7f2fa419caa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0092e0 tx=0x7f2f8c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:39.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.557+0000 7f2fa0ff9700 1 -- 172.21.15.151:0/2751795864 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f8c01d070 con 0x7f2fa4101770 2024-04-16T03:11:39.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.558+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa4196860 con 0x7f2fa4101770 2024-04-16T03:11:39.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.558+0000 7f2fa0ff9700 1 -- 172.21.15.151:0/2751795864 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f8c005dc0 con 0x7f2fa4101770 2024-04-16T03:11:39.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.558+0000 7f2fa0ff9700 1 -- 172.21.15.151:0/2751795864 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f8c017400 con 0x7f2fa4101770 2024-04-16T03:11:39.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.558+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa4196d50 con 0x7f2fa4101770 2024-04-16T03:11:39.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.559+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa41091c0 con 0x7f2fa4101770 2024-04-16T03:11:39.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.559+0000 7f2fa0ff9700 1 -- 172.21.15.151:0/2751795864 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2f8c003a60 con 0x7f2fa4101770 2024-04-16T03:11:39.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.559+0000 7f2fa0ff9700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f9006ecc0 0x7f2f90071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:39.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.560+0000 7f2fa0ff9700 1 -- 172.21.15.151:0/2751795864 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2f8c090340 con 0x7f2fa4101770 2024-04-16T03:11:39.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.560+0000 7f2fa2ffd700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f9006ecc0 0x7f2f90071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:39.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.560+0000 7f2fa2ffd700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f9006ecc0 0x7f2f90071180 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2f94005fd0 tx=0x7f2f94005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:39.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.564+0000 7f2fa0ff9700 1 -- 172.21.15.151:0/2751795864 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2f8c05cc20 con 0x7f2fa4101770 2024-04-16T03:11:39.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.846+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2fa404ea90 con 0x7f2fa4101770 2024-04-16T03:11:39.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.846+0000 7f2fa0ff9700 1 -- 172.21.15.151:0/2751795864 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2f8c05c7b0 con 0x7f2fa4101770 2024-04-16T03:11:39.848 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:11:39.848 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:11:39.848 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:11:39.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:11:39.848 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:11:39.848 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:11:39.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:11:39.848 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:11:39.849 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:11:39.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:11:39.849 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:11:39.849 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:11:39.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:11:39.849 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:11:39.849 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:11:39.849 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:11:39.849 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f9006ecc0 msgr2=0x7f2f90071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f9006ecc0 0x7f2f90071180 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2f94005fd0 tx=0x7f2f94005ee0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 msgr2=0x7f2fa419caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 0x7f2fa419caa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0092e0 tx=0x7f2f8c00be20 comp rx=0 tx=0).stop 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 shutdown_connections 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2f9006ecc0 0x7f2f90071180 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa4100dc0 0x7f2fa419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 --2- 172.21.15.151:0/2751795864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2fa4101770 0x7f2fa419caa0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 >> 172.21.15.151:0/2751795864 conn(0x7f2fa40fc650 msgr2=0x7f2fa40fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 shutdown_connections 2024-04-16T03:11:39.851 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.849+0000 7f2fa9337700 1 -- 172.21.15.151:0/2751795864 wait complete. 2024-04-16T03:11:39.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.954+0000 7f5e719b6700 1 -- 172.21.15.151:0/1130290717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 msgr2=0x7f5e6c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.954+0000 7f5e719b6700 1 --2- 172.21.15.151:0/1130290717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 0x7f5e6c100050 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c009910 tx=0x7f5e5c009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:39.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.954+0000 7f5e719b6700 1 -- 172.21.15.151:0/1130290717 shutdown_connections 2024-04-16T03:11:39.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.954+0000 7f5e719b6700 1 --2- 172.21.15.151:0/1130290717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 0x7f5e6c100050 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.954+0000 7f5e719b6700 1 --2- 172.21.15.151:0/1130290717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e6c105a40 0x7f5e6c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.954+0000 7f5e719b6700 1 -- 172.21.15.151:0/1130290717 >> 172.21.15.151:0/1130290717 conn(0x7f5e6c0755a0 msgr2=0x7f5e6c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:39.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.954+0000 7f5e719b6700 1 -- 172.21.15.151:0/1130290717 shutdown_connections 2024-04-16T03:11:39.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.954+0000 7f5e719b6700 1 -- 172.21.15.151:0/1130290717 wait complete. 2024-04-16T03:11:39.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.955+0000 7f5e719b6700 1 Processor -- start 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.955+0000 7f5e719b6700 1 -- start start 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.955+0000 7f5e719b6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 0x7f5e6c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.955+0000 7f5e719b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e6c105a40 0x7f5e6c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.955+0000 7f5e719b6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c1981b0 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.955+0000 7f5e719b6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c198320 con 0x7f5e6c105a40 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.955+0000 7f5e6affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 0x7f5e6c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.955+0000 7f5e6affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 0x7f5e6c197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43664/0 (socket says 172.21.15.151:43664) 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.955+0000 7f5e6affd700 1 -- 172.21.15.151:0/1224003269 learned_addr learned my addr 172.21.15.151:0/1224003269 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e6a7fc700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e6c105a40 0x7f5e6c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e6affd700 1 -- 172.21.15.151:0/1224003269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e6c105a40 msgr2=0x7f5e6c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:39.958 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e6affd700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e6c105a40 0x7f5e6c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:39.959 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:39 smithi151 ceph-mon[112233]: from='client.35800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:39.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e6affd700 1 -- 172.21.15.151:0/1224003269 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0095f0 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e6affd700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 0x7f5e6c197590 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000d8d0 tx=0x7f5e6000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:39.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e709b4700 1 -- 172.21.15.151:0/1224003269 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e600098a0 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c1a0600 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e709b4700 1 -- 172.21.15.151:0/1224003269 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e60010460 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e709b4700 1 -- 172.21.15.151:0/1224003269 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e6000f5f0 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.956+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c1a0b50 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.957+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c107fd0 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.958+0000 7f5e709b4700 1 -- 172.21.15.151:0/1224003269 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5e60009a00 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.959+0000 7f5e709b4700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5e5406ecc0 0x7f5e54071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:39.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.959+0000 7f5e709b4700 1 -- 172.21.15.151:0/1224003269 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5e6008ec30 con 0x7f5e6c0ffbd0 2024-04-16T03:11:39.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.959+0000 7f5e6a7fc700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5e5406ecc0 0x7f5e54071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:39.961 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.959+0000 7f5e6a7fc700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5e5406ecc0 0x7f5e54071180 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c068cc0 tx=0x7f5e5c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:39.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:39.964+0000 7f5e709b4700 1 -- 172.21.15.151:0/1224003269 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5e60057a60 con 0x7f5e6c0ffbd0 2024-04-16T03:11:40.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.206+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5e6c02ccc0 con 0x7f5e6c0ffbd0 2024-04-16T03:11:40.208 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.207+0000 7f5e709b4700 1 -- 172.21.15.151:0/1224003269 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f5e60016770 con 0x7f5e6c0ffbd0 2024-04-16T03:11:40.208 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:11:40.208 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:11:40.209 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:11:40.210 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:11:40.211 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5e5406ecc0 msgr2=0x7f5e54071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5e5406ecc0 0x7f5e54071180 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c068cc0 tx=0x7f5e5c01a040 comp rx=0 tx=0).stop 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 msgr2=0x7f5e6c197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 0x7f5e6c197590 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000d8d0 tx=0x7f5e6000dbe0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 shutdown_connections 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f5e5406ecc0 0x7f5e54071180 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e6c0ffbd0 0x7f5e6c197590 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 --2- 172.21.15.151:0/1224003269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e6c105a40 0x7f5e6c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 >> 172.21.15.151:0/1224003269 conn(0x7f5e6c0755a0 msgr2=0x7f5e6c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 shutdown_connections 2024-04-16T03:11:40.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.209+0000 7f5e719b6700 1 -- 172.21.15.151:0/1224003269 wait complete. 2024-04-16T03:11:40.213 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:11:40.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:39 smithi153 ceph-mon[87858]: from='client.35800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:40.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.318+0000 7fb75c152700 1 -- 172.21.15.151:0/1962588545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7540ffbd0 msgr2=0x7fb754100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.318+0000 7fb75c152700 1 --2- 172.21.15.151:0/1962588545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7540ffbd0 0x7fb754100050 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb750009910 tx=0x7fb750009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.318+0000 7fb75c152700 1 -- 172.21.15.151:0/1962588545 shutdown_connections 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.318+0000 7fb75c152700 1 --2- 172.21.15.151:0/1962588545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7540ffbd0 0x7fb754100050 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.318+0000 7fb75c152700 1 --2- 172.21.15.151:0/1962588545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb754105a40 0x7fb754105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.318+0000 7fb75c152700 1 -- 172.21.15.151:0/1962588545 >> 172.21.15.151:0/1962588545 conn(0x7fb7540755a0 msgr2=0x7fb7540759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.318+0000 7fb75c152700 1 -- 172.21.15.151:0/1962588545 shutdown_connections 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.318+0000 7fb75c152700 1 -- 172.21.15.151:0/1962588545 wait complete. 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.319+0000 7fb75c152700 1 Processor -- start 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.319+0000 7fb75c152700 1 -- start start 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.319+0000 7fb75c152700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7540ffbd0 0x7fb754195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.319+0000 7fb75c152700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb754105a40 0x7fb754195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.319+0000 7fb75c152700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb754196240 con 0x7fb754105a40 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.319+0000 7fb75c152700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb754199f60 con 0x7fb7540ffbd0 2024-04-16T03:11:40.493 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.319+0000 7fb7596ed700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb754105a40 0x7fb754195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.319+0000 7fb7596ed700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb754105a40 0x7fb754195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43688/0 (socket says 172.21.15.151:43688) 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.319+0000 7fb7596ed700 1 -- 172.21.15.151:0/649682437 learned_addr learned my addr 172.21.15.151:0/649682437 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb759eee700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7540ffbd0 0x7fb754195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb7596ed700 1 -- 172.21.15.151:0/649682437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7540ffbd0 msgr2=0x7fb754195620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb7596ed700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7540ffbd0 0x7fb754195620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb7596ed700 1 -- 172.21.15.151:0/649682437 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7500095f0 con 0x7fb754105a40 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb759eee700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7540ffbd0 0x7fb754195620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb7596ed700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb754105a40 0x7fb754195b60 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb750006010 tx=0x7fb75000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb74affd700 1 -- 172.21.15.151:0/649682437 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb75001d070 con 0x7fb754105a40 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb74affd700 1 -- 172.21.15.151:0/649682437 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb75000f460 con 0x7fb754105a40 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75419a1e0 con 0x7fb754105a40 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb74affd700 1 -- 172.21.15.151:0/649682437 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb750017630 con 0x7fb754105a40 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.320+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75419a6d0 con 0x7fb754105a40 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.321+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb754107fd0 con 0x7fb754105a40 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.322+0000 7fb74affd700 1 -- 172.21.15.151:0/649682437 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb75000f5d0 con 0x7fb754105a40 2024-04-16T03:11:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.322+0000 7fb74affd700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb74006ecc0 0x7fb740071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:40.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.322+0000 7fb74affd700 1 -- 172.21.15.151:0/649682437 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb750090300 con 0x7fb754105a40 2024-04-16T03:11:40.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.322+0000 7fb759eee700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb74006ecc0 0x7fb740071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:40.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.323+0000 7fb759eee700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb74006ecc0 0x7fb740071180 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb744005950 tx=0x7fb7440058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:40.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.327+0000 7fb74affd700 1 -- 172.21.15.151:0/649682437 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb7500590b0 con 0x7fb754105a40 2024-04-16T03:11:40.546 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.545+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb75402cce0 con 0x7fb74006ecc0 2024-04-16T03:11:40.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.546+0000 7fb74affd700 1 -- 172.21.15.151:0/649682437 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb75402cce0 con 0x7fb74006ecc0 2024-04-16T03:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:11:40.548 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb74006ecc0 msgr2=0x7fb740071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb74006ecc0 0x7fb740071180 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb744005950 tx=0x7fb7440058e0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb754105a40 msgr2=0x7fb754195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb754105a40 0x7fb754195b60 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb750006010 tx=0x7fb75000be20 comp rx=0 tx=0).stop 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 shutdown_connections 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb74006ecc0 0x7fb740071180 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7540ffbd0 0x7fb754195620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 --2- 172.21.15.151:0/649682437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb754105a40 0x7fb754195b60 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 >> 172.21.15.151:0/649682437 conn(0x7fb7540755a0 msgr2=0x7fb7540fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.548+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 shutdown_connections 2024-04-16T03:11:40.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.549+0000 7fb75c152700 1 -- 172.21.15.151:0/649682437 wait complete. 2024-04-16T03:11:40.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.653+0000 7f1583938700 1 -- 172.21.15.151:0/3034910787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c101770 msgr2=0x7f157c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.653+0000 7f1583938700 1 --2- 172.21.15.151:0/3034910787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c101770 0x7f157c1057c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f156c009910 tx=0x7f156c009c20 comp rx=0 tx=0).stop 2024-04-16T03:11:40.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 -- 172.21.15.151:0/3034910787 shutdown_connections 2024-04-16T03:11:40.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 --2- 172.21.15.151:0/3034910787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c101770 0x7f157c1057c0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 --2- 172.21.15.151:0/3034910787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157c100dc0 0x7f157c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 -- 172.21.15.151:0/3034910787 >> 172.21.15.151:0/3034910787 conn(0x7f157c0fc650 msgr2=0x7f157c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:40.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 -- 172.21.15.151:0/3034910787 shutdown_connections 2024-04-16T03:11:40.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 -- 172.21.15.151:0/3034910787 wait complete. 2024-04-16T03:11:40.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 Processor -- start 2024-04-16T03:11:40.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 -- start start 2024-04-16T03:11:40.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c100dc0 0x7f157c19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:40.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157c101770 0x7f157c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:40.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c19d130 con 0x7f157c100dc0 2024-04-16T03:11:40.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.654+0000 7f1583938700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c1965e0 con 0x7f157c101770 2024-04-16T03:11:40.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15816d4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c100dc0 0x7f157c19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15816d4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c100dc0 0x7f157c19c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43700/0 (socket says 172.21.15.151:43700) 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15816d4700 1 -- 172.21.15.151:0/2194504041 learned_addr learned my addr 172.21.15.151:0/2194504041 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f1580ed3700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157c101770 0x7f157c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15816d4700 1 -- 172.21.15.151:0/2194504041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157c101770 msgr2=0x7f157c19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15816d4700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157c101770 0x7f157c19caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15816d4700 1 -- 172.21.15.151:0/2194504041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c0095f0 con 0x7f157c100dc0 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f1580ed3700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157c101770 0x7f157c19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15816d4700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c100dc0 0x7f157c19c560 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f157800d8d0 tx=0x7f157800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15727fc700 1 -- 172.21.15.151:0/2194504041 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1578009940 con 0x7f157c100dc0 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15727fc700 1 -- 172.21.15.151:0/2194504041 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1578010460 con 0x7f157c100dc0 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157c1968c0 con 0x7f157c100dc0 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f15727fc700 1 -- 172.21.15.151:0/2194504041 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f157800f5f0 con 0x7f157c100dc0 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.655+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157c196e10 con 0x7f157c100dc0 2024-04-16T03:11:40.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.656+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157c1091c0 con 0x7f157c100dc0 2024-04-16T03:11:40.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.657+0000 7f15727fc700 1 -- 172.21.15.151:0/2194504041 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f15780105d0 con 0x7f157c100dc0 2024-04-16T03:11:40.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.657+0000 7f15727fc700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f156806ecc0 0x7f1568071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:11:40.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.657+0000 7f15727fc700 1 -- 172.21.15.151:0/2194504041 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f157808ee70 con 0x7f157c100dc0 2024-04-16T03:11:40.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.657+0000 7f1580ed3700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f156806ecc0 0x7f1568071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:11:40.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.658+0000 7f1580ed3700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f156806ecc0 0x7f1568071180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f156c006010 tx=0x7f156c005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:11:40.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.662+0000 7f15727fc700 1 -- 172.21.15.151:0/2194504041 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f157805b750 con 0x7f157c100dc0 2024-04-16T03:11:40.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.946+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f157c1970f0 con 0x7f157c100dc0 2024-04-16T03:11:40.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.946+0000 7f15727fc700 1 -- 172.21.15.151:0/2194504041 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f157805b2e0 con 0x7f157c100dc0 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:11:40.948 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:11:40.949 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f156806ecc0 msgr2=0x7f1568071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f156806ecc0 0x7f1568071180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f156c006010 tx=0x7f156c005960 comp rx=0 tx=0).stop 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c100dc0 msgr2=0x7f157c19c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c100dc0 0x7f157c19c560 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f157800d8d0 tx=0x7f157800dc90 comp rx=0 tx=0).stop 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 shutdown_connections 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f156806ecc0 0x7f1568071180 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f157c100dc0 0x7f157c19c560 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 --2- 172.21.15.151:0/2194504041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157c101770 0x7f157c19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 >> 172.21.15.151:0/2194504041 conn(0x7f157c0fc650 msgr2=0x7f157c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 shutdown_connections 2024-04-16T03:11:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:11:40.948+0000 7f1583938700 1 -- 172.21.15.151:0/2194504041 wait complete. 2024-04-16T03:11:41.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:40 smithi151 ceph-mon[112233]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:41.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:40 smithi151 ceph-mon[112233]: pgmap v992: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:41.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:40 smithi151 ceph-mon[112233]: from='client.35808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:41.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:40 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2751795864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:11:41.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:40 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1224003269' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:11:41.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:40 smithi153 ceph-mon[87858]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:40 smithi153 ceph-mon[87858]: pgmap v992: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:40 smithi153 ceph-mon[87858]: from='client.35808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:40 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2751795864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:11:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:40 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1224003269' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:11:42.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:41 smithi153 ceph-mon[87858]: from='client.35820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:42.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:41 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2194504041' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:11:42.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:41 smithi151 ceph-mon[112233]: from='client.35820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:11:42.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:41 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2194504041' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:11:43.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:42 smithi151 ceph-mon[112233]: pgmap v993: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:43.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:42 smithi153 ceph-mon[87858]: pgmap v993: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:45.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:44 smithi151 ceph-mon[112233]: pgmap v994: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:45.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:44 smithi153 ceph-mon[87858]: pgmap v994: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:47.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:46 smithi153 ceph-mon[87858]: pgmap v995: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:47.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:46 smithi151 ceph-mon[112233]: pgmap v995: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:48 smithi151 ceph-mon[112233]: pgmap v996: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:49.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:48 smithi153 ceph-mon[87858]: pgmap v996: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:51.143 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:50 smithi151 ceph-mon[112233]: pgmap v997: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:51.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:50 smithi153 ceph-mon[87858]: pgmap v997: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:53.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:52 smithi151 ceph-mon[112233]: pgmap v998: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:53.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:52 smithi153 ceph-mon[87858]: pgmap v998: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:54.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:11:54.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:11:55.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:54 smithi153 ceph-mon[87858]: pgmap v999: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:55.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:54 smithi151 ceph-mon[112233]: pgmap v999: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:57.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:56 smithi151 ceph-mon[112233]: pgmap v1000: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:57.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:56 smithi153 ceph-mon[87858]: pgmap v1000: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:11:58 smithi151 ceph-mon[112233]: pgmap v1001: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:11:59.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:11:58 smithi153 ceph-mon[87858]: pgmap v1001: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:01.135 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:00 smithi151 ceph-mon[112233]: pgmap v1002: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:01.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:00 smithi153 ceph-mon[87858]: pgmap v1002: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:03.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:02 smithi151 ceph-mon[112233]: pgmap v1003: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:03.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:02 smithi153 ceph-mon[87858]: pgmap v1003: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:05.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:04 smithi151 ceph-mon[112233]: pgmap v1004: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:05.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:04 smithi153 ceph-mon[87858]: pgmap v1004: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:07.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:06 smithi151 ceph-mon[112233]: pgmap v1005: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:07.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:06 smithi153 ceph-mon[87858]: pgmap v1005: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:09.172 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:08 smithi151 ceph-mon[112233]: pgmap v1006: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:09.172 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:12:09.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:08 smithi153 ceph-mon[87858]: pgmap v1006: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:12:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.067+0000 7f99c5ae2700 1 -- 172.21.15.151:0/1363773749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c00ffa40 msgr2=0x7f99c00ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.067+0000 7f99c5ae2700 1 --2- 172.21.15.151:0/1363773749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c00ffa40 0x7f99c00ffec0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009910 tx=0x7f99b4009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.067+0000 7f99c5ae2700 1 -- 172.21.15.151:0/1363773749 shutdown_connections 2024-04-16T03:12:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.067+0000 7f99c5ae2700 1 --2- 172.21.15.151:0/1363773749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c00ffa40 0x7f99c00ffec0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.067+0000 7f99c5ae2700 1 --2- 172.21.15.151:0/1363773749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99c0105a40 0x7f99c0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.067+0000 7f99c5ae2700 1 -- 172.21.15.151:0/1363773749 >> 172.21.15.151:0/1363773749 conn(0x7f99c00fb550 msgr2=0x7f99c00fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.067+0000 7f99c5ae2700 1 -- 172.21.15.151:0/1363773749 shutdown_connections 2024-04-16T03:12:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.067+0000 7f99c5ae2700 1 -- 172.21.15.151:0/1363773749 wait complete. 2024-04-16T03:12:11.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99c5ae2700 1 Processor -- start 2024-04-16T03:12:11.071 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99c5ae2700 1 -- start start 2024-04-16T03:12:11.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99c5ae2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99c00ffa40 0x7f99c0197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:11.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99c5ae2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c0105a40 0x7f99c0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:11.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99c5ae2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c01980f0 con 0x7f99c0105a40 2024-04-16T03:12:11.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99c5ae2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c0198260 con 0x7f99c00ffa40 2024-04-16T03:12:11.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99beffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c0105a40 0x7f99c0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99beffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c0105a40 0x7f99c0197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38682/0 (socket says 172.21.15.151:38682) 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99beffd700 1 -- 172.21.15.151:0/2702969065 learned_addr learned my addr 172.21.15.151:0/2702969065 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.068+0000 7f99bf7fe700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99c00ffa40 0x7f99c0197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.069+0000 7f99beffd700 1 -- 172.21.15.151:0/2702969065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99c00ffa40 msgr2=0x7f99c0197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.069+0000 7f99beffd700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99c00ffa40 0x7f99c0197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.069+0000 7f99beffd700 1 -- 172.21.15.151:0/2702969065 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b40095f0 con 0x7f99c0105a40 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.069+0000 7f99beffd700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c0105a40 0x7f99c0197ad0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f99b40092e0 tx=0x7f99b400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.069+0000 7f99bcff9700 1 -- 172.21.15.151:0/2702969065 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b401d070 con 0x7f99c0105a40 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.069+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c01a0600 con 0x7f99c0105a40 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.069+0000 7f99bcff9700 1 -- 172.21.15.151:0/2702969065 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99b4005e00 con 0x7f99c0105a40 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.069+0000 7f99bcff9700 1 -- 172.21.15.151:0/2702969065 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b4017400 con 0x7f99c0105a40 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.069+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c01a0a70 con 0x7f99c0105a40 2024-04-16T03:12:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.070+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c0107fd0 con 0x7f99c0105a40 2024-04-16T03:12:11.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.071+0000 7f99bcff9700 1 -- 172.21.15.151:0/2702969065 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f99b400f6f0 con 0x7f99c0105a40 2024-04-16T03:12:11.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.071+0000 7f99bcff9700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f99a806ecc0 0x7f99a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:11.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.071+0000 7f99bcff9700 1 -- 172.21.15.151:0/2702969065 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f99b4090310 con 0x7f99c0105a40 2024-04-16T03:12:11.074 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.071+0000 7f99bf7fe700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f99a806ecc0 0x7f99a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:11.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.071+0000 7f99bf7fe700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f99a806ecc0 0x7f99a8071180 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009550 tx=0x7f99b00091b0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:11.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.076+0000 7f99bcff9700 1 -- 172.21.15.151:0/2702969065 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f99b40590c0 con 0x7f99c0105a40 2024-04-16T03:12:11.143 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:10 smithi151 ceph-mon[112233]: pgmap v1007: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:11.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:10 smithi153 ceph-mon[87858]: pgmap v1007: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:11.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.294+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99c002cd30 con 0x7f99a806ecc0 2024-04-16T03:12:11.298 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.296+0000 7f99bcff9700 1 -- 172.21.15.151:0/2702969065 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f99c002cd30 con 0x7f99a806ecc0 2024-04-16T03:12:11.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f99a806ecc0 msgr2=0x7f99a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f99a806ecc0 0x7f99a8071180 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009550 tx=0x7f99b00091b0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c0105a40 msgr2=0x7f99c0197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c0105a40 0x7f99c0197ad0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f99b40092e0 tx=0x7f99b400bd70 comp rx=0 tx=0).stop 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 shutdown_connections 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f99a806ecc0 0x7f99a8071180 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99c00ffa40 0x7f99c0197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 --2- 172.21.15.151:0/2702969065 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99c0105a40 0x7f99c0197ad0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 >> 172.21.15.151:0/2702969065 conn(0x7f99c00fb550 msgr2=0x7f99c00fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 shutdown_connections 2024-04-16T03:12:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.298+0000 7f99c5ae2700 1 -- 172.21.15.151:0/2702969065 wait complete. 2024-04-16T03:12:11.311 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:12:11.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.404+0000 7f23a8b40700 1 -- 172.21.15.151:0/4158564796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a40ffa40 msgr2=0x7f23a40ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.404+0000 7f23a8b40700 1 --2- 172.21.15.151:0/4158564796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a40ffa40 0x7f23a40ffec0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f2398009910 tx=0x7f2398009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:11.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.404+0000 7f23a8b40700 1 -- 172.21.15.151:0/4158564796 shutdown_connections 2024-04-16T03:12:11.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.404+0000 7f23a8b40700 1 --2- 172.21.15.151:0/4158564796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a40ffa40 0x7f23a40ffec0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.404+0000 7f23a8b40700 1 --2- 172.21.15.151:0/4158564796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a4105a40 0x7f23a4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.404+0000 7f23a8b40700 1 -- 172.21.15.151:0/4158564796 >> 172.21.15.151:0/4158564796 conn(0x7f23a40fb570 msgr2=0x7f23a40fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:11.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.404+0000 7f23a8b40700 1 -- 172.21.15.151:0/4158564796 shutdown_connections 2024-04-16T03:12:11.407 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.404+0000 7f23a8b40700 1 -- 172.21.15.151:0/4158564796 wait complete. 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a8b40700 1 Processor -- start 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a8b40700 1 -- start start 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a8b40700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40ffa40 0x7f23a419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a8b40700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a4105a40 0x7f23a419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a8b40700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a419f240 con 0x7f23a4105a40 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a8b40700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a4198780 con 0x7f23a40ffa40 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a4105a40 0x7f23a419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a1d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a4105a40 0x7f23a419ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38694/0 (socket says 172.21.15.151:38694) 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a1d9b700 1 -- 172.21.15.151:0/561063122 learned_addr learned my addr 172.21.15.151:0/561063122 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.405+0000 7f23a259c700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40ffa40 0x7f23a419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.406+0000 7f23a1d9b700 1 -- 172.21.15.151:0/561063122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40ffa40 msgr2=0x7f23a419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.406+0000 7f23a1d9b700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40ffa40 0x7f23a419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.406+0000 7f23a1d9b700 1 -- 172.21.15.151:0/561063122 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980095f0 con 0x7f23a4105a40 2024-04-16T03:12:11.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.406+0000 7f23a1d9b700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a4105a40 0x7f23a419ec40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f2398006010 tx=0x7f239800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:11.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.406+0000 7f23937fe700 1 -- 172.21.15.151:0/561063122 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f239801d070 con 0x7f23a4105a40 2024-04-16T03:12:11.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.406+0000 7f23937fe700 1 -- 172.21.15.151:0/561063122 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f239800f460 con 0x7f23a4105a40 2024-04-16T03:12:11.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.406+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a4198a00 con 0x7f23a4105a40 2024-04-16T03:12:11.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.406+0000 7f23937fe700 1 -- 172.21.15.151:0/561063122 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2398017630 con 0x7f23a4105a40 2024-04-16T03:12:11.410 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.406+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a4198ef0 con 0x7f23a4105a40 2024-04-16T03:12:11.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.407+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a404ea90 con 0x7f23a4105a40 2024-04-16T03:12:11.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.408+0000 7f23937fe700 1 -- 172.21.15.151:0/561063122 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2398003a60 con 0x7f23a4105a40 2024-04-16T03:12:11.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.408+0000 7f23937fe700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f238c06ecc0 0x7f238c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:11.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.408+0000 7f23937fe700 1 -- 172.21.15.151:0/561063122 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2398090440 con 0x7f23a4105a40 2024-04-16T03:12:11.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.408+0000 7f23a259c700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f238c06ecc0 0x7f238c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:11.411 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.409+0000 7f23a259c700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f238c06ecc0 0x7f238c071180 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f239400ac50 tx=0x7f239400a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:11.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.413+0000 7f23937fe700 1 -- 172.21.15.151:0/561063122 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f239805ccc0 con 0x7f23a4105a40 2024-04-16T03:12:11.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.631+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f23a4199240 con 0x7f238c06ecc0 2024-04-16T03:12:11.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.632+0000 7f23937fe700 1 -- 172.21.15.151:0/561063122 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f23a4199240 con 0x7f238c06ecc0 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f238c06ecc0 msgr2=0x7f238c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f238c06ecc0 0x7f238c071180 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f239400ac50 tx=0x7f239400a380 comp rx=0 tx=0).stop 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a4105a40 msgr2=0x7f23a419ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a4105a40 0x7f23a419ec40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f2398006010 tx=0x7f239800bd70 comp rx=0 tx=0).stop 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 shutdown_connections 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f238c06ecc0 0x7f238c071180 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40ffa40 0x7f23a419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 --2- 172.21.15.151:0/561063122 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f23a4105a40 0x7f23a419ec40 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 >> 172.21.15.151:0/561063122 conn(0x7f23a40fb570 msgr2=0x7f23a4068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 shutdown_connections 2024-04-16T03:12:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.635+0000 7f23a8b40700 1 -- 172.21.15.151:0/561063122 wait complete. 2024-04-16T03:12:11.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.739+0000 7f4cc5725700 1 -- 172.21.15.151:0/2248754155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc00ffbd0 msgr2=0x7f4cc0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.739+0000 7f4cc5725700 1 --2- 172.21.15.151:0/2248754155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc00ffbd0 0x7f4cc0100050 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009910 tx=0x7f4cb0009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:11.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.739+0000 7f4cc5725700 1 -- 172.21.15.151:0/2248754155 shutdown_connections 2024-04-16T03:12:11.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.739+0000 7f4cc5725700 1 --2- 172.21.15.151:0/2248754155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc00ffbd0 0x7f4cc0100050 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.739+0000 7f4cc5725700 1 --2- 172.21.15.151:0/2248754155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0105a40 0x7f4cc0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.739+0000 7f4cc5725700 1 -- 172.21.15.151:0/2248754155 >> 172.21.15.151:0/2248754155 conn(0x7f4cc00755a0 msgr2=0x7f4cc00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:11.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.739+0000 7f4cc5725700 1 -- 172.21.15.151:0/2248754155 shutdown_connections 2024-04-16T03:12:11.742 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.740+0000 7f4cc5725700 1 -- 172.21.15.151:0/2248754155 wait complete. 2024-04-16T03:12:11.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.740+0000 7f4cc5725700 1 Processor -- start 2024-04-16T03:12:11.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.740+0000 7f4cc5725700 1 -- start start 2024-04-16T03:12:11.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cc5725700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc00ffbd0 0x7f4cc0197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:11.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cc5725700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc0105a40 0x7f4cc0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:11.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cc5725700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc01981b0 con 0x7f4cc0105a40 2024-04-16T03:12:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cc5725700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc0198320 con 0x7f4cc00ffbd0 2024-04-16T03:12:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cbe7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc0105a40 0x7f4cc0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cbe7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc0105a40 0x7f4cc0197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38712/0 (socket says 172.21.15.151:38712) 2024-04-16T03:12:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2823906749 learned_addr learned my addr 172.21.15.151:0/2823906749 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cbeffd700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc00ffbd0 0x7f4cc0197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2823906749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc00ffbd0 msgr2=0x7f4cc0197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cbe7fc700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc00ffbd0 0x7f4cc0197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2823906749 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb00095f0 con 0x7f4cc0105a40 2024-04-16T03:12:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4cbe7fc700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc0105a40 0x7f4cc0197ad0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0006010 tx=0x7f4cb000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:11.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.741+0000 7f4ca7fff700 1 -- 172.21.15.151:0/2823906749 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cb001d070 con 0x7f4cc0105a40 2024-04-16T03:12:11.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.742+0000 7f4ca7fff700 1 -- 172.21.15.151:0/2823906749 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4cb0005dc0 con 0x7f4cc0105a40 2024-04-16T03:12:11.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.742+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc01a0600 con 0x7f4cc0105a40 2024-04-16T03:12:11.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.742+0000 7f4ca7fff700 1 -- 172.21.15.151:0/2823906749 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cb0017400 con 0x7f4cc0105a40 2024-04-16T03:12:11.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.742+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc01a0a20 con 0x7f4cc0105a40 2024-04-16T03:12:11.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.743+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc0107fd0 con 0x7f4cc0105a40 2024-04-16T03:12:11.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.743+0000 7f4ca7fff700 1 -- 172.21.15.151:0/2823906749 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4cb0003a60 con 0x7f4cc0105a40 2024-04-16T03:12:11.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.744+0000 7f4ca7fff700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ca806ecc0 0x7f4ca8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:11.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.744+0000 7f4ca7fff700 1 -- 172.21.15.151:0/2823906749 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4cb00902f0 con 0x7f4cc0105a40 2024-04-16T03:12:11.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.744+0000 7f4cbeffd700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ca806ecc0 0x7f4ca8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:11.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.744+0000 7f4cbeffd700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ca806ecc0 0x7f4ca8071180 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4005950 tx=0x7f4cb400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:11.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.748+0000 7f4ca7fff700 1 -- 172.21.15.151:0/2823906749 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4cb00590a0 con 0x7f4cc0105a40 2024-04-16T03:12:11.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.958+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cc002cd30 con 0x7f4ca806ecc0 2024-04-16T03:12:11.970 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.968+0000 7f4ca7fff700 1 -- 172.21.15.151:0/2823906749 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4cc002cd30 con 0x7f4ca806ecc0 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (33m) 11m ago 45m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (45m) 11m ago 45m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (44m) 11m ago 44m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (31m) 11m ago 45m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (31m) 11m ago 44m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (33m) 11m ago 44m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (27m) 11m ago 39m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (27m) 11m ago 39m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (11m) 11m ago 39m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (26m) 11m ago 39m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (35m) 11m ago 46m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:12:11.971 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (34m) 11m ago 44m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (32m) 11m ago 46m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (32m) 11m ago 43m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (34m) 11m ago 45m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (34m) 11m ago 44m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (30m) 11m ago 43m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (30m) 11m ago 42m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (29m) 11m ago 42m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (29m) 11m ago 41m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (28m) 11m ago 41m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (27m) 11m ago 41m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:12:11.972 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (34m) 11m ago 44m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:12:11.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ca806ecc0 msgr2=0x7f4ca8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ca806ecc0 0x7f4ca8071180 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4005950 tx=0x7f4cb400b410 comp rx=0 tx=0).stop 2024-04-16T03:12:11.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc0105a40 msgr2=0x7f4cc0197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:11.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc0105a40 0x7f4cc0197ad0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0006010 tx=0x7f4cb000bd70 comp rx=0 tx=0).stop 2024-04-16T03:12:11.973 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 shutdown_connections 2024-04-16T03:12:11.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ca806ecc0 0x7f4ca8071180 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc00ffbd0 0x7f4cc0197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 --2- 172.21.15.151:0/2823906749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc0105a40 0x7f4cc0197ad0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:11.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 >> 172.21.15.151:0/2823906749 conn(0x7f4cc00755a0 msgr2=0x7f4cc00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:11.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 shutdown_connections 2024-04-16T03:12:11.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:11.971+0000 7f4cc5725700 1 -- 172.21.15.151:0/2823906749 wait complete. 2024-04-16T03:12:12.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.077+0000 7f839d63d700 1 -- 172.21.15.151:0/2854505285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8398105a40 msgr2=0x7f8398105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.077+0000 7f839d63d700 1 --2- 172.21.15.151:0/2854505285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8398105a40 0x7f8398105e20 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f838800b0a0 tx=0x7f838800b3b0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.077+0000 7f839d63d700 1 -- 172.21.15.151:0/2854505285 shutdown_connections 2024-04-16T03:12:12.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.077+0000 7f839d63d700 1 --2- 172.21.15.151:0/2854505285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83980ffa40 0x7f83980ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.077+0000 7f839d63d700 1 --2- 172.21.15.151:0/2854505285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8398105a40 0x7f8398105e20 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.077+0000 7f839d63d700 1 -- 172.21.15.151:0/2854505285 >> 172.21.15.151:0/2854505285 conn(0x7f83980fb550 msgr2=0x7f83980fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:12.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.077+0000 7f839d63d700 1 -- 172.21.15.151:0/2854505285 shutdown_connections 2024-04-16T03:12:12.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.077+0000 7f839d63d700 1 -- 172.21.15.151:0/2854505285 wait complete. 2024-04-16T03:12:12.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.078+0000 7f839d63d700 1 Processor -- start 2024-04-16T03:12:12.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.078+0000 7f839d63d700 1 -- start start 2024-04-16T03:12:12.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.078+0000 7f839d63d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83980ffa40 0x7f839819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:12.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.078+0000 7f839d63d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8398105a40 0x7f839819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:12.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.078+0000 7f839d63d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83981965d0 con 0x7f83980ffa40 2024-04-16T03:12:12.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.078+0000 7f839d63d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8398196740 con 0x7f8398105a40 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.078+0000 7f8396ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83980ffa40 0x7f839819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.078+0000 7f8396ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83980ffa40 0x7f839819c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38734/0 (socket says 172.21.15.151:38734) 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.078+0000 7f8396ffd700 1 -- 172.21.15.151:0/2771570092 learned_addr learned my addr 172.21.15.151:0/2771570092 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f83967fc700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8398105a40 0x7f839819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f8396ffd700 1 -- 172.21.15.151:0/2771570092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8398105a40 msgr2=0x7f839819cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f8396ffd700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8398105a40 0x7f839819cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f8396ffd700 1 -- 172.21.15.151:0/2771570092 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8388009d00 con 0x7f83980ffa40 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f83967fc700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8398105a40 0x7f839819cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f8396ffd700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83980ffa40 0x7f839819c5e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f838800b070 tx=0x7f838800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f838ffff700 1 -- 172.21.15.151:0/2771570092 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f838800e040 con 0x7f83980ffa40 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f838ffff700 1 -- 172.21.15.151:0/2771570092 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8388003de0 con 0x7f83980ffa40 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83981969c0 con 0x7f83980ffa40 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f838ffff700 1 -- 172.21.15.151:0/2771570092 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f838801d8e0 con 0x7f83980ffa40 2024-04-16T03:12:12.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.079+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8398196eb0 con 0x7f83980ffa40 2024-04-16T03:12:12.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.080+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8398109050 con 0x7f83980ffa40 2024-04-16T03:12:12.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.081+0000 7f838ffff700 1 -- 172.21.15.151:0/2771570092 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8388004040 con 0x7f83980ffa40 2024-04-16T03:12:12.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.081+0000 7f838ffff700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f838406ecc0 0x7f8384071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:12.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.081+0000 7f838ffff700 1 -- 172.21.15.151:0/2771570092 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f83880906b0 con 0x7f83980ffa40 2024-04-16T03:12:12.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.081+0000 7f83967fc700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f838406ecc0 0x7f8384071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:12.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.082+0000 7f83967fc700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f838406ecc0 0x7f8384071180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8380005950 tx=0x7f83800058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:12.088 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.086+0000 7f838ffff700 1 -- 172.21.15.151:0/2771570092 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8388059460 con 0x7f83980ffa40 2024-04-16T03:12:12.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.371+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f83981970d0 con 0x7f83980ffa40 2024-04-16T03:12:12.373 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.371+0000 7f838ffff700 1 -- 172.21.15.151:0/2771570092 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f838805cb20 con 0x7f83980ffa40 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:12:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:12:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:12:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:12:12.375 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:12:12.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f838406ecc0 msgr2=0x7f8384071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f838406ecc0 0x7f8384071180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8380005950 tx=0x7f83800058e0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83980ffa40 msgr2=0x7f839819c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83980ffa40 0x7f839819c5e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f838800b070 tx=0x7f838800cdd0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 shutdown_connections 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f838406ecc0 0x7f8384071180 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83980ffa40 0x7f839819c5e0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 --2- 172.21.15.151:0/2771570092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8398105a40 0x7f839819cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 >> 172.21.15.151:0/2771570092 conn(0x7f83980fb550 msgr2=0x7f8398069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 shutdown_connections 2024-04-16T03:12:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.374+0000 7f839d63d700 1 -- 172.21.15.151:0/2771570092 wait complete. 2024-04-16T03:12:12.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.480+0000 7f89a1824700 1 -- 172.21.15.151:0/2139011082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c100da0 msgr2=0x7f899c101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.480+0000 7f89a1824700 1 --2- 172.21.15.151:0/2139011082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c100da0 0x7f899c101180 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f898c009910 tx=0x7f898c009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:12.482 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.480+0000 7f89a1824700 1 -- 172.21.15.151:0/2139011082 shutdown_connections 2024-04-16T03:12:12.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.480+0000 7f89a1824700 1 --2- 172.21.15.151:0/2139011082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f899c101750 0x7f899c1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.480+0000 7f89a1824700 1 --2- 172.21.15.151:0/2139011082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c100da0 0x7f899c101180 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.480+0000 7f89a1824700 1 -- 172.21.15.151:0/2139011082 >> 172.21.15.151:0/2139011082 conn(0x7f899c0fc630 msgr2=0x7f899c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:12.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.480+0000 7f89a1824700 1 -- 172.21.15.151:0/2139011082 shutdown_connections 2024-04-16T03:12:12.483 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.480+0000 7f89a1824700 1 -- 172.21.15.151:0/2139011082 wait complete. 2024-04-16T03:12:12.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f89a1824700 1 Processor -- start 2024-04-16T03:12:12.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f89a1824700 1 -- start start 2024-04-16T03:12:12.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f89a1824700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f899c100da0 0x7f899c19c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:12.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f89a1824700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c101750 0x7f899c19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f89a1824700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c19d170 con 0x7f899c101750 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f89a1824700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c196620 con 0x7f899c100da0 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f89927fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c101750 0x7f899c19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f89927fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c101750 0x7f899c19cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38748/0 (socket says 172.21.15.151:38748) 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f89927fc700 1 -- 172.21.15.151:0/2608939513 learned_addr learned my addr 172.21.15.151:0/2608939513 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.481+0000 7f899affd700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f899c100da0 0x7f899c19c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.482+0000 7f89927fc700 1 -- 172.21.15.151:0/2608939513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f899c100da0 msgr2=0x7f899c19c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.482+0000 7f89927fc700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f899c100da0 0x7f899c19c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.482+0000 7f89927fc700 1 -- 172.21.15.151:0/2608939513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c0095f0 con 0x7f899c101750 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.482+0000 7f89927fc700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c101750 0x7f899c19cae0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f898400e8f0 tx=0x7f898400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.482+0000 7f89a0822700 1 -- 172.21.15.151:0/2608939513 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8984009790 con 0x7f899c101750 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.482+0000 7f89a0822700 1 -- 172.21.15.151:0/2608939513 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f898400c810 con 0x7f899c101750 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.482+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899c196900 con 0x7f899c101750 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.482+0000 7f89a0822700 1 -- 172.21.15.151:0/2608939513 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8984005580 con 0x7f899c101750 2024-04-16T03:12:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.482+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899c196e50 con 0x7f899c101750 2024-04-16T03:12:12.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.483+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899c109140 con 0x7f899c101750 2024-04-16T03:12:12.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.483+0000 7f89a0822700 1 -- 172.21.15.151:0/2608939513 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f898400c980 con 0x7f899c101750 2024-04-16T03:12:12.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.484+0000 7f89a0822700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f898806ecc0 0x7f8988071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:12.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.484+0000 7f89a0822700 1 -- 172.21.15.151:0/2608939513 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8984014070 con 0x7f899c101750 2024-04-16T03:12:12.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.484+0000 7f899affd700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f898806ecc0 0x7f8988071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:12.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.485+0000 7f899affd700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f898806ecc0 0x7f8988071180 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f898c0059d0 tx=0x7f898c005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:12.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.489+0000 7f89a0822700 1 -- 172.21.15.151:0/2608939513 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f898405ba00 con 0x7f899c101750 2024-04-16T03:12:12.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.734+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f899c1970d0 con 0x7f899c101750 2024-04-16T03:12:12.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.734+0000 7f89a0822700 1 -- 172.21.15.151:0/2608939513 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f8984062030 con 0x7f899c101750 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:12:12.737 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:12:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:12:12.739 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:12:12.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.736+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f898806ecc0 msgr2=0x7f8988071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f898806ecc0 0x7f8988071180 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f898c0059d0 tx=0x7f898c005940 comp rx=0 tx=0).stop 2024-04-16T03:12:12.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c101750 msgr2=0x7f899c19cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c101750 0x7f899c19cae0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f898400e8f0 tx=0x7f898400ec00 comp rx=0 tx=0).stop 2024-04-16T03:12:12.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 shutdown_connections 2024-04-16T03:12:12.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f898806ecc0 0x7f8988071180 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f899c100da0 0x7f899c19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 --2- 172.21.15.151:0/2608939513 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f899c101750 0x7f899c19cae0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 >> 172.21.15.151:0/2608939513 conn(0x7f899c0fc630 msgr2=0x7f899c0fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:12.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 shutdown_connections 2024-04-16T03:12:12.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.737+0000 7f89a1824700 1 -- 172.21.15.151:0/2608939513 wait complete. 2024-04-16T03:12:12.741 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:12:12.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.843+0000 7fe8dd44c700 1 -- 172.21.15.151:0/4177996980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 msgr2=0x7fe8d810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.843+0000 7fe8dd44c700 1 --2- 172.21.15.151:0/4177996980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 0x7fe8d810ab20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009910 tx=0x7fe8cc009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:12.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.844+0000 7fe8dd44c700 1 -- 172.21.15.151:0/4177996980 shutdown_connections 2024-04-16T03:12:12.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.844+0000 7fe8dd44c700 1 --2- 172.21.15.151:0/4177996980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 0x7fe8d810ab20 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.844+0000 7fe8dd44c700 1 --2- 172.21.15.151:0/4177996980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d8101c80 0x7fe8d8102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.844+0000 7fe8dd44c700 1 -- 172.21.15.151:0/4177996980 >> 172.21.15.151:0/4177996980 conn(0x7fe8d80fb540 msgr2=0x7fe8d80fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:12.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.844+0000 7fe8dd44c700 1 -- 172.21.15.151:0/4177996980 shutdown_connections 2024-04-16T03:12:12.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.844+0000 7fe8dd44c700 1 -- 172.21.15.151:0/4177996980 wait complete. 2024-04-16T03:12:12.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.844+0000 7fe8dd44c700 1 Processor -- start 2024-04-16T03:12:12.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.844+0000 7fe8dd44c700 1 -- start start 2024-04-16T03:12:12.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8dd44c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d8101c80 0x7fe8d8194550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:12.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8dd44c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 0x7fe8d8194a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:12.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8dd44c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d8195170 con 0x7fe8d8102630 2024-04-16T03:12:12.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8dd44c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d8198f00 con 0x7fe8d8101c80 2024-04-16T03:12:12.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8d67fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 0x7fe8d8194a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8d67fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 0x7fe8d8194a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38760/0 (socket says 172.21.15.151:38760) 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8d67fc700 1 -- 172.21.15.151:0/387308352 learned_addr learned my addr 172.21.15.151:0/387308352 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8d6ffd700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d8101c80 0x7fe8d8194550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8d67fc700 1 -- 172.21.15.151:0/387308352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d8101c80 msgr2=0x7fe8d8194550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8d67fc700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d8101c80 0x7fe8d8194550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8d67fc700 1 -- 172.21.15.151:0/387308352 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc0095f0 con 0x7fe8d8102630 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8d67fc700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 0x7fe8d8194a90 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc00bd70 tx=0x7fe8cc00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8bffff700 1 -- 172.21.15.151:0/387308352 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8cc010ac0 con 0x7fe8d8102630 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d8199180 con 0x7fe8d8102630 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8bffff700 1 -- 172.21.15.151:0/387308352 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8cc010c20 con 0x7fe8d8102630 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8bffff700 1 -- 172.21.15.151:0/387308352 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8cc0187f0 con 0x7fe8d8102630 2024-04-16T03:12:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.845+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d81996d0 con 0x7fe8d8102630 2024-04-16T03:12:12.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.847+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d8108220 con 0x7fe8d8102630 2024-04-16T03:12:12.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.847+0000 7fe8bffff700 1 -- 172.21.15.151:0/387308352 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe8cc003680 con 0x7fe8d8102630 2024-04-16T03:12:12.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.847+0000 7fe8bffff700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8c006ecc0 0x7fe8c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:12.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.847+0000 7fe8bffff700 1 -- 172.21.15.151:0/387308352 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe8cc0948a0 con 0x7fe8d8102630 2024-04-16T03:12:12.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.847+0000 7fe8d6ffd700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8c006ecc0 0x7fe8c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:12.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.848+0000 7fe8d6ffd700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8c006ecc0 0x7fe8c0071180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8005950 tx=0x7fe8c800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:12.854 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:12.852+0000 7fe8bffff700 1 -- 172.21.15.151:0/387308352 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe8cc05d790 con 0x7fe8d8102630 2024-04-16T03:12:12.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:12 smithi151 ceph-mon[112233]: from='client.35828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:12.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:12 smithi151 ceph-mon[112233]: pgmap v1008: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:12.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:12 smithi151 ceph-mon[112233]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:12.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:12 smithi151 ceph-mon[112233]: from='client.35836 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:12.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:12 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2771570092' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:12:12.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:12 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2608939513' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:12:13.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.075+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8d8199a00 con 0x7fe8c006ecc0 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.077+0000 7fe8bffff700 1 -- 172.21.15.151:0/387308352 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe8d8199a00 con 0x7fe8c006ecc0 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:12:13.079 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:12:13.080 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:12:13.080 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:12:13.080 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:12:13.080 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:12:13.080 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:12:13.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8c006ecc0 msgr2=0x7fe8c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:13.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8c006ecc0 0x7fe8c0071180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8005950 tx=0x7fe8c800b410 comp rx=0 tx=0).stop 2024-04-16T03:12:13.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 msgr2=0x7fe8d8194a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:13.081 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 0x7fe8d8194a90 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc00bd70 tx=0x7fe8cc00bda0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 shutdown_connections 2024-04-16T03:12:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8c006ecc0 0x7fe8c0071180 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d8101c80 0x7fe8d8194550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 --2- 172.21.15.151:0/387308352 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d8102630 0x7fe8d8194a90 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 >> 172.21.15.151:0/387308352 conn(0x7fe8d80fb540 msgr2=0x7fe8d8105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 shutdown_connections 2024-04-16T03:12:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.080+0000 7fe8dd44c700 1 -- 172.21.15.151:0/387308352 wait complete. 2024-04-16T03:12:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.184+0000 7f0819678700 1 -- 172.21.15.151:0/3012399753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0814105a40 msgr2=0x7f0814105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.184+0000 7f0819678700 1 --2- 172.21.15.151:0/3012399753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0814105a40 0x7f0814105e20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f080c009910 tx=0x7f080c009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.184+0000 7f0819678700 1 -- 172.21.15.151:0/3012399753 shutdown_connections 2024-04-16T03:12:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.184+0000 7f0819678700 1 --2- 172.21.15.151:0/3012399753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08140ffa40 0x7f08140ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.184+0000 7f0819678700 1 --2- 172.21.15.151:0/3012399753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0814105a40 0x7f0814105e20 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.184+0000 7f0819678700 1 -- 172.21.15.151:0/3012399753 >> 172.21.15.151:0/3012399753 conn(0x7f08140fb570 msgr2=0x7f08140fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.184+0000 7f0819678700 1 -- 172.21.15.151:0/3012399753 shutdown_connections 2024-04-16T03:12:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.184+0000 7f0819678700 1 -- 172.21.15.151:0/3012399753 wait complete. 2024-04-16T03:12:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f0819678700 1 Processor -- start 2024-04-16T03:12:13.188 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f0819678700 1 -- start start 2024-04-16T03:12:13.188 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f0819678700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08140ffa40 0x7f08141976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f0819678700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0814105a40 0x7f0814197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f0819678700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814198200 con 0x7f08140ffa40 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f0819678700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814198370 con 0x7f0814105a40 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f0812ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08140ffa40 0x7f08141976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f0812ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08140ffa40 0x7f08141976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38780/0 (socket says 172.21.15.151:38780) 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f0812ffd700 1 -- 172.21.15.151:0/3032615378 learned_addr learned my addr 172.21.15.151:0/3032615378 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.185+0000 7f08127fc700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0814105a40 0x7f0814197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.186+0000 7f0812ffd700 1 -- 172.21.15.151:0/3032615378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0814105a40 msgr2=0x7f0814197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.186+0000 7f0812ffd700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0814105a40 0x7f0814197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.186+0000 7f0812ffd700 1 -- 172.21.15.151:0/3032615378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c0095f0 con 0x7f08140ffa40 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.186+0000 7f0812ffd700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08140ffa40 0x7f08141976a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f080c006010 tx=0x7f080c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.186+0000 7f080bfff700 1 -- 172.21.15.151:0/3032615378 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f080c01d070 con 0x7f08140ffa40 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.186+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08141a0710 con 0x7f08140ffa40 2024-04-16T03:12:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.186+0000 7f080bfff700 1 -- 172.21.15.151:0/3032615378 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f080c004fb0 con 0x7f08140ffa40 2024-04-16T03:12:13.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.186+0000 7f080bfff700 1 -- 172.21.15.151:0/3032615378 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f080c022ea0 con 0x7f08140ffa40 2024-04-16T03:12:13.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.186+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08141a0b80 con 0x7f08140ffa40 2024-04-16T03:12:13.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.187+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0814108040 con 0x7f08140ffa40 2024-04-16T03:12:13.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.188+0000 7f080bfff700 1 -- 172.21.15.151:0/3032615378 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f080c022470 con 0x7f08140ffa40 2024-04-16T03:12:13.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.188+0000 7f080bfff700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f080006ecc0 0x7f0800071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:13.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.188+0000 7f080bfff700 1 -- 172.21.15.151:0/3032615378 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f080c0903d0 con 0x7f08140ffa40 2024-04-16T03:12:13.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.188+0000 7f08127fc700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f080006ecc0 0x7f0800071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:13.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.188+0000 7f08127fc700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f080006ecc0 0x7f0800071180 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f07fc006fd0 tx=0x7f07fc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:13.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.193+0000 7f080bfff700 1 -- 172.21.15.151:0/3032615378 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f080c059230 con 0x7f08140ffa40 2024-04-16T03:12:13.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:12 smithi153 ceph-mon[87858]: from='client.35828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:13.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:12 smithi153 ceph-mon[87858]: pgmap v1008: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:12 smithi153 ceph-mon[87858]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:12 smithi153 ceph-mon[87858]: from='client.35836 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:12 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2771570092' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:12:13.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:12 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2608939513' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:12:13.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.489+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f081404ea90 con 0x7f08140ffa40 2024-04-16T03:12:13.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.489+0000 7f080bfff700 1 -- 172.21.15.151:0/3032615378 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f080c022720 con 0x7f08140ffa40 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:12:13.492 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:12:13.493 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:12:13.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.491+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f080006ecc0 msgr2=0x7f0800071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:13.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.491+0000 7f0819678700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f080006ecc0 0x7f0800071180 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f07fc006fd0 tx=0x7f07fc008040 comp rx=0 tx=0).stop 2024-04-16T03:12:13.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.491+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08140ffa40 msgr2=0x7f08141976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:13.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.491+0000 7f0819678700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08140ffa40 0x7f08141976a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f080c006010 tx=0x7f080c00be20 comp rx=0 tx=0).stop 2024-04-16T03:12:13.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.491+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 shutdown_connections 2024-04-16T03:12:13.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.491+0000 7f0819678700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f080006ecc0 0x7f0800071180 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.491+0000 7f0819678700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08140ffa40 0x7f08141976a0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.491+0000 7f0819678700 1 --2- 172.21.15.151:0/3032615378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0814105a40 0x7f0814197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:13.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.491+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 >> 172.21.15.151:0/3032615378 conn(0x7f08140fb570 msgr2=0x7f08140fc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:13.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.492+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 shutdown_connections 2024-04-16T03:12:13.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:13.492+0000 7f0819678700 1 -- 172.21.15.151:0/3032615378 wait complete. 2024-04-16T03:12:14.223 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:13 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3032615378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:12:14.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:13 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3032615378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:12:15.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:14 smithi151 ceph-mon[112233]: from='client.35848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:15.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:14 smithi151 ceph-mon[112233]: pgmap v1009: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:15.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:14 smithi153 ceph-mon[87858]: from='client.35848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:15.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:14 smithi153 ceph-mon[87858]: pgmap v1009: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:17.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:16 smithi151 ceph-mon[112233]: pgmap v1010: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:17.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:16 smithi153 ceph-mon[87858]: pgmap v1010: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:19.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:18 smithi151 ceph-mon[112233]: pgmap v1011: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:19.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:18 smithi153 ceph-mon[87858]: pgmap v1011: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:21.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:20 smithi151 ceph-mon[112233]: pgmap v1012: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:21.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:20 smithi153 ceph-mon[87858]: pgmap v1012: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:23.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:22 smithi151 ceph-mon[112233]: pgmap v1013: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:23.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:22 smithi153 ceph-mon[87858]: pgmap v1013: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:24.246 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:23 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:12:24.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:23 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:12:25.172 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:24 smithi153 ceph-mon[87858]: pgmap v1014: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:25.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:24 smithi151 ceph-mon[112233]: pgmap v1014: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:27.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:26 smithi151 ceph-mon[112233]: pgmap v1015: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:27.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:26 smithi153 ceph-mon[87858]: pgmap v1015: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:29.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:28 smithi151 ceph-mon[112233]: pgmap v1016: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:29.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:28 smithi153 ceph-mon[87858]: pgmap v1016: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:31.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:30 smithi151 ceph-mon[112233]: pgmap v1017: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:31.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:30 smithi153 ceph-mon[87858]: pgmap v1017: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:33.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:32 smithi151 ceph-mon[112233]: pgmap v1018: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:33.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:32 smithi153 ceph-mon[87858]: pgmap v1018: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:35.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:34 smithi151 ceph-mon[112233]: pgmap v1019: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:35.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:34 smithi153 ceph-mon[87858]: pgmap v1019: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:37.209 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:36 smithi153 ceph-mon[87858]: pgmap v1020: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:37.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:36 smithi151 ceph-mon[112233]: pgmap v1020: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:38 smithi151 ceph-mon[112233]: pgmap v1021: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:12:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:38 smithi153 ceph-mon[87858]: pgmap v1021: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:39.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:12:41.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:40 smithi151 ceph-mon[112233]: pgmap v1022: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:41.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:40 smithi153 ceph-mon[87858]: pgmap v1022: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:43.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:42 smithi151 ceph-mon[112233]: pgmap v1023: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:43.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:42 smithi153 ceph-mon[87858]: pgmap v1023: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.611+0000 7ff7324f0700 1 -- 172.21.15.151:0/1515831376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c106c10 msgr2=0x7ff72c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.611+0000 7ff7324f0700 1 --2- 172.21.15.151:0/1515831376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c106c10 0x7ff72c106ff0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009910 tx=0x7ff71c009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.611+0000 7ff7324f0700 1 -- 172.21.15.151:0/1515831376 shutdown_connections 2024-04-16T03:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.611+0000 7ff7324f0700 1 --2- 172.21.15.151:0/1515831376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff72c0ff310 0x7ff72c0ff770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.611+0000 7ff7324f0700 1 --2- 172.21.15.151:0/1515831376 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c106c10 0x7ff72c106ff0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.611+0000 7ff7324f0700 1 -- 172.21.15.151:0/1515831376 >> 172.21.15.151:0/1515831376 conn(0x7ff72c074bf0 msgr2=0x7ff72c075000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.611+0000 7ff7324f0700 1 -- 172.21.15.151:0/1515831376 shutdown_connections 2024-04-16T03:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.611+0000 7ff7324f0700 1 -- 172.21.15.151:0/1515831376 wait complete. 2024-04-16T03:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff7324f0700 1 Processor -- start 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff7324f0700 1 -- start start 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff7324f0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c0ff310 0x7ff72c19c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff7324f0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff72c106c10 0x7ff72c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff7324f0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c19d190 con 0x7ff72c0ff310 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff7324f0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c196640 con 0x7ff72c106c10 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff72bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c0ff310 0x7ff72c19c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff72bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c0ff310 0x7ff72c19c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57556/0 (socket says 172.21.15.151:57556) 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff72bfff700 1 -- 172.21.15.151:0/1454589563 learned_addr learned my addr 172.21.15.151:0/1454589563 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff723fff700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff72c106c10 0x7ff72c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff72bfff700 1 -- 172.21.15.151:0/1454589563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff72c106c10 msgr2=0x7ff72c19cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff72bfff700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff72c106c10 0x7ff72c19cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.612+0000 7ff72bfff700 1 -- 172.21.15.151:0/1454589563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c0095f0 con 0x7ff72c0ff310 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.613+0000 7ff72bfff700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c0ff310 0x7ff72c19c5c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff71c006010 tx=0x7ff71c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.613+0000 7ff729ffb700 1 -- 172.21.15.151:0/1454589563 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff71c01d070 con 0x7ff72c0ff310 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.613+0000 7ff729ffb700 1 -- 172.21.15.151:0/1454589563 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff71c004fb0 con 0x7ff72c0ff310 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.613+0000 7ff729ffb700 1 -- 172.21.15.151:0/1454589563 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff71c022ea0 con 0x7ff72c0ff310 2024-04-16T03:12:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.613+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72c1968c0 con 0x7ff72c0ff310 2024-04-16T03:12:43.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.613+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c196db0 con 0x7ff72c0ff310 2024-04-16T03:12:43.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.614+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c109130 con 0x7ff72c0ff310 2024-04-16T03:12:43.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.614+0000 7ff729ffb700 1 -- 172.21.15.151:0/1454589563 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff71c022470 con 0x7ff72c0ff310 2024-04-16T03:12:43.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.615+0000 7ff729ffb700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff71806ecc0 0x7ff718071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:43.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.615+0000 7ff723fff700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff71806ecc0 0x7ff718071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:43.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.615+0000 7ff729ffb700 1 -- 172.21.15.151:0/1454589563 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff71c090350 con 0x7ff72c0ff310 2024-04-16T03:12:43.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.616+0000 7ff723fff700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff71806ecc0 0x7ff718071180 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff714006fd0 tx=0x7ff714008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:43.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.620+0000 7ff729ffb700 1 -- 172.21.15.151:0/1454589563 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff71c05cd60 con 0x7ff72c0ff310 2024-04-16T03:12:43.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.837+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff72c1970e0 con 0x7ff71806ecc0 2024-04-16T03:12:43.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.838+0000 7ff729ffb700 1 -- 172.21.15.151:0/1454589563 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff72c1970e0 con 0x7ff71806ecc0 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.840+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff71806ecc0 msgr2=0x7ff718071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.840+0000 7ff7324f0700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff71806ecc0 0x7ff718071180 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff714006fd0 tx=0x7ff714008040 comp rx=0 tx=0).stop 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.840+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c0ff310 msgr2=0x7ff72c19c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.840+0000 7ff7324f0700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c0ff310 0x7ff72c19c5c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff71c006010 tx=0x7ff71c00be20 comp rx=0 tx=0).stop 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.840+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 shutdown_connections 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.840+0000 7ff7324f0700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff71806ecc0 0x7ff718071180 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.840+0000 7ff7324f0700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff72c0ff310 0x7ff72c19c5c0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.840+0000 7ff7324f0700 1 --2- 172.21.15.151:0/1454589563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff72c106c10 0x7ff72c19cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.840+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 >> 172.21.15.151:0/1454589563 conn(0x7ff72c074bf0 msgr2=0x7ff72c0fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.841+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 shutdown_connections 2024-04-16T03:12:43.843 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.841+0000 7ff7324f0700 1 -- 172.21.15.151:0/1454589563 wait complete. 2024-04-16T03:12:43.852 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:12:43.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.946+0000 7f111956b700 1 -- 172.21.15.151:0/1019536928 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 msgr2=0x7f11140ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:43.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.946+0000 7f111956b700 1 --2- 172.21.15.151:0/1019536928 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 0x7f11140ffeb0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f1108009910 tx=0x7f1108009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:43.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.946+0000 7f111956b700 1 -- 172.21.15.151:0/1019536928 shutdown_connections 2024-04-16T03:12:43.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.946+0000 7f111956b700 1 --2- 172.21.15.151:0/1019536928 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 0x7f11140ffeb0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:43.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.946+0000 7f111956b700 1 --2- 172.21.15.151:0/1019536928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1114105a30 0x7f1114105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:43.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.946+0000 7f111956b700 1 -- 172.21.15.151:0/1019536928 >> 172.21.15.151:0/1019536928 conn(0x7f11140fb540 msgr2=0x7f11140fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:43.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.946+0000 7f111956b700 1 -- 172.21.15.151:0/1019536928 shutdown_connections 2024-04-16T03:12:43.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.946+0000 7f111956b700 1 -- 172.21.15.151:0/1019536928 wait complete. 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.947+0000 7f111956b700 1 Processor -- start 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.947+0000 7f111956b700 1 -- start start 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.947+0000 7f111956b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 0x7f11140fbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.947+0000 7f111956b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1114105a30 0x7f11140fc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.947+0000 7f111956b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11140fc910 con 0x7f11140ffa30 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.947+0000 7f111956b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11140fca50 con 0x7f1114105a30 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f1112ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 0x7f11140fbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f1112ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 0x7f11140fbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57574/0 (socket says 172.21.15.151:57574) 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f1112ffd700 1 -- 172.21.15.151:0/2097587717 learned_addr learned my addr 172.21.15.151:0/2097587717 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f11127fc700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1114105a30 0x7f11140fc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f1112ffd700 1 -- 172.21.15.151:0/2097587717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1114105a30 msgr2=0x7f11140fc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f1112ffd700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1114105a30 0x7f11140fc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f1112ffd700 1 -- 172.21.15.151:0/2097587717 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080095f0 con 0x7f11140ffa30 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f11127fc700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1114105a30 0x7f11140fc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f1112ffd700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 0x7f11140fbe90 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f110400d8d0 tx=0x7f110400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f10fbfff700 1 -- 172.21.15.151:0/2097587717 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1104009940 con 0x7f11140ffa30 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11141a28e0 con 0x7f11140ffa30 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f10fbfff700 1 -- 172.21.15.151:0/2097587717 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1104010460 con 0x7f11140ffa30 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f10fbfff700 1 -- 172.21.15.151:0/2097587717 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f110400f5f0 con 0x7f11140ffa30 2024-04-16T03:12:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.948+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11141a2cf0 con 0x7f11140ffa30 2024-04-16T03:12:43.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.949+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1114108fd0 con 0x7f11140ffa30 2024-04-16T03:12:43.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.950+0000 7f10fbfff700 1 -- 172.21.15.151:0/2097587717 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f11040105d0 con 0x7f11140ffa30 2024-04-16T03:12:43.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.951+0000 7f10fbfff700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f10fc06ecc0 0x7f10fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:43.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.951+0000 7f10fbfff700 1 -- 172.21.15.151:0/2097587717 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f110408ee80 con 0x7f11140ffa30 2024-04-16T03:12:43.952 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.951+0000 7f11127fc700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f10fc06ecc0 0x7f10fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:43.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.952+0000 7f11127fc700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f10fc06ecc0 0x7f10fc071180 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f110800b5c0 tx=0x7f110801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:43.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:43.956+0000 7f10fbfff700 1 -- 172.21.15.151:0/2097587717 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1104057c30 con 0x7f11140ffa30 2024-04-16T03:12:44.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.173+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f11140fcc10 con 0x7f10fc06ecc0 2024-04-16T03:12:44.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.174+0000 7f10fbfff700 1 -- 172.21.15.151:0/2097587717 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f11140fcc10 con 0x7f10fc06ecc0 2024-04-16T03:12:44.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f10fc06ecc0 msgr2=0x7f10fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f10fc06ecc0 0x7f10fc071180 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f110800b5c0 tx=0x7f110801a040 comp rx=0 tx=0).stop 2024-04-16T03:12:44.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 msgr2=0x7f11140fbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 0x7f11140fbe90 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f110400d8d0 tx=0x7f110400dc90 comp rx=0 tx=0).stop 2024-04-16T03:12:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 shutdown_connections 2024-04-16T03:12:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f10fc06ecc0 0x7f10fc071180 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11140ffa30 0x7f11140fbe90 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 --2- 172.21.15.151:0/2097587717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1114105a30 0x7f11140fc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 >> 172.21.15.151:0/2097587717 conn(0x7f11140fb540 msgr2=0x7f11140690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 shutdown_connections 2024-04-16T03:12:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.177+0000 7f111956b700 1 -- 172.21.15.151:0/2097587717 wait complete. 2024-04-16T03:12:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.282+0000 7f69bf61a700 1 -- 172.21.15.151:0/4036547297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 msgr2=0x7f69b8106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.282+0000 7f69bf61a700 1 --2- 172.21.15.151:0/4036547297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 0x7f69b8106fe0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009910 tx=0x7f69b4009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.282+0000 7f69bf61a700 1 -- 172.21.15.151:0/4036547297 shutdown_connections 2024-04-16T03:12:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.282+0000 7f69bf61a700 1 --2- 172.21.15.151:0/4036547297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b8100b50 0x7f69b8100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.282+0000 7f69bf61a700 1 --2- 172.21.15.151:0/4036547297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 0x7f69b8106fe0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.282+0000 7f69bf61a700 1 -- 172.21.15.151:0/4036547297 >> 172.21.15.151:0/4036547297 conn(0x7f69b80fc640 msgr2=0x7f69b80fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.282+0000 7f69bf61a700 1 -- 172.21.15.151:0/4036547297 shutdown_connections 2024-04-16T03:12:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.282+0000 7f69bf61a700 1 -- 172.21.15.151:0/4036547297 wait complete. 2024-04-16T03:12:44.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.283+0000 7f69bf61a700 1 Processor -- start 2024-04-16T03:12:44.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.283+0000 7f69bf61a700 1 -- start start 2024-04-16T03:12:44.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.283+0000 7f69bf61a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b8100b50 0x7f69b8196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:44.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.283+0000 7f69bf61a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 0x7f69b8196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:44.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.283+0000 7f69bf61a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8197290 con 0x7f69b8106c00 2024-04-16T03:12:44.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.283+0000 7f69bf61a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b819b020 con 0x7f69b8100b50 2024-04-16T03:12:44.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.283+0000 7f69bcbb5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 0x7f69b8196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:44.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.283+0000 7f69bcbb5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 0x7f69b8196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57598/0 (socket says 172.21.15.151:57598) 2024-04-16T03:12:44.288 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.283+0000 7f69bcbb5700 1 -- 172.21.15.151:0/1322717267 learned_addr learned my addr 172.21.15.151:0/1322717267 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:44.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69bd3b6700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b8100b50 0x7f69b8196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:44.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69bcbb5700 1 -- 172.21.15.151:0/1322717267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b8100b50 msgr2=0x7f69b8196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69bcbb5700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b8100b50 0x7f69b8196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69bcbb5700 1 -- 172.21.15.151:0/1322717267 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b40095f0 con 0x7f69b8106c00 2024-04-16T03:12:44.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69bd3b6700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b8100b50 0x7f69b8196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:12:44.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69bcbb5700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 0x7f69b8196bb0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00d8d0 tx=0x7f69ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:44.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69aa7fc700 1 -- 172.21.15.151:0/1322717267 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69ac009940 con 0x7f69b8106c00 2024-04-16T03:12:44.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b819b300 con 0x7f69b8106c00 2024-04-16T03:12:44.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69aa7fc700 1 -- 172.21.15.151:0/1322717267 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69ac010460 con 0x7f69b8106c00 2024-04-16T03:12:44.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69aa7fc700 1 -- 172.21.15.151:0/1322717267 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69ac00f5f0 con 0x7f69b8106c00 2024-04-16T03:12:44.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.284+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b819b8d0 con 0x7f69b8106c00 2024-04-16T03:12:44.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.285+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b8109030 con 0x7f69b8106c00 2024-04-16T03:12:44.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.286+0000 7f69aa7fc700 1 -- 172.21.15.151:0/1322717267 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f69ac0105d0 con 0x7f69b8106c00 2024-04-16T03:12:44.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.286+0000 7f69aa7fc700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f69a406ecc0 0x7f69a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:44.290 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.286+0000 7f69aa7fc700 1 -- 172.21.15.151:0/1322717267 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f69ac08ebe0 con 0x7f69b8106c00 2024-04-16T03:12:44.291 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.286+0000 7f69bd3b6700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f69a406ecc0 0x7f69a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:44.291 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.287+0000 7f69bd3b6700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f69a406ecc0 0x7f69a4071180 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f69b4005b40 tx=0x7f69b401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:44.292 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.291+0000 7f69aa7fc700 1 -- 172.21.15.151:0/1322717267 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f69ac057990 con 0x7f69b8106c00 2024-04-16T03:12:44.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.500+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f69b819bc00 con 0x7f69a406ecc0 2024-04-16T03:12:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.511+0000 7f69aa7fc700 1 -- 172.21.15.151:0/1322717267 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f69b819bc00 con 0x7f69a406ecc0 2024-04-16T03:12:44.512 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:12:44.512 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (34m) 12m ago 45m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (45m) 12m ago 45m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (44m) 12m ago 44m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (31m) 12m ago 45m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (31m) 12m ago 44m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (33m) 12m ago 45m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (28m) 12m ago 40m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (27m) 12m ago 39m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (12m) 12m ago 39m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (27m) 12m ago 40m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:12:44.513 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (36m) 12m ago 47m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:12:44.514 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (35m) 12m ago 44m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:12:44.514 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (32m) 12m ago 47m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:12:44.514 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (32m) 12m ago 44m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:12:44.514 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (35m) 12m ago 45m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:12:44.514 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (34m) 12m ago 44m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:12:44.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (31m) 12m ago 43m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:12:44.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (30m) 12m ago 43m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:12:44.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (30m) 12m ago 42m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:12:44.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (29m) 12m ago 42m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:12:44.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (29m) 12m ago 41m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:12:44.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (28m) 12m ago 41m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:12:44.515 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (34m) 12m ago 45m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:12:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f69a406ecc0 msgr2=0x7f69a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f69a406ecc0 0x7f69a4071180 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f69b4005b40 tx=0x7f69b401a040 comp rx=0 tx=0).stop 2024-04-16T03:12:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 msgr2=0x7f69b8196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 0x7f69b8196bb0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00d8d0 tx=0x7f69ac00dc90 comp rx=0 tx=0).stop 2024-04-16T03:12:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 shutdown_connections 2024-04-16T03:12:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f69a406ecc0 0x7f69a4071180 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b8100b50 0x7f69b8196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 --2- 172.21.15.151:0/1322717267 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b8106c00 0x7f69b8196bb0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 >> 172.21.15.151:0/1322717267 conn(0x7f69b80fc640 msgr2=0x7f69b80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 shutdown_connections 2024-04-16T03:12:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.514+0000 7f69bf61a700 1 -- 172.21.15.151:0/1322717267 wait complete. 2024-04-16T03:12:44.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.623+0000 7f8e14624700 1 -- 172.21.15.151:0/16315421 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c102680 msgr2=0x7f8e0c10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.623+0000 7f8e14624700 1 --2- 172.21.15.151:0/16315421 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c102680 0x7f8e0c10ab70 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009940 tx=0x7f8dfc009c50 comp rx=0 tx=0).stop 2024-04-16T03:12:44.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.624+0000 7f8e14624700 1 -- 172.21.15.151:0/16315421 shutdown_connections 2024-04-16T03:12:44.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.624+0000 7f8e14624700 1 --2- 172.21.15.151:0/16315421 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c102680 0x7f8e0c10ab70 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.624+0000 7f8e14624700 1 --2- 172.21.15.151:0/16315421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e0c101d60 0x7f8e0c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.627 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.624+0000 7f8e14624700 1 -- 172.21.15.151:0/16315421 >> 172.21.15.151:0/16315421 conn(0x7f8e0c0fb550 msgr2=0x7f8e0c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.624+0000 7f8e14624700 1 -- 172.21.15.151:0/16315421 shutdown_connections 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.624+0000 7f8e14624700 1 -- 172.21.15.151:0/16315421 wait complete. 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.624+0000 7f8e14624700 1 Processor -- start 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.624+0000 7f8e14624700 1 -- start start 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e14624700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c101d60 0x7f8e0c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e14624700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e0c102680 0x7f8e0c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e14624700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c19d110 con 0x7f8e0c101d60 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e14624700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c1965c0 con 0x7f8e0c102680 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e123c0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c101d60 0x7f8e0c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e123c0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c101d60 0x7f8e0c19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57616/0 (socket says 172.21.15.151:57616) 2024-04-16T03:12:44.628 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e123c0700 1 -- 172.21.15.151:0/350954676 learned_addr learned my addr 172.21.15.151:0/350954676 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e11bbf700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e0c102680 0x7f8e0c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e123c0700 1 -- 172.21.15.151:0/350954676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e0c102680 msgr2=0x7f8e0c19ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e123c0700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e0c102680 0x7f8e0c19ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e123c0700 1 -- 172.21.15.151:0/350954676 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc0095f0 con 0x7f8e0c101d60 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e123c0700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c101d60 0x7f8e0c19c540 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800d8d0 tx=0x7f8e0800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e037fe700 1 -- 172.21.15.151:0/350954676 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e080098a0 con 0x7f8e0c101d60 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e037fe700 1 -- 172.21.15.151:0/350954676 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e08010460 con 0x7f8e0c101d60 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0c1968a0 con 0x7f8e0c101d60 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e037fe700 1 -- 172.21.15.151:0/350954676 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e0800f5f0 con 0x7f8e0c101d60 2024-04-16T03:12:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.625+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0c196dc0 con 0x7f8e0c101d60 2024-04-16T03:12:44.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.626+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0c108270 con 0x7f8e0c101d60 2024-04-16T03:12:44.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.627+0000 7f8e037fe700 1 -- 172.21.15.151:0/350954676 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8e08009a00 con 0x7f8e0c101d60 2024-04-16T03:12:44.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.627+0000 7f8e037fe700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8df806ecc0 0x7f8df8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:44.630 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.628+0000 7f8e11bbf700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8df806ecc0 0x7f8df8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:44.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.628+0000 7f8e037fe700 1 -- 172.21.15.151:0/350954676 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8e0808ec30 con 0x7f8e0c101d60 2024-04-16T03:12:44.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.628+0000 7f8e11bbf700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8df806ecc0 0x7f8df8071180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009910 tx=0x7f8dfc005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.632+0000 7f8e037fe700 1 -- 172.21.15.151:0/350954676 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8e0805b590 con 0x7f8e0c101d60 2024-04-16T03:12:44.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.913+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8e0c197060 con 0x7f8e0c101d60 2024-04-16T03:12:44.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.914+0000 7f8e037fe700 1 -- 172.21.15.151:0/350954676 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f8e0805b120 con 0x7f8e0c101d60 2024-04-16T03:12:44.915 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:12:44.916 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:12:44.917 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:12:44.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:12:44.917 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:12:44.917 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:12:44.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:12:44.917 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:12:44.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8df806ecc0 msgr2=0x7f8df8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8df806ecc0 0x7f8df8071180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009910 tx=0x7f8dfc005fd0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c101d60 msgr2=0x7f8e0c19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:44.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c101d60 0x7f8e0c19c540 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800d8d0 tx=0x7f8e0800dbe0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 shutdown_connections 2024-04-16T03:12:44.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8df806ecc0 0x7f8df8071180 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0c101d60 0x7f8e0c19c540 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 --2- 172.21.15.151:0/350954676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e0c102680 0x7f8e0c19ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:44.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 >> 172.21.15.151:0/350954676 conn(0x7f8e0c0fb550 msgr2=0x7f8e0c1053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:44.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 shutdown_connections 2024-04-16T03:12:44.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:44.916+0000 7f8e14624700 1 -- 172.21.15.151:0/350954676 wait complete. 2024-04-16T03:12:45.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.021+0000 7f0da864d700 1 -- 172.21.15.151:0/1865650092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 msgr2=0x7f0da01057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.021+0000 7f0da864d700 1 --2- 172.21.15.151:0/1865650092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 0x7f0da01057c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009910 tx=0x7f0d90009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:45.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 -- 172.21.15.151:0/1865650092 shutdown_connections 2024-04-16T03:12:45.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 --2- 172.21.15.151:0/1865650092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 0x7f0da01057c0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 --2- 172.21.15.151:0/1865650092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0da0100dc0 0x7f0da01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 -- 172.21.15.151:0/1865650092 >> 172.21.15.151:0/1865650092 conn(0x7f0da00fc650 msgr2=0x7f0da00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:45.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 -- 172.21.15.151:0/1865650092 shutdown_connections 2024-04-16T03:12:45.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 -- 172.21.15.151:0/1865650092 wait complete. 2024-04-16T03:12:45.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 Processor -- start 2024-04-16T03:12:45.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 -- start start 2024-04-16T03:12:45.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0da0100dc0 0x7f0da0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:45.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 0x7f0da0198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:45.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da0199540 con 0x7f0da0101770 2024-04-16T03:12:45.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.022+0000 7f0da864d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da019d350 con 0x7f0da0100dc0 2024-04-16T03:12:45.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da5be8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 0x7f0da0198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:45.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da5be8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 0x7f0da0198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57624/0 (socket says 172.21.15.151:57624) 2024-04-16T03:12:45.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da5be8700 1 -- 172.21.15.151:0/3295901105 learned_addr learned my addr 172.21.15.151:0/3295901105 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:45.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da63e9700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0da0100dc0 0x7f0da0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:45.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da5be8700 1 -- 172.21.15.151:0/3295901105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0da0100dc0 msgr2=0x7f0da0198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da5be8700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0da0100dc0 0x7f0da0198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da5be8700 1 -- 172.21.15.151:0/3295901105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d900095f0 con 0x7f0da0101770 2024-04-16T03:12:45.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da5be8700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 0x7f0da0198eb0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f0d900092e0 tx=0x7f0d9000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:45.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0d977fe700 1 -- 172.21.15.151:0/3295901105 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d9001d070 con 0x7f0da0101770 2024-04-16T03:12:45.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da019d5d0 con 0x7f0da0101770 2024-04-16T03:12:45.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0d977fe700 1 -- 172.21.15.151:0/3295901105 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d90005dc0 con 0x7f0da0101770 2024-04-16T03:12:45.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0d977fe700 1 -- 172.21.15.151:0/3295901105 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d90017400 con 0x7f0da0101770 2024-04-16T03:12:45.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.023+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da019dac0 con 0x7f0da0101770 2024-04-16T03:12:45.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.024+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da0104f40 con 0x7f0da0101770 2024-04-16T03:12:45.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.025+0000 7f0d977fe700 1 -- 172.21.15.151:0/3295901105 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f0d9000f6f0 con 0x7f0da0101770 2024-04-16T03:12:45.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.025+0000 7f0d977fe700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d8c06ecc0 0x7f0d8c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:45.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.026+0000 7f0d977fe700 1 -- 172.21.15.151:0/3295901105 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0d900902f0 con 0x7f0da0101770 2024-04-16T03:12:45.029 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.026+0000 7f0da63e9700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d8c06ecc0 0x7f0d8c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:45.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.026+0000 7f0da63e9700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d8c06ecc0 0x7f0d8c071180 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c005950 tx=0x7f0d9c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:45.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.030+0000 7f0d977fe700 1 -- 172.21.15.151:0/3295901105 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0d9005cb30 con 0x7f0da0101770 2024-04-16T03:12:45.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:44 smithi151 ceph-mon[112233]: pgmap v1024: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:45.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:44 smithi151 ceph-mon[112233]: from='client.35856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:45.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:44 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/350954676' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:12:45.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:44 smithi153 ceph-mon[87858]: pgmap v1024: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:45.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:44 smithi153 ceph-mon[87858]: from='client.35856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:45.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:44 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/350954676' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:12:45.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.273+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0da004ea90 con 0x7f0da0101770 2024-04-16T03:12:45.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.274+0000 7f0d977fe700 1 -- 172.21.15.151:0/3295901105 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f0d9005c6c0 con 0x7f0da0101770 2024-04-16T03:12:45.275 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:12:45.277 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:12:45.277 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:12:45.277 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:12:45.277 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:12:45.277 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:12:45.277 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:12:45.277 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:12:45.278 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:12:45.278 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:12:45.278 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:12:45.278 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:12:45.279 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:12:45.279 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:12:45.279 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:12:45.279 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:12:45.279 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:12:45.279 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:12:45.279 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:12:45.279 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:12:45.280 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:12:45.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d8c06ecc0 msgr2=0x7f0d8c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d8c06ecc0 0x7f0d8c071180 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c005950 tx=0x7f0d9c00b410 comp rx=0 tx=0).stop 2024-04-16T03:12:45.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 msgr2=0x7f0da0198eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 0x7f0da0198eb0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f0d900092e0 tx=0x7f0d9000bd70 comp rx=0 tx=0).stop 2024-04-16T03:12:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 shutdown_connections 2024-04-16T03:12:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f0d8c06ecc0 0x7f0d8c071180 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0da0100dc0 0x7f0da0198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 --2- 172.21.15.151:0/3295901105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da0101770 0x7f0da0198eb0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 >> 172.21.15.151:0/3295901105 conn(0x7f0da00fc650 msgr2=0x7f0da00fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 shutdown_connections 2024-04-16T03:12:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.276+0000 7f0da864d700 1 -- 172.21.15.151:0/3295901105 wait complete. 2024-04-16T03:12:45.283 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:12:45.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.381+0000 7f9dba4e2700 1 -- 172.21.15.151:0/1005522794 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 msgr2=0x7f9db4100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.381+0000 7f9dba4e2700 1 --2- 172.21.15.151:0/1005522794 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 0x7f9db4100a00 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f9da4009910 tx=0x7f9da4009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:45.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.381+0000 7f9dba4e2700 1 -- 172.21.15.151:0/1005522794 shutdown_connections 2024-04-16T03:12:45.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.381+0000 7f9dba4e2700 1 --2- 172.21.15.151:0/1005522794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9db4100fd0 0x7f9db41046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.381+0000 7f9dba4e2700 1 --2- 172.21.15.151:0/1005522794 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 0x7f9db4100a00 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.381+0000 7f9dba4e2700 1 -- 172.21.15.151:0/1005522794 >> 172.21.15.151:0/1005522794 conn(0x7f9db4074d80 msgr2=0x7f9db4075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:45.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.381+0000 7f9dba4e2700 1 -- 172.21.15.151:0/1005522794 shutdown_connections 2024-04-16T03:12:45.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.381+0000 7f9dba4e2700 1 -- 172.21.15.151:0/1005522794 wait complete. 2024-04-16T03:12:45.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.382+0000 7f9dba4e2700 1 Processor -- start 2024-04-16T03:12:45.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.382+0000 7f9dba4e2700 1 -- start start 2024-04-16T03:12:45.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.382+0000 7f9dba4e2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 0x7f9db419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:45.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.382+0000 7f9dba4e2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9db4100fd0 0x7f9db419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:45.385 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.382+0000 7f9dba4e2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419f350 con 0x7f9db4100620 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.382+0000 7f9dba4e2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db4198800 con 0x7f9db4100fd0 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 0x7f9db419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db3fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 0x7f9db419e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57636/0 (socket says 172.21.15.151:57636) 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db3fff700 1 -- 172.21.15.151:0/2627681147 learned_addr learned my addr 172.21.15.151:0/2627681147 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db37fe700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9db4100fd0 0x7f9db419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db3fff700 1 -- 172.21.15.151:0/2627681147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9db4100fd0 msgr2=0x7f9db419ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db3fff700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9db4100fd0 0x7f9db419ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db3fff700 1 -- 172.21.15.151:0/2627681147 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da40095f0 con 0x7f9db4100620 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db3fff700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 0x7f9db419e780 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9da40092e0 tx=0x7f9da400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db17fa700 1 -- 172.21.15.151:0/2627681147 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9da401d070 con 0x7f9db4100620 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db17fa700 1 -- 172.21.15.151:0/2627681147 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9da4005e00 con 0x7f9db4100620 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db4198a80 con 0x7f9db4100620 2024-04-16T03:12:45.386 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9db17fa700 1 -- 172.21.15.151:0/2627681147 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9da4017400 con 0x7f9db4100620 2024-04-16T03:12:45.387 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.383+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db4198f70 con 0x7f9db4100620 2024-04-16T03:12:45.387 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.384+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db404ea90 con 0x7f9db4100620 2024-04-16T03:12:45.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.385+0000 7f9db17fa700 1 -- 172.21.15.151:0/2627681147 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9da400f6f0 con 0x7f9db4100620 2024-04-16T03:12:45.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.385+0000 7f9db17fa700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9d9c06ecc0 0x7f9d9c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:45.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.386+0000 7f9db17fa700 1 -- 172.21.15.151:0/2627681147 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9da4090340 con 0x7f9db4100620 2024-04-16T03:12:45.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.386+0000 7f9db37fe700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9d9c06ecc0 0x7f9d9c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:45.388 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.386+0000 7f9db37fe700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9d9c06ecc0 0x7f9d9c071180 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9db4106670 tx=0x7f9da800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:45.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.390+0000 7f9db17fa700 1 -- 172.21.15.151:0/2627681147 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9da405cc20 con 0x7f9db4100620 2024-04-16T03:12:45.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.609+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9db41992f0 con 0x7f9d9c06ecc0 2024-04-16T03:12:45.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.611+0000 7f9db17fa700 1 -- 172.21.15.151:0/2627681147 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9db41992f0 con 0x7f9d9c06ecc0 2024-04-16T03:12:45.612 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:12:45.612 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:12:45.612 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:12:45.612 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:12:45.612 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:12:45.613 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:12:45.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9d9c06ecc0 msgr2=0x7f9d9c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9d9c06ecc0 0x7f9d9c071180 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9db4106670 tx=0x7f9da800b410 comp rx=0 tx=0).stop 2024-04-16T03:12:45.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 msgr2=0x7f9db419e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 0x7f9db419e780 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9da40092e0 tx=0x7f9da400bd70 comp rx=0 tx=0).stop 2024-04-16T03:12:45.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 shutdown_connections 2024-04-16T03:12:45.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9d9c06ecc0 0x7f9d9c071180 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9db4100620 0x7f9db419e780 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 --2- 172.21.15.151:0/2627681147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9db4100fd0 0x7f9db419ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 >> 172.21.15.151:0/2627681147 conn(0x7f9db4074d80 msgr2=0x7f9db41036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:45.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 shutdown_connections 2024-04-16T03:12:45.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.613+0000 7f9dba4e2700 1 -- 172.21.15.151:0/2627681147 wait complete. 2024-04-16T03:12:45.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.718+0000 7f91a58e7700 1 -- 172.21.15.151:0/3609105971 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0101770 msgr2=0x7f91a01057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.718+0000 7f91a58e7700 1 --2- 172.21.15.151:0/3609105971 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0101770 0x7f91a01057c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9194009910 tx=0x7f9194009c20 comp rx=0 tx=0).stop 2024-04-16T03:12:45.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.718+0000 7f91a58e7700 1 -- 172.21.15.151:0/3609105971 shutdown_connections 2024-04-16T03:12:45.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.718+0000 7f91a58e7700 1 --2- 172.21.15.151:0/3609105971 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0101770 0x7f91a01057c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.718+0000 7f91a58e7700 1 --2- 172.21.15.151:0/3609105971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a0100dc0 0x7f91a01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.718+0000 7f91a58e7700 1 -- 172.21.15.151:0/3609105971 >> 172.21.15.151:0/3609105971 conn(0x7f91a00fc650 msgr2=0x7f91a00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:45.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.718+0000 7f91a58e7700 1 -- 172.21.15.151:0/3609105971 shutdown_connections 2024-04-16T03:12:45.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.718+0000 7f91a58e7700 1 -- 172.21.15.151:0/3609105971 wait complete. 2024-04-16T03:12:45.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.719+0000 7f91a58e7700 1 Processor -- start 2024-04-16T03:12:45.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.719+0000 7f91a58e7700 1 -- start start 2024-04-16T03:12:45.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.719+0000 7f91a58e7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0100dc0 0x7f91a019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:45.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.719+0000 7f91a58e7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a0101770 0x7f91a019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:45.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f91a58e7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a019d130 con 0x7f91a0100dc0 2024-04-16T03:12:45.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f91a58e7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a01965e0 con 0x7f91a0101770 2024-04-16T03:12:45.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f919effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0100dc0 0x7f91a019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f919effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0100dc0 0x7f91a019c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57656/0 (socket says 172.21.15.151:57656) 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f919effd700 1 -- 172.21.15.151:0/1593034950 learned_addr learned my addr 172.21.15.151:0/1593034950 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f919e7fc700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a0101770 0x7f91a019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f919effd700 1 -- 172.21.15.151:0/1593034950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a0101770 msgr2=0x7f91a019caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f919effd700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a0101770 0x7f91a019caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f919effd700 1 -- 172.21.15.151:0/1593034950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91940095f0 con 0x7f91a0100dc0 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f919e7fc700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a0101770 0x7f91a019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f919effd700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0100dc0 0x7f91a019c560 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f919000c8a0 tx=0x7f919000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f91a48e5700 1 -- 172.21.15.151:0/1593034950 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919000cea0 con 0x7f91a0100dc0 2024-04-16T03:12:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a01968c0 con 0x7f91a0100dc0 2024-04-16T03:12:45.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f91a48e5700 1 -- 172.21.15.151:0/1593034950 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9190004830 con 0x7f91a0100dc0 2024-04-16T03:12:45.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f91a48e5700 1 -- 172.21.15.151:0/1593034950 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91900055b0 con 0x7f91a0100dc0 2024-04-16T03:12:45.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.720+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a0196e10 con 0x7f91a0100dc0 2024-04-16T03:12:45.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.721+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a01091c0 con 0x7f91a0100dc0 2024-04-16T03:12:45.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.722+0000 7f91a48e5700 1 -- 172.21.15.151:0/1593034950 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9190007ad0 con 0x7f91a0100dc0 2024-04-16T03:12:45.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.722+0000 7f91a48e5700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f918806ecc0 0x7f9188071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:12:45.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.722+0000 7f91a48e5700 1 -- 172.21.15.151:0/1593034950 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f919008ea20 con 0x7f91a0100dc0 2024-04-16T03:12:45.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.722+0000 7f919e7fc700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f918806ecc0 0x7f9188071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:12:45.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.723+0000 7f919e7fc700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f918806ecc0 0x7f9188071180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f91a00fcea0 tx=0x7f919401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:12:45.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:45.727+0000 7f91a48e5700 1 -- 172.21.15.151:0/1593034950 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f919005b380 con 0x7f91a0100dc0 2024-04-16T03:12:46.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.010+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f91a0197160 con 0x7f91a0100dc0 2024-04-16T03:12:46.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.010+0000 7f91a48e5700 1 -- 172.21.15.151:0/1593034950 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f919001d020 con 0x7f91a0100dc0 2024-04-16T03:12:46.012 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:12:46.012 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:12:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:12:46.012 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:12:46.012 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:12:46.012 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:12:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:12:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:12:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:12:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:12:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:12:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:12:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:12:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:12:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:12:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:12:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:12:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:12:46.014 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:12:46.014 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:12:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:12:46.014 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:12:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:12:46.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.012+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f918806ecc0 msgr2=0x7f9188071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:46.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.012+0000 7f91a58e7700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f918806ecc0 0x7f9188071180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f91a00fcea0 tx=0x7f919401a040 comp rx=0 tx=0).stop 2024-04-16T03:12:46.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.012+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0100dc0 msgr2=0x7f91a019c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:12:46.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.012+0000 7f91a58e7700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0100dc0 0x7f91a019c560 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f919000c8a0 tx=0x7f919000cc60 comp rx=0 tx=0).stop 2024-04-16T03:12:46.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.012+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 shutdown_connections 2024-04-16T03:12:46.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.012+0000 7f91a58e7700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f918806ecc0 0x7f9188071180 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:46.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.012+0000 7f91a58e7700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91a0100dc0 0x7f91a019c560 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:46.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.012+0000 7f91a58e7700 1 --2- 172.21.15.151:0/1593034950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a0101770 0x7f91a019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:12:46.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.012+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 >> 172.21.15.151:0/1593034950 conn(0x7f91a00fc650 msgr2=0x7f91a00fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:12:46.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.013+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 shutdown_connections 2024-04-16T03:12:46.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:12:46.013+0000 7f91a58e7700 1 -- 172.21.15.151:0/1593034950 wait complete. 2024-04-16T03:12:46.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:45 smithi151 ceph-mon[112233]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:46.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:45 smithi151 ceph-mon[112233]: from='client.35864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:46.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:45 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3295901105' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:12:46.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:45 smithi153 ceph-mon[87858]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:46.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:45 smithi153 ceph-mon[87858]: from='client.35864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:46.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:45 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3295901105' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:12:47.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:46 smithi151 ceph-mon[112233]: pgmap v1025: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:47.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:46 smithi151 ceph-mon[112233]: from='client.35876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:47.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:46 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1593034950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:12:47.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:46 smithi153 ceph-mon[87858]: pgmap v1025: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:47.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:46 smithi153 ceph-mon[87858]: from='client.35876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:12:47.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:46 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1593034950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:12:49.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:48 smithi151 ceph-mon[112233]: pgmap v1026: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:49.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:48 smithi153 ceph-mon[87858]: pgmap v1026: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:51.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:50 smithi151 ceph-mon[112233]: pgmap v1027: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:51.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:50 smithi153 ceph-mon[87858]: pgmap v1027: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:53.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:52 smithi151 ceph-mon[112233]: pgmap v1028: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:53.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:52 smithi153 ceph-mon[87858]: pgmap v1028: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:54.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:12:54.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:12:55.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:54 smithi151 ceph-mon[112233]: pgmap v1029: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:55.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:54 smithi153 ceph-mon[87858]: pgmap v1029: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:57.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:56 smithi151 ceph-mon[112233]: pgmap v1030: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:57.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:56 smithi153 ceph-mon[87858]: pgmap v1030: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:59.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:12:58 smithi151 ceph-mon[112233]: pgmap v1031: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:12:59.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:12:58 smithi153 ceph-mon[87858]: pgmap v1031: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:01.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:00 smithi151 ceph-mon[112233]: pgmap v1032: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:01.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:00 smithi153 ceph-mon[87858]: pgmap v1032: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:03.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:02 smithi151 ceph-mon[112233]: pgmap v1033: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:03.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:02 smithi153 ceph-mon[87858]: pgmap v1033: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:05.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:04 smithi151 ceph-mon[112233]: pgmap v1034: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:05.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:04 smithi153 ceph-mon[87858]: pgmap v1034: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:07.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:06 smithi151 ceph-mon[112233]: pgmap v1035: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:07.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:06 smithi153 ceph-mon[87858]: pgmap v1035: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:08 smithi151 ceph-mon[112233]: pgmap v1036: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:13:09.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:08 smithi153 ceph-mon[87858]: pgmap v1036: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:09.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:13:11.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:10 smithi151 ceph-mon[112233]: pgmap v1037: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:11.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:10 smithi153 ceph-mon[87858]: pgmap v1037: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:13.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:12 smithi153 ceph-mon[87858]: pgmap v1038: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:13.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:12 smithi151 ceph-mon[112233]: pgmap v1038: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:15.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:14 smithi151 ceph-mon[112233]: pgmap v1039: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:15.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:14 smithi153 ceph-mon[87858]: pgmap v1039: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:16.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.132+0000 7f25513ee700 1 -- 172.21.15.151:0/1572619152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c101d30 msgr2=0x7f254c102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.132+0000 7f25513ee700 1 --2- 172.21.15.151:0/1572619152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c101d30 0x7f254c102110 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f253c009910 tx=0x7f253c009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:16.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.132+0000 7f25513ee700 1 -- 172.21.15.151:0/1572619152 shutdown_connections 2024-04-16T03:13:16.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.132+0000 7f25513ee700 1 --2- 172.21.15.151:0/1572619152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f254c102650 0x7f254c10ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.132+0000 7f25513ee700 1 --2- 172.21.15.151:0/1572619152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c101d30 0x7f254c102110 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.132+0000 7f25513ee700 1 -- 172.21.15.151:0/1572619152 >> 172.21.15.151:0/1572619152 conn(0x7f254c0fb560 msgr2=0x7f254c0fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:16.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.132+0000 7f25513ee700 1 -- 172.21.15.151:0/1572619152 shutdown_connections 2024-04-16T03:13:16.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.132+0000 7f25513ee700 1 -- 172.21.15.151:0/1572619152 wait complete. 2024-04-16T03:13:16.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.132+0000 7f25513ee700 1 Processor -- start 2024-04-16T03:13:16.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f25513ee700 1 -- start start 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f25513ee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f254c101d30 0x7f254c19c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f25513ee700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c102650 0x7f254c19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f25513ee700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c19d090 con 0x7f254c102650 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f25513ee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c1965d0 con 0x7f254c101d30 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f254a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c102650 0x7f254c19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f254a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c102650 0x7f254c19ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53524/0 (socket says 172.21.15.151:53524) 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f254a7fc700 1 -- 172.21.15.151:0/984761751 learned_addr learned my addr 172.21.15.151:0/984761751 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f254affd700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f254c101d30 0x7f254c19c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f254a7fc700 1 -- 172.21.15.151:0/984761751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f254c101d30 msgr2=0x7f254c19c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f254a7fc700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f254c101d30 0x7f254c19c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.133+0000 7f254a7fc700 1 -- 172.21.15.151:0/984761751 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c0095f0 con 0x7f254c102650 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.134+0000 7f254a7fc700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c102650 0x7f254c19ca90 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f254000c8f0 tx=0x7f254000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.134+0000 7f2533fff700 1 -- 172.21.15.151:0/984761751 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25400041f0 con 0x7f254c102650 2024-04-16T03:13:16.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.134+0000 7f2533fff700 1 -- 172.21.15.151:0/984761751 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2540007dc0 con 0x7f254c102650 2024-04-16T03:13:16.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.134+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c1968b0 con 0x7f254c102650 2024-04-16T03:13:16.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.134+0000 7f2533fff700 1 -- 172.21.15.151:0/984761751 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2540003ea0 con 0x7f254c102650 2024-04-16T03:13:16.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.134+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c196e00 con 0x7f254c102650 2024-04-16T03:13:16.137 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.135+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254c108240 con 0x7f254c102650 2024-04-16T03:13:16.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.135+0000 7f2533fff700 1 -- 172.21.15.151:0/984761751 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2540004350 con 0x7f254c102650 2024-04-16T03:13:16.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.136+0000 7f2533fff700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f253406ecc0 0x7f2534071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:16.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.136+0000 7f2533fff700 1 -- 172.21.15.151:0/984761751 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f254008eae0 con 0x7f254c102650 2024-04-16T03:13:16.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.136+0000 7f254affd700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f253406ecc0 0x7f2534071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:16.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.136+0000 7f254affd700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f253406ecc0 0x7f2534071180 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f253c006010 tx=0x7f253c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:16.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.141+0000 7f2533fff700 1 -- 172.21.15.151:0/984761751 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f254005b440 con 0x7f254c102650 2024-04-16T03:13:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.359+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f254c1970d0 con 0x7f253406ecc0 2024-04-16T03:13:16.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.360+0000 7f2533fff700 1 -- 172.21.15.151:0/984761751 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f254c1970d0 con 0x7f253406ecc0 2024-04-16T03:13:16.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.362+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f253406ecc0 msgr2=0x7f2534071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.363 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.362+0000 7f25513ee700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f253406ecc0 0x7f2534071180 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f253c006010 tx=0x7f253c01a040 comp rx=0 tx=0).stop 2024-04-16T03:13:16.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.362+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c102650 msgr2=0x7f254c19ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.362+0000 7f25513ee700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c102650 0x7f254c19ca90 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f254000c8f0 tx=0x7f254000ccb0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.362+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 shutdown_connections 2024-04-16T03:13:16.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.362+0000 7f25513ee700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f253406ecc0 0x7f2534071180 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.362+0000 7f25513ee700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f254c101d30 0x7f254c19c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.362+0000 7f25513ee700 1 --2- 172.21.15.151:0/984761751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f254c102650 0x7f254c19ca90 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.362+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 >> 172.21.15.151:0/984761751 conn(0x7f254c0fb560 msgr2=0x7f254c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:16.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.363+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 shutdown_connections 2024-04-16T03:13:16.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.363+0000 7f25513ee700 1 -- 172.21.15.151:0/984761751 wait complete. 2024-04-16T03:13:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:13:16.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.470+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/2149580950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f80ffbd0 msgr2=0x7fa3f8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.470+0000 7fa3fc9c1700 1 --2- 172.21.15.151:0/2149580950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f80ffbd0 0x7fa3f8100050 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009910 tx=0x7fa3e8009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:16.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.470+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/2149580950 shutdown_connections 2024-04-16T03:13:16.471 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.470+0000 7fa3fc9c1700 1 --2- 172.21.15.151:0/2149580950 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f80ffbd0 0x7fa3f8100050 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.470+0000 7fa3fc9c1700 1 --2- 172.21.15.151:0/2149580950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f8105a40 0x7fa3f8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.470+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/2149580950 >> 172.21.15.151:0/2149580950 conn(0x7fa3f80755a0 msgr2=0x7fa3f80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:16.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.470+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/2149580950 shutdown_connections 2024-04-16T03:13:16.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.470+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/2149580950 wait complete. 2024-04-16T03:13:16.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.471+0000 7fa3fc9c1700 1 Processor -- start 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.471+0000 7fa3fc9c1700 1 -- start start 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.471+0000 7fa3fc9c1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f80ffbd0 0x7fa3f819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.471+0000 7fa3fc9c1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f8105a40 0x7fa3f819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.471+0000 7fa3fc9c1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f819d0d0 con 0x7fa3f8105a40 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.471+0000 7fa3fc9c1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f8196580 con 0x7fa3f80ffbd0 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.471+0000 7fa3f5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f8105a40 0x7fa3f819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.471+0000 7fa3f5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f8105a40 0x7fa3f819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58156/0 (socket says 172.21.15.151:58156) 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.471+0000 7fa3f5d9b700 1 -- 172.21.15.151:0/42559606 learned_addr learned my addr 172.21.15.151:0/42559606 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3f659c700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f80ffbd0 0x7fa3f819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3f5d9b700 1 -- 172.21.15.151:0/42559606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f80ffbd0 msgr2=0x7fa3f819c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3f5d9b700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f80ffbd0 0x7fa3f819c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3f5d9b700 1 -- 172.21.15.151:0/42559606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e80095f0 con 0x7fa3f8105a40 2024-04-16T03:13:16.474 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3f5d9b700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f8105a40 0x7fa3f819ca40 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8006010 tx=0x7fa3e800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:16.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3e77fe700 1 -- 172.21.15.151:0/42559606 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3e801d070 con 0x7fa3f8105a40 2024-04-16T03:13:16.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f8196800 con 0x7fa3f8105a40 2024-04-16T03:13:16.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3e77fe700 1 -- 172.21.15.151:0/42559606 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3e800f460 con 0x7fa3f8105a40 2024-04-16T03:13:16.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3e77fe700 1 -- 172.21.15.151:0/42559606 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3e8017630 con 0x7fa3f8105a40 2024-04-16T03:13:16.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.472+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f8196cf0 con 0x7fa3f8105a40 2024-04-16T03:13:16.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.473+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f8108fe0 con 0x7fa3f8105a40 2024-04-16T03:13:16.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.474+0000 7fa3e77fe700 1 -- 172.21.15.151:0/42559606 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa3e800f5d0 con 0x7fa3f8105a40 2024-04-16T03:13:16.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.475+0000 7fa3e77fe700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3e006ecc0 0x7fa3e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:16.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.475+0000 7fa3f659c700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3e006ecc0 0x7fa3e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:16.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.475+0000 7fa3e77fe700 1 -- 172.21.15.151:0/42559606 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa3e80905d0 con 0x7fa3f8105a40 2024-04-16T03:13:16.476 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.475+0000 7fa3f659c700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3e006ecc0 0x7fa3e0071180 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec005fd0 tx=0x7fa3ec005f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:16.480 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.479+0000 7fa3e77fe700 1 -- 172.21.15.151:0/42559606 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa3e805ce90 con 0x7fa3f8105a40 2024-04-16T03:13:16.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.697+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3f8197000 con 0x7fa3e006ecc0 2024-04-16T03:13:16.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.698+0000 7fa3e77fe700 1 -- 172.21.15.151:0/42559606 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa3f8197000 con 0x7fa3e006ecc0 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3e006ecc0 msgr2=0x7fa3e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3e006ecc0 0x7fa3e0071180 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec005fd0 tx=0x7fa3ec005f60 comp rx=0 tx=0).stop 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f8105a40 msgr2=0x7fa3f819ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f8105a40 0x7fa3f819ca40 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8006010 tx=0x7fa3e800be20 comp rx=0 tx=0).stop 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 shutdown_connections 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa3e006ecc0 0x7fa3e0071180 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f80ffbd0 0x7fa3f819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 --2- 172.21.15.151:0/42559606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f8105a40 0x7fa3f819ca40 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 >> 172.21.15.151:0/42559606 conn(0x7fa3f80755a0 msgr2=0x7fa3f8068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 shutdown_connections 2024-04-16T03:13:16.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.700+0000 7fa3fc9c1700 1 -- 172.21.15.151:0/42559606 wait complete. 2024-04-16T03:13:16.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.806+0000 7f648376a700 1 -- 172.21.15.151:0/4061406961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c100dc0 msgr2=0x7f647c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.806+0000 7f648376a700 1 --2- 172.21.15.151:0/4061406961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c100dc0 0x7f647c1011a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f6478009910 tx=0x7f6478009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:16.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.806+0000 7f648376a700 1 -- 172.21.15.151:0/4061406961 shutdown_connections 2024-04-16T03:13:16.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.806+0000 7f648376a700 1 --2- 172.21.15.151:0/4061406961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c101770 0x7f647c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.806+0000 7f648376a700 1 --2- 172.21.15.151:0/4061406961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c100dc0 0x7f647c1011a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.806+0000 7f648376a700 1 -- 172.21.15.151:0/4061406961 >> 172.21.15.151:0/4061406961 conn(0x7f647c0fc650 msgr2=0x7f647c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:16.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.806+0000 7f648376a700 1 -- 172.21.15.151:0/4061406961 shutdown_connections 2024-04-16T03:13:16.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.806+0000 7f648376a700 1 -- 172.21.15.151:0/4061406961 wait complete. 2024-04-16T03:13:16.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f648376a700 1 Processor -- start 2024-04-16T03:13:16.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f648376a700 1 -- start start 2024-04-16T03:13:16.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f648376a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c100dc0 0x7f647c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:16.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f648376a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c101770 0x7f647c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:16.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f648376a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c19d110 con 0x7f647c101770 2024-04-16T03:13:16.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f648376a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c1965c0 con 0x7f647c100dc0 2024-04-16T03:13:16.809 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f6480d05700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c101770 0x7f647c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f6480d05700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c101770 0x7f647c19ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58180/0 (socket says 172.21.15.151:58180) 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f6480d05700 1 -- 172.21.15.151:0/2666896801 learned_addr learned my addr 172.21.15.151:0/2666896801 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f6481506700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c100dc0 0x7f647c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f6480d05700 1 -- 172.21.15.151:0/2666896801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c100dc0 msgr2=0x7f647c19c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f6480d05700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c100dc0 0x7f647c19c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f6480d05700 1 -- 172.21.15.151:0/2666896801 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64780095f0 con 0x7f647c101770 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.807+0000 7f6480d05700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c101770 0x7f647c19ca80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f646c00d900 tx=0x7f646c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.808+0000 7f64727fc700 1 -- 172.21.15.151:0/2666896801 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f646c009900 con 0x7f647c101770 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.808+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c1968a0 con 0x7f647c101770 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.808+0000 7f64727fc700 1 -- 172.21.15.151:0/2666896801 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f646c010460 con 0x7f647c101770 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.808+0000 7f64727fc700 1 -- 172.21.15.151:0/2666896801 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f646c00f5f0 con 0x7f647c101770 2024-04-16T03:13:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.808+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c196df0 con 0x7f647c101770 2024-04-16T03:13:16.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.809+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c109160 con 0x7f647c101770 2024-04-16T03:13:16.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.809+0000 7f64727fc700 1 -- 172.21.15.151:0/2666896801 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f646c0105d0 con 0x7f647c101770 2024-04-16T03:13:16.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.810+0000 7f64727fc700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f646806ecc0 0x7f6468071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:16.811 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.810+0000 7f64727fc700 1 -- 172.21.15.151:0/2666896801 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f646c08ea30 con 0x7f647c101770 2024-04-16T03:13:16.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.810+0000 7f6481506700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f646806ecc0 0x7f6468071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:16.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.810+0000 7f6481506700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f646806ecc0 0x7f6468071180 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6478006010 tx=0x7f647801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:16.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:16.814+0000 7f64727fc700 1 -- 172.21.15.151:0/2666896801 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f646c05b310 con 0x7f647c101770 2024-04-16T03:13:17.024 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.023+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f647c197090 con 0x7f646806ecc0 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.034+0000 7f64727fc700 1 -- 172.21.15.151:0/2666896801 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f647c197090 con 0x7f646806ecc0 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (34m) 12m ago 46m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (46m) 12m ago 46m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (45m) 12m ago 45m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (32m) 12m ago 46m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (32m) 12m ago 45m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (34m) 12m ago 45m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (28m) 12m ago 40m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:13:17.035 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (28m) 12m ago 40m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (12m) 12m ago 40m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (28m) 12m ago 40m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (36m) 12m ago 47m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (35m) 12m ago 45m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (33m) 12m ago 47m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (33m) 12m ago 45m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (35m) 12m ago 46m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (35m) 12m ago 45m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (31m) 12m ago 44m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (31m) 12m ago 43m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (30m) 12m ago 43m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (30m) 12m ago 42m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (29m) 12m ago 42m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (29m) 12m ago 42m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:13:17.036 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (35m) 12m ago 45m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:13:17.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f646806ecc0 msgr2=0x7f6468071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f646806ecc0 0x7f6468071180 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6478006010 tx=0x7f647801a040 comp rx=0 tx=0).stop 2024-04-16T03:13:17.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c101770 msgr2=0x7f647c19ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c101770 0x7f647c19ca80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f646c00d900 tx=0x7f646c00dc10 comp rx=0 tx=0).stop 2024-04-16T03:13:17.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 shutdown_connections 2024-04-16T03:13:17.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f646806ecc0 0x7f6468071180 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c100dc0 0x7f647c19c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 --2- 172.21.15.151:0/2666896801 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c101770 0x7f647c19ca80 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 >> 172.21.15.151:0/2666896801 conn(0x7f647c0fc650 msgr2=0x7f647c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:17.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 shutdown_connections 2024-04-16T03:13:17.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.036+0000 7f648376a700 1 -- 172.21.15.151:0/2666896801 wait complete. 2024-04-16T03:13:17.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.139+0000 7f398ddd8700 1 -- 172.21.15.151:0/104927857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 msgr2=0x7f39881011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.139+0000 7f398ddd8700 1 --2- 172.21.15.151:0/104927857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 0x7f39881011a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3978009910 tx=0x7f3978009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:17.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.139+0000 7f398ddd8700 1 -- 172.21.15.151:0/104927857 shutdown_connections 2024-04-16T03:13:17.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.139+0000 7f398ddd8700 1 --2- 172.21.15.151:0/104927857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3988101770 0x7f39881057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.139+0000 7f398ddd8700 1 --2- 172.21.15.151:0/104927857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 0x7f39881011a0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.140 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.139+0000 7f398ddd8700 1 -- 172.21.15.151:0/104927857 >> 172.21.15.151:0/104927857 conn(0x7f39880fc650 msgr2=0x7f39880fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:17.140 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:17 smithi151 ceph-mon[112233]: pgmap v1040: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:17.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.139+0000 7f398ddd8700 1 -- 172.21.15.151:0/104927857 shutdown_connections 2024-04-16T03:13:17.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.139+0000 7f398ddd8700 1 -- 172.21.15.151:0/104927857 wait complete. 2024-04-16T03:13:17.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.139+0000 7f398ddd8700 1 Processor -- start 2024-04-16T03:13:17.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f398ddd8700 1 -- start start 2024-04-16T03:13:17.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f398ddd8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 0x7f3988073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:17.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f398ddd8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3988101770 0x7f39880735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:17.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f398ddd8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3988073b10 con 0x7f3988100dc0 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f398ddd8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3988073c50 con 0x7f3988101770 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f39877fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 0x7f3988073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f39877fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 0x7f3988073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58192/0 (socket says 172.21.15.151:58192) 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f39877fe700 1 -- 172.21.15.151:0/3044472025 learned_addr learned my addr 172.21.15.151:0/3044472025 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f3986ffd700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3988101770 0x7f39880735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f39877fe700 1 -- 172.21.15.151:0/3044472025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3988101770 msgr2=0x7f39880735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f39877fe700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3988101770 0x7f39880735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.140+0000 7f39877fe700 1 -- 172.21.15.151:0/3044472025 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39780095f0 con 0x7f3988100dc0 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.141+0000 7f39877fe700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 0x7f3988073090 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3978006010 tx=0x7f397800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.141+0000 7f3984ff9700 1 -- 172.21.15.151:0/3044472025 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f397801d070 con 0x7f3988100dc0 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.141+0000 7f3984ff9700 1 -- 172.21.15.151:0/3044472025 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3978004fb0 con 0x7f3988100dc0 2024-04-16T03:13:17.143 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.141+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3988073ed0 con 0x7f3988100dc0 2024-04-16T03:13:17.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.141+0000 7f3984ff9700 1 -- 172.21.15.151:0/3044472025 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3978022ea0 con 0x7f3988100dc0 2024-04-16T03:13:17.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.141+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39881a2c20 con 0x7f3988100dc0 2024-04-16T03:13:17.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.142+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3988109160 con 0x7f3988100dc0 2024-04-16T03:13:17.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.143+0000 7f3984ff9700 1 -- 172.21.15.151:0/3044472025 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3978022470 con 0x7f3988100dc0 2024-04-16T03:13:17.144 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.143+0000 7f3984ff9700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f397406ecc0 0x7f3974071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:17.145 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.143+0000 7f3984ff9700 1 -- 172.21.15.151:0/3044472025 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3978090350 con 0x7f3988100dc0 2024-04-16T03:13:17.145 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.143+0000 7f3986ffd700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f397406ecc0 0x7f3974071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:17.145 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.144+0000 7f3986ffd700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f397406ecc0 0x7f3974071180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3970005ea0 tx=0x7f3970005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:17.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.150+0000 7f3984ff9700 1 -- 172.21.15.151:0/3044472025 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3978059230 con 0x7f3988100dc0 2024-04-16T03:13:17.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:17 smithi153 ceph-mon[87858]: pgmap v1040: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:17.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.428+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f398804ea90 con 0x7f3988100dc0 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.429+0000 7f3984ff9700 1 -- 172.21.15.151:0/3044472025 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f397805c8f0 con 0x7f3988100dc0 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:13:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:13:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:13:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:13:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:13:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:13:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:13:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:13:17.431 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:13:17.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f397406ecc0 msgr2=0x7f3974071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f397406ecc0 0x7f3974071180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3970005ea0 tx=0x7f3970005e30 comp rx=0 tx=0).stop 2024-04-16T03:13:17.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 msgr2=0x7f3988073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 0x7f3988073090 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3978006010 tx=0x7f397800be20 comp rx=0 tx=0).stop 2024-04-16T03:13:17.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 shutdown_connections 2024-04-16T03:13:17.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f397406ecc0 0x7f3974071180 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3988100dc0 0x7f3988073090 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 --2- 172.21.15.151:0/3044472025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3988101770 0x7f39880735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 >> 172.21.15.151:0/3044472025 conn(0x7f39880fc650 msgr2=0x7f39880fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:17.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 shutdown_connections 2024-04-16T03:13:17.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.431+0000 7f398ddd8700 1 -- 172.21.15.151:0/3044472025 wait complete. 2024-04-16T03:13:17.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.537+0000 7f0835988700 1 -- 172.21.15.151:0/3805636790 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 msgr2=0x7f0830105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.537+0000 7f0835988700 1 --2- 172.21.15.151:0/3805636790 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 0x7f0830105810 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f0824009910 tx=0x7f0824009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:17.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.537+0000 7f0835988700 1 -- 172.21.15.151:0/3805636790 shutdown_connections 2024-04-16T03:13:17.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.537+0000 7f0835988700 1 --2- 172.21.15.151:0/3805636790 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 0x7f0830105810 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.537+0000 7f0835988700 1 --2- 172.21.15.151:0/3805636790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0830100e10 0x7f08301011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.537+0000 7f0835988700 1 -- 172.21.15.151:0/3805636790 >> 172.21.15.151:0/3805636790 conn(0x7f08300fc6c0 msgr2=0x7f08300feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:17.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.537+0000 7f0835988700 1 -- 172.21.15.151:0/3805636790 shutdown_connections 2024-04-16T03:13:17.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.537+0000 7f0835988700 1 -- 172.21.15.151:0/3805636790 wait complete. 2024-04-16T03:13:17.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.538+0000 7f0835988700 1 Processor -- start 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.538+0000 7f0835988700 1 -- start start 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.538+0000 7f0835988700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0830100e10 0x7f083019c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.538+0000 7f0835988700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 0x7f083019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.538+0000 7f0835988700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083019d110 con 0x7f08301017c0 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.538+0000 7f0835988700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08301965c0 con 0x7f0830100e10 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.538+0000 7f082e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 0x7f083019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.538+0000 7f082e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 0x7f083019ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58198/0 (socket says 172.21.15.151:58198) 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.538+0000 7f082e7fc700 1 -- 172.21.15.151:0/1140619129 learned_addr learned my addr 172.21.15.151:0/1140619129 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f082effd700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0830100e10 0x7f083019c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f082e7fc700 1 -- 172.21.15.151:0/1140619129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0830100e10 msgr2=0x7f083019c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f082e7fc700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0830100e10 0x7f083019c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f082e7fc700 1 -- 172.21.15.151:0/1140619129 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08240095f0 con 0x7f08301017c0 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f082e7fc700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 0x7f083019ca80 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f08240092e0 tx=0x7f082400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f0834986700 1 -- 172.21.15.151:0/1140619129 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f082401d070 con 0x7f08301017c0 2024-04-16T03:13:17.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0830196840 con 0x7f08301017c0 2024-04-16T03:13:17.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f0834986700 1 -- 172.21.15.151:0/1140619129 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f082400f460 con 0x7f08301017c0 2024-04-16T03:13:17.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f0834986700 1 -- 172.21.15.151:0/1140619129 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08240175f0 con 0x7f08301017c0 2024-04-16T03:13:17.542 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.539+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0830196d30 con 0x7f08301017c0 2024-04-16T03:13:17.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.540+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0830109120 con 0x7f08301017c0 2024-04-16T03:13:17.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.541+0000 7f0834986700 1 -- 172.21.15.151:0/1140619129 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f082400f5d0 con 0x7f08301017c0 2024-04-16T03:13:17.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.541+0000 7f0834986700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f081806ecc0 0x7f0818071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:17.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.541+0000 7f0834986700 1 -- 172.21.15.151:0/1140619129 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0824090390 con 0x7f08301017c0 2024-04-16T03:13:17.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.541+0000 7f082effd700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f081806ecc0 0x7f0818071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:17.543 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.542+0000 7f082effd700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f081806ecc0 0x7f0818071180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0820005950 tx=0x7f0820005cf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:17.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.546+0000 7f0834986700 1 -- 172.21.15.151:0/1140619129 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f082405cc70 con 0x7f08301017c0 2024-04-16T03:13:17.790 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.788+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f083004ea90 con 0x7f08301017c0 2024-04-16T03:13:17.790 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.789+0000 7f0834986700 1 -- 172.21.15.151:0/1140619129 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f082405c800 con 0x7f08301017c0 2024-04-16T03:13:17.790 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:13:17.790 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:13:17.790 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:13:17.790 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:13:17.790 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:13:17.790 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:13:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:13:17.792 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:13:17.793 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:13:17.793 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:13:17.793 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:13:17.793 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:13:17.793 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:13:17.793 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:13:17.793 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:13:17.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f081806ecc0 msgr2=0x7f0818071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f081806ecc0 0x7f0818071180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0820005950 tx=0x7f0820005cf0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 msgr2=0x7f083019ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 0x7f083019ca80 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f08240092e0 tx=0x7f082400be20 comp rx=0 tx=0).stop 2024-04-16T03:13:17.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 shutdown_connections 2024-04-16T03:13:17.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f081806ecc0 0x7f0818071180 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0830100e10 0x7f083019c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 --2- 172.21.15.151:0/1140619129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08301017c0 0x7f083019ca80 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 >> 172.21.15.151:0/1140619129 conn(0x7f08300fc6c0 msgr2=0x7f0830105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:17.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 shutdown_connections 2024-04-16T03:13:17.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.791+0000 7f0835988700 1 -- 172.21.15.151:0/1140619129 wait complete. 2024-04-16T03:13:17.795 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.896+0000 7f83719ea700 1 -- 172.21.15.151:0/2957937425 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 msgr2=0x7f836c068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.896+0000 7f83719ea700 1 --2- 172.21.15.151:0/2957937425 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 0x7f836c068a50 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f835c009910 tx=0x7f835c009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.896+0000 7f83719ea700 1 -- 172.21.15.151:0/2957937425 shutdown_connections 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.896+0000 7f83719ea700 1 --2- 172.21.15.151:0/2957937425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c069020 0x7f836c105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.896+0000 7f83719ea700 1 --2- 172.21.15.151:0/2957937425 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 0x7f836c068a50 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.896+0000 7f83719ea700 1 -- 172.21.15.151:0/2957937425 >> 172.21.15.151:0/2957937425 conn(0x7f836c0755a0 msgr2=0x7f836c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.896+0000 7f83719ea700 1 -- 172.21.15.151:0/2957937425 shutdown_connections 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.896+0000 7f83719ea700 1 -- 172.21.15.151:0/2957937425 wait complete. 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f83719ea700 1 Processor -- start 2024-04-16T03:13:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f83719ea700 1 -- start start 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f83719ea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 0x7f836c0ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f83719ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c069020 0x7f836c1001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f83719ea700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c100720 con 0x7f836c068670 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f83719ea700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c100860 con 0x7f836c069020 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f836affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 0x7f836c0ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f836affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 0x7f836c0ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58204/0 (socket says 172.21.15.151:58204) 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f836affd700 1 -- 172.21.15.151:0/1537151842 learned_addr learned my addr 172.21.15.151:0/1537151842 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f8363fff700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c069020 0x7f836c1001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.897+0000 7f836affd700 1 -- 172.21.15.151:0/1537151842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c069020 msgr2=0x7f836c1001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.898+0000 7f836affd700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c069020 0x7f836c1001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:17.900 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.898+0000 7f836affd700 1 -- 172.21.15.151:0/1537151842 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c0095f0 con 0x7f836c068670 2024-04-16T03:13:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.898+0000 7f836affd700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 0x7f836c0ffca0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f835c006010 tx=0x7f835c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.898+0000 7f83709e8700 1 -- 172.21.15.151:0/1537151842 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f835c01d070 con 0x7f836c068670 2024-04-16T03:13:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.898+0000 7f83709e8700 1 -- 172.21.15.151:0/1537151842 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f835c00f460 con 0x7f836c068670 2024-04-16T03:13:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.898+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836c100ae0 con 0x7f836c068670 2024-04-16T03:13:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.898+0000 7f83709e8700 1 -- 172.21.15.151:0/1537151842 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f835c017610 con 0x7f836c068670 2024-04-16T03:13:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.898+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836c1a2df0 con 0x7f836c068670 2024-04-16T03:13:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.899+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f836c109130 con 0x7f836c068670 2024-04-16T03:13:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.900+0000 7f83709e8700 1 -- 172.21.15.151:0/1537151842 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f835c00f5d0 con 0x7f836c068670 2024-04-16T03:13:17.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.900+0000 7f83709e8700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f835806ecc0 0x7f8358071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:17.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.900+0000 7f83709e8700 1 -- 172.21.15.151:0/1537151842 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f835c0903e0 con 0x7f836c068670 2024-04-16T03:13:17.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.900+0000 7f8363fff700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f835806ecc0 0x7f8358071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:17.902 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.900+0000 7f8363fff700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f835806ecc0 0x7f8358071180 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f836c101490 tx=0x7f8354009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:17.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:17.905+0000 7f83709e8700 1 -- 172.21.15.151:0/1537151842 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f835c059190 con 0x7f836c068670 2024-04-16T03:13:18.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.129+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f836c0611d0 con 0x7f835806ecc0 2024-04-16T03:13:18.130 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:18 smithi151 ceph-mon[112233]: from='client.35884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:18.130 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:18 smithi151 ceph-mon[112233]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:18.130 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:18 smithi151 ceph-mon[112233]: from='client.35892 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:18.130 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:18 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3044472025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:13:18.130 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:18 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1140619129' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:13:18.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.130+0000 7f83709e8700 1 -- 172.21.15.151:0/1537151842 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f836c0611d0 con 0x7f835806ecc0 2024-04-16T03:13:18.131 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:13:18.131 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:13:18.131 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:13:18.131 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:13:18.131 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:13:18.132 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:13:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f835806ecc0 msgr2=0x7f8358071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f835806ecc0 0x7f8358071180 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f836c101490 tx=0x7f8354009190 comp rx=0 tx=0).stop 2024-04-16T03:13:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 msgr2=0x7f836c0ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 0x7f836c0ffca0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f835c006010 tx=0x7f835c00bd70 comp rx=0 tx=0).stop 2024-04-16T03:13:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 shutdown_connections 2024-04-16T03:13:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f835806ecc0 0x7f8358071180 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f836c068670 0x7f836c0ffca0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 --2- 172.21.15.151:0/1537151842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f836c069020 0x7f836c1001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 >> 172.21.15.151:0/1537151842 conn(0x7f836c0755a0 msgr2=0x7f836c0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:18.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 shutdown_connections 2024-04-16T03:13:18.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.133+0000 7f83719ea700 1 -- 172.21.15.151:0/1537151842 wait complete. 2024-04-16T03:13:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/709305786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94101780 msgr2=0x7f8a941057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 --2- 172.21.15.151:0/709305786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94101780 0x7f8a941057d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8a84009910 tx=0x7f8a84009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/709305786 shutdown_connections 2024-04-16T03:13:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 --2- 172.21.15.151:0/709305786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94101780 0x7f8a941057d0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 --2- 172.21.15.151:0/709305786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a94100dd0 0x7f8a941011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/709305786 >> 172.21.15.151:0/709305786 conn(0x7f8a940fc640 msgr2=0x7f8a940fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/709305786 shutdown_connections 2024-04-16T03:13:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/709305786 wait complete. 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 Processor -- start 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.238+0000 7f8a9b8f1700 1 -- start start 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9b8f1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94100dd0 0x7f8a9419c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9b8f1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a94101780 0x7f8a9419cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9b8f1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9419d160 con 0x7f8a94100dd0 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9b8f1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a94196610 con 0x7f8a94101780 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9968d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94100dd0 0x7f8a9419c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9968d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94100dd0 0x7f8a9419c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58234/0 (socket says 172.21.15.151:58234) 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9968d700 1 -- 172.21.15.151:0/2501568030 learned_addr learned my addr 172.21.15.151:0/2501568030 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a98e8c700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a94101780 0x7f8a9419cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9968d700 1 -- 172.21.15.151:0/2501568030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a94101780 msgr2=0x7f8a9419cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9968d700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a94101780 0x7f8a9419cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9968d700 1 -- 172.21.15.151:0/2501568030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a840095f0 con 0x7f8a94100dd0 2024-04-16T03:13:18.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9968d700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94100dd0 0x7f8a9419c590 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000d8d0 tx=0x7f8a9000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:18.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a8a7fc700 1 -- 172.21.15.151:0/2501568030 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a900098a0 con 0x7f8a94100dd0 2024-04-16T03:13:18.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a941968f0 con 0x7f8a94100dd0 2024-04-16T03:13:18.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a8a7fc700 1 -- 172.21.15.151:0/2501568030 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a90010460 con 0x7f8a94100dd0 2024-04-16T03:13:18.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a8a7fc700 1 -- 172.21.15.151:0/2501568030 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a9000f5f0 con 0x7f8a94100dd0 2024-04-16T03:13:18.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.239+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a94196e40 con 0x7f8a94100dd0 2024-04-16T03:13:18.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.240+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a941091d0 con 0x7f8a94100dd0 2024-04-16T03:13:18.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.241+0000 7f8a8a7fc700 1 -- 172.21.15.151:0/2501568030 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8a90009a00 con 0x7f8a94100dd0 2024-04-16T03:13:18.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.241+0000 7f8a8a7fc700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a8006ecc0 0x7f8a80071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:18.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.242+0000 7f8a8a7fc700 1 -- 172.21.15.151:0/2501568030 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8a9008ed30 con 0x7f8a94100dd0 2024-04-16T03:13:18.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.242+0000 7f8a98e8c700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a8006ecc0 0x7f8a80071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:18.243 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.242+0000 7f8a98e8c700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a8006ecc0 0x7f8a80071180 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8a940fcd70 tx=0x7f8a8401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:18.247 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.246+0000 7f8a8a7fc700 1 -- 172.21.15.151:0/2501568030 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8a9005b610 con 0x7f8a94100dd0 2024-04-16T03:13:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:18 smithi153 ceph-mon[87858]: from='client.35884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:18 smithi153 ceph-mon[87858]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:18 smithi153 ceph-mon[87858]: from='client.35892 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:18 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3044472025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:13:18.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:18 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1140619129' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:13:18.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.534+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8a94197060 con 0x7f8a94100dd0 2024-04-16T03:13:18.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.534+0000 7f8a8a7fc700 1 -- 172.21.15.151:0/2501568030 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f8a9005b1a0 con 0x7f8a94100dd0 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:13:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:13:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:13:18.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a8006ecc0 msgr2=0x7f8a80071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:18.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a8006ecc0 0x7f8a80071180 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8a940fcd70 tx=0x7f8a8401a040 comp rx=0 tx=0).stop 2024-04-16T03:13:18.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94100dd0 msgr2=0x7f8a9419c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:18.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94100dd0 0x7f8a9419c590 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000d8d0 tx=0x7f8a9000dbe0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 shutdown_connections 2024-04-16T03:13:18.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8a8006ecc0 0x7f8a80071180 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a94100dd0 0x7f8a9419c590 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 --2- 172.21.15.151:0/2501568030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a94101780 0x7f8a9419cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:18.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 >> 172.21.15.151:0/2501568030 conn(0x7f8a940fc640 msgr2=0x7f8a940fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:18.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 shutdown_connections 2024-04-16T03:13:18.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:18.536+0000 7f8a9b8f1700 1 -- 172.21.15.151:0/2501568030 wait complete. 2024-04-16T03:13:19.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:19 smithi153 ceph-mon[87858]: pgmap v1041: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:19.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:19 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2501568030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:13:19.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:19 smithi151 ceph-mon[112233]: pgmap v1041: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:19.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:19 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2501568030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:13:20.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:20 smithi151 ceph-mon[112233]: from='client.35904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:20.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:20 smithi153 ceph-mon[87858]: from='client.35904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:21.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:21 smithi153 ceph-mon[87858]: pgmap v1042: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:21.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:21 smithi151 ceph-mon[112233]: pgmap v1042: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:23.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:23 smithi153 ceph-mon[87858]: pgmap v1043: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:23.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:23 smithi151 ceph-mon[112233]: pgmap v1043: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:13:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:13:25.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:25 smithi151 ceph-mon[112233]: pgmap v1044: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:25.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:25 smithi153 ceph-mon[87858]: pgmap v1044: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:27.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:27 smithi151 ceph-mon[112233]: pgmap v1045: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:27.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:27 smithi153 ceph-mon[87858]: pgmap v1045: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:29.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:29 smithi151 ceph-mon[112233]: pgmap v1046: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:29.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:29 smithi153 ceph-mon[87858]: pgmap v1046: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:31.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:31 smithi151 ceph-mon[112233]: pgmap v1047: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:31.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:31 smithi153 ceph-mon[87858]: pgmap v1047: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:33.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:33 smithi153 ceph-mon[87858]: pgmap v1048: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:33.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:33 smithi151 ceph-mon[112233]: pgmap v1048: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:35 smithi151 ceph-mon[112233]: pgmap v1049: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:35.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:35 smithi153 ceph-mon[87858]: pgmap v1049: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:37.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:37 smithi151 ceph-mon[112233]: pgmap v1050: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:37 smithi153 ceph-mon[87858]: pgmap v1050: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:39.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:39 smithi151 ceph-mon[112233]: pgmap v1051: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:39.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:13:39.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:39 smithi153 ceph-mon[87858]: pgmap v1051: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:39.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:13:41.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:41 smithi151 ceph-mon[112233]: pgmap v1052: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:41 smithi153 ceph-mon[87858]: pgmap v1052: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:43.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:43 smithi153 ceph-mon[87858]: pgmap v1053: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:43.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:43 smithi151 ceph-mon[112233]: pgmap v1053: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:45.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:45 smithi151 ceph-mon[112233]: pgmap v1054: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:45.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:45 smithi153 ceph-mon[87858]: pgmap v1054: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:47.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:47 smithi151 ceph-mon[112233]: pgmap v1055: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:47.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:47 smithi153 ceph-mon[87858]: pgmap v1055: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:48.658 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.654+0000 7f7150a28700 1 -- 172.21.15.151:0/2642693307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c100dc0 msgr2=0x7f714c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:48.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.654+0000 7f7150a28700 1 --2- 172.21.15.151:0/2642693307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c100dc0 0x7f714c1011a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f713c009910 tx=0x7f713c009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:48.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.654+0000 7f7150a28700 1 -- 172.21.15.151:0/2642693307 shutdown_connections 2024-04-16T03:13:48.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.654+0000 7f7150a28700 1 --2- 172.21.15.151:0/2642693307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f714c101770 0x7f714c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.654+0000 7f7150a28700 1 --2- 172.21.15.151:0/2642693307 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c100dc0 0x7f714c1011a0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.654+0000 7f7150a28700 1 -- 172.21.15.151:0/2642693307 >> 172.21.15.151:0/2642693307 conn(0x7f714c0fc650 msgr2=0x7f714c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:48.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.654+0000 7f7150a28700 1 -- 172.21.15.151:0/2642693307 shutdown_connections 2024-04-16T03:13:48.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.654+0000 7f7150a28700 1 -- 172.21.15.151:0/2642693307 wait complete. 2024-04-16T03:13:48.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.655+0000 7f7150a28700 1 Processor -- start 2024-04-16T03:13:48.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.655+0000 7f7150a28700 1 -- start start 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.655+0000 7f7150a28700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f714c100dc0 0x7f714c19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.655+0000 7f7150a28700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c101770 0x7f714c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.655+0000 7f7150a28700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c19d130 con 0x7f714c101770 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.655+0000 7f7150a28700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c1965e0 con 0x7f714c100dc0 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.655+0000 7f7149d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c101770 0x7f714c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.655+0000 7f7149d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c101770 0x7f714c19caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40394/0 (socket says 172.21.15.151:40394) 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.655+0000 7f7149d9b700 1 -- 172.21.15.151:0/401970744 learned_addr learned my addr 172.21.15.151:0/401970744 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f714a59c700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f714c100dc0 0x7f714c19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f7149d9b700 1 -- 172.21.15.151:0/401970744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f714c100dc0 msgr2=0x7f714c19c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f7149d9b700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f714c100dc0 0x7f714c19c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f7149d9b700 1 -- 172.21.15.151:0/401970744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c0095f0 con 0x7f714c101770 2024-04-16T03:13:48.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f7149d9b700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c101770 0x7f714c19caa0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f714000d8d0 tx=0x7f714000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:48.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f713b7fe700 1 -- 172.21.15.151:0/401970744 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7140004a00 con 0x7f714c101770 2024-04-16T03:13:48.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c1968c0 con 0x7f714c101770 2024-04-16T03:13:48.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f713b7fe700 1 -- 172.21.15.151:0/401970744 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71400054e0 con 0x7f714c101770 2024-04-16T03:13:48.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f713b7fe700 1 -- 172.21.15.151:0/401970744 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f714000fc20 con 0x7f714c101770 2024-04-16T03:13:48.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.656+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c196e10 con 0x7f714c101770 2024-04-16T03:13:48.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.658+0000 7f713b7fe700 1 -- 172.21.15.151:0/401970744 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7140005020 con 0x7f714c101770 2024-04-16T03:13:48.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.658+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c109160 con 0x7f714c101770 2024-04-16T03:13:48.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.658+0000 7f713b7fe700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f713406ecc0 0x7f7134071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:48.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.658+0000 7f713b7fe700 1 -- 172.21.15.151:0/401970744 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f714008e700 con 0x7f714c101770 2024-04-16T03:13:48.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.659+0000 7f714a59c700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f713406ecc0 0x7f7134071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:48.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.659+0000 7f714a59c700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f713406ecc0 0x7f7134071180 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f713c006010 tx=0x7f713c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:48.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.663+0000 7f713b7fe700 1 -- 172.21.15.151:0/401970744 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f714005afe0 con 0x7f714c101770 2024-04-16T03:13:48.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.880+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f714c1970b0 con 0x7f713406ecc0 2024-04-16T03:13:48.882 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.881+0000 7f713b7fe700 1 -- 172.21.15.151:0/401970744 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f714c1970b0 con 0x7f713406ecc0 2024-04-16T03:13:48.884 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f713406ecc0 msgr2=0x7f7134071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f713406ecc0 0x7f7134071180 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f713c006010 tx=0x7f713c01a040 comp rx=0 tx=0).stop 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c101770 msgr2=0x7f714c19caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c101770 0x7f714c19caa0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f714000d8d0 tx=0x7f714000dbe0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 shutdown_connections 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f713406ecc0 0x7f7134071180 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f714c100dc0 0x7f714c19c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 --2- 172.21.15.151:0/401970744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f714c101770 0x7f714c19caa0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 >> 172.21.15.151:0/401970744 conn(0x7f714c0fc650 msgr2=0x7f714c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 shutdown_connections 2024-04-16T03:13:48.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.883+0000 7f7150a28700 1 -- 172.21.15.151:0/401970744 wait complete. 2024-04-16T03:13:48.895 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:13:48.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.989+0000 7f8d55e96700 1 -- 172.21.15.151:0/2597191355 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50100dc0 msgr2=0x7f8d501011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:48.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.989+0000 7f8d55e96700 1 --2- 172.21.15.151:0/2597191355 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50100dc0 0x7f8d501011a0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009910 tx=0x7f8d40009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:48.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.989+0000 7f8d55e96700 1 -- 172.21.15.151:0/2597191355 shutdown_connections 2024-04-16T03:13:48.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.989+0000 7f8d55e96700 1 --2- 172.21.15.151:0/2597191355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d50101770 0x7f8d501057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.989+0000 7f8d55e96700 1 --2- 172.21.15.151:0/2597191355 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50100dc0 0x7f8d501011a0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.989+0000 7f8d55e96700 1 -- 172.21.15.151:0/2597191355 >> 172.21.15.151:0/2597191355 conn(0x7f8d500fc650 msgr2=0x7f8d500fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:48.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.989+0000 7f8d55e96700 1 -- 172.21.15.151:0/2597191355 shutdown_connections 2024-04-16T03:13:48.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.989+0000 7f8d55e96700 1 -- 172.21.15.151:0/2597191355 wait complete. 2024-04-16T03:13:48.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.989+0000 7f8d55e96700 1 Processor -- start 2024-04-16T03:13:48.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d55e96700 1 -- start start 2024-04-16T03:13:48.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d55e96700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d50100dc0 0x7f8d50073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:48.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d55e96700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50101770 0x7f8d500735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:48.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d55e96700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d50073b10 con 0x7f8d50101770 2024-04-16T03:13:48.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d55e96700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d50073c50 con 0x7f8d50100dc0 2024-04-16T03:13:48.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d4effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50101770 0x7f8d500735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:48.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d4effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50101770 0x7f8d500735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40424/0 (socket says 172.21.15.151:40424) 2024-04-16T03:13:48.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d4effd700 1 -- 172.21.15.151:0/2711949916 learned_addr learned my addr 172.21.15.151:0/2711949916 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:48.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d4f7fe700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d50100dc0 0x7f8d50073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:48.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d4effd700 1 -- 172.21.15.151:0/2711949916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d50100dc0 msgr2=0x7f8d50073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d4effd700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d50100dc0 0x7f8d50073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d4effd700 1 -- 172.21.15.151:0/2711949916 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400095f0 con 0x7f8d50101770 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d4f7fe700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d50100dc0 0x7f8d50073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.990+0000 7f8d4effd700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50101770 0x7f8d500735d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400c8f0 tx=0x7f8d4400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.991+0000 7f8d4cff9700 1 -- 172.21.15.151:0/2711949916 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d44007a50 con 0x7f8d50101770 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.991+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d501a2940 con 0x7f8d50101770 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.991+0000 7f8d4cff9700 1 -- 172.21.15.151:0/2711949916 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d44007bb0 con 0x7f8d50101770 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.991+0000 7f8d4cff9700 1 -- 172.21.15.151:0/2711949916 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d44018630 con 0x7f8d50101770 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.991+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d501a2d00 con 0x7f8d50101770 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.992+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d50109160 con 0x7f8d50101770 2024-04-16T03:13:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.992+0000 7f8d4cff9700 1 -- 172.21.15.151:0/2711949916 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8d44007d20 con 0x7f8d50101770 2024-04-16T03:13:48.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.993+0000 7f8d4cff9700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d3806ecc0 0x7f8d38071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:48.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.993+0000 7f8d4cff9700 1 -- 172.21.15.151:0/2711949916 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8d4408ee50 con 0x7f8d50101770 2024-04-16T03:13:48.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.993+0000 7f8d4f7fe700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d3806ecc0 0x7f8d38071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:48.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.993+0000 7f8d4f7fe700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d3806ecc0 0x7f8d38071180 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8d400059d0 tx=0x7f8d4001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:48.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:48.998+0000 7f8d4cff9700 1 -- 172.21.15.151:0/2711949916 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8d44057c00 con 0x7f8d50101770 2024-04-16T03:13:49.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.216+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d5002cda0 con 0x7f8d3806ecc0 2024-04-16T03:13:49.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.217+0000 7f8d4cff9700 1 -- 172.21.15.151:0/2711949916 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8d5002cda0 con 0x7f8d3806ecc0 2024-04-16T03:13:49.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d3806ecc0 msgr2=0x7f8d38071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d3806ecc0 0x7f8d38071180 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8d400059d0 tx=0x7f8d4001a040 comp rx=0 tx=0).stop 2024-04-16T03:13:49.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50101770 msgr2=0x7f8d500735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50101770 0x7f8d500735d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400c8f0 tx=0x7f8d4400ccb0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 shutdown_connections 2024-04-16T03:13:49.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8d3806ecc0 0x7f8d38071180 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d50100dc0 0x7f8d50073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 --2- 172.21.15.151:0/2711949916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8d50101770 0x7f8d500735d0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 >> 172.21.15.151:0/2711949916 conn(0x7f8d500fc650 msgr2=0x7f8d500fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:49.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 shutdown_connections 2024-04-16T03:13:49.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.219+0000 7f8d55e96700 1 -- 172.21.15.151:0/2711949916 wait complete. 2024-04-16T03:13:49.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:49 smithi151 ceph-mon[112233]: pgmap v1056: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:49.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.324+0000 7f9c50599700 1 -- 172.21.15.151:0/3446472155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 msgr2=0x7f9c48105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.324+0000 7f9c50599700 1 --2- 172.21.15.151:0/3446472155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 0x7f9c48105e20 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9c38009910 tx=0x7f9c38009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:49.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.324+0000 7f9c50599700 1 -- 172.21.15.151:0/3446472155 shutdown_connections 2024-04-16T03:13:49.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.324+0000 7f9c50599700 1 --2- 172.21.15.151:0/3446472155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c480ffa40 0x7f9c480ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.324+0000 7f9c50599700 1 --2- 172.21.15.151:0/3446472155 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 0x7f9c48105e20 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.324+0000 7f9c50599700 1 -- 172.21.15.151:0/3446472155 >> 172.21.15.151:0/3446472155 conn(0x7f9c480fb570 msgr2=0x7f9c480fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.324+0000 7f9c50599700 1 -- 172.21.15.151:0/3446472155 shutdown_connections 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.324+0000 7f9c50599700 1 -- 172.21.15.151:0/3446472155 wait complete. 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c50599700 1 Processor -- start 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c50599700 1 -- start start 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c50599700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c480ffa40 0x7f9c48195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c50599700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 0x7f9c48195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c50599700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48196270 con 0x7f9c48105a40 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c50599700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4819a000 con 0x7f9c480ffa40 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c4db34700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 0x7f9c48195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c4db34700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 0x7f9c48195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40444/0 (socket says 172.21.15.151:40444) 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c4db34700 1 -- 172.21.15.151:0/1152148579 learned_addr learned my addr 172.21.15.151:0/1152148579 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:49.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c4e335700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c480ffa40 0x7f9c48195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c4db34700 1 -- 172.21.15.151:0/1152148579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c480ffa40 msgr2=0x7f9c48195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c4db34700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c480ffa40 0x7f9c48195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c4db34700 1 -- 172.21.15.151:0/1152148579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c380095f0 con 0x7f9c48105a40 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.325+0000 7f9c4e335700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c480ffa40 0x7f9c48195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.326+0000 7f9c4db34700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 0x7f9c48195b90 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400d8d0 tx=0x7f9c4400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.326+0000 7f9c3f7fe700 1 -- 172.21.15.151:0/1152148579 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c44009940 con 0x7f9c48105a40 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.326+0000 7f9c3f7fe700 1 -- 172.21.15.151:0/1152148579 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c44010460 con 0x7f9c48105a40 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.326+0000 7f9c3f7fe700 1 -- 172.21.15.151:0/1152148579 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c4400f5f0 con 0x7f9c48105a40 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.326+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4819a2e0 con 0x7f9c48105a40 2024-04-16T03:13:49.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.326+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4819a830 con 0x7f9c48105a40 2024-04-16T03:13:49.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.327+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c48108040 con 0x7f9c48105a40 2024-04-16T03:13:49.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.327+0000 7f9c3f7fe700 1 -- 172.21.15.151:0/1152148579 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9c440105d0 con 0x7f9c48105a40 2024-04-16T03:13:49.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.328+0000 7f9c3f7fe700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c3406ecc0 0x7f9c34071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:49.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.328+0000 7f9c3f7fe700 1 -- 172.21.15.151:0/1152148579 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9c4408ee80 con 0x7f9c48105a40 2024-04-16T03:13:49.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.328+0000 7f9c4e335700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c3406ecc0 0x7f9c34071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:49.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.328+0000 7f9c4e335700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c3406ecc0 0x7f9c34071180 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9c38005b40 tx=0x7f9c3801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:49.334 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.333+0000 7f9c3f7fe700 1 -- 172.21.15.151:0/1152148579 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9c44057c30 con 0x7f9c48105a40 2024-04-16T03:13:49.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:49 smithi153 ceph-mon[87858]: pgmap v1056: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:49.544 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.543+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c4819ab10 con 0x7f9c3406ecc0 2024-04-16T03:13:49.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.554+0000 7f9c3f7fe700 1 -- 172.21.15.151:0/1152148579 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9c4819ab10 con 0x7f9c3406ecc0 2024-04-16T03:13:49.555 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:13:49.555 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (35m) 13m ago 46m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:13:49.555 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (47m) 13m ago 46m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (45m) 13m ago 45m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (32m) 13m ago 46m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (32m) 13m ago 45m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (35m) 13m ago 46m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (29m) 13m ago 41m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (28m) 13m ago 41m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (13m) 13m ago 41m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (28m) 13m ago 41m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (37m) 13m ago 48m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (36m) 13m ago 45m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (34m) 13m ago 48m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (33m) 13m ago 45m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (36m) 13m ago 46m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:13:49.556 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (36m) 13m ago 45m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:13:49.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (32m) 13m ago 44m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:13:49.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (31m) 13m ago 44m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:13:49.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (31m) 13m ago 43m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:13:49.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (30m) 13m ago 43m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:13:49.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (30m) 13m ago 43m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:13:49.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (29m) 13m ago 42m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:13:49.557 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (35m) 13m ago 46m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:13:49.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c3406ecc0 msgr2=0x7f9c34071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c3406ecc0 0x7f9c34071180 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9c38005b40 tx=0x7f9c3801a040 comp rx=0 tx=0).stop 2024-04-16T03:13:49.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 msgr2=0x7f9c48195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 0x7f9c48195b90 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400d8d0 tx=0x7f9c4400dc90 comp rx=0 tx=0).stop 2024-04-16T03:13:49.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 shutdown_connections 2024-04-16T03:13:49.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9c3406ecc0 0x7f9c34071180 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c480ffa40 0x7f9c48195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 --2- 172.21.15.151:0/1152148579 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c48105a40 0x7f9c48195b90 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 >> 172.21.15.151:0/1152148579 conn(0x7f9c480fb570 msgr2=0x7f9c480fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:49.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 shutdown_connections 2024-04-16T03:13:49.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.556+0000 7f9c50599700 1 -- 172.21.15.151:0/1152148579 wait complete. 2024-04-16T03:13:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.659+0000 7f7db5d33700 1 -- 172.21.15.151:0/2660807165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 msgr2=0x7f7db0105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.659+0000 7f7db5d33700 1 --2- 172.21.15.151:0/2660807165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 0x7f7db0105790 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009910 tx=0x7f7d98009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.659+0000 7f7db5d33700 1 -- 172.21.15.151:0/2660807165 shutdown_connections 2024-04-16T03:13:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.659+0000 7f7db5d33700 1 --2- 172.21.15.151:0/2660807165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 0x7f7db0105790 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.659+0000 7f7db5d33700 1 --2- 172.21.15.151:0/2660807165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db0068670 0x7f7db0068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.659+0000 7f7db5d33700 1 -- 172.21.15.151:0/2660807165 >> 172.21.15.151:0/2660807165 conn(0x7f7db00755a0 msgr2=0x7f7db00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.659+0000 7f7db5d33700 1 -- 172.21.15.151:0/2660807165 shutdown_connections 2024-04-16T03:13:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.659+0000 7f7db5d33700 1 -- 172.21.15.151:0/2660807165 wait complete. 2024-04-16T03:13:49.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7db5d33700 1 Processor -- start 2024-04-16T03:13:49.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7db5d33700 1 -- start start 2024-04-16T03:13:49.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7db5d33700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db0068670 0x7f7db019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:49.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7db5d33700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 0x7f7db019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7db5d33700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db019f3b0 con 0x7f7db0069020 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7db5d33700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db0198860 con 0x7f7db0068670 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7daeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 0x7f7db019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7daeffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 0x7f7db019ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40468/0 (socket says 172.21.15.151:40468) 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7daeffd700 1 -- 172.21.15.151:0/4121967489 learned_addr learned my addr 172.21.15.151:0/4121967489 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.660+0000 7f7daf7fe700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db0068670 0x7f7db019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.661+0000 7f7daeffd700 1 -- 172.21.15.151:0/4121967489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db0068670 msgr2=0x7f7db019e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.661+0000 7f7daeffd700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db0068670 0x7f7db019e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.661+0000 7f7daeffd700 1 -- 172.21.15.151:0/4121967489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d980095f0 con 0x7f7db0069020 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.661+0000 7f7daeffd700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 0x7f7db019ed20 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f7d980092e0 tx=0x7f7d9800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.661+0000 7f7dacff9700 1 -- 172.21.15.151:0/4121967489 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d9801d070 con 0x7f7db0069020 2024-04-16T03:13:49.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.661+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db0198ae0 con 0x7f7db0069020 2024-04-16T03:13:49.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.661+0000 7f7dacff9700 1 -- 172.21.15.151:0/4121967489 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d98005dc0 con 0x7f7db0069020 2024-04-16T03:13:49.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.661+0000 7f7dacff9700 1 -- 172.21.15.151:0/4121967489 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d98017400 con 0x7f7db0069020 2024-04-16T03:13:49.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.661+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db0198fd0 con 0x7f7db0069020 2024-04-16T03:13:49.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.662+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db004ea90 con 0x7f7db0069020 2024-04-16T03:13:49.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.663+0000 7f7dacff9700 1 -- 172.21.15.151:0/4121967489 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7d9800f6f0 con 0x7f7db0069020 2024-04-16T03:13:49.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.663+0000 7f7dacff9700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7d9c06ecc0 0x7f7d9c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:49.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.663+0000 7f7dacff9700 1 -- 172.21.15.151:0/4121967489 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7d98090250 con 0x7f7db0069020 2024-04-16T03:13:49.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.663+0000 7f7daf7fe700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7d9c06ecc0 0x7f7d9c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:49.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.664+0000 7f7daf7fe700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7d9c06ecc0 0x7f7d9c071180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7da0005950 tx=0x7f7da000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.668+0000 7f7dacff9700 1 -- 172.21.15.151:0/4121967489 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7d9805cb30 con 0x7f7db0069020 2024-04-16T03:13:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.951+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7db0199d70 con 0x7f7db0069020 2024-04-16T03:13:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.952+0000 7f7dacff9700 1 -- 172.21.15.151:0/4121967489 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7d9805c6c0 con 0x7f7db0069020 2024-04-16T03:13:49.953 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:13:49.953 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:13:49.953 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:13:49.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:13:49.954 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:13:49.955 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:13:49.955 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:13:49.955 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:13:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7d9c06ecc0 msgr2=0x7f7d9c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7d9c06ecc0 0x7f7d9c071180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7da0005950 tx=0x7f7da000b410 comp rx=0 tx=0).stop 2024-04-16T03:13:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 msgr2=0x7f7db019ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 0x7f7db019ed20 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f7d980092e0 tx=0x7f7d9800bd70 comp rx=0 tx=0).stop 2024-04-16T03:13:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 shutdown_connections 2024-04-16T03:13:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7d9c06ecc0 0x7f7d9c071180 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db0068670 0x7f7db019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 --2- 172.21.15.151:0/4121967489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7db0069020 0x7f7db019ed20 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:49.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 >> 172.21.15.151:0/4121967489 conn(0x7f7db00755a0 msgr2=0x7f7db0102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:49.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 shutdown_connections 2024-04-16T03:13:49.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:49.954+0000 7f7db5d33700 1 -- 172.21.15.151:0/4121967489 wait complete. 2024-04-16T03:13:50.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.060+0000 7f221fdb5700 1 -- 172.21.15.151:0/4050228970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22180ffbd0 msgr2=0x7f2218100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.060+0000 7f221fdb5700 1 --2- 172.21.15.151:0/4050228970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22180ffbd0 0x7f2218100050 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f2214009910 tx=0x7f2214009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:50.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.060+0000 7f221fdb5700 1 -- 172.21.15.151:0/4050228970 shutdown_connections 2024-04-16T03:13:50.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.060+0000 7f221fdb5700 1 --2- 172.21.15.151:0/4050228970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22180ffbd0 0x7f2218100050 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.060+0000 7f221fdb5700 1 --2- 172.21.15.151:0/4050228970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2218105a40 0x7f2218105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.060+0000 7f221fdb5700 1 -- 172.21.15.151:0/4050228970 >> 172.21.15.151:0/4050228970 conn(0x7f22180755a0 msgr2=0x7f22180759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:50.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.060+0000 7f221fdb5700 1 -- 172.21.15.151:0/4050228970 shutdown_connections 2024-04-16T03:13:50.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.060+0000 7f221fdb5700 1 -- 172.21.15.151:0/4050228970 wait complete. 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.061+0000 7f221fdb5700 1 Processor -- start 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.061+0000 7f221fdb5700 1 -- start start 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.061+0000 7f221fdb5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22180ffbd0 0x7f221819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.061+0000 7f221fdb5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2218105a40 0x7f221819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.061+0000 7f221fdb5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221819d0d0 con 0x7f22180ffbd0 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.061+0000 7f221fdb5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2218196580 con 0x7f2218105a40 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.061+0000 7f221db51700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22180ffbd0 0x7f221819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.061+0000 7f221d350700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2218105a40 0x7f221819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.061+0000 7f221d350700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2218105a40 0x7f221819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:42914/0 (socket says 172.21.15.151:42914) 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.062+0000 7f221d350700 1 -- 172.21.15.151:0/4233759101 learned_addr learned my addr 172.21.15.151:0/4233759101 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.062+0000 7f221db51700 1 -- 172.21.15.151:0/4233759101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2218105a40 msgr2=0x7f221819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.062+0000 7f221db51700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2218105a40 0x7f221819ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.062+0000 7f221db51700 1 -- 172.21.15.151:0/4233759101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22140095f0 con 0x7f22180ffbd0 2024-04-16T03:13:50.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.062+0000 7f221db51700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22180ffbd0 0x7f221819c500 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f220c00e970 tx=0x7f220c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:50.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.064+0000 7f220affd700 1 -- 172.21.15.151:0/4233759101 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c00c9d0 con 0x7f22180ffbd0 2024-04-16T03:13:50.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.064+0000 7f220affd700 1 -- 172.21.15.151:0/4233759101 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f220c010430 con 0x7f22180ffbd0 2024-04-16T03:13:50.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.064+0000 7f220affd700 1 -- 172.21.15.151:0/4233759101 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c018630 con 0x7f22180ffbd0 2024-04-16T03:13:50.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.064+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2218196860 con 0x7f22180ffbd0 2024-04-16T03:13:50.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.065+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2218196db0 con 0x7f22180ffbd0 2024-04-16T03:13:50.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.065+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2218108fe0 con 0x7f22180ffbd0 2024-04-16T03:13:50.067 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.066+0000 7f220affd700 1 -- 172.21.15.151:0/4233759101 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f220c0105a0 con 0x7f22180ffbd0 2024-04-16T03:13:50.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.066+0000 7f220affd700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f220406ecc0 0x7f2204071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:50.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.066+0000 7f220affd700 1 -- 172.21.15.151:0/4233759101 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f220c014070 con 0x7f22180ffbd0 2024-04-16T03:13:50.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.066+0000 7f221d350700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f220406ecc0 0x7f2204071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:50.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.067+0000 7f221d350700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f220406ecc0 0x7f2204071180 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2218104d90 tx=0x7f2214005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:50.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.071+0000 7f220affd700 1 -- 172.21.15.151:0/4233759101 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f220c053ff0 con 0x7f22180ffbd0 2024-04-16T03:13:50.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:50 smithi151 ceph-mon[112233]: from='client.35912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:50 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4121967489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.335+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f22181970a0 con 0x7f22180ffbd0 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.336+0000 7f220affd700 1 -- 172.21.15.151:0/4233759101 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f220c05b960 con 0x7f22180ffbd0 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:13:50.337 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:13:50.338 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:13:50.339 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:13:50.340 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:13:50.340 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:13:50.340 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:13:50.340 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f220406ecc0 msgr2=0x7f2204071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f220406ecc0 0x7f2204071180 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2218104d90 tx=0x7f2214005a90 comp rx=0 tx=0).stop 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22180ffbd0 msgr2=0x7f221819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22180ffbd0 0x7f221819c500 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f220c00e970 tx=0x7f220c00ec80 comp rx=0 tx=0).stop 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 shutdown_connections 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f220406ecc0 0x7f2204071180 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22180ffbd0 0x7f221819c500 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 --2- 172.21.15.151:0/4233759101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2218105a40 0x7f221819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 >> 172.21.15.151:0/4233759101 conn(0x7f22180755a0 msgr2=0x7f2218068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 shutdown_connections 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.339+0000 7f221fdb5700 1 -- 172.21.15.151:0/4233759101 wait complete. 2024-04-16T03:13:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:13:50.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.445+0000 7f3251e8c700 1 -- 172.21.15.151:0/4230165826 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 msgr2=0x7f324c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.445+0000 7f3251e8c700 1 --2- 172.21.15.151:0/4230165826 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 0x7f324c1011a0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f323c009910 tx=0x7f323c009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.445+0000 7f3251e8c700 1 -- 172.21.15.151:0/4230165826 shutdown_connections 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.445+0000 7f3251e8c700 1 --2- 172.21.15.151:0/4230165826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f324c101770 0x7f324c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.445+0000 7f3251e8c700 1 --2- 172.21.15.151:0/4230165826 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 0x7f324c1011a0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.445+0000 7f3251e8c700 1 -- 172.21.15.151:0/4230165826 >> 172.21.15.151:0/4230165826 conn(0x7f324c0fc650 msgr2=0x7f324c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.445+0000 7f3251e8c700 1 -- 172.21.15.151:0/4230165826 shutdown_connections 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f3251e8c700 1 -- 172.21.15.151:0/4230165826 wait complete. 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f3251e8c700 1 Processor -- start 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f3251e8c700 1 -- start start 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f3251e8c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 0x7f324c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f3251e8c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f324c101770 0x7f324c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f3251e8c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c073b10 con 0x7f324c100dc0 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f3251e8c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c073c50 con 0x7f324c101770 2024-04-16T03:13:50.449 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f324b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 0x7f324c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f324b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 0x7f324c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40490/0 (socket says 172.21.15.151:40490) 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.446+0000 7f324b7fe700 1 -- 172.21.15.151:0/1248740967 learned_addr learned my addr 172.21.15.151:0/1248740967 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f3243fff700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f324c101770 0x7f324c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f324b7fe700 1 -- 172.21.15.151:0/1248740967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f324c101770 msgr2=0x7f324c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f324b7fe700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f324c101770 0x7f324c0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f324b7fe700 1 -- 172.21.15.151:0/1248740967 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0095f0 con 0x7f324c100dc0 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f324b7fe700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 0x7f324c073090 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f323c006010 tx=0x7f323c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f32497fa700 1 -- 172.21.15.151:0/1248740967 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f323c01d070 con 0x7f324c100dc0 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f32497fa700 1 -- 172.21.15.151:0/1248740967 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f323c00f460 con 0x7f324c100dc0 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324c073ed0 con 0x7f324c100dc0 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f32497fa700 1 -- 172.21.15.151:0/1248740967 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f323c017610 con 0x7f324c100dc0 2024-04-16T03:13:50.450 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.447+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324c1a2c70 con 0x7f324c100dc0 2024-04-16T03:13:50.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.448+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324c109160 con 0x7f324c100dc0 2024-04-16T03:13:50.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.449+0000 7f32497fa700 1 -- 172.21.15.151:0/1248740967 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f323c00f5d0 con 0x7f324c100dc0 2024-04-16T03:13:50.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.449+0000 7f32497fa700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f323806ecc0 0x7f3238071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:50.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.449+0000 7f3243fff700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f323806ecc0 0x7f3238071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:50.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.449+0000 7f32497fa700 1 -- 172.21.15.151:0/1248740967 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f323c090390 con 0x7f324c100dc0 2024-04-16T03:13:50.451 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.449+0000 7f3243fff700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f323806ecc0 0x7f3238071180 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f324c074880 tx=0x7f323400a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:50.455 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.454+0000 7f32497fa700 1 -- 172.21.15.151:0/1248740967 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f323c059140 con 0x7f324c100dc0 2024-04-16T03:13:50.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:50 smithi153 ceph-mon[87858]: from='client.35912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:50.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:50 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4121967489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:13:50.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.670+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f324c0611d0 con 0x7f323806ecc0 2024-04-16T03:13:50.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.671+0000 7f32497fa700 1 -- 172.21.15.151:0/1248740967 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f324c0611d0 con 0x7f323806ecc0 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:13:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:13:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:13:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:13:50.674 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:13:50.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f323806ecc0 msgr2=0x7f3238071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f323806ecc0 0x7f3238071180 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f324c074880 tx=0x7f323400a300 comp rx=0 tx=0).stop 2024-04-16T03:13:50.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 msgr2=0x7f324c073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 0x7f324c073090 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f323c006010 tx=0x7f323c00bd70 comp rx=0 tx=0).stop 2024-04-16T03:13:50.675 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 shutdown_connections 2024-04-16T03:13:50.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f323806ecc0 0x7f3238071180 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f324c100dc0 0x7f324c073090 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 --2- 172.21.15.151:0/1248740967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f324c101770 0x7f324c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 >> 172.21.15.151:0/1248740967 conn(0x7f324c0fc650 msgr2=0x7f324c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:50.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 shutdown_connections 2024-04-16T03:13:50.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.674+0000 7f3251e8c700 1 -- 172.21.15.151:0/1248740967 wait complete. 2024-04-16T03:13:50.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.779+0000 7f0061135700 1 -- 172.21.15.151:0/1246000466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 msgr2=0x7f005c068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.779+0000 7f0061135700 1 --2- 172.21.15.151:0/1246000466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 0x7f005c068a50 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f0044009910 tx=0x7f0044009c20 comp rx=0 tx=0).stop 2024-04-16T03:13:50.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.779+0000 7f0061135700 1 -- 172.21.15.151:0/1246000466 shutdown_connections 2024-04-16T03:13:50.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.779+0000 7f0061135700 1 --2- 172.21.15.151:0/1246000466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f005c069020 0x7f005c105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.779+0000 7f0061135700 1 --2- 172.21.15.151:0/1246000466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 0x7f005c068a50 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.779+0000 7f0061135700 1 -- 172.21.15.151:0/1246000466 >> 172.21.15.151:0/1246000466 conn(0x7f005c0755a0 msgr2=0x7f005c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:50.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.779+0000 7f0061135700 1 -- 172.21.15.151:0/1246000466 shutdown_connections 2024-04-16T03:13:50.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.779+0000 7f0061135700 1 -- 172.21.15.151:0/1246000466 wait complete. 2024-04-16T03:13:50.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f0061135700 1 Processor -- start 2024-04-16T03:13:50.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f0061135700 1 -- start start 2024-04-16T03:13:50.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f0061135700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 0x7f005c0ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:50.782 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f0061135700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f005c069020 0x7f005c1001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f0061135700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c100720 con 0x7f005c068670 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f0061135700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c100860 con 0x7f005c069020 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f005ad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 0x7f005c0ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f005ad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 0x7f005c0ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40510/0 (socket says 172.21.15.151:40510) 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f005ad9d700 1 -- 172.21.15.151:0/2629928464 learned_addr learned my addr 172.21.15.151:0/2629928464 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f005a59c700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f005c069020 0x7f005c1001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f005ad9d700 1 -- 172.21.15.151:0/2629928464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f005c069020 msgr2=0x7f005c1001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f005ad9d700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f005c069020 0x7f005c1001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.780+0000 7f005ad9d700 1 -- 172.21.15.151:0/2629928464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00440095f0 con 0x7f005c068670 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.781+0000 7f005ad9d700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 0x7f005c0ffca0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f0044006010 tx=0x7f004400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.781+0000 7f0053fff700 1 -- 172.21.15.151:0/2629928464 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f004401d070 con 0x7f005c068670 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.781+0000 7f0053fff700 1 -- 172.21.15.151:0/2629928464 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f004400f460 con 0x7f005c068670 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.781+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c100ae0 con 0x7f005c068670 2024-04-16T03:13:50.783 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.781+0000 7f0053fff700 1 -- 172.21.15.151:0/2629928464 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0044017630 con 0x7f005c068670 2024-04-16T03:13:50.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.781+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c1a2c40 con 0x7f005c068670 2024-04-16T03:13:50.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.782+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005c109130 con 0x7f005c068670 2024-04-16T03:13:50.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.782+0000 7f0053fff700 1 -- 172.21.15.151:0/2629928464 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f004400f5d0 con 0x7f005c068670 2024-04-16T03:13:50.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.783+0000 7f0053fff700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f004806ecc0 0x7f0048071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:13:50.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.783+0000 7f0053fff700 1 -- 172.21.15.151:0/2629928464 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0044090300 con 0x7f005c068670 2024-04-16T03:13:50.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.783+0000 7f005a59c700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f004806ecc0 0x7f0048071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:13:50.785 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.783+0000 7f005a59c700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f004806ecc0 0x7f0048071180 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f004c005950 tx=0x7f004c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:13:50.789 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:50.788+0000 7f0053fff700 1 -- 172.21.15.151:0/2629928464 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f00440590b0 con 0x7f005c068670 2024-04-16T03:13:51.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.070+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f005c04ea90 con 0x7f005c068670 2024-04-16T03:13:51.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.070+0000 7f0053fff700 1 -- 172.21.15.151:0/2629928464 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f004405c770 con 0x7f005c068670 2024-04-16T03:13:51.072 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:13:51.073 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:13:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:13:51.073 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:13:51.073 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:13:51.073 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:13:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:13:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:13:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:13:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:13:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:13:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:13:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:13:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:13:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:13:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:13:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:13:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:13:51.075 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:13:51.075 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:13:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:13:51.075 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:13:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:13:51.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.072+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f004806ecc0 msgr2=0x7f0048071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:51.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.072+0000 7f0061135700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f004806ecc0 0x7f0048071180 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f004c005950 tx=0x7f004c0058e0 comp rx=0 tx=0).stop 2024-04-16T03:13:51.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.072+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 msgr2=0x7f005c0ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:13:51.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.072+0000 7f0061135700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 0x7f005c0ffca0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f0044006010 tx=0x7f004400be20 comp rx=0 tx=0).stop 2024-04-16T03:13:51.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.072+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 shutdown_connections 2024-04-16T03:13:51.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.073+0000 7f0061135700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f004806ecc0 0x7f0048071180 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:51.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.073+0000 7f0061135700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f005c068670 0x7f005c0ffca0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:51.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.073+0000 7f0061135700 1 --2- 172.21.15.151:0/2629928464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f005c069020 0x7f005c1001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:13:51.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.073+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 >> 172.21.15.151:0/2629928464 conn(0x7f005c0755a0 msgr2=0x7f005c0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:13:51.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.073+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 shutdown_connections 2024-04-16T03:13:51.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:13:51.073+0000 7f0061135700 1 -- 172.21.15.151:0/2629928464 wait complete. 2024-04-16T03:13:51.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:51 smithi151 ceph-mon[112233]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:51.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:51 smithi151 ceph-mon[112233]: pgmap v1057: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:51.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:51 smithi151 ceph-mon[112233]: from='client.35920 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:51.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:51 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4233759101' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:13:51.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:51 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2629928464' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:13:51.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:51 smithi153 ceph-mon[87858]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:51.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:51 smithi153 ceph-mon[87858]: pgmap v1057: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:51.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:51 smithi153 ceph-mon[87858]: from='client.35920 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:51.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:51 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4233759101' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:13:51.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:51 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2629928464' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:13:52.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:52 smithi151 ceph-mon[112233]: from='client.35932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:52.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:52 smithi153 ceph-mon[87858]: from='client.35932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:13:53.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:53 smithi153 ceph-mon[87858]: pgmap v1058: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:53.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:53 smithi151 ceph-mon[112233]: pgmap v1058: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:54.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:54 smithi151 ceph-mon[112233]: pgmap v1059: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:54.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:13:54.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:54 smithi153 ceph-mon[87858]: pgmap v1059: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:54.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:13:56.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:56 smithi151 ceph-mon[112233]: pgmap v1060: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:56.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:56 smithi153 ceph-mon[87858]: pgmap v1060: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:58.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:13:58 smithi151 ceph-mon[112233]: pgmap v1061: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:13:58.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:13:58 smithi153 ceph-mon[87858]: pgmap v1061: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:00.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:00 smithi151 ceph-mon[112233]: pgmap v1062: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:00.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:00 smithi153 ceph-mon[87858]: pgmap v1062: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:02.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:02 smithi151 ceph-mon[112233]: pgmap v1063: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:02.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:02 smithi153 ceph-mon[87858]: pgmap v1063: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:04.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:04 smithi151 ceph-mon[112233]: pgmap v1064: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:04.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:04 smithi153 ceph-mon[87858]: pgmap v1064: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:06.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:06 smithi151 ceph-mon[112233]: pgmap v1065: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:06.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:06 smithi153 ceph-mon[87858]: pgmap v1065: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:08.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:08 smithi151 ceph-mon[112233]: pgmap v1066: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:08.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:08 smithi153 ceph-mon[87858]: pgmap v1066: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:09.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:14:09.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:14:10.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:10 smithi151 ceph-mon[112233]: pgmap v1067: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:10.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:10 smithi153 ceph-mon[87858]: pgmap v1067: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:12.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:12 smithi151 ceph-mon[112233]: pgmap v1068: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:12.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:12 smithi153 ceph-mon[87858]: pgmap v1068: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:14.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:14 smithi151 ceph-mon[112233]: pgmap v1069: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:14.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:14 smithi153 ceph-mon[87858]: pgmap v1069: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:16.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:16 smithi151 ceph-mon[112233]: pgmap v1070: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:16.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:16 smithi153 ceph-mon[87858]: pgmap v1070: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:18.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:18 smithi151 ceph-mon[112233]: pgmap v1071: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:18.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:18 smithi153 ceph-mon[87858]: pgmap v1071: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:20.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:20 smithi151 ceph-mon[112233]: pgmap v1072: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:20.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:20 smithi153 ceph-mon[87858]: pgmap v1072: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:21.197 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.192+0000 7fce2031f700 1 -- 172.21.15.151:0/947901775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ffec0 msgr2=0x7fce181057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.197 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.192+0000 7fce2031f700 1 --2- 172.21.15.151:0/947901775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ffec0 0x7fce181057b0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fce08009910 tx=0x7fce08009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:21.197 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.192+0000 7fce2031f700 1 -- 172.21.15.151:0/947901775 shutdown_connections 2024-04-16T03:14:21.197 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.192+0000 7fce2031f700 1 --2- 172.21.15.151:0/947901775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ffec0 0x7fce181057b0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.197 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.192+0000 7fce2031f700 1 --2- 172.21.15.151:0/947901775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce180ff510 0x7fce180ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.192+0000 7fce2031f700 1 -- 172.21.15.151:0/947901775 >> 172.21.15.151:0/947901775 conn(0x7fce18074e00 msgr2=0x7fce18075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:21.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.192+0000 7fce2031f700 1 -- 172.21.15.151:0/947901775 shutdown_connections 2024-04-16T03:14:21.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.192+0000 7fce2031f700 1 -- 172.21.15.151:0/947901775 wait complete. 2024-04-16T03:14:21.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce2031f700 1 Processor -- start 2024-04-16T03:14:21.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce2031f700 1 -- start start 2024-04-16T03:14:21.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce2031f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ff510 0x7fce1819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:21.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce2031f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce180ffec0 0x7fce1819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:21.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce2031f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1819d110 con 0x7fce180ff510 2024-04-16T03:14:21.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce2031f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce181965c0 con 0x7fce180ffec0 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce1e0bb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ff510 0x7fce1819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce1e0bb700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ff510 0x7fce1819c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35394/0 (socket says 172.21.15.151:35394) 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce1e0bb700 1 -- 172.21.15.151:0/2808202075 learned_addr learned my addr 172.21.15.151:0/2808202075 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce1d8ba700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce180ffec0 0x7fce1819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce1e0bb700 1 -- 172.21.15.151:0/2808202075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce180ffec0 msgr2=0x7fce1819ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce1e0bb700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce180ffec0 0x7fce1819ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce1e0bb700 1 -- 172.21.15.151:0/2808202075 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce080095f0 con 0x7fce180ff510 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.193+0000 7fce1e0bb700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ff510 0x7fce1819c540 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fce1400c8a0 tx=0x7fce1400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.194+0000 7fce0f7fe700 1 -- 172.21.15.151:0/2808202075 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce1400cea0 con 0x7fce180ff510 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.194+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce181968a0 con 0x7fce180ff510 2024-04-16T03:14:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.194+0000 7fce0f7fe700 1 -- 172.21.15.151:0/2808202075 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce14004830 con 0x7fce180ff510 2024-04-16T03:14:21.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.194+0000 7fce0f7fe700 1 -- 172.21.15.151:0/2808202075 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce140055b0 con 0x7fce180ff510 2024-04-16T03:14:21.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.194+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce18196df0 con 0x7fce180ff510 2024-04-16T03:14:21.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.195+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce18109170 con 0x7fce180ff510 2024-04-16T03:14:21.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.195+0000 7fce0f7fe700 1 -- 172.21.15.151:0/2808202075 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fce140049a0 con 0x7fce180ff510 2024-04-16T03:14:21.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.196+0000 7fce0f7fe700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce0406ecc0 0x7fce04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:21.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.196+0000 7fce0f7fe700 1 -- 172.21.15.151:0/2808202075 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fce1408eb50 con 0x7fce180ff510 2024-04-16T03:14:21.200 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.196+0000 7fce1d8ba700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce0406ecc0 0x7fce04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:21.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.196+0000 7fce1d8ba700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce0406ecc0 0x7fce04071180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fce181076e0 tx=0x7fce0801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:21.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.201+0000 7fce0f7fe700 1 -- 172.21.15.151:0/2808202075 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fce1405b430 con 0x7fce180ff510 2024-04-16T03:14:21.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.419+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce181970d0 con 0x7fce0406ecc0 2024-04-16T03:14:21.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.421+0000 7fce0f7fe700 1 -- 172.21.15.151:0/2808202075 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fce181970d0 con 0x7fce0406ecc0 2024-04-16T03:14:21.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce0406ecc0 msgr2=0x7fce04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce0406ecc0 0x7fce04071180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fce181076e0 tx=0x7fce0801a040 comp rx=0 tx=0).stop 2024-04-16T03:14:21.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ff510 msgr2=0x7fce1819c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ff510 0x7fce1819c540 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fce1400c8a0 tx=0x7fce1400cc60 comp rx=0 tx=0).stop 2024-04-16T03:14:21.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 shutdown_connections 2024-04-16T03:14:21.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fce0406ecc0 0x7fce04071180 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce180ff510 0x7fce1819c540 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 --2- 172.21.15.151:0/2808202075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce180ffec0 0x7fce1819ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 >> 172.21.15.151:0/2808202075 conn(0x7fce18074e00 msgr2=0x7fce180fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:21.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 shutdown_connections 2024-04-16T03:14:21.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.423+0000 7fce2031f700 1 -- 172.21.15.151:0/2808202075 wait complete. 2024-04-16T03:14:21.435 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:14:21.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.531+0000 7fa04fd1d700 1 -- 172.21.15.151:0/175066640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0480ff310 msgr2=0x7fa0480ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.531+0000 7fa04fd1d700 1 --2- 172.21.15.151:0/175066640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0480ff310 0x7fa0480ff770 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa038009910 tx=0x7fa038009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:21.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.531+0000 7fa04fd1d700 1 -- 172.21.15.151:0/175066640 shutdown_connections 2024-04-16T03:14:21.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.531+0000 7fa04fd1d700 1 --2- 172.21.15.151:0/175066640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0480ff310 0x7fa0480ff770 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.531+0000 7fa04fd1d700 1 --2- 172.21.15.151:0/175066640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048106c10 0x7fa048106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.531+0000 7fa04fd1d700 1 -- 172.21.15.151:0/175066640 >> 172.21.15.151:0/175066640 conn(0x7fa048074bf0 msgr2=0x7fa048075000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:21.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.531+0000 7fa04fd1d700 1 -- 172.21.15.151:0/175066640 shutdown_connections 2024-04-16T03:14:21.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.531+0000 7fa04fd1d700 1 -- 172.21.15.151:0/175066640 wait complete. 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04fd1d700 1 Processor -- start 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04fd1d700 1 -- start start 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04fd1d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0480ff310 0x7fa04819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04fd1d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048106c10 0x7fa04819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04fd1d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04819d050 con 0x7fa048106c10 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04fd1d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa048196590 con 0x7fa0480ff310 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04d2b8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048106c10 0x7fa04819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04d2b8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048106c10 0x7fa04819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35422/0 (socket says 172.21.15.151:35422) 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04d2b8700 1 -- 172.21.15.151:0/2641097102 learned_addr learned my addr 172.21.15.151:0/2641097102 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04dab9700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0480ff310 0x7fa04819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:21.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04d2b8700 1 -- 172.21.15.151:0/2641097102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0480ff310 msgr2=0x7fa04819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04d2b8700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0480ff310 0x7fa04819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.532+0000 7fa04d2b8700 1 -- 172.21.15.151:0/2641097102 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0380095f0 con 0x7fa048106c10 2024-04-16T03:14:21.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.533+0000 7fa04d2b8700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048106c10 0x7fa04819ca50 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa038006010 tx=0x7fa03800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:21.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.533+0000 7fa03effd700 1 -- 172.21.15.151:0/2641097102 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa03801d070 con 0x7fa048106c10 2024-04-16T03:14:21.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.533+0000 7fa03effd700 1 -- 172.21.15.151:0/2641097102 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa03800f460 con 0x7fa048106c10 2024-04-16T03:14:21.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.533+0000 7fa03effd700 1 -- 172.21.15.151:0/2641097102 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa038017630 con 0x7fa048106c10 2024-04-16T03:14:21.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.533+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa048196810 con 0x7fa048106c10 2024-04-16T03:14:21.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.533+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa048196d00 con 0x7fa048106c10 2024-04-16T03:14:21.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.534+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa048109110 con 0x7fa048106c10 2024-04-16T03:14:21.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.534+0000 7fa03effd700 1 -- 172.21.15.151:0/2641097102 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa03800f5d0 con 0x7fa048106c10 2024-04-16T03:14:21.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.535+0000 7fa03effd700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:21.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.535+0000 7fa03effd700 1 -- 172.21.15.151:0/2641097102 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa0380903c0 con 0x7fa048106c10 2024-04-16T03:14:21.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.535+0000 7fa04dab9700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:21.539 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.535+0000 7fa04dab9700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa04400ac50 tx=0x7fa04400a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:21.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.539+0000 7fa03effd700 1 -- 172.21.15.151:0/2641097102 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa03805cca0 con 0x7fa048106c10 2024-04-16T03:14:21.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.756+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa048196fe0 con 0x7fa03406ecc0 2024-04-16T03:14:21.758 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.757+0000 7fa03effd700 1 -- 172.21.15.151:0/2641097102 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa048196fe0 con 0x7fa03406ecc0 2024-04-16T03:14:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.759+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 msgr2=0x7fa034071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.759+0000 7fa04fd1d700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa04400ac50 tx=0x7fa04400a380 comp rx=0 tx=0).stop 2024-04-16T03:14:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.759+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048106c10 msgr2=0x7fa04819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.759+0000 7fa04fd1d700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048106c10 0x7fa04819ca50 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa038006010 tx=0x7fa03800bd70 comp rx=0 tx=0).stop 2024-04-16T03:14:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.759+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 shutdown_connections 2024-04-16T03:14:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.759+0000 7fa04fd1d700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa03406ecc0 0x7fa034071180 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.759+0000 7fa04fd1d700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0480ff310 0x7fa04819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.759+0000 7fa04fd1d700 1 --2- 172.21.15.151:0/2641097102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa048106c10 0x7fa04819ca50 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.759+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 >> 172.21.15.151:0/2641097102 conn(0x7fa048074bf0 msgr2=0x7fa0480fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:21.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.760+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 shutdown_connections 2024-04-16T03:14:21.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.760+0000 7fa04fd1d700 1 -- 172.21.15.151:0/2641097102 wait complete. 2024-04-16T03:14:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.865+0000 7ff2d031e700 1 -- 172.21.15.151:0/1069395756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8068670 msgr2=0x7ff2c8068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.865+0000 7ff2d031e700 1 --2- 172.21.15.151:0/1069395756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8068670 0x7ff2c8068a50 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009910 tx=0x7ff2bc009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.865+0000 7ff2d031e700 1 -- 172.21.15.151:0/1069395756 shutdown_connections 2024-04-16T03:14:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.865+0000 7ff2d031e700 1 --2- 172.21.15.151:0/1069395756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2c8069020 0x7ff2c8105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.865+0000 7ff2d031e700 1 --2- 172.21.15.151:0/1069395756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8068670 0x7ff2c8068a50 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.865+0000 7ff2d031e700 1 -- 172.21.15.151:0/1069395756 >> 172.21.15.151:0/1069395756 conn(0x7ff2c80755a0 msgr2=0x7ff2c80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.865+0000 7ff2d031e700 1 -- 172.21.15.151:0/1069395756 shutdown_connections 2024-04-16T03:14:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.865+0000 7ff2d031e700 1 -- 172.21.15.151:0/1069395756 wait complete. 2024-04-16T03:14:21.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2d031e700 1 Processor -- start 2024-04-16T03:14:21.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2d031e700 1 -- start start 2024-04-16T03:14:21.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2d031e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2c8068670 0x7ff2c819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2d031e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8069020 0x7ff2c819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2d031e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c819d0e0 con 0x7ff2c8069020 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2d031e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8196590 con 0x7ff2c8068670 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2cd8b9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8069020 0x7ff2c819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2cd8b9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8069020 0x7ff2c819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35444/0 (socket says 172.21.15.151:35444) 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2cd8b9700 1 -- 172.21.15.151:0/2695458116 learned_addr learned my addr 172.21.15.151:0/2695458116 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.866+0000 7ff2ce0ba700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2c8068670 0x7ff2c819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2cd8b9700 1 -- 172.21.15.151:0/2695458116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2c8068670 msgr2=0x7ff2c819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2cd8b9700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2c8068670 0x7ff2c819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2cd8b9700 1 -- 172.21.15.151:0/2695458116 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc0095f0 con 0x7ff2c8069020 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2ce0ba700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2c8068670 0x7ff2c819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2cd8b9700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8069020 0x7ff2c819ca50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400c8f0 tx=0x7ff2c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2bb7fe700 1 -- 172.21.15.151:0/2695458116 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2c4007a50 con 0x7ff2c8069020 2024-04-16T03:14:21.870 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2bb7fe700 1 -- 172.21.15.151:0/2695458116 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2c4007bb0 con 0x7ff2c8069020 2024-04-16T03:14:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c8196870 con 0x7ff2c8069020 2024-04-16T03:14:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2bb7fe700 1 -- 172.21.15.151:0/2695458116 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2c4018630 con 0x7ff2c8069020 2024-04-16T03:14:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.867+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c8196dc0 con 0x7ff2c8069020 2024-04-16T03:14:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.868+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c8109130 con 0x7ff2c8069020 2024-04-16T03:14:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.868+0000 7ff2bb7fe700 1 -- 172.21.15.151:0/2695458116 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff2c4007d20 con 0x7ff2c8069020 2024-04-16T03:14:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.869+0000 7ff2bb7fe700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff2b406ecc0 0x7ff2b4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.869+0000 7ff2bb7fe700 1 -- 172.21.15.151:0/2695458116 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff2c408ed60 con 0x7ff2c8069020 2024-04-16T03:14:21.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.869+0000 7ff2ce0ba700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff2b406ecc0 0x7ff2b4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:21.872 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.869+0000 7ff2ce0ba700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff2b406ecc0 0x7ff2b4071180 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc005100 tx=0x7ff2bc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:21.874 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:21.874+0000 7ff2bb7fe700 1 -- 172.21.15.151:0/2695458116 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff2c405b6c0 con 0x7ff2c8069020 2024-04-16T03:14:22.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.082+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2c8197110 con 0x7ff2b406ecc0 2024-04-16T03:14:22.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.094+0000 7ff2bb7fe700 1 -- 172.21.15.151:0/2695458116 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff2c8197110 con 0x7ff2b406ecc0 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (35m) 13m ago 47m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (47m) 13m ago 47m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (46m) 13m ago 46m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (33m) 13m ago 47m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (33m) 13m ago 46m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (35m) 13m ago 46m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (29m) 13m ago 41m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (29m) 13m ago 41m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (13m) 13m ago 41m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (29m) 13m ago 41m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (37m) 13m ago 48m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:14:22.096 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (36m) 13m ago 46m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (34m) 13m ago 48m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (34m) 13m ago 46m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (36m) 13m ago 47m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (36m) 13m ago 46m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (33m) 13m ago 45m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (32m) 13m ago 44m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (31m) 13m ago 44m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (31m) 13m ago 44m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (30m) 13m ago 43m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (30m) 13m ago 43m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:14:22.097 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (36m) 13m ago 46m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:14:22.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff2b406ecc0 msgr2=0x7ff2b4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff2b406ecc0 0x7ff2b4071180 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc005100 tx=0x7ff2bc01a040 comp rx=0 tx=0).stop 2024-04-16T03:14:22.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8069020 msgr2=0x7ff2c819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.098 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8069020 0x7ff2c819ca50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400c8f0 tx=0x7ff2c400ccb0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 shutdown_connections 2024-04-16T03:14:22.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff2b406ecc0 0x7ff2b4071180 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2c8068670 0x7ff2c819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 --2- 172.21.15.151:0/2695458116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2c8069020 0x7ff2c819ca50 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 >> 172.21.15.151:0/2695458116 conn(0x7ff2c80755a0 msgr2=0x7ff2c80fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:22.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 shutdown_connections 2024-04-16T03:14:22.099 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.097+0000 7ff2d031e700 1 -- 172.21.15.151:0/2695458116 wait complete. 2024-04-16T03:14:22.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.201+0000 7fc113f94700 1 -- 172.21.15.151:0/3947195992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 msgr2=0x7fc10c105780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.201+0000 7fc113f94700 1 --2- 172.21.15.151:0/3947195992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 0x7fc10c105780 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc100009910 tx=0x7fc100009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:22.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.201+0000 7fc113f94700 1 -- 172.21.15.151:0/3947195992 shutdown_connections 2024-04-16T03:14:22.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.201+0000 7fc113f94700 1 --2- 172.21.15.151:0/3947195992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 0x7fc10c105780 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.201+0000 7fc113f94700 1 --2- 172.21.15.151:0/3947195992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c068670 0x7fc10c068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.201+0000 7fc113f94700 1 -- 172.21.15.151:0/3947195992 >> 172.21.15.151:0/3947195992 conn(0x7fc10c0754f0 msgr2=0x7fc10c075900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:22.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.201+0000 7fc113f94700 1 -- 172.21.15.151:0/3947195992 shutdown_connections 2024-04-16T03:14:22.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.201+0000 7fc113f94700 1 -- 172.21.15.151:0/3947195992 wait complete. 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc113f94700 1 Processor -- start 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc113f94700 1 -- start start 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc113f94700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c068670 0x7fc10c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc113f94700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 0x7fc10c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc113f94700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c19d0d0 con 0x7fc10c069020 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc113f94700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c196580 con 0x7fc10c068670 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc11152f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 0x7fc10c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc11152f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 0x7fc10c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35460/0 (socket says 172.21.15.151:35460) 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc11152f700 1 -- 172.21.15.151:0/2078815639 learned_addr learned my addr 172.21.15.151:0/2078815639 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc111d30700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c068670 0x7fc10c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc11152f700 1 -- 172.21.15.151:0/2078815639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c068670 msgr2=0x7fc10c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc11152f700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c068670 0x7fc10c19c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc11152f700 1 -- 172.21.15.151:0/2078815639 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1000095f0 con 0x7fc10c069020 2024-04-16T03:14:22.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.202+0000 7fc111d30700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c068670 0x7fc10c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:14:22.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.203+0000 7fc11152f700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 0x7fc10c19ca40 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc1000092e0 tx=0x7fc10000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:22.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.203+0000 7fc10affd700 1 -- 172.21.15.151:0/2078815639 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc10001d070 con 0x7fc10c069020 2024-04-16T03:14:22.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.203+0000 7fc10affd700 1 -- 172.21.15.151:0/2078815639 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc100005dc0 con 0x7fc10c069020 2024-04-16T03:14:22.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.203+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10c196800 con 0x7fc10c069020 2024-04-16T03:14:22.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.203+0000 7fc10affd700 1 -- 172.21.15.151:0/2078815639 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc100017400 con 0x7fc10c069020 2024-04-16T03:14:22.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.203+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10c196cf0 con 0x7fc10c069020 2024-04-16T03:14:22.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.204+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10c04ea90 con 0x7fc10c069020 2024-04-16T03:14:22.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.204+0000 7fc10affd700 1 -- 172.21.15.151:0/2078815639 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc10000f460 con 0x7fc10c069020 2024-04-16T03:14:22.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.205+0000 7fc10affd700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:22.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.205+0000 7fc10affd700 1 -- 172.21.15.151:0/2078815639 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc100090370 con 0x7fc10c069020 2024-04-16T03:14:22.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.205+0000 7fc111d30700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:22.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.205+0000 7fc111d30700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009690 tx=0x7fc0fc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:22.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.209+0000 7fc10affd700 1 -- 172.21.15.151:0/2078815639 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc10005cc50 con 0x7fc10c069020 2024-04-16T03:14:22.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.493+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc10c197000 con 0x7fc10c069020 2024-04-16T03:14:22.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.493+0000 7fc10affd700 1 -- 172.21.15.151:0/2078815639 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fc10005c7e0 con 0x7fc10c069020 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:14:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:14:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:14:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:14:22.496 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:14:22.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 msgr2=0x7fc0f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009690 tx=0x7fc0fc009040 comp rx=0 tx=0).stop 2024-04-16T03:14:22.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 msgr2=0x7fc10c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 0x7fc10c19ca40 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc1000092e0 tx=0x7fc10000be20 comp rx=0 tx=0).stop 2024-04-16T03:14:22.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 shutdown_connections 2024-04-16T03:14:22.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c068670 0x7fc10c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 --2- 172.21.15.151:0/2078815639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c069020 0x7fc10c19ca40 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 >> 172.21.15.151:0/2078815639 conn(0x7fc10c0754f0 msgr2=0x7fc10c0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:22.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 shutdown_connections 2024-04-16T03:14:22.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.496+0000 7fc113f94700 1 -- 172.21.15.151:0/2078815639 wait complete. 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.601+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/8413093 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28068670 msgr2=0x7f3e28068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.601+0000 7f3e2ed4c700 1 --2- 172.21.15.151:0/8413093 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28068670 0x7f3e28068a50 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009910 tx=0x7f3e1c009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.601+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/8413093 shutdown_connections 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.601+0000 7f3e2ed4c700 1 --2- 172.21.15.151:0/8413093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e28069020 0x7f3e28105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.601+0000 7f3e2ed4c700 1 --2- 172.21.15.151:0/8413093 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28068670 0x7f3e28068a50 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.601+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/8413093 >> 172.21.15.151:0/8413093 conn(0x7f3e280755a0 msgr2=0x7f3e280759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.601+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/8413093 shutdown_connections 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.601+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/8413093 wait complete. 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e2ed4c700 1 Processor -- start 2024-04-16T03:14:22.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e2ed4c700 1 -- start start 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e2ed4c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e28068670 0x7f3e2819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e2ed4c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28069020 0x7f3e2819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e2ed4c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2819d0e0 con 0x7f3e28069020 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e2ed4c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e28196590 con 0x7f3e28068670 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e27fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28069020 0x7f3e2819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e27fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28069020 0x7f3e2819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35472/0 (socket says 172.21.15.151:35472) 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e27fff700 1 -- 172.21.15.151:0/1118402021 learned_addr learned my addr 172.21.15.151:0/1118402021 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.602+0000 7f3e2cae8700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e28068670 0x7f3e2819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.603+0000 7f3e27fff700 1 -- 172.21.15.151:0/1118402021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e28068670 msgr2=0x7f3e2819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.603+0000 7f3e27fff700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e28068670 0x7f3e2819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.603+0000 7f3e27fff700 1 -- 172.21.15.151:0/1118402021 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c0095f0 con 0x7f3e28069020 2024-04-16T03:14:22.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.603+0000 7f3e27fff700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28069020 0x7f3e2819ca50 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400c8f0 tx=0x7f3e1400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:22.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.603+0000 7f3e25ffb700 1 -- 172.21.15.151:0/1118402021 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e14007a50 con 0x7f3e28069020 2024-04-16T03:14:22.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.603+0000 7f3e25ffb700 1 -- 172.21.15.151:0/1118402021 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e14007bb0 con 0x7f3e28069020 2024-04-16T03:14:22.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.603+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e28196870 con 0x7f3e28069020 2024-04-16T03:14:22.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.603+0000 7f3e25ffb700 1 -- 172.21.15.151:0/1118402021 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e14018630 con 0x7f3e28069020 2024-04-16T03:14:22.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.603+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e28196dc0 con 0x7f3e28069020 2024-04-16T03:14:22.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.604+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e28109130 con 0x7f3e28069020 2024-04-16T03:14:22.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.604+0000 7f3e25ffb700 1 -- 172.21.15.151:0/1118402021 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3e14007d20 con 0x7f3e28069020 2024-04-16T03:14:22.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.605+0000 7f3e25ffb700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e1806ecc0 0x7f3e18071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:22.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.605+0000 7f3e25ffb700 1 -- 172.21.15.151:0/1118402021 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3e1408eeb0 con 0x7f3e28069020 2024-04-16T03:14:22.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.605+0000 7f3e2cae8700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e1806ecc0 0x7f3e18071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:22.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.605+0000 7f3e2cae8700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e1806ecc0 0x7f3e18071180 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00b5c0 tx=0x7f3e1c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:22.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.610+0000 7f3e25ffb700 1 -- 172.21.15.151:0/1118402021 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3e1405b730 con 0x7f3e28069020 2024-04-16T03:14:22.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:22 smithi151 ceph-mon[112233]: pgmap v1073: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:22.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:22 smithi151 ceph-mon[112233]: from='client.35940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:22.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:22 smithi151 ceph-mon[112233]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:22.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:22 smithi153 ceph-mon[87858]: pgmap v1073: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:22.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:22 smithi153 ceph-mon[87858]: from='client.35940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:22.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:22 smithi153 ceph-mon[87858]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:22.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.854+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3e2804ea90 con 0x7f3e28069020 2024-04-16T03:14:22.855 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.854+0000 7f3e25ffb700 1 -- 172.21.15.151:0/1118402021 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f3e1405b2c0 con 0x7f3e28069020 2024-04-16T03:14:22.855 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:14:22.855 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:14:22.855 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:14:22.856 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:14:22.857 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:14:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:14:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:14:22.858 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:14:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:14:22.858 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:14:22.858 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:14:22.858 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:14:22.858 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.856+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e1806ecc0 msgr2=0x7f3e18071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.856+0000 7f3e2ed4c700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e1806ecc0 0x7f3e18071180 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00b5c0 tx=0x7f3e1c01a040 comp rx=0 tx=0).stop 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.856+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28069020 msgr2=0x7f3e2819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.856+0000 7f3e2ed4c700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28069020 0x7f3e2819ca50 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400c8f0 tx=0x7f3e1400ccb0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.856+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 shutdown_connections 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.856+0000 7f3e2ed4c700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e1806ecc0 0x7f3e18071180 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.856+0000 7f3e2ed4c700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e28068670 0x7f3e2819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.856+0000 7f3e2ed4c700 1 --2- 172.21.15.151:0/1118402021 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e28069020 0x7f3e2819ca50 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.856+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 >> 172.21.15.151:0/1118402021 conn(0x7f3e280755a0 msgr2=0x7f3e280fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.857+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 shutdown_connections 2024-04-16T03:14:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.857+0000 7f3e2ed4c700 1 -- 172.21.15.151:0/1118402021 wait complete. 2024-04-16T03:14:22.860 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:14:22.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.961+0000 7fc1112bf700 1 -- 172.21.15.151:0/3828060232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c100e10 msgr2=0x7fc10c1011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.961+0000 7fc1112bf700 1 --2- 172.21.15.151:0/3828060232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c100e10 0x7fc10c1011f0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009910 tx=0x7fc0fc009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:22.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.961+0000 7fc1112bf700 1 -- 172.21.15.151:0/3828060232 shutdown_connections 2024-04-16T03:14:22.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.961+0000 7fc1112bf700 1 --2- 172.21.15.151:0/3828060232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c1017c0 0x7fc10c105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.961+0000 7fc1112bf700 1 --2- 172.21.15.151:0/3828060232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c100e10 0x7fc10c1011f0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.961+0000 7fc1112bf700 1 -- 172.21.15.151:0/3828060232 >> 172.21.15.151:0/3828060232 conn(0x7fc10c0fc6c0 msgr2=0x7fc10c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:22.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.961+0000 7fc1112bf700 1 -- 172.21.15.151:0/3828060232 shutdown_connections 2024-04-16T03:14:22.963 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.962+0000 7fc1112bf700 1 -- 172.21.15.151:0/3828060232 wait complete. 2024-04-16T03:14:22.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.962+0000 7fc1112bf700 1 Processor -- start 2024-04-16T03:14:22.964 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.962+0000 7fc1112bf700 1 -- start start 2024-04-16T03:14:22.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.962+0000 7fc1112bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c100e10 0x7fc10c19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:22.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.962+0000 7fc1112bf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c1017c0 0x7fc10c19cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:22.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc1112bf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c19d220 con 0x7fc10c1017c0 2024-04-16T03:14:22.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc1112bf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c1966d0 con 0x7fc10c100e10 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc10a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c1017c0 0x7fc10c19cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc10a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c1017c0 0x7fc10c19cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35492/0 (socket says 172.21.15.151:35492) 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc10a7fc700 1 -- 172.21.15.151:0/4112623986 learned_addr learned my addr 172.21.15.151:0/4112623986 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc10affd700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c100e10 0x7fc10c19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc10a7fc700 1 -- 172.21.15.151:0/4112623986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c100e10 msgr2=0x7fc10c19c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc10a7fc700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c100e10 0x7fc10c19c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc10a7fc700 1 -- 172.21.15.151:0/4112623986 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc0095f0 con 0x7fc10c1017c0 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc10a7fc700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c1017c0 0x7fc10c19cb90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc10000b480 tx=0x7fc10000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc0f3fff700 1 -- 172.21.15.151:0/4112623986 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1000118c0 con 0x7fc10c1017c0 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc0f3fff700 1 -- 172.21.15.151:0/4112623986 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc100010380 con 0x7fc10c1017c0 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10c1969b0 con 0x7fc10c1017c0 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc0f3fff700 1 -- 172.21.15.151:0/4112623986 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc10000f4f0 con 0x7fc10c1017c0 2024-04-16T03:14:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.963+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10c196f00 con 0x7fc10c1017c0 2024-04-16T03:14:22.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.964+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10c1091b0 con 0x7fc10c1017c0 2024-04-16T03:14:22.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.965+0000 7fc0f3fff700 1 -- 172.21.15.151:0/4112623986 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc1000104f0 con 0x7fc10c1017c0 2024-04-16T03:14:22.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.965+0000 7fc0f3fff700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:22.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.965+0000 7fc0f3fff700 1 -- 172.21.15.151:0/4112623986 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc10008ea90 con 0x7fc10c1017c0 2024-04-16T03:14:22.967 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.965+0000 7fc10affd700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:22.968 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.966+0000 7fc10affd700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00b5c0 tx=0x7fc0fc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:22.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:22.970+0000 7fc0f3fff700 1 -- 172.21.15.151:0/4112623986 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc10005b370 con 0x7fc10c1017c0 2024-04-16T03:14:23.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.190+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc10c1971e0 con 0x7fc0f406ecc0 2024-04-16T03:14:23.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.191+0000 7fc0f3fff700 1 -- 172.21.15.151:0/4112623986 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc10c1971e0 con 0x7fc0f406ecc0 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:14:23.193 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:14:23.194 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:14:23.194 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:14:23.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.193+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 msgr2=0x7fc0f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:23.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.193+0000 7fc1112bf700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00b5c0 tx=0x7fc0fc01a040 comp rx=0 tx=0).stop 2024-04-16T03:14:23.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.193+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c1017c0 msgr2=0x7fc10c19cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:23.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.193+0000 7fc1112bf700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c1017c0 0x7fc10c19cb90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc10000b480 tx=0x7fc10000b840 comp rx=0 tx=0).stop 2024-04-16T03:14:23.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.193+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 shutdown_connections 2024-04-16T03:14:23.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.194+0000 7fc1112bf700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc0f406ecc0 0x7fc0f4071180 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:23.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.194+0000 7fc1112bf700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc10c100e10 0x7fc10c19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:23.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.194+0000 7fc1112bf700 1 --2- 172.21.15.151:0/4112623986 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc10c1017c0 0x7fc10c19cb90 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:23.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.194+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 >> 172.21.15.151:0/4112623986 conn(0x7fc10c0fc6c0 msgr2=0x7fc10c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:23.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.194+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 shutdown_connections 2024-04-16T03:14:23.196 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.194+0000 7fc1112bf700 1 -- 172.21.15.151:0/4112623986 wait complete. 2024-04-16T03:14:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.298+0000 7f9e02ab5700 1 -- 172.21.15.151:0/3468056474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc101750 msgr2=0x7f9dfc1057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.298+0000 7f9e02ab5700 1 --2- 172.21.15.151:0/3468056474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc101750 0x7f9dfc1057a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009910 tx=0x7f9de8009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:23.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.298+0000 7f9e02ab5700 1 -- 172.21.15.151:0/3468056474 shutdown_connections 2024-04-16T03:14:23.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.298+0000 7f9e02ab5700 1 --2- 172.21.15.151:0/3468056474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc101750 0x7f9dfc1057a0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:23.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.298+0000 7f9e02ab5700 1 --2- 172.21.15.151:0/3468056474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9dfc100da0 0x7f9dfc101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:23.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.298+0000 7f9e02ab5700 1 -- 172.21.15.151:0/3468056474 >> 172.21.15.151:0/3468056474 conn(0x7f9dfc0fc630 msgr2=0x7f9dfc0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:23.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.298+0000 7f9e02ab5700 1 -- 172.21.15.151:0/3468056474 shutdown_connections 2024-04-16T03:14:23.300 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.298+0000 7f9e02ab5700 1 -- 172.21.15.151:0/3468056474 wait complete. 2024-04-16T03:14:23.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.299+0000 7f9e02ab5700 1 Processor -- start 2024-04-16T03:14:23.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.299+0000 7f9e02ab5700 1 -- start start 2024-04-16T03:14:23.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.299+0000 7f9e02ab5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc100da0 0x7f9dfc19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:23.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.299+0000 7f9e02ab5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9dfc101750 0x7f9dfc19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.299+0000 7f9e02ab5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc19d0d0 con 0x7f9dfc100da0 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.299+0000 7f9e02ab5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc196580 con 0x7f9dfc101750 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.299+0000 7f9e00851700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc100da0 0x7f9dfc19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.299+0000 7f9e00851700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc100da0 0x7f9dfc19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35518/0 (socket says 172.21.15.151:35518) 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.299+0000 7f9e00851700 1 -- 172.21.15.151:0/4256159821 learned_addr learned my addr 172.21.15.151:0/4256159821 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9dfbfff700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9dfc101750 0x7f9dfc19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9e00851700 1 -- 172.21.15.151:0/4256159821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9dfc101750 msgr2=0x7f9dfc19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9e00851700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9dfc101750 0x7f9dfc19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9e00851700 1 -- 172.21.15.151:0/4256159821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de80095f0 con 0x7f9dfc100da0 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9e00851700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc100da0 0x7f9dfc19c500 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f9df000d900 tx=0x7f9df000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9df9ffb700 1 -- 172.21.15.151:0/4256159821 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df00041f0 con 0x7f9dfc100da0 2024-04-16T03:14:23.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc196860 con 0x7f9dfc100da0 2024-04-16T03:14:23.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9df9ffb700 1 -- 172.21.15.151:0/4256159821 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9df0004cf0 con 0x7f9dfc100da0 2024-04-16T03:14:23.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9df9ffb700 1 -- 172.21.15.151:0/4256159821 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df000b750 con 0x7f9dfc100da0 2024-04-16T03:14:23.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.300+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc196db0 con 0x7f9dfc100da0 2024-04-16T03:14:23.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.301+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc1091a0 con 0x7f9dfc100da0 2024-04-16T03:14:23.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.302+0000 7f9df9ffb700 1 -- 172.21.15.151:0/4256159821 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9df0004750 con 0x7f9dfc100da0 2024-04-16T03:14:23.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.302+0000 7f9df9ffb700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dec06ecc0 0x7f9dec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:23.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.302+0000 7f9dfbfff700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dec06ecc0 0x7f9dec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:23.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.302+0000 7f9df9ffb700 1 -- 172.21.15.151:0/4256159821 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9df008ed60 con 0x7f9dfc100da0 2024-04-16T03:14:23.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.303+0000 7f9dfbfff700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dec06ecc0 0x7f9dec071180 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0fce50 tx=0x7f9de801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:23.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.307+0000 7f9df9ffb700 1 -- 172.21.15.151:0/4256159821 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9df005b640 con 0x7f9dfc100da0 2024-04-16T03:14:23.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.589+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9dfc197050 con 0x7f9dfc100da0 2024-04-16T03:14:23.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.589+0000 7f9df9ffb700 1 -- 172.21.15.151:0/4256159821 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f9df005b1d0 con 0x7f9dfc100da0 2024-04-16T03:14:23.591 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:14:23.591 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:14:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:14:23.591 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:14:23.591 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:14:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:14:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:14:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:14:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:14:23.593 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:14:23.593 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:14:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:14:23.593 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:14:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:14:23.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dec06ecc0 msgr2=0x7f9dec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:23.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dec06ecc0 0x7f9dec071180 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0fce50 tx=0x7f9de801a040 comp rx=0 tx=0).stop 2024-04-16T03:14:23.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc100da0 msgr2=0x7f9dfc19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:23.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc100da0 0x7f9dfc19c500 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f9df000d900 tx=0x7f9df000dc10 comp rx=0 tx=0).stop 2024-04-16T03:14:23.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 shutdown_connections 2024-04-16T03:14:23.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f9dec06ecc0 0x7f9dec071180 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:23.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9dfc100da0 0x7f9dfc19c500 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:23.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 --2- 172.21.15.151:0/4256159821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9dfc101750 0x7f9dfc19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:23.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 >> 172.21.15.151:0/4256159821 conn(0x7f9dfc0fc630 msgr2=0x7f9dfc0fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:23.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 shutdown_connections 2024-04-16T03:14:23.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:23.592+0000 7f9e02ab5700 1 -- 172.21.15.151:0/4256159821 wait complete. 2024-04-16T03:14:23.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:23 smithi151 ceph-mon[112233]: from='client.35948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:23.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:23 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2078815639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:14:23.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:23 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1118402021' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:14:23.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:23 smithi153 ceph-mon[87858]: from='client.35948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:23 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2078815639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:14:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:23 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1118402021' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:14:24.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:24 smithi151 ceph-mon[112233]: from='client.35960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:24.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:24 smithi151 ceph-mon[112233]: pgmap v1074: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:24.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:24 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4256159821' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:14:24.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:14:24.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:24 smithi153 ceph-mon[87858]: from='client.35960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:24.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:24 smithi153 ceph-mon[87858]: pgmap v1074: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:24.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:24 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4256159821' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:14:24.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:14:26.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:26 smithi151 ceph-mon[112233]: pgmap v1075: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:26.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:26 smithi153 ceph-mon[87858]: pgmap v1075: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:28.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:28 smithi153 ceph-mon[87858]: pgmap v1076: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:28.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:28 smithi151 ceph-mon[112233]: pgmap v1076: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:30.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:30 smithi153 ceph-mon[87858]: pgmap v1077: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:30.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:30 smithi151 ceph-mon[112233]: pgmap v1077: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:32.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:32 smithi151 ceph-mon[112233]: pgmap v1078: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:32.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:32 smithi153 ceph-mon[87858]: pgmap v1078: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:34.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:34 smithi153 ceph-mon[87858]: pgmap v1079: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:34.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:34 smithi151 ceph-mon[112233]: pgmap v1079: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:36.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:36 smithi153 ceph-mon[87858]: pgmap v1080: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:36.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:36 smithi151 ceph-mon[112233]: pgmap v1080: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:38.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:38 smithi153 ceph-mon[87858]: pgmap v1081: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:38.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:38 smithi151 ceph-mon[112233]: pgmap v1081: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:39.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:14:39.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:14:40.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:40 smithi153 ceph-mon[87858]: pgmap v1082: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:40.796 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:40 smithi151 ceph-mon[112233]: pgmap v1082: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:42.746 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:42 smithi151 ceph-mon[112233]: pgmap v1083: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:43.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:42 smithi153 ceph-mon[87858]: pgmap v1083: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:44.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:44 smithi151 ceph-mon[112233]: pgmap v1084: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:45.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:44 smithi153 ceph-mon[87858]: pgmap v1084: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:46.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:46 smithi151 ceph-mon[112233]: pgmap v1085: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:47.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:46 smithi153 ceph-mon[87858]: pgmap v1085: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:48.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:48 smithi151 ceph-mon[112233]: pgmap v1086: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:49.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:48 smithi153 ceph-mon[87858]: pgmap v1086: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:50.903 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:50 smithi151 ceph-mon[112233]: pgmap v1087: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:51.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:50 smithi153 ceph-mon[87858]: pgmap v1087: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:52.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:52 smithi151 ceph-mon[112233]: pgmap v1088: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:53.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:52 smithi153 ceph-mon[87858]: pgmap v1088: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:53.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.710+0000 7f66d6ada700 1 -- 172.21.15.151:0/1123486165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 msgr2=0x7f66d0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:53.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.710+0000 7f66d6ada700 1 --2- 172.21.15.151:0/1123486165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 0x7f66d0105e20 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009910 tx=0x7f66c8009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:53.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.710+0000 7f66d6ada700 1 -- 172.21.15.151:0/1123486165 shutdown_connections 2024-04-16T03:14:53.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.710+0000 7f66d6ada700 1 --2- 172.21.15.151:0/1123486165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d00ffa40 0x7f66d00ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:53.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.710+0000 7f66d6ada700 1 --2- 172.21.15.151:0/1123486165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 0x7f66d0105e20 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:53.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.710+0000 7f66d6ada700 1 -- 172.21.15.151:0/1123486165 >> 172.21.15.151:0/1123486165 conn(0x7f66d00fb570 msgr2=0x7f66d00fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:53.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.711+0000 7f66d6ada700 1 -- 172.21.15.151:0/1123486165 shutdown_connections 2024-04-16T03:14:53.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.711+0000 7f66d6ada700 1 -- 172.21.15.151:0/1123486165 wait complete. 2024-04-16T03:14:53.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.711+0000 7f66d6ada700 1 Processor -- start 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.711+0000 7f66d6ada700 1 -- start start 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.711+0000 7f66d6ada700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d00ffa40 0x7f66d019c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.711+0000 7f66d6ada700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 0x7f66d019ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.711+0000 7f66d6ada700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d019d0f0 con 0x7f66d0105a40 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.711+0000 7f66d6ada700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d01965a0 con 0x7f66d00ffa40 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.711+0000 7f66cffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 0x7f66d019ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66cffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 0x7f66d019ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60888/0 (socket says 172.21.15.151:60888) 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66cffff700 1 -- 172.21.15.151:0/853087896 learned_addr learned my addr 172.21.15.151:0/853087896 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66d4876700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d00ffa40 0x7f66d019c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66cffff700 1 -- 172.21.15.151:0/853087896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d00ffa40 msgr2=0x7f66d019c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66cffff700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d00ffa40 0x7f66d019c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66cffff700 1 -- 172.21.15.151:0/853087896 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c80095f0 con 0x7f66d0105a40 2024-04-16T03:14:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66cffff700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 0x7f66d019ca60 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f66c0007e90 tx=0x7f66c000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66cdffb700 1 -- 172.21.15.151:0/853087896 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66c000de60 con 0x7f66d0105a40 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66cdffb700 1 -- 172.21.15.151:0/853087896 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66c000f040 con 0x7f66d0105a40 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66cdffb700 1 -- 172.21.15.151:0/853087896 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66c0004520 con 0x7f66d0105a40 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.712+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d0196880 con 0x7f66d0105a40 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.713+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d0196dd0 con 0x7f66d0105a40 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.714+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d0108040 con 0x7f66d0105a40 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.714+0000 7f66cdffb700 1 -- 172.21.15.151:0/853087896 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f66c0004750 con 0x7f66d0105a40 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.714+0000 7f66cdffb700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f66bc06ecc0 0x7f66bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.715+0000 7f66cdffb700 1 -- 172.21.15.151:0/853087896 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f66c008ea80 con 0x7f66d0105a40 2024-04-16T03:14:53.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.715+0000 7f66d4876700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f66bc06ecc0 0x7f66bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:53.717 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.715+0000 7f66d4876700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f66bc06ecc0 0x7f66bc071180 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f66c8006010 tx=0x7f66c801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:53.720 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.719+0000 7f66cdffb700 1 -- 172.21.15.151:0/853087896 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f66c005b360 con 0x7f66d0105a40 2024-04-16T03:14:53.940 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.939+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f66d0196ff0 con 0x7f66bc06ecc0 2024-04-16T03:14:53.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.940+0000 7f66cdffb700 1 -- 172.21.15.151:0/853087896 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f66d0196ff0 con 0x7f66bc06ecc0 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.942+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f66bc06ecc0 msgr2=0x7f66bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.942+0000 7f66d6ada700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f66bc06ecc0 0x7f66bc071180 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f66c8006010 tx=0x7f66c801a040 comp rx=0 tx=0).stop 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.942+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 msgr2=0x7f66d019ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.942+0000 7f66d6ada700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 0x7f66d019ca60 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f66c0007e90 tx=0x7f66c000d3b0 comp rx=0 tx=0).stop 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.942+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 shutdown_connections 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.942+0000 7f66d6ada700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f66bc06ecc0 0x7f66bc071180 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.942+0000 7f66d6ada700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66d00ffa40 0x7f66d019c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.942+0000 7f66d6ada700 1 --2- 172.21.15.151:0/853087896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d0105a40 0x7f66d019ca60 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.942+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 >> 172.21.15.151:0/853087896 conn(0x7f66d00fb570 msgr2=0x7f66d00fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.943+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 shutdown_connections 2024-04-16T03:14:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:53.943+0000 7f66d6ada700 1 -- 172.21.15.151:0/853087896 wait complete. 2024-04-16T03:14:53.954 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:14:54.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.050+0000 7f9559b6f700 1 -- 172.21.15.151:0/2304674055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9554105b50 msgr2=0x7f9554105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.050+0000 7f9559b6f700 1 --2- 172.21.15.151:0/2304674055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9554105b50 0x7f9554105f30 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9548009910 tx=0x7f9548009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:54.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.050+0000 7f9559b6f700 1 -- 172.21.15.151:0/2304674055 shutdown_connections 2024-04-16T03:14:54.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.050+0000 7f9559b6f700 1 --2- 172.21.15.151:0/2304674055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95540fef70 0x7f95540ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.050+0000 7f9559b6f700 1 --2- 172.21.15.151:0/2304674055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9554105b50 0x7f9554105f30 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.050+0000 7f9559b6f700 1 -- 172.21.15.151:0/2304674055 >> 172.21.15.151:0/2304674055 conn(0x7f9554075e30 msgr2=0x7f95540fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:54.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.050+0000 7f9559b6f700 1 -- 172.21.15.151:0/2304674055 shutdown_connections 2024-04-16T03:14:54.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.050+0000 7f9559b6f700 1 -- 172.21.15.151:0/2304674055 wait complete. 2024-04-16T03:14:54.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.051+0000 7f9559b6f700 1 Processor -- start 2024-04-16T03:14:54.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.051+0000 7f9559b6f700 1 -- start start 2024-04-16T03:14:54.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.051+0000 7f9559b6f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95540fef70 0x7f95540728e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:54.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.051+0000 7f9559b6f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9554105b50 0x7f955406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:54.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f9559b6f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955406de20 con 0x7f9554105b50 2024-04-16T03:14:54.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f9559b6f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955406df90 con 0x7f95540fef70 2024-04-16T03:14:54.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f95537fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95540fef70 0x7f95540728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:54.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f95537fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95540fef70 0x7f95540728e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.151:57830/0 (socket says 172.21.15.151:57830) 2024-04-16T03:14:54.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f95537fe700 1 -- 172.21.15.151:0/3422168683 learned_addr learned my addr 172.21.15.151:0/3422168683 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:54.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f9552ffd700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9554105b50 0x7f955406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:54.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f95537fe700 1 -- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9554105b50 msgr2=0x7f955406d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f95537fe700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9554105b50 0x7f955406d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f95537fe700 1 -- 172.21.15.151:0/3422168683 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95480095f0 con 0x7f95540fef70 2024-04-16T03:14:54.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.052+0000 7f95537fe700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95540fef70 0x7f95540728e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f954800f690 tx=0x7f954800f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:54.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.053+0000 7f9550ff9700 1 -- 172.21.15.151:0/3422168683 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f954801d070 con 0x7f95540fef70 2024-04-16T03:14:54.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.053+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955406e210 con 0x7f95540fef70 2024-04-16T03:14:54.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.053+0000 7f9550ff9700 1 -- 172.21.15.151:0/3422168683 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95480050d0 con 0x7f95540fef70 2024-04-16T03:14:54.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.053+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955406e700 con 0x7f95540fef70 2024-04-16T03:14:54.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.053+0000 7f9550ff9700 1 -- 172.21.15.151:0/3422168683 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9548022470 con 0x7f95540fef70 2024-04-16T03:14:54.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.054+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9540005320 con 0x7f95540fef70 2024-04-16T03:14:54.055 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.055+0000 7f9550ff9700 1 -- 172.21.15.151:0/3422168683 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9548017400 con 0x7f95540fef70 2024-04-16T03:14:54.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.055+0000 7f9550ff9700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f953c06ecc0 0x7f953c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:54.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.055+0000 7f9550ff9700 1 -- 172.21.15.151:0/3422168683 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9548090130 con 0x7f95540fef70 2024-04-16T03:14:54.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.055+0000 7f9552ffd700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f953c06ecc0 0x7f953c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:54.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.056+0000 7f9552ffd700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f953c06ecc0 0x7f953c071180 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f955406f0d0 tx=0x7f9544009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:54.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.061+0000 7f9550ff9700 1 -- 172.21.15.151:0/3422168683 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9548058f60 con 0x7f95540fef70 2024-04-16T03:14:54.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.280+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9540000bf0 con 0x7f953c06ecc0 2024-04-16T03:14:54.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.282+0000 7f9550ff9700 1 -- 172.21.15.151:0/3422168683 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9540000bf0 con 0x7f953c06ecc0 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f953c06ecc0 msgr2=0x7f953c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f953c06ecc0 0x7f953c071180 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f955406f0d0 tx=0x7f9544009260 comp rx=0 tx=0).stop 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95540fef70 msgr2=0x7f95540728e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95540fef70 0x7f95540728e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f954800f690 tx=0x7f954800f6c0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 shutdown_connections 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f953c06ecc0 0x7f953c071180 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95540fef70 0x7f95540728e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 --2- 172.21.15.151:0/3422168683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9554105b50 0x7f955406d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 >> 172.21.15.151:0/3422168683 conn(0x7f9554075e30 msgr2=0x7f9554104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 shutdown_connections 2024-04-16T03:14:54.286 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.284+0000 7f9559b6f700 1 -- 172.21.15.151:0/3422168683 wait complete. 2024-04-16T03:14:54.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.388+0000 7f7cbf024700 1 -- 172.21.15.151:0/2059599116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 msgr2=0x7f7cb8105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.388+0000 7f7cbf024700 1 --2- 172.21.15.151:0/2059599116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 0x7f7cb8105790 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009910 tx=0x7f7ca4009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:54.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.388+0000 7f7cbf024700 1 -- 172.21.15.151:0/2059599116 shutdown_connections 2024-04-16T03:14:54.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.388+0000 7f7cbf024700 1 --2- 172.21.15.151:0/2059599116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 0x7f7cb8105790 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.389 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.388+0000 7f7cbf024700 1 --2- 172.21.15.151:0/2059599116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8068670 0x7f7cb8068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.388+0000 7f7cbf024700 1 -- 172.21.15.151:0/2059599116 >> 172.21.15.151:0/2059599116 conn(0x7f7cb80755a0 msgr2=0x7f7cb80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:54.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.388+0000 7f7cbf024700 1 -- 172.21.15.151:0/2059599116 shutdown_connections 2024-04-16T03:14:54.390 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.388+0000 7f7cbf024700 1 -- 172.21.15.151:0/2059599116 wait complete. 2024-04-16T03:14:54.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cbf024700 1 Processor -- start 2024-04-16T03:14:54.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cbf024700 1 -- start start 2024-04-16T03:14:54.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cbf024700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8068670 0x7f7cb8196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:54.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cbf024700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 0x7f7cb8196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cbf024700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb81972d0 con 0x7f7cb8069020 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cbf024700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb819a000 con 0x7f7cb8068670 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cb7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 0x7f7cb8196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cb7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 0x7f7cb8196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60914/0 (socket says 172.21.15.151:60914) 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cb7fff700 1 -- 172.21.15.151:0/3756216334 learned_addr learned my addr 172.21.15.151:0/3756216334 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.389+0000 7f7cbcdc0700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8068670 0x7f7cb8196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cb7fff700 1 -- 172.21.15.151:0/3756216334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8068670 msgr2=0x7f7cb8196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cb7fff700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8068670 0x7f7cb8196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cb7fff700 1 -- 172.21.15.151:0/3756216334 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca40095f0 con 0x7f7cb8069020 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cbcdc0700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8068670 0x7f7cb8196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cb7fff700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 0x7f7cb8196c40 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40092e0 tx=0x7f7ca400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cb5ffb700 1 -- 172.21.15.151:0/3756216334 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ca401d070 con 0x7f7cb8069020 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cb5ffb700 1 -- 172.21.15.151:0/3756216334 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ca400f460 con 0x7f7cb8069020 2024-04-16T03:14:54.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb819a280 con 0x7f7cb8069020 2024-04-16T03:14:54.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cb5ffb700 1 -- 172.21.15.151:0/3756216334 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ca40175f0 con 0x7f7cb8069020 2024-04-16T03:14:54.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.390+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb819a720 con 0x7f7cb8069020 2024-04-16T03:14:54.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.391+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb81090a0 con 0x7f7cb8069020 2024-04-16T03:14:54.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.392+0000 7f7cb5ffb700 1 -- 172.21.15.151:0/3756216334 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7ca40178b0 con 0x7f7cb8069020 2024-04-16T03:14:54.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.392+0000 7f7cb5ffb700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca806ecc0 0x7f7ca8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:54.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.392+0000 7f7cbcdc0700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca806ecc0 0x7f7ca8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:54.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.392+0000 7f7cb5ffb700 1 -- 172.21.15.151:0/3756216334 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7ca40903c0 con 0x7f7cb8069020 2024-04-16T03:14:54.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.393+0000 7f7cbcdc0700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca806ecc0 0x7f7ca8071180 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7cac005950 tx=0x7f7cac005f50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:54.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.397+0000 7f7cb5ffb700 1 -- 172.21.15.151:0/3756216334 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7ca4059170 con 0x7f7cb8069020 2024-04-16T03:14:54.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.605+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cb802cd30 con 0x7f7ca806ecc0 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.616+0000 7f7cb5ffb700 1 -- 172.21.15.151:0/3756216334 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7cb802cd30 con 0x7f7ca806ecc0 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (36m) 14m ago 47m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (48m) 14m ago 48m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (47m) 14m ago 46m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (34m) 14m ago 48m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (33m) 14m ago 46m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (36m) 14m ago 47m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (30m) 14m ago 42m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (30m) 14m ago 42m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (14m) 14m ago 42m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (29m) 14m ago 42m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (38m) 14m ago 49m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:14:54.617 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (37m) 14m ago 46m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (35m) 14m ago 49m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (34m) 14m ago 46m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (37m) 14m ago 47m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (37m) 14m ago 46m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (33m) 14m ago 45m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (32m) 14m ago 45m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (32m) 14m ago 44m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (31m) 14m ago 44m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (31m) 14m ago 44m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (30m) 14m ago 43m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:14:54.618 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (36m) 14m ago 47m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:14:54.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca806ecc0 msgr2=0x7f7ca8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca806ecc0 0x7f7ca8071180 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7cac005950 tx=0x7f7cac005f50 comp rx=0 tx=0).stop 2024-04-16T03:14:54.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 msgr2=0x7f7cb8196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 0x7f7cb8196c40 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40092e0 tx=0x7f7ca400be20 comp rx=0 tx=0).stop 2024-04-16T03:14:54.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 shutdown_connections 2024-04-16T03:14:54.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7ca806ecc0 0x7f7ca8071180 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cb8068670 0x7f7cb8196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 --2- 172.21.15.151:0/3756216334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb8069020 0x7f7cb8196c40 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 >> 172.21.15.151:0/3756216334 conn(0x7f7cb80755a0 msgr2=0x7f7cb8105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:54.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 shutdown_connections 2024-04-16T03:14:54.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.618+0000 7f7cbf024700 1 -- 172.21.15.151:0/3756216334 wait complete. 2024-04-16T03:14:54.723 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.722+0000 7f03758c1700 1 -- 172.21.15.151:0/1866773975 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 msgr2=0x7f0370105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.723 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.722+0000 7f03758c1700 1 --2- 172.21.15.151:0/1866773975 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 0x7f0370105790 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f0364009910 tx=0x7f0364009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:54.723 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.722+0000 7f03758c1700 1 -- 172.21.15.151:0/1866773975 shutdown_connections 2024-04-16T03:14:54.723 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.722+0000 7f03758c1700 1 --2- 172.21.15.151:0/1866773975 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 0x7f0370105790 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.723 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.722+0000 7f03758c1700 1 --2- 172.21.15.151:0/1866773975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370068670 0x7f0370068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.722+0000 7f03758c1700 1 -- 172.21.15.151:0/1866773975 >> 172.21.15.151:0/1866773975 conn(0x7f03700755a0 msgr2=0x7f03700759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:54.724 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:54 smithi151 ceph-mon[112233]: pgmap v1089: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:54.724 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:14:54.724 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:54 smithi151 ceph-mon[112233]: from='client.35968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.722+0000 7f03758c1700 1 -- 172.21.15.151:0/1866773975 shutdown_connections 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.722+0000 7f03758c1700 1 -- 172.21.15.151:0/1866773975 wait complete. 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.723+0000 7f03758c1700 1 Processor -- start 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.723+0000 7f03758c1700 1 -- start start 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.723+0000 7f03758c1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370068670 0x7f037019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.723+0000 7f03758c1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 0x7f037019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.723+0000 7f03758c1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037019d220 con 0x7f0370069020 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.723+0000 7f03758c1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03701966d0 con 0x7f0370068670 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f036e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 0x7f037019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f036e7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 0x7f037019cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60936/0 (socket says 172.21.15.151:60936) 2024-04-16T03:14:54.726 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f036e7fc700 1 -- 172.21.15.151:0/723874710 learned_addr learned my addr 172.21.15.151:0/723874710 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f036effd700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370068670 0x7f037019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f036e7fc700 1 -- 172.21.15.151:0/723874710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370068670 msgr2=0x7f037019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f036e7fc700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370068670 0x7f037019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f036e7fc700 1 -- 172.21.15.151:0/723874710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03640095f0 con 0x7f0370069020 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f036e7fc700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 0x7f037019cb90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f0364000c00 tx=0x7f036400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f03748bf700 1 -- 172.21.15.151:0/723874710 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f036401d070 con 0x7f0370069020 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0370196950 con 0x7f0370069020 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f03748bf700 1 -- 172.21.15.151:0/723874710 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0364004fb0 con 0x7f0370069020 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f03748bf700 1 -- 172.21.15.151:0/723874710 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f036400f460 con 0x7f0370069020 2024-04-16T03:14:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.724+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0370196e40 con 0x7f0370069020 2024-04-16T03:14:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.725+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03701090e0 con 0x7f0370069020 2024-04-16T03:14:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.726+0000 7f03748bf700 1 -- 172.21.15.151:0/723874710 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f03640228a0 con 0x7f0370069020 2024-04-16T03:14:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.726+0000 7f03748bf700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f035806ecc0 0x7f0358071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.727+0000 7f03748bf700 1 -- 172.21.15.151:0/723874710 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f03640903e0 con 0x7f0370069020 2024-04-16T03:14:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.727+0000 7f036effd700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f035806ecc0 0x7f0358071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.727+0000 7f036effd700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f035806ecc0 0x7f0358071180 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0360006fd0 tx=0x7f0360008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:54.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:54.731+0000 7f03748bf700 1 -- 172.21.15.151:0/723874710 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f036405cd90 con 0x7f0370069020 2024-04-16T03:14:55.018 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.017+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f037004ea90 con 0x7f0370069020 2024-04-16T03:14:55.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.018+0000 7f03748bf700 1 -- 172.21.15.151:0/723874710 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f036405c920 con 0x7f0370069020 2024-04-16T03:14:55.019 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:14:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:14:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:14:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:14:55.020 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:14:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:54 smithi153 ceph-mon[87858]: pgmap v1089: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:14:55.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:54 smithi153 ceph-mon[87858]: from='client.35968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f035806ecc0 msgr2=0x7f0358071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f035806ecc0 0x7f0358071180 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0360006fd0 tx=0x7f0360008040 comp rx=0 tx=0).stop 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 msgr2=0x7f037019cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 0x7f037019cb90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f0364000c00 tx=0x7f036400be20 comp rx=0 tx=0).stop 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 shutdown_connections 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f035806ecc0 0x7f0358071180 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370068670 0x7f037019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 --2- 172.21.15.151:0/723874710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0370069020 0x7f037019cb90 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 >> 172.21.15.151:0/723874710 conn(0x7f03700755a0 msgr2=0x7f03700fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 shutdown_connections 2024-04-16T03:14:55.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.020+0000 7f03758c1700 1 -- 172.21.15.151:0/723874710 wait complete. 2024-04-16T03:14:55.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.125+0000 7f53d9cf1700 1 -- 172.21.15.151:0/1835088686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 msgr2=0x7f53d4102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.125+0000 7f53d9cf1700 1 --2- 172.21.15.151:0/1835088686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 0x7f53d4102050 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f53c400b0a0 tx=0x7f53c400b3b0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.125+0000 7f53d9cf1700 1 -- 172.21.15.151:0/1835088686 shutdown_connections 2024-04-16T03:14:55.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.125+0000 7f53d9cf1700 1 --2- 172.21.15.151:0/1835088686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d4102620 0x7f53d410ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.125+0000 7f53d9cf1700 1 --2- 172.21.15.151:0/1835088686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 0x7f53d4102050 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.125+0000 7f53d9cf1700 1 -- 172.21.15.151:0/1835088686 >> 172.21.15.151:0/1835088686 conn(0x7f53d40fb550 msgr2=0x7f53d40fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:55.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.125+0000 7f53d9cf1700 1 -- 172.21.15.151:0/1835088686 shutdown_connections 2024-04-16T03:14:55.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.125+0000 7f53d9cf1700 1 -- 172.21.15.151:0/1835088686 wait complete. 2024-04-16T03:14:55.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.125+0000 7f53d9cf1700 1 Processor -- start 2024-04-16T03:14:55.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d9cf1700 1 -- start start 2024-04-16T03:14:55.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d9cf1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 0x7f53d40ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:55.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d9cf1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d4102620 0x7f53d40ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:55.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d9cf1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d41003e0 con 0x7f53d4101c70 2024-04-16T03:14:55.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d9cf1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d4100520 con 0x7f53d4102620 2024-04-16T03:14:55.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 0x7f53d40ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 0x7f53d40ff960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60962/0 (socket says 172.21.15.151:60962) 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d37fe700 1 -- 172.21.15.151:0/661120515 learned_addr learned my addr 172.21.15.151:0/661120515 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d2ffd700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d4102620 0x7f53d40ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d37fe700 1 -- 172.21.15.151:0/661120515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d4102620 msgr2=0x7f53d40ffea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d37fe700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d4102620 0x7f53d40ffea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.126+0000 7f53d37fe700 1 -- 172.21.15.151:0/661120515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c4009d00 con 0x7f53d4101c70 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.127+0000 7f53d37fe700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 0x7f53d40ff960 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f53c4000f80 tx=0x7f53c400bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.127+0000 7f53d0ff9700 1 -- 172.21.15.151:0/661120515 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53c400e040 con 0x7f53d4101c70 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.127+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d41007a0 con 0x7f53d4101c70 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.127+0000 7f53d0ff9700 1 -- 172.21.15.151:0/661120515 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53c4003c10 con 0x7f53d4101c70 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.127+0000 7f53d0ff9700 1 -- 172.21.15.151:0/661120515 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53c401d730 con 0x7f53d4101c70 2024-04-16T03:14:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.127+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d41a2c20 con 0x7f53d4101c70 2024-04-16T03:14:55.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.128+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d4108210 con 0x7f53d4101c70 2024-04-16T03:14:55.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.128+0000 7f53d0ff9700 1 -- 172.21.15.151:0/661120515 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f53c4019070 con 0x7f53d4101c70 2024-04-16T03:14:55.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.129+0000 7f53d0ff9700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53bc06ecc0 0x7f53bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:55.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.129+0000 7f53d0ff9700 1 -- 172.21.15.151:0/661120515 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f53c4090400 con 0x7f53d4101c70 2024-04-16T03:14:55.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.129+0000 7f53d2ffd700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53bc06ecc0 0x7f53bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:55.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.129+0000 7f53d2ffd700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53bc06ecc0 0x7f53bc071180 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f53d4101150 tx=0x7f53c8009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:55.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.133+0000 7f53d0ff9700 1 -- 172.21.15.151:0/661120515 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f53c4059150 con 0x7f53d4101c70 2024-04-16T03:14:55.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.375+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f53d404ea90 con 0x7f53d4101c70 2024-04-16T03:14:55.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.376+0000 7f53d0ff9700 1 -- 172.21.15.151:0/661120515 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f53c405c810 con 0x7f53d4101c70 2024-04-16T03:14:55.377 INFO:teuthology.orchestra.run.smithi151.stdout:e33 2024-04-16T03:14:55.377 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:14:55.377 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:14:55.377 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:14:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:14:55.377 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 24 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T02:44:25.408848+0000 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:14:55.378 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:up {} 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:failed 0,1 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:14:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:14:55.380 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:14:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:14:55.380 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:14:55.380 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:14:55.380 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:14:55.380 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:14:55.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53bc06ecc0 msgr2=0x7f53bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53bc06ecc0 0x7f53bc071180 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f53d4101150 tx=0x7f53c8009260 comp rx=0 tx=0).stop 2024-04-16T03:14:55.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 msgr2=0x7f53d40ff960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 0x7f53d40ff960 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f53c4000f80 tx=0x7f53c400bd10 comp rx=0 tx=0).stop 2024-04-16T03:14:55.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 shutdown_connections 2024-04-16T03:14:55.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f53bc06ecc0 0x7f53bc071180 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53d4101c70 0x7f53d40ff960 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 --2- 172.21.15.151:0/661120515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d4102620 0x7f53d40ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 >> 172.21.15.151:0/661120515 conn(0x7f53d40fb550 msgr2=0x7f53d4105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:55.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 shutdown_connections 2024-04-16T03:14:55.382 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.378+0000 7f53d9cf1700 1 -- 172.21.15.151:0/661120515 wait complete. 2024-04-16T03:14:55.382 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:14:55.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.485+0000 7fe91d516700 1 -- 172.21.15.151:0/3973861560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918102630 msgr2=0x7fe91810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.485+0000 7fe91d516700 1 --2- 172.21.15.151:0/3973861560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918102630 0x7fe91810ab20 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fe900009910 tx=0x7fe900009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.485+0000 7fe91d516700 1 -- 172.21.15.151:0/3973861560 shutdown_connections 2024-04-16T03:14:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.485+0000 7fe91d516700 1 --2- 172.21.15.151:0/3973861560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918102630 0x7fe91810ab20 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.485+0000 7fe91d516700 1 --2- 172.21.15.151:0/3973861560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918101c80 0x7fe918102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.485+0000 7fe91d516700 1 -- 172.21.15.151:0/3973861560 >> 172.21.15.151:0/3973861560 conn(0x7fe9180fb540 msgr2=0x7fe9180fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.485+0000 7fe91d516700 1 -- 172.21.15.151:0/3973861560 shutdown_connections 2024-04-16T03:14:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.485+0000 7fe91d516700 1 -- 172.21.15.151:0/3973861560 wait complete. 2024-04-16T03:14:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.485+0000 7fe91d516700 1 Processor -- start 2024-04-16T03:14:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe91d516700 1 -- start start 2024-04-16T03:14:55.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe91d516700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918101c80 0x7fe91819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:55.488 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe91d516700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918102630 0x7fe91819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe91d516700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91819d0f0 con 0x7fe918101c80 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe91d516700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9181965a0 con 0x7fe918102630 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe916ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918101c80 0x7fe91819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe916ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918101c80 0x7fe91819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:60984/0 (socket says 172.21.15.151:60984) 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe916ffd700 1 -- 172.21.15.151:0/194559495 learned_addr learned my addr 172.21.15.151:0/194559495 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe90e7fc700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918102630 0x7fe91819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe916ffd700 1 -- 172.21.15.151:0/194559495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918102630 msgr2=0x7fe91819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe916ffd700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918102630 0x7fe91819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe916ffd700 1 -- 172.21.15.151:0/194559495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9000095f0 con 0x7fe918101c80 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.486+0000 7fe916ffd700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918101c80 0x7fe91819c520 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fe90800d900 tx=0x7fe90800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.487+0000 7fe914ff9700 1 -- 172.21.15.151:0/194559495 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9080041f0 con 0x7fe918101c80 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.487+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe918196880 con 0x7fe918101c80 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.487+0000 7fe914ff9700 1 -- 172.21.15.151:0/194559495 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe908004cf0 con 0x7fe918101c80 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.487+0000 7fe914ff9700 1 -- 172.21.15.151:0/194559495 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe90800b750 con 0x7fe918101c80 2024-04-16T03:14:55.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.487+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe918196da0 con 0x7fe918101c80 2024-04-16T03:14:55.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.488+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe918108220 con 0x7fe918101c80 2024-04-16T03:14:55.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.488+0000 7fe914ff9700 1 -- 172.21.15.151:0/194559495 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe908004750 con 0x7fe918101c80 2024-04-16T03:14:55.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.489+0000 7fe914ff9700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90406ecc0 0x7fe904071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:55.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.489+0000 7fe914ff9700 1 -- 172.21.15.151:0/194559495 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe90808ed50 con 0x7fe918101c80 2024-04-16T03:14:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.489+0000 7fe90e7fc700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90406ecc0 0x7fe904071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.489+0000 7fe90e7fc700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90406ecc0 0x7fe904071180 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe900006010 tx=0x7fe90001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:55.494 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.493+0000 7fe914ff9700 1 -- 172.21.15.151:0/194559495 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe90805b630 con 0x7fe918101c80 2024-04-16T03:14:55.714 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.713+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe918197080 con 0x7fe90406ecc0 2024-04-16T03:14:55.714 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:55 smithi151 ceph-mon[112233]: from='client.45399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:55.714 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:55 smithi151 ceph-mon[112233]: from='client.35976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:55.714 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:55 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/723874710' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:14:55.714 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:55 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/661120515' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:14:55.715 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.714+0000 7fe914ff9700 1 -- 172.21.15.151:0/194559495 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe918197080 con 0x7fe90406ecc0 2024-04-16T03:14:55.715 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:14:55.715 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:14:55.715 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:14:55.715 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:14:55.716 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:14:55.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90406ecc0 msgr2=0x7fe904071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90406ecc0 0x7fe904071180 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe900006010 tx=0x7fe90001a040 comp rx=0 tx=0).stop 2024-04-16T03:14:55.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918101c80 msgr2=0x7fe91819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.718 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918101c80 0x7fe91819c520 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fe90800d900 tx=0x7fe90800dc10 comp rx=0 tx=0).stop 2024-04-16T03:14:55.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 shutdown_connections 2024-04-16T03:14:55.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe90406ecc0 0x7fe904071180 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe918101c80 0x7fe91819c520 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 --2- 172.21.15.151:0/194559495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918102630 0x7fe91819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 >> 172.21.15.151:0/194559495 conn(0x7fe9180fb540 msgr2=0x7fe918105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:55.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 shutdown_connections 2024-04-16T03:14:55.719 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.717+0000 7fe91d516700 1 -- 172.21.15.151:0/194559495 wait complete. 2024-04-16T03:14:55.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.821+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/2208102741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 msgr2=0x7f4ef01057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.821+0000 7f4ef7fd0700 1 --2- 172.21.15.151:0/2208102741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 0x7f4ef01057c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4eec009910 tx=0x7f4eec009c20 comp rx=0 tx=0).stop 2024-04-16T03:14:55.822 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.821+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/2208102741 shutdown_connections 2024-04-16T03:14:55.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.821+0000 7f4ef7fd0700 1 --2- 172.21.15.151:0/2208102741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 0x7f4ef01057c0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.821+0000 7f4ef7fd0700 1 --2- 172.21.15.151:0/2208102741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0100dc0 0x7f4ef01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.821+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/2208102741 >> 172.21.15.151:0/2208102741 conn(0x7f4ef00fc650 msgr2=0x7f4ef00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:55.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.821+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/2208102741 shutdown_connections 2024-04-16T03:14:55.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.821+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/2208102741 wait complete. 2024-04-16T03:14:55.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef7fd0700 1 Processor -- start 2024-04-16T03:14:55.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef7fd0700 1 -- start start 2024-04-16T03:14:55.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef7fd0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0100dc0 0x7f4ef019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:55.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef7fd0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 0x7f4ef019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef7fd0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef019d130 con 0x7f4ef0101770 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef7fd0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef01965e0 con 0x7f4ef0100dc0 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef556b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 0x7f4ef019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef556b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 0x7f4ef019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:32776/0 (socket says 172.21.15.151:32776) 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef556b700 1 -- 172.21.15.151:0/3835889069 learned_addr learned my addr 172.21.15.151:0/3835889069 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.822+0000 7f4ef5d6c700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0100dc0 0x7f4ef019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.823+0000 7f4ef556b700 1 -- 172.21.15.151:0/3835889069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0100dc0 msgr2=0x7f4ef019c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.823+0000 7f4ef556b700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0100dc0 0x7f4ef019c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.823+0000 7f4ef556b700 1 -- 172.21.15.151:0/3835889069 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eec0095f0 con 0x7f4ef0101770 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.823+0000 7f4ef556b700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 0x7f4ef019caa0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f4eec0092e0 tx=0x7f4eec00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.823+0000 7f4ee6ffd700 1 -- 172.21.15.151:0/3835889069 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eec01d070 con 0x7f4ef0101770 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.823+0000 7f4ee6ffd700 1 -- 172.21.15.151:0/3835889069 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4eec005dc0 con 0x7f4ef0101770 2024-04-16T03:14:55.825 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.823+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef0196860 con 0x7f4ef0101770 2024-04-16T03:14:55.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.823+0000 7f4ee6ffd700 1 -- 172.21.15.151:0/3835889069 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eec017400 con 0x7f4ef0101770 2024-04-16T03:14:55.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.823+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef0196d50 con 0x7f4ef0101770 2024-04-16T03:14:55.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.824+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef01091c0 con 0x7f4ef0101770 2024-04-16T03:14:55.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.824+0000 7f4ee6ffd700 1 -- 172.21.15.151:0/3835889069 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4eec00f6f0 con 0x7f4ef0101770 2024-04-16T03:14:55.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.825+0000 7f4ee6ffd700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4edc06ecc0 0x7f4edc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:14:55.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.825+0000 7f4ee6ffd700 1 -- 172.21.15.151:0/3835889069 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4eec090250 con 0x7f4ef0101770 2024-04-16T03:14:55.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.825+0000 7f4ef5d6c700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4edc06ecc0 0x7f4edc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:14:55.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.825+0000 7f4ef5d6c700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4edc06ecc0 0x7f4edc071180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0005950 tx=0x7f4ee000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:14:55.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:55.830+0000 7f4ee6ffd700 1 -- 172.21.15.151:0/3835889069 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4eec05cb30 con 0x7f4ef0101770 2024-04-16T03:14:56.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:55 smithi153 ceph-mon[87858]: from='client.45399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:56.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:55 smithi153 ceph-mon[87858]: from='client.35976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:56.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:55 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/723874710' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:14:56.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:55 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/661120515' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:14:56.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.126+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4ef002ccf0 con 0x7f4ef0101770 2024-04-16T03:14:56.128 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.127+0000 7f4ee6ffd700 1 -- 172.21.15.151:0/3835889069 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4303 (secure 0 0 0) 0x7f4eec05c6c0 con 0x7f4ef0101770 2024-04-16T03:14:56.128 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:14:56.128 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:14:56.128 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:14:56.128 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:14:56.128 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:14:56.128 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp8jdnun_n:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-16T03:14:56.129 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-16T03:14:56.130 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4edc06ecc0 msgr2=0x7f4edc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4edc06ecc0 0x7f4edc071180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0005950 tx=0x7f4ee000b410 comp rx=0 tx=0).stop 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 msgr2=0x7f4ef019caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 0x7f4ef019caa0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f4eec0092e0 tx=0x7f4eec00bd70 comp rx=0 tx=0).stop 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 shutdown_connections 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4edc06ecc0 0x7f4edc071180 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef0100dc0 0x7f4ef019c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 --2- 172.21.15.151:0/3835889069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ef0101770 0x7f4ef019caa0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 >> 172.21.15.151:0/3835889069 conn(0x7f4ef00fc650 msgr2=0x7f4ef00fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 shutdown_connections 2024-04-16T03:14:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:14:56.129+0000 7f4ef7fd0700 1 -- 172.21.15.151:0/3835889069 wait complete. 2024-04-16T03:14:56.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:56 smithi153 ceph-mon[87858]: pgmap v1090: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:56.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:56 smithi153 ceph-mon[87858]: from='client.35988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:56.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:56 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3835889069' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:14:56.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:56 smithi151 ceph-mon[112233]: pgmap v1090: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:56.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:56 smithi151 ceph-mon[112233]: from='client.35988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:14:56.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:56 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3835889069' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:14:58.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:14:58 smithi151 ceph-mon[112233]: pgmap v1091: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:14:59.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:14:58 smithi153 ceph-mon[87858]: pgmap v1091: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:00.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:00 smithi151 ceph-mon[112233]: pgmap v1092: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:01.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:00 smithi153 ceph-mon[87858]: pgmap v1092: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:02.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:02 smithi151 ceph-mon[112233]: pgmap v1093: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:03.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:02 smithi153 ceph-mon[87858]: pgmap v1093: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:04.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:04 smithi151 ceph-mon[112233]: pgmap v1094: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:05.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:04 smithi153 ceph-mon[87858]: pgmap v1094: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:06.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:06 smithi151 ceph-mon[112233]: pgmap v1095: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:07.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:06 smithi153 ceph-mon[87858]: pgmap v1095: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:08.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:08 smithi153 ceph-mon[87858]: pgmap v1096: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:08.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:08 smithi151 ceph-mon[112233]: pgmap v1096: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:09.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:15:10.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:15:10.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:10 smithi151 ceph-mon[112233]: pgmap v1097: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:11.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:10 smithi153 ceph-mon[87858]: pgmap v1097: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:12.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:12 smithi151 ceph-mon[112233]: pgmap v1098: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:13.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:12 smithi153 ceph-mon[87858]: pgmap v1098: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:14.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:14 smithi151 ceph-mon[112233]: pgmap v1099: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:15.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:14 smithi153 ceph-mon[87858]: pgmap v1099: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:16.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:16 smithi151 ceph-mon[112233]: pgmap v1100: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:17.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:16 smithi153 ceph-mon[87858]: pgmap v1100: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:18.937 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:18 smithi153 ceph-mon[87858]: pgmap v1101: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:18.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:18 smithi151 ceph-mon[112233]: pgmap v1101: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:20.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:20 smithi151 ceph-mon[112233]: pgmap v1102: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:21.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:20 smithi153 ceph-mon[87858]: pgmap v1102: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:22.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:22 smithi151 ceph-mon[112233]: pgmap v1103: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:23.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:22 smithi153 ceph-mon[87858]: pgmap v1103: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: pgmap v1104: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:15:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:15:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:15:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:24.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi151.bdpnxq"}]': finished 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi151.slrqog"}]': finished 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T03:15:24.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.mqbgav"}]': finished 2024-04-16T03:15:24.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T03:15:24.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.ogdcjk"}]': finished 2024-04-16T03:15:24.997 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-16T03:15:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: pgmap v1104: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:15:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:15:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:15:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:15:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:25.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi151.bdpnxq"}]: dispatch 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi151.bdpnxq"}]': finished 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi151.slrqog"}]: dispatch 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi151.slrqog"}]': finished 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.mqbgav"}]: dispatch 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.mqbgav"}]': finished 2024-04-16T03:15:25.022 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.ogdcjk"}]: dispatch 2024-04-16T03:15:25.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.ogdcjk"}]': finished 2024-04-16T03:15:25.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-16T03:15:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: pgmap v1105: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: pgmap v1106: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all mds 2024-04-16T03:15:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: Upgrade: Setting filesystem cephfs Joinable 2024-04-16T03:15:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-16T03:15:25.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: Standby daemon mds.cephfs.smithi151.bdpnxq assigned to filesystem cephfs as rank 0 2024-04-16T03:15:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: Standby daemon mds.cephfs.smithi151.slrqog assigned to filesystem cephfs as rank 1 2024-04-16T03:15:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T03:15:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T03:15:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:replay,1=cephfs.smithi151.slrqog=up:replay} 2 up:standby 2024-04-16T03:15:25.996 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:25 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-16T03:15:26.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: pgmap v1105: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: pgmap v1106: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all mds 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: Upgrade: Setting filesystem cephfs Joinable 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: Standby daemon mds.cephfs.smithi151.bdpnxq assigned to filesystem cephfs as rank 0 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: Standby daemon mds.cephfs.smithi151.slrqog assigned to filesystem cephfs as rank 1 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:replay,1=cephfs.smithi151.slrqog=up:replay} 2 up:standby 2024-04-16T03:15:26.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:25 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-16T03:15:26.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.276+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1156176331 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 msgr2=0x7f82d8100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:26.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.276+0000 7f82ddbc5700 1 --2- 172.21.15.151:0/1156176331 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 0x7f82d8100640 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009910 tx=0x7f82cc009c20 comp rx=0 tx=0).stop 2024-04-16T03:15:26.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.276+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1156176331 shutdown_connections 2024-04-16T03:15:26.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.276+0000 7f82ddbc5700 1 --2- 172.21.15.151:0/1156176331 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 0x7f82d8100640 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:26.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.276+0000 7f82ddbc5700 1 --2- 172.21.15.151:0/1156176331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8104a10 0x7f82d8104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:26.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.276+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1156176331 >> 172.21.15.151:0/1156176331 conn(0x7f82d8075e30 msgr2=0x7f82d80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:26.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.276+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1156176331 shutdown_connections 2024-04-16T03:15:26.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.276+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1156176331 wait complete. 2024-04-16T03:15:26.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.277+0000 7f82ddbc5700 1 Processor -- start 2024-04-16T03:15:26.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.278+0000 7f82ddbc5700 1 -- start start 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.278+0000 7f82ddbc5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 0x7f82d8195660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.278+0000 7f82ddbc5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8104a10 0x7f82d8195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.278+0000 7f82ddbc5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d8196280 con 0x7f82d81001c0 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.278+0000 7f82d77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 0x7f82d8195660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.278+0000 7f82d77fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 0x7f82d8195660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34730/0 (socket says 172.21.15.151:34730) 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.278+0000 7f82d77fe700 1 -- 172.21.15.151:0/1215249080 learned_addr learned my addr 172.21.15.151:0/1215249080 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.278+0000 7f82ddbc5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d819a010 con 0x7f82d8104a10 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.279+0000 7f82d6ffd700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8104a10 0x7f82d8195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.279+0000 7f82d77fe700 1 -- 172.21.15.151:0/1215249080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8104a10 msgr2=0x7f82d8195ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:26.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.279+0000 7f82d77fe700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8104a10 0x7f82d8195ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:26.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.279+0000 7f82d77fe700 1 -- 172.21.15.151:0/1215249080 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc0095f0 con 0x7f82d81001c0 2024-04-16T03:15:26.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.279+0000 7f82d77fe700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 0x7f82d8195660 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d900 tx=0x7f82c800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:26.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.279+0000 7f82d4ff9700 1 -- 172.21.15.151:0/1215249080 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c8004a00 con 0x7f82d81001c0 2024-04-16T03:15:26.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.280+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d819a290 con 0x7f82d81001c0 2024-04-16T03:15:26.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.280+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d819a7e0 con 0x7f82d81001c0 2024-04-16T03:15:26.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.280+0000 7f82d4ff9700 1 -- 172.21.15.151:0/1215249080 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82c80054e0 con 0x7f82d81001c0 2024-04-16T03:15:26.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.281+0000 7f82d4ff9700 1 -- 172.21.15.151:0/1215249080 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c8009d00 con 0x7f82d81001c0 2024-04-16T03:15:26.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.281+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c4005320 con 0x7f82d81001c0 2024-04-16T03:15:26.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.282+0000 7f82d4ff9700 1 -- 172.21.15.151:0/1215249080 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f82c8004b60 con 0x7f82d81001c0 2024-04-16T03:15:26.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.283+0000 7f82d4ff9700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f82c006eab0 0x7f82c0070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:26.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.283+0000 7f82d6ffd700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f82c006eab0 0x7f82c0070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:26.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.283+0000 7f82d4ff9700 1 -- 172.21.15.151:0/1215249080 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f82c808ece0 con 0x7f82d81001c0 2024-04-16T03:15:26.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.284+0000 7f82d6ffd700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f82c006eab0 0x7f82c0070f70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f82d8196c80 tx=0x7f82cc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:26.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.292+0000 7f82d4ff9700 1 -- 172.21.15.151:0/1215249080 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f82c8057b10 con 0x7f82d81001c0 2024-04-16T03:15:26.612 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:26 smithi151 ceph-mon[112233]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-16T03:15:26.612 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-16T03:15:26.613 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:26 smithi151 ceph-mon[112233]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:replay,1=cephfs.smithi151.slrqog=up:replay} 2 up:standby 2024-04-16T03:15:26.613 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:26.613 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:26.613 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:26.613 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:26.613 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:26.613 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:26 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:26.750 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.749+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f82c4000bf0 con 0x7f82c006eab0 2024-04-16T03:15:26.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.751+0000 7f82d4ff9700 1 -- 172.21.15.151:0/1215249080 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f82c4000bf0 con 0x7f82c006eab0 2024-04-16T03:15:26.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f82c006eab0 msgr2=0x7f82c0070f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:26.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f82c006eab0 0x7f82c0070f70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f82d8196c80 tx=0x7f82cc01a040 comp rx=0 tx=0).stop 2024-04-16T03:15:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 msgr2=0x7f82d8195660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 0x7f82d8195660 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d900 tx=0x7f82c800dc10 comp rx=0 tx=0).stop 2024-04-16T03:15:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 shutdown_connections 2024-04-16T03:15:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f82c006eab0 0x7f82c0070f70 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82d81001c0 0x7f82d8195660 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 --2- 172.21.15.151:0/1215249080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8104a10 0x7f82d8195ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 >> 172.21.15.151:0/1215249080 conn(0x7f82d8075e30 msgr2=0x7f82d80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.755+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 shutdown_connections 2024-04-16T03:15:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.756+0000 7f82ddbc5700 1 -- 172.21.15.151:0/1215249080 wait complete. 2024-04-16T03:15:26.773 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:15:26.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:26 smithi153 ceph-mon[87858]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-16T03:15:26.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-16T03:15:26.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:26 smithi153 ceph-mon[87858]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:replay,1=cephfs.smithi151.slrqog=up:replay} 2 up:standby 2024-04-16T03:15:26.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:26.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:26.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:26.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:26.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:26.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:26 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:26.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.937+0000 7f65f55a2700 1 -- 172.21.15.151:0/3953742679 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 msgr2=0x7f65f0106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:26.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.937+0000 7f65f55a2700 1 --2- 172.21.15.151:0/3953742679 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 0x7f65f0106da0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009910 tx=0x7f65e0009c20 comp rx=0 tx=0).stop 2024-04-16T03:15:26.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.938+0000 7f65f55a2700 1 -- 172.21.15.151:0/3953742679 shutdown_connections 2024-04-16T03:15:26.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.938+0000 7f65f55a2700 1 --2- 172.21.15.151:0/3953742679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f01009c0 0x7f65f0100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:26.940 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.938+0000 7f65f55a2700 1 --2- 172.21.15.151:0/3953742679 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 0x7f65f0106da0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:26.940 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.938+0000 7f65f55a2700 1 -- 172.21.15.151:0/3953742679 >> 172.21.15.151:0/3953742679 conn(0x7f65f00fc4d0 msgr2=0x7f65f00fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:26.940 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.939+0000 7f65f55a2700 1 -- 172.21.15.151:0/3953742679 shutdown_connections 2024-04-16T03:15:26.940 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.939+0000 7f65f55a2700 1 -- 172.21.15.151:0/3953742679 wait complete. 2024-04-16T03:15:26.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.941+0000 7f65f55a2700 1 Processor -- start 2024-04-16T03:15:26.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.941+0000 7f65f55a2700 1 -- start start 2024-04-16T03:15:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.942+0000 7f65f55a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f01009c0 0x7f65f019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.942+0000 7f65f55a2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 0x7f65f019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.942+0000 7f65f55a2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f019d060 con 0x7f65f01069c0 2024-04-16T03:15:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.942+0000 7f65f55a2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0196520 con 0x7f65f01009c0 2024-04-16T03:15:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.942+0000 7f65ef7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 0x7f65f019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.942+0000 7f65ef7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 0x7f65f019c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44486/0 (socket says 172.21.15.151:44486) 2024-04-16T03:15:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.942+0000 7f65ef7fe700 1 -- 172.21.15.151:0/3286488123 learned_addr learned my addr 172.21.15.151:0/3286488123 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:15:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.942+0000 7f65effff700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f01009c0 0x7f65f019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.943+0000 7f65effff700 1 -- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 msgr2=0x7f65f019c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.943+0000 7f65effff700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 0x7f65f019c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.943+0000 7f65effff700 1 -- 172.21.15.151:0/3286488123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e00095f0 con 0x7f65f01009c0 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.943+0000 7f65ef7fe700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 0x7f65f019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.944+0000 7f65effff700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f01009c0 0x7f65f019c440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009fd0 tx=0x7f65e001ca50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.944+0000 7f65ed7fa700 1 -- 172.21.15.151:0/3286488123 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65e0021070 con 0x7f65f01009c0 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.944+0000 7f65ed7fa700 1 -- 172.21.15.151:0/3286488123 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65e000b610 con 0x7f65f01009c0 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.944+0000 7f65ed7fa700 1 -- 172.21.15.151:0/3286488123 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65e000f690 con 0x7f65f01009c0 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.945+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f01967a0 con 0x7f65f01009c0 2024-04-16T03:15:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.945+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f0196ba0 con 0x7f65f01009c0 2024-04-16T03:15:26.947 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.946+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f0108fc0 con 0x7f65f01009c0 2024-04-16T03:15:26.948 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.947+0000 7f65ed7fa700 1 -- 172.21.15.151:0/3286488123 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f65e0025070 con 0x7f65f01009c0 2024-04-16T03:15:26.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.948+0000 7f65ed7fa700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65d806ecc0 0x7f65d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:26.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.948+0000 7f65ed7fa700 1 -- 172.21.15.151:0/3286488123 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f65e00942c0 con 0x7f65f01009c0 2024-04-16T03:15:26.949 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.948+0000 7f65ef7fe700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65d806ecc0 0x7f65d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:26.950 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.949+0000 7f65ef7fe700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65d806ecc0 0x7f65d8071180 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f65f0197d10 tx=0x7f65e4009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:26.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:26.953+0000 7f65ed7fa700 1 -- 172.21.15.151:0/3286488123 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f65e0060b80 con 0x7f65f01009c0 2024-04-16T03:15:27.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.420+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f65f0196e70 con 0x7f65d806ecc0 2024-04-16T03:15:27.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.421+0000 7f65ed7fa700 1 -- 172.21.15.151:0/3286488123 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f65f0196e70 con 0x7f65d806ecc0 2024-04-16T03:15:27.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65d806ecc0 msgr2=0x7f65d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:27.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65d806ecc0 0x7f65d8071180 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f65f0197d10 tx=0x7f65e4009260 comp rx=0 tx=0).stop 2024-04-16T03:15:27.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f01009c0 msgr2=0x7f65f019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:27.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f01009c0 0x7f65f019c440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009fd0 tx=0x7f65e001ca50 comp rx=0 tx=0).stop 2024-04-16T03:15:27.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 shutdown_connections 2024-04-16T03:15:27.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f65d806ecc0 0x7f65d8071180 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:27.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f01009c0 0x7f65f019c440 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:27.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 --2- 172.21.15.151:0/3286488123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65f01069c0 0x7f65f019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:27.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 >> 172.21.15.151:0/3286488123 conn(0x7f65f00fc4d0 msgr2=0x7f65f00fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:27.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 shutdown_connections 2024-04-16T03:15:27.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.425+0000 7f65f55a2700 1 -- 172.21.15.151:0/3286488123 wait complete. 2024-04-16T03:15:27.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.560+0000 7f4ccf314700 1 -- 172.21.15.151:0/1700285084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc8105b50 msgr2=0x7f4cc8068410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:27.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.560+0000 7f4ccf314700 1 --2- 172.21.15.151:0/1700285084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc8105b50 0x7f4cc8068410 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009910 tx=0x7f4cc4009c20 comp rx=0 tx=0).stop 2024-04-16T03:15:27.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.560+0000 7f4ccf314700 1 -- 172.21.15.151:0/1700285084 shutdown_connections 2024-04-16T03:15:27.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.560+0000 7f4ccf314700 1 --2- 172.21.15.151:0/1700285084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc8068950 0x7f4cc8068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:27.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.560+0000 7f4ccf314700 1 --2- 172.21.15.151:0/1700285084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc8105b50 0x7f4cc8068410 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:27.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.560+0000 7f4ccf314700 1 -- 172.21.15.151:0/1700285084 >> 172.21.15.151:0/1700285084 conn(0x7f4cc80755a0 msgr2=0x7f4cc80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:27.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.561+0000 7f4ccf314700 1 -- 172.21.15.151:0/1700285084 shutdown_connections 2024-04-16T03:15:27.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.561+0000 7f4ccf314700 1 -- 172.21.15.151:0/1700285084 wait complete. 2024-04-16T03:15:27.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccf314700 1 Processor -- start 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccf314700 1 -- start start 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccf314700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc8068950 0x7f4cc81a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccf314700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc8105b50 0x7f4cc81a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccf314700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc81a3680 con 0x7f4cc8105b50 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccf314700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc819cb30 con 0x7f4cc8068950 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccc8af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc8105b50 0x7f4cc81a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccc8af700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc8105b50 0x7f4cc81a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44512/0 (socket says 172.21.15.151:44512) 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccd0b0700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc8068950 0x7f4cc81a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.562+0000 7f4ccc8af700 1 -- 172.21.15.151:0/2497901264 learned_addr learned my addr 172.21.15.151:0/2497901264 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.563+0000 7f4ccd0b0700 1 -- 172.21.15.151:0/2497901264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc8105b50 msgr2=0x7f4cc81a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.563+0000 7f4ccd0b0700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc8105b50 0x7f4cc81a2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.563+0000 7f4ccd0b0700 1 -- 172.21.15.151:0/2497901264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc40095f0 con 0x7f4cc8068950 2024-04-16T03:15:27.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.563+0000 7f4ccd0b0700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc8068950 0x7f4cc81a2ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4006010 tx=0x7f4cc400bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:27.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.563+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2497901264 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cc401d070 con 0x7f4cc8068950 2024-04-16T03:15:27.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.563+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2497901264 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4cc4005dc0 con 0x7f4cc8068950 2024-04-16T03:15:27.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.563+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2497901264 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cc4017400 con 0x7f4cc8068950 2024-04-16T03:15:27.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.563+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc819cdb0 con 0x7f4cc8068950 2024-04-16T03:15:27.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.563+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc819d320 con 0x7f4cc8068950 2024-04-16T03:15:27.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.565+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc804ea90 con 0x7f4cc8068950 2024-04-16T03:15:27.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.566+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2497901264 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4cc400f6f0 con 0x7f4cc8068950 2024-04-16T03:15:27.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.566+0000 7f4cbe7fc700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4cb406ecc0 0x7f4cb4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:27.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.566+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2497901264 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4cc4090100 con 0x7f4cc8068950 2024-04-16T03:15:27.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.566+0000 7f4ccc8af700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4cb406ecc0 0x7f4cb4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:27.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.567+0000 7f4ccc8af700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4cb406ecc0 0x7f4cb4071180 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8075bd0 tx=0x7f4cb800b500 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:27.572 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.571+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2497901264 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4cc405ca60 con 0x7f4cc8068950 2024-04-16T03:15:27.816 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:27 smithi151 ceph-mon[112233]: pgmap v1107: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:27.816 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:27 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all rgw 2024-04-16T03:15:27.816 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:27 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T03:15:27.816 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:27 smithi151 ceph-mon[112233]: from='client.35996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:15:27.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.912+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cc819d7f0 con 0x7f4cb406ecc0 2024-04-16T03:15:27.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.924+0000 7f4cbe7fc700 1 -- 172.21.15.151:0/2497901264 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4cc819d7f0 con 0x7f4cb406ecc0 2024-04-16T03:15:27.925 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:15:27.925 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (37m) 14m ago 48m 25.7M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:15:27.925 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (48m) 14m ago 48m 10.0M - 18.2.1 5be31c24972a c0c715bb40c8 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (47m) 15m ago 47m 9.94M - 18.2.1 5be31c24972a e542166f14fb 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (34m) 14m ago 48m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (34m) 15m ago 47m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (36m) 14m ago 48m 81.7M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (30m) 14m ago 42m 21.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (30m) 14m ago 42m 19.4M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (15m) 15m ago 42m 14.8M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (30m) 15m ago 42m 20.0M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (39m) 14m ago 49m 513M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (37m) 15m ago 47m 459M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (35m) 14m ago 49m 73.3M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (35m) 15m ago 47m 54.8M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (37m) 14m ago 48m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:15:27.926 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (37m) 15m ago 47m 16.3M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:15:27.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (34m) 14m ago 46m 96.3M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:15:27.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (33m) 14m ago 46m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:15:27.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (32m) 14m ago 45m 87.2M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:15:27.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (32m) 15m ago 45m 114M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:15:27.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (31m) 15m ago 44m 94.3M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:15:27.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (31m) 15m ago 44m 104M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:15:27.927 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (37m) 14m ago 47m 84.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:15:27.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4cb406ecc0 msgr2=0x7f4cb4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4cb406ecc0 0x7f4cb4071180 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8075bd0 tx=0x7f4cb800b500 comp rx=0 tx=0).stop 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc8068950 msgr2=0x7f4cc81a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc8068950 0x7f4cc81a2ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4006010 tx=0x7f4cc400bd70 comp rx=0 tx=0).stop 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 shutdown_connections 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4cb406ecc0 0x7f4cb4071180 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc8068950 0x7f4cc81a2ab0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 --2- 172.21.15.151:0/2497901264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4cc8105b50 0x7f4cc81a2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 >> 172.21.15.151:0/2497901264 conn(0x7f4cc80755a0 msgr2=0x7f4cc81014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 shutdown_connections 2024-04-16T03:15:27.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:27.928+0000 7f4ccf314700 1 -- 172.21.15.151:0/2497901264 wait complete. 2024-04-16T03:15:28.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:27 smithi153 ceph-mon[87858]: pgmap v1107: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-16T03:15:28.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:27 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all rgw 2024-04-16T03:15:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:27 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T03:15:28.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:27 smithi153 ceph-mon[87858]: from='client.35996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:15:28.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.088+0000 7fdef6723700 1 -- 172.21.15.151:0/4291282946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 msgr2=0x7fdef01057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:28.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.088+0000 7fdef6723700 1 --2- 172.21.15.151:0/4291282946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 0x7fdef01057a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fded8009910 tx=0x7fded8009c20 comp rx=0 tx=0).stop 2024-04-16T03:15:28.090 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.089+0000 7fdef6723700 1 -- 172.21.15.151:0/4291282946 shutdown_connections 2024-04-16T03:15:28.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.089+0000 7fdef6723700 1 --2- 172.21.15.151:0/4291282946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 0x7fdef01057a0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.089+0000 7fdef6723700 1 --2- 172.21.15.151:0/4291282946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdef0100da0 0x7fdef0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.089+0000 7fdef6723700 1 -- 172.21.15.151:0/4291282946 >> 172.21.15.151:0/4291282946 conn(0x7fdef00fc630 msgr2=0x7fdef00fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.089+0000 7fdef6723700 1 -- 172.21.15.151:0/4291282946 shutdown_connections 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.089+0000 7fdef6723700 1 -- 172.21.15.151:0/4291282946 wait complete. 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.089+0000 7fdef6723700 1 Processor -- start 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.090+0000 7fdef6723700 1 -- start start 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.090+0000 7fdef6723700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdef0100da0 0x7fdef00728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.090+0000 7fdef6723700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 0x7fdef006d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.090+0000 7fdef6723700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef006dea0 con 0x7fdef0101750 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.090+0000 7fdef6723700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef006e010 con 0x7fdef0100da0 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.090+0000 7fdee7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 0x7fdef006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.090+0000 7fdee7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 0x7fdef006d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44526/0 (socket says 172.21.15.151:44526) 2024-04-16T03:15:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.090+0000 7fdee7fff700 1 -- 172.21.15.151:0/3716698978 learned_addr learned my addr 172.21.15.151:0/3716698978 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:15:28.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.090+0000 7fdeeffff700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdef0100da0 0x7fdef00728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:28.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.091+0000 7fdee7fff700 1 -- 172.21.15.151:0/3716698978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdef0100da0 msgr2=0x7fdef00728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:28.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.091+0000 7fdee7fff700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdef0100da0 0x7fdef00728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.091+0000 7fdee7fff700 1 -- 172.21.15.151:0/3716698978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded80095f0 con 0x7fdef0101750 2024-04-16T03:15:28.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.091+0000 7fdee7fff700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 0x7fdef006d960 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fded8005f40 tx=0x7fded80057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:28.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.091+0000 7fdeedffb700 1 -- 172.21.15.151:0/3716698978 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fded8023cb0 con 0x7fdef0101750 2024-04-16T03:15:28.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.091+0000 7fdeedffb700 1 -- 172.21.15.151:0/3716698978 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fded8023e10 con 0x7fdef0101750 2024-04-16T03:15:28.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.092+0000 7fdeedffb700 1 -- 172.21.15.151:0/3716698978 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fded800f9f0 con 0x7fdef0101750 2024-04-16T03:15:28.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.092+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef006e290 con 0x7fdef0101750 2024-04-16T03:15:28.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.092+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef006e780 con 0x7fdef0101750 2024-04-16T03:15:28.094 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.093+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef01091a0 con 0x7fdef0101750 2024-04-16T03:15:28.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.093+0000 7fdeedffb700 1 -- 172.21.15.151:0/3716698978 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fded8029020 con 0x7fdef0101750 2024-04-16T03:15:28.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.094+0000 7fdeedffb700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdedc06ed10 0x7fdedc0711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:28.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.094+0000 7fdeedffb700 1 -- 172.21.15.151:0/3716698978 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fded8098550 con 0x7fdef0101750 2024-04-16T03:15:28.095 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.094+0000 7fdeeffff700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdedc06ed10 0x7fdedc0711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:28.096 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.095+0000 7fdeeffff700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdedc06ed10 0x7fdedc0711d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdee00106c0 tx=0x7fdee000a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:28.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.103+0000 7fdeedffb700 1 -- 172.21.15.151:0/3716698978 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fded8061360 con 0x7fdef0101750 2024-04-16T03:15:28.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.523+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdef004ea90 con 0x7fdef0101750 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.524+0000 7fdeedffb700 1 -- 172.21.15.151:0/3716698978 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fded8064a20 con 0x7fdef0101750 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:15:28.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:15:28.526 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:15:28.526 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:15:28.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:15:28.526 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:15:28.526 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:15:28.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:15:28.526 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:15:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdedc06ed10 msgr2=0x7fdedc0711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdedc06ed10 0x7fdedc0711d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdee00106c0 tx=0x7fdee000a380 comp rx=0 tx=0).stop 2024-04-16T03:15:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 msgr2=0x7fdef006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:28.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 0x7fdef006d960 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fded8005f40 tx=0x7fded80057e0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 shutdown_connections 2024-04-16T03:15:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fdedc06ed10 0x7fdedc0711d0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdef0100da0 0x7fdef00728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 --2- 172.21.15.151:0/3716698978 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0101750 0x7fdef006d960 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 >> 172.21.15.151:0/3716698978 conn(0x7fdef00fc630 msgr2=0x7fdef00fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 shutdown_connections 2024-04-16T03:15:28.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.526+0000 7fdef6723700 1 -- 172.21.15.151:0/3716698978 wait complete. 2024-04-16T03:15:28.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.659+0000 7fa8eae39700 1 -- 172.21.15.151:0/977313405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e4105a40 msgr2=0x7fa8e4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:28.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.659+0000 7fa8eae39700 1 --2- 172.21.15.151:0/977313405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e4105a40 0x7fa8e4105e20 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009910 tx=0x7fa8d8009c20 comp rx=0 tx=0).stop 2024-04-16T03:15:28.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.660+0000 7fa8eae39700 1 -- 172.21.15.151:0/977313405 shutdown_connections 2024-04-16T03:15:28.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.660+0000 7fa8eae39700 1 --2- 172.21.15.151:0/977313405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e40ffbd0 0x7fa8e4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.660+0000 7fa8eae39700 1 --2- 172.21.15.151:0/977313405 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e4105a40 0x7fa8e4105e20 secure :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009910 tx=0x7fa8d8009c20 comp rx=0 tx=0).stop 2024-04-16T03:15:28.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.660+0000 7fa8eae39700 1 -- 172.21.15.151:0/977313405 >> 172.21.15.151:0/977313405 conn(0x7fa8e40755a0 msgr2=0x7fa8e40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:28.661 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.660+0000 7fa8eae39700 1 -- 172.21.15.151:0/977313405 shutdown_connections 2024-04-16T03:15:28.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.660+0000 7fa8eae39700 1 -- 172.21.15.151:0/977313405 wait complete. 2024-04-16T03:15:28.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.661+0000 7fa8eae39700 1 Processor -- start 2024-04-16T03:15:28.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.661+0000 7fa8eae39700 1 -- start start 2024-04-16T03:15:28.662 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.662+0000 7fa8eae39700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e40ffbd0 0x7fa8e419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:28.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.662+0000 7fa8eae39700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e419ab60 0x7fa8e41946a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:28.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.662+0000 7fa8eae39700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e419b130 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.663 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.662+0000 7fa8e8bd5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e40ffbd0 0x7fa8e419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.662+0000 7fa8e8bd5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e40ffbd0 0x7fa8e419a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44546/0 (socket says 172.21.15.151:44546) 2024-04-16T03:15:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.662+0000 7fa8e8bd5700 1 -- 172.21.15.151:0/4254836717 learned_addr learned my addr 172.21.15.151:0/4254836717 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:15:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.662+0000 7fa8e3fff700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e419ab60 0x7fa8e41946a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.662+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e4194be0 con 0x7fa8e419ab60 2024-04-16T03:15:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.662+0000 7fa8e8bd5700 1 -- 172.21.15.151:0/4254836717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e419ab60 msgr2=0x7fa8e41946a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.663+0000 7fa8e8bd5700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e419ab60 0x7fa8e41946a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.663+0000 7fa8e8bd5700 1 -- 172.21.15.151:0/4254836717 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d80095f0 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.663+0000 7fa8e3fff700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e419ab60 0x7fa8e41946a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T03:15:28.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.663+0000 7fa8e8bd5700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e40ffbd0 0x7fa8e419a620 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009fd0 tx=0x7fa8d800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:28.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.663+0000 7fa8e1ffb700 1 -- 172.21.15.151:0/4254836717 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d801d070 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.664+0000 7fa8e1ffb700 1 -- 172.21.15.151:0/4254836717 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8d8005dc0 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.664+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e4194e60 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.665 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.664+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e4195350 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.666 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.664+0000 7fa8e1ffb700 1 -- 172.21.15.151:0/4254836717 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d8003e20 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.666+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e4108040 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.667 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.666+0000 7fa8e1ffb700 1 -- 172.21.15.151:0/4254836717 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa8d8003680 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.667+0000 7fa8e1ffb700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8d4070ed0 0x7fa8d4073390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:28.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.668+0000 7fa8e1ffb700 1 -- 172.21.15.151:0/4254836717 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa8d805c960 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.671+0000 7fa8e3fff700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8d4070ed0 0x7fa8d4073390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:28.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.674+0000 7fa8e3fff700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8d4070ed0 0x7fa8d4073390 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0011490 tx=0x7fa8d0009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:28.677 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.677+0000 7fa8e1ffb700 1 -- 172.21.15.151:0/4254836717 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa8d805c030 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.970 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:28 smithi151 ceph-mon[112233]: from='client.45423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:15:28.970 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:28 smithi151 ceph-mon[112233]: from='client.45427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:15:28.971 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:28 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3716698978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:28.971 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.969+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa8e4195bb0 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.972 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.971+0000 7fa8e1ffb700 1 -- 172.21.15.151:0/4254836717 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1963 (secure 0 0 0) 0x7fa8d805bbc0 con 0x7fa8e40ffbd0 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:e36 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 36 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T03:15:25.548544+0000 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:15:28.973 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=34416,1=34420} 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:15:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 0 members: 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:34416} state up:replay seq 1 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:34420} state up:replay seq 1 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout:Standby daemons: 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:15:28.975 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{-1:44863} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:15:28.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.975+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8d4070ed0 msgr2=0x7fa8d4073390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:28.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.975+0000 7fa8eae39700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8d4070ed0 0x7fa8d4073390 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0011490 tx=0x7fa8d0009260 comp rx=0 tx=0).stop 2024-04-16T03:15:28.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.975+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e40ffbd0 msgr2=0x7fa8e419a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:28.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.975+0000 7fa8eae39700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e40ffbd0 0x7fa8e419a620 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009fd0 tx=0x7fa8d800f690 comp rx=0 tx=0).stop 2024-04-16T03:15:28.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.976+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 shutdown_connections 2024-04-16T03:15:28.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.976+0000 7fa8eae39700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa8d4070ed0 0x7fa8d4073390 secure :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0011490 tx=0x7fa8d0009260 comp rx=0 tx=0).stop 2024-04-16T03:15:28.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.976+0000 7fa8eae39700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa8e40ffbd0 0x7fa8e419a620 secure :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009fd0 tx=0x7fa8d800f690 comp rx=0 tx=0).stop 2024-04-16T03:15:28.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.976+0000 7fa8eae39700 1 --2- 172.21.15.151:0/4254836717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e419ab60 0x7fa8e41946a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:28.977 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.976+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 >> 172.21.15.151:0/4254836717 conn(0x7fa8e40755a0 msgr2=0x7fa8e40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:28.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.976+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 shutdown_connections 2024-04-16T03:15:28.978 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:28.976+0000 7fa8eae39700 1 -- 172.21.15.151:0/4254836717 wait complete. 2024-04-16T03:15:28.982 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 36 2024-04-16T03:15:29.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:28 smithi153 ceph-mon[87858]: from='client.45423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:15:29.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:28 smithi153 ceph-mon[87858]: from='client.45427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:15:29.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:28 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3716698978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:15:29.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.114+0000 7f976d59d700 1 -- 172.21.15.151:0/1880301252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 msgr2=0x7f9768100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:29.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.114+0000 7f976d59d700 1 --2- 172.21.15.151:0/1880301252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 0x7f9768100740 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9758009910 tx=0x7f9758009c20 comp rx=0 tx=0).stop 2024-04-16T03:15:29.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.114+0000 7f976d59d700 1 -- 172.21.15.151:0/1880301252 shutdown_connections 2024-04-16T03:15:29.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.114+0000 7f976d59d700 1 --2- 172.21.15.151:0/1880301252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9768100d10 0x7f97681046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.114+0000 7f976d59d700 1 --2- 172.21.15.151:0/1880301252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 0x7f9768100740 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.114+0000 7f976d59d700 1 -- 172.21.15.151:0/1880301252 >> 172.21.15.151:0/1880301252 conn(0x7f9768075e30 msgr2=0x7f97680fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:29.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.115+0000 7f976d59d700 1 -- 172.21.15.151:0/1880301252 shutdown_connections 2024-04-16T03:15:29.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.115+0000 7f976d59d700 1 -- 172.21.15.151:0/1880301252 wait complete. 2024-04-16T03:15:29.116 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.115+0000 7f976d59d700 1 Processor -- start 2024-04-16T03:15:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.116+0000 7f976d59d700 1 -- start start 2024-04-16T03:15:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.116+0000 7f976d59d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 0x7f97681955a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.116+0000 7f976d59d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9768100d10 0x7f9768195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.116+0000 7f976d59d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681961c0 con 0x7f9768100360 2024-04-16T03:15:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.116+0000 7f976d59d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768199f50 con 0x7f9768100d10 2024-04-16T03:15:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.116+0000 7f9766ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 0x7f97681955a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.116+0000 7f9766ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 0x7f97681955a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44568/0 (socket says 172.21.15.151:44568) 2024-04-16T03:15:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.116+0000 7f9766ffd700 1 -- 172.21.15.151:0/1782354999 learned_addr learned my addr 172.21.15.151:0/1782354999 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:15:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.116+0000 7f97667fc700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9768100d10 0x7f9768195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.117+0000 7f9766ffd700 1 -- 172.21.15.151:0/1782354999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9768100d10 msgr2=0x7f9768195ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.117+0000 7f9766ffd700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9768100d10 0x7f9768195ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.117+0000 7f9766ffd700 1 -- 172.21.15.151:0/1782354999 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97580095f0 con 0x7f9768100360 2024-04-16T03:15:29.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.117+0000 7f9766ffd700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 0x7f97681955a0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f97580092e0 tx=0x7f975800f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:29.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.117+0000 7f974ffff700 1 -- 172.21.15.151:0/1782354999 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975801d070 con 0x7f9768100360 2024-04-16T03:15:29.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.117+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976819a0f0 con 0x7f9768100360 2024-04-16T03:15:29.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.117+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976819a660 con 0x7f9768100360 2024-04-16T03:15:29.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.117+0000 7f974ffff700 1 -- 172.21.15.151:0/1782354999 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97580052a0 con 0x7f9768100360 2024-04-16T03:15:29.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.117+0000 7f974ffff700 1 -- 172.21.15.151:0/1782354999 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975801d070 con 0x7f9768100360 2024-04-16T03:15:29.119 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.119+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9754005320 con 0x7f9768100360 2024-04-16T03:15:29.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.120+0000 7f974ffff700 1 -- 172.21.15.151:0/1782354999 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f9758003680 con 0x7f9768100360 2024-04-16T03:15:29.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.121+0000 7f974ffff700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f975006ecc0 0x7f9750071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:29.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.121+0000 7f974ffff700 1 -- 172.21.15.151:0/1782354999 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f975808fe80 con 0x7f9768100360 2024-04-16T03:15:29.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.121+0000 7f97667fc700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f975006ecc0 0x7f9750071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:29.122 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.122+0000 7f97667fc700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f975006ecc0 0x7f9750071180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9768196bc0 tx=0x7f975c009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:29.131 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.130+0000 7f974ffff700 1 -- 172.21.15.151:0/1782354999 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9758058c30 con 0x7f9768100360 2024-04-16T03:15:29.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.472+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9754000bf0 con 0x7f975006ecc0 2024-04-16T03:15:29.475 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.474+0000 7f974ffff700 1 -- 172.21.15.151:0/1782354999 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f9754000bf0 con 0x7f975006ecc0 2024-04-16T03:15:29.475 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:15:29.475 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f", 2024-04-16T03:15:29.475 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": true, 2024-04-16T03:15:29.475 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T03:15:29.475 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [ 2024-04-16T03:15:29.475 INFO:teuthology.orchestra.run.smithi151.stdout: "mds", 2024-04-16T03:15:29.475 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr", 2024-04-16T03:15:29.476 INFO:teuthology.orchestra.run.smithi151.stdout: "mon", 2024-04-16T03:15:29.476 INFO:teuthology.orchestra.run.smithi151.stdout: "crash", 2024-04-16T03:15:29.476 INFO:teuthology.orchestra.run.smithi151.stdout: "osd" 2024-04-16T03:15:29.476 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-04-16T03:15:29.476 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": "16/23 daemons upgraded", 2024-04-16T03:15:29.476 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "Currently upgrading ceph-exporter daemons", 2024-04-16T03:15:29.476 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:15:29.476 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:15:29.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.477+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f975006ecc0 msgr2=0x7f9750071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:29.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.477+0000 7f976d59d700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f975006ecc0 0x7f9750071180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9768196bc0 tx=0x7f975c009260 comp rx=0 tx=0).stop 2024-04-16T03:15:29.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.477+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 msgr2=0x7f97681955a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:29.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.477+0000 7f976d59d700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 0x7f97681955a0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f97580092e0 tx=0x7f975800f460 comp rx=0 tx=0).stop 2024-04-16T03:15:29.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.477+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 shutdown_connections 2024-04-16T03:15:29.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.477+0000 7f976d59d700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f975006ecc0 0x7f9750071180 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.477+0000 7f976d59d700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9768100360 0x7f97681955a0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.477+0000 7f976d59d700 1 --2- 172.21.15.151:0/1782354999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9768100d10 0x7f9768195ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.477+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 >> 172.21.15.151:0/1782354999 conn(0x7f9768075e30 msgr2=0x7f97680fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:29.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.478+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 shutdown_connections 2024-04-16T03:15:29.479 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.478+0000 7f976d59d700 1 -- 172.21.15.151:0/1782354999 wait complete. 2024-04-16T03:15:29.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.589+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/3990316506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 msgr2=0x7fe8d40ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:29.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.589+0000 7fe8d9e9d700 1 --2- 172.21.15.151:0/3990316506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 0x7fe8d40ff3f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009910 tx=0x7fe8c8009c20 comp rx=0 tx=0).stop 2024-04-16T03:15:29.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.589+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/3990316506 shutdown_connections 2024-04-16T03:15:29.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.589+0000 7fe8d9e9d700 1 --2- 172.21.15.151:0/3990316506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 0x7fe8d40ff3f0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.589+0000 7fe8d9e9d700 1 --2- 172.21.15.151:0/3990316506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d4105b50 0x7fe8d4105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.589+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/3990316506 >> 172.21.15.151:0/3990316506 conn(0x7fe8d4075e30 msgr2=0x7fe8d40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:29.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.589+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/3990316506 shutdown_connections 2024-04-16T03:15:29.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.589+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/3990316506 wait complete. 2024-04-16T03:15:29.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.589+0000 7fe8d9e9d700 1 Processor -- start 2024-04-16T03:15:29.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d9e9d700 1 -- start start 2024-04-16T03:15:29.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d9e9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 0x7fe8d419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:29.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d9e9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d4105b50 0x7fe8d419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:29.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d9e9d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d419f350 con 0x7fe8d40fef70 2024-04-16T03:15:29.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d9e9d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d4198800 con 0x7fe8d4105b50 2024-04-16T03:15:29.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 0x7fe8d419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:29.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 0x7fe8d419e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44580/0 (socket says 172.21.15.151:44580) 2024-04-16T03:15:29.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d37fe700 1 -- 172.21.15.151:0/2333464505 learned_addr learned my addr 172.21.15.151:0/2333464505 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:15:29.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d2ffd700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d4105b50 0x7fe8d419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:29.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.590+0000 7fe8d37fe700 1 -- 172.21.15.151:0/2333464505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d4105b50 msgr2=0x7fe8d419ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:29.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.591+0000 7fe8d37fe700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d4105b50 0x7fe8d419ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.591+0000 7fe8d37fe700 1 -- 172.21.15.151:0/2333464505 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c80095f0 con 0x7fe8d40fef70 2024-04-16T03:15:29.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.591+0000 7fe8d37fe700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 0x7fe8d419e780 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe8c400d8d0 tx=0x7fe8c400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:29.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.591+0000 7fe8d0ff9700 1 -- 172.21.15.151:0/2333464505 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8c40098a0 con 0x7fe8d40fef70 2024-04-16T03:15:29.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.591+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d4198ae0 con 0x7fe8d40fef70 2024-04-16T03:15:29.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.591+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d4199030 con 0x7fe8d40fef70 2024-04-16T03:15:29.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.591+0000 7fe8d0ff9700 1 -- 172.21.15.151:0/2333464505 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8c4010460 con 0x7fe8d40fef70 2024-04-16T03:15:29.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.591+0000 7fe8d0ff9700 1 -- 172.21.15.151:0/2333464505 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8c401b620 con 0x7fe8d40fef70 2024-04-16T03:15:29.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.592+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c0005320 con 0x7fe8d40fef70 2024-04-16T03:15:29.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.594+0000 7fe8d0ff9700 1 -- 172.21.15.151:0/2333464505 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe8c4009a00 con 0x7fe8d40fef70 2024-04-16T03:15:29.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.595+0000 7fe8d0ff9700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8bc06ecc0 0x7fe8bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:15:29.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.595+0000 7fe8d0ff9700 1 -- 172.21.15.151:0/2333464505 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe8c408f160 con 0x7fe8d40fef70 2024-04-16T03:15:29.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.595+0000 7fe8d2ffd700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8bc06ecc0 0x7fe8bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:15:29.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.596+0000 7fe8d2ffd700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8bc06ecc0 0x7fe8bc071180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe8d41071e0 tx=0x7fe8c801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:15:29.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.598+0000 7fe8d0ff9700 1 -- 172.21.15.151:0/2333464505 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe8c405bac0 con 0x7fe8d40fef70 2024-04-16T03:15:29.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:29 smithi151 ceph-mon[112233]: pgmap v1108: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 13 MiB/s rd, 5 op/s 2024-04-16T03:15:29.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:29 smithi151 ceph-mon[112233]: Upgrade: Updating ceph-exporter.smithi151 (1/2) 2024-04-16T03:15:29.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:29 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:29.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:29 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T03:15:29.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:29 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:15:29.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:29 smithi151 ceph-mon[112233]: Deploying daemon ceph-exporter.smithi151 on smithi151 2024-04-16T03:15:29.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:29 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4254836717' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:15:29.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.984+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe8c0005190 con 0x7fe8d40fef70 2024-04-16T03:15:29.985 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.984+0000 7fe8d0ff9700 1 -- 172.21.15.151:0/2333464505 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4177 (secure 0 0 0) 0x7fe8c405b650 con 0x7fe8d40fef70 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_WARN failed to probe daemons or devices; 1 filesystem is degraded 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: host smithi153 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi153/config 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpbzcg2778:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout:Traceback (most recent call last): 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: return _run_code(code, main_globals, None, 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: exec(code, run_globals) 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5618, in 2024-04-16T03:15:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 5606, in main 2024-04-16T03:15:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 408, in _infer_config 2024-04-16T03:15:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 323, in _infer_fsid 2024-04-16T03:15:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 436, in _infer_image 2024-04-16T03:15:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 310, in _validate_fsid 2024-04-16T03:15:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/__main__.py", line 3359, in command_ceph_volume 2024-04-16T03:15:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: File "/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/cephadm.9c7bad676baf498b80bb47210c4bf0a739099cb1046e0675369c31c5f7f0e09d/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-16T03:15:29.987 INFO:teuthology.orchestra.run.smithi151.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f -e NODE_NAME=smithi153 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/run/ceph:z -v /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpbzcg2778:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f inventory --format=json-pretty --filter-for-batch 2024-04-16T03:15:29.987 INFO:teuthology.orchestra.run.smithi151.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-16T03:15:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: fs cephfs is degraded 2024-04-16T03:15:29.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8bc06ecc0 msgr2=0x7fe8bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:29.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8bc06ecc0 0x7fe8bc071180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe8d41071e0 tx=0x7fe8c801a040 comp rx=0 tx=0).stop 2024-04-16T03:15:29.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 msgr2=0x7fe8d419e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:15:29.989 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 0x7fe8d419e780 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe8c400d8d0 tx=0x7fe8c400dbe0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 shutdown_connections 2024-04-16T03:15:29.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe8bc06ecc0 0x7fe8bc071180 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe8d40fef70 0x7fe8d419e780 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 --2- 172.21.15.151:0/2333464505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8d4105b50 0x7fe8d419ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:15:29.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 >> 172.21.15.151:0/2333464505 conn(0x7fe8d4075e30 msgr2=0x7fe8d4104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:15:29.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 shutdown_connections 2024-04-16T03:15:29.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:15:29.988+0000 7fe8d9e9d700 1 -- 172.21.15.151:0/2333464505 wait complete. 2024-04-16T03:15:30.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:29 smithi153 ceph-mon[87858]: pgmap v1108: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 13 MiB/s rd, 5 op/s 2024-04-16T03:15:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:29 smithi153 ceph-mon[87858]: Upgrade: Updating ceph-exporter.smithi151 (1/2) 2024-04-16T03:15:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:29 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:29 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi151", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T03:15:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:29 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:15:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:29 smithi153 ceph-mon[87858]: Deploying daemon ceph-exporter.smithi151 on smithi151 2024-04-16T03:15:30.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:29 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4254836717' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:15:30.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:30 smithi151 ceph-mon[112233]: from='client.36016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:15:30.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:30 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2333464505' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:15:31.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:30 smithi153 ceph-mon[87858]: from='client.36016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:15:31.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:30 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2333464505' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:15:31.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:31 smithi151 ceph-mon[112233]: pgmap v1109: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 8 op/s 2024-04-16T03:15:32.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:31 smithi153 ceph-mon[87858]: pgmap v1109: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 8 op/s 2024-04-16T03:15:33.834 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:33 smithi151 ceph-mon[112233]: pgmap v1110: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9 op/s 2024-04-16T03:15:34.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:33 smithi153 ceph-mon[87858]: pgmap v1110: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9 op/s 2024-04-16T03:15:34.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:34 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:resolve 2024-04-16T03:15:34.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:34 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:resolve 2024-04-16T03:15:34.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:34 smithi151 ceph-mon[112233]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:resolve,1=cephfs.smithi151.slrqog=up:resolve} 2 up:standby 2024-04-16T03:15:35.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:34 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:resolve 2024-04-16T03:15:35.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:34 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:resolve 2024-04-16T03:15:35.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:34 smithi153 ceph-mon[87858]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:resolve,1=cephfs.smithi151.slrqog=up:resolve} 2 up:standby 2024-04-16T03:15:35.683 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:35 smithi151 ceph-mon[112233]: pgmap v1111: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 8 op/s 2024-04-16T03:15:35.683 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:35 smithi151 ceph-mon[112233]: reconnect by client.14546 192.168.0.1:0/1232858561 after 0.000999987 2024-04-16T03:15:35.683 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:35 smithi151 ceph-mon[112233]: reconnect by client.14546 192.168.0.1:0/1232858561 after 0.000999987 2024-04-16T03:15:35.684 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:35 smithi151 ceph-mon[112233]: reconnect by client.24303 192.168.0.1:0/1797148833 after 0.000999987 2024-04-16T03:15:35.684 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:35 smithi151 ceph-mon[112233]: reconnect by client.24303 192.168.0.1:0/1797148833 after 0.000999987 2024-04-16T03:15:35.684 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:35 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:reconnect 2024-04-16T03:15:35.684 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:35 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:reconnect 2024-04-16T03:15:35.684 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:35 smithi151 ceph-mon[112233]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:reconnect,1=cephfs.smithi151.slrqog=up:reconnect} 2 up:standby 2024-04-16T03:15:36.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:35 smithi153 ceph-mon[87858]: pgmap v1111: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 8 op/s 2024-04-16T03:15:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:35 smithi153 ceph-mon[87858]: reconnect by client.14546 192.168.0.1:0/1232858561 after 0.000999987 2024-04-16T03:15:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:35 smithi153 ceph-mon[87858]: reconnect by client.14546 192.168.0.1:0/1232858561 after 0.000999987 2024-04-16T03:15:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:35 smithi153 ceph-mon[87858]: reconnect by client.24303 192.168.0.1:0/1797148833 after 0.000999987 2024-04-16T03:15:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:35 smithi153 ceph-mon[87858]: reconnect by client.24303 192.168.0.1:0/1797148833 after 0.000999987 2024-04-16T03:15:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:35 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:reconnect 2024-04-16T03:15:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:35 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:reconnect 2024-04-16T03:15:36.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:35 smithi153 ceph-mon[87858]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:reconnect,1=cephfs.smithi151.slrqog=up:reconnect} 2 up:standby 2024-04-16T03:15:36.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:36 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:rejoin 2024-04-16T03:15:36.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:36 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:rejoin 2024-04-16T03:15:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:36 smithi151 ceph-mon[112233]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:rejoin,1=cephfs.smithi151.slrqog=up:rejoin} 2 up:standby 2024-04-16T03:15:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:36 smithi151 ceph-mon[112233]: daemon mds.cephfs.smithi151.bdpnxq is now active in filesystem cephfs as rank 0 2024-04-16T03:15:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:36 smithi151 ceph-mon[112233]: daemon mds.cephfs.smithi151.slrqog is now active in filesystem cephfs as rank 1 2024-04-16T03:15:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:36 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:36.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:36 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:37.017 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:36 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:rejoin 2024-04-16T03:15:37.017 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:36 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:rejoin 2024-04-16T03:15:37.017 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:36 smithi153 ceph-mon[87858]: fsmap cephfs:2/2 {0=cephfs.smithi151.bdpnxq=up:rejoin,1=cephfs.smithi151.slrqog=up:rejoin} 2 up:standby 2024-04-16T03:15:37.017 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:36 smithi153 ceph-mon[87858]: daemon mds.cephfs.smithi151.bdpnxq is now active in filesystem cephfs as rank 0 2024-04-16T03:15:37.017 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:36 smithi153 ceph-mon[87858]: daemon mds.cephfs.smithi151.slrqog is now active in filesystem cephfs as rank 1 2024-04-16T03:15:37.017 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:36 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:37.017 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:36 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:37.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:37 smithi151 ceph-mon[112233]: pgmap v1112: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-04-16T03:15:37.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:37 smithi151 ceph-mon[112233]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T03:15:37.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:37 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:active 2024-04-16T03:15:37.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:37 smithi151 ceph-mon[112233]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:active 2024-04-16T03:15:37.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:37 smithi151 ceph-mon[112233]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby 2024-04-16T03:15:37.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:37 smithi151 ceph-mon[112233]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T03:15:38.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:37 smithi153 ceph-mon[87858]: pgmap v1112: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-04-16T03:15:38.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:37 smithi153 ceph-mon[87858]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T03:15:38.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:37 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] up:active 2024-04-16T03:15:38.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:37 smithi153 ceph-mon[87858]: mds.? [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] up:active 2024-04-16T03:15:38.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:37 smithi153 ceph-mon[87858]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby 2024-04-16T03:15:38.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:37 smithi153 ceph-mon[87858]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T03:15:40.126 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:39 smithi153 ceph-mon[87858]: pgmap v1113: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-04-16T03:15:40.126 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:40.127 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:15:40.127 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:40.127 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T03:15:40.127 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:15:40.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:39 smithi151 ceph-mon[112233]: pgmap v1113: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-04-16T03:15:40.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:40.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:15:40.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:40.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-16T03:15:40.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:15:41.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:40 smithi151 ceph-mon[112233]: Upgrade: Updating ceph-exporter.smithi153 (2/2) 2024-04-16T03:15:41.144 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:40 smithi151 ceph-mon[112233]: Deploying daemon ceph-exporter.smithi153 on smithi153 2024-04-16T03:15:41.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:40 smithi153 ceph-mon[87858]: Upgrade: Updating ceph-exporter.smithi153 (2/2) 2024-04-16T03:15:41.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:40 smithi153 ceph-mon[87858]: Deploying daemon ceph-exporter.smithi153 on smithi153 2024-04-16T03:15:42.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:41 smithi153 ceph-mon[87858]: pgmap v1114: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 3.8 KiB/s wr, 17 op/s 2024-04-16T03:15:42.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:41 smithi151 ceph-mon[112233]: pgmap v1114: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 3.8 KiB/s wr, 17 op/s 2024-04-16T03:15:44.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:43 smithi151 ceph-mon[112233]: pgmap v1115: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 3.8 KiB/s wr, 15 op/s 2024-04-16T03:15:44.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:43 smithi153 ceph-mon[87858]: pgmap v1115: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 3.8 KiB/s wr, 15 op/s 2024-04-16T03:15:46.214 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:45 smithi153 ceph-mon[87858]: pgmap v1116: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 8.8 KiB/s wr, 18 op/s 2024-04-16T03:15:46.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:45 smithi151 ceph-mon[112233]: pgmap v1116: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 8.8 KiB/s wr, 18 op/s 2024-04-16T03:15:48.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:47 smithi151 ceph-mon[112233]: pgmap v1117: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 10 KiB/s wr, 19 op/s 2024-04-16T03:15:48.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:47 smithi153 ceph-mon[87858]: pgmap v1117: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 10 KiB/s wr, 19 op/s 2024-04-16T03:15:50.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:49 smithi151 ceph-mon[112233]: pgmap v1118: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 10 KiB/s wr, 19 op/s 2024-04-16T03:15:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:15:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:49 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:49 smithi153 ceph-mon[87858]: pgmap v1118: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 10 KiB/s wr, 19 op/s 2024-04-16T03:15:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:15:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:50.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:49 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:51.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:51 smithi151 ceph-mon[112233]: pgmap v1119: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 9.3 KiB/s wr, 12 op/s 2024-04-16T03:15:52.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:51 smithi153 ceph-mon[87858]: pgmap v1119: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 9.3 KiB/s wr, 12 op/s 2024-04-16T03:15:53.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:53 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:53.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:53 smithi151 ceph-mon[112233]: pgmap v1120: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 9.6 MiB/s rd, 6.7 KiB/s wr, 7 op/s 2024-04-16T03:15:53.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:53.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:53 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:53.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:53 smithi153 ceph-mon[87858]: pgmap v1120: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 9.6 MiB/s rd, 6.7 KiB/s wr, 7 op/s 2024-04-16T03:15:54.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:54.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:54.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:15:54.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:54.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:54.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:15:55.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:55 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:55.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:55 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:55.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:55 smithi153 ceph-mon[87858]: pgmap v1121: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.3 MiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-04-16T03:15:55.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:55 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:55.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:55 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:55.745 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:55 smithi151 ceph-mon[112233]: pgmap v1121: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.3 MiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-04-16T03:15:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:56 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:56 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:56.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:56 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:56.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:56 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:57.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:57 smithi151 ceph-mon[112233]: pgmap v1122: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-16T03:15:57.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:57 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:57.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:15:57 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:58.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:57 smithi153 ceph-mon[87858]: pgmap v1122: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-04-16T03:15:58.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:57 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:15:58.021 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:15:57 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:00.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.118+0000 7f1c1659e700 1 -- 172.21.15.151:0/4210497102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101069a0 msgr2=0x7f1c10106d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:00.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.118+0000 7f1c1659e700 1 --2- 172.21.15.151:0/4210497102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101069a0 0x7f1c10106d80 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009910 tx=0x7f1c0c009c20 comp rx=0 tx=0).stop 2024-04-16T03:16:00.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.119+0000 7f1c1659e700 1 -- 172.21.15.151:0/4210497102 shutdown_connections 2024-04-16T03:16:00.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.119+0000 7f1c1659e700 1 --2- 172.21.15.151:0/4210497102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c101009a0 0x7f1c10100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:00.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.119+0000 7f1c1659e700 1 --2- 172.21.15.151:0/4210497102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101069a0 0x7f1c10106d80 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:00.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.119+0000 7f1c1659e700 1 -- 172.21.15.151:0/4210497102 >> 172.21.15.151:0/4210497102 conn(0x7f1c100fc490 msgr2=0x7f1c100fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:00.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.119+0000 7f1c1659e700 1 -- 172.21.15.151:0/4210497102 shutdown_connections 2024-04-16T03:16:00.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.119+0000 7f1c1659e700 1 -- 172.21.15.151:0/4210497102 wait complete. 2024-04-16T03:16:00.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.119+0000 7f1c1659e700 1 Processor -- start 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1659e700 1 -- start start 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1659e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101009a0 0x7f1c10073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1659e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c101069a0 0x7f1c100735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1659e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c10073ba0 con 0x7f1c101009a0 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1659e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c10073d10 con 0x7f1c101069a0 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1559c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101009a0 0x7f1c10073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1559c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101009a0 0x7f1c10073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50146/0 (socket says 172.21.15.151:50146) 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1559c700 1 -- 172.21.15.151:0/672422452 learned_addr learned my addr 172.21.15.151:0/672422452 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c14d9b700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c101069a0 0x7f1c100735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1559c700 1 -- 172.21.15.151:0/672422452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c101069a0 msgr2=0x7f1c100735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1559c700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c101069a0 0x7f1c100735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1559c700 1 -- 172.21.15.151:0/672422452 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c0095f0 con 0x7f1c101009a0 2024-04-16T03:16:00.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.120+0000 7f1c1559c700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101009a0 0x7f1c10073090 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c005f50 tx=0x7f1c0c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:00.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.122+0000 7f1c067fc700 1 -- 172.21.15.151:0/672422452 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c0c01d070 con 0x7f1c101009a0 2024-04-16T03:16:00.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.122+0000 7f1c067fc700 1 -- 172.21.15.151:0/672422452 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c0c00b610 con 0x7f1c101009a0 2024-04-16T03:16:00.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.122+0000 7f1c067fc700 1 -- 172.21.15.151:0/672422452 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c0c022470 con 0x7f1c101009a0 2024-04-16T03:16:00.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.123+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c101a27a0 con 0x7f1c101009a0 2024-04-16T03:16:00.125 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.123+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c101a2b50 con 0x7f1c101009a0 2024-04-16T03:16:00.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.124+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c10108fa0 con 0x7f1c101009a0 2024-04-16T03:16:00.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.124+0000 7f1c067fc700 1 -- 172.21.15.151:0/672422452 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1c0c003780 con 0x7f1c101009a0 2024-04-16T03:16:00.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.125+0000 7f1c067fc700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1bfc06ecc0 0x7f1bfc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:00.126 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.125+0000 7f1c067fc700 1 -- 172.21.15.151:0/672422452 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1c0c090140 con 0x7f1c101009a0 2024-04-16T03:16:00.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.125+0000 7f1c14d9b700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1bfc06ecc0 0x7f1bfc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:00.127 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.125+0000 7f1c14d9b700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1bfc06ecc0 0x7f1bfc071180 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1c10074880 tx=0x7f1c00009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:00.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.133+0000 7f1c067fc700 1 -- 172.21.15.151:0/672422452 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1c0c058f70 con 0x7f1c101009a0 2024-04-16T03:16:00.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.373+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c1002cce0 con 0x7f1bfc06ecc0 2024-04-16T03:16:00.374 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: pgmap v1123: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 3 op/s 2024-04-16T03:16:00.374 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:00.374 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: pgmap v1124: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.375 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-16T03:16:00.376 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.374+0000 7f1c067fc700 1 -- 172.21.15.151:0/672422452 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f1c1002cce0 con 0x7f1bfc06ecc0 2024-04-16T03:16:00.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1bfc06ecc0 msgr2=0x7f1bfc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:00.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1bfc06ecc0 0x7f1bfc071180 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1c10074880 tx=0x7f1c00009260 comp rx=0 tx=0).stop 2024-04-16T03:16:00.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101009a0 msgr2=0x7f1c10073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:00.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101009a0 0x7f1c10073090 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c005f50 tx=0x7f1c0c00f690 comp rx=0 tx=0).stop 2024-04-16T03:16:00.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 shutdown_connections 2024-04-16T03:16:00.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1bfc06ecc0 0x7f1bfc071180 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:00.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c101009a0 0x7f1c10073090 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:00.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 --2- 172.21.15.151:0/672422452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c101069a0 0x7f1c100735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:00.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 >> 172.21.15.151:0/672422452 conn(0x7f1c100fc490 msgr2=0x7f1c100fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:00.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 shutdown_connections 2024-04-16T03:16:00.378 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:00.376+0000 7f1c1659e700 1 -- 172.21.15.151:0/672422452 wait complete. 2024-04-16T03:16:00.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: pgmap v1123: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 3 op/s 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: pgmap v1124: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:00.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-16T03:16:01.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph orch ps' 2024-04-16T03:16:01.311 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:16:01.434 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: Upgrade: Setting filesystem cephfs Joinable 2024-04-16T03:16:01.434 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: Cluster is now healthy 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all ceph-exporter 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi151"}]: dispatch 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi151"}]': finished 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi153"}]: dispatch 2024-04-16T03:16:01.435 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi153"}]': finished 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all iscsi 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all nfs 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: Upgrade: Setting container_image for all nvmeof 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.436 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: Upgrade: Finalizing container_image settings 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T03:16:01.437 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.438 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:16:01.439 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: Upgrade: Setting filesystem cephfs Joinable 2024-04-16T03:16:01.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-16T03:16:01.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: Cluster is now healthy 2024-04-16T03:16:01.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-16T03:16:01.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: fsmap cephfs:2 {0=cephfs.smithi151.bdpnxq=up:active,1=cephfs.smithi151.slrqog=up:active} 2 up:standby-replay 2024-04-16T03:16:01.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all ceph-exporter 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi151"}]: dispatch 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi151"}]': finished 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi153"}]: dispatch 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi153"}]': finished 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all iscsi 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all nfs 2024-04-16T03:16:01.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: Upgrade: Setting container_image for all nvmeof 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: Upgrade: Finalizing container_image settings 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T03:16:01.523 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T03:16:01.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:16:01.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:16:01.526 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:16:01.526 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:02.315 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:02 smithi151 ceph-mon[112233]: Upgrade: Complete! 2024-04-16T03:16:02.315 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:02 smithi151 ceph-mon[112233]: from='client.36028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:16:02.316 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:02 smithi151 ceph-mon[112233]: pgmap v1125: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T03:16:02.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:02 smithi153 ceph-mon[87858]: Upgrade: Complete! 2024-04-16T03:16:02.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:02 smithi153 ceph-mon[87858]: from='client.36028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:16:02.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:02 smithi153 ceph-mon[87858]: pgmap v1125: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T03:16:03.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.040+0000 7f72f4af4700 1 -- 172.21.15.151:0/2781369865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f0105a60 msgr2=0x7f72f0105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:03.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.040+0000 7f72f4af4700 1 --2- 172.21.15.151:0/2781369865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f0105a60 0x7f72f0105e40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009910 tx=0x7f72e0009c20 comp rx=0 tx=0).stop 2024-04-16T03:16:03.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.040+0000 7f72f4af4700 1 -- 172.21.15.151:0/2781369865 shutdown_connections 2024-04-16T03:16:03.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.040+0000 7f72f4af4700 1 --2- 172.21.15.151:0/2781369865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72f00ffa60 0x7f72f00ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:03.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.040+0000 7f72f4af4700 1 --2- 172.21.15.151:0/2781369865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f0105a60 0x7f72f0105e40 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:03.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.040+0000 7f72f4af4700 1 -- 172.21.15.151:0/2781369865 >> 172.21.15.151:0/2781369865 conn(0x7f72f00fb550 msgr2=0x7f72f00fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:03.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.040+0000 7f72f4af4700 1 -- 172.21.15.151:0/2781369865 shutdown_connections 2024-04-16T03:16:03.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.040+0000 7f72f4af4700 1 -- 172.21.15.151:0/2781369865 wait complete. 2024-04-16T03:16:03.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.040+0000 7f72f4af4700 1 Processor -- start 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72f4af4700 1 -- start start 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72f4af4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f00ffa60 0x7f72f0197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72f4af4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72f0105a60 0x7f72f0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72f4af4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f01981b0 con 0x7f72f00ffa60 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72f4af4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f0198320 con 0x7f72f0105a60 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72ee59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f00ffa60 0x7f72f0197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72ee59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f00ffa60 0x7f72f0197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50160/0 (socket says 172.21.15.151:50160) 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72ee59c700 1 -- 172.21.15.151:0/925195773 learned_addr learned my addr 172.21.15.151:0/925195773 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72edd9b700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72f0105a60 0x7f72f0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72ee59c700 1 -- 172.21.15.151:0/925195773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72f0105a60 msgr2=0x7f72f0197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72ee59c700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72f0105a60 0x7f72f0197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72ee59c700 1 -- 172.21.15.151:0/925195773 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00095f0 con 0x7f72f00ffa60 2024-04-16T03:16:03.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72edd9b700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72f0105a60 0x7f72f0197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72ee59c700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f00ffa60 0x7f72f0197590 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f72e0006010 tx=0x7f72e000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72df7fe700 1 -- 172.21.15.151:0/925195773 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72e001d070 con 0x7f72f00ffa60 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72df7fe700 1 -- 172.21.15.151:0/925195773 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72e000f460 con 0x7f72f00ffa60 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72df7fe700 1 -- 172.21.15.151:0/925195773 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72e0017630 con 0x7f72f00ffa60 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.041+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f01a0600 con 0x7f72f00ffa60 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.042+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f01a0aa0 con 0x7f72f00ffa60 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.042+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f0108060 con 0x7f72f00ffa60 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.043+0000 7f72df7fe700 1 -- 172.21.15.151:0/925195773 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f72e000f5d0 con 0x7f72f00ffa60 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.044+0000 7f72df7fe700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f72d806ecc0 0x7f72d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.044+0000 7f72df7fe700 1 -- 172.21.15.151:0/925195773 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f72e0090330 con 0x7f72f00ffa60 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.044+0000 7f72edd9b700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f72d806ecc0 0x7f72d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.044+0000 7f72edd9b700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f72d806ecc0 0x7f72d8071180 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f72e4005fd0 tx=0x7f72e4005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:03.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.049+0000 7f72df7fe700 1 -- 172.21.15.151:0/925195773 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f72e00590e0 con 0x7f72f00ffa60 2024-04-16T03:16:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.256+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f72f00611d0 con 0x7f72d806ecc0 2024-04-16T03:16:03.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.267+0000 7f72df7fe700 1 -- 172.21.15.151:0/925195773 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f72f00611d0 con 0x7f72d806ecc0 2024-04-16T03:16:03.269 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:16:03.269 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (37m) 8s ago 48m 24.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:16:03.269 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (27s) 8s ago 49m 7922k - 19.0.0-3084-g87dbfb63 a452a57266c7 d31666f7bbb4 2024-04-16T03:16:03.269 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (16s) 11s ago 48m 7109k - 19.0.0-3084-g87dbfb63 a452a57266c7 b064f1a28cbf 2024-04-16T03:16:03.269 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (35m) 8s ago 49m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:16:03.269 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (35m) 11s ago 48m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:16:03.269 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (37m) 8s ago 48m 81.8M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:16:03.269 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (31m) 8s ago 43m 69.2M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (31m) 8s ago 43m 68.2M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (15m) 11s ago 43m 69.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (30m) 11s ago 43m 68.1M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (39m) 8s ago 50m 518M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (38m) 11s ago 47m 460M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (36m) 8s ago 50m 87.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (35m) 11s ago 47m 62.0M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (38m) 8s ago 49m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (38m) 11s ago 47m 16.2M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (34m) 8s ago 47m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:16:03.270 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (34m) 8s ago 46m 115M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:16:03.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (33m) 8s ago 46m 95.4M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:16:03.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (32m) 11s ago 45m 126M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:16:03.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (32m) 11s ago 45m 101M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:16:03.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (31m) 11s ago 44m 113M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:16:03.271 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (37m) 8s ago 48m 92.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f72d806ecc0 msgr2=0x7f72d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f72d806ecc0 0x7f72d8071180 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f72e4005fd0 tx=0x7f72e4005ee0 comp rx=0 tx=0).stop 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f00ffa60 msgr2=0x7f72f0197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f00ffa60 0x7f72f0197590 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f72e0006010 tx=0x7f72e000be20 comp rx=0 tx=0).stop 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 shutdown_connections 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f72d806ecc0 0x7f72d8071180 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72f00ffa60 0x7f72f0197590 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 --2- 172.21.15.151:0/925195773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72f0105a60 0x7f72f0197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 >> 172.21.15.151:0/925195773 conn(0x7f72f00fb550 msgr2=0x7f72f00fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 shutdown_connections 2024-04-16T03:16:03.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:03.269+0000 7f72f4af4700 1 -- 172.21.15.151:0/925195773 wait complete. 2024-04-16T03:16:03.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph orch upgrade status' 2024-04-16T03:16:04.161 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:16:04.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:04 smithi151 ceph-mon[112233]: pgmap v1126: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-16T03:16:04.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:04 smithi153 ceph-mon[87858]: pgmap v1126: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-16T03:16:05.319 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:05 smithi151 ceph-mon[112233]: from='client.36032 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:16:05.319 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:05 smithi153 ceph-mon[87858]: from='client.36032 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:16:05.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:05.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.883+0000 7fc07197b700 1 -- 172.21.15.151:0/3804240867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ffec0 msgr2=0x7fc06c1057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:05.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.883+0000 7fc07197b700 1 --2- 172.21.15.151:0/3804240867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ffec0 0x7fc06c1057b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc054009910 tx=0x7fc054009c20 comp rx=0 tx=0).stop 2024-04-16T03:16:05.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.883+0000 7fc07197b700 1 -- 172.21.15.151:0/3804240867 shutdown_connections 2024-04-16T03:16:05.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.883+0000 7fc07197b700 1 --2- 172.21.15.151:0/3804240867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ffec0 0x7fc06c1057b0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:05.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.883+0000 7fc07197b700 1 --2- 172.21.15.151:0/3804240867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc06c0ff510 0x7fc06c0ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:05.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.883+0000 7fc07197b700 1 -- 172.21.15.151:0/3804240867 >> 172.21.15.151:0/3804240867 conn(0x7fc06c074e00 msgr2=0x7fc06c075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:05.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.883+0000 7fc07197b700 1 -- 172.21.15.151:0/3804240867 shutdown_connections 2024-04-16T03:16:05.887 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.883+0000 7fc07197b700 1 -- 172.21.15.151:0/3804240867 wait complete. 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.883+0000 7fc07197b700 1 Processor -- start 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc07197b700 1 -- start start 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc07197b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ff510 0x7fc06c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc07197b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc06c0ffec0 0x7fc06c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc07197b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c19d110 con 0x7fc06c0ff510 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc07197b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c1965c0 con 0x7fc06c0ffec0 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc06affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ff510 0x7fc06c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc06affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ff510 0x7fc06c19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50180/0 (socket says 172.21.15.151:50180) 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc06affd700 1 -- 172.21.15.151:0/4283427287 learned_addr learned my addr 172.21.15.151:0/4283427287 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc063fff700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc06c0ffec0 0x7fc06c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc06affd700 1 -- 172.21.15.151:0/4283427287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc06c0ffec0 msgr2=0x7fc06c19ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc06affd700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc06c0ffec0 0x7fc06c19ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc06affd700 1 -- 172.21.15.151:0/4283427287 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0540095f0 con 0x7fc06c0ff510 2024-04-16T03:16:05.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc06affd700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ff510 0x7fc06c19c540 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00d8d0 tx=0x7fc05c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:05.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.884+0000 7fc070979700 1 -- 172.21.15.151:0/4283427287 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc05c009940 con 0x7fc06c0ff510 2024-04-16T03:16:05.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.885+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06c1968a0 con 0x7fc06c0ff510 2024-04-16T03:16:05.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.885+0000 7fc070979700 1 -- 172.21.15.151:0/4283427287 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc05c010460 con 0x7fc06c0ff510 2024-04-16T03:16:05.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.885+0000 7fc070979700 1 -- 172.21.15.151:0/4283427287 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc05c00f5f0 con 0x7fc06c0ff510 2024-04-16T03:16:05.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.885+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06c196df0 con 0x7fc06c0ff510 2024-04-16T03:16:05.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.886+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc06c109170 con 0x7fc06c0ff510 2024-04-16T03:16:05.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.886+0000 7fc070979700 1 -- 172.21.15.151:0/4283427287 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc05c0105d0 con 0x7fc06c0ff510 2024-04-16T03:16:05.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.886+0000 7fc070979700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc05806ecc0 0x7fc058071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:05.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.887+0000 7fc070979700 1 -- 172.21.15.151:0/4283427287 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc05c08ee80 con 0x7fc06c0ff510 2024-04-16T03:16:05.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.887+0000 7fc063fff700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc05806ecc0 0x7fc058071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:05.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.887+0000 7fc063fff700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc05806ecc0 0x7fc058071180 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc06c1076e0 tx=0x7fc05401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:05.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:05.891+0000 7fc070979700 1 -- 172.21.15.151:0/4283427287 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc05c05b760 con 0x7fc06c0ff510 2024-04-16T03:16:06.120 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.118+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc06c1970d0 con 0x7fc05806ecc0 2024-04-16T03:16:06.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.119+0000 7fc070979700 1 -- 172.21.15.151:0/4283427287 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fc06c1970d0 con 0x7fc05806ecc0 2024-04-16T03:16:06.121 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:16:06.121 INFO:teuthology.orchestra.run.smithi151.stdout: "target_image": null, 2024-04-16T03:16:06.121 INFO:teuthology.orchestra.run.smithi151.stdout: "in_progress": false, 2024-04-16T03:16:06.121 INFO:teuthology.orchestra.run.smithi151.stdout: "which": "", 2024-04-16T03:16:06.121 INFO:teuthology.orchestra.run.smithi151.stdout: "services_complete": [], 2024-04-16T03:16:06.121 INFO:teuthology.orchestra.run.smithi151.stdout: "progress": null, 2024-04-16T03:16:06.122 INFO:teuthology.orchestra.run.smithi151.stdout: "message": "", 2024-04-16T03:16:06.122 INFO:teuthology.orchestra.run.smithi151.stdout: "is_paused": false 2024-04-16T03:16:06.122 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:16:06.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.121+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc05806ecc0 msgr2=0x7fc058071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.121+0000 7fc07197b700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc05806ecc0 0x7fc058071180 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc06c1076e0 tx=0x7fc05401a040 comp rx=0 tx=0).stop 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.121+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ff510 msgr2=0x7fc06c19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.121+0000 7fc07197b700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ff510 0x7fc06c19c540 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00d8d0 tx=0x7fc05c00dc90 comp rx=0 tx=0).stop 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.121+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 shutdown_connections 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.121+0000 7fc07197b700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc05806ecc0 0x7fc058071180 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.121+0000 7fc07197b700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06c0ff510 0x7fc06c19c540 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.121+0000 7fc07197b700 1 --2- 172.21.15.151:0/4283427287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc06c0ffec0 0x7fc06c19ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.121+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 >> 172.21.15.151:0/4283427287 conn(0x7fc06c074e00 msgr2=0x7fc06c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.122+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 shutdown_connections 2024-04-16T03:16:06.124 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:06.122+0000 7fc07197b700 1 -- 172.21.15.151:0/4283427287 wait complete. 2024-04-16T03:16:06.158 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:06 smithi151 ceph-mon[112233]: pgmap v1127: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-16T03:16:06.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:06 smithi153 ceph-mon[87858]: pgmap v1127: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-16T03:16:06.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph health detail' 2024-04-16T03:16:07.091 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:16:08.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:08 smithi151 ceph-mon[112233]: from='client.36036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:16:08.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:08 smithi151 ceph-mon[112233]: pgmap v1128: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T03:16:08.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:08 smithi153 ceph-mon[87858]: from='client.36036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:16:08.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:08 smithi153 ceph-mon[87858]: pgmap v1128: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T03:16:08.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.765+0000 7f627621a700 1 -- 172.21.15.151:0/186788925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6270105a40 msgr2=0x7f6270105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:08.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.765+0000 7f627621a700 1 --2- 172.21.15.151:0/186788925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6270105a40 0x7f6270105e20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6264009910 tx=0x7f6264009c20 comp rx=0 tx=0).stop 2024-04-16T03:16:08.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.765+0000 7f627621a700 1 -- 172.21.15.151:0/186788925 shutdown_connections 2024-04-16T03:16:08.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.765+0000 7f627621a700 1 --2- 172.21.15.151:0/186788925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62700ffbd0 0x7f6270100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:08.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.765+0000 7f627621a700 1 --2- 172.21.15.151:0/186788925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6270105a40 0x7f6270105e20 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:08.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.765+0000 7f627621a700 1 -- 172.21.15.151:0/186788925 >> 172.21.15.151:0/186788925 conn(0x7f62700755a0 msgr2=0x7f62700759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:08.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.765+0000 7f627621a700 1 -- 172.21.15.151:0/186788925 shutdown_connections 2024-04-16T03:16:08.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.765+0000 7f627621a700 1 -- 172.21.15.151:0/186788925 wait complete. 2024-04-16T03:16:08.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.766+0000 7f627621a700 1 Processor -- start 2024-04-16T03:16:08.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.766+0000 7f627621a700 1 -- start start 2024-04-16T03:16:08.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.766+0000 7f627621a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f62700ffbd0 0x7f6270101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:08.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.766+0000 7f627621a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6270105a40 0x7f6270101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:08.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.766+0000 7f627621a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6270102120 con 0x7f62700ffbd0 2024-04-16T03:16:08.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.766+0000 7f627621a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6270102290 con 0x7f6270105a40 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.766+0000 7f626f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f62700ffbd0 0x7f6270101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f62700ffbd0 0x7f6270101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58620/0 (socket says 172.21.15.151:58620) 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626f7fe700 1 -- 172.21.15.151:0/2269119266 learned_addr learned my addr 172.21.15.151:0/2269119266 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626effd700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6270105a40 0x7f6270101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626f7fe700 1 -- 172.21.15.151:0/2269119266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6270105a40 msgr2=0x7f6270101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626f7fe700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6270105a40 0x7f6270101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626f7fe700 1 -- 172.21.15.151:0/2269119266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640095f0 con 0x7f62700ffbd0 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626effd700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6270105a40 0x7f6270101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626f7fe700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f62700ffbd0 0x7f6270101610 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f62640092e0 tx=0x7f626400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626cff9700 1 -- 172.21.15.151:0/2269119266 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f626401d070 con 0x7f62700ffbd0 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626cff9700 1 -- 172.21.15.151:0/2269119266 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6264005e00 con 0x7f62700ffbd0 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62701a28c0 con 0x7f62700ffbd0 2024-04-16T03:16:08.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f626cff9700 1 -- 172.21.15.151:0/2269119266 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6264017400 con 0x7f62700ffbd0 2024-04-16T03:16:08.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.767+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62701a2c70 con 0x7f62700ffbd0 2024-04-16T03:16:08.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.768+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6270109050 con 0x7f62700ffbd0 2024-04-16T03:16:08.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.769+0000 7f626cff9700 1 -- 172.21.15.151:0/2269119266 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f626400f460 con 0x7f62700ffbd0 2024-04-16T03:16:08.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.769+0000 7f626cff9700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f625806ecc0 0x7f6258071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:08.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.769+0000 7f626cff9700 1 -- 172.21.15.151:0/2269119266 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6264090390 con 0x7f62700ffbd0 2024-04-16T03:16:08.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.769+0000 7f626effd700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f625806ecc0 0x7f6258071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:08.773 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.770+0000 7f626effd700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f625806ecc0 0x7f6258071180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6260005950 tx=0x7f62600058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:08.776 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:08.774+0000 7f626cff9700 1 -- 172.21.15.151:0/2269119266 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6264059140 con 0x7f62700ffbd0 2024-04-16T03:16:09.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.057+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6270108a30 con 0x7f62700ffbd0 2024-04-16T03:16:09.059 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.057+0000 7f626cff9700 1 -- 172.21.15.151:0/2269119266 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f626405c800 con 0x7f62700ffbd0 2024-04-16T03:16:09.060 INFO:teuthology.orchestra.run.smithi151.stdout:HEALTH_OK 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f625806ecc0 msgr2=0x7f6258071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f625806ecc0 0x7f6258071180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6260005950 tx=0x7f62600058e0 comp rx=0 tx=0).stop 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f62700ffbd0 msgr2=0x7f6270101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f62700ffbd0 0x7f6270101610 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f62640092e0 tx=0x7f626400be20 comp rx=0 tx=0).stop 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 shutdown_connections 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f625806ecc0 0x7f6258071180 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f62700ffbd0 0x7f6270101610 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 --2- 172.21.15.151:0/2269119266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6270105a40 0x7f6270101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 >> 172.21.15.151:0/2269119266 conn(0x7f62700755a0 msgr2=0x7f62700685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 shutdown_connections 2024-04-16T03:16:09.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:09.060+0000 7f627621a700 1 -- 172.21.15.151:0/2269119266 wait complete. 2024-04-16T03:16:09.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph versions' 2024-04-16T03:16:09.949 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:16:09.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:09.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:16:09.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:09 smithi151 ceph-mon[112233]: pgmap v1129: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T03:16:09.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:09 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2269119266' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:16:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:16:10.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:16:10.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:09 smithi153 ceph-mon[87858]: pgmap v1129: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-16T03:16:10.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:09 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2269119266' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T03:16:11.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.996+0000 7fbda2d6d700 1 -- 172.21.15.151:0/3845465667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c105a40 msgr2=0x7fbd9c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.996+0000 7fbda2d6d700 1 --2- 172.21.15.151:0/3845465667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c105a40 0x7fbd9c105e20 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fbd90009910 tx=0x7fbd90009c20 comp rx=0 tx=0).stop 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.996+0000 7fbda2d6d700 1 -- 172.21.15.151:0/3845465667 shutdown_connections 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.996+0000 7fbda2d6d700 1 --2- 172.21.15.151:0/3845465667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9c0ffbd0 0x7fbd9c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.996+0000 7fbda2d6d700 1 --2- 172.21.15.151:0/3845465667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c105a40 0x7fbd9c105e20 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.996+0000 7fbda2d6d700 1 -- 172.21.15.151:0/3845465667 >> 172.21.15.151:0/3845465667 conn(0x7fbd9c0755a0 msgr2=0x7fbd9c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.997+0000 7fbda2d6d700 1 -- 172.21.15.151:0/3845465667 shutdown_connections 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.997+0000 7fbda2d6d700 1 -- 172.21.15.151:0/3845465667 wait complete. 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.997+0000 7fbda2d6d700 1 Processor -- start 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.997+0000 7fbda2d6d700 1 -- start start 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.997+0000 7fbda2d6d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c0ffbd0 0x7fbd9c19c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.997+0000 7fbda2d6d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9c105a40 0x7fbd9c19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.997+0000 7fbda2d6d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c19d260 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.997+0000 7fbda2d6d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c1966e0 con 0x7fbd9c105a40 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbda0b09700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c0ffbd0 0x7fbd9c19c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbda0b09700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c0ffbd0 0x7fbd9c19c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:58638/0 (socket says 172.21.15.151:58638) 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbda0b09700 1 -- 172.21.15.151:0/459106660 learned_addr learned my addr 172.21.15.151:0/459106660 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbd9bfff700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9c105a40 0x7fbd9c19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbda0b09700 1 -- 172.21.15.151:0/459106660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9c105a40 msgr2=0x7fbd9c19cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbda0b09700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9c105a40 0x7fbd9c19cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbda0b09700 1 -- 172.21.15.151:0/459106660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd900095f0 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbda0b09700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c0ffbd0 0x7fbd9c19c640 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fbd90006010 tx=0x7fbd9000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbd99ffb700 1 -- 172.21.15.151:0/459106660 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd9001d070 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbd99ffb700 1 -- 172.21.15.151:0/459106660 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd9000f460 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c196960 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.998+0000 7fbd99ffb700 1 -- 172.21.15.151:0/459106660 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd90017630 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.999+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c196e50 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:11.999+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c108040 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.000+0000 7fbd99ffb700 1 -- 172.21.15.151:0/459106660 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fbd9000f5d0 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.000+0000 7fbd99ffb700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd8c06ecc0 0x7fbd8c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:16:12.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.000+0000 7fbd99ffb700 1 -- 172.21.15.151:0/459106660 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbd900903c0 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.000+0000 7fbd9bfff700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd8c06ecc0 0x7fbd8c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:16:12.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.001+0000 7fbd9bfff700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd8c06ecc0 0x7fbd8c071180 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c197ed0 tx=0x7fbd88009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:16:12.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.005+0000 7fbd99ffb700 1 -- 172.21.15.151:0/459106660 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbd9005cca0 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:12 smithi151 ceph-mon[112233]: pgmap v1130: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:12.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.292+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbd9c107a20 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.293 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.293+0000 7fbd99ffb700 1 -- 172.21.15.151:0/459106660 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fbd9005c830 con 0x7fbd9c0ffbd0 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:16:12.294 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:16:12.295 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:16:12.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:16:12.295 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:16:12.295 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:16:12.295 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:16:12.295 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:16:12.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd8c06ecc0 msgr2=0x7fbd8c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:12.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd8c06ecc0 0x7fbd8c071180 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c197ed0 tx=0x7fbd88009190 comp rx=0 tx=0).stop 2024-04-16T03:16:12.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c0ffbd0 msgr2=0x7fbd9c19c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:16:12.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c0ffbd0 0x7fbd9c19c640 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fbd90006010 tx=0x7fbd9000bd70 comp rx=0 tx=0).stop 2024-04-16T03:16:12.296 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 shutdown_connections 2024-04-16T03:16:12.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbd8c06ecc0 0x7fbd8c071180 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:12.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9c0ffbd0 0x7fbd9c19c640 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:12.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 --2- 172.21.15.151:0/459106660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9c105a40 0x7fbd9c19cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:16:12.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 >> 172.21.15.151:0/459106660 conn(0x7fbd9c0755a0 msgr2=0x7fbd9c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:16:12.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 shutdown_connections 2024-04-16T03:16:12.297 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:16:12.295+0000 7fbda2d6d700 1 -- 172.21.15.151:0/459106660 wait complete. 2024-04-16T03:16:12.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:12 smithi153 ceph-mon[87858]: pgmap v1130: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:13.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-16T03:16:13.230 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:16:13.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:13 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/459106660' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:13.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:13 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/459106660' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:16:14.366 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:14 smithi151 ceph-mon[112233]: pgmap v1131: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:14.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:14 smithi153 ceph-mon[87858]: pgmap v1131: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:14.962 INFO:teuthology.orchestra.run.smithi151.stdout:wait for servicemap items w/ changing names to refresh 2024-04-16T03:16:15.388 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:15 smithi151 ceph-mon[112233]: pgmap v1132: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:15.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:15 smithi153 ceph-mon[87858]: pgmap v1132: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:15.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'sleep 60' 2024-04-16T03:16:15.892 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:16:18.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:18 smithi151 ceph-mon[112233]: pgmap v1133: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:18.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:18 smithi153 ceph-mon[87858]: pgmap v1133: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:20.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:20 smithi151 ceph-mon[112233]: pgmap v1134: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:20.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:20 smithi153 ceph-mon[87858]: pgmap v1134: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:22.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:22 smithi151 ceph-mon[112233]: pgmap v1135: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:22.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:22 smithi153 ceph-mon[87858]: pgmap v1135: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:23.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:23 smithi151 ceph-mon[112233]: pgmap v1136: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:23.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:23 smithi153 ceph-mon[87858]: pgmap v1136: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:24.452 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:16:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:16:25.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:25 smithi151 ceph-mon[112233]: pgmap v1137: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:25.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:25 smithi153 ceph-mon[87858]: pgmap v1137: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:28.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:28 smithi151 ceph-mon[112233]: pgmap v1138: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:28.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:28 smithi153 ceph-mon[87858]: pgmap v1138: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:30.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:30 smithi151 ceph-mon[112233]: pgmap v1139: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:30.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:30 smithi153 ceph-mon[87858]: pgmap v1139: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:32.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:32 smithi151 ceph-mon[112233]: pgmap v1140: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:32.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:32 smithi153 ceph-mon[87858]: pgmap v1140: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:33.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:33 smithi151 ceph-mon[112233]: pgmap v1141: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:33.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:33 smithi153 ceph-mon[87858]: pgmap v1141: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:36.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:36 smithi151 ceph-mon[112233]: pgmap v1142: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:36.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:36 smithi153 ceph-mon[87858]: pgmap v1142: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:38.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:38 smithi151 ceph-mon[112233]: pgmap v1143: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:38.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:38 smithi153 ceph-mon[87858]: pgmap v1143: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:39.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:16:39.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:16:40.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:40 smithi151 ceph-mon[112233]: pgmap v1144: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:40.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:40 smithi153 ceph-mon[87858]: pgmap v1144: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:41.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:41 smithi151 ceph-mon[112233]: pgmap v1145: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:41 smithi153 ceph-mon[87858]: pgmap v1145: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:44.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:44 smithi151 ceph-mon[112233]: pgmap v1146: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:44.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:44 smithi153 ceph-mon[87858]: pgmap v1146: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:46.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:46 smithi151 ceph-mon[112233]: pgmap v1147: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:46.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:46 smithi153 ceph-mon[87858]: pgmap v1147: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:47.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:47 smithi151 ceph-mon[112233]: pgmap v1148: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:47.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:47 smithi153 ceph-mon[87858]: pgmap v1148: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:50.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:50 smithi151 ceph-mon[112233]: pgmap v1149: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:50.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:50 smithi153 ceph-mon[87858]: pgmap v1149: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:52.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:52 smithi151 ceph-mon[112233]: pgmap v1150: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:52.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:52 smithi153 ceph-mon[87858]: pgmap v1150: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:53.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:53 smithi151 ceph-mon[112233]: pgmap v1151: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:53.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:53 smithi153 ceph-mon[87858]: pgmap v1151: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:54.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:16:54.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:16:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:55 smithi151 ceph-mon[112233]: pgmap v1152: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:55.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:55 smithi153 ceph-mon[87858]: pgmap v1152: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:58.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:58 smithi151 ceph-mon[112233]: pgmap v1153: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:58.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:58 smithi153 ceph-mon[87858]: pgmap v1153: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:16:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:16:59 smithi151 ceph-mon[112233]: pgmap v1154: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:16:59.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:16:59 smithi153 ceph-mon[87858]: pgmap v1154: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:00.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:00 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:17:00.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:00 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:17:01.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:01 smithi153 ceph-mon[87858]: pgmap v1155: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:01.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:17:01.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:17:01.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:01 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:17:01.949 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:01 smithi151 ceph-mon[112233]: pgmap v1155: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:01.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:17:01.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:17:01.950 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:01 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:17:04.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:04 smithi151 ceph-mon[112233]: pgmap v1156: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:04.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:04 smithi153 ceph-mon[87858]: pgmap v1156: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:05.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:05 smithi153 ceph-mon[87858]: pgmap v1157: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:05.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:05 smithi151 ceph-mon[112233]: pgmap v1157: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:08.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:08 smithi151 ceph-mon[112233]: pgmap v1158: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:08.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:08 smithi153 ceph-mon[87858]: pgmap v1158: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:09.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:09 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:17:09.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:09 smithi151 ceph-mon[112233]: pgmap v1159: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:09.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:09 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:17:09.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:09 smithi153 ceph-mon[87858]: pgmap v1159: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:12.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:12 smithi151 ceph-mon[112233]: pgmap v1160: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:12.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:12 smithi153 ceph-mon[87858]: pgmap v1160: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:13.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:13 smithi151 ceph-mon[112233]: pgmap v1161: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:13.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:13 smithi153 ceph-mon[87858]: pgmap v1161: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:15.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:15 smithi153 ceph-mon[87858]: pgmap v1162: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:15.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:15 smithi151 ceph-mon[112233]: pgmap v1162: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:17.443 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:17 smithi151 ceph-mon[112233]: pgmap v1163: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:17 smithi153 ceph-mon[87858]: pgmap v1163: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:18.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph orch ps' 2024-04-16T03:17:18.356 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:17:19.461 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:19 smithi151 ceph-mon[112233]: pgmap v1164: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:19.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:19 smithi153 ceph-mon[87858]: pgmap v1164: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:20.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.010+0000 7fa291981700 1 -- 172.21.15.151:0/3931940859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 msgr2=0x7fa28c105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:20.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.010+0000 7fa291981700 1 --2- 172.21.15.151:0/3931940859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 0x7fa28c105e50 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fa27c009910 tx=0x7fa27c009c20 comp rx=0 tx=0).stop 2024-04-16T03:17:20.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.010+0000 7fa291981700 1 -- 172.21.15.151:0/3931940859 shutdown_connections 2024-04-16T03:17:20.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.010+0000 7fa291981700 1 --2- 172.21.15.151:0/3931940859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa28c0ffa70 0x7fa28c0ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:20.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.010+0000 7fa291981700 1 --2- 172.21.15.151:0/3931940859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 0x7fa28c105e50 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:20.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.010+0000 7fa291981700 1 -- 172.21.15.151:0/3931940859 >> 172.21.15.151:0/3931940859 conn(0x7fa28c0fb560 msgr2=0x7fa28c0fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:20.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.010+0000 7fa291981700 1 -- 172.21.15.151:0/3931940859 shutdown_connections 2024-04-16T03:17:20.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.010+0000 7fa291981700 1 -- 172.21.15.151:0/3931940859 wait complete. 2024-04-16T03:17:20.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.011+0000 7fa291981700 1 Processor -- start 2024-04-16T03:17:20.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.011+0000 7fa291981700 1 -- start start 2024-04-16T03:17:20.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa291981700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa28c0ffa70 0x7fa28c1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:20.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa291981700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 0x7fa28c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:20.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa291981700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c198100 con 0x7fa28c105a70 2024-04-16T03:17:20.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa291981700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c198240 con 0x7fa28c0ffa70 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa28a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 0x7fa28c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa28a7fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 0x7fa28c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44554/0 (socket says 172.21.15.151:44554) 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa28a7fc700 1 -- 172.21.15.151:0/3702808858 learned_addr learned my addr 172.21.15.151:0/3702808858 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa28affd700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa28c0ffa70 0x7fa28c1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa28a7fc700 1 -- 172.21.15.151:0/3702808858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa28c0ffa70 msgr2=0x7fa28c1975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa28a7fc700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa28c0ffa70 0x7fa28c1975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa28a7fc700 1 -- 172.21.15.151:0/3702808858 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c0095f0 con 0x7fa28c105a70 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa28a7fc700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 0x7fa28c197ae0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa28000d8d0 tx=0x7fa28000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa29097f700 1 -- 172.21.15.151:0/3702808858 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2800098a0 con 0x7fa28c105a70 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa29097f700 1 -- 172.21.15.151:0/3702808858 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa280010460 con 0x7fa28c105a70 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c1a0610 con 0x7fa28c105a70 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa29097f700 1 -- 172.21.15.151:0/3702808858 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28000f5f0 con 0x7fa28c105a70 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.012+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c1a0ad0 con 0x7fa28c105a70 2024-04-16T03:17:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.013+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28c04ea90 con 0x7fa28c105a70 2024-04-16T03:17:20.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.014+0000 7fa29097f700 1 -- 172.21.15.151:0/3702808858 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa280009a00 con 0x7fa28c105a70 2024-04-16T03:17:20.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.014+0000 7fa29097f700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa27406ecc0 0x7fa274071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:20.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.015+0000 7fa29097f700 1 -- 172.21.15.151:0/3702808858 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa28008ec30 con 0x7fa28c105a70 2024-04-16T03:17:20.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.015+0000 7fa28affd700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa27406ecc0 0x7fa274071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:20.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.015+0000 7fa28affd700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa27406ecc0 0x7fa274071180 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa27c006010 tx=0x7fa27c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:20.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.019+0000 7fa29097f700 1 -- 172.21.15.151:0/3702808858 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa280057a60 con 0x7fa28c105a70 2024-04-16T03:17:20.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.228+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 --> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa28c0fc5f0 con 0x7fa27406ecc0 2024-04-16T03:17:20.242 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.240+0000 7fa29097f700 1 -- 172.21.15.151:0/3702808858 <== mgr.34104 v2:172.21.15.151:6800/2362635868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fa28c0fc5f0 con 0x7fa27406ecc0 2024-04-16T03:17:20.242 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T03:17:20.242 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.smithi151 smithi151 *:9093,9094 running (38m) 85s ago 50m 24.9M - 0.25.0 c8568f914cd2 ae1b0207e2ab 2024-04-16T03:17:20.242 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi151 smithi151 running (104s) 85s ago 50m 7922k - 19.0.0-3084-g87dbfb63 a452a57266c7 d31666f7bbb4 2024-04-16T03:17:20.242 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-exporter.smithi153 smithi153 running (93s) 88s ago 49m 7109k - 19.0.0-3084-g87dbfb63 a452a57266c7 b064f1a28cbf 2024-04-16T03:17:20.242 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi151 smithi151 running (36m) 85s ago 50m 6660k - 19.0.0-3084-g87dbfb63 a452a57266c7 c77e1aab330a 2024-04-16T03:17:20.242 INFO:teuthology.orchestra.run.smithi151.stdout:crash.smithi153 smithi153 running (36m) 88s ago 49m 6665k - 19.0.0-3084-g87dbfb63 a452a57266c7 3dd23106e28c 2024-04-16T03:17:20.242 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.smithi151 smithi151 *:3000 running (38m) 85s ago 49m 81.8M - 9.4.12 2bacad6d85d8 43f2781ad9e4 2024-04-16T03:17:20.242 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.bdpnxq smithi151 running (32m) 85s ago 44m 69.2M - 19.0.0-3084-g87dbfb63 a452a57266c7 86186d4bd5dc 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi151.slrqog smithi151 running (32m) 85s ago 44m 68.2M - 19.0.0-3084-g87dbfb63 a452a57266c7 5ce0f1ddd813 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.mqbgav smithi153 running (16m) 88s ago 44m 69.9M - 19.0.0-3084-g87dbfb63 a452a57266c7 7a47a5984e2a 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:mds.cephfs.smithi153.ogdcjk smithi153 running (32m) 88s ago 44m 68.1M - 19.0.0-3084-g87dbfb63 a452a57266c7 9cbaf3624549 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi151.uqqame smithi151 *:8443,9283,8765 running (40m) 85s ago 51m 518M - 19.0.0-3084-g87dbfb63 a452a57266c7 772cbed0f347 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.smithi153.esbtzf smithi153 *:8443,9283,8765 running (39m) 88s ago 49m 460M - 19.0.0-3084-g87dbfb63 a452a57266c7 39f5fb89c561 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi151 smithi151 running (37m) 85s ago 51m 87.6M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 420c23cf907e 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:mon.smithi153 smithi153 running (37m) 88s ago 49m 62.0M 2048M 19.0.0-3084-g87dbfb63 a452a57266c7 cc3cd0d47f4e 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi151 smithi151 *:9100 running (39m) 85s ago 50m 15.5M - 1.5.0 0da6a335fe13 019bee388048 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.smithi153 smithi153 *:9100 running (39m) 88s ago 49m 16.2M - 1.5.0 0da6a335fe13 982dc8b2a415 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (36m) 85s ago 48m 103M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 fee11db8c57b 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (35m) 85s ago 47m 115M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 e3b70d49369d 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (34m) 85s ago 47m 95.4M 1216M 19.0.0-3084-g87dbfb63 a452a57266c7 4f046ee57e95 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi153 running (34m) 88s ago 47m 126M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a731606ec80f 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi153 running (33m) 88s ago 46m 101M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 184b926e242c 2024-04-16T03:17:20.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi153 running (33m) 88s ago 46m 113M 2240M 19.0.0-3084-g87dbfb63 a452a57266c7 a3f762d46f24 2024-04-16T03:17:20.244 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.smithi151 smithi151 *:9095 running (39m) 85s ago 49m 92.0M - 2.43.0 a07b618ecd1d aad8a32f04d8 2024-04-16T03:17:20.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa27406ecc0 msgr2=0x7fa274071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:20.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa27406ecc0 0x7fa274071180 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa27c006010 tx=0x7fa27c01a040 comp rx=0 tx=0).stop 2024-04-16T03:17:20.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 msgr2=0x7fa28c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:20.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 0x7fa28c197ae0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa28000d8d0 tx=0x7fa28000dbe0 comp rx=0 tx=0).stop 2024-04-16T03:17:20.244 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 shutdown_connections 2024-04-16T03:17:20.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa27406ecc0 0x7fa274071180 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:20.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa28c0ffa70 0x7fa28c1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:20.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 --2- 172.21.15.151:0/3702808858 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa28c105a70 0x7fa28c197ae0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:20.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 >> 172.21.15.151:0/3702808858 conn(0x7fa28c0fb560 msgr2=0x7fa28c0fbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:20.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 shutdown_connections 2024-04-16T03:17:20.245 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:20.242+0000 7fa291981700 1 -- 172.21.15.151:0/3702808858 wait complete. 2024-04-16T03:17:20.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph versions' 2024-04-16T03:17:21.137 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:17:21.483 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:21 smithi151 ceph-mon[112233]: from='client.36048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:17:21.483 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:21 smithi151 ceph-mon[112233]: pgmap v1165: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:21 smithi153 ceph-mon[87858]: from='client.36048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T03:17:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:21 smithi153 ceph-mon[87858]: pgmap v1165: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:22.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.772+0000 7ff848f64700 1 -- 172.21.15.151:0/3629358575 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff844101da0 msgr2=0x7ff844102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:22.778 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.772+0000 7ff848f64700 1 --2- 172.21.15.151:0/3629358575 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff844101da0 0x7ff844102180 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff838009910 tx=0x7ff838009c20 comp rx=0 tx=0).stop 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.773+0000 7ff848f64700 1 -- 172.21.15.151:0/3629358575 shutdown_connections 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.773+0000 7ff848f64700 1 --2- 172.21.15.151:0/3629358575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8441026c0 0x7ff84410abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.773+0000 7ff848f64700 1 --2- 172.21.15.151:0/3629358575 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff844101da0 0x7ff844102180 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.773+0000 7ff848f64700 1 -- 172.21.15.151:0/3629358575 >> 172.21.15.151:0/3629358575 conn(0x7ff8440fb550 msgr2=0x7ff8440fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.773+0000 7ff848f64700 1 -- 172.21.15.151:0/3629358575 shutdown_connections 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.773+0000 7ff848f64700 1 -- 172.21.15.151:0/3629358575 wait complete. 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff848f64700 1 Processor -- start 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff848f64700 1 -- start start 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff848f64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff844101da0 0x7ff84419c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff848f64700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8441026c0 0x7ff84419cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff848f64700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84419d1d0 con 0x7ff8441026c0 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff848f64700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff844196680 con 0x7ff844101da0 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff841d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8441026c0 0x7ff84419cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:22.779 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff841d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8441026c0 0x7ff84419cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44578/0 (socket says 172.21.15.151:44578) 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff841d9b700 1 -- 172.21.15.151:0/2389125225 learned_addr learned my addr 172.21.15.151:0/2389125225 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff84259c700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff844101da0 0x7ff84419c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff841d9b700 1 -- 172.21.15.151:0/2389125225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff844101da0 msgr2=0x7ff84419c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff841d9b700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff844101da0 0x7ff84419c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.774+0000 7ff841d9b700 1 -- 172.21.15.151:0/2389125225 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8380095f0 con 0x7ff8441026c0 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.775+0000 7ff841d9b700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8441026c0 0x7ff84419cb40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff83400b480 tx=0x7ff83400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.775+0000 7ff8337fe700 1 -- 172.21.15.151:0/2389125225 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8340118c0 con 0x7ff8441026c0 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.775+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff844196960 con 0x7ff8441026c0 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.775+0000 7ff8337fe700 1 -- 172.21.15.151:0/2389125225 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff834010380 con 0x7ff8441026c0 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.775+0000 7ff8337fe700 1 -- 172.21.15.151:0/2389125225 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff83400f4f0 con 0x7ff8441026c0 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.775+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff844196eb0 con 0x7ff8441026c0 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.776+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8441082b0 con 0x7ff8441026c0 2024-04-16T03:17:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.776+0000 7ff8337fe700 1 -- 172.21.15.151:0/2389125225 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff8340104f0 con 0x7ff8441026c0 2024-04-16T03:17:22.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.777+0000 7ff8337fe700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff82c06ecc0 0x7ff82c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:22.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.777+0000 7ff8337fe700 1 -- 172.21.15.151:0/2389125225 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff83408ea90 con 0x7ff8441026c0 2024-04-16T03:17:22.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.777+0000 7ff84259c700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff82c06ecc0 0x7ff82c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:22.781 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.777+0000 7ff84259c700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff82c06ecc0 0x7ff82c071180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff838006010 tx=0x7ff83801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:22.784 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:22.782+0000 7ff8337fe700 1 -- 172.21.15.151:0/2389125225 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff83405b370 con 0x7ff8441026c0 2024-04-16T03:17:23.062 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.060+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff84404ea90 con 0x7ff8441026c0 2024-04-16T03:17:23.063 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.061+0000 7ff8337fe700 1 -- 172.21.15.151:0/2389125225 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff83405af00 con 0x7ff8441026c0 2024-04-16T03:17:23.063 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-04-16T03:17:23.063 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-04-16T03:17:23.063 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:17:23.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:17:23.063 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 2 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 6 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 4 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)": 14 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-04-16T03:17:23.064 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.063+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff82c06ecc0 msgr2=0x7ff82c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.063+0000 7ff848f64700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff82c06ecc0 0x7ff82c071180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff838006010 tx=0x7ff83801a040 comp rx=0 tx=0).stop 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.063+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8441026c0 msgr2=0x7ff84419cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.063+0000 7ff848f64700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8441026c0 0x7ff84419cb40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff83400b480 tx=0x7ff83400b840 comp rx=0 tx=0).stop 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.064+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 shutdown_connections 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.064+0000 7ff848f64700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff82c06ecc0 0x7ff82c071180 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.064+0000 7ff848f64700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff844101da0 0x7ff84419c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.064+0000 7ff848f64700 1 --2- 172.21.15.151:0/2389125225 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8441026c0 0x7ff84419cb40 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.064+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 >> 172.21.15.151:0/2389125225 conn(0x7ff8440fb550 msgr2=0x7ff8441053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.064+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 shutdown_connections 2024-04-16T03:17:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:23.064+0000 7ff848f64700 1 -- 172.21.15.151:0/2389125225 wait complete. 2024-04-16T03:17:23.393 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:23 smithi151 ceph-mon[112233]: pgmap v1166: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:23.393 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:23 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2389125225' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:17:23.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:23 smithi153 ceph-mon[87858]: pgmap v1166: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:23.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:23 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2389125225' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:17:23.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-16T03:17:23.921 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:17:24.380 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:17:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:17:25.321 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:25 smithi151 ceph-mon[112233]: pgmap v1167: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:25.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:25 smithi153 ceph-mon[87858]: pgmap v1167: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.573+0000 7f1d0db44700 1 -- 172.21.15.151:0/1202559593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d080ffa40 msgr2=0x7f1d080ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.573+0000 7f1d0db44700 1 --2- 172.21.15.151:0/1202559593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d080ffa40 0x7f1d080ffec0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009910 tx=0x7f1cfc009c20 comp rx=0 tx=0).stop 2024-04-16T03:17:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.574+0000 7f1d0db44700 1 -- 172.21.15.151:0/1202559593 shutdown_connections 2024-04-16T03:17:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.574+0000 7f1d0db44700 1 --2- 172.21.15.151:0/1202559593 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d080ffa40 0x7f1d080ffec0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.574+0000 7f1d0db44700 1 --2- 172.21.15.151:0/1202559593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d08105a40 0x7f1d08105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.574+0000 7f1d0db44700 1 -- 172.21.15.151:0/1202559593 >> 172.21.15.151:0/1202559593 conn(0x7f1d080fb570 msgr2=0x7f1d080fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.574+0000 7f1d0db44700 1 -- 172.21.15.151:0/1202559593 shutdown_connections 2024-04-16T03:17:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.574+0000 7f1d0db44700 1 -- 172.21.15.151:0/1202559593 wait complete. 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.574+0000 7f1d0db44700 1 Processor -- start 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.574+0000 7f1d0db44700 1 -- start start 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d0db44700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d080ffa40 0x7f1d0819e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d0db44700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d08105a40 0x7f1d0819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d0db44700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0819f3d0 con 0x7f1d08105a40 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d0db44700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08198880 con 0x7f1d080ffa40 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d06ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d08105a40 0x7f1d0819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d06ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d08105a40 0x7f1d0819ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44586/0 (socket says 172.21.15.151:44586) 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d06ffd700 1 -- 172.21.15.151:0/892997756 learned_addr learned my addr 172.21.15.151:0/892997756 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d077fe700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d080ffa40 0x7f1d0819e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:25.578 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d06ffd700 1 -- 172.21.15.151:0/892997756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d080ffa40 msgr2=0x7f1d0819e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d06ffd700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d080ffa40 0x7f1d0819e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d06ffd700 1 -- 172.21.15.151:0/892997756 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc0095f0 con 0x7f1d08105a40 2024-04-16T03:17:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d06ffd700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d08105a40 0x7f1d0819ed40 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc006010 tx=0x7f1cfc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.575+0000 7f1d04ff9700 1 -- 172.21.15.151:0/892997756 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cfc01d070 con 0x7f1d08105a40 2024-04-16T03:17:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.576+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d08198b00 con 0x7f1d08105a40 2024-04-16T03:17:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.576+0000 7f1d04ff9700 1 -- 172.21.15.151:0/892997756 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cfc004fb0 con 0x7f1d08105a40 2024-04-16T03:17:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.576+0000 7f1d04ff9700 1 -- 172.21.15.151:0/892997756 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cfc022ea0 con 0x7f1d08105a40 2024-04-16T03:17:25.579 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.576+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d08198ff0 con 0x7f1d08105a40 2024-04-16T03:17:25.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.577+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0804ea90 con 0x7f1d08105a40 2024-04-16T03:17:25.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.577+0000 7f1d04ff9700 1 -- 172.21.15.151:0/892997756 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1cfc022470 con 0x7f1d08105a40 2024-04-16T03:17:25.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.578+0000 7f1d04ff9700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1cf006ecc0 0x7f1cf0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:25.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.578+0000 7f1d04ff9700 1 -- 172.21.15.151:0/892997756 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1cfc090420 con 0x7f1d08105a40 2024-04-16T03:17:25.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.578+0000 7f1d077fe700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1cf006ecc0 0x7f1cf0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:25.580 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.578+0000 7f1d077fe700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1cf006ecc0 0x7f1cf0071180 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8006fd0 tx=0x7f1cf8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:25.584 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.582+0000 7f1d04ff9700 1 -- 172.21.15.151:0/892997756 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1cfc05cdb0 con 0x7f1d08105a40 2024-04-16T03:17:25.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.863+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1d08199440 con 0x7f1d08105a40 2024-04-16T03:17:25.866 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.864+0000 7f1d04ff9700 1 -- 172.21.15.151:0/892997756 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f1cfc022720 con 0x7f1d08105a40 2024-04-16T03:17:25.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1cf006ecc0 msgr2=0x7f1cf0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:25.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1cf006ecc0 0x7f1cf0071180 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8006fd0 tx=0x7f1cf8008040 comp rx=0 tx=0).stop 2024-04-16T03:17:25.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d08105a40 msgr2=0x7f1d0819ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:25.868 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d08105a40 0x7f1d0819ed40 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc006010 tx=0x7f1cfc00be20 comp rx=0 tx=0).stop 2024-04-16T03:17:25.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 shutdown_connections 2024-04-16T03:17:25.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1cf006ecc0 0x7f1cf0071180 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:25.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d080ffa40 0x7f1d0819e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:25.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 --2- 172.21.15.151:0/892997756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d08105a40 0x7f1d0819ed40 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:25.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 >> 172.21.15.151:0/892997756 conn(0x7f1d080fb570 msgr2=0x7f1d08068d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:25.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 shutdown_connections 2024-04-16T03:17:25.869 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:25.866+0000 7f1d0db44700 1 -- 172.21.15.151:0/892997756 wait complete. 2024-04-16T03:17:25.878 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-04-16T03:17:26.391 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:26 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/892997756' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:17:26.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-16T03:17:26.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:26 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/892997756' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:17:26.628 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:17:27.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:27 smithi151 ceph-mon[112233]: pgmap v1168: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:27.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:27 smithi153 ceph-mon[87858]: pgmap v1168: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:28.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.395+0000 7f4d276c8700 1 -- 172.21.15.151:0/3935992281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 msgr2=0x7f4d20105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:28.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.395+0000 7f4d276c8700 1 --2- 172.21.15.151:0/3935992281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 0x7f4d20105e20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800b0a0 tx=0x7f4d1800b3b0 comp rx=0 tx=0).stop 2024-04-16T03:17:28.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.395+0000 7f4d276c8700 1 -- 172.21.15.151:0/3935992281 shutdown_connections 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.395+0000 7f4d276c8700 1 --2- 172.21.15.151:0/3935992281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d200ffbd0 0x7f4d20100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.395+0000 7f4d276c8700 1 --2- 172.21.15.151:0/3935992281 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 0x7f4d20105e20 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.395+0000 7f4d276c8700 1 -- 172.21.15.151:0/3935992281 >> 172.21.15.151:0/3935992281 conn(0x7f4d200755a0 msgr2=0x7f4d200759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.395+0000 7f4d276c8700 1 -- 172.21.15.151:0/3935992281 shutdown_connections 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.395+0000 7f4d276c8700 1 -- 172.21.15.151:0/3935992281 wait complete. 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d276c8700 1 Processor -- start 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d276c8700 1 -- start start 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d276c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d200ffbd0 0x7f4d201976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d276c8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 0x7f4d20197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d276c8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20198300 con 0x7f4d20105a40 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d276c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20198470 con 0x7f4d200ffbd0 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d24c63700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 0x7f4d20197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d24c63700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 0x7f4d20197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57214/0 (socket says 172.21.15.151:57214) 2024-04-16T03:17:28.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d24c63700 1 -- 172.21.15.151:0/3478381207 learned_addr learned my addr 172.21.15.151:0/3478381207 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.396+0000 7f4d25464700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d200ffbd0 0x7f4d201976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d24c63700 1 -- 172.21.15.151:0/3478381207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d200ffbd0 msgr2=0x7f4d201976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d24c63700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d200ffbd0 0x7f4d201976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d24c63700 1 -- 172.21.15.151:0/3478381207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d18009d00 con 0x7f4d20105a40 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d25464700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d200ffbd0 0x7f4d201976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d24c63700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 0x7f4d20197c20 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4d1400b4b0 tx=0x7f4d1400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d1e7fc700 1 -- 172.21.15.151:0/3478381207 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d14011800 con 0x7f4d20105a40 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d1e7fc700 1 -- 172.21.15.151:0/3478381207 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d14011e20 con 0x7f4d20105a40 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d201a0750 con 0x7f4d20105a40 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d1e7fc700 1 -- 172.21.15.151:0/3478381207 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d1400f2e0 con 0x7f4d20105a40 2024-04-16T03:17:28.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.397+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d201a0ca0 con 0x7f4d20105a40 2024-04-16T03:17:28.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.398+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d20108040 con 0x7f4d20105a40 2024-04-16T03:17:28.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.399+0000 7f4d1e7fc700 1 -- 172.21.15.151:0/3478381207 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4d14010380 con 0x7f4d20105a40 2024-04-16T03:17:28.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.399+0000 7f4d1e7fc700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0806ecc0 0x7f4d08071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:28.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.399+0000 7f4d25464700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0806ecc0 0x7f4d08071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:28.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.399+0000 7f4d1e7fc700 1 -- 172.21.15.151:0/3478381207 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4d1408ea00 con 0x7f4d20105a40 2024-04-16T03:17:28.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.400+0000 7f4d25464700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0806ecc0 0x7f4d08071180 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4d180062a0 tx=0x7f4d18006210 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:28.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.404+0000 7f4d1e7fc700 1 -- 172.21.15.151:0/3478381207 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4d140577b0 con 0x7f4d20105a40 2024-04-16T03:17:28.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.687+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4d20107a20 con 0x7f4d20105a40 2024-04-16T03:17:28.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.688+0000 7f4d1e7fc700 1 -- 172.21.15.151:0/3478381207 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4d1405ae70 con 0x7f4d20105a40 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0806ecc0 msgr2=0x7f4d08071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0806ecc0 0x7f4d08071180 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4d180062a0 tx=0x7f4d18006210 comp rx=0 tx=0).stop 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 msgr2=0x7f4d20197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 0x7f4d20197c20 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4d1400b4b0 tx=0x7f4d1400b870 comp rx=0 tx=0).stop 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 shutdown_connections 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4d0806ecc0 0x7f4d08071180 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d200ffbd0 0x7f4d201976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 --2- 172.21.15.151:0/3478381207 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20105a40 0x7f4d20197c20 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 >> 172.21.15.151:0/3478381207 conn(0x7f4d200755a0 msgr2=0x7f4d200fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 shutdown_connections 2024-04-16T03:17:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:28.690+0000 7f4d276c8700 1 -- 172.21.15.151:0/3478381207 wait complete. 2024-04-16T03:17:28.701 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.0.0-3084-g87dbfb63 (87dbfb632e974cb2c28d17423071d2d833b53b29) squid (dev)" 2024-04-16T03:17:28.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:28 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3478381207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:17:29.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:28 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3478381207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T03:17:29.371 DEBUG:teuthology.parallel:result is None 2024-04-16T03:17:29.371 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T03:17:29.382 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T03:17:29.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T03:17:29.598 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/mon.smithi151/config 2024-04-16T03:17:29.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:29 smithi151 ceph-mon[112233]: pgmap v1169: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:30.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:29 smithi153 ceph-mon[87858]: pgmap v1169: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:31.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.270+0000 7f7e65b74700 1 -- 172.21.15.151:0/1856294083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 msgr2=0x7f7e60100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:31.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.270+0000 7f7e65b74700 1 --2- 172.21.15.151:0/1856294083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 0x7f7e60100050 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009910 tx=0x7f7e48009c20 comp rx=0 tx=0).stop 2024-04-16T03:17:31.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.270+0000 7f7e65b74700 1 -- 172.21.15.151:0/1856294083 shutdown_connections 2024-04-16T03:17:31.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.270+0000 7f7e65b74700 1 --2- 172.21.15.151:0/1856294083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 0x7f7e60100050 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:31.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.270+0000 7f7e65b74700 1 --2- 172.21.15.151:0/1856294083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e60105a40 0x7f7e60105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:31.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.270+0000 7f7e65b74700 1 -- 172.21.15.151:0/1856294083 >> 172.21.15.151:0/1856294083 conn(0x7f7e600755a0 msgr2=0x7f7e600759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:31.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.270+0000 7f7e65b74700 1 -- 172.21.15.151:0/1856294083 shutdown_connections 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.270+0000 7f7e65b74700 1 -- 172.21.15.151:0/1856294083 wait complete. 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.271+0000 7f7e65b74700 1 Processor -- start 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.271+0000 7f7e65b74700 1 -- start start 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.271+0000 7f7e65b74700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 0x7f7e6019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.271+0000 7f7e65b74700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e60105a40 0x7f7e6019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.271+0000 7f7e65b74700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6019c0d0 con 0x7f7e600ffbd0 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.271+0000 7f7e65b74700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e60195580 con 0x7f7e60105a40 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.271+0000 7f7e5f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 0x7f7e6019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.271+0000 7f7e5f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 0x7f7e6019b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57236/0 (socket says 172.21.15.151:57236) 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.271+0000 7f7e5f7fe700 1 -- 172.21.15.151:0/301328079 learned_addr learned my addr 172.21.15.151:0/301328079 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e5effd700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e60105a40 0x7f7e6019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e5f7fe700 1 -- 172.21.15.151:0/301328079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e60105a40 msgr2=0x7f7e6019ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e5f7fe700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e60105a40 0x7f7e6019ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e5f7fe700 1 -- 172.21.15.151:0/301328079 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e480095f0 con 0x7f7e600ffbd0 2024-04-16T03:17:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e5f7fe700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 0x7f7e6019b500 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000ba50 tx=0x7f7e5000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e5cff9700 1 -- 172.21.15.151:0/301328079 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e5000ca00 con 0x7f7e600ffbd0 2024-04-16T03:17:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e60195860 con 0x7f7e600ffbd0 2024-04-16T03:17:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e5cff9700 1 -- 172.21.15.151:0/301328079 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e5000cb60 con 0x7f7e600ffbd0 2024-04-16T03:17:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e5cff9700 1 -- 172.21.15.151:0/301328079 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e500075a0 con 0x7f7e600ffbd0 2024-04-16T03:17:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.272+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e60195db0 con 0x7f7e600ffbd0 2024-04-16T03:17:31.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.273+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e60108060 con 0x7f7e600ffbd0 2024-04-16T03:17:31.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.274+0000 7f7e5cff9700 1 -- 172.21.15.151:0/301328079 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7e5000ccd0 con 0x7f7e600ffbd0 2024-04-16T03:17:31.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.274+0000 7f7e5cff9700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7e4c06ecc0 0x7f7e4c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:31.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.274+0000 7f7e5cff9700 1 -- 172.21.15.151:0/301328079 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7e50006e40 con 0x7f7e600ffbd0 2024-04-16T03:17:31.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.274+0000 7f7e5effd700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7e4c06ecc0 0x7f7e4c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:31.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.275+0000 7f7e5effd700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7e4c06ecc0 0x7f7e4c071180 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7e480059d0 tx=0x7f7e4801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:31.280 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.279+0000 7f7e5cff9700 1 -- 172.21.15.151:0/301328079 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7e5005af60 con 0x7f7e600ffbd0 2024-04-16T03:17:31.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:31 smithi151 ceph-mon[112233]: pgmap v1170: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:31.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:31 smithi153 ceph-mon[87858]: pgmap v1170: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:31.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.522+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7e6004ea90 con 0x7f7e600ffbd0 2024-04-16T03:17:31.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.522+0000 7f7e5cff9700 1 -- 172.21.15.151:0/301328079 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 42 v42) v1 ==== 76+0+1975 (secure 0 0 0) 0x7f7e5005aaf0 con 0x7f7e600ffbd0 2024-04-16T03:17:31.523 INFO:teuthology.orchestra.run.smithi151.stdout:e42 2024-04-16T03:17:31.523 INFO:teuthology.orchestra.run.smithi151.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:legacy client fscid: 1 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:Filesystem 'cephfs' (1) 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:fs_name cephfs 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 42 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-04-16T02:32:34.941393+0000 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-04-16T03:15:59.179916+0000 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:tableserver 0 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:root 0 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:session_timeout 60 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:session_autoclose 300 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:max_file_size 1099511627776 2024-04-16T03:17:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:max_xattr_size 65536 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:required_client_features {} 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure 0 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:last_failure_osd_epoch 82 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:max_mds 2 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:in 0,1 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:up {0=34416,1=34420} 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:failed 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:damaged 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:stopped 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:data_pools [3] 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:metadata_pool 2 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:inline_data disabled 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:balancer 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:bal_rank_mask -1 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:standby_count_wanted 1 2024-04-16T03:17:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:qdb_cluster leader: 34416 members: 34416,34420 2024-04-16T03:17:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.bdpnxq{0:34416} state up:active seq 469 join_fscid=1 addr [v2:172.21.15.151:6826/2013103473,v1:172.21.15.151:6828/2013103473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:17:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.ogdcjk{0:44311} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6824/1413640526,v1:172.21.15.153:6825/1413640526] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:17:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi151.slrqog{1:34420} state up:active seq 464 join_fscid=1 addr [v2:172.21.15.151:6827/2926701426,v1:172.21.15.151:6829/2926701426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:17:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.cephfs.smithi153.mqbgav{1:44863} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/3861689149,v1:172.21.15.153:6827/3861689149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T03:17:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:31.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.525+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7e4c06ecc0 msgr2=0x7f7e4c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:31.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.525+0000 7f7e65b74700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7e4c06ecc0 0x7f7e4c071180 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7e480059d0 tx=0x7f7e4801a040 comp rx=0 tx=0).stop 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.525+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 msgr2=0x7f7e6019b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.525+0000 7f7e65b74700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 0x7f7e6019b500 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000ba50 tx=0x7f7e5000be10 comp rx=0 tx=0).stop 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.525+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 shutdown_connections 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.525+0000 7f7e65b74700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7e4c06ecc0 0x7f7e4c071180 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.525+0000 7f7e65b74700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e600ffbd0 0x7f7e6019b500 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.525+0000 7f7e65b74700 1 --2- 172.21.15.151:0/301328079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e60105a40 0x7f7e6019ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.525+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 >> 172.21.15.151:0/301328079 conn(0x7f7e600755a0 msgr2=0x7f7e600fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.526+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 shutdown_connections 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:31.526+0000 7f7e65b74700 1 -- 172.21.15.151:0/301328079 wait complete. 2024-04-16T03:17:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 42 2024-04-16T03:17:32.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:32 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/301328079' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:17:32.288 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-16T03:17:32.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T03:17:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.412+0000 7f3b0ae50640 1 -- 172.21.15.151:0/1195595755 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 msgr2=0x7f3b040feb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.412+0000 7f3b0ae50640 1 --2- 172.21.15.151:0/1195595755 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 0x7f3b040feb80 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009880 tx=0x7f3af402bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.412+0000 7f3b0ae50640 1 -- 172.21.15.151:0/1195595755 shutdown_connections 2024-04-16T03:17:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.412+0000 7f3b0ae50640 1 --2- 172.21.15.151:0/1195595755 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 0x7f3b040feb80 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.412+0000 7f3b0ae50640 1 --2- 172.21.15.151:0/1195595755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b04106200 0x7f3b041065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.412+0000 7f3b0ae50640 1 -- 172.21.15.151:0/1195595755 >> 172.21.15.151:0/1195595755 conn(0x7f3b040752f0 msgr2=0x7f3b04075700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.412+0000 7f3b0ae50640 1 -- 172.21.15.151:0/1195595755 shutdown_connections 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.412+0000 7f3b0ae50640 1 -- 172.21.15.151:0/1195595755 wait complete. 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.412+0000 7f3b0ae50640 1 Processor -- start 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b0ae50640 1 -- start start 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b0ae50640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 0x7f3b0419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b0ae50640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b04106200 0x7f3b0419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b0ae50640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b04195ab0 con 0x7f3b040fe720 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b0ae50640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b04195c20 con 0x7f3b04106200 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b08bec640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 0x7f3b0419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b08bec640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 0x7f3b0419ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57246/0 (socket says 172.21.15.151:57246) 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b08bec640 1 -- 172.21.15.151:0/296690070 learned_addr learned my addr 172.21.15.151:0/296690070 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3afbfff640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b04106200 0x7f3b0419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b08bec640 1 -- 172.21.15.151:0/296690070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b04106200 msgr2=0x7f3b0419bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b08bec640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b04106200 0x7f3b0419bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b08bec640 1 -- 172.21.15.151:0/296690070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af4009560 con 0x7f3b040fe720 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3afbfff640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b04106200 0x7f3b0419bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b08bec640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 0x7f3b0419ba40 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000d8d0 tx=0x7f3b0000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3af9ffb640 1 -- 172.21.15.151:0/296690070 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b0000bea0 con 0x7f3b040fe720 2024-04-16T03:17:32.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3af9ffb640 1 -- 172.21.15.151:0/296690070 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b00009c00 con 0x7f3b040fe720 2024-04-16T03:17:32.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.413+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b04195f00 con 0x7f3b040fe720 2024-04-16T03:17:32.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.414+0000 7f3af9ffb640 1 -- 172.21.15.151:0/296690070 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b0000f710 con 0x7f3b040fe720 2024-04-16T03:17:32.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.414+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b04196460 con 0x7f3b040fe720 2024-04-16T03:17:32.417 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.415+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b0404e040 con 0x7f3b040fe720 2024-04-16T03:17:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.415+0000 7f3af9ffb640 1 -- 172.21.15.151:0/296690070 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3b0000f990 con 0x7f3b040fe720 2024-04-16T03:17:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.416+0000 7f3af9ffb640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3aec06ecf0 0x7f3aec0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.416+0000 7f3afbfff640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3aec06ecf0 0x7f3aec0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.416+0000 7f3af9ffb640 1 -- 172.21.15.151:0/296690070 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3b0008ebb0 con 0x7f3b040fe720 2024-04-16T03:17:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.416+0000 7f3afbfff640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3aec06ecf0 0x7f3aec0711b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3af4005ec0 tx=0x7f3af4005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:32.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.421+0000 7f3af9ffb640 1 -- 172.21.15.151:0/296690070 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3b00095080 con 0x7f3b040fe720 2024-04-16T03:17:32.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:32 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/301328079' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T03:17:32.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.607+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b04196750 con 0x7f3b040fe720 2024-04-16T03:17:32.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.608+0000 7f3af9ffb640 1 -- 172.21.15.151:0/296690070 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16697 (secure 0 0 0) 0x7f3b00057960 con 0x7f3b040fe720 2024-04-16T03:17:32.609 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:32.610 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":86,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","created":"2024-04-16T02:25:34.606206+0000","modified":"2024-04-16T03:00:19.177856+0000","last_up_change":"2024-04-16T02:44:24.227194+0000","last_in_change":"2024-04-16T02:30:58.536477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T02:30:03.137221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T02:32:33.681623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T02:32:34.178442+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8da4d50b-626b-4cf3-9524-ea84e7733113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6803","nonce":1997175184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6805","nonce":1997175184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6809","nonce":1997175184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6807","nonce":1997175184}]},"public_addr":"172.21.15.151:6803/1997175184","cluster_addr":"172.21.15.151:6805/1997175184","heartbeat_back_addr":"172.21.15.151:6809/1997175184","heartbeat_front_addr":"172.21.15.151:6807/1997175184","state":["exists","up"]},{"osd":1,"uuid":"8d2980b5-1d30-445c-91ae-f08074b6d8f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6811","nonce":4084991031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6813","nonce":4084991031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6817","nonce":4084991031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6815","nonce":4084991031}]},"public_addr":"172.21.15.151:6811/4084991031","cluster_addr":"172.21.15.151:6813/4084991031","heartbeat_back_addr":"172.21.15.151:6817/4084991031","heartbeat_front_addr":"172.21.15.151:6815/4084991031","state":["exists","up"]},{"osd":2,"uuid":"64e4e7ce-5c3c-48c8-be3d-c9936b6866b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6819","nonce":3234791793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6821","nonce":3234791793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6825","nonce":3234791793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6823","nonce":3234791793}]},"public_addr":"172.21.15.151:6819/3234791793","cluster_addr":"172.21.15.151:6821/3234791793","heartbeat_back_addr":"172.21.15.151:6825/3234791793","heartbeat_front_addr":"172.21.15.151:6823/3234791793","state":["exists","up"]},{"osd":3,"uuid":"1194f132-bf2d-4e0c-ab93-f3344ff64d76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6801","nonce":1577343223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6803","nonce":1577343223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6807","nonce":1577343223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6805","nonce":1577343223}]},"public_addr":"172.21.15.153:6801/1577343223","cluster_addr":"172.21.15.153:6803/1577343223","heartbeat_back_addr":"172.21.15.153:6807/1577343223","heartbeat_front_addr":"172.21.15.153:6805/1577343223","state":["exists","up"]},{"osd":4,"uuid":"9d803d8b-eb35-4a0f-9d36-de11b743a123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6809","nonce":290685586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6811","nonce":290685586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6815","nonce":290685586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6813","nonce":290685586}]},"public_addr":"172.21.15.153:6809/290685586","cluster_addr":"172.21.15.153:6811/290685586","heartbeat_back_addr":"172.21.15.153:6815/290685586","heartbeat_front_addr":"172.21.15.153:6813/290685586","state":["exists","up"]},{"osd":5,"uuid":"7d0d2460-ef76-481b-ad51-afb80d05bc72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6817","nonce":45452282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6819","nonce":45452282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6823","nonce":45452282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6821","nonce":45452282}]},"public_addr":"172.21.15.153:6817/45452282","cluster_addr":"172.21.15.153:6819/45452282","heartbeat_back_addr":"172.21.15.153:6823/45452282","heartbeat_front_addr":"172.21.15.153:6821/45452282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:02.729782+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:31.268279+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:59.891666+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:24.930390+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:50.555079+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:31:15.969056+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6827/1662886502":"2024-04-17T02:44:25.408644+0000","172.21.15.151:6826/1662886502":"2024-04-17T02:44:25.408644+0000","172.21.15.151:6829/37453862":"2024-04-17T02:44:25.408661+0000","172.21.15.151:0/2684608614":"2024-04-17T02:26:34.117371+0000","172.21.15.151:0/3091708247":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/681541897":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/969025109":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/671603278":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/3129110405":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3798542210":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/2753390843":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/2402350166":"2024-04-17T02:39:53.700570+0000","172.21.15.151:6800/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.153:0/986410000":"2024-04-17T02:36:50.207364+0000","172.21.15.151:0/4272586248":"2024-04-17T02:26:01.417600+0000","172.21.15.151:6801/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3454798699":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.153:0/3590885768":"2024-04-17T02:36:50.207364+0000","172.21.15.153:6829/2287826646":"2024-04-17T02:36:50.207364+0000","172.21.15.151:6828/37453862":"2024-04-17T02:44:25.408661+0000","172.21.15.151:0/2293925528":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/1505356478":"2024-04-17T02:39:53.700570+0000","172.21.15.151:0/1299604123":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/661820069":"2024-04-17T02:35:10.400124+0000","172.21.15.153:6824/1319024054":"2024-04-17T02:32:48.923574+0000","172.21.15.151:0/3077174249":"2024-04-17T02:39:53.700570+0000","172.21.15.153:6825/1319024054":"2024-04-17T02:32:48.923574+0000","172.21.15.151:6800/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.153:0/977218519":"2024-04-17T02:36:50.207364+0000","172.21.15.151:6800/2753390843":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/1923729027":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6801/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.153:6828/2287826646":"2024-04-17T02:36:50.207364+0000","172.21.15.153:0/3588036861":"2024-04-17T02:36:50.207364+0000","172.21.15.151:6800/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:6800/1243156277":"2024-04-17T02:39:53.700570+0000","172.21.15.151:6801/1243156277":"2024-04-17T02:39:53.700570+0000","172.21.15.151:0/1770724784":"2024-04-17T02:39:53.700570+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T03:17:32.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.610+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3aec06ecf0 msgr2=0x7f3aec0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.610+0000 7f3b0ae50640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3aec06ecf0 0x7f3aec0711b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3af4005ec0 tx=0x7f3af4005fd0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.610+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 msgr2=0x7f3b0419ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.610+0000 7f3b0ae50640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 0x7f3b0419ba40 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000d8d0 tx=0x7f3b0000dd90 comp rx=0 tx=0).stop 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.610+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 shutdown_connections 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.610+0000 7f3b0ae50640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3aec06ecf0 0x7f3aec0711b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.610+0000 7f3b0ae50640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b04106200 0x7f3b0419bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.610+0000 7f3b0ae50640 1 --2- 172.21.15.151:0/296690070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b040fe720 0x7f3b0419ba40 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.610+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 >> 172.21.15.151:0/296690070 conn(0x7f3b040752f0 msgr2=0x7f3b040fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.611+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 shutdown_connections 2024-04-16T03:17:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.611+0000 7f3b0ae50640 1 -- 172.21.15.151:0/296690070 wait complete. 2024-04-16T03:17:32.623 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T02:30:03.137221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T02:32:33.681623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T02:32:34.178442+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T03:17:32.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T03:17:32.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.730+0000 7fbdcea41640 1 -- 172.21.15.151:0/328266831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc80ff1a0 msgr2=0x7fbdc80ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:32.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.730+0000 7fbdcea41640 1 --2- 172.21.15.151:0/328266831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc80ff1a0 0x7fbdc80ff580 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009880 tx=0x7fbdb802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.730+0000 7fbdcea41640 1 -- 172.21.15.151:0/328266831 shutdown_connections 2024-04-16T03:17:32.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.730+0000 7fbdcea41640 1 --2- 172.21.15.151:0/328266831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc80ffb50 0x7fbdc8103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.730+0000 7fbdcea41640 1 --2- 172.21.15.151:0/328266831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc80ff1a0 0x7fbdc80ff580 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.730+0000 7fbdcea41640 1 -- 172.21.15.151:0/328266831 >> 172.21.15.151:0/328266831 conn(0x7fbdc80faa60 msgr2=0x7fbdc80fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdcea41640 1 -- 172.21.15.151:0/328266831 shutdown_connections 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdcea41640 1 -- 172.21.15.151:0/328266831 wait complete. 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdcea41640 1 Processor -- start 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdcea41640 1 -- start start 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdcea41640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc80ff1a0 0x7fbdc8195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdcea41640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc81962e0 0x7fbdc819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdcea41640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc81968e0 con 0x7fbdc81962e0 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdcea41640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8196a50 con 0x7fbdc80ff1a0 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdbffff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc81962e0 0x7fbdc819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdbffff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc81962e0 0x7fbdc819a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57254/0 (socket says 172.21.15.151:57254) 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.731+0000 7fbdbffff640 1 -- 172.21.15.151:0/150464719 learned_addr learned my addr 172.21.15.151:0/150464719 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.732+0000 7fbdbffff640 1 -- 172.21.15.151:0/150464719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc80ff1a0 msgr2=0x7fbdc8195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:32.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.732+0000 7fbdbffff640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc80ff1a0 0x7fbdc8195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.732+0000 7fbdbffff640 1 -- 172.21.15.151:0/150464719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb8009560 con 0x7fbdc81962e0 2024-04-16T03:17:32.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.732+0000 7fbdbffff640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc81962e0 0x7fbdc819a740 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000d6f0 tx=0x7fbdb000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:32.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.732+0000 7fbdc5ffb640 1 -- 172.21.15.151:0/150464719 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdb00051f0 con 0x7fbdc81962e0 2024-04-16T03:17:32.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.732+0000 7fbdc5ffb640 1 -- 172.21.15.151:0/150464719 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdb0009ce0 con 0x7fbdc81962e0 2024-04-16T03:17:32.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.732+0000 7fbdc5ffb640 1 -- 172.21.15.151:0/150464719 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdb0004e80 con 0x7fbdc81962e0 2024-04-16T03:17:32.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.732+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc819ad40 con 0x7fbdc81962e0 2024-04-16T03:17:32.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.732+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc819b2a0 con 0x7fbdc81962e0 2024-04-16T03:17:32.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.733+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc819b570 con 0x7fbdc81962e0 2024-04-16T03:17:32.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.733+0000 7fbdc5ffb640 1 -- 172.21.15.151:0/150464719 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fbdb00046c0 con 0x7fbdc81962e0 2024-04-16T03:17:32.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.734+0000 7fbdc5ffb640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbdb406ec30 0x7fbdb40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:32.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.734+0000 7fbdc5ffb640 1 -- 172.21.15.151:0/150464719 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbdb008e7b0 con 0x7fbdc81962e0 2024-04-16T03:17:32.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.734+0000 7fbdc7fff640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbdb406ec30 0x7fbdb40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:32.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.734+0000 7fbdc7fff640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbdb406ec30 0x7fbdb40710f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8036040 tx=0x7fbdb8000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:32.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.739+0000 7fbdc5ffb640 1 -- 172.21.15.151:0/150464719 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbdb00575e0 con 0x7fbdc81962e0 2024-04-16T03:17:32.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.916+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbdc80ffbe0 con 0x7fbdc81962e0 2024-04-16T03:17:32.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.917+0000 7fbdc5ffb640 1 -- 172.21.15.151:0/150464719 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbdb00170a0 con 0x7fbdc81962e0 2024-04-16T03:17:32.918 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 1 2024-04-16T03:17:32.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbdb406ec30 msgr2=0x7fbdb40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:32.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbdb406ec30 0x7fbdb40710f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8036040 tx=0x7fbdb8000bc0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc81962e0 msgr2=0x7fbdc819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:32.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc81962e0 0x7fbdc819a740 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000d6f0 tx=0x7fbdb000dbb0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 shutdown_connections 2024-04-16T03:17:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fbdb406ec30 0x7fbdb40710f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbdc81962e0 0x7fbdc819a740 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 --2- 172.21.15.151:0/150464719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc80ff1a0 0x7fbdc8195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 >> 172.21.15.151:0/150464719 conn(0x7fbdc80faa60 msgr2=0x7fbdc8103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 shutdown_connections 2024-04-16T03:17:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:32.919+0000 7fbdcea41640 1 -- 172.21.15.151:0/150464719 wait complete. 2024-04-16T03:17:32.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T03:17:33.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/987095323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 msgr2=0x7f4acc0ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 --2- 172.21.15.151:0/987095323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 0x7f4acc0ff640 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009c20 tx=0x7f4abc02c1b0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/987095323 shutdown_connections 2024-04-16T03:17:33.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 --2- 172.21.15.151:0/987095323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4acc0ffc10 0x7f4acc103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 --2- 172.21.15.151:0/987095323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 0x7f4acc0ff640 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/987095323 >> 172.21.15.151:0/987095323 conn(0x7f4acc074a50 msgr2=0x7f4acc074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:33.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/987095323 shutdown_connections 2024-04-16T03:17:33.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/987095323 wait complete. 2024-04-16T03:17:33.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 Processor -- start 2024-04-16T03:17:33.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.039+0000 7f4ad0ddd640 1 -- start start 2024-04-16T03:17:33.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4ad0ddd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 0x7f4acc195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:33.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4ad0ddd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4acc196370 0x7f4acc19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:33.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4ad0ddd640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc196970 con 0x7f4acc0ff260 2024-04-16T03:17:33.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4ad0ddd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc196ae0 con 0x7f4acc196370 2024-04-16T03:17:33.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4aca59c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 0x7f4acc195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:33.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4aca59c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 0x7f4acc195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57272/0 (socket says 172.21.15.151:57272) 2024-04-16T03:17:33.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4aca59c640 1 -- 172.21.15.151:0/3648319492 learned_addr learned my addr 172.21.15.151:0/3648319492 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4aca59c640 1 -- 172.21.15.151:0/3648319492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4acc196370 msgr2=0x7f4acc19a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4aca59c640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4acc196370 0x7f4acc19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4aca59c640 1 -- 172.21.15.151:0/3648319492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc009900 con 0x7f4acc0ff260 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4aca59c640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 0x7f4acc195e30 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f4abc006010 tx=0x7f4abc032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4abb7fe640 1 -- 172.21.15.151:0/3648319492 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4abc034820 con 0x7f4acc0ff260 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4acc19ad70 con 0x7f4acc0ff260 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4abb7fe640 1 -- 172.21.15.151:0/3648319492 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4abc003e30 con 0x7f4acc0ff260 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4abb7fe640 1 -- 172.21.15.151:0/3648319492 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4abc03f440 con 0x7f4acc0ff260 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.040+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4acc19b2d0 con 0x7f4acc0ff260 2024-04-16T03:17:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.041+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4acc04e040 con 0x7f4acc0ff260 2024-04-16T03:17:33.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.042+0000 7f4abb7fe640 1 -- 172.21.15.151:0/3648319492 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4abc03b040 con 0x7f4acc0ff260 2024-04-16T03:17:33.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.042+0000 7f4abb7fe640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ab406ec30 0x7f4ab40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:33.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.042+0000 7f4abb7fe640 1 -- 172.21.15.151:0/3648319492 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4abc0b1cc0 con 0x7f4acc0ff260 2024-04-16T03:17:33.047 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.042+0000 7f4ac9d9b640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ab406ec30 0x7f4ab40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:33.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.043+0000 7f4ac9d9b640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ab406ec30 0x7f4ab40710f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4acc072e40 tx=0x7f4ac0005bf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:33.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.047+0000 7f4abb7fe640 1 -- 172.21.15.151:0/3648319492 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4abc07aa70 con 0x7f4acc0ff260 2024-04-16T03:17:33.232 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.231+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f4acc197180 con 0x7f4acc0ff260 2024-04-16T03:17:33.233 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.232+0000 7f4abb7fe640 1 -- 172.21.15.151:0/3648319492 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f4abc07e130 con 0x7f4acc0ff260 2024-04-16T03:17:33.233 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 32 2024-04-16T03:17:33.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ab406ec30 msgr2=0x7f4ab40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ab406ec30 0x7f4ab40710f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4acc072e40 tx=0x7f4ac0005bf0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 msgr2=0x7f4acc195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 0x7f4acc195e30 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f4abc006010 tx=0x7f4abc032b60 comp rx=0 tx=0).stop 2024-04-16T03:17:33.235 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 shutdown_connections 2024-04-16T03:17:33.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4ab406ec30 0x7f4ab40710f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4acc196370 0x7f4acc19a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 --2- 172.21.15.151:0/3648319492 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4acc0ff260 0x7f4acc195e30 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 >> 172.21.15.151:0/3648319492 conn(0x7f4acc074a50 msgr2=0x7f4acc1033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:33.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 shutdown_connections 2024-04-16T03:17:33.236 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.234+0000 7f4ad0ddd640 1 -- 172.21.15.151:0/3648319492 wait complete. 2024-04-16T03:17:33.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:33 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/296690070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T03:17:33.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:33 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/150464719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T03:17:33.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:33 smithi151 ceph-mon[112233]: pgmap v1171: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:33.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T03:17:33.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.356+0000 7fc741999640 1 -- 172.21.15.151:0/1553599845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 msgr2=0x7fc73c072ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.356+0000 7fc741999640 1 --2- 172.21.15.151:0/1553599845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 0x7fc73c072ab0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009880 tx=0x7fc72c02bd70 comp rx=0 tx=0).stop 2024-04-16T03:17:33.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.356+0000 7fc741999640 1 -- 172.21.15.151:0/1553599845 shutdown_connections 2024-04-16T03:17:33.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.356+0000 7fc741999640 1 --2- 172.21.15.151:0/1553599845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc73c072ff0 0x7fc73c10c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.356+0000 7fc741999640 1 --2- 172.21.15.151:0/1553599845 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 0x7fc73c072ab0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.356+0000 7fc741999640 1 -- 172.21.15.151:0/1553599845 >> 172.21.15.151:0/1553599845 conn(0x7fc73c0fbb20 msgr2=0x7fc73c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.356+0000 7fc741999640 1 -- 172.21.15.151:0/1553599845 shutdown_connections 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.356+0000 7fc741999640 1 -- 172.21.15.151:0/1553599845 wait complete. 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc741999640 1 Processor -- start 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc741999640 1 -- start start 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc741999640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 0x7fc73c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc741999640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc73c072ff0 0x7fc73c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc741999640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c198bb0 con 0x7fc73c0726d0 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc741999640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c19c980 con 0x7fc73c072ff0 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc73affd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 0x7fc73c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc73affd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 0x7fc73c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57296/0 (socket says 172.21.15.151:57296) 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc73affd640 1 -- 172.21.15.151:0/1476982056 learned_addr learned my addr 172.21.15.151:0/1476982056 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc7327fc640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc73c072ff0 0x7fc73c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc73affd640 1 -- 172.21.15.151:0/1476982056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc73c072ff0 msgr2=0x7fc73c198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc73affd640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc73c072ff0 0x7fc73c198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc73affd640 1 -- 172.21.15.151:0/1476982056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c009560 con 0x7fc73c0726d0 2024-04-16T03:17:33.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.357+0000 7fc7327fc640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc73c072ff0 0x7fc73c198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:17:33.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.358+0000 7fc73affd640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 0x7fc73c197fe0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009230 tx=0x7fc72c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:33.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.358+0000 7fc740997640 1 -- 172.21.15.151:0/1476982056 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc72c0037c0 con 0x7fc73c0726d0 2024-04-16T03:17:33.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.358+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c19cc00 con 0x7fc73c0726d0 2024-04-16T03:17:33.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.358+0000 7fc740997640 1 -- 172.21.15.151:0/1476982056 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc72c004aa0 con 0x7fc73c0726d0 2024-04-16T03:17:33.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.358+0000 7fc740997640 1 -- 172.21.15.151:0/1476982056 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc72c03e450 con 0x7fc73c0726d0 2024-04-16T03:17:33.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.358+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c19d160 con 0x7fc73c0726d0 2024-04-16T03:17:33.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.359+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c04e040 con 0x7fc73c0726d0 2024-04-16T03:17:33.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.359+0000 7fc740997640 1 -- 172.21.15.151:0/1476982056 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc72c005060 con 0x7fc73c0726d0 2024-04-16T03:17:33.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.360+0000 7fc740997640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc72806ec30 0x7fc7280710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:33.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.360+0000 7fc740997640 1 -- 172.21.15.151:0/1476982056 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc72c0b1cd0 con 0x7fc73c0726d0 2024-04-16T03:17:33.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.360+0000 7fc7327fc640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc72806ec30 0x7fc7280710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:33.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.360+0000 7fc7327fc640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc72806ec30 0x7fc7280710f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc73c1995f0 tx=0x7fc724005d30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:33.365 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.364+0000 7fc740997640 1 -- 172.21.15.151:0/1476982056 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc72c07aa80 con 0x7fc73c0726d0 2024-04-16T03:17:33.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:33 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/296690070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T03:17:33.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:33 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/150464719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T03:17:33.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:33 smithi153 ceph-mon[87858]: pgmap v1171: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:33.547 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.546+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fc73c10f4d0 con 0x7fc73c0726d0 2024-04-16T03:17:33.548 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.547+0000 7fc740997640 1 -- 172.21.15.151:0/1476982056 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc72c07e140 con 0x7fc73c0726d0 2024-04-16T03:17:33.548 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 32 2024-04-16T03:17:33.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc72806ec30 msgr2=0x7fc7280710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc72806ec30 0x7fc7280710f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc73c1995f0 tx=0x7fc724005d30 comp rx=0 tx=0).stop 2024-04-16T03:17:33.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 msgr2=0x7fc73c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.550 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 0x7fc73c197fe0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009230 tx=0x7fc72c004270 comp rx=0 tx=0).stop 2024-04-16T03:17:33.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 shutdown_connections 2024-04-16T03:17:33.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc72806ec30 0x7fc7280710f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc73c072ff0 0x7fc73c198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 --2- 172.21.15.151:0/1476982056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc73c0726d0 0x7fc73c197fe0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 >> 172.21.15.151:0/1476982056 conn(0x7fc73c0fbb20 msgr2=0x7fc73c106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:33.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 shutdown_connections 2024-04-16T03:17:33.551 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.549+0000 7fc741999640 1 -- 172.21.15.151:0/1476982056 wait complete. 2024-04-16T03:17:33.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.665+0000 7fc1f5f92640 1 -- 172.21.15.151:0/3637583153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f00ff180 msgr2=0x7fc1f00ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.665+0000 7fc1f5f92640 1 --2- 172.21.15.151:0/3637583153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f00ff180 0x7fc1f00ff560 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009880 tx=0x7fc1e002bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.665+0000 7fc1f5f92640 1 -- 172.21.15.151:0/3637583153 shutdown_connections 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.665+0000 7fc1f5f92640 1 --2- 172.21.15.151:0/3637583153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f00ffb30 0x7fc1f0103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.665+0000 7fc1f5f92640 1 --2- 172.21.15.151:0/3637583153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f00ff180 0x7fc1f00ff560 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.665+0000 7fc1f5f92640 1 -- 172.21.15.151:0/3637583153 >> 172.21.15.151:0/3637583153 conn(0x7fc1f00fa9e0 msgr2=0x7fc1f00fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.665+0000 7fc1f5f92640 1 -- 172.21.15.151:0/3637583153 shutdown_connections 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.665+0000 7fc1f5f92640 1 -- 172.21.15.151:0/3637583153 wait complete. 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.665+0000 7fc1f5f92640 1 Processor -- start 2024-04-16T03:17:33.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1f5f92640 1 -- start start 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1f5f92640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f00ff180 0x7fc1f0196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1f5f92640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f0197070 0x7fc1f01974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1f5f92640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f019fba0 con 0x7fc1f0197070 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1f5f92640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f019fce0 con 0x7fc1f00ff180 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1e7fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f0197070 0x7fc1f01974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1e7fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f0197070 0x7fc1f01974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57316/0 (socket says 172.21.15.151:57316) 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1e7fff640 1 -- 172.21.15.151:0/95675977 learned_addr learned my addr 172.21.15.151:0/95675977 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1ef7fe640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f00ff180 0x7fc1f0196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1e7fff640 1 -- 172.21.15.151:0/95675977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f00ff180 msgr2=0x7fc1f0196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1e7fff640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f00ff180 0x7fc1f0196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1e7fff640 1 -- 172.21.15.151:0/95675977 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e0009560 con 0x7fc1f0197070 2024-04-16T03:17:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1e7fff640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f0197070 0x7fc1f01974d0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800d6f0 tx=0x7fc1d800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:33.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1ed7fa640 1 -- 172.21.15.151:0/95675977 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1d80051f0 con 0x7fc1f0197070 2024-04-16T03:17:33.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1ed7fa640 1 -- 172.21.15.151:0/95675977 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1d8009ce0 con 0x7fc1f0197070 2024-04-16T03:17:33.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1ed7fa640 1 -- 172.21.15.151:0/95675977 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1d8004e80 con 0x7fc1f0197070 2024-04-16T03:17:33.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.666+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f019ff40 con 0x7fc1f0197070 2024-04-16T03:17:33.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.667+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f01a0420 con 0x7fc1f0197070 2024-04-16T03:17:33.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.667+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f004e040 con 0x7fc1f0197070 2024-04-16T03:17:33.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.668+0000 7fc1ed7fa640 1 -- 172.21.15.151:0/95675977 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc1d80046c0 con 0x7fc1f0197070 2024-04-16T03:17:33.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.668+0000 7fc1ed7fa640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc1dc06ecf0 0x7fc1dc0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:33.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.669+0000 7fc1ed7fa640 1 -- 172.21.15.151:0/95675977 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc1d808e7f0 con 0x7fc1f0197070 2024-04-16T03:17:33.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.669+0000 7fc1ef7fe640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc1dc06ecf0 0x7fc1dc0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:33.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.669+0000 7fc1ef7fe640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc1dc06ecf0 0x7fc1dc0711b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0036040 tx=0x7fc1e0005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:33.674 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.673+0000 7fc1ed7fa640 1 -- 172.21.15.151:0/95675977 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc1d8057620 con 0x7fc1f0197070 2024-04-16T03:17:33.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.889+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1f0107480 con 0x7fc1f0197070 2024-04-16T03:17:33.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.890+0000 7fc1ed7fa640 1 -- 172.21.15.151:0/95675977 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5260 (secure 0 0 0) 0x7fc1d805ace0 con 0x7fc1f0197070 2024-04-16T03:17:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:33.892 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T03:15:59.179916+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":35,"state":"up:active","state_seq":469,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":35,"state":"up:active","state_seq":464,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44311":{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44863":{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34416,"qdb_cluster":[34416,34420]},"id":1}]} 2024-04-16T03:17:33.892 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 42 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc1dc06ecf0 msgr2=0x7fc1dc0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc1dc06ecf0 0x7fc1dc0711b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0036040 tx=0x7fc1e0005fd0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f0197070 msgr2=0x7fc1f01974d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f0197070 0x7fc1f01974d0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800d6f0 tx=0x7fc1d800dbb0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 shutdown_connections 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc1dc06ecf0 0x7fc1dc0711b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1f0197070 0x7fc1f01974d0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 --2- 172.21.15.151:0/95675977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f00ff180 0x7fc1f0196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 >> 172.21.15.151:0/95675977 conn(0x7fc1f00fa9e0 msgr2=0x7fc1f00fc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 shutdown_connections 2024-04-16T03:17:33.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:33.892+0000 7fc1f5f92640 1 -- 172.21.15.151:0/95675977 wait complete. 2024-04-16T03:17:33.904 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-04-16T03:17:33.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-16T03:17:34.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.008+0000 7fd82d284640 1 -- 172.21.15.151:0/3525237156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 msgr2=0x7fd82810a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.011 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.008+0000 7fd82d284640 1 --2- 172.21.15.151:0/3525237156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 0x7fd82810a000 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009880 tx=0x7fd81c02bd70 comp rx=0 tx=0).stop 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.008+0000 7fd82d284640 1 -- 172.21.15.151:0/3525237156 shutdown_connections 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.008+0000 7fd82d284640 1 --2- 172.21.15.151:0/3525237156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 0x7fd82810a000 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.008+0000 7fd82d284640 1 --2- 172.21.15.151:0/3525237156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd828101200 0x7fd8281015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.008+0000 7fd82d284640 1 -- 172.21.15.151:0/3525237156 >> 172.21.15.151:0/3525237156 conn(0x7fd8280faa20 msgr2=0x7fd8280fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.008+0000 7fd82d284640 1 -- 172.21.15.151:0/3525237156 shutdown_connections 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.008+0000 7fd82d284640 1 -- 172.21.15.151:0/3525237156 wait complete. 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd82d284640 1 Processor -- start 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd82d284640 1 -- start start 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd82d284640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd828101200 0x7fd828195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd82d284640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 0x7fd828196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd82d284640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd828196920 con 0x7fd828101b20 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd82d284640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82819a6f0 con 0x7fd828101200 2024-04-16T03:17:34.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd8267fc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 0x7fd828196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd8267fc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 0x7fd828196290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57320/0 (socket says 172.21.15.151:57320) 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd8267fc640 1 -- 172.21.15.151:0/1587425888 learned_addr learned my addr 172.21.15.151:0/1587425888 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd826ffd640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd828101200 0x7fd828195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.009+0000 7fd8267fc640 1 -- 172.21.15.151:0/1587425888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd828101200 msgr2=0x7fd828195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.010+0000 7fd8267fc640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd828101200 0x7fd828195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.010+0000 7fd8267fc640 1 -- 172.21.15.151:0/1587425888 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c009560 con 0x7fd828101b20 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.010+0000 7fd826ffd640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd828101200 0x7fd828195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.010+0000 7fd8267fc640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 0x7fd828196290 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009230 tx=0x7fd81c02c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.010+0000 7fd80ffff640 1 -- 172.21.15.151:0/1587425888 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd81c004520 con 0x7fd828101b20 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.010+0000 7fd80ffff640 1 -- 172.21.15.151:0/1587425888 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd81c003ab0 con 0x7fd828101b20 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.010+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82819a970 con 0x7fd828101b20 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.010+0000 7fd80ffff640 1 -- 172.21.15.151:0/1587425888 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd81c005690 con 0x7fd828101b20 2024-04-16T03:17:34.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.010+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82819aed0 con 0x7fd828101b20 2024-04-16T03:17:34.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.011+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82804e040 con 0x7fd828101b20 2024-04-16T03:17:34.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.011+0000 7fd80ffff640 1 -- 172.21.15.151:0/1587425888 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fd81c004ed0 con 0x7fd828101b20 2024-04-16T03:17:34.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.012+0000 7fd80ffff640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ebe0 0x7fd8100710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:34.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.012+0000 7fd80ffff640 1 -- 172.21.15.151:0/1587425888 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd81c0b1a10 con 0x7fd828101b20 2024-04-16T03:17:34.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.012+0000 7fd826ffd640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ebe0 0x7fd8100710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:34.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.012+0000 7fd826ffd640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ebe0 0x7fd8100710a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd818005fd0 tx=0x7fd818005ea0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:34.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.017+0000 7fd80ffff640 1 -- 172.21.15.151:0/1587425888 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd81c07a7c0 con 0x7fd828101b20 2024-04-16T03:17:34.237 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.236+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7fd828107670 con 0x7fd828101b20 2024-04-16T03:17:34.237 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.236+0000 7fd80ffff640 1 -- 172.21.15.151:0/1587425888 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v42) v1 ==== 107+0+4895 (secure 0 0 0) 0x7fd81c07de80 con 0x7fd828101b20 2024-04-16T03:17:34.237 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:34.238 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:34:49.983507+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14478,"mds_1":14490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14478":{"gid":14478,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":3,"state":"up:active","state_seq":4,"addr":"172.21.15.151:6827/1662886502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1662886502},{"type":"v1","addr":"172.21.15.151:6827","nonce":1662886502}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.151:6829/37453862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":37453862},{"type":"v1","addr":"172.21.15.151:6829","nonce":37453862}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14502":{"gid":14502,"name":"cephfs.smithi153.ogdcjk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6825/3333609621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3333609621},{"type":"v1","addr":"172.21.15.153:6825","nonce":3333609621}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24275":{"gid":24275,"name":"cephfs.smithi153.mqbgav","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.153:6827/2749127439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2749127439},{"type":"v1","addr":"172.21.15.153:6827","nonce":2749127439}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:34.238 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 19 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.238+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ebe0 msgr2=0x7fd8100710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.238+0000 7fd82d284640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ebe0 0x7fd8100710a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd818005fd0 tx=0x7fd818005ea0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.238+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 msgr2=0x7fd828196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.238+0000 7fd82d284640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 0x7fd828196290 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009230 tx=0x7fd81c02c870 comp rx=0 tx=0).stop 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.238+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 shutdown_connections 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.238+0000 7fd82d284640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fd81006ebe0 0x7fd8100710a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.238+0000 7fd82d284640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd828101b20 0x7fd828196290 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.238+0000 7fd82d284640 1 --2- 172.21.15.151:0/1587425888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd828101200 0x7fd828195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.238+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 >> 172.21.15.151:0/1587425888 conn(0x7fd8280faa20 msgr2=0x7fd828104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.239+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 shutdown_connections 2024-04-16T03:17:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.239+0000 7fd82d284640 1 -- 172.21.15.151:0/1587425888 wait complete. 2024-04-16T03:17:34.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-16T03:17:34.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.355+0000 7f1c7315a640 1 -- 172.21.15.151:0/3860581452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ffb40 msgr2=0x7f1c6c103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.355+0000 7f1c7315a640 1 --2- 172.21.15.151:0/3860581452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ffb40 0x7f1c6c103b80 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009880 tx=0x7f1c5802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.355+0000 7f1c7315a640 1 -- 172.21.15.151:0/3860581452 shutdown_connections 2024-04-16T03:17:34.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.355+0000 7f1c7315a640 1 --2- 172.21.15.151:0/3860581452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ffb40 0x7f1c6c103b80 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.355+0000 7f1c7315a640 1 --2- 172.21.15.151:0/3860581452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c6c0ff190 0x7f1c6c0ff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.355+0000 7f1c7315a640 1 -- 172.21.15.151:0/3860581452 >> 172.21.15.151:0/3860581452 conn(0x7f1c6c0faa10 msgr2=0x7f1c6c0fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.355+0000 7f1c7315a640 1 -- 172.21.15.151:0/3860581452 shutdown_connections 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.355+0000 7f1c7315a640 1 -- 172.21.15.151:0/3860581452 wait complete. 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c7315a640 1 Processor -- start 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c7315a640 1 -- start start 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c7315a640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ff190 0x7f1c6c195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c7315a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c6c196380 0x7f1c6c19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c7315a640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c196980 con 0x7f1c6c0ff190 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c7315a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c196af0 con 0x7f1c6c196380 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c70ef6640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ff190 0x7f1c6c195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c70ef6640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ff190 0x7f1c6c195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57338/0 (socket says 172.21.15.151:57338) 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c70ef6640 1 -- 172.21.15.151:0/4029125819 learned_addr learned my addr 172.21.15.151:0/4029125819 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c6bfff640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c6c196380 0x7f1c6c19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c70ef6640 1 -- 172.21.15.151:0/4029125819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c6c196380 msgr2=0x7f1c6c19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c70ef6640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c6c196380 0x7f1c6c19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c70ef6640 1 -- 172.21.15.151:0/4029125819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c58009560 con 0x7f1c6c0ff190 2024-04-16T03:17:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c6bfff640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c6c196380 0x7f1c6c19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c70ef6640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ff190 0x7f1c6c195e40 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000b450 tx=0x7f1c6000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c69ffb640 1 -- 172.21.15.151:0/4029125819 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c60011860 con 0x7f1c6c0ff190 2024-04-16T03:17:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c69ffb640 1 -- 172.21.15.151:0/4029125819 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c60011e80 con 0x7f1c6c0ff190 2024-04-16T03:17:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c19ade0 con 0x7f1c6c0ff190 2024-04-16T03:17:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.356+0000 7f1c69ffb640 1 -- 172.21.15.151:0/4029125819 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c6000f570 con 0x7f1c6c0ff190 2024-04-16T03:17:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.357+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c19b340 con 0x7f1c6c0ff190 2024-04-16T03:17:34.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.358+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6c19b610 con 0x7f1c6c0ff190 2024-04-16T03:17:34.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.358+0000 7f1c69ffb640 1 -- 172.21.15.151:0/4029125819 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1c600103e0 con 0x7f1c6c0ff190 2024-04-16T03:17:34.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.358+0000 7f1c69ffb640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c5c06ec30 0x7f1c5c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:34.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.359+0000 7f1c69ffb640 1 -- 172.21.15.151:0/4029125819 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1c6008ecf0 con 0x7f1c6c0ff190 2024-04-16T03:17:34.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.359+0000 7f1c6bfff640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c5c06ec30 0x7f1c5c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:34.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.359+0000 7f1c6bfff640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c5c06ec30 0x7f1c5c0710f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006010 tx=0x7f1c5802cc00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:34.364 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.363+0000 7f1c69ffb640 1 -- 172.21.15.151:0/4029125819 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1c60057b20 con 0x7f1c6c0ff190 2024-04-16T03:17:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:34 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3648319492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T03:17:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:34 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1476982056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T03:17:34.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:34 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/95675977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T03:17:34.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:34 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3648319492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T03:17:34.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:34 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1476982056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T03:17:34.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:34 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/95675977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T03:17:34.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.581+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f1c6c0ffc00 con 0x7f1c6c0ff190 2024-04-16T03:17:34.582 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.582+0000 7f1c69ffb640 1 -- 172.21.15.151:0/4029125819 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v42) v1 ==== 107+0+4896 (secure 0 0 0) 0x7f1c6005b1e0 con 0x7f1c6c0ff190 2024-04-16T03:17:34.583 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:34.583 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:36:05.787890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14478,"mds_1":14490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14478":{"gid":14478,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":3,"state":"up:active","state_seq":4,"addr":"172.21.15.151:6827/1662886502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1662886502},{"type":"v1","addr":"172.21.15.151:6827","nonce":1662886502}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.151:6829/37453862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":37453862},{"type":"v1","addr":"172.21.15.151:6829","nonce":37453862}]},"join_fscid":1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14502":{"gid":14502,"name":"cephfs.smithi153.ogdcjk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6825/3333609621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3333609621},{"type":"v1","addr":"172.21.15.153:6825","nonce":3333609621}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24275":{"gid":24275,"name":"cephfs.smithi153.mqbgav","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.153:6827/2749127439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2749127439},{"type":"v1","addr":"172.21.15.153:6827","nonce":2749127439}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:34.583 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 20 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c5c06ec30 msgr2=0x7f1c5c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c5c06ec30 0x7f1c5c0710f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006010 tx=0x7f1c5802cc00 comp rx=0 tx=0).stop 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ff190 msgr2=0x7f1c6c195e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ff190 0x7f1c6c195e40 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000b450 tx=0x7f1c6000b910 comp rx=0 tx=0).stop 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 shutdown_connections 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1c5c06ec30 0x7f1c5c0710f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c6c196380 0x7f1c6c19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 --2- 172.21.15.151:0/4029125819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c6c0ff190 0x7f1c6c195e40 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 >> 172.21.15.151:0/4029125819 conn(0x7f1c6c0faa10 msgr2=0x7f1c6c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 shutdown_connections 2024-04-16T03:17:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.584+0000 7f1c7315a640 1 -- 172.21.15.151:0/4029125819 wait complete. 2024-04-16T03:17:34.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-16T03:17:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.700+0000 7f70d0b8b640 1 -- 172.21.15.151:0/3395857498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ffc10 msgr2=0x7f70cc103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.700+0000 7f70d0b8b640 1 --2- 172.21.15.151:0/3395857498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ffc10 0x7f70cc103b70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009880 tx=0x7f70bc02bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.700+0000 7f70d0b8b640 1 -- 172.21.15.151:0/3395857498 shutdown_connections 2024-04-16T03:17:34.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.700+0000 7f70d0b8b640 1 --2- 172.21.15.151:0/3395857498 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ffc10 0x7f70cc103b70 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.700+0000 7f70d0b8b640 1 --2- 172.21.15.151:0/3395857498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70cc0ff260 0x7f70cc0ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.700+0000 7f70d0b8b640 1 -- 172.21.15.151:0/3395857498 >> 172.21.15.151:0/3395857498 conn(0x7f70cc074a50 msgr2=0x7f70cc074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:34.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.700+0000 7f70d0b8b640 1 -- 172.21.15.151:0/3395857498 shutdown_connections 2024-04-16T03:17:34.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.700+0000 7f70d0b8b640 1 -- 172.21.15.151:0/3395857498 wait complete. 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.701+0000 7f70d0b8b640 1 Processor -- start 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.701+0000 7f70d0b8b640 1 -- start start 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.701+0000 7f70d0b8b640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ff260 0x7f70cc195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.701+0000 7f70d0b8b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70cc1962d0 0x7f70cc19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.701+0000 7f70d0b8b640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc1968d0 con 0x7f70cc0ff260 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.701+0000 7f70d0b8b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc196a40 con 0x7f70cc1962d0 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.701+0000 7f70ca59c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ff260 0x7f70cc195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.701+0000 7f70ca59c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ff260 0x7f70cc195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57362/0 (socket says 172.21.15.151:57362) 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.701+0000 7f70ca59c640 1 -- 172.21.15.151:0/1846549953 learned_addr learned my addr 172.21.15.151:0/1846549953 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70c9d9b640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70cc1962d0 0x7f70cc19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70ca59c640 1 -- 172.21.15.151:0/1846549953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70cc1962d0 msgr2=0x7f70cc19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70ca59c640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70cc1962d0 0x7f70cc19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70ca59c640 1 -- 172.21.15.151:0/1846549953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc009560 con 0x7f70cc0ff260 2024-04-16T03:17:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70ca59c640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ff260 0x7f70cc195d90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f70b400e9a0 tx=0x7f70b400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70c37fe640 1 -- 172.21.15.151:0/1846549953 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70b400cd50 con 0x7f70cc0ff260 2024-04-16T03:17:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70cc19ad30 con 0x7f70cc0ff260 2024-04-16T03:17:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70c37fe640 1 -- 172.21.15.151:0/1846549953 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70b400ceb0 con 0x7f70cc0ff260 2024-04-16T03:17:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70c37fe640 1 -- 172.21.15.151:0/1846549953 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70b4010660 con 0x7f70cc0ff260 2024-04-16T03:17:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.702+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70cc19b290 con 0x7f70cc0ff260 2024-04-16T03:17:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.703+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70cc19b570 con 0x7f70cc0ff260 2024-04-16T03:17:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.703+0000 7f70c37fe640 1 -- 172.21.15.151:0/1846549953 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f70b40048a0 con 0x7f70cc0ff260 2024-04-16T03:17:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.704+0000 7f70c37fe640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f70b806ec30 0x7f70b80710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.704+0000 7f70c37fe640 1 -- 172.21.15.151:0/1846549953 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f70b4014070 con 0x7f70cc0ff260 2024-04-16T03:17:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.704+0000 7f70c9d9b640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f70b806ec30 0x7f70b80710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.704+0000 7f70c9d9b640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f70b806ec30 0x7f70b80710f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f70bc036040 tx=0x7f70bc005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:34.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.708+0000 7f70c37fe640 1 -- 172.21.15.151:0/1846549953 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f70b405bd60 con 0x7f70cc0ff260 2024-04-16T03:17:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.926+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f70cc1970e0 con 0x7f70cc0ff260 2024-04-16T03:17:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.927+0000 7f70c37fe640 1 -- 172.21.15.151:0/1846549953 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v42) v1 ==== 107+0+4910 (secure 0 0 0) 0x7f70b405b8f0 con 0x7f70cc0ff260 2024-04-16T03:17:34.928 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:34.928 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:39:46.738140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14478,"mds_1":14490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14478":{"gid":14478,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":3,"state":"up:active","state_seq":4,"addr":"172.21.15.151:6827/1662886502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1662886502},{"type":"v1","addr":"172.21.15.151:6827","nonce":1662886502}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.151:6829/37453862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":37453862},{"type":"v1","addr":"172.21.15.151:6829","nonce":37453862}]},"join_fscid":1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14502":{"gid":14502,"name":"cephfs.smithi153.ogdcjk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6825/3333609621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3333609621},{"type":"v1","addr":"172.21.15.153:6825","nonce":3333609621}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24275":{"gid":24275,"name":"cephfs.smithi153.mqbgav","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.153:6827/2749127439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2749127439},{"type":"v1","addr":"172.21.15.153:6827","nonce":2749127439}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14478,"qdb_cluster":[14478,14490]},"id":1}]} 2024-04-16T03:17:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 21 2024-04-16T03:17:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f70b806ec30 msgr2=0x7f70b80710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f70b806ec30 0x7f70b80710f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f70bc036040 tx=0x7f70bc005fd0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ff260 msgr2=0x7f70cc195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ff260 0x7f70cc195d90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f70b400e9a0 tx=0x7f70b400eda0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 shutdown_connections 2024-04-16T03:17:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f70b806ec30 0x7f70b80710f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70cc1962d0 0x7f70cc19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 --2- 172.21.15.151:0/1846549953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ff260 0x7f70cc195d90 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 >> 172.21.15.151:0/1846549953 conn(0x7f70cc074a50 msgr2=0x7f70cc0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 shutdown_connections 2024-04-16T03:17:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:34.929+0000 7f70d0b8b640 1 -- 172.21.15.151:0/1846549953 wait complete. 2024-04-16T03:17:34.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-16T03:17:35.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.045+0000 7f565d421640 1 -- 172.21.15.151:0/2085780026 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101b00 msgr2=0x7f5658109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.045+0000 7f565d421640 1 --2- 172.21.15.151:0/2085780026 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101b00 0x7f5658109fe0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f56400098b0 tx=0x7f564002bcd0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.045+0000 7f565d421640 1 -- 172.21.15.151:0/2085780026 shutdown_connections 2024-04-16T03:17:35.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.045+0000 7f565d421640 1 --2- 172.21.15.151:0/2085780026 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101b00 0x7f5658109fe0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.045+0000 7f565d421640 1 --2- 172.21.15.151:0/2085780026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5658101150 0x7f5658101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.045+0000 7f565d421640 1 -- 172.21.15.151:0/2085780026 >> 172.21.15.151:0/2085780026 conn(0x7f56580faa00 msgr2=0x7f56580fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:35.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.045+0000 7f565d421640 1 -- 172.21.15.151:0/2085780026 shutdown_connections 2024-04-16T03:17:35.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.045+0000 7f565d421640 1 -- 172.21.15.151:0/2085780026 wait complete. 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f565d421640 1 Processor -- start 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f565d421640 1 -- start start 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f565d421640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101150 0x7f5658195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f565d421640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5658101b00 0x7f56581962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f565d421640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5658196940 con 0x7f5658101150 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f565d421640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565819a6c0 con 0x7f5658101b00 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f5656ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101150 0x7f5658195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f5656ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101150 0x7f5658195d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57378/0 (socket says 172.21.15.151:57378) 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f5656ffd640 1 -- 172.21.15.151:0/514569193 learned_addr learned my addr 172.21.15.151:0/514569193 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f564ffff640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5658101b00 0x7f56581962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f5656ffd640 1 -- 172.21.15.151:0/514569193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5658101b00 msgr2=0x7f56581962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f5656ffd640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5658101b00 0x7f56581962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.046+0000 7f5656ffd640 1 -- 172.21.15.151:0/514569193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5640009560 con 0x7f5658101150 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.047+0000 7f5656ffd640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101150 0x7f5658195d70 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f564800c8f0 tx=0x7f564800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.047+0000 7f5654ff9640 1 -- 172.21.15.151:0/514569193 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5648007b80 con 0x7f5658101150 2024-04-16T03:17:35.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.047+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565819a9a0 con 0x7f5658101150 2024-04-16T03:17:35.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.047+0000 7f5654ff9640 1 -- 172.21.15.151:0/514569193 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56480054e0 con 0x7f5658101150 2024-04-16T03:17:35.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.047+0000 7f5654ff9640 1 -- 172.21.15.151:0/514569193 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f564800f450 con 0x7f5658101150 2024-04-16T03:17:35.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.047+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565819aed0 con 0x7f5658101150 2024-04-16T03:17:35.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.048+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565804e040 con 0x7f5658101150 2024-04-16T03:17:35.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.048+0000 7f5654ff9640 1 -- 172.21.15.151:0/514569193 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5648004f60 con 0x7f5658101150 2024-04-16T03:17:35.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.048+0000 7f5654ff9640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f564406ec30 0x7f56440710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:35.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.049+0000 7f564ffff640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f564406ec30 0x7f56440710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:35.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.049+0000 7f5654ff9640 1 -- 172.21.15.151:0/514569193 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f564808ecc0 con 0x7f5658101150 2024-04-16T03:17:35.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.049+0000 7f564ffff640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f564406ec30 0x7f56440710f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5640006010 tx=0x7f5640000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:35.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.053+0000 7f5654ff9640 1 -- 172.21.15.151:0/514569193 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5648057a70 con 0x7f5658101150 2024-04-16T03:17:35.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:35 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1587425888' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-16T03:17:35.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:35 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4029125819' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-16T03:17:35.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:35 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1846549953' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-16T03:17:35.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:35 smithi151 ceph-mon[112233]: pgmap v1172: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:35.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.270+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f5658107650 con 0x7f5658101150 2024-04-16T03:17:35.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.271+0000 7f5654ff9640 1 -- 172.21.15.151:0/514569193 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v42) v1 ==== 107+0+4909 (secure 0 0 0) 0x7f564805b130 con 0x7f5658101150 2024-04-16T03:17:35.272 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:35.272 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:40:05.114230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14478,"mds_1":14490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14478":{"gid":14478,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":3,"state":"up:active","state_seq":4,"addr":"172.21.15.151:6827/1662886502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1662886502},{"type":"v1","addr":"172.21.15.151:6827","nonce":1662886502}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.151:6829/37453862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":37453862},{"type":"v1","addr":"172.21.15.151:6829","nonce":37453862}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14502":{"gid":14502,"name":"cephfs.smithi153.ogdcjk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6825/3333609621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3333609621},{"type":"v1","addr":"172.21.15.153:6825","nonce":3333609621}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24275":{"gid":24275,"name":"cephfs.smithi153.mqbgav","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.153:6827/2749127439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2749127439},{"type":"v1","addr":"172.21.15.153:6827","nonce":2749127439}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14478,"qdb_cluster":[14490,14478]},"id":1}]} 2024-04-16T03:17:35.272 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 22 2024-04-16T03:17:35.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f564406ec30 msgr2=0x7f56440710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f564406ec30 0x7f56440710f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5640006010 tx=0x7f5640000bc0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101150 msgr2=0x7f5658195d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101150 0x7f5658195d70 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f564800c8f0 tx=0x7f564800cdb0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 shutdown_connections 2024-04-16T03:17:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f564406ec30 0x7f56440710f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5658101b00 0x7f56581962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 --2- 172.21.15.151:0/514569193 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5658101150 0x7f5658195d70 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 >> 172.21.15.151:0/514569193 conn(0x7f56580faa00 msgr2=0x7f5658104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 shutdown_connections 2024-04-16T03:17:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.273+0000 7f565d421640 1 -- 172.21.15.151:0/514569193 wait complete. 2024-04-16T03:17:35.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-16T03:17:35.391 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 -- 172.21.15.151:0/809175143 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 msgr2=0x7f02b4104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 --2- 172.21.15.151:0/809175143 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 0x7f02b4104a90 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009880 tx=0x7f02a402bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 -- 172.21.15.151:0/809175143 shutdown_connections 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 --2- 172.21.15.151:0/809175143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b40fe720 0x7f02b40feb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 --2- 172.21.15.151:0/809175143 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 0x7f02b4104a90 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 -- 172.21.15.151:0/809175143 >> 172.21.15.151:0/809175143 conn(0x7f02b4075250 msgr2=0x7f02b4075660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 -- 172.21.15.151:0/809175143 shutdown_connections 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 -- 172.21.15.151:0/809175143 wait complete. 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 Processor -- start 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 -- start start 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b40fe720 0x7f02b419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 0x7f02b419b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b4194b50 con 0x7f02b41046b0 2024-04-16T03:17:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.389+0000 7f02bc77e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b4194cc0 con 0x7f02b40fe720 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02b9d19640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 0x7f02b419b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02b9d19640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 0x7f02b419b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57398/0 (socket says 172.21.15.151:57398) 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02b9d19640 1 -- 172.21.15.151:0/1848340200 learned_addr learned my addr 172.21.15.151:0/1848340200 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02ba51a640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b40fe720 0x7f02b419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02b9d19640 1 -- 172.21.15.151:0/1848340200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b40fe720 msgr2=0x7f02b419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02b9d19640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b40fe720 0x7f02b419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02b9d19640 1 -- 172.21.15.151:0/1848340200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a4009560 con 0x7f02b41046b0 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02b9d19640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 0x7f02b419b020 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f02b000b420 tx=0x7f02b000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02ab7fe640 1 -- 172.21.15.151:0/1848340200 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02b0011800 con 0x7f02b41046b0 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02ab7fe640 1 -- 172.21.15.151:0/1848340200 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02b0011e20 con 0x7f02b41046b0 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b4194fa0 con 0x7f02b41046b0 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02ab7fe640 1 -- 172.21.15.151:0/1848340200 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02b000f2e0 con 0x7f02b41046b0 2024-04-16T03:17:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.390+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b4195500 con 0x7f02b41046b0 2024-04-16T03:17:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.391+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b404e040 con 0x7f02b41046b0 2024-04-16T03:17:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.392+0000 7f02ab7fe640 1 -- 172.21.15.151:0/1848340200 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f02b0010380 con 0x7f02b41046b0 2024-04-16T03:17:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.392+0000 7f02ab7fe640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f02a006ec30 0x7f02a00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:35.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.392+0000 7f02ab7fe640 1 -- 172.21.15.151:0/1848340200 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f02b008e7e0 con 0x7f02b41046b0 2024-04-16T03:17:35.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.392+0000 7f02ba51a640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f02a006ec30 0x7f02a00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:35.395 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.393+0000 7f02ba51a640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f02a006ec30 0x7f02a00710f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f02a4006010 tx=0x7f02a4000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.397+0000 7f02ab7fe640 1 -- 172.21.15.151:0/1848340200 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f02b0057590 con 0x7f02b41046b0 2024-04-16T03:17:35.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:35 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1587425888' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-16T03:17:35.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:35 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4029125819' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-16T03:17:35.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:35 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1846549953' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-16T03:17:35.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:35 smithi153 ceph-mon[87858]: pgmap v1172: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:35.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.610+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f02b4195ff0 con 0x7f02b41046b0 2024-04-16T03:17:35.611 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.611+0000 7f02ab7fe640 1 -- 172.21.15.151:0/1848340200 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v42) v1 ==== 107+0+3331 (secure 0 0 0) 0x7f02b0055e60 con 0x7f02b41046b0 2024-04-16T03:17:35.612 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:35.612 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:24.250523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14478,"mds_1":14490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14478":{"gid":14478,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":3,"state":"up:active","state_seq":4,"addr":"172.21.15.151:6827/1662886502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1662886502},{"type":"v1","addr":"172.21.15.151:6827","nonce":1662886502}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.151:6829/37453862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":37453862},{"type":"v1","addr":"172.21.15.151:6829","nonce":37453862}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14478,"qdb_cluster":[14478,14490]},"id":1}]} 2024-04-16T03:17:35.612 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 23 2024-04-16T03:17:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f02a006ec30 msgr2=0x7f02a00710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f02a006ec30 0x7f02a00710f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f02a4006010 tx=0x7f02a4000bc0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 msgr2=0x7f02b419b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 0x7f02b419b020 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f02b000b420 tx=0x7f02b000b8e0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 shutdown_connections 2024-04-16T03:17:35.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f02a006ec30 0x7f02a00710f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02b41046b0 0x7f02b419b020 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 --2- 172.21.15.151:0/1848340200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b40fe720 0x7f02b419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 >> 172.21.15.151:0/1848340200 conn(0x7f02b4075250 msgr2=0x7f02b41021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:35.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 shutdown_connections 2024-04-16T03:17:35.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.613+0000 7f02bc77e640 1 -- 172.21.15.151:0/1848340200 wait complete. 2024-04-16T03:17:35.625 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 23 2024-04-16T03:17:35.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-16T03:17:35.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.730+0000 7fe583a46640 1 -- 172.21.15.151:0/70293432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c0ffb30 msgr2=0x7fe57c103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.730+0000 7fe583a46640 1 --2- 172.21.15.151:0/70293432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c0ffb30 0x7fe57c103b70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe578009880 tx=0x7fe57802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.730+0000 7fe583a46640 1 -- 172.21.15.151:0/70293432 shutdown_connections 2024-04-16T03:17:35.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.730+0000 7fe583a46640 1 --2- 172.21.15.151:0/70293432 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c0ffb30 0x7fe57c103b70 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.730+0000 7fe583a46640 1 --2- 172.21.15.151:0/70293432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe57c0ff180 0x7fe57c0ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.730+0000 7fe583a46640 1 -- 172.21.15.151:0/70293432 >> 172.21.15.151:0/70293432 conn(0x7fe57c0faa00 msgr2=0x7fe57c0fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:35.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.730+0000 7fe583a46640 1 -- 172.21.15.151:0/70293432 shutdown_connections 2024-04-16T03:17:35.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.730+0000 7fe583a46640 1 -- 172.21.15.151:0/70293432 wait complete. 2024-04-16T03:17:35.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe583a46640 1 Processor -- start 2024-04-16T03:17:35.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe583a46640 1 -- start start 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe583a46640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe57c0ff180 0x7fe57c195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe583a46640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c1962c0 0x7fe57c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe583a46640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c1968c0 con 0x7fe57c1962c0 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe583a46640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c196a30 con 0x7fe57c0ff180 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe580fe1640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c1962c0 0x7fe57c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe580fe1640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c1962c0 0x7fe57c19a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57412/0 (socket says 172.21.15.151:57412) 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe580fe1640 1 -- 172.21.15.151:0/1623669769 learned_addr learned my addr 172.21.15.151:0/1623669769 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe5817e2640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe57c0ff180 0x7fe57c195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe580fe1640 1 -- 172.21.15.151:0/1623669769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe57c0ff180 msgr2=0x7fe57c195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe580fe1640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe57c0ff180 0x7fe57c195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe580fe1640 1 -- 172.21.15.151:0/1623669769 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe578009560 con 0x7fe57c1962c0 2024-04-16T03:17:35.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.731+0000 7fe580fe1640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c1962c0 0x7fe57c19a720 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe578036040 tx=0x7fe57802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:35.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.732+0000 7fe56e7fc640 1 -- 172.21.15.151:0/1623669769 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5780035f0 con 0x7fe57c1962c0 2024-04-16T03:17:35.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.732+0000 7fe56e7fc640 1 -- 172.21.15.151:0/1623669769 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe578003c10 con 0x7fe57c1962c0 2024-04-16T03:17:35.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.732+0000 7fe56e7fc640 1 -- 172.21.15.151:0/1623669769 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe578034440 con 0x7fe57c1962c0 2024-04-16T03:17:35.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.732+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c19acc0 con 0x7fe57c1962c0 2024-04-16T03:17:35.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.732+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c19b220 con 0x7fe57c1962c0 2024-04-16T03:17:35.735 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.733+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57c04e040 con 0x7fe57c1962c0 2024-04-16T03:17:35.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.733+0000 7fe56e7fc640 1 -- 172.21.15.151:0/1623669769 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fe578003750 con 0x7fe57c1962c0 2024-04-16T03:17:35.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.734+0000 7fe56e7fc640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56806ec30 0x7fe5680710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:35.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.734+0000 7fe56e7fc640 1 -- 172.21.15.151:0/1623669769 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe5780b1220 con 0x7fe57c1962c0 2024-04-16T03:17:35.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.734+0000 7fe5817e2640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56806ec30 0x7fe5680710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:35.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.734+0000 7fe5817e2640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56806ec30 0x7fe5680710f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe570005fd0 tx=0x7fe570005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:35.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.739+0000 7fe56e7fc640 1 -- 172.21.15.151:0/1623669769 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe578079fd0 con 0x7fe57c1962c0 2024-04-16T03:17:35.953 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.952+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7fe57c107480 con 0x7fe57c1962c0 2024-04-16T03:17:35.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.953+0000 7fe56e7fc640 1 -- 172.21.15.151:0/1623669769 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v42) v1 ==== 107+0+3298 (secure 0 0 0) 0x7fe578041d00 con 0x7fe57c1962c0 2024-04-16T03:17:35.954 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:35.954 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44245,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1306847002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1306847002},{"type":"v1","addr":"172.21.15.153:6825","nonce":1306847002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44247,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/341639119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":341639119},{"type":"v1","addr":"172.21.15.153:6827","nonce":341639119}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:35.954 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 24 2024-04-16T03:17:35.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56806ec30 msgr2=0x7fe5680710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56806ec30 0x7fe5680710f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe570005fd0 tx=0x7fe570005850 comp rx=0 tx=0).stop 2024-04-16T03:17:35.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c1962c0 msgr2=0x7fe57c19a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:35.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c1962c0 0x7fe57c19a720 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe578036040 tx=0x7fe57802c850 comp rx=0 tx=0).stop 2024-04-16T03:17:35.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 shutdown_connections 2024-04-16T03:17:35.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fe56806ec30 0x7fe5680710f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe57c1962c0 0x7fe57c19a720 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 --2- 172.21.15.151:0/1623669769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe57c0ff180 0x7fe57c195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:35.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 >> 172.21.15.151:0/1623669769 conn(0x7fe57c0faa00 msgr2=0x7fe57c0fc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:35.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 shutdown_connections 2024-04-16T03:17:35.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:35.955+0000 7fe583a46640 1 -- 172.21.15.151:0/1623669769 wait complete. 2024-04-16T03:17:35.967 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:35.967 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 24 2024-04-16T03:17:35.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-16T03:17:36.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.074+0000 7f1e195c0640 1 -- 172.21.15.151:0/2993578956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 msgr2=0x7f1e140feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.074+0000 7f1e195c0640 1 --2- 172.21.15.151:0/2993578956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 0x7f1e140feb70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009c20 tx=0x7f1e0802c1b0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.075 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.074+0000 7f1e195c0640 1 -- 172.21.15.151:0/2993578956 shutdown_connections 2024-04-16T03:17:36.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.074+0000 7f1e195c0640 1 --2- 172.21.15.151:0/2993578956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 0x7f1e140feb70 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.074+0000 7f1e195c0640 1 --2- 172.21.15.151:0/2993578956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e14106070 0x7f1e14106450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.074+0000 7f1e195c0640 1 -- 172.21.15.151:0/2993578956 >> 172.21.15.151:0/2993578956 conn(0x7f1e14074250 msgr2=0x7f1e14074660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:36.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.074+0000 7f1e195c0640 1 -- 172.21.15.151:0/2993578956 shutdown_connections 2024-04-16T03:17:36.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.074+0000 7f1e195c0640 1 -- 172.21.15.151:0/2993578956 wait complete. 2024-04-16T03:17:36.076 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e195c0640 1 Processor -- start 2024-04-16T03:17:36.077 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e195c0640 1 -- start start 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e195c0640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 0x7f1e1419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e195c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e14106070 0x7f1e1419c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e195c0640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e14195b30 con 0x7f1e140fe730 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e195c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e14195c70 con 0x7f1e14106070 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e12ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 0x7f1e1419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e12ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 0x7f1e1419bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57430/0 (socket says 172.21.15.151:57430) 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e12ffd640 1 -- 172.21.15.151:0/543297597 learned_addr learned my addr 172.21.15.151:0/543297597 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e127fc640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e14106070 0x7f1e1419c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e12ffd640 1 -- 172.21.15.151:0/543297597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e14106070 msgr2=0x7f1e1419c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e12ffd640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e14106070 0x7f1e1419c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.075+0000 7f1e12ffd640 1 -- 172.21.15.151:0/543297597 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e08009900 con 0x7f1e140fe730 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.076+0000 7f1e12ffd640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 0x7f1e1419bb50 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00d8d0 tx=0x7f1e0c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.076+0000 7f1dfbfff640 1 -- 172.21.15.151:0/543297597 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e0c00bea0 con 0x7f1e140fe730 2024-04-16T03:17:36.078 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.076+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e14195f50 con 0x7f1e140fe730 2024-04-16T03:17:36.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.076+0000 7f1dfbfff640 1 -- 172.21.15.151:0/543297597 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e0c009c00 con 0x7f1e140fe730 2024-04-16T03:17:36.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.076+0000 7f1dfbfff640 1 -- 172.21.15.151:0/543297597 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e0c00f710 con 0x7f1e140fe730 2024-04-16T03:17:36.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.076+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e141964b0 con 0x7f1e140fe730 2024-04-16T03:17:36.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.077+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1404e040 con 0x7f1e140fe730 2024-04-16T03:17:36.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.077+0000 7f1dfbfff640 1 -- 172.21.15.151:0/543297597 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1e0c00f990 con 0x7f1e140fe730 2024-04-16T03:17:36.079 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.078+0000 7f1dfbfff640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1dfc06ec30 0x7f1dfc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:36.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.078+0000 7f1dfbfff640 1 -- 172.21.15.151:0/543297597 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1e0c08eb00 con 0x7f1e140fe730 2024-04-16T03:17:36.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.078+0000 7f1e127fc640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1dfc06ec30 0x7f1dfc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:36.080 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.078+0000 7f1e127fc640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1dfc06ec30 0x7f1dfc0710f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1e141973f0 tx=0x7f1e08009840 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:36.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.082+0000 7f1dfbfff640 1 -- 172.21.15.151:0/543297597 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1e0c057960 con 0x7f1e140fe730 2024-04-16T03:17:36.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:36 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/514569193' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-16T03:17:36.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:36 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1848340200' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-16T03:17:36.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:36 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1623669769' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-16T03:17:36.301 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.300+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f1e1410d2d0 con 0x7f1e140fe730 2024-04-16T03:17:36.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.301+0000 7f1dfbfff640 1 -- 172.21.15.151:0/543297597 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v42) v1 ==== 107+0+4859 (secure 0 0 0) 0x7f1e0c05b020 con 0x7f1e140fe730 2024-04-16T03:17:36.302 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:36.302 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34382,"name":"cephfs.smithi151.bdpnxq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6828/4272338874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":4272338874},{"type":"v1","addr":"172.21.15.151:6828","nonce":4272338874}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44245,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1306847002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1306847002},{"type":"v1","addr":"172.21.15.153:6825","nonce":1306847002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44247,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/341639119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":341639119},{"type":"v1","addr":"172.21.15.153:6827","nonce":341639119}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44255,"name":"cephfs.smithi151.slrqog","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6829/271070112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":271070112},{"type":"v1","addr":"172.21.15.151:6829","nonce":271070112}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:36.303 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 25 2024-04-16T03:17:36.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1dfc06ec30 msgr2=0x7f1dfc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1dfc06ec30 0x7f1dfc0710f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1e141973f0 tx=0x7f1e08009840 comp rx=0 tx=0).stop 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 msgr2=0x7f1e1419bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 0x7f1e1419bb50 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00d8d0 tx=0x7f1e0c00dd90 comp rx=0 tx=0).stop 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 shutdown_connections 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f1dfc06ec30 0x7f1dfc0710f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e14106070 0x7f1e1419c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 --2- 172.21.15.151:0/543297597 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e140fe730 0x7f1e1419bb50 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 >> 172.21.15.151:0/543297597 conn(0x7f1e14074250 msgr2=0x7f1e140fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 shutdown_connections 2024-04-16T03:17:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.303+0000 7f1e195c0640 1 -- 172.21.15.151:0/543297597 wait complete. 2024-04-16T03:17:36.315 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:36.315 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-04-16T03:17:36.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-16T03:17:36.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 -- 172.21.15.151:0/1589364522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 msgr2=0x7efd640ff590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 --2- 172.21.15.151:0/1589364522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 0x7efd640ff590 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7efd54009880 tx=0x7efd5402bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 -- 172.21.15.151:0/1589364522 shutdown_connections 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 --2- 172.21.15.151:0/1589364522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd640ffb60 0x7efd64103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 --2- 172.21.15.151:0/1589364522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 0x7efd640ff590 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 -- 172.21.15.151:0/1589364522 >> 172.21.15.151:0/1589364522 conn(0x7efd640faa30 msgr2=0x7efd640fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 -- 172.21.15.151:0/1589364522 shutdown_connections 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 -- 172.21.15.151:0/1589364522 wait complete. 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 Processor -- start 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.421+0000 7efd6a051640 1 -- start start 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd6a051640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 0x7efd64196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd6a051640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd64197460 0x7efd641978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd6a051640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6419ff90 con 0x7efd640ff1b0 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd6a051640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd641a00d0 con 0x7efd64197460 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd637fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 0x7efd64196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd637fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 0x7efd64196f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57452/0 (socket says 172.21.15.151:57452) 2024-04-16T03:17:36.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd637fe640 1 -- 172.21.15.151:0/3048366852 learned_addr learned my addr 172.21.15.151:0/3048366852 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd62ffd640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd64197460 0x7efd641978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd637fe640 1 -- 172.21.15.151:0/3048366852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd64197460 msgr2=0x7efd641978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd637fe640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd64197460 0x7efd641978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd637fe640 1 -- 172.21.15.151:0/3048366852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd54009560 con 0x7efd640ff1b0 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd62ffd640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd64197460 0x7efd641978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd637fe640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 0x7efd64196f20 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7efd54009430 tx=0x7efd54004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd60ff9640 1 -- 172.21.15.151:0/3048366852 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd540035f0 con 0x7efd640ff1b0 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd60ff9640 1 -- 172.21.15.151:0/3048366852 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd54003c10 con 0x7efd640ff1b0 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd641a02d0 con 0x7efd640ff1b0 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd60ff9640 1 -- 172.21.15.151:0/3048366852 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd54034440 con 0x7efd640ff1b0 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.422+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd641a0780 con 0x7efd640ff1b0 2024-04-16T03:17:36.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.423+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6404e040 con 0x7efd640ff1b0 2024-04-16T03:17:36.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.424+0000 7efd60ff9640 1 -- 172.21.15.151:0/3048366852 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7efd54004fd0 con 0x7efd640ff1b0 2024-04-16T03:17:36.427 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.424+0000 7efd60ff9640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd4c06ec30 0x7efd4c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:36.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.424+0000 7efd60ff9640 1 -- 172.21.15.151:0/3048366852 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efd540b12c0 con 0x7efd640ff1b0 2024-04-16T03:17:36.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.424+0000 7efd62ffd640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd4c06ec30 0x7efd4c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:36.428 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.425+0000 7efd62ffd640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd4c06ec30 0x7efd4c0710f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7efd580058c0 tx=0x7efd58005d30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:36.430 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.429+0000 7efd60ff9640 1 -- 172.21.15.151:0/3048366852 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efd5407a070 con 0x7efd640ff1b0 2024-04-16T03:17:36.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:36 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/514569193' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-16T03:17:36.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:36 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1848340200' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-16T03:17:36.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:36 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1623669769' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-16T03:17:36.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.647+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7efd641074b0 con 0x7efd640ff1b0 2024-04-16T03:17:36.648 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.647+0000 7efd60ff9640 1 -- 172.21.15.151:0/3048366852 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v42) v1 ==== 107+0+4077 (secure 0 0 0) 0x7efd5407d730 con 0x7efd640ff1b0 2024-04-16T03:17:36.648 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:36.649 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44245,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1306847002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1306847002},{"type":"v1","addr":"172.21.15.153:6825","nonce":1306847002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44247,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/341639119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":341639119},{"type":"v1","addr":"172.21.15.153:6827","nonce":341639119}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44255,"name":"cephfs.smithi151.slrqog","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6829/271070112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":271070112},{"type":"v1","addr":"172.21.15.151:6829","nonce":271070112}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:36.649 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 26 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.649+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd4c06ec30 msgr2=0x7efd4c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.649+0000 7efd6a051640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd4c06ec30 0x7efd4c0710f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7efd580058c0 tx=0x7efd58005d30 comp rx=0 tx=0).stop 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.649+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 msgr2=0x7efd64196f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.649+0000 7efd6a051640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 0x7efd64196f20 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7efd54009430 tx=0x7efd54004040 comp rx=0 tx=0).stop 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.649+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 shutdown_connections 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.649+0000 7efd6a051640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7efd4c06ec30 0x7efd4c0710f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.649+0000 7efd6a051640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd64197460 0x7efd641978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.649+0000 7efd6a051640 1 --2- 172.21.15.151:0/3048366852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd640ff1b0 0x7efd64196f20 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.649+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 >> 172.21.15.151:0/3048366852 conn(0x7efd640faa30 msgr2=0x7efd640fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.650+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 shutdown_connections 2024-04-16T03:17:36.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.650+0000 7efd6a051640 1 -- 172.21.15.151:0/3048366852 wait complete. 2024-04-16T03:17:36.662 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:36.662 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-04-16T03:17:36.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.766+0000 7f1273edf640 1 -- 172.21.15.151:0/1222361302 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c0ff260 msgr2=0x7f126c0ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.766+0000 7f1273edf640 1 --2- 172.21.15.151:0/1222361302 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c0ff260 0x7f126c0ff640 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f125c009880 tx=0x7f125c02bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.766+0000 7f1273edf640 1 -- 172.21.15.151:0/1222361302 shutdown_connections 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.766+0000 7f1273edf640 1 --2- 172.21.15.151:0/1222361302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c0ffc10 0x7f126c103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.766+0000 7f1273edf640 1 --2- 172.21.15.151:0/1222361302 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c0ff260 0x7f126c0ff640 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.766+0000 7f1273edf640 1 -- 172.21.15.151:0/1222361302 >> 172.21.15.151:0/1222361302 conn(0x7f126c074a50 msgr2=0x7f126c074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.766+0000 7f1273edf640 1 -- 172.21.15.151:0/1222361302 shutdown_connections 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.766+0000 7f1273edf640 1 -- 172.21.15.151:0/1222361302 wait complete. 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f1273edf640 1 Processor -- start 2024-04-16T03:17:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f1273edf640 1 -- start start 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f1273edf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c0ff260 0x7f126c194e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f1273edf640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c195360 0x7f126c1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f1273edf640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c195960 con 0x7f126c195360 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f1273edf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c195ad0 con 0x7f126c0ff260 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f127147a640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c195360 0x7f126c1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f127147a640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c195360 0x7f126c1997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59020/0 (socket says 172.21.15.151:59020) 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f127147a640 1 -- 172.21.15.151:0/1890638358 learned_addr learned my addr 172.21.15.151:0/1890638358 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f127147a640 1 -- 172.21.15.151:0/1890638358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c0ff260 msgr2=0x7f126c194e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f127147a640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c0ff260 0x7f126c194e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f127147a640 1 -- 172.21.15.151:0/1890638358 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c009560 con 0x7f126c195360 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.767+0000 7f127147a640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c195360 0x7f126c1997c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f126800b400 tx=0x7f126800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.768+0000 7f1262ffd640 1 -- 172.21.15.151:0/1890638358 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1268011800 con 0x7f126c195360 2024-04-16T03:17:36.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.768+0000 7f1262ffd640 1 -- 172.21.15.151:0/1890638358 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1268011e20 con 0x7f126c195360 2024-04-16T03:17:36.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.768+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126c199dc0 con 0x7f126c195360 2024-04-16T03:17:36.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.768+0000 7f1262ffd640 1 -- 172.21.15.151:0/1890638358 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f126800f2e0 con 0x7f126c195360 2024-04-16T03:17:36.771 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.768+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126c19a320 con 0x7f126c195360 2024-04-16T03:17:36.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.769+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c04e040 con 0x7f126c195360 2024-04-16T03:17:36.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.769+0000 7f1262ffd640 1 -- 172.21.15.151:0/1890638358 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f1268010380 con 0x7f126c195360 2024-04-16T03:17:36.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.769+0000 7f1262ffd640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f125806ec30 0x7f12580710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:36.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.770+0000 7f1262ffd640 1 -- 172.21.15.151:0/1890638358 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f126808eb50 con 0x7f126c195360 2024-04-16T03:17:36.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.770+0000 7f1271c7b640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f125806ec30 0x7f12580710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:36.772 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.770+0000 7f1271c7b640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f125806ec30 0x7f12580710f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f125c036040 tx=0x7f125c000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:36.775 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.774+0000 7f1262ffd640 1 -- 172.21.15.151:0/1890638358 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1268057900 con 0x7f126c195360 2024-04-16T03:17:36.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.989+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f126c196170 con 0x7f126c195360 2024-04-16T03:17:36.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.990+0000 7f1262ffd640 1 -- 172.21.15.151:0/1890638358 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v42) v1 ==== 107+0+4927 (secure 0 0 0) 0x7f126805afc0 con 0x7f126c195360 2024-04-16T03:17:36.991 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:36.992 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44245,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1306847002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1306847002},{"type":"v1","addr":"172.21.15.153:6825","nonce":1306847002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44247,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/341639119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":341639119},{"type":"v1","addr":"172.21.15.153:6827","nonce":341639119}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44255,"name":"cephfs.smithi151.slrqog","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6829/271070112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":271070112},{"type":"v1","addr":"172.21.15.151:6829","nonce":271070112}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:36.992 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 27 2024-04-16T03:17:36.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f125806ec30 msgr2=0x7f12580710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f125806ec30 0x7f12580710f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f125c036040 tx=0x7f125c000bc0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c195360 msgr2=0x7f126c1997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:36.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c195360 0x7f126c1997c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f126800b400 tx=0x7f126800b8c0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 shutdown_connections 2024-04-16T03:17:36.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f125806ec30 0x7f12580710f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f126c195360 0x7f126c1997c0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 --2- 172.21.15.151:0/1890638358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c0ff260 0x7f126c194e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:36.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 >> 172.21.15.151:0/1890638358 conn(0x7f126c074a50 msgr2=0x7f126c0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:36.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 shutdown_connections 2024-04-16T03:17:36.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:36.992+0000 7f1273edf640 1 -- 172.21.15.151:0/1890638358 wait complete. 2024-04-16T03:17:37.005 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:37.005 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-04-16T03:17:37.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-16T03:17:37.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.109+0000 7f8b7fed2640 1 -- 172.21.15.151:0/378654695 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 msgr2=0x7f8b78100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.109+0000 7f8b7fed2640 1 --2- 172.21.15.151:0/378654695 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 0x7f8b78100680 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009880 tx=0x7f8b7002bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.109+0000 7f8b7fed2640 1 -- 172.21.15.151:0/378654695 shutdown_connections 2024-04-16T03:17:37.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.109+0000 7f8b7fed2640 1 --2- 172.21.15.151:0/378654695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b78100c50 0x7f8b78104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.109+0000 7f8b7fed2640 1 --2- 172.21.15.151:0/378654695 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 0x7f8b78100680 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.109+0000 7f8b7fed2640 1 -- 172.21.15.151:0/378654695 >> 172.21.15.151:0/378654695 conn(0x7f8b780fbb20 msgr2=0x7f8b780fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.109+0000 7f8b7fed2640 1 -- 172.21.15.151:0/378654695 shutdown_connections 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.109+0000 7f8b7fed2640 1 -- 172.21.15.151:0/378654695 wait complete. 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7fed2640 1 Processor -- start 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7fed2640 1 -- start start 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7fed2640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 0x7f8b78195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7fed2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b781963c0 0x7f8b7819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7fed2640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b781969c0 con 0x7f8b781002a0 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7fed2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b78196b30 con 0x7f8b781963c0 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7dc6e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 0x7f8b78195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7dc6e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 0x7f8b78195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59046/0 (socket says 172.21.15.151:59046) 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7dc6e640 1 -- 172.21.15.151:0/129990869 learned_addr learned my addr 172.21.15.151:0/129990869 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7d46d640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b781963c0 0x7f8b7819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7dc6e640 1 -- 172.21.15.151:0/129990869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b781963c0 msgr2=0x7f8b7819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7dc6e640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b781963c0 0x7f8b7819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7dc6e640 1 -- 172.21.15.151:0/129990869 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b70009560 con 0x7f8b781002a0 2024-04-16T03:17:37.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7d46d640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b781963c0 0x7f8b7819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T03:17:37.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.110+0000 7f8b7dc6e640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 0x7f8b78195e80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009430 tx=0x7f8b70004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:37.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.111+0000 7f8b76ffd640 1 -- 172.21.15.151:0/129990869 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b700035f0 con 0x7f8b781002a0 2024-04-16T03:17:37.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.111+0000 7f8b76ffd640 1 -- 172.21.15.151:0/129990869 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b70003c10 con 0x7f8b781002a0 2024-04-16T03:17:37.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.111+0000 7f8b76ffd640 1 -- 172.21.15.151:0/129990869 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b70034440 con 0x7f8b781002a0 2024-04-16T03:17:37.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.111+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7819ad50 con 0x7f8b781002a0 2024-04-16T03:17:37.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.111+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7819b260 con 0x7f8b781002a0 2024-04-16T03:17:37.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.112+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7804e040 con 0x7f8b781002a0 2024-04-16T03:17:37.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.112+0000 7f8b76ffd640 1 -- 172.21.15.151:0/129990869 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8b70004fd0 con 0x7f8b781002a0 2024-04-16T03:17:37.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.113+0000 7f8b76ffd640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8b6006ec30 0x7f8b600710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:37.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.113+0000 7f8b76ffd640 1 -- 172.21.15.151:0/129990869 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8b700b12c0 con 0x7f8b781002a0 2024-04-16T03:17:37.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.113+0000 7f8b7d46d640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8b6006ec30 0x7f8b600710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:37.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.113+0000 7f8b7d46d640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8b6006ec30 0x7f8b600710f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8b780696e0 tx=0x7f8b6c0073c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:37.118 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.117+0000 7f8b76ffd640 1 -- 172.21.15.151:0/129990869 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8b7007a070 con 0x7f8b781002a0 2024-04-16T03:17:37.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.335+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f8b7819b4d0 con 0x7f8b781002a0 2024-04-16T03:17:37.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.335+0000 7f8b76ffd640 1 -- 172.21.15.151:0/129990869 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v42) v1 ==== 107+0+4148 (secure 0 0 0) 0x7f8b7007d730 con 0x7f8b781002a0 2024-04-16T03:17:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:37.337 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44245,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1306847002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1306847002},{"type":"v1","addr":"172.21.15.153:6825","nonce":1306847002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44247,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/341639119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":341639119},{"type":"v1","addr":"172.21.15.153:6827","nonce":341639119}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:37.337 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 28 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8b6006ec30 msgr2=0x7f8b600710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8b6006ec30 0x7f8b600710f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8b780696e0 tx=0x7f8b6c0073c0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 msgr2=0x7f8b78195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 0x7f8b78195e80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009430 tx=0x7f8b70004040 comp rx=0 tx=0).stop 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 shutdown_connections 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8b6006ec30 0x7f8b600710f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b781963c0 0x7f8b7819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 --2- 172.21.15.151:0/129990869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b781002a0 0x7f8b78195e80 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 >> 172.21.15.151:0/129990869 conn(0x7f8b780fbb20 msgr2=0x7f8b780fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 shutdown_connections 2024-04-16T03:17:37.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.337+0000 7f8b7fed2640 1 -- 172.21.15.151:0/129990869 wait complete. 2024-04-16T03:17:37.348 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:37 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/543297597' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-16T03:17:37.349 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:37 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3048366852' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-16T03:17:37.349 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:37 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1890638358' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-16T03:17:37.349 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:37 smithi151 ceph-mon[112233]: pgmap v1173: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:37.350 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:37.351 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-04-16T03:17:37.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-16T03:17:37.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.458+0000 7f6427371640 1 -- 172.21.15.151:0/3213209527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ffc10 msgr2=0x7f6420103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.458+0000 7f6427371640 1 --2- 172.21.15.151:0/3213209527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ffc10 0x7f6420103b70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f6410009880 tx=0x7f641002bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 -- 172.21.15.151:0/3213209527 shutdown_connections 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 --2- 172.21.15.151:0/3213209527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ffc10 0x7f6420103b70 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 --2- 172.21.15.151:0/3213209527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64200ff260 0x7f64200ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 -- 172.21.15.151:0/3213209527 >> 172.21.15.151:0/3213209527 conn(0x7f6420074a50 msgr2=0x7f6420074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 -- 172.21.15.151:0/3213209527 shutdown_connections 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 -- 172.21.15.151:0/3213209527 wait complete. 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 Processor -- start 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 -- start start 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ff260 0x7f6420195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6420196310 0x7f642019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:37.462 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.459+0000 7f6427371640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420196910 con 0x7f64200ff260 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f6427371640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420196a80 con 0x7f6420196310 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f642510d640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ff260 0x7f6420195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f642510d640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ff260 0x7f6420195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59068/0 (socket says 172.21.15.151:59068) 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f642510d640 1 -- 172.21.15.151:0/1421997856 learned_addr learned my addr 172.21.15.151:0/1421997856 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f642510d640 1 -- 172.21.15.151:0/1421997856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6420196310 msgr2=0x7f642019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f642510d640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6420196310 0x7f642019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f642510d640 1 -- 172.21.15.151:0/1421997856 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410009560 con 0x7f64200ff260 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f642510d640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ff260 0x7f6420195dd0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f641c00c920 tx=0x7f641c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f64167fc640 1 -- 172.21.15.151:0/1421997856 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641c007b80 con 0x7f64200ff260 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f64167fc640 1 -- 172.21.15.151:0/1421997856 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f641c0054e0 con 0x7f64200ff260 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642019ad70 con 0x7f64200ff260 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f64167fc640 1 -- 172.21.15.151:0/1421997856 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641c00f450 con 0x7f64200ff260 2024-04-16T03:17:37.463 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.460+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642019b2d0 con 0x7f64200ff260 2024-04-16T03:17:37.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.461+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642019b5b0 con 0x7f64200ff260 2024-04-16T03:17:37.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.462+0000 7f64167fc640 1 -- 172.21.15.151:0/1421997856 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f641c004f60 con 0x7f64200ff260 2024-04-16T03:17:37.464 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.462+0000 7f64167fc640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f640c06ec30 0x7f640c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:37.465 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.462+0000 7f64167fc640 1 -- 172.21.15.151:0/1421997856 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f641c08ed10 con 0x7f64200ff260 2024-04-16T03:17:37.465 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.462+0000 7f642490c640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f640c06ec30 0x7f640c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:37.465 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.463+0000 7f642490c640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f640c06ec30 0x7f640c0710f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6410036040 tx=0x7f6410000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:37.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.467+0000 7f64167fc640 1 -- 172.21.15.151:0/1421997856 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f641c057ac0 con 0x7f64200ff260 2024-04-16T03:17:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:37 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/543297597' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-16T03:17:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:37 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3048366852' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-16T03:17:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:37 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1890638358' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-16T03:17:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:37 smithi153 ceph-mon[87858]: pgmap v1173: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:37.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.683+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f64200ffc10 con 0x7f64200ff260 2024-04-16T03:17:37.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.684+0000 7f64167fc640 1 -- 172.21.15.151:0/1421997856 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v42) v1 ==== 107+0+4998 (secure 0 0 0) 0x7f641c05b180 con 0x7f64200ff260 2024-04-16T03:17:37.685 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:37.685 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44245,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1306847002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1306847002},{"type":"v1","addr":"172.21.15.153:6825","nonce":1306847002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44247,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/341639119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":341639119},{"type":"v1","addr":"172.21.15.153:6827","nonce":341639119}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:37.685 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 29 2024-04-16T03:17:37.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f640c06ec30 msgr2=0x7f640c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f640c06ec30 0x7f640c0710f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6410036040 tx=0x7f6410000bc0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ff260 msgr2=0x7f6420195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ff260 0x7f6420195dd0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f641c00c920 tx=0x7f641c00cde0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 shutdown_connections 2024-04-16T03:17:37.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f640c06ec30 0x7f640c0710f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6420196310 0x7f642019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 --2- 172.21.15.151:0/1421997856 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64200ff260 0x7f6420195dd0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 >> 172.21.15.151:0/1421997856 conn(0x7f6420074a50 msgr2=0x7f6420103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:37.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 shutdown_connections 2024-04-16T03:17:37.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.686+0000 7f6427371640 1 -- 172.21.15.151:0/1421997856 wait complete. 2024-04-16T03:17:37.698 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:37.698 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-04-16T03:17:37.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-16T03:17:37.803 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.802+0000 7fb633bf3640 1 -- 172.21.15.151:0/1286007430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 msgr2=0x7fb62c0ffd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.802+0000 7fb633bf3640 1 --2- 172.21.15.151:0/1286007430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 0x7fb62c0ffd00 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb620009880 tx=0x7fb62002bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.802+0000 7fb633bf3640 1 -- 172.21.15.151:0/1286007430 shutdown_connections 2024-04-16T03:17:37.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.802+0000 7fb633bf3640 1 --2- 172.21.15.151:0/1286007430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 0x7fb62c0ffd00 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.802+0000 7fb633bf3640 1 --2- 172.21.15.151:0/1286007430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c103f90 0x7fb62c104370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.802+0000 7fb633bf3640 1 -- 172.21.15.151:0/1286007430 >> 172.21.15.151:0/1286007430 conn(0x7fb62c0741f0 msgr2=0x7fb62c074600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:37.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb633bf3640 1 -- 172.21.15.151:0/1286007430 shutdown_connections 2024-04-16T03:17:37.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb633bf3640 1 -- 172.21.15.151:0/1286007430 wait complete. 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb633bf3640 1 Processor -- start 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb633bf3640 1 -- start start 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb633bf3640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 0x7fb62c19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb633bf3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c103f90 0x7fb62c19b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb633bf3640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c194b30 con 0x7fb62c0ff880 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb633bf3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c194ca0 con 0x7fb62c103f90 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb63198f640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 0x7fb62c19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb63198f640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 0x7fb62c19aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59086/0 (socket says 172.21.15.151:59086) 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.803+0000 7fb63198f640 1 -- 172.21.15.151:0/3580902043 learned_addr learned my addr 172.21.15.151:0/3580902043 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb63198f640 1 -- 172.21.15.151:0/3580902043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c103f90 msgr2=0x7fb62c19b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:37.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb63118e640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c103f90 0x7fb62c19b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb63198f640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c103f90 0x7fb62c19b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb63198f640 1 -- 172.21.15.151:0/3580902043 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb620009560 con 0x7fb62c0ff880 2024-04-16T03:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb63198f640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 0x7fb62c19aac0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb62400d8d0 tx=0x7fb62400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb62affd640 1 -- 172.21.15.151:0/3580902043 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb624009c00 con 0x7fb62c0ff880 2024-04-16T03:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb62affd640 1 -- 172.21.15.151:0/3580902043 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb624009d60 con 0x7fb62c0ff880 2024-04-16T03:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62c194f80 con 0x7fb62c0ff880 2024-04-16T03:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb62affd640 1 -- 172.21.15.151:0/3580902043 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6240049d0 con 0x7fb62c0ff880 2024-04-16T03:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.804+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62c1954e0 con 0x7fb62c0ff880 2024-04-16T03:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.805+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62c04e040 con 0x7fb62c0ff880 2024-04-16T03:17:37.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.805+0000 7fb62affd640 1 -- 172.21.15.151:0/3580902043 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fb624004c90 con 0x7fb62c0ff880 2024-04-16T03:17:37.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.806+0000 7fb62affd640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb61406ec30 0x7fb6140710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:37.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.806+0000 7fb63118e640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb61406ec30 0x7fb6140710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:37.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.806+0000 7fb62affd640 1 -- 172.21.15.151:0/3580902043 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb62408eb90 con 0x7fb62c0ff880 2024-04-16T03:17:37.808 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.807+0000 7fb63118e640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb61406ec30 0x7fb6140710f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb620036040 tx=0x7fb620005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:37.812 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:37.811+0000 7fb62affd640 1 -- 172.21.15.151:0/3580902043 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb624057940 con 0x7fb62c0ff880 2024-04-16T03:17:38.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.026+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7fb62c196130 con 0x7fb62c0ff880 2024-04-16T03:17:38.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.027+0000 7fb62affd640 1 -- 172.21.15.151:0/3580902043 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v42) v1 ==== 107+0+4216 (secure 0 0 0) 0x7fb62405b000 con 0x7fb62c0ff880 2024-04-16T03:17:38.028 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:38.028 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44247,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/341639119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":341639119},{"type":"v1","addr":"172.21.15.153:6827","nonce":341639119}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:38.028 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 30 2024-04-16T03:17:38.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb61406ec30 msgr2=0x7fb6140710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb61406ec30 0x7fb6140710f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb620036040 tx=0x7fb620005fd0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 msgr2=0x7fb62c19aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 0x7fb62c19aac0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb62400d8d0 tx=0x7fb62400dd90 comp rx=0 tx=0).stop 2024-04-16T03:17:38.030 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 shutdown_connections 2024-04-16T03:17:38.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fb61406ec30 0x7fb6140710f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c103f90 0x7fb62c19b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 --2- 172.21.15.151:0/3580902043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb62c0ff880 0x7fb62c19aac0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 >> 172.21.15.151:0/3580902043 conn(0x7fb62c0741f0 msgr2=0x7fb62c0fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:38.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 shutdown_connections 2024-04-16T03:17:38.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.029+0000 7fb633bf3640 1 -- 172.21.15.151:0/3580902043 wait complete. 2024-04-16T03:17:38.041 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:38.041 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-04-16T03:17:38.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-16T03:17:38.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.145+0000 7f33b7b67640 1 -- 172.21.15.151:0/2183102371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 msgr2=0x7f33b0100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.145+0000 7f33b7b67640 1 --2- 172.21.15.151:0/2183102371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 0x7f33b0100680 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009880 tx=0x7f33a402bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.145+0000 7f33b7b67640 1 -- 172.21.15.151:0/2183102371 shutdown_connections 2024-04-16T03:17:38.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.145+0000 7f33b7b67640 1 --2- 172.21.15.151:0/2183102371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b0100c50 0x7f33b0104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.151 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.145+0000 7f33b7b67640 1 --2- 172.21.15.151:0/2183102371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 0x7f33b0100680 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.145+0000 7f33b7b67640 1 -- 172.21.15.151:0/2183102371 >> 172.21.15.151:0/2183102371 conn(0x7f33b00fbb20 msgr2=0x7f33b00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.145+0000 7f33b7b67640 1 -- 172.21.15.151:0/2183102371 shutdown_connections 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.145+0000 7f33b7b67640 1 -- 172.21.15.151:0/2183102371 wait complete. 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b7b67640 1 Processor -- start 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b7b67640 1 -- start start 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b7b67640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 0x7f33b0195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b7b67640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b0196340 0x7f33b019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b7b67640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b0196940 con 0x7f33b01002a0 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b7b67640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b0196ab0 con 0x7f33b0196340 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b5903640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 0x7f33b0195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b5903640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 0x7f33b0195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59098/0 (socket says 172.21.15.151:59098) 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b5903640 1 -- 172.21.15.151:0/4059955353 learned_addr learned my addr 172.21.15.151:0/4059955353 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b5102640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b0196340 0x7f33b019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b5903640 1 -- 172.21.15.151:0/4059955353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b0196340 msgr2=0x7f33b019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b5903640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b0196340 0x7f33b019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.152 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b5903640 1 -- 172.21.15.151:0/4059955353 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a4009560 con 0x7f33b01002a0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.146+0000 7f33b5903640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 0x7f33b0195e00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009430 tx=0x7f33a4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.147+0000 7f33a2ffd640 1 -- 172.21.15.151:0/4059955353 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33a40035f0 con 0x7f33b01002a0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.147+0000 7f33a2ffd640 1 -- 172.21.15.151:0/4059955353 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33a4003c10 con 0x7f33b01002a0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.147+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b019ad40 con 0x7f33b01002a0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.147+0000 7f33a2ffd640 1 -- 172.21.15.151:0/4059955353 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33a4005670 con 0x7f33b01002a0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.147+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b019b2a0 con 0x7f33b01002a0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.148+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b004e040 con 0x7f33b01002a0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.148+0000 7f33a2ffd640 1 -- 172.21.15.151:0/4059955353 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f33a4004dd0 con 0x7f33b01002a0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.149+0000 7f33a2ffd640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f339c06ec30 0x7f339c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.149+0000 7f33a2ffd640 1 -- 172.21.15.151:0/4059955353 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f33a40b1180 con 0x7f33b01002a0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.149+0000 7f33b5102640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f339c06ec30 0x7f339c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.149+0000 7f33b5102640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f339c06ec30 0x7f339c0710f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f33ac005fd0 tx=0x7f33ac005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:38.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.153+0000 7f33a2ffd640 1 -- 172.21.15.151:0/4059955353 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f33a4079fb0 con 0x7f33b01002a0 2024-04-16T03:17:38.374 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.373+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f33b019b5b0 con 0x7f33b01002a0 2024-04-16T03:17:38.375 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.373+0000 7f33a2ffd640 1 -- 172.21.15.151:0/4059955353 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v42) v1 ==== 107+0+5066 (secure 0 0 0) 0x7f33a407d670 con 0x7f33b01002a0 2024-04-16T03:17:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:38.375 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44247,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/341639119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":341639119},{"type":"v1","addr":"172.21.15.153:6827","nonce":341639119}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:38.375 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 31 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.375+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f339c06ec30 msgr2=0x7f339c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.375+0000 7f33b7b67640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f339c06ec30 0x7f339c0710f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f33ac005fd0 tx=0x7f33ac005850 comp rx=0 tx=0).stop 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.375+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 msgr2=0x7f33b0195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.375+0000 7f33b7b67640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 0x7f33b0195e00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009430 tx=0x7f33a4004040 comp rx=0 tx=0).stop 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.376+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 shutdown_connections 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.376+0000 7f33b7b67640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f339c06ec30 0x7f339c0710f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.376+0000 7f33b7b67640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b0196340 0x7f33b019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.376+0000 7f33b7b67640 1 --2- 172.21.15.151:0/4059955353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f33b01002a0 0x7f33b0195e00 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.376+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 >> 172.21.15.151:0/4059955353 conn(0x7f33b00fbb20 msgr2=0x7f33b00fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.376+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 shutdown_connections 2024-04-16T03:17:38.377 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.376+0000 7f33b7b67640 1 -- 172.21.15.151:0/4059955353 wait complete. 2024-04-16T03:17:38.387 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:38 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/129990869' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-16T03:17:38.387 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:38 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1421997856' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-16T03:17:38.387 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:38 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3580902043' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-16T03:17:38.388 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:38.388 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-04-16T03:17:38.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-16T03:17:38.496 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.493+0000 7fc1694e0640 1 -- 172.21.15.151:0/3192965323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1640ffc10 msgr2=0x7fc164103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.493+0000 7fc1694e0640 1 --2- 172.21.15.151:0/3192965323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1640ffc10 0x7fc164103b70 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc158009c20 tx=0x7fc15802c1b0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.493+0000 7fc1694e0640 1 -- 172.21.15.151:0/3192965323 shutdown_connections 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.493+0000 7fc1694e0640 1 --2- 172.21.15.151:0/3192965323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1640ffc10 0x7fc164103b70 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.493+0000 7fc1694e0640 1 --2- 172.21.15.151:0/3192965323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1640ff260 0x7fc1640ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.493+0000 7fc1694e0640 1 -- 172.21.15.151:0/3192965323 >> 172.21.15.151:0/3192965323 conn(0x7fc164074a50 msgr2=0x7fc164074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.493+0000 7fc1694e0640 1 -- 172.21.15.151:0/3192965323 shutdown_connections 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.494+0000 7fc1694e0640 1 -- 172.21.15.151:0/3192965323 wait complete. 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.494+0000 7fc1694e0640 1 Processor -- start 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.494+0000 7fc1694e0640 1 -- start start 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.494+0000 7fc1694e0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1640ff260 0x7fc164195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:38.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.494+0000 7fc1694e0640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc164196310 0x7fc16419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.494+0000 7fc1694e0640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc164196910 con 0x7fc164196310 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.494+0000 7fc1694e0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc164196a80 con 0x7fc1640ff260 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc1627fc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc164196310 0x7fc16419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc1627fc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc164196310 0x7fc16419a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59106/0 (socket says 172.21.15.151:59106) 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc1627fc640 1 -- 172.21.15.151:0/4068102264 learned_addr learned my addr 172.21.15.151:0/4068102264 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc162ffd640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1640ff260 0x7fc164195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc1627fc640 1 -- 172.21.15.151:0/4068102264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1640ff260 msgr2=0x7fc164195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc1627fc640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1640ff260 0x7fc164195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc1627fc640 1 -- 172.21.15.151:0/4068102264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc158009900 con 0x7fc164196310 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc1627fc640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc164196310 0x7fc16419a770 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc158032b60 tx=0x7fc158032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:38.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc14bfff640 1 -- 172.21.15.151:0/4068102264 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc158034650 con 0x7fc164196310 2024-04-16T03:17:38.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16419ad10 con 0x7fc164196310 2024-04-16T03:17:38.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc14bfff640 1 -- 172.21.15.151:0/4068102264 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc158003e30 con 0x7fc164196310 2024-04-16T03:17:38.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc14bfff640 1 -- 172.21.15.151:0/4068102264 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc158004890 con 0x7fc164196310 2024-04-16T03:17:38.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.495+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16419b270 con 0x7fc164196310 2024-04-16T03:17:38.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.496+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16404e040 con 0x7fc164196310 2024-04-16T03:17:38.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.497+0000 7fc14bfff640 1 -- 172.21.15.151:0/4068102264 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc15803b040 con 0x7fc164196310 2024-04-16T03:17:38.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.497+0000 7fc14bfff640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc14c06ebe0 0x7fc14c0710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:38.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.497+0000 7fc14bfff640 1 -- 172.21.15.151:0/4068102264 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc1580b1980 con 0x7fc164196310 2024-04-16T03:17:38.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.497+0000 7fc162ffd640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc14c06ebe0 0x7fc14c0710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:38.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.498+0000 7fc162ffd640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc14c06ebe0 0x7fc14c0710a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc154009bd0 tx=0x7fc154005f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:38.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.502+0000 7fc14bfff640 1 -- 172.21.15.151:0/4068102264 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc15807a730 con 0x7fc164196310 2024-04-16T03:17:38.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:38 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/129990869' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-16T03:17:38.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:38 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1421997856' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-16T03:17:38.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:38 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3580902043' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-16T03:17:38.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.720+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7fc164197120 con 0x7fc164196310 2024-04-16T03:17:38.721 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.720+0000 7fc14bfff640 1 -- 172.21.15.151:0/4068102264 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v42) v1 ==== 107+0+4287 (secure 0 0 0) 0x7fc15807ddf0 con 0x7fc164196310 2024-04-16T03:17:38.721 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:38.722 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:38.722 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 32 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.722+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc14c06ebe0 msgr2=0x7fc14c0710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.722+0000 7fc1694e0640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc14c06ebe0 0x7fc14c0710a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc154009bd0 tx=0x7fc154005f90 comp rx=0 tx=0).stop 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.722+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc164196310 msgr2=0x7fc16419a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.722+0000 7fc1694e0640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc164196310 0x7fc16419a770 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc158032b60 tx=0x7fc158032b90 comp rx=0 tx=0).stop 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.723+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 shutdown_connections 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.723+0000 7fc1694e0640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc14c06ebe0 0x7fc14c0710a0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.723+0000 7fc1694e0640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc164196310 0x7fc16419a770 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.723+0000 7fc1694e0640 1 --2- 172.21.15.151:0/4068102264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1640ff260 0x7fc164195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.723+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 >> 172.21.15.151:0/4068102264 conn(0x7fc164074a50 msgr2=0x7fc1640fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:38.724 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.723+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 shutdown_connections 2024-04-16T03:17:38.725 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.723+0000 7fc1694e0640 1 -- 172.21.15.151:0/4068102264 wait complete. 2024-04-16T03:17:38.735 DEBUG:tasks.fs:max_mds not reduced in epoch 32 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:38.735 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-04-16T03:17:38.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-16T03:17:38.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 -- 172.21.15.151:0/3969057074 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 msgr2=0x7f54440679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 --2- 172.21.15.151:0/3969057074 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 0x7f54440679a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f5434009c20 tx=0x7f543402c1b0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 -- 172.21.15.151:0/3969057074 shutdown_connections 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 --2- 172.21.15.151:0/3969057074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5444067f70 0x7f5444104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 --2- 172.21.15.151:0/3969057074 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 0x7f54440679a0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 -- 172.21.15.151:0/3969057074 >> 172.21.15.151:0/3969057074 conn(0x7f5444074a50 msgr2=0x7f5444074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 -- 172.21.15.151:0/3969057074 shutdown_connections 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 -- 172.21.15.151:0/3969057074 wait complete. 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 Processor -- start 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.841+0000 7f544b68f640 1 -- start start 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544b68f640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 0x7f5444195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544b68f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54441962f0 0x7f544419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544b68f640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54441968f0 con 0x7f54440675c0 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544b68f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5444196a60 con 0x7f54441962f0 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544942b640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 0x7f5444195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544942b640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 0x7f5444195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59122/0 (socket says 172.21.15.151:59122) 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544942b640 1 -- 172.21.15.151:0/4077185174 learned_addr learned my addr 172.21.15.151:0/4077185174 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f5448c2a640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54441962f0 0x7f544419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544942b640 1 -- 172.21.15.151:0/4077185174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54441962f0 msgr2=0x7f544419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544942b640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54441962f0 0x7f544419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544942b640 1 -- 172.21.15.151:0/4077185174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5434009900 con 0x7f54440675c0 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f5448c2a640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54441962f0 0x7f544419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544942b640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 0x7f5444195db0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f5434006010 tx=0x7f5434032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f54427fc640 1 -- 172.21.15.151:0/4077185174 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5434034820 con 0x7f54440675c0 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f54427fc640 1 -- 172.21.15.151:0/4077185174 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5434003e30 con 0x7f54440675c0 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544419acf0 con 0x7f54440675c0 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f54427fc640 1 -- 172.21.15.151:0/4077185174 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5434004830 con 0x7f54440675c0 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.842+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544419b250 con 0x7f54440675c0 2024-04-16T03:17:38.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.843+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544404e040 con 0x7f54440675c0 2024-04-16T03:17:38.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.844+0000 7f54427fc640 1 -- 172.21.15.151:0/4077185174 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f543403b040 con 0x7f54440675c0 2024-04-16T03:17:38.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.844+0000 7f54427fc640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ec30 0x7f542c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:38.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.845+0000 7f5448c2a640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ec30 0x7f542c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:38.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.845+0000 7f54427fc640 1 -- 172.21.15.151:0/4077185174 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f54340b1c10 con 0x7f54440675c0 2024-04-16T03:17:38.847 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.845+0000 7f5448c2a640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ec30 0x7f542c0710f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f54380058c0 tx=0x7f5438005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:38.850 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:38.849+0000 7f54427fc640 1 -- 172.21.15.151:0/4077185174 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f543407a9c0 con 0x7f54440675c0 2024-04-16T03:17:39.068 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.067+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f544419b560 con 0x7f54440675c0 2024-04-16T03:17:39.069 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.068+0000 7f54427fc640 1 -- 172.21.15.151:0/4077185174 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v42) v1 ==== 107+0+5137 (secure 0 0 0) 0x7f543407e080 con 0x7f54440675c0 2024-04-16T03:17:39.069 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:39.069 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T02:44:25.408848+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:39.069 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 33 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.070+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ec30 msgr2=0x7f542c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.070+0000 7f544b68f640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ec30 0x7f542c0710f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f54380058c0 tx=0x7f5438005850 comp rx=0 tx=0).stop 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.070+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 msgr2=0x7f5444195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.070+0000 7f544b68f640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 0x7f5444195db0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f5434006010 tx=0x7f5434032b60 comp rx=0 tx=0).stop 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.070+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 shutdown_connections 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.070+0000 7f544b68f640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f542c06ec30 0x7f542c0710f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.070+0000 7f544b68f640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54441962f0 0x7f544419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.070+0000 7f544b68f640 1 --2- 172.21.15.151:0/4077185174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54440675c0 0x7f5444195db0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.070+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 >> 172.21.15.151:0/4077185174 conn(0x7f5444074a50 msgr2=0x7f54440fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.071+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 shutdown_connections 2024-04-16T03:17:39.072 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.071+0000 7f544b68f640 1 -- 172.21.15.151:0/4077185174 wait complete. 2024-04-16T03:17:39.083 DEBUG:tasks.fs:max_mds not reduced in epoch 33 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-16T03:17:39.083 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-04-16T03:17:39.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-16T03:17:39.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.188+0000 7f05f8d59640 1 -- 172.21.15.151:0/3694994958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f4105110 msgr2=0x7f05f41054f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.188+0000 7f05f8d59640 1 --2- 172.21.15.151:0/3694994958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f4105110 0x7f05f41054f0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009880 tx=0x7f05e402bd70 comp rx=0 tx=0).stop 2024-04-16T03:17:39.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.188+0000 7f05f8d59640 1 -- 172.21.15.151:0/3694994958 shutdown_connections 2024-04-16T03:17:39.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.188+0000 7f05f8d59640 1 --2- 172.21.15.151:0/3694994958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05f40ff810 0x7f05f40ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.188+0000 7f05f8d59640 1 --2- 172.21.15.151:0/3694994958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f4105110 0x7f05f41054f0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.188+0000 7f05f8d59640 1 -- 172.21.15.151:0/3694994958 >> 172.21.15.151:0/3694994958 conn(0x7f05f4076620 msgr2=0x7f05f4076a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:39.190 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.188+0000 7f05f8d59640 1 -- 172.21.15.151:0/3694994958 shutdown_connections 2024-04-16T03:17:39.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.188+0000 7f05f8d59640 1 -- 172.21.15.151:0/3694994958 wait complete. 2024-04-16T03:17:39.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.189+0000 7f05f8d59640 1 Processor -- start 2024-04-16T03:17:39.191 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.189+0000 7f05f8d59640 1 -- start start 2024-04-16T03:17:39.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.189+0000 7f05f8d59640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f40ff810 0x7f05f4195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:39.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.189+0000 7f05f8d59640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05f4105110 0x7f05f4196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.189+0000 7f05f8d59640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f4196b00 con 0x7f05f40ff810 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.189+0000 7f05f8d59640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f419a8d0 con 0x7f05f4105110 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05f259c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f40ff810 0x7f05f4195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05f259c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f40ff810 0x7f05f4195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59142/0 (socket says 172.21.15.151:59142) 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05f259c640 1 -- 172.21.15.151:0/2657301903 learned_addr learned my addr 172.21.15.151:0/2657301903 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05f259c640 1 -- 172.21.15.151:0/2657301903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05f4105110 msgr2=0x7f05f4196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05f259c640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05f4105110 0x7f05f4196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05f259c640 1 -- 172.21.15.151:0/2657301903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e4009560 con 0x7f05f40ff810 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05f259c640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f40ff810 0x7f05f4195f30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009230 tx=0x7f05e4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05eb7fe640 1 -- 172.21.15.151:0/2657301903 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05e40037c0 con 0x7f05f40ff810 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f419ab50 con 0x7f05f40ff810 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05eb7fe640 1 -- 172.21.15.151:0/2657301903 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05e4004aa0 con 0x7f05f40ff810 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05eb7fe640 1 -- 172.21.15.151:0/2657301903 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05e403e450 con 0x7f05f40ff810 2024-04-16T03:17:39.193 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.190+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f419b0b0 con 0x7f05f40ff810 2024-04-16T03:17:39.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.191+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f404e040 con 0x7f05f40ff810 2024-04-16T03:17:39.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.192+0000 7f05eb7fe640 1 -- 172.21.15.151:0/2657301903 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f05e4005060 con 0x7f05f40ff810 2024-04-16T03:17:39.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.192+0000 7f05eb7fe640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f05e006ec30 0x7f05e00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:39.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.192+0000 7f05f1d9b640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f05e006ec30 0x7f05e00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:39.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.192+0000 7f05eb7fe640 1 -- 172.21.15.151:0/2657301903 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f05e40b1cd0 con 0x7f05f40ff810 2024-04-16T03:17:39.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.193+0000 7f05f1d9b640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f05e006ec30 0x7f05e00710f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f05f4197540 tx=0x7f05dc006c90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:39.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.197+0000 7f05eb7fe640 1 -- 172.21.15.151:0/2657301903 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f05e407aa80 con 0x7f05f40ff810 2024-04-16T03:17:39.412 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.412+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f05f410c380 con 0x7f05f40ff810 2024-04-16T03:17:39.413 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.412+0000 7f05eb7fe640 1 -- 172.21.15.151:0/2657301903 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v42) v1 ==== 107+0+5136 (secure 0 0 0) 0x7f05e407e140 con 0x7f05f40ff810 2024-04-16T03:17:39.413 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:39.414 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T03:15:24.542524+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:39.414 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 34 2024-04-16T03:17:39.415 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f05e006ec30 msgr2=0x7f05e00710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f05e006ec30 0x7f05e00710f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f05f4197540 tx=0x7f05dc006c90 comp rx=0 tx=0).stop 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f40ff810 msgr2=0x7f05f4195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f40ff810 0x7f05f4195f30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009230 tx=0x7f05e4004270 comp rx=0 tx=0).stop 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 shutdown_connections 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f05e006ec30 0x7f05e00710f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05f4105110 0x7f05f4196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 --2- 172.21.15.151:0/2657301903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05f40ff810 0x7f05f4195f30 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 >> 172.21.15.151:0/2657301903 conn(0x7f05f4076620 msgr2=0x7f05f40fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 shutdown_connections 2024-04-16T03:17:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.414+0000 7f05f8d59640 1 -- 172.21.15.151:0/2657301903 wait complete. 2024-04-16T03:17:39.425 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:39 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4059955353' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-16T03:17:39.425 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:39 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4068102264' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-16T03:17:39.425 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:39 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:17:39.425 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:39 smithi151 ceph-mon[112233]: pgmap v1174: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:39.425 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:39 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4077185174' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-16T03:17:39.426 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 34 2024-04-16T03:17:39.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-16T03:17:39.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:39 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4059955353' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-16T03:17:39.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:39 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4068102264' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-16T03:17:39.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:39 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:17:39.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:39 smithi153 ceph-mon[87858]: pgmap v1174: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:39.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:39 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4077185174' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-16T03:17:39.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.532+0000 7f3e27e69640 1 -- 172.21.15.151:0/2864392512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e200ffc10 msgr2=0x7f3e20103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.532+0000 7f3e27e69640 1 --2- 172.21.15.151:0/2864392512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e200ffc10 0x7f3e20103b70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009880 tx=0x7f3e1c02bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.532+0000 7f3e27e69640 1 -- 172.21.15.151:0/2864392512 shutdown_connections 2024-04-16T03:17:39.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.532+0000 7f3e27e69640 1 --2- 172.21.15.151:0/2864392512 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e200ffc10 0x7f3e20103b70 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.532+0000 7f3e27e69640 1 --2- 172.21.15.151:0/2864392512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e200ff260 0x7f3e200ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.532+0000 7f3e27e69640 1 -- 172.21.15.151:0/2864392512 >> 172.21.15.151:0/2864392512 conn(0x7f3e20074a50 msgr2=0x7f3e20074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:39.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.532+0000 7f3e27e69640 1 -- 172.21.15.151:0/2864392512 shutdown_connections 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.532+0000 7f3e27e69640 1 -- 172.21.15.151:0/2864392512 wait complete. 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e27e69640 1 Processor -- start 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e27e69640 1 -- start start 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e27e69640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e200ff260 0x7f3e20195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e27e69640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e201962b0 0x7f3e2019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e27e69640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e201968b0 con 0x7f3e201962b0 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e27e69640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20196a20 con 0x7f3e200ff260 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e25404640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e201962b0 0x7f3e2019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e25404640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e201962b0 0x7f3e2019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59164/0 (socket says 172.21.15.151:59164) 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e25404640 1 -- 172.21.15.151:0/2935067615 learned_addr learned my addr 172.21.15.151:0/2935067615 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e25c05640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e200ff260 0x7f3e20195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e25404640 1 -- 172.21.15.151:0/2935067615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e200ff260 msgr2=0x7f3e20195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e25404640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e200ff260 0x7f3e20195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.536 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e25404640 1 -- 172.21.15.151:0/2935067615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c009560 con 0x7f3e201962b0 2024-04-16T03:17:39.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.533+0000 7f3e25404640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e201962b0 0x7f3e2019a710 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c036040 tx=0x7f3e1c02c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:39.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.534+0000 7f3e12ffd640 1 -- 172.21.15.151:0/2935067615 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e1c0035f0 con 0x7f3e201962b0 2024-04-16T03:17:39.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.534+0000 7f3e12ffd640 1 -- 172.21.15.151:0/2935067615 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e1c003c10 con 0x7f3e201962b0 2024-04-16T03:17:39.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.534+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2019acb0 con 0x7f3e201962b0 2024-04-16T03:17:39.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.534+0000 7f3e12ffd640 1 -- 172.21.15.151:0/2935067615 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e1c005630 con 0x7f3e201962b0 2024-04-16T03:17:39.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.534+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2019b210 con 0x7f3e201962b0 2024-04-16T03:17:39.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.535+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2004e040 con 0x7f3e201962b0 2024-04-16T03:17:39.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.535+0000 7f3e12ffd640 1 -- 172.21.15.151:0/2935067615 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f3e1c003750 con 0x7f3e201962b0 2024-04-16T03:17:39.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.536+0000 7f3e12ffd640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e0c06ec30 0x7f3e0c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:39.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.536+0000 7f3e12ffd640 1 -- 172.21.15.151:0/2935067615 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3e1c0b1220 con 0x7f3e201962b0 2024-04-16T03:17:39.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.536+0000 7f3e25c05640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e0c06ec30 0x7f3e0c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:39.538 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.536+0000 7f3e25c05640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e0c06ec30 0x7f3e0c0710f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009b60 tx=0x7f3e14005f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:39.541 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.540+0000 7f3e12ffd640 1 -- 172.21.15.151:0/2935067615 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3e1c079fd0 con 0x7f3e201962b0 2024-04-16T03:17:39.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.756+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f3e201970c0 con 0x7f3e201962b0 2024-04-16T03:17:39.758 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.757+0000 7f3e12ffd640 1 -- 172.21.15.151:0/2935067615 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v42) v1 ==== 107+0+5227 (secure 0 0 0) 0x7f3e1c09ce60 con 0x7f3e201962b0 2024-04-16T03:17:39.758 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:39.758 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T03:15:25.544137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:39.759 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 35 2024-04-16T03:17:39.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e0c06ec30 msgr2=0x7f3e0c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e0c06ec30 0x7f3e0c0710f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009b60 tx=0x7f3e14005f90 comp rx=0 tx=0).stop 2024-04-16T03:17:39.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e201962b0 msgr2=0x7f3e2019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e201962b0 0x7f3e2019a710 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c036040 tx=0x7f3e1c02c750 comp rx=0 tx=0).stop 2024-04-16T03:17:39.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 shutdown_connections 2024-04-16T03:17:39.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f3e0c06ec30 0x7f3e0c0710f0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3e201962b0 0x7f3e2019a710 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 --2- 172.21.15.151:0/2935067615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e200ff260 0x7f3e20195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 >> 172.21.15.151:0/2935067615 conn(0x7f3e20074a50 msgr2=0x7f3e20103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:39.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 shutdown_connections 2024-04-16T03:17:39.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.759+0000 7f3e27e69640 1 -- 172.21.15.151:0/2935067615 wait complete. 2024-04-16T03:17:39.771 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 35 2024-04-16T03:17:39.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-04-16T03:17:39.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.875+0000 7f8bc900e640 1 -- 172.21.15.151:0/1936329598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc40ffc10 msgr2=0x7f8bc4103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.875+0000 7f8bc900e640 1 --2- 172.21.15.151:0/1936329598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc40ffc10 0x7f8bc4103b70 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009880 tx=0x7f8bb802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.875+0000 7f8bc900e640 1 -- 172.21.15.151:0/1936329598 shutdown_connections 2024-04-16T03:17:39.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.875+0000 7f8bc900e640 1 --2- 172.21.15.151:0/1936329598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc40ffc10 0x7f8bc4103b70 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.875+0000 7f8bc900e640 1 --2- 172.21.15.151:0/1936329598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc40ff260 0x7f8bc40ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.875+0000 7f8bc900e640 1 -- 172.21.15.151:0/1936329598 >> 172.21.15.151:0/1936329598 conn(0x7f8bc4074a50 msgr2=0x7f8bc4074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:39.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.876+0000 7f8bc900e640 1 -- 172.21.15.151:0/1936329598 shutdown_connections 2024-04-16T03:17:39.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.876+0000 7f8bc900e640 1 -- 172.21.15.151:0/1936329598 wait complete. 2024-04-16T03:17:39.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.876+0000 7f8bc900e640 1 Processor -- start 2024-04-16T03:17:39.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.876+0000 7f8bc900e640 1 -- start start 2024-04-16T03:17:39.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc900e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc40ff260 0x7f8bc4100c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:39.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc900e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4101150 0x7f8bc41015b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:39.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc900e640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc41a1fa0 con 0x7f8bc4101150 2024-04-16T03:17:39.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc900e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc41a20e0 con 0x7f8bc40ff260 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc259c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4101150 0x7f8bc41015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc259c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4101150 0x7f8bc41015b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59186/0 (socket says 172.21.15.151:59186) 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc259c640 1 -- 172.21.15.151:0/416815101 learned_addr learned my addr 172.21.15.151:0/416815101 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc259c640 1 -- 172.21.15.151:0/416815101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc40ff260 msgr2=0x7f8bc4100c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc2d9d640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc40ff260 0x7f8bc4100c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc259c640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc40ff260 0x7f8bc4100c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc259c640 1 -- 172.21.15.151:0/416815101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb8009560 con 0x7f8bc4101150 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc259c640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4101150 0x7f8bc41015b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8036040 tx=0x7f8bb802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8babfff640 1 -- 172.21.15.151:0/416815101 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb80035f0 con 0x7f8bc4101150 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8babfff640 1 -- 172.21.15.151:0/416815101 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bb8003c10 con 0x7f8bc4101150 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc41a2220 con 0x7f8bc4101150 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8babfff640 1 -- 172.21.15.151:0/416815101 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb8034440 con 0x7f8bc4101150 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.877+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc41a26d0 con 0x7f8bc4101150 2024-04-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.878+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc404e040 con 0x7f8bc4101150 2024-04-16T03:17:39.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.879+0000 7f8babfff640 1 -- 172.21.15.151:0/416815101 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8bb8003750 con 0x7f8bc4101150 2024-04-16T03:17:39.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.879+0000 7f8babfff640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8bac06ec30 0x7f8bac0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:39.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.879+0000 7f8babfff640 1 -- 172.21.15.151:0/416815101 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8bb80b1240 con 0x7f8bc4101150 2024-04-16T03:17:39.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.879+0000 7f8bc2d9d640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8bac06ec30 0x7f8bac0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:39.881 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.880+0000 7f8bc2d9d640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8bac06ec30 0x7f8bac0710f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4005fd0 tx=0x7f8bb4005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:39.885 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:39.884+0000 7f8babfff640 1 -- 172.21.15.151:0/416815101 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8bb8079ff0 con 0x7f8bc4101150 2024-04-16T03:17:40.100 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.099+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7f8bc4102140 con 0x7f8bc4101150 2024-04-16T03:17:40.101 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.100+0000 7f8babfff640 1 -- 172.21.15.151:0/416815101 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v42) v1 ==== 107+0+5226 (secure 0 0 0) 0x7f8bb807d6b0 con 0x7f8bc4101150 2024-04-16T03:17:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:40.101 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T03:15:25.548544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:40.101 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 36 2024-04-16T03:17:40.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8bac06ec30 msgr2=0x7f8bac0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8bac06ec30 0x7f8bac0710f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4005fd0 tx=0x7f8bb4005850 comp rx=0 tx=0).stop 2024-04-16T03:17:40.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4101150 msgr2=0x7f8bc41015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4101150 0x7f8bc41015b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8036040 tx=0x7f8bb802c850 comp rx=0 tx=0).stop 2024-04-16T03:17:40.103 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 shutdown_connections 2024-04-16T03:17:40.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8bac06ec30 0x7f8bac0710f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4101150 0x7f8bc41015b0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 --2- 172.21.15.151:0/416815101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bc40ff260 0x7f8bc4100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 >> 172.21.15.151:0/416815101 conn(0x7f8bc4074a50 msgr2=0x7f8bc4103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:40.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 shutdown_connections 2024-04-16T03:17:40.104 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.102+0000 7f8bc900e640 1 -- 172.21.15.151:0/416815101 wait complete. 2024-04-16T03:17:40.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 -- 172.21.15.151:0/3536307927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f44106070 msgr2=0x7f8f44106450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 --2- 172.21.15.151:0/3536307927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f44106070 0x7f8f44106450 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009c20 tx=0x7f8f3402c1b0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 -- 172.21.15.151:0/3536307927 shutdown_connections 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 --2- 172.21.15.151:0/3536307927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f440fe730 0x7f8f440feb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 --2- 172.21.15.151:0/3536307927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f44106070 0x7f8f44106450 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 -- 172.21.15.151:0/3536307927 >> 172.21.15.151:0/3536307927 conn(0x7f8f44074250 msgr2=0x7f8f44074660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 -- 172.21.15.151:0/3536307927 shutdown_connections 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 -- 172.21.15.151:0/3536307927 wait complete. 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 Processor -- start 2024-04-16T03:17:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.220+0000 7f8f496ea640 1 -- start start 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f496ea640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f440fe730 0x7f8f440ff910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f496ea640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44106070 0x7f8f440ffe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f496ea640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44100420 con 0x7f8f440fe730 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f496ea640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44100560 con 0x7f8f44106070 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f440fe730 0x7f8f440ff910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f440fe730 0x7f8f440ff910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59218/0 (socket says 172.21.15.151:59218) 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f42ffd640 1 -- 172.21.15.151:0/488778464 learned_addr learned my addr 172.21.15.151:0/488778464 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f427fc640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44106070 0x7f8f440ffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f42ffd640 1 -- 172.21.15.151:0/488778464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44106070 msgr2=0x7f8f440ffe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f42ffd640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44106070 0x7f8f440ffe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f42ffd640 1 -- 172.21.15.151:0/488778464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f34009900 con 0x7f8f440fe730 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f42ffd640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f440fe730 0x7f8f440ff910 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f8f34037040 tx=0x7f8f34032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f3bfff640 1 -- 172.21.15.151:0/488778464 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f34032ed0 con 0x7f8f440fe730 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f3bfff640 1 -- 172.21.15.151:0/488778464 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f34003e30 con 0x7f8f440fe730 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f44100790 con 0x7f8f440fe730 2024-04-16T03:17:40.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.221+0000 7f8f3bfff640 1 -- 172.21.15.151:0/488778464 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f34004830 con 0x7f8f440fe730 2024-04-16T03:17:40.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.222+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f441a22c0 con 0x7f8f440fe730 2024-04-16T03:17:40.225 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.222+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4404e040 con 0x7f8f440fe730 2024-04-16T03:17:40.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.223+0000 7f8f3bfff640 1 -- 172.21.15.151:0/488778464 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8f3403b040 con 0x7f8f440fe730 2024-04-16T03:17:40.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.223+0000 7f8f3bfff640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8f3006ec30 0x7f8f300710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:40.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.223+0000 7f8f3bfff640 1 -- 172.21.15.151:0/488778464 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8f340b1bf0 con 0x7f8f440fe730 2024-04-16T03:17:40.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.224+0000 7f8f427fc640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8f3006ec30 0x7f8f300710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:40.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.224+0000 7f8f427fc640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8f3006ec30 0x7f8f300710f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8f44101480 tx=0x7f8f2c005f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:40.229 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.228+0000 7f8f3bfff640 1 -- 172.21.15.151:0/488778464 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8f3407a9a0 con 0x7f8f440fe730 2024-04-16T03:17:40.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.443+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f8f4410d3f0 con 0x7f8f440fe730 2024-04-16T03:17:40.444 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.443+0000 7f8f3bfff640 1 -- 172.21.15.151:0/488778464 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v42) v1 ==== 107+0+5232 (secure 0 0 0) 0x7f8f3407e060 con 0x7f8f440fe730 2024-04-16T03:17:40.445 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:40.445 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T03:15:33.565011+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":35,"state":"up:resolve","state_seq":466,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":35,"state":"up:resolve","state_seq":461,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:40.445 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 37 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8f3006ec30 msgr2=0x7f8f300710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8f3006ec30 0x7f8f300710f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8f44101480 tx=0x7f8f2c005f90 comp rx=0 tx=0).stop 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f440fe730 msgr2=0x7f8f440ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f440fe730 0x7f8f440ff910 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f8f34037040 tx=0x7f8f34032b60 comp rx=0 tx=0).stop 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 shutdown_connections 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f8f3006ec30 0x7f8f300710f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44106070 0x7f8f440ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 --2- 172.21.15.151:0/488778464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f440fe730 0x7f8f440ff910 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 >> 172.21.15.151:0/488778464 conn(0x7f8f44074250 msgr2=0x7f8f440fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 shutdown_connections 2024-04-16T03:17:40.448 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.446+0000 7f8f496ea640 1 -- 172.21.15.151:0/488778464 wait complete. 2024-04-16T03:17:40.457 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:40 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2657301903' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-16T03:17:40.457 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:40 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2935067615' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-16T03:17:40.457 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:40 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/416815101' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-16T03:17:40.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-04-16T03:17:40.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:40 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2657301903' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-16T03:17:40.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:40 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2935067615' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-16T03:17:40.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:40 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/416815101' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-16T03:17:40.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.564+0000 7f91d665f640 1 -- 172.21.15.151:0/3164202646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d0100c50 msgr2=0x7f91d0104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.564+0000 7f91d665f640 1 --2- 172.21.15.151:0/3164202646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d0100c50 0x7f91d0104c90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009880 tx=0x7f91b802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.564+0000 7f91d665f640 1 -- 172.21.15.151:0/3164202646 shutdown_connections 2024-04-16T03:17:40.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.564+0000 7f91d665f640 1 --2- 172.21.15.151:0/3164202646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d0100c50 0x7f91d0104c90 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.564+0000 7f91d665f640 1 --2- 172.21.15.151:0/3164202646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91d01002a0 0x7f91d0100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.564+0000 7f91d665f640 1 -- 172.21.15.151:0/3164202646 >> 172.21.15.151:0/3164202646 conn(0x7f91d00fbb20 msgr2=0x7f91d00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:40.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.564+0000 7f91d665f640 1 -- 172.21.15.151:0/3164202646 shutdown_connections 2024-04-16T03:17:40.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.564+0000 7f91d665f640 1 -- 172.21.15.151:0/3164202646 wait complete. 2024-04-16T03:17:40.567 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91d665f640 1 Processor -- start 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91d665f640 1 -- start start 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91d665f640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d01002a0 0x7f91d0195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91d665f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91d01962e0 0x7f91d019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91d665f640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d01968e0 con 0x7f91d01002a0 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91d665f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d0196a50 con 0x7f91d01962e0 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91cffff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d01002a0 0x7f91d0195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91cffff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d01002a0 0x7f91d0195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59240/0 (socket says 172.21.15.151:59240) 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91cffff640 1 -- 172.21.15.151:0/3700961153 learned_addr learned my addr 172.21.15.151:0/3700961153 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91cffff640 1 -- 172.21.15.151:0/3700961153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91d01962e0 msgr2=0x7f91d019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91c7fff640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91d01962e0 0x7f91d019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91cffff640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91d01962e0 0x7f91d019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.565+0000 7f91cffff640 1 -- 172.21.15.151:0/3700961153 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b8009560 con 0x7f91d01002a0 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.566+0000 7f91cffff640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d01002a0 0x7f91d0195da0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f91c000d720 tx=0x7f91c000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.566+0000 7f91cdffb640 1 -- 172.21.15.151:0/3700961153 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91c00051f0 con 0x7f91d01002a0 2024-04-16T03:17:40.568 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.566+0000 7f91cdffb640 1 -- 172.21.15.151:0/3700961153 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91c0009ce0 con 0x7f91d01002a0 2024-04-16T03:17:40.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.566+0000 7f91cdffb640 1 -- 172.21.15.151:0/3700961153 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91c0010460 con 0x7f91d01002a0 2024-04-16T03:17:40.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.566+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d019ad40 con 0x7f91d01002a0 2024-04-16T03:17:40.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.566+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d019b2a0 con 0x7f91d01002a0 2024-04-16T03:17:40.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.567+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d004e040 con 0x7f91d01002a0 2024-04-16T03:17:40.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.567+0000 7f91cdffb640 1 -- 172.21.15.151:0/3700961153 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f91c0005350 con 0x7f91d01002a0 2024-04-16T03:17:40.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.568+0000 7f91cdffb640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f91bc06ec30 0x7f91bc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:40.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.568+0000 7f91cdffb640 1 -- 172.21.15.151:0/3700961153 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f91c008e9f0 con 0x7f91d01002a0 2024-04-16T03:17:40.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.568+0000 7f91c7fff640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f91bc06ec30 0x7f91bc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:40.570 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.568+0000 7f91c7fff640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f91bc06ec30 0x7f91bc0710f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f91b8036040 tx=0x7f91b8005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:40.573 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.572+0000 7f91cdffb640 1 -- 172.21.15.151:0/3700961153 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f91c0090e60 con 0x7f91d01002a0 2024-04-16T03:17:40.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.792+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7f91d01970f0 con 0x7f91d01002a0 2024-04-16T03:17:40.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.793+0000 7f91cdffb640 1 -- 172.21.15.151:0/3700961153 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v42) v1 ==== 107+0+5236 (secure 0 0 0) 0x7f91c00577a0 con 0x7f91d01002a0 2024-04-16T03:17:40.794 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:40.794 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T03:15:33.695094+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":35,"state":"up:reconnect","state_seq":467,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":35,"state":"up:reconnect","state_seq":462,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:40.795 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 38 2024-04-16T03:17:40.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f91bc06ec30 msgr2=0x7f91bc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f91bc06ec30 0x7f91bc0710f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f91b8036040 tx=0x7f91b8005fd0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d01002a0 msgr2=0x7f91d0195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d01002a0 0x7f91d0195da0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f91c000d720 tx=0x7f91c000dbe0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 shutdown_connections 2024-04-16T03:17:40.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f91bc06ec30 0x7f91bc0710f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91d01962e0 0x7f91d019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 --2- 172.21.15.151:0/3700961153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f91d01002a0 0x7f91d0195da0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 >> 172.21.15.151:0/3700961153 conn(0x7f91d00fbb20 msgr2=0x7f91d00fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:40.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 shutdown_connections 2024-04-16T03:17:40.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.795+0000 7f91d665f640 1 -- 172.21.15.151:0/3700961153 wait complete. 2024-04-16T03:17:40.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-04-16T03:17:40.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.912+0000 7ffa0c821640 1 -- 172.21.15.151:0/953723475 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa080675c0 msgr2=0x7ffa080679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.912+0000 7ffa0c821640 1 --2- 172.21.15.151:0/953723475 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa080675c0 0x7ffa080679a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009880 tx=0x7ff9f802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.912+0000 7ffa0c821640 1 -- 172.21.15.151:0/953723475 shutdown_connections 2024-04-16T03:17:40.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.912+0000 7ffa0c821640 1 --2- 172.21.15.151:0/953723475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa08067f70 0x7ffa08104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.912+0000 7ffa0c821640 1 --2- 172.21.15.151:0/953723475 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa080675c0 0x7ffa080679a0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.912+0000 7ffa0c821640 1 -- 172.21.15.151:0/953723475 >> 172.21.15.151:0/953723475 conn(0x7ffa08074a50 msgr2=0x7ffa08074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:40.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.912+0000 7ffa0c821640 1 -- 172.21.15.151:0/953723475 shutdown_connections 2024-04-16T03:17:40.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.912+0000 7ffa0c821640 1 -- 172.21.15.151:0/953723475 wait complete. 2024-04-16T03:17:40.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.913+0000 7ffa0c821640 1 Processor -- start 2024-04-16T03:17:40.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.913+0000 7ffa0c821640 1 -- start start 2024-04-16T03:17:40.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.913+0000 7ffa0c821640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa080675c0 0x7ffa08195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:40.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.913+0000 7ffa0c821640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa081962f0 0x7ffa0819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:40.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.913+0000 7ffa0c821640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa081968f0 con 0x7ffa081962f0 2024-04-16T03:17:40.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.913+0000 7ffa0c821640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa08196a60 con 0x7ffa080675c0 2024-04-16T03:17:40.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fdd9b640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa081962f0 0x7ffa0819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fdd9b640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa081962f0 0x7ffa0819a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59256/0 (socket says 172.21.15.151:59256) 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fdd9b640 1 -- 172.21.15.151:0/2745046508 learned_addr learned my addr 172.21.15.151:0/2745046508 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ffa0659c640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa080675c0 0x7ffa08195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fdd9b640 1 -- 172.21.15.151:0/2745046508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa080675c0 msgr2=0x7ffa08195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fdd9b640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa080675c0 0x7ffa08195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fdd9b640 1 -- 172.21.15.151:0/2745046508 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f8009560 con 0x7ffa081962f0 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fdd9b640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa081962f0 0x7ffa0819a750 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c920 tx=0x7ff9f000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fffff640 1 -- 172.21.15.151:0/2745046508 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9f0007b80 con 0x7ffa081962f0 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fffff640 1 -- 172.21.15.151:0/2745046508 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9f0007ce0 con 0x7ffa081962f0 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa0819ad50 con 0x7ffa081962f0 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ff9fffff640 1 -- 172.21.15.151:0/2745046508 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9f00187d0 con 0x7ffa081962f0 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.914+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0819b2b0 con 0x7ffa081962f0 2024-04-16T03:17:40.917 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.915+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0819b590 con 0x7ffa081962f0 2024-04-16T03:17:40.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.916+0000 7ff9fffff640 1 -- 172.21.15.151:0/2745046508 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7ff9f000f450 con 0x7ffa081962f0 2024-04-16T03:17:40.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.916+0000 7ff9fffff640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff9f406ec30 0x7ff9f40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:40.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.916+0000 7ff9fffff640 1 -- 172.21.15.151:0/2745046508 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff9f008f030 con 0x7ffa081962f0 2024-04-16T03:17:40.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.916+0000 7ffa0659c640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff9f406ec30 0x7ff9f40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:40.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.917+0000 7ffa0659c640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff9f406ec30 0x7ff9f40710f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8036040 tx=0x7ff9f8005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:40.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:40.921+0000 7ff9fffff640 1 -- 172.21.15.151:0/2745046508 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff9f0057de0 con 0x7ffa081962f0 2024-04-16T03:17:41.135 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.134+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7ffa08197100 con 0x7ffa081962f0 2024-04-16T03:17:41.136 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.135+0000 7ff9fffff640 1 -- 172.21.15.151:0/2745046508 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v42) v1 ==== 107+0+5230 (secure 0 0 0) 0x7ff9f005b4a0 con 0x7ffa081962f0 2024-04-16T03:17:41.136 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:41.136 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T03:15:34.669591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":35,"state":"up:rejoin","state_seq":468,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":35,"state":"up:rejoin","state_seq":463,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T03:17:41.136 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 39 2024-04-16T03:17:41.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff9f406ec30 msgr2=0x7ff9f40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff9f406ec30 0x7ff9f40710f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8036040 tx=0x7ff9f8005fd0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa081962f0 msgr2=0x7ffa0819a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa081962f0 0x7ffa0819a750 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c920 tx=0x7ff9f000cde0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 shutdown_connections 2024-04-16T03:17:41.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7ff9f406ec30 0x7ff9f40710f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.138 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa081962f0 0x7ffa0819a750 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 --2- 172.21.15.151:0/2745046508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa080675c0 0x7ffa08195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 >> 172.21.15.151:0/2745046508 conn(0x7ffa08074a50 msgr2=0x7ffa080fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:41.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 shutdown_connections 2024-04-16T03:17:41.139 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.137+0000 7ffa0c821640 1 -- 172.21.15.151:0/2745046508 wait complete. 2024-04-16T03:17:41.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 40 2024-04-16T03:17:41.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 -- 172.21.15.151:0/3556600601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ff260 msgr2=0x7f24f00ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 --2- 172.21.15.151:0/3556600601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ff260 0x7f24f00ff640 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009880 tx=0x7f24e002bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 -- 172.21.15.151:0/3556600601 shutdown_connections 2024-04-16T03:17:41.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 --2- 172.21.15.151:0/3556600601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f00ffc10 0x7f24f0103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 --2- 172.21.15.151:0/3556600601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ff260 0x7f24f00ff640 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 -- 172.21.15.151:0/3556600601 >> 172.21.15.151:0/3556600601 conn(0x7f24f0074a50 msgr2=0x7f24f0074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:41.259 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 -- 172.21.15.151:0/3556600601 shutdown_connections 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 -- 172.21.15.151:0/3556600601 wait complete. 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 Processor -- start 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.256+0000 7f24f5b00640 1 -- start start 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24f5b00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f00ff260 0x7f24f0195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24f5b00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f0196370 0x7f24f019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24f5b00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f01968e0 con 0x7f24f0196370 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24f5b00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f0196a50 con 0x7f24f00ff260 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24e7fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f0196370 0x7f24f019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24e7fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f0196370 0x7f24f019a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59284/0 (socket says 172.21.15.151:59284) 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24e7fff640 1 -- 172.21.15.151:0/1607212326 learned_addr learned my addr 172.21.15.151:0/1607212326 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24ef7fe640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f00ff260 0x7f24f0195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24e7fff640 1 -- 172.21.15.151:0/1607212326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f00ff260 msgr2=0x7f24f0195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24e7fff640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f00ff260 0x7f24f0195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24e7fff640 1 -- 172.21.15.151:0/1607212326 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e0009560 con 0x7f24f0196370 2024-04-16T03:17:41.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24ef7fe640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f00ff260 0x7f24f0195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:41.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24e7fff640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f0196370 0x7f24f019a7d0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f24d800c8f0 tx=0x7f24d800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:41.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24ed7fa640 1 -- 172.21.15.151:0/1607212326 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24d8004410 con 0x7f24f0196370 2024-04-16T03:17:41.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f019ad70 con 0x7f24f0196370 2024-04-16T03:17:41.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24ed7fa640 1 -- 172.21.15.151:0/1607212326 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24d8007c60 con 0x7f24f0196370 2024-04-16T03:17:41.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24ed7fa640 1 -- 172.21.15.151:0/1607212326 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24d800f450 con 0x7f24f0196370 2024-04-16T03:17:41.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.257+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f019b280 con 0x7f24f0196370 2024-04-16T03:17:41.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.258+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f004e040 con 0x7f24f0196370 2024-04-16T03:17:41.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.259+0000 7f24ed7fa640 1 -- 172.21.15.151:0/1607212326 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f24d80077a0 con 0x7f24f0196370 2024-04-16T03:17:41.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.259+0000 7f24ed7fa640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f24dc06ec30 0x7f24dc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:41.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.260+0000 7f24ed7fa640 1 -- 172.21.15.151:0/1607212326 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f24d808eec0 con 0x7f24f0196370 2024-04-16T03:17:41.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.260+0000 7f24ef7fe640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f24dc06ec30 0x7f24dc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:41.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.260+0000 7f24ef7fe640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f24dc06ec30 0x7f24dc0710f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f24e0000c00 tx=0x7f24e002cc20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:41.269 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.268+0000 7f24ed7fa640 1 -- 172.21.15.151:0/1607212326 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f24d8057c70 con 0x7f24f0196370 2024-04-16T03:17:41.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.483+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 40, "format": "json"} v 0) v1 -- 0x7f24f00ffc10 con 0x7f24f0196370 2024-04-16T03:17:41.484 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.484+0000 7f24ed7fa640 1 -- 172.21.15.151:0/1607212326 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 40, "format": "json"}]=0 dumped fsmap epoch 40 v42) v1 ==== 107+0+5245 (secure 0 0 0) 0x7f24d805b330 con 0x7f24f0196370 2024-04-16T03:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T03:15:36.672978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":35,"state":"up:active","state_seq":469,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":35,"state":"up:active","state_seq":464,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34416,"qdb_cluster":[34416,34420]},"id":1}]} 2024-04-16T03:17:41.485 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 40 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f24dc06ec30 msgr2=0x7f24dc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f24dc06ec30 0x7f24dc0710f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f24e0000c00 tx=0x7f24e002cc20 comp rx=0 tx=0).stop 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f0196370 msgr2=0x7f24f019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f0196370 0x7f24f019a7d0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f24d800c8f0 tx=0x7f24d800cdb0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 shutdown_connections 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f24dc06ec30 0x7f24dc0710f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f0196370 0x7f24f019a7d0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 --2- 172.21.15.151:0/1607212326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f00ff260 0x7f24f0195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 >> 172.21.15.151:0/1607212326 conn(0x7f24f0074a50 msgr2=0x7f24f0103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 shutdown_connections 2024-04-16T03:17:41.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.486+0000 7f24f5b00640 1 -- 172.21.15.151:0/1607212326 wait complete. 2024-04-16T03:17:41.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:41 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/488778464' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-16T03:17:41.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:41 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3700961153' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-16T03:17:41.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:41 smithi151 ceph-mon[112233]: pgmap v1175: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:41.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:41 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2745046508' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-16T03:17:41.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 41 2024-04-16T03:17:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:41 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/488778464' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-16T03:17:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:41 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3700961153' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-16T03:17:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:41 smithi153 ceph-mon[87858]: pgmap v1175: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:41 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2745046508' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.604+0000 7f7a96799640 1 -- 172.21.15.151:0/3670595753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ffc10 msgr2=0x7f7a90103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.604+0000 7f7a96799640 1 --2- 172.21.15.151:0/3670595753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ffc10 0x7f7a90103b70 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009880 tx=0x7f7a7802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.604+0000 7f7a96799640 1 -- 172.21.15.151:0/3670595753 shutdown_connections 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.604+0000 7f7a96799640 1 --2- 172.21.15.151:0/3670595753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ffc10 0x7f7a90103b70 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.604+0000 7f7a96799640 1 --2- 172.21.15.151:0/3670595753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a900ff260 0x7f7a900ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.604+0000 7f7a96799640 1 -- 172.21.15.151:0/3670595753 >> 172.21.15.151:0/3670595753 conn(0x7f7a90074a50 msgr2=0x7f7a90074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.604+0000 7f7a96799640 1 -- 172.21.15.151:0/3670595753 shutdown_connections 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.604+0000 7f7a96799640 1 -- 172.21.15.151:0/3670595753 wait complete. 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a96799640 1 Processor -- start 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a96799640 1 -- start start 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a96799640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ff260 0x7f7a90195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a96799640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a90196290 0x7f7a9019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:41.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a96799640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a90196800 con 0x7f7a900ff260 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a96799640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a90196970 con 0x7f7a90196290 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a8ffff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ff260 0x7f7a90195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a8ffff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ff260 0x7f7a90195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59306/0 (socket says 172.21.15.151:59306) 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a8ffff640 1 -- 172.21.15.151:0/2164580946 learned_addr learned my addr 172.21.15.151:0/2164580946 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a8ffff640 1 -- 172.21.15.151:0/2164580946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a90196290 msgr2=0x7f7a9019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a8ffff640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a90196290 0x7f7a9019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a8ffff640 1 -- 172.21.15.151:0/2164580946 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a78009560 con 0x7f7a900ff260 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a8ffff640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ff260 0x7f7a90195d50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000c8a0 tx=0x7f7a8000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.605+0000 7f7a8dffb640 1 -- 172.21.15.151:0/2164580946 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a80007b80 con 0x7f7a900ff260 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.606+0000 7f7a8dffb640 1 -- 172.21.15.151:0/2164580946 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a80007ce0 con 0x7f7a900ff260 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.606+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9019ac90 con 0x7f7a900ff260 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.606+0000 7f7a8dffb640 1 -- 172.21.15.151:0/2164580946 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a80018810 con 0x7f7a900ff260 2024-04-16T03:17:41.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.606+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9019b1a0 con 0x7f7a900ff260 2024-04-16T03:17:41.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.607+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9004e040 con 0x7f7a900ff260 2024-04-16T03:17:41.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.607+0000 7f7a8dffb640 1 -- 172.21.15.151:0/2164580946 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f7a80018a80 con 0x7f7a900ff260 2024-04-16T03:17:41.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.608+0000 7f7a8dffb640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a7c06ec30 0x7f7a7c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:41.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.608+0000 7f7a8dffb640 1 -- 172.21.15.151:0/2164580946 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7a8008f0b0 con 0x7f7a900ff260 2024-04-16T03:17:41.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.608+0000 7f7a87fff640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a7c06ec30 0x7f7a7c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:41.610 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.608+0000 7f7a87fff640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a7c06ec30 0x7f7a7c0710f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7a7802efd0 tx=0x7f7a78000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:41.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.612+0000 7f7a8dffb640 1 -- 172.21.15.151:0/2164580946 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7a80057e60 con 0x7f7a900ff260 2024-04-16T03:17:41.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.829+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 41, "format": "json"} v 0) v1 -- 0x7f7a9019b480 con 0x7f7a900ff260 2024-04-16T03:17:41.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.830+0000 7f7a8dffb640 1 -- 172.21.15.151:0/2164580946 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 41, "format": "json"}]=0 dumped fsmap epoch 41 v42) v1 ==== 107+0+5260 (secure 0 0 0) 0x7f7a8005b520 con 0x7f7a900ff260 2024-04-16T03:17:41.831 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:41.831 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-16T02:32:34.941393+0000","modified":"2024-04-16T03:15:36.680924+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34416,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34416":{"gid":34416,"name":"cephfs.smithi151.bdpnxq","rank":0,"incarnation":35,"state":"up:active","state_seq":469,"addr":"172.21.15.151:6828/2013103473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2013103473},{"type":"v1","addr":"172.21.15.151:6828","nonce":2013103473}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi151.slrqog","rank":1,"incarnation":35,"state":"up:active","state_seq":464,"addr":"172.21.15.151:6829/2926701426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":2926701426},{"type":"v1","addr":"172.21.15.151:6829","nonce":2926701426}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44311":{"gid":44311,"name":"cephfs.smithi153.ogdcjk","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6825/1413640526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1413640526},{"type":"v1","addr":"172.21.15.153:6825","nonce":1413640526}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44863":{"gid":44863,"name":"cephfs.smithi153.mqbgav","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6827/3861689149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3861689149},{"type":"v1","addr":"172.21.15.153:6827","nonce":3861689149}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34416,"qdb_cluster":[34420,34416]},"id":1}]} 2024-04-16T03:17:41.832 INFO:teuthology.orchestra.run.smithi151.stderr:dumped fsmap epoch 41 2024-04-16T03:17:41.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a7c06ec30 msgr2=0x7f7a7c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a7c06ec30 0x7f7a7c0710f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7a7802efd0 tx=0x7f7a78000bc0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ff260 msgr2=0x7f7a90195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:41.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ff260 0x7f7a90195d50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000c8a0 tx=0x7f7a8000cd60 comp rx=0 tx=0).stop 2024-04-16T03:17:41.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 shutdown_connections 2024-04-16T03:17:41.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f7a7c06ec30 0x7f7a7c0710f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a90196290 0x7f7a9019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 --2- 172.21.15.151:0/2164580946 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a900ff260 0x7f7a90195d50 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:41.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 >> 172.21.15.151:0/2164580946 conn(0x7f7a90074a50 msgr2=0x7f7a90103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:41.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 shutdown_connections 2024-04-16T03:17:41.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:41.832+0000 7f7a96799640 1 -- 172.21.15.151:0/2164580946 wait complete. 2024-04-16T03:17:41.845 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-16T03:17:41.858 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-16T03:17:41.858 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T03:17:41.859 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T03:17:41.876 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T03:17:41.876 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T03:17:41.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T03:17:42.083 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 -- 172.21.15.151:0/1299234612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f41002e0 msgr2=0x7f81f41006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 --2- 172.21.15.151:0/1299234612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f41002e0 0x7f81f41006c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009880 tx=0x7f81e402bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 -- 172.21.15.151:0/1299234612 shutdown_connections 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 --2- 172.21.15.151:0/1299234612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f4100c90 0x7f81f4104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 --2- 172.21.15.151:0/1299234612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f41002e0 0x7f81f41006c0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 -- 172.21.15.151:0/1299234612 >> 172.21.15.151:0/1299234612 conn(0x7f81f40fbb80 msgr2=0x7f81f40fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 -- 172.21.15.151:0/1299234612 shutdown_connections 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 -- 172.21.15.151:0/1299234612 wait complete. 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 Processor -- start 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.081+0000 7f81f9e5c640 1 -- start start 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f9e5c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f41002e0 0x7f81f4195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f9e5c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f4196210 0x7f81f41995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f9e5c640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f4196810 con 0x7f81f4196210 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f9e5c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f4196950 con 0x7f81f41002e0 2024-04-16T03:17:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f2ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f4196210 0x7f81f41995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f2ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f4196210 0x7f81f41995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59328/0 (socket says 172.21.15.151:59328) 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f2ffd640 1 -- 172.21.15.151:0/2353617042 learned_addr learned my addr 172.21.15.151:0/2353617042 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f37fe640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f41002e0 0x7f81f4195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f2ffd640 1 -- 172.21.15.151:0/2353617042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f41002e0 msgr2=0x7f81f4195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f2ffd640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f41002e0 0x7f81f4195cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f2ffd640 1 -- 172.21.15.151:0/2353617042 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e4009560 con 0x7f81f4196210 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f2ffd640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f4196210 0x7f81f41995c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f81e800e970 tx=0x7f81e800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f0ff9640 1 -- 172.21.15.151:0/2353617042 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e800cc90 con 0x7f81f4196210 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f4199bc0 con 0x7f81f4196210 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f0ff9640 1 -- 172.21.15.151:0/2353617042 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81e800cdf0 con 0x7f81f4196210 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f0ff9640 1 -- 172.21.15.151:0/2353617042 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e8010430 con 0x7f81f4196210 2024-04-16T03:17:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.082+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f419a120 con 0x7f81f4196210 2024-04-16T03:17:42.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.083+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f419a400 con 0x7f81f4196210 2024-04-16T03:17:42.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.084+0000 7f81f0ff9640 1 -- 172.21.15.151:0/2353617042 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f81e80046c0 con 0x7f81f4196210 2024-04-16T03:17:42.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.084+0000 7f81f0ff9640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f81dc06ecf0 0x7f81dc0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:42.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.084+0000 7f81f0ff9640 1 -- 172.21.15.151:0/2353617042 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f81e8014070 con 0x7f81f4196210 2024-04-16T03:17:42.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.084+0000 7f81f37fe640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f81dc06ecf0 0x7f81dc0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:42.086 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.085+0000 7f81f37fe640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f81dc06ecf0 0x7f81dc0711b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f81e4006010 tx=0x7f81e4000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:42.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.089+0000 7f81f0ff9640 1 -- 172.21.15.151:0/2353617042 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f81e8058240 con 0x7f81f4196210 2024-04-16T03:17:42.271 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.270+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81f4100c90 con 0x7f81f4196210 2024-04-16T03:17:42.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.272+0000 7f81f0ff9640 1 -- 172.21.15.151:0/2353617042 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16697 (secure 0 0 0) 0x7f81e805b900 con 0x7f81f4196210 2024-04-16T03:17:42.273 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:42.273 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":86,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","created":"2024-04-16T02:25:34.606206+0000","modified":"2024-04-16T03:00:19.177856+0000","last_up_change":"2024-04-16T02:44:24.227194+0000","last_in_change":"2024-04-16T02:30:58.536477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T02:30:03.137221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T02:32:33.681623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T02:32:34.178442+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8da4d50b-626b-4cf3-9524-ea84e7733113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6803","nonce":1997175184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6805","nonce":1997175184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6809","nonce":1997175184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6807","nonce":1997175184}]},"public_addr":"172.21.15.151:6803/1997175184","cluster_addr":"172.21.15.151:6805/1997175184","heartbeat_back_addr":"172.21.15.151:6809/1997175184","heartbeat_front_addr":"172.21.15.151:6807/1997175184","state":["exists","up"]},{"osd":1,"uuid":"8d2980b5-1d30-445c-91ae-f08074b6d8f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6811","nonce":4084991031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6813","nonce":4084991031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6817","nonce":4084991031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6815","nonce":4084991031}]},"public_addr":"172.21.15.151:6811/4084991031","cluster_addr":"172.21.15.151:6813/4084991031","heartbeat_back_addr":"172.21.15.151:6817/4084991031","heartbeat_front_addr":"172.21.15.151:6815/4084991031","state":["exists","up"]},{"osd":2,"uuid":"64e4e7ce-5c3c-48c8-be3d-c9936b6866b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6819","nonce":3234791793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6821","nonce":3234791793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6825","nonce":3234791793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6823","nonce":3234791793}]},"public_addr":"172.21.15.151:6819/3234791793","cluster_addr":"172.21.15.151:6821/3234791793","heartbeat_back_addr":"172.21.15.151:6825/3234791793","heartbeat_front_addr":"172.21.15.151:6823/3234791793","state":["exists","up"]},{"osd":3,"uuid":"1194f132-bf2d-4e0c-ab93-f3344ff64d76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6801","nonce":1577343223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6803","nonce":1577343223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6807","nonce":1577343223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6805","nonce":1577343223}]},"public_addr":"172.21.15.153:6801/1577343223","cluster_addr":"172.21.15.153:6803/1577343223","heartbeat_back_addr":"172.21.15.153:6807/1577343223","heartbeat_front_addr":"172.21.15.153:6805/1577343223","state":["exists","up"]},{"osd":4,"uuid":"9d803d8b-eb35-4a0f-9d36-de11b743a123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6809","nonce":290685586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6811","nonce":290685586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6815","nonce":290685586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6813","nonce":290685586}]},"public_addr":"172.21.15.153:6809/290685586","cluster_addr":"172.21.15.153:6811/290685586","heartbeat_back_addr":"172.21.15.153:6815/290685586","heartbeat_front_addr":"172.21.15.153:6813/290685586","state":["exists","up"]},{"osd":5,"uuid":"7d0d2460-ef76-481b-ad51-afb80d05bc72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6817","nonce":45452282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6819","nonce":45452282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6823","nonce":45452282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6821","nonce":45452282}]},"public_addr":"172.21.15.153:6817/45452282","cluster_addr":"172.21.15.153:6819/45452282","heartbeat_back_addr":"172.21.15.153:6823/45452282","heartbeat_front_addr":"172.21.15.153:6821/45452282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:02.729782+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:31.268279+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:59.891666+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:24.930390+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:50.555079+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:31:15.969056+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6827/1662886502":"2024-04-17T02:44:25.408644+0000","172.21.15.151:6826/1662886502":"2024-04-17T02:44:25.408644+0000","172.21.15.151:6829/37453862":"2024-04-17T02:44:25.408661+0000","172.21.15.151:0/2684608614":"2024-04-17T02:26:34.117371+0000","172.21.15.151:0/3091708247":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/681541897":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/969025109":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/671603278":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/3129110405":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3798542210":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/2753390843":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/2402350166":"2024-04-17T02:39:53.700570+0000","172.21.15.151:6800/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.153:0/986410000":"2024-04-17T02:36:50.207364+0000","172.21.15.151:0/4272586248":"2024-04-17T02:26:01.417600+0000","172.21.15.151:6801/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3454798699":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.153:0/3590885768":"2024-04-17T02:36:50.207364+0000","172.21.15.153:6829/2287826646":"2024-04-17T02:36:50.207364+0000","172.21.15.151:6828/37453862":"2024-04-17T02:44:25.408661+0000","172.21.15.151:0/2293925528":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/1505356478":"2024-04-17T02:39:53.700570+0000","172.21.15.151:0/1299604123":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/661820069":"2024-04-17T02:35:10.400124+0000","172.21.15.153:6824/1319024054":"2024-04-17T02:32:48.923574+0000","172.21.15.151:0/3077174249":"2024-04-17T02:39:53.700570+0000","172.21.15.153:6825/1319024054":"2024-04-17T02:32:48.923574+0000","172.21.15.151:6800/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.153:0/977218519":"2024-04-17T02:36:50.207364+0000","172.21.15.151:6800/2753390843":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/1923729027":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6801/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.153:6828/2287826646":"2024-04-17T02:36:50.207364+0000","172.21.15.153:0/3588036861":"2024-04-17T02:36:50.207364+0000","172.21.15.151:6800/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:6800/1243156277":"2024-04-17T02:39:53.700570+0000","172.21.15.151:6801/1243156277":"2024-04-17T02:39:53.700570+0000","172.21.15.151:0/1770724784":"2024-04-17T02:39:53.700570+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f81dc06ecf0 msgr2=0x7f81dc0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f81dc06ecf0 0x7f81dc0711b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f81e4006010 tx=0x7f81e4000bc0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f4196210 msgr2=0x7f81f41995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f4196210 0x7f81f41995c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f81e800e970 tx=0x7f81e800ee30 comp rx=0 tx=0).stop 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 shutdown_connections 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f81dc06ecf0 0x7f81dc0711b0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f81f4196210 0x7f81f41995c0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 --2- 172.21.15.151:0/2353617042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f41002e0 0x7f81f4195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 >> 172.21.15.151:0/2353617042 conn(0x7f81f40fbb80 msgr2=0x7f81f40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 shutdown_connections 2024-04-16T03:17:42.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.274+0000 7f81f9e5c640 1 -- 172.21.15.151:0/2353617042 wait complete. 2024-04-16T03:17:42.286 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T02:30:03.137221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T02:32:33.681623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T02:32:34.178442+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T03:17:42.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T03:17:42.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.393+0000 7f506549d640 1 -- 172.21.15.151:0/2150440278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 msgr2=0x7f506010a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.393+0000 7f506549d640 1 --2- 172.21.15.151:0/2150440278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 0x7f506010a020 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5054009880 tx=0x7f505402bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.393+0000 7f506549d640 1 -- 172.21.15.151:0/2150440278 shutdown_connections 2024-04-16T03:17:42.396 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.393+0000 7f506549d640 1 --2- 172.21.15.151:0/2150440278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 0x7f506010a020 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.393+0000 7f506549d640 1 --2- 172.21.15.151:0/2150440278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5060101220 0x7f5060101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.393+0000 7f506549d640 1 -- 172.21.15.151:0/2150440278 >> 172.21.15.151:0/2150440278 conn(0x7f50600faa20 msgr2=0x7f50600fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.393+0000 7f506549d640 1 -- 172.21.15.151:0/2150440278 shutdown_connections 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.393+0000 7f506549d640 1 -- 172.21.15.151:0/2150440278 wait complete. 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f506549d640 1 Processor -- start 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f506549d640 1 -- start start 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f506549d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5060101220 0x7f50600fee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f506549d640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 0x7f50600ff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f506549d640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50600ff8f0 con 0x7f5060101b40 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f506549d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50600ffa60 con 0x7f5060101220 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f505e7fc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 0x7f50600ff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f505e7fc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 0x7f50600ff3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59348/0 (socket says 172.21.15.151:59348) 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f505e7fc640 1 -- 172.21.15.151:0/1782689685 learned_addr learned my addr 172.21.15.151:0/1782689685 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f505effd640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5060101220 0x7f50600fee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:42.397 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f505e7fc640 1 -- 172.21.15.151:0/1782689685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5060101220 msgr2=0x7f50600fee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f505e7fc640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5060101220 0x7f50600fee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f505e7fc640 1 -- 172.21.15.151:0/1782689685 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5054009560 con 0x7f5060101b40 2024-04-16T03:17:42.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.394+0000 7f505effd640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5060101220 0x7f50600fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:42.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.395+0000 7f505e7fc640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 0x7f50600ff3b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5054000c00 tx=0x7f505402c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:42.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.395+0000 7f5047fff640 1 -- 172.21.15.151:0/1782689685 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5054004420 con 0x7f5060101b40 2024-04-16T03:17:42.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.395+0000 7f5047fff640 1 -- 172.21.15.151:0/1782689685 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5054003ab0 con 0x7f5060101b40 2024-04-16T03:17:42.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.395+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50600ffc80 con 0x7f5060101b40 2024-04-16T03:17:42.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.395+0000 7f5047fff640 1 -- 172.21.15.151:0/1782689685 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5054005590 con 0x7f5060101b40 2024-04-16T03:17:42.398 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.395+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50601a23d0 con 0x7f5060101b40 2024-04-16T03:17:42.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.396+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506004e040 con 0x7f5060101b40 2024-04-16T03:17:42.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.396+0000 7f5047fff640 1 -- 172.21.15.151:0/1782689685 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5054004dd0 con 0x7f5060101b40 2024-04-16T03:17:42.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.397+0000 7f5047fff640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f504806ec30 0x7f50480710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:42.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.397+0000 7f5047fff640 1 -- 172.21.15.151:0/1782689685 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f50540b18c0 con 0x7f5060101b40 2024-04-16T03:17:42.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.397+0000 7f505effd640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f504806ec30 0x7f50480710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:42.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.397+0000 7f505effd640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f504806ec30 0x7f50480710f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5050009c20 tx=0x7f5050005f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:42.402 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.401+0000 7f5047fff640 1 -- 172.21.15.151:0/1782689685 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f505407a670 con 0x7f5060101b40 2024-04-16T03:17:42.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:42 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1607212326' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-04-16T03:17:42.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:42 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2164580946' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-04-16T03:17:42.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:42 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1607212326' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-04-16T03:17:42.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:42 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2164580946' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-04-16T03:17:42.586 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.585+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f506010d490 con 0x7f5060101b40 2024-04-16T03:17:42.587 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.586+0000 7f5047fff640 1 -- 172.21.15.151:0/1782689685 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f505407dd30 con 0x7f5060101b40 2024-04-16T03:17:42.587 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 1 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f504806ec30 msgr2=0x7f50480710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f504806ec30 0x7f50480710f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5050009c20 tx=0x7f5050005f90 comp rx=0 tx=0).stop 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 msgr2=0x7f50600ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 0x7f50600ff3b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5054000c00 tx=0x7f505402c6f0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 shutdown_connections 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f504806ec30 0x7f50480710f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5060101b40 0x7f50600ff3b0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 --2- 172.21.15.151:0/1782689685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5060101220 0x7f50600fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 >> 172.21.15.151:0/1782689685 conn(0x7f50600faa20 msgr2=0x7f50601047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:42.589 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 shutdown_connections 2024-04-16T03:17:42.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.588+0000 7f506549d640 1 -- 172.21.15.151:0/1782689685 wait complete. 2024-04-16T03:17:42.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T03:17:42.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.706+0000 7f514a0ec640 1 -- 172.21.15.151:0/3599210787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ffb50 msgr2=0x7f5144103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.706+0000 7f514a0ec640 1 --2- 172.21.15.151:0/3599210787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ffb50 0x7f5144103b90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5138009880 tx=0x7f513802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.707+0000 7f514a0ec640 1 -- 172.21.15.151:0/3599210787 shutdown_connections 2024-04-16T03:17:42.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.707+0000 7f514a0ec640 1 --2- 172.21.15.151:0/3599210787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ffb50 0x7f5144103b90 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.707+0000 7f514a0ec640 1 --2- 172.21.15.151:0/3599210787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51440ff1a0 0x7f51440ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.707+0000 7f514a0ec640 1 -- 172.21.15.151:0/3599210787 >> 172.21.15.151:0/3599210787 conn(0x7f51440faa20 msgr2=0x7f51440fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:42.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.707+0000 7f514a0ec640 1 -- 172.21.15.151:0/3599210787 shutdown_connections 2024-04-16T03:17:42.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.707+0000 7f514a0ec640 1 -- 172.21.15.151:0/3599210787 wait complete. 2024-04-16T03:17:42.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.707+0000 7f514a0ec640 1 Processor -- start 2024-04-16T03:17:42.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.707+0000 7f514a0ec640 1 -- start start 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f514a0ec640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ff1a0 0x7f5144195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f514a0ec640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5144196290 0x7f514419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f514a0ec640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5144196890 con 0x7f51440ff1a0 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f514a0ec640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5144196a00 con 0x7f5144196290 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f51437fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ff1a0 0x7f5144195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f51437fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ff1a0 0x7f5144195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59362/0 (socket says 172.21.15.151:59362) 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f51437fe640 1 -- 172.21.15.151:0/3871743128 learned_addr learned my addr 172.21.15.151:0/3871743128 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f51437fe640 1 -- 172.21.15.151:0/3871743128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5144196290 msgr2=0x7f514419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f51437fe640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5144196290 0x7f514419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f51437fe640 1 -- 172.21.15.151:0/3871743128 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5138009560 con 0x7f51440ff1a0 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f51437fe640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ff1a0 0x7f5144195d50 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f513400ea30 tx=0x7f513400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f5140ff9640 1 -- 172.21.15.151:0/3871743128 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f513400cdb0 con 0x7f51440ff1a0 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f5140ff9640 1 -- 172.21.15.151:0/3871743128 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5134005020 con 0x7f51440ff1a0 2024-04-16T03:17:42.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f5140ff9640 1 -- 172.21.15.151:0/3871743128 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51340116a0 con 0x7f51440ff1a0 2024-04-16T03:17:42.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514419acf0 con 0x7f51440ff1a0 2024-04-16T03:17:42.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.708+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514419b200 con 0x7f51440ff1a0 2024-04-16T03:17:42.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.709+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514419b530 con 0x7f51440ff1a0 2024-04-16T03:17:42.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.710+0000 7f5140ff9640 1 -- 172.21.15.151:0/3871743128 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f5134005190 con 0x7f51440ff1a0 2024-04-16T03:17:42.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.710+0000 7f5140ff9640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f512c06ec30 0x7f512c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:42.712 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.710+0000 7f5142ffd640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f512c06ec30 0x7f512c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:42.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.710+0000 7f5140ff9640 1 -- 172.21.15.151:0/3871743128 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f513408f2f0 con 0x7f51440ff1a0 2024-04-16T03:17:42.713 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.711+0000 7f5142ffd640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f512c06ec30 0x7f512c0710f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5138036040 tx=0x7f5138000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:42.716 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.715+0000 7f5140ff9640 1 -- 172.21.15.151:0/3871743128 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f51340580a0 con 0x7f51440ff1a0 2024-04-16T03:17:42.895 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.894+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f51441970a0 con 0x7f51440ff1a0 2024-04-16T03:17:42.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.895+0000 7f5140ff9640 1 -- 172.21.15.151:0/3871743128 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f513405b760 con 0x7f51440ff1a0 2024-04-16T03:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 32 2024-04-16T03:17:42.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f512c06ec30 msgr2=0x7f512c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f512c06ec30 0x7f512c0710f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5138036040 tx=0x7f5138000bc0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.898 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ff1a0 msgr2=0x7f5144195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:42.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ff1a0 0x7f5144195d50 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f513400ea30 tx=0x7f513400eef0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 shutdown_connections 2024-04-16T03:17:42.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f512c06ec30 0x7f512c0710f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5144196290 0x7f514419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 --2- 172.21.15.151:0/3871743128 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51440ff1a0 0x7f5144195d50 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:42.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 >> 172.21.15.151:0/3871743128 conn(0x7f51440faa20 msgr2=0x7f5144103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:42.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 shutdown_connections 2024-04-16T03:17:42.899 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:42.897+0000 7f514a0ec640 1 -- 172.21.15.151:0/3871743128 wait complete. 2024-04-16T03:17:42.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T03:17:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 -- 172.21.15.151:0/1530654151 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 msgr2=0x7fa9480ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 --2- 172.21.15.151:0/1530654151 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 0x7fa9480ff640 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa944009880 tx=0x7fa94402bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 -- 172.21.15.151:0/1530654151 shutdown_connections 2024-04-16T03:17:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 --2- 172.21.15.151:0/1530654151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9480ffc10 0x7fa948103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 --2- 172.21.15.151:0/1530654151 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 0x7fa9480ff640 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 -- 172.21.15.151:0/1530654151 >> 172.21.15.151:0/1530654151 conn(0x7fa948074a50 msgr2=0x7fa948074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 -- 172.21.15.151:0/1530654151 shutdown_connections 2024-04-16T03:17:43.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 -- 172.21.15.151:0/1530654151 wait complete. 2024-04-16T03:17:43.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 Processor -- start 2024-04-16T03:17:43.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.016+0000 7fa95079c640 1 -- start start 2024-04-16T03:17:43.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa95079c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 0x7fa948196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:43.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa95079c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa948197130 0x7fa948197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:43.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa95079c640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94819fc60 con 0x7fa9480ff260 2024-04-16T03:17:43.019 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa95079c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94819fda0 con 0x7fa948197130 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa94e538640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 0x7fa948196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa94e538640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 0x7fa948196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59384/0 (socket says 172.21.15.151:59384) 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa94e538640 1 -- 172.21.15.151:0/3058681152 learned_addr learned my addr 172.21.15.151:0/3058681152 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa94e538640 1 -- 172.21.15.151:0/3058681152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa948197130 msgr2=0x7fa948197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa94e538640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa948197130 0x7fa948197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa94e538640 1 -- 172.21.15.151:0/3058681152 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa944009560 con 0x7fa9480ff260 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa94e538640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 0x7fa948196bf0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa944009430 tx=0x7fa944004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa93f7fe640 1 -- 172.21.15.151:0/3058681152 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9440035f0 con 0x7fa9480ff260 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa93f7fe640 1 -- 172.21.15.151:0/3058681152 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa944003c10 con 0x7fa9480ff260 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94819ffa0 con 0x7fa9480ff260 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa93f7fe640 1 -- 172.21.15.151:0/3058681152 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa944005670 con 0x7fa9480ff260 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.017+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9481a0420 con 0x7fa9480ff260 2024-04-16T03:17:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.018+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94804e040 con 0x7fa9480ff260 2024-04-16T03:17:43.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.019+0000 7fa93f7fe640 1 -- 172.21.15.151:0/3058681152 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fa944004dd0 con 0x7fa9480ff260 2024-04-16T03:17:43.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.019+0000 7fa93f7fe640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa93406ec30 0x7fa9340710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:43.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.019+0000 7fa93f7fe640 1 -- 172.21.15.151:0/3058681152 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa9440b1180 con 0x7fa9480ff260 2024-04-16T03:17:43.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.019+0000 7fa94dd37640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa93406ec30 0x7fa9340710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:43.021 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.020+0000 7fa94dd37640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa93406ec30 0x7fa9340710f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa938005fd0 tx=0x7fa938005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:43.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.024+0000 7fa93f7fe640 1 -- 172.21.15.151:0/3058681152 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa944079fb0 con 0x7fa9480ff260 2024-04-16T03:17:43.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.203+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fa948107480 con 0x7fa9480ff260 2024-04-16T03:17:43.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.203+0000 7fa93f7fe640 1 -- 172.21.15.151:0/3058681152 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa94407d670 con 0x7fa9480ff260 2024-04-16T03:17:43.204 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 32 2024-04-16T03:17:43.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa93406ec30 msgr2=0x7fa9340710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa93406ec30 0x7fa9340710f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa938005fd0 tx=0x7fa938005850 comp rx=0 tx=0).stop 2024-04-16T03:17:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 msgr2=0x7fa948196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 0x7fa948196bf0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa944009430 tx=0x7fa944004040 comp rx=0 tx=0).stop 2024-04-16T03:17:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 shutdown_connections 2024-04-16T03:17:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fa93406ec30 0x7fa9340710f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa948197130 0x7fa948197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 --2- 172.21.15.151:0/3058681152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9480ff260 0x7fa948196bf0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 >> 172.21.15.151:0/3058681152 conn(0x7fa948074a50 msgr2=0x7fa9480fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 shutdown_connections 2024-04-16T03:17:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.205+0000 7fa95079c640 1 -- 172.21.15.151:0/3058681152 wait complete. 2024-04-16T03:17:43.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T03:17:43.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.322+0000 7f2e79209640 1 -- 172.21.15.151:0/3007011350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e740fefe0 msgr2=0x7f2e740ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.322+0000 7f2e79209640 1 --2- 172.21.15.151:0/3007011350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e740fefe0 0x7f2e740ff440 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009880 tx=0x7f2e6802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.322+0000 7f2e79209640 1 -- 172.21.15.151:0/3007011350 shutdown_connections 2024-04-16T03:17:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.322+0000 7f2e79209640 1 --2- 172.21.15.151:0/3007011350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e740fefe0 0x7f2e740ff440 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.322+0000 7f2e79209640 1 --2- 172.21.15.151:0/3007011350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e74103f20 0x7f2e74104300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.322+0000 7f2e79209640 1 -- 172.21.15.151:0/3007011350 >> 172.21.15.151:0/3007011350 conn(0x7f2e74075b40 msgr2=0x7f2e74075f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.322+0000 7f2e79209640 1 -- 172.21.15.151:0/3007011350 shutdown_connections 2024-04-16T03:17:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.322+0000 7f2e79209640 1 -- 172.21.15.151:0/3007011350 wait complete. 2024-04-16T03:17:43.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.323+0000 7f2e79209640 1 Processor -- start 2024-04-16T03:17:43.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.323+0000 7f2e79209640 1 -- start start 2024-04-16T03:17:43.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.323+0000 7f2e79209640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e740fefe0 0x7f2e74194920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:43.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.323+0000 7f2e79209640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e74103f20 0x7f2e74194e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:43.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.323+0000 7f2e79209640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e741953a0 con 0x7f2e74103f20 2024-04-16T03:17:43.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.323+0000 7f2e79209640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74195510 con 0x7f2e740fefe0 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e7259c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e74103f20 0x7f2e74194e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e7259c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e74103f20 0x7f2e74194e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59410/0 (socket says 172.21.15.151:59410) 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e7259c640 1 -- 172.21.15.151:0/4114641489 learned_addr learned my addr 172.21.15.151:0/4114641489 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e72d9d640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e740fefe0 0x7f2e74194920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e7259c640 1 -- 172.21.15.151:0/4114641489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e740fefe0 msgr2=0x7f2e74194920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e7259c640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e740fefe0 0x7f2e74194920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e7259c640 1 -- 172.21.15.151:0/4114641489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68009560 con 0x7f2e74103f20 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e72d9d640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e740fefe0 0x7f2e74194920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e7259c640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e74103f20 0x7f2e74194e60 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009230 tx=0x7f2e68004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e5bfff640 1 -- 172.21.15.151:0/4114641489 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e680035f0 con 0x7f2e74103f20 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e74195790 con 0x7f2e74103f20 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e5bfff640 1 -- 172.21.15.151:0/4114641489 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e68003c10 con 0x7f2e74103f20 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e5bfff640 1 -- 172.21.15.151:0/4114641489 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e68005670 con 0x7f2e74103f20 2024-04-16T03:17:43.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.324+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e741a21d0 con 0x7f2e74103f20 2024-04-16T03:17:43.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.325+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7404e040 con 0x7f2e74103f20 2024-04-16T03:17:43.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.326+0000 7f2e5bfff640 1 -- 172.21.15.151:0/4114641489 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f2e68004dd0 con 0x7f2e74103f20 2024-04-16T03:17:43.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.326+0000 7f2e5bfff640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e5c06ec90 0x7f2e5c071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:43.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.326+0000 7f2e5bfff640 1 -- 172.21.15.151:0/4114641489 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2e680b1070 con 0x7f2e74103f20 2024-04-16T03:17:43.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.326+0000 7f2e72d9d640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e5c06ec90 0x7f2e5c071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:43.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.327+0000 7f2e72d9d640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e5c06ec90 0x7f2e5c071150 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2e640058c0 tx=0x7f2e64005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:43.332 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.331+0000 7f2e5bfff640 1 -- 172.21.15.151:0/4114641489 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2e68079e20 con 0x7f2e74103f20 2024-04-16T03:17:43.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:43 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/2353617042' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T03:17:43.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:43 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1782689685' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T03:17:43.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:43 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3871743128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T03:17:43.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:43 smithi151 ceph-mon[112233]: pgmap v1176: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:43.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:43 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/2353617042' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T03:17:43.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:43 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1782689685' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T03:17:43.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:43 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3871743128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T03:17:43.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:43 smithi153 ceph-mon[87858]: pgmap v1176: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:43.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.522+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2e74195790 con 0x7f2e74103f20 2024-04-16T03:17:43.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.523+0000 7f2e5bfff640 1 -- 172.21.15.151:0/4114641489 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v86) v1 ==== 81+0+2341 (secure 0 0 0) 0x7f2e74195790 con 0x7f2e74103f20 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6827/1662886502 2024-04-17T02:44:25.408644+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6826/1662886502 2024-04-17T02:44:25.408644+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6829/37453862 2024-04-17T02:44:25.408661+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/2684608614 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3091708247 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/681541897 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/969025109 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/671603278 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3129110405 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3798542210 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:43.524 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/2753390843 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/2402350166 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/3544869641 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:0/986410000 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/4272586248 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/3544869641 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3454798699 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/2400549256 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:0/3590885768 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:6829/2287826646 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6828/37453862 2024-04-17T02:44:25.408661+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/2293925528 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/1505356478 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/1299604123 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/661820069 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:6824/1319024054 2024-04-17T02:32:48.923574+0000 2024-04-16T03:17:43.525 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3077174249 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:6825/1319024054 2024-04-17T02:32:48.923574+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/2400549256 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:0/977218519 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/2753390843 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/1923729027 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/39987226 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:6828/2287826646 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:0/3588036861 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/39987226 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/1243156277 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/1243156277 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:43.526 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/1770724784 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:43.527 INFO:teuthology.orchestra.run.smithi151.stderr:listed 39 entries 2024-04-16T03:17:43.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e5c06ec90 msgr2=0x7f2e5c071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e5c06ec90 0x7f2e5c071150 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2e640058c0 tx=0x7f2e64005850 comp rx=0 tx=0).stop 2024-04-16T03:17:43.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e74103f20 msgr2=0x7f2e74194e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e74103f20 0x7f2e74194e60 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009230 tx=0x7f2e68004040 comp rx=0 tx=0).stop 2024-04-16T03:17:43.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 shutdown_connections 2024-04-16T03:17:43.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f2e5c06ec90 0x7f2e5c071150 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e74103f20 0x7f2e74194e60 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 --2- 172.21.15.151:0/4114641489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e740fefe0 0x7f2e74194920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 >> 172.21.15.151:0/4114641489 conn(0x7f2e74075b40 msgr2=0x7f2e740fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:43.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 shutdown_connections 2024-04-16T03:17:43.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.525+0000 7f2e79209640 1 -- 172.21.15.151:0/4114641489 wait complete. 2024-04-16T03:17:43.537 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T03:17:43.537 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T03:17:43.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T03:17:43.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.698+0000 7f6de2c87640 1 -- 172.21.15.151:0/2226368495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc105180 msgr2=0x7f6ddc105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.698+0000 7f6de2c87640 1 --2- 172.21.15.151:0/2226368495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc105180 0x7f6ddc105560 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009880 tx=0x7f6dd002bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.698+0000 7f6de2c87640 1 -- 172.21.15.151:0/2226368495 shutdown_connections 2024-04-16T03:17:43.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.698+0000 7f6de2c87640 1 --2- 172.21.15.151:0/2226368495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc0ff830 0x7f6ddc0ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.698+0000 7f6de2c87640 1 --2- 172.21.15.151:0/2226368495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc105180 0x7f6ddc105560 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.698+0000 7f6de2c87640 1 -- 172.21.15.151:0/2226368495 >> 172.21.15.151:0/2226368495 conn(0x7f6ddc0752f0 msgr2=0x7f6ddc075700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:43.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.698+0000 7f6de2c87640 1 -- 172.21.15.151:0/2226368495 shutdown_connections 2024-04-16T03:17:43.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.698+0000 7f6de2c87640 1 -- 172.21.15.151:0/2226368495 wait complete. 2024-04-16T03:17:43.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de2c87640 1 Processor -- start 2024-04-16T03:17:43.700 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de2c87640 1 -- start start 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de2c87640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc0ff830 0x7f6ddc195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de2c87640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc105180 0x7f6ddc196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de2c87640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc1968e0 con 0x7f6ddc0ff830 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de2c87640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc1995b0 con 0x7f6ddc105180 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de0a23640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc0ff830 0x7f6ddc195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de0a23640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc0ff830 0x7f6ddc195cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59422/0 (socket says 172.21.15.151:59422) 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de0a23640 1 -- 172.21.15.151:0/1746480205 learned_addr learned my addr 172.21.15.151:0/1746480205 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6ddbfff640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc105180 0x7f6ddc196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de0a23640 1 -- 172.21.15.151:0/1746480205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc105180 msgr2=0x7f6ddc196200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de0a23640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc105180 0x7f6ddc196200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.699+0000 7f6de0a23640 1 -- 172.21.15.151:0/1746480205 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd0009560 con 0x7f6ddc0ff830 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.700+0000 7f6de0a23640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc0ff830 0x7f6ddc195cc0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009230 tx=0x7f6dd0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:43.702 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.700+0000 7f6dd9ffb640 1 -- 172.21.15.151:0/1746480205 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd00035f0 con 0x7f6ddc0ff830 2024-04-16T03:17:43.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.700+0000 7f6dd9ffb640 1 -- 172.21.15.151:0/1746480205 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dd0003c10 con 0x7f6ddc0ff830 2024-04-16T03:17:43.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.700+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc199830 con 0x7f6ddc0ff830 2024-04-16T03:17:43.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.700+0000 7f6dd9ffb640 1 -- 172.21.15.151:0/1746480205 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd0005670 con 0x7f6ddc0ff830 2024-04-16T03:17:43.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.700+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc199d30 con 0x7f6ddc0ff830 2024-04-16T03:17:43.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.701+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ddc04e040 con 0x7f6ddc0ff830 2024-04-16T03:17:43.703 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.701+0000 7f6dd9ffb640 1 -- 172.21.15.151:0/1746480205 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f6dd0004dd0 con 0x7f6ddc0ff830 2024-04-16T03:17:43.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.702+0000 7f6dd9ffb640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ec90 0x7f6dcc071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:43.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.702+0000 7f6dd9ffb640 1 -- 172.21.15.151:0/1746480205 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6dd00b1270 con 0x7f6ddc0ff830 2024-04-16T03:17:43.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.702+0000 7f6ddbfff640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ec90 0x7f6dcc071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:43.704 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.702+0000 7f6ddbfff640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ec90 0x7f6dcc071150 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8005fd0 tx=0x7f6dc8005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:43.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.707+0000 7f6dd9ffb640 1 -- 172.21.15.151:0/1746480205 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6dd007a020 con 0x7f6ddc0ff830 2024-04-16T03:17:43.888 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.887+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ddc10c3b0 con 0x7f6ddc0ff830 2024-04-16T03:17:43.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.889+0000 7f6dd9ffb640 1 -- 172.21.15.151:0/1746480205 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16697 (secure 0 0 0) 0x7f6dd007d6e0 con 0x7f6ddc0ff830 2024-04-16T03:17:43.890 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T03:17:43.890 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":86,"fsid":"80d1e958-fb98-11ee-bc8f-c7b262605968","created":"2024-04-16T02:25:34.606206+0000","modified":"2024-04-16T03:00:19.177856+0000","last_up_change":"2024-04-16T02:44:24.227194+0000","last_in_change":"2024-04-16T02:30:58.536477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T02:30:03.137221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T02:32:33.681623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T02:32:34.178442+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8da4d50b-626b-4cf3-9524-ea84e7733113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6803","nonce":1997175184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6805","nonce":1997175184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6809","nonce":1997175184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1997175184},{"type":"v1","addr":"172.21.15.151:6807","nonce":1997175184}]},"public_addr":"172.21.15.151:6803/1997175184","cluster_addr":"172.21.15.151:6805/1997175184","heartbeat_back_addr":"172.21.15.151:6809/1997175184","heartbeat_front_addr":"172.21.15.151:6807/1997175184","state":["exists","up"]},{"osd":1,"uuid":"8d2980b5-1d30-445c-91ae-f08074b6d8f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6811","nonce":4084991031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6813","nonce":4084991031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6817","nonce":4084991031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":4084991031},{"type":"v1","addr":"172.21.15.151:6815","nonce":4084991031}]},"public_addr":"172.21.15.151:6811/4084991031","cluster_addr":"172.21.15.151:6813/4084991031","heartbeat_back_addr":"172.21.15.151:6817/4084991031","heartbeat_front_addr":"172.21.15.151:6815/4084991031","state":["exists","up"]},{"osd":2,"uuid":"64e4e7ce-5c3c-48c8-be3d-c9936b6866b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6819","nonce":3234791793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6821","nonce":3234791793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6825","nonce":3234791793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3234791793},{"type":"v1","addr":"172.21.15.151:6823","nonce":3234791793}]},"public_addr":"172.21.15.151:6819/3234791793","cluster_addr":"172.21.15.151:6821/3234791793","heartbeat_back_addr":"172.21.15.151:6825/3234791793","heartbeat_front_addr":"172.21.15.151:6823/3234791793","state":["exists","up"]},{"osd":3,"uuid":"1194f132-bf2d-4e0c-ab93-f3344ff64d76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6801","nonce":1577343223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6803","nonce":1577343223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6807","nonce":1577343223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1577343223},{"type":"v1","addr":"172.21.15.153:6805","nonce":1577343223}]},"public_addr":"172.21.15.153:6801/1577343223","cluster_addr":"172.21.15.153:6803/1577343223","heartbeat_back_addr":"172.21.15.153:6807/1577343223","heartbeat_front_addr":"172.21.15.153:6805/1577343223","state":["exists","up"]},{"osd":4,"uuid":"9d803d8b-eb35-4a0f-9d36-de11b743a123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6809","nonce":290685586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6811","nonce":290685586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6815","nonce":290685586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":290685586},{"type":"v1","addr":"172.21.15.153:6813","nonce":290685586}]},"public_addr":"172.21.15.153:6809/290685586","cluster_addr":"172.21.15.153:6811/290685586","heartbeat_back_addr":"172.21.15.153:6815/290685586","heartbeat_front_addr":"172.21.15.153:6813/290685586","state":["exists","up"]},{"osd":5,"uuid":"7d0d2460-ef76-481b-ad51-afb80d05bc72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6817","nonce":45452282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6819","nonce":45452282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6823","nonce":45452282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":45452282},{"type":"v1","addr":"172.21.15.153:6821","nonce":45452282}]},"public_addr":"172.21.15.153:6817/45452282","cluster_addr":"172.21.15.153:6819/45452282","heartbeat_back_addr":"172.21.15.153:6823/45452282","heartbeat_front_addr":"172.21.15.153:6821/45452282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:02.729782+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:31.268279+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:29:59.891666+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:24.930390+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:30:50.555079+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T02:31:15.969056+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6827/1662886502":"2024-04-17T02:44:25.408644+0000","172.21.15.151:6826/1662886502":"2024-04-17T02:44:25.408644+0000","172.21.15.151:6829/37453862":"2024-04-17T02:44:25.408661+0000","172.21.15.151:0/2684608614":"2024-04-17T02:26:34.117371+0000","172.21.15.151:0/3091708247":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/681541897":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/969025109":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/671603278":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/3129110405":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3798542210":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/2753390843":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/2402350166":"2024-04-17T02:39:53.700570+0000","172.21.15.151:6800/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.153:0/986410000":"2024-04-17T02:36:50.207364+0000","172.21.15.151:0/4272586248":"2024-04-17T02:26:01.417600+0000","172.21.15.151:6801/3544869641":"2024-04-17T02:27:42.821901+0000","172.21.15.151:0/3454798699":"2024-04-17T02:26:34.117371+0000","172.21.15.151:6801/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.153:0/3590885768":"2024-04-17T02:36:50.207364+0000","172.21.15.153:6829/2287826646":"2024-04-17T02:36:50.207364+0000","172.21.15.151:6828/37453862":"2024-04-17T02:44:25.408661+0000","172.21.15.151:0/2293925528":"2024-04-17T02:26:01.417600+0000","172.21.15.151:0/1505356478":"2024-04-17T02:39:53.700570+0000","172.21.15.151:0/1299604123":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/661820069":"2024-04-17T02:35:10.400124+0000","172.21.15.153:6824/1319024054":"2024-04-17T02:32:48.923574+0000","172.21.15.151:0/3077174249":"2024-04-17T02:39:53.700570+0000","172.21.15.153:6825/1319024054":"2024-04-17T02:32:48.923574+0000","172.21.15.151:6800/2400549256":"2024-04-17T02:26:34.117371+0000","172.21.15.153:0/977218519":"2024-04-17T02:36:50.207364+0000","172.21.15.151:6800/2753390843":"2024-04-17T02:35:10.400124+0000","172.21.15.151:0/1923729027":"2024-04-17T02:27:42.821901+0000","172.21.15.151:6801/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.153:6828/2287826646":"2024-04-17T02:36:50.207364+0000","172.21.15.153:0/3588036861":"2024-04-17T02:36:50.207364+0000","172.21.15.151:6800/39987226":"2024-04-17T02:26:01.417600+0000","172.21.15.151:6800/1243156277":"2024-04-17T02:39:53.700570+0000","172.21.15.151:6801/1243156277":"2024-04-17T02:39:53.700570+0000","172.21.15.151:0/1770724784":"2024-04-17T02:39:53.700570+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.890+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ec90 msgr2=0x7f6dcc071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.890+0000 7f6de2c87640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ec90 0x7f6dcc071150 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8005fd0 tx=0x7f6dc8005850 comp rx=0 tx=0).stop 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.890+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc0ff830 msgr2=0x7f6ddc195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.891+0000 7f6de2c87640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc0ff830 0x7f6ddc195cc0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009230 tx=0x7f6dd0004040 comp rx=0 tx=0).stop 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.891+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 shutdown_connections 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.891+0000 7f6de2c87640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f6dcc06ec90 0x7f6dcc071150 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.891+0000 7f6de2c87640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ddc105180 0x7f6ddc196200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.891+0000 7f6de2c87640 1 --2- 172.21.15.151:0/1746480205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ddc0ff830 0x7f6ddc195cc0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.891+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 >> 172.21.15.151:0/1746480205 conn(0x7f6ddc0752f0 msgr2=0x7f6ddc108290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:43.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.891+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 shutdown_connections 2024-04-16T03:17:43.893 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:43.891+0000 7f6de2c87640 1 -- 172.21.15.151:0/1746480205 wait complete. 2024-04-16T03:17:43.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T02:30:03.137221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T02:32:33.681623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T02:32:34.178442+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T03:17:43.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.010+0000 7f8756641640 1 -- 172.21.15.151:0/11481340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 msgr2=0x7f875010a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.010+0000 7f8756641640 1 --2- 172.21.15.151:0/11481340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 0x7f875010a0d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f8738009880 tx=0x7f873802bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.010+0000 7f8756641640 1 -- 172.21.15.151:0/11481340 shutdown_connections 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.010+0000 7f8756641640 1 --2- 172.21.15.151:0/11481340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 0x7f875010a0d0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.010+0000 7f8756641640 1 --2- 172.21.15.151:0/11481340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87501012d0 0x7f87501016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.010+0000 7f8756641640 1 -- 172.21.15.151:0/11481340 >> 172.21.15.151:0/11481340 conn(0x7f87500faa30 msgr2=0x7f87500fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.010+0000 7f8756641640 1 -- 172.21.15.151:0/11481340 shutdown_connections 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.010+0000 7f8756641640 1 -- 172.21.15.151:0/11481340 wait complete. 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.011+0000 7f8756641640 1 Processor -- start 2024-04-16T03:17:44.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.011+0000 7f8756641640 1 -- start start 2024-04-16T03:17:44.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.011+0000 7f8756641640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87501012d0 0x7f8750196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.011+0000 7f8756641640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 0x7f87501965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.011+0000 7f8756641640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8750196c40 con 0x7f8750101bf0 2024-04-16T03:17:44.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.011+0000 7f8756641640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875019aa10 con 0x7f87501012d0 2024-04-16T03:17:44.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.011+0000 7f8747fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 0x7f87501965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.011+0000 7f8747fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 0x7f87501965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59428/0 (socket says 172.21.15.151:59428) 2024-04-16T03:17:44.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.011+0000 7f8747fff640 1 -- 172.21.15.151:0/1738351131 learned_addr learned my addr 172.21.15.151:0/1738351131 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f874ffff640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87501012d0 0x7f8750196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f8747fff640 1 -- 172.21.15.151:0/1738351131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87501012d0 msgr2=0x7f8750196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f8747fff640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87501012d0 0x7f8750196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f8747fff640 1 -- 172.21.15.151:0/1738351131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8738009560 con 0x7f8750101bf0 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f874ffff640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87501012d0 0x7f8750196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f8747fff640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 0x7f87501965b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8738000c00 tx=0x7f873802c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f874dffb640 1 -- 172.21.15.151:0/1738351131 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8738004420 con 0x7f8750101bf0 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f874dffb640 1 -- 172.21.15.151:0/1738351131 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8738003ab0 con 0x7f8750101bf0 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875019ac90 con 0x7f8750101bf0 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f874dffb640 1 -- 172.21.15.151:0/1738351131 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8738005590 con 0x7f8750101bf0 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.012+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875019b1f0 con 0x7f8750101bf0 2024-04-16T03:17:44.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.013+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875004e040 con 0x7f8750101bf0 2024-04-16T03:17:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.014+0000 7f874dffb640 1 -- 172.21.15.151:0/1738351131 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f8738004dd0 con 0x7f8750101bf0 2024-04-16T03:17:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.014+0000 7f874dffb640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f873c06ec30 0x7f873c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.014+0000 7f874ffff640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f873c06ec30 0x7f873c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.014+0000 7f874dffb640 1 -- 172.21.15.151:0/1738351131 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f87380b1940 con 0x7f8750101bf0 2024-04-16T03:17:44.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.015+0000 7f874ffff640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f873c06ec30 0x7f873c0710f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8740009c20 tx=0x7f8740005f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:44.020 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.019+0000 7f874dffb640 1 -- 172.21.15.151:0/1738351131 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f873807a6f0 con 0x7f8750101bf0 2024-04-16T03:17:44.198 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.198+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87500699a0 con 0x7f8750101bf0 2024-04-16T03:17:44.199 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.198+0000 7f874dffb640 1 -- 172.21.15.151:0/1738351131 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f873807ddb0 con 0x7f8750101bf0 2024-04-16T03:17:44.199 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 1 2024-04-16T03:17:44.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f873c06ec30 msgr2=0x7f873c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f873c06ec30 0x7f873c0710f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8740009c20 tx=0x7f8740005f90 comp rx=0 tx=0).stop 2024-04-16T03:17:44.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 msgr2=0x7f87501965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 0x7f87501965b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8738000c00 tx=0x7f873802c6f0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 shutdown_connections 2024-04-16T03:17:44.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f873c06ec30 0x7f873c0710f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8750101bf0 0x7f87501965b0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 --2- 172.21.15.151:0/1738351131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87501012d0 0x7f8750196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 >> 172.21.15.151:0/1738351131 conn(0x7f87500faa30 msgr2=0x7f8750104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:44.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 shutdown_connections 2024-04-16T03:17:44.202 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.200+0000 7f8756641640 1 -- 172.21.15.151:0/1738351131 wait complete. 2024-04-16T03:17:44.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T03:17:44.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.319+0000 7f4a35609640 1 -- 172.21.15.151:0/3325302645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 msgr2=0x7f4a3010a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.319+0000 7f4a35609640 1 --2- 172.21.15.151:0/3325302645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 0x7f4a3010a000 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009880 tx=0x7f4a2402bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.319+0000 7f4a35609640 1 -- 172.21.15.151:0/3325302645 shutdown_connections 2024-04-16T03:17:44.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.319+0000 7f4a35609640 1 --2- 172.21.15.151:0/3325302645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 0x7f4a3010a000 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.319+0000 7f4a35609640 1 --2- 172.21.15.151:0/3325302645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a30101200 0x7f4a301015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.319+0000 7f4a35609640 1 -- 172.21.15.151:0/3325302645 >> 172.21.15.151:0/3325302645 conn(0x7f4a300faa60 msgr2=0x7f4a300fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:44.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.319+0000 7f4a35609640 1 -- 172.21.15.151:0/3325302645 shutdown_connections 2024-04-16T03:17:44.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.319+0000 7f4a35609640 1 -- 172.21.15.151:0/3325302645 wait complete. 2024-04-16T03:17:44.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a35609640 1 Processor -- start 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a35609640 1 -- start start 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a35609640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a30101200 0x7f4a30195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a35609640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 0x7f4a30196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a35609640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30196900 con 0x7f4a30101b20 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a35609640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3019a6d0 con 0x7f4a30101200 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a2e7fc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 0x7f4a30196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a2e7fc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 0x7f4a30196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59440/0 (socket says 172.21.15.151:59440) 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a2e7fc640 1 -- 172.21.15.151:0/438956893 learned_addr learned my addr 172.21.15.151:0/438956893 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.320+0000 7f4a2effd640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a30101200 0x7f4a30195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a2e7fc640 1 -- 172.21.15.151:0/438956893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a30101200 msgr2=0x7f4a30195d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a2e7fc640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a30101200 0x7f4a30195d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a2e7fc640 1 -- 172.21.15.151:0/438956893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a24009560 con 0x7f4a30101b20 2024-04-16T03:17:44.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a2effd640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a30101200 0x7f4a30195d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:44.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a2e7fc640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 0x7f4a30196270 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009230 tx=0x7f4a2402c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:44.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a17fff640 1 -- 172.21.15.151:0/438956893 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a24004520 con 0x7f4a30101b20 2024-04-16T03:17:44.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a17fff640 1 -- 172.21.15.151:0/438956893 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a24003ab0 con 0x7f4a30101b20 2024-04-16T03:17:44.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3019a950 con 0x7f4a30101b20 2024-04-16T03:17:44.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a17fff640 1 -- 172.21.15.151:0/438956893 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a2403e450 con 0x7f4a30101b20 2024-04-16T03:17:44.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.321+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3019aeb0 con 0x7f4a30101b20 2024-04-16T03:17:44.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.322+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3004e040 con 0x7f4a30101b20 2024-04-16T03:17:44.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.322+0000 7f4a17fff640 1 -- 172.21.15.151:0/438956893 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f4a24004f60 con 0x7f4a30101b20 2024-04-16T03:17:44.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.323+0000 7f4a17fff640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a1806ec30 0x7f4a180710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.323+0000 7f4a17fff640 1 -- 172.21.15.151:0/438956893 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4a240b1a00 con 0x7f4a30101b20 2024-04-16T03:17:44.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.323+0000 7f4a2effd640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a1806ec30 0x7f4a180710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.323+0000 7f4a2effd640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a1806ec30 0x7f4a180710f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4a200058c0 tx=0x7f4a20005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:44.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.328+0000 7f4a17fff640 1 -- 172.21.15.151:0/438956893 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4a2407a7b0 con 0x7f4a30101b20 2024-04-16T03:17:44.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:44 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3058681152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T03:17:44.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:44 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/4114641489' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T03:17:44.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:44 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1746480205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T03:17:44.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.507+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f4a301075f0 con 0x7f4a30101b20 2024-04-16T03:17:44.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.508+0000 7f4a17fff640 1 -- 172.21.15.151:0/438956893 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f4a24043020 con 0x7f4a30101b20 2024-04-16T03:17:44.509 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 32 2024-04-16T03:17:44.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a1806ec30 msgr2=0x7f4a180710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a1806ec30 0x7f4a180710f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4a200058c0 tx=0x7f4a20005850 comp rx=0 tx=0).stop 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 msgr2=0x7f4a30196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 0x7f4a30196270 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009230 tx=0x7f4a2402c7c0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 shutdown_connections 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f4a1806ec30 0x7f4a180710f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a30101b20 0x7f4a30196270 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 --2- 172.21.15.151:0/438956893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a30101200 0x7f4a30195d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 >> 172.21.15.151:0/438956893 conn(0x7f4a300faa60 msgr2=0x7f4a301047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 shutdown_connections 2024-04-16T03:17:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.510+0000 7f4a35609640 1 -- 172.21.15.151:0/438956893 wait complete. 2024-04-16T03:17:44.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:44 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3058681152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T03:17:44.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:44 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/4114641489' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T03:17:44.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:44 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1746480205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T03:17:44.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T03:17:44.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.628+0000 7f59eaa8e640 1 -- 172.21.15.151:0/1812435430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e4100c50 msgr2=0x7f59e4104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.628+0000 7f59eaa8e640 1 --2- 172.21.15.151:0/1812435430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e4100c50 0x7f59e4104c90 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009880 tx=0x7f59d002bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.628+0000 7f59eaa8e640 1 -- 172.21.15.151:0/1812435430 shutdown_connections 2024-04-16T03:17:44.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.628+0000 7f59eaa8e640 1 --2- 172.21.15.151:0/1812435430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e4100c50 0x7f59e4104c90 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.631 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.628+0000 7f59eaa8e640 1 --2- 172.21.15.151:0/1812435430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e41002a0 0x7f59e4100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.628+0000 7f59eaa8e640 1 -- 172.21.15.151:0/1812435430 >> 172.21.15.151:0/1812435430 conn(0x7f59e40fbb20 msgr2=0x7f59e40fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.628+0000 7f59eaa8e640 1 -- 172.21.15.151:0/1812435430 shutdown_connections 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.628+0000 7f59eaa8e640 1 -- 172.21.15.151:0/1812435430 wait complete. 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59eaa8e640 1 Processor -- start 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59eaa8e640 1 -- start start 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59eaa8e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e41002a0 0x7f59e4195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59eaa8e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e41962a0 0x7f59e419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59eaa8e640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e41968a0 con 0x7f59e41962a0 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59eaa8e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4196a10 con 0x7f59e41002a0 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59e3fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e41962a0 0x7f59e419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59e3fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e41962a0 0x7f59e419a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59466/0 (socket says 172.21.15.151:59466) 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59e3fff640 1 -- 172.21.15.151:0/3669542238 learned_addr learned my addr 172.21.15.151:0/3669542238 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59e882a640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e41002a0 0x7f59e4195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59e3fff640 1 -- 172.21.15.151:0/3669542238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e41002a0 msgr2=0x7f59e4195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.632 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59e3fff640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e41002a0 0x7f59e4195d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59e3fff640 1 -- 172.21.15.151:0/3669542238 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d0009560 con 0x7f59e41962a0 2024-04-16T03:17:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59e882a640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e41002a0 0x7f59e4195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.629+0000 7f59e3fff640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e41962a0 0x7f59e419a700 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f59d0036040 tx=0x7f59d002c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.630+0000 7f59e1ffb640 1 -- 172.21.15.151:0/3669542238 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d00035f0 con 0x7f59e41962a0 2024-04-16T03:17:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.630+0000 7f59e1ffb640 1 -- 172.21.15.151:0/3669542238 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d0003c10 con 0x7f59e41962a0 2024-04-16T03:17:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.630+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e419aca0 con 0x7f59e41962a0 2024-04-16T03:17:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.630+0000 7f59e1ffb640 1 -- 172.21.15.151:0/3669542238 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d0034440 con 0x7f59e41962a0 2024-04-16T03:17:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.630+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e419b200 con 0x7f59e41962a0 2024-04-16T03:17:44.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.631+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e404e040 con 0x7f59e41962a0 2024-04-16T03:17:44.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.631+0000 7f59e1ffb640 1 -- 172.21.15.151:0/3669542238 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7f59d0003750 con 0x7f59e41962a0 2024-04-16T03:17:44.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.632+0000 7f59e1ffb640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f59d406ec30 0x7f59d40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.632+0000 7f59e1ffb640 1 -- 172.21.15.151:0/3669542238 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f59d00b1160 con 0x7f59e41962a0 2024-04-16T03:17:44.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.632+0000 7f59e882a640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f59d406ec30 0x7f59d40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.632+0000 7f59e882a640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f59d406ec30 0x7f59d40710f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f59d8005fd0 tx=0x7f59d800b5f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:44.637 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.636+0000 7f59e1ffb640 1 -- 172.21.15.151:0/3669542238 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f59d0079f90 con 0x7f59e41962a0 2024-04-16T03:17:44.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.813+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f59e41085a0 con 0x7f59e41962a0 2024-04-16T03:17:44.815 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.814+0000 7f59e1ffb640 1 -- 172.21.15.151:0/3669542238 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f59d007d650 con 0x7f59e41962a0 2024-04-16T03:17:44.815 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 32 2024-04-16T03:17:44.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f59d406ec30 msgr2=0x7f59d40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f59d406ec30 0x7f59d40710f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f59d8005fd0 tx=0x7f59d800b5f0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e41962a0 msgr2=0x7f59e419a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e41962a0 0x7f59e419a700 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f59d0036040 tx=0x7f59d002c850 comp rx=0 tx=0).stop 2024-04-16T03:17:44.817 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 shutdown_connections 2024-04-16T03:17:44.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7f59d406ec30 0x7f59d40710f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59e41962a0 0x7f59e419a700 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 --2- 172.21.15.151:0/3669542238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e41002a0 0x7f59e4195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 >> 172.21.15.151:0/3669542238 conn(0x7f59e40fbb20 msgr2=0x7f59e40fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:44.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 shutdown_connections 2024-04-16T03:17:44.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.816+0000 7f59eaa8e640 1 -- 172.21.15.151:0/3669542238 wait complete. 2024-04-16T03:17:44.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T03:17:44.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.933+0000 7fc394f39640 1 -- 172.21.15.151:0/2017372570 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3900ff170 msgr2=0x7fc3900ff550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.933+0000 7fc394f39640 1 --2- 172.21.15.151:0/2017372570 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3900ff170 0x7fc3900ff550 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fc380009880 tx=0x7fc38002bce0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.933+0000 7fc394f39640 1 -- 172.21.15.151:0/2017372570 shutdown_connections 2024-04-16T03:17:44.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.933+0000 7fc394f39640 1 --2- 172.21.15.151:0/2017372570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3900ffb20 0x7fc390103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.933+0000 7fc394f39640 1 --2- 172.21.15.151:0/2017372570 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3900ff170 0x7fc3900ff550 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.933+0000 7fc394f39640 1 -- 172.21.15.151:0/2017372570 >> 172.21.15.151:0/2017372570 conn(0x7fc3900fa9d0 msgr2=0x7fc3900fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:44.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.933+0000 7fc394f39640 1 -- 172.21.15.151:0/2017372570 shutdown_connections 2024-04-16T03:17:44.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.933+0000 7fc394f39640 1 -- 172.21.15.151:0/2017372570 wait complete. 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc394f39640 1 Processor -- start 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc394f39640 1 -- start start 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc394f39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3900ff170 0x7fc390196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc394f39640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3901970a0 0x7fc390197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc394f39640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39019fbd0 con 0x7fc3901970a0 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc394f39640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39019fd10 con 0x7fc3900ff170 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc387fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3901970a0 0x7fc390197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc387fff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3901970a0 0x7fc390197500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59492/0 (socket says 172.21.15.151:59492) 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc387fff640 1 -- 172.21.15.151:0/3024070143 learned_addr learned my addr 172.21.15.151:0/3024070143 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc38e59c640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3900ff170 0x7fc390196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc387fff640 1 -- 172.21.15.151:0/3024070143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3900ff170 msgr2=0x7fc390196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:44.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc387fff640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3900ff170 0x7fc390196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:44.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.934+0000 7fc387fff640 1 -- 172.21.15.151:0/3024070143 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc380009560 con 0x7fc3901970a0 2024-04-16T03:17:44.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.935+0000 7fc38e59c640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3900ff170 0x7fc390196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T03:17:44.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.935+0000 7fc387fff640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3901970a0 0x7fc390197500 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fc37800c920 tx=0x7fc37800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:44.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.935+0000 7fc3877fe640 1 -- 172.21.15.151:0/3024070143 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc378004410 con 0x7fc3901970a0 2024-04-16T03:17:44.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.935+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc39019ff10 con 0x7fc3901970a0 2024-04-16T03:17:44.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.935+0000 7fc3877fe640 1 -- 172.21.15.151:0/3024070143 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc378007c60 con 0x7fc3901970a0 2024-04-16T03:17:44.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.935+0000 7fc3877fe640 1 -- 172.21.15.151:0/3024070143 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc37800f450 con 0x7fc3901970a0 2024-04-16T03:17:44.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.935+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3901a03e0 con 0x7fc3901970a0 2024-04-16T03:17:44.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.936+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39004e040 con 0x7fc3901970a0 2024-04-16T03:17:44.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.936+0000 7fc3877fe640 1 -- 172.21.15.151:0/3024070143 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 41) v1 ==== 92747+0+0 (secure 0 0 0) 0x7fc3780077a0 con 0x7fc3901970a0 2024-04-16T03:17:44.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.937+0000 7fc3877fe640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc37c06ecf0 0x7fc37c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T03:17:44.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.937+0000 7fc38e59c640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc37c06ecf0 0x7fc37c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T03:17:44.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.937+0000 7fc3877fe640 1 -- 172.21.15.151:0/3024070143 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc37808edc0 con 0x7fc3901970a0 2024-04-16T03:17:44.942 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.937+0000 7fc38e59c640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc37c06ecf0 0x7fc37c0711b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc380000c00 tx=0x7fc38002cc20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T03:17:44.943 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:44.942+0000 7fc3877fe640 1 -- 172.21.15.151:0/3024070143 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc378057b70 con 0x7fc3901970a0 2024-04-16T03:17:45.133 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.132+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc390107470 con 0x7fc3901970a0 2024-04-16T03:17:45.134 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.133+0000 7fc3877fe640 1 -- 172.21.15.151:0/3024070143 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v86) v1 ==== 81+0+2341 (secure 0 0 0) 0x7fc37805b230 con 0x7fc3901970a0 2024-04-16T03:17:45.135 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6827/1662886502 2024-04-17T02:44:25.408644+0000 2024-04-16T03:17:45.135 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6826/1662886502 2024-04-17T02:44:25.408644+0000 2024-04-16T03:17:45.135 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6829/37453862 2024-04-17T02:44:25.408661+0000 2024-04-16T03:17:45.135 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/2684608614 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:45.135 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3091708247 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:45.135 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/681541897 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:45.135 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/969025109 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:45.136 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/671603278 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:45.136 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3129110405 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:45.136 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3798542210 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:45.136 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/2753390843 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:45.136 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/2402350166 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:45.136 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/3544869641 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:45.136 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:0/986410000 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:45.136 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/4272586248 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:45.137 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/3544869641 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:45.137 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3454798699 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:45.137 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/2400549256 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:45.137 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:0/3590885768 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:45.137 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:6829/2287826646 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:45.137 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6828/37453862 2024-04-17T02:44:25.408661+0000 2024-04-16T03:17:45.137 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/2293925528 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:45.137 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/1505356478 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:45.138 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/1299604123 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:45.138 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/661820069 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:45.138 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:6824/1319024054 2024-04-17T02:32:48.923574+0000 2024-04-16T03:17:45.138 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/3077174249 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:45.138 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:6825/1319024054 2024-04-17T02:32:48.923574+0000 2024-04-16T03:17:45.138 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/2400549256 2024-04-17T02:26:34.117371+0000 2024-04-16T03:17:45.138 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:0/977218519 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:45.138 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/2753390843 2024-04-17T02:35:10.400124+0000 2024-04-16T03:17:45.139 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/1923729027 2024-04-17T02:27:42.821901+0000 2024-04-16T03:17:45.139 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/39987226 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:45.139 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:6828/2287826646 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:45.139 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.153:0/3588036861 2024-04-17T02:36:50.207364+0000 2024-04-16T03:17:45.139 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/39987226 2024-04-17T02:26:01.417600+0000 2024-04-16T03:17:45.139 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6800/1243156277 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:45.139 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:6801/1243156277 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:45.139 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151:0/1770724784 2024-04-17T02:39:53.700570+0000 2024-04-16T03:17:45.141 INFO:teuthology.orchestra.run.smithi151.stderr:listed 39 entries 2024-04-16T03:17:45.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc37c06ecf0 msgr2=0x7fc37c0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:45.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc37c06ecf0 0x7fc37c0711b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc380000c00 tx=0x7fc38002cc20 comp rx=0 tx=0).stop 2024-04-16T03:17:45.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3901970a0 msgr2=0x7fc390197500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T03:17:45.141 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3901970a0 0x7fc390197500 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fc37800c920 tx=0x7fc37800cde0 comp rx=0 tx=0).stop 2024-04-16T03:17:45.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 shutdown_connections 2024-04-16T03:17:45.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:6800/2362635868,v1:172.21.15.151:6801/2362635868] conn(0x7fc37c06ecf0 0x7fc37c0711b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:45.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc3901970a0 0x7fc390197500 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:45.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 --2- 172.21.15.151:0/3024070143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3900ff170 0x7fc390196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T03:17:45.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 >> 172.21.15.151:0/3024070143 conn(0x7fc3900fa9d0 msgr2=0x7fc3900fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T03:17:45.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 shutdown_connections 2024-04-16T03:17:45.142 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T03:17:45.135+0000 7fc394f39640 1 -- 172.21.15.151:0/3024070143 wait complete. 2024-04-16T03:17:45.148 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi151.front.sepia.ceph.com... 2024-04-16T03:17:45.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T03:17:45.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-16T03:17:45.180 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T03:17:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:45 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/1738351131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T03:17:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:45 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/438956893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T03:17:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:45 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3669542238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T03:17:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:45 smithi151 ceph-mon[112233]: pgmap v1177: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:45.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:45 smithi151 ceph-mon[112233]: from='client.? 172.21.15.151:0/3024070143' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T03:17:45.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:45 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/1738351131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T03:17:45.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:45 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/438956893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T03:17:45.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:45 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3669542238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T03:17:45.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:45 smithi153 ceph-mon[87858]: pgmap v1177: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:45.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:45 smithi153 ceph-mon[87858]: from='client.? 172.21.15.151:0/3024070143' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T03:17:47.466 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:47 smithi151 ceph-mon[112233]: pgmap v1178: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:47.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:47 smithi153 ceph-mon[87858]: pgmap v1178: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:49.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:49 smithi151 ceph-mon[112233]: pgmap v1179: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:49.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:49 smithi153 ceph-mon[87858]: pgmap v1179: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:51.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:51 smithi151 ceph-mon[112233]: pgmap v1180: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:51.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:51 smithi153 ceph-mon[87858]: pgmap v1180: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:53.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:53 smithi151 ceph-mon[112233]: pgmap v1181: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:53.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:53 smithi153 ceph-mon[87858]: pgmap v1181: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:54.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:17:54.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:17:55.468 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:55 smithi153 ceph-mon[87858]: pgmap v1182: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:55 smithi151 ceph-mon[112233]: pgmap v1182: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:57.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:57 smithi151 ceph-mon[112233]: pgmap v1183: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:57.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:57 smithi153 ceph-mon[87858]: pgmap v1183: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:17:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:17:59 smithi151 ceph-mon[112233]: pgmap v1184: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:17:59.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:17:59 smithi153 ceph-mon[87858]: pgmap v1184: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:01.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:01 smithi151 ceph-mon[112233]: pgmap v1185: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:01.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:01 smithi153 ceph-mon[87858]: pgmap v1185: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:02.314 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:02 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:18:02.315 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:02 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:18:02.315 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:02 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:18:02.315 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:02 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:18:02.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:02 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:18:02.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:02 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:18:02.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:02 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:18:02.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:02 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:18:03.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:03 smithi151 ceph-mon[112233]: pgmap v1186: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:03 smithi153 ceph-mon[87858]: pgmap v1186: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:05.498 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:05 smithi151 ceph-mon[112233]: pgmap v1187: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:05 smithi153 ceph-mon[87858]: pgmap v1187: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:07.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:07 smithi151 ceph-mon[112233]: pgmap v1188: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:07.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:07 smithi153 ceph-mon[87858]: pgmap v1188: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:18:09.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:18:10.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:09 smithi151 ceph-mon[112233]: pgmap v1189: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:09 smithi153 ceph-mon[87858]: pgmap v1189: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:11.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:11 smithi151 ceph-mon[112233]: pgmap v1190: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:11.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:11 smithi153 ceph-mon[87858]: pgmap v1190: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:13.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:13 smithi151 ceph-mon[112233]: pgmap v1191: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:13.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:13 smithi153 ceph-mon[87858]: pgmap v1191: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:15.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:15 smithi151 ceph-mon[112233]: pgmap v1192: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:15.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:15 smithi153 ceph-mon[87858]: pgmap v1192: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:17.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:17 smithi151 ceph-mon[112233]: pgmap v1193: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:17 smithi153 ceph-mon[87858]: pgmap v1193: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:19.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:19 smithi151 ceph-mon[112233]: pgmap v1194: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:19.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:19 smithi153 ceph-mon[87858]: pgmap v1194: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:21.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:21 smithi151 ceph-mon[112233]: pgmap v1195: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:21 smithi153 ceph-mon[87858]: pgmap v1195: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:23.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:23 smithi151 ceph-mon[112233]: pgmap v1196: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:23.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:23 smithi153 ceph-mon[87858]: pgmap v1196: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:18:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:18:25.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:25 smithi151 ceph-mon[112233]: pgmap v1197: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:25.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:25 smithi153 ceph-mon[87858]: pgmap v1197: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:27.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:27 smithi151 ceph-mon[112233]: pgmap v1198: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:27.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:27 smithi153 ceph-mon[87858]: pgmap v1198: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:29.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:29 smithi151 ceph-mon[112233]: pgmap v1199: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:29.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:29 smithi153 ceph-mon[87858]: pgmap v1199: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:31.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:31 smithi151 ceph-mon[112233]: pgmap v1200: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:31.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:31 smithi153 ceph-mon[87858]: pgmap v1200: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:33.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:33 smithi151 ceph-mon[112233]: pgmap v1201: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:33.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:33 smithi153 ceph-mon[87858]: pgmap v1201: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:35.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:35 smithi151 ceph-mon[112233]: pgmap v1202: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:35.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:35 smithi153 ceph-mon[87858]: pgmap v1202: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:37.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:37 smithi151 ceph-mon[112233]: pgmap v1203: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:37 smithi153 ceph-mon[87858]: pgmap v1203: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:18:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:18:40.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:39 smithi151 ceph-mon[112233]: pgmap v1204: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:40.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:39 smithi153 ceph-mon[87858]: pgmap v1204: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:41.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:41 smithi153 ceph-mon[87858]: pgmap v1205: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:41.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:41 smithi151 ceph-mon[112233]: pgmap v1205: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:43.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:43 smithi151 ceph-mon[112233]: pgmap v1206: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:43.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:43 smithi153 ceph-mon[87858]: pgmap v1206: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:45.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:45 smithi151 ceph-mon[112233]: pgmap v1207: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:45.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:45 smithi153 ceph-mon[87858]: pgmap v1207: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:47.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:47 smithi151 ceph-mon[112233]: pgmap v1208: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:47.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:47 smithi153 ceph-mon[87858]: pgmap v1208: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:49.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:49 smithi151 ceph-mon[112233]: pgmap v1209: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:49.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:49 smithi153 ceph-mon[87858]: pgmap v1209: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:51.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:51 smithi151 ceph-mon[112233]: pgmap v1210: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:51.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:51 smithi153 ceph-mon[87858]: pgmap v1210: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:53.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:53 smithi151 ceph-mon[112233]: pgmap v1211: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:53.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:53 smithi153 ceph-mon[87858]: pgmap v1211: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:54.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:18:54.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:18:55.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:55 smithi151 ceph-mon[112233]: pgmap v1212: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:55.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:55 smithi153 ceph-mon[87858]: pgmap v1212: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:57.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:57 smithi151 ceph-mon[112233]: pgmap v1213: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:57.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:57 smithi153 ceph-mon[87858]: pgmap v1213: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:18:59.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:18:59 smithi151 ceph-mon[112233]: pgmap v1214: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:18:59.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:18:59 smithi153 ceph-mon[87858]: pgmap v1214: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:01.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:01 smithi151 ceph-mon[112233]: pgmap v1215: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:01.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:01 smithi153 ceph-mon[87858]: pgmap v1215: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:02.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:02 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:19:02.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:02 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:19:03.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:19:03.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:19:03.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:19:03.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:03 smithi151 ceph-mon[112233]: pgmap v1216: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:19:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:19:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:19:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:03 smithi153 ceph-mon[87858]: pgmap v1216: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:05.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:05 smithi151 ceph-mon[112233]: pgmap v1217: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:05 smithi153 ceph-mon[87858]: pgmap v1217: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:07.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:07 smithi151 ceph-mon[112233]: pgmap v1218: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:07.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:07 smithi153 ceph-mon[87858]: pgmap v1218: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:09.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:19:09.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:19:10.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:09 smithi151 ceph-mon[112233]: pgmap v1219: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:09 smithi153 ceph-mon[87858]: pgmap v1219: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:11.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:11 smithi151 ceph-mon[112233]: pgmap v1220: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:11.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:11 smithi153 ceph-mon[87858]: pgmap v1220: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:13.440 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:13 smithi151 ceph-mon[112233]: pgmap v1221: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:13.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:13 smithi153 ceph-mon[87858]: pgmap v1221: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:15.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:15 smithi151 ceph-mon[112233]: pgmap v1222: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:15.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:15 smithi153 ceph-mon[87858]: pgmap v1222: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:17.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:17 smithi151 ceph-mon[112233]: pgmap v1223: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:17 smithi153 ceph-mon[87858]: pgmap v1223: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:19.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:19 smithi151 ceph-mon[112233]: pgmap v1224: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:19.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:19 smithi153 ceph-mon[87858]: pgmap v1224: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:21.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:21 smithi153 ceph-mon[87858]: pgmap v1225: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:21.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:21 smithi151 ceph-mon[112233]: pgmap v1225: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:23.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:23 smithi151 ceph-mon[112233]: pgmap v1226: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:23.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:23 smithi153 ceph-mon[87858]: pgmap v1226: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:19:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:19:25.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:25 smithi151 ceph-mon[112233]: pgmap v1227: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:25.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:25 smithi153 ceph-mon[87858]: pgmap v1227: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:27.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:27 smithi151 ceph-mon[112233]: pgmap v1228: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:27.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:27 smithi153 ceph-mon[87858]: pgmap v1228: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:29.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:29 smithi151 ceph-mon[112233]: pgmap v1229: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:29.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:29 smithi153 ceph-mon[87858]: pgmap v1229: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:31.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:31 smithi151 ceph-mon[112233]: pgmap v1230: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:31.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:31 smithi153 ceph-mon[87858]: pgmap v1230: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:33.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:33 smithi151 ceph-mon[112233]: pgmap v1231: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:33.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:33 smithi153 ceph-mon[87858]: pgmap v1231: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:35 smithi151 ceph-mon[112233]: pgmap v1232: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:35.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:35 smithi153 ceph-mon[87858]: pgmap v1232: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:37.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:37 smithi151 ceph-mon[112233]: pgmap v1233: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:37 smithi153 ceph-mon[87858]: pgmap v1233: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:38.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:19:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:19:40.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:39 smithi151 ceph-mon[112233]: pgmap v1234: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:40.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:39 smithi153 ceph-mon[87858]: pgmap v1234: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:41.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:41 smithi151 ceph-mon[112233]: pgmap v1235: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:41 smithi153 ceph-mon[87858]: pgmap v1235: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:43.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:43 smithi151 ceph-mon[112233]: pgmap v1236: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:43.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:43 smithi153 ceph-mon[87858]: pgmap v1236: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:45.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:45 smithi151 ceph-mon[112233]: pgmap v1237: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:45.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:45 smithi153 ceph-mon[87858]: pgmap v1237: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:47.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:47 smithi151 ceph-mon[112233]: pgmap v1238: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:47.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:47 smithi153 ceph-mon[87858]: pgmap v1238: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:49.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:49 smithi151 ceph-mon[112233]: pgmap v1239: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:49.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:49 smithi153 ceph-mon[87858]: pgmap v1239: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:51.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:51 smithi151 ceph-mon[112233]: pgmap v1240: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:51.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:51 smithi153 ceph-mon[87858]: pgmap v1240: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:53.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:53 smithi151 ceph-mon[112233]: pgmap v1241: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:53.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:53 smithi153 ceph-mon[87858]: pgmap v1241: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:54.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:19:54.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:19:55.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:55 smithi151 ceph-mon[112233]: pgmap v1242: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:55.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:55 smithi153 ceph-mon[87858]: pgmap v1242: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:57.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:57 smithi151 ceph-mon[112233]: pgmap v1243: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:57.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:57 smithi153 ceph-mon[87858]: pgmap v1243: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:19:59.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:19:59 smithi151 ceph-mon[112233]: pgmap v1244: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:19:59.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:19:59 smithi153 ceph-mon[87858]: pgmap v1244: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:00.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:00 smithi151 ceph-mon[112233]: overall HEALTH_OK 2024-04-16T03:20:00.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:00 smithi153 ceph-mon[87858]: overall HEALTH_OK 2024-04-16T03:20:01.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:01 smithi151 ceph-mon[112233]: pgmap v1245: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:01.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:01 smithi153 ceph-mon[87858]: pgmap v1245: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:03.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:02 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:20:03.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:02 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:20:04.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:03 smithi151 ceph-mon[112233]: pgmap v1246: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:04.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:20:04.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:20:04.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:03 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:20:04.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:03 smithi153 ceph-mon[87858]: pgmap v1246: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:04.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:20:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:20:04.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:03 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:20:05.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:05 smithi151 ceph-mon[112233]: pgmap v1247: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:05 smithi153 ceph-mon[87858]: pgmap v1247: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:07.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:07 smithi153 ceph-mon[87858]: pgmap v1248: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:07.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:07 smithi151 ceph-mon[112233]: pgmap v1248: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:09.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:20:09.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:20:10.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:09 smithi151 ceph-mon[112233]: pgmap v1249: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:09 smithi153 ceph-mon[87858]: pgmap v1249: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:11.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:11 smithi151 ceph-mon[112233]: pgmap v1250: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:11.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:11 smithi153 ceph-mon[87858]: pgmap v1250: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:13.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:13 smithi151 ceph-mon[112233]: pgmap v1251: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:13.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:13 smithi153 ceph-mon[87858]: pgmap v1251: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:15.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:15 smithi151 ceph-mon[112233]: pgmap v1252: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:15.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:15 smithi153 ceph-mon[87858]: pgmap v1252: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:17.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:17 smithi151 ceph-mon[112233]: pgmap v1253: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:17 smithi153 ceph-mon[87858]: pgmap v1253: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:19.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:19 smithi151 ceph-mon[112233]: pgmap v1254: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:19.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:19 smithi153 ceph-mon[87858]: pgmap v1254: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:21.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:21 smithi151 ceph-mon[112233]: pgmap v1255: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:21 smithi153 ceph-mon[87858]: pgmap v1255: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:23.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:23 smithi151 ceph-mon[112233]: pgmap v1256: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:23.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:23 smithi153 ceph-mon[87858]: pgmap v1256: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:20:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:20:25.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:25 smithi151 ceph-mon[112233]: pgmap v1257: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:25.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:25 smithi153 ceph-mon[87858]: pgmap v1257: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:27.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:27 smithi151 ceph-mon[112233]: pgmap v1258: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:27.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:27 smithi153 ceph-mon[87858]: pgmap v1258: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:29.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:29 smithi151 ceph-mon[112233]: pgmap v1259: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:29.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:29 smithi153 ceph-mon[87858]: pgmap v1259: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:31.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:31 smithi151 ceph-mon[112233]: pgmap v1260: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:31.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:31 smithi153 ceph-mon[87858]: pgmap v1260: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:33.497 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:33 smithi151 ceph-mon[112233]: pgmap v1261: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:33.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:33 smithi153 ceph-mon[87858]: pgmap v1261: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:35 smithi151 ceph-mon[112233]: pgmap v1262: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:35.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:35 smithi153 ceph-mon[87858]: pgmap v1262: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:37.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:37 smithi151 ceph-mon[112233]: pgmap v1263: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:37 smithi153 ceph-mon[87858]: pgmap v1263: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:39.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:20:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:20:40.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:39 smithi151 ceph-mon[112233]: pgmap v1264: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:40.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:39 smithi153 ceph-mon[87858]: pgmap v1264: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:41.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:41 smithi151 ceph-mon[112233]: pgmap v1265: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:41 smithi153 ceph-mon[87858]: pgmap v1265: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:43.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:43 smithi151 ceph-mon[112233]: pgmap v1266: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:43.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:43 smithi153 ceph-mon[87858]: pgmap v1266: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:45.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:45 smithi151 ceph-mon[112233]: pgmap v1267: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:45.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:45 smithi153 ceph-mon[87858]: pgmap v1267: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:47.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:47 smithi151 ceph-mon[112233]: pgmap v1268: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:47.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:47 smithi153 ceph-mon[87858]: pgmap v1268: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:49.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:49 smithi151 ceph-mon[112233]: pgmap v1269: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:49.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:49 smithi153 ceph-mon[87858]: pgmap v1269: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:51.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:51 smithi151 ceph-mon[112233]: pgmap v1270: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:51.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:51 smithi153 ceph-mon[87858]: pgmap v1270: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:53.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:53 smithi151 ceph-mon[112233]: pgmap v1271: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:53.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:53 smithi153 ceph-mon[87858]: pgmap v1271: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:54.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:20:54.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:20:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:55 smithi151 ceph-mon[112233]: pgmap v1272: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:55.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:55 smithi153 ceph-mon[87858]: pgmap v1272: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:57.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:57 smithi151 ceph-mon[112233]: pgmap v1273: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:57.519 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:57 smithi153 ceph-mon[87858]: pgmap v1273: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:20:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:20:59 smithi151 ceph-mon[112233]: pgmap v1274: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:20:59.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:20:59 smithi153 ceph-mon[87858]: pgmap v1274: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:01.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:01 smithi151 ceph-mon[112233]: pgmap v1275: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:01.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:01 smithi153 ceph-mon[87858]: pgmap v1275: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:03.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:03 smithi151 ceph-mon[112233]: pgmap v1276: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:03 smithi153 ceph-mon[87858]: pgmap v1276: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:04.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:04 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:21:04.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:04 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:21:05.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:21:05.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:21:05.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:21:05.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:05 smithi151 ceph-mon[112233]: pgmap v1277: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:21:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:21:05.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:21:05.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:05 smithi153 ceph-mon[87858]: pgmap v1277: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:07.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:07 smithi151 ceph-mon[112233]: pgmap v1278: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:07.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:07 smithi153 ceph-mon[87858]: pgmap v1278: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:09.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:21:09.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:21:10.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:09 smithi151 ceph-mon[112233]: pgmap v1279: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:09 smithi153 ceph-mon[87858]: pgmap v1279: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:11.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:11 smithi151 ceph-mon[112233]: pgmap v1280: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:11.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:11 smithi153 ceph-mon[87858]: pgmap v1280: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:13.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:13 smithi151 ceph-mon[112233]: pgmap v1281: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:13.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:13 smithi153 ceph-mon[87858]: pgmap v1281: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:15.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:15 smithi151 ceph-mon[112233]: pgmap v1282: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:15.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:15 smithi153 ceph-mon[87858]: pgmap v1282: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:17.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:17 smithi151 ceph-mon[112233]: pgmap v1283: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:17 smithi153 ceph-mon[87858]: pgmap v1283: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:19.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:19 smithi151 ceph-mon[112233]: pgmap v1284: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:19.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:19 smithi153 ceph-mon[87858]: pgmap v1284: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:21.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:21 smithi151 ceph-mon[112233]: pgmap v1285: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:21 smithi153 ceph-mon[87858]: pgmap v1285: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:23.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:23 smithi151 ceph-mon[112233]: pgmap v1286: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:23.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:23 smithi153 ceph-mon[87858]: pgmap v1286: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:21:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:21:25.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:25 smithi151 ceph-mon[112233]: pgmap v1287: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:25.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:25 smithi153 ceph-mon[87858]: pgmap v1287: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:27.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:27 smithi151 ceph-mon[112233]: pgmap v1288: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:27.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:27 smithi153 ceph-mon[87858]: pgmap v1288: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:29.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:29 smithi151 ceph-mon[112233]: pgmap v1289: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:29.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:29 smithi153 ceph-mon[87858]: pgmap v1289: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:31.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:31 smithi151 ceph-mon[112233]: pgmap v1290: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:31.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:31 smithi153 ceph-mon[87858]: pgmap v1290: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:33.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:33 smithi151 ceph-mon[112233]: pgmap v1291: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:33.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:33 smithi153 ceph-mon[87858]: pgmap v1291: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:35 smithi151 ceph-mon[112233]: pgmap v1292: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:35.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:35 smithi153 ceph-mon[87858]: pgmap v1292: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:37.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:37 smithi151 ceph-mon[112233]: pgmap v1293: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:37 smithi153 ceph-mon[87858]: pgmap v1293: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:39.245 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:21:39.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:21:40.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:39 smithi153 ceph-mon[87858]: pgmap v1294: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:40.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:39 smithi151 ceph-mon[112233]: pgmap v1294: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:41.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:41 smithi151 ceph-mon[112233]: pgmap v1295: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:41 smithi153 ceph-mon[87858]: pgmap v1295: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:43.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:43 smithi151 ceph-mon[112233]: pgmap v1296: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:43.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:43 smithi153 ceph-mon[87858]: pgmap v1296: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:45.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:45 smithi151 ceph-mon[112233]: pgmap v1297: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:45.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:45 smithi153 ceph-mon[87858]: pgmap v1297: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:47.496 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:47 smithi151 ceph-mon[112233]: pgmap v1298: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:47.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:47 smithi153 ceph-mon[87858]: pgmap v1298: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:49.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:49 smithi151 ceph-mon[112233]: pgmap v1299: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:49.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:49 smithi153 ceph-mon[87858]: pgmap v1299: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:51.396 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:51 smithi151 ceph-mon[112233]: pgmap v1300: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:51.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:51 smithi153 ceph-mon[87858]: pgmap v1300: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:53.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:53 smithi151 ceph-mon[112233]: pgmap v1301: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:53.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:53 smithi153 ceph-mon[87858]: pgmap v1301: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:54.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:54 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:21:54.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:54 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:21:55.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:55 smithi151 ceph-mon[112233]: pgmap v1302: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:55.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:55 smithi153 ceph-mon[87858]: pgmap v1302: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:57.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:57 smithi151 ceph-mon[112233]: pgmap v1303: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:57.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:57 smithi153 ceph-mon[87858]: pgmap v1303: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:21:59.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:21:59 smithi151 ceph-mon[112233]: pgmap v1304: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:21:59.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:21:59 smithi153 ceph-mon[87858]: pgmap v1304: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:01.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:01 smithi151 ceph-mon[112233]: pgmap v1305: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:01.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:01 smithi153 ceph-mon[87858]: pgmap v1305: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:03.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:03 smithi151 ceph-mon[112233]: pgmap v1306: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:03.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:03 smithi153 ceph-mon[87858]: pgmap v1306: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:04.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:04 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:22:05.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:04 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T03:22:05.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:05 smithi151 ceph-mon[112233]: pgmap v1307: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:05.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:22:05.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:22:05.744 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:05 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:22:06.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:05 smithi153 ceph-mon[87858]: pgmap v1307: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:06.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T03:22:06.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T03:22:06.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:05 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' 2024-04-16T03:22:07.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:07 smithi151 ceph-mon[112233]: pgmap v1308: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:22:07.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:07 smithi153 ceph-mon[87858]: pgmap v1308: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:22:09.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:08 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:22:09.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:08 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:22:10.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:09 smithi151 ceph-mon[112233]: pgmap v1309: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:10.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:09 smithi153 ceph-mon[87858]: pgmap v1309: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:11.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:11 smithi151 ceph-mon[112233]: pgmap v1310: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:11.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:11 smithi153 ceph-mon[87858]: pgmap v1310: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:13.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:13 smithi151 ceph-mon[112233]: pgmap v1311: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:13.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:13 smithi153 ceph-mon[87858]: pgmap v1311: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:15.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:15 smithi151 ceph-mon[112233]: pgmap v1312: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:15.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:15 smithi153 ceph-mon[87858]: pgmap v1312: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:17.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:17 smithi151 ceph-mon[112233]: pgmap v1313: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:22:17.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:17 smithi153 ceph-mon[87858]: pgmap v1313: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:22:19.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:19 smithi151 ceph-mon[112233]: pgmap v1314: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:19.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:19 smithi153 ceph-mon[87858]: pgmap v1314: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:21.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:21 smithi151 ceph-mon[112233]: pgmap v1315: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:21.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:21 smithi153 ceph-mon[87858]: pgmap v1315: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:23.480 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:23 smithi153 ceph-mon[87858]: pgmap v1316: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:23.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:23 smithi151 ceph-mon[112233]: pgmap v1316: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:24.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:24 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:22:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:24 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:22:25.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:25 smithi151 ceph-mon[112233]: pgmap v1317: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:25.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:25 smithi153 ceph-mon[87858]: pgmap v1317: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:27.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:27 smithi151 ceph-mon[112233]: pgmap v1318: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:22:27.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:27 smithi153 ceph-mon[87858]: pgmap v1318: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:22:29.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:29 smithi151 ceph-mon[112233]: pgmap v1319: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:29.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:29 smithi153 ceph-mon[87858]: pgmap v1319: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:31.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:31 smithi151 ceph-mon[112233]: pgmap v1320: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:31.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:31 smithi153 ceph-mon[87858]: pgmap v1320: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:33.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:33 smithi151 ceph-mon[112233]: pgmap v1321: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:33.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:33 smithi153 ceph-mon[87858]: pgmap v1321: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:35.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:35 smithi151 ceph-mon[112233]: pgmap v1322: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:35.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:35 smithi153 ceph-mon[87858]: pgmap v1322: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:37.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:37 smithi151 ceph-mon[112233]: pgmap v1323: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:22:37.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:37 smithi153 ceph-mon[87858]: pgmap v1323: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-16T03:22:39.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:38 smithi153 ceph-mon[87858]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:22:39.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:38 smithi151 ceph-mon[112233]: from='mgr.34104 172.21.15.151:0/1178896399' entity='mgr.smithi151.uqqame' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T03:22:40.244 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:39 smithi151 ceph-mon[112233]: pgmap v1324: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:40.270 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:39 smithi153 ceph-mon[87858]: pgmap v1324: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:41.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:41 smithi151 ceph-mon[112233]: pgmap v1325: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:41.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:41 smithi153 ceph-mon[87858]: pgmap v1325: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:43.495 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:43 smithi151 ceph-mon[112233]: pgmap v1326: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:43.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:43 smithi153 ceph-mon[87858]: pgmap v1326: 65 pgs: 65 active+clean; 275 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-16T03:22:44.253 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-16T03:22:44.254 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T03:22:44.256 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-16T03:22:44.268 INFO:tasks.cephadm:Teardown begin 2024-04-16T03:22:44.268 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks/cephadm.py", line 2246, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T03:22:44.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T03:22:44.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T03:22:44.346 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-16T03:22:44.346 DEBUG:teuthology.orchestra.run.smithi151:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-16T03:22:44.368 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-16T03:22:44.405 INFO:tasks.cephadm:Stopping all daemons... 2024-04-16T03:22:44.405 INFO:tasks.cephadm.mon.smithi151:Stopping mon.smithi151... 2024-04-16T03:22:44.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151 2024-04-16T03:22:44.735 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:44 smithi151 systemd[1]: Stopping Ceph mon.smithi151 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T03:22:44.994 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:44 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:22:44.734+0000 7f7be6ae9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi151 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T03:22:44.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:44 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151[112229]: 2024-04-16T03:22:44.734+0000 7f7be6ae9700 -1 mon.smithi151@0(leader) e3 *** Got Signal Terminated *** 2024-04-16T03:22:44.995 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:44 smithi151 podman[167895]: 2024-04-16 03:22:44.775297952 +0000 UTC m=+0.174288787 container died 420c23cf907e8fb172d44e933f14c1ff0d7a82f1d9d6039e6a827d276613fc2b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=main-87dbfb6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T03:22:45.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:45 smithi151 podman[167895]: 2024-04-16 03:22:45.226097359 +0000 UTC m=+0.625088192 container cleanup 420c23cf907e8fb172d44e933f14c1ff0d7a82f1d9d6039e6a827d276613fc2b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-16T03:22:45.494 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Apr 16 03:22:45 smithi151 bash[167895]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi151 2024-04-16T03:22:45.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi151.service' 2024-04-16T03:22:45.770 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T03:22:45.770 INFO:tasks.cephadm.mon.smithi151:Stopped mon.smithi151 2024-04-16T03:22:45.770 INFO:tasks.cephadm.mon.smithi153:Stopping mon.smithi153... 2024-04-16T03:22:45.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi153 2024-04-16T03:22:46.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:45 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T03:22:46.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:46 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153[87854]: 2024-04-16T03:22:46.070+0000 7fc39f5d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T03:22:46.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:46 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153[87854]: 2024-04-16T03:22:46.070+0000 7fc39f5d9700 -1 mon.smithi153@1(peon) e3 *** Got Signal Terminated *** 2024-04-16T03:22:46.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:46 smithi153 podman[124814]: 2024-04-16 03:22:46.163299669 +0000 UTC m=+0.195018702 container died cc3cd0d47f4ee34c97e15c656cc323f6c0f5393a2db1ac0c529f79383c0d9c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-16T03:22:46.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:46 smithi153 podman[124814]: 2024-04-16 03:22:46.613498853 +0000 UTC m=+0.645217886 container cleanup cc3cd0d47f4ee34c97e15c656cc323f6c0f5393a2db1ac0c529f79383c0d9c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-16T03:22:46.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 16 03:22:46 smithi153 bash[124814]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-mon-smithi153 2024-04-16T03:22:47.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@mon.smithi153.service' 2024-04-16T03:22:47.095 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T03:22:47.095 INFO:tasks.cephadm.mon.smithi153:Stopped mon.smithi153 2024-04-16T03:22:47.095 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-16T03:22:47.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.0 2024-04-16T03:22:47.398 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:47 smithi151 systemd[1]: Stopping Ceph osd.0 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T03:22:47.744 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:47 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[123953]: 2024-04-16T03:22:47.397+0000 7f2a4628e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T03:22:47.744 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:47 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[123953]: 2024-04-16T03:22:47.397+0000 7f2a4628e700 -1 osd.0 86 *** Got signal Terminated *** 2024-04-16T03:22:47.744 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:47 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0[123953]: 2024-04-16T03:22:47.397+0000 7f2a4628e700 -1 osd.0 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T03:22:53.245 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:52 smithi151 podman[168029]: 2024-04-16 03:22:52.958431443 +0000 UTC m=+5.667125977 container died fee11db8c57b7e47cae3509050a716d3548bdde89e218d71f973bcb1e9610165 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-87dbfb6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-16T03:22:53.625 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:53 smithi151 podman[168029]: 2024-04-16 03:22:53.35645736 +0000 UTC m=+6.065151901 container cleanup fee11db8c57b7e47cae3509050a716d3548bdde89e218d71f973bcb1e9610165 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6) 2024-04-16T03:22:53.625 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:53 smithi151 bash[168029]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0 2024-04-16T03:22:53.923 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:53 smithi151 podman[168057]: 2024-04-16 03:22:53.624917814 +0000 UTC m=+0.668220911 container remove fee11db8c57b7e47cae3509050a716d3548bdde89e218d71f973bcb1e9610165 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-16T03:22:54.180 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:53 smithi151 podman[168147]: 2024-04-16 03:22:53.825980913 +0000 UTC m=+0.018525767 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T03:22:54.180 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:53 smithi151 podman[168147]: 2024-04-16 03:22:53.942528133 +0000 UTC m=+0.135073002 container create 4e1dc225d4ac7fca7c98b0fc3789fe9ada3294960305a6606d56f41bb160f440 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T03:22:54.180 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:54 smithi151 podman[168147]: 2024-04-16 03:22:54.179221992 +0000 UTC m=+0.371766829 container init 4e1dc225d4ac7fca7c98b0fc3789fe9ada3294960305a6606d56f41bb160f440 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-87dbfb6) 2024-04-16T03:22:54.483 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:54 smithi151 podman[168147]: 2024-04-16 03:22:54.183197836 +0000 UTC m=+0.375742672 container start 4e1dc225d4ac7fca7c98b0fc3789fe9ada3294960305a6606d56f41bb160f440 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-16T03:22:54.483 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:54 smithi151 podman[168147]: 2024-04-16 03:22:54.278655736 +0000 UTC m=+0.471200582 container attach 4e1dc225d4ac7fca7c98b0fc3789fe9ada3294960305a6606d56f41bb160f440 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T03:22:54.745 INFO:journalctl@ceph.osd.0.smithi151.stdout:Apr 16 03:22:54 smithi151 podman[168147]: 2024-04-16 03:22:54.483834463 +0000 UTC m=+0.676379301 container died 4e1dc225d4ac7fca7c98b0fc3789fe9ada3294960305a6606d56f41bb160f440 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-87dbfb6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-16T03:22:55.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.0.service' 2024-04-16T03:22:55.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T03:22:55.167 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-16T03:22:55.167 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-16T03:22:55.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.1 2024-04-16T03:22:55.494 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:22:55 smithi151 systemd[1]: Stopping Ceph osd.1 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T03:22:55.974 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:22:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[129371]: 2024-04-16T03:22:55.493+0000 7f3e3fc06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T03:22:55.975 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:22:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[129371]: 2024-04-16T03:22:55.493+0000 7f3e3fc06700 -1 osd.1 86 *** Got signal Terminated *** 2024-04-16T03:22:55.975 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:22:55 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1[129371]: 2024-04-16T03:22:55.493+0000 7f3e3fc06700 -1 osd.1 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T03:23:01.229 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:00 smithi151 podman[168253]: 2024-04-16 03:23:00.9520432 +0000 UTC m=+5.564354814 container died e3b70d49369dd6efd1f93ae266e6763d6f57c9df7f2f6a03442035ec441ec2bd (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409) 2024-04-16T03:23:01.494 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:01 smithi151 podman[168253]: 2024-04-16 03:23:01.327616814 +0000 UTC m=+5.939928418 container cleanup e3b70d49369dd6efd1f93ae266e6763d6f57c9df7f2f6a03442035ec441ec2bd (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6) 2024-04-16T03:23:01.495 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:01 smithi151 bash[168253]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1 2024-04-16T03:23:01.943 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:01 smithi151 podman[168268]: 2024-04-16 03:23:01.595497674 +0000 UTC m=+0.643782860 container remove e3b70d49369dd6efd1f93ae266e6763d6f57c9df7f2f6a03442035ec441ec2bd (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T03:23:02.207 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:01 smithi151 podman[168365]: 2024-04-16 03:23:01.846631709 +0000 UTC m=+0.018360527 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T03:23:02.207 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:01 smithi151 podman[168365]: 2024-04-16 03:23:01.963026164 +0000 UTC m=+0.134754958 container create 4b2e27a11117d540d550c518fea4e52299c54a1e9a973dea1eb86fb306154673 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-16T03:23:02.207 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:02 smithi151 podman[168365]: 2024-04-16 03:23:02.207222004 +0000 UTC m=+0.378950798 container init 4b2e27a11117d540d550c518fea4e52299c54a1e9a973dea1eb86fb306154673 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-16T03:23:02.495 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:02 smithi151 podman[168365]: 2024-04-16 03:23:02.210664804 +0000 UTC m=+0.382393598 container start 4b2e27a11117d540d550c518fea4e52299c54a1e9a973dea1eb86fb306154673 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-16T03:23:02.495 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:02 smithi151 podman[168365]: 2024-04-16 03:23:02.290556716 +0000 UTC m=+0.462285520 container attach 4b2e27a11117d540d550c518fea4e52299c54a1e9a973dea1eb86fb306154673 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-16T03:23:02.852 INFO:journalctl@ceph.osd.1.smithi151.stdout:Apr 16 03:23:02 smithi151 podman[168365]: 2024-04-16 03:23:02.510000872 +0000 UTC m=+0.681729671 container died 4b2e27a11117d540d550c518fea4e52299c54a1e9a973dea1eb86fb306154673 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-87dbfb6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-16T03:23:03.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.1.service' 2024-04-16T03:23:03.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T03:23:03.198 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-16T03:23:03.198 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-16T03:23:03.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.2 2024-04-16T03:23:03.495 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:03 smithi151 systemd[1]: Stopping Ceph osd.2 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T03:23:03.994 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:03 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[134925]: 2024-04-16T03:23:03.524+0000 7f651969d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T03:23:03.995 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:03 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[134925]: 2024-04-16T03:23:03.524+0000 7f651969d700 -1 osd.2 86 *** Got signal Terminated *** 2024-04-16T03:23:03.995 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:03 smithi151 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2[134925]: 2024-04-16T03:23:03.524+0000 7f651969d700 -1 osd.2 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T03:23:09.166 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:08 smithi151 podman[168472]: 2024-04-16 03:23:08.851477419 +0000 UTC m=+5.424571308 container died 4f046ee57e957df22e18ec5044171e29a94855786fc83303d7c36a01356f3fd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, RELEASE=main-87dbfb6, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-16T03:23:09.167 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:09 smithi151 podman[168472]: 2024-04-16 03:23:09.166038784 +0000 UTC m=+5.739132667 container cleanup 4f046ee57e957df22e18ec5044171e29a94855786fc83303d7c36a01356f3fd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-16T03:23:09.426 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:09 smithi151 bash[168472]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2 2024-04-16T03:23:09.744 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:09 smithi151 podman[168491]: 2024-04-16 03:23:09.425523571 +0000 UTC m=+0.570110493 container remove 4f046ee57e957df22e18ec5044171e29a94855786fc83303d7c36a01356f3fd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-16T03:23:10.121 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:09 smithi151 podman[168583]: 2024-04-16 03:23:09.745478456 +0000 UTC m=+0.018855535 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T03:23:10.121 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:09 smithi151 podman[168583]: 2024-04-16 03:23:09.876418671 +0000 UTC m=+0.149795792 container create 08d09359cdc9b57f428b7b14022eb6d116822adb174594206b097dd4264a8122 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409) 2024-04-16T03:23:10.121 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:10 smithi151 podman[168583]: 2024-04-16 03:23:10.120679482 +0000 UTC m=+0.394056561 container init 08d09359cdc9b57f428b7b14022eb6d116822adb174594206b097dd4264a8122 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-87dbfb6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T03:23:10.424 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:10 smithi151 podman[168583]: 2024-04-16 03:23:10.124300918 +0000 UTC m=+0.397677998 container start 08d09359cdc9b57f428b7b14022eb6d116822adb174594206b097dd4264a8122 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-04-16T03:23:10.424 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:10 smithi151 podman[168583]: 2024-04-16 03:23:10.212408815 +0000 UTC m=+0.485785912 container attach 08d09359cdc9b57f428b7b14022eb6d116822adb174594206b097dd4264a8122 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-87dbfb6, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T03:23:10.744 INFO:journalctl@ceph.osd.2.smithi151.stdout:Apr 16 03:23:10 smithi151 podman[168583]: 2024-04-16 03:23:10.423655954 +0000 UTC m=+0.697033034 container died 08d09359cdc9b57f428b7b14022eb6d116822adb174594206b097dd4264a8122 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-87dbfb6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-16T03:23:11.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.2.service' 2024-04-16T03:23:11.080 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T03:23:11.080 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-16T03:23:11.080 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-16T03:23:11.080 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.3 2024-04-16T03:23:11.400 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:11 smithi153 systemd[1]: Stopping Ceph osd.3 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T03:23:11.770 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:11 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T03:23:11.400+0000 7f0bd3c79700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T03:23:11.770 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:11 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T03:23:11.400+0000 7f0bd3c79700 -1 osd.3 86 *** Got signal Terminated *** 2024-04-16T03:23:11.771 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:11 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T03:23:11.400+0000 7f0bd3c79700 -1 osd.3 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T03:23:14.270 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:13 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T03:23:13.831+0000 7f0bcc265700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:47.827234+0000 front 2024-04-16T03:22:47.827175+0000 (oldest deadline 2024-04-16T03:23:13.727185+0000) 2024-04-16T03:23:15.270 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:14 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T03:23:14.827+0000 7f0bcc265700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:47.827234+0000 front 2024-04-16T03:22:47.827175+0000 (oldest deadline 2024-04-16T03:23:13.727185+0000) 2024-04-16T03:23:16.232 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:16 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:16.058+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:16.233 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:15 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3[97162]: 2024-04-16T03:23:15.830+0000 7f0bcc265700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:47.827234+0000 front 2024-04-16T03:22:47.827175+0000 (oldest deadline 2024-04-16T03:23:13.727185+0000) 2024-04-16T03:23:16.520 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:16 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:16.231+0000 7f439fd04700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:50.647887+0000 front 2024-04-16T03:22:50.647721+0000 (oldest deadline 2024-04-16T03:23:15.347601+0000) 2024-04-16T03:23:17.159 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:16 smithi153 podman[124946]: 2024-04-16 03:23:16.87163436 +0000 UTC m=+5.588645910 container died a731606ec80f696227676ac1863f0bd846bc24cf3648e284a4e127ec7b5a7786 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6) 2024-04-16T03:23:17.159 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:17 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:17.065+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:17.484 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:17 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:17.194+0000 7f439fd04700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:50.647887+0000 front 2024-04-16T03:22:50.647721+0000 (oldest deadline 2024-04-16T03:23:15.347601+0000) 2024-04-16T03:23:17.484 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:17 smithi153 podman[124946]: 2024-04-16 03:23:17.249642284 +0000 UTC m=+5.966653817 container cleanup a731606ec80f696227676ac1863f0bd846bc24cf3648e284a4e127ec7b5a7786 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-87dbfb6) 2024-04-16T03:23:17.484 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:17 smithi153 bash[124946]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3 2024-04-16T03:23:17.771 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:17 smithi153 podman[124965]: 2024-04-16 03:23:17.484211897 +0000 UTC m=+0.610900435 container remove a731606ec80f696227676ac1863f0bd846bc24cf3648e284a4e127ec7b5a7786 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-16T03:23:18.081 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:17 smithi153 podman[125057]: 2024-04-16 03:23:17.717901825 +0000 UTC m=+0.018063112 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T03:23:18.081 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:17 smithi153 podman[125057]: 2024-04-16 03:23:17.835043642 +0000 UTC m=+0.135204929 container create 09f2d9402e7704f665bb2a2aa68289b873bdbbef69a8373a126179d0457a0457 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-16T03:23:18.081 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:18 smithi153 podman[125057]: 2024-04-16 03:23:18.045927747 +0000 UTC m=+0.346089035 container init 09f2d9402e7704f665bb2a2aa68289b873bdbbef69a8373a126179d0457a0457 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-87dbfb6) 2024-04-16T03:23:18.081 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:18 smithi153 podman[125057]: 2024-04-16 03:23:18.049361108 +0000 UTC m=+0.349522404 container start 09f2d9402e7704f665bb2a2aa68289b873bdbbef69a8373a126179d0457a0457 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T03:23:18.343 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:18 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:18.196+0000 7f439fd04700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:50.647887+0000 front 2024-04-16T03:22:50.647721+0000 (oldest deadline 2024-04-16T03:23:15.347601+0000) 2024-04-16T03:23:18.343 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:18 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:18.079+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:18.343 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:18 smithi153 podman[125057]: 2024-04-16 03:23:18.121041026 +0000 UTC m=+0.421202313 container attach 09f2d9402e7704f665bb2a2aa68289b873bdbbef69a8373a126179d0457a0457 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-87dbfb6, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T03:23:18.657 INFO:journalctl@ceph.osd.3.smithi153.stdout:Apr 16 03:23:18 smithi153 podman[125057]: 2024-04-16 03:23:18.343500291 +0000 UTC m=+0.643661579 container died 09f2d9402e7704f665bb2a2aa68289b873bdbbef69a8373a126179d0457a0457 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T03:23:18.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.3.service' 2024-04-16T03:23:19.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T03:23:19.006 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-16T03:23:19.006 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-16T03:23:19.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.4 2024-04-16T03:23:19.226 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:19 smithi153 systemd[1]: Stopping Ceph osd.4 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T03:23:19.226 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:19 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:19.128+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:19.489 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:19 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:19.226+0000 7f439fd04700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:50.647887+0000 front 2024-04-16T03:22:50.647721+0000 (oldest deadline 2024-04-16T03:23:15.347601+0000) 2024-04-16T03:23:19.490 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:19 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:19.362+0000 7f43a3710700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T03:23:19.490 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:19 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:19.362+0000 7f43a3710700 -1 osd.4 86 *** Got signal Terminated *** 2024-04-16T03:23:19.490 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:19 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:19.362+0000 7f43a3710700 -1 osd.4 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T03:23:20.520 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:20 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:20.113+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:20.521 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:20 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:20.229+0000 7f439fd04700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:50.647887+0000 front 2024-04-16T03:22:50.647721+0000 (oldest deadline 2024-04-16T03:23:15.347601+0000) 2024-04-16T03:23:21.520 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:21 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:21.090+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:21.521 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:21 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:21.227+0000 7f439fd04700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:50.647887+0000 front 2024-04-16T03:22:50.647721+0000 (oldest deadline 2024-04-16T03:23:15.347601+0000) 2024-04-16T03:23:22.520 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:22 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:22.134+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:22.521 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:22 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:22.268+0000 7f439fd04700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:50.647887+0000 front 2024-04-16T03:22:50.647721+0000 (oldest deadline 2024-04-16T03:23:15.347601+0000) 2024-04-16T03:23:23.521 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:23 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:23.260+0000 7f439fd04700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:50.647887+0000 front 2024-04-16T03:22:50.647721+0000 (oldest deadline 2024-04-16T03:23:15.347601+0000) 2024-04-16T03:23:23.521 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:23 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:23.180+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:23.521 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:23 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:23.180+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:24.520 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:24 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4[102386]: 2024-04-16T03:23:24.217+0000 7f439fd04700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:50.647887+0000 front 2024-04-16T03:22:50.647721+0000 (oldest deadline 2024-04-16T03:23:15.347601+0000) 2024-04-16T03:23:24.521 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:24 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:24.213+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:24.521 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:24 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:24.213+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:25.095 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:24 smithi153 podman[125164]: 2024-04-16 03:23:24.779393248 +0000 UTC m=+5.521901500 container died 184b926e242cf992b5265d1eae5185442002b53423a5064e5efd247ebaf40a36 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-16T03:23:25.520 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:25 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:25.254+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:25.521 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:25 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:25.254+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:25.521 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:25 smithi153 podman[125164]: 2024-04-16 03:23:25.094821757 +0000 UTC m=+5.837330019 container cleanup 184b926e242cf992b5265d1eae5185442002b53423a5064e5efd247ebaf40a36 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409) 2024-04-16T03:23:25.521 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:25 smithi153 bash[125164]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4 2024-04-16T03:23:25.521 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:25 smithi153 podman[125192]: 2024-04-16 03:23:25.321365143 +0000 UTC m=+0.542099851 container remove 184b926e242cf992b5265d1eae5185442002b53423a5064e5efd247ebaf40a36 (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-87dbfb6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-04-16T03:23:26.020 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:25 smithi153 podman[125284]: 2024-04-16 03:23:25.67204219 +0000 UTC m=+0.018190985 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T03:23:26.020 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:25 smithi153 podman[125284]: 2024-04-16 03:23:25.78887706 +0000 UTC m=+0.135025847 container create 84abf1beabf397e528e6351c40f9138ae7b633edf3a386c293f48dabd984a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-16T03:23:26.020 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:26 smithi153 podman[125284]: 2024-04-16 03:23:26.016835447 +0000 UTC m=+0.362984234 container init 84abf1beabf397e528e6351c40f9138ae7b633edf3a386c293f48dabd984a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-16T03:23:26.020 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:26 smithi153 podman[125284]: 2024-04-16 03:23:26.02009134 +0000 UTC m=+0.366240131 container start 84abf1beabf397e528e6351c40f9138ae7b633edf3a386c293f48dabd984a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-16T03:23:26.270 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:26 smithi153 podman[125284]: 2024-04-16 03:23:26.116648818 +0000 UTC m=+0.462797625 container attach 84abf1beabf397e528e6351c40f9138ae7b633edf3a386c293f48dabd984a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-87dbfb6, org.label-schema.build-date=20240409, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-16T03:23:26.271 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:26 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:26.227+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:26.271 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:26 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:26.227+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:26.612 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 16 03:23:26 smithi153 podman[125284]: 2024-04-16 03:23:26.31021335 +0000 UTC m=+0.656362142 container died 84abf1beabf397e528e6351c40f9138ae7b633edf3a386c293f48dabd984a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, RELEASE=main-87dbfb6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T03:23:26.956 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.4.service' 2024-04-16T03:23:27.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T03:23:27.000 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-16T03:23:27.000 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-16T03:23:27.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.5 2024-04-16T03:23:27.266 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:27 smithi153 systemd[1]: Stopping Ceph osd.5 for 80d1e958-fb98-11ee-bc8f-c7b262605968... 2024-04-16T03:23:27.520 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:27 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:27.266+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:27.521 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:27 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:27.266+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:27.521 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:27 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:27.365+0000 7fb96fe97700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T03:23:27.521 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:27 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:27.365+0000 7fb96fe97700 -1 osd.5 86 *** Got signal Terminated *** 2024-04-16T03:23:27.521 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:27 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:27.365+0000 7fb96fe97700 -1 osd.5 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T03:23:28.520 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:28 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:28.255+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:28.520 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:28 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:28.255+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:29.422 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:29 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:29.300+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:29.422 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:29 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:29.300+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:30.536 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:30 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:30.277+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:30.536 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:30 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:30.277+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:31.771 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:31 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:31.288+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:31.772 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:31 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:31.288+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:32.771 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:32 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:32.300+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-04-16T03:22:51.499330+0000 front 2024-04-16T03:22:51.499422+0000 (oldest deadline 2024-04-16T03:23:15.599380+0000) 2024-04-16T03:23:32.771 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:32 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:32.300+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-04-16T03:22:59.600709+0000 front 2024-04-16T03:22:59.600743+0000 (oldest deadline 2024-04-16T03:23:23.100637+0000) 2024-04-16T03:23:32.771 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:32 smithi153 ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5[107744]: 2024-04-16T03:23:32.300+0000 7fb968483700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-04-16T03:23:08.401547+0000 front 2024-04-16T03:23:08.401528+0000 (oldest deadline 2024-04-16T03:23:31.901503+0000) 2024-04-16T03:23:33.270 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:32 smithi153 podman[125389]: 2024-04-16 03:23:32.819024467 +0000 UTC m=+5.593517160 container died a3f762d46f2455afcd55ec640107288391d9e8816e9f51bb3dc212818b663b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-87dbfb6, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-16T03:23:33.642 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:33 smithi153 podman[125389]: 2024-04-16 03:23:33.373614067 +0000 UTC m=+6.148106745 container cleanup a3f762d46f2455afcd55ec640107288391d9e8816e9f51bb3dc212818b663b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T03:23:33.642 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:33 smithi153 bash[125389]: ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5 2024-04-16T03:23:33.951 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:33 smithi153 podman[125410]: 2024-04-16 03:23:33.642323094 +0000 UTC m=+0.825195646 container remove a3f762d46f2455afcd55ec640107288391d9e8816e9f51bb3dc212818b663b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-16T03:23:34.237 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:33 smithi153 podman[125502]: 2024-04-16 03:23:33.95132125 +0000 UTC m=+0.110039851 container create b83a1df16001be0b95f486f8151ca234eb8e481a0f06d44583aa56cb2218c33b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-87dbfb6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-16T03:23:34.237 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:33 smithi153 podman[125502]: 2024-04-16 03:23:33.859571947 +0000 UTC m=+0.018290570 image pull a452a57266c718c37049de4c2da9a5e566bb7b9c6db62908ada4c8b0e854997b quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f 2024-04-16T03:23:34.238 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:34 smithi153 podman[125502]: 2024-04-16 03:23:34.153910695 +0000 UTC m=+0.312629298 container init b83a1df16001be0b95f486f8151ca234eb8e481a0f06d44583aa56cb2218c33b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, RELEASE=main-87dbfb6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-16T03:23:34.238 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:34 smithi153 podman[125502]: 2024-04-16 03:23:34.157151185 +0000 UTC m=+0.315869793 container start b83a1df16001be0b95f486f8151ca234eb8e481a0f06d44583aa56cb2218c33b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-16T03:23:34.238 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:34 smithi153 podman[125502]: 2024-04-16 03:23:34.23721727 +0000 UTC m=+0.395935887 container attach b83a1df16001be0b95f486f8151ca234eb8e481a0f06d44583aa56cb2218c33b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=main-87dbfb6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-16T03:23:34.520 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 16 03:23:34 smithi153 podman[125502]: 2024-04-16 03:23:34.451816687 +0000 UTC m=+0.610535295 container died b83a1df16001be0b95f486f8151ca234eb8e481a0f06d44583aa56cb2218c33b (image=quay.ceph.io/ceph-ci/ceph@sha256:386dcbdb0b8a78b880705c56a17f715946e178c298e478ab4b58fddb8072ae6f, name=ceph-80d1e958-fb98-11ee-bc8f-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-87dbfb6, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T03:23:35.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80d1e958-fb98-11ee-bc8f-c7b262605968@osd.5.service' 2024-04-16T03:23:35.207 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T03:23:35.207 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-16T03:23:35.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-16T03:23:35.369 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T03:23:39.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi151.stderr:ceph-fuse[77179]: fuse finished with error 0 and tester_r 0 2024-04-16T03:23:58.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-16T03:23:58.526 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T03:24:12.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T03:24:12.847 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T03:24:12.879 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-16T03:24:12.882 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416/remote/smithi151/crash 2024-04-16T03:24:12.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash -- . 2024-04-16T03:24:12.920 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-16T03:24:12.920 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2024-04-16T03:24:12.922 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416/remote/smithi153/crash 2024-04-16T03:24:12.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash -- . 2024-04-16T03:24:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-16T03:24:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-04-16T03:24:12.952 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-16T03:24:12.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-16T03:24:13.002 INFO:teuthology.orchestra.run.smithi151.stdout:2024-04-16T02:50:00.000124+0000 mon.smithi151 (mon.0) 719 : cluster [ERR] Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-16T03:24:13.002 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-16T03:24:13.002 INFO:tasks.cephadm:Compressing logs... 2024-04-16T03:24:13.002 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T03:24:13.044 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T03:24:13.071 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-16T03:24:13.071 INFO:teuthology.orchestra.run.smithi151.stderr:: No such file or directory 2024-04-16T03:24:13.071 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-16T03:24:13.072 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-16T03:24:13.072 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi151.bdpnxq.log 2024-04-16T03:24:13.079 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-16T03:24:13.079 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-04-16T03:24:13.079 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mon.smithi153.log 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.1.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-client.ceph-exporter.smithi153.log 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi153.mqbgav.log 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mon.smithi153.log: /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-client.ceph-exporter.smithi153.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi153.mqbgav.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.4.log 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mgr.smithi153.esbtzf.log 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.4.log: 92.3% -- replaced with /var/log/ceph/cephadm.log.gzgzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.5.log 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mgr.smithi153.esbtzf.log: 91.5% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-16T03:24:13.080 INFO:teuthology.orchestra.run.smithi153.stderr: 94.6% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-client.ceph-exporter.smithi153.log.gz 2024-04-16T03:24:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.1.log 2024-04-16T03:24:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-client.ceph-exporter.smithi151.log 2024-04-16T03:24:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi151.bdpnxq.log: /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.0.log 2024-04-16T03:24:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-client.ceph-exporter.smithi151.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi151.slrqog.log 2024-04-16T03:24:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-16T03:24:13.082 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi151.slrqog.log: 94.6% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-client.ceph-exporter.smithi151.log.gz 2024-04-16T03:24:13.083 INFO:teuthology.orchestra.run.smithi151.stderr: 91.5% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-16T03:24:13.083 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.log 2024-04-16T03:24:13.084 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.2.log 2024-04-16T03:24:13.085 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.log: 85.9% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-16T03:24:13.085 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mgr.smithi151.uqqame.log 2024-04-16T03:24:13.087 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.3.log 2024-04-16T03:24:13.087 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-16T03:24:13.087 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.log 2024-04-16T03:24:13.089 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.2.log: 89.6% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-16T03:24:13.089 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-16T03:24:13.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.cephadm.log: 85.9% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-16T03:24:13.090 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-16T03:24:13.095 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.log: 89.6% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-16T03:24:13.095 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi153.ogdcjk.log 2024-04-16T03:24:13.099 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mgr.smithi151.uqqame.log: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-16T03:24:13.100 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mon.smithi151.log 2024-04-16T03:24:13.119 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-volume.log: /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mon.smithi151.log: 91.0% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-16T03:24:13.127 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-volume.log: /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi153.ogdcjk.log: 91.0% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-16T03:24:13.138 INFO:teuthology.orchestra.run.smithi153.stderr: 90.0% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mgr.smithi153.esbtzf.log.gz 2024-04-16T03:24:13.661 INFO:teuthology.orchestra.run.smithi153.stderr: 92.6% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mon.smithi153.log.gz 2024-04-16T03:24:13.759 INFO:teuthology.orchestra.run.smithi151.stderr: 89.6% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mgr.smithi151.uqqame.log.gz 2024-04-16T03:24:14.937 INFO:teuthology.orchestra.run.smithi151.stderr: 89.9% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mon.smithi151.log.gz 2024-04-16T03:24:20.001 INFO:teuthology.orchestra.run.smithi151.stderr: 93.4% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.2.log.gz 2024-04-16T03:24:20.059 INFO:teuthology.orchestra.run.smithi153.stderr: 93.4% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.4.log.gz 2024-04-16T03:24:21.061 INFO:teuthology.orchestra.run.smithi151.stderr: 93.4% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.0.log.gz 2024-04-16T03:24:21.284 INFO:teuthology.orchestra.run.smithi153.stderr: 93.5% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.3.log.gz 2024-04-16T03:24:21.563 INFO:teuthology.orchestra.run.smithi151.stderr: 93.5% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.1.log.gz 2024-04-16T03:24:22.043 INFO:teuthology.orchestra.run.smithi153.stderr: 93.6% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-osd.5.log.gz 2024-04-16T03:24:22.538 INFO:teuthology.orchestra.run.smithi153.stderr: 95.0% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi153.ogdcjk.log.gz 2024-04-16T03:24:23.365 INFO:teuthology.orchestra.run.smithi153.stderr: 95.0% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi153.mqbgav.log.gz 2024-04-16T03:24:27.926 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-16T03:24:27.926 INFO:teuthology.orchestra.run.smithi153.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-04-16T03:24:27.927 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-04-16T03:24:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m14.868s 2024-04-16T03:24:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m57.565s 2024-04-16T03:24:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m1.712s 2024-04-16T03:24:28.384 INFO:teuthology.orchestra.run.smithi151.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-16T03:25:02.200 INFO:teuthology.orchestra.run.smithi151.stderr: 93.4% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi151.bdpnxq.log.gz 2024-04-16T03:25:05.122 INFO:teuthology.orchestra.run.smithi151.stderr: 93.5% -- replaced with /var/log/ceph/80d1e958-fb98-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi151.slrqog.log.gz 2024-04-16T03:25:05.125 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-04-16T03:25:05.126 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m52.067s 2024-04-16T03:25:05.126 INFO:teuthology.orchestra.run.smithi151.stderr:user 2m17.703s 2024-04-16T03:25:05.126 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m4.493s 2024-04-16T03:25:05.126 INFO:tasks.cephadm:Archiving logs... 2024-04-16T03:25:05.127 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416/remote/smithi151/log 2024-04-16T03:25:05.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T03:25:22.813 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416/remote/smithi153/log 2024-04-16T03:25:22.814 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T03:25:29.274 INFO:tasks.cephadm:Removing cluster... 2024-04-16T03:25:29.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 --force 2024-04-16T03:25:29.434 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T03:25:30.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80d1e958-fb98-11ee-bc8f-c7b262605968 --force 2024-04-16T03:25:30.283 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: 80d1e958-fb98-11ee-bc8f-c7b262605968 2024-04-16T03:25:30.594 INFO:tasks.cephadm:Removing cephadm ... 2024-04-16T03:25:30.595 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-16T03:25:30.612 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-16T03:25:30.628 INFO:tasks.cephadm:Teardown complete 2024-04-16T03:25:30.629 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T03:25:30.641 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T03:25:30.642 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T03:25:30.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T03:25:30.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T03:25:30.750 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T03:25:30.761 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T03:25:30.761 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T03:25:30.764 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T03:25:30.776 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-04-16T03:25:30.779 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-16T03:25:30.781 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T03:25:30.781 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-04-16T03:25:30.781 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 +447us[ +479us] +/- 40ms 2024-04-16T03:25:30.781 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 15 -2472us[-2440us] +/- 54ms 2024-04-16T03:25:30.781 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +2593us[+2626us] +/- 70ms 2024-04-16T03:25:30.781 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T03:25:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T03:25:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-16T03:25:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 14 +528us[ +528us] +/- 40ms 2024-04-16T03:25:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -2381us[-2418us] +/- 54ms 2024-04-16T03:25:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +2689us[+2651us] +/- 70ms 2024-04-16T03:25:30.784 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T03:25:30.785 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T03:25:30.795 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T03:25:30.796 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T03:25:30.807 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T03:25:30.910 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T03:25:30.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T03:25:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T03:25:31.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T03:25:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T03:25:31.056 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T03:25:31.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T03:25:31.077 INFO:teuthology.task.internal:Duration was 4037.021917 seconds 2024-04-16T03:25:31.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T03:25:31.088 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T03:25:31.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T03:25:31.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T03:25:31.134 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T03:25:31.142 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T03:25:31.537 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T03:25:31.538 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T03:25:31.538 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T03:25:31.558 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-04-16T03:25:31.559 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T03:25:31.586 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T03:25:31.586 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T03:25:31.602 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T03:25:31.658 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T03:25:31.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T03:25:31.673 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T03:25:32.868 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T03:25:32.880 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T03:25:32.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T03:25:32.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T03:25:32.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T03:25:32.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T03:25:32.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T03:25:32.982 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-04-16T03:25:33.014 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-04-16T03:25:33.318 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T03:25:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T03:25:33.374 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T03:25:33.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T03:25:33.393 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T03:25:33.405 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T03:25:33.406 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416/remote/smithi151 2024-04-16T03:25:33.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T03:25:33.483 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416/remote/smithi153 2024-04-16T03:25:33.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T03:25:33.531 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T03:25:33.531 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T03:25:33.533 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T03:25:33.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T03:25:33.594 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T03:25:33.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T03:25:33.605 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T03:25:33.605 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T03:25:33.608 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T03:25:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 03:25 /home/ubuntu/cephtest 2024-04-16T03:25:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: 552891 4 d--------- 2 ubuntu ubuntu 4096 Apr 16 02:33 /home/ubuntu/cephtest/mnt.0 2024-04-16T03:25:33.622 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-16T03:25:33.622 INFO:teuthology.orchestra.run.smithi151.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T03:25:33.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T03:25:33.624 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_3f7db60e5ba4c4d053b594b7ec58703d2833c55b/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T03:25:33.625 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T03:25:33.636 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T03:25:33.648 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T03:25:33.649 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T03:25:33.675 DEBUG:teuthology.nuke:shortname: smithi151 2024-04-16T03:25:33.675 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T03:25:33.700 DEBUG:teuthology.nuke:shortname: smithi153 2024-04-16T03:25:33.700 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T03:25:33.704 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 02:09:17.255584', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDcdvOwGEwiTGWVVq3/lH/SMtW1QGOfmchUl+HmD/ASByfF6XNvi0iddJC9FioCIYXYIjr7IqkS4umjtvk+qDJQ='} 2024-04-16T03:25:33.729 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 02:09:17.257462', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOifDiy5cEQMY0nTfO0+g7qJXRNVwrokpMt23C0FmhTNikwXD+9wRdztbEptjIYY2T0+SX93q7RVH8dcfYwC918='} 2024-04-16T03:25:33.757 INFO:teuthology.orchestra.console.smithi151:Power off 2024-04-16T03:25:33.757 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T03:25:33.788 INFO:teuthology.orchestra.console.smithi153:Power off 2024-04-16T03:25:33.788 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T03:25:33.876 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-04-16T03:25:33.877 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T03:25:33.884 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-04-16T03:25:33.884 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T03:25:33.899 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-04-16T03:25:33.907 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-16T03:25:37.901 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T03:25:37.908 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T03:25:38.010 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-04-16T03:25:38.017 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-16T03:25:42.015 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T03:25:42.023 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T03:25:42.125 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-04-16T03:25:42.131 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-16T03:25:42.227 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-04-16T03:25:46.136 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T03:25:46.246 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-04-16T03:25:46.347 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-04-16T03:25:46.493 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4037.0219168663025 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-04-16T03:25:46.494 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T03:25:46.580 INFO:teuthology.run:FAIL